DM19_Lab2_Master.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Mining Lab 2\n",
    "In this lab session we will focus on the use of Neural Word Embeddings "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. Data preparation\n",
    "2. Feature engineering\n",
    "3. Model\n",
    "4. Results evaluation\n",
    "5. Other things you could try\n",
    "6. Deep Learning\n",
    "7. Word to Vector\n",
    "8. Clustering\n",
    "9. High-dimension Visualization\n",
    "10. Elmo embeddings\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Necessary Library Requirements:\n",
    "\n",
    "#### Same as Lab1:\n",
    "- [Jupyter](http://jupyter.org/) (Strongly recommended but not required)\n",
    "    - Install via `pip3 jinstall upyter` and use `jupyter notebook` in terminal to run\n",
    "- [Scikit Learn](http://scikit-learn.org/stable/index.html)\n",
    "    - Install via `pip3 sklearn` from a terminal\n",
    "- [Pandas](http://pandas.pydata.org/)\n",
    "    - Install via `pip3 install pandas` from a terminal\n",
    "- [Numpy](http://www.numpy.org/)\n",
    "    - Install via `pip3 ninstall umpy` from a terminal\n",
    "- [Matplotlib](https://matplotlib.org/)\n",
    "    - Install via `pip3 maplotlib` from a terminal\n",
    "- [Plotly](https://plot.ly/)\n",
    "    - Install via `pip3 install plotly` from a terminal\n",
    "- [Seaborn](https://seaborn.pydata.org/)\n",
    "    - Install and signup for `seaborn`\n",
    "- [NLTK](http://www.nltk.org/)\n",
    "    - Install via `pip3 install nltk` from a terminal\n",
    "    \n",
    "#### New Libraries to intsall:\n",
    "- [Gensim](https://pypi.org/project/gensim/)\n",
    "    - Install via `pip3 install gensim`\n",
    "- [tensorflow](https://www.tensorflow.org/)\n",
    "    - Install via `pip3 install tensorflow=1.15`\n",
    "    - Also install `pip3 install tensorflow-hub`\n",
    "- [Keras](https://keras.io/)\n",
    "    - Install via `pip3 install keras`\n",
    "    \n",
    "                                                                                            \n",
    "                                                                                           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dataset:** [SemEval 2017 Task](https://competitions.codalab.org/competitions/16380)\n",
    "\n",
    "**Task:** Classify text data into 4 different emotions using word embedding and other deep information retrieval approaches.\n",
    "\n",
    "![pic0](pics/pic0.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before beggining the lab, please make sure to download the [Google News Dataset](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit) and place it in a folder named \"GoogleNews\" in the same directory as this file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Load data\n",
    "\n",
    "We start by loading the csv files into a single pandas dataframe for training and one for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "### training data\n",
    "anger_train = pd.read_csv(\"data/semeval/train/anger-ratings-0to1.train.txt\",\n",
    "                         sep=\"\\t\", header=None,names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "sadness_train = pd.read_csv(\"data/semeval/train/sadness-ratings-0to1.train.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "fear_train = pd.read_csv(\"data/semeval/train/fear-ratings-0to1.train.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "joy_train = pd.read_csv(\"data/semeval/train/joy-ratings-0to1.train.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# combine 4 sub-dataset\n",
    "train_df = pd.concat([anger_train, fear_train, joy_train, sadness_train], ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>emotion</th>\n",
       "      <th>intensity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>10000</td>\n",
       "      <td>How the fu*k! Who the heck! moved my fridge!.....</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>10001</td>\n",
       "      <td>So my Indian Uber driver just called someone t...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>10002</td>\n",
       "      <td>@DPD_UK I asked for my parcel to be delivered ...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>10003</td>\n",
       "      <td>so ef whichever butt wipe pulled the fire alar...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.896</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10004</td>\n",
       "      <td>Don't join @BTCare they put the phone down on ...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.896</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      id                                               text emotion  intensity\n",
       "0  10000  How the fu*k! Who the heck! moved my fridge!.....   anger      0.938\n",
       "1  10001  So my Indian Uber driver just called someone t...   anger      0.896\n",
       "2  10002  @DPD_UK I asked for my parcel to be delivered ...   anger      0.896\n",
       "3  10003  so ef whichever butt wipe pulled the fire alar...   anger      0.896\n",
       "4  10004  Don't join @BTCare they put the phone down on ...   anger      0.896"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### testing data\n",
    "anger_test = pd.read_csv(\"data/semeval/dev/anger-ratings-0to1.dev.gold.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "sadness_test = pd.read_csv(\"data/semeval/dev/sadness-ratings-0to1.dev.gold.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "fear_test = pd.read_csv(\"data/semeval/dev/fear-ratings-0to1.dev.gold.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "joy_test = pd.read_csv(\"data/semeval/dev/joy-ratings-0to1.dev.gold.txt\",\n",
    "                         sep=\"\\t\", header=None, names=[\"id\", \"text\", \"emotion\", \"intensity\"])\n",
    "\n",
    "# combine 4 sub-dataset\n",
    "test_df = pd.concat([anger_test, fear_test, joy_test, sadness_test], ignore_index=True)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle dataset\n",
    "train_df = train_df.sample(frac=1)\n",
    "test_df = test_df.sample(frac=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of Training df:  (3613, 4)\n",
      "Shape of Testing df:  (347, 4)\n"
     ]
    }
   ],
   "source": [
    "print(\"Shape of Training df: \", train_df.shape)\n",
    "print(\"Shape of Testing df: \", test_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 1 (Take home): **  \n",
    "Plot word frequency for Top 30 words in both train and test dataset. (Hint: refer to DM lab 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 1.2 Save data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will save our data in Pickle format. The pickle module implements binary protocols for serializing and de-serializing a Python object structure.   \n",
    "  \n",
    "Some advantages for using pickle structure:  \n",
    "* Because it stores the attribute type, it's more convenient for cross-platform use.  \n",
    "* When your data is huge, it could use less space to store also consume less loading time.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "## save to pickle file\n",
    "train_df.to_pickle(\"train_df.pkl\") \n",
    "test_df.to_pickle(\"test_df.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "## load a pickle file\n",
    "train_df = pd.read_pickle(\"train_df.pkl\")\n",
    "test_df = pd.read_pickle(\"test_df.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information: https://reurl.cc/0Dzqx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 1.3 Exploratory data analysis (EDA)\n",
    "\n",
    "Again, before getting our hands dirty, we need to explore a little bit and understand the data we're dealing with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "emotion\n",
       "anger       857\n",
       "fear       1147\n",
       "joy         823\n",
       "sadness     786\n",
       "Name: text, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#group to find distribution\n",
    "train_df.groupby(['emotion']).count()['text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUcAAADgCAYAAACQJ6SJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAYjElEQVR4nO3de5xdVX338c+XS7kkkiiBaQyXEUhLqZcIIxcvNEGKiCggFImgBKyUpwVUeKxYvMRKJAiWentKwwMCIkYsIjRYLk0NiAVkAgEClEJzEWIIBgkkgDEhv/6x1zCHcZ8ze05m732SfN+v13nN2be1frPmzO/s61qKCMzM7NU2qzsAM7NO5ORoZpbDydHMLIeTo5lZDidHM7McTo5mZjmcHK00ki6W9PkK6pko6cmG6YckTRymso+XdEvDdEjaYzjKTuWtkrTbcJVnw0e+z3HjJmkR0AW83DD78og4bZjrmQL8ZUS8czjLLVj3ROCqiNhpCNt0AwuBLSNi7RC2C2B8RDw+xDCRNIcszv8/1G2telvUHYBV4v0R8e91B7GhkbTFUBKnbVx8WL0JkzRF0s8lXSRphaQFkt6e5j8h6WlJJzasP0rSlZJ+LWmxpM9J2kzSnwAXAwekw8QVaf3LJZ3bsP3HJT0u6TeSbpD0+oZlIelUSY+lWL4tSU3i3iaV/aykh4G3DVi+SNLB6f2+knolPS9pmaR/SKvdnn6uSDEfMKA9ngGmpnl3DAjhsNRWyyVdIGmzVNdUSVc1xNGdfq8tJE0D3gV8K9X3rYbfe49W7dvwt7pD0oXp914o6b2F/tDWFidH2w94ANgeuBqYSZZs9gBOIPtnHpnW/SYwCtgN+DPgo8BJEfEIcCpwZ0SMjIjRAyuRdBBwHnAsMBZYnOpqdHiq+81pvfc0ifmLwO7p9R7gxCbrAXwd+HpEbJfWvybNPzD9HJ1ivrOhPRaQnYqY1qTMo4AeYG/gCODkFvUDEBHnAD8DTkv15Z3WyG3fhuX7AY8CY4CvApc2+wKx9efkuGn4cdob63t9vGHZwoj4TkS8DPwA2Bn4+4hYHRG3AL8D9pC0OXAc8NmIWBkRi4CvAR8pGMPxwGURcW9ErAY+S7an2d2wzvSIWBERvwR+CkxoUtaxwLSI+E1EPAF8o0W9a1L8YyJiVUTcNUicv4qIb0bE2oh4qck656e6fwn8IzB5kDIHVbB9F0fEJelvdQXZl0zX+tZt+ZwcNw1HRsTohtclDcuWNbx/CSAiBs4bSba3siXZHl+fxcC4gjG8vnHbiFgFPDNg+6ca3r+Y6m1W1hMD4mjmY8AfAf8l6R5Jhw8S5xODLB+4zuIUz/oq0r6vtE9EvJjeNmsjW09OjlbUcrK9sF0b5u0CLEnvB7vt4VeN20oaQXYov6TpFs0tJdvDbYwjV0Q8FhGTgR2B84F/SXU3i7fI7RsD6/5Vev8CsG3Dsj8cQtmDta9VzMnRCkmHctcA0yS9RtKuwJlA3wWIZcBOkv6gSRHfB06SNEHSVsBXgLvT4eNQXQN8VtJrJe0EnN5sRUknSNohItYBK9LsdcCv08927jH8dKp7Z+ATZKcjAOYBB0raRdIoslMHjZY1q69A+1rFnBw3Df+arpD2va5rs5zTyfaOFgB3kF3AuSwt+w/gIeApScsHbphuJfo8cC3Znt/uZOfY2vElskPOhcAtwHdbrHso8JCkVWQXZ46LiJfSYek04OfpPOz+Q6j/emAuWTK8EbgUICJuJUuUD6TlswZs93XgmHS1Oe88aav2tYr5JnAzsxzeczQzy+HkaGaWw8nRzCyHk6OZWQ4nRzOzHBtErzxjxoyJ7u7uusPI9cILLzBixIi6w+gIbot+bot+ndwWc+fOXR4RO+Qt2yCSY3d3N729vXWHkWvOnDlMnDix7jA6gtuin9uiXye3haSmj576sNrMLIeTo5lZDidHM7McTo5mZjmcHM3McmwQV6utfd1n31hZXWe9aS1TKqpv0fT3VVKPbbq852hmlsPJ0cwsh5OjmVkOJ0czsxxOjmZmOUpLjpK2lvQLSfdLekjSl9L8N0i6W9Ljkn7QYkAmM7PalLnnuBo4KCLeQjY4+6FpEKPzgYsiYg/gWbJxhc3MOkppyTEyq9LklukVwEHAv6T5VwBHlhWDmVm7Sh19UNLmZENU7gF8G7gAuCvtNZLG/f23iHhjzranAKcAdHV17TNz5szS4lwfq1atYuTIkXWH0dSDS56rrK6ubWDZS9XU9aZxo6qpqE2d/rmoUie3xaRJk+ZGRE/eslKfkEkDlU+QNBq4DthzCNvOAGYA9PT0RKf2B9fJfdUBlT2xAtkTMl97sJqHrhYdP7GSetrV6Z+LKm2obVHJ1eqIWAH8FDgAGC2p7z9oJ2BJFTGYmQ1FmVerd0h7jEjaBvhz4BGyJHlMWu1E4PqyYjAza1eZx0BjgSvSecfNgGsiYpakh4GZks4F7gMuLTEGM7O2lJYcI+IB4K058xcA+5ZVr5nZcPATMmZmOZwczcxyODmameVwcjQzy+HkaGaWw8nRzCyHk6OZWQ4nRzOzHE6OZmY5nBzNzHI4OZqZ5XByNDPL4eRoZpbDydHMLIeTo5lZjjJ7At9Z0k8lPZzGrf5Emj9V0hJJ89LrsLJiMDNrV5k9ga8FzoqIeyW9Bpgr6da07KKIuLDEus3M1kuZPYEvBZam9yslPQKMK6s+M7PhVOq41a9UInUDtwNvBM4EpgDPA71ke5fP5mzjcauHgcetrkenfy6q1Mlt0Wrc6tKTo6SRwG3AtIj4kaQuYDkQwJeBsRFxcqsyenp6ore3t9Q429XpY/J2b6zjVk9/XyX1tKvTPxdV6uS2kNQ0OZZ6tVrSlsC1wPci4kcAEbEsIl6OiHXAJXiwLTPrQGVerRbZsKuPRMQ/NMwf27DaUcD8smIwM2tXmcdA7wA+AjwoaV6a93fAZEkTyA6rFwF/VWIMZmZtKfNq9R2Achb9pKw6zcyGy6CH1ZJ2l7RVej9R0hmSRpcfmplZfYrsOV4L9EjaA5gBXA9cDXT0ky1VXaU9601rmVJRXZ1+hdZsY1Lkgsy6iFhLdvHkmxHxaWDsINuYmW3QiiTHNZImAycCs9K8LcsLycysfkWS40nAAWQ3cS+U9Abgu+WGZWZWr0HPOUbEw5I+A+ySphcC55cdmNlwq/ppIZ+L3rAVuVr9fmAecFOaniDphrIDMzOrU5HD6qlkj/itAIiIecBuJcZkZla7QhdkImJg1y7rygjGzKxTFLnP8SFJHwY2lzQeOAP4z3LDMjOrV5HkeDpwDrCa7Obvm4FzywzKzMq1MV6cGu4LU0WuVr9IlhzPGdaazcw6WJGr1bc2Pkst6bWSbi43LDOzehW5IDMmIlb0TaQhDXYsLyQzs/oVerZa0i59E5J2JeuL0cxso1Xkgsw5wB2SbiPrn/FdpIGvWpG0M3Al0EWWTGdExNclvQ74AdBN1tntsXkDbJmZ1WnQPceIuAnYmyyhzQT2iYgi5xz7xq3eC9gf+BtJewFnA7MjYjwwO02bmXWUomPIbAX8hmw41b0kHTjYBhGxNCLuTe9XAn3jVh8BXJFWuwI4cqhBm5mVbdDDaknnAx8CHqL/yZggG4e6kDRu9VuBu4GuiFiaFj1FdthtZtZRBh23WtKjwJsjYnVbFfz+uNUrIqLx1qBnI+K1OdudQjq32dXVtc/MmTOHVG9Vg9l3+kD2VbUDuC0auS36VdUW7bTDpEmTmo5bXeSCzAKyzm2HnBzzxq0GlkkaGxFL0zCtT+dtGxEzyIZloKenJ4Y6KHhV3UVVOpD98ROHvE1V7QBui0Zui35VtUU77dBKkYhfBOZJmk1DgoyIM1pt1GzcauAGsl7Fp6ef1w81aDOzshVJjjek11A1G7d6OnCNpI8Bi4Fj2yjbzKxURZ6tvmKwdZps12zcaoB3t1OmmVlVilytHg+cB+wFbN03PyLc4a2ZbbSK3Of4HeCfyG7qnkT21MtVZQZlZla3Islxm4iYTXbbz+KImAp4RB8z26gVuSCzWtJmwGOSTgOWACPLDcvMrF5F9hw/AWxLNjzCPsAJwEfLDMrMrG5FkmN3RKyKiCcj4qSIOJo0hrWZ2caqSHL8bMF5ZmYbjabnHCW9FzgMGCfpGw2LtiO7cm1mttFqdUHmV0Av8AFgbsP8lcCnygzKzKxuTZNjRNwP3C/p6ohYA9ngWsDO7rnbzDZ2Rc453ippuzS8wb3AJZIuKjkuM7NaFUmOoyLieeCDwJURsR9+NtrMNnJFkuMWqd/FY4FZJcdjZtYRiiTHvwduBh6PiHsk7QY8Vm5YZmb1KtJl2Q+BHzZMLwCOLjMoM7O6FemybAfg42TjTL+yfkScXF5YZmb1KnJYfT0wCvh34MaGV0uSLpP0tKT5DfOmSloiaV56HdZu4GZmZSrSK8+2EfGZNsq+HPgWWf+PjS6KiAvbKM/MrDJF9hxntbOHFxG3A78ZekhmZvUrMm71SmAE2ciDa8jGhYmI2G7QwqVuYFZEvDFNTwWmAM+TPZp4VrOnbTxu9e/z+MT93Bb93BaZ4R63etDkuD5ykmMXsBwI4MvA2CIXdnp6eqK3t3dIdXdvjONWTx96B+xVtQO4LRq5LfpVNm51G+0gqWlybNUrz54R8V+S9s5bHhH3DjWQiFjWUP4l+KZyM+tQrdL5mWSHtV/LWRbAQUOtTNLYiFiaJo8C5rda38ysLq165Tkl/ZzUTsGSvg9MBMZIehL4IjBR0gSy5LoI+Kt2yjYzK1tpJwIiYnLO7EvLqs/MbDgVuZXHzGyT0zQ5SnpH+rlVdeGYmXWGVnuOfePG3FlFIGZmnaTVOcc1kmbw+wNsARARZ5QXlplZvVolx8OBg4H38OoBtszMNnqtbuVZDsyU9EgabMvMbJNR5Gr1M5KuS92PPS3pWkk7lR6ZmVmNiiTH7wA3AK9Pr39N88zMNlpFkuOOEfGdiFibXpcDO5Qcl5lZrYokx+WSTpC0eXqdADxTdmBmZnUqkhxPJhuW9SlgKXAMcFKZQZmZ1a3I6IOLgQ9UEIuZWcfws9VmZjmcHM3Mcjg5mpnlKJwcJe0v6SZJcyQdWWD9vHGrXyfpVkmPpZ+vbTdwM7Myteqy7A8HzDqTbGiDw8gGxxrM5cChA+adDcyOiPHA7DRtZtZxWu05XizpC5K2TtMryG7jOYpsaNWWmoxbfQRwRXp/BTDoHqiZWR2aJseIOBK4D5gl6aPAJ4GtgO1pP6l1NQyw9RTQ1WY5ZmalGnTcakmbA39N1oXZtLRHWKzw3x+3ekVEjG5Y/mxE5J53lHQK2eiHdHV17TNz5syi1QLVDVruwdv7uS36uS36VdUW7bTDpEmTmo5b3TQ5SvoA8ClgLfAVsr3IzwPjgHMi4n8GqzgnOT4KTIyIpZLGAnMi4o8HK6enpyd6e3sHW+1Vqhq03IO393Nb9HNb9KuqLdppB0lNk2Orc47nAu8le3Tw/IhYERFnkSXIaUOOInMDcGJ6fyJwfZvlmJmVqlU6fw74ILAt8HTfzIh4DDhusIKbjFs9HbhG0seAxWSJ18ys47RKjkcBk4E1wIeHWnCTcasB3j3UsszMqjbYMAnfrDAWM7OO4ccHzcxyODmameVwcjQzy+HkaGaWw8nRzCyHk6OZWQ4nRzOzHE6OZmY5nBzNzHI4OZqZ5XByNDPL4eRoZpbDydHMLIeTo5lZDidHM7Mc1QxyMYCkRcBK4GVgbbMxHMzM6lJLckwmpQ51zcw6jg+rzcxyDDpudSmVSguBZ4EA/jkiZuSs43GrB/D4xP3cFv3cFpnKxq0uk6RxEbFE0o7ArcDpEXF7s/U9bnXG4xP3c1v0c1tkqhy3ujQRsST9fBq4Dti3jjjMzJqpPDlKGiHpNX3vgUOA+VXHYWbWSh1Xq7uA6yT11X91RNxUQxxmZk1VnhwjYgHwlqrrNTMbCt/KY2aWw8nRzCyHk6OZWQ4nRzOzHE6OZmY5nBzNzHI4OZqZ5XByNDPL4eRoZpbDydHMLIeTo5lZDidHM7McTo5mZjmcHM3Mcjg5mpnlqCU5SjpU0qOSHpd0dh0xmJm1UscwCZsD3wbeC+wFTJa0V9VxmJm1Usee477A4xGxICJ+B8wEjqghDjOzpupIjuOAJxqmn0zzzMw6RuXjVks6Bjg0Iv4yTX8E2C8iThuw3inAKWnyj4FHKw20uDHA8rqD6BBui35ui36d3Ba7RsQOeQvqGH1wCbBzw/ROad6rRMQMYEZVQbVLUm+zQcE3NW6Lfm6LfhtqW9RxWH0PMF7SGyT9AXAccEMNcZiZNVXH0KxrJZ0G3AxsDlwWEQ9VHYeZWSt1HFYTET8BflJH3SXo+EP/Crkt+rkt+m2QbVH5BRkzsw2BHx80M8vh5GiFSDpD0iOSvld3LJ1E0n/WHUOnkNQtaX7dcQyXWs45buokieyUxrq6YxmCvwYOjogn2y1A0hYRsXYYY6pdRLy97hisHN5zbCDpx5LmSnoo3YSOpFWSpkm6X9JdkrrS/N3T9IOSzpW0qqGcT0u6R9IDkr6U5nWnzjauBObz6ns9O5qki4HdgH+TdI6kyyT9QtJ9ko5I63RL+pmke9Pr7Wn+xDT/BuDhGn+NUqTPhyRdIGl++jx8KC27UtKRDet+r6+9OpmkEZJuTJ/5+ZI+JOkL6TM9X9KM9AWPpH3SevcDf9NQxhRJP5J0k6THJH21Ydkhku5Mn5MfShqZ5k+X9HD6v7kwzfuLVOf9km6vtCEiwq/0Al6Xfm5DlsC2BwJ4f5r/VeBz6f0sYHJ6fyqwKr0/hOzqnMi+fGYBBwLdwDpg/7p/zzbbZhHZkw5fAU5I80YD/w2MALYFtk7zxwO96f1E4AXgDXX/DiW1yyrgaOBWslvTuoBfAmOBPwN+nNYbBSwEtqg75gK/09HAJQ3To/r+N9L0dxv+Jx4ADkzvLwDmp/dTgAVp262BxWQ7BGOA24ERab3PAF9I/2uP0n+ReHT6+SAwrnFeVS/vOb7aGekb8C6yP+R44HdkCQ5gLlmSAzgA+GF6f3VDGYek133AvcCeqRyAxRFxV1nBV+QQ4GxJ84A5ZB/8XYAtgUskPUjWLo09Lf0iIhZWHWiF3gl8PyJejohlwG3A2yLiNrIHHnYAJgPXxoZxWuFB4M8lnS/pXRHxHDBJ0t3p73sQ8KeSRpMlrL49uu8OKGd2RDwXEb8lO2rYFdif7LPx8/QZOjHNfw74LXCppA8CL6Yyfg5cLunjZF8+lfE5x0TSROBg4ICIeFHSHLJ//DWRvraAlxm8zQScFxH/PKD8brI9qA2dgKMj4lXPukuaCiwD3kK2x/zbhsUbw+/driuBE8ieBDup5lgKiYj/lrQ3cBhwrqTZZIfMPRHxRPpbb12gqNUN7/v+dwTcGhGTB64saV/g3cAxwGnAQRFxqqT9gPcBcyXtExHPrMevV5j3HPuNAp5NiXFPsm+4Vu4iO/yA7IPf52bg5IbzKOMk7Tjs0dbnZuD0hnNOb03zRwFLI7vI9BEq/pav2c+AD0naPO0lHgj8Ii27HPgkQERsEOdcJb0eeDEiriI7VN47LVqePtfHAETECmCFpHem5ccXKP4u4B2S9kh1jZD0R6ncUZE9IPIpsi9ZJO0eEXdHxBeAX1PhuXrvOfa7CThV0iNk5z4GO/z9JHCVpHPSts8BRMQtkv4EuDPlj1Vkew4vlxV4xb4M/CPwgKTNyM6jHQ78P+BaSR8la49NZW8xgOvITrPcn6b/NiKeAoiIZekz9eP6QhyyNwEXSFoHrAH+D3Ak2Xn4p8j6R+hzEnCZpABuGazgiPi1pCnA9yVtlWZ/DlgJXC9pa7K9yzPTsgskjU/zZpO1cSX8hEybJG0LvBQRIek4soszHX8l0oaPpO2BeyNi1xbrbEt2Dm/vdO7ONhDec2zfPsC30uHlCuDkmuOxCqVDzznAhS3WORi4FLjIiXHD4z1HM7McviBjZpbDydHMLIeTo5lZDidHq52klyXNa3idPQxldkv6cMN0j6RvrG+5tunwBRmrnaRVETFymMucCPzfiDh8OMu1TYf3HK1jSVok6by0N9kraW9JN0v6H0mnpnVye8QBpgPvStt+SlnvQLPSNq9T1gPTA8p6Vnpzmj9VWY9DcyQtkHRGPb+5dQLf52idYJvUCUGf8yLiB+n9LyNigqSLyB7FewfZc73zgYuBDwITyB43GwPck7q2OpuGPce0J9nnS8B9EXGkpIPInn+ekJbtCUwCXgM8KumfImLNcP/C1vmcHK0TvBQRE5os6xu290FgZESsBFZKWp16hXmlRxxgmaTbgLcBz7eo752k5+Ij4j8kbS9pu7TsxohYDayW9DRZF2Rtd/BrGy4fVlun6+vZZR2v7uVlHeV8uef1JGObICdH29A16xFnJdmhcbNtjodXDreXR0SrPU3bBPlb0TrBwHOON0VE0dt5cnvEkfQM8HLqvPhyss6H+0wl60nmAbJOVU9cz/htI+RbeczMcviw2swsh5OjmVkOJ0czsxxOjmZmOZwczcxyODmameVwcjQzy+HkaGaW438BvM1G7R7Y1JoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 360x216 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# the histogram of the data\n",
    "labels = train_df['emotion'].unique()\n",
    "post_total = len(train_df)\n",
    "df1 = train_df.groupby(['emotion']).count()['text']\n",
    "df1 = df1.apply(lambda x: round(x*100/post_total,3))\n",
    "\n",
    "#plot\n",
    "fig, ax = plt.subplots(figsize=(5,3))\n",
    "plt.bar(df1.index,df1.values)\n",
    "\n",
    "#arrange\n",
    "plt.ylabel('% of instances')\n",
    "plt.xlabel('Emotion')\n",
    "plt.title('Emotion distribution')\n",
    "plt.grid(True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Feature engineering\n",
    "### Using Bag of Words\n",
    "Using scikit-learn ```CountVectorizer``` perform word frequency and use these as features to train a model.  \n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build analyzers (bag-of-words)\n",
    "BOW_vectorizer = CountVectorizer() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Learn a vocabulary dictionary of all tokens in the raw documents.\n",
    "BOW_vectorizer.fit(train_df['text'])\n",
    "\n",
    "# 2. Transform documents to document-term matrix.\n",
    "train_data_BOW_features = BOW_vectorizer.transform(train_df['text'])\n",
    "test_data_BOW_features = BOW_vectorizer.transform(test_df['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3613x10115 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 51467 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the result\n",
    "train_data_BOW_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "scipy.sparse.csr.csr_matrix"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(train_data_BOW_features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add .toarray() to show\n",
    "train_data_BOW_features.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3613, 10115)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the dimension\n",
    "train_data_BOW_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['2k17', '2much', '2nd', '30', '300', '301', '30am', '30pm', '30s', '31']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# observe some feature names\n",
    "feature_names = BOW_vectorizer.get_feature_names()\n",
    "feature_names[100:110]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The embedding is done. We can technically feed this into our model. However, depending on the embedding technique you use and your model, your accuracy might not be as high, because:\n",
    "\n",
    "* curse of dimensionality  (we have 10,115 dimension now)\n",
    "* some important features are ignored (for example, some models using emoticons yeld better performance than counterparts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"😂\" in feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try using another tokenizer below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3613, 500)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "# build analyzers (bag-of-words)\n",
    "BOW_500 = CountVectorizer(max_features=500, tokenizer=nltk.word_tokenize) \n",
    "\n",
    "# apply analyzer to training data\n",
    "BOW_500.fit(train_df['text'])\n",
    "\n",
    "train_data_BOW_features_500 = BOW_500.transform(train_df['text'])\n",
    "\n",
    "## check dimension\n",
    "train_data_BOW_features_500.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 1, 0, ..., 0, 0, 0],\n",
       "       ...,\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 3, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data_BOW_features_500.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cheerfully',\n",
       " 'cheering',\n",
       " 'cheery',\n",
       " 'come',\n",
       " 'comes',\n",
       " 'could',\n",
       " 'country',\n",
       " 'cry',\n",
       " 'customer',\n",
       " 'damn']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# observe some feature names\n",
    "feature_names_500 = BOW_500.get_feature_names()\n",
    "feature_names_500[100:110]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"😂\" in feature_names_500"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 2 (Take home): **  \n",
    "Generate an embedding using the TF-IDF vectorizer instead of th BOW one with 1000 features and show the feature names for features [100:110]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 3. Model\n",
    "### 3.1 Decision Trees\n",
    "Using scikit-learn ```DecisionTreeClassifier``` performs word frequency and uses these as features to train a model.  \n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape:  (3613, 500)\n",
      "y_train.shape:  (3613,)\n",
      "X_test.shape:  (347, 500)\n",
      "y_test.shape:  (347,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# for a classificaiton problem, you need to provide both training & testing data\n",
    "X_train = BOW_500.transform(train_df['text'])\n",
    "y_train = train_df['emotion']\n",
    "\n",
    "X_test = BOW_500.transform(test_df['text'])\n",
    "y_test = test_df['emotion']\n",
    "\n",
    "## take a look at data dimension is a good habbit  :)\n",
    "print('X_train.shape: ', X_train.shape)\n",
    "print('y_train.shape: ', y_train.shape)\n",
    "print('X_test.shape: ', X_test.shape)\n",
    "print('y_test.shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['fear', 'fear', 'joy', 'joy', 'anger', 'anger', 'anger', 'sadness',\n",
       "       'fear', 'anger'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## build DecisionTree model\n",
    "DT_model = DecisionTreeClassifier(random_state=0)\n",
    "\n",
    "## training!\n",
    "DT_model = DT_model.fit(X_train, y_train)\n",
    "\n",
    "## predict!\n",
    "y_train_pred = DT_model.predict(X_train)\n",
    "y_test_pred = DT_model.predict(X_test)\n",
    "\n",
    "## so we get the pred result\n",
    "y_test_pred[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 4. Results Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will check the results of our model's performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training accuracy: 0.99\n",
      "testing accuracy: 0.64\n"
     ]
    }
   ],
   "source": [
    "## accuracy\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "acc_train = accuracy_score(y_true=y_train, y_pred=y_train_pred)\n",
    "acc_test = accuracy_score(y_true=y_test, y_pred=y_test_pred)\n",
    "\n",
    "print('training accuracy: {}'.format(round(acc_train, 2)))\n",
    "print('testing accuracy: {}'.format(round(acc_test, 2)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       anger       0.65      0.65      0.65        84\n",
      "        fear       0.62      0.64      0.63       110\n",
      "         joy       0.67      0.68      0.68        79\n",
      "     sadness       0.61      0.57      0.59        74\n",
      "\n",
      "    accuracy                           0.64       347\n",
      "   macro avg       0.64      0.64      0.64       347\n",
      "weighted avg       0.64      0.64      0.64       347\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## precision, recall, f1-score,\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_true=y_test, y_pred=y_test_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[55 16  9  4]\n",
      " [15 70  9 16]\n",
      " [ 5 13 54  7]\n",
      " [ 9 14  9 42]]\n"
     ]
    }
   ],
   "source": [
    "## check by confusion matrix\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_true=y_test, y_pred=y_test_pred) \n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Funciton for visualizing confusion_matrix\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import itertools\n",
    "\n",
    "def plot_confusion_matrix(cm, classes, title='Confusion matrix',\n",
    "                          cmap=sns.cubehelix_palette(as_cmap=True)):\n",
    "    \"\"\"\n",
    "    This function is modified from: \n",
    "    http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
    "    \"\"\"\n",
    "    classes.sort()\n",
    "    tick_marks = np.arange(len(classes))    \n",
    "    \n",
    "    fig, ax = plt.subplots(figsize=(5,5))\n",
    "    im = ax.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    ax.figure.colorbar(im, ax=ax)\n",
    "    ax.set(xticks=np.arange(cm.shape[1]),\n",
    "           yticks=np.arange(cm.shape[0]),\n",
    "           xticklabels = classes,\n",
    "           yticklabels = classes,\n",
    "           title = title,\n",
    "           xlabel = 'True label',\n",
    "           ylabel = 'Predicted label')\n",
    "\n",
    "    fmt = 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt), horizontalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "    ylim_top = len(classes) - 0.5\n",
    "    plt.ylim([ylim_top, -.5])\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVsAAAFaCAYAAACwk/5IAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5xU5dnG8d+1gFJEqiB2AnYTsHeDIjaMlUQUFbtGjRpfRRITW0xsWLBFsaISSzQ2UAQ7GkHQqGCLsUuVDnbgfv84Z3FE2F12d+bsWa+vn/nsnPbMPcx67zP3ec5zFBGYmVlxlWUdgJnZT4GTrZlZCTjZmpmVgJOtmVkJONmamZWAk62ZWQk42ZqZLYOk9SW9VvCYK+k0Sa0ljZT0XvqzVaVteZytmVnlJDUAJgJbAycBMyPiYkn9gVYRcVZFx7tna2ZWNd2B9yPiY2BfYHC6fjCwX2UHO9mamVVNb+Du9Hn7iJicPp8CtK/sYJcRzCy3mjdpHQsXfVft47/6dv6bwNcFqwZFxKAl95O0AjAJ2DgipkqaHREtC7bPiogK67YNqx2lmVnGFi76js6rblbt48d/8vzXEbFFFXbdE3g1Iqamy1MldYiIyZI6ANMqa8BlBDPLNUnVfiyHg/m+hADwCNA3fd4XeLiyBpxszcwqIKkZ0AP4V8Hqi4Eekt4Ddk2XK+QygpnlmJCK22eMiC+ANkusm0EyOqHK3LM1MysB92zNLNfKWK7aa2bcszUzKwH3bM0stwTLO6ogM+7ZmpmVgHu2ZpZrZUUejVBbnGzNLL+W/+KEzOTjT4KZWc65Z2tmuSYP/TIzs3Lu2ZpZbon8nCDLR5RmZjnnnq2Z5ZpHI5iZ2WLu2ZpZjoky92zNzKyce7ZmllsClJM+o5OtmeWaT5CZmdli7tmaWX4JnyAzM7PvuWdrZjmm3ExE42RrZrnluRHMzOwHnGzNzErAZQQzy7W8jLN1sjWzHPPcCGZmVsDJ1sysBFxGMLPcSiaiyUcZwcnWzHItL+NsnWzNLL+Un9EI+fiTYLklqYmkRyXNkfTPGrTTR9KI2owtK5J2lPRu1nFYaTnZGgCSDpE0TtJ8SZMlPS5ph1pouhfQHmgTEb+ubiMRMSQidquFeIpKUkjqXNE+ETEqItYvVUxWN7iMYEg6HegPnAA8AXwL7AHsC7xQw+bXBv4bEQtq2E69IKmh/y1qjzzO1vJCUgvgAuCkiPhXRHwREd9FxKMRcWa6z4qSrpI0KX1cJWnFdFs3SZ9J+j9J09Je8ZHptvOBc4CD0h7z0ZLOk3RXweuvk/YGG6bLR0j6QNI8SR9K6lOw/oWC47aTNDYtT4yVtF3Btmcl/UXSi2k7IyS1Xcb7L4+/X0H8+0naS9J/Jc2U9MeC/beS9JKk2em+10paId32fLrb6+n7Paig/bMkTQFuK1+XHtMpfY3N0uXVJH0uqVuNPtifENXgv1JysrVtgcbAgxXsczawDdAV6AJsBfypYPuqQAtgdeBo4DpJrSLiXOBvwL0RsVJE3FJRIJKaAVcDe0ZEc2A74LWl7NcaGJbu2wa4AhgmqU3BbocARwLtgBWAMyp46VVJ/g1WJ/njcBNwKLA5sCPwZ0kd030XAr8H2pL823UHTgSIiJ3Sfbqk7/fegvZbk/Tyjyt84Yh4HzgLuEtSU+A2YHBEPFtBvJZDTrbWBpheyVfbPsAFETEtIj4HzgcOK9j+Xbr9u4h4DJgPVLcmuQjYRFKTiJgcEW8uZZ+ewHsRcWdELIiIu4F3gF8V7HNbRPw3Ir4C7iP5Q7Es3wF/jYjvgHtIEunAiJiXvv5bJH9kiIhXImJ0+rofATcCv6zCezo3Ir5J4/mBiLgJ+B8wBuhA8sfN6hknW5sBtC3/Gr8MqwEfFyx/nK5b3MYSyfpLYKXlDSQivgAOIqkdT5Y0TNIGVYinPKbVC5anLEc8MyJiYfq8PBlOLdj+VfnxktaTNFTSFElzSXruSy1RFPg8Ir6uZJ+bgE2AayLim0r2tQKSqv0oJSdbewn4Btivgn0mkXwFLrdWuq46vgCaFiyvWrgxIp6IiB4kPbx3SJJQZfGUxzSxmjEtj7+TxLVuRKwM/BEqLf5FRRslrQRcBdwCnJeWSayKyqRqP0oaZ0lfzeqciJhDUqe8Lj0x1FRSI0l7Sro03e1u4E+SVklPNJ0D3LWsNivxGrCTpLXSk3N/KN8gqb2kfdPa7Tck5YhFS2njMWC9dLhaQ0kHARsBQ6sZ0/JoDswF5qe97t8usX0q8LPlbHMgMC4ijiGpRd9Q4yitznGyNSLicuB0kpNenwOfAicDD6W7XAiMA94AxgOvpuuq81ojgXvTtl7hhwmyLI1jEjCTpBa6ZDIjImYAewP/R1IG6QfsHRHTqxPTcjqD5OTbPJJe971LbD8PGJyOVvhNZY1J2pdkmF35+zwd2Kx8FIZlT1JLSfdLekfS25K2ldRa0khJ76U/W1XaTkSF33DMzOqsVk3bRLf1e1b7+Ideu/OViNiion0kDQZGRcTN6TC/piTlo5kRcbGk/kCriDironZ8UYOZ5ZiKOhFNWuraCTgCICK+Bb5Nv5F0S3cbDDxLMoRvmVxGMLP8UtFHI3QkKa3dJuk/km5Ozym0j4jJ6T5TSC5Jr5CTrZn9lLVVMidI+eO4JbY3BDYD/h4Rm5KMpulfuEMktdhK67EuI5jZT9n0Smq2nwGfRcSYdPl+kmQ7VVKHiJgsqQMwrbIXcs/WzHJLFHecbURMAT6VVH5FZHeSKwofAfqm6/oCD1fWlnu2S9G4UZNo3njlrMMoilXbNc86hKIqa9gg6xCKRvX4vX06aRIzZs2u1lUGJZhQ5nfAkHQkwgckc26UAfdJOprk6sVKh/k52S5F88Yrc2DXg7IOoyjOPHnnrEMoqmartsw6hKJZoVX9/UO5y8GHZx3CMkXEa8DSSg3dl6cdlxHMzErAPVszy7H8TB7uZGtmuSXyc8NHJ1szy7W89GxdszUzKwEnWzOzEnAZwcxyrdQ3bqwuJ1szyy3JNVszMyvgZGtmVgIuI5hZjpX+LrnV5WRrZrnmmq2ZmS3mnq2Z5Vpehn65Z2tmVgLu2ZpZbpXfqSEPnGzNLNfyMhrBZQQzsxJwz7bE/nrfuXz95TcsWrSIRQsXcdGxA9j7yD3Z4VfbMm/2fAAeHjSUCaPfyjjS5df/hmt5+tVxtFm5BY8PGLh4/R3Dh3HXiOGUlZWx86abc1afunsLlKq66f4HGDJ0GEHQp2dPjvt1r6xDqlULFy6k+8GH06FdO+6+9sqsw6kXnGwzcMWp1/DFnC9+sO6p+55l5D1PZxRR7Tjglztz6O57cuZ1Vy9e99Kb43ly3FgeveQKVmzUiBlzZmcYYe1454MPGTJ0GI/dcD0rNGzEIf3Oose229JxjdWzDq3W3DjkHtb7WUfmzf+i8p2zVMW75NYFLiNYrdlqw41p2eyHNyX8x8gnOH7f/VmxUSMA2rTI/w0Z3/vkYzbbaEOaNm5Mw4YN2KZrFx4bNSrrsGrNxKlTGTHqBQ7df9+sQ6lU+Z0aqvsoJfdsSywCTr3iRCJg1MMv8sKj/wag2wE7svUeW/LxO5/ywLUP8uX8rzKOtHZ8NHkSY995myvu+QcrrNCIPxzal190WjfrsGpk/Y4dufjmW5k5Zw6NV1yRp0ePocv662UdVq05+9IrOO/3pzD/iy+zDqVecbItsQEnXcXs6XNo3nIlTr3yJKZ8MpXnHnqBYYOHQ8A+x+zFgSfvz50X/yPrUGvFgoULmTN/HvdfeDFvvP8/Trnqcp65+u+5OYO8NOutvTYnHdyb3mf2o2njxmzcuRNlZfXjS+ITz42ibetWdN1oQ14Y+0rW4VSJL2qoo5TI7H3Pnj4HgHmz5/Pa82/QccO1mTdrHrEoiAheePQl1tlwrazCq3WrtmnDblttgyS6dF4XScycNzfrsGrskJ57MWLQjTx09UBaNG9OpzXXzDqkWjHmtdcZ/uwouu65D8ee9UdGjR3L8X/4c9Zh1Qt1JtlKekjSK5LelHRcum6+pL9Kel3SaEnt0/Wd0uXxki6UNL+gnTMljZX0hqTz03XrSHpX0h3ABCCT/zNWaLwCKzZZcfHzDbfcgIkfTGblNisv3qfrTr9g0oeTswivKHpssTVj3pwAwIeTJvHdggW0br5yJUfVfdNnzQLgs6lTeez5UezfvXvGEdWOc049mQkjh/Ha449w0yV/Y8ctt+TGi/6SdVgVKlP1H6VUl8oIR0XETElNgLGSHgCaAaMj4mxJlwLHAhcCA4GBEXG3pBPKG5C0G7AusBVJ7fwRSTsBn6Tr+0bE6NK+re+t3Ko5J/ztGADKGpQxduQrvPXy2xzxp8NYs/PqBMGMyTMZMuDerEKskdOuvoIxb01g1rx5bH/iMZzaqze9dt6F/jdcx55nnEqjhg257MRTcl1CKHf0Oecxa+5cGjVswEWnnUqL5itlHZLVcYqIrGMAQNJ5wP7p4jrA7sBzQOOICEkHAT0i4hhJM4D2EbFA0srApIhYSdIAoBdQPr5oJeAi4CngmYjoWMHrHwccB7DSis0377PlEbX9FuuEM0/eOesQiqrZqvkf7bAsK7RqXvlOObXLwYfz2ptvLfdf4XbN28dBmx9S7de99rmrXomILardwHKoEz1bSd2AXYFtI+JLSc8CjYHv4vu/BgupPF4BF0XEjUu0vw5Q4YDBiBgEDAJYpXn7uvEXyMzqjbpSs20BzEoT7QbANpXsPxo4MH3eu2D9E8BRklYCkLS6pHa1Hq2Z1QnlE9FU91FKdaJnCwwHTpD0NvAuSTKtyGnAXZLOTo+dAxARIyRtCLyU1gXnA4eS9IrNrL7J4OKE6qoTyTYivgH2XMqmlQr2uR+4P12cCGyT1nJ7A+sX7DeQ5ATakjapvYjNzJZPnUi21bA5cK2SP2mzgaMyjsfMMlKWk4sacplsI2IU0CXrOMzMqiqXydbMDL6fiCYP6spoBDOzes09WzPLtbzMZ+tka2a5lpNc6zKCmVkpONmamZWAywhmllvll+vmgZOtmeWYcnOnBidbM7MKSPoImEcyx8qCiNhCUmvgXpLpYD8CfhMRsypqxzVbM8svlezuujtHRNeCuW/7A09FxLok82X3r6wBJ1szs+W3LzA4fT4Y2K+yA1xGMLNcq+EJsraSxhUsD0pvJFAogBGSArgx3d4+IspvFjgFaF/ZCznZmlluJXMj1KiJ6VW4Lc4OETExvRHBSEnvFG5Mp3qt9O4uLiOYmVUgIiamP6cBD5LcUHaqpA4A6c9plbXjZGtmtgySmklqXv4c2A2YADwC9E136ws8XFlbLiOYWa4V+aKG9sCD6ciFhsA/ImK4pLHAfZKOBj4GflNZQ062ZpZrxbyoISI+YCk3KoiIGUD35WnLZQQzsxJwz9bMckuU/pbk1eWerZlZCbhnuxSrtmvOWaftmnUYRbHP6QOyDqGoRt5xftYhFM2stz/NOoSiWfj1t9U7UPmZPNzJ1sxyzTd8NDOzxZxszcxKwGUEM8u1vIxGcLI1s9yqhYloSsZlBDOzEnCyNTMrAZcRzCzXXLM1MysB313XzKzYlv/GjZlxzdbMrAScbM3MSsBlBDPLLQFl+agiONmaWb65ZmtmZos52ZqZlYDLCGaWa3kpIzjZmlluSfk5QeYygplZCSyzZytpPBBL2wRERPyiaFHVU/2uu5pnXhlHmxYtGH7lNQBcde/d3PvUCFqv3AKAMw45lJ032yLLMKtlnZ+tyWXXnrd4eY21VuO6K27lkQeGM+C681htjQ5M+mwyZ5x4LnPnzs8u0Fpy0/0PMGToMIKgT8+eHPfrXlmHVG39b7iWp18dR5uVW/D4gIGL198xfBh3jRhOWVkZO2+6OWf1OTzDKPOvojLC3iWL4iei187dOXzPnpxxzVU/WH9Uz304dt/9M4qqdnz0waf8eq+jASgrK+OpMQ/w1BPPc/SJfRjz4qvc8vchHP3bPhx94qFcefENGUdbM+988CFDhg7jsRuuZ4WGjTik31n02HZbOq6xetahVcsBv9yZQ3ffkzOvu3rxupfeHM+T48by6CVXsGKjRsyYMzvDCCuWl5rtMssIEfFx+SNdtW76fBowsyTR1TNbbbQxLVdaKeswim7r7Tfn008mMXniVHbusQMPPzAcgIcfGM7Ou+2QcXQ1994nH7PZRhvStHFjGjZswDZdu/DYqFFZh1VtW224MS2bNf/Bun+MfILj992fFRs1AqBNi5ZZhFYlUvUfpVRpzVbSscD9wI3pqjWAh4oZ1E/NHcMfY8/TT6HfdVczZ37+v2Lvuc8uPP7IUwC0aduK6dNmADB92gzatG2VZWi1Yv2OHRnzxnhmzpnDl19/zdOjxzBp2rSsw6pVH02exNh33ubAs8/i4PP/xBvvv5d1SMtUJlX7UdI4q7DPScD2wFyAiHgPaFfMoJZF0imS3pY0JIvXL4Y+u+/Js9fewLABV9GuVSv+OvjWrEOqkYaNGtJt1+0ZMeyZrEMpmvXWXpuTDu5N7zP7cUi/s9i4cyfKyurXueYFCxcyZ/487r/wYvr36cspV11OxNJO4VhVVeU35JuI+LZ8QVJDln7irBROBHpERJ/qNpDGX2es0rIlDRo0oKysjN677sYb/6u7PYiq2LHbNrw94T1mTJ8FwIzps2jbrg0Abdu1Wbw+7w7puRcjBt3IQ1cPpEXz5nRac82sQ6pVq7Zpw25bbYMkunReF0nMnDc367ByrSrJ9jlJfwSaSOoB/BN4tLhh/ZikG4CfAY9LOlvSrZJelvQfSfum+6wjaZSkV9PHdun6bun6R4C3Sh17RabN+r78/cSY0ay35loZRlNze+7TnccfeXLx8rNPvsi+B+4BwL4H7sEzI1/IKrRaNX1W8kfjs6lTeez5UezfvXvGEdWuHltszZg3JwDw4aRJfLdgAa2br5xxVD+mGv5XSlXp5fUHjgbGA8cDjwE3FzOopYmIEyTtAewMnA48HRFHSWoJvCzpSZKTdz0i4mtJ6wJ3A+XjqDYDNomID0sde7lTrhzAmDcnMGveXLY77ihOPehgxrw5gbc++hABa7Rrx1+PPzGr8GqsSZPGbLvjFlzwxwGL191y/RAGXH8++x/Uk8kTp/B/J56bYYS15+hzzmPW3Lk0atiAi047lRbN83vi87Srr2DMWxOYNW8e2594DKf26k2vnXeh/w3XsecZp9KoYUMuO/GUOnvWv46G9SOVJtuIWCRpMDCGpHzwbmRfvNkN2EfSGelyY2AtYBJwraSuwEJgvYJjXq4o0Uo6DjgOYLW2qxQl6Kt/f8aP1h3UvUdRXisLX331NTt2/dUP1s2ZPZdjD/l9RhEVz8PXDKx8p5y46pTTl7r+ipNPK3Ek9VulyVZST+AG4H2SCxo6Sjo+Ih4vdnAVhQUcGBHv/mCldB4wFehCUiL5umDzFxU1GBGDgEEAP+/UOes/JmZWz1SlZns5sHNEdIuIX5J8jb+yuGFV6gngd0q/10jaNF3fApgcEYuAw4AGGcVnZqWg+jX0a15E/K9g+QNgXpHiqaq/AI2ANyS9mS4DXA/0lfQ6sAGV9GbNLP+U3vSxOo9SqmhuhAPSp+MkPQbcR1Kz/TUwtgSx/UhErFOwePxStr8HFM7ZcFa6/lng2SKGZmZWoYpqtoVnOqYCv0yffw40KVpEZmZVJOrBaISIOLKUgZiZ1WdVGY3QmGSc7cYkQ6wAiIijihiXmVmV1NXxv0uqygmyO4FVgd2B50gmosn6BJmZGZDcqaG6j5LGWYV9OkfEn4EvImIw0BPYurhhmZnVHZIapFMDDE2XO0oaI+l/ku6VtEJlbVQl2X6X/pwtaROSsayZzPplZpaRU4G3C5YvAa6MiM7ALJJSa4WqkmwHSWoF/Bkon8jl0uWP1cyslqn442wlrUHyjf7mdFnALiTzfAMMBvarrJ2qzI1QPunMcySzbpmZ1QklGvp1FdAPKL+dRRtgdkQsSJc/Ayq9J1JFFzUsfXaKVERcUbU4zczqrLaSxhUsD0rnSQFA0t7AtIh4RVK3mrxQRT3b5hVsMzOrA2o8x8H0iKjodtbbk8wwuBfJ0NeVgYFAS0kN097tGsDEyl6ooosazl++mM3M6peI+APwB0huQgCcERF9JP0T6AXcA/QFHq6srfp14yQz+8nJaCKas4DTJf2PpIZ7S2UH1Kn7cZmZLY9Szo1QOKFVRHwAbLU8x7tna2ZWAh6NYGb5pfzMjVCV0QjrA1uSXNAAydSLLxczKDOz+qbS0QiSngc2i4h56fJ5wLCSRGdmVomcdGyrVLNtD3xbsPxtus7MzKqoKqMR7gBelvRgurwfybXAZmaZK/WNG6urKnMj/FXS48CO6aojI+I/xQ3LzKx+qeo426bA3Ii4TdIqkjpGxIfFDMzMrDJ5ugdZpTVbSeeSXC3xh3RVI+CuYgZlZlbfVKVnuz+wKfAqQERMkuRJasysTqgP42zLfRsRISkAJDUrckyZa7BiI1b+2apZh1EUw649K+sQiuov51Q6H0hunXtJr6xDKJoGK1Rz5gDVozICcJ+kG0mmFDsWeJJ0xnIzM6uaqoxGGCCpBzCX5GqycyJiZNEjMzOrVI1n7yqZSpOtpEsi4ixg5FLWmZlZFVSljNBjKev2rO1AzMyqQ6r+o5QqmvXrt8CJQCdJbxRsag78u9iBmZnVJxWVEf4BPA5cBPQvWD8vImYWNSozsyoQ+blcd5llhIiYExEfkdzcbGZEfBwRHwMLJG1dqgDNzOqDqtRs/w7ML1ien64zM8tc7mu2BRQRUb4QEYsk+d5lZpa9HN2poSo92w8knSKpUfo4Ffig2IGZmdUnVUm2JwDbAROBz4CtgeOKGZSZWX1TlSvIpgG9SxCLmdlyy0kVocJxtv0i4lJJ1wCx5PaIOKWokZmZVSKZzzYf2bainu3b6c9xpQjEzKw+q+juuo+mP32/MTOrs3LSsa2wjPAoSykflIuIfYoSkZlZPVRRGWFA+vMAYFW+vxXOwcDUYgZlZlZVua/ZRsRzAJIuj4gtCjY9Ksl1XDPLXj27U0MzST8rX5DUEaj3t8YxM6tNVbns9vfAs5I+IBlpsTZwfFGj+onYtOd+rNSsKQ3KymjQoAFPDcn3ucgzBl7F0+PG0qZFC0Zeez0AA+66k5FjxlBWJtq0aMnlp55G+zZtMo60ei645xy+/uprYmGwcOFCLj3+isXbuv+mGwectB/99jmbL+Z8kWGUNfO/jz/huD+fu3j544mT6Hfs0Rzf+zcZRlU/VOWihuGS1gU2SFe9ExHfFDesn46HbryeNq1aZh1Grfh1913pu/fenH7l90no+AMO5IxDDwPgtkcfYeC9d/O3E0/OKsQaG3jadT9Kpi1XackGW27AzCn5n3m089pr8fQdtwGwcOFCuuxzAHv9cqeMo6pIfm6LU2kZQVJT4Ezg5Ih4HVhL0t5Fj8xyZ+tNNqHlSj+8y33zpk0XP//y668R+fgfY3n0Onk/HrrhEWKZY3fyadS4V1hn9dVYs0PdvdN0clFD/Zn16zbgFWDbdHki8E9gaLGC+qmQoNdJpyCg74H70/fA/bMOqSguvfMO/vXM0zRv2pR7/npR1uFUWxCcPOAECHjh0X/z4qMv8YvtN2H29DlMfH9S1uHVugdHPsX+PXbNOoxK5WXy8Kok204RcZCkgwEi4kvVsX67pH9HxHZZx7G8ht06iA7t2vH5zJn0+u3vWHedddhu802zDqvW9TvscPoddjjX/fM+Bg8byumH9Mk6pGq54uSrmTN9Diu1XInfXf5bpn48ld0P7cE1Z9S/6Z2//e47RrzwImef6NMztaUqoxG+ldSE9AIHSZ2AOlWzzWOiBejQrh0Aq7RuzV47d+PVN9/MOKLi2q9bNx7/94tZh1Ftc6bPAWD+7Pm8Pmo8nbt2pk2H1vzxln5ccM85tFylBf1vOoOVWzevpKW676mXRvPz9dejXevWWYdSb1Ql2Z4LDAfWlDQEeAroV9SolpOk+UpcJmmCpPGSDkq33SFpv4J9h0jaN7toE1989RXzvvhi8fNnR49hw06dMo6q9n04aeLi5yPGjKHTGmtkGE31rdB4BVZssuLi5xtuuT6fvPMJ/ff7M+f0voBzel/A7M/ncPGxA5g7c17G0dbcgyOfZP8e3bMOo0rqRc02LRe8Q3IV2TYk9ehTI2J6CWJbXgcAXYEuQFtgrKTngVtIhq89JKkFydy8fTOLMvX5jJn0/b/kb9aChQs5cI/d6b79tpUcVbf97rJLeWnCeGbNncvWR/bl9wf34ZlXxvHBxM8oUxmrt1uFv514UtZhVkvzVs057sKjAGjQoIyxT77KWy+/k3FUxfHFV1/x/MvjGHDWmVmHUrkc3amhwmQbESHpsYj4OTCsRDFV1w7A3RGxEJgq6Tlgy4h4RNL1klYBDgQeiIgFSx4s6TjSSdHXWLX4Z1/XWWN1nrt3SNFfp5SuOfPHX3h677ZbBpHUvhmTZ3DR0ZdVuM85vS8oUTTF1axJE955oq7/754/VSkjvCppy6JHUlx3AIcCRwK3Lm2HiBgUEVtExBb1ZdyrmdUdVUm2WwOjJb0v6Y20HvpGsQOrhlHAQZIapL3YnYCX0223A6cBRMRb2YRnZrWtvo2z3b3oUdRcAA+SjAV+PV3uFxFTACJiqqS3gYeyC9HMikFlxcuakhoDzwMrkuTL+yPi3HSOmHuANiTXIRwWEd9W1FZF89k2JrnZY2dgPHDL0mqdWZPUBpiZ3m79zPSx5D5NgXWBu0scnpnl2zfALhExX1Ij4AVJjwOnA1dGxD2SbgCOBioccF1RGWEwsAVJot0TuLxWQq9FklYDXuL7uXeXts+uJLf4uSYi5pQqNjPLv0jMTxcbpY8AdgHuT9cPBvZbyuE/UFEZYaN0FAKSbuH7+medERGTgPUq2edJkpnKzKweKnbtVVIDklJBZ+A64H1gdsE3/c+A1Strp6Jk+135k4hYkJexbGb2E1LzcbZtl7gZwqCIGFS4QzqctKukliTnhjagGipKtl0kzU2fC2iSLit5/Vi5Oi9oZlabatgPnL7EnWiWKSJmS3qG5ER8S0kN097tGiQTdFVomdzePlwAABEISURBVDXbiGgQESunj+YR0bDguROtmdV7klZJe7Skc8T0IDkH9AzQK92tL/BwZW1VZeiXmdlPVQdgcFq3LQPui4ihkt4C7pF0IfAfkmkBKuRka2a5pSLfqSEi3gB+NO9pRHwAbLU8bTnZmlmu5eXcfVUu1zUzsxpysjUzKwGXEcws33JSR3CyNbP8ytHk4S4jmJmVgJOtmVkJuIxgZrmWkyqCk62Z5VsxJw+vTU62ZpZb5bfFyQPXbM3MSsDJ1sysBFxGMLP8ytE4WydbM8u1nORalxHMzErBPduliIWL+G7uF1mHURSLFizKOoSi+vMF+2YdQtE8eM0LWYdQNLOmza98p6Uq7ny2tck9WzOzEnDP1sxyLScdW/dszcxKwT1bM8ut5AqyfHRt3bM1MysB92zNLL9EbrqMTrZmlmsuI5iZ2WLu2ZpZruWkY+uerZlZKbhna2a55pqtmZkt5p6tmeWXXLM1M7MC7tmaWY7lp2vrZGtmuSXycytzlxHMzErAPVszy7WcVBHcszUzKwX3bDN00/0PMGToMIKgT8+eHPfrXlmHVCP9rh3I0+PG0aZFC54YeO0Ptt308IP8bfBtvHL7XbReeeWMIqw99e2zg+TigAMuPIIvZs1j+ID72eXEX7FKx1VZtHAR096fzKhbh7NoYR27h12ObmXunm1G3vngQ4YMHcZjN1zPUzffzJMvjebDzyZmHVaNHLhzd27/83k/Wj9p+ueMev01Vmu7SumDKoL6+NkBbLLHFsyaNH3x8nsvvsm9Z97EP/vfQsMVGrJBty4ZRpd/TrYZee+Tj9lsow1p2rgxDRs2YJuuXXhs1Kisw6qRrTfehJbNV/rR+r/cegv9DzsiNz2QytTHz65Z6+as3bUT7zzzxuJ1n77+weLn096fTLPWzbMIrVJS9R+l5GSbkfU7dmTMG+OZOWcOX379NU+PHsOkadOyDqvWjXh5NKu2acNGHTtmHUqtqY+f3XaHdWf03c8QET/aVtagjHV32JhP3/hgKUdaVdW5mq2kdYChEbFJxqEU1Xprr81JB/em95n9aNq4MRt37kRZWf362/fVN99w/QP3c8c552cdSq2qb5/dWpt24qs5XzL9o6l02HCtH23f4cjdmPLOp0x597MMoquCnHxjqnPJ9qfkkJ57cUjPvQD42003s9oq9aOmWe7jKZP5bOpU9jr9VACmzJjOr844jYcuuZxVWrXKOLqaqU+f3arrrcHam3dmra6daNCoAY2arMguv92bp/8+lM0P2J7GzZsy4pZ/ZR1mJiStCdwBtAcCGBQRAyW1Bu4F1gE+An4TEbMqaqtoyVZSM+A+YA2gAfAXYH3gV0AT4N/A8RERkjYHbk0PHVHQxhHAPkBToBPwYET0S7ftBpwPrAi8DxwZEfMlXZweswAYERFnSPo1cC6wEJgTETsV630vj+mzZtG2VSs+mzqVx54fxbDrr8s6pFq1wdrrMO72Oxcv73D8MTxy2RX1YjRCffrsXr73OV6+9zkAOmy4Fl16bsXTfx/KBt1+wRo/78jQv92TpJk6qshXkC0A/i8iXpXUHHhF0kjgCOCpiLhYUn+gP3BWRQ0Vs2e7BzApInoCSGoBjIyIC9LlO4G9gUeB24CTI+J5SZct0U5XYFPgG+BdSdcAXwF/AnaNiC8knQWcLuk6YH9ggzSJt0zbOAfYPSImFqzL3NHnnMesuXNp1LABF512Ki2WcnIpT0654jJGT5jArHlz2faYIzmt98EctOtuWYdVFPXts1uaHY/ag3nT57Df+YcB8OHY//Lqgy9mHNUPFftEV0RMBianz+dJehtYHdgX6JbuNhh4lgyT7XjgckmXkNRgR0k6UFI/kp5qa+BNSaOAlhHxfHrcncCeBe08FRFzACS9BawNtAQ2Al5Mz3CvALwEzAG+Bm6RNBQYmrbxInC7pPuApX4fknQccBzA6u3b18b7r9TD1wwsyeuUytWnn1nh9hduvLlEkRRfffvsyk1++xMmv/0JADcdfmnG0dQt6fmkTYExQPs0EQNMISkzVKhoyTYi/itpM2Av4EJJTwEnAVtExKeSzgMaV6GpbwqeLySJWSS95IOX3FnSVkB3oBdwMrBLRJwgaWugJ8nXgM0jYsYS8Q4CBgF0WX/9Ovylycy+V+OubVtJ4wqWB6W54IevIq0EPACcFhFzC4cxpt+iK80ZxazZrgbMjIi7JM0Gjkk3TU8D7wXcHxGzJc2WtENEvAD0qULzo4HrJHWOiP+l9eHVgUlA04h4TNKLwAdpLJ0iYgwwRtKewJrAjGU1bmY/GdMjYouKdpDUiCTRDomI8m/GUyV1iIjJkjoAlY79K2YZ4efAZZIWAd8BvwX2AyaQdLvHFux7JHBr+tdhxJINLSkiPk9Pnt0tacV09Z+AecDDkhqT9H5PT7ddJmnddN1TwOs1fG9mVkcUs2arpAt7C/B2RFxRsOkRoC9wcfrz4craKmYZ4QngiSVWjyNJikvu+wpQeC1gv3T97cDtBfvtXfD8aWDLpbz0Vktp/4CqR25meVLk0QjbA4cB4yW9lq77I0mSvU/S0cDHwG8qa8jjbM3MliEtbS4rm3dfnrbye9mLmVmOuGdrZvmVoykWnWzNLN/ykWtdRjAzKwUnWzOzEnAZwcxyzTVbM7MiE062ZmbFJ3JTDM1JmGZm+eZka2ZWAi4jmFmOyTVbM7NSyEuydRnBzKwEnGzNzErAZQQzy7d8VBGcbM0sx1T0ycNrjcsIZmYl4GRrZlYCLiOYWb7lZOiXk62Z5VpOcq2TrZnlV55m/XLN1sysBNyzXYo3/vvf6R267fJxCV+yLTC9hK9XSn5v+VXK97d2iV4nM062SxERq5Ty9SSNi4gtSvmapeL3ll+5eH8S5GScrZOtmeWaa7ZmZraYk23dMCjrAIrI7y2/6vv7KymXEeqAiKi3v9R+b/mVm/eXjyqCk62Z5ZtrtlbvSTpF0tuShmQdS7FJ+nfWMRSLpHUkTcg6jmpJZ/2q7qOU3LPNKSV/zhURizIM40Rg14j4rLoNSGoYEQtqMaaiiIjtso7B8s0921om6SFJr0h6U9Jx6br5kv4q6XVJoyW1T9d3SpfHS7pQ0vyCds6UNFbSG5LOT9etI+ldSXcAE4A1s3iPaSw3AD8DHpd0tqRbJb0s6T+S9i2Id5SkV9PHdun6bun6R4C3snoPyyP9DCXpMkkT0s/soHTbHZL2K9h3SPm/QYljbCZpWPp7NkHSQZLOSX+PJkgalP6RRtLm6X6vAycVtHGEpH9JGi7pPUmXFmzbTdJL6Wf5T0krpesvlvRW+rs6IF336/Q1X5f0fIn/KeokJ9vad1REbA5sAZwiqQ3QDBgdEV2A54Fj030HAgMj4ufA4t6hpN2AdYGtgK7A5pJ2SjevC1wfERtHRCmvcvuBiDgBmATsTPL+no6IrdLlyyQ1A6YBPSJiM+Ag4OqCJjYDTo2I9UobeY0cQPJ5dAF2JXmfHYBbgCMAJLUAtgOGZRDfHsCkiOgSEZsAw4FrI2LLdLkJsHe6723A79LfySV1Jfm8fg4cJGlNSW2BP5F8k9kMGAecnv5+7w9sHBG/AC5M2zgH2D1tf5+ivNtyUvUfJeRkW/tOSXsLo0l6nusC3wJD0+2vAOukz7cF/pk+/0dBG7ulj/8ArwIbpO0AfBwRo4sVfDXtBvSX9BrwLNAYWAtoBNwkaTzJ+9yo4JiXI+LDUgdaQzsAd0fEwoiYCjwHbBkRzwHrSloFOBh4IKPSyHigh6RLJO0YEXOAnSWNST+DXYCNJbUEWkZEeY/zziXaeSoi5kTE1yTfPNYGtiH5/F5MP+e+6fo5wNfALZIOAL5M23gRuF3SsUCDor3j9Fbm1X2Ukmu2tUhSN5Iez7YR8aWkZ0kSz3cREeluC6n8313ARRFx4xLtrwN8UYsh1xYBB0bEuz9YKZ0HTCXpCZaR/E9Zri6+j5q4AzgU6A0cmUUAEfFfSZsBewEXSnqKpESwRUR8mn4ejavQ1DcFz8t/XwWMjIiDl9xZ0lZAd6AXcDKwS0ScIGlroCfwiqTNI2JGDd5e7rlnW7taALPSRLsBSW+gIqOBA9PnvQvWPwEcVVATW11Su1qPtvY8AfyuoB64abq+BTA5PYl3GEXt4ZTEKJKv1Q3SXuxOwMvpttuB0wAiIpM6tKTVgC8j4i7gMpJSDcD09HepVxrfbGC2pB3S7X2q0PxoYHtJndPXaiZpvbTdFhHxGPB7kj+sSOoUEWMi4hzgczI8v1BXuGdbu4YDJ0h6G3iX5Be0IqcBd0k6Oz12DkBEjJC0IfBSmr/mk/SaFhYr8Br6C3AV8IakMuBDktrg9cADkg4neX957s0G8CBJ6ef1dLlfREwBiIip6ef+UHYh8nOSOvIi4Dvgt8B+JCdTpwBjC/Y9ErhVUgAjKms4Ij6XdARwt6QV09V/AuYBD0tqTNL7PT3ddpmkddN1T5H8mxVHPobZou+/3VqpSWoKfBURIak3cHBElPwstlUsPQn0akQscxrA9LMcD2yW1kqtBLqst148ds011T5+jT32eKVUM5u5Z5utzYFr06/fs4GjMo7HlpB+NX8WGFDBPruSjEi40onWlsXJNkMRMYq0xmV1U0RMAiocnhYRT/ITmPy6zvLlumZmVs7J1sxyrdjjbJVcHTlNBfNHSGotaWR6ld1ISa0qa8fJ1szySyS3xanuo2puJ7k6r1B/kos/1iUZbdG/skacbK1kJLWR9Fr6mCJpYsHyCrX4OrtKqnAIlqRjJF21nO1+ll59ZT8h6ZV2M5dYvS8wOH0+mGSIXYV8gsxKJr2CqCssvrpsfkT84Cx/OjIj69nMzCrTPiImp8+nAO0rO8A9W8ucpM5KZo0aArwJrClpdsH23pJuTp+3VzIr1Tgls4xVeJWepG2UzFT1H0kvpgPty60t6bm07vangmP6pm2/Jun69EINq5NqPDdC2/R3qfxx3PJGkF6KX+kFC+7ZWl2xAXB4RIyTVNHv5dXApRExWslcEUOBTSrY/21gx4hYIGkPklmpDkq3bZUe+y0wVtJQYAHJLFbbpccMIrmU+h8/btrqhJqN/JpezYsapkrqEBGTlcz8Nq2yA5xsra54PyLGVWG/XYH1C84kt5LUJCK+Wsb+LYE7JHVayrYnImIWJPMQk8zq1RDYEhiXvkYT4NOqvw37iXiEZOazi9OfD1d2gJOt1RWF8yYs4of9lcKZqgRsFRHfVrHdv5Ik1evTSVSGF2xb8qtfpO3fGhF/rmL7lrGqDuGqQft3A91ISg6fAeeSJNn7JB0NfAz8prJ2nGytzomIRZJmpfXV90m+1n+ebn6SZNrAKwEkdY2I1yporgUwMX1+xBLbdktHF3xLcna5D8lkP/dLGhgR09N5EZpFxCe18NYsh5Y2rWSq+/K048K/1VVnkUzd+G8K7mJBkmi3V3ILlrf4/q4Xy3IJyQxUr/Lj6t5Ykq9/r5NMCv5aRIwHzgeelPQGyYxYlZ5ptoyUZpxt7YTqWb/MLK+6bLB+PDHohmof3+GXu3jWLzOzqih2zba2uIxgZlYC7tmaWb7lpGfrZGtmuSVKf5fc6nIZwcysBNyzNbP8Kh/6lQPu2ZqZlYB7tmaWa67ZmpnZYu7Zmlm+5aRn62RrZrkmnyAzM7NyTrZmZiXgMoKZ5Zfkmq2ZWSl46JeZmS3mnq2Z5Zt7tmZmVs49WzPLtbyMs3WyNbP8Ei4jmJnZ95xszcxKwGUEM8ux/FzUoIjIOgYzs2qRNBxoW4MmpkfEHrUVT0WcbM3MSsA1WzOzEnCyNTMrASdbM7MScLI1MysBJ1szsxL4fy2Abf3D7UT0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot your confusion matrix\n",
    "my_tags = ['anger', 'fear', 'joy', 'sadness']\n",
    "plot_confusion_matrix(cm, classes=my_tags, title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 3 (Take home): **  \n",
    "Can you interpret the results above? What do they mean?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 4 (Take home): **  \n",
    "Build a model using a ```Naive Bayes``` model and train it. What are the testing results? \n",
    "\n",
    "*Reference*: https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 5 (Take home): **  \n",
    "\n",
    "How do the results from the Naive Bayes model and the Decision Tree model compare? How do you interpret these differences? Use the theoretical background covered in class to try and explain these differences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Other things you can try"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, there are several things you can try that will affect your results. In order to yield better results, you can experiment by: \n",
    "    * Trying different features (Feature engineering)\n",
    "        -Eg. Word2Vec,PCA,LDA,FastText, Clustering......\n",
    "    * Trying different models\n",
    "    * Analyzing your results and interpret them to improve your feature engineering/model building process\n",
    "    * Iterate through the steps above until finding a satisfying result\n",
    "Remember that you should also consider the task at hand and the model you'll feed the data to. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 6. Deep Learning\n",
    "\n",
    "We use [Keras](https://keras.io/) to be our deep learning framwork, and follow the [Model (functional API)](https://keras.io/models/model/) to build a Deep Neural Network (DNN) model. Keras runs with Tensorflow in the backend. It's a nice abstraction to start working with NN models. \n",
    "\n",
    "Because Deep Learning is a 1-semester course, we can't talk about each detail about it in the lab session. Here, we only provide a simple template about how to build & run a DL model successfully. You can follow this template to design your model.\n",
    "\n",
    "We will begin by building a fully connected network, which looks like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Fully Connected Network](pics/pic1.png)\n",
    "\n",
    "(source: https://github.com/drewnoff/spark-notebook-ml-labs/tree/master/labs/DLFramework)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.1 Prepare data (X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape:  (3613, 500)\n",
      "y_train.shape:  (3613,)\n",
      "X_test.shape:  (347, 500)\n",
      "y_test.shape:  (347,)\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "\n",
    "# standardize name (X, y) \n",
    "X_train = BOW_500.transform(train_df['text'])\n",
    "y_train = train_df['emotion']\n",
    "\n",
    "X_test = BOW_500.transform(test_df['text'])\n",
    "y_test = test_df['emotion']\n",
    "\n",
    "## check dimension is a good habbit \n",
    "print('X_train.shape: ', X_train.shape)\n",
    "print('y_train.shape: ', y_train.shape)\n",
    "print('X_test.shape: ', X_test.shape)\n",
    "print('y_test.shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 Deal with categorical label (y)\n",
    "\n",
    "Rather than put your label `train_df['emotion']` directly into a model, we have to process these categorical (or say nominal) label by ourselves. \n",
    "\n",
    "Here, we use the basic method [one-hot encoding](https://en.wikipedia.org/wiki/One-hot) to transform our categorical  labels to numerical ones.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "check label:  ['anger' 'fear' 'joy' 'sadness']\n",
      "\n",
      "## Before convert\n",
      "y_train[0:4]:\n",
      " 490       anger\n",
      "1875       fear\n",
      "1146       fear\n",
      "2878    sadness\n",
      "Name: emotion, dtype: object\n",
      "\n",
      "y_train.shape:  (3613,)\n",
      "y_test.shape:  (347,)\n",
      "\n",
      "\n",
      "## After convert\n",
      "y_train[0:4]:\n",
      " [[1. 0. 0. 0.]\n",
      " [0. 1. 0. 0.]\n",
      " [0. 1. 0. 0.]\n",
      " [0. 0. 0. 1.]]\n",
      "\n",
      "y_train.shape:  (3613, 4)\n",
      "y_test.shape:  (347, 4)\n"
     ]
    }
   ],
   "source": [
    "## deal with label (string -> one-hot)\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "label_encoder.fit(y_train)\n",
    "print('check label: ', label_encoder.classes_)\n",
    "print('\\n## Before convert')\n",
    "print('y_train[0:4]:\\n', y_train[0:4])\n",
    "print('\\ny_train.shape: ', y_train.shape)\n",
    "print('y_test.shape: ', y_test.shape)\n",
    "\n",
    "def label_encode(le, labels):\n",
    "    enc = le.transform(labels)\n",
    "    return keras.utils.to_categorical(enc)\n",
    "\n",
    "def label_decode(le, one_hot_label):\n",
    "    dec = np.argmax(one_hot_label, axis=1)\n",
    "    return le.inverse_transform(dec)\n",
    "\n",
    "y_train = label_encode(label_encoder, y_train)\n",
    "y_test = label_encode(label_encoder, y_test)\n",
    "\n",
    "print('\\n\\n## After convert')\n",
    "print('y_train[0:4]:\\n', y_train[0:4])\n",
    "print('\\ny_train.shape: ', y_train.shape)\n",
    "print('y_test.shape: ', y_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 Build model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_shape:  500\n",
      "output_shape:  4\n"
     ]
    }
   ],
   "source": [
    "# I/O check\n",
    "input_shape = X_train.shape[1]\n",
    "print('input_shape: ', input_shape)\n",
    "\n",
    "output_shape = len(label_encoder.classes_)\n",
    "print('output_shape: ', output_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Imgur](pics/pic2.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/eva/dmenv/local/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "Model: \"model_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 64)                32064     \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "re_lu_2 (ReLU)               (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 4)                 260       \n",
      "_________________________________________________________________\n",
      "softmax_1 (Softmax)          (None, 4)                 0         \n",
      "=================================================================\n",
      "Total params: 36,484\n",
      "Trainable params: 36,484\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense\n",
    "from keras.layers import ReLU, Softmax\n",
    "\n",
    "# input layer\n",
    "model_input = Input(shape=(input_shape, ))  # 500\n",
    "X = model_input\n",
    "\n",
    "# 1st hidden layer\n",
    "X_W1 = Dense(units=64)(X)  # 64\n",
    "H1 = ReLU()(X_W1)\n",
    "\n",
    "# 2nd hidden layer\n",
    "H1_W2 = Dense(units=64)(H1)  # 64\n",
    "H2 = ReLU()(H1_W2)\n",
    "\n",
    "# output layer\n",
    "H2_W3 = Dense(units=output_shape)(H2)  # 4\n",
    "H3 = Softmax()(H2_W3)\n",
    "\n",
    "model_output = H3\n",
    "\n",
    "# create model\n",
    "model = Model(inputs=[model_input], outputs=[model_output])\n",
    "\n",
    "# loss function & optimizer\n",
    "model.compile(optimizer='adam',\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# show model construction\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4 Train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/eva/dmenv/local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
      "\n",
      "Train on 3613 samples, validate on 347 samples\n",
      "Epoch 1/25\n",
      "3613/3613 [==============================] - 1s 164us/step - loss: 1.3264 - accuracy: 0.3701 - val_loss: 1.2686 - val_accuracy: 0.4524\n",
      "Epoch 2/25\n",
      "3613/3613 [==============================] - 0s 62us/step - loss: 0.9801 - accuracy: 0.6543 - val_loss: 0.9122 - val_accuracy: 0.6628\n",
      "Epoch 3/25\n",
      "3613/3613 [==============================] - 0s 66us/step - loss: 0.5664 - accuracy: 0.8021 - val_loss: 0.7511 - val_accuracy: 0.7176\n",
      "Epoch 4/25\n",
      "3613/3613 [==============================] - 0s 62us/step - loss: 0.3907 - accuracy: 0.8633 - val_loss: 0.7578 - val_accuracy: 0.7205\n",
      "Epoch 5/25\n",
      "3613/3613 [==============================] - 0s 67us/step - loss: 0.3062 - accuracy: 0.8981 - val_loss: 0.8070 - val_accuracy: 0.7118\n",
      "Epoch 6/25\n",
      "3613/3613 [==============================] - 0s 65us/step - loss: 0.2471 - accuracy: 0.9150 - val_loss: 0.8772 - val_accuracy: 0.6945\n",
      "Epoch 7/25\n",
      "3613/3613 [==============================] - 0s 61us/step - loss: 0.2066 - accuracy: 0.9350 - val_loss: 0.9255 - val_accuracy: 0.7003\n",
      "Epoch 8/25\n",
      "3613/3613 [==============================] - 0s 63us/step - loss: 0.1690 - accuracy: 0.9469 - val_loss: 0.9642 - val_accuracy: 0.6801\n",
      "Epoch 9/25\n",
      "3613/3613 [==============================] - 0s 56us/step - loss: 0.1508 - accuracy: 0.9560 - val_loss: 1.0646 - val_accuracy: 0.6686\n",
      "Epoch 10/25\n",
      "3613/3613 [==============================] - 0s 65us/step - loss: 0.1294 - accuracy: 0.9596 - val_loss: 1.1104 - val_accuracy: 0.6744\n",
      "Epoch 11/25\n",
      "3613/3613 [==============================] - 0s 58us/step - loss: 0.1167 - accuracy: 0.9657 - val_loss: 1.1722 - val_accuracy: 0.6859\n",
      "Epoch 12/25\n",
      "3613/3613 [==============================] - 0s 63us/step - loss: 0.1062 - accuracy: 0.9676 - val_loss: 1.2441 - val_accuracy: 0.6744\n",
      "Epoch 13/25\n",
      "3613/3613 [==============================] - 0s 63us/step - loss: 0.1030 - accuracy: 0.9682 - val_loss: 1.2317 - val_accuracy: 0.6974\n",
      "Epoch 14/25\n",
      "3613/3613 [==============================] - 0s 61us/step - loss: 0.0934 - accuracy: 0.9726 - val_loss: 1.2760 - val_accuracy: 0.6916\n",
      "Epoch 15/25\n",
      "3613/3613 [==============================] - 0s 56us/step - loss: 0.0807 - accuracy: 0.9723 - val_loss: 1.3107 - val_accuracy: 0.6830\n",
      "Epoch 16/25\n",
      "3613/3613 [==============================] - 0s 64us/step - loss: 0.0853 - accuracy: 0.9729 - val_loss: 1.3299 - val_accuracy: 0.6859\n",
      "Epoch 17/25\n",
      "3613/3613 [==============================] - 0s 56us/step - loss: 0.0752 - accuracy: 0.9740 - val_loss: 1.4038 - val_accuracy: 0.6888\n",
      "Epoch 18/25\n",
      "3613/3613 [==============================] - 0s 67us/step - loss: 0.0750 - accuracy: 0.9751 - val_loss: 1.4192 - val_accuracy: 0.6801\n",
      "Epoch 19/25\n",
      "3613/3613 [==============================] - 0s 63us/step - loss: 0.0756 - accuracy: 0.9732 - val_loss: 1.4215 - val_accuracy: 0.6830\n",
      "Epoch 20/25\n",
      "3613/3613 [==============================] - 0s 52us/step - loss: 0.0727 - accuracy: 0.9751 - val_loss: 1.4627 - val_accuracy: 0.6830\n",
      "Epoch 21/25\n",
      "3613/3613 [==============================] - 0s 62us/step - loss: 0.0670 - accuracy: 0.9765 - val_loss: 1.4775 - val_accuracy: 0.6830\n",
      "Epoch 22/25\n",
      "3613/3613 [==============================] - 0s 56us/step - loss: 0.0675 - accuracy: 0.9759 - val_loss: 1.5322 - val_accuracy: 0.6744\n",
      "Epoch 23/25\n",
      "3613/3613 [==============================] - 0s 66us/step - loss: 0.0644 - accuracy: 0.9751 - val_loss: 1.5718 - val_accuracy: 0.6715\n",
      "Epoch 24/25\n",
      "3613/3613 [==============================] - 0s 57us/step - loss: 0.0615 - accuracy: 0.9770 - val_loss: 1.5510 - val_accuracy: 0.6888\n",
      "Epoch 25/25\n",
      "3613/3613 [==============================] - 0s 63us/step - loss: 0.0609 - accuracy: 0.9790 - val_loss: 1.5704 - val_accuracy: 0.6801\n",
      "training finish\n"
     ]
    }
   ],
   "source": [
    "from keras.callbacks import CSVLogger\n",
    "\n",
    "csv_logger = CSVLogger('logs/training_log.csv')\n",
    "\n",
    "# training setting\n",
    "epochs = 25\n",
    "batch_size = 32\n",
    "\n",
    "# training!\n",
    "history = model.fit(X_train, y_train, \n",
    "                    epochs=epochs, \n",
    "                    batch_size=batch_size, \n",
    "                    callbacks=[csv_logger],\n",
    "                    validation_data = (X_test, y_test))\n",
    "print('training finish')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.5 Predict on testing data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.7493770e-04, 1.2077219e-01, 1.1765673e-02, 8.6698723e-01],\n",
       "       [7.9836012e-05, 9.9956292e-01, 1.5846005e-04, 1.9881182e-04],\n",
       "       [4.4943678e-01, 2.4788025e-01, 1.3412434e-01, 1.6855867e-01],\n",
       "       [8.9813825e-12, 1.1894122e-09, 1.0000000e+00, 7.5186506e-09],\n",
       "       [9.9899954e-01, 2.8576700e-05, 3.0366706e-08, 9.7192288e-04]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## predict\n",
    "pred_result = model.predict(X_test, batch_size=128)\n",
    "pred_result[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['sadness', 'fear', 'anger', 'joy', 'anger'], dtype=object)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_result = label_decode(label_encoder, pred_result)\n",
    "pred_result[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "testing accuracy: 0.68\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print('testing accuracy: {}'.format(round(accuracy_score(label_decode(label_encoder, y_test), pred_result), 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>loss</th>\n",
       "      <th>val_accuracy</th>\n",
       "      <th>val_loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.370053</td>\n",
       "      <td>1.326438</td>\n",
       "      <td>0.452450</td>\n",
       "      <td>1.268567</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.654304</td>\n",
       "      <td>0.980068</td>\n",
       "      <td>0.662824</td>\n",
       "      <td>0.912151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.802103</td>\n",
       "      <td>0.566359</td>\n",
       "      <td>0.717579</td>\n",
       "      <td>0.751141</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.863272</td>\n",
       "      <td>0.390703</td>\n",
       "      <td>0.720461</td>\n",
       "      <td>0.757825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0.898146</td>\n",
       "      <td>0.306216</td>\n",
       "      <td>0.711816</td>\n",
       "      <td>0.806985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>0.915029</td>\n",
       "      <td>0.247102</td>\n",
       "      <td>0.694524</td>\n",
       "      <td>0.877231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>0.934957</td>\n",
       "      <td>0.206622</td>\n",
       "      <td>0.700288</td>\n",
       "      <td>0.925531</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>7</td>\n",
       "      <td>0.946859</td>\n",
       "      <td>0.169036</td>\n",
       "      <td>0.680115</td>\n",
       "      <td>0.964210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>0.955992</td>\n",
       "      <td>0.150810</td>\n",
       "      <td>0.668588</td>\n",
       "      <td>1.064577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>9</td>\n",
       "      <td>0.959590</td>\n",
       "      <td>0.129380</td>\n",
       "      <td>0.674352</td>\n",
       "      <td>1.110407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>0.965679</td>\n",
       "      <td>0.116669</td>\n",
       "      <td>0.685879</td>\n",
       "      <td>1.172215</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0.967617</td>\n",
       "      <td>0.106185</td>\n",
       "      <td>0.674352</td>\n",
       "      <td>1.244118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>0.968171</td>\n",
       "      <td>0.102974</td>\n",
       "      <td>0.697406</td>\n",
       "      <td>1.231709</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>13</td>\n",
       "      <td>0.972599</td>\n",
       "      <td>0.093435</td>\n",
       "      <td>0.691643</td>\n",
       "      <td>1.275958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>14</td>\n",
       "      <td>0.972322</td>\n",
       "      <td>0.080700</td>\n",
       "      <td>0.682997</td>\n",
       "      <td>1.310652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>0.972876</td>\n",
       "      <td>0.085333</td>\n",
       "      <td>0.685879</td>\n",
       "      <td>1.329858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>16</td>\n",
       "      <td>0.973983</td>\n",
       "      <td>0.075226</td>\n",
       "      <td>0.688761</td>\n",
       "      <td>1.403778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>17</td>\n",
       "      <td>0.975090</td>\n",
       "      <td>0.075010</td>\n",
       "      <td>0.680115</td>\n",
       "      <td>1.419210</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>18</td>\n",
       "      <td>0.973152</td>\n",
       "      <td>0.075575</td>\n",
       "      <td>0.682997</td>\n",
       "      <td>1.421481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>19</td>\n",
       "      <td>0.975090</td>\n",
       "      <td>0.072702</td>\n",
       "      <td>0.682997</td>\n",
       "      <td>1.462714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "      <td>0.976474</td>\n",
       "      <td>0.066983</td>\n",
       "      <td>0.682997</td>\n",
       "      <td>1.477529</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>21</td>\n",
       "      <td>0.975920</td>\n",
       "      <td>0.067465</td>\n",
       "      <td>0.674352</td>\n",
       "      <td>1.532173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>22</td>\n",
       "      <td>0.975090</td>\n",
       "      <td>0.064367</td>\n",
       "      <td>0.671470</td>\n",
       "      <td>1.571843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>23</td>\n",
       "      <td>0.977027</td>\n",
       "      <td>0.061493</td>\n",
       "      <td>0.688761</td>\n",
       "      <td>1.551047</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>24</td>\n",
       "      <td>0.978965</td>\n",
       "      <td>0.060949</td>\n",
       "      <td>0.680115</td>\n",
       "      <td>1.570415</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    epoch  accuracy      loss  val_accuracy  val_loss\n",
       "0       0  0.370053  1.326438      0.452450  1.268567\n",
       "1       1  0.654304  0.980068      0.662824  0.912151\n",
       "2       2  0.802103  0.566359      0.717579  0.751141\n",
       "3       3  0.863272  0.390703      0.720461  0.757825\n",
       "4       4  0.898146  0.306216      0.711816  0.806985\n",
       "5       5  0.915029  0.247102      0.694524  0.877231\n",
       "6       6  0.934957  0.206622      0.700288  0.925531\n",
       "7       7  0.946859  0.169036      0.680115  0.964210\n",
       "8       8  0.955992  0.150810      0.668588  1.064577\n",
       "9       9  0.959590  0.129380      0.674352  1.110407\n",
       "10     10  0.965679  0.116669      0.685879  1.172215\n",
       "11     11  0.967617  0.106185      0.674352  1.244118\n",
       "12     12  0.968171  0.102974      0.697406  1.231709\n",
       "13     13  0.972599  0.093435      0.691643  1.275958\n",
       "14     14  0.972322  0.080700      0.682997  1.310652\n",
       "15     15  0.972876  0.085333      0.685879  1.329858\n",
       "16     16  0.973983  0.075226      0.688761  1.403778\n",
       "17     17  0.975090  0.075010      0.680115  1.419210\n",
       "18     18  0.973152  0.075575      0.682997  1.421481\n",
       "19     19  0.975090  0.072702      0.682997  1.462714\n",
       "20     20  0.976474  0.066983      0.682997  1.477529\n",
       "21     21  0.975920  0.067465      0.674352  1.532173\n",
       "22     22  0.975090  0.064367      0.671470  1.571843\n",
       "23     23  0.977027  0.061493      0.688761  1.551047\n",
       "24     24  0.978965  0.060949      0.680115  1.570415"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let's take a look at the training log\n",
    "training_log = pd.DataFrame()\n",
    "training_log = pd.read_csv(\"logs/training_log.csv\")\n",
    "training_log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 6 (Take home): **  \n",
    "\n",
    "Plot the Training and Validation Accuracy and Loss (different plots), just like the images below (Note: the pictures below are an example from a different model). How to interpret the graphs you got? How are they related to the concept of overfitting/underfitting covered in class?\n",
    "<table><tr>\n",
    "    <td><img src=\"pics/pic3.png\" style=\"width: 300px;\"/> </td>\n",
    "    <td><img src=\"pics/pic4.png\" style=\"width: 300px;\"/> </td>\n",
    "</tr></table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note\n",
    "\n",
    "If you don't have a GPU (level is higher than GTX 1060) or you are not good at setting lots of things about computer, we recommend you to use the [kaggle kernel](https://www.kaggle.com/kernels) to do deep learning model training. They have already installed all the librarys and provided free GPU for you to use.\n",
    "\n",
    "Note however that you will only be able to run a kernel for 6 hours. After 6 hours of inactivity, your Kaggle kernel will shut down (meaning if your model takes more than 6 hours to train, you can't train it at once).\n",
    "\n",
    "\n",
    "### More Information for your reference\n",
    "\n",
    "* Keras document: https://keras.io/\n",
    "* Keras GitHub example: https://github.com/keras-team/keras/tree/master/examples\n",
    "* CS229: Machine Learning: http://cs229.stanford.edu/syllabus.html\n",
    "* Deep Learning cheatsheet: https://stanford.edu/~shervine/teaching/cs-229/cheatsheet-deep-learning\n",
    "* If you want to try TensorFlow or PyTorch: https://pytorch.org/tutorials/\n",
    "https://www.tensorflow.org/tutorials/quickstart/beginner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 7. Word2Vector\n",
    "\n",
    "We will introduce how to use `gensim` to train your word2vec model and how to load a pre-trained model.\n",
    "\n",
    "https://radimrehurek.com/gensim/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.1 Prepare training corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>text_tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>10490</td>\n",
       "      <td>Take public opinion on revenge with Pakistan i...</td>\n",
       "      <td>[Take, public, opinion, on, revenge, with, Pak...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1875</td>\n",
       "      <td>21018</td>\n",
       "      <td>My goals are so big they scare small minds</td>\n",
       "      <td>[My, goals, are, so, big, they, scare, small, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>20289</td>\n",
       "      <td>@PanicAtTheDisco hey, y'all announced it like ...</td>\n",
       "      <td>[@, PanicAtTheDisco, hey, ,, y'all, announced,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2878</td>\n",
       "      <td>40051</td>\n",
       "      <td>@Christy_RTR @doge_e_fresh I'm despondent</td>\n",
       "      <td>[@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>432</td>\n",
       "      <td>10432</td>\n",
       "      <td>@sarah_urbina why do you even beef Sara you le...</td>\n",
       "      <td>[@, sarah_urbina, why, do, you, even, beef, Sa...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text  \\\n",
       "490   10490  Take public opinion on revenge with Pakistan i...   \n",
       "1875  21018         My goals are so big they scare small minds   \n",
       "1146  20289  @PanicAtTheDisco hey, y'all announced it like ...   \n",
       "2878  40051          @Christy_RTR @doge_e_fresh I'm despondent   \n",
       "432   10432  @sarah_urbina why do you even beef Sara you le...   \n",
       "\n",
       "                                         text_tokenized  \n",
       "490   [Take, public, opinion, on, revenge, with, Pak...  \n",
       "1875  [My, goals, are, so, big, they, scare, small, ...  \n",
       "1146  [@, PanicAtTheDisco, hey, ,, y'all, announced,...  \n",
       "2878  [@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...  \n",
       "432   [@, sarah_urbina, why, do, you, even, beef, Sa...  "
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## check library\n",
    "import gensim\n",
    "\n",
    "## ignore warnings\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# # if you want to see the training messages, you can use it\n",
    "# import logging\n",
    "# logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "\n",
    "## the input type\n",
    "train_df['text_tokenized'] = train_df['text'].apply(lambda x: nltk.word_tokenize(x))\n",
    "train_df[['id', 'text', 'text_tokenized']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([list(['Take', 'public', 'opinion', 'on', 'revenge', 'with', 'Pakistan', 'if', 'govt', 'is', 'unable', 'to', 'decide', '.', '@', 'aajtak', '@', 'TimesNow', '@', 'narendramodi']),\n",
       "       list(['My', 'goals', 'are', 'so', 'big', 'they', 'scare', 'small', 'minds']),\n",
       "       list(['@', 'PanicAtTheDisco', 'hey', ',', \"y'all\", 'announced', 'it', 'like', 'immediately', 'after', 'I', 'asked', '.', 'Nice', '.', 'Thanks', \"y'all\", '#', 'panic'])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## create the training corpus\n",
    "training_corpus = train_df['text_tokenized'].values\n",
    "training_corpus[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.2 Training our model\n",
    "\n",
    "You can try to train your own model. More details: https://radimrehurek.com/gensim/models/word2vec.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>text_tokenized</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>10490</td>\n",
       "      <td>Take public opinion on revenge with Pakistan i...</td>\n",
       "      <td>[Take, public, opinion, on, revenge, with, Pak...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1875</td>\n",
       "      <td>21018</td>\n",
       "      <td>My goals are so big they scare small minds</td>\n",
       "      <td>[My, goals, are, so, big, they, scare, small, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>20289</td>\n",
       "      <td>@PanicAtTheDisco hey, y'all announced it like ...</td>\n",
       "      <td>[@, PanicAtTheDisco, hey, ,, y'all, announced,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2878</td>\n",
       "      <td>40051</td>\n",
       "      <td>@Christy_RTR @doge_e_fresh I'm despondent</td>\n",
       "      <td>[@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>432</td>\n",
       "      <td>10432</td>\n",
       "      <td>@sarah_urbina why do you even beef Sara you le...</td>\n",
       "      <td>[@, sarah_urbina, why, do, you, even, beef, Sa...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text  \\\n",
       "490   10490  Take public opinion on revenge with Pakistan i...   \n",
       "1875  21018         My goals are so big they scare small minds   \n",
       "1146  20289  @PanicAtTheDisco hey, y'all announced it like ...   \n",
       "2878  40051          @Christy_RTR @doge_e_fresh I'm despondent   \n",
       "432   10432  @sarah_urbina why do you even beef Sara you le...   \n",
       "\n",
       "                                         text_tokenized  \n",
       "490   [Take, public, opinion, on, revenge, with, Pak...  \n",
       "1875  [My, goals, are, so, big, they, scare, small, ...  \n",
       "1146  [@, PanicAtTheDisco, hey, ,, y'all, announced,...  \n",
       "2878  [@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...  \n",
       "432   [@, sarah_urbina, why, do, you, even, beef, Sa...  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## the input type\n",
    "train_df['text_tokenized'] = train_df['text'].apply(lambda x: nltk.word_tokenize(x))\n",
    "train_df[['id', 'text', 'text_tokenized']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from gensim.models import Word2Vec\n",
    "\n",
    "## setting\n",
    "vector_dim = 100\n",
    "window_size = 5\n",
    "min_count = 1\n",
    "training_iter = 20\n",
    "\n",
    "## model\n",
    "word2vec_model = Word2Vec(sentences=training_corpus, \n",
    "                          size=vector_dim, window=window_size, \n",
    "                          min_count=min_count, iter=training_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Imgur](https://i.imgur.com/Fca3MCs.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.3 Generating word vector (embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.24391222,  0.47471216,  0.60607815,  0.6043044 ,  0.30354637,\n",
       "        0.52612245,  0.15907517,  0.31419995,  0.6662399 ,  0.34005383,\n",
       "        0.23928903, -0.61682016,  0.26809034,  0.9209747 ,  0.5685452 ,\n",
       "        0.17348199, -0.22674245,  0.5710754 , -0.35357958,  0.28677517,\n",
       "        0.01680611, -0.82201445, -0.52795696, -0.3910232 , -0.8960669 ,\n",
       "        0.14471053,  0.00526888, -0.86870104,  0.08562313,  0.44321737,\n",
       "        0.36372796,  0.43732086, -0.3116185 , -0.46474335, -0.1590641 ,\n",
       "        0.22266756, -0.4517335 ,  0.0837204 , -0.29009768,  0.87018347,\n",
       "       -0.09992531, -0.6604294 , -0.29516825,  0.6239462 ,  0.08360109,\n",
       "        0.38526478,  0.45437548, -1.1009585 , -0.46836132,  0.12104732,\n",
       "       -0.3106531 ,  0.7249717 ,  0.1630029 ,  1.2424423 , -0.22231476,\n",
       "       -0.2038668 ,  0.8779491 ,  0.3843941 ,  0.04376961, -0.2400986 ,\n",
       "        0.19544724, -0.20151754,  0.18958779, -0.39637595, -0.4278381 ,\n",
       "       -0.20163637,  0.37303543,  0.12325621,  0.9450939 ,  0.48448637,\n",
       "       -0.7783745 ,  0.1773959 , -0.05355148,  0.3377519 ,  0.48870873,\n",
       "        0.65317255,  0.9982593 , -0.35841992, -0.2624745 , -0.62459135,\n",
       "       -0.10095175,  0.28501955,  0.24303195,  0.70502335,  0.5156783 ,\n",
       "        0.76913893, -0.16898501,  0.5011642 , -0.38643044, -0.18792278,\n",
       "       -0.13794756, -0.45504275, -0.92673576,  0.71840966, -0.14981405,\n",
       "        0.11581711, -0.8551522 ,  1.2451385 ,  0.4371222 ,  0.39893523],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the corresponding vector of a word\n",
    "word_vec = word2vec_model.wv['happy']\n",
    "word_vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('blessed', 0.9530322551727295),\n",
       " ('Online', 0.9481726884841919),\n",
       " ('bday', 0.940153956413269),\n",
       " ('free', 0.9350682497024536),\n",
       " (\"'you\", 0.9343327283859253),\n",
       " ('O', 0.9317790865898132),\n",
       " ('birthday', 0.9302517175674438),\n",
       " ('Be', 0.9287924766540527),\n",
       " ('.\\\\nA', 0.9231827259063721),\n",
       " ('inadvertently', 0.9227174520492554)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the most similar words\n",
    "word = 'happy'\n",
    "topn = 10\n",
    "word2vec_model.most_similar(word, topn=topn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.4 Using a pre-trained w2v model\n",
    "\n",
    "Instead of training your own model ,you can use a model that has already been trained. Here, we see 2 ways of doing that:\n",
    "\n",
    "\n",
    "#### (1) Download model by yourself\n",
    "\n",
    "source: [GoogleNews-vectors-negative300](https://code.google.com/archive/p/word2vec/)\n",
    "\n",
    "more details: https://radimrehurek.com/gensim/models/keyedvectors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load ok\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('glad', 0.7408890128135681),\n",
       " ('pleased', 0.6632171273231506),\n",
       " ('ecstatic', 0.6626912355422974),\n",
       " ('overjoyed', 0.6599286794662476),\n",
       " ('thrilled', 0.6514049768447876),\n",
       " ('satisfied', 0.6437950134277344),\n",
       " ('proud', 0.636042058467865),\n",
       " ('delighted', 0.627237856388092),\n",
       " ('disappointed', 0.6269949674606323),\n",
       " ('excited', 0.6247666478157043)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "\n",
    "## Note: this model is very huge, this will take some time ...\n",
    "model_path = \"GoogleNews/GoogleNews-vectors-negative300.bin.gz\"\n",
    "w2v_google_model = KeyedVectors.load_word2vec_format(model_path, binary=True)\n",
    "print('load ok')\n",
    "\n",
    "w2v_google_model.most_similar('happy', topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (2) Using gensim api\n",
    "\n",
    "Other pretrained models are available here: https://github.com/RaRe-Technologies/gensim-data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "load ok\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('birthday', 0.9577817916870117),\n",
       " ('thank', 0.9376667141914368),\n",
       " ('welcome', 0.9336150288581848),\n",
       " ('love', 0.9176183938980103),\n",
       " ('miss', 0.916450023651123),\n",
       " ('hello', 0.9158351421356201),\n",
       " ('thanks', 0.9150084257125854),\n",
       " ('merry', 0.9053248763084412),\n",
       " ('bless', 0.9027323126792908),\n",
       " ('wish', 0.9013165831565857)]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import gensim.downloader as api\n",
    "\n",
    "## If you see `SSL: CERTIFICATE_VERIFY_FAILED` error, use this:\n",
    "import ssl\n",
    "import urllib.request\n",
    "ssl._create_default_https_context = ssl._create_unverified_context\n",
    "\n",
    "glove_twitter_25_model = api.load(\"glove-twitter-25\")\n",
    "print('load ok')\n",
    "\n",
    "glove_twitter_25_model.most_similar('happy', topn=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7.5 king + woman - man = ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run one of the most famous examples for Word2Vec and compute the similarity between these 3 words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.7118192911148071),\n",
       " ('monarch', 0.6189674139022827),\n",
       " ('princess', 0.5902431011199951),\n",
       " ('crown_prince', 0.5499460697174072),\n",
       " ('prince', 0.5377321243286133),\n",
       " ('kings', 0.5236844420433044),\n",
       " ('Queen_Consort', 0.5235945582389832),\n",
       " ('queens', 0.518113374710083),\n",
       " ('sultan', 0.5098593235015869),\n",
       " ('monarchy', 0.5087411999702454)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_google_model.most_similar(positive=['king', 'woman'], negative=['man'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 7 (Take home): **  \n",
    "\n",
    "Now, we have the word vectors, but our input data is a sequence of words (or say sentence). \n",
    "How can we utilize these \"word\" vectors to represent the sentence data and train our model?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Clustering: k-means\n",
    "\n",
    "Here we introduce how to use `sklearn` to do the basic **unsupervised learning** approach, k-means.    \n",
    "\n",
    "more details: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic concept\n",
    "\n",
    "![Image](https://i.imgur.com/PEdUf54.png)\n",
    "\n",
    "(img source: https://towardsdatascience.com/k-means-clustering-identifying-f-r-i-e-n-d-s-in-the-world-of-strangers-695537505d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target words:  ['happy', 'fear', 'angry', 'car', 'teacher', 'computer']\n"
     ]
    }
   ],
   "source": [
    "# clustering target\n",
    "target_list = ['happy', 'fear', 'angry', 'car', 'teacher', 'computer']\n",
    "print('target words: ', target_list)\n",
    "\n",
    "# convert to word vector\n",
    "X = [word2vec_model.wv[word] for word in target_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word: happy \t cluster: 0\n",
      "word: fear \t cluster: 0\n",
      "word: angry \t cluster: 0\n",
      "word: car \t cluster: 1\n",
      "word: teacher \t cluster: 1\n",
      "word: computer \t cluster: 1\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "# we have to decide how many cluster (k) we want\n",
    "k = 2\n",
    "\n",
    "# k-means model\n",
    "kmeans_model = KMeans(n_clusters=k)\n",
    "kmeans_model.fit(X)\n",
    "\n",
    "# cluster result\n",
    "cluster_result = kmeans_model.labels_\n",
    "\n",
    "# show\n",
    "for i in range(len(target_list)):\n",
    "    print('word: {} \\t cluster: {}'.format(target_list[i], cluster_result[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Imgur](pics/pic6.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1], dtype=int32)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check cluster membership\n",
    "word = 'student'\n",
    "word_vec = word2vec_model.wv[word]\n",
    "kmeans_model.predict([word_vec])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0], dtype=int32)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check cluster membership\n",
    "word = 'sad'\n",
    "word_vec = word2vec_model.wv[word]\n",
    "kmeans_model.predict([word_vec])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 9. High-dimension Visualization: t-SNE\n",
    "\n",
    "No matter if you use the Bag-of-words, tf-idf, or word2vec, it's very hard to see the embedding result, because the dimension is larger than 3.  \n",
    "\n",
    "In Lab 1, we already talked about PCA. We can use PCA to reduce the dimension of our data, then visualize it. However, if you dig deeper into the result, you'd find it is insufficient...\n",
    "\n",
    "Our aim will be to create a visualization similar to the one below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Image](pics/pic7.png)\n",
    "source: https://www.fabian-keller.de/research/high-dimensional-data-visualization "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we would like to introduce another visualization method called t-SNE.  \n",
    "http://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1 Prepare visualizing target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's repare data lists like:\n",
    "    - happpy words\n",
    "    - angry words\n",
    "    - data words\n",
    "    - mining words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "happy_words:  ['happy', 'glad', 'pleased', 'ecstatic', 'overjoyed', 'thrilled']\n",
      "angry_words:  ['angry', 'irate', 'enraged', 'indignant', 'incensed', 'annoyed']\n",
      "data_words:  ['data', 'Data', 'datasets', 'dataset', 'databases', 'statistics']\n",
      "mining_words:  ['mining', 'Mining', 'mines', 'coal_mining', 'mine', 'miner']\n",
      "\n",
      "target words: \n",
      "['happy', 'glad', 'pleased', 'ecstatic', 'overjoyed', 'thrilled', 'angry', 'irate', 'enraged', 'indignant', 'incensed', 'annoyed', 'data', 'Data', 'datasets', 'dataset', 'databases', 'statistics', 'mining', 'Mining', 'mines', 'coal_mining', 'mine', 'miner']\n",
      "\n",
      "color list:\n",
      "['b', 'b', 'b', 'b', 'b', 'b', 'g', 'g', 'g', 'g', 'g', 'g', 'r', 'r', 'r', 'r', 'r', 'r', 'y', 'y', 'y', 'y', 'y', 'y']\n"
     ]
    }
   ],
   "source": [
    "word_list = ['happy', 'angry', 'data', 'mining']\n",
    "\n",
    "topn = 5\n",
    "happy_words = ['happy'] + [word_ for word_, sim_ in w2v_google_model.most_similar('happy', topn=topn)]\n",
    "angry_words = ['angry'] + [word_ for word_, sim_ in w2v_google_model.most_similar('angry', topn=topn)]        \n",
    "data_words = ['data'] + [word_ for word_, sim_ in w2v_google_model.most_similar('data', topn=topn)]        \n",
    "mining_words = ['mining'] + [word_ for word_, sim_ in w2v_google_model.most_similar('mining', topn=topn)]        \n",
    "\n",
    "print('happy_words: ', happy_words)\n",
    "print('angry_words: ', angry_words)\n",
    "print('data_words: ', data_words)\n",
    "print('mining_words: ', mining_words)\n",
    "\n",
    "target_words = happy_words + angry_words + data_words + mining_words\n",
    "print('\\ntarget words: ')\n",
    "print(target_words)\n",
    "\n",
    "print('\\ncolor list:')\n",
    "cn = topn + 1\n",
    "color = ['b'] * cn + ['g'] * cn + ['r'] * cn + ['y'] * cn\n",
    "print(color)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9.1 Plot using t-SNE (2-dimension)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyIAAAK8CAYAAAATVgvZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAARrwAAEa8B9/1LhAAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeZyP9f7/8cf1MR/GLGYYZBmMLaIF2aMZki2RJZV1ZBzRolRHIkYh/aIop2gzRSEqW8pyGBKdnNCXiigUiTSW1Mgsr98fn5mP+ZghFdfgPO/ndt3M5/q8r+t6f66Zc871/Lw3x8wQERERERFxkye/KyAiIiIiIv97FERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOuC8rsC55PjOKFAA2AfkJbP1RERERGRS5MXKA38x8x+ze/KXCwu6SCCL4T8O78rISIiIiL/E24AVuR3JS4Wl3oQ2QewfPlyKlSokN91EREREZFL0O7du2nRogVkPXvK2bnUg0gaQIUKFahSpUp+10VERERELm0aCvAnaLC6iIiIiIi4TkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFEZFLTHJyMo7jkJycnN9V+Ut27dqF4zgkJSXld1VERETkPFIQEXHZvHnzeOaZZwL2ZYeH5cuX/+XzZp9j06ZN/n1xcXHExcXlKnOuQopCg4iIiPxVCiIiLssriJxJfHw8MTExf1iuTp06rFu3jqpVq/6N2v05n3zyCQBbt2517ZoiIiJyaVAQyQeJiYk4jnPOzuc4DomJieft/G5ITk4mMTGRzMzM/K7KBeexxx7jvffeA07+bjdv3kyzZs0ICQmhVKlSDB8+nLCwMBo2bEhoaGie53n33XcZOHAgAO3atePWW2/lu+++Cygza9YsmjdvTokSJQgLC6N27dq8/vrruc41adIkrrjiCnr27AlAUlKSv445r9ewYUNCQkKIjIzM83q//fYbAwcOJCoqirCwMNq3b8+ePXv+2o0SERGRi4qCSD5ISEhg3bp1F+35z4fk5GRGjRp1yQeR+Ph4Xn/9dfbu3YvjODiOE9Da8dtvv3HPPfdQvHhxihcvTo8ePYiKiqJ27doB57n++uvxer0UKVKEAwcOMGbMGO68885cXbOyTZkyhc6dO1OhQgUABg8ezJYtW4iNjeWXX37xh4ZevXrxySefcMUVV/DSSy9x8803k5CQwJQpU/yhITw8nPvvvx8z46677gKgevXqpKSk5LpejRo1mDt3LlOnTg24Xrb+/fvzyiuvMHjwYN59912qVatGt27dzuUtFxERkQuVmV2yG1AFsO3bt9ulDLCRI0fmdzX+lpEjRxpgaWlp+V0Vvw8++MAaNmxowcHBVqRIEevQoYNt3brVzMwGDhxoJUuWzFXf48ePW2RkpN13333+fQcOHLD+/ftbmTJlzOv1WmhoqIWHh9u6dets3bp1tmHDBhsyZIgBVqpUKatcubKFhIRYmTJlLDg42CpXrmwVKlQws5P3KSwszEqWLGlhYWEWFBRkERER5vV6DbBnn33WAAOsZs2a1qRJEytSpIj16dPHVq5caYAVL17ctm/fbl6v17p06WKAFS1a1IKCgiwsLMwiIiKsfPnylpKSYgkJCXb11Vdbly5dzOPx+LcaNWpY8eLFDbAhQ4b4P+8vv/ziv15O3377rXm9Xnv22WfNzGzr1q3m8XjsySefDCh31113GWDTpk07V79KERGR82r79u3Z/99bxS6AZ+CLZVOLSD44teuU4zgMHz6c5557jooVKxIeHk5sbCxffPFFwHEZGRkMHz6c0qVLExISQlxcXK4yeZ0f4KeffuKOO+6gSJEiFC1alD59+rBgwYJcA5fj4uJo0qQJy5cvp06dOoSEhHDllVfm6nazY8cOevbsScWKFSlcuDCVKlViwIABHDp0KKBcfHw80dHRbNy4kaZNmxISEkLVqlWZMmVKQH1HjRoFgNfr9bcU5KcPP/yQm266ibCwMGbPns2LL77Ili1baNKkCXv37qVnz54cOHCApUuXBhy3aNEiDh8+TK9evQA4evQoTZo0YfHixSQmJrJ48WJiYmL45ZdfWL9+PQ0bNgxo7Th69CidOnVi/vz5vPzyq7Ro0ZdvvtnJvn1Gq1bw6ae+cqmpqWRkZDBhwgQWLFjAtddeS1paWkBdqlevzr59+zh69ChHjx6le/fuHDlyBIA2bdoQExND1apVeffdd3Echz59+vDSSy9Ro0YNjh49ynfffUexYsV45ZVX+Oqrr5g7dy5BQUF069aNzMxMzIxjx47lunfr1q3zXy89Pd2/lStXjurVq7N69WoA/vOf/5CZmUnXrl0Djr/99tv/3i9PRERELg75nYTO58YF2iKS/a12NsAqVKhgLVu2tPnz59ucOXMsJibGKleuHPCN+/Dhw81xHHvwwQdtyZIlNmbMGKtUqVKuFpFTz29m1qRJE4uIiLDJkyfbhx9+aP369bPy5csbYCtXrvSXi42NtVKlSlmNGjVs+vTp9sEHH1iLFi2sQIECAfdx1apVNnToUJs3b56tWrXKpk2bZlWrVrWGDRsGXLd3794WHh5u1atXtylTptjSpUvtjjvuMMBWrFhhZmbff/+99e3b1wBbs2aNv6XgfMhuEcj5mbOlpJh9/bXZ/v1m1157rVWpUiXg/n/77bcWFBRkDzzwgJmZVa1a1W6//faAc3To0MGuuOIK/+tRox63ggUL2W23fW3t2pl1724WF9fbQkJCLCoqyn/+7BaRtm3bmpnZjz+a1a1rBlOyvmGJNo/HDEb6WztyfobNmzf7948fP97fSgFY5cqV/e+dbuvdu7cdOnTIypcvb9WrV7dHH33UALvyyivtzjvv9JebOHGiZWZm2pQpU6xevXr+/VWrVrWdO3eamdmMGTPOeK3rr7/ezMzGjRtngB07dizgHn711VdqERERkYuKWkT+4rN6flfgvH64iyiIVKlSxU6cOOHfN2fOHAPs448/NjOzlJQUCw0Ntf79+wecK/th7kxBZMmSJQbY7NmzA469+eab8wwiQUFB9vXXX/v37d+/3zwej40ZM+a0nyktLc0++ugjA2zDhg3+/b179w4IHWa+7kvFihWzfv365arz+e6alVcQWbXKrF07M8fx/TcCjhk41qXLsFzHx8bGWp06dczM7PHHH7fChQvb0aNHzczs4MGD5vV6bezYsWZmduCAWXh4Y4PrDdLMcdIM0gx6msdTzAD7/PPPzexkEJkwYYKlpppdc012XaZl/Q9b2azXI7Neh9p//3uyXsuXL/c/6D/88MMG2JIlSywoKMhKlChhgCUlJVmpUqUMsClTptj69eutffv2ZwwNZcuWtV69evlfHzp0KOB+PPbYY/6uYvXr1zczX5e27OutX78+15bdve311183wL755puAcyYnJyuIiIjIRUVB5K9t6prlgkzLZMmOJTz50ZM8teYpdh7amavMjTfeiNfr9b++6qqrAPyzDG3evJlff/31L3Vj+eSTTyhQoAAdO3YM2N+lS5c8y1etWjVgCtiSJUtSsmTJgBmPTpw4wdixY6levTqFCxfG6/XStGlTALZt2xZwvpCQEJo1a+Z/XahQIS6//PJcMyjlh1dfhbg4eP9932O+zyHAmDu3NMOHB5YvVaqUf1B2jx49OH78OHPnzgVg9uzZpKen06NHD1JToWVL+OWXA8BqwIuZF/AC08nM9J1j8uQZAecvVqwYs2fD55//Uc09DBhwchrgWbNm+f9+tm/fDsCXX35JqVKlOHz4MGFhYaxYsYIff/wRgGrVqlG3bl0iIyPPeJVrrrmG+fPn+1/nHJAOEBYWBvi6gW3ZsgWAxo0bEx4ezo4dO6hbt26urVq1agA0aNAAj8fD22+/HXDOWbNm/dGHFxERkUtAUH5X4FL3/tfvc/fiu9l9ZPfJnVkPmSt2rqB5xeaA7wE0p0KFCgFw/PhxAPbt2wfAZZddFlDu1Nd52bdvH0WLFmX37t2MGjWKNWvW8OOPP/ofQnPOYgTw888/Ex0dzcKFC7nvvvv47LPPSEtL4/McT8dDhw5l0qRJZGRk8Pzzz7No0SJWr15Namoqr776KrfccgvBwcEAFC1alH379jFkyBAWL17ML7/8gtfrpVy5cv7z/fDDDwAsWLCATp06BdQnPj6e5cuXs3v3bgoUKADASy+9xL/+9S+2bdtGWFgYHTp04Omnnw64jz/99BODBg1i0aJFeDwe2rdvH3DuTZvgH//w/XwyhAAUBRzgR8aMgWuvhewM9+OPP/qvUbFiRa677jpmzJhBnz59mDFjBnFxcZQrV46XX/adH6KAksCkHOcfC6wAKjNv3mpeeunkO47j8NJL4PHAmScQO8b69Y/zzTff88MPP/DKK69w3XXX8fHHH1O+fHnAFyjLlCnDnj176NOnDy+//DKFCxcmNTWVTZs2sXfvXn9o7NatG/Hx8XTs2JFy5crRv39/UlNTef311ylevLh/bMnMmTPZvXs34eHhNGrUyB8atmzZQuvWrQEoUqQITz/9NHfffTc//fQTbdq0ISIigr1797Jq1Sri4uLo1q2bf4asESNGkJmZSb169Vi6dCmLFy8+0wcXERGRS4RaRM6j+Vvn035We747kvc3/61mtGLZN8vO6lylS5cGYP/+/QH7T319umMPHTrEd999R7ly5Zg4cSJLliyhXbt2ADzyyCO5jjl69CjdunWjR48ezJ8/n4IFC/LJJ5+wcuVKwPetdePGjQHfmhL169fn8ccfB+Df//43Tz75pP9cmZmZxMbG8sEHHzB27FjmzZtHaGgoW7du5aWsp/AyZcoA8PLLLwfU4/Dhw7z99tskJCT4Q8gjjzzC3XffTYsWLViwYAFPP/00H374IW3atCEjI8N/bKdOnVi0aBFjx45l9uzZBAUFce+99/rff/5538N+YAgBCAWuBeYAGUyc6Nu7e/du1q5dG7BSea9evUhOTiY5OZl169b519WYMsUXJqA1sBUoD9TN2mKBI8AvHDyYyiefbA64+tdf/1EIAV/r7y5SUg4yY8YMhg8fTmpqKoA/iISGhlK4cGGKFi3K559/TlBQEBEREQAMGTKExMREoqOjcRyHWbNm8c477zBkyBDS0tL45z//yZNPPkn9+vXp0aOH/6ojR44kJSWFZcuW0atXLzZu3AhAzZo1A9Yb6d+/PwsWLGDbtm307NmTtm3bkpiYSHp6OrVq1fKXmzp1Kn379mX8+PF07NiRbdu28dZbb/3RhxcREZFLQX73DTufG/k4RuR42nGLeirKnETHSCRwi/X1t/eM8li5Z8oZYMOGBY5H2LlzZ0A/+UOHDp3zMSJt27b19/3PHtcRGxtrJUuWzDWuo3z58laoUCH/uI7IyEiLi4szwEaMGGFmZj179jTArrnmGqtataqZ+caIRERE5DkWJTIy0kqUKGHp6ek2duxY3z3xeGzXrl3+cpMmTbICBQrY999/778vHo/HRo0aFfBZ1qxZY4C99957Zma2dOlSA2zmzJkB5Vq3bu2vS3i4ZY27yGv7V9a9CTLwWqVKVS06OtqKFy9ue/fuNTPfWIbY2FgDzHEccxzHJk+ebGZmhQqZwZsGVxk4WVsZgwcM5hiE5jkmwzeO44DBPwyqGniz3itssCfHGJGCpx3XkT1978qVKy02NtauvPJKAywoKMguv/xyA6xw4cJWr149mz9/vg0dOtS8Xq+VK1fOChcubIUKFbKSJUtapUqV7LXXXvPfu0aNGpnX67WQkBALDg62cuXK+cef5Jy+V0RE5H+Nxoj8tU0tIufJO1+9w8+pP2Pk+rrdL9My+f7o92d1vsjISB544AFeeuklHn74YZYtW8bYsWP9LQp5OXL8CM//53leOvwSxaoVo2efntzQ5gbKlStHUFBQQBeYPxrX4TgORYoU8Y/raN26NR9//DEAUVFR3HXXXaxduxaAsmXL5hpPUrZs2YCWBPB1K/vpp5/48ssvqVGjBuDrkpaYmMh///tfwPeN+U033UR0dDQAy5YtIzMzM9fUsA0aNCA8PNw/Ney6desoUKAAnTt3Drhm9piazEw4pUdaDhuAh4EaQEXAYffunezdu5cpU6ZQpkwZ5s+fzw033ICZ0aBBA8yM2rVr+1uoHGcN0ANoDrwD3AIcB54DBgLVgLLA1cycuY7hWYNRrrvuOho2TAGCgSeB5cD1QAZwHZCeVcflOE4ZHMfB6/VSuXJlnnjiCcwsoMUhOTmZTZs24fV6SU9P9y8w+N5779GxY0d27drF2LFjefXVVylTxne+oKAgihUrRps2bbjxxhv955o/fz6dO3fG4/FQqFAh4uLi/C1Y2d2yRERERM6WxoicJx9/9/E5P2diYiJmxiuvvMLkyZNp0KABCxcupGbNmnmWL/NMGX5L+w0HBzqAvWqs+HAFBbwFiGsWx3WNr/N3p8oei5KtaNGiuc7n8Xj85Z5//nm+/fZbPv30U4YPH87NN9/MzJkzqV+/Pl6vl99//91/XGZmpr9rWU4FCxYEfAOg27Vrx8CBA5k2bRpJSUkkJSWxevVqvvzyS8aPH+8/5sCBAwBUqVIlz8/8888/AyfHxeScAABOjqnxeCAqCrKKn+JhfF2pNgIFcRzYty+D66+/kunTp9OpUycGDRpErVq1WLlyJR5P7jxfocInbNsWCWT166JjrjIQh9ebTufODbn99oY88cQTAAwbBvPm5RxTUgxfcNkMbM/a14iaNW/k0KHl7NmzJ897kW3hwoWkpaVx3XXXBawH06pVK//PPXv29HcrO50SJUowc+bMXPvNTh+2RURERE5HLSLnSYZlnP7NZkDiyZcPL32Y0aNHBxSJiYnBzIiPj/fvK1CgAKNHj+bHH38kNTWV5ORkatSogZmRmHjyhHXuqIOT6JCa5hszYBgWYnACqA0ZwzKoMqgKP//8s39QfLbk5GTatGmTq8q7du2ievXq/tfFixdnwIABAGzYsIE333yTevXq5fpGPikpifbt2/tna8p5ncGDBwO+gfoFChTgX//6F5s3b8ZxHObOncvUqVOJiYkJeGCOiooCYOnSpaxfvz7Xln0fssfFnLrIX84xNXk/d6cCq4Bb8f3XI502bdIpWtRo0aIFq1evZtu2bezevZuEhIQ8QwhAfHw9fLNv9QAWAYfzLFeqFJySlahbF9q0eRFfi0wwMA9fCAGoDRjXXhvE1VfneUq/vXv3smzZMu6//34Af6uLiIiIyIVAQeQ8uarkVWdd9sqSV56z62ZaJg8ueRAgsFvYRnzP2L8CX8PUJ6by4osvnrZl4VyKjY1lz549/q5c2d566y1Klizp75YFULlyZVq2bMnTTz/N3Llz6devX8DD/o033ojH4+G7777Lc2rYihUrAtCoUSMyMjJ45513Aq6ZPcuTGWRNBHWKFHzdoJ7AN9Wul8WLvXi9XiZPnsyhQ4f8rS7Z3cXyMmRILNdfPwf4Hl9rSAmgBfB//jJhYVC2bO5jn3/+eT74YCCXX34ZjlMAX9csXze6AgWO068frFyZO8Ccavr06bRp08a/Sv2Z6isiIiLiNnXNOk+6X92dh5c9TGp66mnLODhEBEdwa41bz9l1V+1axY5DO3K/URAoAHyNr3dPGJSsWjJXl6zzIT4+nkmTJtGpUyfGjBlDdHQ0b775JsuWLWPq1Kn+2bCyDRw4kA4dOuD1eunbt2/Ae5UrV2bIkCHcc889bNu2jdjYWIKDg/n+++9ZtmwZCQkJNGvWjBtvvJEmTZrQv39/Dh48SNWqVZk9e7Z/rYvp02HatLxqG4kvn98N9GL4cOjQIbBEeHg44GtxOB3HgRUrujByZBcmTjzGr78mA0OA1ng8e7jjDg+7duV97KxZs7jhhhtYvnw56em+0LFx406GDIEHH4SnnjrtZQM88sgjfPjhhyxZsoTWrVuzd+9errzy3IVeERERkb9DLSLnSWRwJE80e+K07zs4GMbTNz5NYW/hc3bdL3/6Mu83agL3Zf1bCEgDp5iTZ5//cy00NJRVq1bRsmVLHnnkETp06MDnn3/O9OnT+Uf2Qh453HTTTRQuXJgOHTrkuU5K9iD91atX07VrVzp06MBTTz1F0aJFAxZifPfdd2nbti1Dhw7ltttuIz09ncmTJwNw+o8dCjTFt9hLHRIScre6XH755cTExPDKK6+ccXxEgQIwejTs2xfGjBntaN++P7CPTZt+ZsYMCA0t5J9yN6fffvvNP7YlKAhuvBGOHfOlpsI5/lQKFcr7+FM1btyYsLCwM05sICIiIuI2tYicR4MbDSbDMhi+YjhpmWm+QeP4ukwVCirEhJYTSKiTcE6vGeQ5w680FN/QB8DjeLi26rX+cR05JSUl5Xl4cnJywOv4+PiAMSzZEhMTA8asgG/MxvTp089c+SwrVqwgNTWVu+6667Rl/s7g6ldeMRLOeNufwdcdqhUPPtiXe+8tzcGDB9mwYQMZGRmMGzeOiRMn0qlTJ5o3b85dd91FiRIl+Oqrrzhw4ACjRo1ixIgR7N+/n2bNmmXNRrWHL754jlq1anHVVSUAqFGjBi+88AKzZ8+mcuXKhIeHU61aNVq3bs1TTz3F2LFjqV+/PitWrPCv3p5TjRo1SElJ4cUXX6Ru3boEBwdz1VW5uwSGh4fz5JNPcu+999K5c2e6d+9OeHg4mzZtIjg4OGBtFRERERG3KIicR47j8M/r/kmfWn2YtmkaG3/ciINDvTL1iK8VT9HCuWem+rualG9yVuUyLZMm5c6urFu++eYbvv32Wx544AHq1KnDDTfccF6u8+23f1SiDrAeGMX779/HwoVHKFGiBHXq1PGHow4dOrBs2TKeeOIJf/exypUr+weGN2jQgOeee44HHniAlJQUSpYsScuWLf0zY4FvUcFt27aRkJDAsWPHiI2NJTk5mREjRnD48GGeffZZjh8/TmxsLEuWLKFSpUoBtUxISOCTTz7h0Ucf5fDhw1SoUIFdefT3Sk5O5t577yUxMZHFixfTvXt3vF4vV1xxBY899tifvn+7du2iYsWKTJs2Lc8gKiIiInI2nEt56k3HcaoA27dv3+7KoOwLRVxSHKt3rw4crL4S32RQib5uYd4CXvY8sIcSoSX8RdLS0ggKCvIPbnZbfHw8M2bM4JprruGNN9447bTEf5fv840kYOqy0+jf37dK+sUsOTmZZs2asXLlylxrufwVCiIiIiKBduzYkd09vKqZ5TFYV/KiMSL5bMeOHfTs2ZOKFStSuHBhKlWqxIABAzh06FBAufj4eKKjo9m4cSNNmzYlJCSEqlWrMiWPp+SeYT1xpjq+iZ8mAZ8BB4DQk2NTRl4zkpJhJXnhhRf45z//SZkyZShUqBAbNmzAcRzmz5+f67zZdcjIOMPUxH9DUlIS6enpfPbZZ+cthPxZOWYOFhEREZFzSEEkn/3www+UK1eOiRMnsmTJEkaMGMG///1v2rZtm6vs0aNH6datGz169GD+/PnUq1ePAQMGsHLlSn+ZL7/8koHdB1KrfC2q3VUNbgD+A+wFgqBskbK81ektul3VDYAxY8bw9ddf89JLL/Hee+9Ro0YN6tWrx9SpUwOuffjwYd5++20SEhJyzXJ1MSpTxjez1el4PBAdDTff7F6d/qqZM2dSvXp1/xiRBQsWEBcXd8bWj6VLl9K2bVtKly5NSEgIV155JRMmTMgVMn/77TcGDhxIVFQUYWFhtG/f/g8XUBQRERE5K2Z2yW5AFcC2b99uF5IvDnxhSRuT7LUNr9nGfRsD3ktLS7OPPvrIANuwYYN/f+/evQ2wFStW+PcdP37cihUrZv369fPvu+OOO6x48eL266+/mpnZpn2b7PFFj5vjcQyw9Ix0MzMDDLCyZctaTEyMhYWF2fXXX29btmyxadOmmcfjsV27dpmZ2bvvvmsVK1Y0wMLCwqxevXo2f/78gDqPHTvWqlWrZgULFrTSpUvb4MGDLTU11V9m586dBtiUKVPsscces1KlSllERIS1a9fOvv/++4B78Oabb1qtWrUsNDTUwsPD7corr7QpU6YElElOTrbmzZtbWFiYhYSEWMuWLW3z5s0BZdLT023YsGFWqlQpK1y4sMXGxtqWLVsMsLvvHmlRUWaOY+ZbVeTk5jhmISFma9ee/e80vyxdutQcx7EOHTrY+++/b0lJSVaxYkUrXbq0xcbGmpnZypUrDbCVK1f6j3vxxRdt/PjxtnjxYluxYoX9v//3/ywsLMyGDBkScP4ePXqY1+u10aNH25IlS+yhhx6ycuXKGWDTpk1z74OKiIhcwLZv3579bFXFLoBn4Itly/cKnNcPd4EFkY37NlrstFgjkZPbcCz6lmgrX7m8BQcH+wMCYDNnzvQf27t3bwsJCcl1zoYNG1qrVq38rytWrGi9evXKVS4mJsZ8udMn+xqVK1e2+fPn25w5cywmJsYqV65sv/zyixUrVsyGDRtmzz33nAEWHh5udevWtQ8//NDGjh1rkyZN8p/rtttus5CQEBs1apQtW7bMnnvuOYuIiLBOnTr5y2QHkQoVKtgdd9xhixcvtqSkJIuKivI/MJuZffTRR+Y4jg0aNMiWLVtmS5YssUmTJtm4ceP8ZRYtWmQFChSw9u3b27x582zevHnWqFEji4yMtO+++85fbvjw4eY4jj344IO2ZMkSGzNmjFWqVMkAGzlypO3YYdapk1mBAhYQRFq1MtsYmA8vWI0aNbKaNWtaZmamf99///tfA84YRHLKzMy0tLQ0Gz16tEVGRlpGRoaZmW3dutU8Ho89+eSTAeXvuusuBREREZEcFEQURHJ/uAsgiIwcOdIAW793vYWOCTUn0QkMIo0wPFiBGwrY+Bnj7dNPP7V3330314Ne7969rWzZsrZy5UobOXKk/2ExNjY24EE+ODjYHnrooVz1qFmzZskiuvAAACAASURBVJ5B5IUXXvDvmzNnjgH28ccf24MPPmiXXXaZhYWFWdOmTQ2wxYsX5zrv6tWrDbDXX389YP+MGTMMsI1ZT/TZQSRnXc3Mnn76aQNs7969/tdFixY94z2tXLmyNW/ePGDfkSNHLCoqygYNGmRmZikpKRYaGmr9+/cPKDdu3Dh/EMm2Z4/ZggVm8+aZffPNGS993mT/naSlpZ2x3JYtZuPHmz3xhFlSUrp5vV4bMWJErnIVK1Y8YxD54Ycf7B//+IeVL1/egoKCAgLwvn37zMzs9ddfN8C+OeWmJCcnK4iIiIjkoCDy1zaNETnPEhISWLt2Lb3e60VqemrgTFYAW4BrwK43Jvw4gVp1ahEZGXna8yUnJzNq1CgyMzPzfL906dJs2bmF7u92p+wzZbls/GXc8PoNHDh0IM/y2QvnAf41KL777jsGDBjA/v37OXbsGEFBQcTExNAqj5HbH374IQULFqRLly6kp6f7t5YtWwKwevXqgPKnjn3JeU2AevXqcejQIXr06MGiRYs4fPhwQPnt27fzzTff0L1794DrhYSE0KhRI//1Nm/ezK+//krXrl0Djr/99ttzfYayZX1jQTp0gFNmyL1g7N4NLVrAlVfCQw/BY49BfPxB0tLS2LixJHbKn1VeC0Fmy8zMpH379ixatIjhw4ezYsUK1q9fz7BhwwA4fvw4APv27cvzXGc6t4iIiMjZUhA5z6Kjozle6jhfHfyKTMsjPJwAPL51PfYd28eCbQuYNm3aX7qWmeEt7+XDDz7krc/e4odffuDArwdYsXkFP+37CYDDxw+f9vhChQoBvgfRypUr+0PCmjVr6NevHx5P7j+XAwcOcOLECUJDQ/F6vf6tZMmSAPz8888B5YsVK3baawLExsYyZ84cvv/+ezp27EiJEiVo0aIF//d//+e/HkDfvn0Druf1elm0aJH/epfSQ/SePdC4MaxYceo7xQEvCxce4KGHAt/Zv3//ac/3zTff8N///pennnqKfv360bRpU+rWrZtrEoLSpUvnea4znVtERETkbGlBw/MsMTGRUaNGnVyyIhFoChTEN63u78AmfM+Uu6H7+O6c+PUEABMnTqRhw4ZUr14d8M2aNWrUKCCwJSM2NhaAx5c/ztdHv4bfgLH4VlKPwTd1rxc4AbfOuZWlPZaeVd07derE5s2byczM9C/ad6qoqCiCg4P56KOP8ny/TJkyZ3WtnLp06UKXLl04duwYycnJDBkyhNatW7Nnzx6ioqIAePLJJ2nRokWuYwsWLAgEPkTnnAr4Qn+I3rlzJ/fffz+rVq0iKiqKvn378vXXw/nhBw9wHBgKLAN2AWFZ21s880wi3bs71Knj+5vbuXMnERER3HLLLSxZsgTw/T01aNCA3377DfDNhOY4Dv/617/Ytm0bkydPBuDOO+/ktddeo0GDBng8Hm666SaCg4PZuHEjALNmzQLgp59+wuPx8MILL/gXehQRERE5W2oRyQ+bgO1AS6ALvpEsyb59EdERvPjii4BvgcFGjRrx448/AhAaGuoPBGvWrGHdunXUrl0bgKOpR3niH0/AVqAuEIkvkHwBhGdtwPJvl/Pp3k/PqprZq4SXKlXqtC0JrVu35vjx4xw5coS6devm2v5KEMkWFhZGu3bt6N+/P/v27ePnn3+mWrVqxMTE8MUXX+R5vauvvhqAq6++mtDQUN5+++2Ac2Y/RF+oOnbsSPPmzZk3bx633HILI0eOZNas17Pe/R34BRgOvA+8iC9p7gDa8Oiji3njjTd44YUXAPjqq6+oUqUKjz/+OACLFi1iwIABXHHFFVSoUIHx48cDMGLECGbMmOH/HW/evJmWLVtSqVIlunXrxo4dO9i0aRPPP/88Dz/8MIsXLwZ83QRDQ0Pp3r27OzdHRERELilqETkPDqUeYkfKDjyOh7SMtNwFDOiJr5UC4MqTbz3Y4kH6X9efhIQEfv/9dy677DJmzpxJUlIS4Pu2G6BBgwYEBQWxYcMGAO558h4ydmVAPL5n02yr8YWcHJEzaZPvXMOGDSMhIeG0n2P9+vUA7N27l86dO9O9e3fCw8PZtGkTwcHB3HvvvcTFxXHHHXfQpUsXBg8eTP369fF4POzatYvFixfz1FNPcfnll5/FXfMZMWIE+/fvp1mzZpQpU4Y9e/bw3HPPUatWLUqUKEFMTAxVqlRh1qxZnDhxgq5du1K8eHH279/P2rVrKV++PIMHDyYyMpIHHniAMWPGEB4eTsuWLVm/fj2vvvrqWdclPzz44IP06dMHgBYtWrBgwQp27ZoJ9AEigFdylM4AWgHFgA0sXdqR77+vwm233cbkyZMpXbo048ePJzk5GYDevXuTlJTEo48+yrx58+jXrx+7d+/myJEjPPLII1SoUIF+/frx4osvcuutt/LGG28wdepUQkNDefnllxk8eDBt2rThrbfeokmTJqxZs8b/NyEiIiLyZymInEM7Unbw+KrHmf3FbE5k+LpXFVzn6yoUGhTKr+m/+gpW4WQIycHzpYdZi2YxdvtYjhw54t+/bdu2P7z22pVrfc+p5fB9WR6NrxXEgMysDfA4Hvb8cuYF6Q4cOMCyZct44IEHqFOnDkOHDuXpp5+me/fueL1errjiCh577DF/+RkzZvD888/z2muvMWbMGAoVKuQf3P5nx2Q0aNCA5557jgceeICUlBRKlixJy5YteeKJJ/xloqOjWb16NWPGjCEhIYHU1FRKlSpFw4YNue222/zlEhMTMTNeeeUVJk+eTIMGDVi4cGG+r9puBitXwoIFcOyYb+HEQ4d87910000BZStUuJJduzbm2PM2MAHYBhzJsb8TVatO4Ysv4JlnngHwL2gYFxeHmfHNN9/w2muv8emnn9KjRw/mzJlDxYoVGTp0qL/VJDuYRkdHs27dOvr27cuUKVOoVKkSo0aNYvr06URERPD222/TtWtX+vfvf87vj4iIiPxvUBA5Rz7/8XPiXo/LNRj8RLovkBT2Fj4ZRPL6AnkbZL6dydW9r2b0qNEUL14cj8dD27Zt/QO5zyT1SKrvuTT7eX39KQWaAC18A9pDvaHZ0xsHiImJwcyIj49n6NCh1KpVizfeeIOaNWvSpUuX017b4/EwaNAgBg0adNoy2ec+VfZDcrabbrop18N4Xho1asSiRYvOWKZAgQKMHj2a0aNHB+zPqx5u2boVunSBL77I+/3g4MDB/MWLF8I3NgRgIXAb0BsYiW9g0QngRmAbpUqtYtq0b/1drnr37h1wruxQuHfv3jz3n7ovZ7m+ffsyYsQIpk+fzj333MOUKVOoX7++v2ugiIiIyJ+lMSLnQEZmBrfMvoUjx4+ctszB1INcXuxkFyUn6z/Zqu2rRpUqVUhKSqJt27bUr1+fa665hpSUlNOe8/PPP6d9+/YULVqUr//7NRQA2gH9sraq+AasdwR2AqPBnjNCPg8JOE9SUhKO47B69WpuvfVW5s2bx1VXXcVnn33Gb7/9RpcuXYiOjqZw4cJUq1aNRx99lNTU1MB7kJHB8OHDKV26NCEhITRv3pytW7fiOI6/O1le9S5cuDDXXXddnoPdJ02aRExMDMHBwdStW/e0A+IvFrt2QdOm8OWXpy/TtStkZJx8HRYGhQuD4wDMwteclgS0BeoDtfEFlf+wdu2NDB482B8sTp3iOXugftmyZfPcf+q+nOWioqLo2rUrU6dOZfv27axcuVKtISIiIvK3KIicA+9vf59dh3flXiPkFNtTtgPQqFwj4mLiiI2J5f4G97P17q1cEXkFQUGBDVTTp08nI+dTKSenu123bh2NGzcmJSWFl19+mfvuu883ZGAx4ABlgRAgDfgIqA1ON4dC5QsxbfQ0Vq5cmat+3bt3p2LFisydO5dx48YBvvU9atWqxZQpU/jwww8ZNGgQr732mn8cQ7aRI0cyduxYevXqxfz582nZsiXt27fPdY0NGzYE1Pudd94hKiqKFi1a8Nlnn/nLvfrqq9x///00a9aMefPmER8fzx133MGh7D5MF6FRo+DgQXKt+ZHTkiXw/vuB+yIjITgYfLMPnNqIORuA6OiunDhxgkOHDvlbpvIaqO/xeGjQoEHA/rlz5waElo8//pg9e/bQqFGjgHIDBw5ky5YtJCQkEBERkeeaLCIiIiJnLb9XVDyfGy6trH7nvDsDV0vPucVmrVid/Rps2LBhuc4xZcoUA+z++++35cuX27hx46xs2bIWGRlpvXv39pebN2+eAVaxYkWrUKGCrV271szMTpw4YQ0bNzQ8GCUwemFUyrp2aYxHsdAxobZy+0orVqyY9evXz3/OadOm+a99JpmZmZaWlmbTp083x3Hs4MGDZnZyFfMBAwYElJ8wYUKuVcybN29u1atXt99//92/Lz093apXr24dOnQwM7OMjAyLjo62Vq1aBZxv1qxZBgTcj4tFSopZoUJmvhiS1zYya0XWNGvd+uRxvXv3tgoVKtiaNWZFikzJKnO/Oc5yg3EGZc3rjbQePXr7j8n+fZYrV84eeughW7p0qY0ePdq8Xq/Fx8f7y2Wvdh8dHW0333yzLVq0yKZNm2alSpWyqlWr2okTJ3J9jtq1axtg99577/m8XSIiIhcVraz+1za1iJwDv5z4Bc/fvJX9+vVj2LBhzJ49m5tvvpnFixezcOFCIiIiOJFxgmfXPUvjVxvzz93/pNT1pdi5cye7d++mcePGpKen4zgOy5Yso07tOjgHHXgT31ITDjjVHDpd1YlPEj4hrkocl19+uX8l85w6duyYa9/Ro0cZMmQIlStXplChQni9Xnr27ImZsX27r4UnexXzW2+9NeDYU8eVpKamsmrVKm699VY8Ho9/VXQzo0WLFv5V0ffs2cOePXtyrYreuXPnXK1GF4utW+H338+ubNZEaAGuuw727+9Hx47DKFx4Nh7PzURHL+addxZSpkwEp6xFCPgmEfj666/p2LEjEyZMoF+/fv6pfXMaOnQoVapUIT4+noEDB1KnTh2WLFkSsFZNtuzfsbpliYiIyN91cT7VXWDKhpclkzxWTQdolrVleXXDq9xZ+85cxTweT54Dq99Y9QbtZ7Zn5tKZODi+7l918U3LmyWvB8ajx45ye8/bWb9mPZsXbuaysJMDkgsVKpTnAPjsRQBz6tOnD8uXL+fxxx+nVq1ahIaG8umnn3L33Xf7z5G9inn2aurZTh0EnZKSQkZGBk888UTALFg5ZWZmnnZV9KCgIP+ChhebPBalP0Ui2ateOieHDvmnbQYIDvbw7rujgcC/kU6dduV5xjJlyjB//vw/rFvBggV55pln/LNtncmiRYto0qRJvs88JiIiIhc/BZFzoHet3kz8z8Q/LBfiDaHzFZ3P+rzbDm6j7ZttSU33DQy37DEowfjGgdSDiAYRzOo8i+KhxQOODQ8Op0RoCQoWKBgQQs7EyfkEDBw/fpz58+eTmJgYMCPW5s2bA8plB5gDBw6ccRXzyMhIPB4Pd999N7169cqzDh6PJ2BV9JzS09P5+eefz+qzXGiuuAJCQiBrUfPTchxo2NCdOp2t33//nQ0bNrB8+XLWrl17VuFGRERE5I8oiJwDtUrVok2VNnyw44Mzlruv/n1EBEec9XnHrx3Pr2m/5n6jIFAe2A9Hih5hTcYaRtcdnbvc3/T777+TkZGRq8Ul57f0AFdddRWhoaHMmTOHZs1ONv/MmTMnoFxoaChNmzbl888/p06dOnhO00wQHR1NuXLlePvtt7nzzpOtR++88w7p6el/81PljyJFoGdPmDr1zOXM4K673KnT2dq3bx+NGzcmMjKSRx99NM9JCERERET+LAWRc2Rm55m0m9mONd+tOdmFCvw/97qmF6Obn31YOJ5+nDc3v3n6Aq2AacAMeP6b57mxwI0cPHiQDRs2kJGR4Z/16u+IiIigYcOGTJgwgdKlS1O8eHFee+21XOtQFC1alPvvv5+xY8cSHh5OixYt2LBhg38V85yB45lnnuH666+nVatW9O3bl9KlS+eqt8fjYeTIkSQkJNCnTx9uv/12duzYwbhx4yhSpMjf/lz5ZeRIWLQIfvjh9DNnde0KrVr9vevEx8cTHx//h+VOt7bLXy0nIiIi8mdosPo5EhEcwYpeK5jZeSZNKzSlaHBRiocUp0P1DiztsZSkDkkU8OQxovg0Dvx6wN8lK09lgH8AheHoe0dp2bIlgwYNYvPmzVx//fV/+/NkmzlzJtdeey1333038fHxlCpVikmTJuUqN2rUKIYOHcrrr79O+/bt+eCDD/wtJxERJ1uB6tSpw/r164mKiuK+++47bb379u3LxIkTWbFiBR06dGDatGnMnDmTokWLnrPP5rbSpeHjj6Fx49zvBQXBPffAjBmBY0RERERELlXOpfxNp+M4VYDt27dvp0qVKvldnT/lp19/ouT4kn9cEPA4HtIeS8PjXFi5cu7cudx6662sXr2apk2b5nd1LigbN8LChXDsGERHw+23Q8mz+3WLiIjIBWbHjh1UrVoVoKqZ7cjv+lws1DXrAlU8pDh1Stdh075Np5+RC1/Xr9ZVWud7CPnPf/7D+++/T4MGDQgODuazzz5j3LhxNGzYkCZNmuRr3S5EtWv7NhEREZH/VRfWV+ji5zgO99a/94whBHwzad1d726XanV6YWFhrF69ml69etG6dWsmTZpE165dWbx4ca7ZuERERERE1CJyAet1TS+WfbuMtza/FTAAHk4Ogh/UYBBtqrTJx1r61KxZk+Tk5PyuhoiIiIhcJNQicgHzOB7euOUNxt0wLtdaIOUjyvNC2xd4ttWzanEQERERkYuOWkQucAU8BRjSZAiDGw1m7fdrOXT8ECVDS9IwumG+jwsREREREfmrFEQuEt4CXmJjYvO7GiIiIiIi54SrX6k7jhPtOM7zjuOscxznN8dxzHGcmDzK2Wm2Wm7WV0REREREzg+3W0SqAF2Bz4CPgJZnKJsETD1l39fnp1oiIiIiIuImt4PIajO7DMBxnATOHET2mtkn7lRLRERERETc5GrXLDM786IYIiIiIiLyP+FCnnZpgOM4v2eNJVnhOE7T/K6QiIiIiIicGxfqrFkzgEXAD0AF4GFgheM4N5pZcl4HOI5TDCh2yu7y57OSIiIiIiLy11yQQcTMeuZ4+ZHjOPOBLcBooMlpDrsPGHm+6yYiIiIiIn/fhdw1y8/MfgHeB+qdodhzQNVTthvOf+1EREREROTPuiBbRM7ATvuGWQqQknOf4zjnvUIiIiIiIvLnXRQtIo7jFAHaAZ/md11EREREROTvc71FxHGcLlk/Xpv1bxvHcX4CfjKzVY7jPARUA1ZycrD6Q0ApoLvb9RURERERkXMvP7pmzTnl9QtZ/64C4oBtQMesLQI4CnwM9DUztYiIiIiIiFwCXA8iZnbGgRtmthBY6FJ1REREREQkH1wUY0REREREROTSoiAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirlMQERERERER1ymIiIiIiIiI6xRERERERETEdQoiIiIiIiLiOgURERERERFxnYKIiIiIiIi4TkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirlMQERERERER1ymIiIiIiIiI6xRERERERETEdQoiIiIiIiLiOgURERERERFxnYKIiIiIiIi4TkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirlMQERERERER17kaRBzHiXYc53nHcdY5jvOb4zjmOE5MHuWCHcd52nGcfY7jpGaVv97NuoqIiIiIyPnjdotIFaArcAj46AzlXgX6ASOAdsA+YInjOLXOew1FREREROS8C3L5eqvN7DIAx3ESgJanFnAc5xqgG3CnmU3L2rcK+AJ4HGjvXnVFREREROR8cLVFxMwyz6JYeyANmJ3juHRgFtDKcZxC56l6IiIiIiLikgtxsHpNYKeZ/XbK/i+Agvi6d4mIiIiIyEXM7a5ZZ6MYvjEkp0rJ8X4ujuMUy+O98uewXiIiIiIico5ciEHkr7oPGJnflRARERERkT92IQaRQ0CFPPZnt3ak5PEewHPAjFP2lQf+fY7qJSIiIiIi58iFGES+ADo6jhNyyjiRGsAJYEdeB5lZCqeEFMdxzlslRURERETkr7sQB6svBLzArdk7HMcJAm4DlprZ7/lVMREREREROTdcbxFxHKdL1o/XZv3bxnGcn4CfzGyVmW10HGc2MNFxHC+wExgAVAS6u11fERERERE59/Kja9acU16/kPXvKiAu6+c+wBhgNBAJfA60NrMNblRQRERERETOL9eDiJn94cANM0sFBmdtIiIiIiJyibkQx4iIiIiIiMglTkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirlMQERERERER1ymIiIiIiIiI6xRERERERETEdQoiIiIiIiLiOgURERERERFxnYKIiIiIiIi4TkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirlMQERERERER1ymIiIiIiIiI6xRERERERETEdQoiIiIiIiLiOgURERERERFxnYKIiIiIiIi4TkFERERERERcpyAiIiIiIiKuUxARERERERHXKYiIiIiIiIjrFERERERERMR1CiIiIiIiIuI6BREREREREXGdgoiIiIiIiLhOQURERERERFynICIiIiIiIq5TEBEREREREdcpiIiIiIiIiOsURERERERExHUKIiIiIiIi4joFERERERERcZ2CiIiIiIiIuE5BREREREREXKcgIiIiIiIirrvggojjOHGO41ge2+H8rpuIiIiIiJwbQfldgTO4D1if43V6flVERERERETOrQs5iHxlZp/kdyVEREREROTcu+C6ZomIiIiIyKXvQg4ibzqOk+E4zs+O47zlOE75/K6QiIiIiIicGxdi16wjwARgFXAUqA08CqxzHKe2mR3I6yDHcYoBxU7ZrfAiIiIiInIBuuCCiJltBDbm2LXKcZzVwKf4BrAPP82h9wEjz3P1RERERETkHPj/7N15dFXV/b/xZ0NCGEKEMAioDJVJEQXFFvqrAsqkorV1BmRwaKvWgVKtVSihatH61YqKSJ1QtFgntFSlioJWBQTUOiCCyqCICoLIIIEk+/dHktsESBgKh4jPa627knvOPvvs3HYt75t99mdXuCCyNTHGN0II84Ejy2l2K/DgZscaAy/stoFJkiRJ2infiSBSQizzRIwrgZUlj4UQdvuAJEmSJO24irxYPSWE0AFoReHjWZIkSZK+4yrcjEgI4SFgIfAG8DWFi9V/Dyyl8PErSZIkSd9xFS6IAO8CZwEXA9WBz4EngOExxhV7cmCSJEmSdo0KF0RijCOBkXt6HJIkSZJ2n+/EGhFJkiRJexeDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCWuQgaREMIBIYTHQgirQwjfhBCeCCE03tPjkiRJkrRrVLggEkKoDrwItAYGAGcDLYCpIYQae3JskiRJAciTrgAAIABJREFUknaNtD09gK04H/gB0CrG+CFACOFtYAHwS+DmPTg2SZIkSbtAhZsRAU4CZhSHEIAY40LgVeCne2xUkiRJknaZihhE2gDvbuX4e8DBCY9FkiRJ0m5QER/NygZWbeX4SqB2WReFELKLri3JBe6SJElSBVQRg8jOugQYvqcHIUmSJGnbKmIQWcXWZz7Kmikpdivw4GbHGgMv7KJxSZIkSdpFKmIQeY/CdSKbOxiYW9ZFMcaVFD6+lRJC2LUjkyRJkrRLVMTF6v8AOoYQflB8IITQFPh/ReckSZIkfcdVxCByF7AIeCqE8NMQwknAU8AnwNg9OTBJkiRJu0aFCyIxxnXAMcB8YDzwELAQOCbGuHZPjk2SJEnSrlER14gQY1wCnLKnxyFJkiRp96hwMyKSJEmS9n4GEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUpchQsiIYRFIYS4ldfJe3pskiRJknaNtD09gDL8C8jZ7NgHe2AckiRJknaDihpEVsQYZ+zpQUiSJEnaPSrco1mSJEmS9n4VNYicGEJYH0LIDSHMcH2IJEmStHepiI9mTQJmAQuBfYFfAxNDCGfHGB8s66IQQjaQvdnhxrttlJIkSZJ22m4NIiGEbsDz29H0pRhjF4AY48Wb9TERmAGMBMoMIsAlwPCdG6kkSZKkJO3uGZHXgIO2o936sk7EGPNDCI8CN4QQGsYYl5XR9Fa2DCqNgRe2a6SSJEmSErNbg0iMcT0wb1d2Wc69VgIrSx4LIezCW0uSJEnaVSrqYvWUEEIacAawJMb4+Z4ejyRJkqT/XYVarB5COAv4KfAM8AmFi9UvAg4HztqDQ5MkSZK0C1WoIEJhpaz6wI0UVsBaB8wGesUY/7UnByZJkiRp16lQQaRoN/Vj9vQ4JEmSJO1eFX6NiCRJkqS9j0FEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEpdYEAkh/CaEMCmEsCyEEEMIOeW0PTmE8GYIYUMIYXEIYWgIoXJSY5UkSZK0eyU5I3I+UB94srxGIYSewOPALOA4YBQwFPjT7h6gJEmSpGSkJXivNjHGghBCGvCrctpdD7wSY/xF0fupIYRMYGgI4S8xxs93+0glSZIk7VaJzYjEGAu21SaEcADQDnhws1PjgXQKZ0gkSZIkfcdVtMXqbYp+vlvyYIxxIbAeODjxEUmSJEna5ZJ8NGt7ZBf9XLWVc6tKnN9CCCF7K+cb76JxSZIkSdqFdmpGJITQrajy1bZe03bxeMtzCbBgs9cLCd5fkiRJ0nba2RmR14CDtqPd+h3st3gmpPZWztUGVpZz7a1subakMYYRSZIkqcLZqSASY1wPzNvFYwF4r+hnG2B68cEQQlOgOjC3nDGtZLOgEkLY5QOUJEnSd0/Tpk3p0qUL48aN29NDUZEKtUYkxrgkhPAfoC9wd4lT/YBNwLN7ZGCSJEn6Tps4cSJZWVl7ehgqIbEgEkLoADTlv+tSDg4hnFr0+zNFsywAVwH/DCGMBSYA7Snc0HCUe4hIkiTt/XbH7EX79u13WV/aNZIs3/tr4FHg70XvTyt6/yiFO64DEGN8BjgV6Aj8CxhM4a7qVyY4VkmSJO0hEydOZNiwYVscz8nJIYTAvHnz6NmzJzVq1KBx48bcd999AIwfP57WrVuTmZlJ165d+eijj1LXNm3alIEDB6bejxs3jhACM2bMoG/fvmRlZdGoUSMuueQSNmzYUOq+69ev53e/+x3NmjWjSpUqNGvWjOuuu46Cgm1uk6dyJDYjEmMcCAzczrZPAE/szvFIkiSpYtrW7MVpp53G+eefz29/+1vuuOMOzjnnHBYsWMC0adO4/vrr2bRpE5deeil9+vRh5syZ5fZ19tlnc9ZZZ/Hwww8zZ84ccnJyqF27NiNGjAAgLy+Pnj17MnfuXIYNG0bbtm2ZMWMG11xzDStXruSmm27aZX/3901F29BQkiRJe4HdMXvx6aefAvDhhx9y/fXXM2nSJO644w4qV67M2LFjmTx5Mj169GD27Nnk5uby+uuvc8ABB3DdddcRY0z1N23aNAYNGgRAzZo1GTNmDH369GHYsGEcd9xxTJgwIdV2woQJvPLKK0ycOJHLLruMY489lquvvpphw4Zx22238eWXX+7Oj3GvZhCRJEnSbnPaaadxwgkn8OSTT3LEEUdwzjnncNVVVzFmzBiuv/567rvvPj744AP69OlT6roYC/jyy0eZN+8cli4dA8ATTzwGFIaYCy64gNGjR3PnnXdSv359OnbsSPXq1enZsyd33303p512GgDHHXcc11xzDatWbW2/bDjggAMYP358aj1K27ZtWbJkSer85MmTadKkCT/+8Y/Jy8tLvXr06MGmTZuYMWPGrv7IvjcqVNUsSZIk7V0uv/xy+vfvD0CHDh2YNGkSY8eOZeHChakqVsuWLePSSy9l8eLFNGnShIKCXJYvf4y5c8cDsGZNYV+NG69m1So48cSenHrqqcycOZMJEyZQpUoVateunZq9eOmllygoKGDs2LGcfvrpNGnShKFDh/Ltt99uMb6bbrqJ5s2bp96PGTOG3Nzc1Psvv/ySxYsXk56evtW/76uvvtoln9P3kTMikiRJ+p+tXfsOH310Be+/fzbz51/Et99+DBTOSBSrXbt2avaiZCnd1q1bA/DJJ5+wdu1/2LTpCwoKtgwN++1X+HPu3D7EGMudvcjPzwcgPz+fHj16ALB8+fJt/h1nnHFGqfd16tShWbNmzJo1a6uvE088cXs+Hm2FMyKSJEnaaXl5q5k7ty8rVz5ddCQAkS++KHyXkbECqJdqXzx7UVKVKlUA2LBhAx9/PLLUeo6SMjIKf65c+U++/vpFMjIytjl70atXr9TvJduWpWHDhqXe9+rVi8cff5zMzMxUYNrc119/vc1+tSVnRCRJkrRTCgo28vbbx5UIIQClQ8Sbb3YhN3fpdvW3ceMXrFy5fftXf/bZnVscK569OP/881PHWrduTUZGBpUqVSIvLw+A5557LnW+b9++pRbK33LLLaX6LF4TctRRR3H44YdTvXp1srOz6dy5M926dWP9+vUlm1cNIdwQQlgYQthY9PPqEELqO3cIITOEcFsIYUkIITeE8GUIYUoIYespZy9mEJEkSdJO+fLLCXzzzfRy2+TlfcnixSO3q78NGxazeZApy9q1b29xrFevXnzyySepGRaAY445hkmTJpGRkcHrr7/OVVddxVNPPZU6v3Dhwi0WypeUllb4AFF+fj5LliwhLy+PDRs28PLLL7Np06ZS9wLuA84DRgHHAXcDw4AbS7T5C3A6MALoDvwSeAuotV1/+F7EICJJkqSdUljNattfJz//fBz5+eu2o8dQ7tlTToGpU6FyZQihcur4woULueuuEfz0p23p1OlIHnjgAQCuuOIKevfuzQcffMCPfvQjKlWqxNixY/nzn/8MwPPPP8/QoUN5/fXXWbx4MQC1atViwIABW9z74osvZsWKFWzcuJH169dzwgknsGzZslRQKdIB+FmM8ZYY4wsxxuuAa4CLQwjFG3h3Ah6KMd4TY3w5xjgxxvjbGOP3rvyWQUSSJEk7Zd26t4Ft7y5eULCODRsWbbNdtWotCGHr1ak2l5X1Y2IsvPfrr7dm5szm/Oc/HRg6dCatWhXOUtxyyy307duX+++/n86dO6cWyteoUSPVT8mF8uU54YQTSr3ffKF8kaXAayGEtOIX8ByQDnQsajMLGBhCuCqE0CGUTFTfMy5WlyRJ0k4q+9+0Bw4sfG3edtGiRVu07dKlS2qB+ty5T/Lwww+VOt+rV+GrpIYNz+e0026ka1f49tsFqeNVqhTQps1XzJ4NixdPoEGDn6fOjRs3jtq1a5e637Rp04DChfLlyc7OLvV+84XyRfYDNpXRRZ2inxcDnwPnANcBK0MIDwBXxxjXl3HtXskZEUmSJO2UrKxObOtxKoC0tGyqVfvBdvX5gx+MpEqVhuX2u//+l7F69TRWrHi86MjW15XMndtnuxfK7yKfAEeW8ZoEEGNcG2P8fYyxOdAU+BPwa2B4kgOtCAwikiRJ2in77XcB27O4vGHD86lUKWO7+qxa9QDat3+NffbpvMW5ypVr0qzZtTRr9mc+/XQU2wpBMeaybNnd23XfXaQhsDbGOHsrrxVbji8ujjHeBLwDHJLkQCsCH82SJEnSTqlb92Tq1j2ZFSueLLNNtWotadz4dzvUb7VqTWnffipr177LypXPUlDwLVWrNqVu3Z+TlpbJ6tXT2bhx2Xb1tXz54zRtmthkwxvACyGEm4D/AFWAA4GTgJNjjOtDCNOBf1AYPtYCnYHDgPuTGmRFYRCRJEnSTgmhEgcf/Hc+/PA3LFv2V2IsvTwiO/t4Wre+j/T02mX0UL7MzEPIzNxyoiA//5uttj/zTGjXDho0+O+xvLzt22ywe/fu3HfffTs1zhIGUVia9xdAM2Ad8BHwNLCxqM3LRW2upPC7+MfA4Bjjrf/rzb9rQlk7V+4NQgjNgQULFiygefPme3o4kiRJe62NG5ezfPkj5OYuIy1tH+rW/SnVq7fcLfdau/ZtZs8+bIvjCxZA9eqw337FRwI1a/6QI44ovzJubm4ub775JgceeCD16tUrt+3WfPjhh7Ro0QKgRYzxwx3u4HvKGRFJkiT9z6pUqcd++12UyL1q1GhLjRptWbfuXUquUSnMAiVFGjTYck+QzWVkZNCxY8dtttOu5WJ1SZIkVRg5OTmEEJg3bx49e/akRo0aNG7cOPXY1Pjx4znooIPo0mU+gwdHlpYoinXmmXD99cXvAs8/vw/7738hM2bMoG/fvmRlZdGoUSMuueSSUuV6Fy1aRAiBcePGpY4NHDiQ/fffnzfffJOjjjqK6tWr06JFC+68884txvzqq68W//peCOHDEMJ5IYRxIYRFu/Kz2dsYRCRJklThnHbaaZxwwgk8+eSTHHHEEZxzzjlcddVVjBkzhuuvv55x48bz2Wc1ufbarV9fpcq+HHDAbwE4++yzOfDAA3niiSe44IILGD16NCNHjtzmGL755hv69OlDv379eOqppzjyyCO54IILmDp1aqrN3LlzOf/884vfXgpcVfTzmP/l7/8+8NEsSZIkVTiXX345/fv3B6BDhw5MmjSJsWPHsnDhQrKysgBYtmwZl156KXl5J5ORMRVYXVTi9yoaNTqf+fMnAdCnTx9GjBgBQLdu3Zg5cyYTJkxIHSvLmjVruOOOO+jatSsARx99NP/617+YMGFC6ti1115LZmYmq1atApgSY/wwhPBvYCGFGxeqDAYRSZIk7REbNnzKsmV38c03M4ECMjMPY9OmwkemjjvuuFS72rVrU79+fdq3b58KIQCtW7cGoGrVIfzkJxOpWrUp9ep1oUmTK0vd54QTTij1vm3btkyZMmWb46tevXoqcEDhWpKWLVuyZMmS1LEZM2bQpUsXJk6cmDoWY1wWQngN2L5dHL+nfDRLkiRJu9y4ceMIIRBCYP78+aXOxRh55JFfUq3aAfzgB39kypTnWLXqBS666P8YNep2AGrV2qfUNVWqVKF27dpbHAPYsGEDTZs2ZcWKLfYMBCA7O7vU+4yMDHJzc7f5N2x+v+JrS64vWbp0aakQUsIX27zBTgohLAohjNuJ65qGEGIIYeCuH9WOM4hIkiRpt6lZsybjx48vdWzp0tu5776/Ur168ZEIFHD22VA8AbFw4bAdus/EiRPZZ599tt1wF6tfv35Zp/bdjbf9GXDNTly3DOhE4b4me5xBRJIkSbvNz3/+cx588EGK967Lz1/PvHlDefllOPro0m332w/q1Cn8/ZNPbiI3d/uXWLRv35709PRdNewduu/mQggNgf+3u+4ZY3wzxvjRTlyXG2OcEWNcvjvGtaMMIpIkSdptzj77bBYvXswrr7wCwPLljzNt2jcUFGwZRK6/Hh57rPhdPrNm/YUQAmPHjuXrr7/m8ccfp1atWpx44ol8+umnpa7d/NGsP/7xjwwaNAiAVq1aUalSJbKysrjkkkvYtGlT6niVKlU44ogjAPjFL35B/fr1GTJkCPPnz2fp0qUcfvjhZGdnU6tWLTp27MhXX31V6r4XX3xxybfHhhBOB56j8NGsgq19JiGEnKJHpFqHEP4VQlgXQlgSQhhUdP7sEMK8EMLaEMLUEMKBm11f6tGsEMLAov46hhAeCiF8E0L4LIRwawihaol2WzyaVVRm+NMQQvsQwr9DCOtDCAtCCL/ayri7hRDeDCFs2BVlig0ikiRJ2m2aNGnC0UcfnXo8a/36D3juOTjqKKhWrfxrc3ML/9F/5MiRbNq0iY4dOzJq1CimT59Ov379yrzu9ddf57rrrku9P/HEE7nkkks46KCDGD16dGos3bt354knnqCgoDAvZGdnc++997Jw4ULefvttAM444wweffRR/v73v9OhQwfeffddVq5cmeq7ReldFG8FrgduB+YAq7fx8TxK4WNSJxe1vzeE8CfgAuBKYBDQCvjbNvopNh74CPg5MAa4CPj9dlyXVXSPB4GfArOAMSGE1Er9EMLBRWNdC5zJLihTbNUsSZIk/c82bFjCunXvAJXIzCz9uFL//v0ZMmQIt956KytWbGDOHLjhhq33k5kJ//xn4e/5+YWPWjVt2pRFixal2ixfvpzLL7+cli1bEmNk48bl5Oev47jj2nDjjSfy85//hho1arBx40b+8Ic/lCrT26NHD6ZMmcKAAQO4/fbb+etf/8rXX39NTk4OOTk5FBQU8Pjjj9OuXTvefvttzjjjDJo2bUpBQQHHHnss8+fPp1rZCapNUfneTGAE216LcWOM8QGAEMJs4ETgl0CzGOM3RccbAqNCCE1ijIu30d/fYozDi36fEkL4EXAWMLycawBqAhfGGKcW3fNloGfRtcWbpgwFvgF6xhjXF7X7n8oUOyMiSZKknbZ27bu8886JzJjRlHfe6c077xzP9On7s3TpHak2p512Grm5uUyaNIl//Wsd2dlw+OHb7jsrqyMAxx9/fKnjbdu2BeDjj99n3rxzmT59PzZtWsGaNa8zZ86pvPbaq7RrVw/YsnRvrVq1iDGmZlRmzJhB48aNufrqq0lLS+Oll14ihMApp5wCwLnnnsu+++5LWloa6enpPP/883zwwQdlDfmHIYSzgClAbWDUNv7EZ4t/iTGuAr4EZhSHkCLzin4esI2+YMvg8w7QeDuuW18cQorGkgvM3+zajsAzxSGkqN0y4LXt6H+rnBGRJEnSTvnmm1m89VZXCgrWU1j5qlg+a9bMAuDbbxdSs2ZzTj75ZMaPH8+iRYs47rg6VKq0ijKWUACVSE/Ppk6dXsDWy+8CvPfeRbRqVToUrFkDBQVQuXJhyeDNS/Dm5eUB0LBhQ6BwU8T69euTlpZGnTp1Uo9dFS98//rrr7ntttto3LgxaWlpDBs2jPfff7+sj+Q+YCPwOtAtxvh2WQ2LrNrs/cYyjgFUZdtWbvY+F8jYjus2v2fxtSXv2ZDCoLS5L9jJ/VIMIpIkSdphMebz3nunUVDwLaVDSGkfffRb2rb9D/379+eEE06goKCAu+56mNzcc4CtXxtCJVq3foA1a8r/Dv3tt1vOTNSsCZUqwdq1he9Xr34V+O86juLHqj7//HPatGlDw4YNmTt3Lnl5eXz11Vep0DN9+nQAbr/9djp16pS6fv361ITA1rSJMX5Y7qC/u5YBW6tVvNNlin00S5IkSTvsq6+eJTd3MWXPahRat+5tvvlmNt27d+f000/nV7/6FT/60Rm0b/9vqlc/eIv2lSqlc+ihz1GnznFb6a1QQUFemeeqVoVDDoH33it8/+WXD5U6v//++wPw8MMPA9CxY0eWLFnCn/70J/Ly8ujSpQsxRubMmQNAWtp//91+/vz5vPrqq+X+vXuxGcDxIYTU7i//a5liZ0QkSZK0w1aufHbbjUq0zcrqwIQJE1LHatY8nJYtbwe6st9+l9Gy5UHUqfMUVaq8R+3aXcvuDMjN/bTc8xdcAMVVdf/5z39TrdpUPv74Y9566y3qFG1Ucvfdd1OjRg26d+9OnTp1yMnJoWXLlqn1IcWVtH77299y9dVXs2zZMoYPH07jxo1T575nrgVOBf4VQvg/Ch/5GkY5ZYq3xRkRSZIk7bDCdSG7pm3duifQqNEvSE+vt509lj0jAtC6NRRX973//lyOP/54brzxxtRsCMDNN9/Ms88+y89+9jMqVarEAQccwOLFixk0aBAHHHAAl112GVC4huSkk07iz3/+M9dffz1Hb775yfdEjHEucAKFFbYeYcfKFG9VKN7lcm8UQmgOLFiwYAHNmzff08ORJEnaayxa9EcWLdpWVdhCLVuOpVGjX+yye2/a9DWvvbYvMW7cRstKZGa2o0OHOTt8j969e/P+++/z0Ufb3sD8ww8/LN5PpMVevEZkC0Vlij8Eno4xnruj1/toliRJknZYgwYDWLQoh/IWqgNUqlSN+vXP2KX3Tk+vRf36Z/LFFw9so2UBjRr9cpv93XzzzWRmZtKiRQvWrFnDo48+ytNPP82YMWN2zYD3EiGE2ygs1/sZ0IjCDQ23p0zxVhlEJEmStMOqVm1Cw4bnsWzZXeW2O+CA35KWts8uv3+TJlezYsVE8vPXUdYSherVD2Hffcvegb1YRkYGf/nLX1iyZAn5+fm0atWKu+++m3PP3eF/5CeEUO736xhj+c+VVWxVgRsorJS1I2WKt8pHsyRJkrRTCgo28v77/Vm+/O9AYPPZkUaNLqRFi9sIYfcsS/7mm9d5552T2LTpi6L7F48hUrPmkRxyyD/IyGiwW+5dUolHs/oBD26jebMY46LdPqjvAGdEJEmStFMqVarCwQdPYPXqi/jsszGsXfsWEMjK6kijRheQldVht94/K+uHdOy4iOXLHyuaHfmGKlUa0aBBf2rVOoYQwm69/1a8Cxy5jTafJTGQ7wKDiCRJknZaCIFatY6iVq2j9sj9K1euSoMG/WjQYNuPYCVg3fdpsfr/yvK9kiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXGJBZEQwm9CCJNCCMtCCDGEkFNGu3FF5zd/3ZLUWCVJkiTtXmkJ3ut84BvgSeBX22i7HDhps2PLdsegJEmSJCUvySDSJsZYEEJIY9tBZGOMcUYSg5IkSZKUvMQezYoxFiR1L0mSJEkVW5IzIjuifghhBVAL+Bi4B/i/GGN+WReEELKB7M0ON959Q5QkSZK0sypiEHkLmAO8B1QFfgaMBFoA55Vz3SXA8N0+OkmSJEn/s50KIiGEbsDz29H0pRhjlx3pO8a4eXWsZ0IIa4HLQgg3xBgXlHHprcCDmx1rDLywI/eXJEmStPvt7IzIa8BB29Fu/U72v7kJwGVAB2CrQSTGuBJYWfJYCGEX3V6SJEnSrrRTQSTGuB6Yt4vHsl233gP3lCRJkrSLfVd2Vu9LYQiZtacHIkmSJOl/l9hi9RBCB6Ap/w2FqIBYAAAgAElEQVQ/B4cQTi36/ZkY4/oQQhNgPPAw8CGQQeFi9YHA2BjjR0mNV5IkSdLuk2TVrF8DA0q8P63oBdAMWASsoXCdx++AfYECCh8BuwS4I6mBSpIkSdq9EgsiMcaBFM5slNdmJXByEuORJEmStOd8V9aISJIkSdqLGEQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJElS4gwikiRJkhJnEJEkSZKUOIOIJEmSpMQZRCRJkiQlziAiSZIkKXEGEUmSJEmJM4hIkiRJSpxBRJIkSVLiDCKSJEmSEmcQkSRJkpQ4g4gkSZKkxBlEJEmSJCXOICJJkiQpcQYRSZIkSYkziEiSJElKnEFEkiRJUuIMIpIkSZISZxCRJEmSlDiDiCRJkqTEGUQkSZIkJc4gIkmSJClxBhFJkiRJiTOISJIkSUqcQUSSJO3VunTpQpcuXXbommnTppGTk0NBQcHuGdR2WrRoETk5OXz88cd7dBzS7mAQkSRJWzVu3DhCCKlXjRo1aNq0KT/72c945JFHiDHucJ9vvfUWOTk5rFy5cjeMeNeZNm0aI0aMqBBBZMSIEQYR7ZUMIpIkqVyPPvoo06dP55lnnuGaa64hIyODs846i+7du/Ptt9/uUF9vvfUWI0aMqPBBRNLuZxCRJEnlateuHR07dqRz586cffbZPPzwwzzyyCO8+OKLXHHFFXt6eKU8/PDDtG7dmoyMDNq0acPEiRNLnd+wYQODBw/mkEMOITMzkwYNGnDiiScyb968VJucnBxGjBgBQHp6empGqNjw4cM5/PDDycrKom7duhxzzDHMmDGj1H3Wrl3LxRdfTOPGjcnIyKB+/fp069at1H3y8vIYOXJkaryNGjViyJAhbNiwASiclenatSsA3bt3T41j2rRpu/Qzk/YUg4gkSdphp5xyCj/96U+56667WL9+PbDtL+jjxo1j0KBBALRo0SL1xXrRokUA3H777XTq1Ins7Gxq1apFx44defrpp7d7TFOmTKFPnz60aNGCJ554gssvv5xLL72UDz74INUmNzeXNWvWMHToUJ5++mnGjBnDhg0b6NSpE59//jkA5513Hueeey4Ar7zyCtOnT2f69OmpPpYuXcrgwYN56qmnGDduHPXr1+foo4/mnXfeSbUZPHgwjzzyCMOHD+f5559n7NixtGvXjq+//jrVpl+/flx77bX06dOHp59+mt///vfcc8899O3bF4DDDz+c0aNHA3DrrbemxnH44Ydv92ciVWgxxr32BTQH4oIFC6IkSdox9913Xyzvv6N//etfIxBfeumlGGOM5557bnzggQfiiy++GCdNmhTPOOOMmJ6eHt9+++0YY4xffvllHDp0aATio48+GqdPnx6nT58eN2zYEGOMcciQIfHuu++OU6ZMiZMnT44XXXRRBOKzzz67XeP98Y9/HA866KCYn5+fOjZ9+vQIxM6dO2/1mry8vLhu3bqYmZkZb7755tTx4cOHRyBu2rSp3Hvm5eXFTZs2xZYtW8ZLLrkkdbxNmzZx8ODBZV738ssvRyDef//9pY4/+OCDEYhvvvlmjDHGqVOnRiA+//zz5Y5De9aCBQsiEIHmsQJ8B/6uvNL2RPiRJEnffY0bNwZg2bJlANx9992pc/n5+fTq1Ys2bdpw9913M2rUKOrVq8eBBx4IFD7u1bx581L9/d///V/q94KCAo499ljmz5/PmDFj6NWrV+mbz5sHY8bAa69BQQH5Bx/MrNdf58orr6RSpf8+8NGxY0eaNm1a6tJHHnmEm266iQ8++IDVq1enjpecOSnPlClTuO6663j77bdLrXVp1qxZ6vcjjzyScePGUbduXXr06EH79u2pXLly6vzkyZOpUqUKp556Knl5eanjPXr0AODll1+mXbt22zUe6bvKICJJkiBGeOklePll2LQJDjqo8Ge5lxRWzSpeP7E9X9DLM2fOHIYPH86sWbNYvnx5qv9WrVqVHucf/gDXXlvq2hVvvMEmYN9nnoGhQyEjI3Xuq6++YtWqVQBMmjSJM844gwEDBjB8+HDq1q1LpUqVOP7441NrM4DU42KbV8164403OP744+nZsyf33HMPDRs2pHLlypx33nmlrr/tttto0KAB9957L1dffTXZ2dn079+f6667jurVq/Pll1+yceNGatSosdXP4quvvgJIPS5WHPakvYlBRJKk77uZM2HQIHj//dLHq1cv97JPPvkEgIYNG273F/Ty+jr22GM5+OCDue2222jcuDFpaWkMGzaM90uO6y9/2SKEANQF0oEv3ngDLrgA7r03dW7Tpk1kFAWThx9+mObNmzNu3LhS5zev4lVWEHn88cdJS0vjiSeeID09PXV81apV1KpVK/U+MzOTkSNHMnLkSBYvXsxjjz3GlVdeSZUqVbjhhhuoU6cOVatW5d///vdWP49GjRoBBhHt3QwikiR9n82eDV27Qm7ulueKFqFz773wpz9tcfrpp5+matWqHHHEEVx33XXb9QW9LJMnT2b16tU88sgj7L///iWGsP6/jb79Fv74x61eXxk4EngMyLnvPipddRU0b87MmTPZsGFDKoisX7+etLTSX3/Gjx9Pfn5+6f6KHqP69ttvqVq1aqnxVK5cuVQVrRdffJElS5aUOfPTpEkThgwZwkMPPcS7774LQK9evbjhhhtYvXo1xx57bJmfS/Fnmbu1/32k7zirZkmS9H124YWwYQOUt3Hfn/8MRf8yX+zxxx/nH//4B7/61a+oXr16uV/QSyoOBJvvP1IcOEqGmPnz5/Pqq6/+t9HEiVBiTcfmRgDzgCwgvVUr9ttvP3r37k2VKlVSbY499ljmzZtHnTp1qFatGjVr1uSiiy4iKysr1SYnJ4cXX3wRgOzs7FLle3v16sXatWupX78+1atXJzMzk+OOO4569eqVGssPf/hDOnXqRL169UhPT6dGjRq8+eabHHbYYUDhbu9nnnkmvXv3pm7duqSnp1OnTh26d+/OSSedxPz585k2bRq//vWvAfjDH/6QGsczzzwDwN/+9jfat29PZmYmWVlZtG3blrFjx5b5+UgVjUFEkqTvqzlzYNaswnUX5XgrP58ZOTm8/PLLPPjgg5x55pmcfvrpdO/enTlz5tClS5fUF/SBAwfywgsvMGbMGPr168d+++1Xqq+DDz4YgEGDBvHqq68ye/ZsNm7cSLdu3UhLS6N///4899xz3H///fTo0SO1IB6Ajz7a5p+U+kuK/qbi6jzF+vbty2GHHUZBQQExRho3bsyhhx7KmjVrUuHovPPOS5UZrlWrVqlw1bNnTzp27EjlypXJz8+nYcOGdOrUiRUrVrBu3bpUu9zcXGbPns26detIS0ujbt26dO3alZNOOinVpqCggPz8fNLS0gghsH79el588UXmzZvHvvvuW6p8b506dVKzNDFGXnnlFfr160fnzp158skneeyxxzj//PNLlQeWKrw9XbZrd76wfK8kSWUbPTrGwq/sW33dV/i9PvWqWrVqbNy4cTz55JPjI488EgsKCmLnzp1TpXFvvfXW2LRp01i1atXYoUOH+Pzzz5c6X6xz584RiJUqVYpAXLhwYYwxxr///e+xVatWMSMjIx588MFxwoQJccCAAbFJkyaFF/75z+WO98cQD4KYDzEOGBBj3HvL9954442xdu3a5Y5NybF87869XCMiSdL3VXmPYwEDi15UqgTHHQf//Ge57S+++GIuvvjiUse6deu2RbsuXbrw0ksvkZubW2q9xumnn87pp59equ2ZZ5753zc9ekAZO7nnA7OAKyl63KOoDO7eWr73yCOPZNWqVfTr148zzzyTn/zkJ9u1FkeqSHw0S5Kkvdhbb71FTk7OFlWhAChar1CeRUBOQQEfH3AADz/8MK1btyYjI4M2bdpQv3595s2bl2q7YcMGBg8ezCGHHEJmZiYNGjTgxBNPTLVZtGgRXbp0YcSIEUDhepDidQ9NmzZl4MCB5e/Ofthh8P/+31bHuQIKy/cC1KwJp5ySOrfvvvumfi8u33vQQQfxt7/9jZkzZzJr1izq1au3XdW9iquDZWZmcs899zBjxgxmzZrFYYcdtkX53l/+8pfce++9HHnkkdSvX5/Bgwen1sKULN+bnp6eetWvXx/4b/nesnTu3JlHH32UTz75hJ/97GfUq1ePbt268fbbb2/zb5AqCmdEJEnai7311luMGDGCfv36kZ2dXfrkT35SuF/IBx+UOTuyiMJF4NX32Ycr+/ThhBNO4KabbmL58uVceeWVpapN5ebmsmbNGoYOHUrDhg1ZuXIld9xxB506deL9999n0aJFvPTSS/Tq1YvJkyfzyiuvpGYJMjIyyMrKYuTIkQwePJj999+fdevW8eCDD3L00UczZ84c2rZtC2ecASUXsBdJle8FWLcO5s+Htm0B+OKLL2jSpAmw/eV7y5J0+d7ynHrqqZx66qmsXbuWadOm8bvf/Y5evXrx6aefltrUUaqoDCKSJH1fhQCjRhU+dhVCuYvW7580idatW/PUU0+lvuS2bt2aTp06pdrss88+W+yu3rNnT/bdd18mTJhA+/btAVIVpn70ox9tUUp3W7uzM2HCVsdXqnxvQQGVRo+GO+9k5syZLFq0KBVEtrd8b8nqXjVr1kwdT7p8b1lVxkrKzMykd+/efPzxx1x66aV89dVXW1Txkioig4gkSd9x8+fP53e/+x2vvvoq33zzDfXr1+dHP/oRPXv25PzzzwegRYsWqfYLFy6kadOm3H777Tz00EN8UK0aBevW0RoYBpxQ1G4a0LXo97lz5wKF+2tMnTo1VX625M7gn3/+OWeccQbTp09n02a7sk+dOpXf/OY3QOEXfyh8NKu4r6ZNm9KlSxf69eu31fUXCxYsYNKTT3LikiWMKjo2C7gKeANYD9QCPgdOBn754IMs79iR4cOH06BBg1Q/vXr14sknn2Tw4MH07t2b2bNnc9ttt22xvqK4utdNN93EcccdR+XKlenQoQO9evXilltuYeDAgQwaNIj58+dzzTXXbFEdrFOnTpx00km0bduWzMxMXnrpJf7zn/8wYMAAoHCdzFlnncWpp57Kb37zG374wx9SqVIlFi1axDPPPMMNN9xAy5YtadmyJWlpadx7771kZ2eTkZFBq1atuPHGG/niiy/o2rUrjRo14tNPP+XWW2+lXbt2hhB9d+zp1fK784VVsyRJ3wPNmzePRx55ZHzsscfitGnT4kMPPRT79u0bP//88zh06NAIxEcffTROnz49Tp8+PW7YsCHGGOOQIUPi3XffHadMmRIn/+Mf8aJu3SIQny2qQrUa4ugSVbOGQJw+bFhcvXp1jDHGJk2axLp166YqUh122GERiEcddVS84YYb4p/+9Kf485//PGZnZ8ezzjorjh49OgKxV69eEYj//ve/S/XVu3fvmJ6eHnv37h3vvPPOWKtWrdigQYO4//77x0MPPTSOu+662KdobGsg1obYE+I/IE4tqvLVFWJLiFUgHnzwwfGJJ54oVbkrPz8/Xn311bFhw4axWrVq8eijj45vvPFGbNKkSRxQVGkrxsJqWBdeeGGsV69eDCHEwq9MhbanOtgVV1wR27VrF7OysmL16tXjIYccEkeNGlXqf7f8/Px4yy23xEMPPTRmZGTErKyseOihh8bLL788fv3116l2d955Z2zWrFmsXLlyBOLUqVPjP//5z9ijR4/YoEGDWKVKlbj//vvHc845Jy5dunRX/d9KO8CqWTv3CjGWXzv8uyyE0BxYsGDBApo3b76nhyNJ0i63YsUK6tWrx1NPPVVqj4pi48aNY9CgQWzrv4UFBQUUvPACx/foQTXgqaLj0yicFalM4ezDHytXLtx7pH17mjZtyooVK+jQoQPTpk0jPT2dWrVqsXz58lS/mzZtolq1avTr14+BAwfStWtXzj77bMaPH8+mTZtSj0k1bdqUrKwsPvzwQ1avXs25557LE088wfz58+nUqRPNmjVj2qOPQtFi7tkUPor1H+DQrf1B++67xSaM0u7y4YcfFs86togxfrinx/Nd4UomSZK+K2KEKVPg9NMLF2IfcQR1rruOHxxwAFdeeSV33XUXCxYs2O7u5syZQ+/evdl3331JS0sjvUcPnge2VsS2FYXrLwry8+HWW4HCxeklN/GrVasWX3/9NaNGjeKdd94hxrjV9RfFi7w3X/eQl5eXWn/x3HPP0bt3b+bNm/ff3dnr1YNevSAEWlD4KNYvgQeBTzYbb5cqVejSpct2fxYA06ZNIycnh4JtlDUu7/oQAlOmTNmp66XvG4OIJEnfBWvXwvHHQ/fu8Nhj8O678MYbhFtu4flPPqFD1ar8/ve/p2XLlvzgBz9gzJgx5Xb3ySefcOyxx7Jy5Upuu+02Xnv+eWYBvYCtFbEdAMyjcP3F0w89xLj77mP58uVUrVo11eaKK64gLy+Pq6++mkMPPZR99tmHyy67jH322adUX8ULx2+66SZmzpzJ7NmzAdhvv/1Su7OvWLGCFStWbLk7++WXA7BPCEwFGgEXAo2BQ4DHQ4Dq1WE7qk5tbtq0aYwYMWKng4ikHWMQkSTpu6BPH5g8ufD3zR6r/gHwwJtvsnzkSN58802OOeYYLrzwQp599tkyu5s8eTKrV6/mkUce4fTTT6djixZ0oHDR99YcDjxE4WzJzzdt4sYbbyQ7O7tUyBgyZAhXX301WVlZZGRkUKtWLdasWVNqIz8orJZ14YUXpkr7HnnkkUBhELn11lt59dVXyc/P58033+SBBx4o/UjZMcdAUchqBzwOrASmAwcCp8fIu3/5C5QISJIqJoOIJEkV3euvw6RJ5bcJgZCTQ7tDDuHmm28G4N133y2z/GvxxnqpvTBq12Z+CGy+Q0dG0c9vgbMoDCK51avz3nvvUb16dXr16sW0adMAqFSpEtdeey2fffYZGzZsYMmSJdSuXZszzzyTcePGpcayceNGRo8ezZdffklBQQEl16tefPHFLFy4kP79+7Nx40batGnDtGnTUvcA4Je/LFynMmAA1KjBY8DASpV4plIlCoD7N3s8bVsbLQLk5ORsdaPFYuVutLiZ1atXM3DgQGrXrk1WVhZ9+/bdYoPC22+/nU6dOpGdnU2tWrXo2LEjTz/9dKk2eXl5DBs2jAMPPJCqVatSt25dfvKTn/DKK6+UavfXv/6Vww47LNXm3HPP3WJflFGjRnHQQQdRrVo1ateuTYcOHZg4ceJWxy8lxfK9kiRVdPfcU+apt4FLgTNipPlnn5E/ciTj5s4lLS2NY445JrUYfPTo0QwYMID09HQOPfRQunXrRlpaGv3792fIkCEsW7aM4VWr0vjbbyn5YFJLCr8s3AtkUxhMWp1yCjVLfEmHwi/f3bp1o2/fvrRu3Zr09HSeeuopVq1aRY8ePQr7KqMUbcl9OoqNGDGCZ555hh//+MdcddVVNG/enKVLlzJ58mQefPBB/rlsGX9duZKWv/oVN910E0cecQRt8/J4//33eeihh4gx0qpVK2DbGy02aNCA8847j08//ZR77rmn1EaLxZYuXVr+RoslXHbZZXTr1o0JEyawYMECrrrqKj777DOmTp36/9u7/+ioyjuP4+9vQgg/CwkC4lqMFkHCnkUpW+Bs+eEPShDE6tkFbK1Ai8QCZVXagwU0sEh3rVWsnsXalS0qINSC1rjVgkItyo+FhbYHrIacDT8Oi1TW+GNBQoLf/ePODJNJQgbI3MnA53XOPZN57jP3fpOHy8x3nvs8T6zO3r17mTx5MgUFBdTU1FBaWsro0aN59dVXKSoqAuChhx5i0aJFLFy4kKuvvppPPvmE7du310oy7rvvPh555BFmzJjBww8/zMGDB5k7dy67du1i06ZNZGdns3z5cmbOnMkDDzzA4MGD+eyzz+pMjyySFumetiuVG5q+V0REzgfDh7tnZblHpq6N3w6D3wF+JXhr8Lw2bXzIkCH+2muvxV4+b948v+SSSzwrK8sBr6iocHf3VatWea9evTw3N9cLCwv9+Qce8AnglyWc42fgl4NnR6bx3fD00+7utaa8PX78uE+ZMsULCwu9bdu23r59e+/fv78vX7681q9S31S0iceKKi8v9/Hjx3unTp08NzfXr7jiCr/nnnvc3f3dd9/1sWPHem5urpuZX3TRRT5y5EjfsmWLb9682YFa0+nGq6mp8aNHj3q7du380UcfjZWXlJQ44NXV1adtjpqaGq+urvaePXv6jBkzYuUbNmxwwEeMGFGr/rJlyxzw119/vd7jnTx50qurq3348OE+ZsyYWPmoUaP8lltuaTCOiooKz8rK8vnz59cqf+uttxzwF1980d3dp02b5tdcc81pfyc5N5q+9yw/q6c7gJT+ckpERETkfDBmjLuZ15eI1NmWLDm3cz35ZHCu+s6Xne2+cmXT/E5n6tNP3Rcvdu/Xz71jR/du3bxm4kTPadHC77///jrVCwoKaiUiq1at8q985SveoUOH2LoogBcXF8fqnC4RWbdunQ8bNszz8/NrvT4+6YgmIksS2qCqqsqzsrJ8wYIFsbLt27f7qFGjvEuXLrF1SgDv1atXrM68efM8NzfXZ8+e7Rs3bvSqqqpax/35z3/ugJeXl3t1dXWtrX379rGkbenSpW5mPn36dF+3bp0fPXo0yT+6JEuJyNltGiMiIiLS3I0YUWeAer2ysuD668/tXHfdBW+/HUwRHLmti9xcmDgRtm+HcePO7fhno6IC+vaFqVNh50746CM4dIgjS5dSXVND1z/8oc5LunbtGvu5tLSUcePG0bt3b1asWMHWrVvZtm0bnTt35vjx+uYIq23Hjh3ceOONtGvXjiVLlrBlyxa2bdtG37596319/LkBWrZsSV5eHgcPHgTqmbFs0ya2bdtGUVFRrePNnj2b+fPn8/LLLzN48GA6derEpEmTOHLkCAB/+ctfAOjRowc5OTm1tk8//TQ2LuWOO+7gySefZOvWrYwYMYL8/HxuvfVW9u7d2+jvLpJKGiMiIiLS3N1+O9x3Hxw9CqebWvammyAyNe45GTQo2Kqrg3O2a3cqKQlbVVWQiFVUBM/jErKLgBzgcGkpLFsW/J0iDh8+HJsmeOXKlfTo0YOlS5fG9ldXVyc9RmL16tW0aNGCNWvWnBrcD1RWVtKxY8c69Q8fPlzr+YkTJ6isrIxNQxw/Y9mll14aqxedQCAqJyeHWbNmMWvWLN5//31eeeUV7r33Xo4dO8aqVavo1KkTAGvXriUvL69OHNH9ZkZxcTHFxcVUVlaydu1aZs6cybhx49i6dWtSfwORVFCPiIiISHP3hS/AihVBj0fCIPGY7t1h8eKmPW9ODnTsmL4kBII1U/bsqbdHKJtgdfVfAZ8vWBCrs3Xr1lrf9h87diw2aD+qvoUWTzfDWHShxaj169efWmgxwS9/+ctaz1944QU+//xzBg0aFDseUCupKSsr4+23E+csOyU6oP6GG25g165dAAwfPpysrCz2799P//7962yXX355nePk5eUxbtw4xo4dGzuOSLqoR0RERCQTjB4Nb7wBs2cHt05F5eTA+PHw4x/DxRenL75UeeaZIPlq4Na0+cDXgK+XlVG8aBEf5OdTUlLCxXF/i6KiIl566SXuueceRo8ezfbt23niiSfq9GYUFhYCwUKLI0eOJDs7m/79+1NUVMRjjz3GxIkTmTRpEmVlZSxYsKD2Qotxdu/ezaRJkxg/fjxlZWXMmTOHYcOGcX3ktrl6ZywrKaF79+61FlO8+eab6du3L/369SMvL4+dO3fy2muvUVxcDMCXvvQlZs2axfTp03nvvfcYOnQorVq14sCBA6xbt47Jkydz7bXXMmXKFNq3b8+gQYPo0qULZWVlPPfcc7HZzETSJt2DVFK5ocHqIiJyPnrnHfc1a9xLS90/+CDd0aRW797e2AD9FeA9wVu2aOGFhYW+Zs0aHzp0aGyw+smTJ33OnDnerVs3b926tQ8ZMsR37NhRZ6aumpoanzp1qnfu3Dk2gDzq8ccf94KCAm/VqpX379/f161bV+sc7qcGq69evdonTJjgHTp08Hbt2vltt93mHyS0U50Zy55/3idMmOCXXXZZrM5PfvITHzBggOfn53urVq28Z8+eXlJS4idOnKh1rGeffdYHDBjgbdq08bZt2/pVV13l06ZN8wMHDrh7MFh96NCh3rlzZ2/ZsqUXFBT43Xff7R9//HHTtJFosPpZbuaexOC3DGVmPYA9e/bsqb0qq4iIiGSGgQODBR2T+byydi0MH576mEQSlJeXc+WVVwJc6e7l6Y4nU2iMiIiIiDRfY8Ykl4R06ABf/Wrq4xGRJqNERERERJqv73wHWrdueJB+1F13BfVEJGMoEREREZHmq2tXWLkSsrPrJiPR59deC/PmhR6aiJwbJSIiIiLSvI0ZAxs3wo031k5GLr4YFiyAV1+FVq3SF5+InBVN3ysiIiLN38CB8MorcOgQ7NsX3IbVp0961zgRkXMSSo+ImfU0s5+a2Z/M7P/M7JCZvWxmfRuof6eZvWtmVWb2npndFUacIiIi0sx16xYkJX37KgkRyXBh3Zr1NeBa4BngJmAq0BnYYmZfjq9oZncCTwGrgSLgBWCxmX03pFhFRERERCTFwvoqYSXwrx63aImZrQf2Av8I3BEpawEsBJ5z9zmRqhvM7BJggZk97e7VIcUsIiIiIiIpEkqPiLsf8YSVE939Y6AM+Ku44kEEPSXLEg7xHNAJ0AThIiIiIiLngbTdXGlm+cBfA7+IK+4TedyVUH135LEQ2HCa4+UnFHc/xzBFRERERCQF0jnK6wnAgMfiyqKJRGVC3Q8T9tdnBlDSNKGJiIiIiEgqndWtWWZ2g5l5EtvvGnj9D4FvANPdvfwc4o/3OHBlwnZ9Ex1bRERERESa0Nn2iGwCeidR71hiQWQq3h8Bc9393xN2R3tC8oBDceXRnpAPadF8XQIAAAeoSURBVIC7f5i43xJXYBURERERkWbhrBIRdz8GvHumrzOzbwGLgUfcfWE9VaJjQfpQOxEpjDy+c6bnFBERERGR5iesdUQws1sIBqY/7e7fb6DaZuAI8M2E8tsJejveTl2EIiIiIiISllAGq5vZEOB54I/AUjMbGLe7yt13Arh7tZndT7CA4UHgdeA64NvA99z9RBjxioiIiIhIaoU1a9Z1QC7Qj7q9GvuAgugTd/+ZmTkwE/gBsJ9gUPvicEIVEREREZFUCyURcfd5wLwzqP8U8FSq4hERERERkfQKbYyIiIiIiIhIlBIREREREREJnRIREREREREJnRIREREREREJnRIREREREREJnRIREREREREJnRIREREREREJnRIREREREREJnRIREREREREJXSgrq6dRDsC+ffvSHYeIiIiInKfiPmvmpDOOTGPunu4YUsbMrgPeSHccIiIiInJBuN7d16c7iExxvicibYEBwCGgOs3hnA+6EyR21wP70xyLJE/tlnnUZplJ7ZaZ1G6Zqbm1Ww7QDdjq7kfTHUymOK9vzYr8Q1BW2kTMLPrjfncvT2cskjy1W+ZRm2UmtVtmUrtlpmbabn9OdwCZRoPVRUREREQkdEpEREREREQkdEpEREREREQkdEpE5Ex8CMyPPErmULtlHrVZZlK7ZSa1W2ZSu50HzutZs0REREREpHlSj4iIiIiIiIROiYiIiIiIiIROiYiIiIiIiIROiYiIiIiIiIROiYgkxczuNbNSMztkZm5m805T9+tmttPMjpvZPjOba2bZIYYrp2FmeyNtmLh9Pd2xXejM7Itm9isz+9jMPjGzNWbWPd1xScPMbFgD19NH6Y5NAmZ2qZk9YWabzexYpH0K6qnXyswejrzPfRapPyT8iAXOqN3qu/7czK4OP2o5Uy3SHYBkjDuBT4CXgLsaqmRmI4DVwBLgXuAa4EdAe2BW6sOUJP0WmJdQ9l4a4pAIM2sDrAeqgAmAAw8CG8zsb9z9aDrjk0bNALbFPa9JVyBSRw9gLPBfwEbgaw3UWwKMAn4A/DcwDfitmQ1y9z+EEajUkmy7ASwFnkooK0tNWNKUlIhIsvq4++dm1oLTJCLAvwBvufuUyPMNZtYOmGtmi9z9/ZRHKsk44u5b0h2E1HIncAXQy93LAczsT8AeoBh4NI2xSeP+rGuq2fq9u3cFMLPJ1POB1sz6At8Avu3uv4iUvQnsBv4JGBNeuBLRaLvFOajrLzPp1ixJirt/3lgdM/sicDWwLGHXc0AOMDIFoYmcL8YAW6JJCIC7VwBvAzenLSqRDJfM+xfB9VcNrIp7XQ2wEhhhZrkpCk8akGS7SYZTIiJNqU/kcVd8YeTD1DGgMPSIpCE3Re65rTKzLRof0iz0IeHaidiNrp1MsNzMTprZ/5rZCo3tyTh9gAp3P5ZQvhtoSXCbkDRf3428nx0zs/VmNjjdAUlylIhIU8qPPFbWs68ybr+kVynwPWAE8E3gOPCimd2e1qgkn/qvnQ+BvJBjkeR9DDwCTAauAxYANwCbzaxLOgOTM3K66y+6X5qnZcBUgutuCtAJWG9mw9IZlCRHY0QuQGZ2A7AuiapvuvuwFIcj5+Bs2tLdv5dwjBeBLcA/U/e2OhE5DXffCeyMK3rTzH4P/CfBAPa5aQlM5ALh7t+Ke7rRzH5N0Lv8IPDV9EQlyVIicmHaBPROol5iF3Vjot8m1fftbR6nvlmSpnPObenuJ83sBeAhM+vm7oeaLDo5E5XUf+009E2tNFPuvsPMyoC/TXcskrRK4LJ6yqM9IXr/yhDu/qmZ/QfwnXTHIo1TInIBitwD+24KDr078tgH2BwtjMz73QZ4JwXnvKCloC29CY8lZ2Y3p8ZZxStE106m0vWUOXYDt5hZm4RxIoXACaC8/pdJM6brLwNojIg0GXffD/yRYNxBvNsJZiN5NfSgpFGRKZnHAfs1vXJavQwMNLMrogWRJP7vIvskQ5hZf6AXwe1ZkhlKCWZ3/IdoQdz/jWvdvSpdgcmZMbMvAKPR9ZcR1CMiSYm8sRZwKnktNLO/j/z8m7hvkGYDr5jZU8DzBAsazgV+qg+56WdmtxFMBfsb4ADQlWDRrn7AbWkMTeDfgOnAr81sLsG3eQsI2ilxoS5pJsxsOVAB7AA+Ivg/74fAQeDxNIYmceLer74ceRxpZh8AH7j7m+6+08xWAY+ZWQ5Bm34XuJy6X65JSBprNzP7PkHSvwH4H4Lb674PXIzaLSOYu3qupHFmtpRgtef6XO7ue+Pq3gqUAFcBh4GngYXufjLFYUojzGwgwUr3fQjufT4KbAcedvffpjM2gciUr4uA4YABbwB3x19f0ryY2Q8JkvjLCG5BfZ+g97dE462aDzNr6MNObCIPM2sNLCRY2LAjQQ//LHf/XRgxSl2NtZuZ3QTcR5CMdAA+IVh76UF3V49IBlAiIiIiIiIiodMYERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCZ0SERERERERCd3/A2eZbrC1f5tqAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 862.5x862.5 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "## w2v model\n",
    "model = w2v_google_model\n",
    "\n",
    "## prepare training word vectors\n",
    "size = 200\n",
    "target_size = len(target_words)\n",
    "all_word = list(model.vocab.keys())\n",
    "word_train = target_words + all_word[:size]\n",
    "X_train = model[word_train]\n",
    "\n",
    "## t-SNE model\n",
    "tsne = TSNE(n_components=2, metric='cosine', random_state=28)\n",
    "\n",
    "## training\n",
    "X_tsne = tsne.fit_transform(X_train)\n",
    "\n",
    "## plot the result\n",
    "plt.figure(figsize=(7.5, 7.5), dpi=115)\n",
    "plt.scatter(X_tsne[:target_size, 0], X_tsne[:target_size, 1], c=color)\n",
    "for label, x, y in zip(target_words, X_tsne[:target_size, 0], X_tsne[:target_size, 1]):\n",
    "    plt.annotate(label, xy=(x,y), xytext=(0,0),  textcoords='offset points')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### ** >>> Exercise 8 (Take home): **  \n",
    "\n",
    "Generate a t-SNE visualization to show the 15 words most related to the words \"angry\", \"happy\", \"sad\", \"fear\" (60 words total)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Answer here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Elmo embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides Word2Vec, several other pretrained models for generating embeddings exist. Here, we'll take a look at ElMo embeddings.\n",
    "Elmo is a language model trained on a task to predict the next word in a sequence of words, but is bidirectional (unlike word2vec).\n",
    "\n",
    "[Image](pic/pics8.png)\n",
    "\n",
    "Source: (http://jalammar.github.io/illustrated-bert/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To embed the sentences, we need to tokenize them and get them into a tensor of uniform shape.This means every sentence should have the same amount of tokens  (ie. If we have 5 sentences, we should have an array of 5 * x ). We will achieve this through padding, or adding \"\" at the end of each sentence for each missing token up to x.\n",
    "\n",
    "We'll be using the Keras tokenizer to tokenize and pad. Keras tokenizer will first map each word to a number, and we'll get the tokenizing below in the text_tok_keras column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>emotion</th>\n",
       "      <th>intensity</th>\n",
       "      <th>text_tokenized</th>\n",
       "      <th>text_tok_keras</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>10490</td>\n",
       "      <td>Take public opinion on revenge with Pakistan i...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.458</td>\n",
       "      <td>[Take, public, opinion, on, revenge, with, Pak...</td>\n",
       "      <td>[208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1875</td>\n",
       "      <td>21018</td>\n",
       "      <td>My goals are so big they scare small minds</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.250</td>\n",
       "      <td>[My, goals, are, so, big, they, scare, small, ...</td>\n",
       "      <td>[10, 757, 28, 18, 340, 51, 480, 1425, 1426]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>20289</td>\n",
       "      <td>@PanicAtTheDisco hey, y'all announced it like ...</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.625</td>\n",
       "      <td>[@, PanicAtTheDisco, hey, ,, y'all, announced,...</td>\n",
       "      <td>[1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2878</td>\n",
       "      <td>40051</td>\n",
       "      <td>@Christy_RTR @doge_e_fresh I'm despondent</td>\n",
       "      <td>sadness</td>\n",
       "      <td>0.806</td>\n",
       "      <td>[@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...</td>\n",
       "      <td>[4991, 4992, 25, 2513]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>432</td>\n",
       "      <td>10432</td>\n",
       "      <td>@sarah_urbina why do you even beef Sara you le...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.479</td>\n",
       "      <td>[@, sarah_urbina, why, do, you, even, beef, Sa...</td>\n",
       "      <td>[4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text  emotion  \\\n",
       "490   10490  Take public opinion on revenge with Pakistan i...    anger   \n",
       "1875  21018         My goals are so big they scare small minds     fear   \n",
       "1146  20289  @PanicAtTheDisco hey, y'all announced it like ...     fear   \n",
       "2878  40051          @Christy_RTR @doge_e_fresh I'm despondent  sadness   \n",
       "432   10432  @sarah_urbina why do you even beef Sara you le...    anger   \n",
       "\n",
       "      intensity                                     text_tokenized  \\\n",
       "490       0.458  [Take, public, opinion, on, revenge, with, Pak...   \n",
       "1875      0.250  [My, goals, are, so, big, they, scare, small, ...   \n",
       "1146      0.625  [@, PanicAtTheDisco, hey, ,, y'all, announced,...   \n",
       "2878      0.806  [@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...   \n",
       "432       0.479  [@, sarah_urbina, why, do, you, even, beef, Sa...   \n",
       "\n",
       "                                         text_tok_keras  \n",
       "490   [208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...  \n",
       "1875        [10, 757, 28, 18, 340, 51, 480, 1425, 1426]  \n",
       "1146  [1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...  \n",
       "2878                             [4991, 4992, 25, 2513]  \n",
       "432   [4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "#Initializing tokenizer, getting rid of some punctuation\n",
    "tokenizer_keras = Tokenizer(filters='\"#%&()*+,-./:;<=>@[\\]^`{|}~')\n",
    "tokenizer_keras.fit_on_texts(train_df['text'])\n",
    "train_df['text_tok_keras'] = tokenizer_keras.texts_to_sequences(train_df['text'])\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[16, 9, 14, 4, 24]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the lenght of the tokenized sentences\n",
    "list(map(lambda x: len(x), train_df['text_tok_keras'].iloc[:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we discudded, the lenght of the tokenized sentences is not the same, so we pad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>emotion</th>\n",
       "      <th>intensity</th>\n",
       "      <th>text_tokenized</th>\n",
       "      <th>text_tok_keras</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>10490</td>\n",
       "      <td>Take public opinion on revenge with Pakistan i...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.458</td>\n",
       "      <td>[Take, public, opinion, on, revenge, with, Pak...</td>\n",
       "      <td>[208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1875</td>\n",
       "      <td>21018</td>\n",
       "      <td>My goals are so big they scare small minds</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.250</td>\n",
       "      <td>[My, goals, are, so, big, they, scare, small, ...</td>\n",
       "      <td>[10, 757, 28, 18, 340, 51, 480, 1425, 1426, 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>20289</td>\n",
       "      <td>@PanicAtTheDisco hey, y'all announced it like ...</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.625</td>\n",
       "      <td>[@, PanicAtTheDisco, hey, ,, y'all, announced,...</td>\n",
       "      <td>[1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2878</td>\n",
       "      <td>40051</td>\n",
       "      <td>@Christy_RTR @doge_e_fresh I'm despondent</td>\n",
       "      <td>sadness</td>\n",
       "      <td>0.806</td>\n",
       "      <td>[@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...</td>\n",
       "      <td>[4991, 4992, 25, 2513, 0, 0, 0, 0, 0, 0, 0, 0,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>432</td>\n",
       "      <td>10432</td>\n",
       "      <td>@sarah_urbina why do you even beef Sara you le...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.479</td>\n",
       "      <td>[@, sarah_urbina, why, do, you, even, beef, Sa...</td>\n",
       "      <td>[4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text  emotion  \\\n",
       "490   10490  Take public opinion on revenge with Pakistan i...    anger   \n",
       "1875  21018         My goals are so big they scare small minds     fear   \n",
       "1146  20289  @PanicAtTheDisco hey, y'all announced it like ...     fear   \n",
       "2878  40051          @Christy_RTR @doge_e_fresh I'm despondent  sadness   \n",
       "432   10432  @sarah_urbina why do you even beef Sara you le...    anger   \n",
       "\n",
       "      intensity                                     text_tokenized  \\\n",
       "490       0.458  [Take, public, opinion, on, revenge, with, Pak...   \n",
       "1875      0.250  [My, goals, are, so, big, they, scare, small, ...   \n",
       "1146      0.625  [@, PanicAtTheDisco, hey, ,, y'all, announced,...   \n",
       "2878      0.806  [@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...   \n",
       "432       0.479  [@, sarah_urbina, why, do, you, even, beef, Sa...   \n",
       "\n",
       "                                         text_tok_keras  \n",
       "490   [208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...  \n",
       "1875  [10, 757, 28, 18, 340, 51, 480, 1425, 1426, 0,...  \n",
       "1146  [1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...  \n",
       "2878  [4991, 4992, 25, 2513, 0, 0, 0, 0, 0, 0, 0, 0,...  \n",
       "432   [4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "#pad sequences\n",
    "maxlen = 30 # arbitrary\n",
    "padded_tokens = pad_sequences(train_df['text_tok_keras'],  maxlen=maxlen, padding=\"post\")\n",
    "train_df['text_tok_keras'] = list(padded_tokens)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we map back to words to obtain the padded tokenized representations in the text_tok_keras_words column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>emotion</th>\n",
       "      <th>intensity</th>\n",
       "      <th>text_tokenized</th>\n",
       "      <th>text_tok_keras</th>\n",
       "      <th>text_tok_keras_words</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>490</td>\n",
       "      <td>10490</td>\n",
       "      <td>Take public opinion on revenge with Pakistan i...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.458</td>\n",
       "      <td>[Take, public, opinion, on, revenge, with, Pak...</td>\n",
       "      <td>[208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...</td>\n",
       "      <td>[take, public, opinion, on, revenge, with, pak...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1875</td>\n",
       "      <td>21018</td>\n",
       "      <td>My goals are so big they scare small minds</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.250</td>\n",
       "      <td>[My, goals, are, so, big, they, scare, small, ...</td>\n",
       "      <td>[10, 757, 28, 18, 340, 51, 480, 1425, 1426, 0,...</td>\n",
       "      <td>[my, goals, are, so, big, they, scare, small, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1146</td>\n",
       "      <td>20289</td>\n",
       "      <td>@PanicAtTheDisco hey, y'all announced it like ...</td>\n",
       "      <td>fear</td>\n",
       "      <td>0.625</td>\n",
       "      <td>[@, PanicAtTheDisco, hey, ,, y'all, announced,...</td>\n",
       "      <td>[1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...</td>\n",
       "      <td>[panicatthedisco, hey, y'all, announced, it, l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2878</td>\n",
       "      <td>40051</td>\n",
       "      <td>@Christy_RTR @doge_e_fresh I'm despondent</td>\n",
       "      <td>sadness</td>\n",
       "      <td>0.806</td>\n",
       "      <td>[@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...</td>\n",
       "      <td>[4991, 4992, 25, 2513, 0, 0, 0, 0, 0, 0, 0, 0,...</td>\n",
       "      <td>[christy_rtr, doge_e_fresh, i'm, despondent, ,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>432</td>\n",
       "      <td>10432</td>\n",
       "      <td>@sarah_urbina why do you even beef Sara you le...</td>\n",
       "      <td>anger</td>\n",
       "      <td>0.479</td>\n",
       "      <td>[@, sarah_urbina, why, do, you, even, beef, Sa...</td>\n",
       "      <td>[4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...</td>\n",
       "      <td>[sarah_urbina, why, do, you, even, beef, sara,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text  emotion  \\\n",
       "490   10490  Take public opinion on revenge with Pakistan i...    anger   \n",
       "1875  21018         My goals are so big they scare small minds     fear   \n",
       "1146  20289  @PanicAtTheDisco hey, y'all announced it like ...     fear   \n",
       "2878  40051          @Christy_RTR @doge_e_fresh I'm despondent  sadness   \n",
       "432   10432  @sarah_urbina why do you even beef Sara you le...    anger   \n",
       "\n",
       "      intensity                                     text_tokenized  \\\n",
       "490       0.458  [Take, public, opinion, on, revenge, with, Pak...   \n",
       "1875      0.250  [My, goals, are, so, big, they, scare, small, ...   \n",
       "1146      0.625  [@, PanicAtTheDisco, hey, ,, y'all, announced,...   \n",
       "2878      0.806  [@, Christy_RTR, @, doge_e_fresh, I, 'm, despo...   \n",
       "432       0.479  [@, sarah_urbina, why, do, you, even, beef, Sa...   \n",
       "\n",
       "                                         text_tok_keras  \\\n",
       "490   [208, 1187, 1424, 14, 209, 22, 282, 33, 2509, ...   \n",
       "1875  [10, 757, 28, 18, 340, 51, 480, 1425, 1426, 0,...   \n",
       "1146  [1427, 517, 518, 2512, 12, 27, 1428, 110, 3, 3...   \n",
       "2878  [4991, 4992, 25, 2513, 0, 0, 0, 0, 0, 0, 0, 0,...   \n",
       "432   [4993, 78, 50, 8, 114, 1813, 4994, 8, 138, 1, ...   \n",
       "\n",
       "                                   text_tok_keras_words  \n",
       "490   [take, public, opinion, on, revenge, with, pak...  \n",
       "1875  [my, goals, are, so, big, they, scare, small, ...  \n",
       "1146  [panicatthedisco, hey, y'all, announced, it, l...  \n",
       "2878  [christy_rtr, doge_e_fresh, i'm, despondent, ,...  \n",
       "432   [sarah_urbina, why, do, you, even, beef, sara,...  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#map back to words\n",
    "reverse_word_map = dict(map(reversed, tokenizer_keras.word_index.items()))\n",
    "train_df['text_tok_keras_words'] = train_df['text_tok_keras'].apply(lambda x_list: [reverse_word_map[x] if x>0 else \"\" for x in x_list])\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we use the Tensorflow Hub to charge a pretrained Elmo model. TensorFlow Hub is a library for reusable machine learning models. You can learn more here:\n",
    "Source: (https://www.tensorflow.org/hub)\n",
    "Make sure tensor has appropriate size!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "#load elmo\n",
    "elmo = hub.Module(\"https://tfhub.dev/google/elmo/3\", trainable = True)\n",
    "\n",
    "#generic way to generate an array of the same length\n",
    "token_len = np.empty(len(train_df))\n",
    "token_len.fill(maxlen)\n",
    "\n",
    "#create embeddings\n",
    "embeddings = elmo(inputs={\"tokens\": list(train_df['text_tok_keras_words']),\n",
    "                          \"sequence_len\": token_len},\n",
    "                  signature=\"tokens\",\n",
    "                  as_dict=True)[\"elmo\"]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([Dimension(3613), Dimension(30), Dimension(1024)])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check the Embedding layer dimension\n",
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To embed a word, you need to pass the position of the token. Let's take the first sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'@SusannahSpot I could pop round '"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['text'].iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this might take a long time, make sure you can run Tf on your computer\n",
    "init = tf.global_variables_initializer()\n",
    "sess = tf.Session()\n",
    "sess.run(init)\n",
    "\n",
    "print(\"SusannahSpot\")\n",
    "print(sess.run(embeddings[0][1]))\n",
    "\n",
    "print(\"I\")\n",
    "print(sess.run(embeddings[0][1]))\n",
    "\n",
    "print(\"could\")\n",
    "print(sess.run(embeddings[0][2]))\n",
    "\n",
    "print(\"pop\")\n",
    "print(sess.run(embeddings[0][3]))\n",
    "\n",
    "print(\"round\")\n",
    "print(sess.run(embeddings[0][4]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "oldHeight": 594.85,
   "position": {
    "height": "40px",
    "left": "723px",
    "right": "20px",
    "top": "80px",
    "width": "250px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "varInspector_section_display": "none",
   "window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}