svm-spam-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spam detection with Support Vector Machine\n",
    "\n",
    "E-mail is one of the most secure medium for online communication and transferring data through the web. With its increase in popularity, the number of ill-intentioned/unsolicited emails has also increased rapidly. To filtering data, there are different approaches which automatically classify these messages. The one I picked for this project was a support vector classifier, a method I first experimented within this Bootcamp.\n",
    "An SVM works well in a binary classification task such as this one. It separates spam from non-spam (aka ham) by drawing a separator known as a hyperplane between the classes.\n",
    "\n",
    "### Learning goals\n",
    "\n",
    "- Develop a deep understanding of SVM concepts and implement them\n",
    "- Practice data exploration and preprocessing\n",
    "- Feature extraction with PCA/SVD/LDA\n",
    "- Apply SVM to a non linear problem with a custom non linear kernel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:21.875111Z",
     "iopub.status.busy": "2021-12-28T12:25:21.874829Z",
     "iopub.status.idle": "2021-12-28T12:25:21.885031Z",
     "shell.execute_reply": "2021-12-28T12:25:21.884085Z",
     "shell.execute_reply.started": "2021-12-28T12:25:21.875081Z"
    }
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import nltk\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline  \n",
    "\n",
    "from collections import Counter\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn import svm\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.model_selection import GridSearchCV, train_test_split\n",
    "from sklearn.metrics import confusion_matrix, classification_report, accuracy_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:21.888022Z",
     "iopub.status.busy": "2021-12-28T12:25:21.887154Z",
     "iopub.status.idle": "2021-12-28T12:25:22.138615Z",
     "shell.execute_reply": "2021-12-28T12:25:22.137862Z",
     "shell.execute_reply.started": "2021-12-28T12:25:21.887977Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(11120, 2)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"datasets/emails_raw.csv\")\n",
    "\n",
    "df.describe\n",
    "\n",
    "# print the number of rows and columns\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:22.140978Z",
     "iopub.status.busy": "2021-12-28T12:25:22.140727Z",
     "iopub.status.idle": "2021-12-28T12:25:22.311068Z",
     "shell.execute_reply": "2021-12-28T12:25:22.310175Z",
     "shell.execute_reply.started": "2021-12-28T12:25:22.140939Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPUAAADnCAYAAADGrxD1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAX3UlEQVR4nO3deXxU5b3H8c9vZhKWEMMmoKIOAoossohat7pBtQxaa2ldWqsoUJcuqFXH1tbj1dZY9VavVu31pVdbXOpWF44gdakLbohFAigoMFRk3w4JIfu5f5yJDUhIQmbmOefM7/16zStjMpPzDeabc+bMc55HXNdFKRUeEdMBlFKZpaVWKmS01EqFjJZaqZDRUisVMlpqpUJGS61UyGiplQoZLbVSIaOlVipktNRKhYyWWqmQ0VIrFTJaaqVCRkutVMhoqQ0RkbiILDCdQ4WPllqpkNFSmxUVkQdEZKGIzBKRTiIyWUTmiMjHIvKMiHQGEJGHReQ+EXlPRJaJyIki8pCIfCIiDxv+OZSPaKnNGgj8yXXdIcAW4HvAs67rHuG67nDgE+DiJo/vBhwNXAG8APwRGAIME5EROcytfExLbdZy13Xnpe/PBeLAUBF5S0TKgB/ilbbRi643qVwZsNZ13TLXdRuAhennKqWlNqy6yf16IAY8DPzUdd1hwI1Ax108vmGn5zakn6uUltqHioHVIlKAt6dWqk30r7v//AZ4H1if/lhsNo4KGtF5v5UKFz38VipktNRKhYyWWqmQ0RNlIRRP2lHgAKB/+tYX74RbF6Ao/XHnWydgG94gGCf9sel9B9gMrAAWpEoTa3Lz06i20hNlARZP2hHgEGAoMCz9cQjQDyjI8uY34g16WZC+LcQr+6Ysb1e1QEsdMPGkPQAYm76dBHQ1GujrVgMfAq8Cr6RKEwsN58k7WmqfiyftHsApeCUeQ/CGg64GXgFeAmakShOO4Tyhp6X2oXjS7gtcCJwJjCQ8JzRrgTfxLkZ5OlWaWGU4TyhpqX0inrQLgNOBScCphKfIzakHpgP3Ay+nShP6i5ghWmrD4kl7EF6Rzwd6GY5jynLgAeDBVGlinekwQaelNiCetAvxLtaYBBxjOI6f1AJ/B/6cKk28ZjpMUGmpcyietGPAROB6vPeRVfMWA78DHk2VJhpMhwkSLXUOpN9PPg+4ARhgOE7QzAeuS5UmXjIdJCi01FkUT9qCN0XRjcBgw3GC7k3g2lRp4j3TQfxOS50l8aSdAP4LGGU6S8g8h7fn/tR0EL/SUmdYPGkfhPc2zVjTWUKsHm/ap+t1DPrXaakzJH0RxZWABXQ2myZvbAQuTZUmnjIdxE+01BkQT9rDgYfQQ21THgcuT5UmNpsO4gda6nZI752vxTurXWg4Tr5bBVycKk3MNB3ENC31Hoon7f7AX9DBI37zv8BVqdJEhekgpmip90A8aZ8NPIg34YDyn2XABanSxNumg5igpW6D9PvON+JN46v8rQFIpkoTt5kOkmta6laKJ+3OwCPABNNZVJv8BZiSKk1Ut/jIkNBSt0L6+ubn0bPbQfUu8N1UaWKt6SC5oKVuQTxpH4lX6D6ms6h2WQGclg8j0cJ+IX67xJP2ecAbaKHD4EBgdjxpH2s6SLZpqZsRT9rXAI+y46qTKti6A6/Ek/ZZpoNkk5Z6F+JJeypwq+kcKis6Ak/Fk3ZoVxTV19Q7iSftS4F7TedQWVcHfC9VmnjBdJBM01I3EU/aE/EGlYjpLConqoFxYZs6SUudlj4p9lf0JUm+qQDGpEoT75sOkilaaiCetCcATwBR01mUEZuBE1KliTLTQTIh70sdT9qnA8+Q/bWnlL+tAY5PlSY+Nx2kvfK61OnroN/FW/FRqRXAcanSxErTQdojb0sdT9pdgbnAQYajtNvK+y4iUtgJIhEkEmWfC+5ky9uPUvHxy0Q6lwDQ7Zs/plP/I6hauYhNs+5FojF6nn41Bd33o6GqgvXP30qvH9yISN6fUvgIOCbIY8Xzcn3q9NVW0whBoRv1Pvf3RNMFblQ8+kxKjtpxnMXWOX+n1wSLuq1rKZ83g+4nT8J552+UHP19LbRnFPBH4DLTQfZUvv5fvB5ImA5hgkRiuHXVuLXVSCRG7ebV1JVvoOMBh5mO5ieXxpP2OaZD7Km8O/yOJ+1T8ZZVDc0ftJX3X0y0YxcAuoz4NsUjTvMOv8teJdKhM4V9BtDt5ElEO3ahZu0yNr78J6SgkJ6Jq9j8+oN0Pf5HFHTfz/BP4TvlwOhUaWKJ6SBtlVeljiftA/FeR/cwnSWT6so3ECvuSf22Laz92/V0H3sJBd33I9JpLxBhy1vTqK/YRM9xU3d4XtUXC6hc8i7FI8ex5a1pSCRKt5MvJlrUzcwP4j/zgaNSpYkq00HaIjR7q5bEk3YH4GlCVmiAWHFPAKJFXel88NFUr1pCtKgbEokiEqF4+KnUrN5xh+O6rvda+phz2DL7MbqdOJEuw09l69wXTfwIfnUYcLfpEG2VN6UGbgdGmw6RaQ01VTRUV351v2r5vyjc+0DqKjZ99ZjKJe9S0PPAHZ63bcFrdDpoNNFOxbi11SACIt591dSkeNL+kekQbZEXh9/xpH083nXRoRvTXbtlDeufvdn7j4YGigafQMkxZ7Nh+h3UrF0GIsRKetH91J8S69Lde1htFeuevpHeP7gJicao+mIBm2bd95+3uXr0NfgT+dI2YHiqNLHUdJDWCH2p04fdHwOHmM6iAu2lVGkiEO+Y5MPh92/RQqv2GxdP2meYDtEaod5Tx5P2EGAeeTrIRmXccmCw38+Gh31PfQ9aaJU5/YDrTIdoSWj31OkRQY+bzqFCpwoYkipNLDMdpDmh3FPHk3YR3ltYSmVaR+Au0yF2J5SlBq4BdNyjypbx8aQ93nSI5oTu8DuetPfCuy62q+EoKtyWAYNSpYla00F2FsY99eVooVX2HQT4cqRZqPbU8aTdCUgBvQxHUflhCXBoqjTRYDpIU2HbU09GC61y52Dg+6ZD7Cw0pY4n7QLgl6ZzqLzzK9MBdhaaUgPnA/ubDqHyzmHxpP0t0yGaCkWp40k7AiRN51B5a6rpAE2FotTABGCg6RAqb50WT9q+uWgoLKWeYjqAymsC/MJ0iEaBf0srnrT7AF8Snj9QKpjKgV5+uIIrDEU4h3D8HCrYioFTTYeAcJThPNMBlEr7gekAEPDD73jSHgB8ZjqHUmm+OAQP+p5a99LKT4qBb5sOEfRSn2s6gFI7MT5sNLCH3/GkPQpvtQ2l/KQC2NvkIXiQ99SBXcBMhVoXYJzJAEEu9SmmAyjVDKOH4IEsdTxpdwGGm86hVDPGmNx4IEsNHA1ETYdQqhk902+3GhHUUh9nOoBSLfiGqQ1rqZXKDi11a8WTdgw4ynQOpVpwtKkNB67UwCigyHQIpVpwWHoizJwLYqn10FsFQQwYbWLDQSz1saYDKNVKRl5XB7HUQ00HUKqVtNQtiSftKBA3nUOpVjKyAwpUqYEDgELTIZRqJSOLNAat1DpjqAqSonjS7prrjQat1AeZDqBUG/XN9QZju/uiiJQDjRdcS/qjm77vuq67Vxaz7UrO/4GUaqe+wIJcbnC3pXZdtzhXQVpJF5JXQZPzHVGrD79F5DgRmZi+31NE+mUvVrO01Cpocv4726pSi8gNwLXAdelPFQLTshVqN7TUKmh8u6f+LnAGsA3Add1VeDMn5lo3A9tUqj18W+oa15uh0AUQEVMXVBQY2q5Se6p3rjfY2lI/KSJ/BrqKyGTgFeCB7MVqls52ooIm54Oldnv2u5HrureLyFhgK3Aw8FvXdf+R1WS71qq8SvlIzo8u21KSMqAT3iF4WXbitEhLrYIm57+zrT37PQn4ADgLb4H390TkomwGa4aWWgWNb/fUVwMjXdfdCCAiPYB3gIeyFawZ+po6gw6WL5ZPidkrTo3M2buIqn1M5wmjBmQrbM7pNltb6o14K/o1Kk9/LmfiSVsI3lh13zlcFn86JTZ97QmR+X07Sm1/wMQgorwRwd2a6222NPb7yvTdz4H3ReR5vNfU3wHmZznbzvTQew8IDQ0nRj4umxy1txwRWdy/QOoHAYNM58oj9bneYEtFaRxgsjR9a/R8duLsVjBX8jOggLqaROS9+RNjM7cPk+WDIuLqaibm1OV6g4Fa9TKetLcAJaZz+FFnqrZNiL5Zdn50Vv0AWTVURP+dfGIOlnNkLjfYqkNaEdkbuAYYAnRs/LzruidnKVdz1qCl/koJFVvOj/5j4Tmx1wv2Y8MwEXMTyKtmrc31Blv7OvVR4G/AeOAS4AJgfbZC7cYa4BAD2/WNfdi4ZmJsxpKzom8X92DrMBGdXdXnfFvqHq7rPigiv3Bd9w3gDRGZk81gzcj5P5Af9JcvV0yJ2qlvR9/vUcz2ISL0MZ1JtZpvS12b/rhaRBLAKqB7diLt1hoD2zRihHy+eEps+pqTIvP27SQ1A4EDTWdSe8S3pb5ZREqAq4C7gb2AqdkKtRshLrXrfjMyf8Hk6EubvhFZ1K9A6g8hz19qhIQ/S+267vT0XQc4CUBEpmYp0+6EqtQx6mpPi8wpuyg2o2K4LD0kKu4w05lUxvmz1M24ErgzQzlaK/Cl7kj19rOib82/IDqr7mBZOVSEUaYzqazK+e9se0otLT8k41Yb2Ga77UWFc170tQXnRV+N7S/rh4noUrx5ooYdB23lRHtKbWLUymK8YXe+v7CjN5vWXRh7efH3om923hvnMH3rKS99iuXUtvywzGrLvN87fAnv2uqcSpUmtseT9iLAl68947L6i8lRe9n46Hvd9qJyqAi9TGdSRuX6+gggePN+A3yIj0o9TJZ9NjlmrxoTmbtPZ6k5GNjfdCblG/4rtU99CEw0t3nXPTaycNGkqL3+2MjCeKHUDUTX+FK7pqVupQ9zvcEo9XVjI3PLJsVeKh8pnw+MSsOQXGdQgaSlbqWP8Ua4ZXWamA7UVH0nOnv+hdGXawfJvwdHhJHZ3J4KnfVYjpF3awJ16WWjeNL+CDJfsi5Ubj03+vqCH0ZfiRwoa4eK0CXT21B5YwaWM87EhoO4pwbvEDwjpe6Bs2FibOYnE6JvdurN5sNEOCYT31flPRNTaAPBLfUcYPKePnl/WfflpKi99IzoO127sm2oCMdnMJtSALNMbTiopX69rU84VFYsnRKbvvJbkQ97F0n1IHSxPZU9X2I5C01tPJCvqQHiSftTWriK6ShZtGhyzF53fKTsgA5Sd1COoin1CJZzoamNB3VPDTCdnUodoaH+5MhHCybHXnIOlyUDYtIwGBhsJp7KY8YOvSHYpX4RuKqQ2urTI+/OnxibWT1YVhyqM2cqw1wMniSDYJd6tl143azBsuJoEY4wHUaptI+wHBPz930lsCtepEoTdUMiK1aK4Mfx6Sp/PW46QGBLnfak6QBKNVGPN/OuUUEv9avkeE0vpXZjFpZjfHaeYJfacurwweGOUmmPmA4AQS+1517TAZQCtmBmjbmvCX6pLecT4DXTMVTeewrLqTIdAsJQas89pgOovOeLQ28IT6lfAL4wHULlrTlYzmzTIRqFo9SWUw/cbzqGylu3mA7QVDhK7XkAqDYdQuWdT4DnTIdoKjyl9obmTTMdQ+WdW7EcX13qGJ5Se25E99Yqd1bggxFkOwtXqS3nC+A+0zFU3rg9PQDKV8JVas/vgXLTIVTorQMeNB1iV8JXau+19X+bjqFC79dYznbTIXYlfKX23AFsMB1ChdaHwEOmQzQnnKW2nHJ89t6hCg0X+DmW02A6SHPCWWrPPXjvISqVSdOwnHdNh9id8JbacmqASYBv/6KqwCkHrjUdoiXhLTWA5byDXpqpMucmU+tjtUW4S+25Dm+QgGpGfYPLyD9XMP6xSgAufG47/e4qZ8T9FYy4v4J5a+oBeGZRLUPureD4/9vGxkrvAGjppgbOfrrSWPYcWgDcaTpEa4S/1JZTAfzEdAw/u+v9Gg7tueOvwm1jOzLvki7Mu6QLI/pEAbj7gxrmTC7iJ4cX8FiZN+bi+teruPmkDjnPnGNVwHlYTq3pIK0R/lIDWM7LwF9Mx/CjlVsbsD+rY9KowhYfGxGoroPKWiiIwlsr6uhTFGFgj2gOkhp1LZZTZjpEa+VHqT1XAKtMh/CbqTOr+MOYjkRkx8//+rVqDruvgitmVlFd512vcN1xHRjz1228uKSOc4cWcNOb1fzmhNDvpWdgOf9jOkRb5E+pLWcT8H28BesVMH1JLb2KhMP33XFPe8spHfj08iLmTC5iU5XLrbNrABjbP8bcKV148dzOPL+4lnEDYyzZWM+EJyuZ/MJ2Kmt9dbFSJqwDJpoO0Vb5U2poPBt+lekYfjH73/W8sLiO+J3lnPP0dl5bXsePnt3OPsURRIQOMWHiiAI++LJ+h+dV1ro8PK+Wy48o5IZ/VvPImZ047oAoj84P3d/Li7CctaZDtFV+lRrAcu4GHjMdww9uGdORlVcWk5pazBMTOnFyvxjTzurE6nLvzLbrujz3aR1De+10Em12DT8/qpCCqLC9FkS819sh21Pfg+XYpkPsiSCvpdUek4Fh6ZvayQ+f3c76ShfXhRF9otw/vuNXX1tV3sAHq+q54UTvtfTPjizkiAe20bWj8NzZnUxFzrR/AleaDrGnArs+dbtZJQOBOUCJ6SjKV5YCR2E5gV35Jf8OvxtZzmfA+XjrHykF4ABnBLnQkM+lBrCcF9GBKcpTA3wXy1lkOkh75XepASznQeCXpmMoo1xgIpbzuukgmaClBrCcO4DfmY6hjLkWywnNOyL5e6JsV6ySe4DLTcdQOXUNlnOb6RCZpHvqHf0MH075qrJmatgKDVrqHXmTsl+ILgoQdi5wGZZzl+kg2aCl3pk3j/OPgdtNR1FZ0QBMxnJCOz+8vqbeHavkCryZSaWlh6pAqMcbzx3qy3C11C2xSs7BW3u45QuOlZ85wLlYzgzTQbJNS90aVskpwN+BYtNR1B5ZgjdSbLHpILmgr6lbw3JeBY4DPjcdRbXZDODIfCk0aKlbz3LmA4cDT5qOolrtNmA8luOYDpJLevi9J6ySy/DW6wr9XD4BtR3vDHdejjnQUu8pq2Qk3l57gOkoagfvARdgOUtMBzFFD7/3lOX8Cz0c95Ma4FfAcflcaNA9dWZ4b3vdBfQyHSVPzQN+HKRpfLNJ99SZYDlPAIcCDxtOkm/qgJvwzm5rodN0T51pVsmJeCtuDjGcJOxeBa7Ccj42HcRvtNTZYJXE8K74soC9zIYJnU+Aq4M602cuaKmzySrZG7ga7xrtzobTBN064AbgASxH55XbDS11LlglvfDKfRla7rbajrfa5C1YTrnhLIGgpc4lr9zXAJei5W7JWrxzE/cFfXbPXNNSm2CV9MZbsO8iYG/DafxmAd5ovcewnGrTYYJIS22SVVIAfAeYAowhv6/bngXcgeXMMh0k6LTUfmGV9AMuxltlcV/DaXKlDHgceALLWW46TFhoqf3GKokCpwFnAglgH6N5Mu8z4Am8Igd+4nw/0lL7mVUiwChgfPp2OME7RK8D5gKvAc9gOXMN5wk9LXWQWCV9gHHAN4HRwCAgutvn5F4t8CHeypFvALOxnAqjifKMljrIrJIiYCTeHnx0+nYwuRvTXw18infGegFemd/BcipztH21C1rqsLFKOgF9gf3Tt6b39we6402iWIg3yUMhX/8jUA5sATanb433NwDL8KZ1Wgqs0NFd/qOlVo0n5wqBAqAyPfe5CigttVIho9dTKxUyWmqlQkZLrVTIaKlVu4lIkYjYIvKxiCwQkbNFJCUifxCRMhH5QEQGpB97uoi8LyL/EpFXRKR3+vOWiDwiIm+JyAoROavJ82eKSIHZnzI4tNQqE04DVrmuO9x13aHAzPTnHdd1h+FdQnln+nNvA99wXXck3nDRa5p8n/7AycAZeMsJv55+/na8IbOqFbTUKhPKgLEicquIHO+6buOKGI83+Xh0+n5f4GURKcObOKLpXG4zXNetTX+/KP/541AGxLOYP1S01KrdXNddgjdGvQy4WUR+2/ilpg9Lf7wbuCe9B/4J0LHJY6rT368BqHX/835rAxDLUvzQ0VKrdhORfYFK13Wn4a1fNSr9pbObfHw3fb8E+DJ9/4Kchcwj+tdPZcIw4DYRacC7oONS4Gmgm4jMx9sDn5t+rAU8JSKb8a7c6pf7uOGmI8pUVohIChjtuu4G01nyjR5+KxUyuqdWKmR0T61UyGiplQoZLbVSIaOlVipktNRKhYyWWqmQ0VIrFTJaaqVCRkutVMhoqZUKGS21UiGjpVYqZLTUSoWMllqpkNFSKxUy/w8FELqUxQZqswAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "classes = pd.value_counts(df[\"Label\"], sort= True)\n",
    "classes.plot(kind = 'pie',  autopct='%1.0f%%')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:22.314026Z",
     "iopub.status.busy": "2021-12-28T12:25:22.312666Z",
     "iopub.status.idle": "2021-12-28T12:25:22.870688Z",
     "shell.execute_reply": "2021-12-28T12:25:22.869339Z",
     "shell.execute_reply.started": "2021-12-28T12:25:22.313964Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAE6CAYAAAAcDUnBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgG0lEQVR4nO3debhcRZ3/8fcHwr4vUWQJkVVRB9SoKKgs6oAoOAgiAyJrRFFx1HFwBgX9qQM66ijoT4Ig6wiIMCIIikCAsCckLAZkDSKg7BJQRPA7f1Q1Oenc3Nvn3j7dncrn9Tz93D5LLX1u97er69Spo4jAzMzKs0S/K2BmZs1wgDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wFtjJH1F0qOS/tDvuiyqJIWkjWqm2UvSr0ZZ3pGSThtNWhs8DvCFkjRH0nOS1mxbPzMHjYljzH/YwCNpAvAZYLOIWGssZfWapKmSDux3PUYrIk6PiHf1ux7Wfw7wZbsX2LO1IOk1wPI9KnsC8FhEPDzURknjelSPRYKPhzXBAb5spwL7VJY/DJxS3UHSKpJOkfSIpPskHS5pibxtI0mXS/pT7mo5M6+/Iie/SdLTkvZoy/MdwMXA2nn7SZIm5lb/AZJ+B1ya991f0m2SnpD0S0nrV/J5p6Tbc/nH5rocmLfN15VQyX9c5XWdIOkhSQ/k7qIl87Z9JU2T9F+53Hsl7Zi3fRV4K3Bsrvux7QdV0smSPpOfr5PLPSQvbyjp8coxPEjSXXndeZLWruQTkg6RdCdwZ173r7nOD0rav63cd0uaLWlufk2fHeqf3np9beUcLOlOSU9K+p4kDZU2Wzq/J+ZK+o2kSZW8DpN0d942W9I/tZV7laRv53LukfSWvP5+SQ9L+vAw5Vq3RYQfBT6AOcA7gN8CrwSWBH4PrA8EMDHvdwrwM2AlYCJwB3BA3vZj4D9IDYFlga0r+Qew0TDlbwP8vrI8Mac5BVgBWA7YBbgr128ccDhwdd5/TWAusBuwFPAvwPPAgXn7kcBpQ+Q/Li+fCxyXy3oJcD3wkbxtX+BvwEH5uHwUeBBQ3j61Vc5CXtv+wM/z838G7gbOrGz7WX6+HfAo8DpgGeAY4Iq2Y3gxsHo+HjsAfwRenev9P9XjDDwEvDU/Xw143ULqty8wra2c84FVSb+sHgF2WEjaI4FngXfnY/OfwLWV7bsDa+f3xB7AM8DLKuU+D+yX034F+B3wvfz635X/pyv2+/OxuDz6XgE/GvrHzgvwh+cP6Q45mIzLH/iJ+UP4HKmfvJXuI8DU/PwUYAqw7hD5jzbAb1BZdyH5yyQvLwH8mfQltE9bYBHpC2rEAA+8FPgrsFxl+57AZfn5vsBdlW3L57Rr5eWpDB/gNwSeyPX9QT5mv8/bTgY+nZ+fAHy9km5F0hfLxMox3K6y/UTgqMryJswf4H+Xy1p5hP/9viwY4KtfzmcBhy0k7ZHAryvLmwF/GaasWcAulXLvrGx7TS77pZV1jwFb9Pvzsbg83EVTvlNJrcx9aeueIbWSlwLuq6y7D1gnP/8cKbBen3+q78/Y3V95vj7wnfxz/kng8VzeOqRW4ov7RooO1bTDWZ/0uh6q5H0cqSXf8uLInoj4c366YieZR8TdpJbrFqTunPOBByVtCrwduDzvujaVYxsRT5MC3DqV7Kqvae225er/BeD9pJb1fbm76s2d1DerjmT6M8O/1vZ9l610fe0jaVbluL6a9D5q+WPl+V8AIqJ9XUfH2cbOJ3YKFxH3SbqXFBgOaNv8KKlFuT4wO6+bADyQ0/6B1I2BpK2BX0u6IiLuGkuVKs/vB74aEae37yRpY2C9yrKqy6QAWz1hXB2pcz+pBb9mRDw/xjouzOWk7qOlI+IBSZeTznGsRmrVQur2qZ5TWAFYg3x8hyjrIeZ/jRPmq1TEDcAukpYCPk5qiVf3b1Q+P3I8sD1wTUS8IGkW6UvZBpBb8IuHA0hdAc9UV0bEC6Qg8VVJK+UP8KeB0wAk7S5p3bz7E6Rg9Pe8/EdggzHW6wfA5yW9Kpe3iqTd87YLgFdJ2jW3Hj/J/EF8FvA2SRMkrQJ8vvK6HgJ+BXxT0sqSlsgnP9/eYb06eW2Xk4Js64Tz1Lw8LR9XSOcw9pO0haRlgK8B10XEnIXkeRawr6TNJC0PHNHaIGlppfHtq0TE34CnmPe/6JUVSO+BR3Kd9iO14G1AOcAvBiLi7oiYvpDNnyC1hu8BppFO7J2Yt70BuE7S08B5wKERcU/ediRwcv6p/oFR1utc4GjgDElPAbcCO+Ztj5JO6B1F6tbYGLiqkvZi4EzgZmAGqZukah9gadIvkyeAs4GXdVi17wC75RE2313IPpeTTky3Avw00i+K1jIR8WvgC8BPSa3zDYEPLqzQiLgQ+G/SCKO78t+qDwFz8rE6GNirw9fTFRExG/gmcA3pS/A1VP4nNnhaowbMBp6kqaQTqz/sd13MFgVuwZuZFcoB3sysUO6iMTMrlFvwZmaFcoA3MyvUQF3otOaaa8bEiRP7XQ0zs0XGjBkzHo2I8UNtG6gAP3HiRKZPX9hwbTMzayepfUqLF7mLxsysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCtXoKBpJc0i36HoBeD4iJg2fwszMuqUXwyS3zVO/mplZD7mLxsysUE234AP4laQAjouIKe07SJoMTAaYMGG+O5Qx8bALhs18zlE7jViBseYxUvpO62Fm1mtNt+C3jojXke7Sc4ikt7XvEBFTImJSREwaP37Iq23NzGwUGg3wEdG6efPDwLnAG5ssz8zM5mkswEtaQdJKrefAu0j33DQzsx5osg/+pcC5klrl/E9EXNRgeWZmVtFYgI+Ie4DNm8rfzMyG52GSZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRXKAd7MrFAO8GZmhXKANzMrlAO8mVmhHODNzArlAG9mVigHeDOzQjnAm5kVygHezKxQDvBmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRXKAd7MrFCNB3hJS0qaKen8pssyM7N5etGCPxS4rQflmJlZRaMBXtK6wE7AD5ssx8zMFtR0C/6/gc8Bf2+4HDMzazOuqYwlvQd4OCJmSNpmmP0mA5MBJkyY0FR1+mriYRcMu33OUTs1nsdI6Tuth5ktOppswW8F7CxpDnAGsJ2k09p3iogpETEpIiaNHz++weqYmS1eGgvwEfH5iFg3IiYCHwQujYi9myrPzMzm53HwZmaFaqwPvioipgJTe1GWmZklbsGbmRXKAd7MrFAO8GZmhXKANzMrlAO8mVmhHODNzArlAG9mVigHeDOzQjnAm5kVygHezKxQDvBmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRXKAd7MrFAO8GZmhXKANzMrlAO8mVmhHODNzArlAG9mVigHeDOzQjUW4CUtK+l6STdJ+o2kLzVVlpmZLWhcg3n/FdguIp6WtBQwTdKFEXFtg2WamVnWWICPiACezotL5Uc0VZ6Zmc2v0T54SUtKmgU8DFwcEdc1WZ6Zmc3TaICPiBciYgtgXeCNkl7dvo+kyZKmS5r+yCOPNFkdM7PFSk9G0UTEk8BlwA5DbJsSEZMiYtL48eN7UR0zs8VCk6NoxktaNT9fDngncHtT5ZmZ2fyaHEXzMuBkSUuSvkjOiojzGyzPzMwqmhxFczPw2qbyNzOz4flKVjOzQjnAm5kVygHezKxQI/bBS1p9uO0R8Xj3qmNmZt3SyUnWG4H1gCcAAasCv8vbAtigkZqZmdmYdNJFczHw3ohYMyLWAN4D/CoiXh4RDu5mZgOqkwC/ZUT8orUQERcCb2muSmZm1g2ddNE8KOlw4LS8vBfwYHNVMjOzbuikBb8nMB44FzgnP9+zyUqZmdnYjdiCz6NkDpW0QkQ804M6mZlZF4zYgpf0Fkmzgdvy8uaSvt94zczMbEw66aL5NvCPwGMAEXET8LYmK2VmZmPX0ZWsEXF/26oXGqiLmZl1USejaO6X9BYg8s2zDyV315iZ2eDqpAV/MHAIsA7wALBFXjYzswE2bAs+36zjOxGxV4/qY2ZmXTJsCz4iXgDWl7R0j+pjZmZd0kkf/D3AVZLOA14cBx8R32qsVmZmNmYLbcFLOjU/3Rk4P++7UuVhZmYDbLgW/OslrU2aGviYHtXHzMy6ZLgA/wPgEuDlwPTKeuF54M3MBt5Cu2gi4rsR8UrgRxGxQeXheeDNzBYBI46Dj4iP9qIiZmbWXb7ptplZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCtVYgJe0nqTLJM2W9BtJhzZVlpmZLaiTG36M1vPAZyLiRkkrATMkXRwRsxss08zMssZa8BHxUETcmJ/PBW4j3bjbzMx6oCd98JImAq8FrutFeWZm1mwXDQCSVgR+CnwqIp4aYvtkYDLAhAkTmq6OjcHEwy4YcZ85R+00pjxGSt+NPHrxOrqRRyfHwmw4jbbgJS1FCu6nR8Q5Q+0TEVMiYlJETBo/fnyT1TEzW6w0OYpGwAnAbRHxrabKMTOzoTXZgt8K+BCwnaRZ+fHuBsszM7OKxvrgI2Ia6QbdZmbWB76S1cysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRXKAd7MrFAO8GZmhXKANzMrlAO8mVmhHODNzArlAG9mVigHeDOzQjnAm5kVygHezKxQDvBmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRWqsQAv6URJD0u6takyzMxs4ZpswZ8E7NBg/mZmNozGAnxEXAE83lT+ZmY2PPfBm5kValy/KyBpMjAZYMKECX2ujdmiZeJhFwy7fc5RO40pfTfyGCn9oORR0rFo6XsLPiKmRMSkiJg0fvz4flfHzKwYfQ/wZmbWjCaHSf4YuAbYVNLvJR3QVFlmZragxvrgI2LPpvI2M7ORuYvGzKxQDvBmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUA7wZmaFcoA3MyuUA7yZWaEc4M3MCuUAb2ZWKAd4M7NCOcCbmRXKAd7MrFAO8GZmhXKANzMrlAO8mVmhHODNzArlAG9mVigHeDOzQjnAm5kVygHezKxQDvBmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUI0GeEk7SPqtpLskHdZkWWZmNr/GArykJYHvATsCmwF7StqsqfLMzGx+Tbbg3wjcFRH3RMRzwBnALg2WZ2ZmFYqIZjKWdgN2iIgD8/KHgDdFxMfb9psMTM6LmwK/HSbbNYFHx1i1seYxCHUYlDwGoQ6Dkscg1GFQ8hiEOgxKHr2ow/oRMX6oDePGWPCYRcQUYEon+0qaHhGTxlLeWPMYhDoMSh6DUIdByWMQ6jAoeQxCHQYlj37XockumgeA9SrL6+Z1ZmbWA00G+BuAjSW9XNLSwAeB8xosz8zMKhrroomI5yV9HPglsCRwYkT8ZozZdtSV03Aeg1CHQcljEOowKHkMQh0GJY9BqMOg5NHXOjR2ktXMzPrLV7KamRXKAd7MrFAO8IspSWv1uw5m1iz3wS/CJL0MeDwi/jqKtDdGxOtq7P/ptlVBuvhiWkTcWyMfRdubTtIyo3kNg0DSasDGwLKtdRFxRY/KfgXp6vB18qoHgPMi4rZR5LU0sEle/G1E/K07tbR+Wmxa8JK2lrRffj5e0stHkcdawy2PkPalkt6THy+pW/ZCnArcLum/RpFWNfdfqe2xMjAJuFDSB2vkc8J8lZBWBH5Rsy5I2kTSJZJuzcv/IOnwmnksL+kLko7PyxtLek+N9AcCV5BGin0p/z2yTh1GS9K/kab/EHB9fgj4cd2J/SRtA9xJmjvq+8Adkt7WYdpPD/eoUYe5kp5qe9wv6VxJG9R5PW35fkzSHpJ6clFn/pyfIOnCvLyZpANGkc85knaSNLYYHRGL3AM4sub+RwA/B+7Iy2sDV42i3AuGWx4m3QeA+4CTgVOAe4HdunQsBLxqFOk+1qXyVwdurLH/l4Hv5+erAVcD+42i3MtJ8x3NrKy7tWYeZwKfa6UDlgdm1Uh/C6nlPisvvwI4p8O00/LfucBTlcdc4KkO0t8BLDXE+qWBO2sehxnAppXlTYAZHaY9YrhHjTr8P+AjzGs8TAaOBvYApo7h/XkIcAzpl81o85hSY98L8+f9prw8DrhlFGW+AzgduBs4qvr/qZXPaF90Px/Ae2vuPysHwpmVdTf3sL43AS+pLI9vvQFKeFSPa4f7fx34AeliuPePsswb2suuE5zz/tOHyKPj/0ulDrOAZfLz3/TomN9OmoOkff36pC6WOnkt8Fno5edjYce98sVZ53/y8k7W1azb60fxnpjZ/jpGWfYqwMHA/eTG0FBf7At79H0umtGIiJ/XTPJcRISk1OSVVmigWsNZIiIeriw/RiHdY5K2BZ7oYL9dK4vXAV8gdSuEpF0j4pyaRT8qaUPSuYDW5HYP1czjOUnLVfLYEKhzLuD3klYF/he4WNITpF9qvfAp4BJJd5I+/AATgI2Ajy8s0UJMl/RD4LS8vDcwvZOEkj4XEV+XdAz5OFZFxCc7rMOfJX0AODsv7wY828qmwzwAfgq0n1s6G3h9jTzmExEzauz+jKQ1mPee2hL402jKzfnsDXwImElq0W8NfBjYppM8FskAL+k9EXF+h/sKOF/SccCqkg4C9geOb7KObS6U9Evgx3l5D0bR79xPkm5hwQ/a6sCDwD4dZPHetuWZwFJ5fQB1A/whpCv8XiHpAVK311418zgCuAhYT9LpwFbAvp0mjoh/yk+PlHQZqbV1Uc06jEpEXCRpE1I3VfUk6w0R8ULN7D5KOp6fyMtXkvriO9E6odvRF8Iw9gK+k8sN4Fpg7/wFPOIXVj7h/CpglbbGxMpUToCPhqTJkSZF7MSnSVOybCjpKtKv9d1GUea5pNl1TyX1WLQaL2dK6vhYL5KjaCR9KSKOqLH/LaQD/y5SV80vI+Lipuo3RPlHk1qtW+dVVwJbRsS/9aoOYyVp/bZVATwWEc/0qT7LkD44E0lfNE8BERFfrpHH6qT3w5b577XASlFjVNCiTNIuwLoR8b28fD0pIAXwuYg4e7j0bXntHhE/GWldU/JreR+wM/PPeTUXOCMiru4wn/eSzq39vbLuIxFxXI26jCMFZzHKEUmSto2Iy+qmWyCfRTTArxURf6ix/8nAsRFxQ4PVGq78BYYkSro5Iv6hH/XpJ0njgYNIgfnFX5ARsX/NfC4CngRuBF5ssUbEN2vkcRWwY0Q8lZdfCfwkIl5dpy6Lqvz6PxgR9+flWcB2wIrAjyJi+xp5DfUe73gobhffF2+OiGvqpGlLfxrwZlJXz4kRcXuH6XYdbnunXZDdyqdlkeyiAX4IdDycDXgTsJek+4AXW5xNB1hJHwU+Bmwg6ebKppWAq5ose4D9jPQL5tdUAvMorBsRO4yxLl8Dfi7p3aQRMKdQv5tnUbZ0K7hn0yLiceDxTs9TSdoReDewjqTvVjatDDxfoy7del/cJenfGeUXRUTsLWllYE/gpHze7kTSr4C5wyRt74KcL1s674LsVj7AItqCr2uI7gUAIqLRE2KSViENBfxPoDo2eW7+IC12JM2KiC26kM8U4JiIuGWM+byPNFRyJdKInjvGWrdFhaS7ImKjhWy7OyI27CCPzYEtSMNfv1jZNBe4LCJGPAGf8+nW++Jq0hfFDOb/ZffTmvmsQTq5+SnSeYaNgO9GxDFjrWMvLRYB3gaHpK8AV0fEqE4yV072jiNdQXoPaeSLSH3wI/4qG2LEx/ak8cZzoNbIj0VaPrE8NSKOb1v/EWCbiNizRl5LjaavuZJ+TO+LSj5j+qLIffn7kgL6KcDJEfGwpOWB2RExcYT0a5BO3m9Neo9NA74cEY/VrMcywPtZ8JdIx+eYwAHeekzSXGAFUlD+G/MC88odph/y11hLJ7/KJH14hDxO7qQuizqlK6r/l/S/uDGvfj2wDPC+iPhjjby2Il3Fuz4pILX+rx1dhVp5XzxHel9AjfdFJZ+xNiBOBk6IIaabkLR9RFwyQvqLSVc3t4ac7kX6snxHzXpcRBpe2f5LpONzTOAAb7bYk7QdaYghpAu1Lh1FHrcD/8KCAalWy3WsutCA+AJwUvXcRJ1hkpJubT9JL+mWiHhNp69hYfmMhgO89YSkV0TE7ZKGHFURETcOtb6hupwVER9YyNj+xk++l0jSdRHxpjHmsTPQmgNnaqfXunSTpIeBR4CPt4Yp1hwN9C3SBXxn5VW7AW+MiM/WrEd3zjE5wFsvSJoSEZPzBUHtIiK262FdXhYRD/Xr5HuJJB1FujXnOVSuBu70izunfwPpak1Io1imR8TnO0zflQaEpJmkGTp/ApwdEd+QNDMiXjtCurmkxoJIvyBa4+iXAJ4eRVfTbEZ5jmm+fBzgzWysKl/crYDSCkgdfXHnYcRbtC4wkrQkaT6XjgJatxoQrWAuaVng/5OuCXhNRLyik/TdkhsfqwFvzauuAJ6s2/hYVMfB2yJM0ltYcHTAKT0sv9XaWmATozixZwBMHWJd3dbjqkBr+PAqdRJGxOT8d9uaZbabnvN5FthP0iHUnMdG3blHwPuAA0m/iESasuB40syYndfFLXjrJUmnAhuSZmBsnYyLxWVoYqkkfaayuCzpQsTbOr3ASOmeAkcDl5EC2tuAwyLizJr1WIo0t86LffnAcWMZwlmz/AOBQ4F1Se/xLYFr6nZB5l80b448FUi+8Owad9HYQJN0G7BZ+I1XtDyO+5cRsU0H+y5BOhl5JakfHuD6qDEdSSWvH5ImsWsNdf0Q8EJEHNhh+o1JFyZuxrwWeHRy0VdOfwvpNVwbEVsoTYL2tYgYdgqCheWTf0mQu4xuqDsax1001mu3AmtRf2pfW7QsT2rFjigi/q407fBZzD9R2Gi8ISI2ryxfKummGul/RLpQ6dvAtqT51+tM7f1sRDwrCaVbUd4uadMa6av1uE5pVklIXTYnLHz3oTnAW09I+jmpT3YlYLbSzIXV0RY796tuNnZtQ06XJM1KWeeqy19L+izpLlvV+aLqTunxgqQNI+LuXK8NqDe3zXIRcYkk5ROaR0qawfzTMAynK/cIiIhvSZrKvBlo94uImXXzcReN9YSkt5P6Vo8mzf3y4ibg6LGOobb+ahty+jzwx4joeLIxSUNN0dzxlbCVfLYntX7vyasmkoJjR1Pv5rlstibdJORS0hz7R0VE7VZ4fs+vAlwUEc/VTd8NDvDWU0NdNKLFdOpk677cV/0Z0vxCT5JuC/ntVl92B+nfQJpcbFXSfWJXBr4REdeOkG714baP4pdIVzjAW0+oMnUyaWKvlpVIN0Dfuy8Vs4Egaci7gtUdPivpLNLNX1oXTP0zsGpE7N5B2iVJvyZrXXWa097LvAud2tX+JdItDvDWE5462YaTZ/hsWZbUAr8xImrd7k7S7IjYbKR1w6S/NiK2rFPmIPNJVuuJiPgTaXa8jqegtcVHRHyiupxPVJ4xiqxulLRlq0tF0puod7/YmZLOI01VUD3ZO+yNNgZprqUqt+DNbODkC5ZurXtyM19nsSnwu7xqAvBb0onfEedykfSjIVbHSBdsDdJcS1VuwZtZ31WG0UIaZrkZ82ZkrGOst3FcAjg0Ip7M9VoNGHEO9i5OldBVbsGbWd/lIYWtYPQ8cF9EPNCHeiwwc2Qns0m27f9q5r8StqdzLVW5BW9mfSNpWkRsDZzP/KNQQumG14+Thil+v0dVWkLSapHvJZuHP3YcJyUdAWxDCvC/AHYk3bavLwHeLXgzG1hK9zi9ejQXGo2yvH2AfyedZAXYHfhqRJzaYfpbgM1JUx1vLumlwGkR8c5GKjwCt+DNbGBFxGOStulheadImg60ToruGhGza2Txlzy3zvOSVgYeBtbrekU75ABvZgMtIno6MV0O6HWCetX0PMTzeNL9aZ8GrulS1WpzF42ZWQMkTQRWjoib+1YHB3gzs+7IF0mdAfysdbOOfqozz7GZmQ3vm6TZKGdLOlvSbnkCtL5wC97MrMvyxGXbAQcBO/TrPr8+yWpm1kWSlgPeC+wBvA44qV91cReNmVmX5OmKbyO13o8FPkyaeqEvHODNzLrnBOADwFzSnaW+RAr4feEuGjOzMZK0CWkq7D2BR0n3llW/Jx/zSVYzszGS9HfgSuCAiLgrr7unX3dyanEXjZnZ2O0KPARcJun4fPPvoW7f11NuwZuZdYmkFYBdSF0125FmkTw3In7Vl/o4wJuZdV++WcjuwB4RsX1f6uAAb2ZWJvfBm5kVygHezKxQDvBmYyRpX0nH9rseZu0c4M1qyhNJmQ08B3hbrEj6V0mfzM+/LenS/Hw7SadL2lPSLZJulXR0Jd3Tkr4p6SbgzZL2k3SHpOuBrSr77Z7T3iTpil6/PrMqB3hb3FwJvDU/nwSsKGmpvO4O4GjS+OUtgDdIel/edwXguojYHLibNMfIVqS5vzer5P9F4B/zfjs3+krMRuAAb4ubGcDr8w2R/0q6X+YkUoB/EpgaEY9ExPPA6cDbcroXgJ/m52+q7Pccad6RlquAkyQdRB9nETQDB3hbzETE34B7gX2Bq0kt+m2BjYA5wyR9NiJe6CD/g4HDgfWAGZLWGGOVzUbNAd4WR1cCnwWuyM8PBmYC1wNvl7RmPpG6J3D5EOmvy/utkbt3dm9tkLRhRFwXEV8EHiEFerO+8HTBtji6EvgP4JqIeEbSs8CVEfGQpMOAy0gTRV0QET9rT5z3O5LUvfMkMKuy+RuSNs7pLwFuavKFmA3HUxWYmRXKXTRmZoVygDczK5QDvJlZoRzgzcwK5QBvZlYoB3gzs0I5wJuZFcoB3sysUP8HwiW8+Jx4qG0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAE2CAYAAABsoZXqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAmvklEQVR4nO3debgcZZn+8e8NBATZFCKyhYDgIG4oAUHFBZdBBsFRUPjhAqIRxwXHXccFmHFGHHcYRRQQEBUElcjIKMouCCQx7CgIIpsS2YMIBO/fH+97SKfpk9N9TnefJHV/rquvU9VV9dbTy+mn6l2qZJuIiGiuFSY7gIiImFxJBBERDZdEEBHRcEkEERENl0QQEdFwSQQREQ2XRBCTQtJ/SPqLpD9NdizLKkmWtHmP2+wj6eeDiimWTUkEDSTpD5IekrRu2/O/qT8u0ydY/hJ/oCRNAz4AbGX7yRPZ17BJOlvS2yY7jvGyfYLtV052HLF0SSJorhuAvUdmJD0TWG1I+54G3GH79k4LJa00pDiWCXk/YtCSCJrreODNLfNvAY5rXUHSWpKOkzRf0o2SPiFphbpsc0nnSLqnVvGcWJ8/t25+qaQFkt7QVubLgTOADeryb0uaXs8i9pf0R+DMuu5bJV0t6S5JP5O0SUs5r5B0Td3/4TWWt9VlB0n6Tsu6I+Wv1PK6jpJ0m6RbajXVinXZvpLOl/T5ut8bJL2qLvsMsCNweI398PY3VdKxkj5Qpzes+31XnX+KpDtb3sO3S7quPjdL0gYt5VjSuyRdC1xbn/tQjflWSW9t2+8ukq6SdF99TR/s9KGPvL62/Rwg6VpJd0v6H0kaZdvtJM2WdK+kP0v6Ytv7O7PGdlvr/ut2F9byb6uf18ptMfxLjeE+Sf9e36sL6r5Oal0/BsB2Hg17AH8AXg78FngasCJwM7AJYGB6Xe844FRgDWA68Dtg/7rse8C/UQ4mHge8sKV8A5svYf8vAW5umZ9etzkOeDywKrA7cF2NbyXgE8AFdf11gfuAPYApwL8CC4G31eUHAd/pUP5Kdf5HwDfqvp4EXAy8oy7bF3gYeHt9X94J3AqoLj97ZD+jvLa3Aj+p0/8P+D1wYsuyU+v0TsBfgOcCqwCHAee2vYdnAE+s78fOwJ+BZ9S4v9v6PgO3ATvW6ScAzx0lvn2B89v2cxqwNuVMbT6w8yjbXgi8qU6vDmzf9v5+r8b2zFrOy+vybYDt6+c4HbgaeF9bDKcCawJPBx4EfglsBqwFXAW8ZbL/b5bnR84Imm3krOAVlH/OW0YW1CPkvYCP2b7P9h+ALwBvqqs8TEkcG9j+m+3zmbiDbN9v+wHgAOC/bF9teyHwn8DW9axgF+BK2yfbfhj4MtBVo7Ok9er276v7uh34Un2tI260/U3bjwDHAusD63X5Gs4BXliP+l8EfA54QV324rocYB/gaNtzbT8IfAzYQYu3z/yX7Tvr+/F64BjbV9i+n5LsWj0MbCVpTdt32Z7bZbwAn7V9t+0/AmcBW4+y3sPA5pLWtb3A9q/blh9c39PLgWOoVY+259j+te2F9Xv0jfpetPqc7XttXwlcAfzc9vW27wFOB57Tw+uJHiURNNvxlKPWfWmrFqIcdU8Bbmx57kZgwzr9YUDAxZKubK+qGKebWqY3Ab5SqxPuBu6s+9sQ2KB1XZfDytZtl2QTyuu6raXsb1DODEY8mlRs/7VOrt5N4bZ/D9xP+THdkXK0faukf2DxRLABLe+t7QXAHSx6f2l7TRu0zbd+LgCvoyS4G2s12Q7dxFu1JtG/Mvpr3R94KnCNpEsk7dq2vD2+DQAkPVXSaZL+JOleSlJft23bP7dMP9Bhvqv3P8YniaDBbN9IaTTeBfhh2+K/sOiof8Q06lmD7T/ZfrvtDYB3AF9Tj10ZO4XUMn0Tpbpm7ZbHqrYvoFSDbDyyYq3T3rhl2/tZvOG7tWfSTZSqh3Vbyl3T9tPHEeNozqFUW61s+5Y6/xZKlc28us6ttLy3kh4PrEPLWVnbvhZ7zZTPYtGK9iW2d6cktB8DJ3URZ09sX2t777qPQ4GTa9wj2uO7tU5/HbgG2ML2msDHKUk9lhJJBLE/sFOtbnhUrRY5CfiMpDVqlcz7ge8ASNpT0kZ19bsoP1p/r/N/ptTvTsQRwMckPb3uby1Je9Zl/ws8XdJrawPwe1n8x34e8CJJ0yStRal2GXldtwE/B74gaU1JK9SGyfaqitF089rOAd4NjDScn13nz6/vK5T69P0kbS1pFcpR8kW16qSTk4B9JW0laTXg0yMLJK2sMj5grVpVdi+LPou+kfRGSVNt/x24uz7dup9PSlqtfmb7ASfW59eoMS2QtCWl3SWWIkkEDWf797Znj7L4PZSj6+uB8ykNlEfXZdsCF0laAMwCDrR9fV12EHBsrXp5/Tjj+hHlqPP7tTrhCuBVddlfgD2Bz1KqU7YAftWy7RmUH6HLgDmU6plWbwZWpjRC3gWcTGkH6MZXgD1UehR9dZR1zqH8+I0kgvMpZygj89j+BfBJ4BTK0f5TWLydYjG2T6e0hZxJaUQ/s22VNwF/qO/VAZQ2iH7bGbiyfuZfAfaq7Rcjzqmx/RL4vO2RgWsfpFRB3gd8k0UJIpYSIz0hIpZpks6m9BT61mTH0jS1gfsGYEpt2I9lTM4IIiIaLokgIqLhUjUUEdFwOSOIiGi4JIKIiIZb5q5quO6663r69OmTHUZExDJlzpw5f7E9tdOyZS4RTJ8+ndmzR+v2HhERnUhqvyzJo1I1FBHRcEkEERENl0QQEdFwSQQREQ2XRBAR0XADTwSSVpT0G0ntV4BE0iqSTqz3bb2o7e5MERExBMM4IziQchvETvYH7rK9OeV2gYcOIZ6IiGgx0ERQb1zyT8BolwbenXJPWCjXhH9ZvdtUREQMyaDPCL5MubftaHdL2pB6n9N6HfN7KLfrW4ykmZJmS5o9f/789oVjPyIiYlQDSwT1xta3254z0bJsH2l7hu0ZU6d2HCEdERHjNMgzghcAu0n6A/B9YCdJ32lb5xbqDa/rvWfXotx6MCIihmRgicD2x2xvZHs65V6sZ9p+Y9tqs4C31Ok96jq5QUJExBAN/aJzkg4BZtueBRwFHC/pOuBOlnDz7oiIGIyhJALbZwNn1+lPtTz/N2DPYcQQERGdZWRxRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENNwgb17/OEkXS7pU0pWSDu6wzr6S5kuaVx9vG1Q8ERHR2SDvUPYgsJPtBZKmAOdLOt32r9vWO9H2uwcYR0RELMHAEkG9Cf2COjulPnJj+oiIpcxA2wgkrShpHnA7cIbtizqs9jpJl0k6WdLGg4wnIiIea6CJwPYjtrcGNgK2k/SMtlV+Aky3/SzgDODYTuVImilptqTZ8+fPH2TIERGNM5ReQ7bvBs4Cdm57/g7bD9bZbwHbjLL9kbZn2J4xderUgcYaEdE0g+w1NFXS2nV6VeAVwDVt66zfMrsbcPWg4omIiM4G2WtofeBYSStSEs5Jtk+TdAgw2/Ys4L2SdgMWAncC+w4wnoiI6EClc8+yY8aMGZ49e/aiJ6SxN1rGXmNERL9JmmN7RqdlGVkcEdFwSQQREQ2XRBAR0XBJBBERDZdEEBHRcEkEERENl0QQEdFwSQQREQ2XRBAR0XBJBBERDZdEEBHRcEkEERENl0QQEdFwSQQREQ2XRBAR0XBJBBERDZdEEBHRcIO8Z/HjJF0s6VJJV0o6uMM6q0g6UdJ1ki6SNH1Q8URERGeDPCN4ENjJ9rOBrYGdJW3fts7+wF22Nwe+BBw6wHgiIqKDgSUCFwvq7JT6aL958O7AsXX6ZOBlUjc3IY6IiH4ZaBuBpBUlzQNuB86wfVHbKhsCNwHYXgjcA6wzyJgiImJxA00Eth+xvTWwEbCdpGeMpxxJMyXNljR7/vz5fY2x7mDJj2WljLG2z8lWRHQwlF5Dtu8GzgJ2blt0C7AxgKSVgLWAOzpsf6TtGbZnTJ06dcDRRkQ0yyB7DU2VtHadXhV4BXBN22qzgLfU6T2AM223tyNERMQArTTAstcHjpW0IiXhnGT7NEmHALNtzwKOAo6XdB1wJ7DXAOOJiIgOBpYIbF8GPKfD859qmf4bsOegYoiIiLFlZHFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDDfKexRtLOkvSVZKulHRgh3VeIukeSfPq41OdyoqIiMEZ5D2LFwIfsD1X0hrAHEln2L6qbb3zbO86wDgiImIJBnZGYPs223Pr9H3A1cCGg9pfRESMz1DaCCRNp9zI/qIOi3eQdKmk0yU9fRjxRETEIoOsGgJA0urAKcD7bN/btngusIntBZJ2AX4MbNGhjJnATIBp06YNNuCIiIYZ6BmBpCmUJHCC7R+2L7d9r+0FdfqnwBRJ63ZY70jbM2zPmDp16iBDjohonEH2GhJwFHC17S+Oss6T63pI2q7Gc8egYoqIiMcaZNXQC4A3AZdLmlef+zgwDcD2EcAewDslLQQeAPay7QHGFBERbQaWCGyfD2iMdQ4HDh9UDBERMbaMLI6IaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhhtzQJmkJy5pue07+xdOREQMWzcji+cCGwN3UUYKrw38sS4zsNlAIouIiKHopmroDODVtte1vQ6wK/Bz25vaThKIiFjGdZMItq+XiAbA9unA8wcXUkREDFM3VUO3SvoE8J06vw9w6+BCioiIYermjGBvYCrwI+CHdXrvQQYVERHDM+YZQe0VdKCkx9u+fwgxRUTEEI15RiDp+ZKuAq6u88+W9LWBRxYREUPRTdXQl4B/pN5C0valwIsGGVRERAxPVyOLbd/U9tQjY20jaWNJZ0m6StKVkg7ssI4kfVXSdZIuk/TcLuOOiIg+6abX0E2Sng9Y0hTgQGo10RgWAh+wPVfSGsAcSWfYvqplnVcBW9TH84Cv178RETEk3ZwRHAC8C9gQuAXYus4vke3bbM+t0/dRkseGbavtDhzn4tfA2pLW7z78iIiYqCWeEUhaEfiK7X0mshNJ04HnABe1LdoQaK12urk+d9tE9hcREd1bYiKw/YikTSStbPuh8exA0urAKcD7bN87zjJmAjMBpk2bNp4iolvSkpfby0YZY23fbRwRDdBNG8H1wK8kzQIeHUdg+4tjbVjbFE4BTrD9ww6r3EK5oN2Ijepzi7F9JHAkwIwZM/LfGxHRR6O2EUg6vk7uBpxW112j5bFEkgQcBVy9hKQxC3hz7T20PXCP7VQLRUQM0ZLOCLaRtAHlktOHjaPsFwBvAi6XNK8+93FgGoDtI4CfArsA1wF/BfYbx34iImIClpQIjgB+CWwKzG55XnRxHwLb59d1l7SO6aIHUkREDM6oVUO2v2r7acAxtjdreeQ+BBERy5ExxxHYfucwAomIiMmRm9dHRDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENNzAEoGkoyXdLumKUZa/RNI9kubVx6cGFUtERIxuSbeqnKhvA4cDxy1hnfNs7zrAGCIiYgwDOyOwfS5w56DKj4iI/pjsNoIdJF0q6XRJT5/kWCIiGmmQVUNjmQtsYnuBpF2AHwNbdFpR0kxgJsC0adOGFmBERBNM2hmB7XttL6jTPwWmSFp3lHWPtD3D9oypU6cONc6IiOXdpCUCSU+WpDq9XY3ljsmKJyKiqQZWNSTpe8BLgHUl3Qx8GpgCYPsIYA/gnZIWAg8Ae9n2oOKJiIjOBpYIbO89xvLDKd1LIyJiEk12r6GIiJhkSQQREQ2XRBAR0XBJBBERDZdEEBHRcEkEERENl0QQEdFwSQQREQ2XRBAR0XBJBBERDZdEEBHRcEkEERENl0QQEdFwSQQREQ2XRBAR0XBJBBERDZdEEBHRcANLBJKOlnS7pCtGWS5JX5V0naTLJD13ULFERMToBnlG8G1g5yUsfxWwRX3MBL4+wFgiImIUA0sEts8F7lzCKrsDx7n4NbC2pPUHFU9ERHQ2mW0EGwI3tczfXJ+LiIghWmmyA+iGpJmU6iOmTZs2ydFEY0hLXm4vG2WMtX0/ylheXkc/ylhW3osWk3lGcAuwccv8RvW5x7B9pO0ZtmdMnTp1KMFFRDTFZCaCWcCba++h7YF7bN82ifFERDTSwKqGJH0PeAmwrqSbgU8DUwBsHwH8FNgFuA74K7DfoGKJiIjRDSwR2N57jOUG3jWo/UdERHcysjgiouGSCCIiGi6JICKi4ZIIIiIaLokgIqLhkggiIhouiSAiouGSCCIiGi6JICKi4ZIIIiIaLokgIqLhkggiIhouiSAiouGSCCIiGi6JICKi4ZIIIiIaLokgIqLhBpoIJO0s6beSrpP00Q7L95U0X9K8+njbIOOJiIjHGuQ9i1cE/gd4BXAzcImkWbavalv1RNvvHlQcERGxZIM8I9gOuM729bYfAr4P7D7A/UVExDgMMhFsCNzUMn9zfa7d6yRdJulkSRsPMJ6IiOhgshuLfwJMt/0s4Azg2E4rSZopabak2fPnzx9qgBERy7tBJoJbgNYj/I3qc4+yfYftB+vst4BtOhVk+0jbM2zPmDp16kCCjYhoqkEmgkuALSRtKmllYC9gVusKktZvmd0NuHqA8URERAcD6zVke6GkdwM/A1YEjrZ9paRDgNm2ZwHvlbQbsBC4E9h3UPFERERnsj3ZMfRkxowZnj179qInpLE3Gus1jlVGN+/R0lBG3ovut+9HGXkvut++H2UM43X0o4yl9L2QNMf2jE6rTnZjcURETLIkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4gSYCSTtL+q2k6yR9tMPyVSSdWJdfJGn6IOOJiIjHGlgikLQi8D/Aq4CtgL0lbdW22v7AXbY3B74EHDqoeCIiorNBnhFsB1xn+3rbDwHfB3ZvW2d34Ng6fTLwMqmbm3FGRES/rDTAsjcEbmqZvxl43mjr2F4o6R5gHeAvrStJmgnMrLMLJP12jH2vu1gZveeWiW6/PJWxNMTQjzIW374fZeS9mOzX0Y8ymvRebDLahoNMBH1j+0jgyG7XlzTb9ozx7m+i2y9PZSwNMfSjjKUhhqWljKUhhqWljKUhhqWhjEFWDd0CbNwyv1F9ruM6klYC1gLuGGBMERHRZpCJ4BJgC0mbSloZ2AuY1bbOLOAtdXoP4EzbHmBMERHRZmBVQ7XO/93Az4AVgaNtXynpEGC27VnAUcDxkq4D7qQki37ouhppQNsvT2UsDTH0o4ylIYalpYylIYalpYylIYZJL0M5AI+IaLaMLI6IaLgkgoiIhksiiIhouCSCFpKeIGk7SS8aeUx2TN2S9ERJH5f0fklrTqCcLSW9TNLqbc/vPPEoh0vSpt08N+AYVhzm/kYj6ZmTHUMnkjaW9KHJjmO8JL2w/s+9cpL2v05fylleGoslrQdsW2cvtn17j9u/DTiQMt5hHrA9cKHtnbrc/s2dnrd9XC9xjJeks4ALgVWAnYFX276+xzLeC7wLuBrYGjjQ9ql12Vzbz+2ynFWA1wHTaemZZvuQXuKZqE4xS5pje5seyhCwD7CZ7UMkTQOebPviLre/HjgFOMb2VT2E36ms5/PY97Sr75ek8yjfjW8DJ9i+Zxz7Xwc4CHgBYOB84BDbPY39kTQV2BPYG9gA+JHtD3a57WbAV4AdgL9TvvP/2ut3vUO5B9k+qIv1Lra9XZ1+O+X/5UfAK4Gf2P5sD/t8f4en7wHm2J7XZRnXUn6vjgFOH2/3++XijEDS64GLKV+u1wMXSdqjx2IOpCSSG22/FHgOcHcP22/b8tiR8g+zW48xdCTpoC5WW8f2x21/AHg/cI6kyyW9UtJJXe7q7cA2tl8DvAT4pKQDR8LoIeRTKdeRWgjc3/KYMEmnd7HOlpJeB6wl6bUtj32Bx/W4y69RfnT2rvP3US6m2K1nA78DviXp15JmjueMTdLxwOeBF7Loe9b1KFLbO1IS2sbAHEnflfSKHsP4PnA7JcnvAcwHTuxmQ0lrSHqLpJ9R/lefAmxq+yndJoHqu8BJwJMpSeQHwPd62H40c7pcb0rL9EzgFbYPpiSCfXrc5wzgAMqldjYE3kE5iPumpA93WcZTKd1G3wRcK+k/JT21xzjA9jL/AC4FntQyPxW4tMcyLql/5wGr1OkrJxDT2sD/9en1vbqLdX4FTG+ZV/1yrQas3+V+rmybXx34P+CLwLwe4r1igq/3uaM8tgFu62L73SlHSHfUvyOPrwLP7zGWufXvb1q/b+N8XS+mjKa/n3Kxxc172PZq6hn8BN/bFSk/5LfUMq8BXjvezxW4vMttHwDOoRwkjdREXD+O+C/r8Ny4Po9xvn+XAk+gXBNtdtuy3/RY1rnA6i3zq9f3aFXgqnHE9tL6ud5dy9mh222XiWsNdWEFL14VdAe9n+3cLGlt4MfAGZLuAm6cQEz3A32pj7b9ky5Weyuwcss2ZtElPf7a5a7+LGlr19NS2wsk7QocDfRSx3yBpGfavryHbVpdQvkidzoLWXusjV2qs06VtIPtC8cZw4iHaz1/ya6lWuPv3W5ct/0nYD9Ktc4XgBMoP4g/pRzRdeMKylHwbd3uuy2OZ9UY/gk4g3JwMVfSBpTqlR92UczPJe1FOSKHclbwsy5D+BhlwOjXgO9J6upMooPT671Nvk/5TN4A/FTSEwFs3zmeQiXtavu0LlZdi3L2IMCS1rd9W21T6/VKcU8CHmyZfxhYz/YDkh4cZZv2uNcB3kg5I/gz8B7KFRu2ppwtdfcbNKxMOuAs/TnKF3Lf+jgdOHQC5b2YUq2zcg/b/KR+ALOA/wWuBz7bp9e365Dex40o9d+dlr2gh3KuAh4CfgtcBlxOhyO5JWx/BbDFKMtu6qGcpwK/pB7JAs8CPtHje7JP/UxvAT5TX9OePWx/PWUE/WPORICv9lDOWcBd9Xs+8j2b1cP25wBvBlbtsOxNXZZxHyUJLqyPv9fn7gPu7bKMzYCP1+/E34CPAE/t4XXcsIRHz2cYLeUePN5t6/arUaq6etnmk8Bc4NP1MQf4FPB4SjtON2X8rpazUYdlH+k2luWisVjSocBFlPpTgPOA7W1/ZIgxvLhldiGlreHmPpV9sO1P96OsYZDU8XK3trs6w6rtO5fbfszlxiW9xvaPuyznHOBDwDdsP6c+d4XtZ3SzfUs5WwIvoxzx/dL21V1utyLwb+5DI3nb9+tRts+ZaNnDIGma7T+2PfcMStvLG1xuTtU4kmZQGt8BfmV7do/by7YlrWa72zP/x5aznCSCTr1DLrP9rCHHMaGeS0so98m2/9SPsgZJ0pq27x05TW/ncZ62t+1jP9vHdLnuJba3lfSblkQwz/bWPe7zhZQzlGNq1dDqtm/octtHe5lMBkmXU6u12hdRahB7+h+pVUzTWbzn0pjVSq3/o5JOsf26XvbbUs5AeudNxv+YpKcAN9t+UNJLKdWvx9m+u4cydqCcca5ue5qkZwPvsP0vvcSyTLcRSHon8C/AZpIua1m0BqXxdJixvB74b+Bsyj/ZYZI+ZPvkPhT/LWDXPpQzaN+lxDmH8uPTWmdqSrXARB1Mafjtxl/qP9tI/f4e9FjHLunTlN4d/1D3OwX4DouO4sbyK0mHU3rXPNpzyvbcLvd/vu0XSrqPxX/QR37Ix+qB1LfvjaSjKdVrV7KoncR0177Q+l2YyPdg25bpx1HO1OYCE+2mfRSl/WSYTgFmSNocOIJS3fddYJceyvgy8I91W2xfqnGMf1qmzwgkrUVpwf8v4KMti+7rx9Fnj7FcSulKdnudnwr8wvazhxnH0kDSdyh10ufZvmYc21822iJKffIqXZazGaVr3fMp9es3APt0W0VVy5hH6Uo8t+WsouuzTZXxHe3sLsen9IukV9k+ve25A2wf0UMZV9luv+94t9u2nhF0PSali3LXBr5ve1kc8DjX9nNrV9EHbB/WevbaZRkX2X5e21nvpb3+7izTZwQug2LuYVEf78nUj55Ly4ujKL1iDqtH5HMpSeErXW6/HuUo56625wVc0EMcr6H0zDmL8lncD7xcZVDZvC7LeKjWwY6cVTy+h/0D7O+2wU41QQ3bJyU9aPvMGsOHKd0Nu04EwIWStvL4BsY9W9K9lM9w1ToN3Z/ZjOZ+xnmGIelJtIwraW/DGIKHJe1NacR/dX1uyhLW7+QmlYGGljSFMh6qqzasVst0IljKnF4Hy4wMbnkD5UeocWyfJelcymn8SymDZp5BGRHajdModZ7z2hdIOruHUGbUxyzKD84bKb2YDpD0A9uf66KMkyR9A1hbZSTpW4Fv9hDDyZQxEK1+QBkTMUy7AaepXM5hZ2BLyniLXhxHSQZ/onR77LqdwXZfLrUh6ScsqiJbAdiKRd1Zuy1jN0o33g0oA+Q2ofx4Pr0fMfZgP8r/xmds36By+ZPjeyzjAMr/1YaUnm0/p4x27skyXTW0NFG5PMNNlCNhKEfAP5rEkCaNpF9SusBdSOnBdX6/Gs57jONcYBfbC+r86pSuvTtThvEvsZpDkihdarekjBwV8DPbZ3Sx7y0pPyyfo/RcGrEm8CHbw/7RGTkC/gWlDeet7vGfX+UGUu+ndP18dCxFL1Vt41Xr0ddj8YPXhZTP5Dbbv++hrEuBnShVt8+pDbVvtL1/P2NeluSMoH+eBLyXUg1yNN0PtFkeXUY54n0GperubkkX2n5gyHFMaMBOrRL6qe1nUgZh9eIfKA21a7PotB9Kn/u391jWuHVoZF6ZUpWyR+152EuVzHyXOwtOhi8DH3PbIEWVi+l9mcXf47E8bPsOSStIWqGewX65b5GOYYw2sK7OsCQdRufeYFAKeW8vMSUR9IntT0j6JOXIcT/gcJVr/BzVy9HK8sD2v0K5vgxlgN8xlFGxXTXy9tEJlOtOnVrnXw18t9bzd1vPPVfStrYv6WXH7u/o5nGzvQY82oB/LnDueBrwq99I+i5l8OSjibSb7qN9sF57Eqj7vlzS9B7LurueHZ4LnCDpdvp0Lawu/Z3yIz7yXo7nAKl1vMHBlAFp45aqoT6r/Xj3o1Q/nEW5iukZtru9iNQyT+Ve1TtSzgr+QKkeOm+koXLIsUx0wM41wBaU13E/Pfa/r73H3s5j+96/tZc4JqpWf+xYH+NpwEdSp267HsZrkXSt7S1GWXadexiQVg8E/kb5LPehXDbiBPd4FdWJqFWHe1MOTq6iJIWf2144jrJ66mnUsYwkgv5QuUrnm4G/UPr9/9j2w5JWAK61/ZRJDXCIJH2Q8uM/Zzxf7KWB6khYTXyU9AXU9wJ4pGX7U/oSaA9URjq3NuA/YHvLYccxHpK+B5xp+5ttz7+N0m37DZMT2cRJegPliraH2v7vcWw/4e64SQR9Iulg4OhOPxCSnuYuL0sQSwf1byRszyOZB6EfDfiSNgIOY9EZ1nmUe1b05VIqY+x7Pcp1/x9i0SWjZ1DaPP7ZPYwKlvRa4FBKG5KYeBfWnknakHIRvn+mdJM+iXJfhgXjKCuJIGIQ2gbojPvUW9J/ABfYntSuxJK+RKmqe5Ay6v5cyo2Xuq6flnQGpQpjpIvjGykD9Hq9r8G41SqukWtFXTme6sba++nVk3VwpnINrDUoP/6nUMYcPcpdDIZt6wSwGouuMDyupJZEENFBv0bC1n/Y1ShHsg8zCUefbfGMNOB/kHKl2a4b8Dud3SwtZzy9kPQr291eImQQ+/8Di37EO102ZOgDDtNrKKKzfo2EXYvSILmpF93qcv3+h7tkHRrwj6ZU7fTiDklvZNGgyb1pO5pdRsxWuR/Cjxl+7ydsTx/GfnqRM4KIAZL0dUp3wZ1sP03SEyi9Q7YdY9N+xzHhBvzacH4Y5dadplzu4z22b+pboEMwmb2f2uIY6bW0qe1/V4/3w+5rLEkEEYOjRRcWa21z6PmiYEsDSccC77N9V51/IvD5Yf+ALi+WloMESNVQxKBN6FaXS5lnjSQBKI2akibUf32YJH3Y9udGG5Xb62jcPnjeyEFC3f9dklYea6NBSCKIGKyvUro9PknSZyj3+f3E5IY0bitIekLbGcGy9BvyEcq1n37PY69sOxmWmoOEZelDjFjm2D5B0hwW3eryNcvwmJIvUK4++oM6vyflPs7Lij9L2oAy8v8l0PPN5vtt5CBhvck+SEgbQUR0TdJWlCt3QhnpO557E0wKSe+h3tGQcsnmRxcxSd02Nc77Yfc9jiSCiGgSSV+3/c7JjgNAE7gfdl/jSCKIiBg+tdwP2/ZTa7XVDyZjsFtTb6UYETHZ/ply57j7AWzfSrn0xNAlEURETI6HXKpkxns/7L5JIoiImBzt98P+Bb3dD7tv0kYQETFk9fIS47of9kDiSSKIiBg+SZe73A970qVqKCJicsyVNPTrCnWSM4KIiEkw0fth9zWWJIKIiOHp1/2w+xpTEkFExPD0637Y/ZQ2goiI4Wq92N3Qr2/USRJBRMRweZTpSZOqoYiIIZL0CIsah1cF/jqyiN7uh92/mJIIIiKaLVVDERENl0QQEdFwSQQRQyBpX0mHT3YcEZ0kEUQMQL0pecQyIYkgoo2kD0l6b53+kqQz6/ROkk6QtLekyyVdIenQlu0WSPqCpEuBHSTtJ+l3ki4GXtCy3p5120slnTvs1xfRLokg4rHOA3as0zOA1SVNqc/9DjiUcgP3rYFtJb2mrvt44CLbzwZ+DxxMSQAvBLZqKf9TwD/W9XYb6CuJ6EISQcRjzQG2kbQm8CBwISUh7AjcDZxte77thcAJwIvqdo8Ap9Tp57Ws9xBwYkv5vwK+XW9GkiqkmHRJBBFtbD8M3ADsC1xAOUN4KbA55UqRo/mb7Ue6KP8A4BPAxsAcSetMMOSICUkiiOjsPOCDwLl1+gDgN8DFwIslrVsbhPcGzumw/UV1vXVqtdKeIwskPcX2RbY/BcynJISISbPSZAcQsZQ6D/g34ELb90v6G3Ce7dskfRQ4i3JJgP+1fWr7xnW9gyjVSncD81oW/7ekLer2vwQuHeQLiRhLLjEREdFwqRqKiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIb7/61UgPLYYvphAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "count1 = Counter(\"\".join(str(df[df['Label'] == 'ham'][\"EmailText\"])).split()).most_common(20)\n",
    "df1 = pd.DataFrame.from_dict(count1)\n",
    "df1 = df1.rename(columns={0: \"ham\", 1 : \"count\"})\n",
    "\n",
    "count2 = Counter(\"\".join(str(df[df['Label'] =='spam'][\"EmailText\"])).split()).most_common(20)\n",
    "df2 = pd.DataFrame.from_dict(count2)\n",
    "df2 = df2.rename(columns={0: \"spam\", 1 : \"count_\"})\n",
    "\n",
    "df1.plot.bar(legend = False)\n",
    "y_pos = np.arange(len(df1[\"ham\"]))\n",
    "plt.xticks(y_pos, df1[\"ham\"])\n",
    "plt.title('Most frequent words in ham')\n",
    "plt.xlabel('words')\n",
    "plt.ylabel('freq')\n",
    "plt.show()\n",
    "\n",
    "df2.plot.bar(legend = False, color = 'red')\n",
    "y_pos = np.arange(len(df2[\"spam\"]))\n",
    "plt.xticks(y_pos, df2[\"spam\"])\n",
    "plt.title('Most frequent words in spam')\n",
    "plt.xlabel('words')\n",
    "plt.ylabel('freq')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning\n",
    "\n",
    "### Data cleaning method\n",
    "With regexes and nltk I preprocess the data to make it easier to learn for the SVM. In order, I'm: turning all letters to lowercase, encoding URLs, email addresses and \n",
    " numbers, and converting punctuation and newlines to  whitespace. I attempted word stemming with nltk to further increase performance. While scikit learn's feature extraction function does have a \"remove stopwords\" argument, I removed stopwords in the same place as all the other preprocessing ops, with a regular expression. Regular expressions carry out their function at a low level and are heavily optimised making them likely quicker than any other alternatives out there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:22.872391Z",
     "iopub.status.busy": "2021-12-28T12:25:22.872142Z",
     "iopub.status.idle": "2021-12-28T12:25:23.044807Z",
     "shell.execute_reply": "2021-12-28T12:25:23.043880Z",
     "shell.execute_reply.started": "2021-12-28T12:25:22.872351Z"
    }
   },
   "outputs": [],
   "source": [
    "def sanitise(email):\n",
    "    # convert text to lowercase\n",
    "    email = email.lower()\n",
    "\n",
    "    # convert URLs to 'httpaddr'\n",
    "    email = re.sub(r'(http|https)://[^\\s]*', r' httpaddr ', email)\n",
    "\n",
    "    # convert email addresses to 'emailaddr'\n",
    "    email = re.sub(r'[^\\s]+@[^\\s]+[.][^\\s]+', r' emailaddr ', email)\n",
    "\n",
    "    # convert numbers to 'number'\n",
    "    email = re.sub(r'[0-9]+', r' number ', email)\n",
    "\n",
    "    # convert $, ! and ? to proper words\n",
    "    email = re.sub(r'[$]', r' dollar ', email)\n",
    "    email = re.sub(r'[!]', r' exclammark ', email)\n",
    "    email = re.sub(r'[?]', r' questmark ', email)\n",
    "\n",
    "    # convert other punctuation to whitespace\n",
    "    email = re.sub(r'([^\\w\\s]+)|([_-]+)', r' ', email)\n",
    "\n",
    "    # convert newlines and blanklines to special strings and extra whitespace to single\n",
    "    email = re.sub(r'\\n', r' newline ', email)\n",
    "    email = re.sub(r'\\n\\n', r' blankline ', email)\n",
    "    email = re.sub(r'\\s+', r' ', email)\n",
    "    email = email.strip(' ')\n",
    "    \n",
    "    # remove stopwords\n",
    "    pattern = re.compile(r'\\b(' + r'|'.join(nltk.corpus.stopwords.words('english')) + r')\\b\\s*')\n",
    "    email = pattern.sub('', email)\n",
    "\n",
    "    # perform word stemming\n",
    "    emailWords = email.split(' ')\n",
    "    stemmer = nltk.stem.snowball.SnowballStemmer('english')\n",
    "    stemWords = [stemmer.stem(word) for word in emailWords]\n",
    "    email = ' '.join(stemWords)\n",
    "    \n",
    "    return email"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:23.047407Z",
     "iopub.status.busy": "2021-12-28T12:25:23.046476Z",
     "iopub.status.idle": "2021-12-28T12:25:48.156178Z",
     "shell.execute_reply": "2021-12-28T12:25:48.155565Z",
     "shell.execute_reply.started": "2021-12-28T12:25:23.047320Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method NDFrame.describe of       Label                                          EmailText\n",
       "0       ham  go jurong point crazi avail bugi n great world...\n",
       "1       ham                              ok lar joke wif u oni\n",
       "2      spam  free entri number wkli comp win fa cup final t...\n",
       "3       ham                u dun say earli hor u c alreadi say\n",
       "4       ham               nah think goe usf live around though\n",
       "...     ...                                                ...\n",
       "11115   ham  p spread option jeff short follow p option att...\n",
       "11116   ham  resum molli would like invit student interview...\n",
       "11117   ham  confidenti sophi think fair deal vinc sophi ki...\n",
       "11118   ham  chang video teleconfer enron christi meet numb...\n",
       "11119   ham  evalu enron remind pleas pass along other may ...\n",
       "\n",
       "[10537 rows x 2 columns]>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply sanitise data\n",
    "df[\"EmailText\"] = df[\"EmailText\"].apply(sanitise)\n",
    "\n",
    "# Drop all rows with missing data\n",
    "df.dropna(axis=0, how='any')\n",
    "\n",
    "# Remove duplicates\n",
    "df.drop_duplicates(inplace=True)\n",
    "\n",
    "df.describe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the dataset 75-25"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:48.157967Z",
     "iopub.status.busy": "2021-12-28T12:25:48.157510Z",
     "iopub.status.idle": "2021-12-28T12:25:48.166427Z",
     "shell.execute_reply": "2021-12-28T12:25:48.165652Z",
     "shell.execute_reply.started": "2021-12-28T12:25:48.157932Z"
    }
   },
   "outputs": [],
   "source": [
    "x = df[\"EmailText\"]\n",
    "y = df[\"Label\"]\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.25, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Extraction\n",
    "\n",
    "Term frequency-inverse document frequency is proven to improve text classification ([see Jing et al](https://ieeexplore.ieee.org/abstract/document/1174522) and [Guo et al](https://www.sciencedirect.com/science/article/abs/pii/S0925231219310276)). Tf-idf vectoriser transforms each email into a sparse matrix of numerical representations of weight of words. This conversion makes it easier for a learning model to pick up on the information an email holds, as it deals with overall document weightage of a word. Its alternative, `CountVectorizer` only counts the number of times a word appears in the document which results in bias in favour of most frequent words. This way, it ends up ignoring rare words which could have helped in processing our data more efficiently. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T13:24:03.331111Z",
     "iopub.status.busy": "2021-12-28T13:24:03.330784Z",
     "iopub.status.idle": "2021-12-28T13:24:04.441418Z",
     "shell.execute_reply": "2021-12-28T13:24:04.439758Z",
     "shell.execute_reply.started": "2021-12-28T13:24:03.331075Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7902, 38867)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv = TfidfVectorizer()\n",
    "features = cv.fit_transform(x_train)\n",
    "\n",
    "# No of rows and the length of all tfidf vectors \n",
    "features.shape\n",
    "# features[0] returns <1x39653 sparse matrix of type '<class 'numpy.float64'>' \twith 9 stored elements in Compressed Sparse Row format>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and train the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:25:49.205848Z",
     "iopub.status.busy": "2021-12-28T12:25:49.205308Z",
     "iopub.status.idle": "2021-12-28T12:50:01.909129Z",
     "shell.execute_reply": "2021-12-28T12:50:01.907932Z",
     "shell.execute_reply.started": "2021-12-28T12:25:49.205792Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9731713490255631"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuned_parameters1 = {'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': [1e-3, 1e-4],'C': [1, 100, 1000]}\n",
    "model = GridSearchCV(svm.SVC(), tuned_parameters)\n",
    "model.fit(features, y_train)\n",
    "\n",
    "# Training accuracy \n",
    "model.score(cv.transform(x_train), y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 10, 'gamma': 0.01, 'kernel': 'rbf'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Best hyperparameters\n",
    "model.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction and evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-12-28T12:50:01.911073Z",
     "iopub.status.busy": "2021-12-28T12:50:01.910656Z",
     "iopub.status.idle": "2021-12-28T12:50:01.919295Z",
     "shell.execute_reply": "2021-12-28T12:50:01.918111Z",
     "shell.execute_reply.started": "2021-12-28T12:50:01.911032Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAD4CAYAAAAw/yevAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAWSklEQVR4nO3deXhV1bnH8e+bMAWtASdkqoBy8eFarYLzXCwFscWpXBxRU6OidrC3ivVaWlt70ToUrVMqKDgh1XLBYlUKWoeKCsqDAlYjCiQyiBCUMJmc9/6RJUYJyUlykrOy+X189pO9197n7HUk/li+e+19zN0REZG45GS7AyIisi2Fs4hIhBTOIiIRUjiLiERI4SwiEqFWTX2Cz1cv1nQQ2UZel2Oy3QWJUMWWUmvse9Qnc1rv3qvR52sqTR7OIiLNKlWZ7R5khMJZRJLFU9nuQUYonEUkWVIKZxGR6LhGziIiEaqsyHYPMkLhLCLJoguCIiIRUllDRCRCuiAoIhIfXRAUEYmRRs4iIhGq/DzbPcgIhbOIJIvKGiIiEVJZQ0QkQho5i4hESCNnEZH4eEoXBEVE4qORs4hIhFRzFhGJkB58JCISIY2cRUQipJqziEiE9LB9EZEIaeQsIhIfd10QFBGJj0bOIiIRSshsjZxsd0BEJKNSqfSXOpjZeDNbZWZvV2v7g5m9Y2bzzWyKmXWotu8aMys2s3+b2feqtQ8KbcVmNiqdj6FwFpFkqaxIf6nbA8Cgr7XNAPZ39wOAd4FrAMysLzAc+M/wmrvMLNfMcoE7gcFAX+DMcGytFM4ikiyeSn+p663cXwDWfK3tWXf/ItlnA93C+lBgkrtvdvcPgGLg0LAUu/tid98CTArH1krhLCLJUo+yhpkVmtmcakthPc92IfD3sN4VWFZtX0lo2157rXRBUESSpR6zNdy9CChqyGnM7FqgAni4Ia+vi8JZRJKlGWZrmNn5wMnAAHf30FwKdK92WLfQRi3t26VwFpFkaeLbt81sEHAVcJy7b6i2axrwiJndCnQBegOvAQb0NrOeVIXycOCsus6jcBaRZMngTShm9ihwPLC7mZUAo6mandEWmGFmALPd/RJ3X2Bmk4GFVJU7LvNwu6KZXQ48A+QC4919QV3nVjiLSLJksKzh7mfW0DyuluNvAG6oof0p4Kn6nFvhLCLJotu3RUQipHAWEYnQ1skTLZvCWUSSpUIP2xcRiU9CnkqncBaRZFHNWUQkQqo5i4hESCNnEZEIKZxFROLjlfqCVxGR+GjkLCISIU2lExGJUEqzNURE4qOyhohIhHRBUAD+5/e38sLLr7Frxw7830P3AHBH0URmvfQKOZbDrh3zueHan7PnHrvx2fpyRl1/E8tXfkxlRSXnn3U6pw4ZCMABxwyhd68eAHTutAd/uunXWfpE0pT+XHQLQ046kVUfr+bbBw0A4FfXXUnBhWfx8eqqL3m+7rox/P3pWdnsZsuWkJGzeRPfTfP56sXJKABtx5x5b9E+L49f/vbmreG8vrycnXfaCYCH/jKV9z9YyuirrqBowiTWl5dz5cgC1qwt4+QzL+KfTz5C69atOeTEU3n9H1Oy+VGaVV6XY7Ldhaw45ujDWL++nPvvH/uVcF6/vpxbb7s3y73LvootpdbY99hw84/Szpz2/31fo8/XVHKy3YGWrv+3v0X+Lt/4StsXwQywceMmLPzxmxnlGzbi7mzYuIn8Xb5Bbm5uc3ZXsuzFl15lzdqybHcj2TyV/hKxOssaZrYfMBToGppKgWnuvqgpO9bSjb33AaY9PZNv7LQT4+8YA8BZp3+fy6/+DScMPZvyDRu5+fpryMmp+vtxy5YtDLvwx7TKzaHg3GEMOPbIbHZfmtnISy/gnHPOYO7c+fziquspK1uX7S61XAmZrVHryNnMrgYmUfXtsa/x5TfJPmpmo2p5XaGZzTGzOfdNfDST/W0xfnLx+cyc8iBDBp7AI088CcDLr81lv969eG7qwzzxwJ38/ta7WF9eDsCzT0xg8vjbufHXV3Pj2HtZWvJRNrsvzeieeyfyH/sdSb/+A1mxYhV/uOlX2e5Si+apVNpLzOoqaxQAh7j7GHd/KCxjgEPDvhq5e5G793f3/j86r6bvR9xxnDzwBP7x/MsATJk+gxOPOwoz45vdutC18158sKQEgE577A5A966dOeSgA3jnvfez1mdpXqtWrSaVSuHu3DfuYQ455NvZ7lLLVlmZ/hKxusI5BXSpob1z2Cc1WLKsdOv6rBdfoefe3YCqWRiz584DYPWatXy4tIRuXfZi3aefsWXLFgDWlq3jzbcWsk+PbzZ7vyU79tprz63rpwwdzIIF/85ibxIg5ekvEaur5vxTYKaZvQcsC23fBPYFLm/CfrUYvxg9htffnE9Z2acMOOUcRhacy4uvvM6HS0uwHKPLXnvyq19cAcAl55/FtTfcwqnnXoq787ORF9KxQz5vvrWQ62+6A8sxPOUUnDOMfXruneVPJk3hoQfv5Lhjj2D33Xflw8Vz+M31N3PccUdy4IF9cXeWLCnh0pFXZ7ubLVsGyxVmNh44GVjl7vuHtl2Bx4AewIfAMHdfa2YGjAVOAjYA57v7G+E1I4D/CW/7O3efUOe565pKZ2Y5VJUxql8QfN3d0/p/gqRPpZOG2VGn0kntMjGVrvxXw9POnJ2un1Tr+czsWGA9MLFaON8ErHH3MeHaW0d3v9rMTgKuoCqcDwPGuvthIcznAP0BB+YC/dx9bW3nrnO2hrungNl1HSciEoUMTpFz9xfMrMfXmocCx4f1CcDzwNWhfaJXjXhnm1kHM+scjp3h7msAzGwGMAiodbaE5jmLSLLUo+ZcfWZZWArTOEMnd18e1lcAncJ6V74s/wKUhLbttddKt2+LSKJ4RfqzMNy9CChq8Lnc3cyapHSrkbOIJEvTz9ZYGcoVhJ+rQnsp0L3acd1C2/baa6VwFpFkafrbt6cBI8L6CGBqtfbzrMrhwLpQ/ngGGGhmHc2sIzAwtNVKZQ0RSZYMzl82s0epuqC3u5mVAKOBMcBkMysAlgDDwuFPUTVTo5iqqXQXALj7GjP7LfB6OO76Ly4O1kbhLCKJ4hkMZ3ff3i3OA2o41oHLtvM+44Hx9Tm3wllEkqUeFwRjpnAWkWSJ/LbsdCmcRSRZFM4iIvFp6m93ai4KZxFJFo2cRUQipHAWEYmPVyTjUfMKZxFJlmRks8JZRJIlkzehZJPCWUSSReEsIhIhlTVEROKjsoaISIS8QuEsIhIflTVEROKTwe93zSqFs4gki8JZRCQ+GjmLiETIK7Ldg8xQOItIomjkLCISIYWziEiM3LLdg4xQOItIomjkLCISIU8lY+Sck+0OiIhkUqrS0l7qYmY/M7MFZva2mT1qZu3MrKeZvWpmxWb2mJm1Cce2DdvFYX+PxnwOhbOIJIqn0l9qY2ZdgR8D/d19fyAXGA7cCNzm7vsCa4GC8JICYG1ovy0c12AKZxFJFE9Z2ksaWgF5ZtYKaA8sB74DPB72TwBOCetDwzZh/wAza3CNReEsIoninv5iZoVmNqfaUvjl+3gpcDOwlKpQXgfMBcrct97qUgJ0DetdgWXhtRXh+N0a+jl0QVBEEqU+FwTdvQgoqmmfmXWkajTcEygD/gIManwP06NwFpFESedCX5pOBD5w948BzOyvwFFABzNrFUbH3YDScHwp0B0oCWWQfOCThp5cZQ0RSZQM1pyXAoebWftQOx4ALASeA84Ix4wApob1aWGbsH+Wuzf4yf8aOYtIoniG7hB091fN7HHgDaACeJOqEsh0YJKZ/S60jQsvGQc8aGbFwBqqZnY0mMJZRBIlk3cIuvtoYPTXmhcDh9Zw7Cbgh5k6t8JZRBIlpWdriIjEJ1NljWxTOItIomRwtkZWKZxFJFGS8uAjhbOIJIpqziIiEVLNWUQkQg2/7SMuCmcRSRSVNUREIpTSBUERkfho5JymLvsMbupTSAv02YOFdR8k0gC6ICgiEiGNnEVEIpSQyRoKZxFJlspUMh5Tr3AWkUTJ4BNDs0rhLCKJ4qjmLCISnVRCis4KZxFJlJRGziIi8VFZQ0QkQpUKZxGR+Gi2hohIhBTOIiIRSkrNORm30oiIBClLf6mLmXUws8fN7B0zW2RmR5jZrmY2w8zeCz87hmPNzG43s2Izm29mBzfmcyicRSRRUljaSxrGAk+7+37AgcAiYBQw0917AzPDNsBgoHdYCoG7G/M5FM4ikiiV9VhqY2b5wLHAOAB33+LuZcBQYEI4bAJwSlgfCkz0KrOBDmbWuaGfQ+EsIomSMkt7MbNCM5tTban+oPGewMfA/Wb2ppndZ2Y7AZ3cfXk4ZgXQKax3BZZVe31JaGsQXRAUkUSpz93b7l4EFG1ndyvgYOAKd3/VzMbyZQnji9e7mTXJDeMaOYtIoqTqsdShBChx91fD9uNUhfXKL8oV4eeqsL8U6F7t9d1CW4MonEUkUTI1W8PdVwDLzKxPaBoALASmASNC2whgalifBpwXZm0cDqyrVv6oN5U1RCRRMnz79hXAw2bWBlgMXEDVoHaymRUAS4Bh4dingJOAYmBDOLbBFM4ikijpzF9Ol7vPA/rXsGtADcc6cFmmzq1wFpFE0e3bIiIRSsiz9hXOIpIsmSxrZJPCWUQSRWUNEZEIVWrkLCISH42cRUQipHAWEYmQZmuIiERIszVERCKksoaISITqeoh+S6FwFpFEUVlDRCRCKmuIiERIszVERCKUSkg8K5xFJFF0QVBEJEKqOYuIREizNUREIqSas4hIhJIRzQpnEUkY1ZxFRCJUmZCxs8JZRBIlKSPnnGx3QEQkk1J42ks6zCzXzN40s7+F7Z5m9qqZFZvZY2bWJrS3DdvFYX+PxnwOhbOIJIrXY0nTT4BF1bZvBG5z932BtUBBaC8A1ob228JxDaZwFpFESdVjqYuZdQOGAPeFbQO+AzweDpkAnBLWh4Ztwv4B4fgGUTiLSKJU4mkvZlZoZnOqLYVfe7s/AlfxZZbvBpS5e0XYLgG6hvWuwDKAsH9dOL5BdEFQRBKlPjehuHsRUFTTPjM7GVjl7nPN7PiMdK4eNHJuIhePHMGLs//GC688yb3jbqFt2zYA/PK6nzJ77tO8/NpTXHTxuVnupTSV0VNmc8KNT3D6n6ZvbXv27aWcdsd0Dhr9CAtKP9nmNcvLyjnid5OZ8NKX5c2HX3mH0/80ndPumM5D/3qnWfre0mWw5nwU8AMz+xCYRFU5YyzQwcy+GNh2A0rDeinQHSDszwe2/YNOk8K5CezVeU8uuuQ8vnv86Rx7xPfJzc3l1NOHcObZp9Gla2eO6D+Yow49iSlPTK/7zaRF+sFBvbjr3BO+0rZvp3xuPfMYDt57zxpfc8vTb3BU785bt4tXlvHXue/zUOH3mDxyMC++W8rSTz5r0n4nQaZma7j7Ne7ezd17AMOBWe5+NvAccEY4bAQwNaxPC9uE/bPcvcGTrhXOTaRVbi7t8tqRm5tLXl47VqxYxfkFZ3LLTXfyxZ/X6tVrstxLaSr9euzJLnltvtLWa498euy+S43Hz1q0jC4dd2afPfK3ti3++FO+1W038tq0olVuDv167MnMhcuatN9JkMkLgttxNXClmRVTVVMeF9rHAbuF9iuBUQ0/hcK5SaxYvoq77hjPvLef4+13X+LTT9fz/KyX6dGzO6ecdhIznn+CSY//mV699s52VyUCGzZ/zgMvLuSS4/f/Svu+nfJ5Y8nHlG3YzMYtFbz07kes/HRDlnrZcng9/kn7Pd2fd/eTw/pidz/U3fd19x+6++bQvils7xv2L27M52hwOJvZBbXs23oFdNOWsoaeosXK77ALg4YMoN8BA/hWn2No3z6PM4b9gLZt2rBp02a+e/zpPDhhMmPv/H22uyoRuOe5tzj7yP1o37b1V9p77ZHPBUf35dIJs7jswefo07kjOQ2fmbXDqM9sjZg1ZrbGb4D7a9pR/QroHvl94v430ASOO/5Ili4p4ZNP1gIw/clnOeSwg/joo5VMf3JGaJvB7Xf+bza7KZF4q+QTZixcxh+fncdnm7aQY0bb1jkMP6wPp/bbh1P77QPA7TPm0Sm/fZZ7G7+k3L5dazib2fzt7QI6Zb47yVCy7CP69T+QvLx2bNy4iWOPO4J5b77N+s/Wc/Qxh/HIkhKOPPpQ3n//w2x3VSJw/4++u3X97lnzad+mNcMP6wPAmvWb2HXndiwvK2fWohImXjQwW91sMVINvwYXlbpGzp2A71F1i2J1BvyrSXqUAG/Mnc+TU59h5gtTqKio4K35i5j4wGO0y2vHPX++mYtHjqC8fAM/u+LabHdVmsiov7zMnA9WUrZhMwNvnsKlJxxAfl4bxjw1h7Xlm7nioX/SZ68O3D3iO7W+z88nvci6jZtplZPDNUP6b3ORUbaVjGgGq22mh5mNA+5395dq2PeIu59V1wl2xLKG1G1pUZ2/OrIDyvuv0Y0uqp+196lpZ84jS6ZEW8SvdeTs7gW17NN/XSISnfrMwoiZbt8WkUSpUDiLiMRHI2cRkQjtEFPpRERamkY8ziIqCmcRSZT6PDI0ZgpnEUmU2G/LTpfCWUQSRSNnEZEIqeYsIhIhzdYQEYmQ5jmLiERINWcRkQhVejIKGwpnEUkUlTVERCK0ozxsX0SkRUlGNCucRSRhknJBsMHfvi0iEqMUnvZSGzPrbmbPmdlCM1tgZj8J7bua2Qwzey/87BjazcxuN7NiM5tvZgc35nMonEUkUSo9lfZShwrg5+7eFzgcuMzM+gKjgJnu3huYGbYBBgO9w1II3N2Yz6FwFpFE8Xr8U+v7uC939zfC+mfAIqArMBSYEA6bAJwS1ocCE73KbKCDmXVu6OdQOItIorh72ouZFZrZnGpLYU3vaWY9gIOAV4FO7r487FoBdArrXYFl1V5WEtoaRBcERSRR6nNB0N2LgKLajjGznYEngJ+6+6dmX35ht7u7mTXJFUiFs4gkSiafSmdmrakK5ofd/a+heaWZdXb35aFssSq0lwLdq728W2hrEJU1RCRRKkmlvdTGqobI44BF7n5rtV3TgBFhfQQwtVr7eWHWxuHAumrlj3rTyFlEEiWDdwgeBZwLvGVm80LbL4ExwGQzKwCWAMPCvqeAk4BiYANwQWNOrnAWkUTJ1LM13P0lwLaze0ANxztwWUZOjsJZRBJGz9YQEYmQnkonIhIhjZxFRCKkh+2LiERIZQ0RkQi5Rs4iIvFJyvOcFc4ikiiZvH07mxTOIpIoGjmLiESoMqWas4hIdDRbQ0QkQqo5i4hESDVnEZEIaeQsIhIhXRAUEYmQyhoiIhFSWUNEJEJ6ZKiISIQ0z1lEJEIaOYuIRCilR4aKiMRHFwRFRCKkcBYRiVAyohksKX/LtARmVujuRdnuh8RFvxdSk5xsd2AHU5jtDkiU9Hsh21A4i4hESOEsIhIhhXPzUl1RaqLfC9mGLgiKiERII2cRkQgpnEVEIqRwbiZmNsjM/m1mxWY2Ktv9kewzs/FmtsrM3s52XyQ+CudmYGa5wJ3AYKAvcKaZ9c1uryQCDwCDst0JiZPCuXkcChS7+2J33wJMAoZmuU+SZe7+ArAm2/2QOCmcm0dXYFm17ZLQJiJSI4WziEiEFM7NoxToXm27W2gTEamRwrl5vA70NrOeZtYGGA5My3KfRCRiCudm4O4VwOXAM8AiYLK7L8huryTbzOxR4BWgj5mVmFlBtvsk8dDt2yIiEdLIWUQkQgpnEZEIKZxFRCKkcBYRiZDCWUQkQgpnEZEIKZxFRCL0/2txweMJm5NcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = model.predict(cv.transform(x_test))\n",
    "\n",
    "# Show confusion matrix, i.e. all TN TP FN FP\n",
    "cf_matrix = confusion_matrix(y_test, y_pred)\n",
    "sns.heatmap(cf_matrix, annot=True, fmt='d')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'              precision    recall  f1-score   support\\n\\n         ham       0.94      0.99      0.96      1400\\n        spam       0.99      0.93      0.96      1235\\n\\n    accuracy                           0.96      2635\\n   macro avg       0.96      0.96      0.96      2635\\nweighted avg       0.96      0.96      0.96      2635\\n'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Classification report includes recall, precision, F1 score\n",
    "classification_report(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9616698292220114"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test accuracy\n",
    "model.score(cv.transform(x_test), y_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}