knn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# k-Nearest Neighbors Classification\n",
    "\n",
    "This notebooks uses a simple kNN classifier that segmented the image detecting between damage and not damage.\n",
    "Image segmentation follows the following propierties:\n",
    "- Pixels from the same regions have similar greyscale of multivariate values.\n",
    "- Adjacent pixels present in different regions have different values.\n",
    "\n",
    "Also some of the hypterparaters of the kNN classifier to have in account are:\n",
    "- Value of k. Number of neighbours.\n",
    "- Measire of a distance (eg. Euclidean, Manhattan).\n",
    "- If ussing Soft-kNN, way to combine the information of more than one observation\n",
    "- Dominance of points (equally weighted or some points are dominating).\n",
    "\n",
    "For this excercise we will use the euclidean distance and we will use cross-validation to find the best k. \n",
    "We will also try to compare how data preprocessing will affect the performance of the classifier.\n",
    "\n",
    "### Set Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import random\n",
    "import numpy as np\n",
    "from utils.data_utils import load_data\n",
    "\n",
    "# Nice plots\n",
    "%matplotlib inline \n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# Keep notebook stable across the run\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape:  (2400, 51, 71)\n",
      "Training labels shape:  (2400, 51, 71)\n",
      "Test data shape:  (600, 51, 71)\n",
      "Test labels shape:  (600, 51, 71)\n"
     ]
    }
   ],
   "source": [
    "filename = 'datasets/Deltas3.mat'\n",
    "\n",
    "# This is a precaution: clean the data before start using it\n",
    "try:\n",
    "    del X_train, y_train\n",
    "    del X_test, y_test\n",
    "except:\n",
    "    pass\n",
    "\n",
    "# load_data\n",
    "X_train, X_test, y_train, y_test = load_data(filename)\n",
    "\n",
    "print('Training data shape: ', X_train.shape)\n",
    "print('Training labels shape: ', y_train.shape)\n",
    "print('Test data shape: ', X_test.shape)\n",
    "print('Test labels shape: ', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Some usefull functions to plot the images.\n",
    "from utils.visualization_utils import plot_image\n",
    "\n",
    "def plot_gray_image(image):\n",
    "    plt.imshow(image, cmap='gray', interpolation='nearest')\n",
    "    plt.axis('off')\n",
    "    \n",
    "def plot_color_image(image):\n",
    "    plt.imshow(image.astype(np.unit8), interpolation='nearest')\n",
    "    plt.axis('off')\n",
    "\n",
    "def plot_image(image, color=False, reshape=False):\n",
    "    if reshape:\n",
    "        image = np.reshape(image, (51,71))\n",
    "    if color:\n",
    "        plot_color_image(image)\n",
    "    else:\n",
    "        plot_gray_image(image)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAAG+CAYAAAAwWNebAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deaxl2Vnd1+6urrl6ck+4J6MmcWIMirHAgBBYAdpisLCiIBQSYcAgHBJhlIRAIGJICJAEZZARoJCIiDgYMoEcEgfFgBJZNgnYjkMUbMtyem73XNVV1dXzzR/3rne/t97+9tn33le7qt9dP6l07pn22efc+3at8+1vKLPZDMYYY8ZwxaXugDHGbBMedI0xZiAedI0xZiAedI0xZiAedI0xZiAedI0xZiCv+EG3lPLjpZT3DL5mKaX8cinlqVLK/+w4/jWllFkp5dCI/hljLl8ui0G3lHJPKeVrLnU/VuArAHwtgNtms9mXXOrOGLP4G7pQSjlbSjldSvlQKeWdpZTL4m/cLPEXsh53ArhnNpudv9QdMSbw1tlsdgrz3+fPAPhBAP/i0nbJKJfdoFtK+fZSygdLKT+7eH3/f6WUrwv7P7eU8t8W/6P/VwA3yPlfuvhf/nQp5eOllDcvtl9fSnmglPLWxfrJUsqnSynflvTj1aWU95VSnlwc992L7e8A8M8BfFkp5Vwp5Scq51656P/jpZTPAPgG2f8dpZQ/XtzDZ0op3xP2vXnRz79ZSnm0lPJwKeVtpZSvL6V8atGfHw7Hf0kp5cOL+324lPJzpZTDYf/dpZRPllLOlFJ+fvHsvivs/85FX54qpfx2KeXOri/KXLbMZrMzs9nsfQC+BcDbSymvL6V8QynlY6WUp0sp95dSfpzHB/PXdyz2PbVQyV9cSvnfi9/Wz4Xj7yql/G4p5YnFb/xfl1KuDfu/aHGts6WUf1tK+fVSyk+G/d9YSvlfQZF/4aBHc3kwm80u+T8A9wD4msXnbwfwAoDvBnAlgL8M4CEAZbH/wwD+EYAjAL4SwFkA71nsuxXAEwC+HvP/UL52sX7jYv/dAD4L4CYAvwTg3zX69N8B/DyAowD+DIDHAPzZ0McPNs59J4BPALgdwPUAfg/ADMChxf5vAHAXgALgqwA8A+CLFvveDOBFAD8K4KrFc3gMwK8COAXg8wFcAPC5i+PfCOBLARwC8BoAfwzg+xf7bgDwNIA/t9j/rsWz/a7F/m8C8GkAf3qx/28D+NCl/j3432Z/Q7L9vsXf0JsBfMHi7+ILATwC4G2LY16z+H3+4uL3fjeAZwH85uJv5VYAjwL4qsXxn7f42zoC4MbF38o/Wew7DODexW/tqsVv73kAP7nY/4ZFW2/C/O/77Yu+H7nUz3DYd3WpO6A/mMWA9umw7/jiB3ELgDsWA9KJsP9XsRx0fxDAv5K2fxvA28P6uwH8EYAHAbwq6c/tAF4CcCps+2kA/zL0sTXo/i6Ad4b1uxEG3crxvwngXYvPb8Z8UL1ysX5qce6bwvEf4R9Mpa3vB/Abi8/fBuDDYV8BcD+Wg+77Abwj7L8C8/8A7rzUvwn/W+0f8kH39wH8SGX7PwHwjxefOejeGvY/AeBbwvq/x+I/80pbbwPwscXnr1z8bZWw/4Nh0P0FAH9Xzv8kFgP6Nvy77MwLCz7LD7PZ7JnFx5MAXg3gqdluW+q94fOdAL558dpyupRyGvNJr88Jx/wzAK/HfAB9Irn+qwE8OZvNzsp1bu3s/6sxH9xqfUQp5etKKb+/MBWcxlyZRzPJE7PZ7KXF5wuL5SNh/wXMnwdKKX+ylPJbpZTPllKeBvBToa1d/ZjNf+EPhHbuBPBPw7N6EvOBufc+zeXPrQCeLKW8qZTye6WUx0opZzB/G7tBjtXfWPabu7mU8mullAcXv7n3YPdv7sHFb43Ev4U7Afx1+Ru9fXHeVnC5DroZDwO4rpRyImy7I3y+H3Ole234d2I2m/0MMLe1Yj7o/gqA7y2lfF5ynYcAXF9KOSXXeXCFft5e62Mp5QjmquFnAdw8m82uBfCfMR/s1uEXMDdl/InZbHY1gB8ObT0M4LZw7RLXMX9e3yPP69hsNvvQmn0xlxGllC/GfND9IOZvhO8DcPtsNrsGc1PCur+5n8JcGX/B4jf3l7D7N3fr4rdG4t/C/QD+nvzmjs9ms/eu2ZdXHK+oQXc2m90L4A8B/EQp5XAp5SsAvDUc8h4Aby2lvGUxmXV0MTHFgeaHMf+xfCeAfwjgVxYDsV7nfgAfAvDTiza+EMA7Fu338G8AfF8p5bZSynUAfijsO4y5LewxAC+W+STh3Z3t1jiFud32XCnlT2FuvyP/CcAXLCbiDgH4K5ibacgvAvhbpZTPB4BSyjWllG/eoC/mMqCUcnUp5RsB/Brmprc/wvx38uRsNnu2lPIlAL51g0ucAnAOwJlSyq0AfiDs+zDmprm/Wko5VEr5JgDRrfKXALxzobxLKeXEYpIvCpwDzStq0F3wrZgb4Z8E8GOYq1YAO4PlN2E+uD6G+f+qPwDgilLKGwH8NQDftnh1//uYD8A/hDp/AXNb10MAfgPAj81msw909vGXMLclfxzARwH8h9DHswC+D/OB+anF/byvs90af2PRxtnFdX89XOtxAN8M4B9gbqN7Heb/aT232P8bmD+HX1u8Jv4fAF8H80rlP5ZSzmL+u/8RzCecv2Ox73sB/J3F/h/F/Pe3Lj8B4IsAnMH8P/b4+34e88mzdwA4jbkK/i0sf3N/iPnk8M9h/vv/NOZzJFsDPQLMFlDmjvIPAPiLs9ns9y51f8x2UEr5HwB+cTab/fKl7svlwCtR6ZoVWJharl3Ykmnv/f1L3C1zgCmlfFUp5ZaFeeHtmLuo/ZdL3a/LBecCOPh8GeaTKIcB/F/MXc0utE8xZiNei7n54gSAzwD487PZ7OFL26XLB5sXjDFmIDYvGGPMQDzoGmPMQJo23de+9rXzYOrD8/wphw7ND48mieeeew4AcOHC3Ex4/vw8WOyZZ+aBZM8//zwA4Morl+6w/HzFFVfsau/ll19urq+Lnr9JezxXlxH6hWfLWj9039T1W9uytmrb+R3ostXXl156adeSv48TJ07sWh4/fnznnKuuumpXe4899hgA4MEHH1zXQX8jSim2q5mLymw2q/62m4Mu//BuuWXuT3/06FEAyz9MAHjhhRcALAdXDr4cdLnOwRnYO5hmg2A2WNWOydYj2XVaA+km/yGwLzqQ6QBX2zdFrY/7aZ/X5xj7xc8cSI8dOwYAOHnyJIDlYMvfC7D8j5bPzZhtxeYFY4wZSFPpHjlyBMBSydSULtUOt1HJvPjii7uWfBWN24gqtR6Fq+qx9/U8XkfXa+qV29i+KrWaytQ+raJ0oxmmdl81NRuf7X5T66sqXZoXdJ2/H2BpmmJf1dxgzLZgpWuMMQNpKl0qGqovLqP60kmxbCKmRmZL5XKVtnqYUrjZcT19bU2oZdetqWMqwcy2m/X5YtG6ziYTnrbtmm3FStcYYwbiQdcYYwbSNC/wFVCX8dVX9+nEme6P+7LXUzUv1CacdNKth6nX/Nqru27TZcusoJNwq7iM6blZn7M+rEvm8xv7oaYPTpLRbZDuga3fybPPPrtvfTbmlYSVrjHGDKSpdFW1MhCidgzVDZdUPVxGtyaqnUzxqoqMikkn7npcxnonwWoqMguKaCndrG+Z4s221e6rR+muM/E4FbUX28wm/fT+Yht0EeNzY/CMMduGla4xxgyky2WsZpclVLK00XFdAyAiqoi0fY3tr9kTa3bRjJbNtra/R+n2hP/2Lqf2tfraum6LKdvt1DMDlt9x9mzi2w3tvvy+mKPDmG3DStcYYwbSVLpUJ6QWCjs1u19rK7NfZgo0quYstLYVLpuptkw1rpJMpseLIkvKU1O6U+e2WEfpTqnwmtKlbZ/LzM4dnxlDg/m92aZrthUrXWOMGUhT6WqCm5ZCo5LlksqG3gbRvseZbCrYLNxXbbzxs9p71/FiWCV8dSrxTKuNLOS1x6a7iZ22h+zNodZH3cbvj/Z8ftf87uMbitrgWzZ/Yw4yVrrGGDOQptJlSsee9IKESoZqp3YOVaqmf+yBdsSpJDk1eqKtMlbxJuixFU/1bZX+ZOp0neTm2TOp+UqTHrv+6EQ9xlyuWOkaY8xAPOgaY8xAulzGstdHYK+pQF8faxM0OunGc/nayu0a8hs/6/VaYaurJsdpTR4pPXXVNqHHJNLrbrbJRFsr4U1P4iD+TjQ3szHbhpWuMcYMpCvhjYYDRxcuTWyj7kNaNyu2l6mdltuSKi8NRW1VZSCrKN+pY2r7M0W5ThrKTapm9FS3yOgJWc4q/LaUfi2025htwkrXGGMG0lS6Fy5cAADccMMN84MlWTWwVDVMYMLwTh6rwRPAXkd5dS8jDM6IZOpNK+LuV2LvTHFuEjrcolcBRhXZa29upaFchSyRfCvoRBPg72fidWNeSVjpGmPMQLpsurfccguApaKhAgb2ehOorZUKp5YAXasLq7cEOXLkyJ5zVymfs0qQQsZUwpuaqluFXoVbU969ZYh67Nw9Kr33WbSeiYMkzLZipWuMMQNpKt3rrrsOAHDbbbcBWCqX06dP7xyjivbqq68GsLT70sYbk1brOVmJmpp3A223WdKcmoJaVemuooR7yucom5TTqZXCmepTS+lOHbtKmLPuj9/F5ap0N7Et2wPDrIOVrjHGDKSpdE+ePAkAuOuuuwAsVeajjz66cwwLUardldvptxvV8ZkzZwAs1XCmeGtJbDSaTRWvFrvUz7X1qe0959T2T5WL74l8W8cenanUmh24FUVWu4fasVmfW0r3Unsv7Mf1W9+5MRlWusYYMxAPusYYM5CuyhF33HEHgOUrfAx0ePrppwHsdd2iWYHuZTHQgRNkTz75JIDlZJsGTdRe1dS8oMlxWjlye16ddb3XFNGapNqEVdqacvtaZSJNTQc95oUel7FNAkdeCexH+LY52FjpGmPMQJpKl5NUDI7g5FgMjrjpppsA7FUwPObs2bMA6qG+WjlCQ4ipXmuqSI+ppYEkvepqHfevHsXWo1Z7E+v0BH+scl1tQ9XrJkq3NZF20LHiNRlWusYYM5Cm0j1+/DgA4NSpUwD2VnuN+zSggWG/tOVGOzD3UdnqUmumtRJe63XVtttiFVcuPafn2KyvlyOruMS1XNGmzrmcn8HFYFvv2+RY6RpjzEC6bLq0k1KtUt0Cy7BfBkfwGAYpUOnG/+mpdOn5QEXN9WgzjtevbVNbbs3zoTeJec1+OZXGsHaNXnWziY13ndn/TdRWK1n7VMIdIA+AMWbb8F+AMcYMpKl0jx07BmBpw6VyiSG//MxjqY4V+u0CwLlz5wAsw4zZvqrkrBxMhApMFW5UVNlMfMunl2QKt3XOKjZivc46x03ZY1v3mSUbytZr7Wpi8nWS5Bx0bNs1xErXGGMG0lS6/N+5lZCcSWuocNWLgOo1ejxosUou1T6raRtje5n6rSldTft4sdXGfhSTXHd/PKanuKSub1Kscz9Kvhtz0LHSNcaYgTSVLm2v9J+lV8FTTz21cwzzJ1Dx0hOBtl5GsUXfW/XDzXIt1HI99CYIb5VtX4epFIvreC+scr2p7fF66/gOr3POKn66xpg5VrrGGDMQD7rGGDOQpnnhiSeeALCsFEFzwwMPPLBzzP333w9gGSRBNzAGUHDyiiYKYGmK4IScVnvgOdzOJbDb1NBiFZeuVZgKgb3YbDLR1WKV+1nHbGCTgzFzrHSNMWYgTdmYuXL1OMyT1kSNbqOibdU7600RuN/1q6aUWk8S81VcqtZR0he7yvBUG1azxkxjpWuMMQNpKt0bbrgBAPDqV78awDIheQzppX32mmuuAbC07TIsmPsjbEeT5GhgRbTlZmQqa7/UVq/rVs1lTPf1KNts+ya21ottp10lRNqYbcdK1xhjBtKV8IbqlQr0+uuv3zlGk9dwyVBfei3ERDhUtlO2T00t2UNPspVNnPpXCVrYxMaqbVzMgItV27UN15j1sdI1xpiBNJWulr7h+okTJ3aOoT8u1aumdlSPhPiZ9l4uNTyYbUXf3KxYpfr4tgopbjJjnxVhrB0zRasA5n4o3lFliPQcTSifbdsmbOc2ZDv/Aowx5hLRVLpUoLTbcj36yvJ/cCa20cTj9FRgKZ7YHpf0hlBvhZrSJVPqtFb+O1OTF1uF9CjpVVV3zS94E/Yj4U1LWe9H0iFjDgJWusYYMxAPusYYM5CmeeHMmTMAgMceewzA0rxw/vz5nWNoImCuXbqIcVLs9OnTAIDHH3985xwm0mH7bIPnMGhCw4+BvWHAOkmmy9q2/a6KO3VMT/jvOkEYU8e0Jgyn7qNlDljFrNB7vYPKtt63ybHSNcaYgTSVLlUqUzkyFWNUulStVKuc/KIqppqNE2k8h+1zIo1KVOutRbR+mqrYVnAE2c9qCa3JuN5aYpFNKkasQm+Coh5l3dOvWu26g4wVrsnYjr8AY4y5TOhyGXvooYcALBVOVLqsl0ZFqy5kXEalS/sulS5VqypcqqKorOhWRmU7pXjj54sRQruJwu2xsfa4uU2p7/12jet9U1hH0b/SscI1U1jpGmPMQLqqAd93330Alv+Lx9I7PIaqlVCB0l4blS7VMW3EVD0thUu0lE+2Xkt2PmWvXMWroMVUQp11PCCmwoR72trvgIp1uFyU4CZJ27UNY1bBStcYYwbSVLpUpw8++CCA5f/s0aZLRUv1S2VLezDXa+dQjWYKl9erJcuhStbSPjWbbs+M/BRTimgd/9n9ZiqRzzreGrVzpkomvZJCfl8JfTQHCytdY4wZSFPpMlnNI488AmCpSKkygWUkGpeaHIfLWtIa9cPViDGq2Kh0tXy7lnGvFbOc8iZYRXlO2Wtb11vnOlmb66jlVgKaqWVP4c117MzGbBtWusYYM5Cm0mWZHuZeoB9tVJ605eqS9lnNowDsVUpUpzxHE5/H4pZTCpdquRX5tJ823f3wROjdl7W5ib+sHsPn1rLLrpJzgfS0a8w2YKVrjDED8aBrjDEDaZoXWNmXbl/qngXsrW9GE0RP1QetFKFmBZoQGEocr5e5jG1S26v1aj+VqGUdd6zIJu5e66RpXHUirdVuT0KhVUwRxhxkrHSNMWYgTaVLlarBC1H1aQgvlecqyU7UNUwn0KLSzYIjSCuEeKo/NeXWm4pwnRDblhvWOpNjU2pyv5Rub1+3ObWjMRn+CzDGmIE0la7aXKlSYhgo1XAteXg8Nm7XtIxqu1WlWwsD1r5dbDewqfZ73MD20465jr20R+lOtbFK32Ib6jJmxWu2Ff/yjTFmIE2lS+VZU7gkS6moHgrx3EzJ6nrNbltTzvvNKoEHtePWCXBYlR6l2+N5sI7inbreKt4SxmwbVrrGGDOQptJlOsZjx47ND17Yb3tUqyrfml1Wl6psa/bbTRO97FcbPYxW46uoyE2U5lQJoZqXi225xszxX4AxxgykK7Ujk9ZoaRxgqUa1EKUq3XiObsuUb80erOznrHur6ONU+z2qdnQxxlUU71TKylbppExpR1VLP+510mkac5Cw0jXGmIF40DXGmIF0mReuu+46APUQW75ichKMyXG00m+E23SSTd3Baq+p6+S1nXqVzQI6au3uR92znmCJqevWrpeds8qzye6j9p1n3xNNCTHZET/z2Kk6a8YcVKx0jTFmIE2lyyoQLXcfdQ+iguGEWmtyKlONqqjjca2JnV6m2mipc1WNPYqtp6/7MSG4iQrPjtW3kviZ+6hiuayl9eQ2u4yZbcd/AcYYM5Cm0iW00WmKR2Cvyjl8+DCApZ2WNt5agENWN6ul0DKV2lJzU25euqypuqm2WzZWvb7ed9w2Ra2P2fVWCQpR96/suvEYuhIePXoUwDKIhr8BbtfP8Vxjtg0rXWOMGUhT6VKxqJqNUCFRyai9j8uYiFyV7iaVYjexfapq7LFfahur9FHvtxYm25tCMvZR72cV+/Cqnh3Ast+001LhcnnixAkAu9UtlS37wmOM2TasdI0xZiBNpUulokqXtt34OZvVrykl2nenlO5+FTjMbJtZGsqaT2qm4HtsyZkfa0vp9tqha33YDw+PFuw334RU6ariBZaqmH06fvz4RembMZc7VrrGGDOQptKlHY4qhcq3lshE1WKW4lE/RzLbZ031ZbP9PUm3p0rA1/qa2X1bqnJK4cY3hk1SH67j0bGqCq69baiXgi6jhwKP5XWpho3ZNqx0jTFmIB50jTFmIF0uY63giJ2GZLItC5oA8oQpSs28kAUaKK1zeF3eD/vDtmq5f6fMCzWTifa/9Rw3cZtTNnFvy0w/tUm/qe+8NuHKvtBkZcy2YaVrjDED6QoDrlV/IDr5lKmqmiJVhan0qD1VZvuRUKVVf0zV+TppGi82q/RNUUWqbdba63Gnq7njGbONWOkaY8xAmkr36aefBgA8/PDDAJZ22agumdCGaSDPnTsHYFlJmPujfY82P1WlU+G5+rlGS91N1f/aJJXkKgEcPQnIp1ItTvVhan0qCU8rhaWqVv4uGAxx8uRJALsDIDThzX333ZfehzEHGStdY4wZSLGNzRhjxmGla4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxA/Gga4wxAznU2nn99dfPAODUqVMAgHPnzgEALly4sHPM4cOHAQBXXHHFruWLL74IAHj22Wd3bQeAK6+8EgBQSgEAvPzyy7vO4fYjR47sOZfHPPfcc7tv5ND8Vq666ioAwGw229n3wgsv7FpqPwjP4fXjNrbL6/AY7n/ppZf2XI/HHj16dNc6z3n++ef3nKPPk+s8h88+9pF90++AfeJ1at/bsWPHqvfDc/SZxevFPsTr6XMFlt8x+8Zn8tBDD5U9Bw+glDKbPsqY9ZnNZtXfdnPQ5R+P/oHyDwhYDoL8Y+IfrQ5g8Q+R7XEbByPCtnhubUDjMewL2+KA0PrDZ5/ZLrfXBl3CbbV9up33p/dZ6xPRwY7PRPt44sQJAMvvAtg7COp/TPqfQDyfg1/8jy22yf80439ivI4O7kp8Juw3243/ARizTdi8YIwxA2kq3WeeeQbA8jX/7NmzAJZKClgqF6o6VYSqLoG9Sja+ZgN7X1OjymJ7PCY7Np6j21TVaV9bxPuI14/XUxMHl2o+iW3xM4/N+sLj9JkBe982qHCpVvk9xs9qzmC7vC+2EdUs1TH7qOYbfhdUt8Beswz7ZMy2YaVrjDEDaSpdVSVUUNE2SKWkdkVVd9GeqZNPqu5UxcVztX1dVyUcUYXNdtU+XLNf6vXYfs3Wq33R63J7VLM6YacTkvrGEJ+J3ofagbmfChXYa3dWZa124aiS1QavqlhVdLx3VeHGbBtWusYYMxAPusYYM5CmeUEnhOIr5k4Di1fN2qRX3B7JXrf5mso2deIptqev+Xxd1dfxeKxO+mUuT7U+T01wRTKTRwt93VefW05q8j55L/E6vHddqo8xsPc5Ep0cY9vRVBA/x3NoiuD+aM7gtdXlz5htw798Y4wZSFPpEnWGj+gEj04S1QIOMnWs7l9c71FFOglXc+FSssm46MqlE2VTQRKtvukziQqbbxF8Nur+pcESsY+qHtXNSyPhYl/YDs/RgApGI548eXLnXCpZ7Qvvgc+7NflWe2syZhuw0jXGmIE0la66J9UCD4i6AKmqi2q1J1QYWCqmmmuV2knVbhuVqNpH9dws9wOwV3nqOXr9Vh9Jy40ue8Y8Vm3Xem1gqfb1ecZz1C2P6+fPnwewVLMMcGjZZzVcvPY2oN9xKyTamIOMla4xxgykS+lqpquWbVDVXc2rQW2OWbBCzW6q11G1rPbi2B7REOJWmLD2QRWoehvEc7KQZe0HsHwm+iyyt4H4ZpGpbg3OiMcx4Yx6HmggBe8v2mT5WdtXO3TtDcWYbcdK1xhjBtJUuqpOWmpF7Xgt+68qoamENLVUkpqeUT0savZEJQsZjqouC91tqfLM97V1XT1Gn4E+x5pHhz4DTXcZ0XzHpMfmqm8b+vxq4dzsW5bb2JhtwUrXGGMG0lS6LVuuonZDVXuaEjEeU0tWHrfH62YRaLVjSeYPrKpZZ99r7WU+y7UkOZnirdl4Mxt15qPcSrSuNtbWc+R11ce2JzGNJsVRX9z4nWt7TmJuthUrXWOMGYgHXWOMGUhXjTSdRKoFHihqkmglW8kmq2qv0Jk7GSdtuGyZCHQSrpYQRs9V9zJ9lY+TVdlzIzoZF6895aJWy/nLPqgrl5ptYj+ywAYtBspzmHAnXjtLrENi8VBepxYYYsw2YaVrjDEDaSpdVUg1BaqqdapqAlAPYAD21gnLqgXHY3sqR2Qhtlo5OEt3GM/RpDxaOSOeX6sPF8+N52RlzfXZ12qyZYl09H6j0s0m0LIKIPFcnQjMQqXjd6DnOOGN2VasdI0xZiBdwRG0/6l6BfaG32ooqLZV26eqMQsljvu0om6muOM5qnC1HlmrkrC2paqulX4yczurhdaq0lXVWlP9NTtvXFcbdqtPGm5cqzWX1V7Tc+I11NZeuw9jtgErXWOMGUhXNWAqQs5Gt5RgpnSjkz2PyUrU1JJ86/WyJDJ6jXjOVELyWsJ13abpJms2ZFXhmli9ldAnQ+3c8Vz1GtDk7+plUOujhva2gj+ypOXqLVFL6GOFa7YdK11jjBnISkq3J4lNFnoalZOqw8yPtTb7T8XERC3RFxSol6bpSaQTaaVpVPupJgOP52sIrdqUW4nPM4+RWsJ39QbRhDM1LxB99nye2gZplTBqHZvdnxPemG3FStcYYwbSZWDTRDBRpWRJcFp2Un6mSooPuvAAACAASURBVFUbpKqiqCb5WZPU6Ox/VFtZIh21QdY8CDJVp+q5VWZdS9hrQprYvqpxTW5e84lVJZv5B8fS6aqg2Uct11N7nuoVkSUbqtmsa9+pMduEla4xxgzEg64xxgykaV7QRCo19PU3m2yLr9862ZbVLNO8t/Gc48ePA9j7Gtw6p/ZaH/e3+koy97NaMAb7wj7qJFVtMoxoRYysNlwNNX2oeaPVDq/LibWaC6D2n33TCb3oysZnoO0as234l2+MMQNZqUbazkkdoahZpdi4j2hNLz0ubqd6UvWoarkV4JBN/NQCLTKlq2q8VclBE+to0px4TObexmOZYjHu1z7ofekzi8eoy53CCa/aRCH3HTt2bFefav2I19b+G7NN+JdvjDEDaSpdVTe1qrJUNbTVqdN7Kz1jpqRV7dUShKt7lyrdWljuVIXdWhpDVW9ZBeOastbw5pbtOHO30kAKDe2N+1RB63Vr6pJ91IQ0ek58ntn9tFzG9PtfJQTamIOEla4xxgykqXTVEb9VPofQRqgqLyqdmqKM661yPer4r7P7moYSyNM+qg1Xk8oAdXUfaSXeqSXDmTpHVWqWYrGWVDyzqfYo+CyRUM2W2wrTjm3H34Zua3lfGHOQsdI1xpiBdHkvtBLdqDJSH81aORsqJA0DVnVVU55UzKp0qXC14GI8Jku6oqoynquKUBPqsI0LFy7snKNhsuqfy+3Rc4DnMCF4lli95t2Q2Y5b6SeJJjjX9dpbh6riLHVlzYY8lZLTmIOOla4xxgyky3tBk5PUyrBo9BUVm/rTxnNUkalnQs0WmRWg1KKWtaQ1qjS5rrbrWjIZ9SHWfsT9VL+qhrUfLb/gLAE5z6FvLLCMNFMvBm0zPhNVqZn3iaroSOZRUXsz0rcNJ7wx24qVrjHGDGStwpQRKia14VJ9abrB2K6ua+RW7Xj2RW2O2Qx+/KxKV225tbSJWeRUVlyz1pes6GRsO7v3TJ1HG7n6wKo6zhKTxz6oatVik62y9CTLoRGvbe8Fs+1Y6RpjzEA86BpjzEC6JtJ0uauBxastl3zt1dfIWp0zTqbwHG1D+xHRCTM1FfTUH9NAhJopQSf9tK2aC5eaKXQyruccdctSV7iewBH2XU098Xx9Fnz2NC+00npm7mzqvhevo653xmwbVrrGGDOQlZTuzklBpWThneo+FAMBGEigCjBLLh6vr31RpasJW/Rz7ZhMvQJ7XdKyRDQtNT6V2KfWRw3goGqlAo3HZ0EXOslZU5f6/elEaGsSTlEXvJjOkX1qhXgbsw1Y6RpjzEC6yvWQWorAzKWJqE0yojbOLAy4Zg/OQodrylNVcHbdGqp0NdijptJjSHBsI6ssXOsL71Ofa82mq6pYqxu37NxKdl89bxtc8hnV1DGDMNQ1zphtwUrXGGMG0hUcQZtgqyii2uxILYl5lphFQ4prqQN7EnPH/bVjNVhB1Wst+XYMu41tqWqO+zT8V2kFfWhAitqQ47lZoplWak59U1BF3QpmoFpV9X3ixAkAy+cZ3yA0/WOPjdiYg4iVrjHGDKTLe0FnsGv2PQ39zPx3gdzTILMLs9x6PFcVWma3jdtUvRHta1SEauPMvAzidi3CmD2jmlrNku+07MGZd0bmMx3P1+TvWmpIC2bGz+fPnwew9y2gpmL1bWOqIKYxBxUrXWOMGUhXWJDaLWs2VlV3GnlUi4bKZtDVxqptA3sTZ2uSmah0s+tktta4PStVpMqzVXgzK9cT+6VKeqpcfKuMTtb3VhSbRqS1zq15NMT12huR2umtdM22YqVrjDED8aBrjDEDWcm8UKuSoE70mStXzYVLX7/VvUxfdWMf9DqaAKc22ZdNumXVbLWd1v3Vqk1kARu1ihj6ep/dT20STifqauYLvc/eBDStUOZTp07tunetChHNKjQn1CY6jdkmrHSNMWYgXS5jrarAWcIZnRiKqkeVGNc1BWEt8CGrittT8ZaoytJKC1H1sT0qNZ3c47Km5LVycZaOMqITV3pM7TvQ56nBJrXUlRqaTOg6puG6ceKrFvASr9uacG1NjhqzDVjpGmPMQJpKVx3nVTHGbWojVFVXU7pZWLEmw66Fy2b20VpiHaJ1yPRcrbgbj6Uyq1Ub1utO1URrJQ7S59iTulLDjgkT72g4d0SfIxUu12upHTP3Nv0e4/3ps7BN12wrVrrGGDOQptLNwnNrjvmqfloJTtTuSkXUk/Bmala/FgBAW7EmjdEAC95DDDtWm22WUrJ2f5k9lkS1p2o1C0CoqWQNIeb1qVpr1+exWVpN7Ue0wWYeJK37VQ8HVwM224qVrjHGDKSpdLPkKzX7XuabWktik/nFZrbdmg9sK7QV2K3MWOJGfWC1H1qUMX5mH7QkDu2mrQQ0mR9tLSmPljdSpVtTiKowNRWnqtZ4PfU2Yd+0aGgtabr6baudtlWOqGV7N+YgY6VrjDEDaSpdneXXpN/xM1Vx5kVQK02TFbXMyqrH6xC1hWpkV/ysCjCLKovXUJtuVo4o2qkz3+HM8yJ+VhurKs6aDTmL/lP1Woti0+g/9daoeXRkdtmsfLx+jvdlzLZhpWuMMQPpKkxJtVMrwqj2vCwXQy2CSpWuRjpltt5Ilmcg9jGzSas6r3kGZDPyVK20vWZlgyI8J/PjbdFKgJ55UrTKK+kz1++pxxMhS1Rfs5nTk+KZZ54BsPQoMWbbsNI1xpiBeNA1xpiBNM0LTN139uxZAPXqtpl5gegEELB8tdRj+dqqKR1bgRVZ+GxtEiczfWQuXrX2+QzoKsZXaD6r2H9tQ80KLZcxrXOm/agl9slMLXoN/RxR84X2Pfab22qTl8BuEwWfG49xwhuzrVjpGmPMQJpK98YbbwSwN0Vgy1Fe3YVqqosKSZPI1NRbhk4A8fqqQGP/s8TnWmE4urtxokxDa7nOkOFYEVdTOqpqrYUsZxOPWWKflrtZppajss4qFGdKupa4vlUHL95/vL/sLceYbcFK1xhjBtJUutdeey0A4IknngCwNxACyBNaUxFyGVWPKiUNwtCw1ZoC1uu20jNq4ESmdDXENx6jKpjXpdKNNkq9n2x7TekqWQmenrBjdb2r2ZCzBD6ZS1mk1m7WR5I9G2O2BStdY4wZSFdqR9rhajPZVHq0G1Ixqb32zJkzO+dQUWqIrSpbquRagh2dOdfr0Qk/nq+KNysvE0NUswCDrB/x2OzcmtdElvBc29TnHK+j9tjMGyS2p33JlrX+qw1Zyx7V3hhILSG+MduAla4xxgykqXQff/xxALmfa/zM2XsqP/UYiMqMNmKqUbWLqn8uPRJie3p99WKoFZdU26P69tZU5FSZmZrHgxaG1O2adLzGVHL22uy/2mc1LWPNe0Gfge6vJfrR/mvhzVpaSu47ceIEgOVvwJhtw0rXGGMG0lS69913HwDgpptuAlBXMPzMZN9UvFpOvebb+9RTT+06Vv0+2WbNfql206wEPLC3sKamPFQF34pI0/Zr3hnsm5YD0uvF66hXhLahtIo+Zqq1llRcvRZ0ey1hkT4Dfb5qOweWbzNaOsmYbcNK1xhjBuJB1xhjBtI0L7zhDW8AAFx99dUAlhNrXALLSS5u4+sxE8Bw4qTmesRXVrahyVBqyVH0VTZzN6tVSdCJpswkEfuaJYBRl6442ce+ZRUdauHUamLJEu3UQnqnqvLWQnuz0Gv9nrRScmxfn4kmRIo5c1uBGsZsE1a6xhgzkKbSfdvb3gYAOH/+PIClm8+99967c8ynPvUpAMAjjzxSbYOKlxMpwN7Jm9OnTwPY7UwP7FVHwF6FpOqqVqMtq7qQBQL0VNxtVWfQUFedqNOJwrgtO0cDHFrJcnRdXbri58y9TN3Zaspa29Jja3XqsnVjtgUrXWOMGUhT6b7lLW8BANxzzz0AlsnMb7/99p1jmP7x4x//OADgE5/4BIClArzlllsALG27wFL1qpsSXciyJCwRVWqtJN9T9cuy1Iixb1NKt6UE1Vas1Ylj+2oz5rNqpdckWXIc3R/RvmQJ2CNZ7TpV8rXnXrMRG7NNWOkaY8xAmkr3da97HYBluO7NN98MYKluAeD6668HsNd2+7GPfWxXW/GckydPAtid+BtYKjEmx6mlaSRZshUSlZTaK2nvVaVWC7nVYIGehOuqjlXV1ZLkZGHMalutqUxVx1lC9Jrdmc9Gg1k0zWZPKsZWBWF9I1klYb0xBwkrXWOMGUhT6VLl0Af3hhtuALBbtVLZcskZeSqmj370owDq6pF2XqosKmAtiRMVU81PFVjaPGs+slnSdA3TrfngqneCHluz6ao6VZXYUnvs/5Q3Q0v9Z94FrYKimQdHLeFN9kxUhccQZu23la7ZVqx0jTFmIE2l+8EPfhAA8IEPfAAA8PrXvx4AcNddd+0cc9111+3apgluqNw+8pGP7JyjNk8tA8T1c+fOAah7E+gsuyrQmqpTm6OW+mmpMFWAqmZj9JX2jW8Mah+t3Zd6C2QKu1aOqOZpEM/Nyq7H9rIE7LWkQ4p+99FXep3ioxeTHht163kZsy5WusYYMxAPusYYM5CmeeHd7343AOB3fud3AABf/uVfDgD46q/+6p1j3vjGNwJYupO95jWvAbB8taSJINZIYwAFX0fpdqb1zrQKBbD31TYLfa29GmbVGNTtLF5DJ+60NloteEIn+TQMly540c1N+6+v+1n4s7YT+6Y5cWs10jJXLg3WqLl/aYgy0UlGYG8lkVfCq3uPWcaYVbHSNcaYgTSV7mc+8xkAwLve9S4AwHvf+14Ay3BdYKkC3vSmNwEAXvWqVwEAbr31VgDLCbYHH3xw55yHHnoIwDKBDt3NtOoEl7XKvure1VIlOnkzNZnTCn3VoIJatQk9h8qT6pTbY4IfDZ3NUkiy7/FcVcE6cVhLYqMTd7quKrnmMqZ91jeG2nN2+K/Zdqx0jTFmIE2le+eddwIA7r77bgBL5fv+979/5xjaaqn86EJGRUN7bXSpon2PydEZbEHFqzZPJtoBljZiqi1et2WXVfcgDQvOXNeAvTZPqlVNvN5KksM+aeL1aAtVW64eq3Xd4rm6T9VpLUlOFkBRC/aIbcTP+kw0dLkWdsxz+T0as21Y6RpjzEBKj5O4McaY/cFK1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBuJB1xhjBnKotfNzPudzZgBwyy23AACuvPJKAMCLL764c8yFCxcAAGfPngUAnDt3btf2K66Yj+uHDi0vxW2z2QwA8PLLL+9a12UpZc+57Av3ccm2Itx3+PBhAMDRo0cBAMeOHdu1zrZfeumlnXPPnz+/6555Hzz2qquuAgCcOHFi5xxehzz//PMAgGeeeQYAcOTIkT3PhNd89tlnd52j+/lM4rn8zPvksc8999yuNrke2+c5vB9dxmevfeEz4Trv6+TJkwB2PxN+Zl8/+clPAgDOnj279wIDKKXMLsV1zfYwm82qv20rXWOMGUhT6d5www0AgDvuuAPAUtVFNUkVRRVHpUuFSMXbA1UV1VzrGEJFpgo3qk1+pto6fvz4riX38/7i9akOqeSfeuopAEvFdurUKQDA1VdfvXMOFR/VONtThRjhMS+88MKu61KRsq9U5exr7RnwHD57fhdcAnsVNftUe1NQVB3zjYHPV5ex3/pMjNk2rHSNMWYgTaWrdsua0qV6U1trC1W0ql5rdsSsbwqvH22eqmSpRLmdx7I/0Z6qCpfn8i3g2muvBbC0Y8brsI9Ur3wb4DOLapXwWKpSfZNQG3bsU7xnYPk9sc2osPmZS21XlWhcr/Uhrte+A94r92XfnzEHHf/yjTFmIE2lG2fxgb1eBcBezwOew+3aBjCtdHtQ2yPbqtl4M48HnqP2TNqlAeDJJ58EsFRtN954464lle4111yzcw4VtHokcDv7Qztn3EZFSxsur8+2qLxrNlFVzqo8W89Zv7/ad030GD63bFlrp8d2bC4/em3x6/xNbwtWusYYMxAPusYYM5CmeUHdivhKGCdkeEy2pOtTz+vGfh0D7J7EUSd+rutkDvv8xBNP7GzjJNRNN90EYDmBxvWbb74ZQN1ljNc7c+bMrmP4uh8n33gs3bzYB96vBlhwYi+eS9ctDfJQk0/8zGehk21ZoEpEJ1H5zHXiNd4z2+d9mMuXTdz6aufa5DDHStcYYwbSVLpUI1RftTBgujZpUATPrSnd7H+8dZSuTsZxPbqucQKLak4nnHg/nKQ6ffr0zj4qWk6YXXfddQCWCve2227btQ4sFSfv/dFHHwWwfDbcz7YiVMXqnsfn+fTTTwPYPdnHbTyW51KJ8r6jKxw/azDGKpOnGujQCiHW8OUYqGEuLy5W4MomE+cHCStdY4wZSFPpqmpVZ/94TKZ0qZyijVUVUpbwpuaEr/9Lquri/pZNV9WcJu2J52pIK8N+qWzvvPNOAEtFDCyVNZ8Jr8/ttO3Gc3hNqlbejypc2oHpShb7zXunK5qGFsfvjUpXw435PFsJb4g+1x4Fw2fA+zGXD6NCs7dd8VrpGmPMQLq8F7jk/0wxRSDVHBUZl6pAo3pU+2trhjxj6tjomcA+qBrWGXzeV/REoGqkRwKVLr0YXvWqVwHYbZ9V7wHeO5dsP15H00yyL/RiePzxxwEsgyZ4fNxHtarpJ1tvDJq8RvfXwnX1+XHZSuep8wGautKYbcFK1xhjBtJUurUEMLpesxcCS5XTUrq9oaCrJMDR68bPmedDT1pITVZOpcn98XqaRFwToWsiofhZ0yRSWavi5jLeh6aMVLUa+5jtm1LA8XpZAvmamuX3oiHE5tJzqb6LbbXtWukaY8xAulI7qoqNikqViyZZaaXy603p2FJbOtuuy/hZ+6IKt6YI1S6apYGMEVbcRn9f2mXZLhVwLdWiPkdV8C0FukoSmcyGq+3X7MFZ2sdM8QJ7f0NO7Wi2Ff/yjTFmIB50jTFmIF3mBQ0miOu13KlAn3F8ylWs5eqUvUq3rpuZM1pmDq02wYkuTnAxPJjrwNJ1iklp1H2ulvRFJyS1xpyG7bZMCfpcNfikdn72LFpk59R+ExpAYfOC2Vb8yzfGmIE0lS5RVVRLEZhNrmRuWa1jWqorU8U9VQv03Ow68ThN5qKTRpzgim5mPJbVJK6//noAy8AGhhAzsCLChDcawqvVLeLbRpakRr+b2nfQuvep7Vn4dk+wi13GzLZipWuMMQPpCo7I6pHVtrVUlaLJtVXp1tAQYj2nZU+s1WsD9rqS1RLDcBuTy2iCn1jvTNUwVbCG8MbgCE0eQ7cyTUijScfjvW5iJ81Uauu70Ge9Sji3a6SZbcVK1xhjBtKldHuYUsUtm+AmSler2NZK06g9VJWhBnIwwU/8TE8DplT87Gc/C2AZrlurPkxVrCqV6jUmIucx9HjgdbiuCYWi54MqdX1GrXI9U28oPfb1zK7fwkr38kF/L6Ovu21Y6RpjzECaSncdNaKqp5UMuzeJeY8NWVVdLcRW7aYaYktba0ywTTXKkF7649577727+hhVK221tPfSI4H3WfN4oD8uS/s88MADAIDHHnsMwFI1cxmVLtuJSYXiM6l5PGRvGT1vG9p+T/J5xUrXbCtWusYYM5C1bLqtZNhT/rqt9tdRTETVXC3Zitp2CW2ijDqLXgVUqfQ8iF4KwN5k4/EY7qM61aKSMbEOVTGV7cMPPwxgmaCctl22EVUr1bcmxVE7dy1RUeaB0Hrmq/jyrtqGuXSMsu1uqy2XWOkaY8xAPOgaY8xAVppIyxLgtM5tOexPvWbUTBNTOXhr7lHcRrcsrSXG13yt3gAsTQN04dIKCDQLcAIMWE5saSADzRc0VURoctAlzQo0M9DcEfvIdjXXr14/BlRkk4tKTyjvOq+jNi9cvlwsM8O2mxWIla4xxgykK+FNVmVAPwPrTYZl+2pKKpvo0YQ0UblpzS4qPq5z4otKMU6W8VgqTF5Xw4JrNcsYWKGpHamoa/fNPrFdKl26rPE6DMqI7en96n1GpZsFk/SEEvcqICe8eWWzSsKpnnPNHCtdY4wZSJfSXYee+meZGl4lRaDWO6spXVV8VKCZTTcGLZw8eXJX36g42SaDFKLS1fvTKsC1RN5sT8OONaEOFW5Ut2xPFa4my6nZdKdsuT2uYz3f1yb2X3P5YAW7OVa6xhgzkLWUbs3Gquub/I/YaiPbl6V8BPYqQFW6qrRj0AIVpXocsA2qyah0td2spFEtgIPtUQVT2Wo6yNjH7P502VK6WZJ2UvvOexPY17ZZMZltxUrXGGMGslIYcE/JnZ62VvFW6Gk7UvMxpJqj0qOqo71U+xNVK/ep4lU7KRPWAHs9ATIVHpUu7chqs9VkNhriG/ui9mCua2rJ+Fmfeea90OOJ0FLHmdo3Ztuw0jXGmIGsVZiyZavbZNZ76vjWdXpUVuZFENMk6rlUnDyHSypQTYQer5ehbcXzNSE519UTgrZfILfhqi03Kl1V46so3KzgpbbR+t6M2VasdI0xZiAXzU9X6bHpZudsOguepX3MVHJUhBqtpmqupRR1n14vnpO1o31WD4X4WYtZqodCtANPKdwW63gerPp2Y8xBxUrXGGMG4kHXGGMGsnFwxDrHTp3fSi2nNbzWqUCr5gXtVy1ZTjZxVutHNpHWqheX1SrLQnvjRBo/awrHLABCP2v/Iz2JiqaSHtW22bxgthUrXWOMGUhT6a6jYHR7bT1LbLMK6q7USiGZBSdQEbaCFjQtpE6oRVcx7VsWXlybSCPah540jXpspnBrKTn3Izx3KtglfnZwhNl2rHSNMWYgXTbdHnvcKgppSuG2rqdJZFZxeZpSvDXbJ/dlwQs9rldZSsf4zLRPGrqslYxrdmd1DWv1LXtTWMe2q23abmtMjpWuMcYMZCWl20OPHTjbN6V8a8dkNt2WbTdTdbUggsz+mgU+tK7XU7pIy+fos6n1UY9V2zHp8bDI+r4pVr/GzLHSNcaYgazlp1ubBe8tp17btootcB2llPVtFVVOZbmKet3kDUHb6EmNmKnudTwHer6Ddb4/Y7YdK11jjBlIl9JdJ/qLrOL/2WPTXadPWYLuTK1GRZgpzCyJTa1vPQowux9db11PUYW7H8q71r6VrTH9WOkaY8xAPOgaY8xA9t1lrOecTWqg6TkttyiSvbpnr9DRDJGZHjJzQ6tPrfueMiO0zBnZdVqTb+tMfGbX6WHKXc+YbcFK1xhjBrJSNeDWMVPHrqOK9istpKpiqsasam+tPT13EyVfO7d3km+d2nO1twLdtmm6zthWLfy4FQJtzDZhpWuMMQNZy6a7jipaJyHOfh2raq6W2CYet5/pDVv7elzIes7JbLmbfG/6zFZJZl57jpvUZDPmIOG/AGOMGcjGSnc/ZqGzme1NFHXNfjl1bo+9cRVlPXVOq4+rqNT9CMNdReVPhVXXVG2P94Ux24CVrjHGDGTjhDeZglkn5LV3f4tW4nNFbbs1+2VvspwWPQp+E5W6jkru7X/rzUHbaNl07b1gzBwrXWOMGchKCW9qCnRVxdRqV4+p+cT2KsFVlJQmQo+2yHW8LzJattf9VLqrnJOp1RbZ76HV1ir+xsYcZKx0jTFmIF0l2NUOF2P5e5VLK/Whkvmd9pzbUtS9ts/W9Xr8V7Nje5RuL6v4+K4S4beOsu/5DVjpGjPHStcYYwbiQdcYYwbSZV7omSDJaCU/ydZbaRN7E95MbYvsd5jzlDmjNWm2zmt3r6ml5vbVe+82BxizP1jpGmPMQDaeSJs6pxUS2hue23KtWmdCTdf3O+F672TUKqq8FVqsLm+bJBdaJcFN7/dojFlipWuMMQNpKt0sxLfHPqvHXnnllXuOyVRVzXWrl/226WZ9WyctI1klvWGP8p36DkhPMMY63x/DqVdR2E7xaLYV//KNMWYgKyndGlMJrKmQWupYk26vEhLbq/JqTCnt1raeIImsj1Pba/tax2ZpLXuUrr5N6PfW+v42SSlppWu2Ff/yjTFmIPsWBpyVY+mxA/cqtZ6+rmLTnbJrtvZtosJbx+7Hva/ipaEeCPo9tpSuFvakbVePa/XZmG3DStcYYwbSldpR1WvN53aVIo+ZPXQddbfK9cgqttzefrTswD332xP9l/W112a8iu9ty6Y7Re1eXJjSmDn+CzDGmIF40DXGmIE0zQt8heUrZu31NNunLkk9obxT+XVrrOIats72Tfq2br+A6aQ5Pe2uMjk35frXSpbTMyGp5gWddDNmW7DSNcaYgTSVLtXIVVddBaA+CXLo0KFdS0KF++KLLwLYrXqojrMKERerXtiUMtuPig4teia2ehX1Kmkhe5RuNsFVC6LQ8F8u1e0shg7z93H48OE97RmzTVjpGmPMQLqCI44cObJrPaoiKllVw9xOWq5CmeJVBVWjxwWpN2F3TypJsoo71n6yiRpvuahlaTtr7nU85oUXXti15G9AVS0AHD16dNc2B0eYbcVK1xhjBtJUuseOHdu1pDqhsgGWSojKlktuf/755/e0q7Y/3U64/2LZ/zaxse7HdXoCOTYJ4Gip817Phtqz5xsIfwe03R4/fhwAcPLkyV3L+Jlq+MSJE933YcxBwkrXGGMG0lS6VCe0x9FWF30suS2zy7aS5GQJs1teBpuEDu+Hz+0myje739YxPWHA2b79SB1Z817gsVSttPnz93Lq1CkAwNVXV57TdAAAAbZJREFUX71zDpUtz7nmmmvSvhlzkLHSNcaYgXQpXdp0OfMcla56K2S23ZptMCsHRHoitnqSvazjn9uzr2d/ra+15PDZs1hF6WYKtyfpUHY/te9PfbL5+1ClyyWwVLq0/0Z7rzHbhJWuMcYMxIOuMcYMpGle4AQaJ0pqIZw0NfAV89lnn911Ds0MNfNCa0IpUssdu06lhczU0RN+PGWaqIXJZmaFWm7ZzPSwSUKfddroSWaj1+PvRF0MuYzH0LxglzGzrVjpGmPMQLrCgFtoZQFOsuh6TQnux0RaT8VibU+VbeaqVtvWk45yHaWbVVbI3gJqLlzKKs9G+56FA8fPbJdvQPyua9Um9D5iMhxjtgkrXWOMGUhT6d5zzz0A9gZHxGQ2Fy5cAACcOXMGAHD69GkAwLlz53adQxsvsFf10C7MdjVlYCsMeJXaYuu4jGXX0/WW4tX1nhppU6q/ZWOdsvEC01V/W4nrNeENvzfaaa+99loAuwMgNAz4D/7gD6p9NOagY6VrjDEDKfuZpNsYY0wbK11jjBmIB11jjBmIB11jjBmIB11jjBmIB11jjBmIB11jjBnI/we4g3+SyVH42AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x576 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Explore some data and their corresponding ground truth\n",
    "from utils.visualization_utils import plot_image\n",
    "\n",
    "some_images = np.random.randint(2400, size=(3))\n",
    "plt.figure(figsize=(6, 8))\n",
    "for i in range(1,4):\n",
    "    plt.subplot(3,2,2*i-1)\n",
    "    plot_image(X_train[some_images[i-1]])\n",
    "    if i == 1:\n",
    "        plt.title('Index of damage')\n",
    "    plt.subplot(3,2,2*i)\n",
    "    plot_image(y_train[some_images[i-1]])\n",
    "    if i == 1:\n",
    "        plt.title('Damage')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Preprocessing \n",
    "\n",
    "In this first study we will just subsample the data for a quick run and vectorize the matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocessing: Subsample the data for more efficient code execution in this exercise\n",
    "num_test = 50\n",
    "mask = list(range(num_test))\n",
    "X_test = X_test[mask]\n",
    "y_test = y_test[mask]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2400, 3621) (50, 3621)\n",
      "(2400, 3621) (50, 3621)\n"
     ]
    }
   ],
   "source": [
    "# Preprocessing: Reshape the image data into rows\n",
    "X_train = np.reshape(X_train, (X_train.shape[0], -1))\n",
    "X_test = np.reshape(X_test, (X_test.shape[0], -1))\n",
    "y_train = np.reshape(y_train, (y_train.shape[0], -1))\n",
    "y_test = np.reshape(y_test, (y_test.shape[0], -1))\n",
    "print(X_train.shape, X_test.shape)\n",
    "print(y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preprocessing: Normalizing the data\n",
    "from sklearn.preprocessing import Normalizer\n",
    "\n",
    "# Create an instance of the normilizer\n",
    "normalizer = Normalizer().fit(X_train)\n",
    "\n",
    "# Fit the data\n",
    "X_train = normalizer.transform(X_train)\n",
    "X_test = normalizer.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the data is ready to be passed trought a classifier. We will try to use the kNN to classify whether a pixel in an index image is part of the damage zone or not. Just remember that this classifier consist in two steps:\n",
    "- It calculates the distance between the analized pixel and the training data.\n",
    "- Then, it choose the label comparing with the k nearest neighbors.\n",
    "\n",
    "As we want an output the same size of the image we implemented the kNN in a little different way to the typical one. We calculate the distance pixel-wise so at the end of the process each pixel in a test image has a distance to all the pixels in the same position of the training data. As you can imagine, this process is computational expensive that's why we use such a small amount of test data.\n",
    "\n",
    "Note that this is one of the main problems of kNN. Since the trainig function is a constant process, the predict function computation grows as we increase the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50, 2400, 3621)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from classifiers.k_nearest_neighbor import KNearestNeighbor\n",
    "knn_clf = KNearestNeighbor()\n",
    "knn_clf.train(X_train, y_train)\n",
    "\n",
    "dists = knn_clf.predict(X_test)\n",
    "dists.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 EPOCH\n",
      "20 EPOCH\n",
      "30 EPOCH\n",
      "40 EPOCH\n",
      "50 EPOCH\n"
     ]
    }
   ],
   "source": [
    "# Now implement the function predict_labels and run the code below:\n",
    "# We use k = 1 (which is Nearest Neighbor).\n",
    "\n",
    "y_pred = knn_clf.predict_labels(dists, k=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcwAAABsCAYAAADnsILcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAT1klEQVR4nO2d2W4bxxKGi5slyksQI7nIRZAAyQPk/Z8kyF0AOwuQ2I6XaOGic6FTw9LPqp6mbck+h993M+JwprtnKHD4d1X/Nbm+vjYAAABoM/3UAwAAAPhfgAcmAABABzwwAQAAOuCBCQAA0AEPTAAAgA7mrTefPn16bWb29ddfm5nZy5cvzczs3bt3Np3ePGuXy6WZmU0mEzMzW6/Xt7bb7dbMzDabzfC3b53ZbHZr6++v12vzLF5vX7eOv45tV8f41tvebrd7/Th+nc5mszHFj2m1H7eTycTm8/mtc/3Y1Wo1XLvZ7p7EcfixmuGs44j33N87OTkxM7PFYnFrv/d3cXEx7Hvy5MmtMb148eL2zbkjJpMJqdswcH19fef/d/zPQaT6n0NhAgAAdDCmMM1spzSGk+ZzOz8/N7MbtRlxdaOqcT6fD4rIVZpvW4pJFZIqPj/H97tyi2NxfCyqRrfb7d6xOqaoDrVtVXbV2tY4Vh+njsX3V2o8Ut3rqFr9HrtK1Hvubfg5jx8/tocPH5rZ7nO/uroqxwAAcCygMAEAADpoKsxXr16Z2U65vH792sxu4lyuOlQpaYwvqspKYWocr1KcET1H+4vt+z4fs54zmUxGFawqsqj8tD3v14/x9zO1XcVIdezZmFRRqhrdbrd77agSz/r3z9nbu7i42BsDAMCxgcIEAADooKkwPU4ZlaXZTUalZ8d6tqWqOFVX19fXe5mzGj/LYoCqolQROTHWqPHImHWbtRWPVyWmY8pimFVWrB4b26zGOJb5mo2xFW/Vvqt4Z5ZZ6zMMWWYwAMCx0Xxg+peoPyg9ceT09NROT09vGiiSVPzY+GWuX+zV9KcznU5Hp0qzB031oPKHuxOTgLR9fQhWY55MJuUx+jBqPfz0enqWk+gY/cEWr7/6cRKnh/V9/dFT/UgBADgm+CYEAADooKkwHzx4YGY75eKvF4vFoNZcqbiiVHWnCTBm+6rGyZSaHqPHaqJLVKVONa0alWDVT7X0pTUlq/1n07va3iHTnjp+J5pEKJoYpGONU+ua2KTKHADgGEFhAgAAdNBUmK4Eq2QZs30FqQqmZwG+qqyo4qpYpbZXWeZl/bRs9qplKtWC/1Y8sor9xdhilSBUxVBjuz1GCaoWKxu/aEuoS078PQCAYwaFCQAA0EFTYWYL4c1u1FZlCqAL72M8rVJPemzLHKBSqrHNatyOjj0qqLFjXW1rv/GYngxbvXa17dPjsvfGXsfYbGUKoXHPTP3qNQMAHCMoTAAAgA6aClPXUvrruE+VlyuVzKRAVVRmUReJ2aTO2JrA7JyqdFccczWmKjZ7SPwzu87KVL6y/Mvs+/x+aj8xBlkpc41Lx89GrxlrPAAAFCYAAEAXTYXpysLX4UXFWcUbVWE6rezVKkM0i/lVmZvZes+xvqOqbJmdRypFGPvObAG17epaqwzYmPE6lhHsCnM+n+8V9lYXH9/fUqWXl5etWwIAcBSgMAEAADpoKkxXH64wo0+sKr5qPWamxKrYZSs+qcqycrlpeda21kVWRubVGON4KpXdWjOq8VxdD5kVuXZaLkdmu89rsVjsjVvj0Rp7ju1WsV8AgGOky7igMi83qw0EsiojY0tDlDgNqVU2Wv1XVnWHWOPpg6qV9JOZqmfHxmsYmyZWsuUylQVfbKuqUqIP9eyByYMSAGAHU7IAAAAddJmvV8sXzPanDtUMoMeqrvV+ayF/fB1VUaUonWy6uDJsr5aiZDUnx6Zxo8qr1HbLKq9KFNJ7HpN+1N5QTfKzflvT6QAAxwrfhAAAAB10FZDOtqqMqkSbrKRVZZGXqZ7KfL1KzolqsVKj2eveBJfMzFzLaR1iyKDbKlabxXN7lW08xxWnK01VtL3xZQCAYwOFCQAA0EFTYao5QRZzrMp4tTI2q2LMWempKktV+x8rdZX1G80Vqjhtzxir9nV/VHO6T5VlLLflr2NsUvs2y43i/VhVkmMFpat9AADHCt+EAAAAHXQZF2RqS5VWpVii8qtKTfVY42mcUInKVhVeZargxDWOVXZsNebYV2Wv14oPqmpUpev7Hzx4MPx9dnZmZvvrYtWWcLVaDX/r1mkZNuhYAACOGRQmAABAB10FpFUhLRaL0YLHLRN0VXqt+GGl6FpxtbHM2mzMnjXailHG97NYn7rmaFZrPL5XWZ6enpqZ2XK5HJTlcrk0s906Wb03UWG6gf7bt2/NzOzff/9Nz/ExX15ellnGAADHDAoTAACgg6bCrDxSp9NpqprMcjVldqP2xoo0K5nTT9VfjKFWyrJVQmts/WFlvh7HX5nIZ6913KosXUU+fPhweO1q8+TkxMx2McxqXet6vR6OjbFQs53S9HJf0Yzd9+n1AQDcB2Nr2T8VKEwAAIAOuqqVOJmHrKNZsq04mL6nxYxj7K/XeadVdFr7zdRdFX/Utlq/fConnkxhqsr1ex0VpdlOTc5ms+E+uQL0beu6/G9Xp3ovzs/Pb+1frVYoTAD4ZLQ8xMfc4eJ7zsdUp13WeNlDT6chqweK0yqhVbWZnVNN52Y3pXqYZg/7ngd93J8tv/Ap0upeZA8eP+fRo0dmtnuw6Y+Vi4uLvYeoll/Tz2K1Wg0PRF1O4lOzjv9QWK1We2YHAAD3RTR2qcJNY+ffFUzJAgAAdNC1rMTJzMory7rMKk+f/JUCjIqpKrvliigzNOhdDhGvb8wST8+JY62WhvixmTm7H+tLRXwq1hWnK0I/99GjR4PCVGs8P9ZVYyz47X+/efPm1taP1eUr6/X6VgKQ2S4hCADgQ+ktdBGPdVpJoD3TtR8KChMAAKCDLvP1TDlVc8uHxO96knIqVapJRjFhqDqnMoqP11MVTdbriqYLeo5el1oMTiaTIQ7psUtXmKoaff+DBw9K8wa9F5lhuytIV4u+rERjpqenp3ulv+B4eR/zis9lCQB8nvT8f4wdk6nIu1SWDgoTAACggy6FqTE6s7qEVSsGqLG8VoFq31bz0of88u0da3ZspVZb7emxun82mw3xSI9huuJ0ZenKz2Oa2+122Pftt9+amdk333xjZrt4pPfjccpnz57Zb7/9dqtvP9aXjnh/0djAx3J5eVleM/x/8yG2iC3DEYDe78mx91r77woUJgAAQAdNhelUC/LN8nhgPCZmvGqGq7bfYz5QZeVmqlTfq64rrvupfrFUJgTR9KDXKH4+n+/Z27nCc+Xn9zpmwH7//fdmZvbTTz+Zmdl3331nZjuTAz/2n3/+MTOzr776auj7l19+uTUm7S9m1moGLRwPH+sX+33Ek+B/i0Nm78a+jw/p72P+D6IwAQAAOuiyxsvmkas1hpWrTqb8ql8R2a+K3nWShxDH2jKAb73ebDZ790ezitX+bj6fD8eoWb3eTy/P9eWXXw6xyx9//NHMzH744Qcz2ylMjzm+fPlyaOuvv/4yM7M///zTzMz++OOPW2NxVRpV8FjWLwDAodzXbAPrMAEAAD4xXV6yWeHnzMknHpsVeK6yYTUuGeOgY645maKt1kxW45jNZoPS0+tyWmq4GlPVX2a+ru37Wkj3gl0ul/b48WMzM3vy5ImZmX3xxRdmtsuo9SLRrjSXy+WgPv36XLF6dq66A8UMR90CANwVdzWTVSnO98nm7npgascx0UXNBsbqV/b0Ex+g1bKSlp2ePlyrh7uapce/K2P4VtJPr1FwPKcysXdjAX/4bbfbvYeoPyA9cefVq1dmtkv6efPmzdCOn1MZFmTLf6prBgC4K6Lo+VjtmdXPkEPgmxAAAKCDg6zxIlUNxmz61ixXVU6r7Fdr+Ui2zY6tEoZaytKvvZLzmYVeZUivKjIanKuxebS1U3xMaoruY4hq1Pvz91xJ6nIWva44Pc3SAAD42FTfK+9jSvMh/b0PKEwAAIAOusp7ZU/malmEk6mtytygJ1aq5glV7DT7dTKWkJQpTd1qP9nYK3P37BdOlcTk6tH7i0k5/rcn8rgxuxoXeD/L5XJPUbrZu5YRi0lPPcb5AAB3wYfGMO+yzBcKEwAAoIODrPFaT/0xdbXZbPb2VcWhs3b1GFW4MXZaxUir/lr2dlWstufaVYV6HDG2UY1Ny3K9e/fOXr9+bWa7LFh/7f15DDNmxHpWrMc7vT0tIxZLhKn6zYp0AwC8D3edE3GXRisoTAAAgA7eax1mnGPWbFJVI/GcMRP0LNP20MypGCsdm8NuxeZUUerYtM/YXmVWXxWnjsd6zNHvo8cn//77b/v111/NzOzp06e3+nVDA1eTrkCfPXtmv//+u5ndKNTYtxaJ9u1qtRrWdfoxKMzj4WNlKpJZDZ+Ku/zfQ2ECAAB0cJDCdDLFUSnM2EYVQ6wybqPtXGV3pwbhMQY39is5U7892bDVGKv2dTytmKlnwroijFmyP//88632X7x4YWa7zFePYXoB6efPn9vz589vneNjcYs8NWG/vLwc3vPrwunn+PgQpYm6hE/Bffzf8U0IAADQwUEVgjOnnsrZR9+PKi4r/VWdU60J1ExOP2ez2RxcfDSObUxZZkq3KodVecuu1+tBDfpWiza7ObrHEWez2XC+F4P2mKavx/R7EX1o9TNz9ajZvz6Oq6ur4ZiWkT4cB6hF+BS08mc+JV1TstlykCqhpVpWst1uu00Oem5KNhUbX5uNmyrEbe8UVJZQpMtFqraitZ0/1DzJx9vwh58aC1xdXQ37HG/DTdhbBgn+QPT3tC3v5+LiYvj7c/knBYDj5nP5DkI6AAAAdHCQ+XqcenRF5FRlqqqp2nhstT+bkq3UW1SLqj51SjFTsmP96PSqtzGfz0u7vjgFG1+vVquhHTcf0Hvhy0lcgV5eXg6KMjNAqK7Tl4j4lK/ODPj0qx+XTePe5UJgAID35b5nwVCYAAAAHXQl/WQL8lVh6vKPDFVclRlA9qthzDg9WyJSJd1ky1tapb+yMbbKiVXJTTEeWY1ft25lt1gshvb8/NhOJBbI9uQhHa8nE3kb0UJPP2+MCwDgc2dsJozyXgAAAPdEU2FqLDDG6sZiiUosZlwtU8iWkPQWG20tX6lUYxxPZYxQFb2ORg2VstSxuiqfTCaD2vStmw1UxucnJyd7RaB92yqynS1pMdvFLt0gwV9fX1/vxTkBAD5HslnIu4xrojABAAA66IphZhmvlem6KrG4FnIsttiTSVupxfiroloPqW1lircyj6+ygNfrdXfRae8nZtaqkvXSXP7aY4xnZ2d7azar64lj9PZdWXrs0vtxhRnN2NWqEOMCAPgcue8Mfr4JAQAAOmgqzCquF/+uTMQzFTT2a0DdgiK9JbqiQtKxqZl4S5WqMvNjNTv4+vp6cNHRNaeq1Pzc2Wy2p261H23z4uJiWEsZDdnjNlPj0V3IzPZKd/nreG8q5yIAgM+J+/5uQmECAAB00KUws7WPGtPTYzR+l5Xqqn4dZEqziktq9mc8b8x1KLZZxVf9GDV5j+pO1yuqotV4ZVxT6VTX5QpwOp0OqlM/l5aHbeXWU+2P1wMAADtQmAAAAB10Zclq1Y/ojDNWYDlTPb1rN6MvrG6zCiqOKkinOjeLzWosNsYfY1tx3aIqWO0nKlBtp6dQtY5XVW/lMBT/1vul8dbJZHJQ1RgAgGOhq7xXa1nB2NKQOMVYPajGymJl6Bd+HGs1VVmVIttsNmUdz2qJRUwOqtr1aWK97mg/p4btPUs4DrlP1dIa7T/7wdFjdwgAcCwwJQsAANBBlzVepkp0n6qSalo0ttejpiqjAm0rU5iaeFSNeb1elypUFWbPFLNTlRWLCUpjCj2bVlWqhJ7M8k/PyaaCKe8FALAPChMAAKCDg8p7ObPZrFxKUSXjxAX/VbxOz+1RPZnxeGXUXpmxZybvbg6gbamSzsamqq1HnVZLYeLYxxKt9F7EMVafS8ucoLI7BAA4RlCYAAAAHTQV5mKxMLN9M/HFYjH8XdnNZUpQLeIqhelkSxyqTNvYT8vMPR7bsu0bWxrSiu9Vhg9ZzLbHfMDf10LUzoeo0mw8+lm62gYAOGZQmAAAAB00FaYrQVc0USFW2aOqgqLa08X/aoigpbW22+3QTpVNmsUrK7UYj2m9judkY4pEFaftqbqL16sKtopHxvuqmbtVRnJLRfZkAWs7ajgPAHCMoDABAAA6OKi8V9z2rqWMCq3XRi/G0LJSXNmxmYqrDM1VOWXZpNXYVBXHfWOWf3E8Pesu4xiz8WtcV4+L/Whc1ePTWoYtOye7ZgCAYwOFCQAA0EFTYXo5Kc+S9KLD0+m0jGE6WUmtyqO0ta3WJ44pz6y9yls266faZuprTDFn52ZrQVtt6N/xnKzEmZ4zFt+M42iZ0wMAHCtd5usnJydmZvb27Vszu/li9odn9aXt7/s2PpR0GtDJqqI471NBY6zuphOXUui5Y9tolJAZPMT+s3Oc6oEd76+eUxkLZMtK9By9zmhE4TU4Hf8MAQCOGaZkAQAAOmgqzNPTUzMzWy6XZrZTJ5eXl3Z+fn7TwH/Voi7sz6ZdtdyVGhko2+12dEF/a7ozK/0VX8etqtsx0/LqdXZuNo1cXUdlAJG1N7Y0Javz6Z+LLk1xYqmzs7MzM0NhAgCYoTABAAC6aCpMj2W5MvQkoKurq0F1OlV5L2c6nZbG6dX+WHarWtqgKism4aj6dKWk5u+bzWZvLFVMU69vPp+nBakjGrONhgxOlTzVSrhpxS79td5bNY9Qm8Krq6vhc/b7pTFNAIBjBIUJAADQwYTiwAAAAOOgMAEAADrggQkAANABD0wAAIAOeGACAAB0wAMTAACgAx6YAAAAHfwHPeS8oDqzM+0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 576x864 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's see the result of the 1-NN\n",
    "\n",
    "some_image = 6\n",
    "plt.figure(figsize=(8,12))\n",
    "plt.subplot(1,3,1)\n",
    "plot_image(np.reshape(X_test[some_image],(51,71)))\n",
    "plt.subplot(1,3,2)\n",
    "plot_image(np.reshape(y_test[some_image],(51,71)))\n",
    "plt.subplot(1,3,3)\n",
    "plot_image(np.reshape(y_pred[some_image],(51,71)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the performance of the classifier we will use the intersection over union value. This is defined by:\n",
    "$$\n",
    "IoU = \\frac{TP}{FP+TP+FN}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4454083531843866"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_positive = np.sum(np.where(y_pred + y_test == 2, 1, 0), axis = 1)\n",
    "false_positive = np.sum(np.where(y_pred - y_test == 1, 1, 0), axis = 1)\n",
    "false_negative = np.sum(np.where(y_pred - y_test == -1, 1, 0), axis = 1)\n",
    "\n",
    "iou = true_positive / (false_positive + true_positive + false_negative)\n",
    "np.mean(iou)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 EPOCH\n",
      "20 EPOCH\n",
      "30 EPOCH\n",
      "40 EPOCH\n",
      "50 EPOCH\n"
     ]
    }
   ],
   "source": [
    "y_pred = knn_clf.predict_labels(dists, k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcwAAABsCAYAAADnsILcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAR8klEQVR4nO2d23IbxRaGf8my5cRxnEPFrpAQoHgknoB73og34YoL3iMECFChAjngJD5I2hepNWotre5ecrJr146+76Y1o56enmmX2n+vQ48Wi4UAAACgzfh/3QEAAID/B5gwAQAAEjBhAgAAJGDCBAAASMCECQAAkGDS+vLbb79dSNLt27dVlovFQi9evJAk/fHHH5KkX375ZaX866+/JEm7u7uSpP39/eHzzs6OJGk0GoX3nc/nQ3l5eSlJ1fLi4mLlmvF4rPF4HLZfu589U3RsZa2tjJdx1LZd70vfflnaZ3s+e49WTiaTle/H4/HwXs7PzyVJb9++lSTduXNHkvTo0aOV8v79+7p169ZKX3766SdJ0g8//FB/gR+R0WiE6zYMLBaL//rfHX9zUFL7m0NhAgAAJGgqzLt370qSvvnmG0nS8fGxpPeq8c2bN5KkZ8+eSZJ+++03SdKvv/4qSfrzzz8lLRXNfD5fU2ulEpJi5emVnJWz2axa+vt49ebbKpWsKTJrz85bG6birNzZ2Vl7Dn8/f96Oo+fzfSzVtvUpasf6Up4fj8drKvTatWuSlmP74MEDSdLDhw8lSffu3dP169clLVUpAACgMAEAAFI0FabZskxZnpycSJKm06lOT09X6r57906S9Pr1a0kaFKgpG1NK5TmvLEvVVsMrPzs2W+ZsNhs+12ylkcKM2inLmsKcTCZrz1OzlZbf1+p6ZVkqanvmmqKNFKave+PGDUnS0dGRpKXStDE+Pj7WwcGBJOns7EzS0nYNALDNoDABAAASNBWmqRvzbp1Op8OxV4NeKXlbYFnH2+Cs/ZbCtPa8za9UruX58t5eiUUqrubtanVqqnexWKz1LapTfj8ajVa8eq9Kz+4Z4b/zKn93d3cYj1Y7AADbRnPCfPnypSQNy6///vuvpPc/qrb0+urVq5W6VlrYSbl065cz9/b2JC1/mH3YSTkhWB1bljSHFFtCtePz8/PhXM05xignGD/B+6Vfa8NP6hmnn2jC9P9o1Jx/Sqef2sTsj8u2ffv2PPbPj42flQcHB0Mf7J3+/fffAgDYdliSBQAASNBUmL///rukZejIcNFkMigSq2OlJTKw0px/xuPxoM4stGF/f1/SUmmawjSlVi6VeqccUz/mmFIe+1CQnvIrl1W90vRLpy3Hmiw7OzvdvkV9zIbLlPi6Nm72XDYWNgaLxWJwDLL3aCFDnzqt9wcAgMIEAABI0FSYT548kbRMRmBJCEaj0aBUTEn+/PPP4TUWbjKZTIZwBVM1ZkczdWMKtLTn2X/9pnYiRVkeRzZM77ATKbNWOEfZhm9zEwVSKkR/vVeprXZriRE8s9lsxelKWqp6e0elmpfej9fh4aGk5Tt//Phx67EAALYCFCYAAECCpsK0ROrPnz+XtBrUb56TT58+XalrpSlPU4CHh4drCcC9svRKqfQM9V6xkbK0771nay/BuZRPvh61WQtJ8W2UbdXS5dWSspd9raldr6Tn8/nwvsyWbHZie19eSb9582ZIVGB9stUDAIBtBoUJAACQoKkwfYxeqUZ8/OU///wjaalG7XtTKRcXF2vJvL0SNMr71BSmV5Y+blLqe8lmtvvKeKZGidLL0rcV9a1m04zS6dW29fKKs0ynZ6reVgasb3at2Tan0+laO7ZV26cOHrEA0AKFCQAAkKCpMO0/bsvqY8xms+GcZfIxm6JdY1tEmSra29sbPvtsOmW7Uqwwa6rNe3mOx+PuZsxGa5utSBXWqClZ3zdjsVisKcha2dpAureNmKlHaT2rkvc6Nhvn6enpUPcqnsAAAJ8qKEwAAIAETYVpdi1TIaZY5vP5YDv0Gy+byjGFaUyn0zXF4j00jVJp1hReK5F7z+O01laLnids+bmnUsvnyihLK/252rZoUVymxbz69+Vtv2dnZ0PsrNWxuFkAgG2mOWG2gutrP9o+vV3Zll+i9Mur0XHNcaa260aZqm6TpcSrLjuWk19rgvRlz5moNcnX/hFoJYH3ffJ1y7Zru8kAAGwzLMkCAAAkaCpMT6lYakuHPnjeJzEvP9e21CqvaYVmlJSKKbv0mnHoqbXhFWKEbz+qW2un1bfs0m+UIMGvCEShKq2lXQCAbYVfRAAAgARNhemdcYxSxfW2pSrDGEyxeNVoDifeAWU+n6eVZcs+6Pvkj6P0dr3nis71FGvUR/+eWo5KNTunvbfompqt1x+Xdktv3/RtAABsIyhMAACABFdSmCU+NMQHxpfp77IKs2wra2csbYsfEmjfu7alWjPesVb2Uu9tojCt9O++TFzglWwr+btvN/N3AADwqYPCBAAASHAlhdmyLXplGSUh8Kq05SXbS1wQKbSPkcptE6W5ibL09Wvbh0XfZ22lFjdZ2h5r8av+eVCYAAAxKEwAAIAETYVpKjFSNrVk6N6OVqaAq3mGthRmjyirTVZhZmIoN0mnt0nGn54nr/8+2hqs5zVbnqslsY+eobYSAACwzaAwAQAAEjQVZk2VRDbMmpdsGScZqaXyuKZao7r+OONNehUy1/a2Bmspz95WYC288mupx002t+7l7QUA2EZQmAAAAAlSCtOrx/l8vmZvrNklS+XUs/FF9LbVainMbN7XSPFtwqbKsrxflKko6nvtnFRXj1HGn15ZbtrtrwUA2GaaE2YrDMTvg+nrREun/40kBB9j8v1Y1JYyW33MJFD3dT01J6Dyndf+wTGif4Y2ebcAAJ86LMkCAAAkSCnMKOyjtly7idLLJjyP8HWjZcPeUuzHorf06t9NK7nCJmou81y1vviyTDhxcXEhabnlF0uyAAAoTAAAgBSpDaR9urvLy8u11Hc1hRkF62eJtt3qMZ/P01uP+XuV32VVaRSOUbNdRvftbb+VCZtp9a2nLP1KweXlJTZMAIAAFCYAAECClA3T1KTZtubzeTXJektdZVPH1Y6j7zIq7mMqpE08XVuB/2XKwKhuFBqTVc5lH2tezN52aWN7fn6u8/NzSUsbJgAAoDABAABSbGTDNBUym80GFVKek9oKs5XwOzqO7IOeKNWbt/tdxfs2q3Yj22zPhll6yXplmbFh9t5Xy67a8o6V3o/n2dmZpOUG1NgwAQBQmAAAACmuZMO8uLgIPWfLayL7XU09tcpsarzofr1tsMrj3n1a569iwzS8sozS2tlxLY1erW+lqu/ZMsuxLW3VrfsAAGwTKEwAAIAEG+WSjbwvs+Um7bcUZs9eGNXJsGkcZhQXWYvDjPrlvWRr3rGR7TJbRn3xfYxWCvy2YShMAICk00/04+33cYwSf0vxEqnRc46JQlF8+5tQW5otnXCyDkItJ5zapFfWq92ndf9NJ8zWkqxh41cbRwAAeA9LsgAAAAmaCtPUhoUX7O7uDuftswW3e8XpHXyknJNKry/+OHNNtozuk+1PSS80JFK0PcekVt1NlrT9ONn4lWM8nU67zwgAsG2gMAEAABKkFKapSSvH4/GgSEyheKXpy/F4vBYW4e8TXdNTpV5pZlRVpOauqqZaSQj8fX3fyzqZchOlbMfeqcivANi42dju7e2tjLPvLwDAtoLCBAAASHAlhTkajdYUZs2GWaqfXqJxr8TKrbqMmrIs+xylootK/5zRdzV11fKq/Ri20uha/26jd+zb9krSh7NYWY7x3t5e2D4AwDaDwgQAAEiwkZeslaXCrCnNSHHW4hK98mzZMFvKMjpffpc9brUbqbmsTTZSmNmytAHX3nFLYdaUpbdB7+7urqwklHUBALYZfgkBAAASpBSmKQxTk+Xnng2zLGtqp7UdV8+G6TPYZNRidFy7LmMH7fWxZe+s9bHlJZtVmlGffB0fhzmZTIbPrZhaAIBtg19CAACABFeyYUrrtq9NbJi+fV83k72npigz+VprbWbaj44j26tUT6ieadefj2JFW/ZO37eel2ykMLFhAgAsSU2YflJcLBbD597yYDRh1hKpR0uZvaD5TBhIb4m0NRm1lotrdf351j8E2UkwUycKN/Hv1o9b5ARUqwMAsM3wSwgAAJCgqTC9eixVSDaIvqUwa+qtVGI9BRkpwU3DPCKF6d+Bbyuil0TeL9G22v0QpRk9V21br2icUJgAAOvwSwgAAJAgpTC94pjP511lmVEp3qbX2iS6Fk6yiUNNy/aXce5pHZd9+pBk5Rm1mH2+0kkr037tPihMAAAUJgAAQIqmwjSVaGS8SmvJCcq2ZrPZWntlXSNSmL5dT2n37NlIS3W8qQ2xPK6pX090Ta1OpNxb/S9Lr+79ZylWnzU2qQsA8KmCwgQAAEiwkcLcBK9yMm1FdkGvLP3WYF5llRtI9xRmxoaZ8Vb1faqRsW22VHcrxrV3bS1tXu39lqAwAQBQmAAAACk2UpilCqmpJW8r+5BYvkgt1jaQLj1Ue31r2QevEg/ZU46bJIbPKNtWlp7yOPrO7MfRZt3W16y9GABgm+CXEAAAIEEql2yk8nzMYc22eBWFeZU4xtI7t6cwI1tmL86ylvEnygu7SRzjVZRty5O2PF+uEPj2bZxMcdrxfD5fi3FFYQIAoDABAABSpNwfS/VhpT9nSsUrzbL0NrbMFlo1m6X/PrNLSa+tTN0o1jLjaVq7z1WUZU/9llhfLi8vV0ojGicfJ9vz/gUA2AaaE+bFxYWk9UlwNptVJ8qzszNJ8aTkJxJ/ra83n8+rS76140xYSXQ+k6AgouWY1HJYqk2Mre3Eatf4JdTyndg7trG08bF/XqJ/fPw5uxYAYJthSRYAACBBU2GaGplOp5Kk/f19SdLu7q7evn07fJZWN5eWNHxfOpVESdzLMnJA6SUuaIWVZBMXRE4/Rm8pOOpb7XwZprFJYgSjlnzAU74Tey82llbaWFpSAiun0+nwnY3XmzdvmvcDANgGUJgAAAAJmgrTlMqNGzckSUdHR5LeO46YUjk8PJQkXbt2TdJSlZjCNFU0nU6H73qJ2s0xZTQaranQmtIs7aHeSaV2v1b6uSwtpdnqayvEJTqOEhdkUuXZezk/P5ckvXv3TtJy3ExN2vHNmzd18+ZNSVobLwCAbQaFCQAAkKCpME1p3Lp1S9KqwjQV+Pr1a0nS7du3JUmnp6crbVi9nZ2dwU7mU+6Z8rLv9/b2Vs5LS5VWC18p7Z/22dOyG/aSktfaalGrs1gsugnUPVG6u1pSiNJGbP03JWmrBcfHx5Kku3fvSlqO39HR0fDZxsmOAQC2GRQmAABAgqbCfPTokaSlwrRSWnrHejuanTfP2tKWacrRlGTNM7Sl7rwt0yvOMo4wG1NZetb2EjL4tHOj0ahqm81scm1k7IU122stlrNcCbCxM6X52WefSZIePny4Up6cnKyMsyR99dVX1T4BAGwLKEwAAIAETYX55ZdfSpLu3bsnaWn3mkwmg+3r4OBA0no8pqlIs3GWysnqeFUVJXtvfVeWpQ0zs51WSelZa2UtlZyPJZ1MJtXnaSlMr1h7cZhlH3t1y2fxdcyG+eDBA0nSF198IWlVYV6/fn3l2b/++uvwPgAA2wQKEwAAIEFTYZoKOTk5kSTdv39f0ns1ad6wpiRN4ZkqMZXz4sWLob1MNpsatc2saxl/ovvUsvaUGXH8c/g8qt7TN/L+tfv4Ppd999ugtZSl72Pt+SIF6tWuxVuaDdPG2BTm8fHxoDAt1vbzzz8P+wYAsE00J0xz/rBlPPux3dvbW0tQYHVfvnwpSXr16tVKW62JrJVarrdkGSUH2MTZR4qTydtEaQH/hi09WzmZTFaWZ6Wc049/1szyai1cxlPe3/elFipk5c2bN4ckFOakdefOndR9AQA+ZViSBQAASJDaD9MvT47H47VE6V4x9VRWeU3NoSdSixl6CrO1D6Y/9onio/74Z+z1eTQadffMzOyp2WrfSr887N91tBRdW44GANhmUJgAAAAJmgrzxx9/lLRUIWb3ms1mev78uSTp6dOnkqTHjx9Lkp48eSJJevbsmaRlUvb9/f21hAW1UI4ozV1m4+jyfElvC61MsoCWzbFnI41ss70tx6K+RyEtUVkqXnunZmu25OsWKmRhJWV4iYUMWTvff/+9JOm7774LnxMAYBtAYQIAACQYfYitDAAAYFtAYQIAACRgwgQAAEjAhAkAAJCACRMAACABEyYAAEACJkwAAIAE/wFPueWDzBmGHgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 576x864 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "some_image = 9\n",
    "\n",
    "plt.figure(figsize=(8,12))\n",
    "plt.subplot(1,3,1)\n",
    "plot_image(np.reshape(X_test[some_image],(51,71)))\n",
    "plt.subplot(1,3,2)\n",
    "plot_image(np.reshape(y_test[some_image],(51,71)))\n",
    "plt.subplot(1,3,3)\n",
    "plot_image(np.reshape(y_pred[some_image],(51,71)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5245048696511984"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_positive = np.sum(np.where(y_pred + y_test == 2, 1, 0), axis = 1)\n",
    "false_positive = np.sum(np.where(y_pred - y_test == 1, 1, 0), axis = 1)\n",
    "false_negative = np.sum(np.where(y_pred - y_test == -1, 1, 0), axis = 1)\n",
    "\n",
    "iou = true_positive / (false_positive + true_positive + false_negative)\n",
    "np.mean(iou)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-Tuning the Model\n",
    "To get the maximun score of the model we will need to choose the correct hyperparameters. In this classifier the most important hyperparameter is the number of neighbors with whom the model will compare the labels (k).\n",
    "\n",
    "For this case we will use cross-validation ussing 5 different folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_folds = 5\n",
    "k_choices = [1, 3, 5, 8, 10, 12, 15, 20, 50, 100]\n",
    "\n",
    "X_train_folds = np.array_split(X_train, num_folds)\n",
    "y_train_folds = np.array_split(y_train, num_folds)\n",
    "\n",
    "k_to_accuracies = {}\n",
    "\n",
    "num_test_fold = X_train_folds[0].shape[0]\n",
    "for k_value in k_choices:\n",
    "    k_list = []\n",
    "    for fold_index in range(num_folds): \n",
    "        # Train classifier\n",
    "        val_set = X_train_folds.pop(fold_index)\n",
    "        val_lbl = y_train_folds.pop(fold_index)\n",
    "        X_fold = np.vstack(X_train_folds)\n",
    "        y_fold = np.hstack(y_train_folds)\n",
    "        classifier = KNearestNeighbor()\n",
    "        classifier.train(X_fold, y_fold)\n",
    "        \n",
    "        # Calculate the accuracy\n",
    "        y_test_pred = classifier.predict(val_set, k=k_value)\n",
    "        true_positive = np.sum(np.where(y_pred + y_test == 2, 1, 0), axis = 1)\n",
    "        false_positive = np.sum(np.where(y_pred - y_test == 1, 1, 0), axis = 1)\n",
    "        false_negative = np.sum(np.where(y_pred - y_test == -1, 1, 0), axis = 1)\n",
    "\n",
    "        iou = np.mean(true_positive / (false_positive + true_positive + false_negative))   \n",
    "        k_list.append(iou)\n",
    "        \n",
    "        # Restore the varables\n",
    "        X_train_folds = np.array_split(X_train, num_folds)\n",
    "        y_train_folds = np.array_split(y_train, num_folds)\n",
    "        \n",
    "    k_to_accuracies[k_value] = k_list\n",
    "    \n",
    "for k in sorted(k_to_accuracies):\n",
    "    for accuracy in k_to_accuracies[k]:\n",
    "        print('k = %d, accuracy = %f' % (k, accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = pd.DataFrame(\n",
    "    [(true_positive[i], false_positive[i], false_negative[i])\n",
    "     for i in range(len(false_negative))],\n",
    "    columns=['true_positive', 'false_positive', 'false_negative'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df['precision'] = results_df['true_positive']/(results_df['true_positive']+results_df['false_positive'])\n",
    "results_df['recall'] = results_df['true_positive']/(results_df['true_positive']+results_df['false_negative'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7587402204996166"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['precision'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6076917865113795"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['recall'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true_positive</th>\n",
       "      <th>false_positive</th>\n",
       "      <th>false_negative</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>115</td>\n",
       "      <td>3</td>\n",
       "      <td>121</td>\n",
       "      <td>0.974576</td>\n",
       "      <td>0.487288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>149</td>\n",
       "      <td>9</td>\n",
       "      <td>28</td>\n",
       "      <td>0.943038</td>\n",
       "      <td>0.841808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.972973</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>42</td>\n",
       "      <td>5</td>\n",
       "      <td>22</td>\n",
       "      <td>0.893617</td>\n",
       "      <td>0.656250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>23</td>\n",
       "      <td>22</td>\n",
       "      <td>2</td>\n",
       "      <td>0.511111</td>\n",
       "      <td>0.920000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>0.416667</td>\n",
       "      <td>0.555556</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>79</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>0.766990</td>\n",
       "      <td>0.724771</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>139</td>\n",
       "      <td>6</td>\n",
       "      <td>54</td>\n",
       "      <td>0.958621</td>\n",
       "      <td>0.720207</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>138</td>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>0.992806</td>\n",
       "      <td>0.846626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>58</td>\n",
       "      <td>10</td>\n",
       "      <td>11</td>\n",
       "      <td>0.852941</td>\n",
       "      <td>0.840580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>0.400000</td>\n",
       "      <td>0.222222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>136</td>\n",
       "      <td>0</td>\n",
       "      <td>58</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.701031</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>51</td>\n",
       "      <td>33</td>\n",
       "      <td>18</td>\n",
       "      <td>0.607143</td>\n",
       "      <td>0.739130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>121</td>\n",
       "      <td>6</td>\n",
       "      <td>19</td>\n",
       "      <td>0.952756</td>\n",
       "      <td>0.864286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>91</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.957895</td>\n",
       "      <td>0.900990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>106</td>\n",
       "      <td>6</td>\n",
       "      <td>44</td>\n",
       "      <td>0.946429</td>\n",
       "      <td>0.706667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>25</td>\n",
       "      <td>60</td>\n",
       "      <td>10</td>\n",
       "      <td>0.294118</td>\n",
       "      <td>0.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>95</td>\n",
       "      <td>13</td>\n",
       "      <td>101</td>\n",
       "      <td>0.879630</td>\n",
       "      <td>0.484694</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>118</td>\n",
       "      <td>5</td>\n",
       "      <td>59</td>\n",
       "      <td>0.959350</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>23</td>\n",
       "      <td>57</td>\n",
       "      <td>27</td>\n",
       "      <td>0.287500</td>\n",
       "      <td>0.460000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>58</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0.840580</td>\n",
       "      <td>0.840580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>168</td>\n",
       "      <td>10</td>\n",
       "      <td>25</td>\n",
       "      <td>0.943820</td>\n",
       "      <td>0.870466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>147</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>0.954545</td>\n",
       "      <td>0.830508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>162</td>\n",
       "      <td>3</td>\n",
       "      <td>31</td>\n",
       "      <td>0.981818</td>\n",
       "      <td>0.839378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>111</td>\n",
       "      <td>6</td>\n",
       "      <td>21</td>\n",
       "      <td>0.948718</td>\n",
       "      <td>0.840909</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>103</td>\n",
       "      <td>6</td>\n",
       "      <td>60</td>\n",
       "      <td>0.944954</td>\n",
       "      <td>0.631902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>97</td>\n",
       "      <td>25</td>\n",
       "      <td>16</td>\n",
       "      <td>0.795082</td>\n",
       "      <td>0.858407</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>140</td>\n",
       "      <td>2</td>\n",
       "      <td>56</td>\n",
       "      <td>0.985915</td>\n",
       "      <td>0.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>89</td>\n",
       "      <td>7</td>\n",
       "      <td>67</td>\n",
       "      <td>0.927083</td>\n",
       "      <td>0.570513</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.230769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>141</td>\n",
       "      <td>3</td>\n",
       "      <td>66</td>\n",
       "      <td>0.979167</td>\n",
       "      <td>0.681159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>158</td>\n",
       "      <td>8</td>\n",
       "      <td>19</td>\n",
       "      <td>0.951807</td>\n",
       "      <td>0.892655</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>175</td>\n",
       "      <td>5</td>\n",
       "      <td>100</td>\n",
       "      <td>0.972222</td>\n",
       "      <td>0.636364</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>114</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "      <td>0.950000</td>\n",
       "      <td>0.820144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>57</td>\n",
       "      <td>19</td>\n",
       "      <td>18</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>0.760000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>149</td>\n",
       "      <td>7</td>\n",
       "      <td>126</td>\n",
       "      <td>0.955128</td>\n",
       "      <td>0.541818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>66</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>0.891892</td>\n",
       "      <td>0.956522</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>192</td>\n",
       "      <td>1</td>\n",
       "      <td>61</td>\n",
       "      <td>0.994819</td>\n",
       "      <td>0.758893</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>68</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>0.723404</td>\n",
       "      <td>0.985507</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>14</td>\n",
       "      <td>17</td>\n",
       "      <td>4</td>\n",
       "      <td>0.451613</td>\n",
       "      <td>0.777778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>13</td>\n",
       "      <td>5</td>\n",
       "      <td>12</td>\n",
       "      <td>0.722222</td>\n",
       "      <td>0.520000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    true_positive  false_positive  false_negative  precision    recall\n",
       "0             115               3             121   0.974576  0.487288\n",
       "1             149               9              28   0.943038  0.841808\n",
       "2              36              12               1   0.750000  0.972973\n",
       "3              42               5              22   0.893617  0.656250\n",
       "4              23              22               2   0.511111  0.920000\n",
       "5               5               7               4   0.416667  0.555556\n",
       "6              79              24              30   0.766990  0.724771\n",
       "7             139               6              54   0.958621  0.720207\n",
       "8               4               1               1   0.800000  0.800000\n",
       "9               0               0               9        NaN  0.000000\n",
       "10            138               1              25   0.992806  0.846626\n",
       "11             58              10              11   0.852941  0.840580\n",
       "12              2               3               7   0.400000  0.222222\n",
       "13            136               0              58   1.000000  0.701031\n",
       "14              0               0               9        NaN  0.000000\n",
       "15             51              33              18   0.607143  0.739130\n",
       "16            121               6              19   0.952756  0.864286\n",
       "17             91               4              10   0.957895  0.900990\n",
       "18              0               1              13   0.000000  0.000000\n",
       "19            106               6              44   0.946429  0.706667\n",
       "20             25              60              10   0.294118  0.714286\n",
       "21             95              13             101   0.879630  0.484694\n",
       "22            118               5              59   0.959350  0.666667\n",
       "23             23              57              27   0.287500  0.460000\n",
       "24              0               0               5        NaN  0.000000\n",
       "25             58              11              11   0.840580  0.840580\n",
       "26            168              10              25   0.943820  0.870466\n",
       "27              0               0               5        NaN  0.000000\n",
       "28            147               7              30   0.954545  0.830508\n",
       "29            162               3              31   0.981818  0.839378\n",
       "30              0               0               5        NaN  0.000000\n",
       "31            111               6              21   0.948718  0.840909\n",
       "32            103               6              60   0.944954  0.631902\n",
       "33             97              25              16   0.795082  0.858407\n",
       "34            140               2              56   0.985915  0.714286\n",
       "35             89               7              67   0.927083  0.570513\n",
       "36              3               6              10   0.333333  0.230769\n",
       "37            141               3              66   0.979167  0.681159\n",
       "38            158               8              19   0.951807  0.892655\n",
       "39            175               5             100   0.972222  0.636364\n",
       "40            114               6              25   0.950000  0.820144\n",
       "41              0               3               5   0.000000  0.000000\n",
       "42             57              19              18   0.750000  0.760000\n",
       "43            149               7             126   0.955128  0.541818\n",
       "44             66               8               3   0.891892  0.956522\n",
       "45            192               1              61   0.994819  0.758893\n",
       "46              0               4               5   0.000000  0.000000\n",
       "47             68              26               1   0.723404  0.985507\n",
       "48             14              17               4   0.451613  0.777778\n",
       "49             13               5              12   0.722222  0.520000"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "false negative:  [121  28   1  22   2   4  30  54   1   9  25  11   7  58   9  18  19  10\n",
      "  13  44  10 101  59  27   5  11  25   5  30  31   5  21  60  16  56  67\n",
      "  10  66  19 100  25   5  18 126   3  61   5   1   4  12]\n",
      "false positive:  [ 3  9 12  5 22  7 24  6  1  0  1 10  3  0  0 33  6  4  1  6 60 13  5 57\n",
      "  0 11 10  0  7  3  0  6  6 25  2  7  6  3  8  5  6  3 19  7  8  1  4 26\n",
      " 17  5]\n",
      "true positive: [115 149  36  42  23   5  79 139   4   0 138  58   2 136   0  51 121  91\n",
      "   0 106  25  95 118  23   0  58 168   0 147 162   0 111 103  97 140  89\n",
      "   3 141 158 175 114   0  57 149  66 192   0  68  14  13]\n"
     ]
    }
   ],
   "source": [
    "print('false negative: ' , false_negative)\n",
    "print('false positive: ', false_positive)\n",
    "print('true positive:', true_positive)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}