Lecture4-Performance_evaluation_code_only.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Before jumping to the code implementation\n",
    "- Gif를 사용하지 않고 해당 ipynb파일만 다운받고 싶으면 wget이나 curl을 사용하여도 좋습니다. \n",
    "    - wget https://raw.githubusercontent.com/COMBINE-SKKU/bio_data_mining/master/Lecture4-Performance_evaluation_code_only.ipynb\n",
    "    - curl https://raw.githubusercontent.com/COMBINE-SKKU/bio_data_mining/master/Lecture4-Performance_evaluation_code_only.ipynb --output Lecture4-Performance_evaluation_code_only.ipynb\n",
    "<br><br><br>    \n",
    "- 동영상과 같이 git clone을 사용하고 싶은 학생이 주의하여야 할 점은 git은 원칙적으로 이미 존재하는 디렉토리에는 clone을 하지 않으므로 디렉토리 (예를 들자면 /Documents/BioDataMining)를 지워주고 실행하여야 합니다. \n",
    "- Open the terminal and make a directory dedicated for this class code implementation (e.g., mkdir ~/Documents/BioDataMining)\n",
    "- Install Git (https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)\n",
    "- Go to the directory for this class, and type: git clone https://github.com/COMBINE-SKKU/bio_data_mining.git\n",
    "- If you are a newbie for programming a python and jupyter notebook, please install anaconda \n",
    "    - Window: https://problemsolvingwithpython.com/01-Orientation/01.03-Installing-Anaconda-on-Windows/\n",
    "    - Mac: https://problemsolvingwithpython.com/01-Orientation/01.04-Installing-Anaconda-on-MacOS/\n",
    "    - Linux: https://problemsolvingwithpython.com/01-Orientation/01.05-Installing-Anaconda-on-Linux/\n",
    "- And learn how to open a jupyter notebook (https://www.youtube.com/watch?v=OJMILWh6ARY)\n",
    "- Run the following codes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# MNIST Digit Classification\n",
    "***\n",
    "- Examples from TextBook \"O'Reilly book Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow\" p85-99"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src='https://raw.githubusercontent.com/COMBINE-SKKU/combine-skku/master/class/week4/Fig4-12.png'></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'frame', 'feature_names', 'target_names', 'DESCR', 'details', 'categories', 'url'])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_openml\n",
    "mnist = fetch_openml('mnist_784', version=1)\n",
    "mnist.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 784)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X, y = mnist[\"data\"], mnist[\"target\"]\n",
    "X.shape "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000,)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAGaElEQVR4nO3dPUiWfR/G8dveSyprs2gOXHqhcAh6hZqsNRqiJoPKRYnAoTGorWyLpqhFcmgpEmqIIByKXiAHIaKhFrGghiJ81ucBr991Z/Z4XPr5jB6cXSfVtxP6c2rb9PT0P0CeJfN9A8DMxAmhxAmhxAmhxAmhljXZ/Vcu/H1tM33RkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCLZvvG+B//fr1q9y/fPnyVz9/aGio4fb9+/fy2vHx8XK/ceNGuQ8MDDTc7t69W167atWqcr948WK5X7p0qdzngycnhBInhBInhBInhBInhBInhBInhHLOOYMPHz6U+48fP8r92bNn5f706dOG29TUVHnt8PBwuc+nLVu2lPv58+fLfWRkpOG2du3a8tpt27aV+759+8o9kScnhBInhBInhBInhBInhBInhGqbnp6u9nJsVS9evCj3gwcPlvvffm0r1dKlS8v91q1b5d7e3j7rz960aVO5b9iwody3bt0668/+P2ib6YuenBBKnBBKnBBKnBBKnBBKnBBKnBBqUZ5zTk5Olnt3d3e5T0xMzOXtzKlm997sPPDx48cNtxUrVpTXLtbz3zngnBNaiTghlDghlDghlDghlDghlDgh1KL81pgbN24s96tXr5b7/fv3y33Hjh3l3tfXV+6V7du3l/vo6Gi5N3un8s2bNw23a9euldcytzw5IZQ4IZQ4IZQ4IZQ4IZQ4IZQ4IdSifJ/zT339+rXcm/24ut7e3obbzZs3y2tv375d7idOnCh3InmfE1qJOCGUOCGUOCGUOCGUOCGUOCHUonyf80+tW7fuj65fv379rK9tdg56/Pjxcl+yxL/HrcKfFIQSJ4QSJ4QSJ4QSJ4QSJ4Tyytg8+PbtW8Otp6envPbJkyfl/uDBg3I/fPhwuTMvvDIGrUScEEqcEEqcEEqcEEqcEEqcEMo5Z5iJiYly37lzZ7l3dHSU+4EDB8p9165dDbezZ8+W17a1zXhcR3POOaGViBNCiRNCiRNCiRNCiRNCiRNCOedsMSMjI+V++vTpcm/24wsrly9fLveTJ0+We2dn56w/e4FzzgmtRJwQSpwQSpwQSpwQSpwQSpwQyjnnAvP69ety7+/vL/fR0dFZf/aZM2fKfXBwsNw3b948689ucc45oZWIE0KJE0KJE0KJE0KJE0KJE0I551xkpqamyv3+/fsNt1OnTpXXNvm79M+hQ4fK/dGjR+W+gDnnhFYiTgglTgglTgglTgglTgjlKIV/beXKleX+8+fPcl++fHm5P3z4sOG2f//+8toW5ygFWok4IZQ4IZQ4IZQ4IZQ4IZQ4IdSy+b4B5tarV6/KfXh4uNzHxsYabs3OMZvp6uoq97179/7Rr7/QeHJCKHFCKHFCKHFCKHFCKHFCKHFCKOecYcbHx8v9+vXr5X7v3r1y//Tp02/f07+1bFn916mzs7PclyzxrPhvfjcglDghlDghlDghlDghlDghlDghlHPOv6DZWeKdO3cabkNDQ+W179+/n80tzYndu3eX++DgYLkfPXp0Lm9nwfPkhFDihFDihFDihFDihFDihFCOUmbw+fPncn/79m25nzt3rtzfvXv32/c0V7q7u8v9woULDbdjx46V13rla2753YRQ4oRQ4oRQ4oRQ4oRQ4oRQ4oRQC/acc3JysuHW29tbXvvy5ctyn5iYmNU9zYU9e/aUe39/f7kfOXKk3FevXv3b98Tf4ckJocQJocQJocQJocQJocQJocQJoWLPOZ8/f17uV65cKfexsbGG28ePH2d1T3NlzZo1Dbe+vr7y2mbffrK9vX1W90QeT04IJU4IJU4IJU4IJU4IJU4IJU4IFXvOOTIy8kf7n+jq6ir3np6ecl+6dGm5DwwMNNw6OjrKa1k8PDkhlDghlDghlDghlDghlDghlDghVNv09HS1lyMwJ9pm+qInJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Rq9iMAZ/yWfcDf58kJocQJocQJocQJocQJocQJof4DO14Dhyk10VwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "some_digit = X.iloc[0]\n",
    "some_digit_image = some_digit.values.reshape(28, 28)\n",
    "\n",
    "plt.imshow(some_digit_image, cmap=\"binary\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()\n",
    "\n",
    "y = y.astype(np.uint8)\n",
    "y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Digit classification using a Random Forest algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = X[:60000], X[60000:], y[:60000], y[60000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y_train_5 = (y_train == 5) # True for all 5s, False for all other digits\n",
    "y_test_5 = (y_test == 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "forest_clf = RandomForestClassifier(random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_split.py:296: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.\n",
      "  FutureWarning\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.98885\n",
      "0.9863\n",
      "0.98635\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.base import clone\n",
    "\n",
    "skfolds = StratifiedKFold(n_splits=3, random_state=42, shuffle=True)\n",
    "\n",
    "for train_index, test_index in skfolds.split(X_train, y_train_5):\n",
    "    clone_clf = clone(forest_clf)\n",
    "    \n",
    "    X_train_folds = X_train.iloc[train_index]\n",
    "    y_train_folds = y_train_5.iloc[train_index]\n",
    "    X_test_fold = X_train.iloc[test_index]\n",
    "    y_test_fold = y_train_5.iloc[test_index]\n",
    "    clone_clf.fit(X_train_folds, y_train_folds)\n",
    "    y_pred = clone_clf.predict(X_test_fold)\n",
    "    n_correct = sum(y_pred == y_test_fold)\n",
    "    \n",
    "    print(n_correct / len(y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.98885, 0.9863 , 0.98635])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "cross_val_score(forest_clf, X_train, y_train_5, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Just like the cross_val_score() function, cross_val_predict() performs K-fold cross-validation, \n",
    "# but instead of returning the evaluation scores, it returns the predictions made on each test fold.\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "y_train_pred = cross_val_predict(forest_clf, X_train, y_train_5, cv=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center><img src='https://raw.githubusercontent.com/COMBINE-SKKU/combine-skku/master/class/week4/Fig4-13.png'></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4696,    45],\n",
       "       [  725, 54534]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_train_5, y_train_pred).transpose()\n",
    "cm[:,[1,0]] = cm[:,[0,1]]\n",
    "cm[[1,0],:] = cm[[0,1],:]\n",
    "\n",
    "cm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "manual calcualtion of precision: 0.9905083315756169\n",
      "precision_score: 0.9905083315756169\n",
      "manual calcualtion of recall (sensitivity): 0.8662608374838591\n",
      "recall_score: 0.8662608374838591\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "\n",
    "print('manual calcualtion of precision: ', end='')\n",
    "print(cm[0, 0] / ( cm[0, 0] + cm[0, 1] ))\n",
    "\n",
    "print('precision_score: ', end='')\n",
    "print(precision_score(y_train_5, y_train_pred))\n",
    "\n",
    "print('manual calcualtion of recall (sensitivity): ', end='')\n",
    "print(cm[0, 0] / ( cm[0, 0] + cm[1, 0] ))\n",
    "\n",
    "print('recall_score: ', end='')\n",
    "print(recall_score(y_train_5, y_train_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "$F_{1}=\\frac{2}{\\frac{1}{\\text {precision}}+\\frac{1}{\\text {recall}}}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9242275142688446"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1_score(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Precision/Recall Trade-Off\n",
    "<br>\n",
    "<center><img src='https://raw.githubusercontent.com/COMBINE-SKKU/combine-skku/master/class/week4/Fig4-14.png'></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import cross_val_predict\n",
    "\n",
    "forest_clf = RandomForestClassifier(random_state=42)\n",
    "y_probas_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3, method=\"predict_proba\")\n",
    "\n",
    "y_scores_forest = y_probas_forest[:, 1] # score = proba of positive class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.89, 0.01, 0.04, ..., 0.98, 0.08, 0.06])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_scores_forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "threshold = 0.5\n",
    "y_some_digit_pred = (y_scores_forest[0] > threshold)\n",
    "print(y_some_digit_pred)\n",
    "\n",
    "threshold = 0.9\n",
    "y_some_digit_pred = (y_scores_forest[0] > threshold)\n",
    "print(y_some_digit_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Present precision-recall curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores_forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
    "    plt.plot(thresholds, precisions[:-1], \"b--\", label=\"Precision\", linewidth=2)\n",
    "    plt.plot(thresholds, recalls[:-1], \"g-\", label=\"Recall\", linewidth=2)\n",
    "    plt.legend(loc=\"center right\", fontsize=16)\n",
    "    plt.xlabel(\"Threshold\", fontsize=16)      \n",
    "    plt.grid(True)                         "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "cell_style": "split",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAELCAYAAADJF31HAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd3gU1frA8e/JpndIoySE0IQQqaEJYqhS/KFYAVHxqtjAAnIt14J47SL2gooFKaJeFBUBUQJK70gRCD0ESAiB9H5+f0wSQkjIBpJMdvf9PM8+W+bMzHtS3kzOnKK01gghhLB9TmYHIIQQonpIQhdCCDshCV0IIeyEJHQhhLATktCFEMJOOJt14sDAQN20adMLlsnIyMDLy6t2AqpDpN6OxVHrDY5b90up98aNG09qrYPK22ZaQm/atCkbNmy4YJnY2FhiYmJqJ6A6ROrtWBy13uC4db+UeiulDlW0TZpchBDCTkhCF0IIOyEJXQgh7IQkdCGEsBOS0IUQwk5UmtCVUjOUUolKqe0VbFdKqXeUUnFKqW1KqU7VH6YQQojKWHOF/gUw6ALbBwMtix5jgQ8vPSwhhBBVVWk/dK31CqVU0wsUuRb4Shvz8K5RSvkrpRpqrY9VU4zn+Pdv/2ZN/BosThYsyoKzkzOuFteSR+nPvVy88Hb1xtvVm4Y+DQn3C6eJXxOCvYLxdvVGKVUTIQpR47SGwkIoKID8/LOvi2fD9vEBi8V4feYMZGScWxZAKfDwgIYNjfd5ebBwIWRlQXb22WMpZezTuze0aGF8tm0b/PXX2e3FlAJnZ7j77rOfzZgBKSnGNq2N42dmGs/16wdS3B17/3547z2jXNljAjz2GISEGK+nToXlyyE313jk558t366dcRww6jFwYMVfx6efPrt93ryz+5Xl5ga//Xb2/W23waEKeoPfdBOMH2+83rr17OtifftCTXW9r46BRY2BI6Xexxd9dl5CV0qNxbiKJyQkhNjY2AseOD09/bwyy3ctZ13KuksKGMAJJ7ycvfBx9sHHxQdfZ1/8XPzwd/HHz8UPXxdfPCweeFo88bR44uPig5+LH97O3rg7udfoH4Py6u0I6mK9tYbcXCeys53QWuHvnwcYCS4hwYPMTEvRw7nkdU6OhejoU0REZAKwcaM/y5YFk5vrREGBorBQFSVkhZOT5rHHztZ7ypRIjhzxIC/Pibw847zZ2Raysy1ce+1RHnooDoC///bjoYc6Vhj3jBnriYjIAOCll1rz228Nyi3Xrt1p3n57CwBZWU5cd13vCo/55JO7GDjwBADffRfK+++3KLecq2sBLVr8WfJ+ypQuHDpU/qjIYcM8S+q+bZsf06ZVXKeoqHWEhxtf08WLK67TmTOniY016pSd7cSff1Zcp+XLd+HqeqLodSh//ll+ndzcCoiNPVunFSu6cPhw+XUKCTnC5ZfvA2DrVj/+/PPcOjk7JxIdXTM/69WR0MvLbOWumqG1ng5MB4iOjtaVjZQqbzTVZ5GfkZyZTIEuoKCwgPzCfPIK88jJzyG3IPeczzPzMknLTSM1J5WEtAQOnTnEodOHSM5KNrblp5GWnwbZVauws5Mzfm5+1POoR4hXCA19GtLAqwGNfRsT6htKY5/GBHgG4Ofmh7+7P75uvlX6AyCj56pfXh4cPw7JycaVYVaWcdV44IBxNenvb5R75hnjSi0tzXhkZJy9Uu3SBdYVXUtkZYGnZ8XnmzHj7FXY9u3wyy/ll3N2hsmTvUvq/dBDEBdXftnQ0FBiYkIB44pRKeMq3NkZnJyM105Fjajdu3ehTRvj9f/+B3//bZQrXQYgPNy/5NwFBfB//2dctbu5nS2ntbHf0KFt6NnTOGhhofEovT6O1sbD2dlyzvfxwQfhWKnLO09P4+HuDk5OqSVlW7Qwvk9lj1n8fM01XQkIMN57ecEDDxhxurkZ8RX/ivn5+dO+/dk6LV9e/tcT4LLL2hAS0qbk/LfcUn45JycLvXqdrdN33xk/A+Vp3DiM5s3DAOjQATqW+RsVFBTMiRPeNfKzXh0JPR4IK/U+FEiohuOWKyo4qlqOk1eQx5mcM5zOPk1yZjLJWckkZyaTmJFIUmYSyZnJZORlkJ6bzpmcM5zKOsWprFMkZyaTU5BjlM9KJu5UBb99pXi5eNGsXjOa1WtW0tzj4+qDv7s/QV5BBHsFE+wVTEPvhgR6BlZL/RxFWpqRAOPiICnJaF44cwbat4eRI40yGzYYybgiV14J3boZr5OSYM+ec7e7uRkJKKjU7Bnu7tCypfG5jw/4+p599vSkJJkC9OkDH31k7FM6qRYn49K+/tpIasWJysvLeHh6nlu2R4+zTSeVeecd41EZiwUWLLDumH37Gg9rTJpU8bbY2NMlr0NDYeJE647ZpcuFv6fFLBajqcgaoaHGw9rzW8Pfv/zznzhh3f5VVR0JfQEwTik1F+gGnKmp9vPq5GJxIdAzkEDPQFrUL//frIrk5OeUJPkT6Sc4nn6chLQEjqYd5WjaUeJT40nJSin5g5Gem87fiX/zd+LflR7bSTlRz6UezfY2I9Q3lDDfMFrUb0GL+i1oGdCSCP8ILE6Wi622TcrPN9pX//kHrrnm7JXj1VfDkiXl73PTTWcTelCQsU9ICAQGGsnRw8NIvk2bQr16Z/f7z3/gkUeM5OzjYyRTSzlfbqXOT/wVadvWeFSk9H/e7dpZd0whylNpQldKzQFigEClVDzwHOACoLX+CFgIDAHigEzgzpoKtq5wc3Yj2Nm4qm4d2LrS8ilZKRw4fYD9KftJzkwmPTedtNw0UrJSSMxMJCkjiRMZJziWdsy48s9NJjkhmfUJ6887lqvFlVYBrWgd2Jo2gW1oG9SWtsFtaV6vOR4uHjVR3RqltXHjyqMo9K1b4YUXjCvu06chNdW44i6+Gj10CJo0MV4HBBhXsc2bG1fLDRqAn5/xiCr1j1yTJsaNs/ISc1lhYZWXEaKusqaXy8hKtmvgwWqLyA7V86hHPY96dGpYeRf93IJc5v82nyZtmxCfGs+hM4eIOxXH3lN72ZO8h/jUeLYnbmd74vnDAhp6N6RZvWa0D2lPt9BudGvcjZYBLXFS5o8f27XLaMfdtQv27YMjR4xHfLzRhl3cuyAvD77//vz9mzQxmjFKt1t+/DHMnFl5oi5uaxbC3pk2fa4on6vFlRD3EHqE9Sh3e3puOnuS97AzaSc7k3ayI2kHO5N2cvD0QY6lH+NY+jFWHlnJBxs+AIz2+8igSNoGt+Xy4Mvp2KAjHRp0oJ5HvXKPf7EKC2H3bqNppDhZP//82Xbfu++GVavK3zc5+ezrtm2NduTLLjOuwIvbpl1dz9/Px6daqyCEzZOEbmO8Xb3p1LDTeVf7+YX5HE09yt5Te9mYsJG1R9eyJn4Nx9KPsT5h/XnNN+F+4bQLaUe7kHZ0bNCRK8KuoKFPwyrFcvAgTJ8Oa9fC+vXGDcrSbr/dSMxg3EALCIDISKN5pEkTo3kjLMxopy7m4QG33lqlMIQQRSSh2wlnJ2fC/cMJ9w+nf7P+JZ8nZyazI2kHOxJ3sPXEVjYf38y2E9uMLpxnDvHTnp9Kyjb1b0rPsJ7ENI0hpmkMzes1Rymjz/Tu3fDHH1C/PowYYZQ/cwZefvlsDGFhRrNIcaIu3a3vhRdq+isghJCEbucCPAPoHd6b3uFn+07lF+azN3kv205sY+uJrWxI2MCa+DUcPH2Qg6cPMuvvWQA08GqIf24UJ/9pzcl/WkNSJF3C2zJihNF/LzLS6BXSpQt07Xp2xKEQwhyS0B2Qs5MzbYLa0CaoDbdEGSMpCgoL2JG0gxWHVvD9xmWsSljO8YxjHOcYNP8Nmhv7rgdC3gimY4OOdGnUhS6jo+kR1oNgr2DzKiSEACShC4wbmfn5Ftq1MtrUgw+OI/bBQqi3j6iY3bTr+w/ODXey+5RxIzYxI5HF+xazeN/ikmO0CmhF7ya9aRfSrmQQVYv6LXCxuJhYMyEciyR0BxUfD3PmGD1Ktm0zbmB++aWxbdAgmPqGE9de25LmzVsC15Tsp7Xm0JlDbEzYyIaEDaw9upa1R9eyJ3kPe5LPHWlTz70ew1sP5+a2N9M3oq8kdyFqmCR0B1JQYMwp8t57sHTp2Xky/PzA2/tsOV9fmDCh/GMopWjq35Sm/k25IfIGwJhGYdOxTaw8spI9yXvYn7KfPcl7OHTmEDO2zGDGlhl4OHvQsWFHohtG0z20O/2b9SfIK6j8kwghLookdAfywgtG33AwRlj+3//B6NEweHD5/byt5WJxMQYyhXY75/OdSTv5dse3zNs5j51JO1l1ZBWrjqzinXXGxCKdGnZiQLMB9GrSix6h5fe7F0JYTxK6Hdu3z5hsqnt34/3o0TBrFtx/P9x557lzmNSEyKBInot5judiniM5M5lNxzaxPmE9sQdjWXFoBZuObWLTsU28uvJVAMI8wuh7ui/dGnejR1gP2oW0qxOjXIWwFZLQ7dChQ8aV+JdfGiMvt241hr+3aGFMKGXGuh4BngEMaD6AAc0H8NSVT5GZl8mKQytYfnA5q+JXse7oOo5kHeHLrV/y5VajMT/YK5hBLQYxuMVg+jfrLzNRClEJSeh2JCEBXnrJGL2Zl2fMX9Kpk7E6TPFozLqySJOniyeDWgxiUAtjdcPcglxm/DKD/Ab5rD26lhWHVnD4zGG+2voVX239CoAODTrQP6I/fSL60KtJL3zdfM2sghB1jiR0O3D6NEyebMy5nZNjJO1bbzU+a1G1mYFN42pxpbVva2K6xjCOcWit2XVyFwv3LmRR3CL+OvwXW45vYcvxLbyx+g0sykLnRp0ZFTWKsZ3H2uRMk0JUN0nodsDFxeiCmJMDN9xgJPKo6lkHxDRKKSKDIokMiuSxKx4jKy+LVUdW8fuB34k9GMv6hPWsO7qOdUfX8fJfLzPpikncG30v3q7elR9cCDslCd1GHT9uTHbl4mI0p3z2mTF/Svv2ZkdWMzxcPOjXrB/9mvUDjFknF8ct5uW/XmbjsY089ttjTFkxhVFRo7in8z1WTVUshL2RLgQ2RmtjDvCoqHMnvLrmGvtN5uXxdvXmhsgbWH/Pen4Z9QtXhF1Bak4qH238iM7TO9P2g7Y8/OvD/LT7J9Jy0io/oBB2QK7QbcjmzTB+PKxcabzfsMGYh9zJgf8sK6UY0nIIQ1oOYXvidj7Z+Akzt80smS/+nXXv4OniyciokdwXfR/RjaLNDlmIGuPAqcB2pKUZq8FHRxvJPCQEvvgCfv7ZsZN5WVHBUbw9+G2OP3ac5WOW80zvZ+ge2p3MvEw+2/wZXT7pQqePO/H6ytc5ePqg2eEKUe3kCr2OS0gwVqSPjze6IT76KDz3nDFcX5TP1eJaMmXwlD5T2H1yN9M3TufzLZ+z+fhmNh/fzL+X/pvODTvTv1l/YprG0KtJL7mhKmyeXN/VcQ0bGu3lXbrAxo3w5puSzKvqssDLmHr1VI5OOMr/bv4fI6JG4OXixcZjG3l15asMnjWYgNcCGPX9KFYcWoEunuRGCBsjV+h10OHDHhw+bCzTphTMnm1MmCULHV8aDxcPhrcZzvA2w0tGqsYejGXZwWWsP7qeOdvnMGf7HCKDIhl9+WhuansTLerbSEd+IZCEXufMnw/33deZdu1gxQpj0qyannPFEZUdqXro9CE+3fQpn27+lJ1JO3nqj6d46o+n6NSwE2Paj+G29rfh7+5vctRCXJg0udQhX34JN94IWVnOhIcbw/dF7Qj3D+eFvi9w+JHDLBixgNHtRuPj6sOmY5t4aNFDNJraiLt+vIt1R9dJk4yosySh1xHTp8OYMUY3xDvuOMjcuWfnXxG1x8Xiwv9d9n/MHD6TxEmJzLtxHv0i+pGVn8WMLTPo9mk3OnzcgffWvceZ7DNmhyvEOSSh1wHvvAP33mu8fvVVGDPmYJ2ZRMuRuTu7c1Pbm1h6+1J2j9vNxB4TCfQMZNuJbYz/dTwRb0fw8p8vk56bbnaoQgCS0E33xx/w8MPG67ffhn//29x4RPlaBbTijYFvEP9oPHNvmEuvJr1IyU7hqT+eotnbzXht5Wuczj5tdpjCwUlCN1mfPvDgg/Dhh8bgIVG3uTm7cUvULawYs4LfbvuNbo27kZSZxONLHyf0zVAe/vVhDqQcMDtM4aAkoZskN9d4VspY4/O++8yNR1SNUor+zfqz+q7VLBy1kH4R/cjIy+Cdde/Q8t2WjP1pLIfPHDY7TOFgJKGb4N13jdGfJ0+aHYm4VEopBrcczNLbl7Ll3i3c1u42NJpPNn1Cy3db8sAvD7D1+FazwxQOQhJ6LfvgA6NpZcsWWLrU7GhEdWrfoD1fDf+KnQ/sZGTUSPIK8vhww4d0+LgDnad35sP1H5KZl2l2mMKOSUKvRR9/bLSXg9HMMmKEufGImnFZ4GXMvmE2f9//N+O6jKOeez02HdvEAwsfIOLtCF756xVSc1LNDlPYIasSulJqkFJqt1IqTin1RDnbmyillimlNiultimlhlR/qLZt6VK4/37j9dtvn03swn61DW7Lu0PeJWFiAnNumEN0o2gSMxJ58vcnCX8rnA/Xf0ihLjQ7TGFHKk3oSikL8D4wGIgERiqlIssUexqYp7XuCIwAPqjuQG3ZiRNw223G4hTPPCO9WRyNu7M7I6JGsO7udSwZvYTe4b05nX2aBxY+QM8ZPdmXvs/sEIWdsOYKvSsQp7Xer7XOBeYC15Ypo4HiJdj9gITqC9H2zZhhLBl31VXG1LfCMSmlGNB8ALF3xPLdTd/RyKcRa+LXMHbjWCYsniDNMOKSWZPQGwNHSr2PL/qstMnAaKVUPLAQGF8t0dmJJ56Ajz6CWbNkxkRhJPYbIm9g14O7GN/V+FWZtmYal713GbO2zZK5YsRFU5X98CilbgKu1lrfXfT+NqCr1np8qTITio41VSnVA/gMiNL63AZCpdRYYCxASEhI57lz517w3Onp6Xh7O96iA1Jvx7IlcQufxH/CzrSdALT1bcuDzR+kjW8bkyOreY76Pb+Uevfp02ej1rr8tRS11hd8AD2AxaXePwk8WabMDiCs1Pv9QPCFjtu5c2ddmWXLllVapq46dUrr4cO13ru36vvacr0vhSPXu6CwQM/YNEMHvRakmYxmMnrU96P01uNbdWFhodkh1hhH/p5fLGCDriCvWtPksh5oqZSKUEq5Ytz0XFCmzGGgH4BSqg3gDiRV5a+OPdEa7ryzeG5zs6MRtsBJOXFnxzuJeyiOJ3o+gZvFjdl/z6b9R+2J+jCK/674L8fSjpkdpqjjKk3oWut8YBywGNiF0Ztlh1JqilJqWFGxicA9SqmtwBxgTNFfEof0zjvw44/GUnGffGJ2NMKW+Lr58nL/l/ln3D/c1/k+AjwC2Jm0k2eWPUPLd1vy6l+vkpOfY3aYoo6yqh+61nqh1rqV1rq51vrFos+e1VovKHq9U2vdU2vdXmvdQWu9pCaDrsu2bIFJk4zXM2ZARIS58Qjb1NS/KR9e8yHHJh7jl1G/MOyyYWTkZfDE708Q9WEUC/cuNDtEUQfJSNFqVFAA99xjrDT0wANw/fVmRyRsnYvFhSEth/DjiB9ZMnoJbQLbEHcqjqGzh3Ld3Os4ePqg2SGKOkQSejV6/33YsAHCwoyFKoSoTgOaD2DrfVuZOnAqPq4+/Lj7R9q834ZX/npFRpwKQBJ6tXN3N+ZpccCeWKIWuFhcmNBjArvH7ebWy28lOz+bJ39/kmvnXisLbAhJ6NXpoYfgwAEYNqzyskJcioY+Dfn6+q/59dZfqedej5/3/EyXT7qwPXG72aEJE0lCrwal+/M0aGBeHMLxDGoxiI1jN9KhQQfiTsXReXpnHv/tcVnA2kFJQr9EeXnGHC0ffwz5+WZHIxxRRL0IVv5rJXd1vIvcglxeW/UaLd9tyfSN02UaAQcjCf0Sffwx/PknTJ167pW6ELXJ08WTT4d9ytq719IzrCdJmUnc+/O9DJ09lBPpJ8wOT9QSSeiX4PRpmDzZeP3aa+DiYmo4QtC1cVf+vPNP5twwh/oe9fk17lfafdSORXGLzA5N1AJJ6Jfg5ZchORl694Zry04oLIRJlFKMiBrBtvu20adpHxIzEhk8azCPLnpURpnaOUnoF+ngQXjrLeP11KmglKnhCHGexr6N+e2233i538s4Oznz1tq36PZpN3Yl7TI7NFFDJKFfpCefhNxcuPVWiC5/IkshTGdxsvBErydY+a+VNK/XnK0nttJ5emfeW/eeDEayQ5LQL0JGBmzfbgwievFFs6MRonJdG3dl872buaP9HWTlZzH+1/EMnDmQw2cOmx2aqEaS0C+Clxds3gzLl0N4uNnRCGEdHzcfvrjuC76/+XsCPQP5/cDvXP7h5Xy741uzQxPVRBL6RXJ2hq5dzY5CiKq7vs31bL9/O9dedi2pOanc/N3NPP7b4+QXykAKWycJvQqys42pcY/JOgPCxoV4hzD/lvlMu3oaFmXhtVWvMXjWYJIyHHZdGrsgCb0K3nsP3ngDbrjB7EiEuHRKKR7p/gi/3/47wV7BLN2/lKgPo5i/a77ZoYmLJAndSrm58OabxutnnjE3FiGq01VNr2Lj2I3ENI0hMSOR6+ddz63/u5VTWafMDk1UkSR0K333ndHU0rYtDBpkdjRCVK9Q31B+v/133h38Lp4unsz+ezaXf3g5S/cvNTs0UQWS0K309tvG88MPyyAiYZ+clBPjuo5j233b6BnWk4S0BAbMHMCExRPIzs82OzxhBUnoVlizBtatg/r1jYFEQtiz5vWbEzsmlhf6vIBFWZi2Zhp9vuwjU/LaAEnoVige4j92LHh6mhuLELXB2cmZp3s/zaq7VtHErwlr4tcwYOYAUrJSzA5NXIAkdCs88giMGGEs/CyEI+nauCsrxqwgwj+C9Qnr6T+zP8mZyWaHJSogCd0K3bvDnDnG4s9COJpw/3CWj1lO83rN2XRsE32/6svx9ONmhyXKIQn9AmTBCiEMYX5hLB+znMsCLmPbiW30mtGLAykHzA5LlCEJ/QI+/BAGDIDVq82ORAjzNfZtzJ93/knnhp3Zl7KPnjN68veJv80OS5QiCb0CBQXGQKKlS+HoUbOjEaJuCPIK4o87/qBP0z4cSz9Gr8978eM/P5odligiCb0CCxbAvn0QEQHDh5sdjRB1h6+bLwtvXchNkTeRmpPKdd9cx1O/P0VBYYHZoTk8SegVmDrVeH74YbBYzI1FiLrG3dmdb278htcHvI6TcuLlv15m0KxB0lfdZM5mB1AXrV0LK1eCnx/8619mRyMcSWpqKomJieTl5ZkdilWG1htKv+H9SMpMorCwkA3bNhDiFYLFybqrID8/P3btcrwl8cqrt4uLC8HBwfj6+l70cSWhl2PaNOP53nvBx8fcWITjSE1N5cSJEzRu3BgPDw+UDc0xkZOfw95Te8nOz6bQUkjz+s3xcPGodL+0tDR8HPCXrGy9tdZkZWVxtOiG3cUmdWlyKSMjA376CZycYPx4s6MRjiQxMZHGjRvj6elpU8kcwM3ZjdYBrfFy8SK3IJfdybvJyM0wOyyboZTC09OTxo0bk5iYeNHHkYRehpcXHDwI338PoaFmRyMcSV5eHh4elV/V1lXOFmdaBbTCz82P/MJ8difvJi0nzeywbIqHh8clNbdZldCVUoOUUruVUnFKqScqKHOzUmqnUmqHUmr2RUdUBwQFwXXXmR2FcES2dmVelsXJQvP6zanvUZ9CXcieU3vkRmkVXOr3v9KErpSyAO8Dg4FIYKRSKrJMmZbAk0BPrXVb4JFLisokWVlG/3MhxMVzUk5E+EcQ6BmI1pq4U3GS1GuJNVfoXYE4rfV+rXUuMBe4tkyZe4D3tdYpAFrri28EMtHUqRAeDnPnmh2JELZNKUW4XzjBXsFoNPtS9kmbei2wppdLY+BIqffxQLcyZVoBKKVWAhZgstZ6UdkDKaXGAmMBQkJCiI2NveCJ09PTKy1TXbSG6dO7cvSoJ4cObSM21rzlt2qz3nWJo9fbz8+PtDT7anP+ZfYvPFBqmlIvby8imkZwxx13cNddd6GUqpU6v/TSS7zyyiukpqZavc+QIUMAWLhwYbXHU1BQUGG9s7OzL/r3wJqEXl6jTtlpq5yBlkAMEAr8qZSK0lqfPmcnracD0wGio6N1TEzMBU8cGxtLZWWqy8aNcOQIBAfDxIntcDaxQ2dt1rsucfR679q1y+668BXf5H3rs7fwDfIlLyuPtUvWMmnSJFJTU5k0aVKt1PnBBx/k2muvrdK5Pv74Y4Aaie9C3TXd3d3p2LHjRR3XmrQVD5SeODYUSCinzBqtdR5wQCm1GyPBr7+oqEzw9dfG88iRmJrMhbBHg3oNIt8/n6z8LHr37c3+fft56623mDRp0nlltdbk5eXh6upabecPDQ0ltIrd1iIjIysvVMdY04a+HmiplIpQSrkCI4AFZcr8APQBUEoFYjTB7K/OQGtSfr4x3znA6NHmxiKEPbI4WWgZ0BJXiyvpuem0vLwlaWlpJCUl0bRpU0aPHs2MGTNo3bo1rq6u/PLLLwBkZmby+OOPExERgaurKxEREbz44osUFhaec/ykpCQeeOABwsLCcHNzIywsjNtuu42cnBwAJk+efF4Pkrfffps2bdrg4eFBvXr1iI6OZv78+SXbY2JizvuPcffu3QwfPhx/f388PDzo3r07ixad27pcfK69e/cydOhQvL29CQ8PZ8qUKefFXd0qTeha63xgHLAY2AXM01rvUEpNUUoNKyq2GEhWSu0ElgGTtNY2s6zJb7/BiRNw2WXQubPZ0QhxLqUqfkyffrbc9OkXLlta584Vlxs79my5jRurrx6uFlda1m+JRVnYv38/FosFLy8vAJYtW8abb77Jc889x6JFi2jXrh35+flcffXVfPrppzz88MP8+uuv3H333bzwwgvnXNmnpKRwxRVX8M033zBhwgQWLlzIa6+9Rl5eHrm5ueXGMmvWLCZOnMjIkSNZuHAhs2bN4sYbb+TUqYrvnSUkJNCrVy+2bt3Ke++9x7x58/D392fo0KH8+uuv55UfPnw4ffv25YcffuC6667jueee48svv7zEr+KFWdW4oLVeCCws89mzpdMBUn4AAB9DSURBVF5rYELRw+b8/LPxfNtt5//gCyEuXUFBAfn5+WSnZ7P8++Us+3UZVw64kixLFmAk5Y0bN9KgQYOSfWbOnMlff/3F8uXL6d27NwD9+vUD4Pnnn+fxxx8nODiYadOmsX//fjZs2HBO2/PIkSMrjGf16tW0a9eOZ58tSWMlN0Er8uabb5KSksLq1atp0aJFyT6RkZH85z//YfDgweeUnzhxInfeeScA/fv3548//mDOnDkln9UEGSkKvPsuxMbKRFyibtK64kfpq+mxYy9ctrSNGysuV/qqv7r+Y23dujUuLi7Ur1+fiQ9P5MZbbuSZqc9wMvckBbqA7t27n5PMARYtWkR4eDhXXHEF+fn5JY+BAweSl5fHmjVrAFiyZAldunSp0o3ELl26sGXLFsaPH8/SpUvJzMysdJ8VK1bQvXv3kmQOYLFYGDlyJFu2bDmvB83QoUPPeR8VFcXhw4etjvFiyO0/jHlbrrrK7CiEsF/z588nNDQUHx8fwsPDcXd3JzkzmQOnD1BQWIBvoC9a63PauRMTEzl06BAuLi7lHjM5ObnkuX379lWK5/bbbyc7O5vPPvuMDz74ABcXF4YMGcKbb75J06ZNy93n1KlT5f7RaNCgAVprUlJSzplUq379+ueUc3NzIzs7u0pxVpXDJ/SUFKhXz+wohLBvUVFR51zZAgR4BpCbbbRx5xTkcPjMYZr4NSlJ6gEBAURERDBv3rxyj1mceAMDA0tmKbSWUop7772Xe++9l5SUFJYsWcLEiRO55ZZbWLt2bbn71K9fn+PHz18c+/jx4yilzkvgZnDoJpfDhyEkBK6/XhaEFsIM3i7euDi5oFAkZSaRmHF2kPmgQYM4cuQI3t7eREdHn/cIDAwEYODAgaxbt46tW7deVAz16tXjlltu4eabb2b79u0VlrvqqqtYs2YNBw8eLPmsoKCAb775ho4dO9aJMQQOfYU+axbk5YGrq9wMFcIsSim8XIzeLkdSj+Du7I6fux+33norn3/+Of369WPixIm0b9+e3Nxc9u3bx4IFC/jhhx/w9PTk0UcfZfbs2fTv35+nn36ayy+/nJMnT/Ljjz/y0UcflZtox44di4+PDz169CA4OJg9e/Ywc+ZMBg4cWGGcjz76KF988QUDBgzg+eefx9fXlw8++IA9e/aUdLM0m8MmdK1h5kzj9W23mRuLEI7OzdmNht4NOZZ+jH0p+2gT2AYPFw8WL17MK6+8wvTp0zlw4ABeXl40b96coUOHlgw88vf3Z+XKlTz99NO88sorJCcnExISQt++fSscnNSzZ08+//xzZs6cyZkzZ2jUqBGjR4/m+eefrzDGRo0a8ddff/H4449z//33k5OTQ4cOHfjll18YNGhQjXxdqkppk9oaoqOj9YYNGy5YpiaHgm/dCh06GFPlHj0KFdx3MYWjD4F3NKWH/rdp08bscGpV6SHwWmv2p+wnJTsFV4srrQNb42qpvtGidcmFhv5X9nOglNqotY4ub5vDtqF/+63xfMMNdSuZC+GolFJE+EeUrHq0N3kvBYUyn3VVOGRC1/psQr/xRnNjEUKc5eTkRIv6LXCzuJGVn0XcqTgKdc0Ol7cnDpnQDxyAuDgIDJT+50LUNS4WF1oFtMLZyZm03DQOnj6IWU3DtsYhE3qzZpCQAN99JzMrClEXuTm70bJ+S5yUE6eyThGfGm92SDbBIRM6GP3P5epciLrLy9WLFvVaoFCcyDjB8fTzB/WIczlcQs/OlkFEQtgKX3dfmvo3BSA+NZ6TmSfNDaiOc7iE/uqrxrqh33xjdiRCCGsEeAYQ5mussXPo9CEycyufSMtROVxC/+47Y6k5Pz+zIxFCWCvEO4QgzyA0mv2n90t3xgo4VELfvRu2bwd/f+jb1+xohBBVEeYbhoezB9n52RxJPVL5Dg7IoRJ68epSw4YZ87cIIWyHk5MTzeo1QynFycyTnMqqeHUhR+WQCX34cHPjEMJRfPHFFyilSh6urq40b96cp5566qLmBvdw8TinPT0nP+eiY2vatCljxow5L9bSsynaGofphX30KKxbBx4ecIEJ1YQQNeDbb78lNDSUtLQ05s+fz8svv0xaWhovvfRSlY8V5BlEak4qp7NPc+jMIVrWb3neAtCOymESevG6oVdfDZ6e5sYihKPp0KFDyQIXAwYMYO/evXz22Wf897//rfKxlFKE+4WTnptOak4qSZlJBHsFV3fINslhmlzuustYN/TJJ82ORAjRqVMnsrKySpaRAzhw4AC33norQUFBuLm50aFDB+YXt5OWsnXrVm6+8Wb6tu1Lr+a96NquKy+8+ELJ9iVLljBkyBAaNmyIp6cnUVFRTJ06lYIC++8Z4zBX6M7OMjJU2B71fN1oStDPVe9ovIMHD+Ln51eybNuRI0fo1q0bwcHBTJs2jaCgIL755htuuOEGfvjhB4YNGwbAunXriImJoUWLFrw17S2cfJ3YuXsnu3bvKlmTdP/+/fTr14/x48fj7u7Ohg0bmDx5MklJSbzyyivVWo+6xiESutayIpEQZiooKCA/P7+kDf3777/nrbfewmKxADB58mS01ixfvpyAgAAArr76ao4cOcKzzz5bktAfe+wxAgICWLNmDZ6enuQX5LOj+w7yCvM4kXGCBt4NuO+++0rOq7XmyiuvJDc3lzfeeIOXXnoJJyf7bZhwiIQ+ahScPg1vvAFt25odjRDWq+4rY7O0bt36nPcPPPAA48aNIy0tDYBFixYxZMgQ/Pz8yM/PLyl39dVXM2nSJFJTU3F2dmblypVMmjQJz6IbYc4WZ8L9w4k7FcfR1KP4uvly5uQZJk+ezKJFi0hISDjneImJiTRo0KAWamwOu0/omZnw44+QlQWffGJ2NEI4pvnz5xMaGkpSUhJvvvkmH3zwAd26dWN4UR/ixMREvvrqK7766qty909OTsbV1ZXCwkJCQ0PP2ebv7k+QZxBJmUnsS97Hv4b9i4SEBCZPnkzr1q3x8PDghx9+4MUXX7yorpK2xO4T+tKlRjLv0gXK/BwIIWpJVFRUSS+Xvn370q5dOyZNmsTAgQPx8fEhICCAK6+8kscff7zc/Rs1akRBQQFOTk4cPXr0vO2hvqGk5aaxZ88eNmzYwMyZMxk9enTJ9p9++qlmKlbH2G9jUpHi7opFTXBCCJO5ubnx+uuvk5iYyKeffgrAoEGD2LZtG23btiU6Ovq8h5ubG56envTq1Yuvv/6arKysc45pcbIQ4R9BTpYx0Cifs80seXl5zJo1q/YqaCK7vkLX+mxCv+Yac2MRQpw1bNgwunTpwjvvvMPEiROZMmUKXbt2pXfv3owbN46mTZuSkpLC9u3b2b9/PzNmzADgjTfe4KqrrqJHjx5MnDiR0NBQ9u/fz5YtW3j33Xfp3rE7DUMb8uwzz+Ll5oWrqyvTpk0zuba1x66v0DdvhmPHoHFjaN/e7GiEEKX997//JSkpiY8++ogmTZqwYcMG2rdvz1NPPcWAAQO4//77Wb58OX1LzaTXpUsXVq5cSVhYGOPHj2fIkCG8/vrrJe3qTeo34a0v3qJ+UH3uuOMOHnzwQXr37s0TTzxhVjVrlV1foZe+Opdui0LUvjFjxpwzX0ppAwcOJDU1FR8fHwBCQ0NLmmAupGPHjhW2iVucLPS/oj/NfmiGs5MzUcFRODsZae7uu+8+p2zZOVsuFKutsOuE/q9/QVAQdOpkdiRCiNpSz70ePq4+pOWmkZCWQBO/JmaHVGusanJRSg1SSu1WSsUppSr830UpdaNSSiuloqsvxIsXGgr33w/dupkdiRCitiilCPMzZmRMzEgkM89xVjiqNKErpSzA+8BgIBIYqZSKLKecD/AQsLa6gxRCiKrwdPEsmbDr4OmDFOpCkyOqHdZcoXcF4rTW+7XWucBc4Npyyr0AvAbUiZ77jzwCEydCfLzZkQghzNDYpzGuFlcy8zI5nn7c7HBqhTVt6I2B0us9xQPnNGIopToCYVrrn5VSj1V0IKXUWGAsQEhICLGxsRc8cXp6eqVlypOb68THH/ckO9tC9+6rCQq6+EnwzXCx9bZ1jl5vPz8/UlNTHWpu74KCgpLh/zUhxDWEI1lHSEhLwKXABXeLe42dqyoqqrfWmuzs7Iv+PbAmoZf301UywYRSygmYBoyp7EBa6+nAdIDo6GgdExNzwfKxsbFUVqY8ixdDdjZ06AA33dSjyvub7WLrbescvd5xcXE4OzuXzFPiCNLS0kp6udQEH3zIccohMSORxNxE2gS1wUmZ31u7onpnZmbi4+NDx44dL+q41tQsHggr9T4USCj13geIAmKVUgeB7sACM2+M/vqr8TxkiFkRCFF1wcHBHD16lMzMTLS2j0m56oLGPo1xs7iRlZ/FsbRjZodTLq01mZmZHD16lODgi1+sw5or9PVAS6VUBHAUGAGMKhXIGSCw+L1SKhZ4TGu94aKjukRLlhjPgwaZFYEQVefr6wtAQkICeXl5JkdTO7Kzs3F3r/lmkIL8Ak6mn+QkJ0n2ScbF4lLj57yQ8urt4uJCSEhIyc/Bxag0oWut85VS44DFgAWYobXeoZSaAmzQWi+46LPXgCNHYNcu8PGB7t3NjkaIqvH19b2kX2hbExsbe9HNC1U19qexfLLpE64Kv4pldywz9V5FTdXbqsYkrfVCrXUrrXVzrfWLRZ89W14y11rH1IWr8759wcXcP8JCiDrklf6vEOQZxPJDy5m5babZ4dQI8+8OVLP27WH8eBg50uxIhBB1SX2P+kwdOBWAiUsmcirrlMkRVT+7S+jR0fDOO3DLLWZHIoSoa0a3G01M0xhOZp7ksSUV9rC2WXaX0IUQoiJKKT4a+hFuFjc+3/I5i+IWmR1StbKrhP6//8Fnn0FiotmRCCHqqssCL2NKnykA3PPTPZzJPmNyRNXHrhL61Klw992werXZkQgh6rIJPSbQtXFX4lPjmfTbJLPDqTZ2k9BPn4a1a8HZGfr0MTsaIURd5uzkzOfXfo6rxZVPNn3Cb/t+MzukamE3Cf2PP6CgAHr0AAfqxiuEuEiRQZE8d9VzAExYMsEuRufaTUL/regP7MCB5sYhhLAdE3tMpJFPI7YnbreLG6SS0IUQDsvN2Y1Huj0CwKsrXzU5mktnFwn9wAHYtw/8/KBzZ7OjEULYkrGdx+Lr5svyQ8tZG2/b6/PYRUJPTTWG+g8eDBaL2dEIIWyJn7sf90ffD8Brq14zOZpLYxcJvX17+P13mD3b7EiEELbo4W4P42pxZf6u+ew+udvscC6aXST0Yg600IsQoho19GnI7e1uR6N5Y9UbZodz0Ww+oR89avQ/z883OxIhhC2b1HMSCsXMbTNJykgyO5yLYvMJffZsY97z8ePNjkQIYctaBbRiaKuh5BTk8PHGj80O56LYfEIv7q7Yu7e5cQghbF9xF8b3179PTr5tLS4PNp7Qs7Phzz+N1/36mRuLEML29Y3oy+XBl3M8/TjzdswzO5wqs+mEvmqVkdTbt4dLWFdVCCEAY3rdR7obV+nT1kyzuekAbDqhL11qPPfvb24cQgj7MeryUQR5BrH5+Gb+PPyn2eFUiU0n9GXLjOe+fc2NQwhhP9yd3UsGGk1bM83kaKrGZhN6Xh4cOmSMDL3ySrOjEULYk/u73I+rxZUf//mRXUm7zA7Hajab0F1cjD7ou3eDj4/Z0Qgh7EkD7wbc2eFONJpXVr5idjhWs9mEDsbI0ObNzY5CCGGPHu/5OBZlYda2WRxIOWB2OFax2YSemWl2BEIIexZRL4JRl4+iQBfw+qrXzQ7HKjaZ0FNToX596NnTWKVICCFqwhO9ngBgxuYZHEs7ZnI0lbPJhL5yJeTkQGGhTJcrhKg5kUGRXN/menIKcpi6eqrZ4VTKJhN6bKzxHBNjZhRCCEfwVK+nAPhow0ckZyabHM2F2WRCL+5/3qePuXEIIexf50adubr51WTkZfDeuvfMDueCbC6hp6bCxo3g7AxXXGF2NEIIR1Dclv7OunfIyM0wOZqK2VxC/+svo+28Sxfw9jY7GiGEI7gq/Cq6h3bnVNYpPtn0idnhVMjmEnpx+7k0twghaotSiid7PQnA1NVTyS3INTmi8tlcQp8wAebMgVGjzI5ECOFIrml1DZFBkcSnxjNr2yyzwymXVQldKTVIKbVbKRWnlHqinO0TlFI7lVLblFK/K6XCqz9UQ4MGMGIEtG1bU2cQQojzOSknnuhppL9XV75KoS40OaLzVZrQlVIW4H1gMBAJjFRKRZYpthmI1lq3A74DXqvuQIUQwmwjokYQ7hfO7uTd/LznZ7PDOY81V+hdgTit9X6tdS4wF7i2dAGt9TKtdfFg/DVAaPWGKYQQ5nOxuJQsgPHm6jdNjuZ8qrIVOZRSNwKDtNZ3F72/DeimtR5XQfn3gONa6/+Ws20sMBYgJCSk89y5cy947vT0dLwdsCuL1NuxOGq9wTbrnpGfwS1rbiGjIIOPOn3EZT6XVfkYl1LvPn36bNRaR5e3zdmK/VU5n5X7V0ApNRqIBq4qb7vWejowHSA6OlrHVDLUMzY2lsrK2COpt2Nx1HqD7db9vsL7mLp6KivyVnBvzL1V3r+m6m1Nk0s8EFbqfSiQULaQUqo/8B9gmNba9pbLFkIIK43vOh6LsjBvxzyOnDlidjglrEno64GWSqkIpZQrMAJYULqAUqoj8DFGMk+s/jCFEKLuCPcP58bIG8kvzOfdde+aHU6JShO61jofGAcsBnYB87TWO5RSU5RSw4qKvQ54A98qpbYopRZUcDghhLALE3pMAGD6xumk5aSZHI3BmjZ0tNYLgYVlPnu21Ov+1RyXEELUaV0bd6VnWE9WHlnJNzu+4e5Od5sdku2NFBVCiLrink73AMYCGHWBJHQhhLhIN0beiLerN6vjV/PPyX/MDkcSuhBCXCwvVy9uaXsLAF9s+cLcYJCELoQQl+TODncC8OXWL8kvzDc1FknoQghxCa4Iu4JWAa04nn6cxXGLTY1FEroQQlwCpVTJVfqMLebeHJWELoQQl+j29rfjpJz4afdPJGUkmRaHJHQhhLhEjXwaMbjFYPIK8/h629emxSEJXQghqsG/Ov4LgM82f0Zls9jWFEnoQghRDa5pdQ3BXsHsSNrBuqPrTIlBEroQQlQDV4srt7e7HTCu0s0gCV0IIarJXZ3uAmDO9jmk56bX+vkloQshRDVpHdianmE9Sc9N59sd39b6+SWhCyFENbqro3GVbkaziyR0IYSoRje1vQlvV29WHlnJrqRdtXpuSehCCFGNvF29GRk1EoBPN31aq+eWhC6EENWseJ70L7d+SU5+7S2xLAldCCGqWXSjaDo06EByVjLz/5lfa+eVhC6EENVMKcW9ne8FjDVHa4skdCGEqAGjLh+Fp4snyw4uY0/ynlo5pyR0IYSoAb5uviU3Rz/Z+EmtnFMSuhBC1JCxnccC8MXWL2rl5qgkdCGEqCFdGnWhfUh7TmaerJWbo5LQhRCihpS+OTptzbQan1ZXEroQQtSgOzrcQaBnIOuOrmPFoRU1ei5J6EIIUYM8XTwZ33U8AK+ufLVGzyUJXQghatiDXR7E08WTX+N+ZduJbTV2HknoQghRwwI8A7i7490AvL7q9Ro7jyR0IYSoBRN6TMCiLMz5ew7Hs4/XyDkkoQshRC0I9w9nRNQICnQB38bXzOIXktCFEKKW/Lvnv3FxcqFAF9TI8a1K6EqpQUqp3UqpOKXUE+Vsd1NKfVO0fa1Sqml1ByqEELauXUg7EiYm8EjLR2rk+JUmdKWUBXgfGAxEAiOVUpFlit0FpGitWwDTgJrtmyOEEDYq0DOwxo5tzRV6VyBOa71fa50LzAWuLVPmWuDLotffAf2UUqr6whRCCFEZZyvKNAaOlHofD3SrqIzWOl8pdQYIAE6WLqSUGguMBQgJCSE2NvaCJ05PT6+0jD2SejsWR603OG7da6re1iT08q60y05IYE0ZtNbTgekA0dHROiYm5oInjo2NpbIy9kjq7Vgctd7guHWvqXpb0+QSD4SVeh8KJFRURinlDPgBp6ojQCGEENaxJqGvB1oqpSKUUq7ACGBBmTILgDuKXt8I/KFreloxIYQQ56i0yaWoTXwcsBiwADO01juUUlOADVrrBcBnwEylVBzGlfmImgxaCCHE+axpQ0drvRBYWOazZ0u9zgZuqt7QhBBCVIUyq2VEKZUEHKqkWCBleso4CKm3Y3HUeoPj1v1S6h2utQ4qb4NpCd0aSqkNWutos+OobVJvx+Ko9QbHrXtN1VvmchFCCDshCV0IIexEXU/o080OwCRSb8fiqPUGx617jdS7TrehCyGEsF5dv0IXQghhJUnoQghhJ+pEQnfUBTSsqPcEpdROpdQ2pdTvSqlwM+KsbpXVu1S5G5VSWillF93arKm3Uurmou/5DqXU7NqOsSZY8XPeRCm1TCm1uehnfYgZcVY3pdQMpVSiUmp7BduVUuqdoq/LNqVUp0s+qdba1AfGdAL7gGaAK7AViCxT5gHgo6LXI4BvzI67lurdB/Asen2/o9S7qJwPsAJYA0SbHXctfb9bApuBekXvg82Ou5bqPR24v+h1JHDQ7Lirqe69gU7A9gq2DwF+xZittjuw9lLPWReu0B11AY1K6621Xqa1zix6uwZjpktbZ833G+AF4DUguzaDq0HW1Pse4H2tdQqA1jqxlmOsCdbUWwO+Ra/9OH82V5uktV7BhWedvRb4ShvWAP5KqYaXcs66kNDLW0CjcUVltNb5QPECGrbMmnqXdhfGX3NbV2m9lVIdgTCt9c+1GVgNs+b73QpopZRaqZRao5QaVGvR1Rxr6j0ZGK2UiseYM2p87YRmuqrmgEpZNTlXDau2BTRsjNV1UkqNBqKBq2o0otpxwXorpZww1qUdU1sB1RJrvt/OGM0uMRj/jf2plIrSWp+u4dhqkjX1Hgl8obWeqpTqgTFza5TWurDmwzNVtee1unCF7qgLaFhTb5RS/YH/AMO01jm1FFtNqqzePkAUEKuUOojRtrjADm6MWvtz/qPWOk9rfQDYjZHgbZk19b4LmAegtV4NuGNMXmXvrMoBVVEXErqjLqBRab2Lmh4+xkjm9tCeCpXUW2t9RmsdqLVuqrVuinHvYJjWeoM54VYba37Of8C4EY5SKhCjCWZ/rUZZ/ayp92GgH4BSqg1GQk+q1SjNsQC4vai3S3fgjNb62CUd0ew7waXu9u7BuBv+n6LPpmD8IoPxDf4WiAPWAc3MjrmW6r0UOAFsKXosMDvm2qh3mbKx2EEvFyu/3wp4E9gJ/A2MMDvmWqp3JLASowfMFmCg2TFXU73nAMeAPIyr8buA+4D7Sn2/3y/6uvxdHT/nMvRfCCHsRF1ochFCCFENJKELIYSdkIQuhBB2QhK6EELYCUnoQghhJyShC1MUzaJY2eNgUdkvioaFm04pNbkotmoZZV18PCvKxRSdN6Y6zivsU10Y+i8cU48y7+dj9EOeXOozexgZK0StkYQuTKGN2eVKKKVygJNlP79USik3bR9TJghRKWlyETZDKdVRKfWnUipTKbVXKXVfme1jipoleiulvlVKnQbWltp+VdFCIWlKqQyl1GKlVFSZY1ytlFqllDqjlEovWpjh2XLCiVBK/VJU5pBS6tmiicVKH+sypdR8pdRppVSWtTMoKqWClFKzlVKpRft+BfhX6YslHJIkdGErfIHZwNcY80ivBz5USvUpp+ws4ADGvD9PACilhgK/A+nAaGAUxkRgfyqliid+a4Yxv8YB4BZgGMZQfK9yzjEf+AO4DmMOluc5O98QSqlGwF9Ae2AccDNwGvhFKTW4krr+D7gGeKoojnzg3Ur2EaJuzOUiD3kAB4GvK9j2Bca0on1KfeYGnASml/psTFG5aeUcIw74vcxnvkXHeKvo/Y1F+/teIM7JRWXuLPP538CSUu/fwEjELUp9ZsGYQXFT2eOVej+g6Pgjyhz/16LPY8z+Xsmj7j7kCl3Yikyt9bLiN9poF98LNCmn7PzSb5RSLYHmwCyllHPxA8gEVmMsFQbGxFB5wFxlrGcafIF4finzfnuZWHoDa7TWcaViLsCYsKmDUsqX8vUACoDvy3w+9wKxCAFIk4uwHSnlfJaDMRNnWWWnIC1OzJ9hJOzSj2soWv2qKPlejfF7MRM4roxFyctbWKTsfPxlY6lfThwAxzFm2atXzjaAhkCK1jqvzOcnKigvRAnp5SLsUdl+3clFz09iTElcVm7JjsZ/AcuUUm5AT4xpXn9RSjXVWp+sQgyngAblfN6gKL6KFmg5BtRTSrmUSeohVTi3cFCS0IUj2I3RRt9Wa/2KNTsUNen8oZTyBn4EIjDa2621HHik6A/BQQCllAXjJudmrXVaBfutxmhrv4Fzm1lGVOHcwkFJQhd2T2utlVIPAj8WrZozDyM5hwBXAIe11m8WdYPsjbFQ8RGMZdCexFgWbHsVT1u8LupvSqnngFTgAYxViIZeINbflFJ/AR8XrVq0F+OPQFRF+whRTNrQhUPQWi/ESNZewKfAYuA1jCaQ1UXFthZtfxlYAryH0YWxr9Y6q4rnSwB6ATuAD4HvMNrVh2qtF1Wy+/UYf1ReBr7BuPAaV5XzC8ckKxYJIYSdkCt0IYSwE5LQhRDCTkhCF0IIOyEJXQgh7IQkdCGEsBOS0IUQwk5IQhdCCDshCV0IIezE/wPklhjFv8brTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_precision_recall_vs_threshold(precisions, recalls, thresholds)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "cell_style": "split"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF8CAYAAAAuF9n2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAf7UlEQVR4nO3de5QdZZnv8e/TSSCEBIMkiEJAAiEjgyIOgpczEhVHZI3gfWDkjDgcM+rgOSOXEcWJCMvRgVGXHvESR8TbiHeMmhlk0BZ1CRNU5AAajBFDuAUCRNqQhJDn/PHuNpumO+nuvXft3V3fz1q1qmpX7bfefmHlt9+6vBWZiSRJqoe+bldAkiRVx+CXJKlGDH5JkmrE4JckqUYMfkmSasTglySpRioN/oi4JCLWRcSNI2yPiPhwRKyKiBsi4hlV1k+SpMmu6h7/pcBxO9j+EmBBY1oMfKyCOkmSVBuVBn9mXg3ct4NdTgQ+m8U1wOyIeGI1tZMkafLrtWv8+wK3Na2vbXwmSZLaYGq3KzBEDPPZsGMKR8RiyuUA+voe92e7735AJ+tVe5kQw/3XUVtV2c6Z0bTcPI9HrWdGY/7o9cEyWh31u5W/1xHHu6evL+nrg4gkoqxHDK6XbbD9v+/2+eB+MGVKMmVK/vG7g/OpU8vnEf4HHsktt9xyb2bOHc93ey341wLzmtb3A+4YbsfMXAosBVi4cGGuXPmLzteuxvr7+1m0aFG3qzHp2c5jt20bPPwwbN4MW7aUaXD5saED1157Lc961tFkwiOPwNatZd68PGgwoIYuN5c5tPzBeWap27Zt28sfnLZte+x3tm6FTZvgoYe2T3/4w/ZpYKDMN23aXpfBafAYg/UfOh9cHvp3Da3btm2P/u7gcvOxNm2Ce+7Zvn8nzZoFe+1Vpsc/HqZP3/4DdOZM2HPP8vm++8K8ebDPPvCEJ8Dee5d9J7OI+N14v9trwb8MOD0iLgOOBjZk5p1drpOkHtbXB7vuWqbRWLv2IQ46qLN1muy2bYP77y8/RJp/qGzaVKYVK27k4IMPe9QPj+YfEw8/DBs3wvr1cO+95UfNYBkDA+WHxbp18OCDZbr11rHVLwJe8AI46STYf3+YMwcWLCg/JFRx8EfEF4FFwJyIWAu8C5gGkJkfB5YDxwOrgI3A66usnyRp5/r6tvfEh7PbbvfS6omrTNiwofw4uO++Mt+8mT9eQnjwwfLjY/16uP12uO02uOsuuPvuMl11VZmazZ8Phx9efhQccwwccADssUdr9ZyIKg3+zDx5J9sT+PuKqiNJ6lERMHt2mcZ6huaBB+Cyy6C/v5xRuPtuuOUWWL26TN/4xvZ9Z86EhQvh6KPL9Nznjv14E02vneqXJKkls2fDG99YpkEPP1zC/9pr4bvfhZ/+tJwpGBgoyz/9KXz0o2Xf5z0P3vIWeNnLYOokTMlJ+CdJkvRo06bBn/5pmf72b8tnmeVywQ03lB8E114LV14JV19dpv33L+F/8MHw4hfDIYd0929oF4NfklRLEeWpgEWL+OM9Cb//PXzmM/DhD8OqVWUOMGUK/N3flTMBf/In3apxe/TaAD6SJHXNHnuUcF+5svT+L7ywPB2QWS4FPOUpcMQRsGRJuV9gIjL4JUkaoq8Pjj0Wzj4bvvhFuP56OO208kjg9dfDBReUHwFnnlnuHZhIDH5JknbiqU+Ff/u3Mr7A8uXw2teWQaI+8IHyVMCznw3/9V/druXoGPySJI3S9OnwkpfA5z8PK1bAqaeWRwKvuQZe9CJ4/evLDYO9zOCXJGkcjjwSPv3pMk7Ae95TRo+89FI47LDHDh7USwx+SZJaMGMGvOMd8ItfwHOeA3fcUXr/n/pUt2s2PINfkqQ2WLgQfvCDcsd/Znn877vf7XatHsvglySpTaZOhXe/G849t7yY6PTTy6iBvcTglySpzd71rjLi369/DR/5SLdr82gGvyRJbTZtGrz3vWX5jDPg/e/vbn2aGfySJHXAq161PfDPOgve/vZy7b/bDH5JkjrkjDPKM/9TpsD73ldG+us2g1+SpA567Wvh61+HXXaBD34QLr+8u/Ux+CVJ6rATToCLLirLixfDvfd2ry4GvyRJFTj9dHj+8+Gee+Btb+tePQx+SZIq0NcHH/94OeV/ySXwox91qR7dOawkSfVzyCHbe/v/+I/ducvf4JckqUJnnw1z58JPfgLf+U71xzf4JUmq0KxZ5Zl+gHPOga1bqz2+wS9JUsXe/GaYPx9uugk++clqj23wS5JUsV13hQsvLMtLlsDAQHXHNvglSeqCV7wCjjqqPNP/jW9Ud1yDX5KkLoiAU08ty1/4QnXHNfglSeqS17wGpk6FK68sA/tUweCXJKlL9toLjj0Wtm2rbgx/g1+SpC565SvL/Gtfq+Z4Br8kSV104ollON+rroL77+/88Qx+SZK6aO7c8vKerVur6fUb/JIkddkpp5T55z/f+WMZ/JIkddkrXgHTp8MPfgBr1nT2WAa/JEldtsce8NKXluVOD+Zj8EuS1ANe9rIy7/RjfQa/JEk94Pjjy2A+P/whrF/fueMY/JIk9YDZs+HP/xweeaSEf6cY/JIk9YhnPrPMf/7zzh3D4JckqUc84xllbvBLklQDRxxR5j/7WeeOYfBLktQjDj64PM9/++3w4IOdOYbBL0lSj+jrg/nzy/JvftOhY3SmWEmSNB4HHVTmBr8kSTVgj1+SpBqxxy9JUo0ceGCZ/+53nSnf4JckqYfMm1fmt93WmfINfkmSesj++5f5mjWQ2f7yDX5JknrI7Nmw++4wMAAbNrS/fINfkqQeErH9dP+aNe0v3+CXJKnH7LNPma9b1/6yDX5JknrMnnuW+QMPtL9sg1+SpB4zGPz339/+sg1+SZJ6zOzZZW6PX5KkGrDHL0lSjey9d5nfeWf7y648+CPiuIhYGRGrIuKcYbbvHxHfj4ifR8QNEXF81XWUJKmbBl/Us3p1+8uuNPgjYgpwMfAS4FDg5Ig4dMhu7wS+nJlHACcBH62yjpIkddukCX7gKGBVZq7OzC3AZcCJQ/ZJYI/G8uOAOyqsnyRJXTdvHkyZArffDps2tbfsqoN/X6D5tQNrG581Ow84JSLWAsuBt1RTNUmSesO0aWXM/sz2v6VvanuL26kY5rOhryA4Gbg0M98fEc8GPhcRh2XmtkcVFLEYWAwwd+5c+vv7O1FfNQwMDNjGFbCdO8827jzbuD323PNwfvvbPbn88hs4+uj72lZu1cG/FpjXtL4fjz2VfxpwHEBm/iQipgNzgEcNXJiZS4GlAAsXLsxFixZ1qMoC6O/vxzbuPNu582zjzrON2+PP/gx+9jOYOfNptLM5qz7VvwJYEBEHRsQulJv3lg3ZZw3wQoCIeAowHbin0lpKktRlT3ximd/T5gSsNPgzcytwOnAF8EvK3fs3RcT5EXFCY7czgTdExC+ALwKnZnbijcSSJPWuTo3eV/WpfjJzOeWmvebPljQt3ww8t+p6SZLUSx73uDLfsKG95TpynyRJPahTPX6DX5KkHmTwS5JUI57qlySpRgZ7/O1+Q5/BL0lSDxp8Q9+6dWUEv3Yx+CVJ6kEzZ8KMGbBxIwwMtK9cg1+SpB4UAU94Qlm+++72lWvwS5LUo/bZp8zvuqt9ZRr8kiT1KINfkqQaGTzVb/BLklQDgz1+r/FLklQDnuqXJKlG5swp8/Xr21emwS9JUo96/OPL3OCXJKkG9tqrzA1+SZJqYDD477uvfWUa/JIk9ajmU/3tGq/f4JckqUfttluZtmyBP/yhPWUa/JIk9bB2n+43+CVJ6mHtvrPf4JckqYe1+85+g1+SpB42a1aZDwy0pzyDX5KkHjZzZpkb/JIk1YDBL0lSjRj8kiTViMEvSVKN7L57mTuAjyRJNWCPX5KkGjH4JUmqkcFT/Qa/JEk1MNjj9xq/JEk14M19kiTViMEvSVKNGPySJNWIwS9JUo0Y/JIk1Uhz8Ge2Xp7BL0lSD5s2rUyPPAJbtrRensEvSVKPa+fpfoNfkqQeNxj8Gze2XpbBL0lSj7PHL0lSjcyYUeb2+CVJqoFddy3zzZtbL8vglySpx02fXuabNrVelsEvSVKPG+zxG/ySJNXAYI/fU/2SJNWAPX5JkmrEHr8kSTVij1+SpBqxxy9JUo3Y45ckqUZ8jl+SpBoZDP6HHmq9LINfkqQeNzhWv8EvSVIN+HY+SZJqZLDHb/BLklQDgz3+Cfla3og4LiJWRsSqiDhnhH1eExE3R8RNEfHvVddRkqRe0s4e/9TWixi9iJgCXAy8CFgLrIiIZZl5c9M+C4C3A8/NzPsjYu8q6yhJUq+ZyD3+o4BVmbk6M7cAlwEnDtnnDcDFmXk/QGauq7iOkiT1lAnb4wf2BW5rWl8LHD1kn0MAIuLHwBTgvMz8z6EFRcRiYDHA3Llz6e/v70R91TAwMGAbV8B27jzbuPNs4/Zbs2YGcBTr12+kv/+/WyprzMEfEa8DTgb2B6YP2ZyZedCOvj7MZzlMnRYAi4D9gB9GxGGZ+cCQAy0FlgIsXLgwFy1aNNo/QePQ39+Pbdx5tnPn2cadZxu335o1Zb5t24yW23ZMwR8R/wS8G7gRuB4Y6+sC1gLzmtb3A+4YZp9rMvNh4LcRsZLyQ2DFGI8lSdKk0M5r/GPt8Z8GfCgz3zrO460AFkTEgcDtwEnAXw/Z53LKGYVLI2IO5dT/6nEeT5KkCa+bz/HvBXxrvAfLzK3A6cAVwC+BL2fmTRFxfkSc0NjtCmB9RNwMfB84OzPXj/eYkiRNdNOnQwRs2QJbt7ZW1lh7/D8ADge+N94DZuZyYPmQz5Y0LSdwRmOSJKn2Isrp/oGB1k/3jzX4/wH4ekSsp4T3fUN3yMxtrVVJkiQNNWNGd4L/lsb80yNsz3GUKUmSdmK33cq81Tf0jTWkz+exj99JkqQOGwz+Snv8mXlea4eTJEnj0a4e/7iH7I2ImRExLyJ2b60KkiRpZwYf6as8+CPixRFxHfAAcCuwISL+OyJe1FpVJEnSSLpyjT8iXgx8B1gFXADcBTwR+CtgeUQcn5lXtlYlSZI0VFeu8QPnAd8F/rL5sb2IOB/4NmU4X4NfkqQ269Y1/sMpr8x91LP6jfWPAk9vrTqSJGk43brGvxnYY4Rtsxj7S3skSdIodKvH3w9c0HjJzh9FxP6UywDfb606kiRpON0awOdtwI+BlRFxDXAnsA/wLMpd/m9rrTqSJGk47bq5b0w9/sy8BXga8GFgV+AZwHTgQ8DTM/PXrVVHkiQNp1s9fjLzTuCs1g4rSZLGomsD+EiSpOpNn17mmza1Vs5Oe/wR8T3gzZn5q8byjmRmvrC1KkmSpKF23bXMN7f4/NxoTvVH03IfO347X+xgmyRJGqfKgj8zn9+0vKi1w0mSpPFoV/B7jV+SpAmgK8EfESdGxOub1g+IiJ9ExIMR8dWImNladSRJ0nB22aXMq+7xvxOY27T+AWA/YCnwPMrofZIkqc26dar/IOAGgIjYDTgeOCMzzwTeAby8tepIkqThdCv4pwODQwc8h3Jz4Hcb6yuBJ7VWHUmSNJxuBf+twP9oLJ8I/DQzNzTW9wY2DPclSZLUmsHg37KltXLGOmTvJ4B/jYiXA08H3tS07dnAza1VR5IkDafKAXz+KDM/FBH3Ut7G9+HM/GzT5lnAp1urjiRJGk5Xgh8gM78AfGGYz/+utapIkqSROICPJEk1UlnwR8QjEXFUY3lbY32kaWtr1ZEkScOp8lT/+cDapuUdvaRHkiR1wODIfR2/qz8z3920fF5rh5MkSePR1wczZ8LAQIvljGXniJgWEbuPsG33iJjWWnUkSdJIZs1qvYyx3tz3KeCTI2z7RGOSJEkdsMcerZcx1uBfBHxzhG3LgBe2VBtJkjSibvT49wbWjbDtHuAJrVVHkiSNpBvBvw546gjbngqsb606kiRpJN0I/m8D/xQRT2v+MCKeCpwLfKv1KkmSpOG04xr/WIfsXQK8CPhpRKygPN+/L3AU8Fvgna1XSZIkDafyHn9m3gs8E3gvEJQ39AXwHuCZje2SJKkD2hH843lJzwOUnv+S1g8vSZJGqyvBDxARcyiv5t0L+FZm3hcR04Etmbmt9WpJkqShKn+OP4qLKNf2lwGXAE9ubP4m5QY/SZLUAd24q//twOmUl/UcTbm+P+hbwF+2XiVJkjScbpzq/1/A+Zn53oiYMmTbKuCg1qskSZKG040e/77ANSNs2wIM+wIfSZLUum6M1X87cNgI2w6nPMsvSZI6YMaM1ssYa/B/BVgSEc9t+iwj4hDgTOCy1qskSZKGM21a62WMNfjPA34FXA38uvHZV4D/11h/X+tVkiRJw2lH8I/p5r7MfCgiFgF/DbyYckPfeuAC4AuZubX1KkmSpOFUGvwRMQ04HrghMz8HfK71w0uSpNGq9FR/Zj4MfJntA/ZIkqQKdeMa/2pg79YPK0mSxqobwX8hcG5EzG390JIkaSwqv7kPeAHweOC3EXENcCeQTdszM1/XerUkSdJQ3Qj+PwceBu6hDM87dIjefMw3JElSW3Qj+I8EBjJzU+uHliRJY9HXV6Zt21ooY2c7RMSUiDgvIh4A7gZ+HxFfi4jZ4zlgRBwXESsjYlVEnLOD/V4VERkRR47nOJIkTUat9vpH0+N/I7AE6AdWAPOBlwO/B14/loM13uh3MfAiYC2wIiKWZebNQ/abBfxv4NqxlC9J0mQ3bRps3jz+74/mrv43AJ/MzBdk5tsy89XA3wOnRMQuYzzeUcCqzFydmVsoY/ufOMx+F1CeIPCSgiRJTaro8c8Hzhry2ZeAjwEHsH3M/tHYF7itaX0tcHTzDhFxBDAvM78dEUOP27zfYmAxwNy5c+nv7x9DNTRWAwMDtnEFbOfOs407zzbutOe09O3RBP9Mymn9Zg825rPGeLwY5rM/PgkQEX3AB4FTd1ZQZi4FlgIsXLgwFy1aNMaqaCz6+/uxjTvPdu4827jzbOPOmjED7r9//N8f7V39+0bE/Kb1KU2fP9C8Y2au3kE5a4F5Tev7AXc0rc8CDgP6IwJgH2BZRJyQmdeNsq6SJE1aVZzqB/jqCJ9fPsxnU4b5bNAKYEFEHAjcDpxEedMfAJm5AZgzuB4R/cBZhr4kScXUsT6IP/T7o9hnTHfu70hmbo2I04ErKD8QLsnMmyLifOC6zFzWrmNJkjQZxXAXzcdgp8GfmZ9p7RCPKW85sHzIZ0tG2HdRO48tSdJE12rwj/UlPZIkqYsMfkmSasTglySpRgx+SZJqpK/F5Db4JUmaQOzxS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNeLjfJIk1Yg9fkmSasTglySpRgx+SZJqxOCXJKlGvLlPkqQaMfglSaqRKVNa+77BL0nSBGKPX5KkGjH4JUmqEYNfkqQa8Rq/JEk1Yo9fkqQaMfglSaoRT/VLklQj9vglSaoRg1+SpBox+CVJqhGv8UuSVCP2+CVJqhGDX5KkGvFUvyRJNWKPX5KkGjH4JUmqkYjWvm/wS5I0gdjjlySpRuzxS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUI+ef39r3Kw/+iDguIlZGxKqIOGeY7WdExM0RcUNEXBURB1RdR0mSetW0aa19v9Lgj4gpwMXAS4BDgZMj4tAhu/0cODIznwZ8FbiwyjpKkjSZVd3jPwpYlZmrM3MLcBlwYvMOmfn9zNzYWL0G2K/iOkqSNGlVHfz7Arc1ra9tfDaS04D/6GiNJEmqkakVH2+4exFz2B0jTgGOBI4ZYftiYDHA3Llz6e/vb1MVNZyBgQHbuAK2c+fZxp1nG/e2qoN/LTCvaX0/4I6hO0XEscC5wDGZuXm4gjJzKbAUYOHChblo0aK2V1bb9ff3Yxt3nu3cebZx59nGva3qU/0rgAURcWBE7AKcBCxr3iEijgA+AZyQmesqrp8kSZNapcGfmVuB04ErgF8CX87MmyLi/Ig4obHbRcBM4CsRcX1ELBuhOEmSNEZVn+onM5cDy4d8tqRp+diq6yRJUl04cp8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTVi8EuSVCOVB39EHBcRKyNiVUScM8z2XSPiS43t10bEk6uuoyRJk1WlwR8RU4CLgZcAhwInR8ShQ3Y7Dbg/Mw8GPgj8S5V1lCRpMqu6x38UsCozV2fmFuAy4MQh+5wIfKax/FXghRERFdZRkqRJq+rg3xe4rWl9beOzYffJzK3ABmCvSmonSdIkN7Xi4w3Xc89x7ENELAYWN1Y3R8SNLdZNOzYHuLfblagB27nzbOPOs407b+F4v1h18K8F5jWt7wfcMcI+ayNiKvA44L6hBWXmUmApQERcl5lHdqTGAmzjqtjOnWcbd55t3HkRcd14v1v1qf4VwIKIODAidgFOApYN2WcZ8LrG8quA72XmY3r8kiRp7Crt8Wfm1og4HbgCmAJckpk3RcT5wHWZuQz4FPC5iFhF6emfVGUdJUmazKo+1U9mLgeWD/lsSdPyJuDVYyx2aRuqph2zjathO3eebdx5tnHnjbuNw7PokiTVh0P2SpJUIxMq+B3ut/NG0cZnRMTNEXFDRFwVEQd0o54T2c7auGm/V0VERoR3R4/DaNo5Il7T+P/5poj496rrONGN4t+L/SPi+xHx88a/Gcd3o54TWURcEhHrRnpkPYoPN/4b3BARz9hpoZk5ISbKzYC/AeYDuwC/AA4dss+bgY83lk8CvtTtek+kaZRt/HxgRmP5TbZx+9u4sd8s4GrgGuDIbtd7ok2j/H95AfBzYM/G+t7drvdEmkbZxkuBNzWWDwVu7Xa9J9oEPA94BnDjCNuPB/6DMgbOs4Brd1bmROrxO9xv5+20jTPz+5m5sbF6DWUsBo3eaP4/BrgAuBDYVGXlJpHRtPMbgIsz836AzFxXcR0nutG0cQJ7NJYfx2PHbdFOZObVDDOWTZMTgc9mcQ0wOyKeuKMyJ1LwO9xv542mjZudRvmlqdHbaRtHxBHAvMz8dpUVm2RG8//yIcAhEfHjiLgmIo6rrHaTw2ja+DzglIhYS3ma6y3VVK1WxvrvdvWP87WgbcP9akSjbr+IOAU4EjimozWafHbYxhHRR3kr5alVVWiSGs3/y1Mpp/sXUc5c/TAiDsvMBzpct8liNG18MnBpZr4/Ip5NGaPlsMzc1vnq1caYc28i9fjHMtwvOxruVyMaTRsTEccC5wInZObmiuo2WeysjWcBhwH9EXEr5ZrdMm/wG7PR/nvxzcx8ODN/C6yk/BDQ6IymjU8DvgyQmT8BplPG8Vf7jOrf7WYTKfgd7rfzdtrGjdPQn6CEvtdEx26HbZyZGzJzTmY+OTOfTLmP4oTMHPe43DU1mn8vLqfcrEpEzKGc+l9daS0nttG08RrghQAR8RRK8N9TaS0nv2XA3zTu7n8WsCEz79zRFybMqf50uN+OG2UbXwTMBL7SuG9yTWae0LVKTzCjbGO1aJTtfAXwFxFxM/AIcHZmru9erSeWUbbxmcAnI+KtlNPPp9oZG5uI+CLlctScxr0S7wKmAWTmxyn3ThwPrAI2Aq/faZn+N5AkqT4m0ql+SZLUIoNfkqQaMfglSaoRg1+SpBox+CVJqhGDX6qBiDi18aa/wWlLRPwmIv45IqZ3uW63RsSlTeuDdX1y1yolTWIT5jl+SW3xaspIX7OAlwNvbyw7hrpUEwa/VC/XZ+aqxvKVEbEAOC0i/o/jp0v14Kl+qd5+BuxG0/jpjSFYvxAR90TE5oi4PiJePvSLEXF4RHwjItZHxEMRsTIi3t60/S8iYnlE3BkRGyPixog4MyKmVPOnSRqOPX6p3p5MeX31eoCImAdcC6wD3koZV/2vgK9FxMsGhxSOiKOAfsowoW+lXD5YADytqez5wFXA/wU2Ud7meB4wFzino3+VpBEZ/FK9TGm8uXLwGv8rgX/IzEca28+jvObzmKZx669o/CA4n+0vYflXyo+FZ2XmxsZn32s+UGMccQCivNjhh8AuwFkR8Q4vLUjdYfBL9fKrIesfzcyPNK0fR3npx4bGD4RBVwAXRcQewFbgucBFTaH/GBHxRMoPieOAJ/Hof2/2Bu4a7x8hafwMfqleXk45LT8XOAN4c0Rcm5mfbWzfG/ibxjScvYAtlPuD1o50kIjoo5wdeBIl/H8FPAS8DDiX8npWSV1g8Ev1cuPgXf0R8T3gBkpP/muZ+QfK6fsfAv8ywvfvoLyCdRuw7w6OcxDlmv7/zMzPD34YES9t/U+Q1Arv6pdqKjM3A2dTevlvbnz8n5Qb9G7KzOuGmTY3Tu//CDglInYbofgZjfnDgx9ExDTgtR35YySNmj1+qcYyc1lErKDccPcRYAnw38DVjfVbgT2Bw4D5mfm3ja+eBfwA+ElEvJ9y2n8+8PTMfAvwS+B3wHsi4hHKD4C3VveXSRqJPX5J76T0+t+YmWsop+h/AfwzcCXwMeAYmu7az8wVlBv8bqM8rreccvZgbWP7Fsr1/LuAzwIXA1cD76vkL5I0osjMbtdBkiRVxB6/JEk1YvBLklQjBr8kSTVi8EuSVCMGvyRJNWLwS5JUIwa/JEk1YvBLklQjBr8kSTXy/wHESqPz/LiTYwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_precision_vs_recall(precisions, recalls):\n",
    "    plt.plot(recalls, precisions, \"b-\", linewidth=2)\n",
    "    plt.xlabel(\"Recall\", fontsize=16)\n",
    "    plt.ylabel(\"Precision\", fontsize=16)\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.grid(True)\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "plot_precision_vs_recall(precisions, recalls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# The ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeMAAAFlCAYAAADYnoD9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deVxVdeL/8deHRcR9yzWX0szMEo2S1IIsTdM000rNFrPcMrdSa5oaa2wxdysry/I7/UwdS5PSyZoSd80NnbTJjFzQcskNBFk/vz9ABpT0KhfOvYf38/HgIffew+HNB+TN53POPddYaxERERHnBDgdQEREpLhTGYuIiDhMZSwiIuIwlbGIiIjDVMYiIiIOUxmLiIg4LMipT1ylShVbr149r+3v1KlTlC5d2mv7K640jgWnMSw4jWHBaQwLzttjuGnTpiPW2svye8yxMq5Xrx4bN2702v5iYmKIiory2v6KK41jwWkMC05jWHAaw4Lz9hgaY/b82WNaphYREXGYylhERMRhKmMRERGHqYxFREQcpjIWERFxmMpYRETEYSpjERERh6mMRUREHKYyFhERcdgFy9gY86Ex5pAx5oc/edwYY6YZY3YZY7YZY5p7P6aIiIh7eTIzngW0P8/jHYCrst/6Ae8UPJaIiEjxccFrU1trVxhj6p1nky7AP6y1FlhnjKlgjKlhrf3NSxmLjdT0TBJOp5GSnkl6hiU9M5P0TJvzflqGJSPTkp6RSVqmJSP7Pmst1oKF7H8t5LwPWd+arNuQ9bi1eR+3WQ/wY3wav3+/N8++8mxn/7f92fvPzCdHzue0ud7P9TXbXDdsrkfy3p//B1zqfv5sW2/ZtzeVtck/en2/xYnGsOA0hgWTnpbKb7+lck3z01QrV7LQP583XiiiFrAv1+347PvOKWNjTD+yZs9Uq1aNmJgYL3z6LImJiV7dn6fSMi3JaZCUbklKz3o/JcOSmgEpmVn/pp65nQGns7dLSofktOyPSYekNEtqZpHHz98P/3E6gf/7Nc7pBP5PY1hwGsNLcnrfDxz5chJV7h5JRI1A6pQLLPTP6Y0yNvncl+90w1o7A5gBEB4ebr35ahjeeHUNay3HktL4/cRpDp48zZHEFE4kp3HydDonk9P441Qqx06l5vx7LCmVlHTvNWiAgbIlgwkNDiQo0BAUYAgKDMj+1xAUEEBwoCEwwBCcfX9gQACB2QcbDAZjyHo7820xWd8gY0z2vzl359x3ZtMAYzj4++/UqFE9z77AEJBrvybXPsneZ0Cu/efeb+7tA8z/flRyvZv3Byj3Nvnf/b+v7Tz7yXO/yfMZzrs/b4iL+4Urr6zv1X0WNxrDgtMYXrqft55kzuqytGtQmrvatKZKmZBC/5zeKON4oHau25cDB7yw30J1Oi2DjbuPsXnvMbbFH2f3H0nEH0vidNrFlWtQgKFcaDBlSwZRrmTWv6VKBBFaIpCSQQGElggkNDiQktlvZUoGUS7Xtrk/tlSJwHyLoyjFxBwjKqqpoxn8XQz7iIrSL8GC0BgWnMbw4uzdu5evv/6axx9/HKLqM37w/axcubJIihi8U8bRwGBjzFygBXDCV48XHziezNwN+9i85xgb9xzNt3jLlgyiermSVC9fksvKhFAuNJjyocGUCw2mUulgKpUOoVKpElQqU4KKpbJmsU4XqIiIXLqFCxfSt29fMjIy6NKlC5dddhmBgYW/NJ3bBcvYGDMHiAKqGGPigb8BwQDW2neBJcBdwC4gCehTWGEvRWp6Jgs2x7Ngy3427zlGeub/VtAb1yhHxJWVaVq7PA2rlaVWxVDKlQx2MK2IiBSV5ORknn76ad555x1uuOEG5s6dy2WXXeZIFk/Opu55gcct8KTXEnnRtvjjPDVnC3v+SAIgMMDQuWlN7rquBs3rVKBqEZwhJyIivicjI4PIyEg2bNjA008/zauvvkqJEiUcy+ONZWqfFH8siYdmfs+J5DQaVC3DoKj6tGlUlQqlnBtsERFxlrUWYwyBgYEMHDiQl156iQ4dOjgdy51lnJlpGfXpNk4kp9GmUVXe6d2ckKCiXf8XERHfcvz4cfr160e3bt144IEH6NPHd46quvLa1P9v/R7W/PIHlUuX4I3u16uIRUSKuXXr1tGsWTMWLFjAwYMHnY5zDteVcXpGJm8v2wXA2HuaFNlp6SIi4nsyMzN5/fXXad26NQCrVq1iyJAhDqc6l+vK+Nv/HuLgyRSuqFKa9k2qOx1HREQc9O233/Lcc8/RrVs3tmzZQkREhNOR8uW6Y8YLN+8HoOdNtfX8XxGRYmr//v3UqlWLtm3bEhMTw6233urTneCqmXFqeibLdx4GoNP1NR1OIyIiRS01NZVnnnmGBg0a8MMPWa/8GxkZ6dNFDC6bGe88mEByWgZXVClNzQqhTscREZEitGvXLnr27MnGjRsZNGgQ9ev7z+VAXVXGP+w/AcC1Ncs5nERERIrS7NmzGTBgAMHBwSxYsICuXbs6HemiuKqM//3jIQBaXFHJ4SQiIlKUtm7dSlhYGLNnz6ZOnTpOx7lorirj2H3HAYi6uqrDSUREpLBt2bKFlJQUIiIieOWVVzDGEBTkn7XmmhO4ktMtRxJTCAkKoJaOF4uIuJa1lqlTpxIREcHw4cOx1hIcHOy3RQwuKuMjyVmvxlSrYigBAb591pyIiFyaI0eO0LlzZ4YNG8add97JF1984fNnSnvCf/+MOMuJlKzXJq5RXq/EJCLiRrt376ZVq1YcOXKEqVOn8tRTT7miiMFFZZyQmvWvXpVJRMSd6tSpQ+fOnenXrx/NmjVzOo5XuWaZ+nhK1jJ11bK6FrWIiFvs3buXzp07Ex8fT0BAAO+8847rihhcVMZJaVllXCFUM2MRETdYsGABTZs2ZdmyZezYscPpOIXKNWV8KruMy4e6ZuVdRKRYSk5OZtCgQXTr1o0GDRoQGxtLu3btnI5VqFxTxidSs8r4srI6gUtExJ+99NJLvPPOOzzzzDOsXr3ary5realcM408nZ5VxmVKuuZLEhEpNqy1nDhxggoVKvDcc89x++2307ZtW6djFRnXzIyT07P+LVUi0NkgIiJyUY4fP84DDzxAVFQUp0+fpnz58sWqiMFFZZyYfcy4UmmdwCUi4i/Wrl1LWFgYCxcupGfPnpQoUTx/h7umjM+cwFVRzzMWEfF5GRkZvPrqq9xyyy0YY1i1ahWjR48mIMA1tXRRXPNVp2iZWkTEb6SlpTFv3jy6d+9ObGwsLVq0cDqSo1xxtlNGpiXdgjEQEuSavy9ERFznm2++oUWLFpQrV47ly5dTvnx511zSsiBc0VxpGVnXpQ4ODNA3VUTEB6WmpvL000/Trl07Xn/9dQAqVKig39nZXDEzTknPKuMSga7420JExFV27dpFjx492LRpE08++SQvvvii05F8jivKODk1A4BQHS8WEfEpS5cupXv37gQHB7NgwQK6du3qdCSf5Iqp5Om07DIOVhmLiPiSxo0b06ZNG2JjY1XE5+GKMj6zTK2Tt0REnLd582YGDRpEZmYmtWvXZtGiRdSpU8fpWD7NFe2VnplVxkE6Ziwi4hhrLVOmTCEiIoJFixaxb98+pyP5DVe015llas2MRUSccfjwYe6++26GDx9O+/bt2bp1K3Xr1nU6lt9wxQlciSlZZVwmxBVfjoiIX7HW0qlTJ2JjY5k2bRqDBw/WU5YukivaKyPzzPOM9c0XESkq6enpWGsJDg5m8uTJlCpVirCwMKdj+SVXrOumZWRdlzowQGUsIlIU9uzZQ2RkZM5zhlu2bKkiLgBXlHFGZlYZBxXTC4yLiBSlzz77jLCwMP7zn/9w/fXXOx3HFVzRXunZZRyoZWoRkUKTnJzMgAED6N69O1dddRVbtmyhZ8+eTsdyBVeUsbVZZRygEwZERArNL7/8wv/93/8xatQoVq1aRf369Z2O5BquOIHrDFWxiIh3WWtZsWIFkZGRNGnShF27dlGrVi2nY7mOK2bGmdkzY02MRUS85/jx49x///1ERUXx7bffAqiIC4krZsbZr6BIoNpYRMQr1qxZQ69evdi/fz/jxo3jtttuczqSq7ljZpx9AleAntokIlJgU6ZM4dZbbyUgIIBVq1YxatQoAvRslULlitFNz3lqk8pYRKSgqlWrxn333ceWLVto0aKF03GKBVeUcYbVzFhEpCCWLFnCrFmzAOjZsyeffPIJ5cuXdzZUMeKKMj7z1CYdMxYRuTgpKSmMGDGCjh078s4775CRkXWtf11bumi5oozPHDPWz46IiOd+/vlnWrZsyeTJk3nyySdZvnw5gYGBTscqllxxNnV2F+uiHyIiHjp06BA33HADQUFBLFy4kHvuucfpSMWaK8r4zLWp9UIRIiLnl5GRQWBgIFWrVmXChAl06NCB2rVrOx2r2HPHMnXO5TAdDiIi4sM2b95MkyZNWLVqFQD9+vVTEfsIV5Rx9iq1TjgQEcmHtZYpU6YQERFBYmKiflf6II/K2BjT3hjzkzFmlzHm2Xwer2OMWWaM2WKM2WaMucv7UT3I6cQnFRHxYYcPH6ZTp04MHz6cu+66i9jYWFq1auV0LDnLBcvYGBMIvA10ABoDPY0xjc/a7K/AP621zYAewHRvBz0fay+8jYhIcfTJJ5/w7bff8tZbb7Fw4UIqV67sdCTJhycncN0E7LLWxgEYY+YCXYAdubaxQLns98sDB7wZ8kLsmYVqTY1FREhPT2f37t0APPXUU3To0IGGDRs6G0rOy5Nl6lrAvly347Pvy20M0NsYEw8sAZ7ySjoP2ZwuVhuLSPG2Z88eIiMjGTp0KMeOHSMgIEBF7Ac8mRnn13BnLwz3BGZZaycaY24GPjbGNLHWZubZkTH9gH6Qde3TmJiYS4h8rri4VAD27d1LTMzvXtlncZWYmOi170txpTEsOI3hpVm+fDkTJkwgMzOTQYMGsXXrVqcj+bWi/Dn0pIzjgdznvl/OucvQfYH2ANbatcaYkkAV4FDujay1M4AZAOHh4TYqKurSUp/lh8yf4eed1K1bh6ioRl7ZZ3EVExODt74vxZXGsOA0hhcnPT2dwYMH895773HTTTcxZ84c9u7dqzEsoKL8OfRkmXoDcJUx5gpjTAmyTtCKPmubvcDtAMaYa4CSwGFvBj0fqytwiUgxFhgYyIkTJxg1ahQrV67kyiuvdDqSXKQLzoyttenGmMHAUiAQ+NBau90Y8zKw0VobDTwNvG+MGU7WEvaj1hbdOc5nLoepLhaR4sJay/vvv09kZCRXX301s2fP1msO+zGPLodprV1C1olZue97Mdf7OwDHnrh25gpceiK7iBQHx44d44knnuCzzz5jyJAhTJ06VUXs51x1beogXQ9TRFxuzZo19OzZkwMHDjBu3DieeeYZpyOJF7iijM9QFYuImy1dupSOHTtSp04dVq1aRYsWLZyOJF6idQ0RER935hScyMhIRo4cyZYtW1TELuOKMrbnPO1ZRMQdFi9eTMuWLTl58iQlS5bktddeo3z58k7HEi9zRxnrbGoRcZmUlBSGDx9Op06dSEpK4ujRo05HkkLkijI+Q2dTi4gb7Ny5k5YtWzJlyhSeeuop1q9fT7169ZyOJYXIFSdwaZFaRNxk2LBh7N69m0WLFtG5c2en40gRcEcZq41FxM8lJCSQmppK5cqVmTFjBgCXX365w6mkqLhsmdrpBCIiF2/Tpk00b96cRx55BMgqYRVx8eKqMhYR8SfWWiZPnszNN9/M6dOnGTVqlNORxCGuWKYWEfE3R44c4ZFHHmHJkiV06dKFmTNnUrlyZadjiUM0MxYRcchPP/3EW2+9xcKFC1XExZwrZsa66IeI+IO0tDRmzJhBv379qFKlCjt27KBEiRJOxxIf4KqZsdHVqUXER+3evZtbb72VwYMHEx2d9ZLwKmI5wx1lrImxiPiw+fPnExYWxvbt25kzZw7dunVzOpL4GHeUcTY9tUlEfM3YsWO5//77ufrqq4mNjaVHjx5ORxIf5JJjxiIivqlTp06cOnWKl19+meDgYKfjiI9yRRmfoYmxiDjNWst7773Hjh07mDZtGmFhYYSFhTkdS3ycq5apRUScdOzYMe677z4GDhzIzp07SUlJcTqS+AmVsYiIF6xevZqwsDAWLVrE+PHjWbJkCSEhIU7HEj/hqmVqEREnJCQkcPfdd1OxYkXWrFnDjTfe6HQk8TOuKGOrl20SEQccOXKEypUrU7ZsWaKjo7n++uspV66c07HED7limfpMF+upTSJSVBYvXsw111zDe++9B0Dr1q1VxHLJ3FHG2f8GqI1FpJClpKQwfPhwOnXqRK1atYiKinI6kriAK5apM7VMLSJFYOfOnfTo0YMtW7bw1FNP8cYbb1CyZEmnY4kLuKKM/7dMrZmxiBSeuLg49u3bx6JFi+jcubPTccRF3LFMnd3GAepiEfGyhIQEFi1aBED79u2Ji4tTEYvXuaOMs//VMWMR8aaNGzfSrFkz7r//fvbv3w9A2bJlHU4lbuSOMtbZ1CLiRZmZmUyaNImWLVuSkpLCv//9b2rVquV0LHExVxwzFhHxFmstXbt2JTo6mnvuuYeZM2dSqVIlp2OJy6mMRURyMcYQGRlJu3btGDRokE4MlSLhijK2ehFFESmAtLQ0/va3v9GyZUs6derEiBEjnI4kxYwrjhmfob9fReRi7d69m1tvvZXXXnuNFStWOB1HiilXzIxFRC7F/PnzeeKJJ7DWMnfuXB544AGnI0kx5YqZsS7AJSIXa+XKldx///00atSI2NhYFbE4yhVlnEMnWojIBZw6dQrIemGH2bNns3LlSq644gqHU0lx564yFhH5E9Za3n33XerVq8fOnTsxxtCrVy+Cg4OdjiaiMhYR9zt27Bjdu3dn4MCB3HDDDZQvX97pSCJ5qIxFxNVWr15NWFgY0dHRTJgwgSVLllCtWjWnY4nkobOpRcTV5syZQ3BwMGvWrOHGG290Oo5IvjQzFhHX2b9/P//5z38AGD9+PJs3b1YRi09TGYuIq3z55Zc0bdqUhx56CGstoaGhlCtXzulYIuelMhYRV0hJSWHYsGHcfffd1K5dm3nz5um60uI3dMxYRPzewYMH6dChA1u2bGHIkCGMGzeOkiVLOh1LxGOaGYuI36tcuTJ16tRh0aJFTJ06VUUsfkdlLCJ+6eTJkzz11FMcOnSIoKAgPv/8czp37ux0LJFLojIWEb+zceNGmjdvzvTp01m2bJnTcUQKTGUsIn4jMzOTiRMn0rJlS1JTU1m+fLle4EFcwRVlrBdtEikeXn31VZ555hk6depEbGwsrVu3djqSiFe46mxqPYlBxJ3S0tIIDg5mwIAB1KhRg8cee0xPWxJXccfMWFNjEVdKS0vjueeeIyoqirS0NKpUqULfvn1VxOI6HpWxMaa9MeYnY8wuY8yzf7LN/caYHcaY7caYT7wb0zP6/yniHrt37+bWW2/l9ddfp3HjxqSnpzsdSaTQXHCZ2hgTCLwNtAXigQ3GmGhr7Y5c21wFPAe0stYeM8ZULazAIuJ+MTExTJkyBWstc+fO1Ula4nqezIxvAnZZa+OstanAXKDLWds8AbxtrT0GYK095N2YIlJcpKSk8OGHH3LNNdcQGxurIpZiwZMTuGoB+3LdjgdanLVNQwBjzGogEBhjrf3q7B0ZY/oB/QCqVatGTEzMJUQ+128HUgDY+dNOYpJ/9co+i6vExESvfV+KK43hpfn111+pUaMGJUuWZMyYMdSpU4c9e/awZ88ep6P5Jf0cFlxRjqEnZZzfkdizT5kKAq4CooDLgZXGmCbW2uN5PsjaGcAMgPDwcBsVFXWxefO19Oh/IH4vDa9uSFSLul7ZZ3EVExODt74vxZXG8OJYa3n33XcZMWIEgwcPZvz48QAawwLSz2HBFeUYerJMHQ/UznX7cuBAPtssstamWWt/BX4iq5xFRP7U0aNH6datG4MGDSIqKoqRI0c6HUnEEZ6U8QbgKmPMFcaYEkAPIPqsbT4HbgMwxlQha9k6zptBz0/PbRLxNxs2bCAsLIwvv/ySCRMmsHjxYqpW1bmfUjxdcJnaWptujBkMLCXrePCH1trtxpiXgY3W2ujsx9oZY3YAGcBIa+0fhRk8P0aX/RDxGxUrVqRKlSosWLCA8PBwp+OIOMqjK3BZa5cAS86678Vc71tgRPZbkdNFP0T8Q3x8PB999BF//etfadCgAZs2bdIFPERw2RW49H9axHdFR0fTtGlTxo0bx88//wygIhbJ5o4yzj5mrP/WIr7n9OnTDBkyhC5dulC3bl02bdpEw4YNnY4l4lPc9UIRamMRn3PPPfewdOlShg4dyrhx4wgJCXE6kojPcVUZi4hvsNnHjowxPP300wwePJhOnTo5nErEd6mMRcSrTp48ycCBA2ncuDHPP/88bdu2dTqSiM9zxTFjEfENGzZsoHnz5sydO1cnZ4lcBJWxiBRYZmYmEyZMoGXLlqSmprJ8+XL+8pe/OB1LxG+ojEWkwHbs2MGzzz7L3XffTWxsLK1bt3Y6kohf0TFjEblkP//8M1dddRVNmjTJubyllqdFLp5mxiJy0dLS0njuuedo1KgR33zzDQDNmjVTEYtcIs2MReSi/Prrr/Tq1Yt169bxxBNP0KpVK6cjifg9lbGIeOyzzz7jsccewxjDP//5T+677z6nI4m4gspYRDx29OhRGjduzJw5c6hXr57TcURcQ8eMReS8tm3bxqJFiwB4/PHHWblypYpYxMtUxiKSL2stb7/9NjfddBMjR44kPT0dYwxBQVpQE/E2lbGInOPo0aPce++9DB48mDZt2rBq1SqVsEgh0v8uEcnj2LFjhIWF8fvvvzNx4kSGDRtGQID+bhcpTCpjEcmjYsWK9O/fn/bt23PDDTc4HUekWNCfuyJCfHw87dq1Y9OmTQA8//zzKmKRIqQyFinmoqOjadq0KWvWrGHPnj1OxxEpllTGIsXU6dOnGTJkCF26dKFu3bps3ryZe++91+lYIsWSylikmPrggw948803GTp0KGvXrqVhw4ZORxIptlxxApe1TicQ8Q/WWg4ePEj16tUZMGAA1113HZGRkU7HEin2XDUzNugVY0T+zMmTJ+nduzfNmzfnjz/+ICgoSEUs4iNcUcaaGIuc34YNG2jevDnz5s3jySefpEKFCk5HEpFcXFHGOTQxFskjMzOTCRMm0LJlS9LS0li+fDnPP/88gYGBTkcTkVzcVcYikocxhuXLl9O5c2diY2P12sMiPkoncIm40DfffMNVV11FvXr1mDdvHqGhoRijpSMRX+WqmbF+1Uhxl5aWxujRo2nXrh1jxowBoFSpUipiER/njpmxTuESIS4ujp49e/L999/Tv39/Jk2a5HQkEfGQK8r4DP31L8XV2rVrad++PcYY5s+fT/fu3Z2OJCIXwVXL1CLF1XXXXZdzkpaKWMT/uKOMtUotxdC2bdvo3r07SUlJlClTho8//ph69eo5HUtELoE7yjibFqmlOLDW8vbbb3PTTTexevVq4uLinI4kIgXkqjIWcbujR4/StWtXBg8ezO233862bdto0qSJ07FEpIBUxiJ+5LHHHmPJkiVMmjSJL774gssuu8zpSCLiBSpjER+XkZFBYmIiABMmTGDt2rUMHz6cgAD99xVxC1c9tUnEbeLj4+nduzdVqlRh/vz5NGjQwOlIIlII9Ke1iI+Kjo6madOmbNy4kc6dO+t59CIupjIW8TGnT59myJAhdOnShbp167J582Yefvhhp2OJSCFSGYv4mOPHjzNv3jyGDRvG2rVradiwodORRKSQ6ZixiA+w1vLll19y1113Ub16dX788UcqVarkdCwRKSKaGYs47MSJE/Tq1YvOnTsze/ZsABWxSDGjmbGIg77//nt69OjB3r17eeWVV3jwwQedjiQiDtDMWMQhH3zwAa1atSIjI4MVK1bwl7/8hcDAQKdjiYgDVMYiDrn22mvp1q0bsbGxtGzZ0uk4IuIglbFIEfr6668ZO3YsADfffDNz586lYsWKDqcSEaepjEWKQGpqKqNGjeLOO+9k3rx5JCUlOR1JRHyIylikkMXFxXHLLbcwfvx4+vfvz/r16ylVqpTTsUTEh+hsapFClJycTKtWrUhOTmb+/Pl0797d6Ugi4oM8mhkbY9obY34yxuwyxjx7nu26G2OsMSbcexFF/E9KSgoAoaGhvPfee8TGxqqIReRPXbCMjTGBwNtAB6Ax0NMY0zif7coCQ4D13g4p4k927dpFWFgYH3/8MQCdO3emXr16zoYSEZ/mycz4JmCXtTbOWpsKzAW65LPd34E3gNNezCfiN6y1vPXWWwwaNIgTJ05Qq1YtpyOJiJ/w5JhxLWBfrtvxQIvcGxhjmgG1rbVfGmOe+bMdGWP6Af0AqlWrRkxMzEUHzs/vB7OWBP/73x+JSdjllX0WV4mJiV77vhQnJ0+e5I033mD16tWEh4fz/PPPExAQoLG8RPo5LDiNYcEV5Rh6Usb5vYiqzXnQmABgMvDohXZkrZ0BzAAIDw+3UVFRHoW8kOiDsXBgP40aXUPUDZd7ZZ/FVUxMDN76vhQnixYt4vvvv2fSpEmEhYVx2223OR3Jr+nnsOA0hgVXlGPoyTJ1PFA71+3LgQO5bpcFmgAxxpjdQAQQrZO4xO3S09NZu3YtAF26dGHXrl0MHz4cY/L7+1VE5M95UsYbgKuMMVcYY0oAPYDoMw9aa09Ya6tYa+tZa+sB64DO1tqNhZJYxAfs27ePNm3aEBkZya+//gpAnTp1HE4lIv7qgmVsrU0HBgNLgR+Bf1prtxtjXjbGdC7sgCK+ZtGiRYSFhbFlyxY+/PBDrrjiCqcjiYif8+iiH9baJcCSs+578U+2jSp4LBHfY61l2LBhTJs2jRtuuIE5c+Zw1VVXOR1LRFxAl8MU8ZAxhvLlyzNixAjWrFmjIhYRr9HlMEXOw1rLRx99RL169WjTpg0vvfSSTtASEa/TzFjkT5w4cYJevXrRt29fZs6cCaAiFpFCoTIWycf69etp1qwZ8+fP55VXXuEf//iH05FExMW0TC1ylk2bNtG6dWtq1arFihUraNmypdORRMTlNDMWyZaRkQFA8+bNeTboCx4AAByVSURBVOWVV4iNjVURi0iRUBmLAF9//TXXXnste/bswRjDqFGjqFChgtOxRKSYUBlLsZaamsqoUaO48847CQoKIjk52elIIlIM6ZixFFtxcXH07NmT77//nv79+zNp0iRKlSrldCwRKYZUxlJsvfHGG+zcuZP58+fTvXt3p+OISDGmZWopVk6dOsXu3bsBmDBhArGxsSpiEXGcZsZSbMTGxtKjRw9CQkLYvHkzZcqUoUyZMk7HEhHRzFjcz1rLm2++SYsWLUhISGDKlCkEBgY6HUtEJIdmxuJqJ06c4OGHHyY6OppOnTrx0UcfUaVKFadjiYjkoZmxuFpoaChHjhxhypQpREdHq4hFxCdpZiyuk56ezpQpU+jbty8VK1ZkxYoVWpYWEZ+mMhZX2bdvHw8++CArV66kdOnSDBw4UEUsIj5PZSyu8fnnn/PYY4+RlpbGxx9/TO/evZ2OJCLiER0zFleYPn06Xbt25corr2Tz5s0qYhHxK5oZi1+z1mKM4Z577uHAgQO8+OKLlChRwulYIiIXRTNj8UvWWmbOnEmXLl3IyMigZs2ajB07VkUsIn5JZSx+58SJE/Ts2ZPHH3+cU6dOkZCQ4HQkEZECURmLX1m/fj3NmjXj008/5ZVXXuHrr7/W6w6LiN/TMWPxG+np6fTu3ZvMzExWrFhBy5YtnY4kIuIVKmPxeQcPHqRChQqEhITw+eefU6tWLc2GRcRVtEwtPm3p0qVcf/31/PWvfwXg2muvVRGLiOuojMUnpaamMnLkSNq3b0/VqlXp06eP05FERAqNlqnF58TFxdGjRw82bNjAwIEDmThxIqGhoU7HEhEpNCpj8TnJycns37+fzz77jHvvvdfpOCIihU7L1OITEhMTmTlzJpB1XDguLk5FLCLFhspYHBcbG0t4eDhPPPEE27ZtAyAkJMThVCIiRUdlLI6x1jJt2jRatGhBQkIC3377Lddff73TsUREipyOGYtjHn30Uf7xj3/QqVMnPvroI6pUqeJ0JBERR6iMxTFdunShefPmDBkyBGOM03FERByjMpYik56ezt///ncqV67MkCFDdIKWiEg2HTOWIrFv3z5uu+02Xn75ZX744Qen44iI+BTNjKXQLVy4kL59+5KWlsbHH39M7969nY4kIuJTVMZSqHbu3Em3bt1o3rw5c+fOpUGDBk5HEhHxOSpjKRRHjx6lUqVKNGzYkMWLF3P77bdTokQJp2OJiPgkHTMWr7LW8sEHH1CnTh2WLVsGQIcOHVTEIiLnoTIWrzl+/Dg9evTgiSee4Oabb6ZRo0ZORxIR8QsqY/GKdevW0axZMz777DNee+01li5dSo0aNZyOJSLiF3TMWLxi3bp1AKxatYqIiAiH04iI+BfNjOWS/f777yxfvhyAoUOHsm3bNhWxiMgl0MxYLslXX33Fww8/THBwMHFxcYSEhFC2bFmnY4mI+CXNjOWipKamMnLkSDp06EC1atX45ptv9HKHIiIFpJmxeCwxMZHbbruNjRs3MmjQICZMmEBoaKjTsURE/J5mxuKxMmXKcPPNN7NgwQLefvttFbGIiJeojOW8EhMT6d+/P9u3bwdg2rRpdO3a1eFUIiLuojKWPxUbG0t4eDjvv/8+K1eudDqOiIhreVTGxpj2xpifjDG7jDHP5vP4CGPMDmPMNmPMt8aYut6PKkXFWsu0adNo0aIFCQkJfPfddwwYMMDpWCIirnXBMjbGBAJvAx2AxkBPY0zjszbbAoRba68HPgXe8HZQKTozZ85k6NChtGvXjq1btxIVFeV0JBERV/PkbOqbgF3W2jgAY8xcoAuw48wG1tplubZfB+gFa/1QUlISAA899BAhISH07t0bY4zDqURE3M+TZepawL5ct+Oz7/szfYF/FSSUFK309HReeOEFrrvuOhITEwkJCeGhhx5SEYuIFBFPZsb5/Ua2+W5oTG8gHIj8k8f7Af0AqlWrRkxMjGcpL+D3gykA/Pe/PxKTsMsr+ywuDh48yNixY/nhhx9o3749ycnJXvu+FFeJiYkawwLSGBacxrDginIMPSnjeKB2rtuXAwfO3sgYcwfwPBBprU3Jb0fW2hnADIDw8HDrrWOR0Qdj4cB+GjW6hqgbLvfKPouDBQsWMGDAADIyMpg9eza9evUiJiZGx4gLSGNYcBrDgtMYFlxRjqEnZbwBuMoYcwWwH+gB9Mq9gTGmGfAe0N5ae8jrKcXrrLVMnz6dBg0aMHfuXOrXr+90JBGRYuuCZWytTTfGDAaWAoHAh9ba7caYl4GN1tpoYDxQBpiffZxxr7W2cyHmlku0Y8cOKlasSI0aNZg3bx5ly5alRIkSTscSESnWPLo2tbV2CbDkrPtezPX+HV7OJV5mreWDDz5g6NChdO7cmblz51K5cmWnY4mICLoCV7Fw/PhxHnjgAfr160erVq2YMmWK05FERCQXvWqTy23fvp1OnToRHx/P66+/zsiRIwkI0N9gIiK+RGXscjVr1qRevXrMmTOHiIgIp+OIiEg+NEVyod9++42hQ4eSmppKxYoVWbZsmYpYRMSHqYxd5quvvqJp06a8//77bN682ek4IiLiAZWxS6SmpvLMM8/QoUMHqlevzsaNGzUbFhHxEypjl+jbty8TJ05k0KBBrF+/nsaNz35hLRER8VU6gcvPpaenExQUxOjRo7n33nvp2rWr05FEROQiqYz9VGJiIk8++STGGGbNmkWTJk1o0qSJ07FEROQSaJnaD23evJnmzZvz//7f/6Nu3bpkZmY6HUlERApAZexHrLVMmTKFm2++maSkJL777jteeuklXcRDRMTP6be4H/n999956aWXuPPOO4mNjSUyMt+XjRYRET+jY8Z+YMuWLYSFhVGjRg02bNhA/fr1yX51LBERcQHNjH1Yeno6L7zwAjfccAMffvghAA0aNFARi4i4jGbGPmrv3r306tWL1atX06dPHx544AGnI4mISCFRGfugL7/8koceeoiMjAw++eQTevbs6XQkEREpRCpjH1SqVCmuvvpqZs+eTf369Z2OIyIihUzHjH3Ejh07ePfddwFo06YNa9euVRGLiBQTKmOHWWuZMWMG4eHhvPTSS5w8eRJAJ2mJiBQjKmMHHT9+nAceeID+/fvTqlUrtmzZQrly5ZyOJSIiRUzHjB2SmppKixYtiIuL4/XXX2fkyJG6kpaISDGlMi5i1lqMMZQoUYJnn32Wxo0b06JFC6djiYiIgzQVK0IHDhygbdu2LFiwAIA+ffqoiEVERGVcVJYsWULTpk1Zs2YNSUlJTscREREfojIuZCkpKYwYMYKOHTtSs2ZNNm3aRO/evZ2OJSIiPkRlXMj+9a9/MXnyZJ588knWr1/PNddc43QkERHxMTqBq5DExcVx5ZVXcs8997BhwwbCw8OdjiQiIj5KM2MvS0xM5JFHHqFJkyb8/PPPACpiERE5L82MvWjz5s306NGDX375hRdeeIErrrjC6UgiIuIHNDP2kqlTpxIREUFSUhLfffcdY8aMIShIf+uIiMiFqYy9ZPfu3XTo0IGtW7cSGRnpdBwREfEjmroVwLJlyyhVqhQtWrRg/PjxBAYG6gUeRETkomlmfAnS09N54YUXuP322xkzZgwAQUFBKmIREbkkmhlfpD179tCrVy/WrFnDY489xrRp05yOJOL3Tp48yaFDh0hLSwOgfPny/Pjjjw6n8m8aw4LzdAyDg4OpWrVqgV51T2V8EX788UdatmxJRkYGn3zyCT179nQ6kojfO3nyJAcPHqRWrVqEhoZijCEhIYGyZcs6Hc2vaQwLzpMxtNaSnJzM/v37AS65kLVMfREaNmzII488wpYtW1TEIl5y6NAhatWqRalSpXSoR/yOMYZSpUpRq1YtDh06dMn7URlfwPbt22nTpg2//fYbgYGBTJkyhfr16zsdS8Q10tLSCA0NdTqGSIGEhobmHGa5FCrjP2Gt5b333iM8PJzt27ezZ88epyOJuJZmxOLvCvozrDLOx7Fjx7jvvvsYMGAAt9xyC1u3biUiIsLpWCIi4lIq43z89a9/ZdGiRYwbN46vvvqK6tWrOx1JRERcTGWcLSMjgyNHjgAwduxYVq9ezahRowgI0BCJiOdmzZqFMSbnrUSJEtSvX5+//OUvnD592pFMY8aM8ZlDAWePT+63f//7307HO8esWbP48MMPC/3z6KlNwIEDB3jooYc4ceIEa9eupWLFitx0001OxxIRPzZ//nwuv/xyEhISWLhwIa+99hoJCQm8+eabTkfzCWfGJ7fGjRs7lObPzZo1i/T0dB577LFC/TzFvowXL17Mo48+SlJSEm+++aZe3EFEvCIsLIwGDRoA0LZtW37++WdmzpzJ1KlTteJG3vHxppSUFEJCQry+38JWbH8iUlJSGDFiBJ06daJmzZps3LiRxx57zGeWckTEXZo3b05ycnLO4TCAw4cP079/fxo2bEipUqWoXbs2vXr1yrmAxBlnlpl//vlnOnbsSJkyZahbty4vv/wymZmZebbdsmULt9xyC5dddhm1atXi73//O9bac/KcPHmSwYMHU7NmTUJCQrj66quZPHlynm1jYmIwxvD555/Tv39/KlWqRMWKFRk+fDgZGRls2LCB1q1bU7p0aa699lqWLl3qtfH66aef6Nq1KxUqVCA0NJSIiAi++uqrfMflhx9+4M4776RMmTLcf//9OY8vWLCAiIgISpUqRYUKFbjvvvvYu3dvnn188sknNGvWjDJlylC+fHmuu+463nvvPQDuuusuli9fzurVq3OW0qOiorz2NeZWbKeBGRkZfP311wwePJjx48dTsmRJpyOJiIvt3r2b8uXLU7ly5Zz7jh49SsmSJXnttde47LLLOHDgABMnTqRVq1b897//Pef3UteuXenTpw/Dhw/niy++4G9/+xu1a9emT58+ABw5coQ2bdpQvXp13n33XSpUqMD48ePPKaDMzEw6duzI5s2befnll7nuuutYvHgxI0aM4PDhw7z66qt5th82bBj33nsv8+bNY8WKFYwdO5b09HT+/e9/M3LkSGrVqsXYsWO599572bNnD1WqVLngeGRkZJCenp5z2xhDYGAgkHXosHXr1pQtW5a33nqL8uXL8/bbb9OxY0e+/PJLOnTokGdfXbp0oW/fvowePTpn1eHdd99l4MCB9OnThxdffJGEhATGjBlDZGQk27Zto2zZsqxatYrevXszZMgQxo8fT2ZmJv/97385fvw4AJMmTaJ///5kZGTkFHRBLnl5PsWujD/99FPat29PmTJlWL9+PaVLl3Y6koic5bpXVjgdAYDdr3e85I89UzZnjhl/9tlnTJkyJadwAK6++mqmTp2a52NatWpFnTp1+Ne//kXXrl3z7PPpp5/OKd477riD7777jjlz5uTcN3nyZE6dOsXSpUupWLEiZcuWpW3bttStWzfPfpYsWcKqVav46KOPePTRRwFo164dp06dYuLEiYwYMSJPobZp04ZJkyYBWUvuixcv5q233mLlypW0bt0agBo1atC0aVMWL17MI488csHxadSoUZ7brVq1YtWqVUBWCR47doy1a9fmLGXfddddNG7cmOeff/6cMh4yZAhDhw7NuZ2YmMjo0aPp06dPnpOvWrRoQcOGDZk5cybDhg1j3bp1VKhQgSlTpuRs065duzwZy5UrR3p6eqE/vbXYLFMnJCTw8MMPc9999/HWW28BqIhFpNA0atSI4OBgKlWqRN++fenfvz+DBw8+Z7t33nmHpk2bUqZMGYKCgqhTpw6QtUx7to4d8/5x0KRJkzyz3rVr1xIREZGzD8j6PXf33Xfn+bgVK1YQEBBwzmV9e/fuTWpqKmvXrs1z/9nl16hRI0qXLp1TxGfuA9i3b9+5g5GPhQsXsmHDhpy3mTNn5skXERGR55hyYGAgPXv2JDY2lpMnT+bZ19l/tKxdu5aTJ0/y4IMPkp6envN2+eWX06hRI1asyPpj78Ybb+TYsWP07t2bL7/8MmdG7IRiMTPevHkzPXr04JdffmHMmDGMHDnS6Ugich7/ef5Wv3+Rg4ULF3L55Zdz+PBhJk2axPTp02nRogUPP/xwzjZvvvkmQ4YMYcSIEYwfP56KFSuSmZlJREREvk+DqlSpUp7bISEhebb77bffaNKkyTkfV61atTy3jx49SqVKlc450enMNRWOHj2a5/6KFSvmuV2iRAkqVKhwzn2Ax0/fatKkyZ+ewHX06FGaNWt2zv3Vq1fHWsuxY8fyLBfXqFEjz3ZnrhF9xx135Lv/M19PZGQk8+fP580338wp9MjISCZNmsT111/v0dfhLa4v4/nz5/Pggw9StWpVli1bxq233up0JBEpBnKXTZs2bbj++usZOXIk3bp1y1mVmzt3LrfffjsTJ07M+bhff/31kj9njRo1OHjw4Dn3n31fpUqVOHr0KKmpqTklCvD7778D5Dmu7YRKlSrlZMnt999/xxhzzh8lZ594eyb/rFmzuPbaa8/ZT+4/9Lp370737t1JTEwkJiaG0aNH0759e+Lj473xpXjM9cvUN954Iw888ABbt25VEYuII0JCQhg/fjyHDh1i+vTpOfcnJSURHBycZ9uPPvrokj/PzTffzLp16/IsFZ86dYovvvgiz3aRkZFkZmYyf/78PPfPnj2bEiVKOH7538jISNatW8fu3btz7svIyGDevHk0a9bsgqsmLVu2pGzZsuzatYvw8PBz3q6++upzPqZMmTJ06tSJ/v3789tvv/HHH38AWd+75ORkr359+XHlzPi7777jk08+4f3336devXp8/PHHTkcSkWKuc+fO3HjjjUyYMIHBgwcTGhpK+/btGTduHK+++io33XQT3333HZ9++uklf47hw4czffp02rVrx+jRo3POpj77VbE6dOhA69atGTBgAIcPH+baa69lyZIlfPDBBzz33HMenQ1dmIYPH86sWbNo27YtL730EuXKlWP69Ons3LmTxYsXX/Djy5Urx/jx43nyySc5fPgwHTp0oHz58uzfv5/ly5cTFRVFr169ePHFFzl48CC33XYbNWvWJD4+nmnTphEWFsZll11GQkICjRs3Zvr06cybN4/69etTtmzZfMu8oDyaGRtj2htjfjLG7DLGPJvP4yHGmHnZj683xtTzdlBPpKen8fzzz3PHHXewevVqDh8+7EQMEZF8jR07lkOHDvHuu+8C8OKLL9K/f38mT55M165d2bZtW4Geq1ulShW+/fZbqlSpwoABA3jyySdp3779OVePCggIyDnredy4cXTs2JHFixczadIkXnnllQJ9jd5Qs2ZNVq1axbXXXsvAgQPp3r07R48eZfHixbRv396jffTv35/o6Gh++uknHnroITp06MDf/vY30tPTCQsLA7LOrt69ezfDhw+nbdu2jB49msjIyDyFP3r0aG6//XYef/xxbrzxRvr3718oX7PJ78ngeTYwJhDYCbQF4oENQE9r7Y5c2wwCrrfWDjDG9AC6WmsfON9+w8PD7caNGwuaH4AR82L5Z8xmSq+Zzs5tm+jbty9Tp07V2dKXICYmptCe1F5caAwvzo8//sg111yT576EhAS/P4HLaRrDgrvYMczvZzk3Y8wma214fo95skx9E7DLWhuXvbO5QBdgR65tugBjst//FHjLGGPshZreS2xmJofmjyHo9DHmzJlDjx49iuLTioiIeIUnZVwLyP3EsXigxZ9tY61NN8acACoDR3JvZIzpB/SDrFPtY2JiLi31WY4cSaHqXUPo0bwa1atX99p+i6MzZxTKpdMYXpzy5cuTkJCQ576MjIxz7pOLozEsuIsdw9OnT1/y/31Pyji/izWfPeP1ZBustTOAGZC1TO2tpbyoKIiJCdHSoBdoibXgNIYX58cffzxnKVBLrAWnMSy4ix3DkiVL5vv8aE94cgJXPFA71+3LgQN/to0xJggoDxxFRERELsiTMt4AXGWMucIYUwLoAUSftU00cOZipN2B74rqeLGIiIi/u+AydfYx4MHAUiAQ+NBau90Y8zKw0VobDcwEPjbG7CJrRqwzqETEY9ZavXyp+LWCzj89uuiHtXYJsOSs+17M9f5p4L4CJRGRYik4OJjk5GRKlSrldBSRS5acnHzO1dQuhusvhykivq1q1ars37+fpKSkAs8uRIqatZakpCT2799P1apVL3k/rrwcpoj4jzOvvnPgwAHS0tKArKeIlCxZ0slYfk9jWHCejmFwcDDVqlXL80pSF0tlLCKOK1euXJ5fZDExMZf8FBHJojEsuKIcQy1Ti4iIOExlLCIi4jCVsYiIiMNUxiIiIg5TGYuIiDhMZSwiIuIw49ST7I0xh4E9XtxlFc56yUa5JBrHgtMYFpzGsOA0hgXn7TGsa629LL8HHCtjbzPGbLTWhjudw99pHAtOY1hwGsOC0xgWXFGOoZapRUREHKYyFhERcZibyniG0wFcQuNYcBrDgtMYFpzGsOCKbAxdc8xYRETEX7lpZiwiIuKX/K6MjTHtjTE/GWN2GWOezefxEGPMvOzH1xtj6hV9St/mwRiOMMbsMMZsM8Z8a4yp60ROX3ahMcy1XXdjjDXG6KzWfHgyjsaY+7N/HrcbYz4p6oy+zoP/z3WMMcuMMVuy/0/f5UROX2WM+dAYc8gY88OfPG6MMdOyx3ebMaZ5oQSx1vrNGxAI/AJcCZQAtgKNz9pmEPBu9vs9gHlO5/alNw/H8DagVPb7AzWGFz+G2duVBVYA64Bwp3P72puHP4tXAVuAitm3qzqd25fePBzDGcDA7PcbA7udzu1Lb8CtQHPghz95/C7gX4ABIoD1hZHD32bGNwG7rLVx1tpUYC7Q5axtugD/l/3+p8DtxhhThBl93QXH0Fq7zFqblH1zHXB5EWf0dZ78HAL8HXgDOF2U4fyIJ+P4BPC2tfYYgLX2UBFn9HWejKEFzrxYdHngQBHm83nW2hXA0fNs0gX4h82yDqhgjKnh7Rz+Vsa1gH25bsdn35fvNtbadOAEULlI0vkHT8Ywt75k/VUo/3PBMTTGNANqW2u/LMpgfsaTn8WGQENjzGpjzDpjTPsiS+cfPBnDMUBvY0w8sAR4qmiiucbF/s68JEHe3mEhy2+Ge/bp4J5sU5x5PD7GmN5AOBBZqIn8z3nH0BgTAEwGHi2qQH7Kk5/FILKWqqPIWqFZaYxpYq09XsjZ/IUnY9gTmGWtnWiMuRn4OHsMMws/nisUSaf428w4Hqid6/blnLvkkrONMSaIrGWZ8y1BFDeejCHGmDuA54HO1tqUIsrmLy40hmWBJkCMMWY3WceZonUS1zk8/f+8yFqbZq39FfiJrHKWLJ6MYV/gnwDW2rVASbKuuSye8eh3ZkH5WxlvAK4yxlxhjClB1gla0WdtEw08kv1+d+A7m30UXgAPxjB7ifU9sopYx+jOdd4xtNaesNZWsdbWs9bWI+u4e2dr7UZn4vosT/4/f07WCYUYY6qQtWwdV6QpfZsnY7gXuB3AGHMNWWV8uEhT+rdo4OHss6ojgBPW2t+8/Un8apnaWptujBkMLCXrLMIPrbXbjTEvAxuttdHATLKWYXaRNSPu4Vxi3+PhGI4HygDzs89922ut7exYaB/j4RjKBXg4jkuBdsaYHUAGMNJa+4dzqX2Lh2P4NPC+MWY4Wcurj2qC8j/GmDlkHQapkn1c/W9AMIC19l2yjrPfBewCkoA+hZJD3xMRERFn+dsytYiIiOuojEVERBymMhYREXGYylhERMRhKmMRERGHqYxFREQcpjIWERFxmMpYRETEYf8fh2ffxS6P26QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "0.9983436731328145"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "fpr_forest, tpr_forest, thresholds_forest = roc_curve(y_train_5,y_scores_forest)\n",
    "\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.plot(fpr_forest, tpr_forest, linewidth=2, label=\"Random Forest\")\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.grid(True)\n",
    "plt.legend(loc=\"lower right\", fontsize=16)\n",
    "plt.show()\n",
    "\n",
    "from sklearn.metrics import roc_auc_score\n",
    "roc_auc_score(y_train_5, y_scores_forest)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}