aula18-knn-avaliacao.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aula 18 &mdash; Classificação pelos K vizinhos mais próximos e Métricas de avaliação\n",
    "\n",
    "Renato Vimieiro\n",
    "\n",
    "rv2 {em} cin.ufpe.br\n",
    "\n",
    "maio 2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KNN &dash; K vizinhos mais próximos\n",
    "\n",
    "O método supervisionado de classificação pelos $k$ vizinhos mais próximos é talvez o método mais simples de classificação. Como vimos os classificadores Naive Bayes e Análise de discriminantes consistem em ajustar modelos para estimar a probabilidade condicional $P(c_i~|~X)$. Os classificadores para isso supunham que os atributos seguiam alguma distribuição pré-definida (e.g. normal). O KNN, por outro lado, estima essa probabilidade a partir dos dados. A ideia é usar os $k$ vizinhos mais próximos da amostra a ser classificada para tomar a decisão.\n",
    "\n",
    "A definição de vizinhos mais próximos é dada por uma função de distância (em geral, Euclidiana, mas qualquer distância é válida). A distância $\\delta(X,X_i)$ é a distância Euclidiana entre a amostra a ser classificada $X$ e uma amostra do conjunto de treinamento $X_i$ no espaço p-dimensional. Mais precisamente,\n",
    "\n",
    "$$\n",
    "\\delta(X,X_i) = ||X-X_i||_2 = \\sqrt{\\sum (X_j - X_{ij})^2}\n",
    "$$\n",
    "\n",
    "Pode-se demonstrar (ver Zaki e Meira Jr. seção 18.3) que \n",
    "\n",
    "$$\n",
    "P(c_i~|~X) = \\frac{K_i}{K}\n",
    "$$\n",
    "\n",
    "onde $K_i$ é a quantidade de vizinhos da classe $c_i$ entre os $K$ vizinhos mais próximos de $X$.\n",
    "\n",
    "Logo, a classe predita para a amostra $X$ é\n",
    "\n",
    "$$\n",
    "\\hat{y} = argmax{P(c_i~|~X)} = argmax{\\frac{K_i}{K}} = argmax{K_i}\n",
    "$$\n",
    "\n",
    "Em outras palavras, o kNN atribui a $X$ a classe da maioria de seus vizinhos mais próximos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "prostate = pd.read_csv(\"https://web.stanford.edu/~hastie/ElemStatLearn/datasets/prostate.data\",\n",
    "                       header=0,index_col=0,sep='\\t')\n",
    "prostate['prognostico'] = prostate.gleason.map(lambda x: x >=7 and 'mau' or 'bom')\n",
    "treino = prostate[prostate.train=='T'].drop('train',axis=1)\n",
    "teste = prostate[prostate.train=='F'].drop('train',axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
       "           metric_params=None, n_jobs=1, n_neighbors=4, p=2,\n",
       "           weights='uniform')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "model = KNeighborsClassifier(n_neighbors=4)\n",
    "attrs = ['lpsa','lcavol','lweight','lcp','lbph','svi']\n",
    "model.fit(treino[attrs],treino.prognostico.map(lambda x: x=='mau' and 1 or 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>prognostico</th>\n",
       "      <th>bom</th>\n",
       "      <th>mau</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row_0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "prognostico  bom  mau\n",
       "row_0                \n",
       "0             20   10\n",
       "1              5   32"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(model.predict(treino[attrs]),treino.prognostico)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>prognostico</th>\n",
       "      <th>bom</th>\n",
       "      <th>mau</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row_0</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "prognostico  bom  mau\n",
       "row_0                \n",
       "0              8    7\n",
       "1              2   13"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.crosstab(model.predict(teste[attrs]),teste.prognostico)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Métricas de avaliação de classificadores\n",
    "\n",
    "Como havíamos discutido antes, a tabela de contingência nos permite calcular diversas métricas de qualidade. Segue uma lista das métricas mais usadas\n",
    "\n",
    "- **acurácia**: $(TP+TN)/(TP+FP+TN+FN)$\n",
    "- **taxa de erro**: $(FP+FN)/(TP+FP+TN+FN)$\n",
    "- **precisão**: $TP/(TP+FP)$\n",
    "- **revocação/cobertura/sensitividade**: $TP/(TP+FN)$\n",
    "- **especificidade**: $TN/(FP+TN)$\n",
    "- **F-measure**: $2(precisao*sensitividade)/(precisao + sensitividade) = 2TP/(2TP+FP+FN)$ (média harmônica da precisão e sensitividade)\n",
    "- **MCC (Matthew's Correlation Coefficient)**: $(TP*TN-FP*FN)/\\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}$ \n",
    "\n",
    "A F-measure é uma boa métrica de avaliação pois leva em consideração a precisão do modelo em classificar corretamente as amostras positivas (mostra o quanto ele acerta e quantas amostras positivas ele classifica como tal). Seu valor máximo é 1, exatamente quando o modelo consegue classificar corretamente todas as amostras (FP=FN=0).\n",
    "\n",
    "O MCC também é bastante valioso principalmente quando há um desbalanço na distribuição das classes. Seu valor varia de -1 a +1, sendo +1 perfeita classificação, 0 aleatória, e -1 classificação completamente oposta.\n",
    "\n",
    "A maior parte dessas métricas podem ser computadas diretamente em `sklearn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "        bom       0.67      0.80      0.73        25\n",
      "        mau       0.86      0.76      0.81        42\n",
      "\n",
      "avg / total       0.79      0.78      0.78        67\n",
      "\n",
      "MCC treino = 0.547, Acuracia treino = 0.776\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "        bom       0.53      0.80      0.64        10\n",
      "        mau       0.87      0.65      0.74        20\n",
      "\n",
      "avg / total       0.76      0.70      0.71        30\n",
      "\n",
      "MCC teste = 0.424, Acuracia teste = 0.700\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import matthews_corrcoef, classification_report, accuracy_score, roc_curve, roc_auc_score\n",
    "pred_teste = model.predict(teste[attrs])\n",
    "pred_treino = model.predict(treino[attrs])\n",
    "real_teste = teste.prognostico.map(lambda x: x=='mau' and 1 or 0)\n",
    "real_treino = treino.prognostico.map(lambda x: x=='mau' and 1 or 0)\n",
    "print(classification_report(real_treino,pred_treino,target_names=['bom','mau']))\n",
    "print('MCC treino = {:.3f}, Acuracia treino = {:.3f}'.format(matthews_corrcoef(real_treino,pred_treino),\n",
    "                                                            accuracy_score(real_treino,pred_treino)))\n",
    "print(classification_report(real_teste,pred_teste,target_names=['bom','mau']))\n",
    "print('MCC teste = {:.3f}, Acuracia teste = {:.3f}'.format(matthews_corrcoef(real_teste,pred_teste),\n",
    "                                                            accuracy_score(real_teste,pred_teste)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Outra métrica muito popular é a curva ROC (Receive Operating Characteristic). Essa curva mostra a relação entre a proporção de verdadeiros positivos (sensibilidade) e falsos positivos (FPR=FP/FP+TN) quando se varia os limiares de corte para classificação. Como vimos, alguns modelos usam um limiar na probabilidade condicional $P(c_i~|~X)$ para classificar as amostras como positivas e negativas. Cada limiar escolhido resulta em um valor de sensibilidade e FPR. Os limiares são escolhidos com base nas probabilidades de pertencerem à classe positiva das $m$ amostras. Os valores são variados da menor pra maior probabilidade."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhcAAAF2CAYAAADdmboqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3Xd4FFXbx/HvbElvG7pSFIVDERWQYqUJgoJiQdBHLCgP\ngooK2MD2+mBH7GBHQRREKYogCAiiFAEFqQdQBOklm55Nsrvz/rER6STLbmaT3J/r4iI7mez8OI7J\nnTOnGKZpIoQQQggRKjarAwghhBCifJHiQgghhBAhJcWFEEIIIUJKigshhBBChJQUF0IIIYQIKSku\nhBBCCBFSUlwIIYQQIqSkuBBCCCFESDmsDlDa9u3LOrhqmM1mkJoaT1paDn6/LCYWatK+4SXtG17S\nvuEjbRte4WzfKlUSjWJlCOlVyxibzcAwDGy2YrWVKCFp3/CS9g0vad/wkbYNr0ho3wpdXAghhBAi\n9KS4EEIIIURISXEhhBBCiJCS4kIIIYQQISXFhRBCCCFCSooLIYQQQoRUhVvn4lCO72dDYgy0vszq\nKEIIIUS5EVHFhVIqGlgO3KO1/vE45zQFRgNNgDVAf631ryW+mNdL3OOPgsMOCxYjnThCCCFEaETM\nT9SiwuJzoNEJzokDvgUWAM2AxcC3SqnYkl4vdsz72DdthPXrif7o/SBTCyGEKAnTNNmw1c2Pv21n\nw1Y3pikrdIZSpLRvRPRcKKUaAp8V49ReQK7W+pGi1w8opa4EegBji3s9w51G3IgXDr6OefF5cq/t\ngelKLUFqIYQQJbFC72PSD5vZm5538FjVlFh6tDub5qqKhcnKh0hq30jpuWgDzAUuBE60Xmkr4Kcj\njv1c9HXFFv/Sc9jc7oOvbe404l5+viRvIYQQogRW6H2Mmrr6sB98AHvT8xg1dTUr9D6LkpUPkda+\nEdFzobV+55+PlVInOrUGgXEWh9oDNC7utewbNTGffHTU8diPP8RzR1989eoX962EKPNyPV52peVY\nHSMiOew2EjPzycry4PX5rY5TppmmyWdzNnK8HnrThM/mbCQ5wYlhyH4jJVWc9p00fzPN6lcutfaN\niOKiBOKA/COO5QPRxX2DhKeGYni9Rx03vF4SnxpK9heTTy2hOMhutx32twitU23fXI+Xh99ZRK7n\n6P8fhCht7qx8nhtX8rH5onj2uvP4c1cmqrarVK5X1ooLD0cXEtFAbnHfwOm0H/9z+HG54oNLJo4r\nKanE421FCQTbvnu3pklhIUQFUug3Su1nXFkrLnYA1Y84Vh3YVdw3yHjyfyTPmXPM3gvfxk1kL1+F\n/6yzTy2lAAK/USclxZKZmYdPupVD7lTbNyvLc/DjO69qSM2qCaGMV+bZbAbx8dHk5OTj98uMhlOx\ndXcWH8/ccNLzbu/SgDrVE0shUfmyZNtq5qzZgG/PGSc8z2kzcbtP7TFocYuTslZcLAEeOeLYRcCz\nxX2DwrMVntv6EPvhe0d9zr5tK4kd25I5ZjyFF196aknFQT6fH69XiotwCbZ9Dx1HUL1SHHWqyTf1\nQzkcNlyueNzuHLl/T1HtqgnMWLz1qMGGh6rqiuXSc2vImIsS8Hg9TN48nZ+zfiGqtoEny4WZm3zM\nc6u6YqlbI6nU7uWIfxiulKqmlIopevklkKKUelUp1VAp9ToQD3xRkvfMeXgofte/z538LhdZjz+N\nabNhS08nucc1xHz6Scj+DUIIUZEZhkGPdmdzvLrBMKBH27OlsCiBDWmbePaXV/l55y8AxNhjsTkL\nj3muFe0bicXFkf2Pu4AbAbTWWUBX4DICK3m2BLporY9fDh/rAq5Ucoc8evC155GheAYOInP8F/gT\nEgODOwfdR/xTw8DnO5V/ixBCCKC5qsKA7k2o6jp8jFBVVywDujeRdS6KyePNZ6Kewpsr3yfNE1hS\noXnV8/i/ix7i7vbtIqZ9jYq2Otq+fVmBf7DXS2qb1tgddtwLFuMtqrPsG9aTfMuN2LdtBSD/ii5k\njf4AM0G6jEtKupXD61Tb94+dGTw7dgUAw25tzlmnHbs7taKS+zc8TNPkj52ZeE0Dp82kbo0k6bEo\npi0ZW/l47efs96QBkOCMp6e6lmZVzz14Trjbt0qVxGK9WVkbcxE6Dge5w18gMTEGHA4o+ubha9AQ\n93c/kHz7zTh/WUL0rJnYu15BxqcT8desZXFoIYQo2wzDoEEdlxRuQbAbdtLy0wE4v8o59FLXkRh1\n+EDsSGnfSHwsUmq8HTtBly5HHTcrVyb9q2/w9OgFgGPdGlxXtMOxYllpRxRCCCEAqJ1Uk6vrduaO\nRjdx1zm9jyosIkmFLi5OKDqarLfeJWfokwDY9u0lpfuVRE/50uJgQgghKqqOddpyQfWmEf8oSYqL\nEzEMch8YQsaH4zBjYzHy80nq14e4l57juOusCiGEEBWcFBfFUNDtGtK//g5f9RoAxI94gcS7+0Be\niSapCCGEEMdV6Peyyf2H1TFCQoqLYvKe15T0WT9QeO75AMRM+YqU667C2LPH4mRCCCHKum2Z23lx\n2eu8tfIDdmbvtjrOKZPiogT8NU4jfdpM8q+6GgDniuW4OrfDvma1xcmEEEKURV6/l+l/zublFW+x\nK2cPXtPHoqKFscoyKS5KKj6ezA/Hknv/YADsO7bj6tqJqFkzLQ4mhBCiLNmetZOXl7/FzL/m4Df9\nRNmj6Fn/Wq6v183qaKes4q5zcSpsNnKGPYX37HokDroPIzeHpFt7kfPUcPL638tx17gVQghR4fn8\nPmZvnc/Mv+bgMwOrQNdLqcstDXtQObaSxelCQ3ouTkF+z5tJ/2o6/kqVMEyThKeHkTDoPigosDqa\nEEKICLUjexffbpmNz/ThtDm5od7VDGz633JTWIAUF6fM2/pC3DPn4VUNAIgdP5bkG7tjpB2wOJkQ\nJ3bo0v/b9mRR0bYCEMIqtZNq0r72pdRNrsPQlg/QrtYl2Izy9eO4fP1rLOI/40zSv/2egnYdAIha\n9BMpXTpg37zJ4mRCHNsKvY/RU9cefD1u1kYee3cJK/Q+C1MJUXFcXbczDzbrT9W48rlhmxQXIWIm\nJZMxfhK5d/UDwLHlT1K6dMD543xrgwlxhBV6H6OmrsadlX/Y8b3peYyauloKDCFKgcPmKHe9FYeS\nAZ2h5HCQ89zL+M6uT8Kwh7FlpJPc81qyX3gFz219rE5X6nI9XvZuTSMry4PXJ5sThZrDbiMxM79E\n7WuaJp/N2XjcBWZNEybN30yz+pUjfnlhISJZvq+AaHuU1TEsI8VFGHj69MVX9yyS7roNW2YGiQ89\ngH2TJuf/ngO73ep4pSLX4+XhdxaR6/FaHUWU0F53Hpu2Z1C/VorVUYQoc/ymnx+3L2bGX98zuPk9\nVCunjz1Opvz2yVissG170mfMwVfnDADi3htNUu+eGFmZ1gYrJbvScqSwKMPSs/NPfpIQ4jD78w7w\nxm/vMWnTNHIKc/l8w1dWR7KM9FyEka++wv3dDyT1uYWoxT8TPWc29q6dyBg3EX/tOlbHKzV3XtWQ\n6pXirI5R7jjsNhITY0r0WGTbnizGzdp40vNSEqJPNZ4QFYZpmvy0cwmTN39LgS+wFMHpCTW4od7V\nFiezjhQXYWZWqkTGpGkkPPQAsZ9/imP9Olyd25Hx8ed4W7ayOl6pqFk1gTrVEq2OUe44HDZcrnjc\n7hy83uIVF3VrJDFr6d/sTT/+pntVXbHUq5kcqphClGtpHjfj13/JBndgdqDNsHFFnfZ0PqM9DlvF\n/RErj0VKQ1QU2a+9TfYTz2AaBrb9+0m57iqiv5xodTJRwRiGQY92Zx93EVnDgB5tz5bBnEIUQ1ZB\nNs/98urBwuK0+Oo81PxeutbtVKELC5DiovQYBnn3PUDmmPGYcXEYBQUkDehL3PPPgF9mUojS01xV\nYUD3JlR1xR52vKorlgHdm9BcVcwBaEKUVGJUAi2rN8PAoFOddjzcYiC1k2paHSsiVOzSygIFV3Yl\n/ZtZJPXuhX3nDuJfHYFj82Yy33wH4mRcgigdzVUVmtWvzMa/08nIKSAlIZp6NZOlx0KIErrmrCtp\nVb05dZJqWR0lokjPhQW8Tc4jfdYPFDZtBkD0N1NJ6d4F2+5dFicTFYlhGKjaLlo2rEb9WilSWAgR\nhGh7lBQWxyDFhUX81aqTPmUGnquvBcC58jdSrmiHY/Uqi5MJIYQQp0aKCyvFxZH13hhyBj0MgH3X\nTlK6XUHUt99YHEwIIURWQTYfr/2c/XlpVkcpc6S4sJrNRu6jj5M56n3M6GiM3FyS7/gPsW+8ynHX\naBZCCBFWv+79neFLX2HZnt8Yv34SflMG3peEFBcRIv+GnqRPno6/cmCkfsLwp0gc2B/yZaVEIYQo\nLdkFOXy0ZjwfrvmU7MIcAKrHV8MnxUWJyGyRCOJt0Qr3rB9IvuVGHOvXETPxM2xb/wpMX61Uyep4\nQghRrq3at4bPN0wmqzAbgNQYF7c06IFKPdviZGWP9FxEGH+t2qR/+z35Ha8AIGrJIlxXtMOuN1ic\nTAghyie/6efjtRN4b/XYg4XFxae1YmjLB6WwCJIUFxHITEgkc+wEcvvdA4B921+kXHk5znlzLE4m\nhBDlj82w4SxaUTMlOpl7z7uLmxtcT6wjxuJkZZcUF5HKbifnf8+TNeJ1TIcDW1Ymyf/pQcyH71md\nTAghyp3r6nWlQ63LGNZyEA0r1bc6TpknxUWE89x6BxkTp+BPTsHw+Uh8bAgJjw4Gr2xnLoQQoRLr\niOG6el2Jc8ae/GRxUlJclAGFl7Yh/bu5eOueBUDsR++TfPMNGJkZFicTQgghjibFRRnhO6se6TPn\nUnDJZQBEzZ9HypWXY/tri7XBhBCiDNiQtgm3J93qGBWGFBdliOlKJWPCZPJuuQ0Ax0aNq3M7nEsW\nWZxMCCEik8ebzwQ9hTdXvs9nG77ClMUJS4UUF2VNVBTZr7xB9jPPYRoGtrQ0kq/vRvSE8VYnE0KI\niLLJ/QfP/fIqC3csBmBb1nYOeNwWp6oYZBGtssgwyLv7XnxnnkXi3Xdiy8kmaWB/cjdvImfok2CT\nmlEIUXEV+AqY9sdM5m//+eCx86ucQy91HYlRCRYmqzikuCjDCq7oQvr02ST37ol9+9/EvTES++ZN\nZL79HsTHWx1PCCFK3R/pfzFu/UT25R0AIM4RS8/63Wle7XwMw7A4XcUhv+KWcb7G5+CeOY/C5i0A\niJ7xDSlXd8a2c4fFyYQQovT9tHPJwcKiSeWGPN5qMBdUbyqFRSmT4qIcMKtVI33Kt3iuuwEA5+pV\npFzRDsfKXy1OJoQQpatHvaupHl+NWxv2pF+T20mOTrI6UoUkxUV5ERND1ugPyXl4KAD2PbtJuaYL\nUd9MtTiYEEKUnjhnHMNaPkirGs2lt8JCUlyUJ4ZB7pBHyXxvDGZMDEZeHsl33krcqy+DTL8SQlQQ\nNkN+tFlN/guUQ/ndryd96gz8VaoCEP/8/0gc0Bc8HouTCSHEqfH6veQW5lkdQ5yEFBfllLfZBbhn\n/YC3cRMAYr76gpTru2Hs22dxMiGECM7fWTt5afmbfLr+C1kMK8JJcVGO+WvWwv3NLPI7XwmAc9lS\nXF3aY1+/zuJkQghRfD6/j5lb5vDS8jfYkb2LVfvXsi5NWx1LnIAUF+VdQgKZY8aTe8/9ANi3bSXl\nqo5EzZllcTAhhDi5ndm7GbHiLaZvmY3f9OO0OelR7xoapsq26JFMFtGqCOx2cp76H7569UkYcj+2\n7CySbulJzjPPkde3P8iIaiFEhPH5fczd9iPfbpmN1/QBUDf5DHo3vJGqcZUtTidORoqLCsRzc298\ndc4gqc8t2NxuEh5/FPumTWQ/9xI4nVbHE0KIg2Zs+Z7vts4DwGlz0K1uZ9rVukRmgpQREVFcKKWi\ngVHAdUAu8IrWeuRxzr0WGA7UBn4D7tda/1ZaWcu6wosvxT1zHsm33Ihj8yZiP/kQ+59/kPnhJ5gp\nLqvjCSEEAG1rXcJPO5dSObYSvRveSPX4qlZHEiUQKSXgCKAZ0BYYADyllLruyJOUUo2A8cBzwLnA\nKuBbpVRM6UUt+/x1zyJ9xhwKLmsHQNTC+aR06YD9z80WJxNCiIDEqAQebNafQc36S2FRBlleXCil\n4oA7gYFa61Va62nAS8C9xzi9E7BGaz1ea70FeAyoDjQqtcDlhJniIuPzL8m77U4AHH9sJqVze5w/\nL7Q4mRBCBFSPr4rdZrc6hgiC5cUFcB6BxzOLDzn2E9DqGOceABorpS5SShlAHyAD+CPsKcsjp5Ps\nl0aS/eyLmDYbtvR0kntcQ8z4sVYnE0IIUYZFQnFRA9ivtfYecmwPEKOUqnTEuROBGQSKjwICPRw3\naK0zSiVpeWQY5PXtT+b4L/AnJGJ4vSQ+eC/xTw0Dn8/qdEKIcshv+pm1aQHj1k6yOooIk0gY0BkH\n5B9x7J/X0Uccr0TgMcgAYCnQH/hYKdVUa72/OBez2QxstsDUS7vddtjfFZn/is5kfTeHhJtvxL5t\nK3Gj38Sx5Q9y3vsIEhJK/H6OQ9rUZjNwOKSNQ03u3/CS9g2PA3lpjF01iQ0HNgHQuLKiWbVzLU5V\nvkTCvRsJxYWHo4uIf17nHnH8ReB3rfU7AEqpfsB64A7g5eJcLDU1/qid8pKSYksYuZy6uCUs+wWu\nvRYWLSLquxlEde0E33wDtWuX6K0SM/+tF+Pjo3G54kOdVhSR+ze8pH1DwzRN5v75M2NXfonHG/j+\nUCv5NOpWqynfH8LEyns3EoqLHUBlpZRNa+0vOlYdyNNapx9xbnPg9X9eaK1NpdQqoE5xL5aWlnNY\nz0VSUiyZmXn4fP6TfGUF4YyHL78m7oF7if5iAvz+O/4WLcj+dCK+C1oU+22ysv7dJC0nJx+3Oycc\naSs0uX/DS9o3dNyedMauncS6A4Eluw0Muje8gk612mPDJt8fQiyc925xC8FIKC5WAoVAa2BR0bFL\ngWXHOHcnR88MUcAvxb2Y32/i9x++4Y3P58frlW8eBzmiyHzzXeLOrk/8c89g27uXxG6dyXpjNPnX\n3lCst/AeckP7/aa0bxjJ/Rte0r6nZkPaJj5YM448b+AXjupxVbm9SS+andEQtztH2jaMrLx3LS8u\ntNZ5SqmxwDtKqT5ATWAwcDuAUqoakKG19gDvA2OUUssJzC7pS2AxrU+syF6uGQa5DwzBe9bZJN3b\nDyMvj6R+fcjZvIncIY/KkuFCiGIJrFFhYGDQofZldD2zE7HRRz4JF+WN5cVFkUEEVuicR2Bq6RNa\n66lFn9tFoNAYq7X+QikVDwwFTifQ69GuuIM5RckVdOtOeq3aJN16E/bdu4h/+XnsmzeS9dooiJVn\n0UKIE0uJTubWhjeSEBVP3eQzrI4jSklEFBda6zwCgzLvOMbnbEe8HgOMKaVoAvCe34z0WT+Q1LsX\nzt9XEjPlK+zbtpLx8eeY1apZHU8IEeHOrdLY6giilMkcK1Es/hqnkT5tJvlXdgPAuWI5rs7tsK9d\nc8zzTfPfcS1bd2cd9loIIUT5JsWFKL74eDI/Gkfu/YMBsO/YjuuqjkTNmnnYaSv0PkZPXXvw9ccz\nN/DYu0tYofeValwhRPhlF+SwcMcSq2OICCPFhSgZm42cYU+R+cZoTKcTIzeHpFt7ETvqTTBNVuh9\njJq6GndWPs23rKD5lhUA7E3PY9TU1VJgCFGOrNy3huFLX2GCnsya/eutjiMiSESMubBKrsfL3q1p\nZGV5Dps6KYrhsq4kvleV+oP64nSnkfD0MHJWrmbiOTdjmmDz+7hzwRhM4Lc65+O32TFNmDR/M83q\nVz5qITMhRNmRU5jLFxunsnzPyoPHtmRu45zKDS1MJSJJhS0ucj1eHn5nEbke78lPFsdhp9q1z/Hk\n1OHUTttOtakTeHD5Gp7v9gjt1i+gVtp2AK5cNZPpTbsCsNedx6btGdSvlWJlcCFEkFbvX8dnG74i\nsyALCMwGuaVBDxpWqm9xMhFJKmxxsSstRwqLENiTUp2HbnqRh6ePoPnW3zh3+xpGjh9CYn72wXNu\nWjyR+Q3akB2bCEB69pFbyQghIl1uYR5fbvqapbtXHDx2YY0WXF+vK7EOmZYuDldhi4tD3XlVQ6pX\nirM6RplWeMdFbPrf49SbNp4amXsO+1ySJ4ubF0/gvfZ9AUhJkAV0hChr8rweVu0LzA5Ljkri5gbX\ny2MQcVxSXAA1qyZQp1qi1THKPPO9UXydbtJtwWccOaKiy+/fMeP8LhScVY96NZMtySeECF6lWBfX\n1evK5vQt9Kh3NXFO+YVMHJ8UFyJkDMPgQs/2owoLAIffx50LPmL/nZNkMKcQZdRFNVpy8WmtrI4h\nygApLkRIJcdHHfdzZ8f6qKOqlGIaIUQoyS8GorhknQsRUtnPPI/pOHbNmrxuFfH/9wTky4BOISLR\n3lzZpkmEhhQXIqR89RWe2/ocddyMisIA4t5+HVfn9tjXryv9cEKIY8r3FfDFxmk8s+RlNqRtsjqO\nKAekuBAhl/PwUPwu18HXflcq7jk/UtCmHQCOtatxdWpD7Ltvg18WLxPCSpvTt/D8L6+yYPvPmJhM\n3fyt7AUkTpkUFyLkTFcquUMePfja88hj+Bo0ImPiFLKHv4AZHY2Rn0/CE4+R3KM7tp07LEwrRMVU\n4Cvkq03f8Nqv77Av7wAATSo3ov95d8rYCnHKpLgQYZF3R1989epDw4bk9wmsb4HNRt5/B+D+/ke8\njZsAELVwPq62FxI9bbKFaYWoWLZkbOX5Za8y7++FmJjEOmK4tWFP+jW5jeRomZYvTl1Qs0WUUucB\n9wMNgB7ANcA6rfX80EUTZZrDQe7wF0hMjAGHA7z/Pv7wNWiI+7t5xL/4LLFvv44tPZ2kvrfjmf0d\n2c+/jJkk62AIES65hXm8tfIDPL7AwOpGlRT/aXADKdHy/50InRL3XCilmgNLgLpAcyAaaArMVkpd\nGdp4oizzduwEXboc+5PR0eQ8+QwZU77FV7MWADGTJuBqdzHOJYtKMaUQFUucM5ZudTsTY4/mPw16\nMODcPlJYiJAL5rHIi8ArWuu2QAGA1rov8BbwdMiSiQqh8KJLcM9fhOeGngDY/95G8jVdiB/+NBQU\nWJpNiPLqspoX8kTrIVx0WgsZXyHCIpji4gJg7DGOvw00OrU4oiIyk5LJGvU+me+NwZ+cgmGaxL0x\nkpQuHbDrDVbHE6LcsRk26a0QYRVMcVEAJB3jeC0g59TiiIosv/v1uBcspuDSNgA4V6/C1fEyYj54\nR6asClECMpVUWC2Y4mIq8KxSKqXotamUagC8DkwPWTJRIflPO52MSdPIfua5wMJbHg+JQx8mudd1\n2HbvsjqeEBFvZ/ZuXlr+JpvTt1gdRVRgwRQXQ4AEYD8QD/wKrAV8wEOhiyYqLJuNvLvvxT17Ad6G\njQGImj8PV5vWRH0zzdpsQkQon9/HrL/m8eKy19mWtZ1x678g3yfjloQ1SlxcaK0ztdYXA1cADwPP\nA92AC7TWaSHOJyowX6PGuGfPJ3fAQEzDwOZ2k3xnbxIH9sfIyrQ6nhARY3fOHl75dRRf//kdXtOH\n0+agzekX4rTJ3pTCGiW+85RS84DrtNZzgbmHHK+qlJqltW4ayoCigouOJufp4RRc3onEe/th37mD\nmAnjcS76icy33sPb+kKrEwphGb/pZ97fC/nmz1l4/V4AzkyqTe+GN1ItvqrF6URFVqziomj9iguK\nXrYBhiqlso84rR5wRuiiCfGvwksuwz1/EQmPDiZm8pfYt20lpXsXcgcOCiw1HnX8rd6FKI9M0+TN\nlR+w0b0ZAIdhp2vdK+hQ+zJshiy+LKxV3J6LvwisY/HPhOheBMZY/MMEspExFyKMzBQXWe98REHH\nziQ8MhhbZgbxr40gat4cskZ/EFhuXIgKwjAMGqXWZ6N7M7UTa3Jro57UiK9mdSwhgGIWF1rrdQRW\n5EQptQVoobXeH85gQhxP/vU3UtjqQhLvu5uonxfi/H0lrssvJfvJ/+Hp0xdkUSBRQXSofRnxzjha\nVW+O3Wa3Oo4QBwUzoPPM4xUWSqmYU48kxMn5a9Yi46tvyH5qeGDKal4eiY8NIenmGzD27LE6nhCl\nwmbYuOi0llJYiIgTzIDOSsAwoAnwzx1tENhjpBGQcpwvFSK0bDby7hlIQZt2JA24C8eG9UTP/R5n\nm1ZkvfImBVd1szqhEEJUSMGM+hkF3EpgnYvLgB1AItCawLRUIUqV75wmuGcvILffPQDY0tJIvuM/\nJDx4L0Z2lsXphAiOaZr8vGMpWzP/tjqKECUWTHFxOXCb1vomQAMva60vAD4AGocynBDFFhNDzv+e\nJ33SNHw1TgMgdvxYXO0uxrFsqcXhhCgZtyedt1d9yGf6K8aum0ihr9DqSEKUSDDFRQLwe9HHG4Dz\niz5+E2gXilBCBKuwTbvALqvXXAeAfetfpHS7grgXhkOhfIMWkc00TRbvWs7wpSNZn7bx4PGMAumB\nE2VLMMXFDqBO0ccbgXOLPs4FUkMRSohTYbpSyXpvDJlvv4c/MQnD7yd+5EukdO2I/Y9NVscT4pjS\n8zN45/cxfLr+Czw+DwYGHWu35dEW91M5Vr61irIlmOLiK+BjpdTFwPfAbUqpG4D/A+Q7t4gMhkF+\nj1645y+i4MKLAXD+9iuuDpcS8/GHILtGigiyfPdvDF86kjUHNgBQNa4yg5oPoPvZV+K0Oy1OJ0TJ\nBVNcDCOw+2kdrfU8AsXGF8BVBDY1EyJi+GvVJmPydLKfeAbT6cTIzSXx4QdJuuVGjL17rY4nBAD7\n8g6Q583DwKB9rUt5rMWD1E2uc/IvFCJClXgqqta6AHjgkNd3K6WGAhmAP4TZhAgNu528+x6gsG07\nEgf0xaE3EP39LJxtW5M18i0KOl9pdUJRwXWq044dObtpW/Nizk450+o4QpyyEvVcKKXOUUqpI48X\n7YZ6DvBLqIIJEWreJucFpqz2vRsA2/79JN/ai4TBAyH7yK1yhCg9dpudu865RQoLUW4Uq7hQSp2p\nlFoNrALWKaUWK6VSiz7nVEo9BywDaoUvqhAhEBtLzrMvkT5xCr5q1QOHxn2Mq8MlOFYsszicEEKU\nD8XtuRhUc3/iAAAgAElEQVQJJAG3AzcRmI76klKqKrAEeBSYQGCFTiEiXmG7DrgXLCa/6zUAOLb8\nSUrXTsS99Bx4vRanE+VNdmEOe3NlOyZRcRS3uLgYuEdrPU5r/QVwM9Ad+BKoAVyltb616PGIEGWC\nmVqJzA/HkvnGaPwJiRg+H/EjXiClWyfsf262Op4oJ37ft5Znl47kgzXj8PqlcBUVQ3GLCxew8p8X\nWuvVBHoyEoDztdYzw5BNiPAzDPJ7/Qf3Dz9T2LI1AM4Vy3G1v4SYcR/LlFURtNzCXMaum8i7qz8h\nsyCLHdm7Dk41FaK8K25xYQcKjjiWDwzSWst8PlHm+eucQfq0mWQPewrT4QhMWR08kKTbbsLYt8/q\neKKMWXtgA8OXjmTp7hUAJEcl0f/cOzi/yjkWJxOidASzzsWhtoUkhRCRwG4n7/7BpM+ci7defQCi\nv5tBapvWRH3/ncXhRFmQ581j/PpJjFr1ERkFmQC0qt6cx1sN4pzKDS1OJ0TpKW5xYRb9OdZxIcoV\n73lNcX//I3l9+gJg27+P5P/cSMJDD0JOjsXpRCT7cftiFu0KzDpKjEqgX5PbuLVRT+KccRYnE6J0\nFXcRLQNYrpTyHXIsDliglDpshJLWum6owglhmbg4sl94hYKOV5Bw/z3Y9+4h9pMPcS6cT9boD/A2\nbW51QhGB2te+jOV7VnJaQnV61L+GBGe81ZGEsERxi4v/C2sKISJUQYdOuBcsIXHwQKJnfIPjzz9I\nufJycoc8Su79g8FR4kVuRTnmtDkY1Lw/sY5Yq6MIYalifWfUWktxISoss1IlMsd8SvSE8SQMfRhb\nTjbxLz5L1NzvAzuvnimddeJfUlgIceoDOkNCKRWtlPpQKeVWSu1QSg06wblNlFILlVK5SqlVSqm2\npRhVVFSGQf5NtwSmrLZoBYBz+S+BKaufjZMpqxWIrFUhxMlFRHEBjACaAW2BAcBTSqnrjjxJKZUE\nzAbWENjLZAowRSlVufSiiorMf8aZpE+bSc6jj2M6HNhyskl84B6Sbv8PxoEDVscTYVTgK+SrTd/w\n6q/v4PP7Tv4FQlRglhcXSqk44E5goNZ6ldZ6GvAScO8xTr8dyNJa99da/6m1fhrYCFxQWnmFwOEg\nd9DDpH/7Pd6zzgYgeuZ0XG1aEzV3tsXhRDhsydjK88teZd7fC/krcxvz/l5odSQhIprlxQVwHoGx\nH4sPOfYT0OoY57YBph16QGvdSmstixCIUudt2hz3nIXk3XYnAPa9e0i+6QYSHhkEubkWpxOhUOAr\nZPLGb3llxaiDe4M0rtSAFtWbWpxMiMgW9FB3pVRtoCHwI5B4Cit11gD2a60PfZC5B4hRSlXSWh/a\n11wX+EUp9S5wNbAFGKK1XhTktYU4NfHxZL/8KgWdriDx/nuw7d9H7JgPcC5cEJiyep78ECqr/sr4\nm3FLvmB75i4AYuwx3FCvG61rXIBhGBanEyKylbi4UEpFAWOBGwE/UB8YUTQe4jqtdWYJ3zKOwFLi\nh/rndfQRxxOAR4DXgc4EdmidrZRSWusdxbmYzWZgsxk47LbDjjkckdCJU77Yi9rYbi//bevvciWZ\nLX4h7oH7iJoxHcfmTaR06YDn0WF47h8EdnvIr1mR2re0rdm/gbd/+wi/6QegUaX69G58I6kxKRYn\nKx/k3g2vSGjfYHouHifwKKM9ML3o2BvAGOAFAgMyS8LD0UXEP6+P7Fv2Ar8dMjV2lVKqE9C76Non\nlZoaj2EYJGb+W8/Ex0fjcsliN+GSlFRBpua54mH61/Dhh/DAAxg5OcQO/z9if5gD48bBmWeG5bIV\npn1LUavEJny5qTLuvAxuPf96OtS9RHorwkDu3fCysn2DKS5uAvprrecrpUyAoo/vItCjUdLiYgdQ\nWSll01r7i45VB/K01ulHnLsLOHJbwY1AreJeLC0tB5vNICvLc/BYTk4+brcs6xxqdruNpKRYMjPz\n8Pn8J/+C8uL6m7A1bUl8v7twrFgGP/+Mee555L44goJeN0OIfkhV2PYtJf89tzdVXC5ifHGkp8sY\nmlCSeze8wtm+xf1FPJji4nRg8zGObwNSg3i/lUAh0Br4Z+zEpcCyY5y7BLjsiGMNgPHFvZjfb+L3\nm3gPaXC/38TrlRs8XHw+f8Vr39pnUvDNLOJefZm4kS9hZGcRf08/HDNnkDXiNczUSiG7VIVs31Jw\nWnwNXPHxuN050r5hIvdueFnZvsE8kFkHXH6M472KPlciWus8Aj0e7yilLlBKdQcGExhXgVKqmlIq\npuj0d4BzlVJPKqXOUko9A5wJfBrEv0OI8HI4yH3oMdKnz8ZbtIpn9PRpuNpciHPeHIvDCSFE+ART\nXDwNvK6UGkmg5+M2pdQE4CnguSBzDAJWAPOAN4EntNZTiz63i8DgUbTW24ArCMwUWQ1cBXTRWu8K\n8rpChJ23eQvcc38ir/cdANj37Cal13XED30I8vIsTlcx7crZw0Q99eCATSFEaJX4sYjWerpS6npg\nKOADHiKwYmZPrfVXwYQo6r24o+jPkZ+zHfF6MbJolihrEhLIfuV1Cjp1JvHBe7Dt30/cB+8S9WPR\nLqtNzrM6YYXgN/3M3fYj07fMxuv3UiWuEu1rXWp1LCHKnWCmotYtWrRKFq4SooQKruhC2vwlJA66\nl+jZ3+HYqEnp3J6cRx4n756BYZmyKgL25O5j3Lov2JK5FQCHzYGBzAARIhyCeSyyWSm1QCl1e9HS\n3UKIEjCrViVz3ESyXn4NMy4Oo7CQhOFPkXztVdi2bbU6XrnjN/3M+3shz//y6sHCok5iLR5tcT/t\nal1icTohyqdgiou2BKaDvgLsUUp9opRqF9JUQpR3hoHntj645y6ksGkzAKKWLMLV7mKiv/hcdlkN\nkfT8DF779V2+2vQNhX4vdsNOt7qdGdx8ADXiq1kdT4hyq8TFhdb6R611PwJrUdxKYIXNb5VSW5RS\nT4c4nxDlmu+seqRP/56cwY9g2mzYsjJJurcfiX1vx3CnWR2vzIuxx5CenwFArYTTeKTFQDqf0R67\nTR4/CRFOQa8NqrUu1FpPAe4hsGpnKjAsVMGEqDCcTnIfGUb6N7PwnRFYxTPm6ymBKasLfrA4XNkW\n44imd8MeXHlmRx664D5OT6hhdSQhKoSgigulVLxS6hal1ExgO3A38DKBjcWEEEHwtmhF2ryfybvl\nNgDsu3eR0uMa4p94FDyek3y1OJ56rrO46syO0lshRCkqcXGhlPqcwK6lowisQdFBa11faz1ca/13\nqAMKUaEkJJA98k0yPv4Mf6XAKp5x747C1akN9jWrLQ4nhBDFE0zPRXUCj0Kqa637aK0XhjiTEBVe\nwZVdSZu/hPwOHQFwbFiPq3M7Yt9+A/yy8NM/TNNkzf71mDIAVoiIEsyAznZa60+01rKTjxBhZFar\nRuZnX5L1wiuYsbEYBQUk/N/jJF/fDdt26SRMz89g9O9jGP37GBbuWGJ1HCHEIYq1iJZS6k+ghdb6\ngFJqC3DcXxO01jLuQohQMQw8ffpSeGkbEgf0xbnqN6J+Xoir7UXkjngV+h61qG25Z5omv+z+lUmb\nvibPG1g+fdGuX7jk9FbYjKDHqAshQqi4K3R+AvyzCcLH4YkihDgeX736pM+YQ9yI54l7fSS2zAwS\n/tsH5s3GeO5lSEi2OmKpyMjPYoKezO/71wJgYNCu1iV0q9tZCgshIkixigut9f8d8vIHYLHWuvDQ\nc4p2Lr0qhNmEEIdyOsl97EkK2nci6Z7/Yt/2F0yYQNKPC8l88x0KL21jdcKwMU2TFXtX8YWeSo43\n8ES2cmwleje8kbNTzrQ4nRDiSMGU+j8AKcc43gjZ+lyIsPO2ao37h5/Iv7k3ALadO0i5vhvxTw2D\n/HyL04VHgb+QKZu/PVhYtKl5MUNbPiiFhRARqrhjLh4gsNw3gAHsVkod69RfQpRLCHECZmISuW+N\nJvr67vjv6ovNnUbc6DeJmj+PzNEf4GvU2OqIIRVtj+I/DW5ggp7CLQ17UN91ltWRhBAnUNwxF28B\naQR6Oj4CHgQyDvm8CWQD80KaTghxYtddR2aDc4m7926ifpiLY/1aXJ3akDPsafL6DQBb+RmH0KiS\n4snWQ3DYSryZsxCilBV3zIUXGAuglDKBCVrr8tn/KkQZY9aoQcaEycR89B4J//cEhsdDwlNDiZoz\ni6w338F/2ulWRwwZKSyEKBuK+1jkVmBiUUFhAj2P81gErfXY0MUTQhSLYeC5sx+Fl7Ylsf9dOFev\nImrhAlxtLiT7pZHkX3uD1QlPyuP1EG2PxjAMq6MIIU5RcftMPwaSD/n4eH/GhCqYEKLkfPUV6TPn\nknv/YEzDwJaRTlK/PiT2vwsjI/2o86PmziZq7mwLkh5u7YENPLNkBEt2r7A6ihAiBIr7WMR2rI+F\nEBEoKoqcYU9R0KEjiff8F/vf24j56gucSxaR9da7FF58aeA8r5f4J4eCYVDQpj04Sv+RQ543j8mb\nprNo1zIAvtr0NedWbkS8M67UswghQueUCwWlVBWl1A1KqTNCkKfUHLoXwdbdWbI3gSh3CltfhPuH\nn/HceBMA9h3bSb6uK/HPPAn5+cSOeR/Hpo04NmpiPv6g1POtT9vIs0tfPVhYJEUlcmvDnlJYCFEO\nGCX9oaqUOgeYDNwF/A6sI7CZWT5wpdb6h1CHDKV9+7LMFXofn83ZiDvr3zGpVVNi6dHubJqrKham\nK18cDhsuVzxudw5er2y2FWolad+or6eQOOR+bOmBRyNe1RDbrh3YMjMB8LtcpC35DdOVGvbcHq+H\nKZu/5aedSw8eu6Da+fSofw0JzviwX7+45P4NH2nb8Apn+1apklisQVHB9FyMADYBG4CbACdQE3gZ\nGB7E+5WqFXofo6auPqywANibnseoqatZofdZlEyI8Cm4+lrcC5ZQ0KYdAA69/mBhAWBzu4l7+fmw\n5zBNk7dWfniwsEhwxtP3nN7c0fjmiCoshBCnJpji4iJgsNZ6L9AZmKG13klgQOf5IcwWFpN+2Mzx\nOmtMEybN3yyPSES55K9xGhkTp5AzcNAxdx6M/fhD7Js2hjWDYRh0rNMWgKZVz+XxVoM5v2qTsF5T\nCFH6giku/ECBUsoBtAXmFh1PBCJ+G/a96Xkn/rw7j03bM054jhBlls2Gc83vHKtf0/B6SXzgnrBH\nOK9KY4Y0v4e7zrmFxKiEsF9PCFH6gikuFgOPAc8AscAMpdTpwHPAkhBms0x6tqwPJiom57KlJF9/\nNc4FP3DcLr4QODO5TtjeWwhhvWCKi/uAZkB/4H6t9X7gUaAhMCSE2SyTkhBtdQQhwib7mecxjzHt\n9J9SImrhfFJ6XENKp7ZEfT0FfL7SDSiEKPNKXFxorTdrrZtrrV1a69FFh58B6mmtdWjjhV7VlNgT\nf94VS72aySc8R4iyzFdf4bmtz1HHPb1vJ+fRx/FXrgyAc9VvJN91G66LmhMzdgx4PCd970JfIVM3\nz2DFnpUhzy2EKDtKPBUVQCmVANwCNAEKgbUElgfPPOEXRoDvfvrTHDV19TF7fA0DBnRvItNRQ0Sm\nm4XXqbSv4U4jtXVTbG43cMRU1NxcYj7/lLjRb2LftvXg1/iqViOv3z14bu+DmZh01Htuzfybsesm\nsjt3L/GOOIa1GkxydOKp/SMtJPdv+EjbhleZnIqqlKoNrAFGEpg50g54HfhdKVWzpO9X2pqrKgzo\n3oSqrsN7MKq6YqWwEBWG6Uold8ijB1/nPPTYv2tcxMXhufO/pC35jczRH+BtGNi+3b53Dwn/e5LU\npo2JH/40xp49ABT6vXz9x3eMWPE2u3P3AlA7qSYm8kNDiIoqmEW0JhFYNOsGrfWeomPVgC+AHVrr\nm0OeMoT27csyITDf/o+dmXhNA6fNpG6NJNkwKcTkt5PwOuX29XpxtWkNhoF7/uLjL/9tmkTNnU3s\nm68Rtfjnfw9HR7P/umt4t201fk8uBCDaHsX1Z3fjotNalvn/n+T+DR9p2/CKhJ6LYIqLdKCj1nrZ\nEcdbAjO11pVK9Ial7J/iAuQGDzdp3/AKRfv+s2lZQYdOxbvmL0uJe+tVor+bcfCY3zBY3LouK2+5\nmg5dH6BSrCuoLJFG7t/wkbYNr0goLoKZLeLl2OtZ5AEyzUKIMqSgQ6diFxYA3patyBw7gbSFv7Dj\n6s547TZspsnFi//gnntepe5td+BcuCCs01iFEJEvmOLiZ+AJpZTznwNFHw8r+pwQopzzqQZEffAF\n0ya/w4E+fTDjApuNRc2fR8r13Ujp3I6o6V+DX34rFaIiCqa4eBS4AvhDKTVZKfUV8CeBpcAfDmU4\nIURku+zCXvhfeI0Dv64l5+Gh+FMDg0Kdv/1Kcp9bcF3SgpjxYyFfFqYToiIJZp2L9QT2EPmcwGOQ\nWGA8cJ7WelVo4wkhygIztRK5Qx7lwIq1ZD/7Ir6atQBwbN5E4oP3ktriXGLffgMjO8vipEKI0lCi\nAZ1KqSSgQGt98tV0IpQM6Cw90r7hFe723ZOzl4yCLOq7zir5FxcWEj3lS+Leeg3HhvUHD/uTU8jr\ncxd5d/XHrBLZ077l/g0fadvwKjMDOpVSKUqpr4E0IEspNUUpVflUAgohIpPf9DNv2488v+w1Ploz\nnuyCnJK/idNJ/o034Z6/mIxPJ1LYsjUAtox04l8dQaXmjUl4ZBC2rX+FNrwQIiIU97HIy0Ar4AkC\nAzdbAO+EK5QQwhp7c/fz2q/v8tXm6RT6veR68/gj46/g39Bmo6BTF9Knz8b99SzyO3UGwPB4iB3z\nAamtm5J4dx/sa1aH5h8ghIgIx1k15yhdgFu11rMAlFKLgDlKKYfW2hu2dEKIUuE3/fy4YzHTNs+g\nwB9YEKtWwmn0btST0xNqhOQa3tYXktn6Quzr1xH31mtET56E4fMRM/lLYiZ/SX6HjuTd9yCFF14c\nWItfCFFmFbfnoipw6K8WiwkUJtVCnkgIUaoO5KXx5m/vM2njNAr8hdgMG1ed2ZGHLrgvZIXFoXwN\nG5H19nukLV1J7l39MGMDS/FHz/2elO5XknLl5UTNmC7TWIUow4pbXDgILJ4FgNbahyyaJUS5sCn9\nTzam/wHA6Qk1ePiCgVx5ZkfsNntYr+uvXYec517mwIq15Ax+BH9KCgDOFctIvv1mXJe2JHrCeCgo\nCGsOIUToBbPOhRCiHGlVvTnnVTmHzmd04OEL7qNW4mmlen2zcmVyHxnGgV/Xkf2/5/GddjoAjk0b\nSRrYn9SW5xH7zluQnV2quYQQwSvWVFSllB+4ANh/yOG1BMZibDv0XK31Ya8jjUxFLT3SvuEVyvb1\nm35sRoT8rlFQQPTkSYFprBv1wcP+lBTy+vyXvLvuxqwc/slqcv+Gj7RteJWZqahFlgFbDvkTDyw4\n5PVfRX8LIcqYiCksAKKiyO/1H9w/LiXjk88pbN4CAFt6OvEjX6JS88bED30I298R/XuMEBVacWeL\ntAtrCiFE2JimSUZBJinRyVZHKRmbjYIuV1HQ+UqcSxYR+8ZIoud+j5GXR9wH7xI75gPyr72B3Hsf\nwNeosdVphRCHKPGW62WdPBYpPdK+4VWc9s3Iz2KCnsy2rO0MazmIOGdsKacMLfua1YFprFO/wjhk\nNkl+xyvIvW8Q3tYXhuxacv+Gj7RteJW1xyJCiDLCNE2W71nJs0tf4ff9a0nPz2DW1nlWxzplvnOa\nkPXOh6Qt+Y28O+7CjIkBIPr7WbiuvoKUrp2ImjVTprEKYbGIKC6UUtFKqQ+VUm6l1A6l1KBifM0Z\nSqlspdRlpZFRiLIiqyCbD9Z8ypi1n5HjzQWgbc2LufLMjhYnCx3/GWeS/eLIwDTWB4fgTy6axvrL\nEpJ798TV9kKiJ34GhYUWJxWiYoqI4gIYATQD2gIDgKeUUted5GtGE9iRVQhR5Le9qxm+9BVW7gus\neVcpJpX7m/ajR/1riLZHWZwu9MwqVch97EnSfltL9tPP4qseWPTLsWE9SffdTWqr84l9fzTkBLE/\nihAiaJYXF0qpOOBOYKDWepXWehrwEnDvCb7mP0BCKUUUokxYn7aRD9aMI7sw8IP00tMvZGjLB4Pb\n1bSMMRMSyRtwH2nLfifrtbfxnnU2APbtf5Mw7BEqNW9M3MvPY6QdsDipEBVDUMWFUqqGUupJpdTn\nSqmqSqkblFIqyAznEZi1sviQYz8R2CjtWNeuBLwA9ANkAwIhijRw1aOBqx6u6BTuO78vvdS1xDgq\n2CK60dF4bu6N+6dlZHz0KYVNmwFgS0sj/uXnqdSsMfGPP4Jt+98WBxWifCtxcaGUOhtYA9wOXE+g\nB6EnsFwpdcyC4CRqAPuP2ABtDxBTVEgcaSTwsdZ6XRDXEqLcMgyDWxv1YlirQTRIrWd1HGvZ7RR0\nvZr0734gffJ0Ctq2B8DIzSXuvdGktjyPxPvuxq43WBxUiPKpuOtcHOoVYIrW+i6lVFbRsZuAsQR6\nFEq6JkYckH/EsX9eH/Zrl1LqcuAioG8Jr3GQzWZgswU6POx222F/i9CS9g2vY7VvJUcZW8uiFJht\n25LTti2e31cR8/pInNOmYHi9xEz8jJiJn1HQ5So8Ax/E16r1YV8n92/4SNuGVyS0bzDFxUVAm0MP\naK29SqlngKVBvJ+HozdA++d17j8HlFIxBAZxDtBaB72TUWpqPMYR2zknJcm40HCS9g0vad9ianNR\n4M8ff8CIETBmDOTnEzXzW6JmfguXXgqPPgpduhy25bu0b/hI24aXle0bTHHh4NiPU5IAXxDvtwOo\nrJSyaa3/mZxeHcjTWqcfcl5LoC7wlVLq0OpgplLqE631gOJcLC0t57Cei6SkWDIz8/D5ZF58qEn7\nhta6/ZoZW+Zyb9M+xDhipH2DlVodnhuBcf9DRL87ipgP38fIyoSFC2HhQryNGpN//yB8N/QgKTVR\n2jcM5N4Nr3C2r8sVX6zzgikuZgGPKaV6F702lVKpwIvA3CDebyVQCLQGFhUdu5TAXiaHWgoc+SB5\nM4GZJnOKezG/38TvP3xVUp/PL6vEhZG076nxeD1M2fwtP+0MdAxO0tO56ZCZ2tK+QapUhcKhT5Fz\n7wPEfDKG2Hffxr53D451a3H0uxPfs8/AQ0PwXdsTb1SM1WnLJbl3w8vK9i3x8t9KqdOA+UAyUAnY\nANQB0oDLtNZbSxpCKTUauBjoA9QEPgZu11pPVUpVAzK01p5jfJ0faKu1/rG415Llv0uPtO+p02mb\n+XTDJNI8bgASnQn0anAd51c5R9o31DweYiZNIPat13Bs+fPgYX+lSuT17R9YEdSVamHA8kPu3fAq\nk8t/a613AucDQ4F3gB+BR4AmwRQWRQYBK4B5wJvAE1rrqUWf2wXceJyvq1gbo4gKw+PNZ6Keyhsr\n3ztYWDSrei6PtxrM+VXOsThdORUTg6f37bgXrSDjw7F4z28KgO3AAeJfGE6lpo2Jf3Iotp07LA4q\nROSTjcukeg4bad/guD3pvPbrO+z3pAEQ74yjZ/1raV7tvMPOk/YNL4fdwPXrEgqHP4tzwfyDx02n\nE88NPcm79wF89epbF7AMk3s3vCKh56LEYy6UUifc/Uhr3b6k7ymE+FdydBLJ0cns96RxXpVz6KWu\nJSkq0epYFY9hwOWXk938Qli+nLg3XyNq+jSMwkJiP/+UmAnjKejSldz7HsDbvIXVaYWIKMFMgt16\nxJ8dBNaqaMW/AzKFEEGyGTZ6N7yR2xr1ou85vaWwiADe85uR+eFY3IuWk9f7dsyoKAzTJHrGN7i6\ndCD52qtwzvseKlhPsBDHE7LHIkqpJ4BaWuv/huQNw0Qei5Qead/wkvYNrxO1r233LmLfHUXMJx9h\ny846eNzbuAm5Ax8kv1t3cAQzGa9ikHs3vCLhsUgol+8ax/EHXgohRLnhr16DnKf+F9iNddhT+CtX\nAcCxdjVJ/fqQemEzYsZ8AHl5FicVwhqhLC4uArwnPUuICq7Q7+X7rfMp8AW90KyIEGZyCnn3D+bA\nijVkvfQqvjpnAGDf+heJjwyiUvNziHttBEZG+onfSIhyJpgBnT9w9BTQJAK7m74dilBClFfbsrYz\nbt0X7MzZTUZ+JjfUv9rqSCIUYmPx3H4nnltuI3r6NGLfeBXnmt+x7d9H/HPPEPvGq3huvYO8u+/B\nX72G1WmFCLtgei7+4uhBnSsIbCY2JGTJhChHvH4v0/+czcvL32Jnzm4AduXswecPZsV8EbEcDvK7\nX0/63IWkT5hMwSWXAWDLziJu1BukXtCEhEH3Yf9jk8VBhQivYEYczQZmaa3TQh1GiPJoe9ZOxq3/\ngu3ZOwGIskdx7VlXccnprbAZsitkuWQYFLa/nIz2l+P4tWga64xvMAoKiP30E2LGj6XgqqsD01ib\nNrc6rRAhF8x3trcJbCwmhDgBn9/HzC1zeWn5mwcLi3opdRnW8kEuq3mhFBYVhLfZBWSO+RT3T8vI\nu7k3ptMZmMY6fRquK9qRfP3VOOfPk2msolwJ5rvbRqBJqIMIUR6tPrAOn+nDaXNyQ72rGdj0v1SO\nrWR1LGEBX736ZL/2NmnLV5Pb/z788QkARC2cT8qN3Unp2IboaZPBJ4/KRNkXzMZl7xLYiXQlsAk4\nbK6V1rpPyNKFgaxzUXqkfWF3zl7+v717j7Op3v84/tqXmT33iyFU7vRNEVGUro5y76ZcfuR+nEqd\nLuoU1Umno3RBnS5ydERKUoTkVnJXSYqIvhRCJMz9Pvvy+2MtTGOGGfaetfeez/PxmIfZa62993u+\ns8z67O/3u9aatWMefdRt1Iyp4dfXlvYNrEC3ry0tleip/yP6f5OwHzlyfLm7QUPyht9Pfu++EBWe\nd2OVfTewQvU6FxcAa4AsjOGRBiW+hBCmWrHn8MClf/N7YSFCny+5GrkjHuXot1vJGjsOT916ADh3\n7yL+Hw+S0roZ0a++jC0zw+KkQlSc3LhMqueAkfYNLGnfwKr09nW7cc3/mJhXX8a5/cfji73xCeQP\nGvxeAgAAACAASURBVEru34bjq1kz8Dkqgey7gRUyPRdKKY9S6pyziyRE+DmSd5Qir1w7TviB00nB\n7b1IW/klGe9/ROGVVwFgz8ok5rWXSbmsGXEPP4B91y8WBxXi9Mo7LFKuSkWIqsLr87Jy/zqeXT+B\nRbs/tzqOCCc2G4U3dCJj/mLSFn5OQeduxuKCAqLfnUq1dq2JHzYI5w+bLA4qRNnkXDghKuhIXiqv\nfj+Zj3bMp9BbxIp9a8gszDr9E4WoIPflbcmcPpPUNd+Q36cfPqcTm9dL1PyPSb7hWhJ73kLEmlVy\nGqsIOhW5iFYvpVTm6TbSWk8/izxCBC2fz8faA+uZ+/OnFJj3BTkvrjb9m/aW26KLgPKoC8l69U1y\nHnuC6EmvE/3uNGy5uUSuWkHkqhUUtbyU3L+PoLBrd3A4rI4rRPkmdCqlyjsjxKe1Duo9WyZ0Vp5w\nat/U/DRmbJ/NT2nGZZvtNjud6rWnc/0OOO3W3Fo7nNo3GAVz+9pSjxL99lvGaaypJy6W7G7UmLx7\nHyC/Zx9wuSxMeGrB3LbhIBgmdFakuKiltf7jbINZTYqLyhNO7fvKd5PYmb4LgNqxNRnQtDd1E863\nNFM4tW8wCon2zckhaua7xEx8Dcf+fccXe2rWIu+ue8kfOBhffIKFAUsXEm0bwoKhuCjvnAsZ0BNV\n2h1NbibC7qRjvfY8dvkDlhcWQgAQG0v+X+8mdf0mMl//L+4LmwLgOPQ7cc/8k2qXXkzMc89g+yPk\nPxeKECNniwhRDufHn8sz7UZxS6MuRFg0DCJEmSIiKOj1f6St/IqM92ZR1OYKAOyZGcS+Ms44jfXR\nh7Dv2W1xUFFVlLe4eIcSl/kWoqqRSZsi6NntFHbsQvqnn5H2yVIKOnYGwJafT/S0KVS74lLi7xqM\nY8sPFgcV4a5cxYXWerDWWs61E2HL5/Ph9cnYrwgf7iuuJPO9D0ld9TX5PfvgcziM01jnzqFah6tJ\n7NODiHVr5DRWERBynQtR5WUVZvO/re+yZM8XVkcRwu88TS8i643JpH6zmdxhd+OLjgYgcvkykm7r\nRlLXDkQu+hS8UlwL/5HiQlRp3/3xA2PWj2fT4a0s3vMF+7MOWB1JiIDw1qlLzrMvcvS7beQ8/Bje\npCQAIjZ+S+KgviRf0wbXzPegsNDipCIcSHEhqqTswhze3jqDKVvfI7soB4Crz21L9egUi5MJEVi+\nlBRyH3uCo99tI/vfY/Gcex4Azp07SHhgONUuv4ToN1/Hli0j4eLMSXEhqpzNh7cyZv14Nv6xGYBk\nVxJ/bzmM3uo2opzBe+EhIfwqLo68u+4l9ZvNZL76Ju4LFACOgweIG/041VpdTMzz/8Z25IjFQUUo\nkuJCVCmf7lrK5C3TySrKBuCqc9vyRNsRXFiticXJhLBIZCQFffqRtno9Ge/MpKj15QDY09OJnfAS\nKa0vJm7UI9j3/mpxUBFKpLgQVUrz6hdht9lJciVyb4uh9L3wdqKdUVbHEsJ6djuFXbqRvmgZ6fMX\nU9DhRgBseXlET5lMtbYtib/nrzi2/WhxUBEKpLgQVUq9hDoMbXYnT7QZwUUpyuo4QgQfm42iK68i\nc+YcUpevI79HT3x2OzaPh6g5H1Lt+itJ6HsHEV9/KaexijJJcSGqnJY1mhETEW11DCGCnqdZc7Im\nTSF1/SbyhgzDF2X08rmWfUbSzZ1J6t6RyCWL5DRWcRIpLoQQQpySt159sp8fz9GNP5Lz0CN4E83T\nWDesJ3FAH5KvuwLXBzOgqMjipCJYSHEhwspPqTtZd2C91TGECEu+GjXIHfUUqd//SPbTz+KpVRsA\np/6JhPvvoVqbFkRPngg5Oad8Hefnn8HixZURWVhEigsRFvLdBczSc3lt01t8qOdxIPt3qyMJEbZ8\ncfHkDf87qRt+IOuVN3A3Ns62cvy2n7gnR5LS6iJiXnwO29GjJz/Z7SbmyZHw8MPgdldyclFZpLgQ\nIW9n2i7GfvMyq3/7CgCX00VGQabFqYSoAlwu8vv2J23tBjKmzqCoVWsA7GlpxI57npTWFxP7xKPY\n9+87/pToqW/h2LkDtm/H9fZbViUXAWbzVbHZvocPZx3/gZ1OO8nJsaSl5eB2y4Qkfwt0+xZ6Cvnk\nlyWs2L/2+LIWNZrRR91WJe5gKvtvYEn7ngGfj4h1a4h57WUiV5y4V4/P6aSgR0/yBgwmsX9v7Glp\nAHiTq5H69Xf4kqtZlTgsBXLfrVEj3lae7aS4kD8eARPI9t2TuZd3fvyAP/KMqwfGOKPpdcGtXFaz\nJTZbufb9kCf7b2BJ+54d55bNRL/2Mq5P5mE7xdkkuX+9i5znXqrEZOEvGIoLGRYRIcnj9XI4zxjP\nbZbSlCfbPszltS6tMoWFEMHO3bwFWZOnkfrVd+QNHIovIqLU7aKnTTGGSURYcVodQIgz0SipPl0a\n3EBKVDJta7WWokKIIOVt0JDsl17G8fNOItetPmm9ze0m9qlRZM6cY0E6EShSXIiQ1a3BjVZHEEKU\nlyuyzFWOfXuNq33Kh4SwIcMiQgghAi77mbH4nKV/nnXu0MQ9OkIuwhVGpLgQQcntdbM7Q+7CKES4\n8FygyB845KTl3sREAKLfmUJin9uxpadVdjQRAFJciKCzP+sAL337Ov/5fjJ/5B62Oo4Qwk9yHn0c\nb3Ly8cfe5Gqkfb6awmvbAxC5ZiVJXW/AsetnqyIKP5HiQgQNj9fD4t1f8OK3r7E/+wBF3qLjF8YS\nQoQ+X3I1ch8Zefxx/mOj8NZvQMbM2eQNHAqA8+edJHX+CxHr1lgVU/iBFBciKBzI/p1xG1/n091L\n8fg8RNgjuKPJzfRo3N3qaEIIP8obPAxPkwugaVMKhgwzFkZEkP3iBLKffQGf3Y49PZ3EnrcQNWO6\ntWHFGZOzRYSlPF4PX+xbzcJdn+H2eQBomFiP/k17cU5MDYvTCSH8zukkd8zzxMdHgdMJxy7yZLOR\nN+wePA0bET9sMPbsLOIfug/Hzh3k/PNf4HBYm1tUiPRcCEvtzdrP/F8W4/Z5cNqd9GjcnYda3SOF\nhRBhzH1jR+jSpdR1hR06kr7wczx16wEQM/FVEgb1hezsyowozlJQ9FwopVzARKAHkAuM11pPKGPb\nbsAYoDHwC/BPrfWCysoq/KtBYj2uPa8dv2btY0DT3tSKPcfqSEIIi3maXkTa4uUkDupLxIb1uJYu\nJvmmTmS8+wHe8+tYHU+UQ7D0XIwDWgHXA8OB0UqpHiU3Uko1B+YA/wNaAJOB2eZyEaJua9yNh1sN\nl8JCCHGcr0YN0ucsIP+O3gA4f9xCcqf2ODdusDiZKA/LiwulVAwwFLhfa71Zaz0feBG4r5TN+wJf\naK3f0Frv0lpPBFYAvSovsfC3SEcEDruMpwohSoiKIuuNyeQ8/hQA9sN/kHRrV1xzZ1scTJyO5cUF\nRg+EEyh+zuFaoG0p204DRpayPNH/sYS/FHnkqntCiDNks5H74CNkTJmOLzoaW0EBCXcNIealscYl\nw0VQCobiojZwRGvtLrbsEBCllEopvqE2bDn2WCl1MdABWFYpSUWFeH1ePvt5FU+sHcvRPLnqnhDi\nzBXedCvp8xfjqVUbgNiXxhJ/9xDIy7M4mShNMEzojAEKSiw79thV1pOUUtUx5l+s0Vp/Ut43s9tt\n2O3GzXEcDvuf/hX+czQvlXc3z2b7UeNWyh/s+JgHWg+zOFV4kf03sKR9A+eM2/ayy8hatoq4fr1w\nbt5E1Nw5OPfuJfu9D/DVrBmApKEpGPbdYCgu8jm5iDj2OLe0JyilagKfAz6gZ0XerFq12JNuz52Q\nEF2RlxCn4PP5WL5rHdM3zSHPnQ9AncRzGdDqNpKTYy1OF55k/w0sad/AOaO2TW4C69ZC//4wdy7O\njRtI6ng9fPopXHKJ3zOGMiv33WAoLn4Dqiul7Fpr82oq1ALytNbpJTdWSp0HLAc8wPVa66MVebPU\n1Jw/9VwkJESTmZmHx+M9zTPF6aTlp/Pujx/x41ENgA0btzTtSKc6HbBjJy0tx+KE4UX238CS9g0c\nv7TtW+8QVb8R0S+Pg3378LVrR85bUynq3NW/YUNQIPfd8n5IDIbiYhNQBFwBfGkuuwY46Xwj88yS\nJeb27bXWFb6rldfrw+v98yQgj8eL2y1/PM5GRkEW/14/njy3Mf5ZM+YcBjfvQ6v6TUlLy5H2DSDZ\nfwNL2jdwzrZts0c9RVHDxsSP+Du2nBxi+/UmZ/QY8u65D0r0UFdFVu67lhcXWus8pdR0YJJSaghw\nPvAwMAiOD4FkaK3zgSeABhjXw7Cb68Do5cis7OzihERXPC1rNOPrg9/yl7rXcFODTkS7ypwyI4QQ\nflHQuy+eeg1IHNwX+9GjxD39BI6dmuwXJkBkpNXxqqxgmak0AtiIMdzxGsZVN+eZ6w5y4joWPYBo\nYD1woNjXK5WaVpTq9ibdGdF6OD0adyfCEWF1HCFEFeG+4krSFi/HfYECIHrGdBJ734YtLdXiZFWX\nzVfFzhM+fDjr+A/sdNpJTo6VbvsAkfYNLGnfwJL2DZxAta0tM4OEYYOIXPEFAO4GDcmc8RGexk38\n9h6hIJD7bo0a8eUabwqWngshhBDirPgSEsmY8RG5f70LAOfuXSR16UDE6pXWBquCpLgQ5ZJdmMN7\n2z8ivSDD6ihCCFE2p5Oc514i6/nx+BwO7BnpJPa+jah33rY6WZUixYU4rc2HtzJm/Xi+OriBGT/N\npqoNpQkhQk/+kGFkvD8bb0IiNo+H+H88SOyTj4HHY3W0KkGKC1GmnKJcpv04k8lbppNVlA1AsisJ\nt0/+cwohgl9R+w6kL1qGp159AGImv0lC/97YsuTkwkCT4kKUasuRbYxZP54Nh74HIMmVyL0thtL3\nwtuJsFt+BrMQQpSL5wJF2pIVFF7RDgDXss9I6t4R+95fLU4W3qS4EH/i8Xp4d9uHTPphGpmFWQBc\nUfsynmgzgotSlMXphBCi4nwpKWR8NJ/8Pv0AcG7fRnLn9ji/WW9xsvAlxYX4E4fdgdtn3KA2MTKe\nuy8ZRP+mvYiJkPsrCCFCmMtF1n8mkv3PZ/DZbNiPHCGpRzdcs2dZnSwsSf+2OEmvC24lxhlD94Yd\niY2IsTqOEEL4h81G3t8fxNOoMQnD/4otN5eE4cPI+XkHuY8+AXb5vO0v0pLiJLERMfRWt0phIYQI\nS4Vdu5O+YCmec88DIHbCSyQMGwS5pd6IW5wBKS6EEEJUOe7mLUhfuoKiS1sB4Fowj6Rbu2D//aDF\nycKDFBdV0M60XccnawohRFXlrVmL9LmLyL/5NgAiNn1PUqf2OLdstjhZ6JPiogop9BQye8cnvPL9\nJD7Qc+ViWEIIERND1uSp5Ix4FADHwQMk3dSJyEWfWhwstElxUUX8kr6Hsd+8wor9awHYmfYLaQXp\nFqcSQoggYLeTO/JJMie+hc/lMiZ6Du5H9Ksvg3wIOyNytkiYK/QU8emupSzftwYfxn+S5tWb8n/q\ndhJdCRanE0KI4FFwR2889eqTOLAv9iOHiRszGudOTda4/4DLZXW8kCLFRRjbnbGXd7d/yKHcPwCI\ndkbRs8kttKnVCputXHfNFUKIKsV9eVvSlq4g8c5eOLdvI2rW+9h/3UPm1Bn4UlKsjhcyZFgkjH2x\nb/XxwuKiFMWTbR+mbe3WUlgIIcQpeOvUJX3h5xTc2AmAyK+/JLlzexz6J4uThQ4pLsJY7wtupXpU\nNfpd2JPhlwwhyZVodSQhhAgJvrh4Mqd/QO5d9wLg+HUPSV1vIGL5MouThQYpLsJYfGQcT13xD9qd\ne7n0VgghREU5HOT8eyxZ4/6Dz+nEnpVJYr+eRE2ZbHWyoCfFRZhz2B1WRxBCiJCWP2AwGbPm4k1M\nwubxED/qEeJGPgxut9XRgpYUFyHM4/WQ7863OoYQQoS9omuuI33JF7gbNgIg+u23SOx7B7bMDIuT\nBScpLkLUgezfeWnj68zUH1sdRQghqgRPoyakL/6CwquvBSBy5XKSut6Afc9ua4MFISkuQozH62Hp\nnuW8sOE/7Mv6jW8PbeKn1J1WxxJCiCrBl1yNjA8+Ju/OgQA4d2iSO7cn4usvLU4WXKS4CCG/5xxi\n/HcT+WTXEtw+DxF2Jz0ad+eC5EZWRxNCiKojMpLs8a+S/cxz+Gw27KmpJN5+E64PZlidLGjIRbRC\ngNfnZfm+NSzYtRS315hAVD+hLv2b9qJW7DkWpxNCiCrIZiPv7vvwNGhE/N1Dsedkk3D/PeT+vJOc\nx58Ce9X+7F61f/oQ8ckvS5j780LcXjdOm4NbG3Xl4dbDpbAQQgiLFXbqQvqnn+E5vw4AMa9OIGFI\nf8jJsTiZtaS4CAHX17mKaGcUdePP57HLH+DGetdjt8mvTgghgoHn4makLVlBUevLAXAtWkDSzZ2x\nH/jN4mTWkSNUCEhyJfJQq3t4pPW9nBtXy+o4QgghSvCdcw7pcxeS3+MOACK2bCapU3ucm76zOJk1\npLgIEefF1ZYLYgkhRDCLiiLrzSnkPPo4AI5Dv5N0SxciF8yzOFjlk+JCCCGE8BebjdxHRpL51jR8\nUVHY8vJIHDqAmAkvgs9ndbpKI8WFxXw+H+sOrGf2zk+sjiKEEMJPCm7pQfq8RXhrGBPvY58fQ/zw\nYZBfNa6qLMWFhdLy03lj8xTe/2kOK/atZeuR7VZHEkII4SfuVpeR9tlK3Bc3ByBqzock3X4TtsOH\nLU4WeFJcWMDn8/HVwW959psJbE/dAUDNmHOIj4yzOJkQQgh/8p53PmkLllLQuSsAERvWk9zlLzi2\nb7M4WWBJcVHJ0gsymPTDNN7b/iF57nxs2Lih7nWMuvwB6iXUsTqeEEIIf4uLI3PqDHLvfQAAx95f\nSep2I5HLllocLHCkuKhEP6Xu5Nn1E9h61Bj+OCe6OiNa38NtjbsR4YiwOJ0QQoiAcTjIGf1vsl55\nA19EBPbsLBLu7E305IlhOdFTiotKVCM6BY/Pgw0b7etczag2D9Iwsb7VsYQQQlSS/L79yfhoPt7k\nZGxeL3FPjiTuHw9BUZHV0fxK7i1SiVKiq9Hvwp4kRMbTJLmh1XGEEEJYoKjd1aQtXk7inb1w/ryT\n6Olv49i9i8wp7+BLSrY6nl9Iz0Ula12zhRQWQghRxXkbNiJ90TIKr20PQOSalSR16YBj188WJ/MP\nKS6EEEIIC/iSksmYOZu8QUMBcP7yM0md/0LEujUWJzt7Ulz4UU5RLusPbrQ6hhBCiFAREUH2CxPI\neu5FfHY79vR0EnveQtR771id7KzInAs/2XJkG+//NIfMwiySXImoao2tjiSEECIU2Gzk//VuvA0a\nEj9sMPbsLOJH/B3Hzh3kPPUMOELvvlLSc3GWcovymL5tFpN+mEZmYRYAO9LCY8xMCCFE5Sns0JH0\nRcvw1K0HQMybr5EwqC+27CyLk1WcFBdn4cejmme/mcD6342hkMTIBO65ZDA3NepscTIhhBChyHNh\nU9KWrKCozRUAuJYuJql7J+z791mcrGKkuDgDee58ZmyfzcTNU0gvyACgTa1WPNl2BM2qN7U4nRBC\niFDmq16d9DkLyO/ZBwDntq0kd2qPc+MGi5OVnxQXZyCrMIsNh74HID4yjr81H8jAi/oQExFjcTIh\nhBBhweUi6/X/kvP4UwDYD/9B0q1dcc2dbXGw8pEJnWfgnJga3NKoC7szfqWXupW4iFirIwkhhAg3\nNhu5Dz6Cu1ETEu77G7a8PBLuGkLOzh3k/mMU2GxWJyyT9FycoevPv4ohzfpJYSGEECKgCm+6hfRP\nluCpVRuA2HHPE3/3EMjLszhZ2aS4OEO2IK4YhRBChBd3i0tJX7qCoktaAhA1dw5JPbphO3TopG2d\nn38GixdXdsQ/Z7D03U1KKRcwEegB5ALjtdYTytj2UuBNoDmwFbhHa/2dvzOl5qdRLSo8rvEuhBAi\n9Hlrn0v6/MUk3HcXroWfELHxW5I7tyfj3Vl4mjU3NnK7iXlyJDgdsOorrOpDCJaei3FAK+B6YDgw\nWinVo+RGSqkYYCGwytz+K2ChUiraX0EKPUV8vPNTRn/1Ar+k7/HXywohhBBnLzaWzCnTyX3gYQAc\nv+0nuXtHIpcaPRXRU9/CsXMHbN+O6+23LItpeXFhFgxDgfu11pu11vOBF4H7Stm8D5CrtX5MGx4E\nsoCe/siyO2Mvz294hS/2rcbr8zJn5wJ8Pp8/XloIIYTwD7udnCdGk/naJHyRkdhyc0gY0IfocS8Q\nM+7545tFvTAWW1qqNREtedc/a4ExPPNVsWVrgbalbNvWXFfcOuDKswlQ5Cli3s+LGL/xDQ7lHgbg\nohTF3y4ZIHMrhBBCBKWC3n1Jn70Ab0oKNp+PuBefxZ6Wdny9PS2VmJfGWpItGIqL2sARrbW72LJD\nQJRSKqWUbQ+UWHYIOP9M3/yX1F959utX+HzvSnz4iHK46HdhT4ZfMoQkV+KZvqwQQggRcO4rriRt\n8XLc9RuUuj562hRjmKSSBcOEzhigoMSyY49d5dy25HZlsttt2O1Gb0SeJ4+nV71Mgdt4yabVmjDg\n4l5Ui5aJnP7gcNj/9K/wL2nfwJL2DRxpWz9r3AhfvfqwZ/dJq2xuN/GjHyf7w48rNVIwFBf5nFwc\nHHucW85tS25XppSUuOPjHMnE8u7tr5T3qeIMJST4bb6tKIW0b2BJ+waOtK0frVqxGCj1xlYRyz5b\nnJwc27Uy4wRD2fgbUF0pVTxLLSBPa51eyra1SiyrBRwMYD4hhBAiuPl8XfD5bGV8VWphAcFRXGwC\nioArii27BijtDi1fA+1KLGtnLhdCCCFEELAFw6mWSqk3gauAIRiTM6cBg7TW85RSNYEMrXW+Uioe\n2AnMBCYDdwN3AI211sF7HVQhhBCiCgmGnguAEcBGYDnwGvBPrfU8c91BoBeA1joL6A5cC3wLtAG6\nSGEhhBBCBI+g6LkQQgghRPgIlp4LIYQQQoQJKS6EEEII4VdSXAghhBDCr6S4EEIIIYRfSXEhhBBC\nCL8Khst/B5RSygVMBHpgXCZ8vNZ6QhnbXgq8CTQHtgL3aK2/q6ysoaiC7dsNGAM0Bn7BOOV4QWVl\nDUUVad9iz6mPsf921VqvDnjIEFbB/be5uW1rjOvtPKC1XllJUUNOBdv2Noy/DXWB7zHa9vvKyhrK\nzHb+Fri3rP/vVhzbqkLPxTigFXA9MBwYrZTqUXIjpVQMsBBYZW7/FbBQKSUXvz+18rZvc2AO8D+g\nBcZF0Gaby0XZytW+JbwJyH5bPuXdfxOAzzD+MDcD5gJzlVLVKy9qyClv214EzACeAy4BNmP87Y2q\nvKihySwsZgIXnWIbS45tYV1cmI06FLhfa71Zaz0feBG4r5TN+wC5WuvHtOFBIAvoWXmJQ0sF27cv\n8IXW+g2t9S6t9URgBeYF0sTJKti+x57TD4irpIghrYLtOwjI0lrfY+6/TwM7gMsqK28oqWDbdgS2\naq1naK13A6Mw7hlV5gFTgFKqKcatL0q/1/oJlhzbwrq4wPiE7MSo1I5ZC7QtZdu25rri1gFXBiZa\nWKhI+04DRpayPNH/scJGRdoXpVQK8DxwF2ArbRvxJxVp3+uA+cUXaK3baq2XBC5eSKtI2x4FLlZK\ntVNK2TBuA5GBMXQqynYd8AXGMepU/98tObaFe3FRGziitXYXW3YIiDL/EJfc9kCJZYcw7nUiSlfu\n9jUr5i3HHiulLgY6AMsqJWloqsj+CzABmKa13lYp6UJfRdq3IXBEKfVfpdRBpdSXSqmSN1EUJ1Sk\nbWcBizAOgIUYPRx3aK0zKiVpiNJaT9JaP6K1zj/NppYc28K9uIgBCkosO/bYVc5tS24nTqhI+x5n\njlPPAdZorT8JULZwUO72VUrdgHGH4H9XQq5wUZH9Nw54DOOPdGdgNfCZUuq8gCYMXRVp2xSMYZDh\nGPeLmg5Mk/ksfmPJsS3ci4t8Tm7AY49zy7ltye3ECRVpXwDMu9wuB3zIfJbTKVf7mhPf3gSGa60L\nKylbOKjI/usGvtda/8ucQzASY85F/wBnDFUVadsXgB/MT+LfYwzr5QCDAxuxyrDk2BbuxcVvQHWl\nVPGfsxaQp7VOL2XbWiWW1cK4K6soXUXaF/NT3mqMsdjrtdZHKydmyCpv+7bB6Lafo5TKUkplmcsX\nK6UmVlLWUFSR/fcg8FOJZTuAOgHMF8oq0ratMc4QAUBr7TMf1wt4yqrBkmNbuBcXm4Ai4Ipiy64B\nNpSy7dcY3crFtTOXi9KVu33N2eNLzO2v01ofqpSEoa287bseaAK0xJhI18JcPhR4KsAZQ1lF/z60\nKLHsQmBPQJKFvoq07QFOPjNEAbsDE63KseTYFtYX0dJa5ymlpgOTlFJDMCawPIxxWtmxLvoMc0LM\nbGCsUupljGsw3A3EAh9akT0UVLB9n8A4Zep6wG6uA+OTTGZlZw8FFWzfXcWfq5QCOKC1PlKpoUNI\nBdt3EnCfUuopjGsyDMTYn9+zInuwq2DbvgVMVUp9i3F2yTCMi2m9Y0X2cBAMx7Zw77kAGAFsxBjn\nfw3jqpDzzHUHMa+zoLXOAroD12Jc7awN0EVrnVfpiUNLudoX4yp90Rifsg8U+3qlUtOGnvK2b0m+\nSsgWDsr792Ev0Am4GdgCdMP4+yDDpmUrb9t+iHH9i8eB7zBOkWwvhXGFlPz/bvmxzebzyd8gIYQQ\nQvhPVei5EEIIIUQlkuJCCCGEEH4lxYUQQggh/EqKCyGEEEL4lRQXQgghhPArKS6EEEII4VdSXAgh\nhBDCr6S4EEIIIYRfSXEhhBBCCL8K63uLCBFqlFIrMS7TW5IPGK+1frQcr3EdsAKob1622q+UUvU4\n+aZSHiDVfN9/aK33+em9dgNTtdbPmI8HAIu01keUUgOBt7XWDn+8VynvPRCYitH2NnOxF8jEuIzy\no1rrTRV4vTpAO631LH9nFSLYSM+FEMHFB8wCamLcFvnYV23gXxV8nUDyAbdxIl9djPvHXAosANR7\nSAAABvpJREFU8OP7XAaMA1BKXQtMA2LMdR9gtEsg+fjz76EucDvG72eJebff8noH4/4kQoQ96bkQ\nIvjkaa0PWx3iNGxAmtb6j2LLDiqlngbeU0o111pvOds30VofLfbQTrGiSWtdAPxx0pP8rJTfxQGl\n1H3ASuAvwKflfCnb6TcRIjxIcSFEiFFKJQEvAV2Ac4A0YD5wv3mL5ZLbN8a4K+WVGAfoL4FHtNZb\nzfUJGL0DtwKRGF3+j2mtN55BPI/5b7752ucDzwMdgHhgLcawyRZzfQ3gDaA9xm2gvwMe11qvNtfv\nxhiaWIVxd02A3UqpwRgH66laa7tSaipwodb6ymI/d12M4ZsbtNYrlFLtgLHA5cBhjB6WUeZdIyuq\nwHz/IvO9bMBIjFux1zfXrwPu1VrvVkqtAK4DrlNKXa+1bqiUigDGAP2ARIy7rY7WWn9+BnmECCoy\nLCJE6JkGtMAoBhoDDwIDgL+Vsf0sYD/QCuN2yx7g42LrFwP1gK7m+q+BtUqpFuUNpJSyKaVaAk8C\n32utdyql4jAKmXMxbvl8JZALrDbnHwBMAqKAa4BmwA5gnlIqusRbrMMYjvBhFAfH5i0c68mYBrRR\nSjUs9pw7gb1mYXEJ8DmwyHyf/zPbY2l5f8ZiP2sD4AVgD7DaXPwA8DDwENAEuAW4ABhvru8BfGXm\nvsxc9g5wg5mlJfAhsEAp1aWimYQINtJzIUTwuVMp1bPEstVa627m958Bq7TWP5qP9yql7geal/F6\nDTEOonu11m7zU/+FAEqpDkBboLrWOt3c/kml1NUYB8whp8i5WCnlNb93mf+uAu4yv+8PVAPu0Fqn\nmu/XF/gFuBfjk35D4Adgj9Y6Xyn1APAeJ3pAADBzp5oPj2itC5RSxdevMns5+mL0BmB+/475/SPA\nUq31C+bjXUqpfsAvSqlrj/WUlMKmlMrkxJBGBFAILAEGaq3zzOU7gQFa68Xm431KqY+AO8x8aUqp\nQowhr1SzN6kP0FJr/YP5nFfMAu1RjIJPiJAlxYUQwWc+xgGm+Bh9XrHv3wRuNouEJsDFGF3x28t4\nvceB/wD3mmejLAFmmusuxejB3Ff8YI0xPBJ5mpxDgW/M74uAP8x5EMc0A3YcKywAzALiG04UQv/C\nKCZ6KqXWYhRB72utC0/z3qWZjjHEMEYpdSnQlBPFRSugsVKq5BCIz9yurOLCh9FLZMMYghqDMZnz\nyeJn4mitFyql2iil/gUo8+tijB6j0rQ0/11rDqkc48QY5hIipElxIUTwydJalzzVEzg+tr8QuAh4\nH+OMie+At8p6Ma31m+an6K4Ycx+eweidaIlRWGRgHHxLTjgs4NQOaK13nWK9jdLPWrFjzlXQWs9T\nStUGOmMMETwEjFZKtdVal1UslWUa8JRSqhXGUMPaYu1oB2ZgFAclf85TTp4t9hq7lFI3YRRUnyul\nWmqt0wCUUiOBf2LMD1kGTMAYtupTxssem5x6NZBdYp3n5M2FCC0y50KI0NIS40B8h9b6ca31TGAX\nxtyLk85GUErVUEq9Bri01tO11gMxPonXxphguBVIMNfvOvYFjMKYN3A2fjAiqOrF8kRhzDn4USkV\nqZQaDzTSWn+ktb4LoyfGC3Qr5fVOeXqt2ZOwEugJ9MIoNo7ZClyktd5d7GeMBF4B6lBO5jBIP4zT\nUt8otmoU8LTW+j6t9f+01t9g9F4U/50Uz7/VXHduiXYfCgwubx4hgpX0XAgRWn7H+NTfWyl1BKiO\nMexRkxPzHuDEQS0V40DdUCn1OJAFDMLolfgW2AtsBmaZ8x32YcyHGIjxSf9svI9x0P1QKfUoxlyF\n0RhnhfxXa12olLocuNqcM/I7Ru9KLMZE0JKyzZ+rpVLqaCnrwSgo3sD44PRhseXjMSaSvg68DiSb\n27kwJpGWm9b6B6XUCxi9PzO01gsx2q2jUupTjJ6HARjXAfm9RP76SqnztNbbzG0nmae1/ohRFD2G\n8fsRIqRJz4UQIURrfRDjwH8zsA3jALofeJkTZyGA+SlZa+3BOGXVi9FdvwVjaKSr1nqP1tqLMRzx\nLcaZDJsxuupv1VqvPEWU016kS2udidE7kma+92qMg/lVxeYr9MLoeZkP/IRxxktfrfWx4qL4+2zB\nONtjFmWfGTPHfM4crXVOsSzrMS5g1QLYCMzDmKNyo9bafbqfpRRjzOdPVErFYkxejQE2YExqvRhj\nYus55um4YJwZ0xzYbA5v9TbzTsIoLvoDQ7TW751BHiGCis3nC/SF/IQQQghRlUjPhRBCCCH8SooL\nIYQQQviVFBdCCCGE8CspLoQQQgjhV1JcCCGEEMKvpLgQQgghhF9JcSGEEEIIv5LiQgghhBB+JcWF\nEEIIIfxKigshhBBC+JUUF0IIIYTwq/8H8A6Qk8IvteQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x118b39668>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr, tpr, thresholds = roc_curve(treino.prognostico.map(lambda x: x=='mau' and 1 or 0),\n",
    "                                 model.predict_proba(treino[attrs])[:,1],\n",
    "                                 drop_intermediate=False)\n",
    "plt.plot(fpr,tpr, marker='o',drawstyle='steps')\n",
    "plt.plot(fpr,fpr,linestyle='--')\n",
    "plt.plot(fpr,thresholds,marker='d',color='red')\n",
    "plt.xlim([-0.01, 1.05])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Podemos interpretar esse gráfico da seguinte forma. O ideal é estarmos o mais próximo possível do canto esquerdo superior. Pontos acima da linha tracejada indicam uma predição melhor que aleatória (sobre a linha). Finalmente, podemos resumir o desempenho do classificador calculando a área sob a curva ROC (AUC). Como a área do gráfico é 1, a AUC será um valor entre 0 e 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.86749999999999994"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "roc_auc_score(teste.prognostico.map(lambda x: x=='mau' and 1 or 0),\n",
    "                                 model.predict_proba(teste[attrs])[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assim, como o valor é próximo de 1, concluimos que o classificador teve um bom desempenho.\n",
    "\n",
    "Finalmente, podemos avaliar o desempenho do classificador testando a hipótese de que a classe predita e a classe real são independentes e não estão relacionadas. Para isso, podemos usa o teste do $\\chi^2$.\n",
    "\n",
    "O teste avalia se a diferença entre os valores observados e esperados (sob a hipótese de independência) na tabela de contigência através da estatística $\\chi^2$.\n",
    "\n",
    "O valor esperado para a célula $E_{ij}$ é dado por\n",
    "\n",
    "$$\n",
    "E_{ij} = \\frac{R_iC_j}{N}\n",
    "$$\n",
    "\n",
    "A estatística $\\chi^2$ é dada por\n",
    "\n",
    "$$\n",
    "\\chi^2 = \\sum_i^k\\sum_j^k\\frac{(O_{ij}-E_{ij})^2}{E_{ij}}\n",
    "$$\n",
    "\n",
    "Depois avalia-se a probabilidade de se observar uma estatística tão alta quanto a calculada usando a distribuição $\\chi^2$. Se o valor for suficientemente pequeno, rejeita-se a hipótese nula, ou seja, aceita-se que existe relação entre as classes preditas e reais."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prognostico  bom  mau\n",
      "row_0                \n",
      "0              8    7\n",
      "1              2   13\n",
      "[10 20]\n",
      "[15 15]\n"
     ]
    }
   ],
   "source": [
    "tabcont = pd.crosstab(model.predict(teste[attrs]),teste.prognostico)\n",
    "print(tabcont)\n",
    "cols = tabcont.sum(axis=0).values\n",
    "print(cols)\n",
    "rows = tabcont.sum(axis=1).values\n",
    "print(rows)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  5.,  10.],\n",
       "       [  5.,  10.]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exp = np.fromiter([cols[j]*rows[i]/np.sum(cols) for i in [0,1] for j in [0,1]],dtype='float').reshape(2,2)\n",
    "exp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.4000000000000004"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chisq = np.sum((tabcont.values-exp)**2/exp)\n",
    "chisq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.020136751550346329"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from scipy.stats import chi2_contingency\n",
    "from scipy.stats.distributions import chi2\n",
    "chi2.sf(chisq,df=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5.4000000000000004, 0.020136751550346329, 1, array([[  5.,  10.],\n",
       "        [  5.,  10.]]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi2_contingency(tabcont.values,correction=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Como o p-value é menor que 0.05, rejeitamos a hipótese de independência e concluimos que há relação entre o valor predito e o valor real."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Investigando a correção dos rótulos\n",
    "\n",
    "Todas as métricas de qualidade discutidas acima assumem que os rótulos do conjunto de treinamento são corretos. Podemos analisar a correção dos rótulos, avaliando a concordância na atribuição dos rótulos por vários classificadores. Se houver alta concordância na atribuição dos rótulos, temos mais indícios de que os rótulos são corretos.\n",
    "\n",
    "Existem diversas métricas para se quantificar o nível de concordância entre os rótulos atribuidos por diversos codificadores (classificadores). Entre essas métricas as mais conhecidas são [Fleiss Kappa](https://en.wikipedia.org/wiki/Fleiss%27_kappa) e [Krippendorf alpha](https://en.wikipedia.org/wiki/Krippendorff%27s_alpha).\n",
    "\n",
    "O Fleiss Kappa possui uma escala de valores associados que facilita sua compreensão (veja o artigo da Wikipedia). Logo, focaremos nela nessa aula. Imagine que possuimos $n$ amostras que serão classificadas em $k$ classes por $m$ codificadores. O Fleiss Kappa informa o quanto a probabilidade/proporção média de concordância desvia do valor esperado. Formalmente,\n",
    "\n",
    "$$\n",
    "\\kappa = \\frac{\\bar{P}-\\bar{P_e}}{1-\\bar{P_e}}\n",
    "$$\n",
    "\n",
    "A probabilidade de concordância esperada é o valor esperado sobre todas as classes\n",
    "\n",
    "$$\n",
    "\\bar{P_e} = \\sum_j^k p_j^2 = \\sum_j^k(\\frac{1}{nm} \\sum_i n_{ij})^2\n",
    "$$\n",
    "onde $n_{ij}$ é a quantidade de codificadores que assinalaram a classe $j$ ao exemplo $i$.\n",
    "\n",
    "A probabilidade de concordância observada para cada amostra é dada pela proporção de pares de classificadores que concordam na atribuição da classe sobre o total de pares.\n",
    "\n",
    "$$\n",
    "P_i = \\frac{1}{m(m-1)}\\sum_j^km_{ij}(m_{ij}-1)\n",
    "$$\n",
    "\n",
    "Assim, a probabilidade de concordância média observada é dada por:\n",
    "\n",
    "$$\n",
    "\\bar{P} = \\frac{1}{n}\\sum P_i = \\frac{1}{n}\\sum_i^n\\frac{1}{m(m-1)}\\sum_j^km_{ij}(m_{ij}-1) = \\frac{1}{nm(m-1)}\\sum_i^n\\sum_j^km_{ij}^2-nm\n",
    "$$\n",
    "\n",
    "Quando $\\bar{P}$ é menor ou igual a $\\bar{P_e}$, temos uma concordância simplesmente por chance e $\\kappa \\leq 0$. Por outro lado, $\\kappa = 1$ indica que todos os classificadores concordam com as classes atribuídas. A escala de interpretação proposta por [Landis e Koch (1977)](www.jstor.org/stable/2529310) é a seguinte:\n",
    "\n",
    "\n",
    "| ${\\displaystyle \\kappa }$ | Interpretation |\n",
    "|-------------------------|----------------|\n",
    "|< 0 |\tPoor agreement|\n",
    "|0.01 – 0.20 |\tSlight agreement|\n",
    "|0.21 – 0.40 |\tFair agreement|\n",
    "|0.41 – 0.60 |\tModerate agreement|\n",
    "|0.61 – 0.80 |\tSubstantial agreement|\n",
    "|0.81 – 1.00 |\tAlmost perfect agreement|\n",
    "\n",
    "Podemos aplicar essa metodologia para avaliar os rótulos das amostras de nosso exemplo. Para isso, vamos avaliar a concordância de várias classificadores e a classe real."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DecisionTreeClassifier</th>\n",
       "      <th>ExtraTreesClassifier</th>\n",
       "      <th>GaussianNB</th>\n",
       "      <th>KNeighborsClassifier</th>\n",
       "      <th>LogisticRegression</th>\n",
       "      <th>RandomForestClassifier</th>\n",
       "      <th>prognostico</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    DecisionTreeClassifier  ExtraTreesClassifier  GaussianNB  \\\n",
       "1                        0                     0           0   \n",
       "2                        0                     0           0   \n",
       "3                        1                     0           0   \n",
       "4                        0                     0           0   \n",
       "5                        0                     0           0   \n",
       "6                        0                     0           0   \n",
       "8                        0                     0           0   \n",
       "11                       0                     0           0   \n",
       "12                       0                     0           0   \n",
       "13                       1                     1           0   \n",
       "\n",
       "    KNeighborsClassifier  LogisticRegression  RandomForestClassifier  \\\n",
       "1                      0                   0                       0   \n",
       "2                      0                   0                       0   \n",
       "3                      0                   0                       0   \n",
       "4                      0                   0                       0   \n",
       "5                      0                   0                       0   \n",
       "6                      0                   0                       0   \n",
       "8                      0                   0                       0   \n",
       "11                     0                   0                       0   \n",
       "12                     0                   0                       0   \n",
       "13                     1                   0                       1   \n",
       "\n",
       "    prognostico  \n",
       "1             0  \n",
       "2             0  \n",
       "3             1  \n",
       "4             0  \n",
       "5             0  \n",
       "6             0  \n",
       "8             0  \n",
       "11            0  \n",
       "12            0  \n",
       "13            1  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "models = [DecisionTreeClassifier(), RandomForestClassifier(min_samples_leaf=5,n_estimators=100),\n",
    "         ExtraTreesClassifier(min_samples_leaf=5,n_estimators=100),LogisticRegression(), GaussianNB(),\n",
    "         KNeighborsClassifier(n_neighbors=3)]\n",
    "attrs = ['lpsa','lcavol','lweight','lcp','lbph']\n",
    "models = map(lambda x: x.fit(treino[attrs],\n",
    "                             treino.prognostico.map(lambda x: x=='mau' and 1 or 0)), models)\n",
    "import re\n",
    "prediction = pd.DataFrame(dict(map(lambda x: (re.sub('.*\\.','',str(type(x))).rstrip(\"'>\"),\n",
    "                                              x.predict(treino[attrs])),models)))\n",
    "prediction['prognostico'] = treino.prognostico.map(lambda x: x=='mau' and 1 or 0).values\n",
    "prediction.index = treino.index\n",
    "prediction.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6813079161745185"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.metrics.agreement import AnnotationTask\n",
    "at = AnnotationTask(data=[(coder,item,prediction.loc[item,coder]) for coder in prediction.columns for item in prediction.index])\n",
    "at.kappa()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "De acordo com a tabela, temos concordância substancial acerca dos rótulos atribuídos pelos codificadores.\n",
    "De qualquer forma, podemos ainda verificar quais são as amostras onde houve maior discordância entre os classificadores e o rótulo real, e no geral."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      0    1\n",
       "3   5.0  2.0\n",
       "13  2.0  5.0\n",
       "14  4.0  3.0\n",
       "16  6.0  1.0\n",
       "17  5.0  2.0\n",
       "18  3.0  4.0\n",
       "27  4.0  3.0\n",
       "30  2.0  5.0\n",
       "33  3.0  4.0\n",
       "35  6.0  1.0\n",
       "38  1.0  6.0\n",
       "41  2.0  5.0\n",
       "43  4.0  3.0\n",
       "51  2.0  5.0\n",
       "52  2.0  5.0\n",
       "58  4.0  3.0\n",
       "59  2.0  5.0\n",
       "60  1.0  6.0\n",
       "61  5.0  2.0\n",
       "87  2.0  5.0\n",
       "91  2.0  5.0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "geral = prediction.apply(lambda x: x.value_counts(),axis=1).dropna()\n",
    "geral"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gleason</th>\n",
       "      <th>prognostico</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>9</td>\n",
       "      <td>mau</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>7</td>\n",
       "      <td>mau</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>6</td>\n",
       "      <td>bom</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    gleason prognostico    0    1\n",
       "3         7         mau  5.0  2.0\n",
       "13        7         mau  2.0  5.0\n",
       "14        7         mau  4.0  3.0\n",
       "16        6         bom  6.0  1.0\n",
       "17        7         mau  5.0  2.0\n",
       "18        6         bom  3.0  4.0\n",
       "27        7         mau  4.0  3.0\n",
       "30        6         bom  2.0  5.0\n",
       "33        6         bom  3.0  4.0\n",
       "35        6         bom  6.0  1.0\n",
       "38        7         mau  1.0  6.0\n",
       "41        9         mau  2.0  5.0\n",
       "43        6         bom  4.0  3.0\n",
       "51        7         mau  2.0  5.0\n",
       "52        6         bom  2.0  5.0\n",
       "58        6         bom  4.0  3.0\n",
       "59        7         mau  2.0  5.0\n",
       "60        7         mau  1.0  6.0\n",
       "61        6         bom  5.0  2.0\n",
       "87        6         bom  2.0  5.0\n",
       "91        6         bom  2.0  5.0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(treino.loc[geral.index.values.tolist(),['gleason','prognostico']],geral,left_index=True,right_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}