HEART.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64ac3fa1",
   "metadata": {},
   "source": [
    "#  About the project and data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e0d484",
   "metadata": {},
   "source": [
    "In this project, we examine six examples of machine learning models in the field of classification.<br>\n",
    "These six examples are:\n",
    "**KNN, GradientBoostingClassifier, LogisticRegression, SVM, DecisionTreeClassifier, RandomForestClassifier**<br>\n",
    "In each of the models, I tried to put the best value for each parameter to have the best fit and the closest prediction.<br>\n",
    "I use 4 methods to evaluate each algorithm.\n",
    "**accuracy_score , f1-csore , log_loss , jacard**<br>\n",
    "You can see the results below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67c73754",
   "metadata": {},
   "source": [
    "# Load library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6b83608f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d046d102",
   "metadata": {},
   "source": [
    "# Read data from csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "73fbf7dd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trtbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>restecg</th>\n",
       "      <th>thalachh</th>\n",
       "      <th>exng</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slp</th>\n",
       "      <th>caa</th>\n",
       "      <th>thall</th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>63</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>145</td>\n",
       "      <td>233</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>150</td>\n",
       "      <td>0</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>130</td>\n",
       "      <td>250</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>187</td>\n",
       "      <td>0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>130</td>\n",
       "      <td>204</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172</td>\n",
       "      <td>0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>56</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>178</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>354</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>163</td>\n",
       "      <td>1</td>\n",
       "      <td>0.6</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  sex  cp  trtbps  chol  fbs  restecg  thalachh  exng  oldpeak  slp  \\\n",
       "0   63    1   3     145   233    1        0       150     0      2.3    0   \n",
       "1   37    1   2     130   250    0        1       187     0      3.5    0   \n",
       "2   41    0   1     130   204    0        0       172     0      1.4    2   \n",
       "3   56    1   1     120   236    0        1       178     0      0.8    2   \n",
       "4   57    0   0     120   354    0        1       163     1      0.6    2   \n",
       "\n",
       "   caa  thall  output  \n",
       "0    0      1       1  \n",
       "1    0      2       1  \n",
       "2    0      2       1  \n",
       "3    0      2       1  \n",
       "4    0      2       1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df =pd.read_csv(\"heart.csv\")\n",
    "df.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12d44054",
   "metadata": {},
   "source": [
    "**Standardize** the names of some columns according to the needs of the issues"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fd07e7b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns ={\"caa\":\"ca\",'thalachh':'thalach','restecg':'rest-ecg'}, inplace =True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84569e79",
   "metadata": {},
   "source": [
    "# Knowing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a3d6614c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(303, 14)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6e48b75c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>sex</th>\n",
       "      <th>cp</th>\n",
       "      <th>trtbps</th>\n",
       "      <th>chol</th>\n",
       "      <th>fbs</th>\n",
       "      <th>rest-ecg</th>\n",
       "      <th>thalach</th>\n",
       "      <th>exng</th>\n",
       "      <th>oldpeak</th>\n",
       "      <th>slp</th>\n",
       "      <th>ca</th>\n",
       "      <th>thall</th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "      <td>303.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>54.366337</td>\n",
       "      <td>0.683168</td>\n",
       "      <td>0.966997</td>\n",
       "      <td>131.623762</td>\n",
       "      <td>246.264026</td>\n",
       "      <td>0.148515</td>\n",
       "      <td>0.528053</td>\n",
       "      <td>149.646865</td>\n",
       "      <td>0.326733</td>\n",
       "      <td>1.039604</td>\n",
       "      <td>1.399340</td>\n",
       "      <td>0.729373</td>\n",
       "      <td>2.313531</td>\n",
       "      <td>0.544554</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>9.082101</td>\n",
       "      <td>0.466011</td>\n",
       "      <td>1.032052</td>\n",
       "      <td>17.538143</td>\n",
       "      <td>51.830751</td>\n",
       "      <td>0.356198</td>\n",
       "      <td>0.525860</td>\n",
       "      <td>22.905161</td>\n",
       "      <td>0.469794</td>\n",
       "      <td>1.161075</td>\n",
       "      <td>0.616226</td>\n",
       "      <td>1.022606</td>\n",
       "      <td>0.612277</td>\n",
       "      <td>0.498835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>94.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>47.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>120.000000</td>\n",
       "      <td>211.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>133.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>55.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>240.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>153.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>61.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>274.500000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>166.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>77.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>200.000000</td>\n",
       "      <td>564.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>202.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              age         sex          cp      trtbps        chol         fbs  \\\n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000   \n",
       "mean    54.366337    0.683168    0.966997  131.623762  246.264026    0.148515   \n",
       "std      9.082101    0.466011    1.032052   17.538143   51.830751    0.356198   \n",
       "min     29.000000    0.000000    0.000000   94.000000  126.000000    0.000000   \n",
       "25%     47.500000    0.000000    0.000000  120.000000  211.000000    0.000000   \n",
       "50%     55.000000    1.000000    1.000000  130.000000  240.000000    0.000000   \n",
       "75%     61.000000    1.000000    2.000000  140.000000  274.500000    0.000000   \n",
       "max     77.000000    1.000000    3.000000  200.000000  564.000000    1.000000   \n",
       "\n",
       "         rest-ecg     thalach        exng     oldpeak         slp          ca  \\\n",
       "count  303.000000  303.000000  303.000000  303.000000  303.000000  303.000000   \n",
       "mean     0.528053  149.646865    0.326733    1.039604    1.399340    0.729373   \n",
       "std      0.525860   22.905161    0.469794    1.161075    0.616226    1.022606   \n",
       "min      0.000000   71.000000    0.000000    0.000000    0.000000    0.000000   \n",
       "25%      0.000000  133.500000    0.000000    0.000000    1.000000    0.000000   \n",
       "50%      1.000000  153.000000    0.000000    0.800000    1.000000    0.000000   \n",
       "75%      1.000000  166.000000    1.000000    1.600000    2.000000    1.000000   \n",
       "max      2.000000  202.000000    1.000000    6.200000    2.000000    4.000000   \n",
       "\n",
       "            thall      output  \n",
       "count  303.000000  303.000000  \n",
       "mean     2.313531    0.544554  \n",
       "std      0.612277    0.498835  \n",
       "min      0.000000    0.000000  \n",
       "25%      2.000000    0.000000  \n",
       "50%      2.000000    1.000000  \n",
       "75%      3.000000    1.000000  \n",
       "max      3.000000    1.000000  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e43b21ef",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 303 entries, 0 to 302\n",
      "Data columns (total 14 columns):\n",
      " #   Column    Non-Null Count  Dtype  \n",
      "---  ------    --------------  -----  \n",
      " 0   age       303 non-null    int64  \n",
      " 1   sex       303 non-null    int64  \n",
      " 2   cp        303 non-null    int64  \n",
      " 3   trtbps    303 non-null    int64  \n",
      " 4   chol      303 non-null    int64  \n",
      " 5   fbs       303 non-null    int64  \n",
      " 6   rest-ecg  303 non-null    int64  \n",
      " 7   thalach   303 non-null    int64  \n",
      " 8   exng      303 non-null    int64  \n",
      " 9   oldpeak   303 non-null    float64\n",
      " 10  slp       303 non-null    int64  \n",
      " 11  ca        303 non-null    int64  \n",
      " 12  thall     303 non-null    int64  \n",
      " 13  output    303 non-null    int64  \n",
      "dtypes: float64(1), int64(13)\n",
      "memory usage: 33.3 KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47b63b2c",
   "metadata": {},
   "source": [
    "Here we can see the values ​​that each column has"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1c52921b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------  age  ------\n",
      "58    19\n",
      "57    17\n",
      "54    16\n",
      "59    14\n",
      "52    13\n",
      "51    12\n",
      "62    11\n",
      "60    11\n",
      "44    11\n",
      "56    11\n",
      "64    10\n",
      "41    10\n",
      "63     9\n",
      "67     9\n",
      "65     8\n",
      "43     8\n",
      "45     8\n",
      "55     8\n",
      "42     8\n",
      "61     8\n",
      "53     8\n",
      "46     7\n",
      "48     7\n",
      "66     7\n",
      "50     7\n",
      "49     5\n",
      "47     5\n",
      "70     4\n",
      "39     4\n",
      "35     4\n",
      "68     4\n",
      "38     3\n",
      "71     3\n",
      "40     3\n",
      "69     3\n",
      "34     2\n",
      "37     2\n",
      "29     1\n",
      "74     1\n",
      "76     1\n",
      "77     1\n",
      "Name: age, dtype: int64\n",
      "------  sex  ------\n",
      "1    207\n",
      "0     96\n",
      "Name: sex, dtype: int64\n",
      "------  cp  ------\n",
      "0    143\n",
      "2     87\n",
      "1     50\n",
      "3     23\n",
      "Name: cp, dtype: int64\n",
      "------  trtbps  ------\n",
      "120    37\n",
      "130    36\n",
      "140    32\n",
      "110    19\n",
      "150    17\n",
      "138    13\n",
      "128    12\n",
      "160    11\n",
      "125    11\n",
      "112     9\n",
      "132     8\n",
      "118     7\n",
      "124     6\n",
      "135     6\n",
      "108     6\n",
      "152     5\n",
      "134     5\n",
      "145     5\n",
      "122     4\n",
      "170     4\n",
      "100     4\n",
      "105     3\n",
      "126     3\n",
      "115     3\n",
      "180     3\n",
      "136     3\n",
      "142     3\n",
      "102     2\n",
      "148     2\n",
      "178     2\n",
      "94      2\n",
      "144     2\n",
      "146     2\n",
      "200     1\n",
      "114     1\n",
      "154     1\n",
      "123     1\n",
      "192     1\n",
      "174     1\n",
      "165     1\n",
      "104     1\n",
      "117     1\n",
      "101     1\n",
      "156     1\n",
      "106     1\n",
      "155     1\n",
      "129     1\n",
      "172     1\n",
      "164     1\n",
      "Name: trtbps, dtype: int64\n",
      "------  chol  ------\n",
      "204    6\n",
      "197    6\n",
      "234    6\n",
      "269    5\n",
      "254    5\n",
      "      ..\n",
      "284    1\n",
      "224    1\n",
      "167    1\n",
      "276    1\n",
      "131    1\n",
      "Name: chol, Length: 152, dtype: int64\n",
      "------  fbs  ------\n",
      "0    258\n",
      "1     45\n",
      "Name: fbs, dtype: int64\n",
      "------  rest-ecg  ------\n",
      "1    152\n",
      "0    147\n",
      "2      4\n",
      "Name: rest-ecg, dtype: int64\n",
      "------  thalach  ------\n",
      "162    11\n",
      "160     9\n",
      "163     9\n",
      "152     8\n",
      "173     8\n",
      "       ..\n",
      "202     1\n",
      "184     1\n",
      "121     1\n",
      "192     1\n",
      "90      1\n",
      "Name: thalach, Length: 91, dtype: int64\n",
      "------  exng  ------\n",
      "0    204\n",
      "1     99\n",
      "Name: exng, dtype: int64\n",
      "------  oldpeak  ------\n",
      "0.0    99\n",
      "1.2    17\n",
      "1.0    14\n",
      "0.6    14\n",
      "1.4    13\n",
      "0.8    13\n",
      "0.2    12\n",
      "1.6    11\n",
      "1.8    10\n",
      "0.4     9\n",
      "2.0     9\n",
      "0.1     7\n",
      "2.8     6\n",
      "2.6     6\n",
      "1.5     5\n",
      "3.0     5\n",
      "1.9     5\n",
      "0.5     5\n",
      "3.6     4\n",
      "2.2     4\n",
      "2.4     3\n",
      "0.9     3\n",
      "3.4     3\n",
      "4.0     3\n",
      "0.3     3\n",
      "2.3     2\n",
      "3.2     2\n",
      "2.5     2\n",
      "4.2     2\n",
      "1.1     2\n",
      "3.1     1\n",
      "0.7     1\n",
      "3.5     1\n",
      "6.2     1\n",
      "1.3     1\n",
      "5.6     1\n",
      "2.9     1\n",
      "2.1     1\n",
      "3.8     1\n",
      "4.4     1\n",
      "Name: oldpeak, dtype: int64\n",
      "------  slp  ------\n",
      "2    142\n",
      "1    140\n",
      "0     21\n",
      "Name: slp, dtype: int64\n",
      "------  ca  ------\n",
      "0    175\n",
      "1     65\n",
      "2     38\n",
      "3     20\n",
      "4      5\n",
      "Name: ca, dtype: int64\n",
      "------  thall  ------\n",
      "2    166\n",
      "3    117\n",
      "1     18\n",
      "0      2\n",
      "Name: thall, dtype: int64\n",
      "------  output  ------\n",
      "1    165\n",
      "0    138\n",
      "Name: output, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "for column in df.columns:\n",
    "    print(\"------  \" + column + \"  ------\")\n",
    "    print(df[column].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da913ecc",
   "metadata": {},
   "source": [
    "# Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9728c0e",
   "metadata": {},
   "source": [
    "In the **ca column**, we see the value of 4, which is our  **noise** in this column.\n",
    "Considering that the number of our data is small, instead of removing the noises,<br> we tried to go the noises with fillna to the conventional data for the ca column and used the median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9dc8bc4f",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    175\n",
      "1     65\n",
      "2     38\n",
      "3     20\n",
      "4      5\n",
      "Name: ca, dtype: int64\n",
      "0.0    175\n",
      "1.0     65\n",
      "2.0     38\n",
      "3.0     20\n",
      "Name: ca, dtype: int64\n",
      "0.0    180\n",
      "1.0     65\n",
      "2.0     38\n",
      "3.0     20\n",
      "Name: ca, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df['ca'].value_counts())\n",
    "\n",
    "df['ca'] =df['ca'].replace(4,np.NAN)\n",
    "print(df['ca'].value_counts())\n",
    "\n",
    "df['ca'] =df['ca'].fillna(df['ca'].median())\n",
    "print(df['ca'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c99e0f9",
   "metadata": {},
   "source": [
    "If there are duplicate rows in the data, we delete them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c352a2af",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(303, 14)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop_duplicates()\n",
    "df.shape\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b340458",
   "metadata": {},
   "source": [
    "In this section, we try to **measure the importance** of each data column.<br>\n",
    "So that important columns for existing algorithms are separated and train data is examined on these columns.<br>\n",
    "**Tree-based feature selection**<br>\n",
    "\n",
    "I used **ExtraTreesClassifier** and its methods to check each feature.<br>\n",
    "I observed the importance of each column and selected the columns thatI thought would give better output to our algorithms.<br>\n",
    "I must say that at first all the columns were checked for the algorithms and the result was not the removal of some columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "836ff33b",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.0690264  0.05627477 0.14189538 0.05720631 0.06090602 0.02069798\n",
      " 0.03289084 0.07905552 0.09776715 0.08860827 0.06356649 0.13920897\n",
      " 0.0928959 ]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEXCAYAAACjyo8UAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdmElEQVR4nO3de5gdVZ3u8e9LIopcRKU9QBJN9GTAjKLECCgz430eAmgYdRRGYQ5eIg4IjDoaPTpexvGC6AgOhxgVH1EYjncjRsEbXgHT4WoIPLYZlUg49HgBHhgJkff8saphZ7PTXZ2u3Zfi/TxPP921qmr91u7s/Lr2qrVWyTYREdFeO011AyIior+S6CMiWi6JPiKi5ZLoIyJaLok+IqLlkugjIlpu9lQ3oJe99trL8+fPn+pmRETMGOvWrfsv2wO99k3LRD9//nwGBwenuhkRETOGpF9tb1+6biIiWi6JPiKi5ZLoIyJaLok+IqLlkugjIlouiT4iouWS6CMiWq5Wopd0mKQbJA1JWtFj//6SLpV0l6Q39tg/S9KVki5sotEREVHfmBOmJM0CzgKeB2wC1kpabfu6jsN+B5wMHLWdak4BNgB7TKSx81d8fdzn/PL9R0wkZETEjFfniv4gYMj2RttbgAuAZZ0H2L7F9lrg7u6TJc0FjgA+0UB7IyJinOok+jnAjR3bm6qyuj4CvAm4ZxznREREQ+okevUoq/WgWUlHArfYXlfj2OWSBiUNDg8P16k+IiJqqLOo2SZgXsf2XOCmmvUfCrxA0uHAQ4A9JH3W9su7D7S9ClgFsGTJkil9YnnuBUREm9S5ol8LLJS0QNLOwNHA6jqV236L7bm251fnfbdXko+IiP4Z84re9lZJJwEXAbOAc2yvl3RCtX+lpL2BQcqomnsknQossn1b/5oeERF11FqP3vYaYE1X2cqOn2+mdOmMVsclwCXjbmFERExIZsZGRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFytRC/pMEk3SBqStKLH/v0lXSrpLklv7CifJ+l7kjZIWi/plCYbHxERY5s91gGSZgFnAc8DNgFrJa22fV3HYb8DTgaO6jp9K/AG21dI2h1YJ+lbXedGREQf1bmiPwgYsr3R9hbgAmBZ5wG2b7G9Fri7q3yz7Suqn28HNgBzGml5RETUUifRzwFu7NjexA4ka0nzgQOBy7ezf7mkQUmDw8PD460+IiK2o06iV48yjyeIpN2ALwKn2r6t1zG2V9leYnvJwMDAeKqPiIhR1En0m4B5HdtzgZvqBpD0IEqSP8/2l8bXvIiImKg6iX4tsFDSAkk7A0cDq+tULknAJ4ENtj+8482MiIgdNeaoG9tbJZ0EXATMAs6xvV7SCdX+lZL2BgaBPYB7JJ0KLAIOAI4FrpV0VVXlW22vafyVRERET2MmeoAqMa/pKlvZ8fPNlC6dbj+idx9/RERMksyMjYhouST6iIiWS6KPiGi5JPqIiJZLoo+IaLkk+oiIlkuij4houST6iIiWS6KPiGi5JPqIiJZLoo+IaLkk+oiIlkuij4houST6iIiWS6KPiGi5JPqIiJZLoo+IaLkk+oiIlkuij4houST6iIiWq5XoJR0m6QZJQ5JW9Ni/v6RLJd0l6Y3jOTciIvprzEQvaRZwFrAUWAQcI2lR12G/A04GTt+BcyMioo/qXNEfBAzZ3mh7C3ABsKzzANu32F4L3D3ecyMior/qJPo5wI0d25uqsjpqnytpuaRBSYPDw8M1q4+IiLHUSfTqUeaa9dc+1/Yq20tsLxkYGKhZfUREjKVOot8EzOvYngvcVLP+iZwbERENmF3jmLXAQkkLgN8ARwN/V7P+iZzbevNXfH1cx//y/Uf0qSUR0WZjJnrbWyWdBFwEzALOsb1e0gnV/pWS9gYGgT2AeySdCiyyfVuvc/v0WiIiooc6V/TYXgOs6Spb2fHzzZRumVrnRkTE5MnM2IiIlkuij4houST6iIiWS6KPiGi5WjdjY+Ya7xBOyDDOiLbJFX1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRchldGIzKMM2L6yhV9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcrUQv6TBJN0gakrSix35JOrPaf42kxR37/lHSekk/k/Qfkh7S5AuIiIjRjZnoJc0CzgKWAouAYyQt6jpsKbCw+loOnF2dOwc4GVhi+wnALODoxlofERFjqnNFfxAwZHuj7S3ABcCyrmOWAee6uAzYU9I+1b7ZwC6SZgMPBW5qqO0REVFDnUQ/B7ixY3tTVTbmMbZ/A5wO/BrYDNxq++JeQSQtlzQoaXB4eLhu+yMiYgx1Er16lLnOMZIeTrnaXwDsC+wq6eW9gtheZXuJ7SUDAwM1mhUREXXUWb1yEzCvY3su9+9+2d4xzwX+0/YwgKQvAU8HPrujDY4HtqySGTF+da7o1wILJS2QtDPlZurqrmNWA8dVo28OoXTRbKZ02Rwi6aGSBDwH2NBg+yMiYgxjXtHb3irpJOAiyqiZc2yvl3RCtX8lsAY4HBgC7gSOr/ZdLukLwBXAVuBKYFU/XkhERPRW68EjttdQknln2cqOnw2cuJ1z3wG8YwJtjIiICcjM2IiIlkuij4houST6iIiWS6KPiGi5JPqIiJZLoo+IaLkk+oiIlkuij4houST6iIiWS6KPiGi5WksgRDzQjHeVzKyQGdNZrugjIlouiT4iouWS6CMiWi6JPiKi5ZLoIyJaLok+IqLlkugjIlou4+gjpsh4x+pDxuvHjskVfUREy9VK9JIOk3SDpCFJK3rsl6Qzq/3XSFrcsW9PSV+QdL2kDZKe1uQLiIiI0Y2Z6CXNAs4ClgKLgGMkLeo6bCmwsPpaDpzdse8M4Ju29weeBGxooN0REVFTnT76g4Ah2xsBJF0ALAOu6zhmGXCubQOXVVfx+wB3AH8F/C8A21uALc01PyLGknsBUafrZg5wY8f2pqqszjGPBYaBT0m6UtInJO3aK4ik5ZIGJQ0ODw/XfgERETG6OolePcpc85jZwGLgbNsHUq7w79fHD2B7le0ltpcMDAzUaFZERNRRJ9FvAuZ1bM8Fbqp5zCZgk+3Lq/IvUBJ/RERMkjqJfi2wUNICSTsDRwOru45ZDRxXjb45BLjV9mbbNwM3StqvOu45bNu3HxERfTbmzVjbWyWdBFwEzALOsb1e0gnV/pXAGuBwYAi4Ezi+o4rXAedVfyQ2du2LiIg+qzUz1vYaSjLvLFvZ8bOBE7dz7lXAkh1vYkTMBBndM31lCYSImFHymMfxyxIIEREtl0QfEdFySfQRES2XRB8R0XJJ9BERLZdEHxHRckn0EREtl0QfEdFySfQRES2XRB8R0XJJ9BERLZdEHxHRckn0EREtl0QfEdFySfQRES2XRB8R0XJJ9BERLZdEHxHRckn0EREtVyvRSzpM0g2ShiSt6LFfks6s9l8jaXHX/lmSrpR0YVMNj4iIesZM9JJmAWcBS4FFwDGSFnUdthRYWH0tB87u2n8KsGHCrY2IiHGrc0V/EDBke6PtLcAFwLKuY5YB57q4DNhT0j4AkuYCRwCfaLDdERFRU51EPwe4sWN7U1VW95iPAG8C7tmxJkZExETUSfTqUeY6x0g6ErjF9roxg0jLJQ1KGhweHq7RrIiIqKNOot8EzOvYngvcVPOYQ4EXSPolpcvn2ZI+2yuI7VW2l9heMjAwULP5ERExljqJfi2wUNICSTsDRwOru45ZDRxXjb45BLjV9mbbb7E91/b86rzv2n55ky8gIiJGN3usA2xvlXQScBEwCzjH9npJJ1T7VwJrgMOBIeBO4Pj+NTkiIsZjzEQPYHsNJZl3lq3s+NnAiWPUcQlwybhbGBERE5KZsRERLZdEHxHRckn0EREtl0QfEdFySfQRES1Xa9RNRMQDyfwVXx/3Ob98/xF9aEkzckUfEdFySfQRES2XRB8R0XJJ9BERLZdEHxHRckn0EREtl0QfEdFySfQRES2XRB8R0XJJ9BERLZdEHxHRckn0EREtl0QfEdFySfQRES1XK9FLOkzSDZKGJK3osV+Szqz2XyNpcVU+T9L3JG2QtF7SKU2/gIiIGN2YiV7SLOAsYCmwCDhG0qKuw5YCC6uv5cDZVflW4A22Hw8cApzY49yIiOijOlf0BwFDtjfa3gJcACzrOmYZcK6Ly4A9Je1je7PtKwBs3w5sAOY02P6IiBhDnSdMzQFu7NjeBBxc45g5wOaRAknzgQOBy3ekoRERbTNZT7Kqc0WvHmUezzGSdgO+CJxq+7aeQaTlkgYlDQ4PD9doVkRE1FEn0W8C5nVszwVuqnuMpAdRkvx5tr+0vSC2V9leYnvJwMBAnbZHREQNdRL9WmChpAWSdgaOBlZ3HbMaOK4afXMIcKvtzZIEfBLYYPvDjbY8IiJqGbOP3vZWSScBFwGzgHNsr5d0QrV/JbAGOBwYAu4Ejq9OPxQ4FrhW0lVV2Vttr2n0VURExHbVuRlLlZjXdJWt7PjZwIk9zvsRvfvvIyJikmRmbEREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETLJdFHRLRcEn1ERMsl0UdEtFwSfUREyyXRR0S0XBJ9RETL1Ur0kg6TdIOkIUkreuyXpDOr/ddIWlz33IiI6K8xE72kWcBZwFJgEXCMpEVdhy0FFlZfy4Gzx3FuRET0UZ0r+oOAIdsbbW8BLgCWdR2zDDjXxWXAnpL2qXluRET0kWyPfoD0YuAw26+qto8FDrZ9UscxFwLvt/2javs7wJuB+WOd21HHcsqnAYD9gBvG8Tr2Av5rHMfvqMSZnjESZ/rGSJzJi/EY2wO9dsyucbJ6lHX/ddjeMXXOLYX2KmBVjfbcj6RB20t25NzE6W+cNr2WtsVp02tpW5ymY9RJ9JuAeR3bc4Gbah6zc41zIyKij+r00a8FFkpaIGln4Ghgddcxq4HjqtE3hwC32t5c89yIiOijMa/obW+VdBJwETALOMf2ekknVPtXAmuAw4Eh4E7g+NHO7cPr2KEun8SZlDhtei1ti9Om19K2OI3GGPNmbEREzGyZGRsR0XJJ9BERLZdEHxHRcjM60UvaWdIBkp5YjeqJaUDSI3qULZiKtjRJ0q5T3YaZRtJOkvaY6nY80M3Ym7GSjgBWAr+gTMxaALzG9jcajjNAmeW7CHjISLntZzcY42vcfyLZrcAg8DHbf5xg/YtH22/7ionU3yPej4Gltm+rthcBn7P9hIbjPJ0y+/re0WO2z20yRkecTwC72X60pCdR3mv/0GCMRwLvBA6lvBd+BLzb9m8bqv9aek9WFGDbBzQRp4p1PnAC8CdgHfAw4MO2P9hUjI5Yr7T9yY7tWcDbbL+r4Ti9fn8j/0ff08S/k6RDgats3yHp5cBi4Azbv5pw3TM40V8PHGl7qNp+HPB12/s3HOdi4P8Cb6S8ef8eGLb95gZjnAEMAP9RFb0UuBnYBdjD9rETrP97o+x2k3+0qnhHAG8CjqAsZ3Eu8DLbVzUY4zPA44CrKAkFyms5uakYHbEuB14MrLZ9YFX2syb/cEn6FvAD4LNV0cuAZ9p+bkP1P2a0/U0kk45YV9l+sqSXAU+hXCita/KPSUes84E9gVcCjwQ+BXzf9hsbjnMa5X12flV0dPX9NuAvbD+/gRjXAE8CDgA+A3wSeKHtZ0y0bmzPyC/gB13b6i5rKM666vs1HWXf7+dr6SwD1k/173oHX9NRwE+Aa4GFfah/A9WFyiS8lsur71d2lF3dcIx1PcoG+/R6/gdwZPX1qD7Uvx54EPB54Bn9+H11xXspZV2YXwOH9inGj7dXBlzbUIwrqu//DLyys2yiX3WWQJiu1ktaA3yO8pHqb4G1kl4IYPtLDcW5u/q+ubpSvYmylEOTBiQ92vavASQ9mrKoEcCWiVY+8jvZnqZ+V5I+yrYfb/cANgKvk4Sbvdr+GbA3sLnBOrfnxqr7xtW9oJMpf2ia9D1JR1Pez1A+QXy94RhIegnwQeASysXRRyX9k+0vNBjmY8AvgauBH1SfJm5rsP57SVoInAJ8EXg8cKykK23f2XCo3SQdbPvyKu5BwG7Vvq0Nxbhd0luAY4G/rLqhHtRExTO56+ZTPYpHFlKz7Vc0FOdI4IeUNXs+Skle77T9tSbqr2Iczv3vN/wD5T/jq21/ZIL19/pdjWjyd/X3o+23/ekGYozcz9gdeDLwU+CujhgvmGiMHjH3As4Ankv597kYOMUN9Z9XMW4HdqV0D4gyUOKOardtN3JDU9LVwPNs31JtDwDftv2kJuofJe5s200lxM56rwdOsv1tSQJeD7zC9p83HOepwDncl9xvB15F+fRyhO3Pbe/cccTYG/g7YK3tH1YXfM90A/edZnKi/zTlP9sfqu2HAx9qKmmNEucRwOl9iPNgYH/Kf/LrPcEbsG0ladT+Stvfn6y2zESSrrX9xI7tnSjdKk8c5bTxxngk8A7gL+jDjeWuWHu4uunfUbbQ9s+bjlXV/TBK3vxDn+rfm/IcD1MS/s1N1DuTu24O6Pxl2/69pAMnIc7v+hTnKdw3guSAqqujHyNIjgD+nG1HEL274RgLgfdx/5FKj51o3SOJvBquuXnkD6KkXSh9z42TdGaP4lspfehfnWDdkzoiCvimpIvY9sb/moZjXEC5sfyiavtllAENjdxY7rKLpH8D5tg+rBrh9TSg0UQv6b3AaV0Xlm+w/bYGY7yK0j//Xe7rVnu37XMmXPcMvqK/mvKx5vfV9iMoN0kbuzKZrDiTNYJE0krgocCzKMMFXwz81PYrG47zI8oV3b8Bz6cscifb72gwxiDwdJcnl1H1nf/Y9lObitERaxXl09bnq6IXUT6yzwM22j51AnX3GhF1739KNzwiqor5IsowzpEBDF9uuP51tp/SVdaXNdwlfYMy0uZ/236SpNmUm+ZN54ErXY246ii7wvaof6jHGeMGynv6t9X2I4Gf2N5vwpU3cUd3Kr6A4yg3xP4FeDdwPXDsTIzDJI0goRo51PF9N+DiPsQZGal0bUfZDxuOcVWPsr6M7KBcYc3u2J5dlc0CrmsoxksoQ2kB3g58GVjc7/dEn35fp1OGH+5Ufb0EeFefYq2tvl852nujgTjXAA/u2N6FhkfEAd8Bdu7Y3ply/2TCdc/Yrhvb51ZXdc+mXJm80PZ1MzTOZI0g+e/q+52S9gV+S7nx27Q/Vn2/P1dZpvo3wKMajjEs6QW2VwNIWkb/Hu82h3Kj9NZqe1dgX9t/knTX9k8bl7fZ/pykvwCeB3wIOBs4uKH6gXtHYH2A8u8h7hu80OTs1ddQboqOzAnYCbhD0uv7EOuO6sq3jLGunofRYP0jPgt8pxrYYOAVwIQHFwBUvxco/08ul/TVKsYyymCDCZuxiR6gSriNJ/cpiLMXcJ2kfo8guVDSnpThdVdQ3kyfaDgGwKmULqKTKZ+EnkWZaNakE4DzJP07JVndSPn01Q+nAVdJuqSK9VfAe1WWRPh2QzFGuuyOAFba/qqkdzZUd6fTgOfbbnp46L1s796vunt4PeVhRo9TmZE9QOmSbJTt06oJTSMjr/7F9kUNVX8g5Vkezwc+0lE+ofs/nWZsH32bbG8kifs4gqQa5fMQ2/24+hmJsavtO8Y+ckIxdqO8j2/vc5x9KeObr6dc0W+y/YMG67+QckX3XMqN+f+m3D9pdNijpB/bPrTJOrcT54XcN+rmh7a/0qc4f0t5sNE8yr2Tg4G3u/mb2COzixe6DOV8KDCrifedpOuApcDXgGd277f9uwnHSKKfepIWdXcHSXqm7Uv6EKvv68NIehpl+nbf1oap4vR9BFEV51WUSTlzKTfMDwEudbPrHT0UOIxyX+PnkvYBnmj74obqH5k09wxKN+FX2PbTY1MTDJH0f4D/ybYje35h+8SmYnTEusb2AVWX13spXV5vtd10l9ergeXAI2w/rhpZttL2cxqo+2TgtZRu1M5nao90q014tFoS/TQg6WeU9WA+SElapwFLbD+t4TiTNbpnMtaGmZQRRFWsa4GnApe5rOGyP+Xm4kubjtUvHZPmRiYVdrIbnBciaT3wBFfJpbpfc60bnsRU1X2l7QMlva+KcX6vETINxLmKMr798o739DZzEhqIcbbt1zZVX6cZ3UffIgdTbpD9hDLj8zzK8LemLQEWeRL+utu+sUxUvNeftnfsDnp6dSV3je13SfoQ0NhVaZc/2v6jJCQ92Pb1kiY+5G0S2T4etj/RsOFwNwCPBkYWSptHGbXSD7+R9DFKl9cHqi7Jfiy/fpftLSPv6WoYZ6P/j/qV5CGJfrq4m9Inuwvliv4/bd/ThziTNbpnMtaGmawRRACbqpvYXwG+Jen3bPsReybp20RD3bc8xcOADdXgAlMuZH7SRIweXkLp8jrd9h+qLq9/6kOc70t6K2WC1vMoS5Q0tgxKv6XrZhqoJmV9lTJOfy/KolB3225k9IAmeX0Ybbs2zE6Um2WnuNm1Yd5OWXvoOcBZVCOIbL+9qRjbifsMSiL7pqvJWjNJPycAbm9QwYh+Di7ot6r76ZXAX1O6vi6ivN9mRAJNop8GVFbC2w9YYPvdKosZHWf7PQ3V/wzKm/MDlHXi790FfKDpG1eTbTJGELWFpOOAtwBfoPxxfAnwr7Y/02CMD7jreQ29ymYqSYv7Maqnn5LopwFJZwP3AM+2/fiq3/RiNzydv9eU7ZFRCw3HeSzliv4QSjK5FPhH2xsbqHtSllxuM5X1YEYmAH6n6QmAk/U+mypNL30wGdJHPz0cbHuxpCvh3n7TRtahBpD0Wkqf4mOrSR8jdgd+3FScDudTulP+pto+mjLUrolPDqM9ycf074Zsa/RrAmDH++xxk/Q+myrdo5amvVzRTwPVcMSnU9btWKyyRvjFTQ0RU1la9eGUFSVXdOy6vYnJGD3iXd7dHSTpMtuHNB0rpo/Jfp9NFUlH9WsCWL8k0U8DKs/WfCnlYcCfpowJf5vtz4964jQl6f3AHyjL1Zry2h5MucpvZKZfFWdSJkxFfdVNy2uanDMxHUj6TvfkqF5l01W6bqYB2+dJWkcZQSLgqH6uRTIJRiYSvaar/BWUxD/xmX7bmTA10XpjYmzfI+lqdTwacyaT9BDK+2yv6t7ZSLfNHsC+U9awccoVfcxIHVPfR77vBnzJ9l9Pddse6CR9lzKT+Kfc9zjEfizS13eSTqEs0rcvZS2ikUR/G/Bx2/8+RU0bl1zRR2MmeUTMZE6YivHZDTiyY3tkaO+MY/sM4AxJr7P90aluz45Koo8mjYyIeRTl5vJ3q+1nUR503mSin6wll2P8ZndPjlJ51ONMdrOk3W3fLultlPtp75kp4+nTdRONU1ly99W2N1fb+wBn2R71in8C8TJhahroHMYL/KJj1+6Uxzy+fEoa1oCuVTLfR3mKVuOrZPZLruijH+aPJPnK/wP+rOkg3Usuq08PVI/azge+QTuHV3Y+GOZs9+/BMH2RK/ponMpTnxZSJkmZMmFqyPbrGowxKUsuR8DkPRimX5Looy+qG7N/WW3+wPaXG65/A5O05HKE+vxgmH5L1030RTXCpp/LEUzWkssR2L5T0i2UxyP+HNhafZ8RckUfjZF0O/c9wajzjTXySLQ9GogxqUsuRwBIegflwT372f6zakjv5z0Jz99tQq7oozG2dx/5WdKT2bbr5uqGwpzOfeOyj+oon7FjtWNG+BvgQMpQXmzfJGn30U+ZPpLoo3EqDzt+NaXrRsBnJH28iQknI+OzJT2ohWO1Y/raYtuSRp6Du+tUN2g8kuijH14FHGL7DigPnaCsST/hRD8FSy7HA5zKg2IvrJ5Nu6ekV1PWbfr41LasviT66Aex7cPA/0Rza3i3eax2TEPVlfxRwJspa9zsB/yz7W9NacPGIYk++uFTwOWSRoZUHgV8somKq9mvtwLHNFFfRE2XAn+w3Y8Hj/ddRt1EX0haTBmKJsrN2CunuEkRO0zSdZTZ3b9i2xU5Z8TjEZPoIyLGIOkxvcpt/2qy27IjkugjIlpup6luQERE9FcSfUREyyXRR0S0XBJ9RETLJdFHRLTc/wd1TX8VWL1B/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['age', 'sex', 'cp', 'trtbps', 'chol', 'fbs', 'rest-ecg', 'thalach', 'exng', 'oldpeak', 'slp', 'ca', 'thall']\n"
     ]
    }
   ],
   "source": [
    "X_shape =df.iloc[:,0:13]\n",
    "Y_shape =df['output']\n",
    "\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "model =ExtraTreesClassifier()\n",
    "\n",
    "model.fit(X_shape, Y_shape)\n",
    "print(model.feature_importances_)\n",
    "feat_importances =pd.Series(model.feature_importances_, index=X_shape.columns)\n",
    "#show 13 columns of our fit\n",
    "feat_importances.nlargest(13).plot.bar()\n",
    "plt.show()\n",
    "\n",
    "list1=feat_importances.keys().to_list()\n",
    "print(list1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c17b1c4",
   "metadata": {},
   "source": [
    "Show hist of values have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9922c1ce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<AxesSubplot:title={'center':'age'}>,\n",
       "        <AxesSubplot:title={'center':'sex'}>,\n",
       "        <AxesSubplot:title={'center':'cp'}>,\n",
       "        <AxesSubplot:title={'center':'trtbps'}>],\n",
       "       [<AxesSubplot:title={'center':'chol'}>,\n",
       "        <AxesSubplot:title={'center':'fbs'}>,\n",
       "        <AxesSubplot:title={'center':'rest-ecg'}>,\n",
       "        <AxesSubplot:title={'center':'thalach'}>],\n",
       "       [<AxesSubplot:title={'center':'exng'}>,\n",
       "        <AxesSubplot:title={'center':'oldpeak'}>,\n",
       "        <AxesSubplot:title={'center':'slp'}>,\n",
       "        <AxesSubplot:title={'center':'ca'}>],\n",
       "       [<AxesSubplot:title={'center':'thall'}>,\n",
       "        <AxesSubplot:title={'center':'output'}>, <AxesSubplot:>,\n",
       "        <AxesSubplot:>]], dtype=object)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA28AAAJOCAYAAAAgSD/bAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABrD0lEQVR4nO39f9xldV3v/z+eIRoOKBI6IYwOnsgTSpJOpNGpMVMRVPRz0gOHFI50qI6UnjOnHOx81PLwaeobWllaqAQminxKDhwxk4Ne+e2HIhgKiOikowwzMgkIDJU2+Pr8sdYVm4vr9/619rUf99tt3/Ze7/Vjv9ba673Xfu31XuudqkKSJEmS1G3fNe4AJEmSJElLM3mTJEmSpAlg8iZJkiRJE8DkTZIkSZImgMmbJEmSJE0AkzdJkiRJmgAmb5IkSZIWleSMJH817jimncmbJEmSNEWS7EjyU4uM35ikkjxslHFpaSZvkiRJkgAwYes2k7cJlGRrkr9Pcm+Szyd5aVu+X5LzknwjyVeSnN37r0mSRyd5d5LdSW5L8j+T7DfetZEmR5LXtXXn3iS3JHlOku/qqZN3JLk0ySHt9O9I8qc98/9mkquTZHxrIU2uJBuSfDDJP7T17ffbplx/neRtSe5O8oUkzxl3rFJXJfkT4AnA/06yN8mvtL8Xz0zyNeBjwCfayb/ZTvOsB2afv64lmUnyG0muacdf3nM8/O4k723r7TeTfDrJ+lGu91ph8jaZ/h74d8CjgV8D3pvkMOA/Ay8AjgWeDrxkznwXAfuA7wN+CHge8LMjiViacEmeDJwN/HBVHQQ8H9gB/BJNXfsJ4PHAXcAftLNtAX6w/XH574AzgdOrqkYbvTT52j8bPwR8FdgIHA5c0o7+EeDLwKHAG4EPzv5olPRgVfUK4GvAi6rqQODSdtRPAD9Ac3z78bbs4Ko6sKr+th1eqq69EngVzfFwH/B7bfnpNL9bNwDfA/w88E+DX7u1z+RtAlXV/1tVu6rqO1X1AeBLwHHAy4HfraqdVXUXsG12nvbfjRcAr62q+6pqD/BW4JQxrII0ie4HHgEcnWT/qtpRVX8P/Bzwq229+xbwJuCnkzysqv4R+BngLcB7gV+sqp1jil+adMfR/CD85fY49s9VNXvzhD3A71TVv7THxVuAk8YVqDSh3tTWrcWSqqXq2p9U1Y1VdR/wfwMvb/94+ReapO37qur+qrququ4Z1oqsZSZvEyjJK5Nc3552/ibwVJp/QB4P3Nozae/rJwL7A7t75vsj4HGjiVqabFW1HXgtTXK2J8klSR5PU7cu66lXN9Mkeuvb+a6h+ZcyPPDvpqSV2wB8tar2zTPutjlntL9Kc0yUtHy3Lj3JknXt1jnj9qf5jfonwF8AlyTZleS3kuzfb8DTyORtwiR5IvBOmuZb31NVBwM30vww3A0c0TP5hp7XtwLfAg6tqoPbx6Oq6imjiVyafFX1vqr6MZqErYDfpKlbL+ipVwdX1XdX1W0ASV5Nc8ZuF/Ar44pdWgNuBZ6wwM0UDp9zLekTaOqcpPnN13y/lhgPS9e1DXPG/QvwjfZM3a9V1dHAjwIvpGliqRUyeZs862gq1D8AJPlPNGfeoPlX/zVJDk9yMPC62ZmqajfwUeC8JI9qb7Lwb5L8xEijlyZUkicn+ckkjwD+maat/v3AHwLntn+skOSxSU5uX38/8D9pmk6+AviVJMeOI35pDbiG5k/KbUnWtTdAOL4d9zjgl5Lsn+RlNNftfHhcgUoT4HbgSYuM/wfgO/NMs1Rd+5kkRyd5JPDrwJ9W1f1Jnp3kmLYJ5T00Sd39g1qZaWLyNmGq6vPAecDf0lS8Y4C/bke/kyZB+xzwdzSVaR8PVI5XAg8HPk9zU4U/BQ4bVezShHsEzXWk3wC+TnMAez3wu8AVwEeT3At8EviR9uzAe4HfrKrPVtWX2un/pE0AJa1AVd0PvIjmpltfA3YC/6Ed/SngKJr6eS7w01V1xzjilCbEbwD/o23u/9NzR7bXbJ8L/HV7WcAz21FL1bU/AS6kOU5+N81NvQC+l+Z35z00lxf8Jc0xUisUb3q2diV5AfCHVfXEccciSdIwJDkD+Nm2SbOkMUkyA7y3qt417ljWMs+8rSFJDkhyYpKHJTmc5haul407LkmSJEn9M3lbW0LT79tdNM0mbwbeMNaIJGlCJLkgyZ4kN/aUvSlNx+zXt48Te8adk2R7mg7bnz+eqCVJ08Rmk5IkAUl+HNgLvKeqntqWvQnYW1W/PWfao4H380DfY/8H+P72uixJkobCM2+SJAFV9QngzmVOfjJwSVV9q6q+AmynSeQkSRqa+fpKGblDDz20Nm7cOO4wlu2+++5j3bp14w5jbKZ5/a+77rpvVNVjxx3HICxV77r8ORvb6nQ1tqXi6kC9OzvJK4FrgS1VdRdwOM2dRWftbMseIslZwFkABxxwwDM2bNgw32QAfOc73+G7vmvt/K/q+nTXUuvyxS9+cdz1bqD6/a3Zxe/PrsXUtXhg8mJa1vGuqsb+eMYznlGT5OMf//i4QxiraV5/4NrqQJ0ZxGOpetflz9nYVqersS0V1yjrHbARuLFneD2wH01LlXOBC9ryPwB+pme6dwP/fqnlT3K9Ww3Xp7u6VO9G8ej3t2YXP/uuxdS1eKomL6bl1Lu18feRJElDUFW3V9X9VfUdmr40Z5tG7gR6T6EdAewadXySpOli8iZJ0gKSHNYz+FJg9k6UVwCnJHlEkiNpOq29ZtTxSZKmSyeueZMkadySvB/YDByaZCdNX5mbkxwLFLAD+DmAqropyaXA54F9wKvLO01KkobM5G2Kbdx65arm23LMPs7omXfHtpMGFZKkCbPa75FeF57QjYvJq+rUeYrfvcj059JcBzcwN9x294O+X1fD72Rpug3ie9nvke6y2aTUQUk2JPl4kpuT3JTkNW35IUmuSvKl9vkxPfPYYbAkaeIkuSDJniQ39pR5vJPmYfImddM+mluS/wDwTODVbafAW4Grq+oo4Op2eLbD4FOApwAnAG9Pst9YIpckaWUupDl29fJ4J83D5E3qoKraXVWfaV/fC9xM04fUycBF7WQXAS9pX9thsCRpIlXVJ4A75xR7vJPm4TVvUscl2Qj8EPApYH1V7YYmwUvyuHayZXUY3NtZ8Pr165mZmVnwfffu3bvo+HEyttUZRmxbjtnX9zK6vM0kjU1fxztY2TFvKV38nloopkF8L69mXSdpG41TvzGZvEkdluRA4M+A11bVPUkWnHSesnpIQdX5wPkAmzZtqs2bNy/43jMzMyw2fpyMbXWGEVu/N9eA5oYlXd1mkjpnWcc7WNkxbyld/G5fKKZBfC/vOO2hy11tPOO0FmOy2aTUUUn2p0ncLq6qD7bFt8/2O9U+72nL7TBYkrSWeLyT5mHyJnVQmlNs7wZurqq39Iy6Aji9fX06cHlPuR0GS5LWCo930jxsNil10/HAK4Abklzflr0e2AZcmuRM4GvAy8AOgyVJkyvJ+4HNwKFJdgJvxOOdNC+TN6mDquqvmL9dP8BzFphn4B0GS5I0bFV16gKjPN5Jc9hsUpIkSZImgMmbJEmSJE0AkzdJkiRJmgAmb5IkSZI0AUzeJEmSJGkC9JW8JTk4yZ8m+UKSm5M8K8khSa5K8qX2+TGDClaSJEmSplW/Z95+F/hIVf1b4GnAzcBW4OqqOgq4uh2WJEmSJPVh1clbkkcBPw68G6Cqvl1V3wROBi5qJ7sIeEl/IUqSJEmS+umk+0nAPwB/nORpwHXAa4D1VbUboKp2J3ncfDMnOQs4C2D9+vXMzMz0Ecpo7d27d6LiXciWY/atar71Bzx43rWwLSRJkqSu6yd5exjwdOAXq+pTSX6XFTSRrKrzgfMBNm3aVJs3b+4jlNGamZlhkuJdyBlbr1zVfFuO2cd5Nzyw6+w4bfOAIpIkSZK0kH6uedsJ7KyqT7XDf0qTzN2e5DCA9nlPfyFKkiRJkladvFXV14Fbkzy5LXoO8HngCuD0tux04PK+IpQkSZIk9dVsEuAXgYuTPBz4MvCfaBLCS5OcCXwNeFmf76F5bFxlk0dJkiR100p+3205Zt+qL4HR5Ooreauq64FN84x6Tj/LlSRJkiQ9WL/9vEmSJEmSRsDkTZIkSZImQL/XvEmSJEkD194U7wM9RU8C3gAcDPxnmv6GAV5fVR8ebXTSeHjmTeqgJBck2ZPkxp6yNyW5Lcn17ePEnnHnJNme5JYkzx9P1JIkDU5V3VJVx1bVscAzgH8ELmtHv3V2nImbpoln3tS3Qdz5cse2kwYQyZpyIfD7wHvmlL+1qn67tyDJ0cApwFOAxwP/J8n3V9X9owhUkqQReA7w91X11STjjkUaG5M3qYOq6hNJNi5z8pOBS6rqW8BXkmwHjgP+dljxSWtRkguAFwJ7quqpbdkhNM22NgI7gJdX1V3tuHOAM4H7gV+qqr8YQ9jStDgFeH/P8NlJXglcC2yZrZe9kpwFnAWwfv16ZmZmVv3me/fu7Wv+5dpyzL5lT7v+gJVNvxKrWddRbaOVWIsxmbxJk2W+g9XhwCd7ptnZlj3ESg5kXfzCm2VsqzOM2Abxw6FD2+xCHnrGeytwdVVtS7K1HX6dZ7yl0Wn7E34xcE5b9A7gzUC1z+cBr5o7X1WdD5wPsGnTptq8efOqY5iZmaGf+ZdrJf22bTlmH+fdMJyf8jtO27zieUa1jVZiLcZk8iZNjoUOVvO1H6n5FrCSA1kXv/BmGdvqDCO2QXQQe+EJ6zqxzRY4430ysLl9fREwA7wOz3hLo/QC4DNVdTvA7DNAkncCHxpXYNKombxJE2KRg9VOYEPPpEcAu0YYmrSWra+q3QBVtTvJ49ryoZzxHkQzqI6cxQQ6dVZ1INbS+kzYupxKT5PJJIfN1kvgpcCN884lrUEmb9KEWORgdQXwviRvoWm+dRRwzRhClKbJUM54v+3iy/tuBrWa5k7D0uUz0auxltZnUtYlySOB5wI/11P8W0mOpalzO+aMk9Y0kzepg5K8n6ap1qFJdgJvBDbPd7CqqpuSXAp8HtgHvNrrbqSBuX32j5MkhwF72nLPeEsjUFX/CHzPnLJXjCkcaexM3qQOqqpT5yl+9yLTnwucO7yIpKl1BXA6sK19vryn3DPekqSRMnmTJIkFz3hvAy5NcibwNeBl4BlvSdJ4mLxJksSCZ7yh6Rx4vuk94y1JGqnvGncAkiRJkqSlmbxJkiRJ0gQweZMkSZKkCWDyJkmSJEkTwBuWSJIkLcMNt93NGVuv7GsZO7adNKBoJE0jz7xJkiRJ0gToO3lLsl+Sv0vyoXb4kCRXJflS+/yY/sOUJEmSpOk2iDNvrwFu7hneClxdVUcBV7fDkiRJkqQ+9JW8JTkCOAl4V0/xycBF7euLgJf08x6SJEmSpP5vWPI7wK8AB/WUra+q3QBVtTvJ4+abMclZwFkA69evZ2Zmps9QRuOG2+5m/QHwtosvX/Uyjjn80X3HseWYfX0vY7XWHzD495+Uz1+SJEkal1Unb0leCOypquuSbF7p/FV1PnA+wKZNm2rz5hUvYizO2HolW47Zx3k3rD7v3XHa5oHEMS79rv98BrFNJEmSpLWsn1/gxwMvTnIi8N3Ao5K8F7g9yWHtWbfDgD2DCHQt2TjGxEuSJEnSZFr1NW9VdU5VHVFVG4FTgI9V1c8AVwCnt5OdDqy+faEkSZIkCRhOP2/bgOcm+RLw3HZYkiRJktSHgSRvVTVTVS9sX99RVc+pqqPa5zsH8R7SNElyQZI9SW7sKVuwD8Uk5yTZnuSWJM8fT9SSJA1Wkh1JbkhyfZJr2zL7FNbUGsaZN0n9uxA4YU7ZvH0oJjmapunyU9p53p5kv9GFKknSUD27qo6tqk3tsH0Ka2qZvEkdVFWfAOaetV6oD8WTgUuq6ltV9RVgO3DcKOKUJGkM7FNYU2uw93uXNEwL9aF4OPDJnul2tmUPsZL+Fffu3dvZ/veMbXWGEdsg+nzs8jaTNHYFfDRJAX/UdjU18j6FR/U9tZLv1GH0uztrNevaxe/ytRiTyZs0+TJPWc034Ur6V5yZmaGr/S8a2+oMI7ZB9Dl54QnrOrvNJI3d8VW1q03QrkryheXOOMg+hUf13b6S79Rh9Ls7azX973bx+LcWYzJ5kybHQn0o7gQ29Ex3BLCr3ze74ba7+/5hvmPbSf2GIUmaYlW1q33ek+QymssC7FNYU8tr3qTJsVAfilcApyR5RJIjgaOAa8YQnyRJA5NkXZKDZl8DzwNuxD6FNcU88yZ1UJL3A5uBQ5PsBN5I02fipUnOBL4GvAygqm5KcinweWAf8Oqqun8sgUuSNDjrgcuSQPOb9X1V9ZEkn2ae46E0DUzepA6qqlMXGPWcBaY/Fzh3eBFJkjRaVfVl4GnzlN/BAsdDaa2z2aQkSZIkTQCTN0mSJEmaACZvkiRJkjQBTN4kSZIkaQJ4wxJJkiSpTxv77BtVWg6TN0mSliHJDuBe4H5gX1VtSnII8AFgI7ADeHlV3TWuGCVJa5vNJiVJWr5nV9WxVbWpHd4KXF1VRwFXt8OSJA2FyZskSat3MnBR+/oi4CXjC0WStNbZbFKSpOUp4KNJCvijqjofWF9VuwGqaneSx82dKclZwFkA69evZ2ZmZsE3WH8AbDlmX19BLrb8Udtz59287eLL+1rGMYc/ekDR9G8tfT579+7tTCzqntVcv7flmH2cMWe+HdtOGlRIapm8SZK0PMdX1a42QbsqyReWM1Ob5J0PsGnTptq8efOC077t4ss574b+Ds07Tlt4+aPm+jxUV9ZnZmaGxfZFSd1ks0lJkpahqna1z3uAy4DjgNuTHAbQPu8ZX4SSpLXOM2/qhEHcXtdT85KGJck64Luq6t729fOAXweuAE4HtrXP/bURlCRpESZvkiQtbT1wWRJojp3vq6qPJPk0cGmSM4GvAS8bY4ySpDVu1clbkg3Ae4DvBb4DnF9Vv2ufN5Kktaaqvgw8bZ7yO4DnjD4iSdI06ueat33Alqr6AeCZwKuTHI193kiSJEnSwK06eauq3VX1mfb1vcDNwOHY540kSZIkDdxArnlLshH4IeBTLKPPm3aeZfd70yVbjtk3kH5eJllX139S9iFJkrS0RS7ReRPwn4F/aCd9fVV9eDxRSqPVd/KW5EDgz4DXVtU97cXcS1pJvzddcsbWK9lyzL6++3mZZF1d/670nTNsSXYA9wL3A/uqapPXmkqS1qDZS3Q+k+Qg4LokV7Xj3lpVvz3G2KSx6KuftyT70yRuF1fVB9ti+7yRhu/ZVXVsVW1qh73WVJK0pixyiY40tfq522SAdwM3V9VbekbZ5400eicDm9vXFwEzwOvGFYwkSYM05xKd44Gzk7wSuJbm7NxDWpsM8hKdvXv3Ljn/qC8p6dplLPPFM+5LWpbzuY1avzH10/bteOAVwA1Jrm/LXk+TtNnnjTQ8BXw0SQF/1DZBHvi1poM4KAzrC7OLX8azpi22Qfxw6PI2kzR+81yi8w7gzTTHwzcD5wGvmjvfIC/RmZmZYan5z9h65aqXvxpdu4xlvnjGfUnLcj63Ues3plV/4lX1V8BCF7jZ5400PMdX1a42QbsqyReWO+NKDmRvu/jyvg8Kw/rS7uKX8axpi20QP1YuPGFdZ7eZpPGa7xKdqrq9Z/w7gQ+NKTxp5Pq65k3S6FXVrvZ5D3AZcBxeaypJWmMWukRn9njXeilw46hjk8bF5E2aIEnWtXfcIsk64Hk0B63Za03Ba00lSWvD7CU6P5nk+vZxIvBbSW5I8jng2cB/HWuU0gh1p6GspOVYD1zWdsnxMOB9VfWRJJ/Ga00lSWvIIpfo2KebppbJmzRBqurLwNPmKb8DrzWVJEla06Ymeds44jsASZIkSdIgec2bJEmSJE2AqTnzJkmStBYMojXRhSesG0AkkkbNM2+SJEmSNAE88yZJkiRp4AZxlnjHtpMGEMna4Zk3SZIkSZoAnnmTJEnS1FrO2aEtx+zjDO9crg7wzJskSZIkTQCTN0mSJEmaACZvkiRJkjQBTN4kSZIkaQKYvEmSJEnSBPBuk5IkSZI6qZ++4mbvErqW+orzzJskSZIkTQDPvEmSJElas/o5ezerK2fvJiJ5G8QG19q3liqmJEmSNNfQmk0mOSHJLUm2J9k6rPeR9ADrnTR61jtp9Kx3mlZDOfOWZD/gD4DnAjuBTye5oqo+P4z3k2S9k8bBeieNnvVO4zColoAXnrCur/mHdebtOGB7VX25qr4NXAKcPKT3ktSw3kmjZ72TRs96p6mVqhr8QpOfBk6oqp9th18B/EhVnd0zzVnAWe3gk4FbBh7I8BwKfGPcQYzRNK//E6vqseMOYj5DqHdd/pyNbXW6GttScVnvJpfr011rut615YP8rdnFz75rMXUtHpi8mJasd8O6YUnmKXtQllhV5wPnD+n9hyrJtVW1adxxjMu0r3+HDbTedflzNrbV6WpsXY1rmaam3q2G69NdE74uS9Y7GOxvzS5ur67F1LV4YG3GNKxmkzuBDT3DRwC7hvRekhrWO2n0rHfS6FnvNLWGlbx9GjgqyZFJHg6cAlwxpPeS1LDeSaNnvZNGz3qnqTWUZpNVtS/J2cBfAPsBF1TVTcN4rzGZyOaeAzTt699JQ6h3Xf6cjW11uhpbV+Na0pTVu9VwfbprYtdlTL8zu7i9uhZT1+KBNRjTUG5YIkmSJEkarKF10i1JkiRJGhyTN0mSJEmaACZvy5BkvyR/l+RD7fAhSa5K8qX2+THjjnGYkhyc5E+TfCHJzUmeNW3bYK1JckKSW5JsT7J1nvFJ8nvt+M8lefpy5x1yXKe18Xwuyd8keVrPuB1JbkhyfZJrBxnXMmPbnOTu9v2vT/KG5c47gth+uSeuG5Pcn+SQdtzQtluSC5LsSXLjAuPHsp911Vpa56U++0mSZEOSj7fHv5uSvGbcMfUjyXcnuSbJZ9v1+bVxx9QF8+2zi/3WSXJOW1dvSfL8EcXz/2t/i30uyWVJDm7LNyb5p57v+T8cdDyLxPSmJLf1vPeJPePGsY0+0BPLjiTXt+Wj2kbzfl8MdF+qKh9LPID/BrwP+FA7/FvA1vb1VuA3xx3jkNf/IuBn29cPBw6etm2wlh40F3f/PfCk9vP8LHD0nGlOBP6cpi+dZwKfWu68Q47rR4HHtK9fMBtXO7wDOHSM22zz7HfESucddmxzpn8R8LERbbcfB54O3LjA+JHvZ119rLV1Xuqzn6QHcBjw9Pb1QcAXJ/yzCXBg+3p/4FPAM8cd17gf8+2zC/3WAY5u6+gjgCPburvfCOJ5HvCw9vVv9sSzcRR1bYGY3gT893mmHcs2mjP+POANI95G835fDHJf8szbEpIcAZwEvKun+GSahIb2+SUjDmtkkjyKpnK8G6Cqvl1V32SKtsEadBywvaq+XFXfBi6h+Tx7nQy8pxqfBA5Octgy5x1aXFX1N1V1Vzv4SZq+fUahn/Ue5jZbzfJPBd4/wPdfUFV9ArhzkUnGsZ911Zpa52V89hOjqnZX1Wfa1/cCNwOHjzeq1Wvr2952cP/2MfV3r1tgn13ot87JwCVV9a2q+gqwnaYODzWeqvpoVe1rB0d5DFwwpkWMZRvNShLg5YzoeNcT00LfFwPbl0zelvY7wK8A3+kpW19Vu6H5kIDHjSGuUXkS8A/AH6dpOvquJOuYrm2w1hwO3NozvJOH/hBZaJrlzDvMuHqdSXPWZlYBH01yXZKzBhTTSmN7VtsU6c+TPGWF8w47NpI8EjgB+LOe4mFut6WMYz/rqmlc54mTZCPwQzRnqyZWmstBrgf2AFdV1USvzxAt9FunC/X1VTz4GHhk+zvtL5P8uxHHcnbblPOCnuaA495G/w64vaq+1FM20m005/tiYPuSydsikrwQ2FNV1407ljF6GM0p6XdU1Q8B99Gc7tXkyjxlc/91XWia5cy7WstedpJn0yRvr+spPr6qnk7TnPLVSX58QHEtN7bPAE+sqqcBbwP+1wrm7cdKlv8i4K+rqvefymFut6WMYz/rqmlc54mS5ECaPz5eW1X3jDueflTV/VV1LM2Zm+OSPHXMIU2asdbXJL8K7AMubot2A09of6f9N+B9bcupUXgH8G+AY9s4zpsNc55pR/mdNreVyUi30Qq+L1a8nUzeFnc88OIkO2iasPxkkvcCt7dNe2if94wvxKHbCezs+VfuT2mSuWnaBmvNTmBDz/ARwK5lTrOceYcZF0l+kKYZ88lVdcdseVXtap/3AJcx2OYZS8ZWVffMNkWqqg8D+yc5dDnzDju2HqcwpwnJkLfbUsaxn3XVNK7zxEiyP80PsYur6oPjjmdQ2ssgZmjOyOuhFvqtM7b6muR04IXAadVeNNU2ubujfX0dzXVT3z+KeKrq9vbPgO8A7+SBY8g4t9HDgP8L+EBPnCPbRgt8XwxsXzJ5W0RVnVNVR1TVRpofPR+rqp8BrgBObyc7Hbh8TCEOXVV9Hbg1yZPboucAn2eKtsEa9GngqCRHJnk4zb59xZxprgBemcYzgbvb0/zLmXdocSV5AvBB4BVV9cWe8nVJDpp9TXNR9yDvcrec2L63bWNPkuNovl/vWM68w46tjenRwE/QU1dHsN2WMo79rKumcZ0nQluv3w3cXFVvGXc8/Ury2Dxwl8IDgJ8CvjDWoLprod86VwCnJHlEkiOBo4Brhh1MkhNoWpy8uKr+saf8sUn2a18/qY3ny8OOp32/w3oGX8oDx5CxbKPWTwFfqKqdPXGOZBst8n0xuH1psbuZ+HjQ3WM288DdJr8HuBr4Uvt8yLjjG/K6HwtcC3yOpinYY6ZtG6y1B81d/r5I88/Tr7ZlPw/8fPs6wB+0428ANi027wjjehdwF3B9+7i2LX8Szd2aPgvcNOi4lhnb2e17f5bmQvIfHcU2W05s7fAZNBdF98431O1Gc5ZvN/AvNP8untmF/ayrj7W0zvN99uOOqY91+TGaZkyf6/nuOXHccfWxPj8I/F27PjfS3o1v2h8LfF8t+FsH+NW2rt4CvGBE8WynuT5qdj/8w3baf99z/PkM8KIRbqM/ab+/P0eTiBw2zm3Ull/Ye/wb8Taa9/tikPtS2pkkSZIkSR1ms0lJkiRJmgAmb2tAkjOS/NUq531TexMWScuQ5MntrYbvTXJnkv857pgkSVquJBuTVHtjj9XMX0m+r88YLvT4uTomb5K0Mr8CzFTVQXhDCWlk+vmjUpp2SXYk+alxx6H+mbxJ0so8keaiZ0mrsNp/+yVJJm8TJ8mGJB9M8g9J7kjy+z3jfjvJXUm+kuQFPeWPT3JF28Rre5L/PJ7opcmW5GPAs4HfT7IXeDhwaJKr2maUf5nkie20SfLWJHuS3J3kc7EjXE2p9l//1yX5HHBfkh9L8jdJvpnks0k290x7RpIvt3XqK0lOS/IDwB8Cz0qyN8k3F3mvFya5vl3236TpG3J23LzH0CT7JTkvyTfa9zy7n2ZlUpck+RPgCcD/bo9dL29HnZbka+1+/6s90x+X5G/bOrQ7ye+3XZjMt+yT2ksJ7klya5I3zRnfW9dvTXJGz+jHJLmyreufSvJvBrria5TJ2wRp+6f4EPBVYCNwOE3n4QA/QnOL0UOB3wLe3fY1Ac2tVHcCjwd+Gvh/kjxndJFLa0NV/STw/wfOrqoDgW8DpwFvpql71wMXt5M/D/hxmk5ADwb+A02/b9K0OhU4iaaLisuB/wkcAvx34M/afpjWAb9Hc7vsg4AfBa6vqptpupn426o6sKoOnu8NkjwduAD4OZpbc/8RcEXbh9Jix9D/DLyApmucpwMvGeSKS+NUVa8AvkZze/wDgUvbUT8GPJmmD983tH+SANwP/Fea49qz2vH/ZYHF3we8kuY4dxLwC0leAv/aN+ufA28DHktTv67vmfdU4NdouqDaDpzbz3pOC5O3yXIcTQL2y1V1X1X9c1XNtv//alW9s6ruBy4CDgPWJ9lAUzlf105/PU1fWa8YQ/zSWnRlVX2iqr5F01fLs9p69y/AQcC/BVJVN1fTAbU0rX6vqm4Ffgb4cFV9uKq+U1VX0fQlemI73XeApyY5oKp2V9VKmin/Z+CPqupTVXV/VV0EfAt4JosfQ18O/G5V7ayqu4Bt/a+u1Hm/VlX/VFWz/X0+DaCqrquqT1bVvqraQfMnyE/Mt4CqmqmqG9q6/DmaEwaz054G/J+qen9V/UtV3dH+Dp31waq6pqr20fzxeewwVnKtMXmbLBtokrR984z7+uyLqvrH9uWBNAeqO6vq3p5pv0rzj6Ok/t06+6Kq9gJ3Ao+vqo8Bv0/TCfXtSc5P8qgxxSh1wWxdeSLwsrYZ1TfbJpA/RtO57300Z6l/HtjdNqn6t/MtLMkT2iaUe9umYLPL3jJn2RtojoWLHUMf3xMfc15La9XXe17/I83vRpJ8f5IPJfl6knuA/4fmLNxDJPmRJB9vmyLfTVN3Z6fdQNP59IreX4szeZsstwJPWGEb/F3AIUkO6il7AnDbQCOTpteG2RdJDqRpBrYLoKp+r6qeATyFpvnkL48lQqkbqn2+FfiTqjq457GuqrYBVNVfVNVzaVqQfAF455z5aaf7WtuE8sC2Kdjsss+ds+xHVtX7WfwYuhs4omd4wzzTSJOslp7kX72Dpu4dVVWPAl4PZIFp30dz5+UNVfVommtTZ6e9FfA6tgEzeZss19AcYLYlWZfku5Mcv9gMbROVvwF+o53+B4EzeeC6HEn9ObG9IPvhNNe+faqqbk3yw+0/kvvTXBPwzzTXEUjT7r3Ai5I8v71RyHcn2ZzkiCTrk7y4vfbtW8BeHqg3twNHLHTjhNY7gZ9v617aY+VJ7R+Yix1DLwVek+TwJAcDrxvCekvjdDvN9abLcRBwD7C3PfP9C0tMe2dV/XOS44D/2DPuYuCnkrw8ycOSfE+SY1cRu3qYvE2Q9nq2FwHfR3Ph6U6a5iVLOZXm4uxdwGXAG9trDCT1733AG2maSz6Dpo0/wKNofkjeRdNU+Q7gt8cRoNQl7Z+KJ9P8m/8PNP/O/zLNb5LvArbQHK/upLl2ZvZGCR+j6abj60m+scCyr6W57u33aereduCMdtxix9B3Ah8FPgf8HfBhYB/+4aK14zeA/9E2Jf7pJab97zRJ2L00deMDi0z7X4BfT3Iv8AYeuBkKVfU1mmtZt9DU5+tpr6vT6qVqJWdRJUmS1rY03e38YVU9cdyxSFIvz7xJkqSpluSAJCe2TbsOpzmbftm445KkuTzzJkmSplqSRwJ/SdO1xz8BVwKvqap7xhqYJM1h8iZJkiRJE8Bmk5IkSZI0AVbSX9jQHHroobVx48Zxh8F9993HunXrxh3Gv+pSPF2KBcYXz3XXXfeNqnrsyN94CJaqd137zHsZ2+p0Nbal4rLedYOxrU5XY5umegfd+a25lK7uL8Pguj7UsupdVY398YxnPKO64OMf//i4Q3iQLsXTpViqxhcPcG11oM4M4rFUvevaZ97L2Fanq7EtFZf1rhuMbXW6Gts01btaRt3riq7uL8Pguj7UcuqdzSYlSZIkaQKYvEmSJEnSBDB5kyQJSHJBkj1Jbuwpe1OS25Jc3z5O7Bl3TpLtSW5J8vzxRC1JmiYmb5IkNS4ETpin/K1VdWz7+DBAkqOBU4CntPO8Pcl+I4tUkjSVOnG3yUmyceuVfS9jx7aTBhCJNFw33HY3Z/S5v7uva5JU1SeSbFzm5CcDl1TVt4CvJNkOHAf87bDim1bLOe5uOWbfot9XfhdJk8vf3g9m8iZJ0uLOTvJK4FpgS1XdBRwOfLJnmp1t2UMkOQs4C2D9+vXMzMws+EZ79+5ddPw4jSu2LcfsW3Ka9QcsPt04t2lXP9OuxiVpcSZvkiQt7B3Am4Fqn88DXgVknmlrvgVU1fnA+QCbNm2qzZs3L/hmMzMzLDZ+nMYV23JaAGw5Zh/n3bDwT5odp20eYEQr09XPtKtxSVqcyZskSQuoqttnXyd5J/ChdnAnsKFn0iOAXf2+n82VJUmLMXmTJGkBSQ6rqt3t4EuB2TtRXgG8L8lbgMcDRwHXjCFESeqsxa5XW+paVc3P5E2SJCDJ+4HNwKFJdgJvBDYnOZamSeQO4OcAquqmJJcCnwf2Aa+uqvvHELYkaYqYvEmSBFTVqfMUv3uR6c8Fzh1eRJIkPZj9vEmSJEnSBDB5kyRJkqQJYPImSZIkSRPA5E2SJEmSJsCSyVuSC5LsSXJjT9khSa5K8qX2+TE9485Jsj3JLUmeP6zAJUmSJGmaLOfM24XACXPKtgJXV9VRwNXtMEmOBk4BntLO8/Yk+w0sWkmSJEmaUksmb1X1CeDOOcUnAxe1ry8CXtJTfklVfauqvgJsB44bTKiSJEmSNL1W28/b+qraDVBVu5M8ri0/HPhkz3Q727KHSHIWcBbA+vXrmZmZWWUog7N3794l49hyzL6+32e567qceEalS7FA9+KRJEmShm3QnXRnnrKab8KqOh84H2DTpk21efPmAYeycjMzMywVxxlbr+z7fXactvh7rCSeUelSLNC9eCRJkqRhW+3dJm9PchhA+7ynLd8JbOiZ7ghg1+rDkyRJkiTB6pO3K4DT29enA5f3lJ+S5BFJjgSOAq7pL0RJkiRJ0pLNJpO8H9gMHJpkJ/BGYBtwaZIzga8BLwOoqpuSXAp8HtgHvLqq7h9S7JIkSZI0NZZM3qrq1AVGPWeB6c8Fzu0nKEmSJCnJwcC7gKfS3EfhVcAtwAeAjcAO4OVVddd4IpRGa7XNJiUNUZINST6e5OYkNyV5TVt+SJKrknypfX5MzzznJNme5JYkzx9f9JIkDczvAh+pqn8LPA24mQX6G5amgcmb1E37gC1V9QPAM4FXJzmaBQ5Y7bhTgKcAJwBvT7LfWCKXJGkAkjwK+HHg3QBV9e2q+iYL9zcsrXmD7iqgszYu4xb/W47ZN5CuAKR+tf0ozvaleG+Sm2n6TDyZ5hpUaA5YM8Dr2vJLqupbwFeSbAeOA/52tJFLkjQwTwL+AfjjJE8DrgNew8L9DT9IF/sUXspa68d2sf6R1x8wmP6Tl2Pc23SQn+vUJG/SpEqyEfgh4FMsfMA6HPhkz2w727K5y1r2gWwQX6rD+rLs8sHN2Fauq3FJGruHAU8HfrGqPpXkd1lBE8ku9im8lLXWj+1iJ0W2HLOP824YTSqy3D6Wh2WQn6vJm9RhSQ4E/gx4bVXdk2TBSecpq4cUrOBA9raLL+/7S3VYX5ZdPrgZ28p1NS5JY7cT2FlVn2qH/5Qmebs9yWHtn5i9/Q1La57XvEkdlWR/msTt4qr6YFt8e3ugYs4BayewoWf2I4Bdo4pVkqRBq6qvA7cmeXJb9Bya7qgW6m9YWvNM3qQOSnOK7d3AzVX1lp5RCx2wrgBOSfKIJEcCRwHXjCpeSZKG5BeBi5N8DjgW+H9o+ht+bpIvAc9th6WpYLNJqZuOB14B3JDk+rbs9TQHqEuTnAl8DXgZQFXdlORSmn8k9wGvrqr7Rx61JEkDVFXXA5vmGTVvf8PSWmfyJnVQVf0V81/HBgscsKrqXODcoQUlSZKksbLZpCRJkiRNAJM3SZIkSZoAJm+SJEmSNAFM3iRJkiRpApi8SZIEJLkgyZ4kN/aUHZLkqiRfap8f0zPunCTbk9yS5PnjiVqSNE1WnbwleXKS63se9yR5bZI3Jbmtp/zEQQYsSdKQXAicMKdsK3B1VR0FXN0Ok+Ro4BTgKe08b0+y3+hClSRNo1Unb1V1S1UdW1XHAs8A/hG4rB391tlxVfXhAcQpSdJQVdUngDvnFJ8MXNS+vgh4SU/5JVX1rar6CrAdOG4UcUqSpteg+nl7DvD3VfXVZKGuqSRJmjjrq2o3QFXtTvK4tvxw4JM90+1syx4iyVnAWQDr169nZmZm4Tc7ALYcs6+vgBdbfj/27t07tGUvZjnbY6ntNo64Z41ruy2lq3FJWtygkrdTgPf3DJ+d5JXAtcCWqrpr7gwrOZgNwiC+/AdluevapS/WLsUC3YtH0tSZ75/Kmm/CqjofOB9g06ZNtXnz5gUX+raLL+e8G/o7NO84beHl92NmZobFYh+WM7ZeueQ0W47Zt+h2G9Y2WY5xbbeldDUuSYvrO3lL8nDgxcA5bdE7gDfTHMTeDJwHvGrufCs5mA3CIL78B2W5B5EufbF2KRboXjyS1qzbkxzWnnU7DNjTlu8ENvRMdwSwa+TRSZKmyiAylRcAn6mq2wFmnwGSvBP40ADeQ5KkcbgCOB3Y1j5f3lP+viRvAR4PHAVcM5YIJWnANi7jpIfGYxDJ26n0NJmc/YeyHXwpcOO8c0mS1CFJ3g9sBg5NshN4I03SdmmSM4GvAS8DqKqbklwKfB7YB7y6qu4fS+CSpKnRV/KW5JHAc4Gf6yn+rSTH0jSb3DFnnCRJnVRVpy4w6jkLTH8ucO7wIpIk6cH6St6q6h+B75lT9oq+IpIkSZIkPcSq+3mTJEmSJI2OyZskSZIkTQCTN0mSJEmaACZvkiRJkjQBTN4kSZLUSUn2S/J3ST7UDh+S5KokX2qfHzPuGKVRGkQ/b5IkSdIwvAa4GXhUO7wVuLqqtiXZ2g6/blzBaTIMotPxHdtOGkAk/fPMmyRJkjonyRHAScC7eopPBi5qX18EvGTEYUlj5Zk3SZIkddHvAL8CHNRTtr6qdgNU1e4kj1to5iRnAWcBrF+/npmZmeFFOiB79+7tRJxbjtk39PdYf8Bo3mdQ+vlcBvm5mryNwXJP3W45Zh9nLDBtV07dSpIkDVqSFwJ7quq6JJtXs4yqOh84H2DTpk21efOqFjNSMzMzdCHOhX5/DtKWY/Zx3g2Tk4rsOG3zqucd5Oc6OVtMkiRJ0+J44MVJTgS+G3hUkvcCtyc5rD3rdhiwZ6xRSiPmNW+SJEnqlKo6p6qOqKqNwCnAx6rqZ4ArgNPbyU4HLh9TiNJYmLxJkiRpUmwDnpvkS8Bz22FpathsUpIkSZ1VVTPATPv6DuA544xHGifPvEmSJEnSBOjrzFuSHcC9wP3AvqralOQQ4APARmAH8PKququ/MCVJkiRpug3izNuzq+rYqtrUDm+l6fn+KODqdliSJEmS1IdhNJs8mabHe9rnlwzhPSRJkiRpqvR7w5ICPpqkgD9qO0NcVs/3o+71fjk9uHetp/fF4hn29pprkD3DD0LX4pEkSZKGrd/k7fiq2tUmaFcl+cJyZxx1r/fL6Sm+az29LxZPP728r8Yge4YfhK7FM2hJLgBeCOypqqe2ZQteT5rkHOBMmutPf6mq/mIMYUuSpDHbuIzfvJpcfTWbrKpd7fMe4DLgONqe7wHa5z39BilNoQuBE+aUzXs9aZKjaTowfUo7z9uT7De6UCVJkjQKq07ekqxLctDsa+B5wI3Y873Ut6r6BHDnnOKFric9Gbikqr5VVV8BttP8kSJJkqQ1pJ82guuBy5LMLud9VfWRJJ8GLk1yJvA14GX9hymJha8nPRz4ZM90O9uyh1jJtaaDuAZ0WNcldvmaR2Nbua7GJUlS16w6eauqLwNPm6f8Duz5XhqlzFNW8024kmtN33bx5X1fAzqsazO7fM2jsa1cV+OSJKlrhtFVgKThWOh60p3Ahp7pjgB2jTg2SZIkDZnJmzQ5Frqe9ArglCSPSHIkcBRwzRjikyRJ0hB15774kv5VkvcDm4FDk+wE3ghsY57rSavqpiSXAp8H9gGvrqr7xxK4tIYl2QHcS9Mlx76q2rRYFx6SJA2ayZvUQVV16gKj5r2etKrOBc4dXkSSWs+uqm/0DM924bEtydZ2+HXjCU2StNbZbFKSpNVbqAsPSZIGzjNvkiQtTwEfTVLAH7V3b12oC49/ZRcd/VnO9lhqu42zK4qudoXR1bgkLc7kTZKk5Tm+qna1CdpVSb6wnJnsoqM/Z2y9cslpthyzb9HtNqxtshxd7Qqjq3FJWpzNJiVJWoaq2tU+7wEuA45j4S48JEkaOJM3SZKWkGRdkoNmXwPPA25k4S48JEkaOJtNSpK0tPXAZUmgOXa+r6o+kuTTzNOFh6T+JdkAvAf4XuA7wPlV9bt20aFpZvImSdISqurLwNPmKb+DBbrwkNS3fcCWqvpMe+b7uiRXAWdgFx2aUjablCRJUudU1e6q+kz7+l7gZuBw7KJDU8wzb5IkSeq0JBuBHwI+xTK66GjnWXY3HV0xiC4c+u1uZFQG0TXKKPXzuQyyaw6TN0mSJHVWkgOBPwNeW1X3tNeeLmkl3XR0xdsuvpzz/uq+PpcyGT/vl+rio2v66XJkkF1zrLrZZJINST6e5OYkNyV5TVv+piS3Jbm+fZw4kEglSZI0VZLsT5O4XVxVH2yL7aJDU6ufa95mLyL9AeCZwKuTHN2Oe2tVHds+Ptx3lJIkSZoqaU6xvRu4uare0jPKLjo0tVZ9rrJtazzb3vjeJLMXkUqSJEn9Oh54BXBDkuvbstcD27CLDk2pgTQ0nXMR6fHA2UleCVxLc3buIX1vjPoi0uVcENm1CycXi2fUF90O8kLLQehaPJIkabCq6q+AhS5ws4sOTaW+k7d5LiJ9B/BmoNrn84BXzZ1v1BeRnrH1yiWn6dqFk4vF089Fk6sxyAstB6Fr8UiSJEnD1lc/b/NdRFpVt1fV/VX1HeCdwHH9hylJkiRJ062fu03OexHp7N1/Wi8Fblx9eJIkSZIk6K/Z5EIXkZ6a5FiaZpM7gJ/r4z0kSR22cRlN0pdy4QnrBhCJJElrXz93m1zoIlK7BpAkSZKkAevrmjdJkiRJ0miYvEmSJEnSBOjOffG1IoO4zmTHtpMGEIkkSZKkUfDMmyRJkiRNgIk48zaIs0ySJEmSNMk88yZJkiRJE8DkTZIkSZImwEQ0m5QkSZK6bBCX+Ww5ZgCBaE0zeZMkSZKkRfSTnG85Zh9ntPP3e7d3m01KkiRJ0gQweZMkSZKkCWDyJkmSJEkTwORNkiRJkibA0G5YkuQE4HeB/YB3VdW2Yb2XVmclF172Xmg5V78XXmpwrHfS6FnvpNGz3mlaDSV5S7If8AfAc4GdwKeTXFFVnx/G+0my3knjYL3TOAzilvQXnrBuAJGMh/VO02xYZ96OA7ZX1ZcBklwCnAxYqTQ0gziYTfhZROudNHrWO2n0BlrvBvH7QRqVVNXgF5r8NHBCVf1sO/wK4Eeq6uyeac4CzmoHnwzcMvBAVu5Q4BvjDqJHl+LpUiwwvnieWFWPHcP7LmkI9a5rn3kvY1udrsa2VFzWu24wttXpamxrut615V38rbmUru4vw+C6PtSS9W5YZ94yT9mDssSqOh84f0jvvypJrq2qTeOOY1aX4ulSLNC9eDpioPWuy9vY2Fanq7F1Na5lst51gLGtXFfjWqYl6x1087fmUib8c1kR13V1hnW3yZ3Ahp7hI4BdQ3ovSQ3rnTR61jtp9Kx3mlrDSt4+DRyV5MgkDwdOAa4Y0ntJaljvpNGz3kmjZ73T1BpKs8mq2pfkbOAvaG7hekFV3TSM9xqwrp1a71I8XYoFuhfP2A2h3nV5Gxvb6nQ1tq7GtSTrXWcY28p1Na4lTfDvzOWY2M9lFVzXVRjKDUskSZIkSYM1rGaTkiRJkqQBMnmTJEmSpAkwVclbkg1JPp7k5iQ3JXlNW35IkquSfKl9fkzPPOck2Z7kliTPH2As353kmiSfbWP5tXHF0rP8/ZL8XZIPdSCWHUluSHJ9kmvHHc9ak+SEdlttT7J1nvFJ8nvt+M8lefpy5x1yXKe18Xwuyd8keVrPuIfsMyOObXOSu9v3vz7JG5Y77whi++WeuG5Mcn+SQ9pxQ9tuSS5IsifJjQuMH8t+1gX91MEOxLbgvj7kuFa9P3UgtnFts3l/98yZZmzbbdrMt59kjf62WWBd35Tktp56cGLPuEle19HmF1U1NQ/gMODp7euDgC8CRwO/BWxty7cCv9m+Phr4LPAI4Ejg74H9BhRLgAPb1/sDnwKeOY5YemL6b8D7gA+1w+OMZQdw6JyyscWzlh40F3f/PfAk4OHttjt6zjQnAn/e7qfPBD613HmHHNePAo9pX79gNq6F9pkRb7PNs3VnpfMOO7Y5078I+NiIttuPA08Hblxg/Mj3sy48+qmDHYlt3n19BLGtan/qSGzj2mbz/u7pynabtsd8+wlr9LfNAuv6JuC/zzPtpK/rSPOLqTrzVlW7q+oz7et7gZuBw4GTgYvayS4CXtK+Phm4pKq+VVVfAbYDxw0olqqqve3g/u2jxhELQJIjgJOAd/UUjyWWRXQtnkl1HLC9qr5cVd8GLqHZhr1OBt7T7qefBA5Octgy5x1aXFX1N1V1Vzv4SZq+fUahn/Ue5jZbzfJPBd4/wPdfUFV9ArhzkUnGsZ91QT91sAuxjUUf+1MXYhuLRX739Brbdps2C+wna/K3zQrrxKSv60jzi6lK3nol2Qj8EM0Zr/VVtRuaDwB4XDvZ4cCtPbPt5KFfev3EsF+S64E9wFVVNbZYgN8BfgX4Tk/ZuGKBJpH9aJLrkpzVgXjWkuVsr4WmGea2Xumyz6T5t3jWfPvMoCw3tmelaQr950messJ5hx0bSR4JnAD8WU/xMLfbUsaxn3VBP3Vw2PrZ18et6/vNWLfZnN89vbq+3da6afttc3bbPPeCnmaEa2ZdR5FfDKWft65LciDNj5fXVtU9SRacdJ6ygfWtUFX3A8cmORi4LMlTF5l8aLEkeSGwp6quS7J5ObMMK5Yex1fVriSPA65K8oUxx7OWLGd7LTTNMLf1sped5Nk0yduP9RQ/ZJ9p//kbVWyfAZ5YVXvbdvz/CzhqmfMOO7ZZLwL+uqp6/w0d5nZbyjj2sy7opw4OWz/7+rh1eb8Z6zab+7tn7uh5ZunKdptma/FzeQfwZpr1eDNwHvAq1si6jiq/mLozb0n2p9mwF1fVB9vi22ebCLTPe9ryncCGntmPAHYNOqaq+iYwQ/OP+DhiOR54cZIdNE1kfjLJe8cUCwBVtat93gNcRnM6eayf0xqynO210DTD3NbLWnaSH6Rp3ntyVd0xW77APjMoS8ZWVffMNoWuqg8D+yc5dDnzDju2Hqcwp8nkkLfbUsaxn3VBP3Vw2PrZ18ets/vNOLfZAr97enV2u02JqfltU1W3V9X9VfUd4J08cLyZ+HUdZX4xVclbmhT43cDNVfWWnlFXAKe3r08HLu8pPyXJI5IcSfMv2TUDiuWx7Rk3khwA/BTwhXHEUlXnVNURVbWR5sfdx6rqZ8YRC0CSdUkOmn0NPA+4cVzxrEGfBo5KcmSSh9N85lfMmeYK4JVpPBO4uz3lv5x5hxZXkicAHwReUVVf7ClfaJ8ZlOXE9r3tdwxJjqP5fr1jOfMOO7Y2pkcDP8ED9WYU220p49jPuqCfOjj22BbZ18dtXNtsSePaZov87unV2e02Jabmt00efC3lS3ngeDPR6zry/KI6cJeWUT1omlgV8Dng+vZxIvA9wNXAl9rnQ3rm+VWau8DcArxggLH8IPB3bSw3Am9oy0cey5y4NvPA3SbHEgvNXc4+2z5uAn61C9tmLT3a/f6L7Tab3b4/D/x8+zrAH7TjbwA2LTbvCON6F3BXT/29drF9ZsSxnd2+92dpbqbyo6PYZsuJrR0+g+YC6d75hrrdaM7y7Qb+heafxjO7sJ914dFPHexAbAvu60OOa9X7UwdiG9c2W+h3Tye227Q9FthP1uRvmwXW9U/afexzNAnMYWtkXUeaX6RdgCRJkiSpw6aq2aQkSZIkTSqTN0mSJEmaACZvktQjyYVJ/uci4yvJ9w05ho3t+0xldy7SciV5U3t3ZEmaCiZvkiRJkjQBTN4kSZIkaQKYvE2IJI9P8mdJ/iHJV5L8UpJDkuxM8qJ2mgOTbE/yynb4wiR/kOTKJPcm+VSSf9OzzOcluSXJ3UnenuQvk/zsuNZRGqUkP5BkJsk3k9yU5MULTPfLSXYn2ZXkVXPGXZjkD5Nc1daxv0zyxJ7x/7Ydd2db117eM+6kJH+X5J4ktyZ50yKx/vskO5I8dQCrLk2kJK9Lcltb125J8pw542ebG5/V1tfdSbaMK15pLUmyIckH29+hdyT5/ST/JsnH2uFvJLk4bR/GGh6TtwmQ5LuA/03TP8zhwHOA1wI/DLwKeGeSxwFvBa6vqvf0zH4q8GvAY4DtwLntMg8F/hQ4h6YfiluAHx3B6khjl2R/mjr1UeBxwC8CFyd58pzpTgD+O/Bcmk40f2qexZ0GvBk4lKZvl4vbedcBVwHva9/jVODtSZ7Szncf8ErgYOAk4BeSvGSeWP8T8JvAT1XVKDvQljqjrZtnAz9cVQcBzwd2LDD5s2nq6/OArUnmq7eSlinJfsCHgK8CG2l+i15C00fgbwCPB34A2AC8aSxBThGTt8nww8Bjq+rXq+rbVfVl4J3AKVX1UeD/pen87yTg5+bM+8Gquqaq9tH8qDy2LT8RuKmqPtiO+z3g6yNYF6kLngkcCGxr69THaA5Mp86Z7uXAH1fVjVV1H/MflK6sqk9U1bdoOt18VpINwAuBHVX1x1W1r6o+A/wZ8NMAVTVTVTdU1Xeq6nM0HZr+xJxlvxb4ZWBzVW0fwHpLk+p+4BHA0Un2r6odVfX3C0z7a1V1X1XdAPwxD63XklbmOJoE7ZfbuvXPVfVXVbW9qq6qqm9V1T8Ab+GhxzENmMnbZHgi8Pi2edc3k3wTeD2wvh1/PvBUmh+Zd8yZtzch+0eaH6zQVMJbZ0dU01v7ziHELnXR44Fbq+o7PWVfpfk38SHTzZlmrt56tBe4s53vicCPzKm3pwHfC5DkR5J8vG2Ccjfw8zRn73r9MvAHVWXd1FRr/7x4Lc0fKHuSXJLk8QtMPrfOLjSdpOXZAHy1/bP/XyV5XFsXb0tyD/BeHnoc04CZvE2GW4GvVNXBPY+DqurE9lT2HwHvoWl2tdxbmO8GjpgdSJLeYWmN2wVsaJskz3oCcNuc6XbTHLR6p5nrX8cnORA4pF3+rcBfzqm3B1bVL7STvw+4AthQVY8G/pCmCUqv5wH/I8m/X9nqSWtPVb2vqn6M5o+RomlOPJ+5dXbXsGOT1rhbgSfM033Nb9DUxR+sqkcBP8NDj2MaMJO3yXANcE97sfYBSfZL8tQkP0xzBg6aa99+G3hPm9At5UrgmCQvaSvjq2nPCEhT4FM015z9SpL9k2wGXkTThr/XpcAZSY5O8kjgjfMs68QkP5bk4TTXvn2qqm6laYb5/Ule0b7H/kl+OMkPtPMdBNxZVf+c5DjgP86z7JuAE4A/WOiGKtI0SPLkJD+Z5BHAPwP/RNOUcj7/d5JHtteX/ifgA6OKU1qjrqH5M3NbknVJvjvJ8TTHsb3AN5McTtNaRENm8jYBqup+mh+WxwJfAb4BvAv4SeC/Aa9sp/lNmn9Ati5jmd8AXgb8FnAHcDRwLfCtwa+B1C1V9W3gxcALaOrT22nq0RfmTPfnwO8AH6O54c/H5lnc+2iSujuBZ9A0jaSq7qU5c3YKzT//X6epo49o5/svwK8nuRd4A02iOF+sn6W5fu6dSV6wqhWWJt8jgG009fXrNDcBev0C0/4lTX29Gvjt9tpwSavU8zv0+4Cv0Vxm8x9oboj3dOBumpMCHxxXjNMkzaVOmnZt87GdwGlV9fFxxyNNgiQXAjur6n+MOxZp2iXZSPMH5/5zr82RpLXCM29TLMnzkxzcNkN5PU075U+OOSxJkiRJ8zB5m27PAv6ephnKi4CXVNU/jTckSZIkSfOx2aQkSZIkTQDPvEmSJEnSBJjbX8NYHHroobVx48YFx993332sW7dudAGtgLGtTldjWyqu66677htV9dgRhjQ0k1zvVsL16JbVrIf1rhuMbXW6Gts0He9g8brX1c9otVyfbltsfZZV76pq7I9nPOMZtZiPf/zji44fJ2Nbna7GtlRcwLU1gjoBXADsAW7sKXsTTSfS17ePE3vGnUNza+xbgOcv5z0mud6thOvRLatZj1HVu1E8JrneGdvqdDW2rhzvRvVYrO519TNaLden2xZbn+XUO5tNSt10IU3nzHO9taqObR8fBkhyNE1fYk9p53n7MjtqlyRJ0gQxeZM6qKo+QdPp83KcDFxSVd+qqq/QnIE7bmjBSZIkaSyWvOYtyQbgPcD3At8Bzq+q301yCPABYCOwA3h5Vd3VznMOcCZwP/BLVfUXQ4lemj5nJ3klcC2wpa1zh/Pg/vl2tmUPkeQs4CyA9evXMzMzs+Ab7d27d9Hxk8L16Ja1sh6SJI3Dcm5Yso/mR+JnkhwEXJfkKuAM4Oqq2pZkK7AVeN2cJlyPB/5Pku+vqvtXG+QNt93NGVuvXO3sAOzYdlJf80sd8A7gzUC1z+cBr6LpXH2uefsAqarzgfMBNm3aVJs3b17wzd528eWc91f39RVwF+rdzMwMi63npHA9JM3a2OdvIoALT1g7N4Do1yB+Z0I3jnla+5ZM3qpqN7C7fX1vkptp/tU/GdjcTnYRMAO8jp4mXMBXksw24frbQQcvTZOqun32dZJ3Ah9qB3cCG3omPQLYNcLQJGlolpOobDlm36I/vv1RLWmtWFFXAUk2Aj8EfApY3yZ2VNXuJI9rJ1tWE66VNN9af0DzxdyPYTXT6XITIGNbua7GBZDksNk6B7wUuLF9fQXwviRvoTnbfRRwzRhClCRJ0hAtO3lLciDwZ8Brq+qeZL6WWs2k85Q9pAnXiptv3dBfl3Q7Tlt4+f3ochMgY1u5rsSV5P00Z7YPTbITeCOwOcmxNPVpB/BzAFV1U5JLgc/TNHN+dT/NlCVJktRNy8qIkuxPk7hdXFUfbItvnz0TkOQwmj6pwCZcUt+q6tR5it+9yPTnAucOLyJJkiSN25JdBaQ5xfZu4OaqekvPqCuA09vXpwOX95SfkuQRSY7EJlySJEmS1LflnHk7HngFcEOS69uy1wPbgEuTnAl8DXgZ2IRLkiRJkoZhOXeb/Cvmv44N4DkLzGMTLkmSJEkaoCWbTUqSJEmSxs/kTZIkSZImgMmbJEmSJE0AkzdJkiRJmgAmb5IkSZI0AUzeJEmSJGkCmLxJkiRJ0gQweZMkSdLYJLkgyZ4kN/aUvSnJbUmubx8n9ow7J8n2JLckef54opbGw+RNkiRJ43QhcMI85W+tqmPbx4cBkhwNnAI8pZ3n7Un2G1mk0piZvEmSJGlsquoTwJ3LnPxk4JKq+lZVfQXYDhw3tOCkjnnYuAOQJEmS5nF2klcC1wJbquou4HDgkz3T7GzLHiLJWcBZAOvXr2dmZmbeN1l/AGw5Zl/fwS60/FHbu3dvZ2IZBNfnwUzeJEmS1DXvAN4MVPt8HvAqIPNMW/MtoKrOB84H2LRpU23evHneN3rbxZdz3g39/yTecdr8yx+1mZkZFlrXSeT6PJjNJiVJktQpVXV7Vd1fVd8B3skDTSN3Aht6Jj0C2DXq+KRxMXmTJElSpyQ5rGfwpcDsnSivAE5J8ogkRwJHAdeMOj5pXGw2KUmSpLFJ8n5gM3Bokp3AG4HNSY6laRK5A/g5gKq6KcmlwOeBfcCrq+r+MYQtjYXJmyRJksamqk6dp/jdi0x/LnDu8CKSustmk5IkSZI0AZZM3uz1XpIkSZLGbzln3i7EXu8lSZIkaayWTN7s9V6SJEmSxq+fG5aMpNd7YCA93w+rZ/Yu9/pubCvX1bgkDV+SC4AXAnuq6qlt2SHAB4CNNHe8e3l7vCPJOcCZwP3AL1XVX4whbEnSFFlt8jayXu9hMD3fD6vX+y73+m5sK9fVuCSNxIXA7wPv6SnbClxdVduSbG2HXzfnMoHHA/8nyfd7y3JJ0jCt6m6T9novSVprFrhM4GTgovb1RcBLesq9TECSNFKrOp2V5LCq2t0Ozu31/n1J3kLzT6S93kuSJtn62eNdVe1O8ri2fCiXCXS56fa4YlvOZRNLXV4xzm06jO3W76Uk0O19TdLClkze7PVeGj2vvZE6byiXCXS56fa4Yjtj65VLTrPlmH2LXl4xrEsnlmMY220522QpF56wrrP7mqSFLZm82eu9NBYX4rU3UhfcPtvaJMlhwJ623MsEJEkjt6pr3iQNl9feSJ1xBXB6+/p04PKe8lOSPCLJkXiZgCRpBPq7haOkURrptTdd7qJjJdbKdR2ux/AtcJnANuDSJGcCXwNeBl4mIEkaD5M3afJNXRcdK9Hla4hWwvUYvgUuEwB4zgLTe5mAJGmkbDYpTY7b22tu8NobSZKk6WPyJk0Or72RJEmaYjablDrIa28kSZI0l8mb1EFeeyNJkqS5bDYpSZIkSRPA5E2SJEmSJoDJmyRJkiRNAJM3SZIkjU2SC5LsSXJjT9khSa5K8qX2+TE9485Jsj3JLUmeP56opfEweZMkSdI4XQicMKdsK3B1VR0FXN0Ok+Ro4BTgKe08b0+y3+hClcbL5E2SJEljU1WfAO6cU3wycFH7+iLgJT3ll1TVt6rqK8B24LhRxCl1gV0FSJIkqWvWV9VugKraneRxbfnhwCd7ptvZlj1EkrOAswDWr1/PzMzM/G90AGw5Zl/fAS+0/FHbu3dvZ2IZBNfnwUzeJEmSNCkyT1nNN2FVnQ+cD7Bp06bavHnzvAt828WXc94N/f8k3nHa/MsftZmZGRZa10nk+jyYzSYlSZLUNbcnOQygfd7Tlu8ENvRMdwSwa8SxSWNj8iZJkqSuuQI4vX19OnB5T/kpSR6R5EjgKOCaMcQnjcWSyZu3b5UkSdKwJHk/8LfAk5PsTHImsA14bpIvAc9th6mqm4BLgc8DHwFeXVX3jydyafSW08D3QuD3gff0lM3evnVbkq3t8Ovm3L718cD/SfL9VipJkiTNp6pOXWDUcxaY/lzg3OFFJHXXkmfevH2rJEmSJI3fam+tM7Lbt8JgbuE6rFuMdvn2pdMW2w233d33Mo589H6d3WaSJEmaboPuKmDgt2+FwdzCdVi3b+3y7UunLbYztl7Z9zIuPGFdZ7eZJEmSpttq7zbp7VslSZIkaYRWm7x5+1ZJkiRJGqEl2yK2t2/dDByaZCfwRprbtV7a3sr1a8DLoLl9a5LZ27fuw9u3SpIkSdJALJm8eftWSZIkSRq/1TablCRJkiSNkMmbJEmSJE0AkzdJkiRJmgAmb5IkSZI0AUzeJEmSJGkCLHm3SUmSBEl2APcC9wP7qmpTkkOADwAbgR3Ay6vqrnHFKEla2zzzJknS8j27qo6tqk3t8Fbg6qo6Cri6HZYkaSg88yZNGP/9lzrlZGBz+/oiYAZ43biCkaQbbrubM7Ze2dcydmw7aUDRaNBM3qTJ9Oyq+kbP8Oy//9uSbG2H18wPyI19HIS2HLOPM7Ze6YFIg1DAR5MU8EdVdT6wvqp2A1TV7iSPmztTkrOAswDWr1/PzMzMgm+wd+/eRceP07hi23LMviWnWX/A4tONc5sOY7stZ5sspcv7mqSFmbxJa0Mn//3vJ+mSOuj4qtrVJmhXJfnCcmZqk7zzATZt2lSbN29ecNqZmRkWGz9O44ptOWcQthyzj/NuWPgnzY7TNg8wopUZxnbr96wKwIUnrOvsviZpYSZv0uRZ1b//sLIzAEv9kz0pZtdj0v9hXiv/kk/yelTVrvZ5T5LLgOOA25Mc1ta7w4A9Yw1SkrSmmbxJk2dV//7Dys4AvO3iyxf9J3tSzP4jP85/3gehy2dkVmJS1yPJOuC7qure9vXzgF8HrgBOB7a1z5ePL0pJ0lo3+b/MpCnjv//SWKwHLksCzbHzfVX1kSSfBi5NcibwNeBlY4xRkrTGmbxJE8R//6XxqKovA0+bp/wO4Dmjj0iaDt5hWXow+3mTJst64K+SfBa4Briyqj5Ck7Q9N8mXgOe2w5IkrQX2ryi1PPMmTRD//ZckqZt3WJZGweRNkiRJXTX0OywP6u7KXbmT7iDWpyvrApN9l+L59Ls+fSVvtkOWJEnSEA39DsuDurtyV+5qPIj16cq6wOTepXgh/a7PIK55sx2yJEmSBq73DsvAg+6wDOAdljVthnHDkpNp2h/TPr9kCO8hSZKkNSzJuiQHzb6mucPyjTxwh2XwDsuaMv2eIx56O2TodtvdLrfDnbbYBtFevcvbTJKkKWP/itIc/SZvQ2+HDN1uu9vldrjTFtsZW6/sexkXnrCus9tMkqRp4h2WpYfqq9mk7ZAlSZIkaTRWfTqrbXv8XVV1b0875F/ngXbI27AdsiRJkqRVuuG2u/tuXbVj20kDimb8+mmLaDtkSZIkSRqRVSdvtkOWJEmSpNEZRlcBkiRJkqQBM3mTJEmSpAlg8iZJkiRJE8DkTZIkSZImQL+ddEuSpAHxltiSpMV45k2SJEmSJoDJmyRJkiRNAJtNSpoKG/tsijYoNmmTJEmrZfImSZIkSYsY1J/AF56wrq/5bTYpSZIkSRPA5E2SJEmSJoDJmyRJkiRNAJM3SZIkSZoA3rBEkkZotRc8bzlm34M6b/aulZIkTR/PvEmSJEnSBDB5kyRJkqQJYPImSZIkSRNgaNe8JTkB+F1gP+BdVbVtWO8lqWG9mx6D6CzU6+YGw3onjZ71TtNqKMlbkv2APwCeC+wEPp3kiqr6/DDeT5L1TuOx0iRy7o1XYLKTSOudNHrWO02zYTWbPA7YXlVfrqpvA5cAJw/pvSQ1rHfS6FnvpNGz3mlqpaoGv9Dkp4ETqupn2+FXAD9SVWf3THMWcFY7+GTglkUWeSjwjYEHOhjGtjpdjW2puJ5YVY8dVTArMWX1biVcj25ZzXpY77rB2Fanq7Gt6eNdW77cutfVz2i1XJ9uW2x9lqx3w7rmLfOUPShLrKrzgfOXtbDk2qraNIjABs3YVqersXU1rmWamnq3Eq5Ht6yV9egxNfXO2Fanq7F1Na5lWrLewfLr3oRvi4dwfbqt3/UZVrPJncCGnuEjgF1Dei9JDeudNHrWO2n0rHeaWsNK3j4NHJXkyCQPB04BrhjSe0lqWO+k0bPeSaNnvdPUGkqzyaral+Rs4C9obuF6QVXd1Mcil9XcZEyMbXW6GltX41rSlNW7lXA9umWtrAcwdfXO2Fanq7F1Na4lTVm9Ww3Xp9v6Wp+h3LBEkiRJkjRYw2o2KUmSJEkaIJM3SZIkSZoAY0/ekpyQ5JYk25NsnWd8kvxeO/5zSZ6+3HmHHNdpbTyfS/I3SZ7WM25HkhuSXJ/k2kHGtczYNie5u33/65O8YbnzjiC2X+6J68Yk9yc5pB03tO2W5IIke5LcuMD4sexnXbUW1jnJhiQfT3JzkpuSvGbcMfUjyX5J/i7Jh8Ydy2olOTjJnyb5Qvu5PGvcMY1LP8e+DsS24DFmyHGt+nu8A7GNa5st+T04zu3WBWvheDdrqf1wkqzBY/h3J7kmyWfb9fm1VS+sqsb2oLnI9O+BJwEPBz4LHD1nmhOBP6fp0+OZwKeWO++Q4/pR4DHt6xfMxtUO7wAOHeM22wx8aDXzDju2OdO/CPjYiLbbjwNPB25cYPzI97OuPtbKOgOHAU9vXx8EfHES16Nnff4b8L756vakPICLgJ9tXz8cOHjcMY1pO6z62NeR2OY9xowgtlV9j3cktnFtsyW/B8e53cb9WCvHu571WXQ/nKTHGjyGBziwfb0/8CngmatZ1rjPvB0HbK+qL1fVt4FLgJPnTHMy8J5qfBI4OMlhy5x3aHFV1d9U1V3t4Cdp+hgZhX7We5jbbDXLPxV4/wDff0FV9QngzkUmGcd+1lVrYp2randVfaZ9fS9wM3D4eKNanSRHACcB7xp3LKuV5FE0PyzeDVBV366qb441qPHp59jXhdjGoo/v8S7ENhbL/B4c23brgM7u76vR1f1wNdbSMRygrV9728H928eq7ho57uTtcODWnuGdPPSDWWia5cw7zLh6nUnzr9WsAj6a5LokZw0oppXG9qz21OyfJ3nKCucddmwkeSRwAvBnPcXD3G5LGcd+1lVrbp2TbAR+iOafrkn0O8CvAN8Zcxz9eBLwD8Aft80/35Vk3biDGpN+jn3D1s8xZty6/t011m22yPdg17fbME3zuk+MNXAMB/718ofrgT3AVVW1qvUZd/KWecrmZqELTbOceVdr2ctO8mya5O11PcXHV9XTaZpTvjrJjw8oruXG9hngiVX1NOBtwP9awbz9WMnyXwT8dVX1/kM0zO22lHHsZ121ptY5yYE0fxK8tqruGXc8K5XkhcCeqrpu3LH06WE0zXneUVU/BNwHTPT1JX3o59g3bP0cY8aty99dY91mS3wPdnm7Dds0r/tEmPRjeK+qur+qjqVprXdckqeuZjnjTt52Aht6ho8Adi1zmuXMO8y4SPKDNM2YTq6qO2bLq2pX+7wHuIzmtPygLBlbVd0ze2q2qj4M7J/k0OXMO+zYepzCnCaTQ95uSxnHftZVa2adk+xP86V/cVV9cNzxrNLxwIuT7KBp0vOTSd473pBWZSews+efxj+lSeamUT/HvmHr5xgzbp397hrnNlvG92Bnt9sITPO6d94aOYY/RHvJwAxNC7QVG3fy9mngqCRHJnk4zQ/6K+ZMcwXwyvZuSM8E7q6q3cucd2hxJXkC8EHgFVX1xZ7ydUkOmn0NPA8Y5F1/lhPb9yZJ+/o4ms/5juXMO+zY2pgeDfwEcHlP2bC321LGsZ911ZpY57YOvBu4uareMu54VquqzqmqI6pqI81n8bGq+pkxh7ViVfV14NYkT26LngN8fowhjVM/x76xx7bIMWbcxrXNljSubbbM78HObrcRWBPHu7VorRzDZyV5bJKD29cHAD8FfGE1y3rYAONasaral+Rs4C9o7vhzQVXdlOTn2/F/CHyY5k5I24F/BP7TYvOOMK43AN8DvL39Pt5XVZuA9cBlbdnDgPdV1UcGEdcKYvtp4BeS7AP+CTilqgoY2jZbQWwALwU+WlX39cw+1O2W5P00d/s6NMlO4I00F4uObT/rqjW0zscDrwBuaNuYA7y+/ddb4/GLwMXtj6Qv09azadPPsa8jsS10jBmq1X6Pj8IyYhvLNmOB70HgCT2xjW27jdsaOt4B8++HVfXu8Ua1amvtGH4YcFGS/Wj+vLm0qlbV9U9G890hSZIkSerHuJtNSpIkSZKWweRtQiXZmKSSrKrpazvv97WvL0zyPwcboSRJkqRBMnmbIEl2JPmpccch6QG9f4R0cXmSJGntMHmTJEmSpAlg8jYhkvwJzd2h/neSvcDL21GnJflakm8k+dWe6Y9L8rdJvplkd5Lfb+/wJmkeSX4gyUxbZ25K8uK2fCbJz/ZMd0aSv2pff6It/mySvUn+Q5LNSXYmeX1bL3ckOa1n/hUtb9jrLUmSJofJ24SoqlcAXwNeVFUHApe2o34MeDJNn0lvSPIDbfn9wH8FDgWe1Y7/LyMNWpoQaToC/d/AR4HH8cAt7Z+82HxV9ePty6dV1YFV9YF2+Htp6t7hwOnA+Usta4nlSZIkmbytAb9WVf9UVZ8FPgs8DaCqrquqT1bVvqraAfwRTcfYkh7qmcCBwLaq+nZVfQz4EHBqH8v8v6vqW1X1l8CVPHC2XJIkaVXG2km3BuLrPa//keYHKEm+H3gLsAl4JM1nfd3Io5Mmw+OBW6vqOz1lX6U5c7Yad83phP6r7XtIkiStmmfeJstKelR/B/AF4KiqehTweiBDiUqafLuADUl6vxOfANwG3EfzB8is713G8h6TZN2cZe1qX69meZIkSSZvE+Z24EnLnPYg4B5gb5J/C/zC0KKSJt+naJKqX0myf5LNwIuAS4Drgf8rySPbW/ifOWfeherlryV5eJJ/B7wQ+H/b8tUuT5IkTTmTt8nyG8D/SPJN4KeXmPa/A/8RuBd4J+CND6QFVNW3gRcDLwC+AbwdeGVVfQF4K/BtmqTqIuDiObO/CbiovUvl7HVtXwfuojnbdjHw8+2yWOXyJEmSSNVKWuJJkhbTnrV7b1UdMeZQJEnSGuOZN0mSJEmaACZvkiRJkjQBbDYpSZIkSRPAM2+SJEmSNAE60Un3oYceWhs3blxw/H333ce6desWHD9pXJ/uWmpdrrvuum9U1WNHGNLQTHK9M7bV6Wps01TvJEnqRyeSt40bN3LttdcuOH5mZobNmzePLqAhc326a6l1SfLV0UUzXJNc74xtdboa2zTVO0mS+mGzSUmSJEmaACZvkiRJkjQBTN4kSZIkaQJ04po3TbYbbrubM7Ze2dcydmw7aUDRSBqljX3WfYALT+jeTVQkSeoiz7xJkiRJ0gQweZMkSZKkCWDyJkmSJEkTwORNkiRJkiaAyZskSZIkTQCTN0mSJEmaAEsmb0kuSLInyY09ZW9KcluS69vHiT3jzkmyPcktSZ4/rMCltcx6J0mSpLmWc+btQuCEecrfWlXHto8PAyQ5GjgFeEo7z9uT7DeoYKUpciHWO0mSJPVYMnmrqk8Ady5zeScDl1TVt6rqK8B24Lg+4pOmkvVOkiRJcz2sj3nPTvJK4FpgS1XdBRwOfLJnmp1t2UMkOQs4C2D9+vXMzMws+EZ79+5ddPykWWvrs/4A2HLMvr6W0ZXtMQGfjfUOY1utYcTWb92Hbm8zSZK6ZLXJ2zuANwPVPp8HvArIPNPWfAuoqvOB8wE2bdpUmzdvXvDNZmZmWGz8pFlr6/O2iy/nvBv6+R8Adpy2eTDB9Knjn431rmVsqzOM2M7YemXfy7jwhHWd3WaSJHXJqu42WVW3V9X9VfUd4J080ERrJ7ChZ9IjgF39hSgJrHeSJEnTblXJW5LDegZfCszeEe8K4JQkj0hyJHAUcE1/IUoC650kSdK0W7KtW5L3A5uBQ5PsBN4IbE5yLE3TrB3AzwFU1U1JLgU+D+wDXl1V9w8lcmkNs95JkiRpriWTt6o6dZ7idy8y/bnAuf0EJU07650kSZLmWlWzSUmSJEnSaPV3i0BJa9YNt93d950Ed2w7aUDRSJIkyTNvkiRJkjQBTN4kSZIkaQKYvEmSJEnSBDB5kyRJkqQJYPImSZIkSRPA5E2SJEmSJoDJmyRJkiRNAJM3SZIkSZoAJm+SJEmSNAFM3iRJkiRpApi8SZIkSdIEMHmTJEmSpAlg8iZJkiRJE+Bh4w5AGpSNW6/sexkXnrBuAJFIkiRJg+eZN0mSJEmaACZvkiRJkjQBTN4kSZIkaQKYvEmSJEnSBDB5kyRJkqQJYPImSZIkSRPA5E2SJEmSJsCSyVuSC5LsSXJjT9khSa5K8qX2+TE9485Jsj3JLUmeP6zApbXMeidJkqS5lnPm7ULghDllW4Grq+oo4Op2mCRHA6cAT2nneXuS/QYWrTQ9LsR6J0mSpB5LJm9V9QngzjnFJwMXta8vAl7SU35JVX2rqr4CbAeOG0yo0vSw3kmSJGmuh61yvvVVtRugqnYneVxbfjjwyZ7pdrZlD5HkLOAsgPXr1zMzM7Pgm+3du3fR8ZNmra3P+gNgyzH7+lrGILZHvzFA5z+bkda7rnyu8+ny5zRtsU1BvZMkqTNWm7wtJPOU1XwTVtX5wPkAmzZtqs2bNy+40JmZGRYbP2nW2vq87eLLOe+G/nalHadt7juOM7Ze2fcyLjxh3SR+NkOpd135XOfT5To0bbFNcb2TJGnkVnu3yduTHAbQPu9py3cCG3qmOwLYtfrwJPWw3kmSJE2x1SZvVwCnt69PBy7vKT8lySOSHAkcBVzTX4iSWtY7SZKkKbZkm6gk7wc2A4cm2Qm8EdgGXJrkTOBrwMsAquqmJJcCnwf2Aa+uqvuHFLu0ZlnvJEmSNNeSyVtVnbrAqOcsMP25wLn9BCVNO+udJEmS5lpts0lJkiRJ0giZvEmSJEnSBDB5kyRJkqQJYPImSZIkSRPA5E2SJEmSJoDJmyRJkiRNAJM3SZIkSZoAJm+SJEmSNAFM3iRJkiRpApi8SZIkSdIEMHmTJEmSpAlg8iZJkiRJE8DkTZIkSZImgMmbJEmSJE0AkzdJkiRJmgAmb5IkSZI0AUzeJEmSJGkCmLxJkiRJ0gQweZMkSZKkCWDyJkmSJEkTwORNkiRJkiaAyZskSZIkTQCTN0mSJEmaAA/rZ+YkO4B7gfuBfVW1KckhwAeAjcAO4OVVdVd/YUqaZb2TJEmaToM48/bsqjq2qja1w1uBq6vqKODqdljSYFnvJEmSpswwmk2eDFzUvr4IeMkQ3kPSg1nvJEmS1rhU1epnTr4C3AUU8EdVdX6Sb1bVwT3T3FVVj5ln3rOAswDWr1//jEsuuWTB99m7dy8HHnjgquPsmrW2PnvuvJvb/6m/ZRxz+KP7juOG2+7uexlHPnq/RT+bZz/72df1nO0ai1HVu658rvPpch2attimpd5JktQFfV3zBhxfVbuSPA64KskXljtjVZ0PnA+wadOm2rx584LTzszMsNj4SbPW1udtF1/OeTf0tyvtOG1z33GcsfXKvpdx4QnrJuGzGUm968rnOp8u16Fpi22K6p0kSWPXV7PJqtrVPu8BLgOOA25PchhA+7yn3yAlPcB6J0mSNJ1WnbwlWZfkoNnXwPOAG4ErgNPbyU4HLu83SEkN650kSdL06qdN1HrgsiSzy3lfVX0kyaeBS5OcCXwNeFn/YUpqWe8kSZKm1KqTt6r6MvC0ecrvAJ7TT1CS5me9kyRJml7D6CpAkiRJkjRgJm+SJEmSNAFM3iRJkiRpApi8SZIkSdIEMHmTJEmSpAlg8iZJkiRJE8DkTZIkSZImgMmbJEmSJE0AkzdJkiRJmgAmb5IkSZI0AUzeJEmSJGkCmLxJkiRJ0gQweZMkSZKkCWDyJkmSJEkT4GHjDmAa3XDb3Zyx9cq+lrFj20kDikaSJEnSJPDMmyRJkiRNAJM3SZIkSZoAJm+SJEmSNAFM3iRJkiRpApi8SZIkSdIEMHmTJEmSpAlg8iZJkiRJE8DkTZIkSZImwNCStyQnJLklyfYkW4f1PpIeYL2TJElau4aSvCXZD/gD4AXA0cCpSY4exntJaljvJEmS1raHDWm5xwHbq+rLAEkuAU4GPr+ahd1w292csfXKvgLase2kvuaXJsBA650kSZK6ZVjJ2+HArT3DO4Ef6Z0gyVnAWe3g3iS3LLK8Q4Fv9BNQfrOfuQfO9ZmjK+vz7N9ccl2eOKpYVmGa6l3fsQ2Rsa3QhNc7SZJGZljJW+YpqwcNVJ0PnL+shSXXVtWmQQTWBa5Pd034ukxNvTO21elqbF2NS5KkrhnWDUt2Aht6ho8Adg3pvSQ1rHeSJElr2LCSt08DRyU5MsnDgVOAK4b0XpIa1jtJkqQ1bCjNJqtqX5Kzgb8A9gMuqKqb+ljkspp5TRDXp7smdl2mrN4Z2+p0NbauxiVJUqekqpaeSpIkSZI0VkPrpFuSJEmSNDgmb5IkSZI0ATqfvCU5IcktSbYn2TruePqR5IIke5LcOO5Y+pVkQ5KPJ7k5yU1JXjPumPqR5LuTXJPks+36/Nq4YxqmpepVGr/Xjv9ckqcvd94hx3VaG8/nkvxNkqf1jNuR5IYk1ye5dpBxLTO2zUnubt//+iRvWO68I4jtl3viujHJ/UkOaccNbbst9Z03rv1MkqSJVVWdfdDcdOHvgScBDwc+Cxw97rj6WJ8fB54O3DjuWAawLocBT29fHwR8ccI/mwAHtq/3Bz4FPHPccQ1pXZesV8CJwJ+32+WZwKeWO++Q4/pR4DHt6xfMxtUO7wAOHeM22wx8aDXzDju2OdO/CPjYiLbbot9549jPfPjw4cOHj0l+dP3M23HA9qr6clV9G7gEOHnMMa1aVX0CuHPccQxCVe2uqs+0r+8FbgYOH29Uq1eNve3g/u1jrd7NZzn16mTgPe12+SRwcJLDljnv0OKqqr+pqrvawU/S9GU3Cv2s97C/x1a6/FOB9w/w/Re0jO+8cexnkiRNrK4nb4cDt/YM72SCE4S1KslG4IdozlZNrCT7Jbke2ANcVVUTvT6LWE69WmiaYdbJlS77TJqzNrMK+GiS65KcNaCYVhrbs9qmt3+e5CkrnHfYsZHkkcAJwJ/1FA9zuy1lHPuZJEkTayj9vA1Q5ilbq2dDJlKSA2l+CL62qu4Zdzz9qKr7gWOTHAxcluSpVTXx1yfOYzn1aqFphlknl73sJM+mSd5+rKf4+KraleRxwFVJvtCe+RlVbJ8BnlhVe5OcCPwv4Khlzjvs2Ga9CPjrquo9GzbM7baUcexnkiRNrK6fedsJbOgZPgLYNaZYNEeS/WkSt4ur6oPjjmdQquqbwAzNGYq1aDn1aqFphlknl7XsJD8IvAs4uarumC2vql3t8x7gMpqmd4OyZGxVdc9s09uq+jCwf5JDlzPvsGPrcQpzmkwOebstZRz7mSRJE6vrydungaOSHJnk4TQ/PK4Yc0yiuUsc8G7g5qp6y7jj6VeSx7Zn3EhyAPBTwBfGGtTwLKdeXQG8sr0b4DOBu6tq9zLnHVpcSZ4AfBB4RVV9sad8XZKDZl8DzwMGedZ0ObF9b1svSHIczffrHcuZd9ixtTE9GvgJ4PKesmFvt6WMYz+TJGlidbrZZFXtS3I28Bc0dx+7oKpuGnNYq5bk/TR3pDs0yU7gjVX17vFGtWrHA68AbmivEwN4fXvGYRIdBlyUZD+aH92XVtWHxhzTUCxUr5L8fDv+D4EP09wJcDvwj8B/WmzeEcb1BuB7gLe3edK+qtoErKdp6grN99r7quojg4hrBbH9NPALSfYB/wScUlUFDPV7bJmxAbwU+GhV3dcz+1C323zfeTQ3AxrbfiZJ0iRL89tCkiRJktRlXW82KUmSJEnC5E2SJEmSJoLJmyRJkiRNAJM3SZIkSZoAJm+SJEmSNAFM3iRJkiRpApi8SZIkSdIE+P8AiAN1oRXZRakAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1080x720 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.hist(figsize=(15,10), bins=9)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bd36883",
   "metadata": {},
   "source": [
    "# Selection of features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "589adf19",
   "metadata": {},
   "source": [
    "In this part, I tried to remove the columns based on the importance of each column in the above model and my own knowledge, which resulted in 2 **rest-ecg and fbs** columns <br> and one **trtbps** column, which I think and the searches I did. It did not give better results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "94719365",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 63. ,   1. ,   3. , 233. , 150. ,   0. ,   2.3,   0. ,   0. ,\n",
       "          1. ],\n",
       "       [ 37. ,   1. ,   2. , 250. , 187. ,   0. ,   3.5,   0. ,   0. ,\n",
       "          2. ],\n",
       "       [ 41. ,   0. ,   1. , 204. , 172. ,   0. ,   1.4,   2. ,   0. ,\n",
       "          2. ],\n",
       "       [ 56. ,   1. ,   1. , 236. , 178. ,   0. ,   0.8,   2. ,   0. ,\n",
       "          2. ],\n",
       "       [ 57. ,   0. ,   0. , 354. , 163. ,   1. ,   0.6,   2. ,   0. ,\n",
       "          2. ]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x=df[['age','sex','cp','chol','thalach','exng','oldpeak','slp','ca','thall']].values\n",
    "x[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fa78b68a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 1, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y=df['output'].values\n",
    "y[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ab648a9",
   "metadata": {},
   "source": [
    "# Data normalization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c3b39859",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.9521966 ,  0.68100522,  1.97312292, -0.25633371,  0.01544279,\n",
       "        -0.69663055,  1.08733806, -2.27457861, -0.71113139, -2.14887271],\n",
       "       [-1.91531289,  0.68100522,  1.00257707,  0.07219949,  1.63347147,\n",
       "        -0.69663055,  2.12257273, -2.27457861, -0.71113139, -0.51292188],\n",
       "       [-1.47415758, -1.46841752,  0.03203122, -0.81677269,  0.97751389,\n",
       "        -0.69663055,  0.31091206,  0.97635214, -0.71113139, -0.51292188],\n",
       "       [ 0.18017482,  0.68100522,  0.03203122, -0.19835726,  1.23989692,\n",
       "        -0.69663055, -0.20670527,  0.97635214, -0.71113139, -0.51292188],\n",
       "       [ 0.29046364, -1.46841752, -0.93851463,  2.08204965,  0.58393935,\n",
       "         1.43548113, -0.37924438,  0.97635214, -0.71113139, -0.51292188]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import preprocessing\n",
    "x= preprocessing.StandardScaler().fit(x).transform(x.astype(float))\n",
    "x[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7c9e43",
   "metadata": {},
   "source": [
    "# Training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "d534cfb9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set: (242, 10) (242,)\n",
      "Test set: (61, 10) (61,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train ,x_test , Y_train , y_test =train_test_split(x , y, test_size=0.2 , random_state=4) \n",
    "print ('Train set:', X_train.shape,  Y_train.shape)\n",
    "print ('Test set:', x_test.shape,  y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28df9c5d",
   "metadata": {},
   "source": [
    "# Models for find best"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e0b8ccb",
   "metadata": {},
   "source": [
    "Here we review 6 of the most famous classification models for this dataset.<br>\n",
    "Then we evaluate each of the models and tried to put the best parameters in the models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a855bd82",
   "metadata": {},
   "source": [
    "We will try **KNN** as the first model.<br>\n",
    "We find the best value of n in this way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fe0e313d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9b0a8272",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.7704918 , 0.78688525, 0.83606557, 0.85245902, 0.85245902,\n",
       "       0.85245902, 0.8852459 , 0.85245902, 0.8852459 ])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "kn =10\n",
    "Accuracy =np.zeros((kn-1))\n",
    "\n",
    "for number in range(1,kn):\n",
    "    neighbors =KNeighborsClassifier(n_neighbors = number)\n",
    "    neighbors.fit(X_train,Y_train)\n",
    "    yhat =neighbors.predict(x_test)\n",
    "    Accuracy[number-1] =metrics.accuracy_score(y_test, yhat)\n",
    "    \n",
    "Accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0cd5e97",
   "metadata": {},
   "source": [
    "Show accuracy in plot "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f196be23",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1iElEQVR4nO3dd3xW9fn/8dfFMswoQ0S2yJAtRlyIVqviwI3iqL9qv1KtWutEW1trrX5d2KFWi7MWBQFFseKqC0RFWWGDEFYAJYywIev6/XFO/N6GhBwgd+77Tt7PxyMPcp/7jHcC5Mo553Ouj7k7IiIiyaZGogOIiIiURgVKRESSkgqUiIgkJRUoERFJSipQIiKSlGolOkBFatq0qbdr1y7RMUREZC9MmzZtnbs3K7m8ShWodu3aMXXq1ETHEBGRvWBmy0tbrkt8IiKSlFSgREQkKalAiYhIUqpS96BKk5+fT3Z2Njt37kx0lDKlpaXRqlUrateunegoIiJJo8oXqOzsbBo2bEi7du0ws0TH2Y27s379erKzs2nfvn2i44iIJI0qf4lv586dNGnSJCmLE4CZ0aRJk6Q+wxMRSYQqX6CApC1OxZI9n4hIIlSLAiUiIqlHBaqSjBs3DjNjwYIFiY4iUm1t2JbHo+8vYP3WXYmOUiU8OGE+I74q9RnbCqECVUlGjhxJv379GDVqVKKjiFRL7s5dr8/iqU+WcPcbs9Fkrftn4qIchk/MYtm6bXE7hgpUJdi6dSuTJ0/m+eefV4ESSZBxM1bxwbzv6d36QD6Y9z3jZqxKdKSUtWlHPneOncXhBzfg9jM6x+04VX6Yeaz73p7LvNWbK3SfXQ9txL0Du+1xnTfffJMBAwbQqVMnGjduzPTp0+nTp0+F5hCRsq3O3cG94+dydLuDePXaY7ni2SncO34uxx7WhEMPrJvoeCnnvvFzydm6i3/+7CjSateM23F0BlUJRo4cyeDBgwEYPHgwI0eOTHAikerD3Rn6+iwKCp3HBvWids0aPDqoJ4VFwXJd6ts77835jjdmrOKGkzvQq/WBcT1WtTqDKu9MJx7Wr1/Pxx9/zJw5czAzCgsLMTMeeeQRDS8XqQQjpqxg0rfruP/87rRtUh+Atk3q89uzjuCeN+cwYsoKfnZs2wSnTA3rtu7id+Nm0+3QRtx4Sse4H09nUHE2duxYrrrqKpYvX86yZctYuXIl7du35/PPP090NJEqb9m6bTz4znxO7NiUK49p86P3rjimDf07NePBd+bH9UZ/VeHu/G7cbLbsLODxS3pTp1b8y4cKVJyNHDmSCy644EfLLrroIl599dUEJRKpHgqLnNvHZFKrpvHIxT13u2JhZjxyUU9q1zRuH5NJYZEu9e3JmzNX8f7c77n19E50PqRhpRwzrgXKzAaY2UIzW2xmd5XyfrqZvW1mmWY218yujnnvlnDZHDMbaWZp8cwaL59++ikDBgz40bJf//rXPP300wlKJFI9PDcpi6nLN3Lfud1okV76QIhD0tO477xuTF2+kecmZVVywtSxZtMO/vDWXDLaHsS1Jx5WaceNW4Eys5rAU8CZQFfgMjPrWmK1G4B57t4LOBkYZmZ1zKwl8Gsgw927AzWBwfHKKiJVy8LvtjDsg0Wc0a05FxzZco/rnt+7JQO6HcKwDxax8LstlZQwdbg7d479v0EmNWtU3r3zeJ5B9QUWu3uWu+cBo4DzSqzjQEMLzr0bABuAgvC9WkBdM6sF1ANWxzGriFQR+YVF3Dp6Jg3TavHABT3KHYxkZjxwQXcaptXi1tEzyS8sqqSkqeGVcJDJb8/qQrum9Sv12PEsUC2BlTGvs8NlsZ4EjiAoPrOBm929yN1XAY8BK4A1wCZ3/6C0g5jZEDObamZTc3JySg2S7MNIkz2fSCp54uPFzF29mQcu6EHTBgdE2qZJgwN48MIezF29mSc+XhznhKlj+fptPDghHGSSgJGO8SxQpf3aUvIn8RnATOBQoDfwpJk1MrODCM622ofv1TezK0s7iLsPd/cMd89o1qzZbu+npaWxfv36pC0CxfNBpaWl5C02kaSSuTKXpz5ZzIVHtmRA90P2atszuh3ChX1a8tQni8lcmRufgCmkeJBJzRrGwxftPsikMsTzOahsoHXM61bsfpnuauAhD6rHYjNbCnQB2gJL3T0HwMzeAI4HRuxtiFatWpGdnU1ZZ1fJoHhGXRHZdzvzC7ltTCbNGhzAvefu2zOP9w7sxpdL1nPbmEz+c1O/uHZJSHbPf57FN8s2MmxQr4R124hngfoG6Ghm7YFVBIMcLi+xzgrgVGCSmTUHOgNZBGdfx5pZPWBHuM7UfQlRu3ZtzVQrUg089v5CFq/dysvX9CW9bu192kd63do8cnFPfvb81zz2/kLuOafkuK7qYdH3W3js/UWc3rU5F/bZ8yCTeIrbJT53LwBuBN4H5gOj3X2umV1nZteFq90PHG9ms4GPgKHuvs7dpwBjgekE96ZqAMPjlVVEUtuUrPU8P3kpVx4bPHy7P07s2IyfHduW5ycv5aus9RWUMHUUDzJpkFaLBy8sf5BJPFmy3pvZFxkZGT516j6daIlIitq6q4Az/zYRw3j35hOpf8D+XxjanlfAmX+bRJE7797cnwYVsM9U8ZcPF/G3j77lmSv7MKB7i0o5pplNc/eMksvVSUJEUtoD78wne+MOhl3Sq0KKE0C9OrUYNqgX2Rt38MA78ytkn6lgdvYmnvxkMRcc2bLSitOeqECJSMr6dOFaRn69giEnHsbR7RpX6L4z2jVmyImHMfLrFXyycG2F7jsZ7cwv5NbRM2nW4AD+mIDG2qVRgRKRlLRpez5DX59Fp+YNuOW0TnE5xi2ndaJT8wYMHTuL3O15cTlGshj2wUK+XbuVhy/uSXq9fRtkUtFUoEQkJf1h/BzWb83j8Ut6x204eFrtmjx+SW82bMvj3vFz43KMZPD10g089/lSrjimDSft5yCTiqQCJSIpZ8LsNbw1czU3ndKR7i3T43qs7i3TuemUjrw1czUTZq+J67ESYduuAm4fk0nrg+rx27OOSHScH1GBEpGUkrMlmDSvR8t0fvWTDpVyzF/9pAM9W6Xzu3Gzydmyq1KOWVkemDCflRu389igihtkUlFUoEQkZbg7d78xm215hTx+STB9e2WoXbMGwwb1YlteIXe/MTtpW6ftrc8W5fDqlBVce+Jh9G1fsYNMKoIKlIikjNenr+K/87/njtM707F55UyaV6xj84bceUZn/jv/e16fvqpSjx0Pm7bnc+fYTDoe3IBb4zTIZH+pQIlISliVu4P7xs+lb7vGXNMvMe3LrjmhPX3bN+a+8XNZlbsjIRkqyh/fnhv3QSb7SwVKRJJeUZEzdOwsCr3yJ82LVaOG8djFvSh0586xmRSl6DTx781Zw7gZq7jxlMPp0Sq+g0z2hwqUiCS9EVOW8/nidfzu7CNo06ReQrO0aVKPe87uyuTF6xkxZXlCs+yLnC27+O24OfRomc4NPzk80XH2SAVKRJLa0nXBpHkndWrG5X3bJDoOAJf1bc1JnZrx4IT5LF23LdFxInN3fjduNlt3FTCsEgeZ7KvkTici1VphkXPb6JnUqVkjYZPmlcYsmMSvTs0a3DZ6JoUpcqnvjemr+GDe99x+eic6VfIgk32hAiUiSWv4xCymr8jlT+d155D05Jp1+pD0NO4/vzvTV+QyfGJWouOUa3XuDv4YDjL5Rb/DEh0nEhUoEUlKC77bzF8+XMSZ3Q/hvN6HJjpOqc7tdShndj+Ev3y4iAXfbU50nDK5O3eGg0weHdQzYYNM9pYKlIgknbyCIm59LZNGdWvx5/O7J82lvZLMjD+f351GdWtx62uZ5BUUJTpSqUZ8FQwy+e1ZR9C2Sf1Ex4lMBUpEks4TH3/LvDWbefCCHjRpcECi4+xRkwYH8L8X9mTems088fG3iY6zm2XrtvHghAX079SMK45JjkEmUalAiUhSmbkyl398uoSL+rTi9G6HJDpOJKd1bc5FfVrxj0+XMHNlbqLj/KCwyLl9TCa1axqPJNEgk6hUoEQkaezML+S20TM5uOEB/GFg10TH2Sv3ntuV5g0P4NbRM9mZX5joOAA8OymLqcs3ct953ZJukEkUKlAikjQeeW8hS3K28ejFvUivmxyT5kXVKK02j1zci6ycbTzy3sJEx2Hhd1t4/INFDOh2COf3bpnoOPtEBUpEksKXS9bzwuSlXHVcW/p1bJroOPukX8emXHVcW16YvJQvl6xPWI68giJuHT2Thmm1eOCC5B1kUh4VKBFJuK27CrhjbCbtmtTjrjO7JDrOfrnrzC60a1KP28dksnVXQUIyPPnxt8xdvZkHL0z+QSZ7sscCZWY1zWxEZYURkerpgXfmsTp3B8Mu6UW9Osk1ad7eqlenFsMu6cWaTTv483/mVfrxM1fm8tSnS7iwT0vOSJFBJmXZY4Fy90KgmZnVqaQ8IlLNfLJgLSO/XsmQ/h04qm3yTZq3L45q25gh/Tsw6puVfLJgbaUdd2d+IbeGg0zuHdit0o4bL1Eu8S0DJpvZ783s1uKPOOcSkWogd3seQ1+fRefmDbnltI6JjlOhbjmtI52bN2To67PI3Z5XKcd87P1gkMkjF/dMuUEmpYlSoFYD/wnXbRjzISKyX/7w1lw2bMtj2CW9OKBWck6at68OqFWTYZf0YsO2PH7/1ty4H++rrPU8P3kpPzu2LSd2bBb341WGci/2uvt9AGZW391Tp6+8iCS1d2atYXzmam49rRPdWybvpHn7o3vLdG4+tSPDPlzEGd2ac07P+PQU3LqrgNvHZNKmcT3uPiu1B5nEKvcMysyOM7N5wPzwdS8z+0fck4lIlbV2y07ueXM2vVql86uTOyQ6Tlxdf3IHerVK5/dvzmHtlp1xOcYD78xnVe4Ohg1K/UEmsaJc4vsrcAawHsDdM4H+UXZuZgPMbKGZLTazu0p5P93M3jazTDOba2ZXx7x3oJmNNbMFZjbfzI6L9BWJSFJzd+5+fTbb8woZdklvaiX5pHn7q1bNGgy7pDfb8wq5+/XZuFfs3FGfLFzLyK9XMKT/YWS0qxqDTIpF+pfh7itLLCq3j4eZ1QSeAs4EugKXmVnJ3iU3APPcvRdwMjAsZsTg34D33L0L0IvwDE5EUtuYadl8tGAtd5zRmcMPbpDoOJXi8IMbcOeALny0YC1jpmVX2H5zt+cxdOwsOjVvwC0/7VRh+00WUQrUSjM7HnAzq2NmtxOtWPQFFrt7lrvnAaOA80qs40BDCx5zbgBsAArMrBHBWdrzAO6e5+65kb4iEUla2Ru386e353FM+8Zcc0L7RMepVFcf345j2jfmT2/PI3vj9grZ573jg0Emj1/Sm7TaVWuQCUQrUNcRnOm0BFYBvcPX5WkJxJ55ZYfLYj0JHEEwUnA2cLO7FwGHATnAi2Y2w8yeM7NSJzExsyFmNtXMpubk5ESIJSKJUFQUTJrn7jw2qBc1UmTSvIpSo4bx2KBeuDt3jJlF0X5OEz9h9hremrmaX5/ascoOMim3QLn7One/wt2bu3szd7/S3aM0mSrtX1/Jv5EzgJnAoQSF78nw7KkW0Ad42t2PBLYBu93DCvMNd/cMd89o1qxqDK0UqYpe/nIZXyxZzz3ndKV143qJjpMQrRvX4/fndOXLrPW8/OWyfd5PzpZd/G7cbHq2Suf6KjzIJMoovsPCgQw5ZrbWzN4ysygT2mcDrWNetyI4U4p1NfCGBxYDS4Eu4bbZ7j4lXG8sQcESkRSUlbOVh95bwMmdmzH46Nblb1CFXXp0a07u3IyH3ltAVs7Wvd7e3bn7jdlsyyvk8Ut6UbsKDzKJ8pW9CowGWhCc6YwBRkbY7hugo5m1Dwc+DAbGl1hnBXAqgJk1BzoDWe7+HcG9r87heqcCld/USkT2W0FhEbeNyeSAWjV5OAUnzatoZsbDF/XkgFo1uW1MJgWFezdN/Nhp2fx3/vfceUZnDj+4avdMiFKgzN3/7e4F4ccIdr9Utxt3LwBuBN4nGFQx2t3nmtl1ZnZduNr9wPFmNhv4CBjq7uvC924CXjGzWQSX/x7cq69MRJLCPydmMWNFLn86rxvNG6XepHnx0LxRGn86rxszVuTyz4lZkbdblbuDP709j77VZJBJmU90mVnxgPpPwmeYRhEUpkuBd6Ls3N0nABNKLHsm5vPVwOllbDsTyIhyHBFJTvPXbOav/13E2T1acG6v+HRRSFXn9jqUD+Z+z1//u4ifdD6Yroc22uP6wSCTTArdGVZNBpns6ZHjaQQFqfi78MuY95zg7EdEpFTBpHmZpNetw/3np+6kefFiZtx/fnemLN3AraNnMv7GftSpVfZFrRFTljN58XoevKBHtRlkUuZ3w93bu/th4Z8lP6IMkhCRauzvH33L/DWbeejCHjSurxl7StO4fh0eurAHC77bwt8+WlTmekvXbePBCfM5uXMzLutbfQaZlNu0KewIcTbQLnZ9d388frFEJJXNWLGRf3y6mEFHteKnXZsnOk5S+2nX5gw6qhVPf7qEU49oTp82B/3o/cIi57bRM6vlIJMogyTeBn4ONEHTbYhIOXbkFXLb6ExapNfl9wNLdjeT0vx+YFdapNfl9tGZ7Mj7cSe54ROzmF5NB5lEaXvbyt17xj2JiFQJj7y/gKx123j1f46hUVrqT5pXGRql1ebRi3ty+XNTePi9Bfzx3GA23AXfbeYvHy7irB6HVMtBJlHOoN41s1JH2omIxPpiyTpenLyMnx/fjuMPb5roOCnl+MOb8vPj2/HSF8v4YvE68gqKuOW1TBrVrcWfz+9RrS7tFYtyBvUVMM7MagD5BKP63N33PCZSRKqVLTvzuWPMLNo3rc/QAVVn0rzKNHRAFz5blMMdY2dxerfmzF+zmWevyqi2g0yiFKhhwHHAbK/oiUxEEmhV7g5en5a910/yS+lmrMxlzaYdjL3+eOrWqXqdtStD3TrBNPEXP/0FL05exsVHteK0ajzIJEqB+haYo+IkVcl7c75j6Ouz2LQjn2p45SQuaphx2+mddxuFJnunT5uDuP2MzkyYvYY/VPNBJlEK1BrgUzN7F9hVvFDDzCUV7cwv5IF35vPvr5bTo2U6T1x2JO2aljqTi0jC/Orkw/nVyYcnOkbCRSlQS8OPOuGHSEpavHYLN746gwXfbeHaE9tzxxld9vjkvogkVrkFyt3vq4wgIvHi7oyZls29b82lbp2avHj10fyk88GJjiUi5YjSSeITSule7u6nxCWRSAXasjOf342bw/jM1RzfoQl/ubR3tXvYUSRVRbnEd3vM52nARUBBfOKIVJzMlbn8etQMsjfu4PbTO3H9yYdTsxp0gBapKqJc4ptWYtFkM/ssTnlE9ltRkfP850t5+L0FNG+UxmtDjiWjXePyNxSRpBLlEl/s/+wawFHAIXFLJLIf1m3dxe1jMvl0YQ5ndGvOIxf1Ir2e2u2IpKIol/hi54UqIBjR94t4hhLZF18sXsdvXptJ7o587j+/O1ce06ZatocRqSqiXOKr+vMKS0orKCziL/9dxD8+XcJhTevzr2v6ckQLdeISSXVRzqAws+PZfT6ol+OUSSSy7I3buXnUTKYt38ilGa2599yu1KsT6Z+1iCS5KPeg/g10AGYCxROVOKACJQn13pw13Dl2FkUOf7/syGo5HYFIVRblV80MoKt68Umy2JlfyJ/fmceIr1bQq1U6f7/sSNo2UbsikaomSoGaQzBqb02cs4iUK7Zd0ZD+h3H76Z3VrkikiopSoJoC88zsa37cLPbcuKUSKcHdGT11JX8cP496alckUi1EKVB/jHcIkT3ZHLYrejtzNScc3oS/XNKbg9WuSKTKizLMXF0jJGEyV+Zy08gZrMrdwR1ndOa6kzqoXZFINaHxuJKUioqc5z7P4pH3FtK8URqjf3ksR7VVuyKR6kQFSpLOuq27uG10Jp8tymFAt0N4+KKealckUg1FeQ7qqJINY81soLu/Hb9YUl1NDtsVbdqRz5/P784ValckUm1FGZ/7rJn1KH5hZpcB90TZuZkNMLOFZrbYzO4q5f10M3vbzDLNbK6ZXV3i/ZpmNsPM/hPleJK68guLePT9BVz5/BTS69Zm/I0ncOWxbVWcRKqxKJf4LgbGmtkVQD/gKuD08jYys5rAU8BpQDbwjZmNd/d5MavdAMxz94Fm1gxYaGavuHte+P7NwHxAjdWqsOyN2/n1yBlMX5HL4KNb84eBalckItFG8WWZ2WDgTWAlcLq774iw777AYnfPAjCzUcB5QGyBcqChBb8mNwA2EE6GaGatgLOBB4Bbo35Bklrenb2Goa+rXZGI7K7MAmVms/nxVO+NgZrAFDPD3XuWs++WBAWtWDZwTIl1ngTGA6uBhsCl7l4UvvdX4M5weZnMbAgwBKBNmzblRJJksTO/kPv/M49XpgTtip64rA9tmtRLdCwRSSJ7OoM6Zz/3XdrNg5L9/M4gaEJ7CkFD2g/NbBLQH1jr7tPM7OQ9HcTdhwPDATIyMtQvMAV8+33Qrmjh91v4Zf/DuE3tikSkFGUWKHdfvp/7zgZax7xuRXCmFOtq4KGwEe1iM1sKdAFOAM41s7OANKCRmY1w9yv3M5MkkLvz2jcr+ePbc6lfpxb/uqYvJ3VqluhYIpKk4vlr6zdARzNrb2Z1gMEEl/NirQBOBTCz5kBnIMvd73b3Vu7eLtzuYxWn1LZ5Zz43jZzBXW/M5qi2B/HuzSeqOInIHsVtqJS7F5jZjcD7BPeuXnD3uWZ2Xfj+M8D9wEvh/S4Dhrr7unhlksSYuTKXm0ZOZ3XuTu44ozPXn9SBGmpXJCLlsKo0zVNGRoZPnTo10TEkVFTkPDspi0ffD9oV/f2y3mpXJCK7MbNp7p5RcnmUThIXAg8DBxOc5Rjg7q5nk6RMOVt2cduYTCYuyuHM7ofw0EU9Sa+rdkUiEl2US3yPAAPdfX68w0jVsGlHPgOf+JyN2/N44ILuXN5X7YpEZO9FKVDfqzjJ3nhlynK+27yTsdcdR0Y7XdITkX2zpwd1Lww/nWpmrxF0koidUfeN+EaTVLSroJCXJi+j3+FNVZxEZL/s6QxqYMzn2/lx/z0HVKBkN+Nnrmbtll08NqhXoqOISIrb04O6V5f1nkhp3INRe10OaciJHZsmOo6IpLhyH9Q1s3+Z2YExrw8ysxfimkpS0qeLclj0/VaG9D9MgyJEZL9F6STR091zi1+4+0bgyLglkpQ1/LMsDmmUxjk91ZFcRPZflAJVw8wOKn5hZo3RVPFSwuzsTXyZtZ5r+rVT41cRqRBRCs0w4AszGxu+HkQwR5PID4ZPyqLBAbUY3FdTnohIxYgyYeHLZjYN+AlBF4kLS8yKK9Xcyg3bmTB7Ddec0I5GaeoWISIVI9KlurDJaw7B1BeYWRt3XxHXZJIyXpy8DAOuPqF9oqOISBUSZRTfuWb2LbAU+AxYBrwb51ySIjZtz2fUNysY2OtQDj2wbqLjiEgVEuVu9v3AscAid29PMH/T5LimkpTxytfL2Z5XyLUnHpboKCJSxUQpUPnuvp5gNF8Nd/8E6B3fWJIKdhUU8uLkZZzYsSldD1VzexGpWFHuQeWaWQNgEvCKma0FCuIbS1LBWzNXk7NlF8PU1khE4iDKGdR5BL34fgO8Byzhx336pBpyd56dqLZGIhI/UYaZbzOztkBHd/+XmdUjmMJdqrFPF+bw7dqtPH5JL7U1EpG4iDKK71pgLPDPcFFLgqk3pBobPjFoazSwl9oaiUh8RLnEdwNwArAZwN2/JZj+Xaqp2LZGtWuqrZGIxEeUny673D2v+IWZ1SKYD0qqKbU1EpHKEKVAfWZmvwXqmtlpwBjg7fjGkmRV3Nbo8mPaqK2RiMRVlAJ1F5ADzAZ+CUwA7olnKEleL0xeigE/P75doqOISBUXZRRfEfBs+CHV2Kbt+bz2zUrOVVsjEakEusMtkY2YErQ1+h+1NRKRSqACJZHsKijkpS/U1khEKs8eC5SZ1TSzRysrjCSvt2YEbY2G9NfZk4hUjj0WKHcvBI4ytQqo1oqKnOGTsjiiRSP6Ha62RiJSOaI0i50BvGVmY4BtxQvd/Y24pZKk8tmiHBav3cpfLlVbIxGpPFHuQTUG1gOnEDSJHQicE2XnZjbAzBaa2WIzu6uU99PN7G0zyzSzuWZ2dbi8tZl9Ymbzw+U3R/+SpKL9c+ISWqSncU5PtTUSkcoTZZj51fuyYzOrCTwFnAZkA9+Y2Xh3nxez2g3APHcfaGbNgIVm9grBdB63uft0M2sITDOzD0tsK5VgVnYuX2Vt4HdnHaG2RiJSqaI0i21lZuPMbK2ZfW9mr5tZqwj77gssdvessFXSKIKpO2I50DC8x9UA2AAUuPsad58O4O5bgPkETWqlkg2fmEXDA2oxuG/rREcRkWomyq/ELwLjgUMJisTb4bLytARWxrzOZvci8yRwBLCaoFPFzeGDwT8ws3bAkcCU0g5iZkPMbKqZTc3JyYkQS6Iqbmt02TFtaKi2RiJSyaIUqGbu/qK7F4QfLwHNImxX2t30kk1mzwBmEhS/3sCTZvbDQzbhTL6vA79x982lHcTdh7t7hrtnNGsWJZZE9fznS6lhxtUntEt0FBGphqIUqHVmdmX4TFRNM7uSYNBEebKB2OtCrQjOlGJdDbzhgcXAUqALgJnVJihOr2jEYOXL3Z7H6KlBW6MW6WprJCKVL0qBuga4BPgOWANcHC4rzzdARzNrb2Z1gMEElwpjrQBOBTCz5kBnICu8J/U8MN/dH4/yhUjFemXKCrbnFXKtHswVkQQpcxSfmT3s7kOBY9z93L3dsbsXmNmNwPsEU8S/4O5zzey68P1ngPuBl8xsNsElwaHuvs7M+gE/A2ab2cxwl7919wl7m0P2XmxboyNaqK2RiCTGnoaZn2Vm9wB3E8wBtdfCgjKhxLJnYj5fDZxeynafU/o9LKkExW2N/nJJ70RHEZFqbE8F6j1gHVDfzDYTFAwv/tPd9at1FRTb1uiEw5skOo6IVGNl3oNy9zvcPR14x90buXvD2D8rMaNUok8XrWXx2q0M6d9ebY1EJKHKHSTh7iUfrpUqbPjELLU1EpGkoN418oPitkbXnNBebY1EJOH0U0h+oLZGIpJMovTiO8fMVMiquOK2RperrZGIJIkohWcw8K2ZPWJmR8Q7kCRGcVujn6utkYgkiSiDJK4kaNa6BHjRzL4MG7Q2jHs6qRQ/tDXqrbZGIpI8Il26Cxu1vk4wZUYL4AJgupndFMdsUkl+aGt0otoaiUjyiHIPaqCZjQM+BmoDfd39TKAXcHuc80mc7Soo5MXJy+jfqZnaGolIUil3Rl1gEPAXd58Yu9Ddt5tZlKaxksTenLGKdVt3MURnTyKSZKIUqHsJupgDYGZ1gebuvszdP4pbMom7oiLn2UlL6aq2RiKShKLcgxoDxM5yW8g+No+V5PLJwuK2RoeprZGIJJ0oBaqWu+cVvwg/rxO/SFJZhk/M4tD0NM7u2SLRUUREdhOlQOWY2Q/zQZnZeQRdziWFZa7MZcrSDVzTT22NRCQ5RbkHdR3wipk9STDVxkrgqrimkrgbPiloa3Tp0WprJCLJqdwC5e5LgGPNrAFg7r4l/rEknlZu2M67s9dwbf/D1NZIRJJWlDMozOxsoBuQVnwz3d3/FMdcEkfFbY2uPr59oqOIiJQpyoO6zwCXAjcRXOIbBLSNcy6Jk9ztebz2TdDW6JD0tETHEREpU5S748e7+1XARne/DzgO0I2LFPXKlBXsyC9kSH89mCsiyS1KgdoZ/rndzA4F8gFdG0pBO/P/r61Rl0PU1khEkluUe1Bvm9mBwKPAdMCBZ+MZSuLjrZlBW6Nf6uxJRFLAHgtUOFHhR+6eC7xuZv8B0tx9U2WEk4pTVOQMn5hF1xaNOL6D2hqJSPLb4yU+dy8ChsW83qXilJo+WbiWJTnb1NZIRFJGlHtQH5jZRaafaintn2prJCIpJso9qFuB+kCBme0kGGru7q677Cli5spcvl66gXvOPkJtjUQkZUTpJKGp3VPcs5OyaJhWi8F92yQ6iohIZOUWKDPrX9rykhMYSnJasf7/2ho1OCBS4xARkaQQ5SfWHTGfpwF9gWnAKeVtaGYDgL8BNYHn3P2hEu+nAyOANmGWx9z9xSjbSjQvTF5KzRpqayQiqSfKJb6Bsa/NrDXwSHnbmVlN4CngNCAb+MbMxrv7vJjVbgDmuftAM2sGLDSzVwgmRSxvWynHxm1hW6NeLdXWSERSzr7cMc8GukdYry+w2N2zwkkORwHnlVjHgYbhCMEGwAagIOK2Uo5XpixnR34h1/bX2ZOIpJ4o96CeICgkEBS03kBmhH23JJg7qlg2cEyJdZ4ExgOrgYbApe5eZGZRti3ONwQYAtCmjQYBFNuZX8hLXyznJLU1EpEUFeUe1NSYzwuAke4+OcJ2pT035SVenwHMJLif1QH40MwmRdw2WOg+HBgOkJGRUeo61dGbM4K2RmoKKyKpKkqBGgvsdPdCCO4tmVk9d99eznbZ/LjreSuCM6VYVwMPubsDi81sKdAl4rZShqIi59lJWXQ7VG2NRCR1RbkH9RFQN+Z1XeC/Ebb7BuhoZu3NrA4wmOByXqwVwKkAZtYc6AxkRdxWyvDxArU1EpHUF+UMKs3dtxa/cPetZlavvI3cvcDMbgTeJxgq/oK7zzWz68L3nwHuB14ys9kEl/WGuvs6gNK23cuvrdoaPiloa3RWD7U1EpHUFaVAbTOzPu4+HcDMjgJ2RNm5u08AJpRY9kzM56uB06NuK+VTWyMRqSqiFKjfAGPMrPgeUAuCKeAlCT07UW2NRKRqiPKg7jdm1oXg/pABC9w9P+7JZK+tWL+dd+esYUj/DmprJCIpr9xrQGZ2A1Df3ee4+2yggZn9Kv7RZG89/3lW0NbohHaJjiIist+i3KS4NpxRFwB33whcG7dEsk82bstj9NRszuvdkuaN1NZIRFJflAJVI3aywrDHXp34RZJ9MeKrsK3RiXowV0Sqhig3Kt4HRpvZMwTdHK4D3otrKtkrO/ML+deXyzipUzM6H6Lpu0SkaohSoIYS9Lq7nmCQxAfAs/EMJXsnaGuUxy/V1khEqpByL/G5e5G7P+PuF7v7RcBc4In4R5Moioqc4WFbo+PU1khEqpBIT3KaWW8ze9jMlhF0f1gQ11QS2ccL1pKltkYiUgWVeYnPzDoR9MC7DFgPvAaYu/+kkrJJBMMnZtHywLpqayQiVc6ezqAWEDRyHeju/dz9CYKZbiVJzFixka+XbeDqE9qprZGIVDl7+ql2EfAd8ImZPWtmp1L6PE2SIM9NWqq2RiJSZZVZoNx9nLtfSjA/06fALUBzM3vazEpt8CqVp7it0RXHtFVbIxGpkqKM4tvm7q+4+zkEEwfOBO6KdzDZM7U1EpGqbq9uXLj7Bnf/p7ufEq9AUj61NRKR6kB31lOQ2hqJSHWgApViitsandxZbY1EpGpTgUox48K2RkN09iQiVZwKVAp5Z9YaHpwwnx4t09XWSESqPI1PTgE78gr503/mMfLrFfRufSBPXHak2hqJSJWnApXkFn63hZtGTmfR91u57qQO3HZ6J3WNEJFqQQUqSbk7r369gj+9PY+GabV4+Zq+9O/ULNGxREQqjQpUEtq0I5+735jFhNnfcWLHpgy7pBcHN9TzTiJSvahAJZnpKzZy06sz+H7zTu46swtDTjyMGjV0v0lEqh8VqCRRVOQ8M3EJwz5YRIv0NEZfdxx92hyU6FgiIgmjApUE1m7Zya2vZfL54nWc3aMFD17Yg/S6tRMdS0QkoVSgEuyzRTncNnomW3YW8L8X9mDw0a01hFxEBBWohMkvLOKxDxbyz8+y6NS8Aa9eeyydmqt1kYhIsbg+UGNmA8xsoZktNrPdpugwszvMbGb4McfMCs2scfjeLWY2N1w+0syqzDC2Feu3c/EzX/LPz7K4/Jg2vHVDPxUnEZES4lagzKwm8BRwJtAVuMzMusau4+6Puntvd+8N3A185u4bzKwl8Gsgw927AzWBwfHKWpnezlzN2X+fRFbOVv5xRR8evKAHdevUTHQsEZGkE89LfH2Bxe6eBWBmo4DzgHllrH8ZMLJEtrpmlg/UA1bHMWvc7cgr5L635zLqm5Uc2eZA/j74SFo3rpfoWCIiSSueBaolsDLmdTZwTGkrmlk9YABwI4C7rzKzx4AVwA7gA3f/oIxthwBDANq0aVNh4SvSgu82c9OrM1ics5XrT+7AraepXZGISHni+VOytKFoXsa6A4HJ7r4BwMwOIjjbag8cCtQ3sytL29Ddh7t7hrtnNGuWXK2A3J1XpiznvCcns3F7Pi9f05ehA7qoOImIRBDPM6hsoHXM61aUfZluMD++vPdTYKm75wCY2RvA8cCIOOSMi03b87nrjVm8O+c7+ndqxrBBvWjW8IBExxIRSRnxLFDfAB3NrD2wiqAIXV5yJTNLB04CYs+QVgDHhpf+dgCnAlPjmLVCTVu+kV+PDNoV3X1mF65VuyIRkb0WtwLl7gVmdiPwPsEovBfcfa6ZXRe+/0y46gUE95i2xWw7xczGAtOBAmAGMDxeWStKUZHz9GdLePzDoF3RmOuO40i1KxIR2SfmXtZtodSTkZHhU6cm5kTrR+2Kerbgfy/sQaM0tSsSESmPmU1z94ySy9VJogJ8unAtt43OZFteAQ9d2INL1a5IRGS/qUDth7yCoF3R8IlZdG7ekFGXH0tHdYQQEakQKlD7aMX67dw0cjqZ2Zu44pg2/P6crqTVVkcIEZGKogK1D97OXM1v35gNBk9f0Ycze7RIdCQRkSpHBWovbM8r4L7x83ht6kr6tDmQv6ldkYhI3KhARbTgu83c+OoMluRs5YafdOA3P1W7IhGReFKBKoe7M2LKCu7/zzzS69bm39ccQ7+OTRMdS0SkylOB2oNN2/MZ+vos3psbtCt6/JJeNG2gdkUiIpVBBaoMU5dt4OZRM/l+805+e1YX/qef2hWJiFQmFagSCoucZ8J2RS0PrMvY64+nd+sDEx1LRKTaUYGKsXbzTn7z2ky+WLKec3q24EG1KxIRSRgVqFBsu6KHL+rBJRlqVyQikkgqUKG1W3bRrOEBjLpM7YpERJKBClRo0FGtOL93S+rU0rNNIiLJQAUqZGbUqaVLeiIiyUKnCyIikpRUoEREJCmpQImISFJSgRIRkaSkAiUiIklJBUpERJKSCpSIiCQlFSgREUlK5u6JzlBhzCwHWL4fu2gKrKugOPGknBUnFTKCcla0VMiZChmhYnK2dfdmJRdWqQK1v8xsqrtnJDpHeZSz4qRCRlDOipYKOVMhI8Q3py7xiYhIUlKBEhGRpKQC9WPDEx0gIuWsOKmQEZSzoqVCzlTICHHMqXtQIiKSlHQGJSIiSUkFSkREkpIKFGBmL5jZWjObk+gse2Jmrc3sEzObb2ZzzezmRGcqyczSzOxrM8sMM96X6Ex7YmY1zWyGmf0n0VnKYmbLzGy2mc00s6mJzlMaMzvQzMaa2YLw3+dxic5Ukpl1Dr+HxR+bzew3ic5VGjO7Jfz/M8fMRppZWqIzlWRmN4f55sbr+6h7UICZ9Qe2Ai+7e/dE5ymLmbUAWrj7dDNrCEwDznf3eQmO9gMzM6C+u281s9rA58DN7v5VgqOVysxuBTKARu5+TqLzlMbMlgEZ7p60D22a2b+ASe7+nJnVAeq5e26CY5XJzGoCq4Bj3H1/Hu6vcGbWkuD/TVd332Fmo4EJ7v5SYpP9HzPrDowC+gJ5wHvA9e7+bUUeR2dQgLtPBDYkOkd53H2Nu08PP98CzAdaJjbVj3lga/iydviRlL8FmVkr4GzguURnSWVm1gjoDzwP4O55yVycQqcCS5KtOMWoBdQ1s1pAPWB1gvOUdATwlbtvd/cC4DPggoo+iApUijKzdsCRwJQER9lNeNlsJrAW+NDdky5j6K/AnUBRgnOUx4EPzGyamQ1JdJhSHAbkAC+Gl0ufM7P6iQ5VjsHAyESHKI27rwIeA1YAa4BN7v5BYlPtZg7Q38yamFk94CygdUUfRAUqBZlZA+B14DfuvjnReUpy90J37w20AvqGlwOSipmdA6x192mJzhLBCe7eBzgTuCG8JJ1MagF9gKfd/UhgG3BXYiOVLbwEeS4wJtFZSmNmBwHnAe2BQ4H6ZnZlYlP9mLvPBx4GPiS4vJcJFFT0cVSgUkx4X+d14BV3fyPRefYkvMzzKTAgsUlKdQJwbnh/ZxRwipmNSGyk0rn76vDPtcA4guv+ySQbyI45Ux5LULCS1ZnAdHf/PtFByvBTYKm757h7PvAGcHyCM+3G3Z939z7u3p/gFkmF3n8CFaiUEg5AeB6Y7+6PJzpPacysmZkdGH5el+A/24KEhiqFu9/t7q3cvR3B5Z6P3T2pfksFMLP64YAYwstmpxNcXkka7v4dsNLMOoeLTgWSZuBOKS4jSS/vhVYAx5pZvfD//KkE95uTipkdHP7ZBriQOHxPa1X0DlORmY0ETgaamlk2cK+7P5/YVKU6AfgZMDu8xwPwW3efkLhIu2kB/CscJVUDGO3uSTuEOwU0B8YFP6eoBbzq7u8lNlKpbgJeCS+fZQFXJzhPqcL7JacBv0x0lrK4+xQzGwtMJ7hsNoPkbHv0upk1AfKBG9x9Y0UfQMPMRUQkKekSn4iIJCUVKBERSUoqUCIikpRUoEREJCmpQImISFJSgZKUY2ZuZsNiXt9uZn+soH2/ZGYXV8S+yjnOoLDr9ycllrcLv76bYpY9aWY/L2d/15nZVeWs83Mze7KM97aWtrwimVmL4q7xZnZybAd5M/uzmb1vZgeY2Sgz6xjvPJL8VKAkFe0CLjSzpokOEit89iuqXwC/cveflPLeWuDm8JmiSNz9GXd/eS+OX2HChqZR3Ao8W8r2vyN4xu98d98FPE3QI1GqORUoSUUFBA8u3lLyjZJnQMVnBuFv7J+Z2WgzW2RmD5nZFRbMXTXbzDrE7OanZjYpXO+ccPuaZvaomX1jZrPM7Jcx+/3EzF4FZpeS57Jw/3PM7OFw2R+AfsAzZvZoKV9fDvAR8P9K2V8HM3svbBw7ycy6hMv/aGa3h58fHWb8Mswc23ni0HD7b83skRL7HmZm083sIzNrFi7rbWZfhfsbF/aJw8w+NbMHzewzgmI6KPwaM81sYilfE8BFBH3bYo95G0Gj0YHuviNcPCn8O1AjgWpOBUpS1VPAFWaWvhfb9AJuBnoQdOTo5O59CabbuClmvXbASQRTcTxjwWRxvyDoKn00cDRwrZm1D9fvC/zO3bvGHszMDiVoqHkK0Bs42szOd/c/AVOBK9z9jjKyPgTcVspZ2XDgJnc/Crgd+Ecp274IXOfuxwGFJd7rDVwafg8uNbPiDtT1CfrT9SGYOuHecPnLwFB370lQgO+N2deB7n6Suw8D/gCc4e69CBqx/kj4vdoYniEVOwG4DjgzZooW3L0IWEzw9yXVmAqUpKSwi/vLwK/3YrNvwjm1dgFLgOIpDGYTFKVio929KJx8LQvoQtAD76qwxdQUoAlQfJ/ka3dfWsrxjgY+DZt+FgCvEMybFOXrWwp8DVxevMyCLvbHA2PCHP8kaC1FzDoHAg3d/Ytw0asldv2Ru29y950E/fLahsuLgNfCz0cA/cLif6C7fxYu/1eJ/K/FfD4ZeMnMrgVKu9TZguDMMNZiwAi+tyWtJejkLdWYTqEllf2VoF/ZizHLCgh/8Qobbcbex4n97b0o5nURP/6/ULL/lxP8IL3J3d+PfcPMTiaYXqI0Vk7+8jxI0Bm8+JJZDSA3nMqkLOUdM/Z7UEjZPwOi9ED74et29+vM7BiCs86ZZtbb3dfHrLsDKDlt+ffAFcBHZrbe3WMHjKSF20g1pjMoSVnuvgEYTXD5rdgy4Kjw8/MIZvTdW4PMrEZ4X+owYCHwPnC9BdOdYGadrPxJ+aYAJ5lZ0/BS3WUEl88icfcFBGc554SvNwNLzWxQmMHMrFeJbTYCW8zs2HDR4IiHqwEU37u7HPjc3TcBG83sxHD5z8rKb2Yd3H2Ku/8BWMfuk9ct4sdnqcV5FxF0wh5hZr1j3uoEzI2YXaoonUFJqhsG3Bjz+lngLTP7mmCgQVlnN3uykOAHcXOCezk7zew5gh+w08Mzsxzg/D3txN3XmNndwCcEZzYT3P2tvczyAEE362JXAE+b2T0ExXcUwWRxsX4BPGtm2wjm49oU4TjbgG5mNi1c/9Jw+f8juA9Xjz13KX80HBpuBN/3H2Vy921mtsTMDnf3xSXe+8bMrgbGm9lPgK3ADndfEyG3VGHqZi5SxZhZg+JBB2Z2F9DC3W9OcCzM7ALgKHe/p5z1bgE2J+mUN1KJdAYlUvWcHZ651QKWAz9PbJyAu4+zYP6g8uQC/45zHEkBOoMSEZGkpEESIiKSlFSgREQkKalAiYhIUlKBEhGRpKQCJSIiSen/AyYiXLQ5VPEaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1,kn), Accuracy)\n",
    "plt.legend('Accuracy')\n",
    "plt.ylabel('Accuracy for each k number')\n",
    "plt.xlabel('Number of Neighbors (K)')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad6806e",
   "metadata": {},
   "source": [
    "We get the most accuracy from the َAccuracy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "19f23d98",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=7)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_k_neighbor =Accuracy.argmax()\n",
    "\n",
    "neigh = KNeighborsClassifier(max_k_neighbor + 1)\n",
    "neigh.fit(X_train,Y_train)\n",
    "neigh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2670a6b3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = neigh.predict(x_test)\n",
    "yhat[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5c8092",
   "metadata": {},
   "source": [
    "# Evaluation KNN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1473ead",
   "metadata": {},
   "source": [
    "Now we implemented 4 classification model evaluation methods for each model.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "ac92efac",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import jaccard_score\n",
    "from sklearn.metrics import log_loss\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1154172",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "232cfae5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8852459016393442"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_k7=metrics.accuracy_score(y_test, yhat)\n",
    "acc_k7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ca1296c",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "be6ab947",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7407407407407407"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42a5c70d",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "e461a201",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9635316844512425"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test, yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac2be0b6",
   "metadata": {},
   "source": [
    "**f1_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cedca5ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.80      0.85        25\n",
      "           1       0.87      0.94      0.91        36\n",
      "\n",
      "    accuracy                           0.89        61\n",
      "   macro avg       0.89      0.87      0.88        61\n",
      "weighted avg       0.89      0.89      0.88        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print (classification_report(y_test, yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "0f7368f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a776f7ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiIklEQVR4nO3dedyVdZ3/8df7vm9AFBQBMSSVTHMJE5VEMxWzMS1LrMzUsXKcMZvxN+UyZcsvt6lpnEnNNMulwTF1tBT3XMZU3BLByN30F7iBIhoIiMmNn98f1/fGw819n42zXZz308f1OOdcy/f6nPvifPwu16KIwMzMKtPR7ADMzPLIydPMrApOnmZmVXDyNDOrgpOnmVkVnDzNzKrg5Gl1IWmwpBskLZL06zUo53BJt9UytmaRtIekp5sdh9WGfJ5ne5N0GHA8sA2wGJgF/CAi7l3Dco8A/g/wkYjoXtM4W52kALaKiGebHYs1hmuebUzS8cDZwA+BjYHNgJ8BB9ag+M2BP7VD4iyHpK5mx2A1FhGe2nACNgCWAAcXWWcQWXKdm6azgUFp2STgReAEYD4wDzgyLTsVeBtYnvZxFHAK8KuCsscCAXSlz18B/kxW+50NHF4w/96C7T4CPAQsSq8fKVh2F3A6cF8q5zZgZD/frSf+bxbEPxn4JPAn4HXgOwXr7wI8ACxM654LDEzLpqXvsjR930MKyv8W8DJwac+8tM370z52Sp83ARYAk5r9b8NTeZNrnu1rN2AdYGqRdb4L7AqMB3YgSyDfK1j+HrIkPIYsQZ4nacOIOJmsNntlRAyJiIuLBSJpPeAcYP+IGEqWIGf1sd5w4Ka07gjgTOAmSSMKVjsMOBIYBQwETiyy6/eQ/Q3GAN8HLgT+FtgZ2AP4vqQt0rorgOOAkWR/u32AfwSIiD3TOjuk73tlQfnDyWrhRxfuOCL+H1livUzSusB/AVMi4q4i8VoLcfJsXyOABVG8WX04cFpEzI+IV8lqlEcULF+eli+PiJvJal1bVxnPO8A4SYMjYl5EPN7HOp8CnomISyOiOyKuAJ4CPl2wzn9FxJ8iYhlwFVni789ysv7d5cD/kCXGn0TE4rT/x4EPAUTEzIj4fdrvHOAXwF5lfKeTI+KvKZ5VRMSFwDPAg8Bosv9ZWU44ebav14CRJfriNgGeK/j8XJq3soxeyfdNYEilgUTEUrKm7jHAPEk3SdqmjHh6YhpT8PnlCuJ5LSJWpPc9ye2VguXLeraX9AFJN0p6WdIbZDXrkUXKBng1It4qsc6FwDjgpxHx1xLrWgtx8mxfDwBvkfXz9WcuWZOzx2ZpXjWWAusWfH5P4cKIuDUi/oasBvYUWVIpFU9PTC9VGVMlzieLa6uIWB/4DqAS2xQ9lUXSELJ+5IuBU1K3hOWEk2ebiohFZP1850maLGldSQMk7S/pjLTaFcD3JG0kaWRa/1dV7nIWsKekzSRtAHy7Z4GkjSV9JvV9/pWs+b+ijzJuBj4g6TBJXZIOAbYDbqwypkoMBd4AlqRa8dd6LX8F2GK1rYr7CTAzIv6erC/352scpTWMk2cbi4gzyc7x/B7wKvACcCxwbVrlX4EZwCPAo8DDaV41+7oduDKVNZNVE14H2aj9XLIR6L1IgzG9yngNOCCt+xrZSPkBEbGgmpgqdCLZYNRislrxlb2WnwJcImmhpC+UKkzSgcB+ZF0VkB2HnSQdXrOIra58kryZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYF36ygQuoaHBo4tNlhWIEdt92s2SFYL889N4cFCxaUOg+2Ip3rbx7RvdqFWquJZa/eGhH71XLffXHyrJAGDmXQ1iXPRLEGuu/Bc5sdgvWy+8QJNS8zut9i0DZfLLneW3/4aakrv2rCydPM8kGAalqZXSNOnmaWH2qdYRonTzPLCUFHZ7ODWMnJ08zyw812M7MKCTfbzcwqJ9c8zcyq4j5PM7NKyc12M7OK+TxPM7NqCDpaJ2W1TiRmZqV0uOZpZlYZn6pkZlYl93mamVXKl2eamVXHzXYzswrJVxiZmVXHNU8zs0q5z9PMrDputpuZVcjneZqZVaO1bgzSOpGYmZXS0Vl6KkHSOpKmS/qjpMclnZrmD5d0u6Rn0uuGRUOp0VcyM6u/ntOVik2l/RX4WETsAIwH9pO0K3AScEdEbAXckT73y8nTzPJBqdleaiohMkvSxwFpCuBA4JI0/xJgcrFynDzNLD/Kq3mOlDSjYDp69WLUKWkWMB+4PSIeBDaOiHkA6XVUsVA8YGRmuSCgo6Os+t6CiJhQbIWIWAGMlzQMmCppXKXxuOZpZvmgMqcKRMRC4C5gP+AVSaMB0uv8Yts6eZpZTgip9FSyFGmjVONE0mDg48BTwPXAl9NqXwauK1aOm+1mlhtlNttLGQ1cIqmTrAJ5VUTcKOkB4CpJRwHPAwcXK8TJ08xyo5yaZSkR8QiwYx/zXwP2KbccJ08zy4cq+jTrycnTzHJBlNen2ShOnmaWGzXq86wJJ08zyw3XPM3MKuU+TzOz6rjmaWZWISH3eZqZVaV1Kp5OnmaWE3Kz3cysKk6eZmYVcp+nmVm1Wqfi6VvStYNBA7u459ITefDKk5j5m+/yvWM+ucrybxyxD8v+cC4jhq3XpAjb2wsvvMAnPr4347fflp12+CDnnvOTZofUmlKf55rekq5WXPNsA399u5v9jj6Hpcvepqurg9/98nhuu+8Jpj86h/duPIyP7boNz897vdlhtq2uri5+dMaP2XGnnVi8eDEfmbgz+3z8b9h2u+2aHVrLaaU+T9c828TSZW8DMKCrk66uTiICgDNO/Bzf/cm1Kz9b440ePZodd9oJgKFDh7LNNtsyd+5LTY6qNalDJadGcc2zTXR0iPsv/xbv33QjfnHlNB567Dk+tdf2zJ2/kEf/5B9qq3huzhxmzfoDH95lYrNDaUmuefZB0lckbbIG24+VdFiF2/xA0guSlpReO9/eeSfY9Ys/YstPfI8J4zZn3Fab8K2jPsFp59/U7NAsWbJkCYd+4XP8x4/PZv311292OC2nnP7ORibXlkmewFeAqpMnMBaoKHkCNwC7rME+c2fRkmVMm/EMB0z6EJuPGcH0K7/NUzedyphRw3jg8m+x8YihzQ6xLS1fvpxDv/A5Djn0cCYf9Nlmh9OyWil51rXZLul44O/Sx4uAa4EbI2JcWn4iMAR4DJgAXCZpGbAb8CRwJbB32v6wiHhW0pRUxm9SGUsiYgjwI2Db9CzmSyLirFLxRcTvUxlr/mVb2MgNh7B8+QoWLVnGOoMG8LGJW/PjKf/L5vt8e+U6T910KrsffgavLVzaxEjbU0RwzD8cxdbbbMvXjzu+2eG0tEb2aZZSt+QpaWfgSGAi2dlZDwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WZMm3L5PSI0fL/S5HA0cDMGBIuZu1jPeMXJ8LTzuCzo4OOjrE1bc/zG/veazZYVly/333cflllzJu3PZM3Hk8AKf+6w/Zb/9PFt+wDbVSRaeeNc+PAlMjYimApGuAPSos44qC15I1yUIR8TQwvsL99VfWBcAFAB3rjsrdsPRjz8xlt0P/veg623zq5AZFY73t/tGPsmx57v5ZNV4bXdve17ccxqr9rOuUKCP6eN/dU4ayv+TAPndew5qnmTVfdnlm6yTPeg4YTQMmS1pX0nrAQcBvgVGSRkgaxKrN8MVA79GKQwpeH0jv5wA7p/cHAgP62j4ino6I8f1MC2vzFc2skaTSU+kytKmkOyU9KelxSV9P80+R9JKkWWkq2m9St5pnRDycBnemp1kXRcRDkk4j6/+cDTxVsMkU4OcFA0YAgyQ9SJbkD03zLgSukzQduAPoGeF4BOiW9EdgSjkDRpLOIBuhX1fSiynGU6r5vmZWfzVqtncDJ6QcNRSYKen2tOysiPjPcgqp62h7RJwJnNlr3jnAOX2sezVwdc/n9Ec6LyJO7bXeK8CuBbO+neYvp4IH1qdtvgl8s5JtzKxJyqxZlhIR84B56f1iSU8CYyotp5XO8zQz65eAzk6VnICRkmYUTEf3W6Y0FtiRrDUMcKykRyT9UtKGxeJp2cszI2Jss2Mws9ZSZrN9QURMKKOsIWSt3W9ExBuSzgdOJxucPh34Me+ep76alk2eZmarqFGzHUDSALLEeVlEXAMruwR7ll8I3FisDDfbzSwXRG0uz0ynOF4MPJnGZXrmjy5Y7SCyKx/75ZqnmeVEzc7z3B04Ang0Xc4N8B3gUEnjyZrtc4CvFivEydPMcqMWpypFxL30fRHPzZWU4+RpZvlQwz7PWnDyNLNc6OnzbBVOnmaWG610bbuTp5nlRgtVPJ08zSwn2uiWdGZmNZP1eTY7inc5eZpZTrTW/TydPM0sN9xsNzOrlM/zNDOrnICOjta5HYeTp5nlhmueZmZVcJ+nmVml3OdpZla5Vnv0sJOnmeVGRwtVPZ08zSw3Wih3OnmaWT4oL9e2S/op2e3o+xQR/1yXiMzM+tGZkz7PGQ2LwsysDC1U8ew/eUbEJYWfJa0XEUvrH5KZ2epENuLeKkpe6yRpN0lPAE+mzztI+lndIzMz66VDpaeGxVLGOmcDnwBeA4iIPwJ71jEmM7PVKTvPs9TUKGWNtkfEC71GuVbUJxwzs76J1jrPs5ya5wuSPgKEpIGSTiQ14c3MGkkqPZUuQ5tKulPSk5Iel/T1NH+4pNslPZNeNyxWTjnJ8xjgn4AxwEvA+PTZzKyhJJWcytANnBAR2wK7Av8kaTvgJOCOiNgKuCN97lfJZntELAAOLyciM7N6kWpznmdEzAPmpfeLJT1JVjk8EJiUVrsEuAv4Vn/llDPavoWkGyS9Kmm+pOskbbGG8ZuZVUxlTMBISTMKpqP7LU8aC+wIPAhsnBJrT4IdVSyWcgaMLgfOAw5Kn78IXAFMLGNbM7OaKbNZviAiJpRR1hDgauAbEfFGpZd+ltPnqYi4NCK60/Qrily2aWZWD5Lo7Cg9lVnWALLEeVlEXJNmvyJpdFo+GphfrIx+k2caeRoO3CnpJEljJW0u6ZvATWVFaGZWQzUabRdwMfBkRJxZsOh64Mvp/ZeB64qVU6zZPpOshtkTzlcLlgVweukwzcxqp0Z3VdodOAJ4VNKsNO87wI+AqyQdBTwPHFyskGLXtr+vFlGamdVCdpL8mpcTEfdCvxfJ71NuOWVdYSRpHLAdsE5BAP9d7k7MzGqhla4wKpk8JZ1Mdu7TdsDNwP7AvYCTp5k1jNRaybOc0fbPk1VlX46II4EdgEF1jcrMrA+1GDCqlXKa7csi4h1J3ZLWJxu+90nyZtZwuXgMR4EZkoYBF5KNwC8BptczKDOz3kT553E2QjnXtv9jevtzSbcA60fEI/UNy8yslwY3y0sp9gC4nYoti4iH6xNSa9t+6025+Xc/bnYYVuBLv2rLf4otbfbrb9al3Lw024tliAA+VuNYzMyKKmeEu1GKnSS/dyMDMTMrRuTn0cNmZi2lhXKnk6eZ5UN2HmfrZE8nTzPLjVaqeZZzJ3lJ+ltJ30+fN5O0S/1DMzN7V0+fZy3u51kL5Qxe/QzYDTg0fV5Mdmd5M7OG6ihjapRymu0TI2InSX8AiIi/SBpY57jMzFbTQl2eZSXP5ZI6SY/ekLQR8E5dozIz66XnMRytopxa7jnAVGCUpB+Q3Y7uh3WNysysDx0qPTVKOde2XyZpJtlt6QRMjogn6x6ZmVmB7E7yrVPzLOdmyJsBbwI3FM6LiOfrGZiZWW8tlDvL6vO8iXcfBLcO8D7gaeCDdYzLzGxVgs4Wyp7lNNu3L/yc7rb01X5WNzOri1o9AK5WKr7CKCIelvThegRjZlZMrpKnpOMLPnYAOwGv1i0iM7N+tNK17eWcqjS0YBpE1gd6YD2DMjPrTYLOjtJT6XL0S0nzJT1WMO8USS9JmpWmT5Yqp2jNM50cPyQi/qWM72ZmVlc1OlVpCnAuqz8+/ayI+M9yCyn2GI6uiOgu9jgOM7NGqdWAUURMkzR2TcspVvOcTta/OUvS9cCvgaUFAVyzpjs3M6tEmRXPkZJmFHy+ICIuKGO7YyV9CZgBnBARfym2cjmj7cOB18ieWdRzvmcATp5m1jBC5Z7nuSAiJlRY/PnA6WS57XSyZ7j9XbENiiXPUWmk/THeTZo9osLAzMzWTB2vXY+IV1buRroQuLHUNsWSZycwhFWT5sp9VRydmdkaqte17ZJGR8S89PEgskpjUcWS57yIOK0mkZmZrSFRm2vbJV0BTCLrG30ROBmYJGk8WcVwDmVcRVksebbO2ahmZtTm0cMRcWgfsy+utJxiyXOfSgszM6sX0djHbJTSb/KMiNcbGYiZWVF+9LCZWeVEzm5JZ2bWKlondTp5mlmOtFDF08nTzPJC7vM0M6uU+zzNzKrUOqnTydPM8sKnKpmZVS43J8mbmbWaet0YpBpOnmaWGy2UO508zSwfsmZ762RPJ08zyw3XPM3MKib3eZqZVcrNdjOzasjNdjOzqjh5WlPtusMHWG/IUDo7O+nq6uLm393f7JDazoh1B/BPe4xl2OABRAT/+6cF/PbJV1lvYCfHTXofGw0ZyKtL3uasu2az9O0VzQ63JfjadmsJv77+VoaPGNnsMNrWiggufehFZr++jHW6OvjRp7fhkbmLmbTlCB6dt5jrHn2FA7ffmMnbb8xlM+c2O9yWoRbq82ylq53M2sbCZd3Mfn0ZAG91v8NLi95i+LoD+PBmG3D3s68BcPezr/HhzYY1McrWI5WeGsXJsw1J4rDPHcD+e+/Gr6Zc1Oxw2t5GQwbyvuHr8uyCpWwwuIuFy7qBLMGuv44bhz16mu2lpkZpmeQp6SuSNlmD7cdKOqzCbXaW9KikZyWdo1a6ZUsdTf3tndxy1++59KrruOTiX/D7++9pdkhta1BXBydM2oIp019k2fJ3mh1Oi1NZ/5UsRfqlpPmSHiuYN1zS7ZKeSa8bliqnZZIn8BWg6uQJjAUqSp7A+cDRwFZp2m8N9p8b7xmd/ZlHbjSK/T71GWbNnNHkiNpTp+CEvbfgnj+/zvTnFwKwaFk3wwZntc1hg7t4463uJkbYYspospdZ/ZnC6r/1k4A7ImIr4I70uai6Jk9Jx0t6LE3fSLXDwmx/oqRTJH0emABcJmmWpMGS5kj6d0nT07Rl2mZKWr+njCXp7Y+APdL2x5UR22hg/Yh4ICIC+G9gcu2+fWt6c+lSlixevPL9tDvvYOttP9jkqNrTMbtvzkuL3uKmJ+avnDfjhUXsteUIAPbacgQPPb+oWeG1JJUxlRIR04Dej1Y/ELgkvb+EMnJB3TpUJO0MHAlMJPtODwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WwJX9rDsJGAO8WDDvxTSvr+9yNFkNlTHv3bRICK3v1Vdf4e+POASAFd3dTP78Iez98X2bHFX72XrUeuy15Qiee30ZZ3xmGwCumDmXax99meP2eh8f22oEC5a8zZl3zW5ypK2jglOVRkoqbE5dEBEXlNhm44iYBxAR8ySNKrWTevZGfxSYGhFLASRdA+xRYRlXFLyeVcmGEfE0ML6/5f30b0Y/ZV0AXACww44797lOXmw+dgtuv+ehZofR9p6ev5QvTHm4z2Wn3/Zsg6PJkfKa5QsiYkKdI6lr8uzraw5j1a6CdUqUEX287+4pIyXAgX3uvHTN80XgvQXz3gv4hDqzFlbH8zxfkTQ61TpHA/NLbVDPPs9pwGRJ60paDzgI+C0wStIISYNYtRm+GBjaq4xDCl4fSO/nADun9wcCA/raPiKejojx/UwLUxV9saRdUxL+EnBdLb64mdVHHc/zvB74cnr/ZcrIBXWreUbEw5KmANPTrIsi4iFJp5H1f84GnirYZArwc0nLgN3SvEGSHiRL8oemeRcC10maTjYqtjTNfwTolvRHYEpElNPM/1ra72CyxP7bSr+nmTVOLU4mlHQFWetzpKQXgZPJBpyvknQU8DxwcKly6noGbkScCZzZa945wDl9rHs1cHXP59QleV5EnNprvVeAXQtmfTvNXw7sU2F8M4BxlWxjZs2RjaavefaMiEP7WVRR/vDlC2aWD74lXXkiYmyzYzCz1tJCubN1k6eZ2apE32cYNoeTp5nlRgvlTidPM8uHci+/bBQnTzPLjxbKnk6eZpYbfvSwmVkVWid1OnmaWV60WKenk6eZ5UYrPQDOydPMckFAR+vkTidPM8sRJ08zs8q52W5mVgU3283MquHkaWZWmVrdz7NWnDzNLB98P08zs+o4eZqZVUxutpuZVcM1TzOzCrXYpe1OnmaWH34Mh5lZFVoodzp5mll+1Cp3SpoDLAZWAN0RMaHSMpw8zSwfan+e594RsaDajZ08zSwXRGv1eXY0OwAzs3KpjAkYKWlGwXR0H0UFcJukmf0sL8k1TzPLjTIrngvK6MPcPSLmShoF3C7pqYiYVkksrnmaWW6ojP/KERFz0+t8YCqwS6WxOHmaWW5IpafSZWg9SUN73gP7Ao9VGoub7WaWC+UmxzJsDExNg09dwOURcUulhTh5mllu1OLGIBHxZ2CHNS3HydPMcqOFzlRy8jSz/HDyNDOrmO/naWZWsewKo2ZH8S4nTzPLDSdPM7MquNluZlYpPz3TzKxyfgyHmVmVWumWdE6eZpYbLZQ7nTzNLD9aKHc6eZpZjrRQ9nTyNLNcENDRQu12RUSzY8gVSa8CzzU7jhoYCVT98Curm7XluGweERvVskBJt5D9fUpZEBH71XLffcbj5NmeJM2o5nGrVl8+LvnhO8mbmVXBydPMrApOnu3rgmYHYH3ycckJ93mamVXBNU8zsyo4eZqZVcHJ08ysCk6eZjmmVrrNUJtx8rSySfLlvC1C0nBJIym42tuJtLGcPK0oSeMknQ8QEd2SOpsdU7uTtD1wHXAFcKqkQwAiIpxAG8enKlm/JL0XuA14LzA9Ij6e5ndGxIqmBtemJA0CbgUuAe4CdgcOBO6OiHObGFrbcc3TihkDnBMR6wMrJP0OICJWuAnfNN3AHGBaRMwGrgd+BkyS9LfNDKzdOHlavyLiQeCa9P4TwHJJd6bP3ZJGNDO+dpRq/C8DF0kaHBFvAA8B/wN8SNJgN90bw8nT+tTTtxkR83tqmQUJ9AZJuwJnpEELa4CC2v63gceB81ICXQL8HpgAjAj3xTWEk6etpqdPU5nOVMvsSaD7Au8H7gdujIi14d6TLa/gOHSQjbCfASwBrpe0CbAJMAxYp3lRthcnT1tFQeLsAO4DPglZMz0tfz/ZD/TAiJjqJmL99XFM9o+I54GvA48C55H1e/5bRDzbxFDbikfbbaVeP9LrgHvIfpjHAT9Ip8LsQXZWzL09idPNxPrp55j8DPhGRPxrWmcI0BURCyXJx6MxXPM0YLUf6fVkP9JzyX6w7/T8ICPinoi4N70P/1Drp8gxuRZY+XePiCURsTC99/FoECfPNlbY5C44b3MacDfZj/QG4IaI+GHv9a0+KjgmP2hCeFbAzfY2JWkd4KMR8b+StiM7p3M2sCdwJVnt5saI+Ela383BOvMxyRcnzzYlaUPgCOCzZE8k/HREzE41nwuAP0fEv6V1/SNtAB+TfPFVIm0qIv4iaQ7wYeDOdLUKZF05342I+eAfaSP5mOSL+zzblKQPRsT1wCeAxyX9UtKw1M9WOBjhH2mD+Jjki5vtbaan1iLpQWBRROwraQzwHWAQ8GvgYLKazivNjLVd+Jjkk2ue7WcQQERMBAZKuiEiXgJOBd4AzgSu9Y+0oXxMcsg1zzaS7gP5UeCWnv40SfcDCyLiM+nzeyLiZferNYaPSX655tleRgP7AvtK2jzNOxjYT9LU9Hk+uF+tgXxMcsqj7Wuxgr60ccAysjvvvAh8Ly2fCmwI/AdwB0BEvNOkcNuCj8naw832tVTBj3RvYApwJzAROBJ4HfgWMBTYFTgiIu52s7C+fEzWLk6eazFJE4HJZJfz3S9pMnAxsA/Z3cjHAIMjYkazYmw3PiZrDyfPtUjBDYx7biZxA7A92Y91VkS8I+nrwKYRcWLzIm0fPiZrLw8YrSUkDQQ+DWwq6bNkdxs/GHgK+FrBqovILv2zOvMxWbt5wGgtERFvSxoM3Ax0AsdFxJupWXgbcLuk64DPAD9tXqTtw8dk7eaa51qg4DZmNwB/ABYDcyStFxFvAn8DvA0cAnw1Iq5LTUirEx+TtZ8P1logjeBuAxwLHA+cnaYJaZVBwOeBN4H/m7bx6S915GOy9nPyzLmC2so4YCzZD/Iy4CrgO5L+BXiBrE9tMrCxpNGNj7R9+Ji0B4+255ykTSPihfR+MjAJmB0RP5H0KbKR3Ycj4rbmRdlefEzag5NnjknaALgL+O+IOCvN+xzwD8CtwIWRPdN7ZR+cT7iuLx+T9uFme870/OCUPQJ4c7JTXg6VdCRARFxNdieeHYCNeraLpPERr/18TNqTT1XKmTQQMRk4hewHOZ3stJd/kbQu2RMWh5E9Knh2P8VYDfmYtCc323NG0giywYcTIuLxVLsZQ3Zt9BFkN5v4aURMLVKM1ZCPSXtyzTN/usluHtHT/PsVcB6wguy+kEMiYpFvKNFQPiZtyH2eORMRi4CrgT0ljYuI5WSPpB0DdKTlHoRoIB+T9uTkmU9XAgOAsySdRlbLuTn9aK05fEzajPs8c0rSUGA3YEuyu/Pc3+SQ2p6PSXtx8jQzq4Kb7WZmVXDyNDOrgpOnmVkVnDzNzKrg5GlmVgUnT6uYpBWSZkl6TNKv0/Xb1ZY1RdLn0/uLJG1XZN1Jkj5SxT7mSFrtGUH9ze+1zpIK93WKJD/IrQ04eVo1lkXE+IgYR/YoiWMKF/Y8MbJSEfH3EfFEkVUmARUnT7N6cPK0NXUPsGWqFd4p6XLgUUmdkv5D0kOSHpH0Vchu3ybpXElPSLoJGNVTkKS7JE1I7/eT9LCkP0q6Q9JYsiR9XKr17iFpI0lXp308JGn3tO0ISbdJ+oOkXwCiBEnXSpop6XFJR/da9uMUyx2SNkrz3i/plrTNPemRG9ZGfGMQq5qkLmB/4JY0axdgXETMTgloUUR8WNIg4D5JtwE7AluT3U19Y+AJ4Je9yt0IuBDYM5U1PCJel/RzYElE/Gda73LgrIi4V9JmZDcb3hY4Gbg3Ik5Ld25fJRn24+/SPgYDD0m6OiJeA9Yju+v7CZK+n8o+FrgAOCYinpE0EfgZ8LEq/oyWU06eVo3Bkmal9/cAF5M1p6cX3K9yX+BDPf2ZwAbAVsCewBURsQKYK+l3fZS/KzCtp6yIeL2fOD4ObKeVD6pk/XSJ5J7AZ9O2N0n6Sxnf6Z8lHZTeb5pifQ14h+y6dcjulnSNpCHp+/66YN+DytiHrUWcPK0ayyJifOGMlESWFs4C/k9E3NprvU8Cpa4JVhnrQNbttFtELOsjlrKvO5Y0iSwR75aeq34XsE4/q0fa78LefwNrL+7ztHq5FfiapAEAkj4gaT1gGvDF1Cc6Gti7j20fAPaS9L607fA0fzHZfTN73EbWhCatNz69nQYcnubtD2xYItYNgL+kxLkNWc23RwfZ0y8BDiPrDngDmC3p4LQPSdqhxD5sLePkafVyEVl/5sOSHgN+QdbSmQo8AzwKnA/c3XvDiHiVrJ/yGkl/5N1m8w3AQT0DRsA/AxPSgNQTvDvqfyrZvTUfJus+eL5ErLcAXZIeAU4Hfl+wbCnwQUkzyfo0T0vzDweOSvE9DhxYxt/E1iK+q5KZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYFJ08zsyo4eZqZVcHJ08ysCv8f+oA5YmiE2l4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute confusion matrix\n",
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "\n",
    "# Plot non-normalized confusion matrix\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize= False,  title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01fd7559",
   "metadata": {},
   "source": [
    "We will try **GradientBoostingClassifier** as the second model.<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b6335693",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ab1f5d75",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GradientBoostingClassifier()"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradientB =GradientBoostingClassifier(n_estimators=100, max_depth =3)\n",
    "gradientB.fit(X_train, Y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "66beea08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1,\n",
       "       0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1,\n",
       "       1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1], dtype=int64)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat=gradientB.predict(x_test)\n",
    "yhat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "193e0102",
   "metadata": {},
   "source": [
    "# Evaluation GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eda26dbc",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3c604e50",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8852459016393442"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_GBC=metrics.accuracy_score(y_test, yhat)\n",
    "acc_GBC"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d41df8",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "bf05a8e7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7407407407407407"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5356e81",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "e5b5e47a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9635316844512425"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test, yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d42cab",
   "metadata": {},
   "source": [
    "**f1_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "e06ec090",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.80      0.85        25\n",
      "           1       0.87      0.94      0.91        36\n",
      "\n",
      "    accuracy                           0.89        61\n",
      "   macro avg       0.89      0.87      0.88        61\n",
      "weighted avg       0.89      0.89      0.88        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print( classification_report(y_test, yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "559ca885",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "d0b1060e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiIklEQVR4nO3dedyVdZ3/8df7vm9AFBQBMSSVTHMJE5VEMxWzMS1LrMzUsXKcMZvxN+UyZcsvt6lpnEnNNMulwTF1tBT3XMZU3BLByN30F7iBIhoIiMmNn98f1/fGw819n42zXZz308f1OOdcy/f6nPvifPwu16KIwMzMKtPR7ADMzPLIydPMrApOnmZmVXDyNDOrgpOnmVkVnDzNzKrg5Gl1IWmwpBskLZL06zUo53BJt9UytmaRtIekp5sdh9WGfJ5ne5N0GHA8sA2wGJgF/CAi7l3Dco8A/g/wkYjoXtM4W52kALaKiGebHYs1hmuebUzS8cDZwA+BjYHNgJ8BB9ag+M2BP7VD4iyHpK5mx2A1FhGe2nACNgCWAAcXWWcQWXKdm6azgUFp2STgReAEYD4wDzgyLTsVeBtYnvZxFHAK8KuCsscCAXSlz18B/kxW+50NHF4w/96C7T4CPAQsSq8fKVh2F3A6cF8q5zZgZD/frSf+bxbEPxn4JPAn4HXgOwXr7wI8ACxM654LDEzLpqXvsjR930MKyv8W8DJwac+8tM370z52Sp83ARYAk5r9b8NTeZNrnu1rN2AdYGqRdb4L7AqMB3YgSyDfK1j+HrIkPIYsQZ4nacOIOJmsNntlRAyJiIuLBSJpPeAcYP+IGEqWIGf1sd5w4Ka07gjgTOAmSSMKVjsMOBIYBQwETiyy6/eQ/Q3GAN8HLgT+FtgZ2AP4vqQt0rorgOOAkWR/u32AfwSIiD3TOjuk73tlQfnDyWrhRxfuOCL+H1livUzSusB/AVMi4q4i8VoLcfJsXyOABVG8WX04cFpEzI+IV8lqlEcULF+eli+PiJvJal1bVxnPO8A4SYMjYl5EPN7HOp8CnomISyOiOyKuAJ4CPl2wzn9FxJ8iYhlwFVni789ysv7d5cD/kCXGn0TE4rT/x4EPAUTEzIj4fdrvHOAXwF5lfKeTI+KvKZ5VRMSFwDPAg8Bosv9ZWU44ebav14CRJfriNgGeK/j8XJq3soxeyfdNYEilgUTEUrKm7jHAPEk3SdqmjHh6YhpT8PnlCuJ5LSJWpPc9ye2VguXLeraX9AFJN0p6WdIbZDXrkUXKBng1It4qsc6FwDjgpxHx1xLrWgtx8mxfDwBvkfXz9WcuWZOzx2ZpXjWWAusWfH5P4cKIuDUi/oasBvYUWVIpFU9PTC9VGVMlzieLa6uIWB/4DqAS2xQ9lUXSELJ+5IuBU1K3hOWEk2ebiohFZP1850maLGldSQMk7S/pjLTaFcD3JG0kaWRa/1dV7nIWsKekzSRtAHy7Z4GkjSV9JvV9/pWs+b+ijzJuBj4g6TBJXZIOAbYDbqwypkoMBd4AlqRa8dd6LX8F2GK1rYr7CTAzIv6erC/352scpTWMk2cbi4gzyc7x/B7wKvACcCxwbVrlX4EZwCPAo8DDaV41+7oduDKVNZNVE14H2aj9XLIR6L1IgzG9yngNOCCt+xrZSPkBEbGgmpgqdCLZYNRislrxlb2WnwJcImmhpC+UKkzSgcB+ZF0VkB2HnSQdXrOIra58kryZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYF36ygQuoaHBo4tNlhWIEdt92s2SFYL889N4cFCxaUOg+2Ip3rbx7RvdqFWquJZa/eGhH71XLffXHyrJAGDmXQ1iXPRLEGuu/Bc5sdgvWy+8QJNS8zut9i0DZfLLneW3/4aakrv2rCydPM8kGAalqZXSNOnmaWH2qdYRonTzPLCUFHZ7ODWMnJ08zyw812M7MKCTfbzcwqJ9c8zcyq4j5PM7NKyc12M7OK+TxPM7NqCDpaJ2W1TiRmZqV0uOZpZlYZn6pkZlYl93mamVXKl2eamVXHzXYzswrJVxiZmVXHNU8zs0q5z9PMrDputpuZVcjneZqZVaO1bgzSOpGYmZXS0Vl6KkHSOpKmS/qjpMclnZrmD5d0u6Rn0uuGRUOp0VcyM6u/ntOVik2l/RX4WETsAIwH9pO0K3AScEdEbAXckT73y8nTzPJBqdleaiohMkvSxwFpCuBA4JI0/xJgcrFynDzNLD/Kq3mOlDSjYDp69WLUKWkWMB+4PSIeBDaOiHkA6XVUsVA8YGRmuSCgo6Os+t6CiJhQbIWIWAGMlzQMmCppXKXxuOZpZvmgMqcKRMRC4C5gP+AVSaMB0uv8Yts6eZpZTgip9FSyFGmjVONE0mDg48BTwPXAl9NqXwauK1aOm+1mlhtlNttLGQ1cIqmTrAJ5VUTcKOkB4CpJRwHPAwcXK8TJ08xyo5yaZSkR8QiwYx/zXwP2KbccJ08zy4cq+jTrycnTzHJBlNen2ShOnmaWGzXq86wJJ08zyw3XPM3MKuU+TzOz6rjmaWZWISH3eZqZVaV1Kp5OnmaWE3Kz3cysKk6eZmYVcp+nmVm1Wqfi6VvStYNBA7u459ITefDKk5j5m+/yvWM+ucrybxyxD8v+cC4jhq3XpAjb2wsvvMAnPr4347fflp12+CDnnvOTZofUmlKf55rekq5WXPNsA399u5v9jj6Hpcvepqurg9/98nhuu+8Jpj86h/duPIyP7boNz897vdlhtq2uri5+dMaP2XGnnVi8eDEfmbgz+3z8b9h2u+2aHVrLaaU+T9c828TSZW8DMKCrk66uTiICgDNO/Bzf/cm1Kz9b440ePZodd9oJgKFDh7LNNtsyd+5LTY6qNalDJadGcc2zTXR0iPsv/xbv33QjfnHlNB567Dk+tdf2zJ2/kEf/5B9qq3huzhxmzfoDH95lYrNDaUmuefZB0lckbbIG24+VdFiF2/xA0guSlpReO9/eeSfY9Ys/YstPfI8J4zZn3Fab8K2jPsFp59/U7NAsWbJkCYd+4XP8x4/PZv311292OC2nnP7ORibXlkmewFeAqpMnMBaoKHkCNwC7rME+c2fRkmVMm/EMB0z6EJuPGcH0K7/NUzedyphRw3jg8m+x8YihzQ6xLS1fvpxDv/A5Djn0cCYf9Nlmh9OyWil51rXZLul44O/Sx4uAa4EbI2JcWn4iMAR4DJgAXCZpGbAb8CRwJbB32v6wiHhW0pRUxm9SGUsiYgjwI2Db9CzmSyLirFLxRcTvUxlr/mVb2MgNh7B8+QoWLVnGOoMG8LGJW/PjKf/L5vt8e+U6T910KrsffgavLVzaxEjbU0RwzD8cxdbbbMvXjzu+2eG0tEb2aZZSt+QpaWfgSGAi2dlZDwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WZMm3L5PSI0fL/S5HA0cDMGBIuZu1jPeMXJ8LTzuCzo4OOjrE1bc/zG/veazZYVly/333cflllzJu3PZM3Hk8AKf+6w/Zb/9PFt+wDbVSRaeeNc+PAlMjYimApGuAPSos44qC15I1yUIR8TQwvsL99VfWBcAFAB3rjsrdsPRjz8xlt0P/veg623zq5AZFY73t/tGPsmx57v5ZNV4bXdve17ccxqr9rOuUKCP6eN/dU4ayv+TAPndew5qnmTVfdnlm6yTPeg4YTQMmS1pX0nrAQcBvgVGSRkgaxKrN8MVA79GKQwpeH0jv5wA7p/cHAgP62j4ino6I8f1MC2vzFc2skaTSU+kytKmkOyU9KelxSV9P80+R9JKkWWkq2m9St5pnRDycBnemp1kXRcRDkk4j6/+cDTxVsMkU4OcFA0YAgyQ9SJbkD03zLgSukzQduAPoGeF4BOiW9EdgSjkDRpLOIBuhX1fSiynGU6r5vmZWfzVqtncDJ6QcNRSYKen2tOysiPjPcgqp62h7RJwJnNlr3jnAOX2sezVwdc/n9Ec6LyJO7bXeK8CuBbO+neYvp4IH1qdtvgl8s5JtzKxJyqxZlhIR84B56f1iSU8CYyotp5XO8zQz65eAzk6VnICRkmYUTEf3W6Y0FtiRrDUMcKykRyT9UtKGxeJp2cszI2Jss2Mws9ZSZrN9QURMKKOsIWSt3W9ExBuSzgdOJxucPh34Me+ep76alk2eZmarqFGzHUDSALLEeVlEXAMruwR7ll8I3FisDDfbzSwXRG0uz0ynOF4MPJnGZXrmjy5Y7SCyKx/75ZqnmeVEzc7z3B04Ang0Xc4N8B3gUEnjyZrtc4CvFivEydPMcqMWpypFxL30fRHPzZWU4+RpZvlQwz7PWnDyNLNc6OnzbBVOnmaWG610bbuTp5nlRgtVPJ08zSwn2uiWdGZmNZP1eTY7inc5eZpZTrTW/TydPM0sN9xsNzOrlM/zNDOrnICOjta5HYeTp5nlhmueZmZVcJ+nmVml3OdpZla5Vnv0sJOnmeVGRwtVPZ08zSw3Wih3OnmaWT4oL9e2S/op2e3o+xQR/1yXiMzM+tGZkz7PGQ2LwsysDC1U8ew/eUbEJYWfJa0XEUvrH5KZ2epENuLeKkpe6yRpN0lPAE+mzztI+lndIzMz66VDpaeGxVLGOmcDnwBeA4iIPwJ71jEmM7PVKTvPs9TUKGWNtkfEC71GuVbUJxwzs76J1jrPs5ya5wuSPgKEpIGSTiQ14c3MGkkqPZUuQ5tKulPSk5Iel/T1NH+4pNslPZNeNyxWTjnJ8xjgn4AxwEvA+PTZzKyhJJWcytANnBAR2wK7Av8kaTvgJOCOiNgKuCN97lfJZntELAAOLyciM7N6kWpznmdEzAPmpfeLJT1JVjk8EJiUVrsEuAv4Vn/llDPavoWkGyS9Kmm+pOskbbGG8ZuZVUxlTMBISTMKpqP7LU8aC+wIPAhsnBJrT4IdVSyWcgaMLgfOAw5Kn78IXAFMLGNbM7OaKbNZviAiJpRR1hDgauAbEfFGpZd+ltPnqYi4NCK60/Qrily2aWZWD5Lo7Cg9lVnWALLEeVlEXJNmvyJpdFo+GphfrIx+k2caeRoO3CnpJEljJW0u6ZvATWVFaGZWQzUabRdwMfBkRJxZsOh64Mvp/ZeB64qVU6zZPpOshtkTzlcLlgVweukwzcxqp0Z3VdodOAJ4VNKsNO87wI+AqyQdBTwPHFyskGLXtr+vFlGamdVCdpL8mpcTEfdCvxfJ71NuOWVdYSRpHLAdsE5BAP9d7k7MzGqhla4wKpk8JZ1Mdu7TdsDNwP7AvYCTp5k1jNRaybOc0fbPk1VlX46II4EdgEF1jcrMrA+1GDCqlXKa7csi4h1J3ZLWJxu+90nyZtZwuXgMR4EZkoYBF5KNwC8BptczKDOz3kT553E2QjnXtv9jevtzSbcA60fEI/UNy8yslwY3y0sp9gC4nYoti4iH6xNSa9t+6025+Xc/bnYYVuBLv2rLf4otbfbrb9al3Lw024tliAA+VuNYzMyKKmeEu1GKnSS/dyMDMTMrRuTn0cNmZi2lhXKnk6eZ5UN2HmfrZE8nTzPLjVaqeZZzJ3lJ+ltJ30+fN5O0S/1DMzN7V0+fZy3u51kL5Qxe/QzYDTg0fV5Mdmd5M7OG6ihjapRymu0TI2InSX8AiIi/SBpY57jMzFbTQl2eZSXP5ZI6SY/ekLQR8E5dozIz66XnMRytopxa7jnAVGCUpB+Q3Y7uh3WNysysDx0qPTVKOde2XyZpJtlt6QRMjogn6x6ZmVmB7E7yrVPzLOdmyJsBbwI3FM6LiOfrGZiZWW8tlDvL6vO8iXcfBLcO8D7gaeCDdYzLzGxVgs4Wyp7lNNu3L/yc7rb01X5WNzOri1o9AK5WKr7CKCIelvThegRjZlZMrpKnpOMLPnYAOwGv1i0iM7N+tNK17eWcqjS0YBpE1gd6YD2DMjPrTYLOjtJT6XL0S0nzJT1WMO8USS9JmpWmT5Yqp2jNM50cPyQi/qWM72ZmVlc1OlVpCnAuqz8+/ayI+M9yCyn2GI6uiOgu9jgOM7NGqdWAUURMkzR2TcspVvOcTta/OUvS9cCvgaUFAVyzpjs3M6tEmRXPkZJmFHy+ICIuKGO7YyV9CZgBnBARfym2cjmj7cOB18ieWdRzvmcATp5m1jBC5Z7nuSAiJlRY/PnA6WS57XSyZ7j9XbENiiXPUWmk/THeTZo9osLAzMzWTB2vXY+IV1buRroQuLHUNsWSZycwhFWT5sp9VRydmdkaqte17ZJGR8S89PEgskpjUcWS57yIOK0mkZmZrSFRm2vbJV0BTCLrG30ROBmYJGk8WcVwDmVcRVksebbO2ahmZtTm0cMRcWgfsy+utJxiyXOfSgszM6sX0djHbJTSb/KMiNcbGYiZWVF+9LCZWeVEzm5JZ2bWKlondTp5mlmOtFDF08nTzPJC7vM0M6uU+zzNzKrUOqnTydPM8sKnKpmZVS43J8mbmbWaet0YpBpOnmaWGy2UO508zSwfsmZ762RPJ08zyw3XPM3MKib3eZqZVcrNdjOzasjNdjOzqjh5WlPtusMHWG/IUDo7O+nq6uLm393f7JDazoh1B/BPe4xl2OABRAT/+6cF/PbJV1lvYCfHTXofGw0ZyKtL3uasu2az9O0VzQ63JfjadmsJv77+VoaPGNnsMNrWiggufehFZr++jHW6OvjRp7fhkbmLmbTlCB6dt5jrHn2FA7ffmMnbb8xlM+c2O9yWoRbq82ylq53M2sbCZd3Mfn0ZAG91v8NLi95i+LoD+PBmG3D3s68BcPezr/HhzYY1McrWI5WeGsXJsw1J4rDPHcD+e+/Gr6Zc1Oxw2t5GQwbyvuHr8uyCpWwwuIuFy7qBLMGuv44bhz16mu2lpkZpmeQp6SuSNlmD7cdKOqzCbXaW9KikZyWdo1a6ZUsdTf3tndxy1++59KrruOTiX/D7++9pdkhta1BXBydM2oIp019k2fJ3mh1Oi1NZ/5UsRfqlpPmSHiuYN1zS7ZKeSa8bliqnZZIn8BWg6uQJjAUqSp7A+cDRwFZp2m8N9p8b7xmd/ZlHbjSK/T71GWbNnNHkiNpTp+CEvbfgnj+/zvTnFwKwaFk3wwZntc1hg7t4463uJkbYYspospdZ/ZnC6r/1k4A7ImIr4I70uai6Jk9Jx0t6LE3fSLXDwmx/oqRTJH0emABcJmmWpMGS5kj6d0nT07Rl2mZKWr+njCXp7Y+APdL2x5UR22hg/Yh4ICIC+G9gcu2+fWt6c+lSlixevPL9tDvvYOttP9jkqNrTMbtvzkuL3uKmJ+avnDfjhUXsteUIAPbacgQPPb+oWeG1JJUxlRIR04Dej1Y/ELgkvb+EMnJB3TpUJO0MHAlMJPtODwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WwJX9rDsJGAO8WDDvxTSvr+9yNFkNlTHv3bRICK3v1Vdf4e+POASAFd3dTP78Iez98X2bHFX72XrUeuy15Qiee30ZZ3xmGwCumDmXax99meP2eh8f22oEC5a8zZl3zW5ypK2jglOVRkoqbE5dEBEXlNhm44iYBxAR8ySNKrWTevZGfxSYGhFLASRdA+xRYRlXFLyeVcmGEfE0ML6/5f30b0Y/ZV0AXACww44797lOXmw+dgtuv+ehZofR9p6ev5QvTHm4z2Wn3/Zsg6PJkfKa5QsiYkKdI6lr8uzraw5j1a6CdUqUEX287+4pIyXAgX3uvHTN80XgvQXz3gv4hDqzFlbH8zxfkTQ61TpHA/NLbVDPPs9pwGRJ60paDzgI+C0wStIISYNYtRm+GBjaq4xDCl4fSO/nADun9wcCA/raPiKejojx/UwLUxV9saRdUxL+EnBdLb64mdVHHc/zvB74cnr/ZcrIBXWreUbEw5KmANPTrIsi4iFJp5H1f84GnirYZArwc0nLgN3SvEGSHiRL8oemeRcC10maTjYqtjTNfwTolvRHYEpElNPM/1ra72CyxP7bSr+nmTVOLU4mlHQFWetzpKQXgZPJBpyvknQU8DxwcKly6noGbkScCZzZa945wDl9rHs1cHXP59QleV5EnNprvVeAXQtmfTvNXw7sU2F8M4BxlWxjZs2RjaavefaMiEP7WVRR/vDlC2aWD74lXXkiYmyzYzCz1tJCubN1k6eZ2apE32cYNoeTp5nlRgvlTidPM8uHci+/bBQnTzPLjxbKnk6eZpYbfvSwmVkVWid1OnmaWV60WKenk6eZ5UYrPQDOydPMckFAR+vkTidPM8sRJ08zs8q52W5mVgU3283MquHkaWZWmVrdz7NWnDzNLB98P08zs+o4eZqZVUxutpuZVcM1TzOzCrXYpe1OnmaWH34Mh5lZFVoodzp5mll+1Cp3SpoDLAZWAN0RMaHSMpw8zSwfan+e594RsaDajZ08zSwXRGv1eXY0OwAzs3KpjAkYKWlGwXR0H0UFcJukmf0sL8k1TzPLjTIrngvK6MPcPSLmShoF3C7pqYiYVkksrnmaWW6ojP/KERFz0+t8YCqwS6WxOHmaWW5IpafSZWg9SUN73gP7Ao9VGoub7WaWC+UmxzJsDExNg09dwOURcUulhTh5mllu1OLGIBHxZ2CHNS3HydPMcqOFzlRy8jSz/HDyNDOrmO/naWZWsewKo2ZH8S4nTzPLDSdPM7MquNluZlYpPz3TzKxyfgyHmVmVWumWdE6eZpYbLZQ7nTzNLD9aKHc6eZpZjrRQ9nTyNLNcENDRQu12RUSzY8gVSa8CzzU7jhoYCVT98Curm7XluGweERvVskBJt5D9fUpZEBH71XLffcbj5NmeJM2o5nGrVl8+LvnhO8mbmVXBydPMrApOnu3rgmYHYH3ycckJ93mamVXBNU8zsyo4eZqZVcHJ08ysCk6eZjmmVrrNUJtx8rSySfLlvC1C0nBJIym42tuJtLGcPK0oSeMknQ8QEd2SOpsdU7uTtD1wHXAFcKqkQwAiIpxAG8enKlm/JL0XuA14LzA9Ij6e5ndGxIqmBtemJA0CbgUuAe4CdgcOBO6OiHObGFrbcc3TihkDnBMR6wMrJP0OICJWuAnfNN3AHGBaRMwGrgd+BkyS9LfNDKzdOHlavyLiQeCa9P4TwHJJd6bP3ZJGNDO+dpRq/C8DF0kaHBFvAA8B/wN8SNJgN90bw8nT+tTTtxkR83tqmQUJ9AZJuwJnpEELa4CC2v63gceB81ICXQL8HpgAjAj3xTWEk6etpqdPU5nOVMvsSaD7Au8H7gdujIi14d6TLa/gOHSQjbCfASwBrpe0CbAJMAxYp3lRthcnT1tFQeLsAO4DPglZMz0tfz/ZD/TAiJjqJmL99XFM9o+I54GvA48C55H1e/5bRDzbxFDbikfbbaVeP9LrgHvIfpjHAT9Ip8LsQXZWzL09idPNxPrp55j8DPhGRPxrWmcI0BURCyXJx6MxXPM0YLUf6fVkP9JzyX6w7/T8ICPinoi4N70P/1Drp8gxuRZY+XePiCURsTC99/FoECfPNlbY5C44b3MacDfZj/QG4IaI+GHv9a0+KjgmP2hCeFbAzfY2JWkd4KMR8b+StiM7p3M2sCdwJVnt5saI+Ela383BOvMxyRcnzzYlaUPgCOCzZE8k/HREzE41nwuAP0fEv6V1/SNtAB+TfPFVIm0qIv4iaQ7wYeDOdLUKZF05342I+eAfaSP5mOSL+zzblKQPRsT1wCeAxyX9UtKw1M9WOBjhH2mD+Jjki5vtbaan1iLpQWBRROwraQzwHWAQ8GvgYLKazivNjLVd+Jjkk2ue7WcQQERMBAZKuiEiXgJOBd4AzgSu9Y+0oXxMcsg1zzaS7gP5UeCWnv40SfcDCyLiM+nzeyLiZferNYaPSX655tleRgP7AvtK2jzNOxjYT9LU9Hk+uF+tgXxMcsqj7Wuxgr60ccAysjvvvAh8Ly2fCmwI/AdwB0BEvNOkcNuCj8naw832tVTBj3RvYApwJzAROBJ4HfgWMBTYFTgiIu52s7C+fEzWLk6eazFJE4HJZJfz3S9pMnAxsA/Z3cjHAIMjYkazYmw3PiZrDyfPtUjBDYx7biZxA7A92Y91VkS8I+nrwKYRcWLzIm0fPiZrLw8YrSUkDQQ+DWwq6bNkdxs/GHgK+FrBqovILv2zOvMxWbt5wGgtERFvSxoM3Ax0AsdFxJupWXgbcLuk64DPAD9tXqTtw8dk7eaa51qg4DZmNwB/ABYDcyStFxFvAn8DvA0cAnw1Iq5LTUirEx+TtZ8P1logjeBuAxwLHA+cnaYJaZVBwOeBN4H/m7bx6S915GOy9nPyzLmC2so4YCzZD/Iy4CrgO5L+BXiBrE9tMrCxpNGNj7R9+Ji0B4+255ykTSPihfR+MjAJmB0RP5H0KbKR3Ycj4rbmRdlefEzag5NnjknaALgL+O+IOCvN+xzwD8CtwIWRPdN7ZR+cT7iuLx+T9uFme870/OCUPQJ4c7JTXg6VdCRARFxNdieeHYCNeraLpPERr/18TNqTT1XKmTQQMRk4hewHOZ3stJd/kbQu2RMWh5E9Knh2P8VYDfmYtCc323NG0giywYcTIuLxVLsZQ3Zt9BFkN5v4aURMLVKM1ZCPSXtyzTN/usluHtHT/PsVcB6wguy+kEMiYpFvKNFQPiZtyH2eORMRi4CrgT0ljYuI5WSPpB0DdKTlHoRoIB+T9uTkmU9XAgOAsySdRlbLuTn9aK05fEzajPs8c0rSUGA3YEuyu/Pc3+SQ2p6PSXtx8jQzq4Kb7WZmVXDyNDOrgpOnmVkVnDzNzKrg5GlmVgUnT6uYpBWSZkl6TNKv0/Xb1ZY1RdLn0/uLJG1XZN1Jkj5SxT7mSFrtGUH9ze+1zpIK93WKJD/IrQ04eVo1lkXE+IgYR/YoiWMKF/Y8MbJSEfH3EfFEkVUmARUnT7N6cPK0NXUPsGWqFd4p6XLgUUmdkv5D0kOSHpH0Vchu3ybpXElPSLoJGNVTkKS7JE1I7/eT9LCkP0q6Q9JYsiR9XKr17iFpI0lXp308JGn3tO0ISbdJ+oOkXwCiBEnXSpop6XFJR/da9uMUyx2SNkrz3i/plrTNPemRG9ZGfGMQq5qkLmB/4JY0axdgXETMTgloUUR8WNIg4D5JtwE7AluT3U19Y+AJ4Je9yt0IuBDYM5U1PCJel/RzYElE/Gda73LgrIi4V9JmZDcb3hY4Gbg3Ik5Ld25fJRn24+/SPgYDD0m6OiJeA9Yju+v7CZK+n8o+FrgAOCYinpE0EfgZ8LEq/oyWU06eVo3Bkmal9/cAF5M1p6cX3K9yX+BDPf2ZwAbAVsCewBURsQKYK+l3fZS/KzCtp6yIeL2fOD4ObKeVD6pk/XSJ5J7AZ9O2N0n6Sxnf6Z8lHZTeb5pifQ14h+y6dcjulnSNpCHp+/66YN+DytiHrUWcPK0ayyJifOGMlESWFs4C/k9E3NprvU8Cpa4JVhnrQNbttFtELOsjlrKvO5Y0iSwR75aeq34XsE4/q0fa78LefwNrL+7ztHq5FfiapAEAkj4gaT1gGvDF1Cc6Gti7j20fAPaS9L607fA0fzHZfTN73EbWhCatNz69nQYcnubtD2xYItYNgL+kxLkNWc23RwfZ0y8BDiPrDngDmC3p4LQPSdqhxD5sLePkafVyEVl/5sOSHgN+QdbSmQo8AzwKnA/c3XvDiHiVrJ/yGkl/5N1m8w3AQT0DRsA/AxPSgNQTvDvqfyrZvTUfJus+eL5ErLcAXZIeAU4Hfl+wbCnwQUkzyfo0T0vzDweOSvE9DhxYxt/E1iK+q5KZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYFJ08zsyo4eZqZVcHJ08ysCv8f+oA5YmiE2l4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Compute confusion matrix\n",
    "cnf_matrix =confusion_matrix(y_test, yhat, labels =[1,0])\n",
    "np.set_printoptions(precision =2)\n",
    "\n",
    "\n",
    "# Plot non-normalized confusion matrix\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize =False,  title ='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cacc7ae5",
   "metadata": {},
   "source": [
    "We will try **LogisticRegression** as the third model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "ba5a87c8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.01)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "logisticR =LogisticRegression(C=0.01)\n",
    "logisticR.fit(X_train,Y_train)\n",
    "logisticR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "40de05d6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1,\n",
       "       0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1,\n",
       "       1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1], dtype=int64)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat =logisticR.predict(x_test)\n",
    "yhat"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcd5cf93",
   "metadata": {},
   "source": [
    "# Evaluation  LogisticRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2433cb8e",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5cfa951d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8852459016393442"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_lr=metrics.accuracy_score(y_test,yhat)\n",
    "acc_lr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72d791bb",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e653c6a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7307692307692307"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "292e1842",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2d8d37db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9635447926058363"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test, yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1a5bf4e",
   "metadata": {},
   "source": [
    "**f1-score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "388944c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.76      0.84        25\n",
      "           1       0.85      0.97      0.91        36\n",
      "\n",
      "    accuracy                           0.89        61\n",
      "   macro avg       0.90      0.87      0.88        61\n",
      "weighted avg       0.89      0.89      0.88        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "21a93131",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[35  1]\n",
      " [ 6 19]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "c0c75db9",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[35  1]\n",
      " [ 6 19]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAieklEQVR4nO3debwcVZn/8c/3JmQjYQkhEMISBYTBaAJENgURHQQGJfiTHwKyO6AOigLOICqrOoyjgCyCIAyoiIAQWWUR2YJASGKAhNUxYQ0JYctCFJI880edhs7l3t7SW6W/b1716u5aTj19i35yzqlTVYoIzMysOl2tDsDMLI+cPM3MauDkaWZWAydPM7MaOHmamdXAydPMrAZOntYQkgZKukHSG5KuXoFyDpB0Wz1jaxVJO0p6stVxWH3I4zw7m6T9gWOAzYEFwDTgBxExcQXLPRD4GrBDRCxZ0TjbnaQANo2Iv7Y6FmsO1zw7mKRjgLOAHwLrABsCPwP2qkPxGwFPdULirISkvq2OweosIjx14ASsDiwE9imxTn+y5Ppims4C+qdlOwPPA8cCc4HZwKFp2SnAW8DbaR+HAycDvy4qexQQQN/0+RDgb2S135nAAUXzJxZttwPwEPBGet2haNldwGnAfamc24BhvXy3Qvz/XhT/eGAP4CngVeCEovW3Ae4HXk/rngv0S8vuSd9lUfq++xaV/x/AS8CvCvPSNhunfWyVPq8HzAN2bvX/G54qm1zz7FzbAwOACSXW+Q6wHTAWGEOWQL5btHxdsiQ8kixBnidpzYg4iaw2e2VEDI6Ii0sFImlV4Gxg94gYQpYgp/Ww3lDgprTuWsAZwE2S1ipabX/gUGA40A84rsSu1yX7G4wETgQuAr4IbA3sCJwo6f1p3aXAN4FhZH+7TwJfBYiIndI6Y9L3vbKo/KFktfAjinccEf9LllgvlzQI+B/g0oi4q0S81kacPDvXWsC8KN2sPgA4NSLmRsTLZDXKA4uWv52Wvx0RN5PVujarMZ5lwGhJAyNidkTM6GGdfwGejohfRcSSiLgCeAL4TNE6/xMRT0XEYuAqssTfm7fJ+nffBn5Llhh/GhEL0v5nAB8GiIgpEfFA2u8s4OfAxyv4TidFxD9SPMuJiIuAp4EHgRFk/1hZTjh5dq5XgGFl+uLWA54p+vxMmvdOGd2S75vA4GoDiYhFZE3dLwOzJd0kafMK4inENLLo80tVxPNKRCxN7wvJbU7R8sWF7SV9QNKNkl6SNJ+sZj2sRNkAL0fE38uscxEwGjgnIv5RZl1rI06enet+4O9k/Xy9eZGsyVmwYZpXi0XAoKLP6xYvjIhbI+KfyWpgT5AllXLxFGJ6ocaYqnE+WVybRsRqwAmAymxTciiLpMFk/cgXAyenbgnLCSfPDhURb5D1850nabykQZJWkbS7pB+l1a4AvitpbUnD0vq/rnGX04CdJG0oaXXg24UFktaR9NnU9/kPsub/0h7KuBn4gKT9JfWVtC+wBXBjjTFVYwgwH1iYasVf6bZ8DvD+92xV2k+BKRHxJbK+3AtWOEprGifPDhYRZ5CN8fwu8DLwHHAU8Pu0yveBycAjwKPA1DSvln3dDlyZyprC8gmvi+ys/YtkZ6A/TjoZ062MV4A907qvkJ0p3zMi5tUSU5WOIzsZtYCsVnxlt+UnA5dJel3S/y9XmKS9gN3IuiogOw5bSTqgbhFbQ3mQvJlZDVzzNDOrgZOnmXUUSQMkTZL0sKQZkk5J80+W9IKkaWnao2Q5brabWSeRJGDViFgoaRVgInA0WR/0woj4cSXl+HpbM+sokdUYF6aPq6Sp6lqkk2eV1HdgqN+QVodhRbb8pw1bHYJ188wzs5g3b165cbBV6bPaRhFL3nOh1nvE4pdnkI1hLrgwIi4sXkdSH7JRH5sA50XEg5J2B46SdBDZKJNjI+K13vbjZnuVugYNj/6blR2JYk302kPntjoE6+aj245jypTJdU2eXYPWif6b71t2vb//5ZwpETGukjIlrUF2f4evkQ3Xm0dWCz0NGBERh/UaTyU7MDNrOQFS+akKEfE62d24douIORGxNCKWkY3l3abUtk6eZpYf6io/lSsiu2JujfR+IPAp4AlJI4pW2xuYXqoc93maWU4IuvrUo6ARZFeD9SGrQF4VETdK+pWksWTN9lnAkaUKcfI0s/yoslnek4h4BNiyh/kH9rB6r5w8zSwfREXN8mZx8jSznKj+hFAjOXmaWX7Up8+zLpw8zSwn5Ga7mVnVCuM824STp5nlhKCrfVJW+0RiZlZOl2ueZmbV8VAlM7Mauc/TzKxadbs8sy6cPM0sP9xsNzOrUg23nGskJ08zyw/XPM3MquU+TzOz2rjZbmZWJY/zNDOrhW8MYmZWG/d5mpnVwH2eZmZVkpvtZma1aaOaZ/ukcTOzEgR0dXWVncqWIw2QNEnSw5JmSDolzR8q6XZJT6fXNUuV4+RpZvmgCqfy/gHsEhFjgLHAbpK2A44H7oiITYE70udeOXmaWU4IqfxUTmQWpo+rpCmAvYDL0vzLgPGlynHyNLPcqLDZPkzS5KLpiO7lSOojaRowF7g9Ih4E1omI2QDpdXipWHzCyMxyo5KaJTAvIsaVWiEilgJjJa0BTJA0utpYXPM0s3yoX5/nOyLideAuYDdgjqQRAOl1bqltnTzNLBdUpz5PSWunGieSBgKfAp4ArgcOTqsdDFxXqhw3280sNyoZilSBEcBlkvqQVSCviogbJd0PXCXpcOBZYJ9ShTh5mlluVNjnWVJEPAJs2cP8V4BPVlqOk6eZ5UMNfZqN5ORpZrlRj5pnvTh5mlkuCNWrz7MunDzNLD/ap+Lp5GlmOSE3283MauLkaWZWJfd5mpnVqn0qnk6enaB/v7788eJv0K9fX/r26cOEP/6F719wM985cg8O+9wOvPxadneuk869nlsnPtbiaDvTkV86jD/cfCNrDx/OlGnTWx1Oe3KfpzXbP95awm5HnM2ixW/Rt28Xf7rkGG67L0uS5/z6Ts761R0tjtAOPPgQvvzVo/jSYQe1OpS21k7Js306EKyhFi1+C4BV+vahb98+RESLI7JiH9txJ4YOHdrqMNqeulR2ahYnzw7R1SUe+O3xPHvH6fzpgSd4aPozAHz5Czsx6cpvc8FJB7DGkIEtjtKstHrcVale2iZ5SjpE0norsP0oSftXuc0PJD0naWH5tfNt2bJguy+cziaf/i7jRm/EFhuP4KKr72WLz5zMtl84nZfmzef0Yz7X6jDNelVJ4uzI5AkcAtScPIFRQFXJE7gB2GYF9pk7byxczD2Tn2bXHbZg7qsLWLYsiAguufY+xo3eqNXhmZXUMclT0jGSpqfpG6l2OL1o+XGSTpb0eWAccLmkaZIGSpol6b/SI0InSdokbXNpWr9QRqHWeDqwY9r+m5XEFxEPFJ5ZsjIbtuZgVh+cNckH9F+FXbbdjCdnzWHdYau9s85eu4zhsf9d6f8UlnPt1OfZsLPtkrYGDgW2JRud9SBwd0/rRsTvJB0FHBcRk9P2APMjYhtJBwFnAXuW2OXxafs90/abAVf2su7O6fb7lX6XI4DsIVKrDK50s7ax7rDVuOjUA+nT1UVXl7jm9qn84d7pXHzaQXx4s/WJCJ6Z/Spf+/4VrQ61Yx30xf249+67mDdvHhuPWp/vnXgKhxx2eKvDajvtdLa9kUOVPgZMiIhFAJKuBXassowril7PrGbDiHiS7JnMKywiLgQuBOgaNDx3p6mnP/0i2+/3X++Zf/j3ftmCaKwnv/y1/+Eqq4PGefb0Lddg+a6CAWXKiB7eLymUoewv2a/Hndex5mlmrZddntk+ybORfZ73AOMlDZK0KrA38AdguKS1JPVn+Wb4AmBItzL2LXq9P72fBWyd3u9F9sD692wfEU9GxNheptfr8xXNrJmk8lOzNCx5RsRU4FJgEll/5y8i4iHg1PT5RrIn1hVcClxQOGGU5vWX9CBwNFA4CXQR8HFJk8j6Uxel+Y8ASyQ9XOkJI0k/kvQ8MEjS85JOrunLmllTtNPZ9oZenhkRZwBndJt3NnB2D+teA1xT+Jz+COdFxCnd1psDbFc069tp/ttU8fCmtM2/A/9ezTZm1iJNrlmW007jPM3MeiWgTx+VncqWI20g6U5Jj0uaIenoNP9kSS+k1u80SXuUKqdtbwwSEaNaHYOZtZc6NcuXAMdGxFRJQ4Apkm5Py86MiB9XUkjbJk8zs+XUqdmeLoyZnd4vkPQ4MLLactxsN7NcEBWfMBomaXLRdESvZUqjgC3JTmIDHCXpEUmXSFqzVDyueZpZTlQ8znNeRIwrW5o0mOwk9TciYr6k84HTyMaUnwb8BDist+2dPM0sN+o1FEnSKmSJ8/KIuBbeGclTWH4R2XDKXrnZbmb5UMEA+Upya7oy8WLg8TScsjB/RNFqewMln4fimqeZ5UKhz7MOPgocCDwqaVqadwKwn6SxZM32WcCRpQpx8jSz3KjHte0RMZGe771xczXlOHmaWW600xVGTp5mlg8ddEs6M7O6yfo8Wx3Fu5w8zSwn2ut+nk6eZpYbbrabmVWrzW5J5+RpZrkgoKurfa7rcfI0s9xwzdPMrAbu8zQzq5b7PM3Mqtdujx528jSz3Ohqo6qnk6eZ5UYb5U4nTzPLB+Xl2nZJ55Dd165HEfH1hkRkZtaLPjnp85zctCjMzCrQRhXP3pNnRFxW/FnSqhGxqPEhmZm9l8jOuLeLstc6Sdpe0mPA4+nzGEk/a3hkZmbddKn81LRYKljnLODTwCsAEfEwsFMDYzIzey9l4zzLTc1S0dn2iHiu21mupY0Jx8ysZyJ/4zyfk7QDEJL6AV8nNeHNzJqpjXJnRc32LwP/BowEXgDGps9mZk0lqexUQRkbSLpT0uOSZkg6Os0fKul2SU+n1zVLlVO25hkR84ADKv1yZmaNINVtnOcS4NiImCppCDBF0u3AIcAdEXG6pOOB44H/6K2QSs62v1/SDZJeljRX0nWS3l+Pb2BmVg1VMJUTEbMjYmp6v4CsG3IksBdQGKJ5GTC+VDmVNNt/A1wFjADWA64GrqhgOzOzuqqw2T5M0uSi6YgS5Y0CtgQeBNaJiNmQJVhgeKlYKjlhpIj4VdHnX0s6qoLtzMzqRlKlzfZ5ETGugvIGA9cA34iI+dVeN1/q2vah6e2dqf3/W7Jr3fcFbqpqL2ZmdVCvs+2SViFLnJdHxLVp9hxJIyJitqQRwNxSZZSqeU4hS5aFcI8sWhbAabWFbWZWm3rcVUlZIRcDj0fEGUWLrgcOBk5Pr9eVKqfUte3vW+EozczqJBskX5eiPgocCDwqaVqadwJZ0rxK0uHAs8A+pQqp6AojSaOBLYABhXkR8cvqYzYzq109rjCKiIn0fmL+k5WWUzZ5SjoJ2Jksed4M7A5MBJw8zaxppPa6PLOSoUqfJ8vGL0XEocAYoH9DozIz64FUfmqWSprtiyNimaQlklYjOwPlQfJm1nS5eAxHkcmS1gAuIjsDvxCY1MigzMy6ExWP82yKSq5t/2p6e4GkW4DVIuKRxoZlZtZNk5vl5ZQaJL9VqWWFa0M7zQc3XZ9rb/lRq8OwIl/93aOtDsG6eea1xQ0pNy/N9p+UWBbALnWOxcyspErOcDdLqUHyn2hmIGZmpYj8PHrYzKyttFHudPI0s3zIxnG2T/Z08jSz3Ginmmcld5KXpC9KOjF93lDSNo0PzczsXYU+z3JTs1Ry8upnwPbAfunzAuC8hkVkZtaLrgqmZqmk2b5tRGwl6S8AEfFaegSxmVlTtVGXZ0XJ821JfcjGdiJpbWBZQ6MyM+umisdwNEUltdyzgQnAcEk/ILsd3Q8bGpWZWQ+6VH5qlkqubb9c0hSy29IJGB8Rjzc8MjOzItmd5Nun5lnJzZA3BN4EbiieFxHPNjIwM7Pu2ih3VtTneRPvPghuAPA+4Enggw2My8xseYI+bZQ9K2m2f6j4c7rb0pG9rG5m1hB1fABcXVQ9LCrdiu4jDYjFzKykepwwknSJpLmSphfNO1nSC5KmpWmPcuVU0ud5THHswFbAy+VDNDOrrzpd234pcC7vfYjlmRHx40oLqaTPc0jR+yVkfaDXVLoDM7N6kKBPHS4hioh7JI1a0XJKJs80OH5wRHxrRXdkZraiGjxU6ShJBwGTgWMj4rWSsfS2QFLfiFhK1kw3M2upwgmjCvo8h0maXDQdUUHx5wMbA2OB2ZR+kgZQuuY5iSxxTpN0PXA1sKiwMCKurSAgM7O6qbDiOS8ixlVTbkTMeXcfugi4sdw2lfR5DgVeIXtmUWG8ZwBOnmbWNEING+cpaUREzE4f9waml1ofSifP4elM+3TeTZoFUXOUZma1qNO165KuAHYma94/D5wE7CxpLFlum0UFY9lLJc8+wGCWT5oFTp5m1nT1OGEUEfv1MPviassplTxnR8Sp1RZoZtYIIj/XtrdRmGZm+Xn08CebFoWZWRmiuY/ZKKfX5BkRrzYzEDOzkvzoYTOz6omc3ZLOzKxdtE/qdPI0sxxpo4qnk6eZ5YXc52lmVi33eZqZ1ah9UqeTp5nlhYcqmZlVLzeD5M3M2k2D7yRfFSdPM8uNNsqdTp5mlg9Zs719sqeTp5nlhmueZmZVk/s8zcyq5Wa7mVkt5Ga7mVlNnDytpea/8TrfOebfeOrJx5DEf555PluO27bVYXWcQ7cZyZj1VmP+35dw4i1PA7DBGgM4cNxIBvTtYt6it7jw/uf4+5JlLY60PbTbte3tNGDfmuT73/0WO+7yz9w68S9cf8cDbLzpZq0OqSPdN/M1zrh75nLzDvnISH738EuceMvTTH1+Prv/09otiq49qYL/ypYhXSJprqTpRfOGSrpd0tPpdc1y5Th5dpiFC+Yz+YH72Gf/gwHo168fq62+RmuD6lBPvfwmi95auty8dVfrz1MvLwJgxpyFbL3+aq0IrW1J5acKXArs1m3e8cAdEbEpcEf6XJKTZ4d59pmZrLnWMI4/+kj2+tT2nHDMV3lz0aJWh2XJC2/8nbEjhwDwkQ1WZ+igVVocUfsoNNvLTeVExD1A92e07QVclt5fBowvV07bJE9Jh0habwW2HyVp/yq32VrSo5L+KulstdMtWxpk6ZKlPPboNPY/5F+57o/3M2jQIC489yetDsuSSx58gV02XYsTd92EAX27WLIsWh1SG6mk0S6AYZImF01HVFD4OhExGyC9Di+3QTudMDoEmA68WOP2o4D9gd9Usc35wBHAA8DNZFX5P9S4/1xYd731WHfESMZs9REAPr3n3lx4jpNnu3hpwT84465ZAKwzpB8fXm9IawNqJ5U3y+dFxLgGR9PYmqekYyRNT9M3Uu2wuJP2OEknS/o8MA64XNI0SQMlzZL0X5ImpWmTtM2laf1CGQvT29OBHdP236wgthHAahFxf0QE8EsqqKrn3drD12Xdkevzt78+BcD9997FJh/YvLVB2TuG9O8DZE3Uz2wxnLv+6ieAF1MFU43mpJxQyA1zy23QsJqnpK2BQ4Ftyb7Tg8DdPa0bEb+TdBRwXERMTtsDzI+IbSQdBJwF7Flil8en7fdM228GXNnLujsDI4Hni+Y9n+b19F2OIKuhst76G5QIIR++94Mfc9xXD+Ptt99i/Y3ex+lnXdDqkDrSkdtvwGbDV2Vw/778+LObc930OfTv28Uum6wFwNTn32DizNdaHGX7aPBQpeuBg8kqYQcD15XboJHN9o8BEyJiEYCka4EdqyzjiqLXM6vZMCKeBMb2tryX/s0eO5gi4kLgQoAPjdkq951QW4wew7W3TWx1GB3v5/c/1+P8Pz71SpMjyZE65E5JV5BVoIZJeh44iSxpXiXpcOBZYJ9y5TQyefb0Nddg+a6CAWXKiB7eLymUkRJgvx53Xr7m+TywftG89am9v9XMmqCScZzlRMR+vSz6ZDXlNLLP8x5gvKRBklYF9iY7GTNc0lqS+rN8M3wB0L13fN+i1/vT+1nA1un9XkBhLMdy20fEkxExtpfp9XRGbYGk7VISPogKqupm1jp1GudZFw2reUbEVEmXApPSrF9ExEOSTiXr/5wJPFG0yaXABZIWA9unef0lPUiW5Av/WlwEXCdpEtlg1sIgxUeAJZIeBi6NiEqa+V9J+x1IlthX6jPtZnnXToMJGzpUKSLOAM7oNu9s4Owe1r0GuKbwOXVJnhcRp3Rbbw6wXdGsb6f5b1NltTudnBpdzTZm1hrZ2fT2yZ7tNM7TzKx3viVdZSJiVKtjMLP20ka5s32Tp5nZ8kTPIwxbw8nTzHKjjXKnk6eZ5cMKXn5Zd06eZpYfbZQ9nTzNLDf86GEzsxq0T+p08jSzvGizTk8nTzPLDV9hZGZWJQFd7ZM7nTzNLEecPM3Mqudmu5lZDdxsNzOrhZOnmVl1fD9PM7Na+H6eZma1cfI0M6ua6tZslzSL7KGRS4ElETGu2jKcPM0sN+pc8/xERMyrdWMnTzPLhTa7tL2hz203M6srSWUnYJikyUXTET0UFcBtkqb0srws1zzNLDcqbLbPq6AP86MR8aKk4cDtkp6IiHuqicU1TzPLDVUwVSIiXkyvc4EJwDbVxuLkaWb5kMZ5lpvKFiOtKmlI4T2wKzC92nDcbDezXBDU69HD6wATUll9gd9ExC3VFuLkaWa5UY/UGRF/A8asaDlOnmaWG77CyMysBr4xiJlZDVzzNDOrUqVn05vFydPMcsPNdjOzGrjmaWZWAydPM7Oq1e9+nvXg5GlmuZBdYdTqKN7l5GlmueHkaWZWAzfbzcyq5XGeZmbVa7fHcDh5mllu1OmWdHXh5GlmudFGudPJ08zyo41yp5OnmeVIG2VPJ08zywUBXW3UbldEtDqGXJH0MvBMq+Oog2HAvFYHYe+xshyXjSJi7XoWKOkWsr9POfMiYrd67rvHeJw8O5OkyRU829qazMclP/zoYTOzGjh5mpnVwMmzc13Y6gCsRz4uOeE+TzOzGrjmaWZWAydPM7MaOHmamdXAydMsx9ROtxnqME6eVjFJvpy3TUgaKmkYRVd7O5E2l5OnlSRptKTzASJiiaQ+rY6p00n6EHAdcAVwiqR9ASIinECbx0OVrFeS1gduA9YHJkXEp9L8PhGxtKXBdShJ/YFbgcuAu4CPAnsBd0fEuS0MreO45mmljATOjojVgKWS/gQQEUvdhG+ZJcAs4J6ImAlcD/wM2FnSF1sZWKdx8rReRcSDwLXp/aeBtyXdmT4vkbRWK+PrRKnG/xLwC0kDI2I+8BDwW+DDkga66d4cTp7Wo0LfZkTMLdQyixLoDZK2A36UTlpYExTV9r8NzADOSwl0IfAAMA5YK9wX1xROnvYehT5NZfqkWmYhge4KbAz8GbgxIlaGe0+2vaLj0EV2hv1HwELgeknrAesBawADWhdlZ3HytOUUJc4u4D5gD8ia6Wn5xmQ/0L0iYoKbiI3XwzHZPSKeBY4GHgXOI+v3/M+I+GsLQ+0oPttu7+j2I70OuJfsh/lN4AdpKMyOZKNiJhYSp5uJjdPLMfkZ8I2I+H5aZzDQNyJelyQfj+ZwzdOA9/xIryf7kZ5L9oNdVvhBRsS9ETExvQ//UBunxDH5PfDO3z0iFkbE6+m9j0eTOHl2sOImd9G4zXuAu8l+pDcAN0TED7uvb41RxTH5QQvCsyJutncoSQOAj0XEHyVtQTamcyawE3AlWe3mxoj4aVrfzcEG8zHJFyfPDiVpTeBA4HNkTyT8TETMTDWfC4G/RcR/pnX9I20CH5N88VUiHSoiXpM0C/gIcGe6WgWyrpzvRMRc8I+0mXxM8sV9nh1K0gcj4nrg08AMSZdIWiP1sxWfjPCPtEl8TPLFzfYOU6i1SHoQeCMidpU0EjgB6A9cDexDVtOZ08pYO4WPST655tl5+gNExLZAP0k3RMQLwCnAfOAM4Pf+kTaVj0kOuebZQdJ9ID8G3FLoT5P0Z2BeRHw2fV43Il5yv1pz+Jjkl2uenWUEsCuwq6SN0rx9gN0kTUif54L71ZrIxySnfLZ9JVbUlzYaWEx2553nge+m5ROANYH/Bu4AiIhlLQq3I/iYrDzcbF9JFf1IPwFcCtwJbAscCrwK/AcwBNgOODAi7nazsLF8TFYuTp4rMUnbAuPJLuf7s6TxwMXAJ8nuRj4SGBgRk1sVY6fxMVl5OHmuRIpuYFy4mcQNwIfIfqzTImKZpKOBDSLiuNZF2jl8TFZePmG0kpDUD/gMsIGkz5HdbXwf4AngK0WrvkF26Z81mI/Jys0njFYSEfGWpIHAzUAf4JsR8WZqFt4G3C7pOuCzwDmti7Rz+Jis3FzzXAkU3cbsBuAvwAJglqRVI+JN4J+Bt4B9gSMj4rrUhLQG8TFZ+flgrQTSGdzNgaOAY4Cz0jQurdIf+DzwJvC9tI2HvzSQj8nKz8kz54pqK6OBUWQ/yMuBq4ATJH0LeI6sT208sI6kEc2PtHP4mHQGn23POUkbRMRz6f14YGdgZkT8VNK/kJ3ZnRoRt7Uuys7iY9IZnDxzTNLqwF3ALyPizDTv/wH/CtwKXBTZM73f6YPzgOvG8jHpHG6250zhB6fsEcAbkQ152U/SoQARcQ3ZnXjGAGsXtouk+RGv/HxMOpOHKuVMOhExHjiZ7Ac5iWzYy7ckDSJ7wuIaZI8KntlLMVZHPiadyc32nJG0FtnJh2MjYkaq3Ywkuzb6QLKbTZwTERNKFGN15GPSmVzzzJ8lZDePKDT/fg2cBywluy/k4Ih4wzeUaCofkw7kPs+ciYg3gGuAnSSNjoi3yR5JOxLoSst9EqKJfEw6k5NnPl0JrAKcKelUslrOzelHa63hY9Jh3OeZU5KGANsDm5DdnefPLQ6p4/mYdBYnTzOzGrjZbmZWAydPM7MaOHmamdXAydPMrAZOnmZmNXDytKpJWippmqTpkq5O12/XWtalkj6f3v9C0hYl1t1Z0g417GOWpPc8I6i3+d3WWVjlvk6W5Ae5dQAnT6vF4ogYGxGjyR4l8eXihYUnRlYrIr4UEY+VWGVnoOrkadYITp62ou4FNkm1wjsl/QZ4VFIfSf8t6SFJj0g6ErLbt0k6V9Jjkm4ChhcKknSXpHHp/W6Spkp6WNIdkkaRJelvplrvjpLWlnRN2sdDkj6atl1L0m2S/iLp54AoQ9LvJU2RNEPSEd2W/STFcoektdO8jSXdkra5Nz1ywzqIbwxiNZPUF9gduCXN2gYYHREzUwJ6IyI+Iqk/cJ+k24Atgc3I7qa+DvAYcEm3ctcGLgJ2SmUNjYhXJV0ALIyIH6f1fgOcGRETJW1IdrPhfwJOAiZGxKnpzu3LJcNeHJb2MRB4SNI1EfEKsCrZXd+PlXRiKvso4ELgyxHxtKRtgZ8Bu9TwZ7SccvK0WgyUNC29vxe4mKw5PanofpW7Ah8u9GcCqwObAjsBV0TEUuBFSX/qofztgHsKZUXEq73E8SlgC73zoEpWS5dI7gR8Lm17k6TXKvhOX5e0d3q/QYr1FWAZ2XXrkN0t6VpJg9P3vbpo3/0r2IetRJw8rRaLI2Js8YyURBYVzwK+FhG3dltvD6DcNcGqYB3Iup22j4jFPcRS8XXHknYmS8Tbp+eq3wUM6GX1SPt9vfvfwDqL+zytUW4FviJpFQBJH5C0KnAP8IXUJzoC+EQP294PfFzS+9K2Q9P8BWT3zSy4jawJTVpvbHp7D3BAmrc7sGaZWFcHXkuJc3Oymm9BF9nTLwH2J+sOmA/MlLRP2ockjSmzD1vJOHlao/yCrD9zqqTpwM/JWjoTgKeBR4Hzgbu7bxgRL5P1U14r6WHebTbfAOxdOGEEfB0Yl05IPca7Z/1PIbu35lSy7oNny8R6C9BX0iPAacADRcsWAR+UNIWsT/PUNP8A4PAU3wxgrwr+JrYS8V2VzMxq4JqnmVkNnDzNzGrg5GlmVgMnTzOzGjh5mpnVwMnTzKwGTp5mZjX4P/6PSw2kDehIAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize= False,  title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d9cfbf5",
   "metadata": {},
   "source": [
    "We will try **SVM** as the forth model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "45f5a69a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import svm\n",
    "sVm= svm.SVC(kernel='linear')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "8cc9324b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(kernel='linear')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sVm.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "1f51b5b3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = sVm.predict(x_test)\n",
    "yhat [0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da5ad344",
   "metadata": {},
   "source": [
    "# Evaluation SVM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "845e18f0",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "03963780",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8852459016393442"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_sVm=metrics.accuracy_score(y_test,yhat)\n",
    "acc_sVm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40cf5ec2",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "81adc1fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7407407407407407"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f17bfc01",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "d8522084",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.9635316844512425"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test, yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf59ffaa",
   "metadata": {},
   "source": [
    "**f1-score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "54f1b367",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.80      0.85        25\n",
      "           1       0.87      0.94      0.91        36\n",
      "\n",
      "    accuracy                           0.89        61\n",
      "   macro avg       0.89      0.87      0.88        61\n",
      "weighted avg       0.89      0.89      0.88        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f14fdb98",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "6bd8f0a1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[34  2]\n",
      " [ 5 20]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiIklEQVR4nO3dedyVdZ3/8df7vm9AFBQBMSSVTHMJE5VEMxWzMS1LrMzUsXKcMZvxN+UyZcsvt6lpnEnNNMulwTF1tBT3XMZU3BLByN30F7iBIhoIiMmNn98f1/fGw819n42zXZz308f1OOdcy/f6nPvifPwu16KIwMzMKtPR7ADMzPLIydPMrApOnmZmVXDyNDOrgpOnmVkVnDzNzKrg5Gl1IWmwpBskLZL06zUo53BJt9UytmaRtIekp5sdh9WGfJ5ne5N0GHA8sA2wGJgF/CAi7l3Dco8A/g/wkYjoXtM4W52kALaKiGebHYs1hmuebUzS8cDZwA+BjYHNgJ8BB9ag+M2BP7VD4iyHpK5mx2A1FhGe2nACNgCWAAcXWWcQWXKdm6azgUFp2STgReAEYD4wDzgyLTsVeBtYnvZxFHAK8KuCsscCAXSlz18B/kxW+50NHF4w/96C7T4CPAQsSq8fKVh2F3A6cF8q5zZgZD/frSf+bxbEPxn4JPAn4HXgOwXr7wI8ACxM654LDEzLpqXvsjR930MKyv8W8DJwac+8tM370z52Sp83ARYAk5r9b8NTeZNrnu1rN2AdYGqRdb4L7AqMB3YgSyDfK1j+HrIkPIYsQZ4nacOIOJmsNntlRAyJiIuLBSJpPeAcYP+IGEqWIGf1sd5w4Ka07gjgTOAmSSMKVjsMOBIYBQwETiyy6/eQ/Q3GAN8HLgT+FtgZ2AP4vqQt0rorgOOAkWR/u32AfwSIiD3TOjuk73tlQfnDyWrhRxfuOCL+H1livUzSusB/AVMi4q4i8VoLcfJsXyOABVG8WX04cFpEzI+IV8lqlEcULF+eli+PiJvJal1bVxnPO8A4SYMjYl5EPN7HOp8CnomISyOiOyKuAJ4CPl2wzn9FxJ8iYhlwFVni789ysv7d5cD/kCXGn0TE4rT/x4EPAUTEzIj4fdrvHOAXwF5lfKeTI+KvKZ5VRMSFwDPAg8Bosv9ZWU44ebav14CRJfriNgGeK/j8XJq3soxeyfdNYEilgUTEUrKm7jHAPEk3SdqmjHh6YhpT8PnlCuJ5LSJWpPc9ye2VguXLeraX9AFJN0p6WdIbZDXrkUXKBng1It4qsc6FwDjgpxHx1xLrWgtx8mxfDwBvkfXz9WcuWZOzx2ZpXjWWAusWfH5P4cKIuDUi/oasBvYUWVIpFU9PTC9VGVMlzieLa6uIWB/4DqAS2xQ9lUXSELJ+5IuBU1K3hOWEk2ebiohFZP1850maLGldSQMk7S/pjLTaFcD3JG0kaWRa/1dV7nIWsKekzSRtAHy7Z4GkjSV9JvV9/pWs+b+ijzJuBj4g6TBJXZIOAbYDbqwypkoMBd4AlqRa8dd6LX8F2GK1rYr7CTAzIv6erC/352scpTWMk2cbi4gzyc7x/B7wKvACcCxwbVrlX4EZwCPAo8DDaV41+7oduDKVNZNVE14H2aj9XLIR6L1IgzG9yngNOCCt+xrZSPkBEbGgmpgqdCLZYNRislrxlb2WnwJcImmhpC+UKkzSgcB+ZF0VkB2HnSQdXrOIra58kryZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYF36ygQuoaHBo4tNlhWIEdt92s2SFYL889N4cFCxaUOg+2Ip3rbx7RvdqFWquJZa/eGhH71XLffXHyrJAGDmXQ1iXPRLEGuu/Bc5sdgvWy+8QJNS8zut9i0DZfLLneW3/4aakrv2rCydPM8kGAalqZXSNOnmaWH2qdYRonTzPLCUFHZ7ODWMnJ08zyw812M7MKCTfbzcwqJ9c8zcyq4j5PM7NKyc12M7OK+TxPM7NqCDpaJ2W1TiRmZqV0uOZpZlYZn6pkZlYl93mamVXKl2eamVXHzXYzswrJVxiZmVXHNU8zs0q5z9PMrDputpuZVcjneZqZVaO1bgzSOpGYmZXS0Vl6KkHSOpKmS/qjpMclnZrmD5d0u6Rn0uuGRUOp0VcyM6u/ntOVik2l/RX4WETsAIwH9pO0K3AScEdEbAXckT73y8nTzPJBqdleaiohMkvSxwFpCuBA4JI0/xJgcrFynDzNLD/Kq3mOlDSjYDp69WLUKWkWMB+4PSIeBDaOiHkA6XVUsVA8YGRmuSCgo6Os+t6CiJhQbIWIWAGMlzQMmCppXKXxuOZpZvmgMqcKRMRC4C5gP+AVSaMB0uv8Yts6eZpZTgip9FSyFGmjVONE0mDg48BTwPXAl9NqXwauK1aOm+1mlhtlNttLGQ1cIqmTrAJ5VUTcKOkB4CpJRwHPAwcXK8TJ08xyo5yaZSkR8QiwYx/zXwP2KbccJ08zy4cq+jTrycnTzHJBlNen2ShOnmaWGzXq86wJJ08zyw3XPM3MKuU+TzOz6rjmaWZWISH3eZqZVaV1Kp5OnmaWE3Kz3cysKk6eZmYVcp+nmVm1Wqfi6VvStYNBA7u459ITefDKk5j5m+/yvWM+ucrybxyxD8v+cC4jhq3XpAjb2wsvvMAnPr4347fflp12+CDnnvOTZofUmlKf55rekq5WXPNsA399u5v9jj6Hpcvepqurg9/98nhuu+8Jpj86h/duPIyP7boNz897vdlhtq2uri5+dMaP2XGnnVi8eDEfmbgz+3z8b9h2u+2aHVrLaaU+T9c828TSZW8DMKCrk66uTiICgDNO/Bzf/cm1Kz9b440ePZodd9oJgKFDh7LNNtsyd+5LTY6qNalDJadGcc2zTXR0iPsv/xbv33QjfnHlNB567Dk+tdf2zJ2/kEf/5B9qq3huzhxmzfoDH95lYrNDaUmuefZB0lckbbIG24+VdFiF2/xA0guSlpReO9/eeSfY9Ys/YstPfI8J4zZn3Fab8K2jPsFp59/U7NAsWbJkCYd+4XP8x4/PZv311292OC2nnP7ORibXlkmewFeAqpMnMBaoKHkCNwC7rME+c2fRkmVMm/EMB0z6EJuPGcH0K7/NUzedyphRw3jg8m+x8YihzQ6xLS1fvpxDv/A5Djn0cCYf9Nlmh9OyWil51rXZLul44O/Sx4uAa4EbI2JcWn4iMAR4DJgAXCZpGbAb8CRwJbB32v6wiHhW0pRUxm9SGUsiYgjwI2Db9CzmSyLirFLxRcTvUxlr/mVb2MgNh7B8+QoWLVnGOoMG8LGJW/PjKf/L5vt8e+U6T910KrsffgavLVzaxEjbU0RwzD8cxdbbbMvXjzu+2eG0tEb2aZZSt+QpaWfgSGAi2dlZDwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WZMm3L5PSI0fL/S5HA0cDMGBIuZu1jPeMXJ8LTzuCzo4OOjrE1bc/zG/veazZYVly/333cflllzJu3PZM3Hk8AKf+6w/Zb/9PFt+wDbVSRaeeNc+PAlMjYimApGuAPSos44qC15I1yUIR8TQwvsL99VfWBcAFAB3rjsrdsPRjz8xlt0P/veg623zq5AZFY73t/tGPsmx57v5ZNV4bXdve17ccxqr9rOuUKCP6eN/dU4ayv+TAPndew5qnmTVfdnlm6yTPeg4YTQMmS1pX0nrAQcBvgVGSRkgaxKrN8MVA79GKQwpeH0jv5wA7p/cHAgP62j4ino6I8f1MC2vzFc2skaTSU+kytKmkOyU9KelxSV9P80+R9JKkWWkq2m9St5pnRDycBnemp1kXRcRDkk4j6/+cDTxVsMkU4OcFA0YAgyQ9SJbkD03zLgSukzQduAPoGeF4BOiW9EdgSjkDRpLOIBuhX1fSiynGU6r5vmZWfzVqtncDJ6QcNRSYKen2tOysiPjPcgqp62h7RJwJnNlr3jnAOX2sezVwdc/n9Ec6LyJO7bXeK8CuBbO+neYvp4IH1qdtvgl8s5JtzKxJyqxZlhIR84B56f1iSU8CYyotp5XO8zQz65eAzk6VnICRkmYUTEf3W6Y0FtiRrDUMcKykRyT9UtKGxeJp2cszI2Jss2Mws9ZSZrN9QURMKKOsIWSt3W9ExBuSzgdOJxucPh34Me+ep76alk2eZmarqFGzHUDSALLEeVlEXAMruwR7ll8I3FisDDfbzSwXRG0uz0ynOF4MPJnGZXrmjy5Y7SCyKx/75ZqnmeVEzc7z3B04Ang0Xc4N8B3gUEnjyZrtc4CvFivEydPMcqMWpypFxL30fRHPzZWU4+RpZvlQwz7PWnDyNLNc6OnzbBVOnmaWG610bbuTp5nlRgtVPJ08zSwn2uiWdGZmNZP1eTY7inc5eZpZTrTW/TydPM0sN9xsNzOrlM/zNDOrnICOjta5HYeTp5nlhmueZmZVcJ+nmVml3OdpZla5Vnv0sJOnmeVGRwtVPZ08zSw3Wih3OnmaWT4oL9e2S/op2e3o+xQR/1yXiMzM+tGZkz7PGQ2LwsysDC1U8ew/eUbEJYWfJa0XEUvrH5KZ2epENuLeKkpe6yRpN0lPAE+mzztI+lndIzMz66VDpaeGxVLGOmcDnwBeA4iIPwJ71jEmM7PVKTvPs9TUKGWNtkfEC71GuVbUJxwzs76J1jrPs5ya5wuSPgKEpIGSTiQ14c3MGkkqPZUuQ5tKulPSk5Iel/T1NH+4pNslPZNeNyxWTjnJ8xjgn4AxwEvA+PTZzKyhJJWcytANnBAR2wK7Av8kaTvgJOCOiNgKuCN97lfJZntELAAOLyciM7N6kWpznmdEzAPmpfeLJT1JVjk8EJiUVrsEuAv4Vn/llDPavoWkGyS9Kmm+pOskbbGG8ZuZVUxlTMBISTMKpqP7LU8aC+wIPAhsnBJrT4IdVSyWcgaMLgfOAw5Kn78IXAFMLGNbM7OaKbNZviAiJpRR1hDgauAbEfFGpZd+ltPnqYi4NCK60/Qrily2aWZWD5Lo7Cg9lVnWALLEeVlEXJNmvyJpdFo+GphfrIx+k2caeRoO3CnpJEljJW0u6ZvATWVFaGZWQzUabRdwMfBkRJxZsOh64Mvp/ZeB64qVU6zZPpOshtkTzlcLlgVweukwzcxqp0Z3VdodOAJ4VNKsNO87wI+AqyQdBTwPHFyskGLXtr+vFlGamdVCdpL8mpcTEfdCvxfJ71NuOWVdYSRpHLAdsE5BAP9d7k7MzGqhla4wKpk8JZ1Mdu7TdsDNwP7AvYCTp5k1jNRaybOc0fbPk1VlX46II4EdgEF1jcrMrA+1GDCqlXKa7csi4h1J3ZLWJxu+90nyZtZwuXgMR4EZkoYBF5KNwC8BptczKDOz3kT553E2QjnXtv9jevtzSbcA60fEI/UNy8yslwY3y0sp9gC4nYoti4iH6xNSa9t+6025+Xc/bnYYVuBLv2rLf4otbfbrb9al3Lw024tliAA+VuNYzMyKKmeEu1GKnSS/dyMDMTMrRuTn0cNmZi2lhXKnk6eZ5UN2HmfrZE8nTzPLjVaqeZZzJ3lJ+ltJ30+fN5O0S/1DMzN7V0+fZy3u51kL5Qxe/QzYDTg0fV5Mdmd5M7OG6ihjapRymu0TI2InSX8AiIi/SBpY57jMzFbTQl2eZSXP5ZI6SY/ekLQR8E5dozIz66XnMRytopxa7jnAVGCUpB+Q3Y7uh3WNysysDx0qPTVKOde2XyZpJtlt6QRMjogn6x6ZmVmB7E7yrVPzLOdmyJsBbwI3FM6LiOfrGZiZWW8tlDvL6vO8iXcfBLcO8D7gaeCDdYzLzGxVgs4Wyp7lNNu3L/yc7rb01X5WNzOri1o9AK5WKr7CKCIelvThegRjZlZMrpKnpOMLPnYAOwGv1i0iM7N+tNK17eWcqjS0YBpE1gd6YD2DMjPrTYLOjtJT6XL0S0nzJT1WMO8USS9JmpWmT5Yqp2jNM50cPyQi/qWM72ZmVlc1OlVpCnAuqz8+/ayI+M9yCyn2GI6uiOgu9jgOM7NGqdWAUURMkzR2TcspVvOcTta/OUvS9cCvgaUFAVyzpjs3M6tEmRXPkZJmFHy+ICIuKGO7YyV9CZgBnBARfym2cjmj7cOB18ieWdRzvmcATp5m1jBC5Z7nuSAiJlRY/PnA6WS57XSyZ7j9XbENiiXPUWmk/THeTZo9osLAzMzWTB2vXY+IV1buRroQuLHUNsWSZycwhFWT5sp9VRydmdkaqte17ZJGR8S89PEgskpjUcWS57yIOK0mkZmZrSFRm2vbJV0BTCLrG30ROBmYJGk8WcVwDmVcRVksebbO2ahmZtTm0cMRcWgfsy+utJxiyXOfSgszM6sX0djHbJTSb/KMiNcbGYiZWVF+9LCZWeVEzm5JZ2bWKlondTp5mlmOtFDF08nTzPJC7vM0M6uU+zzNzKrUOqnTydPM8sKnKpmZVS43J8mbmbWaet0YpBpOnmaWGy2UO508zSwfsmZ762RPJ08zyw3XPM3MKib3eZqZVcrNdjOzasjNdjOzqjh5WlPtusMHWG/IUDo7O+nq6uLm393f7JDazoh1B/BPe4xl2OABRAT/+6cF/PbJV1lvYCfHTXofGw0ZyKtL3uasu2az9O0VzQ63JfjadmsJv77+VoaPGNnsMNrWiggufehFZr++jHW6OvjRp7fhkbmLmbTlCB6dt5jrHn2FA7ffmMnbb8xlM+c2O9yWoRbq82ylq53M2sbCZd3Mfn0ZAG91v8NLi95i+LoD+PBmG3D3s68BcPezr/HhzYY1McrWI5WeGsXJsw1J4rDPHcD+e+/Gr6Zc1Oxw2t5GQwbyvuHr8uyCpWwwuIuFy7qBLMGuv44bhz16mu2lpkZpmeQp6SuSNlmD7cdKOqzCbXaW9KikZyWdo1a6ZUsdTf3tndxy1++59KrruOTiX/D7++9pdkhta1BXBydM2oIp019k2fJ3mh1Oi1NZ/5UsRfqlpPmSHiuYN1zS7ZKeSa8bliqnZZIn8BWg6uQJjAUqSp7A+cDRwFZp2m8N9p8b7xmd/ZlHbjSK/T71GWbNnNHkiNpTp+CEvbfgnj+/zvTnFwKwaFk3wwZntc1hg7t4463uJkbYYspospdZ/ZnC6r/1k4A7ImIr4I70uai6Jk9Jx0t6LE3fSLXDwmx/oqRTJH0emABcJmmWpMGS5kj6d0nT07Rl2mZKWr+njCXp7Y+APdL2x5UR22hg/Yh4ICIC+G9gcu2+fWt6c+lSlixevPL9tDvvYOttP9jkqNrTMbtvzkuL3uKmJ+avnDfjhUXsteUIAPbacgQPPb+oWeG1JJUxlRIR04Dej1Y/ELgkvb+EMnJB3TpUJO0MHAlMJPtODwJ397VuRPxG0rHAiRExI20P8EZE7CLpS8DZwAFFdnlS2v6AtP3WwJX9rDsJGAO8WDDvxTSvr+9yNFkNlTHv3bRICK3v1Vdf4e+POASAFd3dTP78Iez98X2bHFX72XrUeuy15Qiee30ZZ3xmGwCumDmXax99meP2eh8f22oEC5a8zZl3zW5ypK2jglOVRkoqbE5dEBEXlNhm44iYBxAR8ySNKrWTevZGfxSYGhFLASRdA+xRYRlXFLyeVcmGEfE0ML6/5f30b0Y/ZV0AXACww44797lOXmw+dgtuv+ehZofR9p6ev5QvTHm4z2Wn3/Zsg6PJkfKa5QsiYkKdI6lr8uzraw5j1a6CdUqUEX287+4pIyXAgX3uvHTN80XgvQXz3gv4hDqzFlbH8zxfkTQ61TpHA/NLbVDPPs9pwGRJ60paDzgI+C0wStIISYNYtRm+GBjaq4xDCl4fSO/nADun9wcCA/raPiKejojx/UwLUxV9saRdUxL+EnBdLb64mdVHHc/zvB74cnr/ZcrIBXWreUbEw5KmANPTrIsi4iFJp5H1f84GnirYZArwc0nLgN3SvEGSHiRL8oemeRcC10maTjYqtjTNfwTolvRHYEpElNPM/1ra72CyxP7bSr+nmTVOLU4mlHQFWetzpKQXgZPJBpyvknQU8DxwcKly6noGbkScCZzZa945wDl9rHs1cHXP59QleV5EnNprvVeAXQtmfTvNXw7sU2F8M4BxlWxjZs2RjaavefaMiEP7WVRR/vDlC2aWD74lXXkiYmyzYzCz1tJCubN1k6eZ2apE32cYNoeTp5nlRgvlTidPM8uHci+/bBQnTzPLjxbKnk6eZpYbfvSwmVkVWid1OnmaWV60WKenk6eZ5UYrPQDOydPMckFAR+vkTidPM8sRJ08zs8q52W5mVgU3283MquHkaWZWmVrdz7NWnDzNLB98P08zs+o4eZqZVUxutpuZVcM1TzOzCrXYpe1OnmaWH34Mh5lZFVoodzp5mll+1Cp3SpoDLAZWAN0RMaHSMpw8zSwfan+e594RsaDajZ08zSwXRGv1eXY0OwAzs3KpjAkYKWlGwXR0H0UFcJukmf0sL8k1TzPLjTIrngvK6MPcPSLmShoF3C7pqYiYVkksrnmaWW6ojP/KERFz0+t8YCqwS6WxOHmaWW5IpafSZWg9SUN73gP7Ao9VGoub7WaWC+UmxzJsDExNg09dwOURcUulhTh5mllu1OLGIBHxZ2CHNS3HydPMcqOFzlRy8jSz/HDyNDOrmO/naWZWsewKo2ZH8S4nTzPLDSdPM7MquNluZlYpPz3TzKxyfgyHmVmVWumWdE6eZpYbLZQ7nTzNLD9aKHc6eZpZjrRQ9nTyNLNcENDRQu12RUSzY8gVSa8CzzU7jhoYCVT98Curm7XluGweERvVskBJt5D9fUpZEBH71XLffcbj5NmeJM2o5nGrVl8+LvnhO8mbmVXBydPMrApOnu3rgmYHYH3ycckJ93mamVXBNU8zsyo4eZqZVcHJ08ysCk6eZjmmVrrNUJtx8rSySfLlvC1C0nBJIym42tuJtLGcPK0oSeMknQ8QEd2SOpsdU7uTtD1wHXAFcKqkQwAiIpxAG8enKlm/JL0XuA14LzA9Ij6e5ndGxIqmBtemJA0CbgUuAe4CdgcOBO6OiHObGFrbcc3TihkDnBMR6wMrJP0OICJWuAnfNN3AHGBaRMwGrgd+BkyS9LfNDKzdOHlavyLiQeCa9P4TwHJJd6bP3ZJGNDO+dpRq/C8DF0kaHBFvAA8B/wN8SNJgN90bw8nT+tTTtxkR83tqmQUJ9AZJuwJnpEELa4CC2v63gceB81ICXQL8HpgAjAj3xTWEk6etpqdPU5nOVMvsSaD7Au8H7gdujIi14d6TLa/gOHSQjbCfASwBrpe0CbAJMAxYp3lRthcnT1tFQeLsAO4DPglZMz0tfz/ZD/TAiJjqJmL99XFM9o+I54GvA48C55H1e/5bRDzbxFDbikfbbaVeP9LrgHvIfpjHAT9Ip8LsQXZWzL09idPNxPrp55j8DPhGRPxrWmcI0BURCyXJx6MxXPM0YLUf6fVkP9JzyX6w7/T8ICPinoi4N70P/1Drp8gxuRZY+XePiCURsTC99/FoECfPNlbY5C44b3MacDfZj/QG4IaI+GHv9a0+KjgmP2hCeFbAzfY2JWkd4KMR8b+StiM7p3M2sCdwJVnt5saI+Ela383BOvMxyRcnzzYlaUPgCOCzZE8k/HREzE41nwuAP0fEv6V1/SNtAB+TfPFVIm0qIv4iaQ7wYeDOdLUKZF05342I+eAfaSP5mOSL+zzblKQPRsT1wCeAxyX9UtKw1M9WOBjhH2mD+Jjki5vtbaan1iLpQWBRROwraQzwHWAQ8GvgYLKazivNjLVd+Jjkk2ue7WcQQERMBAZKuiEiXgJOBd4AzgSu9Y+0oXxMcsg1zzaS7gP5UeCWnv40SfcDCyLiM+nzeyLiZferNYaPSX655tleRgP7AvtK2jzNOxjYT9LU9Hk+uF+tgXxMcsqj7Wuxgr60ccAysjvvvAh8Ly2fCmwI/AdwB0BEvNOkcNuCj8naw832tVTBj3RvYApwJzAROBJ4HfgWMBTYFTgiIu52s7C+fEzWLk6eazFJE4HJZJfz3S9pMnAxsA/Z3cjHAIMjYkazYmw3PiZrDyfPtUjBDYx7biZxA7A92Y91VkS8I+nrwKYRcWLzIm0fPiZrLw8YrSUkDQQ+DWwq6bNkdxs/GHgK+FrBqovILv2zOvMxWbt5wGgtERFvSxoM3Ax0AsdFxJupWXgbcLuk64DPAD9tXqTtw8dk7eaa51qg4DZmNwB/ABYDcyStFxFvAn8DvA0cAnw1Iq5LTUirEx+TtZ8P1logjeBuAxwLHA+cnaYJaZVBwOeBN4H/m7bx6S915GOy9nPyzLmC2so4YCzZD/Iy4CrgO5L+BXiBrE9tMrCxpNGNj7R9+Ji0B4+255ykTSPihfR+MjAJmB0RP5H0KbKR3Ycj4rbmRdlefEzag5NnjknaALgL+O+IOCvN+xzwD8CtwIWRPdN7ZR+cT7iuLx+T9uFme870/OCUPQJ4c7JTXg6VdCRARFxNdieeHYCNeraLpPERr/18TNqTT1XKmTQQMRk4hewHOZ3stJd/kbQu2RMWh5E9Knh2P8VYDfmYtCc323NG0giywYcTIuLxVLsZQ3Zt9BFkN5v4aURMLVKM1ZCPSXtyzTN/usluHtHT/PsVcB6wguy+kEMiYpFvKNFQPiZtyH2eORMRi4CrgT0ljYuI5WSPpB0DdKTlHoRoIB+T9uTkmU9XAgOAsySdRlbLuTn9aK05fEzajPs8c0rSUGA3YEuyu/Pc3+SQ2p6PSXtx8jQzq4Kb7WZmVXDyNDOrgpOnmVkVnDzNzKrg5GlmVgUnT6uYpBWSZkl6TNKv0/Xb1ZY1RdLn0/uLJG1XZN1Jkj5SxT7mSFrtGUH9ze+1zpIK93WKJD/IrQ04eVo1lkXE+IgYR/YoiWMKF/Y8MbJSEfH3EfFEkVUmARUnT7N6cPK0NXUPsGWqFd4p6XLgUUmdkv5D0kOSHpH0Vchu3ybpXElPSLoJGNVTkKS7JE1I7/eT9LCkP0q6Q9JYsiR9XKr17iFpI0lXp308JGn3tO0ISbdJ+oOkXwCiBEnXSpop6XFJR/da9uMUyx2SNkrz3i/plrTNPemRG9ZGfGMQq5qkLmB/4JY0axdgXETMTgloUUR8WNIg4D5JtwE7AluT3U19Y+AJ4Je9yt0IuBDYM5U1PCJel/RzYElE/Gda73LgrIi4V9JmZDcb3hY4Gbg3Ik5Ld25fJRn24+/SPgYDD0m6OiJeA9Yju+v7CZK+n8o+FrgAOCYinpE0EfgZ8LEq/oyWU06eVo3Bkmal9/cAF5M1p6cX3K9yX+BDPf2ZwAbAVsCewBURsQKYK+l3fZS/KzCtp6yIeL2fOD4ObKeVD6pk/XSJ5J7AZ9O2N0n6Sxnf6Z8lHZTeb5pifQ14h+y6dcjulnSNpCHp+/66YN+DytiHrUWcPK0ayyJifOGMlESWFs4C/k9E3NprvU8Cpa4JVhnrQNbttFtELOsjlrKvO5Y0iSwR75aeq34XsE4/q0fa78LefwNrL+7ztHq5FfiapAEAkj4gaT1gGvDF1Cc6Gti7j20fAPaS9L607fA0fzHZfTN73EbWhCatNz69nQYcnubtD2xYItYNgL+kxLkNWc23RwfZ0y8BDiPrDngDmC3p4LQPSdqhxD5sLePkafVyEVl/5sOSHgN+QdbSmQo8AzwKnA/c3XvDiHiVrJ/yGkl/5N1m8w3AQT0DRsA/AxPSgNQTvDvqfyrZvTUfJus+eL5ErLcAXZIeAU4Hfl+wbCnwQUkzyfo0T0vzDweOSvE9DhxYxt/E1iK+q5KZWRVc8zQzq4KTp5lZFZw8zcyq4ORpZlYFJ08zsyo4eZqZVcHJ08ysCv8f+oA5YmiE2l4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize= False,  title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "551a67df",
   "metadata": {},
   "source": [
    "We will try **DecisionTreeClassifier** as the fifth model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "c51d50ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "cb3899a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(criterion='entropy')"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deTree =DecisionTreeClassifier(criterion=\"entropy\")\n",
    "deTree "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "40f29644",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(criterion='entropy')"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deTree.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "427abf23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = deTree.predict(x_test)\n",
    "yhat[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "857829e8",
   "metadata": {},
   "source": [
    "# Evaluation DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cec73c2",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "7b67d456",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8688524590163934"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_deTree=metrics.accuracy_score(y_test,yhat)\n",
    "acc_deTree"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ab3b9f",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "80e04f7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7241379310344828"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca00768",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f02120f7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.5297280253935455"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test,yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21ba2e09",
   "metadata": {},
   "source": [
    "**f1-score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "891fc11f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.84      0.84        25\n",
      "           1       0.89      0.89      0.89        36\n",
      "\n",
      "    accuracy                           0.87        61\n",
      "   macro avg       0.86      0.86      0.86        61\n",
      "weighted avg       0.87      0.87      0.87        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "695f8ecb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[32  4]\n",
      " [ 4 21]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "ab07d2ef",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[32  4]\n",
      " [ 4 21]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAhr0lEQVR4nO3dabgcVbn28f+9MxEIcwiECEQBIxhlioRBQgAZHYi+IiIy6ot65AgIHFE5Momg5wgyaxANCAIqxEBABhFkTggQIEwGTZDIEMKYQJBk5zkfajV0Nnv3lO7dXen751XX7q6uWvV0in5ca9WqVYoIzMysOh3NDsDMLI+cPM3MauDkaWZWAydPM7MaOHmamdXAydPMrAZOntYQkgZKulbSa5J+vwzl7C/ppnrG1iySdpD0ZLPjsPqQx3m2N0lfAr4NfAiYD0wHTo2IO5ex3AOA/wS2i4jFyxpnq5MUwMYR8VSzY7He4ZpnG5P0beBnwI+AtYH1gfOBvetQ/AbA39ohcVZCUt9mx2B1FhFe2nABVgUWAPuU2GYAWXJ9Ni0/Awakz8YCc4CjgbnAc8Ah6bOTgLeBRekYXwFOBC4tKns4EEDf9P5g4B9ktd9ZwP5F6+8s2m874D7gtfR3u6LPbgNOAe5K5dwEDO7huxXi/6+i+McBewF/A14Gvle0/dbAPcCradtzgf7ps9vTd3kjfd99i8r/DvA88JvCurTPhukYW6b36wLzgLHN/m/DS2WLa57ta1tgBWBiiW2+D2wDbA5sRpZAji/6fB2yJDyMLEGeJ2n1iDiBrDZ7ZUQMioiLSgUiaSXgbGDPiFiZLEFO72a7NYDr0rZrAmcA10las2izLwGHAEOA/sAxJQ69Dtm/wTDgB8CFwJeBrYAdgB9I+kDathM4ChhM9m+3C/AfABExJm2zWfq+VxaVvwZZLfyw4gNHxN/JEutlklYEfg1MiIjbSsRrLcTJs32tCcyL0s3q/YGTI2JuRLxIVqM8oOjzRenzRRFxPVmta0SN8SwBRkoaGBHPRcSj3WzzSWBmRPwmIhZHxOXAE8Cni7b5dUT8LSIWAr8jS/w9WUTWv7sIuIIsMZ4VEfPT8R8FPgoQEfdHxL3puLOBXwA7VvCdToiIf6d4lhIRFwIzgSnAULL/s7KccPJsXy8Bg8v0xa0LPF30/um07p0yuiTfN4FB1QYSEW+QNXW/Djwn6TpJH6ognkJMw4reP19FPC9FRGd6XUhuLxR9vrCwv6QPSpos6XlJr5PVrAeXKBvgxYh4q8w2FwIjgXMi4t9ltrUW4uTZvu4B3iLr5+vJs2RNzoL107pavAGsWPR+neIPI+LGiNiVrAb2BFlSKRdPIaZ/1RhTNS4gi2vjiFgF+B6gMvuUHMoiaRBZP/JFwImpW8JywsmzTUXEa2T9fOdJGidpRUn9JO0p6Sdps8uB4yWtJWlw2v7SGg85HRgjaX1JqwLfLXwgaW1Jn0l9n/8ma/53dlPG9cAHJX1JUl9J+wKbApNrjKkaKwOvAwtSrfgbXT5/AfjAe/Yq7Szg/oj4Kllf7s+XOUrrNU6ebSwiziAb43k88CLwDHA48Me0yQ+BacDDwCPAA2ldLce6GbgylXU/Sye8DrKr9s+SXYHekXQxpksZLwGfStu+RHal/FMRMa+WmKp0DNnFqPlkteIru3x+InCxpFclfaFcYZL2BvYg66qA7DxsKWn/ukVsDeVB8mZmNXDN08ysBk6eZmY1cPI0M6uBk6eZWQ08WUGV1HdgqP/KzQ7DimyxyfrNDsG6ePrp2cybN6/cONiq9Fllg4jF77lR6z1i4Ys3RsQe9Tx2d5w8q6T+KzNgRNmRKNaL7ppybrNDsC62Hz2q7mXG4rcY8KEvlt3urQfPKXfnV104eZpZPghQXSuzy8TJ08zyQ61zmcbJ08xyQtDRp9lBvMPJ08zyw812M7MqCTfbzcyqJ9c8zcxq4j5PM7Nqyc12M7OqeZynmVktBB2tk7Japw5sZlZOh8ovZUhaQdJUSQ9JelTSSWn9GpJuljQz/V29ZCh1+kpmZo1VGKpUbinv38DOEbEZ2aOp95C0DXAccEtEbAzckt73yMnTzPJDKr+UEZkF6W2/tASwN3BxWn8xpZ8s6+RpZnmRbs8st8BgSdOKlsPeU5LUR9J0YC5wc0RMAdaOiOcA0t8hpaJpnd5XM7NyKmuWz4uIknPiRUQnsLmk1YCJkkZWG4prnmaWD5U02ascyhQRrwK3kT0G+gVJQ7NDaShZrbRHTp5mlh91uGAkaa1U40TSQOATwBPANcBBabODgEmlynGz3cxyom5T0g0FLpbUh6wC+buImCzpHuB3kr4C/BPYp1QhTp5mlh91uMMoIh4Gtuhm/UvALpWW4+RpZvngKenMzGrhiUHMzGrjKenMzGrgWZXMzKokN9vNzGrjmqeZWXUEdHS45mlmVh2lpUU4eZpZTgi52W5mVj03283MauCap5lZtdznaWZWPbnP08ysNu7zNDOrgWueZmbVcp+nmVltXPM0M6uSkPs8zcxq0joVTydPM8sJudluZlYTJ08zsyq5z9PMrFatU/F08mwHA/r35c8XHUn//n3p26cPE//8ID/8+fX86Mhx7DVmJG8v6mTWnHkcdsKlvLZgYbPDbVudnZ1sP3oU6w4bxtWTJjc7nNbTYn2erVMHtob599uL2eOwsxm97+mM/uJp7Lbdpmz9keHccu8TbLXPj9h639OY+fRcjj10t2aH2tbOPfssRmyySbPDaGmSyi69xcmzTbyx8G0A+vXtQ9++fYgIbrn3CTo7lwAw9ZFZDFt7tSZG2N7mzJnDDX+6jkMO/WqzQ2lp6lDZpbc4ebaJjg5x7xXH8c9bTucv9z7BfTOeXurzA/felhvveqxJ0dmxRx/Jqaf9pKUuiLQi1zy7IelgSesuw/7DJX2pyn1OlfSMpAW1HjcvliwJtvni6Wy0+/GMGrkBm2449J3P/usru9PZuYQrrr+viRG2r+uvm8yQtYaw5VZbNTuUllZJ4qwkeUpaT9Ktkh6X9KikI9L6EyX9S9L0tOxVqpyWSZ7AwUDNyRMYDlSVPIFrga2X4Zi589qChdw+bSa7bbcpAPt/ejR7jRnJwd+f0NzA2tg9d9/F5MnXMGKj4Ry4/xe57da/cMiBX252WC2pTjXPxcDREbEJsA3wTUmbps/OjIjN03J9qUIamjwlfVvSjLQcmWqHM4o+PyZl+88Do4DLUsYfKGm2pB9LmpqWjdI+E9L2hTIKtcbTgR3S/kdVEl9E3BsRz9XvG7emwasPYtVBAwFYYUA/dh49gidnv8Cu223C0Qd/gs8f+QsWvrWoyVG2r1NOPY2/z57Dk0/N5pLLrmDsTjvz60subXZYLakefZ4R8VxEPJBezwceB4ZVG0vDhipJ2go4BBhNNjprCvDX7raNiD9IOhw4JiKmpf0BXo+IrSUdCPwM+FSJQx6X9v9U2n8EcGUP246NiFer+C6HAYcB0G9Qpbu1jHUGr8KFJx9An44OOjrEVTc/wJ/umMGMSScwoH9fJl9wOABTH5nNt069osnRmvWswprlYEnTit6Pj4jxPZQ3HNiCLD9tDxye8s00strpKz0dpJHjPD8OTIyIN1KQVwM7VFnG5UV/z6xmx4h4Eti8yuP1VNZ4YDxAx4pDoh5l9qYZM59l2/1+/J71I/c+qQnRWCljdhzLmB3HNjuM1lT5OM95ETGqbHHSIOAq4MiIeF3SBcApQKS/PwUO7Wn/RibP7r7laizdVbBCmTKim9eLC2Uo+5fs3+3B61jzNLPmy27PrM/VdEn9yBLnZRFxNUBEvFD0+YVAyTsVGtnneTswTtKKklYCPgv8CRgiaU1JA1i6GT4fWLlLGfsW/b0nvZ4NFC5L7g30627/iHiyqOO36/Jqfb6imfUmqfxSvgwJuAh4PCLOKFo/tGizzwIzuu5brGE1z4h4QNIEYGpa9cuIuE/SyWT9C7OAJ4p2mQD8XNJCYNu0boCkKWRJfr+07kJgkqSpwC3AG2n9w8BiSQ8BEyKibDNf0k/IrtCvKGlOivHEWr6vmTVencZxbg8cADwiaXpa9z1gP0mbk7VyZwNfKxlLRGt24UmaDYyKiHnNjqVYx4pDYsCILzQ7DCvyyn3nNjsE62L70aO4//5pdR2xvsLQD8bwg84pu92TP97j/kr6PJeVJwYxs1wQ0KdP60wM0rLJMyKGNzsGM2strTSrUssmTzOzpVR4Qai3OHmaWS4I1zzNzGpQv3Ge9eDkaWa54ZqnmVm13OdpZlY993mamdXIfZ5mZjVooYqnk6eZ5USLPXrYydPMciHr82x2FO9y8jSznPA4TzOzmrjZbmZWLY/zNDOrnoCOjtZ5WrqTp5nlhmueZmY1cJ+nmVm13OdpZla9ej56uB6cPM0sNzpaqOrp5GlmudFCudPJ08zyQXm5t13SOWQPf+9WRHyrIRGZmfWgT076PKf1WhRmZhVooYpnz8kzIi4ufi9ppYh4o/EhmZm9l8iuuLeKsvc6SdpW0mPA4+n9ZpLOb3hkZmZddKj80muxVLDNz4DdgZcAIuIhYEwDYzIzey9l4zzLLeWL0XqSbpX0uKRHJR2R1q8h6WZJM9Pf1UuVU9Fd9hHxTJdVnZXsZ2ZWLyIb51luqcBi4OiI2ATYBvimpE2B44BbImJj4Jb0vkeVJM9nJG0HhKT+ko4hNeHNzHqTVH4pJyKei4gH0uv5ZPlsGLA3ULjWczEwrlQ5lYzz/DpwVir8X8CNwDcr2M/MrK4qHOc5WFLxaKHxETG+h/KGA1sAU4C1I+I5yBKspCGlDlI2eUbEPGD/SiI2M2sUqeJxnvMiYlT58jQIuAo4MiJer3YAfiVX2z8g6VpJL0qaK2mSpA9UdRQzszpQBUtF5Uj9yBLnZRFxdVr9gqSh6fOhwNxSZVTS5/lb4HfAUGBd4PfA5RXGaGZWN5LKLhWUIeAi4PGIOKPoo2uAg9Lrg4BJpcqpJHkqIn4TEYvTciklbts0M2sESfTpKL9UYHvgAGBnSdPTshdwOrCrpJnArul9j0rd275GenmrpOOAK8iS5r7AdZVEaGZWT/W4PTMi7qTnFv4ulZZT6oLR/WTJsnCQrxUfHzil0oOYmdVDLmZVioj392YgZmalZIPkmx3Fuyqaz1PSSGBTYIXCuoi4pFFBmZl1J1czyUs6ARhLljyvB/YE7gScPM2s10itlTwrudr+ebJO1Ocj4hBgM2BAQ6MyM+tGPW7PrJdKmu0LI2KJpMWSViEbOOpB8mbW63JxwajINEmrAReSXYFfAExtZFBmZl2Jisdx9opK7m3/j/Ty55JuAFaJiIcbG5aZWRe93Cwvp9Qg+S1LfVaY0qndbLHJ+tw15dxmh2FFxo2f0uwQrIun5jXmiT15abb/tMRnAexc51jMzEqqaPb2XlJqkPxOvRmImVkpIj+PHjYzayktlDudPM0sH7JxnK2TPZ08zSw3WqnmWclM8pL0ZUk/SO/Xl7R140MzM3tXoc+zDvN51kUlF6/OB7YF9kvv5wPnNSwiM7MedFSw9JZKmu2jI2JLSQ8CRMQrkvo3OC4zs/dooS7PipLnIkl9SI/ekLQWsKShUZmZdVF4DEerqKSWezYwERgi6VSy6eh+1NCozMy60aHyS2+p5N72yyTdTzYtnYBxEfF4wyMzMyuSzSTfOjXPSiZDXh94E7i2eF1E/LORgZmZddVCubOiPs/rePdBcCsA7weeBD7cwLjMzJYm6NNC2bOSZvtHit+n2Za+1sPmZmYNkcsHwBWLiAckfawRwZiZlZKr5Cnp20VvO4AtgRcbFpGZWQ/ydm/7ykWvF5P1gV7VmHDMzLonQZ8WmtCzZPJMg+MHRcSxvRSPmVmPWmmoUo95XFLfiOgka6abmTVV4YLRsg6Sl/QrSXMlzShad6Kkf0manpa9ypVTquY5lSxxTpd0DfB74J0Hk0TE1eXDNDOrnzpVPCcA5wKXdFl/ZkT8b6WFVNLnuQbwEtkziwrjPQNw8jSzXiNUl3GeEXG7pOHLWk6p5DkkXWmfwbtJ853jL+uBzcyqUvm964MlTSt6Pz4ixlew3+GSDgSmAUdHxCulNi6VPPsAg1g6aRY4eZpZr6vwgtG8iBhVZdEXAKeQ5bZTyJ4efGipHUolz+ci4uQqAzAzawjRuHvbI+KFd44jXQhMLrdPqeTZOmMCzMxo3KOHJQ2NiOfS28+SdVeWVCp57lKXqMzM6kDU5zEbki4HxpL1jc4BTgDGStqcrNk+mwrm7+gxeUbEy3WI08ysPur06OGI2K+b1RdVW44fPWxmuSByNiWdmVmraJ3U6eRpZjnSQhVPJ08zywvlbko6M7Omc5+nmVmNWid1OnmaWV7UaahSvTh5mlku1GuQfL04eZpZbrTSTPJOnmaWGy2UO508zSwfsmZ762RPJ08zyw3XPM3Mqib3eZqZVcvNdjOzWsjNdjOzmjh5WtN1dnay/ehRrDtsGFdPKvu4FquzwYP6c+wuG7L6iv2ICK5/bC6THn6BHTZcgy9/bBjrrT6QI/7wKDNffKPZobYM39tuLeHcs89ixCabMP/115sdSltasiS48K6neWremwzs18E5+4zkwWdeZ/bLb3LKDTP51o7vb3aILUkt1OfZSnc7WS+ZM2cON/zpOg459KvNDqVtvfzmIp6a9yYACxct4ZlX3mLNlfrxzCtvMefVt5ocXeuSyi+9xTXPNnTs0Udy6mk/YcGC+c0OxYC1V+7PhoNX5MkX3EQvpdWa7S1T85R0sKR1l2H/4ZK+VOU+W0l6RNJTks5WK03Z0iDXXzeZIWsNYcuttmp2KAas0LeD43f/IL+462neXNTZ7HBanCr6X29pmeQJHAzUnDyB4UBVyRO4ADgM2DgteyzD8XPhnrvvYvLkaxix0XAO3P+L3HbrXzjkwC83O6y21KdD/PceG3PrzHnc9Y9Xmh1O66ugyd6b1Z+GJk9J35Y0Iy1HptrhjKLPj5F0oqTPA6OAyyRNlzRQ0mxJP5Y0NS0bpX0mpO0LZSxIL08Hdkj7H1VBbEOBVSLinogI4BJgXP2+fWs65dTT+PvsOTz51GwuuewKxu60M7++5NJmh9WWjtrp/fzzlYVc/dDzzQ4lN1TB0lsa1ucpaSvgEGA02XeaAvy1u20j4g+SDgeOiYhpaX+A1yNia0kHAj8DPlXikMel/T+V9h8BXNnDtmOBYcCconVz0rruvsthZDVU1lt//RIhmFXmw+sM4hMj1mLWS29y3hdGAjDh3mfo16eDb+wwnFUH9uXkT47gH/Pe4PuTn2xytK2h1fo8G3nB6OPAxIh4A0DS1cAOVZZxedHfM6vZMSKeBDbv6fMe+jejh7LGA+MBttpqVLfb5NGYHccyZsexzQ6jLT36/AL2OH9Kt5/dPctN+B61Tu5saPLs7muuxtJdBSuUKSO6eb24UEZKgP27PXj5mucc4H1F694HPFsmHjNronYZ53k7ME7SipJWAj4L/AkYImlNSQNYuhk+H1i5Sxn7Fv29J72eDRQuFe8N9Otu/4h4MiI272F5NSKeA+ZL2iYl4QOBSfX44mbWGG1xwSgiHgAmAFPJ+jt/GRH3ASen95OBJ4p2mQD8vHDBKK0bIGkKcARQuAh0IbCjpKlk/amFwXEPA4slPVTJBaPkG8AvgaeAv5MldzNrUfVInpJ+JWlul4vXa0i6WdLM9Hf1cuU0dJB8RJwBnNFl3dnA2d1sexVwVeF96pI8LyJO6rLdC8A2Rau+m9YvAnapMr5pwMhq9jGz5siuptelajkBOJdshE3BccAtEXG6pOPS+++UKqSVxnmamfWsTuM8I+J24OUuq/cGLk6vL6aCYYste3tmRAxvdgxm1loa2KW5droOQkQ8J2lIuR1aNnmamS1NdD/C8D0GS5pW9H58Gm5YV06eZpYbFV5NnxcRo6os+gVJQ1Otcygwt9wO7vM0s1yo5NbMZWjWXwMclF4fRAXDFp08zSw/6pA9JV1ONm58hKQ5kr5CNjfGrpJmArum9yW52W5muVGPRw9HxH49fFTVUEcnTzPLjda5OdPJ08zyorfnnCvDydPMcqOVJgZx8jSzXBDQ0Tq508nTzHLEydPMrHputpuZ1cDNdjOzWjh5mplVp47zedaFk6eZ5UMvP2ajHCdPM8sNJ08zs6rJzXYzs1q45mlmVqUWu7XdydPM8qPCx3D0CidPM8uNFsqdTp5mlh8tlDudPM0sJzzO08ysesJ9nmZmNWmd1OnkaWY50kIVTydPM8sP32FkZlYD1zzNzKokX203M6uNm+1mZjVwzdPMrAZOnmZmVfN8nmZmVcvuMKpTWdJsYD7QCSyOiFHVluHkaWa5Uedm+04RMa/WnZ08zSw3WqnZ3tHsAMzMKqJ3x3qWWoDBkqYVLYd1U1oAN0m6v4fPy3LN08xyoYrHcMyroA9z+4h4VtIQ4GZJT0TE7dXE45qnmeWGpLJLJSLi2fR3LjAR2LraWJw8zSw3Kmy2lylDK0laufAa2A2YUW0sbrabWW7U6XLR2sDEVEvtC/w2Im6othAnTzPLjzpkz4j4B7DZspbj5GlmuSCgo4Xuz1RENDuGXJH0IvB0s+Oog8FAzQOErWGWl/OyQUSsVc8CJd1A9u9TzryI2KOex+42HifP9iRpWi23pFlj+bzkh6+2m5nVwMnTzKwGTp7ta3yzA7Bu+bzkhPs8zcxq4JqnmVkNnDzNzGrg5GlmVgMnT7McU6XTCFndOXlaxST5dt4WIWkNSYMputvbibR3OXlaSZJGSroAICIWS+rT7JjanaSPAJOAy4GTJO0LEBHhBNp7PFTJeiTpfcBNwPuAqRHxibS+T0R0NjW4NiVpAHAjcDFwG7A9sDfw14g4t4mhtR3XPK2UYcDZEbEK0CnpLwAR0ekmfNMsBmYDt0fELOAa4HxgrKQvNzOwduPkaT2KiCnA1en17sAiSbem94slrdnM+NpRqvE/D/xS0sCIeB24D7gC+KikgW669w4nT+tWoW8zIuYWaplFCfRaSdsAP0kXLawXFNX2vws8CpyXEugC4F5gFLBmuC+uVzh52nsU+jSV6ZNqmYUEuhuwIXA3MDkiloe5J1te0XnoILvC/hNgAXCNpHWBdYHVgBWaF2V7cfK0pRQlzg7gLmAvyJrp6fMNyX6ge0fERDcRG6+bc7JnRPwTOAJ4BDiPrN/ztIh4qomhthVfbbd3dPmRTgLuIPthHgWcmobC7EA2KubOQuJ0M7Fxejgn5wNHRsQP0zaDgL4R8aok+Xz0Dtc8DXjPj/Qash/puWQ/2CWFH2RE3BERd6bX4R9q45Q4J38E3vl3j4gFEfFqeu3z0UucPNtYcZO7aNzm7cBfyX6k1wLXRsSPum5vjVHFOTm1CeFZETfb25SkFYCPR8SfJW1KNqZzFjAGuJKsdjM5Is5K27s52GA+J/ni5NmmJK0OHAB8juyJhJ+OiFmp5jMe+EdEnJa29Y+0F/ic5IvvEmlTEfGKpNnAx4Bb090qkHXlfD8i5oJ/pL3J5yRf3OfZpiR9OCKuAXYHHpX0K0mrpX624osR/pH2Ep+TfHGzvc0Uai2SpgCvRcRukoYB3wMGAL8H9iGr6bzQzFjbhc9JPrnm2X4GAETEaKC/pGsj4l/AScDrwBnAH/0j7VU+JznkmmcbSfNAfhy4odCfJuluYF5EfCa9Xycinne/Wu/wOckv1zzby1BgN2A3SRukdfsAe0iamN7PBfer9SKfk5zy1fblWFFf2khgIdnMO3OA49PnE4HVgf8BbgGIiCVNCrct+JwsP9xsX04V/Uh3AiYAtwKjgUOAl4HvACsD2wAHRMRf3SxsLJ+T5YuT53JM0mhgHNntfHdLGgdcBOxCNhv5MGBgRExrVoztxudk+eHkuRwpmsC4MJnEtcBHyH6s0yNiiaQjgPUi4pjmRdo+fE6WX75gtJyQ1B/4NLCepM+RzTa+D/AE8I2iTV8ju/XPGsznZPnmC0bLiYh4W9JA4HqgD3BURLyZmoU3ATdLmgR8BjineZG2D5+T5ZtrnsuBomnMrgUeBOYDsyWtFBFvArsCbwP7Al+LiEmpCWkN4nOy/PPJWg6kK7gfAg4Hvg38LC2j0iYDgM8DbwL/nfbx8JcG8jlZ/jl55lxRbWUkMJzsB3kZ8Dvge5KOBZ4h61MbB6wtaWjvR9o+fE7ag6+255yk9SLimfR6HDAWmBURZ0n6JNmV3Qci4qbmRdlefE7ag5NnjklaFbgNuCQizkzr/h/w/4EbgQsje6b3O31wHnDdWD4n7cPN9pwp/OCUPQJ4A7IhL/tJOgQgIq4im4lnM2Ctwn6R9H7Eyz+fk/bkoUo5ky5EjANOJPtBTiUb9nKspBXJnrC4Gtmjgmf1UIzVkc9Je3KzPWckrUl28eHoiHg01W6Gkd0bfQDZZBPnRMTEEsVYHfmctCfXPPNnMdnkEYXm36XAeUAn2byQgyLiNU8o0at8TtqQ+zxzJiJeA64CxkgaGRGLyB5JOwzoSJ/7IkQv8jlpT06e+XQl0A84U9LJZLWc69OP1prD56TNuM8zpyStDGwLbEQ2O8/dTQ6p7fmctBcnTzOzGrjZbmZWAydPM7MaOHmamdXAydPMrAZOnmZmNXDytKpJ6pQ0XdIMSb9P92/XWtYESZ9Pr38padMS246VtF0Nx5gt6T3PCOppfZdtFlR5rBMl+UFubcDJ02qxMCI2j4iRZI+S+Hrxh4UnRlYrIr4aEY+V2GQsUHXyNGsEJ09bVncAG6Va4a2Sfgs8IqmPpP+RdJ+khyV9DbLp2ySdK+kxSdcBQwoFSbpN0qj0eg9JD0h6SNItkoaTJemjUq13B0lrSboqHeM+SdunfdeUdJOkByX9AhBlSPqjpPslPSrpsC6f/TTFcouktdK6DSXdkPa5Iz1yw9qIJwaxmknqC+wJ3JBWbQ2MjIhZKQG9FhEfkzQAuEvSTcAWwAiy2dTXBh4DftWl3LWAC4Exqaw1IuJlST8HFkTE/6btfgucGRF3SlqfbLLhTYATgDsj4uQ0c/tSybAHh6ZjDATuk3RVRLwErEQ26/vRkn6Qyj4cGA98PSJmShoNnA/sXMM/o+WUk6fVYqCk6en1HcBFZM3pqUXzVe4GfLTQnwmsCmwMjAEuj4hO4FlJf+mm/G2A2wtlRcTLPcTxCWBTvfOgSlZJt0iOAT6X9r1O0isVfKdvSfpser1eivUlYAnZfeuQzZZ0taRB6fv+vujYAyo4hi1HnDytFgsjYvPiFSmJvFG8CvjPiLixy3Z7AeXuCVYF20DW7bRtRCzsJpaK7zuWNJYsEW+bnqt+G7BCD5tHOu6rXf8NrL24z9Ma5UbgG5L6AUj6oKSVgNuBL6Y+0aHATt3sew+wo6T3p33XSOvnk82bWXATWROatN3m6eXtwP5p3Z7A6mViXRV4JSXOD5HVfAs6yJ5+CfAlsu6A14FZkvZJx5Ckzcocw5YzTp7WKL8k6898QNIM4BdkLZ2JwEzgEeAC4K9dd4yIF8n6Ka+W9BDvNpuvBT5buGAEfAsYlS5IPca7V/1PIptb8wGy7oN/lon1BqCvpIeBU4B7iz57A/iwpPvJ+jRPTuv3B76S4nsU2LuCfxNbjnhWJTOzGrjmaWZWAydPM7MaOHmamdXAydPMrAZOnmZmNXDyNDOrgZOnmVkN/g+D0R1DG0SHtQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize= False,  title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b876735",
   "metadata": {},
   "source": [
    "We will try **RandomForestClassifier** as the sixth model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "562d2fb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "ec1358b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "forestRandom= RandomForestClassifier(n_estimators=100 ,max_depth=3 , criterion=\"entropy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "5b48aa37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(criterion='entropy', max_depth=3)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "forestRandom.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "662c17ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 1, 1, 1], dtype=int64)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat=forestRandom.predict(x_test)\n",
    "yhat[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e7eeb98",
   "metadata": {},
   "source": [
    "# Evaluation RandomForestClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d520e1a2",
   "metadata": {},
   "source": [
    "**accuracy_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "beeb5ae8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9180327868852459"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc_forest=metrics.accuracy_score(y_test,yhat)\n",
    "acc_forest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "646194f1",
   "metadata": {},
   "source": [
    "**jaccard_score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "aeb05882",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8076923076923077"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard_score(y_test, yhat,pos_label=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2f84bbc",
   "metadata": {},
   "source": [
    "**log_loss**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "8f3c8ccd",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.8310996781028566"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_loss(y_test,yhat)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d0ca87c",
   "metadata": {},
   "source": [
    "**f1-score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d832b542",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.84      0.89        25\n",
      "           1       0.90      0.97      0.93        36\n",
      "\n",
      "    accuracy                           0.92        61\n",
      "   macro avg       0.93      0.91      0.91        61\n",
      "weighted avg       0.92      0.92      0.92        61\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "536a02e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[35  1]\n",
      " [ 4 21]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report, confusion_matrix\n",
    "import itertools\n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    Normalization can be applied by setting `normalize=True`.\n",
    "    \"\"\"\n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        print(\"Normalized confusion matrix\")\n",
    "    else:\n",
    "        print('Confusion matrix, without normalization')\n",
    "\n",
    "    print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "print(confusion_matrix(y_test, yhat, labels=[1,0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "7de2b8e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix, without normalization\n",
      "[[35  1]\n",
      " [ 4 21]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU8AAAEmCAYAAADiNhJgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAiPElEQVR4nO3debwcVZ338c/3ZocAIYRACEsUEMUoQTLsS1wGgdEh+MiDgCCLE3QeVARUZFQCqMM4DiCbDAEMCiIwENmXDLIqEBIMkLCISoBASEjYEoiY5ff8UaehudzbW3qr9PfNq17dXcupX9+ifznn1KkqRQRmZladrlYHYGaWR06eZmY1cPI0M6uBk6eZWQ2cPM3MauDkaWZWAydPawhJgyRdL+k1SVetQjkHS7qtnrG1iqTdJD3Z6jisPuRxnp1N0kHAscAHgcXATOBHEXHvKpZ7CPA1YOeIWL6qcbY7SQFsGRF/bnUs1hyueXYwSccCZwI/BjYANgXOA/atQ/GbAX/qhMRZCUl9Wx2D1VlEeOrACVgHWALsX2KdAWTJ9YU0nQkMSMvGAXOB44AFwDzg8LTsZODvwLK0jyOBicClRWWPAgLomz4fBvyVrPb7NHBw0fx7i7bbGXgQeC297ly07E7gVOD3qZzbgGG9fLdC/N8uin88sA/wJ+Bl4MSi9bcH7gNeTeueA/RPy+5O3+WN9H0PKCr/O8CLwK8K89I2m6d9fCx93ghYCIxr9f8bniqbXPPsXDsBA4EpJdb5N2BHYAywDVkC+V7R8g3JkvBIsgR5rqR1I+IkstrsFRExOCIuKhWIpDWBs4C9I2ItsgQ5s4f1hgI3pnXXA04HbpS0XtFqBwGHA8OB/sDxJXa9IdnfYCTwA2AS8EVgO2A34AeS3p/WXQF8ExhG9rf7JPCvABGxe1pnm/R9rygqfyhZLXxC8Y4j4i9kifUySWsAvwAmR8SdJeK1NuLk2bnWAxZG6Wb1wcApEbEgIl4iq1EeUrR8WVq+LCJuIqt1bVVjPCuB0ZIGRcS8iJjdwzr/BDwVEb+KiOURcTnwBPDZonV+ERF/ioilwJVkib83y8j6d5cBvyFLjD+LiMVp/7OBjwJExIyIuD/tdw7w38AeFXynkyLirRTPu0TEJOAp4AFgBNk/VpYTTp6daxEwrExf3EbAM0Wfn0nz3i6jW/J9ExhcbSAR8QZZU/crwDxJN0r6YAXxFGIaWfT5xSriWRQRK9L7QnKbX7R8aWF7SR+QdIOkFyW9TlazHlaibICXIuJvZdaZBIwGzo6It8qsa23EybNz3Qf8jayfrzcvkDU5CzZN82rxBrBG0ecNixdGxK0R8Y9kNbAnyJJKuXgKMT1fY0zV+DlZXFtGxNrAiYDKbFNyKIukwWT9yBcBE1O3hOWEk2eHiojXyPr5zpU0XtIakvpJ2lvST9JqlwPfk7S+pGFp/Utr3OVMYHdJm0paB/huYYGkDST9c+r7fIus+b+ihzJuAj4g6SBJfSUdAGwN3FBjTNVYC3gdWJJqxV/ttnw+8P73bFXaz4AZEfFlsr7c81c5SmsaJ88OFhGnk43x/B7wEvAccDTw27TKD4HpwCPAo8BDaV4t+5oKXJHKmsG7E14X2Vn7F8jOQO9BOhnTrYxFwGfSuovIzpR/JiIW1hJTlY4nOxm1mKxWfEW35ROBSyS9Kun/litM0r7AXmRdFZAdh49JOrhuEVtDeZC8mVkNXPM0M6uBk6eZdRRJAyVNk/SwpNmSTk7zJ0p6XtLMNO1Tshw3282sk0gSsGZELJHUD7gX+AZZH/SSiPhpJeX4elsz6yiR1RiXpI/90lR1LdLJs0rqOyjUf61Wh2FFtv3Qpq0Owbp55pk5LFy4sNw42Kr0WXuziOXvuVDrPWLpS7PJxjAXXBARFxSvI6kP2aiPLYBzI+IBSXsDR0s6lGyUyXER8Upv+3GzvUpdawyPAVuVHYliTfTKg+e0OgTrZpcdxjJjxvS6Js+uNTaIAR88oOx6f/vj2TMiYmwlZUoaQnZ/h6+RDddbSFYLPRUYERFH9BpPJTswM2s5AVL5qQoR8SrZ3bj2ioj5EbEiIlaSjeXdvtS2Tp5mlh/qKj+VKyK7Ym5Iej8I+BTwhKQRRavtB8wqVY77PM0sJwRdfepR0Aiyq8H6kFUgr4yIGyT9StIYsmb7HOCoUoU4eZpZflTZLO9JRDwCbNvD/EN6WL1XTp5mlg+iomZ5szh5mllOVH9CqJGcPM0sP+rT51kXTp5mlhNys93MrGqFcZ5twsnTzHJC0NU+Kat9IjEzK6fLNU8zs+p4qJKZWY3c52lmVq26XZ5ZF06eZpYfbrabmVWphlvONZKTp5nlh2ueZmbVcp+nmVlt3Gw3M6uSx3mamdXCNwYxM6uN+zzNzGrgPk8zsyrJzXYzs9q0Uc2zfdK4mVkJArq6uspOZcuRBkqaJulhSbMlnZzmD5U0VdJT6XXdUuU4eZpZPqjCqby3gE9ExDbAGGAvSTsCJwC3R8SWwO3pc6+cPM0sJ4RUfionMkvSx35pCmBf4JI0/xJgfKlynDzNLDcqbLYPkzS9aJrQvRxJfSTNBBYAUyPiAWCDiJgHkF6Hl4rFJ4zMLDcqqVkCCyNibKkVImIFMEbSEGCKpNHVxuKap5nlQ/36PN8WEa8CdwJ7AfMljQBIrwtKbevkaWa5oDr1eUpaP9U4kTQI+BTwBHAd8KW02peAa0uV42a7meVGJUORKjACuERSH7IK5JURcYOk+4ArJR0JPAvsX6oQJ08zy40K+zxLiohHgG17mL8I+GSl5Th5mlk+1NCn2UhOnmaWG/WoedaLk6eZ5YJQvfo868LJ08zyo30qnk6eZpYTcrPdzKwmTp5mZlVyn6eZWa3ap+Lp5NkJBvTvy/9edAz9+/elb58+TPnfP/LD82/i347ahyM+tzMvvZLdneukc67j1nsfa3G0nemoLx/BzTfdwPrDhzNj5qxWh9Oe3OdpzfbW35ez14SzeGPp3+nbt4vfXXwst/0+S5JnX3oHZ/7q9hZHaId86TC+8q9H8+UjDm11KG2tnZJn+3QgWEO9sfTvAPTr24e+ffsQES2OyIrtutvuDB06tNVhtD11qezULE6eHaKrS9z/mxN49vbT+N39T/DgrGcA+MoXdmfaFd/l/JMOZshag1ocpVlp9birUr20TfKUdJikjVZh+1GSDqpymx9Jek7SkvJr59vKlcGOXziNLT79PcaO3oytNx/BpKvuYevPTmSHL5zGiwtf57RjP9fqMM16VUni7MjkCRwG1Jw8gVFAVckTuB7YfhX2mTuvLVnK3dOfYs+dt2bBy4tZuTKICC6+5veMHb1Zq8MzK6ljkqekYyXNStMxqXY4q2j58ZImSvo8MBa4TNJMSYMkzZH0H+kRodMkbZG2mZzWL5RRqDWeBuyWtv9mJfFFxP2FZ5aszoatO5h1BmdN8oED+vGJHbbiyTnz2XDY2m+vs+8ntuGxv6z2fwrLuXbq82zY2XZJ2wGHAzuQjc56ALirp3Uj4n8kHQ0cHxHT0/YAr0fE9pIOBc4EPlNilyek7T+Ttt8KuKKXdcel2+9X+l0mANlDpPoNrnSztrHhsLWZdMoh9OnqoqtLXD31IW6+ZxYXnXooH91qYyKCZ+a9zNd+eHmrQ+1Yh37xQO65604WLlzI5qM25vs/OJnDjjiy1WG1nXY6297IoUq7AlMi4g0ASdcAu1VZxuVFr2dUs2FEPEn2TOZVFhEXABcAdK0xPHenqWc99QI7Hfgf75l/5Pd/2YJorCe/vNT/cJXVQeM8e/qWQ3h3V8HAMmVED++XF8pQ9pfs3+PO61jzNLPWyy7PbJ/k2cg+z7uB8ZLWkLQmsB9wMzBc0nqSBvDuZvhiYK1uZRxQ9Hpfej8H2C6935fsgfXv2T4inoyIMb1Mr9bnK5pZM0nlp2ZpWPKMiIeAycA0sv7OCyPiQeCU9PkGsifWFUwGzi+cMErzBkh6APgGUDgJNAnYQ9I0sv7UN9L8R4Dlkh6u9ISRpJ9ImgusIWmupIk1fVkza4p2Otve0MszI+J04PRu884Czuph3auBqwuf0x/h3Ig4udt684Edi2Z9N81fRhUPb0rbfBv4djXbmFmLNLlmWU47jfM0M+uVgD59VHYqW460iaQ7JD0uabakb6T5EyU9n1q/MyXtU6qctr0xSESManUMZtZe6tQsXw4cFxEPSVoLmCFpalp2RkT8tJJC2jZ5mpm9S52a7enCmHnp/WJJjwMjqy3HzXYzywVR8QmjYZKmF00Tei1TGgVsS3YSG+BoSY9IuljSuqXicc3TzHKi4nGeCyNibNnSpMFkJ6mPiYjXJf0cOJVsTPmpwH8BR/S2vZOnmeVGvYYiSepHljgvi4hr4O2RPIXlk8iGU/bKzXYzy4cKBshXklvTlYkXAY+n4ZSF+SOKVtsPKPk8FNc8zSwXCn2edbALcAjwqKSZad6JwIGSxpA12+cAR5UqxMnTzHKjHte2R8S99HzvjZuqKcfJ08xyo52uMHLyNLN86KBb0pmZ1U3W59nqKN7h5GlmOdFe9/N08jSz3HCz3cysWm12SzonTzPLBQFdXe1zXY+Tp5nlhmueZmY1cJ+nmVm13OdpZla9dnv0sJOnmeVGVxtVPZ08zSw32ih3OnmaWT4oL9e2Szqb7L52PYqIrzckIjOzXvTJSZ/n9KZFYWZWgTaqePaePCPikuLPktaMiDcaH5KZ2XuJ7Ix7uyh7rZOknSQ9BjyePm8j6byGR2Zm1k2Xyk9Ni6WCdc4EPg0sAoiIh4HdGxiTmdl7KRvnWW5qlorOtkfEc93Ocq1oTDhmZj0T+Rvn+ZyknYGQ1B/4OqkJb2bWTG2UOytqtn8F+H/ASOB5YEz6bGbWVJLKThWUsYmkOyQ9Lmm2pG+k+UMlTZX0VHpdt1Q5ZWueEbEQOLjSL2dm1ghS3cZ5LgeOi4iHJK0FzJA0FTgMuD0iTpN0AnAC8J3eCqnkbPv7JV0v6SVJCyRdK+n99fgGZmbVUAVTORExLyIeSu8Xk3VDjgT2BQpDNC8Bxpcqp5Jm+6+BK4ERwEbAVcDlFWxnZlZXFTbbh0maXjRNKFHeKGBb4AFgg4iYB1mCBYaXiqWSE0aKiF8Vfb5U0tEVbGdmVjeSKm22L4yIsRWUNxi4GjgmIl6v9rr5Ute2D01v70jt/9+QXet+AHBjVXsxM6uDep1tl9SPLHFeFhHXpNnzJY2IiHmSRgALSpVRquY5gyxZFsI9qmhZAKfWFraZWW3qcVclZYVcBDweEacXLboO+BJwWnq9tlQ5pa5tf98qR2lmVifZIPm6FLULcAjwqKSZad6JZEnzSklHAs8C+5cqpKIrjCSNBrYGBhbmRcQvq4/ZzKx29bjCKCLupfcT85+stJyyyVPSScA4suR5E7A3cC/g5GlmTSO11+WZlQxV+jxZNn4xIg4HtgEGNDQqM7MeSOWnZqmk2b40IlZKWi5pbbIzUB4kb2ZNl4vHcBSZLmkIMInsDPwSYFojgzIz605UPM6zKSq5tv1f09vzJd0CrB0RjzQ2LDOzbprcLC+n1CD5j5VaVrg2tNN8ZKtNuPmO08uvaE1zwC8ebHUI1s1fFjXmiT15abb/V4llAXyizrGYmZVUyRnuZik1SP7jzQzEzKwUkZ9HD5uZtZU2yp1OnmaWD9k4zvbJnk6eZpYb7VTzrORO8pL0RUk/SJ83lbR940MzM3tHoc+z3NQslZy8Og/YCTgwfV4MnNuwiMzMetFVwdQslTTbd4iIj0n6I0BEvJIeQWxm1lRt1OVZUfJcJqkP2dhOJK0PrGxoVGZm3VTxGI6mqKSWexYwBRgu6Udkt6P7cUOjMjPrQZfKT81SybXtl0maQXZbOgHjI+LxhkdmZlYku5N8+9Q8K7kZ8qbAm8D1xfMi4tlGBmZm1l0b5c6K+jxv5J0HwQ0E3gc8CXy4gXGZmb2boE8bZc9Kmu0fKf6c7rZ0VC+rm5k1RB0fAFcXVQ+LSrei+4cGxGJmVlI9ThhJuljSAkmziuZNlPS8pJlp2qdcOZX0eR5bHDvwMeCl8iGamdVXna5tnwycw3sfYnlGRPy00kIq6fNcq+j9crI+0Ksr3YGZWT1I0KcOlxBFxN2SRq1qOSWTZxocPzgivrWqOzIzW1UNHqp0tKRDgenAcRHxSslYelsgqW9ErCBrppuZtVThhFEFfZ7DJE0vmiZUUPzPgc2BMcA8Sj9JAyhd85xGljhnSroOuAp4+8EkEXFNBQGZmdVNhRXPhRExtppyI2L+O/vQJOCGcttU0uc5FFhE9syiwnjPAJw8zaxphBo2zlPSiIiYlz7uB8wqtT6UTp7D05n2WbyTNAui5ijNzGpRp2vXJV0OjCNr3s8FTgLGSRpDltvmUMFY9lLJsw8wmHcnzQInTzNrunqcMIqIA3uYfVG15ZRKnvMi4pRqCzQzawSRn2vb2yhMM7P8PHr4k02LwsysDNHcx2yU02vyjIiXmxmImVlJfvSwmVn1RM5uSWdm1i7aJ3U6eZpZjrRRxdPJ08zyQu7zNDOrlvs8zcxq1D6p08nTzPLCQ5XMzKqXm0HyZmbtpsF3kq+Kk6eZ5UYb5U4nTzPLh6zZ3j7Z08nTzHLDNU8zs6rJfZ5mZtVys93MrBZys93MrCZOntZyK1asYO+P78SGIzbil1f8ttXhdJxha/bnmHHvY8igfgRw6+MvccPs+ez8vnU5cLuRbDxkIN/67WP8eeGbrQ61bbTbte3tNGDfmujC889myw98sNVhdKwVK4OL73+Oo/9nFt++9jH2+fBwNhkykGdfWcppU//M7HmLWx1iW1IF/5UtQ7pY0gJJs4rmDZU0VdJT6XXdcuU4eXagF56fy+233cyBhx7e6lA61itLl/HXRVmtcumylcx9ZSlD1+zP3Ff/xvOv/a3F0bUvqfxUgcnAXt3mnQDcHhFbArenzyU5eXagk048nu+d/O90dfnwt4Phg/vz/mFr8KcFS1odSlsrNNvLTeVExN1A92e07Qtckt5fAowvV07b/HokHSZpo1XYfpSkg6rcZjtJj0r6s6Sz1E63bGmQqbfcyLBh6/PRMR9rdSgGDOzbxXc+tQUX3vccS5etbHU4ba6SRrsAhkmaXjRNqKDwDSJiHkB6HV5ug3Y6YXQYMAt4ocbtRwEHAb+uYpufAxOA+4GbyKryN9e4/1yY/sB93HbLjfxu6q289dbfWLz4db424TDOvmByq0PrOH0kTvjHLbjrL4u4f84rrQ6n/VXeLF8YEWMbHE1ja56SjpU0K03HpNphcSft8ZImSvo8MBa4TNJMSYMkzZH0H5KmpWmLtM3ktH6hjEJb5zRgt7T9NyuIbQSwdkTcFxEB/JIKqup5992TfsiM2X/lgUf+xHkX/YpddhvnxNkiX9tjFM+9spTrHp3f6lByQxVMNZqfckIhNywot0HDap6StgMOB3Yg+04PAHf1tG5E/I+ko4HjI2J62h7g9YjYXtKhwJnAZ0rs8oS0/WfS9lsBV/Sy7jhgJDC3aN7cNK+n7zKBrIbKyI03LRGCWWU+tMFgPr7lMOYsepMzPvdhAC59cC79+oh/2Wkz1hnUl+9/+gM8/fKbTLz5Ty2Otj00eKjSdcCXyCphXwKuLbdBI5vtuwJTIuINAEnXALtVWcblRa9nVLNhRDwJjOlteS/9m9FLWRcAFwBss+12Pa6TRzvvugc777pHq8PoSI/PX8K+kx7scdn9c15tbjB5UofcKelysgrUMElzgZPIkuaVko4EngX2L1dOI5NnT19zCO/uKhhYpozo4f3yQhkpAfbvcefla55zgY2L5m1M7f2tZtYElYzjLCciDuxl0SerKaeRfZ53A+MlrSFpTWA/spMxwyWtJ2kA726GLwbW6lbGAUWv96X3c4Dt0vt9gX49bR8RT0bEmF6mV9MZtcWSdkxJ+FAqqKqbWevUaZxnXTSs5hkRD0maDExLsy6MiAclnULW//k08ETRJpOB8yUtBXZK8wZIeoAsyRf+tZgEXCtpGtlg1jfS/EeA5ZIeBiZHRCXN/K+m/Q4iS+yr9Zl2s7xrp8GEDR2qFBGnA6d3m3cWcFYP614NXF34nLokz42Ik7utNx/YsWjWd9P8ZVRZ7U4np0ZXs42ZtUZ2Nr19smc7jfM0M+udb0lXmYgY1eoYzKy9tFHubN/kaWb2bqLnEYat4eRpZrnRRrnTydPM8mEVL7+sOydPM8uPNsqeTp5mlht+9LCZWQ3aJ3U6eZpZXrRZp6eTp5nlhq8wMjOrkoCu9smdTp5mliNOnmZm1XOz3cysBm62m5nVwsnTzKw6vp+nmVktfD9PM7PaOHmamVVNdWu2S5pD9tDIFcDyiBhbbRlOnmaWG3WueX48IhbWurGTp5nlQptd2t7Q57abmdWVpLITMEzS9KJpQg9FBXCbpBm9LC/LNU8zy40Km+0LK+jD3CUiXpA0HJgq6YmIuLuaWFzzNLPcUAVTJSLihfS6AJgCbF9tLE6eZpYPaZxnualsMdKaktYqvAf2BGZVG46b7WaWC4J6PXp4A2BKKqsv8OuIuKXaQpw8zSw36pE6I+KvwDarWo6Tp5nlhq8wMjOrgW8MYmZWA9c8zcyqVOnZ9GZx8jSz3HCz3cysBq55mpnVwMnTzKxq9bufZz04eZpZLmRXGLU6inc4eZpZbjh5mpnVwM12M7NqeZynmVn12u0xHE6eZpYbdbolXV04eZpZbrRR7nTyNLP8aKPc6eRpZjnSRtnTydPMckFAVxu12xURrY4hVyS9BDzT6jjqYBiwsNVB2HusLsdls4hYv54FSrqF7O9TzsKI2Kue++4xHifPziRpegXPtrYm83HJDz962MysBk6eZmY1cPLsXBe0OgDrkY9LTrjP08ysBq55mpnVwMnTzKwGTp5mZjVw8jTLMbXTbYY6jJOnVUySL+dtE5KGShpG0dXeTqTN5eRpJUkaLennABGxXFKfVsfU6SR9BLgWuBw4WdIBABERTqDN46FK1itJGwO3ARsD0yLiU2l+n4hY0dLgOpSkAcCtwCXAncAuwL7AXRFxTgtD6ziueVopI4GzImJtYIWk3wFExAo34VtmOTAHuDsingauA84Dxkn6YisD6zROntariHgAuCa9/zSwTNId6fNySeu1Mr5OlGr8LwIXShoUEa8DDwK/AT4qaZCb7s3h5Gk9KvRtRsSCQi2zKIFeL2lH4CfppIU1QVFt/7vAbODclECXAPcDY4H1wn1xTeHkae9R6NNUpk+qZRYS6J7A5sAfgBsiYnW492TbKzoOXWRn2H8CLAGuk7QRsBEwBBjYuig7i5OnvUtR4uwCfg/sA1kzPS3fnOwHum9ETHETsfF6OCZ7R8SzwDeAR4Fzyfo9/z0i/tzCUDuKz7bb27r9SK8F7iH7YX4T+FEaCrMb2aiYewuJ083ExunlmJwHHBMRP0zrDAb6RsSrkuTj0RyueRrwnh/pdWQ/0nPIfrArCz/IiLgnIu5N78M/1MYpcUx+C7z9d4+IJRHxanrv49EkTp4drLjJXTRu827gLrIf6fXA9RHx4+7rW2NUcUx+1ILwrIib7R1K0kBg14j4X0lbk43pfBrYHbiCrHZzQ0T8LK3v5mCD+Zjki5Nnh5K0LnAI8DmyJxJ+NiKeTjWfC4C/RsS/p3X9I20CH5N88VUiHSoiXpE0B/gH4I50tQpkXTn/FhELwD/SZvIxyRf3eXYoSR+OiOuATwOzJV0saUjqZys+GeEfaZP4mOSLm+0dplBrkfQA8FpE7ClpJHAiMAC4CtifrKYzv5Wxdgofk3xyzbPzDACIiB2A/pKuj4jngZOB14HTgd/6R9pUPiY55JpnB0n3gdwVuKXQnybpD8DCiPjn9HnDiHjR/WrN4WOSX655dpYRwJ7AnpI2S/P2B/aSNCV9XgDuV2siH5Oc8tn21VhRX9poYCnZnXfmAt9Ly6cA6wL/CdwOEBErWxRuR/AxWX242b6aKvqRfhyYDNwB7AAcDrwMfAdYC9gROCQi7nKzsLF8TFYvTp6rMUk7AOPJLuf7g6TxwEXAJ8nuRj4SGBQR01sVY6fxMVl9OHmuRopuYFy4mcT1wEfIfqwzI2KlpG8Am0TE8a2LtHP4mKy+fMJoNSGpP/BZYBNJnyO72/j+wBPAV4tWfY3s0j9rMB+T1ZtPGK0mIuLvkgYBNwF9gG9GxJupWXgbMFXStcA/A2e3LtLO4WOyenPNczVQdBuz64E/AouBOZLWjIg3gX8E/g4cABwVEdemJqQ1iI/J6s8HazWQzuB+EDgaOBY4M01j0yoDgM8DbwLfT9t4+EsD+Zis/pw8c66otjIaGEX2g7wMuBI4UdK3gOfI+tTGAxtIGtH8SDuHj0ln8Nn2nJO0SUQ8l96PB8YBT0fEzyT9E9mZ3Yci4rbWRdlZfEw6g5NnjklaB7gT+GVEnJHm/R/gX4BbgUmRPdP77T44D7huLB+TzuFme84UfnDKHgG8GdmQlwMlHQ4QEVeT3YlnG2D9wnaRND/i1Z+PSWfyUKWcSScixgMTyX6Q08iGvXxL0hpkT1gcQvao4Kd7KcbqyMekM7nZnjOS1iM7+XBcRMxOtZuRZNdGH0J2s4mzI2JKiWKsjnxMOpNrnvmznOzmEYXm36XAucAKsvtCDo6I13xDiabyMelA7vPMmYh4Dbga2F3S6IhYRvZI2pFAV1rukxBN5GPSmZw88+kKoB9whqRTyGo5N6UfrbWGj0mHcZ9nTklaC9gJ2ILs7jx/aHFIHc/HpLM4eZqZ1cDNdjOzGjh5mpnVwMnTzKwGTp5mZjVw8jQzq4GTp1VN0gpJMyXNknRVun671rImS/p8en+hpK1LrDtO0s417GOOpPc8I6i3+d3WWVLlviZK8oPcOoCTp9ViaUSMiYjRZI+S+ErxwsITI6sVEV+OiMdKrDIOqDp5mjWCk6etqnuALVKt8A5JvwYeldRH0n9KelDSI5KOguz2bZLOkfSYpBuB4YWCJN0paWx6v5ekhyQ9LOl2SaPIkvQ3U613N0nrS7o67eNBSbukbdeTdJukP0r6b0CUIem3kmZImi1pQrdl/5ViuV3S+mne5pJuSdvckx65YR3ENwaxmknqC+wN3JJmbQ+MjoinUwJ6LSL+QdIA4PeSbgO2BbYiu5v6BsBjwMXdyl0fmATsnsoaGhEvSzofWBIRP03r/Ro4IyLulbQp2c2GPwScBNwbEaekO7e/Kxn24oi0j0HAg5KujohFwJpkd30/TtIPUtlHAxcAX4mIpyTtAJwHfKKGP6PllJOn1WKQpJnp/T3ARWTN6WlF96vcE/hooT8TWAfYEtgduDwiVgAvSPpdD+XvCNxdKCsiXu4ljk8BW+vtB1WydrpEcnfgc2nbGyW9UsF3+rqk/dL7TVKsi4CVZNetQ3a3pGskDU7f96qifQ+oYB+2GnHytFosjYgxxTNSEnmjeBbwtYi4tdt6+wDlrglWBetA1u20U0Qs7SGWiq87ljSOLBHvlJ6rficwsJfVI+331e5/A+ss7vO0RrkV+KqkfgCSPiBpTeBu4AupT3QE8PEetr0P2EPS+9K2Q9P8xWT3zSy4jawJTVpvTHp7N3Bwmrc3sG6ZWNcBXkmJ84NkNd+CLrKnXwIcRNYd8DrwtKT90z4kaZsy+7DVjJOnNcqFZP2ZD0maBfw3WUtnCvAU8Cjwc+Cu7htGxEtk/ZTXSHqYd5rN1wP7FU4YAV8HxqYTUo/xzln/k8nurfkQWffBs2VivQXoK+kR4FTg/qJlbwAfljSDrE/zlDT/YODIFN9sYN8K/ia2GvFdlczMauCap5lZDZw8zcxq4ORpZlYDJ08zsxo4eZqZ1cDJ08ysBk6eZmY1+P+9ZkvWJTtncQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnf_matrix = confusion_matrix(y_test, yhat, labels=[1,0])\n",
    "np.set_printoptions(precision=2)\n",
    "\n",
    "plt.figure()\n",
    "plot_confusion_matrix(cnf_matrix, classes=['output=1','output=0'],normalize= False,  title='Confusion matrix')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "297d45d2",
   "metadata": {},
   "source": [
    "According to the given outputs, it seems that the two algorithms **RandomForestClassifier** and **LogisticRegression** were able to have the lowest FN value for testing data, which is good.<br> Of course, the percentage of this value was also good for other algorithms.<br>\n",
    "But in general, the **Randomforest algorithm** has given better output to our data and its fp value is also lower,<br> which means a lower error percentage of this algorithm."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}