Lab1-Anomaly Detection in Network Data using GPU-Accelerated XGBoost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/en-us/deep-learning-ai/education/\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anomaly Detection in Network Data using GPU-Accelerated XGBoost\n",
    "\n",
    "- Ananth Sankar, Solutions Architect at NVIDIA.\n",
    "- Eric Harper, Solutions Architect, Global Telecoms at NVIDIA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As network traffic continues to grow exponentially, the number of network attacks and the different kinds of attacks is also growing. The ability to quickly and frequently train machine learning models to detect network intrusions is more important now than ever.\n",
    "\n",
    "In this series of labs, we will learn how to use machine learning and deep learning models for detecting network intrusions in the full KDD99 dataset. The data processing and model training techniques that will be learned in these labs can be applied to many datasets for anomaly detection problems.\n",
    "\n",
    "The KDD99 dataset consists of normal data points and points that have been labeled as Denial of Service (DoS), Remote to User (R2L), User to Root (U2R), and Probing (Probe) by logging network packet information. More information about the dataset can be found at https://kdd.ics.uci.edu/databases/kddcup99/task.html. \n",
    "\n",
    "\n",
    "We'll start off by exploring the dataset and then we will use the NVIDIA RAPIDS library to train GPU-accelerated XGBoost models for network intrusion detection. The RAPIDS suite of software libraries, built on CUDA-X AI, gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. It relies on NVIDIA® CUDA® primitives for low-level compute optimization, but exposes that GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.\n",
    "\n",
    "Notice that we will also be using the Pandas and Scikit-learn packages in this lab. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries that will be needed for the lab\n",
    "\n",
    "import xgboost as xgb\n",
    "import numpy as np\n",
    "from collections import OrderedDict\n",
    "import gc\n",
    "from glob import glob\n",
    "import os\n",
    "import pandas as pd\n",
    "from copy import copy\n",
    "from time import time\n",
    "from sklearn.metrics import roc_auc_score,confusion_matrix,accuracy_score,classification_report,roc_curve\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from timeit import default_timer\n",
    "import matplotlib.pyplot as plt\n",
    "import pickle\n",
    "\n",
    "# Set the seed for numpy\n",
    "np.random.seed(123)\n",
    "\n",
    "# Display all columns of Pandas' dataframes by default\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "data_path = './data/kddcup.data.corrected'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Load the Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin by first importing the KDD99 Dataset using Pandas and then doing some basic data exploration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration</th>\n",
       "      <th>protocol_type</th>\n",
       "      <th>service</th>\n",
       "      <th>flag</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>land</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>urgent</th>\n",
       "      <th>hot</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>num_root</th>\n",
       "      <th>num_file_creations</th>\n",
       "      <th>num_shells</th>\n",
       "      <th>num_access_files</th>\n",
       "      <th>num_outbound_cmds</th>\n",
       "      <th>is_host_login</th>\n",
       "      <th>is_guest_login</th>\n",
       "      <th>count</th>\n",
       "      <th>srv_count</th>\n",
       "      <th>serror_rate</th>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>215</td>\n",
       "      <td>45076</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>162</td>\n",
       "      <td>4528</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>236</td>\n",
       "      <td>1228</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>233</td>\n",
       "      <td>2032</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.33</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>tcp</td>\n",
       "      <td>http</td>\n",
       "      <td>SF</td>\n",
       "      <td>239</td>\n",
       "      <td>486</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   duration protocol_type service flag  src_bytes  dst_bytes  land  \\\n",
       "0         0           tcp    http   SF        215      45076     0   \n",
       "1         0           tcp    http   SF        162       4528     0   \n",
       "2         0           tcp    http   SF        236       1228     0   \n",
       "3         0           tcp    http   SF        233       2032     0   \n",
       "4         0           tcp    http   SF        239        486     0   \n",
       "\n",
       "   wrong_fragment  urgent  hot  num_failed_logins  logged_in  num_compromised  \\\n",
       "0               0       0    0                  0          1                0   \n",
       "1               0       0    0                  0          1                0   \n",
       "2               0       0    0                  0          1                0   \n",
       "3               0       0    0                  0          1                0   \n",
       "4               0       0    0                  0          1                0   \n",
       "\n",
       "   root_shell  su_attempted  num_root  num_file_creations  num_shells  \\\n",
       "0           0             0         0                   0           0   \n",
       "1           0             0         0                   0           0   \n",
       "2           0             0         0                   0           0   \n",
       "3           0             0         0                   0           0   \n",
       "4           0             0         0                   0           0   \n",
       "\n",
       "   num_access_files  num_outbound_cmds  is_host_login  is_guest_login  count  \\\n",
       "0                 0                  0              0               0      1   \n",
       "1                 0                  0              0               0      2   \n",
       "2                 0                  0              0               0      1   \n",
       "3                 0                  0              0               0      2   \n",
       "4                 0                  0              0               0      3   \n",
       "\n",
       "   srv_count  serror_rate  srv_serror_rate  rerror_rate  srv_rerror_rate  \\\n",
       "0          1          0.0              0.0          0.0              0.0   \n",
       "1          2          0.0              0.0          0.0              0.0   \n",
       "2          1          0.0              0.0          0.0              0.0   \n",
       "3          2          0.0              0.0          0.0              0.0   \n",
       "4          3          0.0              0.0          0.0              0.0   \n",
       "\n",
       "   same_srv_rate  diff_srv_rate  srv_diff_host_rate  dst_host_count  \\\n",
       "0            1.0            0.0                 0.0               0   \n",
       "1            1.0            0.0                 0.0               1   \n",
       "2            1.0            0.0                 0.0               2   \n",
       "3            1.0            0.0                 0.0               3   \n",
       "4            1.0            0.0                 0.0               4   \n",
       "\n",
       "   dst_host_srv_count  dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                   0                     0.0                     0.0   \n",
       "1                   1                     1.0                     0.0   \n",
       "2                   2                     1.0                     0.0   \n",
       "3                   3                     1.0                     0.0   \n",
       "4                   4                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.00                          0.0   \n",
       "1                         1.00                          0.0   \n",
       "2                         0.50                          0.0   \n",
       "3                         0.33                          0.0   \n",
       "4                         0.25                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate    label  \n",
       "0                       0.0  normal.  \n",
       "1                       0.0  normal.  \n",
       "2                       0.0  normal.  \n",
       "3                       0.0  normal.  \n",
       "4                       0.0  normal.  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = [\"duration\",\"protocol_type\",\"service\",\"flag\",\"src_bytes\",\"dst_bytes\",\"land\",\"wrong_fragment\",\"urgent\",\"hot\",\"num_failed_logins\",\"logged_in\",\n",
    "             \"num_compromised\",\"root_shell\",\"su_attempted\",\"num_root\",\"num_file_creations\",\"num_shells\",\"num_access_files\",\"num_outbound_cmds\",\n",
    "             \"is_host_login\",\"is_guest_login\",\"count\",\"srv_count\",\"serror_rate\",\"srv_serror_rate\",\"rerror_rate\",\"srv_rerror_rate\",\"same_srv_rate\",\n",
    "             \"diff_srv_rate\",\"srv_diff_host_rate\",\"dst_host_count\",\"dst_host_srv_count\",\"dst_host_same_srv_rate\",\"dst_host_diff_srv_rate\",\n",
    "             \"dst_host_same_src_port_rate\",\"dst_host_srv_diff_host_rate\",\"dst_host_serror_rate\",\"dst_host_srv_serror_rate\",\"dst_host_rerror_rate\",\n",
    "             \"dst_host_srv_rerror_rate\",\"label\"]\n",
    "\n",
    "df =  pd.read_csv(data_path, header=None, names=col_names, index_col=False)\n",
    "\n",
    "# Display the first few rows of the dataset\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each row of the KDD Cup 99 dataset is a network connection, with a total of 41 independent variables and 1 dependent variable. The independent variables can be broadly divided into three groups:\n",
    "\n",
    "1. Basic input features of network connections such as duration, protocol type, and number of bytes from source IP addresses \n",
    "2. Content input features of network connections\n",
    "3. The statistical input features computed over a time window\n",
    "\n",
    "Scroll to the right of the above cell to see all of the features.  The last column on the right is the \"Label\" column. This indicates whether the row is normal or some type of anomalous network traffic.\n",
    "\n",
    "Let's see what types of anomalies are in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>smurf.</th>\n",
       "      <td>2807886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neptune.</th>\n",
       "      <td>1072017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>normal.</th>\n",
       "      <td>972781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>satan.</th>\n",
       "      <td>15892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ipsweep.</th>\n",
       "      <td>12481</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>portsweep.</th>\n",
       "      <td>10413</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nmap.</th>\n",
       "      <td>2316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>back.</th>\n",
       "      <td>2203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>warezclient.</th>\n",
       "      <td>1020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>teardrop.</th>\n",
       "      <td>979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>pod.</th>\n",
       "      <td>264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>guess_passwd.</th>\n",
       "      <td>53</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>buffer_overflow.</th>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>land.</th>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>warezmaster.</th>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imap.</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>rootkit.</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>loadmodule.</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ftp_write.</th>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>multihop.</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>phf.</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>perl.</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>spy.</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    label\n",
       "smurf.            2807886\n",
       "neptune.          1072017\n",
       "normal.            972781\n",
       "satan.              15892\n",
       "ipsweep.            12481\n",
       "portsweep.          10413\n",
       "nmap.                2316\n",
       "back.                2203\n",
       "warezclient.         1020\n",
       "teardrop.             979\n",
       "pod.                  264\n",
       "guess_passwd.          53\n",
       "buffer_overflow.       30\n",
       "land.                  21\n",
       "warezmaster.           20\n",
       "imap.                  12\n",
       "rootkit.               10\n",
       "loadmodule.             9\n",
       "ftp_write.              8\n",
       "multihop.               7\n",
       "phf.                    4\n",
       "perl.                   3\n",
       "spy.                    2"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(df['label'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['back.' 'buffer_overflow.' 'ftp_write.' 'guess_passwd.' 'imap.'\n",
      " 'ipsweep.' 'land.' 'loadmodule.' 'multihop.' 'neptune.' 'nmap.' 'normal.'\n",
      " 'perl.' 'phf.' 'pod.' 'portsweep.' 'rootkit.' 'satan.' 'smurf.' 'spy.'\n",
      " 'teardrop.' 'warezclient.' 'warezmaster.']\n"
     ]
    }
   ],
   "source": [
    "# here we train a label encoder so that we can map our classes to integers later for model training\n",
    "le = LabelEncoder()\n",
    "le.fit(df.label)\n",
    "print(le.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Dataset Modification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the dataset has more anomalies than normal data. Reflect for a moment about the implications of having more anomalies might be. Reflect either here in the notebook, on a piece of paper, or with a peer sitting next to you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll come back to test your hypothesis shortly. \n",
    "\n",
    "<a id='return'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to train an XGBoost model, we have to encode the strings in categorical variables to numeric terms. \n",
    "\n",
    "We will use one-hot encoding to translate each of the 7 categorical features:  `protocol type`, `service`, `flag`, `land`, `logged_in`, `is_host_login`, `is_guest_login` using the Pandas function `get_dummies()`. One-hot encoding will transform the categorical variable into a numerical variable for each category. If a category takes ten values, then that categorical variable will be transformed into 10 numerical variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 One-hot Encode the Categorical Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>land</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>is_host_login</th>\n",
       "      <th>is_guest_login</th>\n",
       "      <th>protocol_type_icmp</th>\n",
       "      <th>protocol_type_tcp</th>\n",
       "      <th>protocol_type_udp</th>\n",
       "      <th>service_IRC</th>\n",
       "      <th>service_X11</th>\n",
       "      <th>service_Z39_50</th>\n",
       "      <th>service_auth</th>\n",
       "      <th>service_bgp</th>\n",
       "      <th>service_courier</th>\n",
       "      <th>service_csnet_ns</th>\n",
       "      <th>service_ctf</th>\n",
       "      <th>service_daytime</th>\n",
       "      <th>service_discard</th>\n",
       "      <th>service_domain</th>\n",
       "      <th>service_domain_u</th>\n",
       "      <th>service_echo</th>\n",
       "      <th>service_eco_i</th>\n",
       "      <th>service_ecr_i</th>\n",
       "      <th>service_efs</th>\n",
       "      <th>service_exec</th>\n",
       "      <th>service_finger</th>\n",
       "      <th>service_ftp</th>\n",
       "      <th>service_ftp_data</th>\n",
       "      <th>service_gopher</th>\n",
       "      <th>service_hostnames</th>\n",
       "      <th>service_http</th>\n",
       "      <th>service_http_443</th>\n",
       "      <th>service_imap4</th>\n",
       "      <th>service_iso_tsap</th>\n",
       "      <th>service_klogin</th>\n",
       "      <th>service_kshell</th>\n",
       "      <th>service_ldap</th>\n",
       "      <th>service_link</th>\n",
       "      <th>service_login</th>\n",
       "      <th>service_mtp</th>\n",
       "      <th>service_name</th>\n",
       "      <th>service_netbios_dgm</th>\n",
       "      <th>service_netbios_ssn</th>\n",
       "      <th>service_netstat</th>\n",
       "      <th>service_nnsp</th>\n",
       "      <th>service_nntp</th>\n",
       "      <th>service_ntp_u</th>\n",
       "      <th>service_other</th>\n",
       "      <th>service_pop_2</th>\n",
       "      <th>service_pop_3</th>\n",
       "      <th>service_printer</th>\n",
       "      <th>service_private</th>\n",
       "      <th>service_red_i</th>\n",
       "      <th>service_remote_job</th>\n",
       "      <th>service_rje</th>\n",
       "      <th>service_shell</th>\n",
       "      <th>service_smtp</th>\n",
       "      <th>service_sql_net</th>\n",
       "      <th>service_ssh</th>\n",
       "      <th>service_sunrpc</th>\n",
       "      <th>service_systat</th>\n",
       "      <th>service_telnet</th>\n",
       "      <th>service_tftp_u</th>\n",
       "      <th>service_tim_i</th>\n",
       "      <th>service_time</th>\n",
       "      <th>service_urh_i</th>\n",
       "      <th>service_urp_i</th>\n",
       "      <th>service_uucp</th>\n",
       "      <th>service_uucp_path</th>\n",
       "      <th>service_vmnet</th>\n",
       "      <th>service_whois</th>\n",
       "      <th>flag_OTH</th>\n",
       "      <th>flag_REJ</th>\n",
       "      <th>flag_RSTO</th>\n",
       "      <th>flag_RSTR</th>\n",
       "      <th>flag_S0</th>\n",
       "      <th>flag_S1</th>\n",
       "      <th>flag_S2</th>\n",
       "      <th>flag_S3</th>\n",
       "      <th>flag_SF</th>\n",
       "      <th>flag_SH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   land  logged_in  is_host_login  is_guest_login  protocol_type_icmp  \\\n",
       "0     0          1              0               0                   0   \n",
       "1     0          1              0               0                   0   \n",
       "2     0          1              0               0                   0   \n",
       "3     0          1              0               0                   0   \n",
       "4     0          1              0               0                   0   \n",
       "\n",
       "   protocol_type_tcp  protocol_type_udp  service_IRC  service_X11  \\\n",
       "0                  1                  0            0            0   \n",
       "1                  1                  0            0            0   \n",
       "2                  1                  0            0            0   \n",
       "3                  1                  0            0            0   \n",
       "4                  1                  0            0            0   \n",
       "\n",
       "   service_Z39_50  service_auth  service_bgp  service_courier  \\\n",
       "0               0             0            0                0   \n",
       "1               0             0            0                0   \n",
       "2               0             0            0                0   \n",
       "3               0             0            0                0   \n",
       "4               0             0            0                0   \n",
       "\n",
       "   service_csnet_ns  service_ctf  service_daytime  service_discard  \\\n",
       "0                 0            0                0                0   \n",
       "1                 0            0                0                0   \n",
       "2                 0            0                0                0   \n",
       "3                 0            0                0                0   \n",
       "4                 0            0                0                0   \n",
       "\n",
       "   service_domain  service_domain_u  service_echo  service_eco_i  \\\n",
       "0               0                 0             0              0   \n",
       "1               0                 0             0              0   \n",
       "2               0                 0             0              0   \n",
       "3               0                 0             0              0   \n",
       "4               0                 0             0              0   \n",
       "\n",
       "   service_ecr_i  service_efs  service_exec  service_finger  service_ftp  \\\n",
       "0              0            0             0               0            0   \n",
       "1              0            0             0               0            0   \n",
       "2              0            0             0               0            0   \n",
       "3              0            0             0               0            0   \n",
       "4              0            0             0               0            0   \n",
       "\n",
       "   service_ftp_data  service_gopher  service_hostnames  service_http  \\\n",
       "0                 0               0                  0             1   \n",
       "1                 0               0                  0             1   \n",
       "2                 0               0                  0             1   \n",
       "3                 0               0                  0             1   \n",
       "4                 0               0                  0             1   \n",
       "\n",
       "   service_http_443  service_imap4  service_iso_tsap  service_klogin  \\\n",
       "0                 0              0                 0               0   \n",
       "1                 0              0                 0               0   \n",
       "2                 0              0                 0               0   \n",
       "3                 0              0                 0               0   \n",
       "4                 0              0                 0               0   \n",
       "\n",
       "   service_kshell  service_ldap  service_link  service_login  service_mtp  \\\n",
       "0               0             0             0              0            0   \n",
       "1               0             0             0              0            0   \n",
       "2               0             0             0              0            0   \n",
       "3               0             0             0              0            0   \n",
       "4               0             0             0              0            0   \n",
       "\n",
       "   service_name  service_netbios_dgm  service_netbios_ssn  service_netstat  \\\n",
       "0             0                    0                    0                0   \n",
       "1             0                    0                    0                0   \n",
       "2             0                    0                    0                0   \n",
       "3             0                    0                    0                0   \n",
       "4             0                    0                    0                0   \n",
       "\n",
       "   service_nnsp  service_nntp  service_ntp_u  service_other  service_pop_2  \\\n",
       "0             0             0              0              0              0   \n",
       "1             0             0              0              0              0   \n",
       "2             0             0              0              0              0   \n",
       "3             0             0              0              0              0   \n",
       "4             0             0              0              0              0   \n",
       "\n",
       "   service_pop_3  service_printer  service_private  service_red_i  \\\n",
       "0              0                0                0              0   \n",
       "1              0                0                0              0   \n",
       "2              0                0                0              0   \n",
       "3              0                0                0              0   \n",
       "4              0                0                0              0   \n",
       "\n",
       "   service_remote_job  service_rje  service_shell  service_smtp  \\\n",
       "0                   0            0              0             0   \n",
       "1                   0            0              0             0   \n",
       "2                   0            0              0             0   \n",
       "3                   0            0              0             0   \n",
       "4                   0            0              0             0   \n",
       "\n",
       "   service_sql_net  service_ssh  service_sunrpc  service_systat  \\\n",
       "0                0            0               0               0   \n",
       "1                0            0               0               0   \n",
       "2                0            0               0               0   \n",
       "3                0            0               0               0   \n",
       "4                0            0               0               0   \n",
       "\n",
       "   service_telnet  service_tftp_u  service_tim_i  service_time  service_urh_i  \\\n",
       "0               0               0              0             0              0   \n",
       "1               0               0              0             0              0   \n",
       "2               0               0              0             0              0   \n",
       "3               0               0              0             0              0   \n",
       "4               0               0              0             0              0   \n",
       "\n",
       "   service_urp_i  service_uucp  service_uucp_path  service_vmnet  \\\n",
       "0              0             0                  0              0   \n",
       "1              0             0                  0              0   \n",
       "2              0             0                  0              0   \n",
       "3              0             0                  0              0   \n",
       "4              0             0                  0              0   \n",
       "\n",
       "   service_whois  flag_OTH  flag_REJ  flag_RSTO  flag_RSTR  flag_S0  flag_S1  \\\n",
       "0              0         0         0          0          0        0        0   \n",
       "1              0         0         0          0          0        0        0   \n",
       "2              0         0         0          0          0        0        0   \n",
       "3              0         0         0          0          0        0        0   \n",
       "4              0         0         0          0          0        0        0   \n",
       "\n",
       "   flag_S2  flag_S3  flag_SF  flag_SH  \n",
       "0        0        0        1        0  \n",
       "1        0        0        1        0  \n",
       "2        0        0        1        0  \n",
       "3        0        0        1        0  \n",
       "4        0        0        1        0  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# capture the categorical variables and one-hot encode them\n",
    "cat_vars = ['protocol_type', 'service', 'flag', 'land', 'logged_in','is_host_login', 'is_guest_login']\n",
    "\n",
    "# find unique labels for each category\n",
    "cat_data = pd.get_dummies(df[cat_vars])\n",
    "\n",
    "# check that the categorical variables were created correctly\n",
    "cat_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Scroll to the right to notice the new categories that were created. For example, the categorical variable \"protocol_type\" is split into three categories, protocol_type_icmp, protocol_type_tcp and protocol_type_udp. Now that the one hot encoding of the categorical data is done, we need to merge the numerical data from the original data. \n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <th>num_access_files</th>\n",
       "      <th>count</th>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>hot</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>duration</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>srv_count</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>urgent</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>serror_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>num_file_creations</th>\n",
       "      <th>num_root</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>num_outbound_cmds</th>\n",
       "      <th>num_shells</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>215</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>45076</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>162</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4528</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>236</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1228</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.33</td>\n",
       "      <td>233</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2032</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.25</td>\n",
       "      <td>239</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>486</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dst_host_srv_diff_host_rate  srv_rerror_rate  num_access_files  count  \\\n",
       "0                          0.0              0.0                 0      1   \n",
       "1                          0.0              0.0                 0      2   \n",
       "2                          0.0              0.0                 0      1   \n",
       "3                          0.0              0.0                 0      2   \n",
       "4                          0.0              0.0                 0      3   \n",
       "\n",
       "   srv_serror_rate  dst_host_count  num_failed_logins  hot  \\\n",
       "0              0.0               0                  0    0   \n",
       "1              0.0               1                  0    0   \n",
       "2              0.0               2                  0    0   \n",
       "3              0.0               3                  0    0   \n",
       "4              0.0               4                  0    0   \n",
       "\n",
       "   dst_host_same_srv_rate  dst_host_srv_count  su_attempted  \\\n",
       "0                     0.0                   0             0   \n",
       "1                     1.0                   1             0   \n",
       "2                     1.0                   2             0   \n",
       "3                     1.0                   3             0   \n",
       "4                     1.0                   4             0   \n",
       "\n",
       "   dst_host_diff_srv_rate  dst_host_same_src_port_rate  src_bytes  duration  \\\n",
       "0                     0.0                         0.00        215         0   \n",
       "1                     0.0                         1.00        162         0   \n",
       "2                     0.0                         0.50        236         0   \n",
       "3                     0.0                         0.33        233         0   \n",
       "4                     0.0                         0.25        239         0   \n",
       "\n",
       "   srv_diff_host_rate  srv_count  dst_host_srv_serror_rate  urgent  \\\n",
       "0                 0.0          1                       0.0       0   \n",
       "1                 0.0          2                       0.0       0   \n",
       "2                 0.0          1                       0.0       0   \n",
       "3                 0.0          2                       0.0       0   \n",
       "4                 0.0          3                       0.0       0   \n",
       "\n",
       "   wrong_fragment  rerror_rate  dst_host_srv_rerror_rate  num_compromised  \\\n",
       "0               0          0.0                       0.0                0   \n",
       "1               0          0.0                       0.0                0   \n",
       "2               0          0.0                       0.0                0   \n",
       "3               0          0.0                       0.0                0   \n",
       "4               0          0.0                       0.0                0   \n",
       "\n",
       "   dst_bytes  serror_rate  diff_srv_rate  num_file_creations  num_root  \\\n",
       "0      45076          0.0            0.0                   0         0   \n",
       "1       4528          0.0            0.0                   0         0   \n",
       "2       1228          0.0            0.0                   0         0   \n",
       "3       2032          0.0            0.0                   0         0   \n",
       "4        486          0.0            0.0                   0         0   \n",
       "\n",
       "   root_shell  dst_host_serror_rate  same_srv_rate  dst_host_rerror_rate  \\\n",
       "0           0                   0.0            1.0                   0.0   \n",
       "1           0                   0.0            1.0                   0.0   \n",
       "2           0                   0.0            1.0                   0.0   \n",
       "3           0                   0.0            1.0                   0.0   \n",
       "4           0                   0.0            1.0                   0.0   \n",
       "\n",
       "   num_outbound_cmds  num_shells  \n",
       "0                  0           0  \n",
       "1                  0           0  \n",
       "2                  0           0  \n",
       "3                  0           0  \n",
       "4                  0           0  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_vars = list(set(df.columns.values.tolist()) - set(cat_vars))\n",
    "numeric_vars.remove('label')\n",
    "numeric_data = df[numeric_vars].copy()\n",
    "\n",
    "# check that the numeric data has been captured accurately\n",
    "numeric_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(982508, 80)\n",
      "(982508, 34)\n",
      "(982508, 114)\n"
     ]
    }
   ],
   "source": [
    "# concat numeric and the encoded categorical variables\n",
    "numeric_cat_data = pd.concat([numeric_data, cat_data], axis=1)\n",
    "\n",
    "# here we do a quick sanity check that the data has been concatenated correctly by checking the dimension of the vectors\n",
    "print(cat_data.shape)\n",
    "print(numeric_data.shape)\n",
    "print(numeric_cat_data.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "Now let's split the data into training set and test set in the ratio of 75:25. We will be using <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html\">LabelEncoder</a>, <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler.fit_transform\">fit_transform</a> and <a href=\"https://scikit-https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html\">train_test_split</a> from scikit-learn.\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# capture the labels\n",
    "labels = df['label'].copy()\n",
    "\n",
    "# convert labels to integers\n",
    "integer_labels = le.transform(labels)\n",
    "\n",
    "# split data into test and train\n",
    "x_train, x_test, y_train, y_test = train_test_split(numeric_cat_data,\n",
    "                                                    integer_labels,\n",
    "                                                    test_size=.25, \n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "We can inspect the dimension of the testing set and the training set to confirm that the data has been split correctly. We will also save the dataset to be used in the later portion of this lab and in lab-2 by \"pickling\" the data. Pickling allows us to save a python object as a binary file.\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(736881, 114)\n",
      "(736881,)\n",
      "(245627, 114)\n",
      "(245627,)\n"
     ]
    }
   ],
   "source": [
    "# check that the dimensions of our train and test sets are okay\n",
    "print(x_train.shape)\n",
    "print(y_train.shape)\n",
    "print(x_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the datasets for later use\n",
    "preprocessed_data = {\n",
    "    'x_train':x_train,\n",
    "    'y_train':y_train,\n",
    "    'x_test':x_test,\n",
    "    'y_test':y_test,\n",
    "    'le':le\n",
    "}\n",
    "\n",
    "# pickle the preprocessed_data\n",
    "path = 'preprocessed_data_full.pkl'\n",
    "out = open(path, 'wb')\n",
    "pickle.dump(preprocessed_data, out)\n",
    "out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will approach this anomaly detection problem in two ways:\n",
    "\n",
    "1. Implementing binary classification where we will label the Normal frames as '0' and Anomalous frames as '1' and use a 'one vs all' approach to detect an anomalous frame\n",
    "2. Implementing multi-class classification where we will be able to detect the *type* of anomaly as well using our original y_train and y_test labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Training a Binary Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, you will convert to binary labels. Remember what you do here as you will implement the multi-class solution yourself later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of anomalies in y_train:  7261\n",
      "Number of anomalies in y_test:   2466\n"
     ]
    }
   ],
   "source": [
    "def convert_label_to_binary(label_encoder, labels):\n",
    "    normal_idx = np.where(label_encoder.classes_ == 'normal.')[0][0]\n",
    "    my_labels = labels.copy()\n",
    "    my_labels[my_labels != normal_idx] = 1 \n",
    "    my_labels[my_labels == normal_idx] = 0\n",
    "    return my_labels\n",
    "\n",
    "binary_y_train = convert_label_to_binary(le, y_train)\n",
    "binary_y_test = convert_label_to_binary(le, y_test)\n",
    "\n",
    "# check how many anomalies are in our labels\n",
    "print('Number of anomalies in y_train: ', binary_y_train.sum())\n",
    "print('Number of anomalies in y_test:  ', binary_y_test.sum())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the anomalies are optimally split between the training set and the testing set. Now that we have prepared the data, we are ready to train an XGBoost model to detect anomalies. \n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Training GPU XGBoost Models with RAPIDS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBoost is widely known today as a \"go to\" model when working with structured data such as the KDD99 dataset. In fact, there are a disproportionately large number of XGBoost-based winning entries of Kaggle competitions.\n",
    "\n",
    "What is XGBoost and why is it so popular?\n",
    "\n",
    "Traditionally, in tree-based ensemble methods such as Random Forests, we train each tree independently. The predictions of multiple trees are summed to obtain the final score. The figure below classifies whether someone would like to play computer games or not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/twocart.png\" alt=\"Alt text that describes the graphic\" title=\"Title text\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\" style=\"color:gray;\">\n",
    "    <em>Image courtesy <a href=\"https://goo.gl/eTxVtA\">goo.gl/eTxVtA<a/></em>\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of XGBoost, an implementation of Gradient Boosted Decision Trees, we repeatedly build new models and combine them into an ensemble. Unlike Random Forests, we build trees one at a time, where each new tree helps to correct errors made by previously trained tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://drive.google.com/uc?id=1WDN-kDMOsIhY3vWy8UBOR0qgBRHqbf-2\" alt=\"Alt text that describes the graphic\" title=\"Title text\" width=700px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The objective of the XGBoost model is as follows:*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=images/xgboost-objective.PNG alt=\"Alt text that describes the graphic\" title=\"Title text\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss function controls the predictive power of the model, and the XGBoost regularization term ensures simplicity and manages overfitting.\n",
    "\n",
    "Since Boosting focuses step by step on the difficult examples it gives it a nice strategy to deal with unbalanced datasets by strengthening the impact of the positive class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Why Use GPU to Accelerate XGBoost?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generally, GPU accelerated XGBoost models will train much faster. It's also extremely easy.  In order to use the GPU version of XGBoost, only one parameter in the code has to be set (see below): ```{'tree_method': 'gpu_hist'```} Also, the environment must have the GPU version of XGBoost: https://xgboost.readthedocs.io/en/latest/gpu/index.html. The easiest way to use GPU XGBoost is to use NVIDIA's RAPIDS container: https://hub.docker.com/r/rapidsai/rapidsai/. \n",
    "\n",
    "Machine learning models using XGBoost on large datasets can take several hours to train and achieve the best predictions. Oftentimes, models have to be trained for 1000s of iterations, and usually many different combinations of hyperparameters must be tested.\n",
    "\n",
    "The more models that can be trained, the better the accuracy will be."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=images/moremodels.PNG alt=\"Alt text that describes the graphic\" title=\"Title text\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fortunately, the XGBoost algorithm is highly parallelizable by virtue of requiring scans across gradient values and using these partial sums to evaluate the quality of splits at every possible split in the training set. By utilizing fast parallel prefix operations to scan through all possible splits as well as parallel radix sorting to repartition data, the GPU accelerated version builds a decision tree for a given boosting iteration one level at a time, processing the entire dataset concurrently on the GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperparameters can have a huge impact on the performance of XGBoost models.\n",
    "Some of the more important ones are listed below. See [here](https://xgboost.readthedocs.io/en/latest/parameter.html) for a description of all parameters and see [here](https://xgboost.readthedocs.io/en/latest/tutorials/param_tuning.html) for a guide to tuning XGBoost models. Feel free to try out different combinations: \n",
    "<ul>\n",
    "<li>\n",
    "<b>objective:</b>\n",
    "Depending on whether we treat our problem as a binary classification problem or as a multi-class classification problem this parameter would be either binary:logistic or multi:softmax respectively.\n",
    "    </li>\n",
    "<li>\n",
    "<b>tree_method:</b>\n",
    "gpu_exact to use the GPU optimized exact greedy algorithm and gpu_hist to use the histogram optimized approximate greedy algorithm. For GPU accelerated XGBoost it is best to use gpu_hist.\n",
    "    </li>\n",
    "<li>\n",
    "<b>n_gpus:</b>\n",
    "The number of GPUs to use for training.  Note, if using GPU DASK, n_gpus should be set to 1.\n",
    "    </li>\n",
    "    \n",
    "<li>\n",
    "<b>max_depth:</b>\n",
    "Increasing this value will give the model more capacity to learn and it will also be more likely to overfit. If using a large max_depth value, make sure to increase the regularization.\n",
    "</li>\n",
    "\n",
    "<li>\n",
    "<b>learning_rate:</b>\n",
    "Used to control the weighting of new trees added to the model. If using a low value here, then the number of rounds of training should be high.\n",
    "</li>\n",
    "\n",
    "<li>\n",
    "<b>num_class:</b>\n",
    "    Indicates the number of classes in the label column. For a binary classification problem this should be 2 and the number of classes in the case of a multi-class problem.\n",
    "</li>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'num_rounds':        10,\n",
    "    'max_depth':         8,\n",
    "    'max_leaves':        2**8,\n",
    "    'alpha':             0.9,\n",
    "    'eta':               0.1,\n",
    "    'gamma':             0.1,\n",
    "    'learning_rate':     0.1,\n",
    "    'subsample':         1,\n",
    "    'reg_lambda':        1,\n",
    "    'scale_pos_weight':  2,\n",
    "    'tree_method':       'gpu_hist',\n",
    "    'n_gpus':            1,\n",
    "    'objective':         'binary:logistic',\n",
    "    'verbose':           True\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Binary Classification Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us look at the feature and label columns of our training set before we start training our XGBoost model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>srv_rerror_rate</th>\n",
       "      <th>num_access_files</th>\n",
       "      <th>count</th>\n",
       "      <th>srv_serror_rate</th>\n",
       "      <th>dst_host_count</th>\n",
       "      <th>num_failed_logins</th>\n",
       "      <th>hot</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>su_attempted</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>src_bytes</th>\n",
       "      <th>duration</th>\n",
       "      <th>srv_diff_host_rate</th>\n",
       "      <th>srv_count</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>urgent</th>\n",
       "      <th>wrong_fragment</th>\n",
       "      <th>rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>num_compromised</th>\n",
       "      <th>dst_bytes</th>\n",
       "      <th>serror_rate</th>\n",
       "      <th>diff_srv_rate</th>\n",
       "      <th>num_file_creations</th>\n",
       "      <th>num_root</th>\n",
       "      <th>root_shell</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>same_srv_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>num_outbound_cmds</th>\n",
       "      <th>num_shells</th>\n",
       "      <th>land</th>\n",
       "      <th>logged_in</th>\n",
       "      <th>is_host_login</th>\n",
       "      <th>is_guest_login</th>\n",
       "      <th>protocol_type_icmp</th>\n",
       "      <th>protocol_type_tcp</th>\n",
       "      <th>protocol_type_udp</th>\n",
       "      <th>service_IRC</th>\n",
       "      <th>service_X11</th>\n",
       "      <th>service_Z39_50</th>\n",
       "      <th>service_auth</th>\n",
       "      <th>service_bgp</th>\n",
       "      <th>service_courier</th>\n",
       "      <th>service_csnet_ns</th>\n",
       "      <th>service_ctf</th>\n",
       "      <th>service_daytime</th>\n",
       "      <th>service_discard</th>\n",
       "      <th>service_domain</th>\n",
       "      <th>service_domain_u</th>\n",
       "      <th>service_echo</th>\n",
       "      <th>service_eco_i</th>\n",
       "      <th>service_ecr_i</th>\n",
       "      <th>service_efs</th>\n",
       "      <th>service_exec</th>\n",
       "      <th>service_finger</th>\n",
       "      <th>service_ftp</th>\n",
       "      <th>service_ftp_data</th>\n",
       "      <th>service_gopher</th>\n",
       "      <th>service_hostnames</th>\n",
       "      <th>service_http</th>\n",
       "      <th>service_http_443</th>\n",
       "      <th>service_imap4</th>\n",
       "      <th>service_iso_tsap</th>\n",
       "      <th>service_klogin</th>\n",
       "      <th>service_kshell</th>\n",
       "      <th>service_ldap</th>\n",
       "      <th>service_link</th>\n",
       "      <th>service_login</th>\n",
       "      <th>service_mtp</th>\n",
       "      <th>service_name</th>\n",
       "      <th>service_netbios_dgm</th>\n",
       "      <th>service_netbios_ssn</th>\n",
       "      <th>service_netstat</th>\n",
       "      <th>service_nnsp</th>\n",
       "      <th>service_nntp</th>\n",
       "      <th>service_ntp_u</th>\n",
       "      <th>service_other</th>\n",
       "      <th>service_pop_2</th>\n",
       "      <th>service_pop_3</th>\n",
       "      <th>service_printer</th>\n",
       "      <th>service_private</th>\n",
       "      <th>service_red_i</th>\n",
       "      <th>service_remote_job</th>\n",
       "      <th>service_rje</th>\n",
       "      <th>service_shell</th>\n",
       "      <th>service_smtp</th>\n",
       "      <th>service_sql_net</th>\n",
       "      <th>service_ssh</th>\n",
       "      <th>service_sunrpc</th>\n",
       "      <th>service_systat</th>\n",
       "      <th>service_telnet</th>\n",
       "      <th>service_tftp_u</th>\n",
       "      <th>service_tim_i</th>\n",
       "      <th>service_time</th>\n",
       "      <th>service_urh_i</th>\n",
       "      <th>service_urp_i</th>\n",
       "      <th>service_uucp</th>\n",
       "      <th>service_uucp_path</th>\n",
       "      <th>service_vmnet</th>\n",
       "      <th>service_whois</th>\n",
       "      <th>flag_OTH</th>\n",
       "      <th>flag_REJ</th>\n",
       "      <th>flag_RSTO</th>\n",
       "      <th>flag_RSTR</th>\n",
       "      <th>flag_S0</th>\n",
       "      <th>flag_S1</th>\n",
       "      <th>flag_S2</th>\n",
       "      <th>flag_S3</th>\n",
       "      <th>flag_SF</th>\n",
       "      <th>flag_SH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>36292</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>220</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>220</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>282</td>\n",
       "      <td>0</td>\n",
       "      <td>0.50</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1227</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>811334</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.63</td>\n",
       "      <td>90</td>\n",
       "      <td>0</td>\n",
       "      <td>0.04</td>\n",
       "      <td>0.01</td>\n",
       "      <td>757</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>331</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4496690</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.87</td>\n",
       "      <td>222</td>\n",
       "      <td>0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>105</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36337</th>\n",
       "      <td>0.01</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>255</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.33</td>\n",
       "      <td>226</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4895936</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>0.0</td>\n",
       "      <td>255</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>255</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>328</td>\n",
       "      <td>0</td>\n",
       "      <td>0.12</td>\n",
       "      <td>16</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>471</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         dst_host_srv_diff_host_rate  srv_rerror_rate  num_access_files  \\\n",
       "36292                           0.00              0.0                 0   \n",
       "811334                          0.00              0.0                 0   \n",
       "4496690                         0.00              0.0                 0   \n",
       "36337                           0.01              0.0                 0   \n",
       "4895936                         0.00              0.0                 0   \n",
       "\n",
       "         count  srv_serror_rate  dst_host_count  num_failed_logins  hot  \\\n",
       "36292        4              0.0             220                  0    0   \n",
       "811334       1              0.0             142                  0    0   \n",
       "4496690      1              0.0             255                  0    0   \n",
       "36337        3              0.0               3                  0    0   \n",
       "4895936     13              0.0             255                  0    0   \n",
       "\n",
       "         dst_host_same_srv_rate  dst_host_srv_count  su_attempted  \\\n",
       "36292                      1.00                 220             0   \n",
       "811334                     0.63                  90             0   \n",
       "4496690                    0.87                 222             0   \n",
       "36337                      1.00                 255             0   \n",
       "4895936                    1.00                 255             0   \n",
       "\n",
       "         dst_host_diff_srv_rate  dst_host_same_src_port_rate  src_bytes  \\\n",
       "36292                      0.00                         0.00        282   \n",
       "811334                     0.04                         0.01        757   \n",
       "4496690                    0.01                         0.00        105   \n",
       "36337                      0.00                         0.33        226   \n",
       "4895936                    0.00                         0.00        328   \n",
       "\n",
       "         duration  srv_diff_host_rate  srv_count  dst_host_srv_serror_rate  \\\n",
       "36292           0                0.50          6                       0.0   \n",
       "811334          0                0.00          1                       0.0   \n",
       "4496690         0                0.00          1                       0.0   \n",
       "36337           0                0.00          3                       0.0   \n",
       "4895936         0                0.12         16                       0.0   \n",
       "\n",
       "         urgent  wrong_fragment  rerror_rate  dst_host_srv_rerror_rate  \\\n",
       "36292         0               0          0.0                       0.0   \n",
       "811334        0               0          0.0                       0.0   \n",
       "4496690       0               0          0.0                       0.0   \n",
       "36337         0               0          0.0                       0.0   \n",
       "4895936       0               0          0.0                       0.0   \n",
       "\n",
       "         num_compromised  dst_bytes  serror_rate  diff_srv_rate  \\\n",
       "36292                  0       1227          0.0            0.0   \n",
       "811334                 0        331          0.0            0.0   \n",
       "4496690                0          0          0.0            0.0   \n",
       "36337                  0        729          0.0            0.0   \n",
       "4895936                0        471          0.0            0.0   \n",
       "\n",
       "         num_file_creations  num_root  root_shell  dst_host_serror_rate  \\\n",
       "36292                     0         0           0                   0.0   \n",
       "811334                    0         0           0                   0.0   \n",
       "4496690                   0         0           0                   0.0   \n",
       "36337                     0         0           0                   0.0   \n",
       "4895936                   0         0           0                   0.0   \n",
       "\n",
       "         same_srv_rate  dst_host_rerror_rate  num_outbound_cmds  num_shells  \\\n",
       "36292              1.0                   0.0                  0           0   \n",
       "811334             1.0                   0.0                  0           0   \n",
       "4496690            1.0                   0.0                  0           0   \n",
       "36337              1.0                   0.0                  0           0   \n",
       "4895936            1.0                   0.0                  0           0   \n",
       "\n",
       "         land  logged_in  is_host_login  is_guest_login  protocol_type_icmp  \\\n",
       "36292       0          1              0               0                   0   \n",
       "811334      0          1              0               0                   0   \n",
       "4496690     0          0              0               0                   0   \n",
       "36337       0          1              0               0                   0   \n",
       "4895936     0          1              0               0                   0   \n",
       "\n",
       "         protocol_type_tcp  protocol_type_udp  service_IRC  service_X11  \\\n",
       "36292                    1                  0            0            0   \n",
       "811334                   1                  0            0            0   \n",
       "4496690                  0                  1            0            0   \n",
       "36337                    1                  0            0            0   \n",
       "4895936                  1                  0            0            0   \n",
       "\n",
       "         service_Z39_50  service_auth  service_bgp  service_courier  \\\n",
       "36292                 0             0            0                0   \n",
       "811334                0             0            0                0   \n",
       "4496690               0             0            0                0   \n",
       "36337                 0             0            0                0   \n",
       "4895936               0             0            0                0   \n",
       "\n",
       "         service_csnet_ns  service_ctf  service_daytime  service_discard  \\\n",
       "36292                   0            0                0                0   \n",
       "811334                  0            0                0                0   \n",
       "4496690                 0            0                0                0   \n",
       "36337                   0            0                0                0   \n",
       "4895936                 0            0                0                0   \n",
       "\n",
       "         service_domain  service_domain_u  service_echo  service_eco_i  \\\n",
       "36292                 0                 0             0              0   \n",
       "811334                0                 0             0              0   \n",
       "4496690               0                 0             0              0   \n",
       "36337                 0                 0             0              0   \n",
       "4895936               0                 0             0              0   \n",
       "\n",
       "         service_ecr_i  service_efs  service_exec  service_finger  \\\n",
       "36292                0            0             0               0   \n",
       "811334               0            0             0               0   \n",
       "4496690              0            0             0               0   \n",
       "36337                0            0             0               0   \n",
       "4895936              0            0             0               0   \n",
       "\n",
       "         service_ftp  service_ftp_data  service_gopher  service_hostnames  \\\n",
       "36292              0                 0               0                  0   \n",
       "811334             0                 0               0                  0   \n",
       "4496690            0                 0               0                  0   \n",
       "36337              0                 0               0                  0   \n",
       "4895936            0                 0               0                  0   \n",
       "\n",
       "         service_http  service_http_443  service_imap4  service_iso_tsap  \\\n",
       "36292               1                 0              0                 0   \n",
       "811334              0                 0              0                 0   \n",
       "4496690             0                 0              0                 0   \n",
       "36337               1                 0              0                 0   \n",
       "4895936             1                 0              0                 0   \n",
       "\n",
       "         service_klogin  service_kshell  service_ldap  service_link  \\\n",
       "36292                 0               0             0             0   \n",
       "811334                0               0             0             0   \n",
       "4496690               0               0             0             0   \n",
       "36337                 0               0             0             0   \n",
       "4895936               0               0             0             0   \n",
       "\n",
       "         service_login  service_mtp  service_name  service_netbios_dgm  \\\n",
       "36292                0            0             0                    0   \n",
       "811334               0            0             0                    0   \n",
       "4496690              0            0             0                    0   \n",
       "36337                0            0             0                    0   \n",
       "4895936              0            0             0                    0   \n",
       "\n",
       "         service_netbios_ssn  service_netstat  service_nnsp  service_nntp  \\\n",
       "36292                      0                0             0             0   \n",
       "811334                     0                0             0             0   \n",
       "4496690                    0                0             0             0   \n",
       "36337                      0                0             0             0   \n",
       "4895936                    0                0             0             0   \n",
       "\n",
       "         service_ntp_u  service_other  service_pop_2  service_pop_3  \\\n",
       "36292                0              0              0              0   \n",
       "811334               0              0              0              0   \n",
       "4496690              0              0              0              0   \n",
       "36337                0              0              0              0   \n",
       "4895936              0              0              0              0   \n",
       "\n",
       "         service_printer  service_private  service_red_i  service_remote_job  \\\n",
       "36292                  0                0              0                   0   \n",
       "811334                 0                0              0                   0   \n",
       "4496690                0                1              0                   0   \n",
       "36337                  0                0              0                   0   \n",
       "4895936                0                0              0                   0   \n",
       "\n",
       "         service_rje  service_shell  service_smtp  service_sql_net  \\\n",
       "36292              0              0             0                0   \n",
       "811334             0              0             1                0   \n",
       "4496690            0              0             0                0   \n",
       "36337              0              0             0                0   \n",
       "4895936            0              0             0                0   \n",
       "\n",
       "         service_ssh  service_sunrpc  service_systat  service_telnet  \\\n",
       "36292              0               0               0               0   \n",
       "811334             0               0               0               0   \n",
       "4496690            0               0               0               0   \n",
       "36337              0               0               0               0   \n",
       "4895936            0               0               0               0   \n",
       "\n",
       "         service_tftp_u  service_tim_i  service_time  service_urh_i  \\\n",
       "36292                 0              0             0              0   \n",
       "811334                0              0             0              0   \n",
       "4496690               0              0             0              0   \n",
       "36337                 0              0             0              0   \n",
       "4895936               0              0             0              0   \n",
       "\n",
       "         service_urp_i  service_uucp  service_uucp_path  service_vmnet  \\\n",
       "36292                0             0                  0              0   \n",
       "811334               0             0                  0              0   \n",
       "4496690              0             0                  0              0   \n",
       "36337                0             0                  0              0   \n",
       "4895936              0             0                  0              0   \n",
       "\n",
       "         service_whois  flag_OTH  flag_REJ  flag_RSTO  flag_RSTR  flag_S0  \\\n",
       "36292                0         0         0          0          0        0   \n",
       "811334               0         0         0          0          0        0   \n",
       "4496690              0         0         0          0          0        0   \n",
       "36337                0         0         0          0          0        0   \n",
       "4895936              0         0         0          0          0        0   \n",
       "\n",
       "         flag_S1  flag_S2  flag_S3  flag_SF  flag_SH  \n",
       "36292          0        0        0        1        0  \n",
       "811334         0        0        0        1        0  \n",
       "4496690        0        0        0        1        0  \n",
       "36337          0        0        0        1        0  \n",
       "4895936        0        0        0        1        0  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,\n",
       "       11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,\n",
       "       11, 11, 11, 11, 11, 11, 11, 11, 18, 11, 11, 11, 11, 11, 11, 11, 11,\n",
       "       11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 18, 11, 11, 11,\n",
       "       11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,\n",
       "       11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.38 s, sys: 282 ms, total: 3.66 s\n",
      "Wall time: 612 ms\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "dtrain = xgb.DMatrix(x_train, label=binary_y_train)\n",
    "dtest = xgb.DMatrix(x_test, label=binary_y_test)\n",
    "evals = [(dtest, 'test',), (dtrain, 'train')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_rounds = params['num_rounds']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10:12:07] WARNING: ../src/learner.cc:576: \n",
      "Parameters: { \"n_gpus\", \"num_rounds\", \"verbose\" } might not be used.\n",
      "\n",
      "  This could be a false alarm, with some parameters getting used by language bindings but\n",
      "  then being mistakenly passed down to XGBoost core, or some parameter actually being used\n",
      "  but getting flagged wrongly here. Please open an issue if you find any such cases.\n",
      "\n",
      "\n",
      "[10:12:07] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'binary:logistic' was changed from 'error' to 'logloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
      "[0]\ttest-logloss:0.59818\ttrain-logloss:0.59817\n",
      "[1]\ttest-logloss:0.52040\ttrain-logloss:0.52039\n",
      "[2]\ttest-logloss:0.45554\ttrain-logloss:0.45553\n",
      "[3]\ttest-logloss:0.40071\ttrain-logloss:0.40070\n",
      "[4]\ttest-logloss:0.35388\ttrain-logloss:0.35386\n",
      "[5]\ttest-logloss:0.31353\ttrain-logloss:0.31352\n",
      "[6]\ttest-logloss:0.27854\ttrain-logloss:0.27853\n",
      "[7]\ttest-logloss:0.24802\ttrain-logloss:0.24800\n",
      "[8]\ttest-logloss:0.22127\ttrain-logloss:0.22126\n",
      "[9]\ttest-logloss:0.19773\ttrain-logloss:0.19772\n"
     ]
    }
   ],
   "source": [
    "model = xgb.train(params, dtrain, num_rounds, evals=evals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `nvidia-smi` shell command to see that data has been moved automatically to the GPU for training. Note the memory utilization under the column *Memory-Usage* after running the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed Mar  1 10:12:09 2023       \n",
      "+-----------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 470.82.01    Driver Version: 470.82.01    CUDA Version: 11.7     |\n",
      "|-------------------------------+----------------------+----------------------+\n",
      "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                               |                      |               MIG M. |\n",
      "|===============================+======================+======================|\n",
      "|   0  NVIDIA A10G         On   | 00000000:00:1E.0 Off |                    0 |\n",
      "|  0%   21C    P0    58W / 300W |   1489MiB / 22731MiB |      0%      Default |\n",
      "|                               |                      |                  N/A |\n",
      "+-------------------------------+----------------------+----------------------+\n",
      "                                                                               \n",
      "+-----------------------------------------------------------------------------+\n",
      "| Processes:                                                                  |\n",
      "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
      "|        ID   ID                                                   Usage      |\n",
      "|=============================================================================|\n",
      "+-----------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Testing and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you scroll to the section where we declared our XGboost parameters, we see that we have set the objective to be `binary:logistic`.\n",
    "\n",
    "In a binary logistic classifier we usually set the threshold at 0.5 meaning all probability scores >= 0.5 will be assigned 1 (Anomaly) and those with scores < 0.5 will be assigned 0 (Normal).\n",
    "\n",
    "This will help us calculate the accuracy as we will see in the coming sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's look at our predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2466"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold = .5\n",
    "true_labels = binary_y_test.astype(int)\n",
    "true_labels.sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.17932051 0.17932051 0.17932051 ... 0.17932051 0.17932051 0.17932051]\n"
     ]
    }
   ],
   "source": [
    "# make predictions on the test set using our trained model\n",
    "preds = model.predict(dtest)\n",
    "print(preds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 0 0 ... 0 0 0]\n"
     ]
    }
   ],
   "source": [
    "pred_labels = (preds > threshold).astype(int)\n",
    "print(pred_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2453"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_labels.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Area under the Curve (AUC)\n",
    "\n",
    "The area under the ROC curve is a great metric for determining how well your classification model is performing, even in the case of imbalanced classes. A score of 1 means your model is performing perfectly, while a score of .5 means that your model is the same as randomly guessing. [See here for more information](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9983689528503293\n"
     ]
    }
   ],
   "source": [
    "# compute the auc\n",
    "auc = roc_auc_score(true_labels, preds)\n",
    "print(auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should be getting near a perfect score here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 Accuracy\n",
    "Accuracy is the ratio of correct classifications to the total number of samples.  Accuracy is often misleading in anomaly detection problems where the classes are highly imbalanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.9999389317949574\n"
     ]
    }
   ],
   "source": [
    "print ('Accuracy:', accuracy_score(true_labels, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy should be almost perfect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.8 Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Confusion Matrix represents the quality of the output of a classifier on any dataset. The diagonal elements represent correct or true labels, whereas the off-diagonal (white box in Figures 5 and 6) elements represent the elements misclassified by the classification model. Therefore, the higher the values of the diagonal elements of the confusion matrix, the better and more accurate the classification model becomes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix :\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAEmCAYAAAC+vFFEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAoG0lEQVR4nO3deZxU1Zn/8c+3GwUUUBFE4ppEYtwRURBxN4qZZDCO4xIXdDTuyWQxiTPJLybRrDMuYRJNTETBGI3GuMUFCYlxSYzivisqKIgioKCirM/vj3saiqa6qxq7u271/b551aurzj333NPd9FOnnnvuuYoIzMwsXxpq3QEzM1udg7OZWQ45OJuZ5ZCDs5lZDjk4m5nlkIOzmVkOOThbu5LUU9ItkuZLuu5DtHO0pDvbs2+1IOl2SWNq3Q+rPw7OBSXp85KmSHpX0qwUREa2Q9OHAQOADSPi39e0kYi4KiIObIf+rELSPpJC0g3NyndK5XdV2c53Jf22Ur2IODgixq9hd63AHJwLSNJXgYuAH5IF0s2Bi4HR7dD8FsDzEbG0HdrqKG8Cu0vasKRsDPB8ex1AGf992Rrzf56CkbQe8H3gjIj4Y0S8FxFLIuKWiPh6qtNd0kWSXkuPiyR1T9v2kTRD0tckzU6j7hPStu8B3wGOSCPyE5uPMCVtmUao3dLr4yW9JOkdSS9LOrqk/N6S/UZIejClSx6UNKJk212SzpV0X2rnTkn9WvkxLAZuBI5M+zcCRwBXNftZ/UzSq5IWSHpI0p6pfBTw3yXf52Ml/fiBpPuAhcDHUtlJafslkq4vaf8nkiZLUrW/PysOB+fi2R3oAdzQSp1vAcOBwcBOwG7At0u2bwysB2wCnAj8QtIGEXEO2Wj89xHRKyIua60jktYFxgIHR0RvYATwaJl6fYFbU90NgQuAW5uNfD8PnABsBKwNnNXasYEJwHHp+UHAk8Brzeo8SPYz6Av8DrhOUo+IuKPZ97lTyT7HAicDvYHpzdr7GrBDeuPZk+xnNya8hoKV4eBcPBsCcyqkHY4Gvh8RsyPiTeB7ZEGnyZK0fUlE3Aa8C2y9hv1ZDmwvqWdEzIqIp8rU+RfghYi4MiKWRsTVwLPAZ0vqXB4Rz0fE+8C1ZEG1RRHxd6CvpK3JgvSEMnV+GxFz0zHPB7pT+fu8IiKeSvssadbeQrKf4wXAb4EvRsSMCu1ZQTk4F89coF9TWqEFH2HVUd/0VLaijWbBfSHQq60diYj3yNIJpwKzJN0q6ZNV9KepT5uUvH59DfpzJXAmsC9lPklIOkvSMymV8jbZp4XW0iUAr7a2MSL+CbwEiOxNxKwsB+fi+QewCDiklTqvkZ3Ya7I5q3/kr9Z7wDolrzcu3RgREyPiU8BAstHwr6voT1OfZq5hn5pcCZwO3JZGtSuktMM3gMOBDSJifWA+WVAFaCkV0WqKQtIZZCPw11L7ZmU5OBdMRMwnO2n3C0mHSFpH0lqSDpb001TtauDbkvqnE2vfIfsYviYeBfaStHk6GflfTRskDZA0OuWeF5GlR5aXaeM24BNp+l83SUcA2wJ/WsM+ARARLwN7k+XYm+sNLCWb2dFN0neAPiXb3wC2bMuMDEmfAM4DjiFLb3xD0uA16711dQ7OBZTyp18lO8n3JtlH8TPJZjBAFkCmAI8DTwAPp7I1OdYk4PeprYdYNaA2pH68BswjC5SnlWljLvAZshNqc8lGnJ+JiDlr0qdmbd8bEeU+FUwE7iCbXjcd+IBVUxZNF9jMlfRwpeOkNNJvgZ9ExGMR8QLZjI8rm2bCmJWSTxSbmeWPR85mZjnk4GxmlkMOzmZmOeTgbGaWQ61diGCA1m4IevjHlEdDPrF9rbtgZUyf9gpz5sxp1/VC1K9HsLjcLMtm3lkyMSJGteexa8VRp5Ie3WDYRrXuhZVx3x33Vq5knW6PYe2x8mwzi5dX93f455mVruCsGw7OZlYfCrZ4n4OzmeWfgEYHZzOz/ClWbHZwNrN6IKc1zMxyRxRu4q+Ds5nVhwaPnM3M8kU4OJuZ5VKxYrODs5nVCZ8QNDPLGc9zNjPLqWLFZgdnM6sHnudsZpY/nq1hZpZTDs5mZjlUrNjs4GxmdcBpDTOznCpWbHZwNrM64XnOZmY5I0+lMzPLp2LFZgdnM6sTHjmbmeWM19YwM8upYsVmB2czqxOe52xmlkPOOZuZ5YyEqhg5Ryd0pbM4OJtZXVAVI2cHZzOzTlawrIaDs5nlX7buUeXovKzju9JpHJzNLP8EDQ0Nte5Fp3JwNrM6oKpyzl2Jg7OZ1YWCxWaK9TnBzOqSyGZrVHpUbEfaTNJfJT0t6SlJ/5nK+0qaJOmF9HWDVC5JYyVNlfS4pCElbY1J9V+QNKakfBdJT6R9xip1rKVjtMTB2czyT9CghoqPKiwFvhYR2wLDgTMkbQucDUyOiEHA5PQa4GBgUHqcDFwCWaAFzgGGAbsB55QE20uAL5TsNyqVt3SMshyczawutMfIOSJmRcTD6fk7wDPAJsBoYHyqNh44JD0fDUyIzP3A+pIGAgcBkyJiXkS8BUwCRqVtfSLi/ogIYEKztsodoyznnM2sLlSZc+4naUrJ60sj4tLy7WlLYGfgn8CAiJiVNr0ODEjPNwFeLdltRiprrXxGmXJaOUZZDs5mlntCVc1zBuZExNCK7Um9gOuBL0fEgtJRd0SEpA692LCaYzitYWb5l+Y5V3pU1ZS0Fllgvioi/piK30gpCdLX2al8JrBZye6bprLWyjctU97aMcpycDazutB0G8HWHpXbkIDLgGci4oKSTTcDTTMuxgA3lZQfl2ZtDAfmp9TEROBASRukE4EHAhPTtgWShqdjHdesrXLHKMtpDTPLvaapdO1gD+BY4AlJj6ay/wZ+DFwr6URgOnB42nYb8GlgKrAQOAEgIuZJOhd4MNX7fkTMS89PB64AegK3pwetHKMsB2czqwvtEZwj4l5avqfK/mXqB3BGC22NA8aVKZ8CbF+mfG65Y7TEwdnM8k/y2hpmZnlUtMu3HZzNLPfaMedcNxyczawuODibmeWNoNF337bc+WApPPUWLF6evd5kXdi818rt09+BFxbAXhvD2o0w+314aQGg7PPg1uvB+t2zuo/MgfmLYf21YXC/lW1EwIsLsn0RbJqOEQHPz4c5H0CjYNsNoM/anfSNd32nnHQqt996O/036s9Dj02pvENBqYDrOdfs9KekkHR+yeuzJH23k/twl6SKl3rWnASD1oPdB8Cu/WHGu/DukmzbB0th7iLo0biyft/uMGwjGL5RFkyffnvlti16wXZlViqctRA+WJYdY8QA2LhnVj53ESxcmpVtswE8+/bq+9oaO/a4Y7jp1htr3Y26oCr+dSW1nJuyCDhUUr+KNcuQVJxRf/fGlaPVbg2wzlqwKN0t7fn5WeAu1a1h5antZctX3da3BzSW+bXPeA8+1mflfmunYP/m+zBwnax8vbVhaaw8tn1oI/caSd++fWvdjbrQHqvS1ZNaBrilwKXAV4BvlW5Iq0WNA/oBbwInRMQrkq4APiBbSeq+tKbq++n1RsB/kF0uuTvwz4g4PrV3CbAr2RU7f4iIczr4e+s47y+Fd5ZkgXL2+1ng7r3W6vVmvw9TF8DiZTB4w+rafeP9bL+1G2Dr9WGdblkgLh2Vd2/Myro3ttiUWUdoKFjOudazun8BHC2p2dCP/wPGR8SOwFXA2JJtmwIjIuKr6fUGZMH4K2TXrl8IbAfsIGlwqvOttFLVjsDeknZsrVOSTpY0RdIUlixvrWrnWrocHp+X5ZAFTHsHPt6nfN2NemapiJ02hJfeqdz2crL/DcM2ynLaT7/Vjh03+3CytTOKNXKuaXCOiAVki1F/qdmm3YHfpedXAiNLtl0XEaWfq29Jl1g+AbwREU9ExHLgKWDLVOdwSQ8Dj5AF7m0r9OvSiBgaEUNZq9bvX8nyyALzxj2zwPv+suxx/2y49/VsNPvPN1dPOWzQPRsVL66QiujemLUL0L9HNjpvKv+gZF+Pmq0mKgfmrhac85C3vQh4GLi8yvrvNXu9KH1dXvK86XU3SR8FzgJ2jYi3Umqkxxr3thYispHsut1gi95ZWa+1YO+BK+vc+zrs1j/LFS9cCj0bs+HGgsVZYK/0JtO/B8xbBJt0g7cWZ8cC6N8TXn0XBvSEBUugmxycrSa6WvCtpObDwrSS07XAiSXFfweOTM+PBu75EIfoQxbQ50saQHZPsPoyfzG8/j68tSgbKd8/O5va1pLZ76+s99zbsEPflSf6prwJT8zLAvE9s2BuamfL3tl+/3gDXpyfzcwA2LA79OwGf38je4P45Pod+Z0WznFHj2Gfkfvy/HMv8PEtBnHFuPGVdyqohgZVfHQleRg5A5wPnFny+ovA5ZK+TjohuKYNR8Rjkh4BniW7rcx9H6ajNbF+dzhgk9brjNx45fMte2ePcob2L1++VgPsXGbijOSA3IEmXOVgXI2mnHOR1Cw4R0SvkudvAOuUvJ4O7Fdmn+Nbeh0R0yhZpq/ZtlX2Kynfp80dN7OacHA2M8uhgsVmB2czqwdez9nMLHecczYzy6mCxWYHZzOrDx45m5nljpBzzmZmOSOnNczMcsf3EDQzyykHZzOzHOpqa2dU4uBsZvnXBZcErcTB2cxyzzlnM7OccnA2M8shB2czs7yRTwiameWO8AlBM7NccnA2M8uhgsXm2t/g1cysIoEaGio+KjYjjZM0W9KTJWXflTRT0qPp8emSbf8laaqk5yQdVFI+KpVNlXR2SflHJf0zlf9e0tqpvHt6PTVt37JSXx2czawuKF2I0tqjClcAo8qUXxgRg9PjtnS8bYEjge3SPhdLapTUCPwCOBjYFjgq1QX4SWprK+At4MRUfiLwViq/MNVrlYOzmeWegAZVflQSEXcD86o87GjgmohYFBEvA1OB3dJjakS8FBGLgWuA0creHfYD/pD2Hw8cUtJW063W/wDsrwrvJi3mnCUNaW3HiHi4te1mZu2n6pFxP0lTSl5fGhGXVrHfmZKOA6YAX4uIt4BNgPtL6sxIZQCvNisfBmwIvB0RS8vU36Rpn4hYKml+qj+npQ61dkLw/Fa2Bdk7hJlZh5OgsbrF9udExNA2Nn8JcC5ZXDuXLPb9RxvbaHctBueI2LczO2Jm1pqOysFGxBtNzyX9GvhTejkT2Kyk6qapjBbK5wLrS+qWRs+l9ZvamiGpG7Beqt+iit+vpHUkfVvSpen1IEmfqbSfmVl7apAqPtaEpIElLz8HNM3kuBk4Ms20+CgwCHgAeBAYlGZmrE120vDmiAjgr8Bhaf8xwE0lbY1Jzw8D/pLqt6iaec6XAw8BI9LrmcB1rHx3MTPrUO21Kp2kq4F9yHLTM4BzgH0kDSZLa0wDTgGIiKckXQs8DSwFzoiIZamdM4GJQCMwLiKeSof4JnCNpPOAR4DLUvllwJWSppKdkDyyUl+rCc4fj4gjJB2VOryw0llGM7N2JVWbc25VRBxVpviyMmVN9X8A/KBM+W3AbWXKXyKbzdG8/APg39vS12qC82JJPcneVZD0cWBRWw5iZvZhiOLN+60mOJ8D3AFsJukqYA/g+I7slJlZc2uaU65XFYNzREyS9DAwnOwN7D8josW5eWZmHaFo2dRqFz7aGxhJltpYC7ihw3pkZtaMgEYH51VJuhjYCrg6FZ0i6YCIOKNDe2ZmtsKaT5WrV9WMnPcDtmmakydpPPBU67uYmbUfqXg552pOgE4FNi95vVkqMzPrNO20Kl3daG3ho1vIcsy9gWckPZBeDyO7SsbMrFM457yq/+20XpiZVVC0tEZrCx/9rTM7YmbWsuKdEKxm4aPhkh6U9K6kxZKWSVrQGZ0zM4PshKBzzqv7OdkiHdcBQ4HjgE90ZKfMzJorWs65qsvVI2Iq0BgRyyLicsrfg8vMrENkt6nqmCVD86qakfPCtGbpo5J+CsyieGuQmFmNdbXgW0k1QfbYVO9M4D2yec6HdmSnzMxWVTnfXLicc0RMT08/AL4HIOn3wBEd2C8zsxWk4uWcq134qLnd27UXZmYVFC2tsabB2cys0zSdECyS1i7fHtLSJrJlQwthyCe257477q11N6yM5bG81l2wMlq/bema62o55UpaGzmf38q2Z9u7I2ZmLRONKtYksdYu3963MztiZtaSIi4Z6pyzmdUF4eBsZpY7zjmbmeWMEI1qrHU3OlU1q9JJ0jGSvpNeby5pt47vmpnZSkVbW6Oa058Xk110clR6/Q7wiw7rkZlZGb58e3XDImKIpEcAIuKttBCSmVmnUPpXJNUE5yWSGsnuH4ik/oBn/5tZ5xE0Nniec3NjgRuAjST9ADgM+HaH9srMrISAhoKtVFzNqnRXSXoI2J/sZ3RIRDzT4T0zM1uh6+WUK6kYnCVtDiwEbikti4hXOrJjZmalHJxXdytZvllAD+CjwHPAdh3YLzOzFQReW6O5iNih9HVare70DuuRmVlzBVxbo81vRRHxMDCsA/piZtYCVfWvYivSOEmzJT1ZUtZX0iRJL6SvG6RySRoraaqkx0uXUZY0JtV/QdKYkvJdJD2R9hmrlItp6RitqeYKwa+WPM6S9DvgtYo/BTOzdpIttt9Q8VGFK4BRzcrOBiZHxCBgcnoNcDAwKD1OBi6BLNAC55ANUncDzikJtpcAXyjZb1SFY7Somu+md8mjO1kOenQV+5mZtZv2CM4RcTcwr1nxaGB8ej4eOKSkfEJk7gfWlzQQOAiYFBHzIuItYBIwKm3rExH3R0QAE5q1Ve4YLWo155wuPukdEWdVasjMrONUvXZGP0lTSl5fGhGXVthnQETMSs9fBwak55sAr5bUm5HKWiufUaa8tWO0qLXbVHWLiKWS9qjUiJlZRxJVr+c8JyKGrulxIiIkddCNttp2jNZGzg8AQ4BHJd0MXAe8V3KAP37oXpqZVakDZ2u8IWlgRMxKqYnZqXwmsFlJvU1T2Uxgn2bld6XyTcvUb+0YLaom59wDmAvsB3wG+Gz6ambWOSQa1FjxsYZuBppmXIwBbiopPy7N2hgOzE+piYnAgZI2SCcCDwQmpm0LJA1PszSOa9ZWuWO0qLWR80aSvgo8ycqLUJp06LDfzKxUNlvjw4+cJV1NNurtJ2kG2ayLHwPXSjoRmA4cnqrfBnwamEp2lfQJABExT9K5wIOp3vcjoukk4+lkM0J6ArenB60co0WtBedGoBeUTfQ4OJtZp2qPy7cj4qgWNu1fpm4AZ7TQzjhgXJnyKcD2ZcrnljtGa1oLzrMi4vttaczMrKM0eD3nFYr1kzCz3BKioaFY9xBsLTi3aQhuZtaRPHJOShLcZmY1JXnJUDOzXPI9BM3Mckdez9nMLG8EyMHZzCxvqluvuStxcDazuuATgmZmOVTlYvpdhoOzmeWe8DxnM7P8kZzWMDPLI7X9ftR1zcHZzOqCR85mZjkjX4RiZpZPnudsZpZDTmuYmeVMdvdtpzXMzHLGOWczs/zxes5mZvmTpTUcnM3McscjZzOz3BGNKtYNXouVYS+QU046lc0HbsEuOw1dbdtFF/yMnt3WZc6cOTXoWTHMeHUGBx/waXbZcShDd9qVX4y9eJXtP7twLOuu1XvF7+Duv93DwA03YfguIxi+ywh+dN6Pq2qnKJrSGpX+dSUdOnKWdAhwA7BNRDzbkcdqpQ/vRkSvWhy7lo497hhOPf0UTjrhC6uUv/rqDCZPmsxmm29Wo54VQ2O3bvzwpz9k5yGDeeeddxg5bE/2O2A/ttn2k8x4dQaTJ/1ltd/BiJG7c/1Nf6i6naIpWlqjo0fORwH3pq/WiUbuNZK+ffuuVv6Nr32TH/z4vML9R+9sAwduzM5DBgPQu3dvtv7k1rz22msAfPOssznvR+dW9TtorZ1iqWbc3LX+T3dYcJbUCxgJnAgcmcr2kXSXpD9IelbSVUr/QyXtL+kRSU9IGiepeyqfJulHkh6VNEXSEEkTJb0o6dSmY0maLOnhtP/oMv2ZkEbyTa+vKlevK7vl5j/xkU0GsuNOO9a6K4Uyfdp0Hnv0cXbdbSh/uvlPDPzIR9hxpx1Wq/fA/Q8wbMjuHPKZQ3n6qWdabadoBDSqseKjK+nItMZo4I6IeF7SXEm7pPKdge2A14D7gD0kTQGuAPZP9ScApwEXpX1eiYjBki5M9fYAegBPAr8EPgA+FxELJPUD7pd0c0RESX8uA74C3ChpPWAEMKZcxyWdDJwMdJmP/wsXLuSnP/of/nTHzbXuSqG8++67fP7wY/jp+T+mW7du/M+Pz+fm229crd7gnXfimRefplevXtxx+0SOPOwoHn/m0bLt9OnTp/O+gbwo4DznjkxrHAVck55fw8rUxgMRMSMilgOPAlsCWwMvR8Tzqc54YK+StpoiyhPAPyPinYh4E1gkaX2yN9YfSnoc+DOwCTCgtDMR8TdgkKT+qS/XR8TSch2PiEsjYmhEDO3fv98affN589KLLzF92jR2GzKcrT++DTNnzGT3Xffg9ddfr3XXuqwlS5bw+cOP4YijDmf050bz0osvM23aNIbvMoJtttqOmTNmssdue/L662/Qp08fevXKTo2MOvgglixZsuJkYfN2iql4aY0OGTlL6gvsB+wgKYBGIIBbgUUlVZdV2YemfZY323952v9ooD+wS0QskTSNbGTd3ATgGLI0ywnVfj9dwfY7bM8rs6aveL31x7fhvn/eQ79+XePNJ28igtO+cAZbf3JrvvSVLwKw/Q7bMf21l1fU2War7bjn/r/Rr18/Xn/9DQYM2AhJTHlgCsuXL2fDDTcs205ReeTcPg4DroyILSJiy4jYDHgZ2LOF+s8BW0raKr0+FvhbG463HjA7BeZ9gS1aqHcF8GWAiHi6De3XneOOHsM+I/fl+ede4ONbDOKKceNr3aVC+cd9/+Dqq67mb3/924rpcXfcPrHF+jdefyO7Dt6NYUN256yvfJ3xv70cSW1up6vK7iFY+V9X0lE556OAnzQru54sj/xi88oR8YGkE4DrJHUDHiTLJVfrKuAWSU8AU4Cy0/Yi4g1JzwA3tqHtujThqtaD8XMvrn7CydrPiJEjeG/JO63WeWbqUyuen3rGKZx6xilr1E5hFGzk3CHBOSL2LVM2FhjbrOzMkueTyU4WNt9vy5LnV5CNflfbBuzeQl9WzHGWtA4wCLi60vdgZnnS9XLKlXStzwGtkHQA8AzwfxExv9b9MbO2UboDd2uPKtuZlqbcPppmiiGpr6RJkl5IXzdI5ZI0VtJUSY9LGlLSzphU/wVJY0rKd0ntT037rtG7SmGCc0T8OeXAL6p1X8ys7do557xvRAyOiKZJ42cDkyNiEDA5vQY4mOzT9iCy6bWXwIpJD+cAw4DdgHOaAnqq84WS/Uat2fdrZpZzov1Gzi0YTTaFl/T1kJLyCZG5H1hf0kDgIGBSRMyLiLeAScCotK1PRNyfrrOYUNJWmzg4m1kdaNd5zgHcKemhdMEZwICImJWev87K6yQ2AV4t2XdGKmutfEaZ8jbzkqFmVheqDL79mvLIyaURcWmzOiMjYqakjYBJklaZ3RURka7PqCkHZzPLP0FDdfcQnFOSRy4rImamr7Ml3UCWM35D0sCImJVSE7NT9ZlA6RoOm6aymcA+zcrvSuWblqnfZk5rmFnutVfOWdK6kno3PQcOJFuj52ZWrrUzBrgpPb8ZOC7N2hgOzE/pj4nAgZI2SCcCDwQmpm0LJA1PszSOK2mrTTxyNrM60G7znAcAN6RA3g34XUTcIelB4FpJJwLTgcNT/duATwNTgYWkZR8iYp6kc8kumAP4fkTMS89PJ7seoydwe3q0mYOzmdWF9gjOEfESsFOZ8rnA/mXKAzijhbbGAePKlE8Btv+wfXVwNrO6UGXOuctwcDaz3Gu6h2CRODibWR340BeZ1B0HZzOrEw7OZmb5Uv085y7DwdnM6oJzzmZmOSPnnM3M8skjZzOzHHLO2cwshzxyNjPLGeeczcxyyiNnM7MccnA2M8shpzXMzHLJwdnMLHeKFZodnM2sDmS3oSrWPOdifbdmZnXCI2czqwuerWFmlkMOzmZmOVS0qXTOOZuZ5ZBHzmZWB+S0hplZPjk4m5nlioCGguWcHZzNrE44OJuZ5U6xQrODs5nVjWKFZwdnM6sDvhOKmVnuCF8haGaWUw7OZma5U6zQ7OBsZnWheOs5OzibWV3wyNnMLJeKFZ4dnM0s/1S8JUMdnM0s94o4lU4RUes+5JqkN4Hpte5HO+kHzKl1J6ysrvS72SIi+rdng5LuIPsZVTInIka157FrxcG5QCRNiYihte6Hrc6/G2uuWHNTzMzqhIOzmVkOOTgXy6W17oC1yL8bW4VzzmZmOeSRs5lZDjk4m5nlkIOzmVkOOTjbCpK2k7RlrfthZj4haCUkjSd7w/52RHSVqyK7HEmbAu9GxNu17ot1HI+crdR/AIuBb3kEnU+S+gPnAwPS62ItOFEgDs4FV/rHHRHLgFOAtYBvO0Dng6TGpucR8SbwKnCepLXCH327LAfnApOkpj9uScMk7RoRS4ETgSAL0FvUtJMFJqkvZG+akj4q6ZNp03eB11g5evbfcRfknLMh6WvAvwILgFeAC4CXgYuBDYCzIuLV2vWweCR1A54AboqIsyVdDjQCbwPnAVcAd0TE2Jp10jqU33ELTtLngE9FxN7A88ABwJeALYDTgdeBpbXrYTGlTzCjgWMknRYRJwD/STZaPhnoC5wkaesadtM6kEfOBVOaykivh5CNxj4FfI4sIF8JzAb+OyKeqUU/i05St4hYKmkQ8HdgbEScm7ZtAYwkO4F7QUTcWsOuWgfxnVAKpFmOeVvgpYh4OL3eCTg/Il6S9FdgM+DN2vW2mEp+R90lrRsRL0jaA7hHUkNEfC9Nc5wu6T3gTEmTI+KD2vbc2puDc4GUBOYvAicBcyRdAEwGngYulHQtcBBwRER0lTtz1IWmwCxpNNmouIekn0fELSlA35VG1P8v7bII6E3R7nxaEM45F0DpdDlJGwEjgL2Ba4DDgEOAG4GfkOWax0TES53e0YJLgXkU8P/I0kuvABMkHRsRU4H9gNMlbSVpLeA94KSIeL92vbaO4pFzAZSMmE8hG2l1T1eX/VrSMuDAVDZe0u/SfGerjU3JAvNQYBBwFnCRpO4R8RtJm5YE47tr1UnreB45F4SkQ4EzgYXADpIuBIiIccCDwAhJfRyYO1fTpxpJ3QEi4jfAi2QXA50VEZcBfwXOkjQA+KB0P+u6PFuji2p28m9v4AvAdRFxU7ry7zLgsYj4aqqzXkTMr1mHC0zSvwCfBRZHxJdS2cXAC8CTwPHAzyLigZp10jqdR85dULPAfChwKLAhsIekjSNiGtlVgHtL+lHabUFNOltwaRGjnwC3AttLukHSOsBdZGmNi4FrHJiLxyPnLiydXPoGsH96jAH+AtwaEbMlbU72f8Ar0NWApF2AdYBdI+KCVHYzMB84NSLeSznmGc3np1vX5xOCXZSkfYDTgAfTH/WfJfUmu+qsp6TrIuKVGnaxkEqmy+0JXAs8Bmwi6ZmIuD0i/lXSZOD3wGeAmbDypK4Vh4NzF1FmZPUyMAv4mKSdIuKxiLhB0tpkU7KuqklHCy4F5j2Ag8muyHySbE7zaEnLI2JiROyfRtUOygXmtEYX0CzH/FmytTDeBqYAPwPmAb+PiCdSnV4R8W6NultY6Qq/5elk3xHAv0XEXZI+BnwaGAZcHRG31bSjlgs+IdiFSDod+B7ZugvjgK+kx/rA8ZK2A3Bg7lwl097WA4iI04FfAj+QtH664Od24CGytZrNPHKuZ+mE3tx04mgjshzmaRHxjKQNyP7Yv0N2efY3gfN8SXZtSDqIbFW5WWSXyl8A/BTYFTg0IuZJ6umr/ayJR851Kl2Q8DXgtJSmmA3MIbvNFBHxFvBlYIeImAV83YG5NtKiUpeQ3V7qDmBj4BcR8XVgGnBzWjPDgdlWcHCuX2+SXdn3EeCE9NF5KnBNWqgdsnUyNlV2myOvydzJStIZfcimL04GbiCbu9xH0uCIOB44Pa3fbLaCg3OdkTRI0tYRsZxsxsVfgW2AL0TE2WQnAe+W9EuyWQA/iIhlPuvfecpcWj2XbDbGqIhYGhEvA0vIfm+Q3fHEbBXOOdcRSRuSjZjnkJ34WwZcCnwe2AqYFRG/kjQM6AG8kgKBdTJJ+wP/RnaibzIwCjiV7A31aeDXwMm+8s9a4uBcZyTtB/yZ7OTSDmT3+HuXLNfcD5gEXB4Ri2rWyYIqucBkONkUxsnAzmR55r+Q5ZrPBt4gW+fkhpp11nLPwbkOSfoUMBbYieyecvsBRwK7kc0G2MOLGNWGsjtk/wb4cUT8SdK+wOFkixhdkWZlNN2CypdkW4scnOtUWsnsQmB4+oPfAFgLWCctbGSdQNkNVncG7ouIVyVtQ5ZqejciDk519iRbFfAp4P8iYmHNOmx1w8G5jkk6mOzj8+4RMbfW/SmadOLvf8nuVv4HYAbwXeATwLFAd+DMlOrYm2xO+pM16q7VGQfnOpfuN/ddYJc0g8M6UUoxnUO25vL5ZOmL5WR55s8A3ZrWaDZrCwfnLsBrZdSWpBuBhyLiXEljyD7NvAg8C+wOHBQRL9Swi1aHHJzN1lDJ7IxdgX8lS21cRXYu4CXgU8CdEXFX7Xpp9crB2exDSuuaXAnsBXw5In6VyntExAc17ZzVLQdns3aQRs9jyRYxmtW0PGit+2X1y5dvm7WPR8mmyu3pwGztwXdCMWsHEbFE0q+AtRyYrT04rWFmlkNOa5iZ5ZCDs5lZDjk4m5nlkIOzmVkOOThbRZKWSXpU0pOSrpO0zodo6wpJh6Xnv5G0bSt195E0Yg2OMU1Sv2rLW2jjeEk/b4/jmq0JB2erxvsRMTgitidb1P/U0o0l9yxsk4g4KSKebqXKPkCbg7NZV+DgbG11D7BVGtXeI+lm4GlJjZL+R9KDkh6XdApk609I+rmk5yT9GdioqSFJd0kamp6PkvSwpMckTZa0JdmbwFfSqH1PSf0lXZ+O8aCkPdK+G0q6U9JTkn4DNL+HX4sk7SbpH5IekfT3tD5zk81SH1+QdE7JPsdIeiD161fpBrpm7coXoVjV0gj5YLLlMAGGANtHxMuSTgbmR8SukroD90m6k2wh+q2Bbcnu2vI0MK5Zu/3J7qm3V2qrb7qBwC/JFq3/31Tvd8CFEXGvpM2BiWQ3ST0HuDcivp9uQnBiG76tZ4E9051JDgB+SHbvP8juLLM9sBB4UNKtwHvAEWR3m1ki6WLgaGBCG45pVpGDs1Wjp6RH0/N7gMvI0g0PlNxA9kBgx6Z8MrAeMIhsMaCrI2IZ8Jqkv5Rpfzhwd1NbETGvhX4cAGxbcnPrPpJ6pWMcmva9VdJbbfje1gPGSxoEBNndZJpMarqJgaQ/AiOBpcAuZMEaoCcwuw3HM6uKg7NV4/2IGFxakALTe6VFwBcjYmKzep9ux340kN2Wa5WV3kqC9Zo4F/hrRHwupVLuKtnW/PLZIPs+x0fEf32Yg5pV4pyztZeJwGmS1gKQ9AlJ6wJ3A0eknPRAYN8y+94P7CXpo2nfvqn8HaB3Sb07gS82vZA0OD29G/h8KjuY7I7k1VoPmJmeH99s26ck9ZXUEzgEuI/sjtqHpWVCSdu3aMPxzKri4Gzt5Tdk+eSHJT0J/Irsk9kNZLdueposL/uP5jtGxJvAycAfJT0G/D5tugX4XNMJQbJ79Q1NJxyfZuWske+RBfenyNIbr7TSz8clzUiPC4CfAj+S9Airf5J8ALgeeBy4PiKmpNkl3wbulPQ4MAkYWOXPyKxqXvjIzCyHPHI2M8shB2czsxxycDYzyyEHZzOzHHJwNjPLIQdnM7MccnA2M8uh/w8i3bpfKnIMlAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results = confusion_matrix(true_labels, pred_labels) \n",
    "\n",
    "print ('Confusion Matrix :')\n",
    "\n",
    "def plot_confusion_matrix(cm, target_names, title='Confusion Matrix', cmap=plt.cm.Greens):\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(target_names))\n",
    "    plt.xticks(tick_marks, target_names, rotation=45)\n",
    "    plt.yticks(tick_marks, target_names)\n",
    "    plt.tight_layout()\n",
    "\n",
    "    width, height = cm.shape\n",
    "\n",
    "    for x in range(width):\n",
    "        for y in range(height):\n",
    "            plt.annotate(str(cm[x][y]), xy=(y, x), \n",
    "                        horizontalalignment='center',\n",
    "                        verticalalignment='center')\n",
    "    plt.ylabel('True Label')\n",
    "    plt.xlabel('Predicted Label')\n",
    "\n",
    "\n",
    "plot_confusion_matrix(results, ['Normal','Anomaly'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the confusion matrix we can see that our model is performing really well.  It is able to classify almost all of the anomalies.  And this is without any tuning of XGBoost.  We can see why XGBoost is so popular."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.9 AUC Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The AUC curve is a plot of the True Positive Rate (TPR) against the False Positive Rate (FPR). The TPR is a measure of how often the prediction is correct, when there is a positive value, and the FPR is how often the prediction is incorrect when the value is in fact negative. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6hklEQVR4nO3dd3gUVffA8e8hgRB6RyAgRaT3UAVEihQRLAiCihRFmqDyggULP8EOVkDE8oIVRKQISlMQFHmlSO8iQuggvaac3x8zhCUmmwWy2WRzPs+zT3Z27sycmd3s2bl35l5RVYwxxpikZAp0AMYYY9I2SxTGGGO8skRhjDHGK0sUxhhjvLJEYYwxxitLFMYYY7yyRGGuiohsEJEmgY4j0ERknIg8l8rbnCAiI1Jzm/4iIveJyLyrXNY+g6lE7D6K9E9EdgKFgVjgFDAH6K+qpwIZV7ARkW7AQ6raMMBxTACiVPXZAMcxDLhBVe9PhW1NIA3sc0ZlZxTB43ZVzQFUB2oATwc2nCsnIqEZcduBZMfc+MISRZBR1f3AXJyEAYCI1BORpSJyTETWeJ6ui0g+EfmviOwVkaMiMt1jXlsRWe0ut1REqnrM2ykizUWkqIicFZF8HvNqiMhhEcnsTvcQkU3u+ueKyPUeZVVE+onINmBbYvskIu3caoZjIrJIRCokiONpEdnorv+/IpL1CvbhSRFZC5wWkVAReUpE/hSRk+4673TLVgDGAfVF5JSIHHNfj68GEpEmIhIlIoNE5KCI7BOR7h7byy8i34nICRFZLiIjROSXpN5LEWno8b7tds9oLsorIrPdOP8nImU8lnvHLX9CRFaKSCOPecNE5BsR+VxETgDdRKSOiPzmbmefiIwWkSwey1QSkfki8o+IHBCRZ0SkFfAM0Mk9HmvcsrlF5GN3PXvcfQxx53UTkV9F5C0ROQIMc1/7xZ0v7ryDbuzrRKSyiPQC7gOGuNv6zuP9a+4+D3HjuvjerRSR4kkdW3OFVNUe6fwB7ASau88jgHXAO+50MeAI0Abnh0ELd7qgO382MBnIC2QGbnZfrwEcBOoCIcCD7nbCEtnmT8DDHvG8AYxzn7cHtgMVgFDgWWCpR1kF5gP5gPBE9u1G4LQbd2ZgiLu+LB5xrAeKu+v4FRhxBfuw2l023H3tHqCoe6w6udsu4s7rBvySIL4JHttrAsQAL7qxtgHOAHnd+ZPcRzagIrA74fo81ns9cBLo7K4rP1DdY5tHgDruMf0CmOSx7P1u+VBgELAfyOrOGwZEA3e4+xgO1ALqueVLApuAx9zyOYF97nqyutN1Pdb1eYK4pwEfANmBQsDvwCMexy8GeNTdVrjnMQVaAiuBPIDgfGaKJDzOSXzuB+N87su5y1YD8gf6fzNYHgEPwB4p8CY6/zCn3C8WBX4E8rjzngQ+S1B+Ls6XZhEg7uIXWYIy7wPDE7y2hUuJxPOf9CHgJ/e5uF+Ajd3pH4CeHuvIhPPleb07rUBTL/v2HPB1guX3AE084ujtMb8N8OcV7EOPZI7taqC9+zz+S81jfvwXGE6iOAuEesw/iPMlHILzBV3OY96IhOvzmPc0MC2JeROAjxLs82Yv+3AUqOY+HwYsTmafH7u4bZxE9UcS5YbhkShw2snO45Hw3eUXehy/XQnWEX9MgabAVvd4ZUrqOCf43F/8DG65+D7ZI+UfVvUUPO5Q1Zw4X1blgQLu69cD97jVCsfcKpOGOEmiOPCPqh5NZH3XA4MSLFcc59d2QlNxqmSKAI1xks8Sj/W847GOf3CSSTGP5Xd72a+iwN8XJ1Q1zi2f1PJ/e8Toyz5ctm0R6epRVXUMqMylY+mLI6oa4zF9BsgBFMT5Fe25PW/7XRz408v8/YlsAwAR+Y84VX3H3X3IzeX7kHCfbxSRWSKy362OetmjfHJxeLoe5+xnn8fx+wDnzCLRbXtS1Z+A0cAY4KCIjBeRXD5u+0riNFfIEkWQUdWfcX59jXRf2o1zRpHH45FdVV915+UTkTyJrGo38FKC5bKp6leJbPMoMA+nqqYLTjWIeqznkQTrCVfVpZ6r8LJLe3G+gACnHhvnS2GPRxnPuugS7jK+7kP8tsVpO/kQ6I9TbZEHp1pLfIgzOYdwql0ikog7od1AGS/zE+W2RwwBOuKcKeYBjnNpH+Df+/E+sBkoq6q5cNoeLpbfDZROYnMJ17Mb54yigMfxzqWqlbwsc/kKVd9V1Vo4VXM34lQpJbscV3m8jG8sUQSnt4EWIlIN+By4XURaug1+Wd1G1whV3YdTNTRWRPKKSGYRaeyu40Ogt4jUdRsZs4vIbSKSM4ltfgl0BTq4zy8aBzwtIpUgvrHznivYl6+B20SkmTiN44Nwvow8E00/EYkQp0F9KE6by9XsQ3acL6RDbqzdcc4oLjoARHg29PpKVWOBb3EacLOJSHmc45WUL4DmItJRnEb2/CJS3YdN5cRJSIeAUBF5HkjuV3lO4ARwyo2rj8e8WUAREXlMRMJEJKeI1HXnHQBKikgmdx/34fxgGCUiuUQkk4iUEZGbfYgbEantvleZcdqGzuGcnV7cVlIJC+AjYLiIlHXf66oikt+X7ZrkWaIIQqp6CPgUeF5Vd+M0KD+D8+WxG+dX2sX3/gGcuvPNOPXpj7nrWAE8jFMVcBSnAbmbl83OBMoC+1V1jUcs04DXgElutcZ6oPUV7MsWnMbZ94DDwO04lwJf8Cj2Jc4X1A6c6ocRV7MPqroRGAX8hvPFVAWncfyin4ANwH4ROezrPnjoj1MNtB/4DPgKJ+klFssunLaHQTjVdatxGmiTMxfnPpqtONVw5/BexQXwH5wzwZM4yfViokVVT+JcSHC7G/c24BZ39hT37xERWeU+7wpkATbiHPNvcKo5fZHL3f5RN/YjOBdGAHwMVHSrtKYnsuybOD8q5uEkvY9xGstNCrAb7ky6Js7Nhg+p6oJAx3KlROQ14DpVfTDQsRjjjZ1RGJNKRKS8WyUiIlIH6IlzOakxaZrdGWlM6smJU91UFKdqaxQwI6ARGeMDq3oyxhjjlVU9GWOM8SrdVT0VKFBAS5YsGegwjDEmXVm5cuVhVS14Ncumu0RRsmRJVqxYEegwjDEmXRGRv5MvlTirejLGGOOVJQpjjDFeWaIwxhjjlSUKY4wxXlmiMMYY45UlCmOMMV75LVGIyCfu2Lfrk5gvIvKuiGwXkbUiUtNfsRhjjLl6/jyjmAC08jK/NU631GWBXjiDpxhjjElj/HbDnaouFpGSXoq0Bz51R0JbJiJ5RKSIO/iJCUaqgP77b2LzfC2XWPmUKOdZPiXLxfetltz6wOuxSIljF6zH+GqOXXp5L67wPVNVpi0vxLQV13EtAnlndjEuH1Alyn3tX4lCRHrhnHVQqXh2mNMD728aic+7mnKQhv8hfCzn65dTSpfzjO3ydxREvP9N0XJuGV/Lxa/7GspBIq/5WC6x8ilaLj2+B5lS5r3y17H1LO9TuSs4tsnuY+Lldu4+R/9nNzP7x8NUrZjUoI6+SRddeKjqeGA8QGSZXEqxhlf+YUiJclf8YfCxnK//SJ7r8rVcqn2ZJXPM4ssYY/xNVbm73Yds2XKcUaNuZcCAumTOPOiq1xfIRLGHyweXj3Bf8y40K1Tp4a+YjDEm3Vq6dDdVqhQiZ84wPvrodgoUyEbx4rmveb2BvDx2JtDVvfqpHnDc2ieMMebKHTlyhocfnslNN33CqFG/AVCjRpEUSRLgxzMKEfkKaAIUEJEo4AUgM4CqjgO+xxk8fjtwBujur1iMMSYYqSqffrqG//xnPkePnmXw4AYMHtwgxbfjz6ueOiczX4F+/tq+McYEuyefXMAbbyylQYPijBt3G1WqFPbLdtJFY7YxxhjH2bPRnD4dTYEC2ejZswZly+ajZ8+aZMrkvwtGrAsPY4xJJ+bM2U7lyu/zyCOzAChXrgAPP1zLr0kCLFEYY0yat3fvSTp2nELr1l+QOXMm+vevnarbt6onY4xJw378cQd33jmZCxdiGT78FgYPbkBYWOp+dVuiMMaYNCg6OpbMmUOoVu062rQpy4gRTbnhhnwBicWqnowxJg05ceI8Awf+QKNG/yU2No4CBbIxaVKHgCUJsERhjDFpgqoyZcoGypcfzXvv/U5kZFHOn48NdFiAVT0ZY0zAHTp0mgcfnM4PP2ynRo3rmDHjXmrXLhbosOJZojDGmADLlSuMw4fP8PbbLenXrw6hoWmrsidtRWOMMRnE4sV/07Ll55w6dYGwsFCWLXuIgQPrpbkkAZYojDEmVR0+fIbu3Wdw880T2Lr1CDt3HgPw+01z18KqnowxJhWoKv/972oGD57PiRPnefrphjz7bGOyZcsc6NCSZYnCGGNSyeefr6VixYKMG3cblSoVCnQ4PrNEYYwxfnLmTDQvv7yE3r0jiYjIxdSpHcmdO2uarmZKjLVRGGOMH3z//TYqVRrLSy8t4bvvtgCQN294uksSYGcUxhiToqKiTvDYY3OYOnUTFSoU4Oefu9G48fWBDuuaWKIwxpgU9NJLi5k9exsvv9yUQYMakCVLSKBDumbiDDSXfkSWK6QrthwMdBjGGBPv99/3EB4eSpUqhTly5AzHj5+ndOm8gQ7rMiKyUlUjr2ZZa6MwxpirdPz4Ofr1m029eh8xdOhPAOTPny3NJYlrZVVPxhhzhVSVyZM38Pjjczl48DSPPlqH4cObBjosv7FEYYwxV+jzz9fStet0IiOLMmtWZ2rVKhrokPzKEoUxxvjg/PkYduw4SoUKBenYsRIxMXF07VqNkJDgr8EP/j00xphrtHDhX1SrNo6WLT/n/PkYwsJC6d69RoZIEmCJwhhjknTw4Gm6dp1G06afEh0dx/jxt6f6eNVpQcbbY2OM8cH27f9Qp86HnDp1gaFDGzF0aCPCw9N+B37+YInCGGM8nDhxnly5wihTJi89e9agR48aVKhQMNBhBZRVPRljDHD69AWefHI+JUu+TVTUCUSEN964NcMnCbAzCmOM4bvvttC//w/s2nWcnj1rpIsxIlKTJQpjTIYVExNHx45TmDZtM5UqFWTJku40bFgi0GGlOZYojDEZjqoiIoSGZqJIkRy8+mozHn+8flB04OcP1kZhjMlQli2LIjLyQ1at2gfAmDG38eSTDS1JeGGJwhiTIRw9epY+fWbRoMHHHDhwiqNHzwY6pHTDr4lCRFqJyBYR2S4iTyUyv4SILBSRP0RkrYi08Wc8xpiMafLk9ZQvP4bx41fx2GP12LSpH82alQ50WOmG39ooRCQEGAO0AKKA5SIyU1U3ehR7FvhaVd8XkYrA90BJf8VkjMmYNm8+TMmSeZgz5z5q1CgS6HDSHX+eUdQBtqvqDlW9AEwC2icoo0Au93luYK8f4zHGZBDnzsXwf/+3KH6s6meeacTSpT0sSVwlfyaKYsBuj+ko9zVPw4D7RSQK52zi0cRWJCK9RGSFiKw4e9bqFY0xSVuwYAdVq77PsGE/8/PPfwOQOXNIhunAzx8CfeQ6AxNUNQJoA3wmIv+KSVXHq2qkqkaGh4enepDGmLTvwIFT3Hfft7Ro8RmqMG/e/YwceWugwwoK/ryPYg9Q3GM6wn3NU0+gFYCq/iYiWYECgA2KbYy5IvPn7+Cbbzby/PONefrpRmTNareJpRR/HsnlQFkRKYWTIO4FuiQoswtoBkwQkQpAVuCQH2MyxgSRNWv2s23bP3ToUJH77qvCTTcVp1Sp4BqvOi3wW9WTqsYA/YG5wCacq5s2iMiLItLOLTYIeFhE1gBfAd1UVf0VkzEmOJw6dYFBg+ZSq9Z4nnpqATExcYiIJQk/kfT2vRxZrpCu2GI1U8ZkVNOnb+bRR38gKuoEvXrV5JVXmpMvn7VdJkdEVqpq5NUsa5V4xph0Y926A9x552SqVCnE5MkdaNCgePILmWtmicIYk6ZFR8eyZMkumjYtRZUqhZk9uwstWpQmc2brmym1BPryWGOMSdLSpbupVWs8LVp8xvbt/wDQpk1ZSxKpzBKFMSbN+eefs/Tq9R033fQJx46d49tvO3LDDfkCHVaGZVVPxpg05dy5GKpXH8fevScZNKg+w4Y1IUeOLIEOK0OzRGGMSROiok4QEZGLrFlDGT78FqpXv45q1a4LdFgGq3oyxgTY2bPRPP/8QsqUeTe+E78HH6xuSSINsTMKY0zAzJv3J337zubPP49y//1VqVMnYb+hJi3wOVGISDZVPePPYIwxGcejj37P6NHLKVs2HwsWPGADCaVhySYKEWkAfATkAEqISDXgEVXt6+/gjDHBJTY2DoCQkEzUqxdBgQLZePLJhtaBXxrnSxvFW0BL4AiAqq4BGvszKGNM8Fm1ah/163/M2LHLAbjvvqq88EITSxLpgE+N2aq6O8FLsX6IxRgThE6ePM/jj8+hdu0P2bXrOEWK5Ax0SOYK+ZLKd7vVTyoimYGBOL3BGmOMV/Pm/UmPHjPYu/ckvXtH8vLLzciTJ2ugwzJXyJdE0Rt4B2cY0z3APMDaJ4wxycqSJYRChbIzdWpH6taNCHQ45iol2824iNykqr8m91pqsW7GjUm7oqNjefPN3zhx4jwvvdQMgLg4JVMmCXBk5lq6GfeljeI9H18zxmRgv/yyixo1PuCpp35k27Z/iItzfoRakkj/kqx6EpH6QAOgoIg84TErF2BdNxpjADhy5AxPPrmAjz/+gxIlcvPdd51p2/bGQIdlUpC3NoosOPdOhAKelymcADr4MyhjTPpx5MhZJk1az5AhDXj++ZvJnt068As2vrRRXK+qf6dSPMmyNgpjAm/TpkN8/fUGXnihCeB0C27DkaZt/h4K9YyIvAFUAuKva1PVplezQWNM+nXmTDQvvbSYN95YSo4cWejZsyYREbksSQQ5XxqzvwA2A6WA/wN2Asv9GJMxJg2aM2c7lSuP5eWXf6FLlyps2dKfiIhcgQ7LpAJfzijyq+rHIjJQVX8GfhYRSxTGZCCnTl3ggQemkT9/OAsXPkiTJiUDHZJJRb4kimj37z4RuQ3YC9iYhMYEudjYOL76aj2dO1cmR44sLFjwAOXLFyAszPpmymh8ecdHiEhuYBDO/RO5gMf8GZQxJrBWrtzLI4/MYuXKfYSHh3L33RVtIKEMLNk2ClWdparHVXW9qt6iqrWAf1IhNmNMKjt+/BwDBvxAnTofsWfPSSZNupu77qoQ6LBMgHm74S4E6IjTx9McVV0vIm2BZ4BwoEbqhGiMSS133/01P/30F/361WbEiKbkzm0d+BnvVU8fA8WB34F3RWQvEAk8parTUyE2Y0wq2LHjKAULZiNnzjBeeqkpmTIJtWvbkKTmEm+JIhKoqqpxIpIV2A+UUdUjqROaMcafLlyIZeTIpQwfvpgBA+rw2mstrIdXkyhvieKCqsYBqOo5EdlhScKY4LB48d/07j2LTZsO06FDRQYMqBvokEwa5i1RlBeRte5zAcq40wKoqlb1e3TGmBT31lu/8cQT8yhZMg+zZ3ehTZuygQ7JpHHeEoVd6mBMkIiLU06fvkDOnGHcdtuNHDp0hmefbUy2bJkDHZpJB5LtFDCtsU4BjbkyGzYcpHfv2fEjzZmMyd8DF101EWklIltEZLuIPJVEmY4islFENojIl/6Mx5iM5MyZaJ5+egHVq3/Apk2HaNu2LOnth6FJG/x2L757H8YYoAUQBSwXkZmqutGjTFngaeAmVT0qIoX8FY8xGckff+zjrru+ZufOY3TvXp3XX29BgQLZAh2WSad8ShQiEg6UUNUtV7DuOsB2Vd3hrmMS0B7Y6FHmYWCMqh4FUFWrUzLmGqgqIkKJErkpUSI3EyfeQePG1wc6LJPOJVv1JCK3A6uBOe50dRGZ6cO6iwG7Paaj3Nc83QjcKCK/isgyEWnlU9TGmMvExMTx9tvLaNbsU2Jj48ifPxs//9zNkoRJEb60UQzDOTs4BqCqq3HGpkgJoUBZoAnQGfhQRPIkLCQivURkhYisOHv2bApt2pjg8Pvve6hT50Mef3wuWbOGcuLE+UCHZIKML4kiWlWPJ3jNlxaxPThdgFwU4b7mKQqYqarRqvoXsBUncVy+MdXxqhqpqpHh4TaSljHgjBHRr99s6tX7iAMHTjNlyj3Mnt2FvHntf8SkLF8SxQYR6QKEiEhZEXkPWOrDcsuBsiJSSkSyAPcCCauspuOcTSAiBXCqonb4GLsxGVrmzJlYtOhvHn20Dps29aNDh4qISKDDMkHIl0TxKM542eeBL4Hj+DAeharGAP2BucAm4GtV3SAiL4pIO7fYXOCIiGwEFgKDrZsQY5K2ffs/dO06jZMnzxMWFsrKlb14553W5MoVFujQTBBL9oY7EampqqtSKZ5k2Q13JiM6fz6G11//lZdeWkKWLCHMnt2FRo2sodr47lpuuPPl8thRInId8A0wWVXXX82GjDFXZ+HCv+jTZzZbthyhU6dKvPlmS4oWzRnosEwGkmyiUNVb3ETREfhARHLhJIwRfo/OmAxOVXnppSVER8cxZ859tGx5Q6BDMhnQFfX1JCJVgCFAJ1XN4reovLCqJxPs4uKUjz9eRatWN1C8eG727TtJnjxZCQ+3DvzM1fNrX08iUkFEhonIOuDiFU82uokxfrB27QEaNvyEXr1m8dFHTtNgkSI5LUmYgPKljeITYDLQUlX3+jkeYzKkU6cu8H//t4i33lpG3rzhTJjQnq5dqwU6LGMA39oo6qdGIMZkZMOGLWLUqN946KEavPpqc/Lntw78TNqRZBuFiHytqh3dKifPQgEd4c7aKEyw2L37OKdPR1O+fAEOHz7D5s2HadiwRKDDMkHKX5fHDnT/tr2aFRtjEhcTE8e77/6P559fSK1aRfn5524UKJDNkoRJs5JszFbVfe7Tvqr6t+cD6Js64RkTXJYtiyIycjyDBs2jSZOSTJx4R6BDMiZZvnTh0SKR11qndCDGBLvZs7fSoMHHHD58hm+/7ch333WmZMk8gQ7LmGQlWfUkIn1wzhxKi8haj1k5gV/9HZgxwUBV2bv3JMWK5aJ589K8+OItDBxYl5w5rW8mk354a8zODeQFXgE8x7s+qar/pEJsibLGbJNebN16hL59Z7N16xE2buxHjhwBuUfVGMB/jdmqqjtFpF8iG8wXyGRhTFp27lwMr776C6+88gvh4aG88kozwsP9Njy9MX7n7dP7Jc4VTytxLo/17OhegdJ+jMuYdGn//lM0bvxftm37h86dK/Pmmy257rocgQ7LmGuSZKJQ1bbu35Qa9tSYoBUdHUvmzCEULpydxo2vZ8yYNrRoUSbQYRmTInzp6+kmEcnuPr9fRN4UEbvg2xicDvzGjVtBmTLvEhV1AhHho4/aWZIwQcWXy2PfB86ISDVgEPAn8JlfozImHVizZj8NGnxMnz6zKVs2P9HRsYEOyRi/8CVRxKhzaVR7YLSqjsG5RNaYDElV+c9/5lGr1nh27DjKZ5/dyYIFD1CqVN5Ah2aMX/hyKcZJEXkaeABoJCKZAOvz2GRYIsLRo2fp2dPpwC9v3vBAh2SMX/lyRtEJOA/0UNX9OGNRvOHXqIxJY/7++xh33DGJVaucnm0+/LAdH3xwuyUJkyEkmyjc5PAFkFtE2gLnVPVTv0dmTBoQHR3L66//SsWKY5k/fwdbthwGIFMmSWZJY4JHslVPItIR5wxiEc69FO+JyGBV/cbPsRkTUEuX7uaRR2axfv1B2rcvx7vvtqZEidyBDsuYVOdLG8VQoLaqHgQQkYLAAsAShQlqCxbs4Pjxc0yf3on27csHOhxjAibJvp7iC4isU9UqHtOZgDWer6Um6+vJ+Iuq8tlnaylYMButW5fl/PkYoqPjrI8mExT81dfTRXNEZC7wlTvdCfj+ajZmTFq1efNh+vSZzaJFO7nnnoq0bl2WsLBQwqyTV2N8GjN7sIjcBTR0XxqvqtP8G5YxqePs2WhefnkJr732K9mzZ+GDD9ry0EM1Ax2WMWmKt/EoygIjgTLAOuA/qrontQIzJjV8991WRoxYwv33V2XkyBYULmwd+BmTkLczik+AT4HFwO3Ae8BdqRGUMf60f/8pVq/eT6tWN3DPPRUpWfIh6tQpFuiwjEmzvCWKnKr6oft8i4isSo2AjPGX2Ng4PvhgJU8//SNZsoSwa9djhIdntiRhTDK8JYqsIlKDS+NQhHtOq6olDpNurFq1j969Z7F8+V6aNy/N2LFtCA+3nmiM8YW3RLEPeNNjer/HtAJN/RWUMSnpr7+OUqfOhxQokI0vv7yLe++tjIjdWW2Mr7wNXHRLagZiTEpSVdatO0jVqoUpVSov//1ve26/vRx58mQNdGjGpDu+dApoTLry119Hadv2K2rU+IC1aw8A8MAD1SxJGHOV/JooRKSViGwRke0i8pSXcneLiIrIVd01aAzAhQuxvPrqL1SqNJaff97JyJEtqFixYKDDMibd8+XO7KsiIiHAGKAFEAUsF5GZqroxQbmcwEDgf/6KxQS/2Ng4GjT4mJUr93HXXRV4++2WFC9uHfgZkxJ8GTNb3LGyn3enS4hIHR/WXQfYrqo7VPUCMAlnlLyEhgOvAeeuIG5jADhx4jwAISGZ6NGjBt9915mpUztakjAmBflS9TQWqA90dqdP4pwpJKcYsNtjOsp9LZ6I1ASKq+psbysSkV4iskJEVpw9e9aHTZtgp6pMmLCa0qXfYcaMzQD07Vubtm1vDHBkxgQfXxJFXVXth/uLX1WPAtfcnabbC+2bwKDkyqrqeFWNVNXI8HAbUSyj27jxEE2aTKR79xmUL1+AMmXyBTokY4KaL20U0W57g0L8eBRxPiy3ByjuMR3hvnZRTqAysMi9pv06YKaItFPVFT6s32RAr7/+K0OH/kSuXGF89NHtdO9ew0abM8bPfEkU7wLTgEIi8hLQAXjWh+WWA2VFpBROgrgX6HJxpqoeBwpcnBaRRTgdD1qSMP+iqogI112Xg/vuq8Ibb7SgYMHsgQ7LmAzBl27GvxCRlUAznO477lDVTT4sFyMi/YG5QAjwiapuEJEXgRWqOvMaYzcZwN69Jxk4cA6NGpVgwIC6dO1aja5dqwU6LGMyFF/GzC4BnAG+83xNVXclt6yqfk+CQY5U9fkkyjZJbn0m44iNjWPs2OUMHfoT0dFxNGgQEeiQjMmwfKl6mo3TPiFAVqAUsAWo5Me4TAa2evV+HnpoJitX7uPWW8swdmwba7A2JoB8qXq6bGxs95LWvn6LyGR4x4+fY+/ek0ye3IF77qloHfgZE2BXfGe2qq4Skbr+CMZkTKrKlCkb2bbtCEOHNubmm0uyY8dAsmb1W8cBxpgr4EsbxRMek5mAmsBev0VkMpQ///yH/v1/YM6c7dSuXZQhQ24ic+YQSxLGpCG+/Dfm9Hgeg9NmMdU/4ZiM4vz5GEaOXMqIEUvInDkT77zTir59axMaah0aG5PWeE0U7o12OVX1P6kUj8kgdu8+wfDhi7n99nK8/XZLihXLFeiQjDFJSDJRiEioey/ETakZkAlehw6dZvLkDfTvX4cbbsjHxo39KF06b6DDMsYkw9sZxe847RGrRWQmMAU4fXGmqn7r59hMkIiLU/773z8YMmQBJ0+ep0WL0pQrV8CShDHphC9tFFmBIzhjZF+8n0IBSxQmWevXH6RPn9n88ssuGjUqwbhxbSlXrkDyCxpj0gxviaKQe8XTei4liIvUr1GZoHDhQiy33voZFy7E8skn7ejWrbrdE2FMOuQtUYQAObg8QVxkicIk6aef/uLmm68nS5YQvv76HsqXL0CBAtkCHZYx5ip5SxT7VPXFVIvEpHtRUScYOHAO3367iU8+aUf37jVo2LBEoMMyxlwjb4nC6giMT2Ji4hg9+neee24hsbFxvPJKM+67r2qgwzLGpBBviaJZqkVh0rUHHpjGpEnrad36BsaMaUOpUnY1kzHBJMlEoar/pGYgJn05duwcoaGZyJEjC/361ebuuytw990VrLHamCBk/SWYK6KqTJq0ngoVxvDccz8B0LBhCTp0sF5ejQlWliiMz7Zv/4eWLT+nc+epRETk4v77rR3CmIzAuug0Pvnyy3X06DGDsLBQRo9uTe/ekYSE2O8MYzICSxTGq+joWDJnDiEysigdOlTk9ddbULRozuQXNMYEDVFNX/fORZYrpCu2HAx0GEHv4MHTDBo0j9OnL/Dtt50CHY4x5hqJyEpVjbyaZa3uwFwmLk4ZP34l5cqNZvLk9VSqVJDY2LhAh2WMCSCrejLxduw4yv33f8tvv0XRpElJ3n//NsqXtw78jMnoLFGYeLlzh3Hs2DkmTryDBx6oape7GmMAq3rK8GbO3MJdd00mNjaO/PmzsX59X7p2rWZJwhgTzxJFBrVr13HuuGMS7dtPYuvWI+zbdwqATJksQRhjLmdVTxlMTEwcb7+9jBdeWISq8tprzXn88XpkzhwS6NCMMWmUJYoMJjY2jo8+WkXTpqV4773WlCyZJ9AhGWPSOKt6ygCOHj3Lk0/O5+TJ84SFhfLrrz2YOfNeSxLGGJ9YoghiqsoXX6ylfPkxjBr1GwsX7gQgf/5s1lhtjPGZVT0Fqa1bj9C372x+/PEv6tQpxty591O9+nWBDssYkw5ZoghSjz02hxUr9jJ2bBt69aplHfgZY66aJYogMn/+n5QvX4DixXPz/vu3ERYWynXX5Qh0WMaYdM6vPzNFpJWIbBGR7SLyVCLznxCRjSKyVkR+FJHr/RlPsNq//xRdukzl1ls/57XXfgXg+uvzWJIwxqQIvyUKEQkBxgCtgYpAZxGpmKDYH0CkqlYFvgFe91c8wSguThk3bgXly49m6tRNvPDCzYwceWugwzLGBBl/nlHUAbar6g5VvQBMAtp7FlDVhap6xp1cBkT4MZ6g88orS+jTZza1ahVl7dreDBvWhKxZrTbRGJOy/PmtUgzY7TEdBdT1Ur4n8ENiM0SkF9ALoGLxjF2dcvLkeQ4fPkOpUnnp3TuSUqXy0rlzZbvc1RjjN2niUhgRuR+IBN5IbL6qjlfVSFWNDA8PT93g0ghVZdq0TVSsOJZOnb5BVcmfPxtdulSxJGGM8St/Joo9QHGP6Qj3tcuISHNgKNBOVc/7MZ506++/j9Gu3STuuutr8uUL5913W1tyMMakGn9WPS0HyopIKZwEcS/QxbOAiNQAPgBaqaqNb5qI337bTfPmnwEwcmQLBg6sR2homjgRNMZkEH5LFKoaIyL9gblACPCJqm4QkReBFao6E6eqKQcwxf2FvEtV2/krpvTkxInz5MoVRs2aRejRozqDB99EiRK5Ax2WMSYDElUNdAxXJLJcIV2xJXhPPo4cOcNTTy1g3rwdbNjQlxw5sgQ6JGNMEBCRlaoaeTXL2rWUaYSq8tlnaxk0aB5Hj57liSfqY80Qxpi0wBJFGnD8+DnuuGMyixbtpH79CMaNa0vVqoUDHZYxxgCWKAJKVRERcuUKo0CBbIwf35aePWvacKTGmDTFLp8JkLlzt1Oz5niiok4gIkyZcg8PP1zLkoQxJs2xRJHK9u07yb33fkOrVl9w5kw0Bw+eDnRIxhjjlVU9paIxY37nmWd+4vz5GP7v/5rw5JM3ERZmb4ExJm2zb6lUtHLlPurWLcaYMW0oWzZ/oMMxxhifWKLwoxMnzvP88wt54IGq1KpVlLFjbyMsLMS63zDGpCuWKPxAVZk6dRMDB85h376TlCiRm1q1iloX4MaYdMm+uVLYX38dpX//H/j++21Ur34d337bkbp1bZgNY0z6ZYkihX3xxToWL/6bt95qSf/+dawDP2NMumd9PaWAJUv+5vz5WJo3L8358zEcOnSGiIhcgQ7LGGPiXUtfT/Zz9xocPnyGHj1m0LjxBF588WcAwsJCLUkYY4KKVT1dBVVlwoTVDB48n+PHz/Pkkzfx3HONAx2WSWOio6OJiori3LlzgQ7FZCBZs2YlIiKCzJkzp9g6LVFche+/30aPHjO56abijBvXlsqVCwU6JJMGRUVFkTNnTkqWLGmXRJtUoaocOXKEqKgoSpUqlWLrtaonH505E82vv+4CoE2bssyYcS+LF3e3JGGSdO7cOfLnz29JwqQaESF//vwpfhZricIHP/ywjcqVx9K69RccO3YOEaFdu3LWgZ9JliUJk9r88ZmzROHFnj0nuOeeKbRp8yVhYaF8911n8uTJGuiwjDEmVVmiSMLBg6epWHEss2ZtZcSIW1izpjc331wy0GEZc0VCQkKoXr06lStX5vbbb+fYsWPx8zZs2EDTpk0pV64cZcuWZfjw4XheLv/DDz8QGRlJxYoVqVGjBoMGDQrAHnj3xx9/0LNnz0CHkaTFixdTs2ZNQkND+eabb5Ist3LlSqpUqcINN9zAgAED4t+Hf/75hxYtWlC2bFlatGjB0aNHAZg1axbPP/98quwD4DR+pKdHrRsLqj9FRR2Pf/7OO8t0+/Yjft2eCV4bN24MdAiaPXv2+Oddu3bVESNGqKrqmTNntHTp0jp37lxVVT19+rS2atVKR48eraqq69at09KlS+umTZtUVTUmJkbHjh2borFFR0df8zo6dOigq1evTtVtXom//vpL16xZow888IBOmTIlyXK1a9fW3377TePi4rRVq1b6/fffq6rq4MGD9ZVXXlFV1VdeeUWHDBmiqqpxcXFavXp1PX36dKLrS+yzB6zQq/zetTMK1/Hj53j00e8pVeodVq3aB8CAAXUpUyZfgCMzJmXUr1+fPXv2APDll19y0003ceuttwKQLVs2Ro8ezauvvgrA66+/ztChQylfvjzgnJn06dPnX+s8deoU3bt3p0qVKlStWpWpU6cCkCNHjvgy33zzDd26dQOgW7du9O7dm7p16zJkyBBKlix52VlO2bJlOXDgAIcOHeLuu++mdu3a1K5dm19//fVf2z558iRr166lWrVqAPz+++/Ur1+fGjVq0KBBA7Zs2QLAhAkTaNeuHU2bNqVZs2acPn2aHj16UKdOHWrUqMGMGTMA2LlzJ40aNaJmzZrUrFmTpUuXXvWxvqhkyZJUrVqVTJmS/qrdt28fJ06coF69eogIXbt2Zfr06QDMmDGDBx98EIAHH3ww/nURoUmTJsyaNeuaY/RFhr88VlWZMmUjjz02h/37T9G/fx3KlMkb6LBMMBrlh4btQb71rBAbG8uPP/4YX02zYcMGatWqdVmZMmXKcOrUKU6cOMH69et9qmoaPnw4uXPnZt26dQDxVSPeREVFsXTpUkJCQoiNjWXatGl0796d//3vf1x//fUULlyYLl268Pjjj9OwYUN27dpFy5Yt2bRp02XrWbFiBZUrV46fLl++PEuWLCE0NJQFCxbwzDPPxCeuVatWsXbtWvLly8czzzxD06ZN+eSTTzh27Bh16tShefPmFCpUiPnz55M1a1a2bdtG586dWbFixb/ib9SoESdPnvzX6yNHjqR58+bJ7n9Ce/bsISLiUn9wERER8Qn9wIEDFClSBIDrrruOAwcOxJeLjIxkyZIldOzY8Yq3eaUydKJQVe6662umT99MzZpFmDmzM5GRRQMdlglWPn6pp6SzZ89SvXp19uzZQ4UKFWjRokWKrn/BggVMmjQpfjpv3uR/ZN1zzz2EhIQA0KlTJ1588UW6d+/OpEmT6NSpU/x6N27cGL/MiRMnOHXq1GVnKvv27aNgwYLx08ePH+fBBx9k27ZtiAjR0dHx81q0aEG+fE7twLx585g5cyYjR44EnMuYd+3aRdGiRenfvz+rV68mJCSErVu3Jhr/kiVLkt1HfxCRy65oKlSoEHv37k2VbWfIRBEdHUvmzM64EA0bFqdp05L07VubkBCriTPBJTw8nNWrV3PmzBlatmzJmDFjGDBgABUrVmTx4sWXld2xYwc5cuQgV65cVKpUiZUrV8ZX61wpzy+0hNf0Z8+ePf55/fr12b59O4cOHWL69Ok8++yzAMTFxbFs2TKyZk36KsPw8PDL1v3cc89xyy23MG3aNHbu3EmTJk0S3aaqMnXqVMqVK3fZ+oYNG0bhwoVZs2YNcXFxSW47pc8oihUrRlRUVPx0VFQUxYoVA6Bw4cLs27ePIkWKsG/fPgoVunTf1rlz5wgPD7/i7V2NDPfNuGjRTqpWHceMGZsBGDSoAY8+WteShAlq2bJl491332XUqFHExMRw33338csvv7BgwQLAOfMYMGAAQ4YMAWDw4MG8/PLL8b+q4+LiGDdu3L/W26JFC8aMGRM/fbHqqXDhwmzatIm4uDimTZuWZFwiwp133skTTzxBhQoVyJ/fGfnx1ltv5b333osvt3r16n8tW6FCBbZv3x4/ffz48fgv2AkTJiS5zZYtW/Lee+/FX1n0xx9/xC9fpEgRMmXKxGeffUZsbGyiyy9ZsoTVq1f/63E1SQKgSJEi5MqVi2XLlqGqfPrpp7Rv3x6Adu3aMXHiRAAmTpwY/zrA1q1bL6t686cM8+146NBpHnxwOrfcMpHz52PImTMs0CEZk6pq1KhB1apV+eqrrwgPD2fGjBmMGDGCcuXKUaVKFWrXrk3//v0BqFq1Km+//TadO3emQoUKVK5cmR07dvxrnc8++yxHjx6lcuXKVKtWjYULFwLw6quv0rZtWxo0aBBfx56UTp068fnnn8dXOwG8++67rFixgqpVq1KxYsVEk1T58uU5fvx4/K/7IUOG8PTTT1OjRg1iYmKS3N5zzz1HdHQ0VatWpVKlSjz33HMA9O3bl4kTJ1KtWjU2b9582VnI1Vq+fDkRERFMmTKFRx55hEqVKsXPq169evzzsWPH8tBDD3HDDTdQpkwZWrduDcBTTz3F/PnzKVu2LAsWLOCpp56KX2bhwoXcdttt1xyjLzJEN+NffbWOfv2+59SpCwwe3IChQxuTLVvKdZhlTGI2bdpEhQoVAh1GUHvrrbfImTMnDz30UKBDSVUHDhygS5cu/Pjjj4nOT+yzZ92MJyMmJo7KlQuxenVvXnqpmSUJY4JEnz59CAvLeLUDu3btYtSoUam2vaA8ozh9+gLDhy+mRInc9O1bO74u0vrdManJzihMoNgZRTJmzdpKpUpjee21X9m69Qjw78vKjEkt6e2HmEn//PGZC5rLY6OiTjBgwA9Mm7aZihULsnhxNxo1uj7QYZkMLGvWrBw5csS6GjepRt3xKLxdVnw1giZR7NhxlLlz/+SVV5rxxBP1yZIlJNAhmQwuIiKCqKgoDh06FOhQTAZycYS7lJSu2yh+/30Pv/22m4ED6wFw5MgZ8ufPFsjwjDEmTUqzbRQi0kpEtojIdhF5KpH5YSIy2Z3/PxEp6ct6jx07R9++s6lX7yPefHMZp09fALAkYYwxfuC3RCEiIcAYoDVQEegsIhUTFOsJHFXVG4C3gNeSW+8/JzJRvvxoPvhgJQMG1GXduj5kz54lpcM3xhjj8mcbRR1gu6ruABCRSUB7YKNHmfbAMPf5N8BoERH1Uh+280AmatXKzfff30fNmt7v+DTGGHPt/JkoigG7PaajgLpJlVHVGBE5DuQHDnsWEpFeQC938vyKFb3WJ+ghOaMqQIJjlYHZsbjEjsUldiwuKZd8kcSli6ueVHU8MB5ARFZcbYNMsLFjcYkdi0vsWFxix+ISEfn34Bo+8mdj9h6guMd0hPtaomVEJBTIDRzxY0zGGGOukD8TxXKgrIiUEpEswL3AzARlZgIPus87AD95a58wxhiT+vxW9eS2OfQH5gIhwCequkFEXsQZ5Hsm8DHwmYhsB/7BSSbJGe+vmNMhOxaX2LG4xI7FJXYsLrnqY5HubrgzxhiTuoKuU0BjjDEpyxKFMcYYr9JsovBX9x/pkQ/H4gkR2Sgia0XkRxEJ2m5zkzsWHuXuFhEVkaC9NNKXYyEiHd3PxgYR+TK1Y0wtPvyPlBCRhSLyh/t/0iYQcfqbiHwiIgdFZH0S80VE3nWP01oRqenTilU1zT1wGr//BEoDWYA1QMUEZfoC49zn9wKTAx13AI/FLUA293mfjHws3HI5gcXAMiAy0HEH8HNRFvgDyOtOFwp03AE8FuOBPu7zisDOQMftp2PRGKgJrE9ifhvgB0CAesD/fFlvWj2jiO/+Q1UvABe7//DUHpjoPv8GaCbB2el/ssdCVReq6hl3chnOPSvByJfPBcBwnH7DzqVmcKnMl2PxMDBGVY8CqOqVDTaffvhyLBTI5T7PDexNxfhSjaouxrmCNCntgU/VsQzIIyLJ9oWUVhNFYt1/FEuqjKrGABe7/wg2vhwLTz1xfjEEo2SPhXsqXVxVZ6dmYAHgy+fiRuBGEflVRJaJSKtUiy51+XIshgH3i0gU8D3waOqEluZc6fcJkE668DC+EZH7gUjg5kDHEggikgl4E+gW4FDSilCc6qcmOGeZi0WkiqoeC2RQAdIZmKCqo0SkPs79W5VVNS7QgaUHafWMwrr/uMSXY4GINAeGAu1U9XwqxZbakjsWOYHKwCIR2YlTBzszSBu0fflcRAEzVTVaVf8CtuIkjmDjy7HoCXwNoKq/AVlxOgzMaHz6PkkorSYK6/7jkmSPhYjUAD7ASRLBWg8NyRwLVT2uqgVUtaSqlsRpr2mnqlfdGVoa5sv/yHScswlEpABOVdSOVIwxtfhyLHYBzQBEpAJOosiIY9TOBLq6Vz/VA46r6r7kFkqTVU/qv+4/0h0fj8UbQA5gituev0tV2wUsaD/x8VhkCD4ei7nArSKyEYgFBqtq0J11+3gsBgEfisjjOA3b3YLxh6WIfIXz46CA2x7zApAZQFXH4bTPtAG2A2eA7j6tNwiPlTHGmBSUVquejDHGpBGWKIwxxnhlicIYY4xXliiMMcZ4ZYnCGGOMV5YoTJokIrEistrjUdJL2VMpsL0JIvKXu61V7t27V7qOj0Skovv8mQTzll5rjO56Lh6X9SLynYjkSaZ89WDtKdWkHrs81qRJInJKVXOkdFkv65gAzFLVb0TkVmCkqla9hvVdc0zJrVdEJgJbVfUlL+W74fSg2z+lYzEZh51RmHRBRHK4Y22sEpF1IvKvXmNFpIiILPb4xd3Iff1WEfnNXXaKiCT3Bb4YuMFd9gl3XetF5DH3tewiMltE1rivd3JfXyQikSLyKhDuxvGFO++U+3eSiNzmEfMEEekgIiEi8oaILHfHCXjEh8PyG26HbiJSx93HP0RkqYiUc+9SfhHo5MbSyY39ExH53S2bWO+7xlwu0P2n28MeiT1w7iRe7T6m4fQikMudVwDnztKLZ8Sn3L+DgKHu8xCcvp8K4HzxZ3dffxJ4PpHtTQA6uM/vAf4H1ALWAdlx7nzfANQA7gY+9Fg2t/t3Ee74Fxdj8ihzMcY7gYnu8yw4PXmGA72AZ93Xw4AVQKlE4jzlsX9TgFbudC4g1H3eHJjqPu8GjPZY/mXgfvd5Hpz+n7IH+v22R9p+pMkuPIwBzqpq9YsTIpIZeFlEGgNxOL+kCwP7PZZZDnzilp2uqqtF5GacgWp+dbs3yYLzSzwxb4jIszh9APXE6RtomqqedmP4FmgEzAFGichrONVVS65gv34A3hGRMKAVsFhVz7rVXVVFpINbLjdOB35/JVg+XERWu/u/CZjvUX6iiJTF6aIicxLbvxVoJyL/caezAiXcdRmTKEsUJr24DygI1FLVaHF6h83qWUBVF7uJ5DZggoi8CRwF5qtqZx+2MVhVv7k4ISLNEiukqlvFGfeiDTBCRH5U1Rd92QlVPScii4CWQCecQXbAGXHsUVWdm8wqzqpqdRHJhtO3UT/gXZzBmhaq6p1uw/+iJJYX4G5V3eJLvMaAtVGY9CM3cNBNErcA/xoXXJyxwg+o6ofARzhDQi4DbhKRi20O2UXkRh+3uQS4Q0SyiUh2nGqjJSJSFDijqp/jdMiY2LjD0e6ZTWIm43TGdvHsBJwv/T4XlxGRG91tJkqdEQ0HAIPkUjf7F7uL7uZR9CROFdxFc4FHxT29EqfnYWO8skRh0osvgEgRWQd0BTYnUqYJsEZE/sD5tf6Oqh7C+eL8SkTW4lQ7lfdlg6q6Cqft4necNouPVPUPoArwu1sF9AIwIpHFxwNrLzZmJzAPZ3CpBeoM3QlOYtsIrBKR9Tjdxns943djWYszKM/rwCvuvnsutxCoeLExG+fMI7Mb2wZ32hiv7PJYY4wxXtkZhTHGGK8sURhjjPHKEoUxxhivLFEYY4zxyhKFMcYYryxRGGOM8coShTHGGK/+H0vgYEL2WkQ7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr, tpr, thresholds = roc_curve(true_labels, pred_labels)\n",
    "roc_auc = roc_auc_score(true_labels, pred_labels)\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(fpr, tpr, color='darkorange', lw=1, label='ROC curve (area = %0.2f)' % roc_auc)\n",
    "plt.plot([0, 1], [0, 1], color='navy', linestyle='--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The closer the curve is to the left-hand and the top-hand borders of the ROC plotting area, the better the model is performing over all thresholds. The area under the ROC curve (AUC) is a way of quantifying this performance. 1 is a perfect score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.10 Model Inspection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is our model performing so well? One tool we can use from XGBoost is the feature importance.  XGBoost will rank the features that it uses in determining a classification.  We can visualize the feature importance using the \"plot_importance\" method.  \n",
    "\n",
    "Knowing which features are contributing the model can be really useful for solving business problems.  In the case of network security, those features can give us way to possibly preempt malicious activity.\n",
    "\n",
    "Feature importance can also be a great way to debug your model. \n",
    "\n",
    "You might have to run this cell twice to see the feature importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAJcCAYAAADD8tPLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAB7jklEQVR4nOzde5xVdb3/8ddbQARUlPACDoo0hBdAEkjsKI4ZYqCoZV6yg4hmWIrlJTmaHC1N1KxM7cdBTVA7iHpUTBHoqDvJC8hFxUtEHigdR4ybOmjK4Of3x1pMm7kwA649N97Px4MHe33Xd32/n7W/U372dz5ro4jAzMzMzMyysV1jB2BmZmZm1pI4wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzM2vWJF0m6fbGjsPMbCP5e7DNzLZdkpYDewAb8pq/EBFvf8Yxz46I//1s0TU/kq4EiiPi240di5k1Hu9gm5nZcRGxY96frU6usyCpdWPOv7Waa9xmlj0n2GZmVo2kjpLukFQmqVTS1ZJapec+L+lJSaskrZT0O0m7pOfuBvYGfi+pXNKPJJVIeqvK+MslfTV9faWkByTdI+l9YNTm5q8h1isl3ZO+7i4pJJ0p6U1JaySNkTRQ0suS1kq6Je/aUZKekXSLpPck/VnSUXnnu0p6RNJqSX+V9J0q8+bHPQa4DDglvfeX0n5nSnpd0geS/k/Sd/PGKJH0lqSLJL2b3u+ZeefbSbpR0t/S+P4kqV16bpCkZ9N7eklSyVYstZkVgBNsMzOryWSgAigGvggcDZydnhNwLdAV2B/oBlwJEBH/Dvydf+2KX1/P+Y4HHgB2AX5Xx/z1cQjQEzgF+BVwOfBV4EDgZElHVOn7BtAZ+E/gQUmd0nP3Am+l93oS8DNJX6kl7juAnwHT0ns/KO3zLnAssDNwJvBLSQfnjbEn0BHYCzgLuFXSrum5nwP9gS8DnYAfAZ9K2gt4DLg6bb8Y+B9Ju23Be2RmBeIE28zMHk53QddKeljSHsAw4AcRsS4i3gV+CZwKEBF/jYg/RMTHEfEP4BfAEbUPXy/PRcTDEfEpSSJa6/z19NOI+GdEzAbWAVMj4t2IKAXmkCTtG70L/Coi1kfENGAJMFxSN+DfgEvTsV4EbgdG1hR3RHxUUyAR8VhEvBGJPwKzgcPzuqwHfpLOPwMoB3pJ2g4YDVwQEaURsSEino2Ij4FvAzMiYkY69x+A+en7ZmaNzPViZmZ2Qv4DiZK+BLQByiRtbN4OeDM9vwdwE0mSuFN6bs1njOHNvNf7bG7+elqR9/qjGo53zDsujU2f+P8byY51V2B1RHxQ5dyAWuKukaSvkeyMf4HkPtoDi/O6rIqIirzjD9P4OgM7kOyuV7UP8E1Jx+W1tQGeqiseMys8J9hmZlbVm8DHQOcqid9GPwMC6BMRqyWdANySd77q11OtI0kqAUhrqauWMuRfU9f8WdtLkvKS7L2BR4C3gU6SdspLsvcGSvOurXqvmxxLagv8D8mu9/SIWC/pYZIym7qsBP4JfB54qcq5N4G7I+I71a4ys0bnEhEzM9tERJSRlDHcKGlnSdulDzZuLAPZiaSM4b20FviSKkOsAHrkHf8F2EHScEltgB8DbT/D/FnbHRgrqY2kb5LUlc+IiDeBZ4FrJe0gqS9JjfQ9mxlrBdA9Le8A2J7kXv8BVKS72UfXJ6i0XOa3wC/Shy1bSTo0TdrvAY6TNDRt3yF9YLJoy2/fzLLmBNvMzGoykiQ5fI2k/OMBoEt67irgYOA9kgftHqxy7bXAj9Oa7osj4j3geyT1y6UkO9pvsXmbmz9rc0keiFwJXAOcFBGr0nOnAd1JdrMfAv6zju/3vj/9e5WkhenO91jgPpL7+BbJ7nh9XUxSTvICsBq4DtguTf6PJ/nWkn+Q7Ghfgv+7btYk+B+aMTOzbZakUST/KM5hjR2LmbUc/qRrZmZmZpYhJ9hmZmZmZhlyiYiZmZmZWYa8g21mZmZmliF/D7Y1KbvssksUFxc3dhiWgXXr1tGhQ4fGDsMy4LVsWbyeLYfXsvEtWLBgZURU/V5/J9jWtOyxxx7Mnz+/scOwDORyOUpKSho7DMuA17Jl8Xq2HF7LxifpbzW1u0TEzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzKzFePPNNznyyCM54IADOPDAA7npppsAuOSSS9hvv/3o27cvJ554ImvXrq3x+pkzZ9KrVy+Ki4uZMGHCVsWgiNja+M3qRdIPgEkR8WFdfffuURzbnXxT4YOygruoTwU3Lm7d2GFYBryWLYvXs+Vormu5fMLwgo5fVlZGWVkZBx98MB988AH9+/fn4Ycf5q233uIrX/kKrVu35tJLLwXguuuu2+TaDRs28IUvfIE//OEPFBUVMXDgQKZOncoBBxxQ41ySFkTEgKrt3sG2hvADoH1jB2FmZmYtX5cuXTj44IMB2Gmnndh///0pLS3l6KOPpnXr5APJoEGDeOutt6pdO2/ePIqLi+nRowfbb789p556KtOnT9/iGJxgGwCSRkp6WdJLku6W1F3Sk2nbE5L2TvtNlnRS3nXl6d8lknKSHpD0Z0m/U2Is0BV4StJTjXN3ZmZmti1avnw5ixYt4pBDDtmk/be//S1f+9rXqvUvLS2lW7dulcdFRUWUlpZu8bzN7/cKljlJBwI/Br4cESsldQKmAFMiYoqk0cCvgRPqGOqLwIHA28AzwL9FxK8lXQgcGREra5n/HOAcgM6dd2N8n4osbssa2R7tkl9fWvPntWxZvJ4tR3Ndy1wu1yDzfPTRR1xwwQWcffbZLFy4sLL9nnvuYe3atey1117VYnn11VcpKyurbH/99dcpLS3d4pidYBvAV4D7NybAEbFa0qHA19PzdwPX12OceRHxFoCkF4HuwJ/quigiJgGTIKnBbo71ZFZdc60NtOq8li2L17PlaK5rufz0koLPsX79eo499ljGjBnDhRdeWNk+efJkXn31VZ544gnat69evdq2bVueffZZSkqSGJ977jm+9KUvVR7Xl0tEbEtVkP7cSNoO2D7v3Md5rzfgD3BmZmbWwCKCs846i/3333+T5HrmzJlcf/31PPLIIzUm1wADBw5k6dKlLFu2jE8++YR7772XESNGbHEMToAM4EngIUm/iIhVaYnIs8CpJLvXpwNz0r7Lgf7AfcAIoE09xv8A2AmosUQkX7s2rVhS4KeLrWHkcrkG2aWwwvNatixez5bDa1mzZ555hrvvvps+ffrQr18/AH72s58xduxYPv74Y4YMGQIkDzpOnDiRt99+m7PPPpsZM2bQunVrbrnlFoYOHcqGDRsYPXo0Bx544BbH4ATbiIhXJV0D/FHSBmARcD5wp6RLgH8AZ6bdbwOmS3oJmAmsq8cUk4CZkt6OiCOzvwMzMzOzxGGHHUZNX0M9bNiwGvt37dqVGTNmbNKvtr715QTbAIiIKSQPNub7Sg39VgCD8pouTdtzQC6v33l5r28Gbs4uWjMzM7OmyzXYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZlbNm2++yZFHHskBBxzAgQceyE033QTA6tWrGTJkCD179mTIkCGsWbOmxuunTJlCz5496dmzJ1OmTGnI0BudE2zLjKQSSV9u7DjMzMzss2vdujU33ngjr732Gs8//zy33norr732GhMmTOCoo45i6dKlHHXUUUyYMKHatatXr+aqq65i7ty5zJs3j6uuuqrWRLwlat3YAVjDk9Q6IioKcG0JUA48u7WxfbR+A93HPba1l1sTclGfCkZ5LVsEr2XL4vVsOSYf06Gg43fp0oUuXboAsNNOO7H//vtTWlrK9OnTyeVyAJxxxhmUlJRw3XXXbXLtrFmzGDJkCJ06dQJgyJAhzJw5k9NOO62gMTcV3sEuAEkdJD0m6SVJr0g6RdJ4SS+kx5MkKe2bk/RLSfMlvS5poKQHJS2VdHXemN+WNE/Si5L+S1KrWuZuJWlyOs9iST/Mm+dXkuYDl0v6m6Tt8uJ9U1KbWsbMv/YCScdJmitpkaT/lbSHpO7AGOCHaYyHS9pN0v+k9/2CpH/L9I02MzOzBrF8+XIWLVrEIYccwooVKyoT7z333JMVK1ZU619aWkq3bt0qj4uKiigtLW2weBubd7AL4xjg7YgYDiCpI/CHiPhJenw3cCzw+7T/JxExQNIFwHSgP7AaeEPSL4HdgVOAf4uI9ZJ+A5wO3FXD3P2AvSKidzrXLnnnto+IAWn7wcARwFNpLLMiYv1m7in/2l2BQRERks4GfhQRF0maCJRHxM/Tfv8N/DIi/iRpb2AWsH/VgSWdA5wD0Lnzbozvs1Wb69bE7NEu2Smz5s9r2bJ4PVuO8vLyyp3kQvroo4+44IILOPvss1m4cCEVFRWbzLthw4Zqcbzxxht88sknle3Lli2jbdu2DRJvU+AEuzAWAzdKug54NCLmSPqGpB8B7YFOwKv8K8F+JO+6VyOiDEDS/wHdgMNIku4X0o3vdsC7tcz9f0APSTcDjwGz885Nq/L6FJIE+1TgN3XcU/61RcA0SV2A7YFltVzzVeCANGaAnSXtGBHl+Z0iYhIwCWDvHsVx42L/WLYEF/WpwGvZMngtWxavZ8sx+ZgOlJSUFHSO9evXc+yxxzJmzBguvPBCAPbaay969epFly5dKCsro2vXrtXiKCsrI5fLVbZPnTqVwYMHFzzepsIlIgUQEX8BDiZJmK+WNJ4kgT0pIvoAtwE75F3ycfr3p3mvNx63BgRMiYh+6Z9eEXFlLXOvAQ4CciQlG7fnnV6X9/oR4BhJnUiS9yfruK38a28Gbknv5btV7iXfdiQ73Rvj3qtqcm1mZmZNU0Rw1llnsf/++1cm1wAjRoyo/FaQKVOmcPzxx1e7dujQocyePZs1a9awZs0aZs+ezdChQxss9sbmj7AFIKkrsDoi7pG0Fjg7PbVS0o7AScADWzDkE8B0Sb+MiHfTpHiniPhbDXN3Jik5+R9JS4B7ahowIsolvQDcRLLLvmEL4ukIbCykOiOv/QNg57zj2cD5wA1pbP0i4sXNDdyuTSuWTBi+BaFYU5XL5Vh+ekljh2EZ8Fq2LF7PlqPQ5RbPPPMMd999N3369KFfv34A/OxnP2PcuHGcfPLJ3HHHHeyzzz7cd999AMyfP5+JEydy++2306lTJ6644goGDhwIwPjx4ysfeNwWOMEujD7ADZI+BdYD5wInAK8A7wAvbMlgEfGapB8Ds9MHE9cD3weqJdjAXsCdGx9gBP5jM0NPA+4n+faPLXElcL+kNSQ73/um7b8HHpB0PEliPRa4VdLLJD9rT5PsqpuZmVkTd9hhhxERNZ574oknqrUNGDCA22//1y/OR48ezejRowsWX1PmBLsAImIWyQN9+eYDP66hb0ne6xxJaUdN56axaR10bXO/RFKeUus8eW0PkJSf1DVmSZXj6SQPY1bt9xegb5XmU+oa38zMzKwlcQ22mZmZmVmGvIPdjEmaC7St0vzvEbF4K8e7Faj6XdU3RcSdWzOemZmZ2bbICXYzFhGHZDze97Mcz8zMzGxb5BIRMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzM7MCGj16NLvvvju9e/eubHvxxRcZNGgQ/fr1Y8CAAcybN6/Ga6dMmULPnj3p2bMnU6ZMaaiQ7TNSRDR2DGaV9u5RHNudfFNjh2EZuKhPBTcubt3YYVgGvJYti9ezuuUThhd0/Keffpodd9yRkSNH8sorrwBw9NFH88Mf/pCvfe1rzJgxg+uvv55cLrfJdatXr2bAgAHMnz8fSfTv358FCxaw6667ApDL5SgpKSlo7LZ5khZExICq7d7BLhBJV0q6uJZzJZK+XMf1kyWdlEEcl33WMTKI4QeS2jd2HGZmZo1h8ODBdOrUaZM2Sbz//vsAvPfee3Tt2rXadbNmzWLIkCF06tSJXXfdlSFDhjBz5swGidk+G3+EbRwlQDnwbAPMdRnwswaYZ3N+ANwDfNjIcZiZmTUJv/rVrxg6dCgXX3wxn376Kc8+Wz0lKC0tpVu3bpXHRUVFlJaWNmSYtpWcYGdI0uXAGcC7wJvAAkljgTFABfAaMC493iDp28D5ETGnliEHS7oQ2BP4UUQ8IEnA9cDXgACujohpkroA04CdSdb1XGA40E7Si8CrEXF6LXGPBC5Ox3s5Iv5dUnfgt0Bn4B/AmRHxd0mTgUcj4oH02vKI2FFSCXAlsBLoDSwAvg2cD3QFnpK0MiKOrGH+c4BzADp33o3xfSpqf5Ot2dijXfKraGv+vJYti9ezuqqlGYXwzjvvsG7dusq5fv3rX3PWWWdxxBFH8NRTT/H1r3+dG2+8cZNr3njjDT755JPKa5YtW0bbtm0rj8vLyxskdttyrsHOiKT+wGTgEJIEdyEwEbgQ2DciPpa0S0SslXQlUB4RP9/MeJOBDsApwH7AIxFRLOkbJAn6MSTJ7wvpnN8CdoiIayS1AtpHxAcbE+DNzHMg8BDw5YhYKalTRKyW9HvggYiYImk0MCIiTqgjwZ4OHAi8DTwDXBIRf5K0HBgQESvreh9dg91yuM6z5fBatixez+oKXYMNsHz5co499tjKGuyOHTuydu1aJBERdOzYsbJkZKOpU6eSy+X4r//6LwC++93vUlJSwmmnnQa4BrspcA124R0OPBQRH0bE+8AjafvLwO/S3eot3TJ4OCI+jYjXgD3StsOAqRGxISJWAH8EBpIk2memyXufiPignnN8Bbh/Y/IbEavT9kOB/05f353OW5d5EfFWRHwKvAh0r2cMZmZm25SuXbvyxz/+EYAnn3ySnj17VuszdOhQZs+ezZo1a1izZg2zZ89m6NChDR2qbQV/hC284cBg4Djgckl9tuDaj/Nea3MdI+JpSYPT+SZL+kVE3LXF0datgvSDmaTtgO3zzuXHu4Gt+Plq16YVSxpgJ8EKL5fLsfz0ksYOwzLgtWxZvJ4N77TTTiOXy7Fy5UqKioq46qqruO2227jggguoqKhghx12YNKkSQDMnz+fiRMncvvtt9OpUyeuuOIKBg4cCMD48eOrPSxpTZMT7Ow8TZLYXkvyvh4H3AZ0i4inJP0JOBXYEfiApFZ6a8wBvitpCtCJJHm/RNI+wFsRcZuktsDBwF3AekltImJ9LeM9CTyUJuSrNpaIkDyAeSrJ7vXp6bwAy4H+wH3ACKBNPWL+ANiJpD7bzMxsmzJ16tQa2xcsWFCtbcCAAdx+++2Vx6NHj2b06NEFi80KwyUiGYmIhSQPGb4EPE5SshHAPZIWA4uAX0fEWuD3wImSXpR0+BZO9RBJ2clLJMnxjyLiHZJvJnlJ0iKSuu2NhcyTgJcl/a6WuF8FrgH+KOkl4BfpqfNJSk5eBv4duCBtvw04Iu17KLCuHjFPAmZKempLbtTMzMysOfJDjtak9OrVK5YsWdLYYVgG/PBNy+G1bFm8ni2H17Lx+SFHMzMzM7MG4BrsRpZ+d/Y3qzTfHxHXZDzP54Anajh1VESsynIuMzMzs22ZE+xGlibSmSbTtcyzCuhX6HnMzMzMtnUuETEzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzOzbczo0aPZfffd6d27d7VzN954I5JYuXJljddOmTKFnj170rNnT6ZMmVLoUM2apdaNHYBZvo/Wb6D7uMcaOwzLwEV9KhjltWwRvJYNb/mE4QUdf9SoUZx33nmMHDlyk/Y333yT2bNns/fee9d43erVq7nqqquYP38+kujfvz8jRoxg1113LWi8Zs3NNrmDLelKSRfXcq5E0pfruH6ypJMyiOOyzzpGUyLpB5LaN3YcZma2eYMHD6ZTp07V2n/4wx9y/fXXI6nG62bNmsWQIUPo1KkTu+66K0OGDGHmzJmFDtes2dkmE+w6lACbTbAzVO8EW4k616tqvy24rs7fZtRjrB8ATrDNzJqh6dOns9dee3HQQQfV2qe0tJRu3bpVHhcVFVFaWtoQ4Zk1K9tMiYiky4EzgHeBN4EFksYCY4AK4DVgXHq8QdK3gfMjYk4tQw6WdCGwJ/CjiHhAyUf+64GvAQFcHRHTJHUBpgE7k7zn5wLDgXaSXgRejYjTa4i5OzALmAv0B4ZJOhk4GWgLPBQR/1lDv+9JmlTluvNqiKsE+CmwBtgP+EI9YxgHDATaAQ+kMYwFugJPSVoZEUdKOhq4Ko31DeDMiCivYY5zgHMAOnfejfF9Kmp5y6052aNdUlpgzZ/XsuHlcrmCjV1eXk4ul+Odd95h3bp15HI5/vnPfzJu3DhuuOGGyuNnnnmGjh07bnLtG2+8wSeffFIZ37Jly2jbtm1B47XabVxLa3q2iQRbUn/gVKAfyT0vBBaQJNT7RsTHknaJiLWSJgLlEfHzOobtAhxGkpg+AjwAfD2d4yCgM/CCpKeBbwGzIuIaSa2A9hExR9J5EdGvjnl6AmdExPNpwtoT+BIg4BFJg4G/V+nXvcrxN2qJC+BgoHdELKtPDJB8WImI1em9PCGpb0T8Ov3AcWRErJTUGfgx8NWIWCfpUuBC4CdVB4+IScAkgL17FMeNi7eJH8sW76I+FXgtWwavZcNbfnpJwcbO5XKUlJSwfPlyOnToQElJCYsXL2bVqlWcd955AKxcuZLzzz+fefPmseeee1ZeW1ZWVnk9wNSpUxk8eHDlsTWs/LWwpmVb+X/Mw0l2ez8EkPRI2v4y8DtJDwMPb+GYD0fEp8BrkvZI2w4DpkbEBmCFpD+S7PS+APxWUpv0uhe3YJ6/bUxsgaPTP4vS4x1Jkt+/V+lX9bra4nofmFdHcl11LICT013n1iQfNA4geS/zDUrbn0lr+bYHnqvPDZuZWcPq06cP7777buVx9+7dmT9/Pp07d96k39ChQ7nssstYs2YNALNnz+baa69t0FjNmoNtJcGuzXBgMHAccLmkPltw7cd5r2t+GiQVEU+nO83DgcmSfhERd9VznnVV5rk2Iv4rv0O6Y53fr+p19R2/zj6S9gUuBgZGxBpJk4EdarhGwB8i4rR6xgFAuzatWFLgp+etYeRyuYLuwlnD8Vq2PKeddhq5XI6VK1dSVFTEVVddxVlnnVVj3/nz5zNx4kRuv/12OnXqxBVXXMHAgQMBGD9+fI0PS5pt67aVhxyfBk6Q1E7STiQJ9XZAt4h4CrgU6EiyI/wBsNNWzjMHOEVSK0m7kSTv8yTtA6yIiNuA20nKMgDWp7va9TULGC1pRwBJe0nafWvj2oJ58+1MknC/l+7cfy3vXP579zzwb5KK01g7SKpW421mZg1v6tSplJWVsX79et56661qyfXy5csrd68HDBjA7bffXnlu9OjR/PWvf+Wvf/0rZ555ZoPGbdZcbBM72BGxUNI04CWShxxfIHnY7x5JHUl2W3+d1mD/HnhA0vFs/iHHmjwEHJrOEyQPP74j6QzgEknrgXJg4xePTgJelrSwpocca7iP2ZL2B55Lyy7KgW8DG7Yyrv224N42xvCSpEXAn0keFn0m7/QkYKakt9OHHEcBUyW1Tc//GPjLls5pZmZm1pwoIho7BrNKvXr1iiVLljR2GJYBP3zTcngtWxavZ8vhtWx8khZExICq7dtKiYiZmZmZWYPYJkpEtlb63dnfrNJ8f0Rck/E8nwOeqOHUURGxKsu5mnIMZmZmZi2BE+zNSBPpTJPpWuZZRfI91Y2mKcRgZmZm1hK4RMTMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMbCuNHj2a3Xffnd69e1e2XXLJJey333707duXE088kbVr19Z47cyZM+nVqxfFxcVMmDChgSI2s4agiGjsGMwq7d2jOLY7+abGDsMycFGfCm5c3Lqxw7AMNNe1XD5heMHnePrpp9lxxx0ZOXIkr7zyCgCzZ8/mK1/5Cq1bt+bSSy8F4Lrrrtvkug0bNvCFL3yBP/zhDxQVFTFw4ECmTp3KAQccUPCYc7kcJSUlBZ/HCs9r2fgkLYiIAVXbm/QOtqQrJV1cy7kSSV+u4/rJkk7KII7LPusYLZ2k7pK+1dhxmJk1pMGDB9OpU6dN2o4++mhat04+kAwaNIi33nqr2nXz5s2juLiYHj16sP3223Pqqacyffr0BonZzAqvSSfYdSgBNptgZ+gzJ9iSWmURSD3mab254/petxV9ugNOsM3M8vz2t7/la1/7WrX20tJSunXrVnlcVFREaWlpQ4ZmZgXU5H7nJ+ly4AzgXeBNYIGkscAYoAJ4DRiXHm+Q9G3g/IiYU8uQgyVdCOwJ/CgiHpAk4Hrga0AAV0fENEldgGnAziTvzbnAcKCdpBeBVyPi9Bpi7gDcBxQBrYCfpuMtT8cbAtwn6esR8aX0mu7A7yOiTy3vwwRgRHrPsyPiYkm7AROBvdNuP4iIZyRdCXwe6AH8XdKSKsf/AfwW6Az8AzgzIv4uaTLwT+CLwDPAhTXEUXXs/wDuBjqkXc6LiGeBCcD+6fs0Bfh12lYCtAVujYj/quVezwHOAejceTfG96moqZs1M3u0S0oLrPlrrmuZy+UaZJ533nmHdevWVZvvnnvuYe3atey1117Vzr366quUlZVVtr/++uuUlpY2SMzl5eUN9t5YYXktm64mlWBL6g+cCvQjiW0hsIAkod43Ij6WtEtErJU0ESiPiJ/XMWwX4DBgP+AR4AHg6+kcB5EknS9IeppkB3ZWRFyT7ji3j4g5ks6LiH6bmeMY4O2IGJ7eR8e8c6si4uC0/VRJ+0bEMuAUkuS7pvfhc8CJwH4REZJ2SU/dBPwyIv4kaW9gFrB/eu4A4LCI+ChNivOPfw9MiYgpkkaTJL8npNcVAV+OiA2bub/8sdoDQyLin5J6AlOBASRrdHFEHJvewznAexExUFJb4BlJs9N730RETAImQVKD3RxrPa265lq3a9U117VcfnpJw8yzfDkdOnTYpBZ28uTJvPrqqzzxxBO0b9++2jVt27bl2Wefrbzmueee40tf+lKD1NO6brfl8Fo2XU2tRORw4KGI+DAi3idJiAFeBn6X7lZv6TbKwxHxaUS8BuyRth0GTI2IDRGxAvgjMBB4ATgzTVD7RMQH9ZxjMTBE0nWSDo+I9/LO5SfR95Ek1rCZBBt4j2Rn+Q5JXwc+TNu/CtyS7hI/Auwsacf03CMR8VHeGPnHhwL/nb6+m+T+N7q/juS66lhtgNskLQbuJ0m+a3I0MDKNdS7wOaBnHfOYmTV7M2fO5Prrr+eRRx6pMbkGGDhwIEuXLmXZsmV88skn3HvvvYwYMaKBIzWzQmkuWxLDgcHAccDlkmosq6jFx3mvtbmOEfG0pMHpfJMl/SIi7qprgoj4i6SDgWHA1ZKeiIifpKfX5XWdBtwv6cHkslhay3gVkr4EHAWcBJwHfIXkA9GgiPhnfv+k4mWTeWo6rk19+uX3+SGwgmT3fzuSDwI1EUnpzqx6xgFAuzatWNIAT/5b4eVyuQbbQbTC8lrW7rTTTiOXy7Fy5UqKioq46qqruPbaa/n4448ZMmQIkDzoOHHiRN5++23OPvtsZsyYQevWrbnlllsYOnQoGzZsYPTo0Rx44IGNfDdmlpWmlmA/TZLYXksS23HAbUC3iHhK0p9ISkh2BD4gqZXeGnOA70qaAnQiSd4vkbQP8FZE3JaWNRwM3AWsl9QmItbXNJikrsDqiLhH0lrg7Jr6RcQbkjYAV1D77jXprnT7iJgh6Rng/9JTs4HzgRvSfv0i4sV63O+zJO/b3cDp6f1vrY4k79Gnks4gqTmHZD12yus3CzhX0pMRsV7SF4DSiKhv4m9m1uRNnTq1WttZZ51VY9+uXbsyY8aMyuNhw4YxbNiwgsVmZo2nSSXYEbFQ0jTgJZKHHF8geQjxnrSuWcCv0xrs3wMPSDqezT/kWJOHSMomXkrH/1FEvJMmjJdIWg+UAyPT/pOAlyUtrOkhR6APcIOkT4H1JA9H1mYaSYK872b67ARMl7QDyT1vfPhwLHCrpJdJ1u5pkoc963I+cKekS0gfcqzHNbX5DfA/kkYCM/nX7vbLJA+dvgRMJqkX7w4sTB8q/Qf/qvs2MzMza7H8D81Yk9KrV69YsmRJY4dhGfDDNy2H17Jl8Xq2HF7Lxtcs/6EZMzMzM7PmpkmViGyt9Luzv1ml+f6IuCbjeT4HPFHDqaMiYtVWjvkQ1ctFLt3ShwM/K0lnAhdUaX4mIr7fkHGYmZmZNXctIsFOE+lMk+la5llF8v3ZWY55Ypbjba2IuBO4s7HjMDMzM2vuXCJiZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZm1qKNHj2a3Xffnd69e1e23X///Rx44IFst912zJ8/v9ZrZ86cSa9evSguLmbChAkNEa6ZtQCtGzsAs3wfrd9A93GPNXYYloGL+lQwymvZIhRyLZdPGF6QcfONGjWK8847j5EjR1a29e7dmwcffJDvfve7tV63YcMGvv/97/OHP/yBoqIiBg4cyIgRIzjggAMKHrOZNW/ewTYkXSnp4i3o30/SsELGZGaWlcGDB9OpU6dN2vbff3969eq12evmzZtHcXExPXr0YPvtt+fUU09l+vTphQzVzFoIJ9gtlKRWBRy+H+AE28xatNLSUrp161Z5XFRURGlpaSNGZGbNhUtEmilJHYD7gCKgFfBT4DpgGjAEuF7SWuBn6fmVEXHUZoY8SNJzQGfg+oi4TdJdwIMR8XA65+/SOX8CtJN0GHAt8ChwM9AbaANcGRHTJR0I3AlsT/Jh7hsRsbSGezkHOAegc+fdGN+nYqvfF2s69miXlBZY81fItczlcgUZt6p33nmHdevWVZtv7dq1LFiwgPLy8mrXvPrqq5SVlVVe8/rrr1NaWtpgMRdKeXl5s78HS3gtmy4n2M3XMcDbETEcQFJHkgR7VUQcLGk3YCEwOCKWSeq0mbEA+gKDgA7AIkmPAXcAPwQeTsf/MnAGsCswICLOS+f+GfBkRIyWtAswT9L/AmOAmyLid5K2J0n0q4mIScAkgL17FMeNi/1j2RJc1KcCr2XLUMi1XH56SUHGrTbP8uV06NCBkpJN59tll13o378/AwYMqHZN27ZtefbZZyuvee655/jSl75UbYzmJpfLNft7sITXsulyiUjztRgYIuk6SYdHxHtp+7T070HA0xGxDCAiVtcx3vSI+CgiVgJPAV+KiD8CPdNk/TTgfyKipm2so4Fxkl4EcsAOwN7Ac8Blki4F9omIj7b2Zs3MGtrAgQNZunQpy5Yt45NPPuHee+9lxIgRjR2WmTUD3l5qpiLiL5IOJqmFvlrSE+mpdVs7ZC3HdwHfBk4FzqzlWpGUfyyp0v66pLnAcGCGpO9GxJObC6Jdm1YsaYBvFbDCy+VyDbY7aYXV3NfytNNOI5fLsXLlSoqKirjqqqvo1KkT559/Pv/4xz8YPnw4/fr1Y9asWbz99tucffbZzJgxg9atW3PLLbcwdOhQNmzYwOjRoznwwAMb+3bMrBlwgt1MSeoKrI6Ie9Ja67OrdHke+I2kfTeWiNSxi328pGtJSkRKgHFp+2RgHvBORLyWtn0A7JR37SzgfEnnR0RI+mJELJLUA/i/iPi1pL1JylA2m2CbmWVt6tSpNbafeOKJ1dq6du3KjBkzKo+HDRvGsGF+ptvMtoxLRJqvPiS1zi8C/wlcnX8yIv5B8uDgg5Je4l+lI7V5maQ05HngpxHxdjrOCuB1kocVN3oKOEDSi5JOIXnAsg3wsqRX02OAk4FX0hh7k+yGm5mZmbVo3sFupiJiFsnOcb7uVfo8Djxej7GurO2cpPZAT2BqXv/VwMAqXav9aw0RMQHwP31mZmZm2xTvYFutJH2VZPf65ryHKM3MzMxsM7yDvQ2RdCZwQZXmZyLi+zX1j4j/BfYpeGBmZmZmLYgT7G1IRNzJprXUZmZmZpYxl4iYmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZWaMaPXo0u+++O717965sW716NUOGDKFnz54MGTKENWvW1HjtlClT6NmzJz179mTKlCkNFbKZ2WY5wTYzs0Y1atQoZs6cuUnbhAkTOOqoo1i6dClHHXUUEyZMqHbd6tWrueqqq5g7dy7z5s3jqquuqjURNzNrSK0bOwDbMpLGAucCOwMPRcR5GY69AVhM8nOxDPj3iFgraTvgV8BXgAD+CZwM3Au0BToB7YDSdKgTgBzwQdp/DTAyIv5WVwwfrd9A93GPZXVL1ogu6lPBKK9lizD5mA4FHX/w4MEsX758k7bp06eTy+UAOOOMMygpKeG6667bpM+sWbMYMmQInTp1AmDIkCHMnDmT0047raDxmpnVxTvYzc/3gCHA5QUY+6OI6BcRvYHVwPfT9lOArkDfiOgDnAisjYhDIqIfMB6Yll7bLyKWp9cdGRF9SZLtHxcgXjNroVasWEGXLl0A2HPPPVmxYkW1PqWlpXTr1q3yuKioiNLS0mr9zMwamnewmxFJE4EewOPAb/PajyNJYLcHVgGnR8QKSbsB/02SHD9Hkpj3j4iV9ZjuOaBv+roLUBYRnwJExFtbGPpzwNjN3Nc5wDkAnTvvxvg+FVs4vDVFe7RLdrGt+SsvL6/cTS6Ud955h3Xr1lXOU1FRscmcGzZsqBbDG2+8wSeffFLZvmzZMtq2bVvwWJu7hlhPaxhey6bLCXYzEhFjJB0DHAkcm3fqT8CgiAhJZwM/Ai4C/hN4MiKuTa87qz7zSGoFHAXckTbdB/xJ0uHAE8A9EbFoC0I/Bnh4M/c1CZgEsHeP4rhxsX8sW4KL+lTgtWwZJh/TgZKSkoLOsXz5cjp0+Nc8e+21F7169aJLly6UlZXRtWvXajGUlZWRy+Uq26dOncrgwYMLHmtzl/+eWfPmtWy6XCLSMhQBsyQtBi4BDkzbDyOpkyYiZpLUQm9OO0kvAu8AewB/SK99C+gF/AfwKfCEpKPqEddTkkqBrwFTt+SGzGzbNmLEiMpvBZkyZQrHH398tT5Dhw5l9uzZrFmzhjVr1jB79myGDh3a0KGamVXj7aWW4WbgFxHxiKQS4MqtHOejiOgnqT0wi6QG+9cAEfExSWnK45JWkDzI+EQd4x0JrAV+B1wFXFhXAO3atGLJhOFbGb41JblcjuWnlzR2GJaBQv8K+rTTTiOXy7Fy5UqKioq46qqrGDduHCeffDJ33HEH++yzD/fddx8A8+fPZ+LEidx+++106tSJK664goEDBwIwfvz4ygcezcwakxPslqEj//oGjzPy2p8h+baP6yQdDexan8Ei4sP020oelvQbklrsdyLi7fQbRfoCL9dzrApJPwAWS7o6IlbX647MbJsxdWrNv+B64onqn+EHDBjA7bffXnk8evRoRo8eXbDYzMy2hktEWoYrgfslLQDyH2C8Cjha0ivAN0lKPz6oz4BpjfXLwGnA7sDv03FeBiqAW+obXESUkZSIfL+uvmZmZmbNnXewm5mI6J6+nJz+ISKmA9Nr6P4eMDTdRT4UGJiWetQ29o5Vjo/LO5xJLSKiMpYa4tx4fH5t15uZmZm1JE6wW7a9gfvSso5PgO80cjxmZmZmLZ4T7BYsIpYCX8xvk/Q5an448aiIWNUggZmZmZm1YE6wtzFpEt2vseMwMzMza6n8kKOZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5lZoxo9ejS77747vXv3rmxbvXo1Q4YMoWfPngwZMoQ1a9bUeO2UKVPo2bMnPXv2ZMqUKQ0VspnZZikiGjsGs0p79yiO7U6+qbHDsAxc1KeCGxe3buwwLAOTj+lASUlJwcZ/+umn2XHHHRk5ciSvvPIKAD/60Y/o1KkT48aNY8KECaxZs4brrrtuk+tWr17NgAEDmD9/PpLo378/CxYsYNdddy1YrC1BLpcr6Hpaw/FaNj5JCyJiQNV272C3AJKulHRxLedKJH25jusnSzppC+arc0wzs/oaPHgwnTp12qRt+vTpnHHGGQCcccYZPPzww9WumzVrFkOGDKFTp07suuuuDBkyhJkzZzZEyGZmm+UEu+UrAbJOhgsxpplZpRUrVtClSxcA9txzT1asWFGtT2lpKd26das8LioqorS0tMFiNDOrjX9/20xJuhw4A3gXeBNYIGksMAaoAF4DxqXHGyR9Gzg/IubUMuRXJY0DdgYujIhHJT0NjI2IF9M5/wR8v+qYwJ+BicDe6Vg/iIhnJB0BbKz3CGBwRHxQw72cA5wD0LnzbozvU7G1b4s1IXu0S8pErPkrLy8nl8sVdI533nmHdevWVc5TUVGxyZwbNmyoFsMbb7zBJ598Utm+bNky2rZtW/BYm7uGWE9rGF7LpssJdjMkqT9wKtCPZA0XAgtIEup9I+JjSbtExFpJE4HyiPh5HcN2B74EfB54SlIxcAcwCviBpC8AO0TES1XHlPTfwC8j4k+S9gZmAfsDFwPfT5PtHYF/1jRxREwCJkFSg+263ZbBNdgtR6FrsAGWL19Ohw7/mmevvfaiV69edOnShbKyMrp27VothrKysk1qUKdOncrgwYNdk1oH1+22HF7LpsslIs3T4cBDEfFhRLwPPJK2vwz8Lt1Z3tKtw/si4tOIWAr8H7AfcD9wrKQ2wGhgci3XfhW4RdKLaSw7pwn1M8Av0p31XSLC25lmVi8jRoyo/FaQKVOmcPzxx1frM3ToUGbPns2aNWtYs2YNs2fPZujQoQ0dqplZNd5ealmGA4OB44DLJfXZgmurfp1MRMSHkv4AHA+cDPSv5drtgEERUXWHeoKkx4BhwDOShkbEnzcXRLs2rVgyYfgWhG1NVS6XY/npJY0dhmWg0L+CPu2008jlcqxcuZKioiKuuuoqxo0bx8knn8wdd9zBPvvsw3333QfA/PnzmThxIrfffjudOnXiiiuuYODAgQCMHz++2sOSZmaNwQl28/Q0MFnStSRreBxwG9AtIp5Ka6VPBXYEPiCpq67LNyVNAfYFegBL0vbbgd8DcyJi4xfRVh1zNkkt9g0AkvpFxIuSPh8Ri4HFkgaS7IpvNsE2s23P1KlTa2x/4oknqrUNGDCA22+/vfJ49OjRjB49umCxmZltDZeINEMRsRCYBrwEPA68QLIDfY+kxcAi4NcRsZYkOT5R0ouSDt/MsH8H5qXjjdm4Gx0RC4D3gTvz+lYdcywwQNLLkl4jeQgSktrtVyS9DKxPxzYzMzNr0byD3UxFxDXANVWab6ih31+AvnWMNaq2c5K6knwQm13HmKfUMO75m5vXzMzMrCXyDrbVStJIYC5weUR82tjxmJmZmTUH3sHehqTfnf3NKs33p7vh1UTEXcBdBQ/MzMzMrAVxgr0NqaWsxMzMzMwy5BIRMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8tQvRJsSZ+X1DZ9XSJprKRdChqZmZmZmVkzVN8d7P8BNkgqBiYB3YD/LlhU1ixJOkHSAY0dh5k1L6NHj2b33Xend+/elW2rV69myJAh9OzZkyFDhrBmzZoar50yZQo9e/akZ8+eTJkypaFCNjPbrNb17PdpRFRIOhG4OSJulrSokIG1VJJaRcSGQl5btd/WXrcVfU4AHgVeq2uu2ny0fgPdxz22tZdbE3JRnwpGeS1bhMnHdCjo+KNGjeK8885j5MiRlW0TJkzgqKOOYty4cUyYMIEJEyZw3XXXbXLd6tWrueqqq5g/fz6S6N+/PyNGjGDXXXctaLxmZnWp7w72ekmnAWeQJFAAbQoTUvMmqYOkxyS9JOkVSadIWi7pOkkLgUskzcvr313S4s2Ml3/tNyUdLek5SQsl3S9px1r6VT0+TdLiNKbr8sYvl3SjpJeAQ+sZw3ckvZDe4/9Iai/py8AI4AZJL6ZlRZ+XNFPSAklzJO2XxXtsZi3L4MGD6dSp0yZt06dP54wzzgDgjDPO4OGHH6523axZsxgyZAidOnVi1113ZciQIcycObMhQjYz26z6JthnkiRf10TEMkn7AncXLqxm7Rjg7Yg4KCJ6Axv/335VRBwcEROA7dP3EOAUYFodY66KiIOB/wV+DHw1PZ4PXFi1X0TcW+W6p4HrgK8A/YCBkk5I+3QA5qbx/qmuGNKxH4yIgRFxEPA6cFZEPAs8AlwSEf0i4g2ScqLzI6I/cDHwmzru08wMgBUrVtClSxcA9txzT1asWFGtT2lpKd26das8LioqorS0tMFiNDOrTb1KRCLiNUmXAnunx8tIEjarbjFwY7pL/GhEzJEEmybR95Ek1hPSv0+pY8yN1w4CDgCeScfcHniuhn5VjwcCuYj4B4Ck3wGDgYeBDSQ19nXJH7u3pKuBXYAdgVlVO6c7618G7k9jBWhb08CSzgHOAejceTfG96moRzjW1O3RLikTseavvLycXC5X0Dneeecd1q1bVzlPRUXFJnNu2LChWgxvvPEGn3zySWX7smXLaNu2bcFjbe4aYj2tYXgtm656JdiSjgN+TpLQ7SupH/CTiBhRwNiapYj4i6SDgWHA1ZKeSE+ty+s2jSTxfDC5JJbWMezGawX8ISJOq6Nfbcc1+Wc9a8Lzx5oMnBARL0kaBZTU0H87YG1E9Ktr4IiYRLLbzd49iuPGxfV9NMCasov6VOC1bBkmH9OBkpKSgs6xfPlyOnT41zx77bUXvXr1okuXLpSVldG1a9dqMZSVlZHL5Srbp06dyuDBgwsea3OX/55Z8+a1bLrqWyJyJfAlYC1ARLwI9ChIRM2cpK7AhxFxD3ADcHDVPmn5xAbgCuouD8n3PPBv6be5bKz3/kI9rpsHHCGps6RWwGnAH7dg3qp2AsoktQFOz2v/ID1HRLwPLJP0zTRWSTroM8xpZtuQESNGVH4ryJQpUzj++OOr9Rk6dCizZ89mzZo1rFmzhtmzZzN06NCGDtXMrJr6bi+tj4j38n7VD/BpAeJpCfqQPOj3KbAeOBd4oIZ+00gS8H1rOFejiPhHumM8deP3kpPUZP+ljuvKJI0DniLZBX8sIqbXd94aXAHMBf6R/r1T2n4vcJukscBJJMn3/5P0Y5KHYu8FXtrcwO3atGLJhOGfITRrKnK5HMtPL2nsMCwDhf4V9GmnnUYul2PlypUUFRVx1VVXMW7cOE4++WTuuOMO9tlnH+677z4A5s+fz8SJE7n99tvp1KkTV1xxBQMHDgRg/Pjx1R6WNDNrDIqIujtJdwBPAOOAbwBjgTYRMaaw4dm2plevXrFkyZLGDsMy4F9dthxey5bF69lyeC0bn6QFETGgant9S0TOBw4EPib5B2beA36QWXRmZmZmZi1EnSUiac3uYxFxJHB54UPaNkl6iOrlIpdGRLVv6GjJMZiZmZk1d3Um2BGxQdKnkjpGxHsNEdS2KCJOdAxmZmZmzV99H3IsBxZL+gN5X9cWEWMLEpWZmZmZWTNV3wT7wfSPmZmZmZltRn3/JccphQ7EzMzMzKwlqO+/5LgMqPZ9fhHhf2zGzMzMzCxPfUtE8r/fbwfgm4C/zd/MzMzMrIp6fQ92RKzK+1MaEb8C/M/tmZmZmZlVUd8SkYPzDrcj2dGu7+63mZmZmdk2o75J8o15ryuAZcDJ2YdjZmZmZta81TfBPisi/i+/QVLVf/HPzMzMzGybV68abOCBeraZmZmZmW3TNruDLWk/4ECgo6Sv553ameTbRMzMzMzMLE9dJSK9gGOBXYDj8to/AL5ToJjMzMzMzJqtzSbYETEdmC7p0Ih4roFiMjMzMzNrtur7kOMiSd8nKRepLA2JiNEFicrMzMzMrJmq70OOdwN7AkOBPwJFJGUiZmZmZmaWp74JdnFEXAGsi4gpJP+K4yGFC8vMzMzMrHmqb4K9Pv17raTeQEdg98KEZGZmTdFNN91E7969OfDAA/nVr35V7XxEMHbsWIqLi+nbty8LFy5s+CDNzJqA+tZgT5K0K3AF8AiwIzC+YFHZNuuj9RvoPu6xxg7DMnBRnwpGeS0bzPIJwws6/iuvvMJtt93GvHnz2H777TnmmGM49thjKS4uruzz+OOPs3TpUpYuXcrcuXM599xzmTt3bkHjMjNriuq1gx0Rt0fEmoj4Y0T0iIjdI2JioYOzukn6iaSvNnYcdZE0RtLIxo7DzLbO66+/ziGHHEL79u1p3bo1RxxxBA8++OAmfaZPn87IkSORxKBBg1i7di1lZWWNFLGZWeOpV4ItaQ9Jd0h6PD0+QNJZhQ3NNpJU628aImJ8RPxvQ8azNSJiYkTc1dhxmNnW6d27N3PmzGHVqlV8+OGHzJgxgzfffHOTPqWlpXTr1q3yuKioiNLS0oYO1cys0dW3RGQycCdweXr8F2AacEcBYmqxJHUA7iP5FpZWwE+BvwK/ICm7WQmMiogySTngReAw4PeSRgP7RsSn6Th/BnoAtwGPRsQDkgYCNwEdgI+Bo4APgQlACdAWuDUi/mszMV4CnJz2fSgi/jNtHwlcDATwckT8u6TuwG+BzsA/gDMj4u+1jHslUB4RP6/h3DnAOQCdO+/G+D4Vm30frXnYo11SJmINI5fLFWzs8vJyAI4//ngOPfRQ2rVrR/fu3SkrK9tk3lWrVrFo0SIqKpJ1X7NmDQsWLKi83pqG8vLygv68WMPxWjZd9U2wO0fEfZL+AyAiKiRtKGBcLdUxwNsRMRxAUkfgceD4iPiHpFOAa4CN3y++fUQMSPseDBwBPEXyr2vOioj1kkjPb0/yoeeUiHhB0s7AR8BZwHsRMVBSW+AZSbMjYlnV4CQdDfQEvgQIeETSYGAV8GPgyxGxUlKn9JKbgSkRMSX9APBr4IQtfVMiYhIwCWDvHsVx4+L6/lhaU3ZRnwq8lg1n+eklBRs7l8tRUlJCSUkJN9xwAwCXXXYZRUVFlJT8a96+ffvSuXPnyrZ169YxYsQIunTpUrDYbMttXE9r/ryWTVd9v0VknaTPkexeImkQ8F7Bomq5FgNDJF0n6XCgG9Ab+IOkF0mS2KK8/tOqvD4lfX1qlXOQ/LP2ZRHxAkBEvB8RFcDRwMh0/LnA50iS6Jocnf5ZBCwE9kv7fgW4PyJWpmOvTvsfCvx3+vpukt12M2uh3n33XQD+/ve/8+CDD/Ktb31rk/MjRozgrrvuIiJ4/vnn6dixo5NrM9sm1Xd76UKSbw/5vKRngN2AkwoWVQsVEX9Jd6KHAVcDTwKvRsShtVyyLu/1I8DP0t3j/um19SHg/IiYVc++11YtIZF0fj3n+szatWnFkgJ/G4I1jFwuV9BdVWt43/jGN1i1ahVt2rTh1ltvZZdddmHixOR59zFjxjBs2DBmzJhBcXEx7du3584772zkiM3MGsdmE2xJe0fE3yNioaQjSHZJBSyJiPWbu9aqk9QVWB0R90haC3wP2E3SoRHxnKQ2wBci4tWq10ZEuaQXSGqsH42IqiU6S4AukgamJSI7kZSIzALOlfRkWlLyBaA0ItZR3Szgp5J+l863F8l3oD8JPCTpFxGxSlKndBf7WZLd9LuB04E5n/lNMrMma86c6v8THzNmTOVrSdx6660NGZKZWZNU1w72w8DB6etpEfGNwobT4vUBbpD0KUniei5QAfw6rcduDfwKqJZgp6YB95M8sLiJiPgkreG+WVI7kuT6q8DtQHdgoZKC7X9QS510RMyWtD/wXFrbXQ58OyJelXQN8Me09n4RMAo4H7gzfTDyH8CZW/BemJmZmbVIdSXYynvdo5CBbAvSMo2aSjUG19C3pIa2B9h0TYiIUXmvXwAG1TD+Zemf+sR4E8kuedX2KcCUKm1/I6nPrs+4V9ann5mZmVlzV9dDjlHLazMzMzMzq0FdO9gHSXqfZNe0Xfqa9DgiYueCRmcFIakPSd10vo8j4pAMxr4c+GaV5vsj4prPOraZmZlZc7DZBDsiWjVUINZwImIx0K9AY19D8l3eZmZmZtuk+n4PtpmZmZmZ1YMTbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzOrl5tuuonevXtz4IEH8qtf/ara+Yhg7NixFBcX07dvXxYuXNjwQZqZNQGtGzsAazkk/QCYFBEfbu0YH63fQPdxj2UXlDWai/pUMMpr2WCWTxhe0PFfeeUVbrvtNubNm8f222/PMcccw7HHHktxcXFln8cff5ylS5eydOlS5s6dy7nnnsvcuXMLGpeZWVPkHexmRInNrlnVPvW5Ju1X54eteoz1A6B9XeOYWfPz+uuvc8ghh9C+fXtat27NEUccwYMPPrhJn+nTpzNy5EgkMWjQINauXUtZWVkjRWxm1nicYDdxkrpLWiLpLuAV4ApJL0h6WdJVtfQ5vMpxN0k3SHpF0mJJp6TXlUiaI+kR4LV6zt9N0v+TNF/Sq3kxjAW6Ak9JeiptO1rSc5IWSrpf0o4FfbPMrGB69+7NnDlzWLVqFR9++CEzZszgzTff3KRPaWkp3bp1qzwuKiqitLS0oUM1M2t0LhFpHnoCZwA7AycBXwIEPCJpMPD3jX0i4nlJ3ascfwPoBxwEdAZekPR0OvbBQO+IWFbX/BHxPICkyyNitaRWwBOS+kbEryVdCBwZESsldQZ+DHw1ItZJuhS4EPhJ1cElnQOcA9C5826M71Oxte+TNSF7tEvKRKxh5HK5go1dXl4OwPHHH8+hhx5Ku3bt6N69O2VlZZvMu2rVKhYtWkRFRbLua9asYcGCBZXXW9NQXl5e0J8Xazhey6bLCXbz8Lc0Uf45cDSwKG3fkST5/fvGPlWvSV8fBkyNiA3ACkl/BAYC7wPz6kiuq44FcHKaFLcGugAHAC9XuWZQ2v6MJIDtgedqGjwiJgGTAPbuURw3LvaPZUtwUZ8KvJYNZ/npJQUbO5fLUVJSQklJCTfccAMAl112GUVFRZSU/Gvevn370rlz58q2devWMWLECLp06VKw2GzLbVxPa/68lk2X/+vXPKxL/xZwbUT8V/7JdMd6XS3X1HfsevWRtC9wMTAwItZImgzsUMM1Av4QEafVMw4za+Leffdddt99d/7+97/z4IMP8vzzz29yfsSIEdxyyy2ceuqpzJ07l44dOzq5NrNtkhPs5mUW8FNJv4uIckl7Aevrcd0c4LuSpgCdgMHAJcB+WxHDziQJ93uS9gC+BuTScx8AOwErgeeBWyUVR8RfJXUA9oqIv2xu8HZtWrGkwN+GYA0jl8sVdFfVGt43vvENVq1aRZs2bbj11lvZZZddmDhxIgBjxoxh2LBhzJgxg+LiYtq3b8+dd97ZyBGbmTUOJ9jNSETMlrQ/8FxadlEOfBvYUMelDwGHAi8BAfwoIt6RtMUJdkS8JGkR8GfgTeCZvNOTgJmS3o6IIyWNAqZKapue/zGw2QTbzJquOXPmVGsbM2ZM5WtJ3HrrrQ0ZkplZk+QEu4mLiOVA77zjm4CbaujaezPXBMmO9SVVxs7xr93nes2fto2qpe/NwM15x0+S1HqbmZmZbTP8NX1mZmZmZhnyDrYBIOlzwBM1nDoqIlY1dDxmZmZmzZUTbAMgTaL7NXYcZmZmZs2dS0TMzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzcysXm666SZ69+7NgQceyK9+9atq5yOCsWPHUlxcTN++fVm4cGHDB2lm1gS0bqiJJF0JlEfEz2s4VwJ8EhHPbub6ycCjEfHAZ4zjsoj42WcZ4zPOXwJcHBHHShoBHBAREyTtBjwKbA+MBfYEfgK8ExFH1jDOKGBARJz3GeMZBcyOiLc/yzjpWCcAf4mI17Z2jI/Wb6D7uMc+ayjWBFzUp4JRXssGs3zC8IKO/8orr3Dbbbcxb948tt9+e4455hiOPfZYiouLK/s8/vjjLF26lKVLlzJ37lzOPfdc5s6dW9C4zMyaoqayg10CfLmB5rrssw4gqVUWgUTEIxExIT08ClgcEV+MiDnAWcB3akquMzYK6FrfznXc+wnAAZ8xHjNrgl5//XUOOeQQ2rdvT+vWrTniiCN48MEHN+kzffp0Ro4ciSQGDRrE2rVrKSsra6SIzcwaT0ETbEmXS/qLpD8BvdK2sZJek/SypHsldQfGAD+U9KKkwzcz5GBJz0r6P0knpeNJ0g2SXpG0WNIpaXsXSU+nY74i6XBJE4B2advvaom5g6THJL2UXrdxvOWSrpO0ELhE0ry8a7pLWryZ9+EYSX9Or/16XvsoSbdI6gdcDxyfxvafwGHAHZJu2Mz70VXSTElLJV2fN+5p6XvxiqTr0rZWkibnvU8/TN/DAcDv0nnb1RJ//r1/U9J3JL2Qvkf/I6m9pC8DI4Ab0rE+n/6ZKWmBpDmS9tvMvZhZE9a7d2/mzJnDqlWr+PDDD5kxYwZvvvnmJn1KS0vp1q1b5XFRURGlpaUNHaqZWaMrWImIpP7AqUC/dJ6FwAJgHLBvRHwsaZeIWCtpIrWUj1TRhSTx3A94BHiAJGHtBxwEdAZekPQ08C1gVkRck+66to+IOZLOi4h+m5njGODtiBie3kfHvHOrIuLgtP1USftGxDLgFGBaLe/DDsBtwFeAv9bULyJelDSevJIPSUeSlJLM30ys/YAvAh8DSyTdDGwArgP6A2uA2WnpxpvAXhHROx1/43t/Xj3mqXrvn4uI29LXVwNnRcTNkh4hr4xH0hPAmIhYKukQ4Dfp+1D1PToHOAegc+fdGN+noo5QrDnYo11SJmINI5fLFWzs8vJyAI4//ngOPfRQ2rVrR/fu3SkrK9tk3lWrVrFo0SIqKpJ1X7NmDQsWLKi83pqG8vLygv68WMPxWjZdhazBPhx4KCI+BEiTL4CXSXZMHwYe3sIxH46IT4HXJO2Rth0GTI2IDcAKSX8EBgIvAL+V1Ca97sV6zrEYuDHd+X00LdfYKD85vo8ksZ6Q/n1KLePtByyLiKUAku4hTSYz8EREvJeO+xqwD/A5IBcR/0jbfwcMBn4K9EiT8MeA2Vs4V/69904T612AHYFZVTtL2pGk7Od+SRub29Y0cERMAiYB7N2jOG5c3GCPBlgBXdSnAq9lw1l+eknBxs7lcpSUlFBSUsINNyS/VLvssssoKiqipORf8/bt25fOnTtXtq1bt44RI0bQpUuXgsVmW27jelrz57Vsuhrjv37DSRK+44DLJfXZgms/znutWnsBEfG0pMHpfJMl/SIi7qprgoj4i6SDgWHA1ZKeiIifpKfX5XWdRpI8PphcliTQDSz//djAZtYzItZIOggYSlKSczIwegvmyr/3ycAJEfGSkockS2rovx2wto7fFlTTrk0rlhT4YS1rGLlcrqBJnzW8d999l913352///3vPPjggzz//PObnB8xYgS33HILp556KnPnzqVjx45Ors1sm1TIGuyngRMktZO0E0lCvR3QLSKeAi4FOpLsgH4A7LSV88wBTklrjHcjSd7nSdoHWJGWMtwOHJz2X5/uatdIUlfgw4i4B7gh77pNRMQbJEntFdRSHpL6M9Bd0ufT49Pqf2tbZR5whKTOaWnMacAfJXUGtouI/wF+zL/ua2ve+52AsvR9PD2vvXKsiHgfWCbpm1BZK3/Q1t6UmTW+b3zjGxxwwAEcd9xx3Hrrreyyyy5MnDiRiRMnAjBs2DB69OhBcXEx3/nOd/jNb37TyBGbmTWOgu1gR8RCSdOAl4B3SUo2ArgnrWsW8Ou0Dvj3wAOSjgfOr1KWUZeHgEPTeQL4UUS8I+kMkocR1wPlwMi0/yTgZUkLI+L0GsbrQ/Kg3qfAeuDczcw9jSQJ37e2DhHxz7TG+DFJH5J8INjaDxN1iogySeOAp0je48ciYnqa3N4paeOHqv9I/54MTJT0EXBoRHxUj2muAOYC/0j/3ng/9wK3SRoLnESSfP8/ST8G2qTnX/qs92hmjWPOnOr/1zxmzJjK15K49dZbGzIkM7MmSRHR2DGYVerVq1csWbKkscOwDLg2sOXwWrYsXs+Ww2vZ+CQtiIgBVdubyvdgm5mZmZm1CE3uEX9JlwPfrNJ8f0Rck/E8nwOeqOHUURGxaivHfIjq5SKXRkS1b9nYgjGHknztXr5lEXHi1o65mbkyj9/MzMxsW9PkEuw0kc40ma5lnlUk3yOd5ZiZJ71pctsgCW4h4jczMzPb1rhExMzMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzMzMzDLkBNvMzMzMLENOsM3MzMzMMuQE28zMzMwsQ06wzczMzMwy5ATbzMzMzCxDTrDNzGqwdu1aTjrpJPbbbz/2339/nnvuuU3ORwRjx46luLiYvn37snDhwkaK1MzMmprWjR2AWb6P1m+g+7jHGjsMy8BFfSoYVaC1XD5heEHGzXfBBRdwzDHH8MADD/DJJ5/w4YcfbnL+8ccfZ+nSpSxdupS5c+dy7rnnMnfu3ILHZWZmTZ93sFsgST+R9NWMxiqR9Ggt534gqX3e8WVZzGnW2N577z2efvppzjrrLAC23357dtlll036TJ8+nZEjRyKJQYMGsXbtWsrKyhohWjMza2qcYDdTkmr97UNEjI+I/22AMH4AtM87doJtLcKyZcvYbbfdOPPMM/niF7/I2Wefzbp16zbpU1paSrdu3SqPi4qKKC0tbehQzcysCXKJSCOT1AG4DygCWgE/Bf4K/ALYEVgJjIqIMkk54EXgMOD3kkYD+0bEp+k4fwZ6ALcBj0bEA5IGAjcBHYCPgaOAD4EJQAnQFrg1Iv5rM2HuKOkBoDewAPg2cD7QFXhK0kpgLtBO0ovAq8DlwMy0/8Fp28iI+LDq4JLOAc4B6Nx5N8b3qdiCd9Caqj3aJWUihZDL5Qoy7kZLlixhwYIFjBo1ilGjRnHzzTdz7rnnMnr06Mo+q1atYtGiRVRUJPe4Zs0aFixYQHl5eUFjawzl5eUFf8+t4Xg9Ww6vZdPlBLvxHQO8HRHDASR1BB4Hjo+If0g6BbgG2Phf9u0jYkDa92DgCOAp4FhgVkSsl0R6fntgGnBKRLwgaWfgI+As4L2IGCipLfCMpNkRsayWGL8IHAi8DTwD/FtE/FrShcCREbEyne+8iOiXvu4O9ALOiohnJP0W+B7w86qDR8QkYBLA3j2K48bF/rFsCS7qU0Gh1nL56SUFGXej/fbbj2uvvZbvfe97ALRq1YoJEyZQUvKvefv27Uvnzp0r29atW8eIESPo0qVLQWNrDLlcbpN7t+bN69lyeC2bLpeINL7FwBBJ10k6HOhGslP8h3Q3+Mcku9sbTavy+pT09alVzkGS4JZFxAsAEfF+RFQARwMj0/HnAp8Dem4mxnkR8VZEfEqyg969nvf2ZkQ8k76+h2Tn3azJ23PPPenWrRtLliwB4IknnuCAAw7YpM+IESO46667iAief/55Onbs2CKTazMz23LeKmxkEfGXdCd6GHA18CTwakQcWssl+YWgjwA/k9QJ6J9eWx8Czo+IWfXs/3He6w3U/+cm6jiupl2bVixpgG+IsMLL5XIF32kupJtvvpnTTz+dTz75hB49enDnnXcyceJEAMaMGcOwYcOYMWMGxcXFtG/fnjvvvLORIzYzs6bCCXYjk9QVWB0R90haS1JGsZukQyPiOUltgC9ExKtVr42IckkvkNRYPxoRG6p0WQJ0kTQwLRHZiaREZBZwrqQn05KSLwClEbGOLfMBsBNJnTjAekltImJ9erz3xvsAvgX8aQvHN2s0/fr1Y/78+Zu0jRkzpvK1JG699daGDsvMzJoBJ9iNrw9wg6RPgfXAuUAF8Ou0Hrs18CuShwRrMg24n+SBxU1ExCdpDffNktqRJNdfBW4nKfNYqKRg+x/ACVsR+yRgpqS3I+LI9PhlSQtJHnJcAnw/rb9+Dfh/WzGHmZmZWbPiBLuRpWUaNZVqDK6hb0kNbQ+QlHzkt43Ke/0CMKiG8S+jHl+rFxE5IJd3fF7e65uBm/OOLwUuhcqHHCsi4tt1zWFmZmbWkvghRzMzMzOzDHkH2wCQ1Ae4u0rzxxFxyNaMFxHLSb4NxczMzGyb4gTbAIiIxUC/xo7DzMzMrLlziYiZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm5mZmZllyAm2mZmZmVmGnGCbmZmZmWXICbaZmZmZWYacYJuZmZmZZcgJtpmZmZlZhpxgm1mz1b17d/r06UO/fv0YMGBAtfMRwdixYykuLqZv374sXLiwEaI0M7NtTevGDsAs30frN9B93GONHYZlYPIxHRpknqeeeorOnTvXeO7xxx9n6dKlLF26lLlz53Luuecyd+7cBonLzMy2Xd7BTkm6UtLFtZwrkfTlOq6fLOmkDOK47LOO0dRIOkHSAY0dh217pk+fzsiRI5HEoEGDWLt2LWVlZY0dlpmZtXBOsOunBNhsgp2hz5xgS2qVRSAZOgFwgm2Zk8TRRx9N//79mTRpUrXzpaWldOvWrfK4qKiI0tLShgzRzMy2Qdt0iYiky4EzgHeBN4EFksYCY4AK4DVgXHq8QdK3gfMjYk4tQw6WdCGwJ/CjiHhAkoDrga8BAVwdEdMkdQGmATuTrMO5wHCgnaQXgVcj4vQaYu4A3AcUAa2An6bjLU/HGwLcJ+nrEfGl9JruwO8jok8t78NA4CagA/AxcBSwHvh/wID0vbgwIp6SNAoYEBHnpdc+Cvw8InKSytNxjgU+Ao4HPg+MAI6Q9GPgGxHxRpX5zwHOAejceTfG96mo5e215qS8vJxcLlfQOa6//np222031qxZw8UXX8xHH33EQQcdVHl+1apVLFq0iIqK5GdqzZo1LFiwgPLy8oLG1dI0xFpaw/F6thxey6Zrm02wJfUHTgX6kbwPC4EFJAn1vhHxsaRdImKtpIlAeUT8vI5huwCHAfsBjwAPAF9P5zgI6Ay8IOlp4FvArIi4Jt1xbh8RcySdFxH9NjPHMcDbETE8vY+OeedWRcTBafupkvaNiGXAKSTJd03vw/bpuVMi4gVJO5MkxxcAERF9JO0HzJb0hTruvwPwfERcLul64DsRcbWkR4BHI+KBmi6KiEnAJIC9exTHjYu32R/LFmXyMR0oKSlpsPleeukl1q9fv8mcffv2pXPnzpVt69atY8SIEXTp0qXB4moJcrlcg66lFZbXs+XwWjZd23KJyOHAQxHxYUS8T5IQA7wM/C7drd7SrdSHI+LTiHgN2CNtOwyYGhEbImIF8EdgIPACcKakK4E+EfFBPedYDAyRdJ2kwyPivbxz+Un0fSSJNWwmwQZ6AWUR8QJARLwfERVp3PekbX8G/gbUlWB/Ajyavl4AdK/XHZlthXXr1vHBBx9Uvp49eza9e/fepM+IESO46667iAief/55Onbs6OTazMwKzluF1Q0HBgPHAZdLqrGsohYf573W5jpGxNOSBqfzTZb0i4i4q64JIuIvkg4GhgFXS3oiIn6Snl6X13UacL+kB5PLYukW3MfmVLDpB7Md8l6vj4hIX29gK36+2rVpxZIJwz9DeNZUFPrXlitWrODEE08EoKKigm9961scc8wxTJw4EYAxY8YwbNgwZsyYQXFxMe3bt+fOO+8saExmZmawbSfYT5MktteSvA/HAbcB3dJa4z+RlJDsCHxAUiu9NeYA35U0BehEkrxfImkf4K2IuE1SW+Bg4C5gvaQ2EbG+psEkdQVWR8Q9ktYCZ9fULyLekLQBuILad68BlgBdJA1MS0R2IikRmQOcDjyZlobsnfbdGfiepO2AvYAv1eM9+ADYqR79zOqtR48evPTSS9Xax4wZU/laErfeemtDhmVmZrbtJtgRsVDSNOAlkoccXyB5CPGetK5ZwK/TGuzfAw9IOp7NP+RYk4eAQ9N5guThx3cknUGSaK8HyoGRaf9JwMuSFtb0kCPQB7hB0qckDyKeu5m5pwE3APvW1iEiPpF0CnCzpHYkyfVXgd8A/0/SYpJd61FpXfozwDKSB0BfJ6ldr8u9wG3pA6QnVX3I0czMzKwl0b9+o2/W+Hr16hVLlixp7DAsA374puXwWrYsXs+Ww2vZ+CQtiIhq/5TwtvyQo5mZmZlZ5rbZEpGtlX539jerNN8fEddkPM/ngCdqOHVURKzayjEfonq5yKURMWtrxjMzMzOz6pxgb6E0kc40ma5lnlUk35+d5ZgnZjmemZmZmVXnEhEzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDNrtrp3706fPn3o168fAwYMqHY+Ihg7dizFxcX07duXhQsXNkKUZma2rWnd2AGY5fto/Qa6j3usscOwDEw+pkODzPPUU0/RuXPnGs89/vjjLF26lKVLlzJ37lzOPfdc5s6d2yBxmZnZtqvZ7GBLulLSxbWcK5H05TqunyzppAziuOyzjtFS1WcdzBrS9OnTGTlyJJIYNGgQa9eupaysrLHDMjOzFq7ZJNh1KAEaKrFr9gm2pK3+zUUd15bQcOtghiSOPvpo+vfvz6RJk6qdLy0tpVu3bpXHRUVFlJaWNmSIZma2DWrSJSKSLgfOAN4F3gQWSBoLjAEqgNeAcenxBknfBs6PiDm1DDlY0oXAnsCPIuIBSQKuB74GBHB1REyT1AWYBuxM8j6dCwwH2kl6EXg1Ik6vIeYOwH1AEdAK+Gk63njgOKAd8Czw3YgISTlgEXA40AEYCfwH0AeYFhE/Tsf9NjAW2B6YC3wvIjbUMH8r4A5gQHo/v42IX6bzvAgcBvxe0mhg34j4NI35z0CPiFhfw5j5106V9Bfgx2ksq4DT0/vaZB3SMScCe6dD/SAinqlh/HOAcwA6d96N8X0qqnaxZqi8vJxcLlfQOa6//np222031qxZw8UXX8xHH33EQQcdVHl+1apVLFq0iIqK5GdqzZo1LFiwgPLy8oLG1dI0xFpaw/F6thxey6arySbYkvoDpwL9SOJcCCwgSaj3jYiPJe0SEWslTQTKI+LndQzbhSRJ3A94BHgA+Ho6x0FAZ+AFSU8D3wJmRcQ1adLaPiLmSDovIvptZo5jgLcjYnh6Hx3T9lsi4idp293AscDv03OfRMQASRcA04H+wGrgDUm/BHYHTgH+LSLWS/oNSVJ7Vw3z9wP2ioje6Vy75J3bPiIGpO0HA0cAT6WxzKopua7l2l2BQekHhLNJPqxcVHUdJP038MuI+JOkvYFZwP5VB46IScAkgL17FMeNi5vsj6VtgcnHdKCkpKTB5nvppZdYv379JnP27duXzp07V7atW7eOESNG0KVLlwaLqyXI5XINupZWWF7PlsNr2XQ15RKRw4GHIuLDiHifJCEGeBn4XbpLuqVbnQ9HxKcR8RqwR9p2GDA1IjZExArgj8BA4AXgTElXAn0i4oN6zrEYGCLpOkmHR8R7afuRkuZKWgx8BTgw75pH8q59NSLKIuJj4P+AbsBRJEn3C+nu+VFAj1rm/z+gh6SbJR0DvJ93blqV16ekr0+tcq4m+eeLgFnpvVxS5V7yfRW4JY35EWBnSTvWMY9Zvaxbt44PPvig8vXs2bPp3bv3Jn1GjBjBXXfdRUTw/PPP07FjRyfXZmZWcM1xq3A4MJik3OJySX224NqP815rcx0j4mlJg9P5Jkv6RUTUtGNc9bq/pLvDw4CrJT1BUoLyG2BARLyZJu071BDXp1Vi/JRkjQRMiYj/qMf8ayQdBAwlKdk4GRidnl6X1/UR4GeSOpEk70/WMXT+tTcDv4iIRySVAFfWcs12JDvd/6wr7o3atWnFkgnD69vdmrBC/9pyxYoVnHjiiQBUVFTwrW99i2OOOYaJEycCMGbMGIYNG8aMGTMoLi6mffv23HnnnQWNyczMDJp2gv00SWJ7LUmcxwG3Ad0i4ilJfyLZed0R+ICkVnprzAG+K2kK0Ikkeb9E0j7AWxFxm6S2wMEkJRnrJbWprZxCUldgdUTcI2ktcDb/SqZXpju4J5GUp9TXE8B0Sb+MiHfTpHiniPhbDfN3Jik5+R9JS4B7ahowIsolvQDcBDxaUz33ZnQENj4pdkZee9V1mE1Si31DGlu/iHhxC+Yxq1WPHj146aWXqrWPGTOm8rUkbr311oYMy8zMrOkm2BGxUNI04CWShxxfIHlo7560rlnAr9Ma7N8DD0g6ns0/5FiTh4BD03mCpJ74HUlnkCTa64FykocPIakVflnSwpoeciR5OPEGSZ8C64Fz0xhvA14B3knvZUvei9ck/RiYLWm7dNzvA9USbGAv4M60HyQPTNZmGnA/ybd/bIkrgfslrSHZ+d43bd9kHUgeyrxV0sskP2tPk+yqm5mZmbVYiojGjsGsUq9evWLJkiWNHYZlwA/ftBxey5bF69lyeC0bn6QFG78EIl9TfsjRzMzMzKzZabIlIlsr/e7sb1Zpvj8irsl4ns+R1EZXdVRErMpyrs3EMBdoW6X53yNi8VaOdyvwb1Wab4oIPxlmZmZmVk8tLsFOE+lMk+la5llF8p3TjSYiDsl4vO9nOZ6ZmZnZtsglImZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZgW1YcMGvvjFL3LsscdWO/fxxx9zyimnUFxczCGHHMLy5csbPkAzM7OMtW7sAMzyfbR+A93HPdbYYWwzlk8YXvA5brrpJvbff3/ef//9aufuuOMOdt11V/76179y7733cumllzJt2rSCx2RmZlZILW4HW9KVki6u5VyJpC/Xcf1kSSdlEMdlW9i/Mm5JP5H01fT14ZJelfSipHaSbkiPb/isMWZJ0i6SvtfYcVjT8tZbb/HYY49x9tln13h++vTpnHHGGQCcdNJJPPHEE0REQ4ZoZmaWuRaXYNehBNhsgp2hLUqw80XE+Ij43/TwdODaiOgXER8B5wB9I+KSrRlb0lb/1qKOa3cBnGDbJn7wgx9w/fXXs912Nf9fTWlpKd26dQOgdevWdOzYkVWrVjVkiGZmZplrESUiki4HzgDeBd4EFkgaC4wBKoDXgHHp8QZJ3wbOj4g5tQw5WNKFwJ7AjyLiAUkCrge+BgRwdURMk9QFmAbsTPJ+ngsMB9pJehF4NSJOr2/caftk4FGSpPVkYKikrwE7ATum93dtRFT7XbqkbwL/CWwA3ouIwZJGAV9Pr20lqQy4OyIey58vIh6oYbyq1w4HpgO7Am2AH0fEdGAC8Pn0nv8QEZdIuiSNvy3wUET8Zy3vwzkkHxzo3Hk3xvepqKmbFUAulyvY2E8++STr16/ngw8+4MUXX2TVqlXV5lu3bh3PPfccu+22GwD//Oc/eeaZZ+jYsWPB4rItV15eXtCfFWtYXs+Ww2vZdDX7BFtSf+BUoB/J/SwkSVTHAftGxMeSdomItZImAuUR8fM6hu0CHAbsBzwCPECSZPYDDgI6Ay9Iehr4FjArIq6R1ApoHxFzJJ0XEf22Iu5KEXG7pMPIS34llW9uXGA8MDQiSiXtktd+MMnO92pJJ5Ikvo9J2h44iuSDQW3yr20NnBgR70vqDDwv6RGS97v3xtgkHQ30BL4ECHhE0uCIeLrq4BExCZgEsHeP4rhxcbP/sWw2lp9eUrCxb7vtNhYsWMCoUaP45z//yfvvv8/tt9/OPffcU9nnC1/4AkVFRRx66KFUVFTw8ccfM2LECJLPs9ZU5HI5SkpKGjsMy4jXs+XwWjZdLaFE5HCS3dEPI+J9koQY4GXgd+lu9ZZuiT4cEZ9GxGvAHmnbYcDUiNgQESuAPwIDgReAMyVdCfSJiA8+Y9yf1TPAZEnfAVrltf8hIlanrx8HjpTUlmRH/um0/KQ2+dcK+Jmkl4H/BfbiX+9RvqPTP4tIPjzsR5Jw2zbiO9/5Dm+99RbLly/n3nvv5Stf+comyTXAiBEjmDJlCgAPPPAAX/nKV5xcm5lZs9eStwqHA4OB44DLJfXZgms/znu92f/aR8TTkgan802W9IuIuGuLo81IRIyRdEgaz4J0pxxgXV6ff0rKAUOBU4B76xh2Xd7r04HdgP4RsV7ScmCHGq4RSe34f21J/O3atGJJA3yzhTWe8ePHM2DAAEaMGMFZZ53Fv//7v1NcXEynTp249966fhTNzMyavpawg/00cEL6DRs7kSTU2wHdIuIp4FKgI0kN8QckdcxbYw5wiqRWknYjSd7nSdoHWBERtwG3k5RTAKyX1GYL4/7MJH0+IuZGxHjgH0C3WrpOA84k2UmfuQVTdATeTZPrI4F90vaq7+0sYLSkHdO49pK0+xbMYy1ISUkJjz76KAA/+clPGDFiBAA77LAD999/P3/961+ZN28ePXr0aMwwzczMMtHsd7AjYqGkacBLJA8LvkDyEOI9kjqS7KT+Oq3B/j3wgKTj2fxDjjV5CDg0nSdIHn58R9IZwCWS1gPlwMi0/yTgZUkLa3rIsZa4s3CDpJ4k9/1EOn6/GvrNBu4GpkfEJ1sw/u+A30taDMwH/gwQEaskPSPpFeDx9CHH/YHn0l/5lwPfJrlXMzMzsxZL/s5Za0p69eoVS5YsaewwLAN++Kbl8Fq2LF7PlsNr2fgkLYiIAVXbW0KJiJmZmZlZk9HsS0S2Vvod1N+s0nx/RFyT8TyfIynVqOqoiNjqf1Ej6/glDQWuq9K8LCJO3JrxzMzMzLZV22yCnSaimSbTtcyzipproD/ruJnGHxGzSB5MNDMzM7PPwCUiZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm1mZmZmliEn2GZmZmZmGXKCbWZmZmaWISfYZmZmZmYZcoJtZmZmZpYhJ9hmZmZmZhlygm0FJ+kESQc0dhzN2ZtvvsmRRx7JAQccwIEHHshNN91UrU9EMHbsWIqLi+nbty8LFy5shEjNzMysdWMHYNmT1CoiNjR2HHlOAB4FXqur40frN9B93GMFDyhryycML+j4rVu35sYbb+Tggw/mgw8+oH///gwZMoQDDvjX55bHH3+cpUuXsnTpUubOncu5557L3LlzCxqXmZmZVecd7GZIUgdJj0l6SdIrkk6RtFzSdZIWApdImpfXv7ukxZsZb6CkZ9Px5knaSdIOku6UtFjSIklHpn1HSbol79pHJZWkr8slXZOO87ykPSR9GRgB3CDpRUmfL9Db0qJ16dKFgw8+GICddtqJ/fffn9LS0k36TJ8+nZEjRyKJQYMGsXbtWsrKyhojXDMzs22aE+zm6Rjg7Yg4KCJ6AzPT9lURcXBETAC2l7Rv2n4KMK2mgSRtn567ICIOAr4KfAR8H4iI6AOcBkyRtEMdcXUAnk/HeRr4TkQ8CzwCXBIR/SLija29aUssX76cRYsWccghh2zSXlpaSrdu3SqPi4qKqiXhZmZmVnguEWmeFgM3SroOeDQi5kiCTZPo+0gS6wnp36fUMlYvoCwiXgCIiPcBJB0G3Jy2/VnS34Av1BHXJySlIAALgCH1uRlJ5wDnAHTuvBvj+1TU57ImJZfLNcg8H330ERdccAFnn312tRrrVatWsWjRIioqkvdvzZo1LFiwgPLy8gaJrary8vIGe1+ssLyWLYvXs+XwWjZdTrCboYj4i6SDgWHA1ZKeSE+ty+s2Dbhf0oPJJbE0o+kr2PQ3H/m72usjItLXG6jnz1dETAImAezdozhuXNz8fiyXn15S8DnWr1/Psccey5gxY7jwwgurne/bty+dO3empCSJZd26dYwYMYIuXboUPLaa5HK5ylisefNatixez5bDa9l0Nb9MxpDUFVgdEfdIWgucXbVPRLwhaQNwBbWUh6SWAF0kDYyIFyTtRFIiMgc4HXhS0heAvdO+OwPfk7QdsBfwpXqE/AGwU33urV2bViwp8AODzVFEcNZZZ7H//vvXmFwDjBgxgltuuYVTTz2VuXPn0rFjx0ZLrs3MzLZlTrCbpz4kDw1+CqwHzgUeqKHfNOAGYN8azgEQEZ9IOgW4WVI7kuT6q8BvgP+XPhxZAYyKiI8lPQMsI/lGkNeB+nwX3L3AbZLGAie5DnvLPfPMM9x999306dOHfv36AfCzn/2Mv//97wCMGTOGYcOGMWPGDIqLi2nfvj133nlnI0ZsZma27XKC3QxFxCxgVpXm7jX0+znw83qM9wIwqIZTZ9bQN0h2tmsaZ8e81w+QJv0R8Qzg78H+DA477DD+VX1TM0nceuutDRSRmZmZ1cbfImJmZmZmliHvYG9DJD1E9XKRS9MdcTMzMzPLgBPsbUhEnNjYMZiZmZm1dC4RMTMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTYzMzMzy5ATbDMzMzOzDDnBNjMzMzPLkBNsMzMzM7MMOcE2MzMzM8uQE2wzMzMzsww5wTbLyOjRo9l9993p3bt3jecjgrFjx1JcXEzfvn1ZuHBhA0doZmZmDaF1Ywdglu+j9RvoPu6xgoy9fMLwgoy70ahRozjvvPMYOXJkjecff/xxli5dytKlS5k7dy7nnnsuc+fOLWhMZmZm1vCaxQ62pCslXVzLuRJJX67j+smSTsogjss+6xgtmaQTJB3Q2HE0lsGDB9OpU6daz0+fPp2RI0ciiUGDBrF27VrKysoaMEIzMzNrCM0iwa5DCbDZBDtDnznBltSq0NdW7be1121FnxOAbTbBrktpaSndunWrPC4qKqK0tLQRIzIzM7NCaLIlIpIuB84A3gXeBBZIGguMASqA14Bx6fEGSd8Gzo+IObUMOVjShcCewI8i4gFJAq4HvgYEcHVETJPUBZgG7EzyHp0LDAfaSXoReDUiTq8h5g7AfUAR0Ar4aTre8nS8IcB9kr4eEV9Kr+kO/D4i+tTyPuRfe72k1cBVQFvgDeDMiCj//+3dfayf5V3H8fcHyszaMhTakMqT3VjQKrV2m5JYap3MIGQFXKZ0qMxMGQnzIS4LizoHBuJMHNMtVUeVh26s2xjQVcXA0lELxLKu9AG22YG2xmF56or0FDbb8vWP+y45Hk4faO/T3wPvV9Kc+3dd931d39+5zp1+z5Xv7z7jnPexMa9D8wtCgH+qqqvb8UeATwPnAVcBDxxCDMcDVwCvAx4Hfh2YAywEfi7JHwPvai9fDEwHXgB+u6r+bZzxr2jHY9q06fzJ2XvG+1YcsVWrVk3IuKM9+eST7Nq1a9y5tm/fzvr169mzp3l/O3bsYN26dYyMjEx4XL0wMjJyVL7nmniu5XBxPYeHa9m/+jLBTvIW4FKapG0S8DCwjiahnllV30/yg1X1XJK/BUaq6i8OMuwMYB7wo8AK4EvAL7dz/CQwDVibZDXwHuCeqrq+3bGdXFX3J/lAVc05wBznA/9dVRe27+OEUX3bq2pu235pkplVtQX4VZrk9UC2V9XcJNOAO4HzqmpXkquBPwD+dJw5Pjbquh8G1gBvAXYA9ya5uKqWA1OAh6rqg4cSQzv2SVW1pD2+DnhfVX0qyQrgH6vqS23fSuDKqnosyc8Afw28fezAVXUjcCPA6W88sz7+yMT8WG69bMGEjPv/5ti6lSlTprBgwSvnmj17NtOmTXu5b9euXSxcuJAZM2ZMeFy9sGrVqnG/Dxo8ruVwcT2Hh2vZv/q1RORc4K6qeqGqnqdJiAE2Abe1u9WvdptzeVW9VFXfBE5u2+YBy6pqb1U9BfwL8DZgLfCbSa4Bzq6qnYc4xyPAO5L8eZJzq+p/RvWNTqK/SJNYw6El2Pv6z6EpwXiw3Um/HDhjP3OMfv02YFVVPVNVe4DbgPlt317gjoPMP3bsn0hyf5JHgMuAHx97cpKpNKU7t7exfprml5zXrIULF7J06VKqijVr1nDCCScMbXItSdJrWV/uYB/AhTSJ4TuBP0oyblnFfnx/1HEOdGJVrU4yv53vliQ3VNXSg01QVd9OMhe4ALguycqq2re7vGvUqV+gSTzvbC6rxw4y9L5rA3ylqhYd5Lz9vR7P96pq7yGcN3qsW4CLq2pjkvfS1MGPdQzw3EF2/F/h9ccdy+YJftrHRFm0aBGrVq3i2Wef5dRTT+Xaa69l9+7dAFx55ZVccMEF3H333Zx55plMnjyZm2++uccRS5KkidCvCfZqmsT2z2hifCewBDitqu5L8gBNCclUYCdNrfThuB94f5JbgRNpkvcPJTkD+E5VLUnyA8BcYCmwO8lxVbV7vMHaUozvVtVnkzwH/NZ451XVvyfZC3yEg+9ej7YGWJzkzKp6vK35PqWqvn2Q674GfLItMdkBLAI+9SrmHet4YFuS42h2sPd9Um9n20dVPZ9kS5J3V9XtbQ347KraeATz9rVly5YdsD8JixcvPkrRSJKkXunLEpGqepgm8dwI/DNNyUYBn23LEtYDn6yq54B/AC5JsiHJua9yqrtoyk42Al+l+fDjkzQ7shuTrKcp4fir9vwbgU1JbtvPeGcDX2tLIj4KXHeAub8A/BpNucghqapngPcCy5JsAv6Vpqb8YNdto6lfv4/mva6rqi8f6rzj+AjwEPAgMPpDi5+n+QVlfZI30STf70uyEfgGcNERzClJkjQQUlW9jkF62VlnnVWbN2/udRjqgB++GR6u5XBxPYeHa9l7SdZV1VvHtvflDrYkSZI0qPq1BvuwtM/OfveY5tur6vqO5zkJWDlO1y9U1fbDHPMuYOaY5qur6p7DGW9QY5AkSRp0Q5Vgt4l0p8n0fubZTvP87C7HvKTL8QY1BkmSpEFniYgkSZLUIRNsSZIkqUMm2JIkSVKHTLAlSZKkDplgS5IkSR0ywZYkSZI6ZIItSZIkdcgEW5IkSeqQCbYkSZLUIRNsSZIkqUMm2JIkSVKHTLAlSZKkDplgS5IkSR0ywZYkSZI6ZIItSZIkdcgEW5IkSeqQCbYkSZLUIRNsSZIkqUMm2JIkSVKHUlW9jkF6WZKdwOZex6FOTAOe7XUQ6oRrOVxcz+HhWvbeGVU1fWzjpF5EIh3A5qp6a6+D0JFL8nXXcji4lsPF9RwermX/skREkiRJ6pAJtiRJktQhE2z1mxt7HYA641oOD9dyuLiew8O17FN+yFGSJEnqkDvYkiRJUodMsCVJkqQOmWCrLyQ5P8nmJI8n+XCv49GRSbI1ySNJNiT5eq/j0aFLclOSp5M8OqrtxCRfSfJY+/WHehmjDs1+1vKaJE+09+aGJBf0MkYdmiSnJbkvyTeTfCPJ77Xt3pt9ygRbPZfkWGAx8EvALGBRklm9jUod+PmqmuMzWgfOLcD5Y9o+DKysqjcDK9vX6n+38Mq1BPhEe2/Oqaq7j3JMOjx7gA9W1SzgHOCq9v9J780+ZYKtfvDTwONV9R9V9b/A54GLehyT9JpUVauB745pvgi4tT2+Fbj4aMakw7OftdQAqqptVfVwe7wT+BZwCt6bfcsEW/3gFOC/Rr3+TtumwVXAvUnWJbmi18HoiJ1cVdva4yeBk3sZjI7YB5JsaktILCkYMEl+BPgp4CG8N/uWCbakiTCvqubSlP1clWR+rwNSN6p5tqvPdx1cfwO8CZgDbAM+3tNo9KokmQrcAfx+VT0/us97s7+YYKsfPAGcNur1qW2bBlRVPdF+fRq4i6YMSIPrqSQzANqvT/c4Hh2mqnqqqvZW1UvAErw3B0aS42iS69uq6s622XuzT5lgqx+sBd6cZGaS1wGXAit6HJMOU5IpSY7fdwz8IvDoga9Sn1sBXN4eXw58uYex6AjsS8Zal+C9ORCSBPh74FtVdcOoLu/NPuVfclRfaB8V9ZfAscBNVXV9byPS4UryRppda4BJwOdcz8GRZBmwAJgGPAV8FFgOfBE4HfhP4Feqyg/P9bn9rOUCmvKQArYC7x9Vw6s+lWQecD/wCPBS2/yHNHXY3pt9yARbkiRJ6pAlIpIkSVKHTLAlSZKkDplgS5IkSR0ywZYkSZI6ZIItSZIkdWhSrwOQJOlAkuyleTzZPhdX1dYehSNJB+Vj+iRJfS3JSFVNPYrzTaqqPUdrPknDxxIRSdJASzIjyeokG5I8muTctv38JA8n2ZhkZdt2YpLlSTYlWZNkdtt+TZLPJHkQ+EyS6UnuSLK2/fezPXyLkgaMJSKSpH73+iQb2uMtVXXJmP73APdU1fVJjgUmJ5kOLAHmV9WWJCe2514LrK+qi5O8HVhK85cNAWYB86rqxSSfAz5RVQ8kOR24B/ixCXuHkoaKCbYkqd+9WFVzDtC/FrgpyXHA8qrakGQBsLqqtgCM+vPR84B3tW1fTXJSkje0fSuq6sX2+DxgVpJ9c7whydSqGunqTUkaXibYkqSBVlWrk8wHLgRuSXIDsOMwhto16vgY4Jyq+l4XMUp6bbEGW5I00JKcATxVVUuAvwPmAmuA+UlmtufsKxG5H7isbVsAPFtVz48z7L3A74yaY84EhS9pCLmDLUkadAuADyXZDYwAv1FVzyS5ArgzyTHA08A7gGtoykk2AS8Al+9nzN8FFrfnTQJWA1dO6LuQNDR8TJ8kSZLUIUtEJEmSpA6ZYEuSJEkdMsGWJEmSOmSCLUmSJHXIBFuSJEnqkAm2JEmS1CETbEmSJKlD/wcdNIsibzC4YgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ax = xgb.plot_importance(model)\n",
    "fig = ax.figure\n",
    "fig.set_size_inches(10, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Impact of Skewed Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we prepared our data, we pointed out that there were more anomalies than normal data and considered the implications of this dataset skew that doesn't match the real world. Take a moment now see how adjusting our dataset impacts performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reduce_anomalies(df, pct_anomalies=.01):\n",
    "    labels = df['label'].copy()\n",
    "    is_anomaly = labels != 'normal.'\n",
    "    num_normal = np.sum(~is_anomaly)\n",
    "    num_anomalies = int(pct_anomalies * num_normal)\n",
    "    all_anomalies = labels[labels != 'normal.']\n",
    "    anomalies_to_keep = np.random.choice(all_anomalies.index, size=num_anomalies, replace=False)\n",
    "    anomalous_data = df.iloc[anomalies_to_keep].copy()\n",
    "    normal_data = df[~is_anomaly].copy()\n",
    "    new_df = pd.concat([normal_data, anomalous_data], axis=0)\n",
    "    return new_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = reduce_anomalies(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what anomalies we have after the reduction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>normal.</th>\n",
       "      <td>972781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>smurf.</th>\n",
       "      <td>6903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>neptune.</th>\n",
       "      <td>2705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>portsweep.</th>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ipsweep.</th>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>satan.</th>\n",
       "      <td>28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>back.</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>nmap.</th>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>warezclient.</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>teardrop.</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               label\n",
       "normal.       972781\n",
       "smurf.          6903\n",
       "neptune.        2705\n",
       "portsweep.        41\n",
       "ipsweep.          34\n",
       "satan.            28\n",
       "back.              7\n",
       "nmap.              6\n",
       "warezclient.       2\n",
       "teardrop.          1"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(df['label'].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Return to [data preprocessing](#return) and rerun cells to this point, comparing and contrasting performance. Again, reflect below, on paper, or with a peer. Reflect on *why* the reduction of anomalies had the impact that it did."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the impact of reducing anomalies in the dataset and why do you think that is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4:  Multi-Class Classification Model [Exercise]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've seen that XGBoost can identify anomalies in the KDD99 dataset really easily.  Let's see if XGBoost can detect the different types of anomalies.  To do that we will train a multi-classification model.\n",
    "\n",
    "We'll point out the difference, then have you implement the pieces that are the same using the binary classifier as a model.\n",
    "\n",
    "For multi-classification models, the labels have to be converted to integers (we've already done this) and the 'objective\n",
    "' and 'num_class' parameters must be changed accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1 Training the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'num_rounds': 10, 'max_depth': 8, 'max_leaves': 256, 'alpha': 0.9, 'eta': 0.1, 'gamma': 0.1, 'learning_rate': 0.1, 'subsample': 1, 'reg_lambda': 1, 'scale_pos_weight': 2, 'tree_method': 'gpu_hist', 'n_gpus': 1, 'objective': 'multi:softprob', 'verbose': True, 'num_class': 23}\n"
     ]
    }
   ],
   "source": [
    "num_labels = len(le.classes_)\n",
    "params['objective'] = 'multi:softprob'\n",
    "params['num_class'] = num_labels\n",
    "print(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see how adding multiple classes doesn't increase the complexity in training this type of model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10:16:20] WARNING: ../src/learner.cc:576: \n",
      "Parameters: { \"n_gpus\", \"num_rounds\", \"scale_pos_weight\", \"verbose\" } might not be used.\n",
      "\n",
      "  This could be a false alarm, with some parameters getting used by language bindings but\n",
      "  then being mistakenly passed down to XGBoost core, or some parameter actually being used\n",
      "  but getting flagged wrongly here. Please open an issue if you find any such cases.\n",
      "\n",
      "\n",
      "[10:16:20] WARNING: ../src/learner.cc:1115: Starting in XGBoost 1.3.0, the default evaluation metric used with the objective 'multi:softprob' was changed from 'merror' to 'mlogloss'. Explicitly set eval_metric if you'd like to restore the old behavior.\n",
      "[0]\ttest-mlogloss:2.03009\ttrain-mlogloss:2.03008\n",
      "[1]\ttest-mlogloss:1.66617\ttrain-mlogloss:1.66616\n",
      "[2]\ttest-mlogloss:1.41777\ttrain-mlogloss:1.41776\n",
      "[3]\ttest-mlogloss:1.22860\ttrain-mlogloss:1.22860\n",
      "[4]\ttest-mlogloss:1.07658\ttrain-mlogloss:1.07658\n",
      "[5]\ttest-mlogloss:0.95044\ttrain-mlogloss:0.95043\n",
      "[6]\ttest-mlogloss:0.84352\ttrain-mlogloss:0.84351\n",
      "[7]\ttest-mlogloss:0.75157\ttrain-mlogloss:0.75155\n",
      "[8]\ttest-mlogloss:0.67162\ttrain-mlogloss:0.67161\n",
      "[9]\ttest-mlogloss:0.60155\ttrain-mlogloss:0.60154\n",
      "CPU times: user 4.6 s, sys: 279 ms, total: 4.88 s\n",
      "Wall time: 1.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "dtrain = xgb.DMatrix(x_train, label=y_train)\n",
    "dtest = xgb.DMatrix(x_test, label=y_test)\n",
    "evals = [(dtest, 'test',), (dtrain, 'train')]\n",
    "model = xgb.train(params, dtrain, num_rounds, evals=evals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2 Testing and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us again evaluate our model on the test set and see how it performs. Note that accuracy now is a metric of how well we do on each of the 23 classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = model.predict(dtest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_labels = np.argmax(preds, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11, 11, 11, ..., 11, 11, 11])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "true_labels = y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([11, 11, 11, ..., 11, 11, 11])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3 Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Score : 0.9998656499489063\n"
     ]
    }
   ],
   "source": [
    "print ('Accuracy Score :', accuracy_score(true_labels, pred_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should be able to get near perfect accuracy. Let's look at the confusion matrix to make sure that we are detecting the anomalies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4 Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix :\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr8AAALICAYAAAB2PpiXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABHU0lEQVR4nO3de7xcdXXw/89KQriqEBMgOYdLiAiGW0wOBCSlVqpyEaMtoBZNUBSstpRan9a2Wlr79PmVihdsqy0VqtYWISiCkWIQRS5CIAGUm2KUKElQEm4qF0MO6/fH7IRDTE5us2dP5vt55zWvzOzZM3uts72srLP2d0dmIkmSJJVgRNMBSJIkSZ1i8StJkqRiWPxKkiSpGBa/kiRJKobFryRJkooxqukAJEmS9Jti7HbJymebDgN++czXM/OYpsNoF4tfSZKkbrTyWZi+a9NRwDeWjm06hHZy7EGSJEnFsPiVJElSMRx7kCRJ6lYRTUfQc+z8SpIkqRh2fiVJkrpRYJuyBv5IJUmSVAyLX0mSJBXDsQdJkqRu5QVvbWfnV5IkScWw+JUkSVIxHHuQJEnqVk49tJ2dX0mSJBXDzq8kSVJXCi94q4GdX0mSJBXD4leSJEnFcOxBkiSpG3l741r4I5UkSVIxLH4lSZJUDMceJEmSupWrPbSdnV9JkiQVw+JXkiRJxXDsQZIkqVs59dB2dn4lSZJUDDu/kiRJ3SiAEbZ+283OryRJkoph8StJkqRiOPYgSZLUrZx6aDs7v5IkSSqGxa8kSZKK4diDJElSt/L2xm1n51eSJEnFsPMrSZLUrWz8tp2dX0mSJBXD4leSJEnFcOxBkiSpG3l741rY+ZUkSVIxLH4lSZJUDMceJEmSupVTD21n51eSJEnFsPMrSZLUlcI7vNXAzq8kSZKKYfErSZKkYjj2IEmS1I1c57cWdn4lSZJUDItfSZIkFcOxB0mSpG7l1EPb2fmVJElSMSx+JUmSVAzHHiRJkrqVN7loOzu/kiRJKoadX0mSpG5l47ft7PxKkiSpGBa/kiRJKoZjD5IkSd3I2xvXws6vJEmSimHxK6lWEbF9RHw1Ih6PiDlb8D2nRMS8dsbWlIj4rYj4QdNxSFKJLH4lARARfxARCyLiVxHxYET8b0TMaMNXnwjsBrw4M0/a3C/JzP/OzNe0IZ5aRURGxEuG2yczr8/M/ToVk6StWHTBo8dY/EoiIt4HfAL4f7QK1T2BTwEz2/D1ewH3ZeaqNnzXVi8ivNZCkhpk8SsVLiJeBHwYeG9mfjkzn8jMZzLzq5n5f6p9to2IT0TEsurxiYjYtnrvlRGxJCL+LCIeqrrGb6/e+zvgb4A3VR3l0yLibyPiC0OOv3fVLR1VvT41In4cEb+MiPsj4pQh228Y8rlXRMSt1TjFrRHxiiHvXRsRfx8RN1bfMy8ixq4n/9Xx//mQ+N8QEcdFxH0R8UhE/NWQ/Q+LiJsi4rFq33+JiNHVe9dVu323yvdNQ77/LyLiZ8B/rt5WfWZSdYyp1esJEbE8Il65JedVUo+IaP7RYyx+JR0BbAdcNsw+fw0cDkwBDgEOAz445P3dgRcBfcBpwL9GxC6ZeTatbvLFmblTZl4wXCARsSPwSeDYzHwB8ArgjnXsNwb4WrXvi4GPAV+LiBcP2e0PgLcDuwKjgfcPc+jdaf0M+mgV6/8BvBWYBvwW8KGImFjtOwj8KTCW1s/uaOA9AJl5VLXPIVW+Fw/5/jG0uuCnDz1wZv4I+AvgCxGxA/CfwOcy89ph4pUkbSaLX0kvBlZsYCzhFODDmflQZi4H/g5425D3n6nefyYzrwR+BWzuTOuzwIERsX1mPpiZd69jn+OBH2bmf2Xmqsy8CPg+cMKQff4zM+/LzKeAS2gV7uvzDPAPmfkM8EVahe15mfnL6vj30Cr6ycyFmXlzddzFwL8Dv70ROZ2dmb+u4nmezPwPYBEwHxhP6x8bkqQaWPxKehgYu4FZ1AnAT4a8/km1bc13rFU8PwnstKmBZOYTwJuAdwMPRsTXImL/jYhndUx9Q17/bBPieTgzB6vnq4vTnw95/6nVn4+Il0bE3Ij4WUT8glZne50jFUMsz8ynN7DPfwAHAv+cmb/ewL6SSjGiCx49pgdTkrSJbgJ+DbxhmH2W0fqV/Wp7Vts2xxPADkNe7z70zcz8ema+mlYH9Pu0isINxbM6pqWbGdOm+DStuPbNzBcCf8WGr4fO4d6MiJ1oXXB4AfC31ViHJKkGFr9S4TLzcVpzrv9aXei1Q0RsExHHRsQ/VbtdBHwwIsZVF479DfCF9X3nBtwBHBURe1YX2/3l6jciYreImFnN/v6a1vjEs+v4jiuBl1bLs42KiDcBk4G5mxnTpngB8AvgV1VX+g/Xev/nwD6b+J3nAQsy8520Zpn/bYujlCStk8WvJDLzo8D7aF3Ethx4APgj4CvVLv8XWAB8D7gTuK3atjnHuhq4uPquhTy/YB1RxbEMeITWLO3axSWZ+TDwOuDPaI1t/DnwusxcsTkxbaL307qY7pe0utIXr/X+3wKfq1aDOHlDXxYRM4FjeC7P9wFTV69yIalgTa/y0KOrPUTmsL+NkyRJUgNi3PbJzL2bDgMu+P7CzBxoOox2sfMrSZKkYninIUmSpG7Ve1MHjbPzK0mSpGLY+ZUkSepWPXjBWdPs/EqSJKkYXdX5HTt2bO61955NhyFJkgr2k8U/ZcWKFbZce1RXFb977b0nN86/oekwJElSwY6cPqPpEJ7j7+jbzh+pJEmSimHxK0mSpGJ01diDJEmSKoGrPdTAzq8kSZKKYedXkiSpW9n4bTs7v5IkSSqGxa8kSZKK4diDJElStxrh3EO72fmVJElSMSx+JUmSVAzHHiRJkrqV6/y2nZ1fSZIkFcPOryRJUjcKXOe3BnZ+JUmSVAyLX0mSJBXDsQdJkqSuFEQXXPCWTQfQZnZ+JUmSVAyLX0mSJBXDsQdJkqQu5dhD+9n5lSRJUjEsfiVJklQMxx4kSZK6VBdMPfQcO7+SJEkqhp1fSZKkLhTAiC5o/Q42HUCbFd/5nXfVPA6ePIUD9juIj5xzbtPh1K60fMGcS8n5jHe+mz3H78W0QwaaDqVjSjzP5mzO0paqtfiNiGMi4gcRsSgiPlDnsTbH4OAgZ535Pi6fexm337mQORfP4d577m06rNqUli+Ycyk5A7xt1lu5/GtfaTqMjinxPJuzOUvtUFvxGxEjgX8FjgUmA2+JiMl1HW9z3HrLAiZN2oeJ+0xk9OjRnHTyicy9Ym7TYdWmtHzBnEvJGWDGUTMYM2ZM02F0TInn2ZzNuTjRWue36UevqbPzexiwKDN/nJkrgS8CM2s83iZbtmwZ/Xv0r3nd19/H0mUPNhhRvUrLF8wZysi5RCWeZ3M2Z6kd6ix++4AHhrxeUm17nog4PSIWRMSC5ctX1BiOJEmSStf4ag+ZeT5wPsC0gakdvYPehAkTWPLAkjWvly5ZSt+E8Z0MoaNKyxfMGcrIuUQlnmdzNucS9eLYQdPq7PwuBfYY8rq/2tY1Bg6dxqJFP2Lx/YtZuXIlcy65lONPOL7psGpTWr5gzqXkXKISz7M5m7PUDnV2fm8F9o2IibSK3jcDf1Dj8TbZqFGj+Ph5H+WE42YyODjI7FNnMfmArromr61KyxfMuZScAWadMpvrv309K1Y8zKS99uVDZ3+QU98xu+mwalPieTZncy5Pb15w1rTIrG/SICKOAz4BjAQuzMx/GG7/aQNT88b5N9QWjyRJ0oYcOX0GCxfc1njVOWL3HXPbt+3fdBg8fe5tCzOzZxZRr3XmNzOvBK6s8xiSJEnSxmr8gjdJkiStm1MP7Vf87Y0lSZJUDotfSZIkFcOxB0mSpC4UuM5vHez8SpIkqRh2fiVJkrpR2Pmtg51fSZIkFcPiV5IkScVw7EGSJKlLBY49tJudX0mSJBXD4leSJEnFsPiVJEnqUhHR+GMjYtwjIr4VEfdExN0R8SfV9jERcXVE/LD6e5dqe0TEJyNiUUR8LyKmDvmu2dX+P4yI2UO2T4uIO6vPfDKqwNZ3jOFY/EqSJGlLrAL+LDMnA4cD742IycAHgGsyc1/gmuo1wLHAvtXjdODT0CpkgbOB6cBhwNlDitlPA+8a8rljqu3rO8Z6WfxKkiRps2Xmg5l5W/X8l8C9QB8wE/hctdvngDdUz2cCn8+Wm4GdI2I88Frg6sx8JDMfBa4Gjqnee2Fm3pyZCXx+re9a1zHWy9UeJEmSulSX3ONibEQsGPL6/Mw8f107RsTewMuB+cBumflg9dbPgN2q533AA0M+tqTaNtz2JevYzjDHWC+LX0mSJA1nRWYObGiniNgJ+BJwVmb+Yui8cGZmRGSNMW70MRx7kCRJ6kJBMCKaf2xUrBHb0Cp8/zszv1xt/nk1skD190PV9qXAHkM+3l9tG257/zq2D3eM9bL4lSRJ0marVl64ALg3Mz825K0rgNUrNswGLh+yfVa16sPhwOPV6MLXgddExC7VhW6vAb5evfeLiDi8Otastb5rXcdYL8ceJEmStCWOBN4G3BkRd1Tb/gr4R+CSiDgN+AlwcvXelcBxwCLgSeDtAJn5SET8PXBrtd+HM/OR6vl7gM8C2wP/Wz0Y5hjrZfErSZLUpTZmnd2mZeYNsN77MB+9jv0TeO96vutC4MJ1bF8AHLiO7Q+v6xjDcexBkiRJxbD4lSRJUjEce5AkSepGsXWMPWxt7PxKkiSpGHZ+JUmSupSN3/az8ytJkqRiWPxKkiSpGI49SJIkdaHAC97qYOdXkiRJxbD4lSRJUjEce5AkSepSjj20n51fSZIkFcPiV5IkScVw7EGSJKkrhWMPNbDzK0mSpGLY+ZUkSepG4QVvdbDzK0mSpGJY/EqSJKkYjj1IkiR1Kace2s/OryRJkoph8StJkqRiOPYgSZLUhQJXe6iDnV9JkiQVw86vJElSl7Lz2352fiVJklQMi19JkiQVw7EHSZKkLjXCsYe2s/MrSZKkYlj8SpIkqRiOPUiSJHWj8PbGdbDzK0mSpGIUX/zOu2oeB0+ewgH7HcRHzjm36XBqV1q+YM6l5PzJT/wzUw8eYNohA8w6ZTZPP/100yHVrsTzbM7mXJIgiGj+0WtqK34j4sKIeCgi7qrrGFtqcHCQs858H5fPvYzb71zInIvncO899zYdVm1KyxfMuZScly5dxqf+5dPcOP96Fn53AYODzzLn4jlNh1WrEs+zOZuz1A51dn4/CxxT4/dvsVtvWcCkSfswcZ+JjB49mpNOPpG5V8xtOqzalJYvmHMpOQOsWrWKp556qvX3k08yfvz4pkOqVYnn2ZzNWWqH2orfzLwOeKSu72+HZcuW0b9H/5rXff19LF32YIMR1au0fMGcoYyc+/omcNb7/oSXTtyfif2TeOGLXsjvvuZ3mw6rViWeZ3M25xJFF/zpNY3P/EbE6RGxICIWLF++oulwJG2FHn30UeZeMZd7F93Njx9YxBNPPMlF/31R02FJkrpQ48VvZp6fmQOZOTBu3NiOHnvChAkseWDJmtdLlyylb0Lv/qq0tHzBnKGMnL95zbfYe+LejBs3jm222YY3vPH13HzT/KbDqlWJ59mczVlqh8aL3yYNHDqNRYt+xOL7F7Ny5UrmXHIpx59wfNNh1aa0fMGcS8l5jz324Jb5t/Lkk0+SmXzrm9ey3/77NR1WrUo8z+ZsziVqeqWHXlztoeibXIwaNYqPn/dRTjhuJoODg8w+dRaTD5jcdFi1KS1fMOdScj5s+qG88ffewBGHHsmoUSM5ZMohnPaudzQdVq1KPM/mbM5SO0Rm1vPFERcBrwTGAj8Hzs7MC4b7zLSBqXnj/BtqiUeSJGljHDl9BgsX3NZ4y3Pb/hfmhD8+rOkwWPyBaxZm5kDTcbRLbZ3fzHxLXd8tSZJUgl4cO2ha0TO/kiRJKkvRM7+SJEndzMZv+9n5lSRJUjEsfiVJklQMxx4kSZK6UIQXvNXBzq8kSZKKYfErSZKkYjj2IEmS1JV68/bCTbPzK0mSpGLY+ZUkSepSdn7bz86vJEmSimHxK0mSpGI49iBJktSlnHpoPzu/kiRJKobFryRJkorh2IMkSVKXcrWH9rPzK0mSpGJY/EqSJKkYjj1IkiR1oQjHHupg51eSJEnFsPMrSZLUpez8tp+dX0mSJBXD4leSJEnFcOxBkiSpSzn10H52fiVJklQMi19JkiQVw7EHSZKkrhSu9lADO7+SJEkqhp1fSZKkLmXnt/3s/EqSJKkYFr+SJEkqhmMPkiRJXSjCsYc62PmVJElSMSx+JUmSVAzHHiRJkrqUUw/tZ+dXkiRJxbDzK0mS1KW84K397PxKkiSpGBa/kiRJKoZjD5IkSd3KsYe2s/MrSZKkYlj8SpIkqRiOPUiSJHWlcLWHGtj5lSRJUjEsfiVJklQMxx4kSZK6UbjYQx3s/EqSJKkYdn4lSZK6UODtjetg51eSJEnFsPiVJElSMRx7kCRJ6lKOPbRf8Z3feVfN4+DJUzhgv4P4yDnnNh1O7Xo538cee4y3nHwKhxzwcqYcOJWbb5oPwKf+5dMccsDLmXrwAH/1F38NwEX/80WmTzt8zWOHbXbiu3d8t8nw26qXz/P6mLM59ypzLiNndU5kZj1fHLEH8HlgNyCB8zPzvOE+M21gat44/4Za4lmXwcFBDnrZIXztqq/S19/HjMN/i8994bO8bPLLOhZDJ/V6vu98+7s4csaRvP20U1m5ciVPPvkk3739u5zz//0Tl331y2y77bY89NBD7Lrrrs/73F133sXJv/9m7rnvroYib69eP8/rYs7m3KvMuZmcj5w+g4ULbmu85brj3rvkyz70qqbDYOE7v7wwMweajqNd6uz8rgL+LDMnA4cD742IyTUeb5PdessCJk3ah4n7TGT06NGcdPKJzL1ibtNh1aaX83388ce54fobOfUdswEYPXo0O++8M+f/+2d4/5//Gdtuuy3AbxS+AJd8cQ4nnXxiR+OtUy+f5/UxZ3PuVeZcRs7DiYjGH72mtuI3Mx/MzNuq578E7gX66jre5li2bBn9e/Sved3X38fSZQ82GFG9ejnfxfcvZuzYsZx+2hkcPnAEf3j6e3jiiSdY9MMfcuMN3+G3jvhtXv07r2XBrQt/47OXzvkSJ7/5pAairkcvn+f1MWdz7lXmXEbO6qyOzPxGxN7Ay4H563jv9IhYEBELli9f0Ylw1INWrRrkjtvv4F1nvIubF9zEDjvuwLnnfJRVq1bxyKOPct13ruX/nfMPvPUtb2PoqM8t829lhx2254ADD2gwekmS1i2i+Uevqb34jYidgC8BZ2XmL9Z+PzPPz8yBzBwYN25s3eE8z4QJE1jywJI1r5cuWUrfhPEdjaGTejnfvv4J9PX3cdj0QwF44++9kTtuv4O+vj7e8IbXExEcetgAI0aMYMWK5/6RNefiOZz8ppObCrsWvXye18eczblXmXMZOauzai1+I2IbWoXvf2fml+s81uYYOHQaixb9iMX3L2blypXMueRSjj/h+KbDqk0v57v77rvT39/PfT+4D4Brv3kt+79sf06YeQLfvvY6AH543w9ZuXIlY8e2/pH17LPP8qVLv8xJb+qdeV/o7fO8PuZszr3KnMvIWZ1V2zq/0ZqQvgC4NzM/VtdxtsSoUaP4+Hkf5YTjZjI4OMjsU2cx+YCuuiavrXo934+ddy5vn/UOVq5cyd4TJ3L+Bf/GjjvuyBnvfDfTDhlg9OjRfObC89cM799w3Q309/czcZ+JDUfeXr1+ntfFnM25V5lzGTmvV49ecNa0Opc6mwFcD9wJPFtt/qvMvHJ9n+n0UmeSJElr65qlziaOyQPPPrrpMLjl7Zf21FJntXV+M/MGoPH/4EiSJEmreXtjSZKkLhR4e+M6FH97Y0mSJJXD4leSJEnFcOxBkiSpSzn20H52fiVJklQMO7+SJEldys5v+9n5lSRJUjEsfiVJklQMxx4kSZK6UYBTD+1n51eSJEnFsPiVJElSMRx7kCRJ6lKu9tB+dn4lSZJUDDu/kiRJXSgIO781sPMrSZKkYlj8SpIkqRiOPUiSJHUpxx7az86vJEmSimHxK0mSpGI49iBJktSlnHpoPzu/kiRJKoadX0mSpG4UXvBWBzu/kiRJKobFryRJkorh2IMkSVK3cuyh7ez8SpIkqRgWv5IkSSqGYw+SJEldytUe2s/OryRJkoph8StJkqRiOPYgSZLUhQIY4dRD29n5lSRJ0haJiAsj4qGIuGvItr+NiKURcUf1OG7Ie38ZEYsi4gcR8doh24+pti2KiA8M2T4xIuZX2y+OiNHV9m2r14uq9/feUKwWv5IkSV0piGj+sZE+Cxyzju0fz8wp1eNKgIiYDLwZOKD6zKciYmREjAT+FTgWmAy8pdoX4Jzqu14CPAqcVm0/DXi02v7xar9hWfxKkiRpi2TmdcAjG7n7TOCLmfnrzLwfWAQcVj0WZeaPM3Ml8EVgZrQq8FcBl1af/xzwhiHf9bnq+aXA0bGBit3iV5IkSXX5o4j4XjUWsUu1rQ94YMg+S6pt69v+YuCxzFy11vbnfVf1/uPV/utl8StJktSNAkZENP4AxkbEgiGP0zcyg08Dk4ApwIPAR+v5QW0aV3uQJEnScFZk5sCmfigzf776eUT8BzC3erkU2GPIrv3VNtaz/WFg54gYVXV3h+6/+ruWRMQo4EXV/utl51eSJEltFxHjh7x8I7B6JYgrgDdXKzVMBPYFbgFuBfatVnYYTeuiuCsyM4FvASdWn58NXD7ku2ZXz08Evlntv152fiVJkrpQsPXc3jgiLgJeSWtEYglwNvDKiJgCJLAYOAMgM++OiEuAe4BVwHszc7D6nj8Cvg6MBC7MzLurQ/wF8MWI+L/A7cAF1fYLgP+KiEW0Lrh784ZitfiVJEnSFsnMt6xj8wXr2LZ6/38A/mEd268ErlzH9h/TWg1i7e1PAydtSqwWv5IkSV3K+dT282cqSZKkYlj8SpIkqRiOPUiSJHWpEVvJBW9bEzu/kiRJKobFryRJkorh2IMkSVIX2prW+d2a2PmVJElSMez8SpIkdaXwgrca2PmVJElSMSx+JUmSVIzii995V83j4MlTOGC/g/jIOec2HU7tSssXzLmUnM9457vZc/xeTDtkoOlQOqbE82zO5lyUaF3w1vSj19RW/EbEdhFxS0R8NyLujoi/q+tYm2twcJCzznwfl8+9jNvvXMici+dw7z33Nh1WbUrLF8y5lJwB3jbrrVz+ta80HUbHlHiezdmcpXaos/P7a+BVmXkIMAU4JiIOr/F4m+zWWxYwadI+TNxnIqNHj+akk09k7hVzmw6rNqXlC+ZcSs4AM46awZgxY5oOo2NKPM/mbM5SO9RW/GbLr6qX21SPrOt4m2PZsmX079G/5nVffx9Llz3YYET1Ki1fMGcoI+cSlXiezdmcSxO0CrWmH72m1pwiYmRE3AE8BFydmfPXsc/pEbEgIhYsX76iznAkSZJUuFqL38wczMwpQD9wWEQcuI59zs/MgcwcGDdubJ3h/IYJEyaw5IEla14vXbKUvgnjOxpDJ5WWL5gzlJFziUo8z+ZszlI7dKSbnZmPAd8CjunE8TbWwKHTWLToRyy+fzErV65kziWXcvwJxzcdVm1KyxfMuZScS1TieTZncy7RiIjGH72mtju8RcQ44JnMfCwitgdeDZxT1/E2x6hRo/j4eR/lhONmMjg4yOxTZzH5gMlNh1Wb0vIFcy4lZ4BZp8zm+m9fz4oVDzNpr3350Nkf5NR3zG46rNqUeJ7N2ZyldojMeq5Bi4iDgc8BI2l1mC/JzA8P95lpA1Pzxvk31BKPJEnSxjhy+gwWLrit8ZbnLvuOy6M/8XtNh8GXXnf+wszsmUXUa+v8Zub3gJfX9f2SJEnSpurFFSwkSZKkdaqt8ytJkqTNF9CTF5w1zc6vJEmSimHxK0mSpGI49iBJktSlHHpoPzu/kiRJKoadX0mSpK7Um3dYa5qdX0mSJBXD4leSJEnFcOxBkiSpC0W4zm8d7PxKkiSpGBa/kiRJKoZjD5IkSV0qHHtoOzu/kiRJKobFryRJkorh2IMkSVKXcrWH9rPzK0mSpGLY+ZUkSepCUT3UXnZ+JUmSVAyLX0mSJBXDsQdJkqQu5QVv7WfnV5IkScWw+JUkSVIxHHuQJEnqSuHYQw0sfiXVYvtjXtp0CB331FX3NR2CJGkDLH4lSZK6UASEnd+2c+ZXkiRJxbD4lSRJUjEce5AkSepSXvDWfnZ+JUmSVAyLX0mSJBXDsQdJkqQu5dBD+9n5lSRJUjHW2/mNiH8Gcn3vZ+aZtUQkSZIkAi94q8NwYw8LOhaFJEmS1AHrLX4z83NDX0fEDpn5ZP0hSZIkSfXY4MxvRBwREfcA369eHxIRn6o9MkmSpMKNiGj80Ws25oK3TwCvBR4GyMzvAkfVGJMkSZJUi41a7SEzH1hr02ANsUiSJEm12ph1fh+IiFcAGRHbAH8C3FtvWJIkSaULogfHDpq2MZ3fdwPvBfqAZcCU6rUkSZK0Vdlg5zczVwCndCAWSZIkqVYbs9rDPhHx1YhYHhEPRcTlEbFPJ4KTJEkqVdAq1Jp+9JqNyel/gEuA8cAEYA5wUZ1BSZIkSXXYmAvedsjM/xry+gsR8X/qCkiSJElA4AVvNVhv8RsRY6qn/xsRHwC+CCTwJuDKDsQmSZIktdVwnd+FtIrd1f/kOGPIewn8ZV1BSZIkSXVYb/GbmRM7GYgkSZKerxdvL9y0jZn5JSIOBCYD263elpmfrysoSZIkqQ4bLH4j4mzglbSK3yuBY4EbAItfSZIkbVU2ZqmzE4GjgZ9l5tuBQ4AX1RpVB827ah4HT57CAfsdxEfOObfpcGpXWr4AZ7zz3ew5fi+mHTLQdCgds9Wd56dXwcLlcNPPW4+f/ur57//kl/CNpbBysPX6oafg5p/DzQ/B/IfgsV8/t+/tK+DaZXDHiud/RyYsehy+8zP4zpBjZMIPHoMbf9b6zl+srC3NdtvqznMbmLM5lyRojT00/eg1G1P8PpWZzwKrIuKFwEPAHht7gIgYGRG3R8TczQ2yLoODg5x15vu4fO5l3H7nQuZcPId777m36bBqU1q+q71t1lu5/GtfaTqMjtkqz3ME7PsiOGI3OHQcLPkV/OqZ1ntPr4KHfw3bjXxu/zHbwvRd4fBdYfIucM9jz723105wwC6/eYwHn4SnB1vHeMVusPv2re0P/xqeXNXa9rJd4PuP/eZnu9BWeZ63kDmbs9QOG1P8LoiInYH/oLUCxG3ATZtwjD8BuvI/tbfesoBJk/Zh4j4TGT16NCedfCJzr+i6Gr1tSst3tRlHzWDMmDEb3rFHbJXneduR8MLRreejRsAO28Cvqy7vfY+3CuOhRo1oFcwAg88+/70x28HIdfxP25InYJ8XPve50VUxvfwpGL9Da/uLRsOqfO7YXWyrPM9byJzNuUQR0fij12yw+M3M92TmY5n5b8CrgdnV+MMGRUQ/cDzwmS0Lsx7Lli2jf4/+Na/7+vtYuuzBBiOqV2n5lmqrP89PrYJfPtMqRB96qlUYv2Cb39zvoada4wt3PAyTd9647/35U60xidtXtLq90Cp0h3aVtx25VRS/W/153gzmbM5SOwx3k4upw72XmbdtxPd/Avhz4AWbHpqk4qx6Fr73COz3otaw2+JfwtSx69531+1bj0d/DT/+JUzddvjvfpbWP/en79oqnO95FAbGtTkBSVK3G261h48O814CrxruiyPidcBDmbkwIl45zH6nA6cD7LHnRo8St8WECRNY8sCSNa+XLllK34TxHY2hk0rLt1Rb7Xl+NluF7+5VUfurZ+CpwdZFbdDqxs5fDoeNa3VnV9tl21Yhu3LwuVGGddl2ZOt7AcZtB3c/+tz2p4d0en89+Pzv71Jb7XneAuZszuUJRtB7YwdNW+/YQ2b+zjCPYQvfypHA6yNiMa1bI78qIr6wjuOcn5kDmTkwbtx6Ojw1GTh0GosW/YjF9y9m5cqVzLnkUo4/4fiOxtBJpeVbqq3yPGe2CtgdR8Fe1S+KdtoGfns8zNi99dh2JEyvCt8nV7U+A63VGZ5N2GYDU1zjtoNHqlUhHl3ZOhbAuO1bF8NlwuMrYVRsFcXvVnmet5A5m7PUDht1k4vNkZl/SXUL5Krz+/7MfGtdx9sco0aN4uPnfZQTjpvJ4OAgs0+dxeQDJjcdVm1Ky3e1WafM5vpvX8+KFQ8zaa99+dDZH+TUd8xuOqzabJXn+fGV8LOnYKdRz3V6X/JCGLvduvd/6KlWwRoBI4GDxjx3IduC5fDEqtaFcNc/2FoN4sXbwd4vgLseaS1xNipaKzsAvHhbWPF0a354RKx7pYgutFWe5y1kzuYstUPk6u5JnQd5rvh93XD7TRuYmjfOv6H2eCTVb/tjXtp0CB331FX3NR2CpDY4cvoMFi64rfF5g933H5+zLzy16TD4pyP/cWFm9sxi+bV1fofKzGuBaztxLEmSJGl9NrjUWbS8NSL+pnq9Z0QcVn9okiRJUnttTOf3U7QWCXoV8GHgl8CXgENrjEuSJKloEfTk7YWbtjHF7/TMnBoRtwNk5qMRMbrmuCRJkqS225ji95mIGElrbV8iYhytTrAkSZJqFK7z23YbnPkFPglcBuwaEf8A3AD8v1qjkiRJkmqwwc5vZv53RCwEjqZ1w9E3ZOa9tUcmSZIktdkGi9+I2BN4Evjq0G2Z+dM6A5MkSSpdeMFb223MzO/XaM37BrAdMBH4AXBAjXFJkiRJbbcxYw8HDX0dEVOB99QWkSRJklSTTb7DW2beFhHT6whGkiRJLUG4zm8NNmbm931DXo4ApgLLaotIkiRJqsnGdH5fMOT5KlozwF+qJxxJkiStFhu1Kq02xbDFb3Vzixdk5vs7FI8kSZJUm/X+cyIiRmXmIHBkB+ORJEmSajNc5/cWWvO9d0TEFcAc4InVb2bml2uOTZIkqWhe8NZ+GzPzux3wMPAqnlvvNwGLX0mSJG1Vhit+d61WeriL54re1bLWqCRJkqQaDFf8jgR24vlF72oWv5IkSTXz9sbtN1zx+2BmfrhjkUiSJEk1G6749Z8akiRJDYnqj9pruJWTj+5YFJIkSVIHrLf4zcxHOhmIJEmSVLeNWepMkiRJnRau81sHbxgtSZKkYlj8SpIkqRiOPUiqxVNX3dd0CJK01XOd3/az8ytJkqRiWPxKkiSpGI49SJIkdaEARtinbDt/opIkSSqGnV9JkqSuFF7wVgM7v5IkSSqGxa8kSZKK4diDJElSl3Lsof3s/EqSJKkYFr+SJEkqhmMPkiRJXWoEjj20m51fSZIkFcPOryRJUhcKvOCtDnZ+JUmSVAyLX0mSJBXDsQdJkqRuFDDCsYe2s/MrSZKkYlj8SpIkqRiOPUiSJHWlIFznt+3s/EqSJKkYdn4lSZK6UAAjwj5lu/kTlSRJUjEsfiVJklQMxx4kSZK6lLc3bj87v5IkSSpG8cXvvKvmcfDkKRyw30F85Jxzmw6ndqXlC+bcqzmf8c53s+f4vZh2yMCabW99yyymTzuc6dMOZ79JL2P6tMMbjLB+JZzntZmzOUtbqtbiNyIWR8SdEXFHRCyo81ibY3BwkLPOfB+Xz72M2+9cyJyL53DvPfc2HVZtSssXzLmXc37brLdy+de+8rxtX7jo88xfeDPzF97MG944k5lvmNlMcB1QynkeypzNuUTRBX96TSc6v7+TmVMyc2DDu3bWrbcsYNKkfZi4z0RGjx7NSSefyNwr5jYdVm1KyxfMuZdznnHUDMaMGbPO9zKTL136ZU5+80kdjqpzSjnPQ5mzOat7RcSFEfFQRNw1ZNuYiLg6In5Y/b1LtT0i4pMRsSgivhcRU4d8Zna1/w8jYvaQ7dOqhuqi6rMx3DGGU/TYw7Jly+jfo3/N677+PpYue7DBiOpVWr5gzlBGzmu78fob2W23XXnJvi9pOpTalHiezdmc1dU+Cxyz1rYPANdk5r7ANdVrgGOBfavH6cCnoVXIAmcD04HDgLOHFLOfBt415HPHbOAY61V38ZvAvIhYGBGnr2uHiDg9IhZExILly1fUHI6kElxy8RxOelPvdn0llSIYEc0/NkZmXgc8stbmmcDnquefA94wZPvns+VmYOeIGA+8Frg6Mx/JzEeBq4FjqvdemJk3Z2YCn1/ru9Z1jPWqe6mzGZm5NCJ2Ba6OiO9XP5w1MvN84HyAaQNTs+Z4nmfChAkseWDJmtdLlyylb8L4TobQUaXlC+YMZeQ81KpVq7j8ssu58ZYbmw6lViWeZ3M2ZzVm7FrXbp1f1W8bsltmrm7b/wzYrXreBzwwZL8l1bbhti9Zx/bhjrFetXZ+M3Np9fdDwGW0WthdY+DQaSxa9CMW37+YlStXMueSSzn+hOObDqs2peUL5lxKzkN98xvf5KX77Ud/f9+Gd96KlXiezdmcSxN0zQVvKzJzYMhjYwrf56k6trU2OTf2GLV1fiNiR2BEZv6yev4a4MN1HW9zjBo1io+f91FOOG4mg4ODzD51FpMPmNx0WLUpLV8w517OedYps7n+29ezYsXDTNprXz509gc59R2zmXPJpT19odtqpZznoczZnLXV+XlEjM/MB6vRhYeq7UuBPYbs119tWwq8cq3t11bb+9ex/3DHWK9oFcntFxH70Or2QqvI/p/M/IfhPjNtYGreOP+GWuKRJEnaGEdOn8HCBbc1vsbXXgfsmX9x8fubDoP3HvQnCzdm1a6I2BuYm5kHVq8/Ajycmf8YER8AxmTmn0fE8cAfAcfRurjtk5l5WHXB20Jg9eoPtwHTMvORiLgFOBOYD1wJ/HNmXrm+YwwXZ22d38z8MXBIXd8vSZLU6zb2grOmRcRFtLq2YyNiCa1VG/4RuCQiTgN+Apxc7X4lrcJ3EfAk8HaAqsj9e+DWar8PZ+bqi+jeQ2tFie2B/60eDHOM9ar7gjdJkiT1uMx8y3reOnod+ybw3vV8z4XAhevYvgA4cB3bH17XMYZT9Dq/kiRJKoudX0mSpG4UEGGfst38iUqSJKkYdn4lSZK60pp1dtVGdn4lSZJUDItfSZIkFcOxB0mSpC4UbD3r/G5N7PxKkiSpGBa/kiRJKoZjD5IkSV0qHHtoOzu/kiRJKobFryRJkorh2IMkSVKXGuFNLtrOzq8kSZKKYedXkiSpCwVe8FYHO7+SJEkqhsWvJEmSiuHYgyRJUlcKIuxTtps/UUmSJBXD4leSJEnFcOxBkiSpS7nOb/vZ+ZUkSVIx7PxKkiR1oQjX+a2DnV9JkiQVw+JXkiRJxXDsQZIkqUuFF7y1nZ1fSZIkFcPiV5IkScVw7EGSJKkrhas91MDOryRJkoph51eSJKlLeYe39rPzK0mSpGJY/EqSJKkYjj1IkiR1oQAi7FO2mz9RSZIkFcPiV5IkScVw7EGSJKkrhbc3roGdX0mSJBXD4leSJEnFcOxBkiSpS3l74/az8ytJkqRi2PmVJEnqUl7w1n52fiVJklQMi19JkiQVw7EHSZKkLuUFb+1n51eSJEnFsPiVJElSMRx7kCRJ6kIBjHC1h7az8ytJkqRi2PmVJEnqRhFe8FaD4ju/866ax8GTp3DAfgfxkXPObTqc2pWWL5hzKTmf8c53s+f4vZh2yEDToXRMief5scce4y0nn8IhB7ycKQdO5eab5jcdUu1KPM8l5qzOqbX4jYidI+LSiPh+RNwbEUfUebxNNTg4yFlnvo/L517G7XcuZM7Fc7j3nnubDqs2peUL5lxKzgBvm/VWLv/aV5oOo2NKPc/v/9P/w2te+2q+e/ft3HLbzez/sv2aDqlWJZ7nEnNWZ9Xd+T0PuCoz9wcOAbrqP7233rKASZP2YeI+Exk9ejQnnXwic6+Y23RYtSktXzDnUnIGmHHUDMaMGdN0GB1T4nl+/PHHueH6Gzn1HbMBGD16NDvvvHOzQdWsxPNcYs7DCUY0/ug1tWUUES8CjgIuAMjMlZn5WF3H2xzLli2jf4/+Na/7+vtYuuzBBiOqV2n5gjlDGTmXqMTzvPj+xYwdO5bTTzuDwweO4A9Pfw9PPPFE02HVqsTzXGLO6qw6y/mJwHLgPyPi9oj4TETsuPZOEXF6RCyIiAXLl6+oMRxJ0tZs1apB7rj9Dt51xru4ecFN7LDjDpx7zkebDkvSVqbO4ncUMBX4dGa+HHgC+MDaO2Xm+Zk5kJkD48aNrTGc3zRhwgSWPLBkzeulS5bSN2F8R2PopNLyBXOGMnIuUYnnua9/An39fRw2/VAA3vh7b+SO2+9oNqialXieS8x5OFGt+NDko9fUWfwuAZZk5upLcS+lVQx3jYFDp7Fo0Y9YfP9iVq5cyZxLLuX4E45vOqzalJYvmHMpOZeoxPO8++6709/fz30/uA+Aa795Lfu/bP+Go6pXiee5xJzVWbWt85uZP4uIByJiv8z8AXA0cE9dx9sco0aN4uPnfZQTjpvJ4OAgs0+dxeQDJjcdVm1KyxfMuZScAWadMpvrv309K1Y8zKS99uVDZ39wzYVRvajU8/yx887l7bPewcqVK9l74kTOv+Dfmg6pViWe5xJzVmdFZtb35RFTgM8Ao4EfA2/PzEfXt/+0gal54/wbaotHkiRpQ46cPoOFC25r/Pf9+x48KT/21X9qOgxev/eJCzOzZxZRr/UOb5l5B9AzPyxJkiRt3by9sSRJUlcKRvTgBWdN672ViyVJkqT1sPiVJElSMRx7kCRJ6lKBYw/tZudXkiRJxbD4lSRJUjEce5AkSepSvXh74abZ+ZUkSVIx7PxKkiR1oQDCPmXb+ROVJElSMSx+JUmSVAzHHiRJkrpSeMFbDez8SpIkqRgWv5IkSSqGYw+SJEldaoS3N247O7+SJEkqhp1fSZKkbhTe4a0Odn4lSZJUDItfSZIkFcOxB0mSpC7Uur2xYw/tZudXkiRJxbD4lSRJUjEce5AkSepSrvbQfnZ+JUmSVAyLX0mSJBXDsQdJkjZBZjYdQsf5q/emBGGfsu38iUqSJKkYdn4lSZK61Ai77m1n51eSJEnFsPiVJElSMRx7kCRJ6kLe3rgedn4lSZJUDItfSZIkFcOxB0mSpC7lGsvtZ+dXkiRJxbDzK0mS1JXCC95qYOdXkiRJxbD4lSRJUjEce5AkSepSXvDWfnZ+JUmSVAyLX0mSJBXDsQdJkqQuFMAI+5Rt509UkiRJxbDzK0mS1I3CC97qYOdXkiRJxbD4lSRJUjEce5AkSepK3t64DnZ+JUmSVAyLX0mSJBXDsQdJkqQu5WoP7WfnV5IkScUovvidd9U8Dp48hQP2O4iPnHNu0+HUrrR8wZxLyPnpp59mxuFHcdjU6Uw9eIC//9v/23RIHVHaeYbeyvmMd/4he03Ym4Eph67Z9rY/mMX0aUcwfdoR7P+SyUyfdsSa9+783l28csarmHbIAIdOOYynn34agLM/9LfsO3E/xu28W8dzqEsvnWd1n9qK34jYLyLuGPL4RUScVdfxNsfg4CBnnfk+Lp97GbffuZA5F8/h3nvubTqs2pSWL5hzKTlvu+22XPWNK7nltvnMX3gT875+NfNvvqXpsGpV4nnutZzfNvsUvjL3K8/b9l//83nmL7yJ+Qtv4g1vnMnMN74egFWrVnHa7NP45L+ex8LvLuCqa/6XbbbZBoDjjz+O677z7U6HX5teO89bKrrgT6+prfjNzB9k5pTMnAJMA54ELqvreJvj1lsWMGnSPkzcZyKjR4/mpJNPZO4Vc5sOqzal5QvmXErOEcFOO+0EwDPPPMOqVc/0/Jxciee513Ke8VszGDNml3W+l5l86dIvc/KbTgLgG1dfw4EHHcjBhxwEwItf/GJGjhwJwGGHH8b48bt3JugO6LXzrO7TqbGHo4EfZeZPOnS8jbJs2TL69+hf87qvv4+lyx5sMKJ6lZYvmDOUkTO0ukXTpx3OnuP35lVHv4rDph+64Q9txUo8zyXlfOMNN7Lrrrvykn1fAsCi+xYREbz+uJkcceiRfOzcjzccYX1KOs8bEtj5rUOnit83Axet642IOD0iFkTEguXLV3QoHEm9ZuTIkcxfeDOLfnIfC25dyN133d10SNJmu+SLczj5zSeteb1qcBXf+c5NXPj5C7jm21dzxVe+yre++a0GI5S2XrUXvxExGng9MGdd72fm+Zk5kJkD48aNrTuc55kwYQJLHliy5vXSJUvpmzC+ozF0Umn5gjlDGTkPtfPOO/PbrzyKeV+/uulQalXieS4l51WrVnHFV67g90/6/TXb+vomMGPGkYwdO5YddtiB1x77Gu64/bsNRlmfUs6zmtOJzu+xwG2Z+fMOHGuTDBw6jUWLfsTi+xezcuVK5lxyKcefcHzTYdWmtHzBnEvJefny5Tz22GMAPPXUU1zzjW+y3377NRtUzUo8z6Xk/M1rvsVL93sp/f19a7b97mt+l7vuupsnn3ySVatWccN1N7D/y/ZvMMr6lHKeN1pE848e04mbXLyF9Yw8NG3UqFF8/LyPcsJxMxkcHGT2qbOYfMDkpsOqTWn5gjmXkvPPHvwZ73rH6QwODvLss8/y+yf+Pse97timw6pViee513Ke/dZTue7b1/Pwiod5yd4v5YN/89ec+o7ZXHrxpZz0ppOet+8uu+zCmWf9Mb91xFFEBK895rUce9wxAPz1Bz7IxV+8hCeffJKX7P1STn3HbD74N3/dREpt0WvnWd0nMrO+L4/YEfgpsE9mPr6h/acNTM0b599QWzySJG2pOv9/s1v1+uopazty+gwWLrit8aQnT9k/P/+NC5sOg0PHHbkwMweajqNdau38ZuYTwIvrPIYkSVJv6s3VFppW/B3eJEmSVI5OzPxKkiRpM5Q2ctIJdn4lSZJUDItfSZIkFcOxB0mSpC7lBW/tZ+dXkiRJxbD4lSRJUjEce5AkSepSjj20n51fSZIkFcPiV5IkScVw7EGSJKkLBd7kog52fiVJklQMi19JkqSuFF3xZ6MijVgcEXdGxB0RsaDaNiYiro6IH1Z/71Jtj4j4ZEQsiojvRcTUId8zu9r/hxExe8j2adX3L6o+u9ktcYtfSZIktcPvZOaUzByoXn8AuCYz9wWuqV4DHAvsWz1OBz4NrWIZOBuYDhwGnL26YK72edeQzx2zuUFa/EqSJKkOM4HPVc8/B7xhyPbPZ8vNwM4RMR54LXB1Zj6SmY8CVwPHVO+9MDNvzswEPj/kuzaZF7xJkiR1qS5Z53fs6lGGyvmZef5a+yQwLyIS+Pfq/d0y88Hq/Z8Bu1XP+4AHhnx2SbVtuO1L1rF9s1j8SpIkaTgrhowyrM+MzFwaEbsCV0fE94e+mZlZFcaNc+xBkiRJWyQzl1Z/PwRcRmtm9+fVyALV3w9Vuy8F9hjy8f5q23Db+9exfbNY/EqSJHWjaK3z2/Rjg2FG7BgRL1j9HHgNcBdwBbB6xYbZwOXV8yuAWdWqD4cDj1fjEV8HXhMRu1QXur0G+Hr13i8i4vBqlYdZQ75rkzn2IEmSpC2xG3BZVSiPAv4nM6+KiFuBSyLiNOAnwMnV/lcCxwGLgCeBtwNk5iMR8ffArdV+H87MR6rn7wE+C2wP/G/12CwWv5IkSV2qSy54G1Zm/hg4ZB3bHwaOXsf2BN67nu+6ELhwHdsXAAducbA49iBJkqSCWPxKkiSpGI49SJIkdaGAjbrgTJvGzq8kSZKKYedXkqRNYCdO2rpZ/EqSJHWl2CpWe9jaOPYgSZKkYtj5lSRJ6lJ2ftvPzq8kSZKKYfErSZKkYjj2IEmS1KVcXaT97PxKkiSpGBa/kiRJKoZjD5IkSV3K1R7az86vJEmSimHxK0mSpGI49iBJktSFAsce6mDnV5IkScWw8ytJktSVwnV+a2DnV5IkScWw+JUkSVIxHHuQJEnqWo49tJudX0mSJBXD4leSJEnFcOxBkiSpGwWu9lADO7+SJEkqhp1fSZKkLuUd3trPzq8kSZKKYfErSZKkYjj2IEmS1KUce2i/4ju/866ax8GTp3DAfgfxkXPObTqc2pWWL5hzKTmf8c53s+f4vZh2yEDToXRMiefZnM1Z2lK1Fr8R8acRcXdE3BURF0XEdnUeb1MNDg5y1pnv4/K5l3H7nQuZc/Ec7r3n3qbDqk1p+YI5l5IzwNtmvZXLv/aVpsPomBLPszmbs9QOtRW/EdEHnAkMZOaBwEjgzXUdb3PcessCJk3ah4n7TGT06NGcdPKJzL1ibtNh1aa0fMGcS8kZYMZRMxgzZkzTYXRMiefZnM25NEEQ0fyj19Q99jAK2D4iRgE7AMtqPt4mWbZsGf179K953dffx9JlDzYYUb1KyxfMGcrIuUQlnmdzNmepHWorfjNzKXAu8FPgQeDxzJy39n4RcXpELIiIBcuXr6grHEmSJKnWsYddgJnARGACsGNEvHXt/TLz/MwcyMyBcePG1hXOOk2YMIElDyxZ83rpkqX0TRjf0Rg6qbR8wZyhjJxLVOJ5NmdzLlF0wZ9eU+fYw+8C92fm8sx8Bvgy8Ioaj7fJBg6dxqJFP2Lx/YtZuXIlcy65lONPOL7psGpTWr5gzqXkXKISz7M5m7PUDnWu8/tT4PCI2AF4CjgaWFDj8TbZqFGj+Ph5H+WE42YyODjI7FNnMfmAyU2HVZvS8gVzLiVngFmnzOb6b1/PihUPM2mvffnQ2R/k1HfMbjqs2pR4ns3ZnEvUi53XpkVm1vflEX8HvAlYBdwOvDMzf72+/acNTM0b599QWzySJEkbcuT0GSxccFvjVefBUw/Kudd9pekw2OsFL1mYmT2ziHqtd3jLzLOBs+s8hiRJkrSxvL2xJElSl+rFdXabVvztjSVJklQOi19JkiQVw7EHSZKkLuVqD+1n51eSJEnFsPMrSZLUhYLwgrca2PmVJElSMSx+JUmSVAzHHiRJkrqUF7y1n51fSZIkFcPiV5IkScVw7EGSJKlrOfbQbnZ+JUmSVAw7v5IkSV3Kvm/72fmVJElSMSx+JUmSVAzHHiRJkrqUtzduPzu/kiRJKobFryRJkorh2IMkSVLXcuyh3ez8SpIkqRgWv5IkSSqGYw+SJEldyqGH9rPzK0mSpGLY+ZUkSepKgb3f9uuq4ve2hbev2H7Ujj9p4NBjgRUNHLdJ5lwGc+59peUL5lyKJnPeq6HjqgO6qvjNzHFNHDciFmTmQBPHboo5l8Gce19p+YI5l6LEnNUZXVX8SpIkqSXC2xvXwQveJEmSVAyL35bzmw6gAeZcBnPufaXlC+ZcihJzVgdEZjYdgyRJktYyZdohOe/G/206DHbbvm9hL81f2/mVJElSMYovfiPimIj4QUQsiogPNB1P3SLiwoh4KCLuajqWToiIPSLiWxFxT0TcHRF/0nRMdYuI7SLiloj4bpXz3zUdU6dExMiIuD0i5jYdSydExOKIuDMi7oiIBU3H0wkRsXNEXBoR34+IeyPiiKZjqlNE7Fed39WPX0TEWU3HVbeI+NPqf7/uioiLImK7pmNqSnTBn15TdPEbESOBfwWOBSYDb4mIyc1GVbvPAsc0HUQHrQL+LDMnA4cD7y3gHP8aeFVmHgJMAY6JiMObDalj/gS4t+kgOux3MnNKL/1KcgPOA67KzP2BQ+jx852ZP6jO7xRgGvAkcFmzUdUrIvqAM4GBzDwQGAm8udmo1EuKLn6Bw4BFmfnjzFwJfBGY2XBMtcrM64BHmo6jUzLzwcy8rXr+S1r/R9nXbFT1ypZfVS+3qR49P9wfEf3A8cBnmo5F9YiIFwFHARcAZObKzHys0aA662jgR5nZxM2gOm0UsH1EjAJ2AJY1HI96SOnFbx/wwJDXS+jxwqhkEbE38HJgfsOh1K769f8dwEPA1ZnZ8zkDnwD+HHi24Tg6KYF5EbEwIk5vOpgOmAgsB/6zGm/5TETs2HRQHfRm4KKmg6hbZi4FzgV+CjwIPJ6Z85qNqjlNjzw49iBtpSJiJ+BLwFmZ+Yum46lbZg5WvybtBw6LiAMbDqlWEfE64KHMXNh0LB02IzOn0hrdem9EHNV0QDUbBUwFPp2ZLweeAHr+Wg2AiBgNvB6Y03QsdYuIXWj9FnYiMAHYMSLe2mxU6iWlF79LgT2GvO6vtqmHRMQ2tArf/87MLzcdTydVvxL+Fr0/530k8PqIWExrfOlVEfGFZkOqX9UhIzMfojUHelizEdVuCbBkyG8yLqVVDJfgWOC2zPx504F0wO8C92fm8sx8Bvgy8IqGY1IPKb34vRXYNyImVv+qfjNwRcMxqY2idV/IC4B7M/NjTcfTCRExLiJ2rp5vD7wa+H6jQdUsM/8yM/szc29a/z3+Zmb2dKcoInaMiBesfg68BujpVVwy82fAAxGxX7XpaOCeBkPqpLdQwMhD5afA4RGxQ/W/4UfT4xc2qrNGNR1AkzJzVUT8EfB1WleTXpiZdzccVq0i4iLglcDYiFgCnJ2ZFzQbVa2OBN4G3FnNwAL8VWZe2VxItRsPfK5azWQEcElmFrH0V2F2Ay5r1QaMAv4nM69qNqSO+GPgv6uGxY+BtzccT+2qf9y8Gjij6Vg6ITPnR8SlwG20Vuy5He/2pjbyDm+SJEldaMq0Q/Ka7zR/rd/Y7Xb3Dm+SJEnS1sjiV5IkScWw+JUkSVIxLH4lSZJUDItfSZIkFcPiV9Jmi4jBiLgjIu6KiDkRscMWfNdnI+LE6vlnImLyMPu+MiI2edH7iFgcEWM3dvta+/xqE4/1txHx/k2NUZKe0/SNjb29sSSt7anMnJKZBwIrgXcPfTMiNmst8cx8Z2YOd/OCV+IdnyRJm8HiV1K7XA+8pOrKXh8RVwD3RMTIiPhIRNwaEd+LiDOgdfe9iPiXiPhBRHwD2HX1F0XEtRExUD0/JiJui4jvRsQ1EbE3rSL7T6uu829Vd7X7UnWMWyPiyOqzL46IeRFxd0R8BjbcwoiIr0TEwuozp6/13ser7ddExLhq26SIuKr6zPURsX9bfpqSpFoUfYc3Se1RdXiPBVbfYWwqcGBm3l8VkI9n5qERsS1wY0TMA14O7AdMpnW3snuAC9f63nHAfwBHVd81JjMfiYh/A36VmedW+/0P8PHMvCEi9qR118aXAWcDN2TmhyPieOC0jUjnHdUxtgdujYgvZebDwI7Agsz804j4m+q7/4jWnafenZk/jIjpwKeAV23Gj1GS1qH3xg6aZvEraUtsP+S20dcDF9AaR7glM++vtr8GOHj1PC/wImBf4CjgoswcBJZFxDfX8f2HA9et/q7MfGQ9cfwuMLm61S/ACyNip+oYv1d99msR8ehG5HRmRLyxer5HFevDwLPAxdX2LwBfro7xCmDOkGNvuxHHkCQ1xOJX0pZ4KjOnDN1QFYFPDN0E/HFmfn2t/Y5rYxwjgMMz8+l1xLLRIuKVtArpIzLzyYi4FthuPbtnddzH1v4ZSFI7BPZ96+DMr6S6fR34w4jYBiAiXhoROwLXAW+qZoLHA7+zjs/eDBwVEROrz46ptv8SeMGQ/eYBf7z6RURMqZ5eB/xBte1YYJcNxPoi4NGq8N2fVud5tRHA6u71H9Aap/gFcH9EnFQdIyLikA0cQ5LUIItfSXX7DK153tsi4i7g32n91uky4IfVe58Hblr7g5m5HDid1ojBd3lu7OCrwBtXX/AGnAkMVBfU3cNzq078Ha3i+W5a4w8/3UCsVwGjIuJe4B9pFd+rPQEcVuXwKuDD1fZTgNOq+O4GZm7Ez0SS1JDIzKZjkCRJ0lpePm1KfuumbzQdBrtsO25hZg40HUe72PmVJElSMSx+JUmSVAxXe5AkSeparvfQbnZ+JUmSVAw7v5IkSV3Kvm/72fmVJElSMSx+JUmSVAzHHiRJkrqWgw/tZudXkiRJxbD4lSRJUjEce5AkSepKQYRjD+1m51eSJEnFsPiVJElSMSx+JUmSVAyLX0mSJBXDC94kSZK6UADhOr9tZ+dXkiRJxbD4lSRJUjEce5AkSepajj20m51fSZIkFcPiV5IkScVw7EGSJKlLOfTQfnZ+JUmSVAw7v5IkSV0qwt5vu9n5lSRJUjEsfiVJklQMxx4kSZK6UuAlb+1n51eSJEnFsPiVJElSMRx7kCRJ6lIOPbSfnV9JkiQVw86vJElS17L32252fiVJklQMi19JkiQVw7EHSZKkbhTe3rgOdn4lSZJUDItfSZIkFcPiV5IkScWw+JUkSVIxLH4lSZK0RSLimIj4QUQsiogPNB3PcFztQZIkqQsFEFvBTS4iYiTwr8CrgSXArRFxRWbe02xk62bnV5IkSVviMGBRZv44M1cCXwRmNhzTetn5lSRJ6kK3Lbz969uP2nFs03EA20XEgiGvz8/M84e87gMeGPJ6CTC9I5FtBotfSZKkLpSZxzQdQy9y7EGSJElbYimwx5DX/dW2rmTxK0mSpC1xK7BvREyMiNHAm4ErGo5pvRx7kCRJ0mbLzFUR8UfA14GRwIWZeXfDYa1XZGbTMUiSJEkd4diDJEmSimHxK0mSpGJY/EqSJKkYFr+SJEkqhsWvJEmSimHxK0mSpGJY/EqSJKkY/z9H6rdu4PlzswAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = confusion_matrix(true_labels, pred_labels)\n",
    "\n",
    "print ('Confusion Matrix :')\n",
    "\n",
    "def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Greens):\n",
    "    plt.figure(figsize=(10,10),)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    \n",
    "    #tick_marks = np.arange(len(target_names))\n",
    "    #plt.xticks(tick_marks, target_names, rotation=45)\n",
    "    #plt.yticks(tick_marks, target_names)\n",
    "    plt.tight_layout()\n",
    "\n",
    "    width, height = cm.shape\n",
    "\n",
    "    for x in range(width):\n",
    "        for y in range(height):\n",
    "            plt.annotate(str(cm[x][y]), xy=(y, x), \n",
    "                        horizontalalignment='center',\n",
    "                        verticalalignment='center')\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "\n",
    "plot_confusion_matrix(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even the multi-class problem can be solved near perfectly with XGBoost.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- As we saw in the binary and multi-class classification problems, XGBoost can be very effective at detecting anomalies when you have labeled data. In labs 2 and 3, we will consider the same KDD99 dataset but we will train deep learning models to detect anomalies without using the labels.  This will mimic a more likely situation is the real world.\n",
    "\n",
    "- GPU-Accelerating XGboost through RAPIDS is easy and fast.  The only change we had to make to use the GPU was to set the 'tree_method' parameter to 'gpu_hist'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ol>\n",
    "<li>\n",
    "Dhaliwal, S., Nahid, A., & Abbas, R. (2018). Effective Intrusion Detection System Using XGBoost. Information, 9(7), 149. doi:10.3390/info9070149\n",
    "</li>\n",
    "<li>\n",
    "Brownlee, J. A Gentle Introduction to XGBoost for Applied Machine Learning. Machine Learning\n",
    "Mastery. Available online: http://machinelearningmastery.com/gentle-introduction-xgboost-applied-machine-learning/\n",
    "(accessed on 2 March 2018).\n",
    "    </li>\n",
    "    <li>A Study on NSL-KDD Dataset for Intrusion Detection System Based on Classification Algorithms. Available\n",
    "online: https://pdfs.semanticscholar.org/1b34/80021c4ab0f632efa99e01a9b073903c5554.pdf (accessed on\n",
    "        26 March 2018)</li>\n",
    "    <li>\n",
    "        XGBoost Parameters—Xgboost 0.7 Documentation. Available online: http://xgboost.readthedocs.io/en/\n",
    "latest/parameter.html (accessed on 12 March 2018)\n",
    "    </li>\n",
    "    <li>\n",
    "        RAPIDS Documentation and Cheat Sheet.Available online: https://rapids.ai/documentation.html\n",
    "    </li>\n",
    "    \n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/en-us/deep-learning-ai/education/\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}