Lab3-Anomaly Detection in Network Data using GANS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/en-us/deep-learning-ai/education/\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anomaly Detection in Network Data using GANs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to the third lab of this series!\n",
    "\n",
    "In the previous labs, we tried our hand at supervised and unsupervised anomaly detection using XGBoost and Deep Autoencoders on the KDD-99 network intrusion dataset.\n",
    "\n",
    "We addressed the issue of unlabelled training data through the use of Deep Autoencoders in the second lab. However, unsupervised methods such as PCA and Autoencoders tend to be effective only on highly correlated data such as the KDD dataset, and these algorithms might also require the data to follow a Gaussian Distribution.\n",
    "\n",
    "> \"Adversarial training (also called GAN for Generative Adversarial Networks), and the variations that are now being proposed, is the most interesting idea in the last 10 years in ML, in my opinion.\".\n",
    "> Yann LeCun, 2016.\n",
    "\n",
    "What do GANs bring to the table and how are they different from Deep Autoencoders?\n",
    "\n",
    "GANs are generative models that generate samples similar to the training dataset by learning the true data distribution. So instead of compressing the input into a latent space and classifying the test samples based on the reconstruction error, we actually train a classifier that outputs a probability score of a sample being Normal or Anomalous. As we will see later in the lab, this has positioned GANs as very attaractive unsupervised learning techniques.\n",
    "\n",
    "GANs can be pretty tough to train and improving their stability is an active area of research today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning: The sklearn.preprocessing.label module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.preprocessing. Anything that cannot be imported from sklearn.preprocessing is now part of the private API.\n",
      "  warnings.warn(message, FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# Import system packages\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "import logging\n",
    "import importlib\n",
    "\n",
    "#Import data manipulation libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import collections\n",
    "from tqdm import tqdm\n",
    "\n",
    "#Import visualization libraries\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "#Importing ML/DL libraries\n",
    "from sklearn.preprocessing import MinMaxScaler, label\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import roc_curve, auc,precision_recall_fscore_support, average_precision_score\n",
    "from sklearn.metrics import precision_recall_curve, auc, confusion_matrix,accuracy_score\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "# from tensorflow.keras import layer\n",
    "\n",
    "from tensorflow.keras import initializers\n",
    "from tensorflow.keras.models import Model, Sequential\n",
    "from tensorflow.keras.optimizers import Adam, RMSprop\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from tensorflow.keras.layers import Dense, Dropout\n",
    "from tensorflow.keras.layers import LeakyReLU\n",
    "from tensorflow.keras.layers import Input, BatchNormalization, LeakyReLU, Dense, Reshape, Flatten, Activation \n",
    "from tensorflow.keras.layers import Dropout, multiply, GaussianNoise, MaxPooling2D, concatenate\n",
    "import pickle\n",
    "\n",
    "import random\n",
    "random.seed(123)\n",
    "\n",
    "data_path = './data/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us load the same KDD-99 dataset that we used in the previous labs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.8/dist-packages/sklearn/base.py:313: UserWarning: Trying to unpickle estimator LabelEncoder from version 0.20.0 when using version 0.22.1. This might lead to breaking code or invalid results. Use at your own risk.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "#loading the pickled file\n",
    "\n",
    "filename = data_path + 'preprocessed_data_full.pkl'\n",
    "input_file = open(filename,'rb')\n",
    "preprocessed_data = pickle.load(input_file)\n",
    "input_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like in Lab 2, we will split the pickled data into vectors and assign them to the label encoder `le`, training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train\n",
      "y_train\n",
      "x_test\n",
      "le\n",
      "y_test\n"
     ]
    }
   ],
   "source": [
    "for key in preprocessed_data:\n",
    "    print(key)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "le = preprocessed_data['le']\n",
    "x_train = preprocessed_data['x_train']\n",
    "y_train = preprocessed_data['y_train']\n",
    "x_test = preprocessed_data['x_test']\n",
    "y_test = preprocessed_data['y_test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  5,  9, 10, 11, 15, 17, 18, 20, 21])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.unique(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first consider the binary classification problem. Similar to previous labs, *Normal* data points will be labeled as '0' and *Anomalous* points will be labeled as '1'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>back.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>buffer_overflow.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ftp_write.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>guess_passwd.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>imap.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ipsweep.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>land.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>loadmodule.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>multihop.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>neptune.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>nmap.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>normal.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>perl.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>phf.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>pod.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>portsweep.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>rootkit.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>satan.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>smurf.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>spy.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>teardrop.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>warezclient.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>warezmaster.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                Type\n",
       "0              back.\n",
       "1   buffer_overflow.\n",
       "2         ftp_write.\n",
       "3      guess_passwd.\n",
       "4              imap.\n",
       "5           ipsweep.\n",
       "6              land.\n",
       "7        loadmodule.\n",
       "8          multihop.\n",
       "9           neptune.\n",
       "10             nmap.\n",
       "11           normal.\n",
       "12             perl.\n",
       "13              phf.\n",
       "14              pod.\n",
       "15        portsweep.\n",
       "16          rootkit.\n",
       "17            satan.\n",
       "18            smurf.\n",
       "19              spy.\n",
       "20         teardrop.\n",
       "21      warezclient.\n",
       "22      warezmaster."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Obtain the class number for Normal entries \n",
    "pd.DataFrame(le.classes_, columns = ['Type'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting labels to Binary\n",
    "\n",
    "y_test[y_test != 11] = 1 \n",
    "y_test[y_test == 11] = 0\n",
    "y_train[y_train != 11] = 1\n",
    "y_train[y_train == 11] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now split the dataset into normal and anomalous data. We will need to do this in order to be able to train GANs to generate Normal packets only and then predict the anomaly based on the Discriminator output. The details regarding this will be covered later in the lab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Subsetting only Normal Network packets in our training set\n",
    "\n",
    "temp_df = x_train.copy()\n",
    "temp_df['label'] = y_train\n",
    "temp_df = temp_df.loc[temp_df['label'] == 0]\n",
    "temp_df = temp_df.drop('label', axis = 1)\n",
    "x_train = temp_df.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to Lab 2, we scale the input training data between 0 and 1 before feeding it to the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Normalize the above splits using the MinMaxScaler from the scikit learn package\n",
    "scaler = MinMaxScaler()\n",
    "\n",
    "# Make sure to only fit the scaler on the training data\n",
    "x_train = scaler.fit_transform(x_train)\n",
    "x_test = scaler.transform(x_test)\n",
    "\n",
    "#Creating dataset dictionary \n",
    "dataset = {}\n",
    "dataset['x_train'] = x_train.astype(np.float32)\n",
    "dataset['y_train'] = y_train.astype(np.float32)\n",
    "dataset['x_test']  = x_test.astype(np.float32)\n",
    "dataset['y_test']  = y_test.astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Summary of the datasets** :\n",
    "- The Training set consists of only normal network packets.\n",
    "- The Testing set comprises a small number of anomalous network packets of about 1%, reflecting what we see in the real world. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Normal Network packets in the Training set: 729620\n",
      "Number of Normal Network packets in the Testing set: 243161\n",
      "Number of Anomalous Network packets in the Testing set: 2466\n"
     ]
    }
   ],
   "source": [
    "# check how many anomalies are in our Testing set\n",
    "print('Number of Normal Network packets in the Training set:', x_train.shape[0])\n",
    "print('Number of Normal Network packets in the Testing set:', collections.Counter(y_test)[0])\n",
    "print('Number of Anomalous Network packets in the Testing set:', collections.Counter(y_test)[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Generative Adversarial Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generative Adversarial Networks (GAN) were introduced by Ian Goodfellow in [Generative Adversarial Networks](https://arxiv.org/abs/1406.2661), Goodfellow, 2014.\n",
    "\n",
    "<br>\n",
    "<img src=\"https://drive.google.com/uc?id=14f7ZwXxyigpwx7bIVFxAOSb_aV28bFav\" alt=\"Alt text that describes the graphic\" title=\"Title text\" />\n",
    "<br>\n",
    "\n",
    "\n",
    "The GAN consists of two networks namely: \n",
    "- The generator `G` that produces fake samples\n",
    "- The discriminator `D` that that receives samples from both `G` and the dataset.\n",
    "\n",
    "During Training the two networks have competing goals. The generator tries to fool the discriminator by outputting values that resemble real data and the discriminator tries to become better at distinguishing between the real and fake data.\n",
    "\n",
    "Mathematically, this means that the Generator's weights are optimized to maximize the probability that fake data is classified as belonging to the real data. The discriminators's weights are optimized to maximize the probability that the real input data is classified as real while minimizing the probability of fake input data being classified as real.\n",
    "\n",
    "Optimality is reached when the generator produces an output that the disciminator cannot concretely label as real or fake and this, happens when either of the networks cannot improve anymore.\n",
    "\n",
    "<img src=./images/gan-optimality.PNG alt=\"Alt text that describes the graphic\" title=\"Title text\" />\n",
    "\n",
    "The first part of the above equation reflects the log probability of the discriminator predicting that the input sample is genuine and the second half reflects the probability of the Discriminator predicting that the Generator's output is not genuine.\n",
    "\n",
    "In this lab, we will be train our GAN on normal network packets. The generator inputs noise and as training progresses the GAN learns the mapping between these random values to the input distribution. The discriminator outputs a score of how likely the the generated output resembles the real data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Generator Network "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Generator is used to synthesize fake data points. As shown below, it consists of 5 Dense Layers with a `tanh` activation function and uses `binary cross-entropy` for calculating the generator loss. Binary cross-entropy loss measures the performance of a two class classification model whose output is a probability value between 0 and 1. A perfect model would have a loss of 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![layers](images/layers.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the Generator network\n",
    "\n",
    "def get_generator(optimizer):\n",
    "    \n",
    "    generator = Sequential()\n",
    "    generator.add(Dense(64, input_dim=114, kernel_initializer=initializers.glorot_normal(seed=42)))\n",
    "    generator.add(Activation('tanh'))\n",
    "    \n",
    "    generator.add(Dense(128))\n",
    "    generator.add(Activation('tanh'))\n",
    "    \n",
    "    generator.add(Dense(256))\n",
    "    generator.add(Activation('tanh'))\n",
    "    \n",
    "    generator.add(Dense(256))\n",
    "    generator.add(Activation('tanh'))\n",
    "       \n",
    "    generator.add(Dense(512))\n",
    "    generator.add(Activation('tanh'))\n",
    "   \n",
    "    generator.add(Dense(114, activation='tanh'))\n",
    "    \n",
    "    generator.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "    \n",
    "    return generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Discriminator Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![layers2](images/layers2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Discriminator basically outputs the score of a sample belonging to the real dataset or the synthetic dataset. It consists of 6 dense layers-each followed by a dropout layer to help prevent overfitting. The sigmoid activation function is applied to the final layer to obtain a value in the range 0 to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "##### Defining the Discriminator network\n",
    "\n",
    "def get_discriminator(optimizer):\n",
    "    \n",
    "    discriminator = Sequential()\n",
    "    \n",
    "    discriminator.add(Dense(256, input_dim=114, kernel_initializer=initializers.glorot_normal(seed=42)))\n",
    "    discriminator.add(Activation('relu'))\n",
    "    discriminator.add(Dropout(0.2))\n",
    "\n",
    "    discriminator.add(Dense(128))\n",
    "    discriminator.add(Activation('relu'))\n",
    "    discriminator.add(Dropout(0.2))\n",
    "       \n",
    "    discriminator.add(Dense(128))\n",
    "    discriminator.add(Activation('relu'))\n",
    "    discriminator.add(Dropout(0.2))\n",
    "    \n",
    "    discriminator.add(Dense(128))\n",
    "    discriminator.add(Activation('relu'))\n",
    "    discriminator.add(Dropout(0.2))\n",
    "\n",
    "    discriminator.add(Dense(128))\n",
    "    discriminator.add(Activation('relu'))\n",
    "    discriminator.add(Dropout(0.2))\n",
    "    \n",
    "    discriminator.add(Dense(1))\n",
    "    discriminator.add(Activation('sigmoid'))\n",
    "   \n",
    "    discriminator.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "\n",
    "    return discriminator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Building the Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously we built the individual networks that constitute the GAN. Let us now stich them together by using the generator and discriminator as individual layers of the overall generative adversarial network. We set the field `discriminator.trainable = False` as we want to alter the weights of only one of them during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_gan_network(discriminator, generator, optimizer,input_dim=114):\n",
    "\n",
    "    discriminator.trainable = False   \n",
    "    gan_input = Input(shape=(input_dim,))  \n",
    "    x = generator(gan_input)        \n",
    "    gan_output = discriminator(x)\n",
    "    \n",
    "    gan = Model(inputs=gan_input, outputs=gan_output)    \n",
    "    gan.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
    "    \n",
    "    return gan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Setting the Hyperparameters "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training a GAN can be quite tricky and time consuming. Refer to this <a href=\"https://github.com/soumith/ganhacks#authors\">  link </a> to get a few ideas on how you could get your models to converge faster by choosing appropriate hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.00001\n",
    "batch_size = 512\n",
    "epochs = 10\n",
    "adam = Adam(learning_rate = learning_rate,beta_1 = 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Training the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generator first predicts on a batch of noise samples. As the generator has randomly initialized weights initially, the output of the generator at this stage is nothing but meaningless values.\n",
    "<br>\n",
    "<img src=./images/Gan-Page-04.jpg alt=\"Alt text that describes the graphic\" title=\"Title text\" />\n",
    "<br>\n",
    "The Discriminator inputs a stack of samples - the first half of which is the output of the generator and the second half is a batch of data samples from the real dataset. We train the Discriminator on this stack with the target labels  0 (Fake) for half the stack and 1 for the second half of the stack. The result of this is that the Discrimator is able to distinguish between the Real and Fake samples.\n",
    "<br>\n",
    "<img src=./images/Gan-Page-2.jpg alt=\"Alt text that describes the graphic\" title=\"Title text\" />\n",
    "<br>\n",
    "The weights of the discriminator are frozen by setting the trainable parameter to False.\n",
    "\n",
    "To train the Generator, We first feed it random noise and let the entire GAN output a probability with the Discriminator weights remaining frozen. As expected this value would be less than 0.5 since the Discriminator was previously set to output a value close to 0 if the input was not genuine .\n",
    "<br>\n",
    "<img src=./images/Gan-Page-3.jpg alt=\"Alt text that describes the graphic\" title=\"Title text\" />\n",
    "<br>\n",
    "\n",
    "Now comes the trick. We tell the GAN that the expected output is 1. This results in the errors being backpropagated only to the Generator. With every sample in the batch the generator's weights are tuned such that the output of the GAN is close to 1, meaning the Generator is now learning to produce samples that resemble the real data.\n",
    "\n",
    "This process loops back to the first step for each batch in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 10%|█         | 1434/14250 [00:29<04:30, 47.40it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 Batch 1424/1425 [D loss: 0.112755] [G loss:0.029275]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 20%|██        | 2858/14250 [00:56<03:37, 52.27it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Batch 1424/1425 [D loss: 0.032169] [G loss:0.003187]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 30%|███       | 4283/14250 [01:24<03:11, 52.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2 Batch 1424/1425 [D loss: 0.014987] [G loss:0.001494]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 40%|████      | 5708/14250 [01:52<02:40, 53.16it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 3 Batch 1424/1425 [D loss: 0.011992] [G loss:0.006911]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 50%|█████     | 7133/14250 [02:19<02:14, 53.00it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 4 Batch 1424/1425 [D loss: 0.005663] [G loss:0.001886]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 60%|██████    | 8560/14250 [02:47<01:51, 51.04it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 5 Batch 1424/1425 [D loss: 0.024951] [G loss:0.000566]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 70%|███████   | 9981/14250 [03:15<01:22, 51.74it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 6 Batch 1424/1425 [D loss: 0.024059] [G loss:0.000072]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 80%|████████  | 11406/14250 [03:43<00:55, 51.45it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 7 Batch 1424/1425 [D loss: 0.018900] [G loss:0.001260]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 90%|█████████ | 12831/14250 [04:11<00:27, 51.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 8 Batch 1424/1425 [D loss: 0.019725] [G loss:0.001055]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████▉| 14249/14250 [04:39<00:00, 51.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 9 Batch 1424/1425 [D loss: 0.004175] [G loss:0.000253]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 14250/14250 [04:50<00:00, 51.36it/s]"
     ]
    }
   ],
   "source": [
    "#Training the GAN\n",
    "x_train, y_train, x_test, y_test = dataset['x_train'], dataset['y_train'],dataset['x_test'],dataset['y_test']\n",
    "\n",
    "#Calculating the number of batches based on the batch size\n",
    "batch_count = x_train.shape[0] // batch_size\n",
    "pbar = tqdm(total=epochs * batch_count)\n",
    "gan_loss = []\n",
    "discriminator_loss = []\n",
    "\n",
    "#Inititalizing the network\n",
    "generator = get_generator(adam)\n",
    "discriminator = get_discriminator(adam)\n",
    "gan = get_gan_network(discriminator, generator, adam,input_dim=114)\n",
    "\n",
    "\n",
    "for epoch in range(epochs):        \n",
    "    for index in range(batch_count):        \n",
    "        pbar.update(1)        \n",
    "        # Creating a random set of input noise and images\n",
    "        noise = np.random.normal(0, 1, size=[batch_size,114])\n",
    "        \n",
    "        # Generate fake samples\n",
    "        generated_images = generator.predict_on_batch(noise)\n",
    "        \n",
    "        #Obtain a batch of normal network packets\n",
    "        image_batch = x_train[index * batch_size: (index + 1) * batch_size]\n",
    "            \n",
    "        X = np.vstack((generated_images,image_batch))       \n",
    "        y_dis = np.ones(2*batch_size) \n",
    "        y_dis[:batch_size] = 0\n",
    "\n",
    "        # Train discriminator\n",
    "        discriminator.trainable = True\n",
    "        d_loss= discriminator.train_on_batch(X, y_dis)\n",
    "\n",
    "        # Train generator\n",
    "        noise = np.random.uniform(0, 1, size=[batch_size, 114])\n",
    "        y_gen = np.ones(batch_size)\n",
    "        discriminator.trainable = False\n",
    "        g_loss = gan.train_on_batch(noise, y_gen)\n",
    "        \n",
    "        #Record the losses\n",
    "        discriminator_loss.append(d_loss)\n",
    "        gan_loss.append(g_loss)\n",
    "        \n",
    "    print(\"Epoch %d Batch %d/%d [D loss: %f] [G loss:%f]\" % (epoch,index,batch_count, d_loss, g_loss))\n",
    "              "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.6 Training Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's visualize the generator and discriminator training losses!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fe2e3971f10>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3dUlEQVR4nO3dd3xUVfr48c8zk0avoZcEF5AeMICIq/5soCjsuvoVdRXXdVlWse664rqLyK69omLBgmV1UbGhoihNREEITekECBBq6CSQMpnz++PeTGaSSTJJZjKZyfN+vfLKLefe+8yd5Jk75557jhhjUEopFfkc4Q5AKaVUcGhCV0qpKKEJXSmlooQmdKWUihKa0JVSKkpoQldKqSihCV1FDBH5SkTGBLusUtFCtB26CiURyfaarQ/kAYX2/J+NMe/WfFRVJyLnAf81xnQIcyhKlRIT7gBUdDPGNCyaFpEM4GZjzNyS5UQkxhjjqsnYlIo2WuWiwkJEzhORTBG5V0T2AdNFpJmIfCEiWSJyxJ7u4LXNQhG52Z6+UUQWi8iTdtntInJJFcsmi8giETkhInNFZKqI/LcKr6mHfdyjIrJOREZ6rbtURNbbx9gtIn+zl7e0X+dRETksIt+LiMNe105EPrLPx3YRud1rf4NEJE1EjovIfhF5urLxquijCV2FUxugOdAZGIv19zjdnu8EnAJeKGf7wcAmoCXwOPC6iEgVyr4HLANaAJOA6yv7QkQkFvgc+AZoBdwGvCsi3e0ir2NVMTUCegPz7eV/BTKBRKA18A/A2En9c2AN0B64ALhTRIbZ200BphhjGgOnAR9UNmYVfTShq3ByAw8YY/KMMaeMMYeMMR8ZY04aY04ADwHnlrP9DmPMq8aYQuAtoC1WUgy4rIh0AgYCE40x+caYxcCsKryWM4GGwKP2fuYDXwDX2OsLgJ4i0tgYc8QYs9JreVugszGmwBjzvbFubA0EEo0xk+39bQNeBUZ7bfcrEWlpjMk2xiytQswqymhCV+GUZYzJLZoRkfoi8oqI7BCR48AioKmIOMvYfl/RhDHmpD3ZsJJl2wGHvZYB7Krk68Dezy5jjNtr2Q6sq2uA3wGXAjtE5DsRGWIvfwJIB74RkW0iMsFe3hloZ1fFHBWRo1hX70UfWH8EugEbRWS5iFxWhZhVlNGboiqcSjax+ivQHRhsjNknIinAKqCsapRg2As0F5H6Xkm9YxX2swfoKCIOr6TeCdgMYIxZDoyyq2bGY1WRdLS/ifwV+KuI9Abmi8hyrA+V7caYrv4OZozZAlxjV81cAcwUkRbGmJwqxK6ihF6hq9qkEVa9+VERaQ48EOoDGmN2AGnAJBGJs6+cL69oOxFJ8P7BqoM/CfxdRGLt5o2XAzPs/V4nIk2MMQXAcazqJkTkMhH5lV2ffwyrSafb3t8J+6ZxPRFxikhvERlob/d7EUm0PzyO2mF5fztQdZAmdFWbPAvUAw4CS4Gva+i41wFDgEPAf4D3sdrLl6U91geP909HrAR+CVb8LwI3GGM22ttcD2TYVUnj7GMCdAXmAtnAEuBFY8wCu67/MiAF2G7v8zWgib3dcGCd3c5/CjDaGHOq6qdARQN9sEipEkTkfWCjMSbk3xCUCia9Qld1nogMFJHTRMQhIsOBUcCnYQ5LqUrTm6JKWe3hP8Zqh54J/MUYsyq8ISlVeVrlopRSUUKrXJRSKkqErcqlZcuWJikpKVyHV0qpiLRixYqDxphEf+vCltCTkpJIS0sL1+GVUioiiciOstZplYtSSkUJTehKKRUlNKErpVSU0HboStVxBQUFZGZmkpubW3FhVWMSEhLo0KEDsbGxAW+jCV2pOi4zM5NGjRqRlJRE2eODqJpkjOHQoUNkZmaSnJwc8HYBVbmIyHAR2SQi6V79NXuvf0ZEVts/m+2+m5VSESA3N5cWLVpoMq9FRIQWLVpU+ltThVfo9uACU4GLsB6LXi4is4wx64vKGGPu8ip/G9C/UlEopcJKk3ntU5X3JJAr9EFAujFmmzEmH5iB1XlRWa4B/lfpSAK0Zf8JbnhjGdplgVJK+QokobfHd0iuTIqH1fIhIp2BZIoHwC25fqw9UnlaVlZWZWMF4KJnFrFocxZ3vr+6StsrpWofp9NJSkoKvXr1ol+/fjz11FO43dZ4HWlpadx+++3VPsbLL7/M22+/XaltzjrrrCof780332TPnj1V3r4qgn1TdDQw0+6cvxRjzDRgGkBqamq1LrE/W72HKaO1ZkepaFCvXj1Wr14NwIEDB7j22ms5fvw4Dz74IKmpqaSmplZr/y6Xi3HjxlV6ux9//LHKx3zzzTfp3bs37dq1C3ibwsJCnM6yhtCtWCBX6LvxHWOxg73Mn9GEsLoFYM3Eiz3TWu2iVPRp1aoV06ZN44UXXsAYw8KFC7nsMmsM7O+++46UlBRSUlLo378/J06cAOCxxx6jT58+9OvXjwkTrHYb5513HnfeeSepqalMmTKFSZMm8eSTT3rW3XXXXaSmptKjRw+WL1/OFVdcQdeuXfnnP//piaVhQ2vM8YULF3Leeedx5ZVXcvrpp3Pdddd58s/kyZMZOHAgvXv3ZuzYsRhjmDlzJmlpaVx33XWkpKRw6tQp5s2bR//+/enTpw833XQTeXnWoFhJSUnce++9DBgwgA8//LBa5y6QK/TlQFcRScZK5KOBa0sWEpHTgWZYw2iFTJP6xW0y5288wAU9WpdTWilVGQ9+vo71e44HdZ892zXmgct7VWqbLl26UFhYyIEDB3yWP/nkk0ydOpWhQ4eSnZ1NQkICX331FZ999hk//fQT9evX5/Dhw57y+fn5nj6jJk2a5LOvuLg40tLSmDJlCqNGjWLFihU0b96c0047jbvuuosWLVr4lF+1ahXr1q2jXbt2DB06lB9++IGzzz6b8ePHM3HiRACuv/56vvjiC6688kpeeOEFnnzySVJTU8nNzeXGG29k3rx5dOvWjRtuuIGXXnqJO++8E4AWLVqwcuXKSp0jfyq8QjfGuLBGKZ8DbAA+MMasE5HJIjLSq+hoYIapwcvmP76lnXspVZcMHTqUu+++m+eee46jR48SExPD3Llz+cMf/kD9+vUBaN68uaf81VdfXea+Ro600lefPn3o1asXbdu2JT4+ni5durBr165S5QcNGkSHDh1wOBykpKSQkZEBwIIFCxg8eDB9+vRh/vz5rFu3rtS2mzZtIjk5mW7dugEwZswYFi1aFFCclRFQHboxZjYwu8SyiSXmJwUlogB8cdvZXPb84po6nFJ1RmWvpENl27ZtOJ1OWrVqxYYNGzzLJ0yYwIgRI5g9ezZDhw5lzpw55e6nQYMGZa6Lj48HwOFweKaL5l0uV5nlwbqJ63K5yM3N5ZZbbiEtLY2OHTsyadKkKj1xW16clRGRfbn0bt/EM13o1np0paJJVlYW48aNY/z48aXaYm/dupU+ffpw7733MnDgQDZu3MhFF13E9OnTOXnyJIBPlUuoFSXvli1bkp2dzcyZMz3rGjVq5Knj7969OxkZGaSnpwPwzjvvcO655wY9noh/9P9/y3by+zM7hzsMpVQ1nDp1ipSUFAoKCoiJieH666/n7rvvLlXu2WefZcGCBTgcDnr16sUll1xCfHw8q1evJjU1lbi4OC699FIefvjhGom7adOm/OlPf6J37960adOGgQMHetbdeOONjBs3jnr16rFkyRKmT5/OVVddhcvlYuDAgVVqdVORsI0pmpqaaqozwMXUBek8MWcTABmPjghWWErVORs2bKBHjx7hDkP54e+9EZEVxhi/7TgjssoF4Df9/T7bpJRSdVbEJvT2TeuFOwSllKpVIjahK6WU8hV5CT07CzJXgFfdf0GhO4wBKaVU7RB5CX31u/Da+VBwim6trcdyv/i5ZjvAUUqp2ijyEnqc3QA/P4fN+7MBuOv9NWEMSCmlaofIS+jxjazf+Sf46C9V79pSKVV77N+/n2uvvZYuXbpwxhlnMGTIED755JOwxLJw4cJq9bIYTpGX0L2u0Hu1axzeWJRS1WaM4Te/+Q3nnHMO27ZtY8WKFcyYMYPMzMyQHdPfo/1FqpLQy9tfTYrAhG7Vm5OXTUJs1fsNVkrVDvPnzycuLs7nycnOnTtz2223UVhYyD333MPAgQPp27cvr7zyClB+d7YrVqzg3HPP5YwzzmDYsGHs3bsXKN2d7ueff87gwYPp378/F154Ifv37ycjI4OXX36ZZ555hpSUFL7//nsyMjI4//zz6du3LxdccAE7d+4Eip8EHTx4MH//+99r+Kz5F3mP/uces35/9xjc8GlYQ1Eq6nw1Afb9Etx9tukDlzxa5up169YxYMAAv+tef/11mjRpwvLly8nLy2Po0KFcfLE1JoK/7mwHDx7MbbfdxmeffUZiYiLvv/8+999/P2+88Qbg253ukSNHWLp0KSLCa6+9xuOPP85TTz3FuHHjaNiwIX/7298AuPzyyxkzZgxjxozhjTfe4Pbbb+fTTz8FIDMzkx9//LFag1IEU+QldIcd8q5l4Y1DKRUSt956K4sXLyYuLo7OnTvz888/ezq9OnbsGFu2bCEuLs7TnS3g6c62adOmrF27losuugiwRgBq27atZ9/e3dRmZmZy9dVXs3fvXvLz80lOTvYbz5IlS/j4448Bq79z76vxq666qtYkc4jEhN7pTOt3x0E+i91ug8OhI5crVS3lXEmHSq9evfjoo48881OnTuXgwYOkpqbSqVMnnn/+eYYNG+azzcKFC/12Z2uMoVevXixZ4n+cHe9uam+77TbuvvtuRo4cycKFC0sNgBGIYHV7GyyRV4eeYHed29q33+as7LwwBKOUqq7zzz+f3NxcXnrpJc+yoq5whw0bxksvvURBQQEAmzdvJicnp8x9de/enaysLE9CLygo8DvgBFhX++3bW31CvfXWW57l3t3egjVQ9IwZMwB49913+fWvf12Vl1kjIi+hO2PBEWv99nLPzJ/DFJBSqjpEhE8//ZTvvvuO5ORkBg0axJgxY3jssce4+eab6dmzJwMGDKB37978+c9/LrdFSVxcHDNnzuTee++lX79+pKSklNliZdKkSVx11VWcccYZtGzZ0rP88ssv55NPPvHcFH3++eeZPn06ffv25Z133mHKlClBPwfBEpnd5z7aCfqOhksfJ2nClwCIwPZHtBtdpSpLu8+tvepG97mx9aHA+tr14bghgE/XLkopVScFlNBFZLiIbBKRdBGZUEaZ/xOR9SKyTkTeC26YJcTWh4JTgHajq5RSRSps5SIiTmAqcBGQCSwXkVnGmPVeZboC9wFDjTFHRKRVqAIGICcLdv4EQIxTW7YoVV3GmFLjd6rwqkp1eCBX6IOAdGPMNmNMPjADGFWizJ+AqcaYI3YgByodSWXkHYfj1mPBjeJjKyislCpPQkIChw4dqlICUaFhjOHQoUMkJCRUartA2qG3B3Z5zWcCg0uU6QYgIj8ATmCSMebrkjsSkbHAWIBOnTpVKtCy1IurPY36lYpEHTp0IDMzk6ysrHCHorwkJCR4HpwKVLAeLIoBugLnAR2ARSLSxxhz1LuQMWYaMA2sVi5VPlr7VNhd9QGmlVLFYmNjy3xKUkWWQKpcdgMdveY72Mu8ZQKzjDEFxpjtwGasBB8aHQcXd9LlRb8yKqXqskAS+nKgq4gki0gcMBqYVaLMp1hX54hIS6wqmG3BC7OEuAaQn1OqreKiLQdDdkillKrtKkzoxhgXMB6YA2wAPjDGrBORySIy0i42BzgkIuuBBcA9xphDoQqa/GzAFPe8aJu/YX/IDqmUUrVdQHXoxpjZwOwSyyZ6TRvgbvsn9E7anxVHMqBeCref/yuem5/OWb9qWe5mSikVzSLzSdGirnN/ehmAAZ2bAeDQdrRKqTosMhN6UcdcWRsBiIuxXsZbP2aEKSCllAq/yEzoOfbNzz2rAMg6YXWduzhdb4oqpequyEzoRYNc2OKckfkylFIqmCIzE176hM/sRT1bhykQpZSqPSIzocf5DvsUo1foSikVoQm9XrNwR6CUUrVOZCZ0b+5Cn9lDOraoUqqOivyEnvaGz6wOFq2UqqsiP6GvmeEzu1j7c1FK1VGRn9BLdKPbpknlOoRXSqloEbkJPaFp8fT+9XRuUR+Ab9drB11KqbopchP6/71dPP3SELq2svpH37I/O0wBKaVUeEVuQk/s7jPrdFgdcxW6dZALpVTdFLkJvVEbn9k/n3saADsPnwxHNEopFXaRm9BL6NGmMQCnCgorKKmUUtEpahJ6fEzUvBSllKqSyM6CKb/3TDocOriFUqpui+yEfsG/iqdd+eGLQymlaoGAErqIDBeRTSKSLiIT/Ky/UUSyRGS1/XNz8EP1w/vG6NGdnsmCQneNHF4ppWqTCgeJFhEnMBW4CMgElovILGPM+hJF3zfGjA9BjIExxTdD81xuYrVLXaVUHRNI1hsEpBtjthlj8oEZwKjQhlUJlz1r/Y6t71m0YOOB8MSilFJhFEhCbw/s8prPtJeV9DsR+VlEZopIR387EpGxIpImImlZWVlVCNePGLvvFneBZ9GRk1qfrpSqe4JVL/E5kGSM6Qt8C7zlr5AxZpoxJtUYk5qYmBicIx/eZv3eX1wD9Mmq3cHZt1JKRZBAEvpuwPuKu4O9zMMYc8gYU9QR+WvAGcEJLwAbPrd+r5juWbRq59EaO7xSStUWgST05UBXEUkWkThgNDDLu4CItPWaHQlsCF6IFTi+x/qdPpeECm/xKqVU9KowoRtjXMB4YA5Wov7AGLNORCaLyEi72O0isk5E1gC3AzeGKuBSGrfzTP62l441qpSqu8SY8PROmJqaatLS0iouWJFlr8LsvwFwMq4lPY8/B0DGoyOqv2+llKplRGSFMSbV37rIb6zd9WLPZP18HX5OKVV3RX5CbxCk1jJKKRXhIj+hxxU/UESj4nuzbh3oQilVx0R+QveW9GvP5JrMo+GLQymlwiC6EroztnhSu9NVStUx0ZHQHXYD9NXvehbNXb8/TMEopVR4REdCHzCm1KLn5qeHIRCllAqf6Ejolz5p/R48LrxxKKVUGEVHQnfYLyNjcXjjUEqpMIqOhF5k/1oGJzcPdxRKKRUW0ZXQgfhYp2f62KmCckoqpVR0ibqE3r5pgmd6/Z7jYYxEKaVqVtQl9AdH9vZMX/Pq0jBGopRSNSvqEnpcTNS9JKWUCohmP6WUihLRl9DD1L+7UkqFW/QldLcr3BEopVRYRF9Cn3O/Z/Lp2Bdh6/wwBqOUUjUn+hL61nmeySuci+Gd34YxGKWUqjnRl9APpZORcC2NyQ53JEopVaMCSugiMlxENolIuohMKKfc70TEiIjfAUxr0o3Ob8IdglJK1agKE7qIOIGpwCVAT+AaEenpp1wj4A7gp2AHGZB6zXxmW8uRsIShlFLhEsgV+iAg3RizzRiTD8wARvkp92/gMSA3iPEFrkVXn9l8YsIShlJKhUsgCb09sMtrPtNe5iEiA4COxpgvy9uRiIwVkTQRScvKyqp0sOVyxvnMxqMdcyml6pZq3xQVEQfwNPDXisoaY6YZY1KNMamJiYnVPbSv2ASf2TjR9uhKqbolkIS+G+joNd/BXlakEdAbWCgiGcCZwKwavzH6a9/Pkzi9QldK1TGBJPTlQFcRSRaROGA0MKtopTHmmDGmpTEmyRiTBCwFRhpj0kIScVk6DPSZ7S3ba/TwSikVbhUmdGOMCxgPzAE2AB8YY9aJyGQRGRnqAAPm8L0J2sWxL0yBKKVUeATUFMQYMxuYXWLZxDLKnlf9sKpAJCyHVUqp2iK6nhT9w9fhjkAppcImuhJ65yFwxWull2uXukqpOiC6EjpAnp9xRLVLXaVUHRB9CT37QKlF7kJN6Eqp6Bd9Cb39GaUWHc4+GYZAlFKqZkVfQqd0ffkHy3aEIQ6llKpZ0ZfQE08vtWjaws1hCEQppWpW9CX0Zp1LLXLiDkMgSilVs6IvofvRSLQOXSkV/epEQr8n5v1wh6CUUiFXJxL6IMcmlm47FO4wlFIqpOpEQk+UY4yetjTcYSilVEjViYT+izsp3CEopVTIRWdCT7nOZ9YdpS9TKaW8RWem63Olz+wid98wBaKUUjUnOhN6XrbP7OmyM0yBKKVUzYnOhB4T7zN7kXNlmAJRSqmaE50J3eEMdwRKKVXjojOhK6VUHRRQQheR4SKySUTSRWSCn/XjROQXEVktIotFpGfwQ62EdgOs3x0GeRbdEzMjTMEopVTNqDChi4gTmApcAvQErvGTsN8zxvQxxqQAjwNPBzvQSqnfHCYdg9SbPItujZmF261D0SmlolcgV+iDgHRjzDZjTD4wAxjlXcAY4z3uWwP8dUoeDr2v8Jn9wyvzwxSIUkqFXiAJvT2wy2s+017mQ0RuFZGtWFfot/vbkYiMFZE0EUnLysqqSryV44zzmb1499TQH1MppcIkaDdFjTFTjTGnAfcC/yyjzDRjTKoxJjUxMTFYhy6biM/sdTHzQn9MpZQKk0AS+m6go9d8B3tZWWYAv6lGTEoppaogkIS+HOgqIskiEgeMBmZ5FxCRrl6zI4AtwQsxuH7JPBbuEJRSKiQqTOjGGBcwHpgDbAA+MMasE5HJIjLSLjZeRNaJyGrgbmBMqAKurnH/XRHuEJRSKiRiAilkjJkNzC6xbKLX9B1Bjitkdh89Fe4QlFIqJOrkk6Jb9p8IdwhKKRV0dTKhX/TMonCHoJRSQVfnEnocBeEOQSmlQqLOJfQrnXp1rpSKTnUuoXeVzHCHoJRSIVHnErpBKi6klFIRKPoTeo+RPrOJcjQ8cSilVIhFf0KPre8ze7lzKf1lC2QfCFNASikVGtGf0Ic/Amfd5rPok/gH4OWzwxSQUkqFRvQn9PrN4eL/lF6evb/mY1FKqRCK/oSulFJ1hCZ0pZSKEnU6oesYo0qpaFKnE/rMFcUPGWXnucIYiVJKVV/dSejDHi616O8f/QzAgk0H6P3AHJImfMn8jXqzVCkVmepOQh9ya5mrlm495Jme/kNGDQSjlFLBV3cSeoBEtGsApVRkqtMJ/UrndxzO2suyjMOeZZrOlVKRqk4n9CdjX2Htc1exaudRzzK9QFdKRaqAErqIDBeRTSKSLiIT/Ky/W0TWi8jPIjJPRDoHP9TQaFWisy5XoTZlVEpFpgoTuog4ganAJUBP4BoR6Vmi2Cog1RjTF5gJPB7sQEPldMcuBspGz/zi9INhjEYppaoukCv0QUC6MWabMSYfmAGM8i5gjFlgjDlpzy4FOgQ3zND6MH5yuENQSqlqCyShtwd2ec1n2svK8kfgK38rRGSsiKSJSFpWVlbgUdawR7/ayNGT+eEOQymlKiWoN0VF5PdAKvCEv/XGmGnGmFRjTGpiYmIwDx1UL3+3lZEv/MCxUzqgtFIqcgSS0HcDHb3mO9jLfIjIhcD9wEhjTF5wwgufnYdP0u/Bb8IdhlJKBSyQhL4c6CoiySISB4wGZnkXEJH+wCtYyVyHAlJKqTCoMKEbY1zAeGAOsAH4wBizTkQmi0jRgJ1PAA2BD0VktYjMKmN3SimlQiQmkELGmNnA7BLLJnpNXxjkuGqN0/4xm83/uQSnQ584UkrVbnX6SdFAFLoNBYXucIehlFIV0oSulFJRom4l9O4jqrTZC/PTgxyIUkoFX91K6LH1qrTZCwvSyXMVBjkYpZQKrrqV0J2xVd70sa82sWhzFm8vyQhePEopFUQBtXKJGo6qv9ydh09ywxvLALhhSFKQAlJKqeCpW1foDmeZqxb+7TyeuLJvmeuN0W51lVK1W91K6DEJZa5KWvovrkrtWOZ6TedKqdqubiX0cgaKJu11AJb94wK/q+dv1B4NlFK1W91K6LENrN/1mpdZpFXjsq/ilVKqNqtbCT2hCTTvApc/W26xZvXLbw1zMDviO5NUSkWhupXQnTFw+yroOcpq8dLlPL/FnryqX7m7GT1taQiCU0qp6qlbzRa9TTxk/Z7UpNSqC3q0LnfT9APZnMovpF5c2a1mlFKqptWtK/Qg+tPbaRzO0WHqlFK1hyb0KlqcfpBrX9WqF6VU7aEJvQyv3ZBaYZmN+07wQ/rBGohGKaUqpgndm7u43/MLe7ZmRJ+2FW5y3Ws/hTIipZQKmCZ0b+4Cn9mp1w0IUyBKKVV5mtDjvVq5FJa+ybngb+fVXCxKKVUNASV0ERkuIptEJF1EJvhZf46IrBQRl4hcGfwwQyj518XThQWlVie1qF+DwSilVNVVmNBFxAlMBS4BegLXiEjPEsV2AjcC7wU7wJC75PHiB4z8XKGLVDw4tPbEqOqKt37MIGnClxzRJru1UiBX6IOAdGPMNmNMPjADGOVdwBiTYYz5GYi80ZSbtIfev7Om3S6/Rfp1bFruLu796OcgB6VU7fS/ZTsB2Hc8N8yRKH8CSejtgV1e85n2suhRNPBFGQn93ZsHl7v5B2mZJE34MthRKVXruO1vo44AvrmqmlejN0VFZKyIpIlIWlZWVk0eunxFCb3Qf0JvGB/DtocvZc0DF5e7m6wT2mmXim57j1lX5g7N57VSIAl9N+A98kMHe1mlGWOmGWNSjTGpiYmJVdlFaFRwhQ7gcAhN6pXfC+PAh+by3ebiDyq32/Dw7A3sOnwyKGEqFU5f/LyHE7nW/4heoNdOgST05UBXEUkWkThgNDArtGHVME9CL93KpbLGvLGMsx6ZR/qBE7y9JINpi7Yx/r2V1d6vUuE2/r1VnulAGguomldhQjfGuIDxwBxgA/CBMWadiEwWkZEAIjJQRDKBq4BXRGRdKIMOuhz7qvrQ1gqLjj2nS4Vl9hzL5cKnFzHp8/UArMk8xonc6n9YKFVbbN53ItwhKD8CqkM3xsw2xnQzxpxmjHnIXjbRGDPLnl5ujOlgjGlgjGlhjOkVyqCDztiNc2bfU2HRqweWPe5oefpM+qZS5Y/k5OMqjLxGQ6pu+Mu7K/k582i4w1Al6JOiUFzlklPxuKGnJTZk/eRhIQvlwIlc9hw9Rf9/f8t9H/8SsuMoVVWNyWGAbGbkCz+EOxRVgiZ0gLgGxdPbvquweP24GK46o0NIQhn00DzOenQ+AJ//vCckx1CqOt6Me4yP4yfhpDDcoagSNKED9Ppt8fTbIwPapEfbxpU+zL+/WM8lU74nO8/FsZMFHDgR2MMZ27KyyckruwWOUjWpj2wHQDDk5LlImvCl54EjFV6a0AGcJZoj7qy4S9zrzuxU6cO8vng7G/YeZ/n2w5z5yDwGPTTPZ32ey/eKJ99l1aGf/9R33PTm8kofT6lgKXSX7t5CgMwjpwB4cs6mGo5I+aMJ3Z/FT/vO718Hr5zj0196fIyTDZOHV2n32XkuThWU/rr66Srf5v1uA09/Y/2j/LT9cJWOpVQwPDt3s2faO7XPWG5dmR/Svl1qBU3oRWLqFU9v/tp33Utnwd418FyKz+J6cU4yHh1Bm8YJlTrUbf8rbs+789BJCt2G177fxv2frC1V9rn56Z7pvcdO+d3fq4u2sWLHYfYdy2XVziOViqW2O5Sdx/o9x0st/35Llj6wVYO8q1TE89sw/YeMsMSj/IsJdwC1xv174cGm5Zc5usPv4oX3nMfp//ra77qKnPPEAvq0b8Ivu49VWHbII/MZe04XtmXlMKBzU37bvz1DHplfqtyrN6RybrdE4mJKf15/tzmLLi0b0LF57ewW2O029HvwG/5wdjJ3X9SNIY/OJ9/l5ovbzqZ3++K+669/fRlOh7D14UvDGG3dcTC74ivwdXuO0atdkwrLqdDRK/QiJZ98K6pe+eLuCjdNiHXSOKHqn42BJPMi0xZtY+6G/Tz+9Sb+35ML/Zb509tpPDFno991Y95YxgVPVdySJxy+25xFl3/M5kSei+fmbeHpbzd77iM8YdfR5hYUejpC81evq0Kv6KwLvuf/xul6nyfcNKGX5Wt7HI+01wMq7ghDb0W5BWU/ePTq99t95t1u4+m3Pb/EA0ve62rKwew80g9kAzBvw36SJnzJmDeW+ZR5bt4Wv9uVJetEHrl+7k2omlGbOqdbuOkAH6btqrhglNEqF2+t+8B++2GeZa9A98Bver56QyovzE/36Zwr3C54aiFbs3I4p1siizZnMSqlnc96YwzvL9/FhI9/oUOzelwzqBNPzNlE2j8vpGXD+JDE9MBna8nOK+SjlZkALPvHBbz5Y0aF2xWd10dm+//mAVbnaCkdm/LprUMDisXtNjw0ewM3nZ1M+6b1Kix/JCef2BgHDePr1r/N9oM5PvPGrkUveYVemxR9W7gqtfwnu4+ezKdp/biaCKlG1K2/zIr8/iN4qlvx/Du/LV3GlQ8xpf8ABiY1562bBpF1Io+BD831LB/Zrx2z1oTnAaGtWdY/4iI7GX62ujiOG6cvo2/7Jp6brplHTnmqNZZuO8Rlfdux99gp2jaxEt2cdftI6diU1o0TcBW62ZqVw/q9x2jftD6DkpsDMGXuFk7kFvDPy0oOaGXZdfgkby3xvQ9x38e/sGFv6ZueZfnyl70+80XVL89c3Q+A1buO+t1uz9FTrN19jIt7tQGsm6rXv259I/gl8xgfjBtS5jH/MH0ZDRNi+XzNHpo3iGPlvy4KON5I9b9lO7nv41+4/szO1I9z+i1TmxN6IBZtzuKGN5bx9k2DOKdbLer9tRq0ysVbo9YVl3GV/zBQYqN40h+6xDP/3DX9mTI6pZqBBd/CTVk+LWi8jX9vFT9tO8SQR+bz2erd5OS5+PM7K/i/V5YA8NS3mxn27CLuen8N//fKEu56fzUAz8zdzGuLt/vdJ8CSbYdKLZu38UBAN9yAcgcRuev9NZ7pLvd9ydasbA54japz1qPzGfvOCs/8E17tpgvc5feZs2BTFp/bH8qH7eZ5BYVuXlyYHpIqnpU7j7A1Kzvo+y3Lxn3HOWRXZeW5CnG7jafbiXeW7uCVRdv8buevkvGz1bt5Y/F2kiZ8ycn8wB+Gc7sND325nt1H/bfkqq5Ct2H30VOeezIAaTusFmEro6hlmF6hl3TnWni2d9nrH+0Ik8q/iRnjdLDs/gvIs+u4z+zSIpgR1oirpy0F4I4Zqz3Ldhw6yeXPLy51E/eTVbt55uoUz/y/v1hPYqN4xp17mk+5v8+smaH63AbPjd8po1N8XsO36/fzp7fTfMpX9vaB9wfL52v2cvPZyfRo25ie7Sr/9LA/V7z4IwAZj47wLMstKOT+T9Zy7yXdadWocs1kKzL82e9p1Sie+0f04I4Zq3FWcD+o6Mrc3xW697k+lJ1Pbryb7FwXnezB1gsK3ew4lMOvWjXy2W7Rlixe/X47a3Yd48mr+vHn/67gf38aTKOEWNzGUOg2bN5/gr4dmlb69R3MzmPiZ2uZ/cs+hvVqzQ1Dkkg/kO25R/Ps3C3sP57Lw7/tE/HdAku4BjhOTU01aWlpFRcMh0klml7dsQam9Cue//t2qN+8UrtMmvAlqbKRFaYbJgq/GJVMnOCbkKD8K+zaYM6d59C9TXGiuWPGKp9qqkDcfv6vuPvi7uWWMcaw8/BJOjSr75M8i672i5rAep+/j1Zk8tcP13D+6a1448aBlYqpIpV9X9ITrieGQnrlvk4OFd97APjmrnO4+JlFnvlZ44f6JGfvGM7rnsjCTf7vRS297wKa1o9l/d7jnNayIU3q+z7lfexUAa8u2sYLC/x/+6zIWzcN4txyql9e+34b//lyA5v+M5z4GP9VUaEmIiuMMan+1kVfZgmGriWGmmuWBNe8Xzz/eDJ8OxHS5xKojM6PMDN+MtsTfh+cGGuZkskcrH/SlTuP8Ny8LZVKGiMcS2lB4E05g2X8eyvJtvvMWb/neKWTOVgPgnX751d8tno3X/2ylx/SD5Yq80HaLs59YiH9HvzGU9UBViLvMbH4eYZ8l5tHv9rIJVO+568fWlVK8zcW9wi6ad8JdhzyvWFZlrW7j/HiwtJJrjpNPytTh+6dzAFPT43HThWUehiurGQOkLbjMKf/62uuePFH+k3+hpkrMunzwBw+WpHJ9oM5/OmttConc7Ca9T4xZyPvLN3BidwCCgrdHDtVPJbBiwutMRPeX+7bgibPVcioFxaTlnGY9AMn+PJn33s9NUWv0MtyeDvEN4YGdnWJuxAm+7kqT2gCf0v3e6PUo9AF/y6udunLBxzPdfGf3/SmY/P6pZrrzb37XHLyXIyaWve6J23BMVYk/IUcE0+vvOme5U05wS0xs3jCdTUFIa4pHJzcPKhdLWz893BW7jzCut3H6dWuMR+t3O1p5QPWlbj3TdpA3Hx2sud+xYU9WjNldAoN7NY3W/afoG3Tej6tcYo+UEt+a3roy/WlmrhWZHv9GxC3iz65r3GCqj+gtvJfFzHg399WeftQ+1Wrhp6mtf/942DO7tqy1IVJvw5NOLtrS0b0acelz33P6W0asdEe/KNLYgO+vuMc1u45xqn8QtIPZHNZ37Y0iI8hIbbqV/flXaFrQq+MklUxPuvKuaJc/xl8cEPxfIkqmwPHc3n6281MHtXb5+nOa6YtZUDnpvRp35Rx/11BXdCeLH5IuAOApNz3PMs/j/sHfRwZ3JF/C5+5zw5XeEFRL9bp05fP8F5t+HrdvmrtUwRmjhvC12v38er32+nTvglTRqdwOCefdk3rebpkznh0BFe8+AMrdx6t0nGW3X8BrZ5pD24XjxdczYuFo6oVd7T412U9+fcX6wMuX/KDtTI0oQeL2w2TmwVe/rTzYch4+O8VvstvWQqtelTq0DsO5XDuEwsB+HDcELJO5HHLu9E3Vmkn2c+i+LuA4oTehGzWJIwF4BXXCB5xXRe2+CLd1GsHcGs1xrjNeHQEPNgcjPWBlJT7ns9VqQpMqBK61qFXhsNhXYk/cBQ6nllx+a3zSydzgBe9tj2wAV4937r6f/OyMptcdG7RgB8mnM9rN6QyMKk5l/Zpy0vXDaja66jFYilu6tZddnKWY60nmQP8OaZ231it7aqTzD1M8beLppzgs/GBPcilihX1ohpseoVeHfk5MP8hWDo1sPIDb4blr1Vc7h97fEdRKsfx3AJ2HT5Jr3ZNSD+QzYxlOz0P9mw/mMOJ3IKIGSps6K9aMK5TJr/+8Y/llnu49dNM29GmhqJSRcYM6cyDo3r7bQWW27BjlTuoq4vO7NKcGWPLfpitPNWuchGR4cAUwAm8Zox5tMT6eOBt4AzgEHC1MSajvH1GRUL3dvIw/Pg8LJkKhX76tLh7AzRqW3GPjkUm7LRuyh7JgKadoDAfxAEx8dZV/OFt0OK0CncDcDLfRW6Bm8YJMcQ4rS9lN7+1nLkbKh5DtSZ0kCwmxrzNube/RvzU/hVv0LwLr/abycAuLfhs9e6QduEquKOymWlVeKoJ/N1LSvo1/+v5Ii0bxpdq519TGnGSGFwcITjPA4RaVatdqpXQRcQJbAYuAjKB5cA1xpj1XmVuAfoaY8aJyGjgt8aYq8vbb9QldH/cbsg/AXENwWHf1c4/CQ+3DW9cZXjZdTkHG/Xgnxd2gL2r+XZvPXJNLGce/wpX9kES6zlYlN2e+pLHy67L+U2nXHISWvP/Osbw9ertLDjUlKZk01qOcJlzCXnEsc6dRCEONro78bPpwrgrLmZEvw7s+WQiXTa8GNbXO7FgDAXE8Afn13Rz7K54A2CK67ccNQ3Zb5rR+fQz+GT9cRLjXeTmF9CIk2wwnYjFRR5x5BGL/+cpI8/z1/Tn8j5t4PkBcCTwVjGFsQ1wFlhNK/fFdeLpnGEMdmzguGlAIznFRndHtpgO7DfNiKOAIzTkcof1txPfsBlb3W3Iyi7gCI1oxgmSZZ/nHO8zzenj2E432cXfYz/wOe4yd3f2mhbsNS1o3KEHnQp3sXxPPoeTL2dOeg5NJZsOzqNsK2zJ2Y61/CfWalH1r4Ib+bpwILnEY7DePQPkkBD0D/ZwJfQhwCRjzDB7/j4AY8wjXmXm2GWWiEgMsA9INOXsvE4k9PK43dZVt3FDnFfTr/euLj3ARrQRh/W6y3PVW/DhmOL521ZC8y6Bf8OpRQqIId84KSAGh912243gwOCkkBjc1m+xzkmOiacQJ4LBgRsHBgfG820hjxivDrIgnnxOUJ9CHLjs7dx28jHGt7W4Qeyf4ukiDeOdtKgfYy0xLpzuAgpd+bhd+cS6/LR3v3cHPNY5+Cesljpl4nAjuLDeSzcOCnHgxE0hDhy4EcBJoaeNfiEOfD/UDS5i+Ln7HQy79s4qxVFeQg+kQW97wLsVfSYwuKwyxhiXiBwDWgA+T1WIyFhgLECnTpUfkzOqOBzg8PMI97Xvl15WHrcbTh60kmR+DuxZZSVLVx7sWAzZWVCvKTRqA44Y2L/eKt+4PbTqaZX9zq5Bi28CeSWaXw68GXKyYPMc335s4htDnt2pVrNk66rt9MusMt4PXDXvYlUPFWnUDvqNLh7mL6EpjF1gLXfGQu6x4iadvfw0BX3gKOQehR1L4NO/WNMAbfrAPrunzIat4dTR4qqvdv2h01nWb+OGFdNh5xKrCqx5F9hh32No0RUOpUPPkdb04D9Dg0TIOwGLnoCMxVZ8rjw4ngl9R0P2fmtf7fpbH9BNOkBeNnQcbN08NIZYdwGxrnxwFwBitTF0F1q/nXHF8648yN5P/ZgEaNASEae1ThzFP24X8UUXAghHDh/ATT4J9VpQz2k4fOIksU4HjRJiOH4qnz1HT9IoPpZDOXkkNozDIeAQqBfrwCnWU6uN4p14ko7DnnbEgDMWpzMWpzPOel9PHrKa4LZPhes/gYTGvs11C05Z94i++Sd0HwG7V0DjtnAs09pfQhMYcit8+VdrhLCEJtDlXOt9KDgJp45A2xT46h5IPhf2rrbOd9NO0PM3EN/IOn7uMWjYyip/6iicOmy9F90uBkcsfHB9cUzNkqxjZW2w5s+5x3p/HE5Y8oLv32hcQ+g81Hpd9ZpDzgHrXLTpDa586hWctOIsyIWYOPILCsC4iYtxgnGT74YYpxOHw2m9Vwi4XRhjcGO1QCl0GzCFDEs5o/TfdhAEcoV+JTDcGHOzPX89MNgYM96rzFq7TKY9v9UuU/oxOVudv0JXSqkqqG6zxd2Ad6fCHexlfsvYVS5NsG6OKqWUqiGBJPTlQFcRSRaROGA0MKtEmVlAUYXnlcD88urPlVJKBV+Fdeh2nfh4YA5Ws8U3jDHrRGQykGaMmQW8DrwjIunAYaykr5RSqgYF1MuRMWY2MLvEsole07nAVcENTSmlVGXoExNKKRUlNKErpVSU0ISulFJRQhO6UkpFibD1tigiWcCOKm7ekhJPodZiGmvwRUqcoLGGSqTEGoo4Oxtj/A58GraEXh0iklbWk1K1jcYafJESJ2isoRIpsdZ0nFrlopRSUUITulJKRYlITejTwh1AJWiswRcpcYLGGiqREmuNxhmRdehKKaVKi9QrdKWUUiVoQldKqSgRcQldRIaLyCYRSReRCWE4fkcRWSAi60VknYjcYS9vLiLfisgW+3cze7mIyHN2vD+LyACvfY2xy28RkTFlHTMIMTtFZJWIfGHPJ4vIT3ZM79vdIiMi8fZ8ur0+yWsf99nLN4nIsBDF2VREZorIRhHZICJDauN5FZG77Pd+rYj8T0QSass5FZE3ROSAPehM0bKgnUMROUNEfrG3eU5EqjxoahmxPmG//z+LyCci0tRrnd/zVVZOKOs9CVasXuv+KiJGRFra8+E7r8aYiPnB6r53K9AFiAPWAD1rOIa2wAB7uhHWANo9gceBCfbyCcBj9vSlwFdYY3ydCfxkL28ObLN/N7Onm4Uo5ruB94Av7PkPgNH29MvAX+zpW4CX7enRwPv2dE/7XMcDyfZ74AxBnG8BN9vTcUDT2nZesYZb3A7U8zqXN9aWcwqcAwwA1notC9o5BJbZZcXe9pIgx3oxEGNPP+YVq9/zRTk5oaz3JFix2ss7YnUtvgNoGe7zGvTkEcofYAgwx2v+PuC+MMf0GXARsAloay9rC2yyp18BrvEqv8lefw3witdyn3JBjK8DMA84H/jC/oM56PVP4zmn9h/mEHs6xi4nJc+zd7kgxtkEK1FKieW16rxSPH5uc/scfQEMq03nFEjCN0kG5Rza6zZ6LfcpF4xYS6z7LfCuPe33fFFGTijv7zyYsQIzgX5ABsUJPWznNdKqXPwNWN0+TLFgf33uD/wEtDbG7LVX7QNa29NlxVxTr+VZ4O+A255vARw1xrj8HNdnsG+gaLDvmog1GcgCpotVPfSaiDSglp1XY8xu4ElgJ7AX6xytoHae0yLBOoft7emSy0PlJqyrVSqIyd/y8v7Og0JERgG7jTFrSqwK23mNtIRea4hIQ+Aj4E5jzHHvdcb6mA17e1ARuQw4YIxZEe5YAhCD9ZX2JWNMfyAHq3rAozacV7v+eRTWB1A7oAEwPJwxVUZtOIeBEJH7ARfwbrhj8UdE6gP/ACZWVLYmRVpCD2TA6pATkVisZP6uMeZje/F+EWlrr28LHLCXlxVzTbyWocBIEckAZmBVu0wBmoo1mHfJ45Y12HdNxJoJZBpjfrLnZ2Il+Np2Xi8EthtjsowxBcDHWOe5Np7TIsE6h7vt6ZDGLCI3ApcB19kfQFWJ9RBlvyfBcBrWh/oa+/+rA7BSRNpUIdbgnddg1NnV1A/WVdw2+0QW3QDpVcMxCPA28GyJ5U/ge+PpcXt6BL43SJbZy5tj1Rk3s3+2A81DGPd5FN8U/RDfm0W32NO34nsD7wN7uhe+N6S2EZqbot8D3e3pSfY5rVXnFRgMrAPq28d+C7itNp1TStehB+0cUvrm3aVBjnU4sB5ILFHO7/minJxQ1nsSrFhLrMuguA49bOc1JMkjlD9Yd5A3Y93Zvj8Mxz8b6yvrz8Bq++dSrDq7ecAWYK7XGyXAVDveX4BUr33dBKTbP38IcdznUZzQu9h/QOn2H328vTzBnk+313fx2v5++zVsohotGyqIMQVIs8/tp/Yffa07r8CDwEZgLfCOnWRqxTkF/odVt1+A9a3nj8E8h0Cq/bq3Ai9Q4iZ2EGJNx6pnLvrfermi80UZOaGs9yRYsZZYn0FxQg/bedVH/5VSKkpEWh26UkqpMmhCV0qpKKEJXSmlooQmdKWUihKa0JVSKkpoQldKqSihCV0ppaLE/wfJaC5DJk38twAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots()\n",
    "plt.plot(discriminator_loss, label='Discriminator')\n",
    "plt.plot(gan_loss, label='Generator')\n",
    "plt.title(\"Training Losses\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss convergence towards the end signifies that the GAN model has reached optimality"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 Anomaly Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the result of all the training we did?  \n",
    "\n",
    "We now have a generator that can input a random seed value and produce an output that closely resembles the data it was trained on. You can read about how this strategy was used to come up with imaginary celebrities <a href =\"https://research.nvidia.com/publication/2017-10_Progressive-Growing-of\" >here</a>\n",
    "\n",
    "The Discriminator that we trained ended up being a very powerful classifier that can tell if a sample point is representative of the true data distribution it was trained on or not and hence can be used for Anomaly Detection!\n",
    "\n",
    "Let us feed our test data into the discriminator and obtain scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predictions on the test set\n",
    "\n",
    "nr_batches_test = np.ceil(x_test.shape[0] // batch_size).astype(np.int32)\n",
    "\n",
    "results =[]\n",
    "\n",
    "for t in range(nr_batches_test +1):    \n",
    "        ran_from = t * batch_size\n",
    "        ran_to = (t + 1) * batch_size\n",
    "        image_batch = x_test[ran_from:ran_to]             \n",
    "        tmp_rslt = discriminator.predict(x=image_batch,batch_size=128,verbose=0)        \n",
    "        results = np.append(results, tmp_rslt)   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us calculate the mean score for normal and anomalous samples in our test set. Ideally, we would like to see a score close to 1 for normal samples and 0 for anomalous samples. This would mean our classifier is doing well in distinguishing between the 2 classes.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean score for normal packets : 0.9988151310747622\n",
      "Mean score for anomalous packets : 0.08214563426548624\n"
     ]
    }
   ],
   "source": [
    "pd.options.display.float_format = '{:20,.7f}'.format\n",
    "results_df = pd.concat([pd.DataFrame(results),pd.DataFrame(y_test)], axis=1)\n",
    "results_df.columns = ['results','y_test']\n",
    "print ('Mean score for normal packets :', results_df.loc[results_df['y_test'] == 0, 'results'].mean() )\n",
    "print ('Mean score for anomalous packets :', results_df.loc[results_df['y_test'] == 1, 'results'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But how exaclty do we identify our Anomalies?\n",
    "\n",
    "Although there are several ways to do this, let us use a more straight forward way for detection. Remember 1% of our test set comprised of anomalies. So, the lowest 1% of the scores should ideally constitute anomalies. Let us test our hypothesis below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Obtaining the lowest 1% score\n",
    "per = np.percentile(results,1)\n",
    "y_pred = results.copy()\n",
    "y_pred = np.array(y_pred)\n",
    "\n",
    "#Thresholding based on the score\n",
    "inds = (y_pred > per)\n",
    "inds_comp = (y_pred <= per)\n",
    "y_pred[inds] = 0\n",
    "y_pred[inds_comp] = 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Accuracy, Precision and Recall values of our model on the test set are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Score : 0.9987175676941052\n",
      "Precision : 0.9377289377289377\n",
      "Recall : 0.9343065693430657\n",
      "F1 : 0.9360146252285192\n"
     ]
    }
   ],
   "source": [
    "precision, recall, f1,_ = precision_recall_fscore_support(y_test,y_pred,average='binary')\n",
    "print ('Accuracy Score :',accuracy_score(y_test, y_pred) )\n",
    "print ('Precision :',precision )\n",
    "print ('Recall :',recall )\n",
    "print ('F1 :',f1 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Confusion Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix is a good way to understand if our predictions agree with the target labels. You've implemented one of these in both of the previous labs, so in this section, you'll get a chance to build your own. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix :\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAALICAYAAACaS/2BAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABB/UlEQVR4nO3deZxkVXn/8c932FFRYARGNlFxwYVtHHCJwQ3BXyLk9zMCakQDQY1LEmPiGjEohpjFhKjIqChoBI1RmQQUCWoQdZABEQWFGXGbARx2RBAYeH5/1G0omu6eHujbVdX38+ZVr64699x7T92uHp5++rnnpKqQJEmSumjeoAcgSZIkDYrBsCRJkjrLYFiSJEmdZTAsSZKkzjIYliRJUmetP+gBSJIkaWqZv3Fx+12DHkbPr+84o6r2G/QwZorBsCRJ0rC7/S7Ya6tBj6Lnf1bNH/QQZpJlEpIkSeosg2FJkiR1lmUSkiRJoyAZ9AjmJDPDkiRJ6iwzw5IkScMumMJsiZdVkiRJnWUwLEmSpM6yTEKSJGkUeANdK8wMS5IkqbMMhiVJktRZlklIkiSNAqskWmFmWJIkSZ1lMCxJkqTOskxCkiRp6MXZJFpiZliSJEmdZWZYkiRp2Lkcc2u8rJIkSeosg2FJkiR1lmUSkiRJo8Ab6FphZliSJEmdZTAsSZKkzrJMQpIkaRRYJdEKM8OSJEnqLDPDkiRJwy7APFPDbTAzLEmSpM4yGJYkSVJnWSYhSZI0CqySaIWZYUmSJHWWwbAkSZI6yzIJSZKkUeByzK0wMyxJkqTOMhiWJElSZ1kmIUmSNAqskmiFmWFJkiR1lplhSZKkYedyzK0xMyxJkqTOMhiWJElSZ1kmIUmSNAqskmiFmWFJkiR1lsGwJEmSOssyCUmSpKEXl2NuiZlhSZIkdZbBsCRJkjrLMglJkqRh56IbrTEzLEmSpM4yMyxJkjQKTAy3wsywJEmSOstgWJIkSZ1lmYQkSdIocJ7hVpgZliRJUmcZDEuSJKmzLJOQJEkaBVZJtMLMsCRJkjrLzLAkSdKwcwW61pgZliRJUmcZDEuSJKmzLJOQJEkaBVZJtMLMsCRJkjrLYFiSJEmdZZmEJEnSKHA55laYGZYkSVJnGQxLkiSpsyyTkCRJGgWmMFvhZZUkSVJnmRmWJEkadok30LXEzLAkSZI6y2BYkiRJnWWZhCRJ0iiwSqIVZoYlDa0kmyT5ryQ3JvmPB3CclyX56kyObVCS/E6SSwc9DkmaKwyGJT1gSV6aZFmSm5NcmeTLSZ45A4d+MbA1sGVV/eH9PUhV/XtV7TsD42lVkkrymKn6VNU3q+pxszUmSZrrLJOQ9IAkeRPwVuA1wBnA7cB+wAHAOQ/w8DsCl1XVmgd4nDkhyfpeC6nDnE2iFWaGJd1vSR4KHAW8rqq+UFW/qao7quq/quqvmj4bJfmXJFc0j39JslGzbZ8kK5P8ZZLVTVb5Vc22vwXeBRzUZJwPS/LuJJ/uO/8jm2zq+s3rVya5PMmvk/w0ycv62s/p2+/pSc5ryi/OS/L0vm3fSPKeJN9qjvPVJPMnef9j4//rvvEfmOSFSS5Lcl2St/f1X5TkO0luaPp+MMmGzbazm27fb97vQX3Hf0uSq4BPjLU1+zy6OccezetHJLk6yT4P5PsqSV1iMCzpgXgasDHwxSn6vAPYG9gN2BVYBLyzb/s2wEOBbYHDgA8l2byqjgTeB3y2qh5cVR+faiBJHgQcC+xfVQ8Bng5cOEG/LYDTmr5bAv8MnJZky75uLwVeBWwFbAi8eYpTb0PvGmxLL3j/KPByYE/gd4C/SbJT0/dO4C+A+fSu3XOBPwWoqmc1fXZt3u9n+46/Bb0s+RH9J66qnwBvAT6dZFPgE8CJVfWNKcYrSepjMCzpgdgSuGYtf7p/GXBUVa2uqquBvwX+qG/7Hc32O6rqdOBm4P7WxN4FPCnJJlV1ZVVdPEGf/wMsr6pPVdWaqjoZ+DHw+319PlFVl1XVrcDn6AXyk7kDOLqq7gBOoRfo/mtV/bo5/yX0fgmgqs6vqqXNeX8GHA/87jTe05FVdVsznnupqo8CK4BzgQX0fvmQNBfNG5LHHDMH35KkWXQtMH+sTGESjwB+3vf6503b3ccYF0zfAjx4XQdSVb8BDqJXu3xlktOSPH4a4xkb07Z9r69ah/FcW1V3Ns/HgtVf9W2/dWz/JI9N8t9JrkpyE73M94QlGH2urqrfrqXPR4EnAf9WVbetpa8kqY/BsKQH4jvAbcCBU/S5gt6f+Mfs0LTdH78BNu17vU3/xqo6o6qeTy9D+mN6QeLaxjM2plX3c0zr4jh649q5qjYD3s7aZw6tqTYmeTDwL8DHgXc3ZSCS5ppwz5LMg37MMQbDku63qrqRXp3sh5obxzZNskGS/ZO8v+l2MvDOJA9vbkR7F/DpyY65FhcCz0qyQ3Pz3tvGNiTZOskBTe3wbfTKLe6a4BinA49tpoNbP8lBwC7Af9/PMa2LhwA3ATc3WevXjtv+K+BR63jMfwWWVdXh9GqhP/KARylJHWIwLOkBqap/At5E76a4q4FfAq8HvtR0eS+wDLgI+AFwQdN2f851JvDZ5ljnc+8Adl4zjiuA6+jV4o4PNqmqa4HfA/6SXpnHXwO/V1XX3J8xraM307s579f0stafHbf93cCJzWwTL1nbwZIcQG8au7H3+SZgj7FZNCRJa5eqKf8CJ0mSpAHLVpsUL17XPxy15LhLzq+qhYMexkwxMyxJkqTOGkgwnGSLJGcmWd583XySfncmubB5LOlr3ynJuUlWJPns2KT1kiRJ0roYVGb4rcBZVbUzcFbzeiK3VtVuzeNFfe1/D3ygqh4DXE9von5JkqS5a16G4zHHDCoYPgA4sXl+IlNPy3QvSQI8B/j8/dlfkiRJGjPVRPlt2rqqrmyeXwVsPUm/jZMsA9YAx1TVl+iteHVD3yT9K7n3ZPn3kuQIxpYwXS97sumg3rKkYbbHY5806CFIGlI//9kvuOaaawafEp2Dc/wOg9YiwyT/w7gJ8Rv3Wiq0qirJZFNa7FhVq5I8Cvhakh8AN67LOKpqMbAYIJttWOy11brsLqkjvvWVcwY9BElD6hl7PXPQQ1CLWguGq+p5k21L8qskC6rqyiQLgNWTHGNV8/XyJN8Adgf+E3hYkvWb7PB2zM7KUZIkSZpjBlUzvAQ4tHl+KHDq+A5JNk+yUfN8PvAM4JLqTYz8deDFU+0vSZI0Z2SIHnPMoILhY4DnJ1kOPK95TZKFST7W9HkCsCzJ9+kFv8dU1SXNtrcAb0qygl4N8cdndfSSJEmaEwZyN1mzHOpzJ2hfBhzePP828ORJ9r8cWNTmGCVJkjT3ObWCJEnS0AsZktkkJpv1YFS5HLMkSZI6y2BYkiRJnWWZhCRJ0giwTKIdZoYlSZLUWWaGJUmSRsCQJIbnHDPDkiRJ6iyDYUmSJHWWZRKSJElDLsC8IamTuHPQA5hhZoYlSZLUWQbDkiRJ6izLJCRJkoZdhmee4bnGzLAkSZI6y2BYkiRJnWWZhCRJ0giwTKIdZoYlSZLUWWaGJUmShl7MDLfEzLAkSZI6y2BYkiRJnWWZhCRJ0giwSqIdZoYlSZLUWQbDkiRJ6izLJCRJkoZccJ7htpgZliRJUmeZGZYkSRp2MTPcFjPDkiRJ6iyDYUmSJHWWZRKSJEkjIFgm0QYzw5IkSeosg2FJkiR1lsGwJEnSCEgyFI9pjHP7JF9PckmSi5P8WdO+RZIzkyxvvm7etCfJsUlWJLkoyR59xzq06b88yaF97Xsm+UGzz7FpBjbZOaZiMCxJkqSZtAb4y6raBdgbeF2SXYC3AmdV1c7AWc1rgP2BnZvHEcBx0AtsgSOBvYBFwJF9we1xwJ/07bdf0z7ZOSZlMCxJkqQZU1VXVtUFzfNfAz8CtgUOAE5sup0IHNg8PwA4qXqWAg9LsgB4AXBmVV1XVdcDZwL7Nds2q6qlVVXASeOONdE5JuVsEpIkSSNgiNbcmJ9kWd/rxVW1eKKOSR4J7A6cC2xdVVc2m64Ctm6ebwv8sm+3lU3bVO0rJ2hninNMymBYkiRJ6+Kaqlq4tk5JHgz8J/DnVXVTf71xVVWSanGM0z6HwbAkSdKQC2HeEKWG1ybJBvQC4X+vqi80zb9KsqCqrmxKHVY37auA7ft2365pWwXsM679G037dhP0n+ock7JmWJIkSTOmmdnh48CPquqf+zYtAcZmhDgUOLWv/RXNrBJ7Azc2pQ5nAPsm2by5cW5f4Ixm201J9m7O9Ypxx5roHJMyMyxJkqSZ9Azgj4AfJLmwaXs7cAzwuSSHAT8HXtJsOx14IbACuAV4FUBVXZfkPcB5Tb+jquq65vmfAp8ENgG+3DyY4hyTMhiWJEkaAdOZ43cYVNU5MOna0c+doH8Br5vkWCcAJ0zQvgx40gTt1050jqlYJiFJkqTOMhiWJElSZ1kmIUmSNOwyOmUSo8bMsCRJkjrLYFiSJEmdZZmEJEnSCLBKoh1mhiVJktRZZoYlSZKGXPAGuraYGZYkSVJnGQxLkiSpsyyTkCRJGgGWSbTDzLAkSZI6y2BYkiRJnWWZhCRJ0tCLZRItMTMsSZKkzjIzLEmSNOziDXRtMTMsSZKkzjIYliRJUmcNJBhOskWSM5Msb75uPkGf3ZJ8J8nFSS5KclDftk8m+WmSC5vHbrP6BiRJkmZZMhyPuWZQmeG3AmdV1c7AWc3r8W4BXlFVTwT2A/4lycP6tv9VVe3WPC5se8CSJEmaewYVDB8AnNg8PxE4cHyHqrqsqpY3z68AVgMPn60BSpIkae4bVDC8dVVd2Ty/Cth6qs5JFgEbAj/paz66KZ/4QJKNptj3iCTLkizjjrse8MAlSZJmW+jNJjEMj7mmtanVkvwPsM0Em97R/6KqKklNcZwFwKeAQ6tqLJp9G70gekNgMfAW4KiJ9q+qxU0fstmGk55HkiRJ3dNaMFxVz5tsW5JfJVlQVVc2we7qSfptBpwGvKOqlvYdeyyrfFuSTwBvnsGhS5IkqSMGVSaxBDi0eX4ocOr4Dkk2BL4InFRVnx+3bUHzNfTqjX/Y5mAlSZIGbdDlEXO1TGJQwfAxwPOTLAee17wmycIkH2v6vAR4FvDKCaZQ+/ckPwB+AMwH3juro5ckSdKcMJDlmKvqWuC5E7QvAw5vnn8a+PQk+z+n1QFKkiQNmXlzMCs7DFyBTpIkSZ1lMCxJkqTOGkiZhCRJktbBHF0KeRiYGZYkSVJnGQxLkiSpsyyTkCRJGnJhbs7xOwzMDEuSJKmzDIYlSZLUWZZJSJIkjYBgmUQbzAxLkiSps8wMS5IkjQBvoGuHmWFJkiR1lsGwJEmSOssyCUmSpBFgmUQ7zAxLkiSpswyGJUmS1FmWSUiSJI0AqyTaYWZYkiRJnWVmWJIkacgl3kDXFjPDkiRJ6iyDYUmSJHWWZRKSJElDL5ZJtMTMsCRJkjrLYFiSJEmdZZmEJEnSCLBMoh1mhiVJktRZBsOSJEnqLMskJEmSRoBVEu0wMyxJkqTOMjMsSZI0AryBrh1mhiVJktRZBsOSJEnqLMskJEmShlximURbzAxLkiSpswyGJUmS1FmWSUiSJI0AyyTaYWZYkiRJnWUwLEmSpM6yTEKSJGkEWCXRDjPDkiRJ6iwzw5IkSUMv3kDXEjPDkiRJ6iyDYUmSJHWWZRKSJEkjwDKJdpgZliRJUmcZDEuSJKmzLJOQJEkacollEm0xMyxJkqTOMhiWJElSZ1kmIUmSNAKskmiHmWFJkiR1lplhSZKkEeANdO0wMyxJkqTOMhiWJElSZ1kmIUmSNAosk2iFmWFJkiR1lsGwRs9v18D5V8N3ftV7/OLme2//+a/hf1bB7Xf2Xq++FZb+CpauhnNXww233dP3it/At67qPa74zT3tN93eO/a3roJLb4CqXvuvb4fvrr7nWDfe3upbldSOVx/+GnZYsCN77rrw7rb3/u3RPGqHx7DXnnuz155785XTvwLAed9ddnfboj324tQvLRnUsCW1YKDBcJL9klyaZEWSt06wfaMkn222n5vkkX3b3ta0X5rkBbM6cA1WAjs/FJ62NTz14bDyZrj5jt62366Ba2+Djde7p/8WG8FeW8HeW8Eum8MlN/Ta77gLfvprWLRV7/HTX/faAH58Q6/v07eGW5pjAiy/CR61We9Yj94Mlt84W+9a0gz6o1e8nFNP+9J92t/wZ6/n3POXcu75S9nvhfsB8MQn7cK3zj2Hc89fyqmnfYk3vPYNrFmzZpZHLIVkOB5zzcCC4STrAR8C9gd2AQ5Jssu4bocB11fVY4APAH/f7LsLcDDwRGA/4MPN8dQFG60Hm23Ye77+PNh0A7ityQJfdmMvUO63/rx76qzuvOue9mt/2wuUN5jXe2yxUa/ttjthTcFDN+ztt2BTuPrWe/Zbc9c9XzfyYyeNomc+65lsscUW0+q76aabsv76vVtsbvvtbXMyGJC6bJCZ4UXAiqq6vKpuB04BDhjX5wDgxOb554Hnpvev0AHAKVV1W1X9FFjRHE9dc+sa+PUdvcB19a294PQhG9y33+pb4du/gguvhV0e1mu77c57B7Mbrddru+3Oe2eWx9oBHvfQXnb4m1f1ssKP2ay1tyZp9n3kw8fz1N0X8erDX8P1119/d/t3zz2PPZ6ykIW7LeLYDx97d3AszZr08jPD8JhrBhkMbwv8su/1yqZtwj5VtQa4EdhymvsCkOSIJMuSLLv7T+CaG9bcBRdd1wtQA/zs173ShYlstUmv5GHXLeHyX9//c678DTz2ofA72/S+/uiG+38sSUPlT15zOJdc9kPOPX8p22yzDW/9q7fdvW3RXk/lgouWcc7Ss/mHY/6R3/72twMcqaSZNOdvoKuqxVW1sKoWssGcf7vdcVf1AuFtNukFurfe2XssXQ3nXNXL5J579T0Z3TGbb9TLJt9+570zvnBPpnij9eC3E7QDXHELbLVx7/lWm3gDnTSHbL311qy33nrMmzePPz78VSw7b9l9+jz+CY/nwQ9+EBf/8JIBjFBSGwYZHa4Ctu97vV3TNmGfJOsDDwWunea+mquq4JLr4UHrw44P6bU9eAP43QXwzG16j43Wg70e3vt6y5p7ZoO46fZeIL3BPNhy496NcXfc1Xtce1uvbaP1YP30At0quPIWePgmvf03Wg+ubwLg62+DTf1TqTRXXHnllXc/P/VLS9jliU8E4Gc//dndN8z9/Oe/4NJLL2PHR+4wkDGquwIDv3Furt5AN8j/k58H7JxkJ3qB7MHAS8f1WQIcCnwHeDHwtaqqJEuAzyT5Z+ARwM7Ad2dt5BqsG2+Hq26FB6/fywRDr3Z3/sYT9199ay+gTWA94Mlb9J5vENjpIb2p0gAe9RDu/uvB4x8GF1/fC5y33Bi23KjXvsvD4NIbe0HyvMATHtbe+5TUmle87FC++b/f5JprruXRO+7M3xz5Ts7+37O56PsXkYQdd9yRfzvuWAC+/a1v84/v/2c22GB95s2bx79+8F+YP3/+gN+BpJmSGsuYDeLkyQuBf6EXopxQVUcnOQpYVlVLkmwMfArYHbgOOLiqLm/2fQfwx8Aa4M+r6strPd9mGxZ7bdXOm5E00m79ymWDHoKkIfWMvZ7J+csuGGhK9EGP3Lwe/85nD3IId7vgT754flUtXHvP0TDQv/FW1enA6ePa3tX3/LfAH06y79HA0a0OUJIkaUjMxRKFYeAdZZIkSeosg2FJkiR1lrfCS5IkjQDLJNphZliSJEmdZWZYkiRpBJgYboeZYUmSJHWWwbAkSZI6yzIJSZKkYTdHl0IeBmaGJUmS1FkGw5IkSeosyyQkSZKGXHCe4baYGZYkSVJnGQxLkiSpsyyTkCRJGgGWSbTDzLAkSZI6y8ywJEnSCDAz3A4zw5IkSeosg2FJkiR1lmUSkiRJwy5glUQ7zAxLkiSpswyGJUmS1FmWSUiSJI0AZ5Noh5lhSZIkdZaZYUmSpCEXYma4JWaGJUmS1FkGw5IkSeosyyQkSZJGgGUS7TAzLEmSpM4yGJYkSVJnWSYhSZI0AqySaIeZYUmSJHWWwbAkSZI6yzIJSZKkYRdnk2iLmWFJkiR1lplhSZKkUWBmuBVmhiVJktRZBsOSJEnqLMskJEmSRoA30LXDzLAkSZI6y2BYkiRJnWWZhCRJ0pALMM8qiVaYGZYkSVJnGQxLkiRpRiU5IcnqJD/sa3t3klVJLmweL+zb9rYkK5JcmuQFfe37NW0rkry1r32nJOc27Z9NsmHTvlHzekWz/ZFrG6vBsCRJ0tALyXA8pumTwH4TtH+gqnZrHqcDJNkFOBh4YrPPh5Osl2Q94EPA/sAuwCFNX4C/b471GOB64LCm/TDg+qb9A02/KRkMS5IkaUZV1dnAddPsfgBwSlXdVlU/BVYAi5rHiqq6vKpuB04BDkgvIn8O8Plm/xOBA/uOdWLz/PPAc7OWCN5gWJIkadgF5iVD8QDmJ1nW9zhiHd7J65Nc1JRRbN60bQv8sq/PyqZtsvYtgRuqas249nsdq9l+Y9N/UgbDkiRJWhfXVNXCvsfiae53HPBoYDfgSuCf2hrgujAYliRJUuuq6ldVdWdV3QV8lF4ZBMAqYPu+rts1bZO1Xws8LMn649rvdaxm+0Ob/pMyGJYkSRpygYHfOLeON9Dd9z0kC/pe/gEwNtPEEuDgZiaInYCdge8C5wE7NzNHbEjvJrslVVXA14EXN/sfCpzad6xDm+cvBr7W9J+Ui25IkiRpRiU5GdiHXn3xSuBIYJ8kuwEF/Ax4NUBVXZzkc8AlwBrgdVV1Z3Oc1wNnAOsBJ1TVxc0p3gKckuS9wPeAjzftHwc+lWQFvRv4Dl7bWA2GJUmSNKOq6pAJmj8+QdtY/6OBoydoPx04fYL2y7mnzKK//bfAH67LWA2GJUmSRoC1re3wukqSJKmzzAxLkiSNgHkP4OY1Tc7MsCRJkjrLYFiSJEmdZZmEJEnSkBubZ1gzb6CZ4ST7Jbk0yYokb51g+5uSXNKsYX1Wkh37tt2Z5MLmsWR2Ry5JkqS5YGCZ4STrAR8Cng+sBM5LsqSqLunr9j1gYVXdkuS1wPuBg5ptt1bVbrM5ZkmSJM0tgyyTWASsaCZNJskpwAH0Vh8BoKq+3td/KfDyWR2hJEnSUIizSbRkkGUS2wK/7Hu9smmbzGHAl/teb5xkWZKlSQ5sYXySJEma40biBrokLwcWAr/b17xjVa1K8ijga0l+UFU/mWDfI4AjANh4vdkYriRJkkbEIIPhVcD2fa+3a9ruJcnzgHcAv1tVt421V9Wq5uvlSb4B7A7cJxiuqsXAYoBstmHN4PglSZJmR5xNoi2DLJM4D9g5yU5JNgQOBu41K0SS3YHjgRdV1eq+9s2TbNQ8nw88g75aY0mSJGk6BpYZrqo1SV4PnAGsB5xQVRcnOQpYVlVLgH8AHgz8R/Pb0C+q6kXAE4Djk9xFL6A/ZtwsFJIkSXNGcKW0tgy0ZriqTgdOH9f2rr7nz5tkv28DT253dJIkSZrr/CVDkiRJnTUSs0lIkiR1nfMMt8PMsCRJkjrLYFiSJEmdZZmEJEnSCHCe4XaYGZYkSVJnGQxLkiSpsyyTkCRJGnLB2STaYmZYkiRJnWVmWJIkaQSYF26HmWFJkiR1lsGwJEmSOssyCUmSpKEXb6BriZlhSZIkdZbBsCRJkjrLMglJkqQhlzjPcFvMDEuSJKmzzAxLkiSNgJgZboWZYUmSJHWWwbAkSZI6yzIJSZKkEeANdO0wMyxJkqTOMhiWJElSZ1kmIUmSNOTSPDTzzAxLkiSpswyGJUmS1FmWSUiSJI0AZ5Noh5lhSZIkdZaZYUmSpKEXM8MtMTMsSZKkzjIYliRJUmdZJiFJkjTkEohlEq0wMyxJkqTOMhiWJElSZ1kmIUmSNAKcTaIdZoYlSZLUWQbDkiRJ6izLJCRJkkaARRLtMDMsSZKkzpo0M5zk34CabHtVvbGVEUmSJOlegjfQtWWqMollszYKSZIkaQAmDYar6sT+10k2rapb2h+SJEmSNDvWWjOc5GlJLgF+3LzeNcmHWx+ZJEmS7jYvGYrHXDOdG+j+BXgBcC1AVX0feFaLY5IkSZJmxbRmk6iqX45rurOFsUiSJEmzajrzDP8yydOBSrIB8GfAj9odliRJku4RMgdLFIbBdDLDrwFeB2wLXAHs1ryWJEmSRtpaM8NVdQ3wslkYiyRJkiYQXCmtLdOZTeJRSf4rydVJVic5NcmjZmNwkiRJUpum80vGZ4DPAQuARwD/AZzc5qAkSZKk2TCdG+g2rapP9b3+dJK/amtAkiRJGid4A11LJg2Gk2zRPP1ykrcCpwAFHAScPgtjkyRJklo1VWb4fHrB79ivIa/u21bA29oalCRJkjQbJg2Gq2qn2RyIJEmSJjcXl0IeBtOpGSbJk4BdgI3H2qrqpLYGJUmSJM2GtQbDSY4E9qEXDJ8O7A+cAxgMS5IkaaRNJzP8YmBX4HtV9aokWwOfbndYkiRJGhMsk2jLdOYZvrWq7gLWJNkMWA1sPxMnT7JfkkuTrGhmrBi//ZXNYh8XNo/D+7YdmmR58zh0JsYjSZKkbplOZnhZkocBH6U3w8TNwHce6ImTrAd8CHg+sBI4L8mSqrpkXNfPVtXrx+27BXAksJDezBbnN/te/0DHJUmSNIycZ7gdaw2Gq+pPm6cfSfIVYLOqumgGzr0IWFFVlwMkOQU4ABgfDE/kBcCZVXVds++ZwH64Mp4kSZLWwVSLbuwx1baquuABnntb4Jd9r1cCe03Q7/8leRZwGfAXVfXLSfbddm0n3OOxT+JbXznn/o9Y0px1V9016CFIGlJVgx6B2jRVZvifpthWwHNmeCwT+S/g5Kq6LcmrgRPX9bxJjgCOANh+hxkpdZYkSZplYR6WSbRhqkU3nt3yuVdx7xvxtmva+sdwbd/LjwHv79t3n3H7fmOik1TVYmAxwJ4L9/B3O0mSJN1tOrNJtOU8YOckOyXZEDgYWNLfIcmCvpcvAn7UPD8D2DfJ5kk2B/Zt2iRJkqRpm9YKdG2oqjVJXk8viF0POKGqLk5yFLCsqpYAb0zyImANcB3wymbf65K8h15ADXDU2M10kiRJc5GzSbRjYMEwQFWdTm9Vu/62d/U9fxvwtkn2PQE4odUBSpIkaU5ba5lEel6e5F3N6x2SLGp/aJIkSVK7ppMZ/jBwF71ZHI4Cfg38J/DUFsclSZKkRuJyzG2ZTjC8V1XtkeR7AFV1fXPDmyRJkjTSphMM39EsnVwASR5OL1MsSZKkWRLnGW7FdKZWOxb4IrBVkqOBc4D3tToqSZIkaRasNTNcVf+e5HzguUCAA6vqR2vZTZIkSRp6aw2Gk+wA3EJvaeS726rqF20OTJIkSfdwnuF2TKdm+DR69cIBNgZ2Ai4FntjiuCRJkqTWTadM4sn9r5PsAfxpayOSJEmSZsk6r0BXVRck2auNwUiSJOm+QpxnuCXTqRl+U9/LecAewBWtjUiSJEmaJdPJDD+k7/kaejXE/9nOcCRJkjSRTGtGXK2rKYPhZrGNh1TVm2dpPJIkSdKsmfRXjCTrV9WdwDNmcTySJEnSrJkqM/xdevXBFyZZAvwH8JuxjVX1hZbHJkmSpIY30LVjOjXDGwPXAs/hnvmGCzAYliRJ0kibKhjeqplJ4ofcEwSPqVZHJUmSJM2CqYLh9YAHc+8geIzBsCRJ0ixyOeZ2TBUMX1lVR83aSCRJkqRZNtWEdf76IUmSpDltqszwc2dtFJIkSZpUmv808ybNDFfVdbM5EEmSJGm2TWdqNUmSJA1SnGe4LS5yLUmSpM4yGJYkSVJnWSYhSZI0ApxnuB1mhiVJktRZBsOSJEnqLMskJEmShlyAeeYwW+FVlSRJUmcZDEuSJKmzLJOQJEkaenE2iZaYGZYkSVJnmRmWJEkaAWaG22FmWJIkSZ1lMCxJkqTOskxCkiRpBMzDMok2mBmWJElSZxkMS5IkqbMsk5AkSRpywdkk2mJmWJIkSZ1lZliSJGnYBeaZGW6FmWFJkiR1lsGwJEmSOssyCUmSpKEX4jzDrTAzLEmSpM4yGJYkSVJnWSYhSZI05ALMiznMNnhVJUmS1FkGw5IkSeosyyQkSZJGgMsxt8PMsCRJkjrLzLAkSdIIcJ7hdpgZliRJ0oxKckKS1Ul+2Ne2RZIzkyxvvm7etCfJsUlWJLkoyR59+xza9F+e5NC+9j2T/KDZ59g0NSSTnWMqBsOSJEmaaZ8E9hvX9lbgrKraGTireQ2wP7Bz8zgCOA56gS1wJLAXsAg4si+4PQ74k7799lvLOSZlMCxJkjT0wrwMx2M6qups4LpxzQcAJzbPTwQO7Gs/qXqWAg9LsgB4AXBmVV1XVdcDZwL7Nds2q6qlVVXASeOONdE5JmUwLEmSpNmwdVVd2Ty/Cti6eb4t8Mu+fiubtqnaV07QPtU5JjXQYDjJfkkubeo97pPGTvKBJBc2j8uS3NC37c6+bUtmdeCSJEndNT/Jsr7HEet6gCajWy2MbZ3PMbDZJJKsB3wIeD69iP68JEuq6pKxPlX1F3393wDs3neIW6tqt1kariRJ0sCEoZpN4pqqWng/9vtVkgVVdWVT6rC6aV8FbN/Xb7umbRWwz7j2bzTt203Qf6pzTGqQmeFFwIqquryqbgdOoVfnMZlDgJNnZWSSJEmaaUuAsRkhDgVO7Wt/RTOrxN7AjU2pwxnAvkk2b26c2xc4o9l2U5K9m1kkXjHuWBOdY1KDDIYnqwO5jyQ7AjsBX+tr3rhJzS9NcmBro5QkSdI6SXIy8B3gcUlWJjkMOAZ4fpLlwPOa1wCnA5cDK4CPAn8KUFXXAe8BzmseRzVtNH0+1uzzE+DLTftk55jUqCy6cTDw+aq6s69tx6paleRRwNeS/KCqfjJ+x6aO5QiA7XfYfvxmSZKkkTDdmRyGQVUdMsmm507Qt4DXTXKcE4ATJmhfBjxpgvZrJzrHVAaZGZ6sPmQiBzOuRKKqVjVfL6dXP7L7fXeDqlpcVQurauHDHz7/gY5ZkiRJc8ggM8PnATsn2YleEHww8NLxnZI8HticXqp9rG1z4Jaqui3JfOAZwPtnZdSSJEmzLZA4I24bBhYMV9WaJK+nVxy9HnBCVV2c5ChgWVWNTZd2MHBKk0If8wTg+CR30ctuH9M/C4UkSZI0HQOtGa6q0+kVTfe3vWvc63dPsN+3gSe3OjhJkiTNeaNyA50kSVKHZZjmGZ5TLD6RJElSZxkMS5IkqbMsk5AkSRpyYbTmGR4lZoYlSZLUWWaGJUmSRkDMDLfCzLAkSZI6y2BYkiRJnWWZhCRJ0giY5zzDrTAzLEmSpM4yGJYkSVJnWSYhSZI05IKzSbTFzLAkSZI6y2BYkiRJnWWZhCRJ0tALiTnMNnhVJUmS1FlmhiVJkkaA8wy3w8ywJEmSOstgWJIkSZ1lmYQkSdKQS5xnuC1mhiVJktRZBsOSJEnqLMskJEmSRkCcTaIVZoYlSZLUWQbDkiRJ6izLJCRJkoZenE2iJWaGJUmS1FlmhiVJkkaAyzG3w8ywJEmSOstgWJIkSZ1lmYQkSdKQC5CYw2yDV1WSJEmdZTAsSZKkzrJMQpIkaejF5ZhbYmZYkiRJnWVmWJIkaQS4Al07zAxLkiSpswyGJUmS1FmWSUiSJI0Ab6Brh5lhSZIkdZbBsCRJkjrLMglJkqQR4GwS7TAzLEmSpM4yGJYkSVJnWSYhSZI05ALMczaJVpgZliRJUmeZGZYkSRp2iTfQtcTMsCRJkjrLYFiSJEmdZZmEJEnSCIg5zFZ4VSVJktRZBsOSJEnqLMskJEmSRoCzSbTDzLAkSZI6y2BYkiRJnWWZhCRJ0pALEJdjboWZYUmSJHWWwbDmrFcf/hp2WLAje+668F7tH/7gcez6xN3Z4ykLeftb3gHAWWeexdMXPYOFuz2Vpy96Bt/42jcGMGJJbVn5y5Xs/7wXsudTFrJw16fyoWM/DMBRR76HRbvvzd57Pp3f3/8ArrziSgCqijf/+V/x5MfvyqLd9+Z7F1x4r+PddNNN7PzIx/GmN/7lbL8VdVaYl+F4zDUDDYaTnJBkdZIfTrI9SY5NsiLJRUn26Nt2aJLlzePQ2Ru1RsUfveLlnHral+7V9r9f/1/+e8l/890LlnLBRcv487/8MwC2nL8ln//S51l24Xl89ITF/PErDx/AiCW1Zb311+d9738f51+0jK+f8zUWf2QxP7rkx/z5X/4Z3/3eUpae/232f+F+/N17jwHgjK98lRUrfsJFP7qQDx53LH/++r+41/GOOvK9PON3njGItyJphg06M/xJYL8ptu8P7Nw8jgCOA0iyBXAksBewCDgyyeatjlQj55nPeiZbbLHFvdoWH/8x3vzXf8lGG20EwFZbbQXAbrvvxiMesQCAXZ64C7+99bfcdtttsztgSa1ZsGAbdt9jNwAe8pCH8LjHP44rrriCzTbb7O4+v7nlN3dPXXXaktN46csPIQmL9l7EjTfewJVXXgXA987/HlevXs1zn/ecWX8fkmbeQIPhqjobuG6KLgcAJ1XPUuBhSRYALwDOrKrrqup64EymDqolAFYsX863zvk2v/O03+X5z34By847/z59vviFL7Hb7rveHTBLmlt+/rOf8/0LL+Kpi3olVO/+m7/lsTs9ns+e/Dne+e5e6dQVV1zBdttte/c+j9h2W65cdQV33XUXb/vrt/O+vz96IGNXt2VI/ptrBp0ZXpttgV/2vV7ZtE3Wfh9JjkiyLMmyq6++prWBajSsWbOG666/nrO//Q3e9/dH8/JD/oiqunv7JRdfwjvf9jd88Lh/G+AoJbXl5ptv5qUveTnv/6dj7s4Kv/s9R3LZT3/MQYe8hOM/vHjK/Rcf91H23X9ftt1uwv/lSBpBwx4MP2BVtbiqFlbVwoc/fP6gh6MB23bbbTnwwBeRhKcuWsi8efO45preL0krV67ioBcfwsc+8VEe9ehHDXikkmbaHXfcwUtf8nIOOuQlHPAHB9xn+8GHHMSXvngqAI94xCNYuXLV3duuWLWKBds+gnOXfpfjP7yYJzzmibzjLe/gM58+mb95+7tm7T1ImnnDHgyvArbve71d0zZZuzSl3z/g9/nfb5wNwPLLlnP77bczf/58brjhBv7vi/4v73nfUTz9GU8b8CglzbSq4rV/8joe9/jH8ca/eMPd7SuWr7j7+X8vOY3HPe6xAPyf338hn/n0yVQV3136XTbb7KEsWLANn/jUx7n08h/xoxUXc/TfH81LX34I73nfUbP+ftRNSYbiMdcM+6IbS4DXJzmF3s1yN1bVlUnOAN7Xd9PcvsDbBjVIDadXvOxQvvm/3+Saa67l0TvuzN8c+U4OfdUrePXhr2HPXRey4YYb8rETFpOEj3zoeH6y4nL+7r1/x9+99+8A+K8vL7n7BjtJo+073/oOJ//7yTzxSU9k7z2fDsC733skJ33iJC67bDnzMo8ddtyeYz/0rwC8YP8XcMaXv8qTH78rm2yyCcd/7LhBDl9Si9JfLznrJ09OBvYB5gO/ojdDxAYAVfWR9H79+CC9m+NuAV5VVcuaff8YeHtzqKOr6hNrO9+eC/eob517zky/DUlzwF1116CHIGlIPXOvZ3HB+RcMNCW681MeU8ee9o+DHMLdXrjDH5xfVQvX3nM0DDQzXFWHrGV7Aa+bZNsJwAltjEuSJGmY9JZjHvbq1tHkVZUkSVJnDXvNsCRJkpibN68NAzPDkiRJ6iyDYUmSJHWWZRKSJEkjYN4cXAp5GJgZliRJUmcZDEuSJKmzLJOQJEkadsHZJFpiZliSJEmdZWZYkiRpyPVWoDMz3AYzw5IkSeosg2FJkiR1lmUSkiRJI8Ab6NphZliSJEmdZTAsSZKkzrJMQpIkaeiFmMNshVdVkiRJnWUwLEmSpM6yTEKSJGkEzHM2iVaYGZYkSVJnmRmWJEkaci7H3B4zw5IkSeosg2FJkiR1lmUSkiRJI8DlmNthZliSJEmdZTAsSZKkzrJMQpIkaejF2SRaYmZYkiRJnWUwLEmSpM6yTEKSJGkEOJtEO8wMS5IkqbPMDEuSJA25APPMYbbCqypJkqTOMhiWJElSZ1kmIUmSNOziDXRtMTMsSZKkzjIYliRJUmdZJiFJkjT0XI65LWaGJUmS1FlmhiVJkkaAN9C1w8ywJEmSOstgWJIkSZ1lmYQkSdII8Aa6dpgZliRJUmcZDEuSJKmzLJOQJEkacsEyibaYGZYkSVJnGQxLkiSpsyyTkCRJGgUuutEKM8OSJEnqLDPDkiRJQy/eQNcSM8OSJEnqLINhSZIkdZZlEpIkSSMg3kDXCjPDkiRJ6iyDYUmSJHWWZRKSJEkjwNkk2jHQzHCSE5KsTvLDSba/LMlFSX6Q5NtJdu3b9rOm/cIky2Zv1JIkSZorBl0m8Ulgvym2/xT43ap6MvAeYPG47c+uqt2qamFL45MkSdIcNtAyiao6O8kjp9j+7b6XS4HtWh+UJEnSELJMoh2Dzgyvi8OAL/e9LuCrSc5PcsRkOyU5IsmyJMuuvvqa1gcpSZKk0TESN9AleTa9YPiZfc3PrKpVSbYCzkzy46o6e/y+VbWYprxiz4V71KwMWJIkaQYF5xluy9BnhpM8BfgYcEBVXTvWXlWrmq+rgS8CiwYzQkmSJI2qoQ6Gk+wAfAH4o6q6rK/9QUkeMvYc2BeYcEYKSZIkaTKDnlrtZOA7wOOSrExyWJLXJHlN0+VdwJbAh8dNobY1cE6S7wPfBU6rqq/M+huQJEmaFRma/6Y12gmmwE2yRZIzkyxvvm7etCfJsUlWNFPq7tF3nEOb/suTHNrXvmdz/BXNvve7hmTQs0kcspbthwOHT9B+ObDrffeQJEnSkHh2VfXPXvBW4KyqOibJW5vXbwH2B3ZuHnsBxwF7JdkCOBJYSG/ihPOTLKmq65s+fwKcC5xOb6re/okWpm2oyyQkSZI0ZxwAnNg8PxE4sK/9pOpZCjwsyQLgBcCZVXVdEwCfCezXbNusqpZWVQEn9R1rnY3EbBKSJEldN0TzDM8ft/rv4mb2rn5jU+AWcHyzfeuqurLZfhW9sleAbYFf9u27smmbqn3lBO33i8GwJEmS1sU101j99z5T4PZvrKpqAuWBs0xCkiRp2KU3z/AwPKZjkilwf9WUONB8Xd10XwVs37f7dk3bVO3bTdB+vxgMS5IkacZMMQXuEmBsRohDgVOb50uAVzSzSuwN3NiUU5wB7Jtk82bmiX2BM5ptNyXZu5lF4hV9x1pnlklIkiRpJm0NfLHJIq8PfKaqvpLkPOBzSQ4Dfg68pOl/OvBCYAVwC/AqgKq6Lsl7gPOafkdV1XXN8z8FPglsQm8Wifs1k8TYACVJkjTkhugGuilNNgVus5LwcydoL+B1kxzrBOCECdqXAU96wIPFMglJkiR1mMGwJEmSOssyCUmSpCEXmPZMDlo3ZoYlSZLUWQbDkiRJ6izLJCRJkoZeRmY2iVFjZliSJEmdZWZYkiRpBJgZboeZYUmSJHWWwbAkSZI6yzIJSZKkEeA8w+0wMyxJkqTOMhiWJElSZ1kmIUmSNAKcTaIdZoYlSZLUWQbDkiRJ6izLJCRJkoZcsEyiLWaGJUmS1FlmhiVJkoZenGe4JWaGJUmS1FkGw5IkSeosyyQkSZJGgmUSbTAzLEmSpM4yGJYkSVJnWSYhSZI07IKzSbTEzLAkSZI6y8ywJEnSCHAFunaYGZYkSVJnGQxLkiSpsyyTkCRJGgGWSbTDzLAkSZI6y2BYkiRJnWWZhCRJ0pALcZ7hlpgZliRJUmcZDEuSJKmzLJOQJEkaAc4m0Q4zw5IkSeosM8OSJEkjwMxwO8wMS5IkqbMMhiVJktRZlklIkiSNAOcZboeZYUmSJHWWwbAkSZI6yzIJSZKkEeBsEu0wMyxJkqTOMhiWJElSZ1kmIUmSNORCnE2iJWaGJUmS1FlmhiVJkkaAN9C1w8ywJEmSOmugwXCSE5KsTvLDSbbvk+TGJBc2j3f1bdsvyaVJViR56+yNWpIkSXPFoMskPgl8EDhpij7frKrf629Ish7wIeD5wErgvCRLquqStgYqSZI0WJZJtGGgmeGqOhu47n7sughYUVWXV9XtwCnAATM6OEmSJM15g84MT8fTknwfuAJ4c1VdDGwL/LKvz0pgr4l2TnIEcETz8rZN1n/QhCUZHTMfuGbQgxgCXocer0OP16HH69DjdejxOvQ8btADUHuGPRi+ANixqm5O8kLgS8DO63KAqloMLAZIsqyqFs74KEeM16HH69DjdejxOvR4HXq8Dj1eh54kywY9BrBIoi1DPZtEVd1UVTc3z08HNkgyH1gFbN/XdbumTZIkSZq2oc4MJ9kG+FVVVZJF9IL3a4EbgJ2T7EQvCD4YeOnABipJktQyV6Brx0CD4SQnA/sA85OsBI4ENgCoqo8ALwZem2QNcCtwcFUVsCbJ64EzgPWAE5pa4rVZPPPvYiR5HXq8Dj1ehx6vQ4/Xocfr0ON16PE6zGHpxZaSJEkaVrvu8ZQ641unDXoYACzYdIfz51It+VCXSUiSJGmMZRJtGOob6CRJkqQ2zblgOMkWSc5Msrz5uvkk/e7sW+Z5SV/7TknObZZ5/mySDWdv9DNnOtchyW5JvpPk4iQXJTmob9snk/y07xrtNqtv4AFa23LdSTZqvr8rmu/3I/u2va1pvzTJC2Z14DNsGtfhTUkuab7/ZyXZsW/bhD8jo2ga1+GVSa7ue7+H9207tPk5Wp7k0Nkd+cyaxnX4QN81uCzJDX3b5sTnIckJSVYnmXDO+fQc21yji5Ls0bdtLn0W1nYdXta8/x8k+XaSXfu2/axpv3BYphy7v6ZxHfZJcmPfZ/9dfdum/HnS6JhzNcNJ3g9cV1XHNB/OzavqLRP0u7mqHjxB++eAL1TVKUk+Any/qo5rf+QzazrXIcljgaqq5UkeAZwPPKGqbkjySeC/q+rzsz74Byi95bovo2+5buCQ/uW6k/wp8JSqek2Sg4E/qKqDkuwCnExvlcNHAP8DPLaq7pzt9/FATfM6PBs4t6puSfJaYJ+qOqjZNuHPyKiZ5nV4JbCwql4/bt8tgGXAQqDo/YzsWVXXz87oZ850rsO4/m8Adq+qP25ez5XPw7OAm4GTqupJE2x/IfAG4IX0FnP616raay59FmBa1+HpwI+q6vok+wPvrqq9mm0/o/fzMvKLcUzjOuxDb8Gv3xvXvk4/TzNh1z2eUl/91ultHX6dbLPp9nOqZnjOZYbpLct8YvP8RODA6e6YJMBzgLEAcJ32HzJrvQ5VdVlVLW+eXwGsBh4+WwNs0XSW6+6/Pp8Hntt8/w8ATqmq26rqp8CK5nijaK3Xoaq+XlW3NC+X0puze655IMu3vwA4s6qua4KeM4H9Whpn29b1OhxC7xfDOaWqzgaum6LLAfQCo6qqpcDDkixgbn0W1nodqurbfYH+XP23YTqfh8k8kH9XNGTmYjC8dVVd2Ty/Cth6kn4bJ1mWZGmSA5u2LYEbqmpN83olvaWfR9F0rwMA6c3jvCHwk77mo5s/k30gyUYtjbMNEy3XPf77eHef5vt9I73v/3T2HRXr+l4OA77c93qin5FRNN3r8P+az/vnk4wt6tPJz0NTLrMT8LW+5rnyeVibya7TXPosrKvx/zYU8NUk5yc5YkBjmk1PS/L9JF9O8sSmrcufhzlnJGeTSPI/wDYTbHpH/4tmsY7J6kB2rKpVSR4FfC3JD+gFRCNjhq4DTdbjU8ChVXVX0/w2ekH0hvTmV3wLcNRMjFvDJ8nL6f3593f7mu/zM1JVP5n4CCPvv4CTq+q2JK+m91eD5wx4TIN0MPD5ceVBXfo8qNGUUh0GPLOv+ZnNZ2Er4MwkP24yrHPRBfQ++zc3JTRfAnYezFCCs0m0YyQzw1X1vKp60gSPU4FfNcHdWJC3epJjrGq+Xg58A9id3up2D0sy9kvCUC/zPBPXIclmwGnAO5o/CY4d+8rmz4S3AZ9gtEoFprNc9919mu/3Q+l9/+fSUt/Tei9JnkfvF6gXNd9vYNKfkVG01utQVdf2vfePAXtOd98Rsi7v5WDGlUjMoc/D2kx2nebSZ2FakjyF3s/DAVV17Vh732dhNfBFRuv/D+ukqm6qqpub56cDGySZTwc/D3PZSAbDa7EEGLvL91Dg1PEdkmw+9mf/5kP9DOCSqirg6/RWvpt0/xExneuwIb1/yE4af6NcXyAdevXGE95pO6TOo1muu3mPB9O7Hv36r8+Lga813/8lwMHpzTaxE70MwHdnadwzba3XIcnuwPH0AuHVfe0T/ozM2shn1nSuw4K+ly8CftQ8PwPYt7kemwP7Nm2jaDo/FyR5PLA58J2+trn0eVibJcAr0rM3cGNTcjaXPgtrlWQH4AvAH1XVZX3tD0rykLHn9K7DKP3/YZ0k2ab5/+BYOeE8eomTaf08zexYessxD8NjrhnJMom1OAb4XJLDgJ8DLwFIshB4TVUdDjwBOD7JXfQ+2Mf03QH6FuCUJO8Fvgd8fLbfwAyZznV4CfAsYMv07qYHeGVVXQj8e5KH0/ubzIXAa2Z19A9AVU24XHeSo4BlVbWE3vf1U0lW0Lt54uBm34vTm1HkEmAN8LoawZkkYNrX4R+ABwP/0fwD94uqehFT/4yMlGlehzcmeRG97/l1wCubfa9L8h56/+MDOKqq7s/NNgM3zesAvZ+FU5pfDsfMmc9DkpOBfYD5SVYCRwIbAFTVR4DT6c0ksQK4BXhVs23OfBZgWtfhXfTuo/hw82/DmurNHrA18MWmbX3gM1X1lVl/AzNkGtfhxcBrk6wBbgUObn42Jvx5GsBb0AyYc1OrSZIkzTW77blrffVbX157x1mw9SbbOrWaJEmSNBcYDEuSJKmzDIYlSZLUWXPxBjpJkqQ5J84z3Aozw5IkSeosg2FJA5HkziQXJvlhkv9IsukDONYnk7y4ef6xJLtM0XefJE+/H+f4WTPH7rTax/W5eR3P9e4kb17XMUqS1p3BsKRBubWqdquqJwG3M24u69yzEuQ6qarD1zIH7j7AOgfDkjRoGZL/5hqDYUnD4JvAY5qs7TeTLAEuSbJekn9Icl6Si5K8GnorIyb5YJJLk/wPsNXYgZJ8o1lchiT7JbkgyfeTnJXkkfSC7r9ostK/k+ThSf6zOcd5SZ7R7Ltlkq8muTjJx2Dt/wdI8qUk5zf7HDFu2wea9rOaBW1I8ugkX2n2+WZ6q79JkmaRN9BJGqgmA7w/MLaK1R7Ak6rqp01AeWNVPTW95YC/leSrwO7A44Bd6K2IdQlwwrjjPhz4KPCs5lhbNKuIfQS4uar+sen3GeADVXVOekvQnkFvxbUjgXOq6qgk/wc4bBpv54+bc2wCnJfkP6vqWuBB9FZ5+4sk72qO/XpgMb0VIZcn2Qv4MPCc+3EZJUn3k8GwpEHZJMmFzfNv0lsi++nAd6vqp037vsBTxuqBgYcCO9NbRvzkZqnsK5J8bYLj7w2cPXasKZbOfR6wS7O8LMBmSR7cnOP/NvueluT6abynNyb5g+b59s1YrwXuAj7btH8a+EJzjqdzz1LYABtN4xySpBlkMCxpUG6tqt36G5qg8Df9TcAbquqMcf1eOIPjmAfsXVW/nWAs05ZkH3qB9dOq6pYk3wA2nqR7Nee9Yfw1kCTNLmuGJQ2zM4DXJtkAIMljkzwIOBs4qKkpXgA8e4J9lwLPSrJTs+8WTfuvgYf09fsq8IaxF0l2a56eDby0adsf2HwtY30ocH0TCD+eXmZ6zDxgLLv9UnrlFzcBP03yh805kmTXtZxDUoclGYrHXGMwLGmYfYxePfAFSX4IHE/vL1pfBJY3204CvjN+x6q6GjiCXknC97mnTOG/gD8Yu4EOeCOwsLlB7xLumdXib+kF0xfTK5f4xVrG+hVg/SQ/Ao6hF4yP+Q2wqHkPzwGOatpfBhzWjO9i4IBpXBNJ0gxKVQ16DJIkSZrCbnvuWmd9+6uDHgYA8zfe5vyqWjjoccwUM8OSJEnqLINhSZIkdZazSUiSJA29ubn62zAwMyxJkqTOMhiWJElSZ1kmIUmSNBIsk2iDmWFJkiR1lsGwJEmSOssyCUmSpCEXLJJoi5lhSZIkdZbBsCRJkjrLMglJkqQRkFgo0QYzw5IkSeosM8OSJEkjwcxwG8wMS5IkqbMMhiVJktRZlklIkiSNAIsk2mFmWJIkSZ1lMCxJkqTOskxCkiRpJFgo0QYzw5IkSeosg2FJkiR1lmUSkiRJQy8ux9wSM8OSJEnqLINhSZIkdZbBsCRJkjrLYFiSJEmd5Q10kiRJQy5AnGe4FWaGJUmS1FkGw5IkSeosyyQkSZJGgmUSbTAzLEmSpM4yMyxJkjQCzAu3w8ywJEmSOstgWJIkSZ1lmYQkSdIISCyUaIOZYUmSJHWWwbAkSZI6yzIJSZKkoRecT6IdZoYlSZLUWQbDkiRJ6izLJCRJkkaARRLtMDMsSZKkzjIzLEmSNBLMDbfBzLAkSZI6y2BYkiRJnWWZhCRJ0rCLyzG3xcywJEmSOstgWJIkSZ1lMCxJkqTOMhiWJElSZxkMS5IkqbOcTUKSJGnIBYiLbrTCzLAkSZI6K1U16DFIkiRpCkm+Aswf9Dga11TVfoMexEwxGJYkSVJnWSYhSZKkzjIYliRJUmcZDEuSJKmzDIYlSZLUWQbDkiRJ6qz/D6ooN62ylbAgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cm = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "print ('Confusion Matrix :')\n",
    "\n",
    "def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.Greens):\n",
    "    plt.figure(figsize=(10,10),)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    plt.tight_layout()\n",
    "\n",
    "    width, height = cm.shape\n",
    "\n",
    "    for x in range(width):\n",
    "        for y in range(height):\n",
    "            plt.annotate(str(cm[x][y]), xy=(y, x), \n",
    "                        horizontalalignment='center',\n",
    "                        verticalalignment='center')\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "\n",
    "\n",
    "plot_confusion_matrix(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 ROC Curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how our ROC curve looks with the predictions. The closer the AUC is to 1, the better the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3bUlEQVR4nO3deZxN9f/A8dfbDAYhe8kydrOQZAkhS5aklG/flKR+g0pKi2/RIvnKlxJFZC2SLEml8s23na9vkj1LMslOmCS7Wd6/P+6ZcWfjjpl779x738/H4z7cc+7n3PM+Zua8z2c5nyOqijHGmNBVwN8BGGOM8S9LBMYYE+IsERhjTIizRGCMMSHOEoExxoQ4SwTGGBPiLBEYY0yIs0Rggo6I7BSR0yJyQkQOishMEbksQ5nmIvK1iBwXkWMi8omIRGcoU0JEXhOR3c53/eosl/XtERnjXZYITLDqqqqXAQ2Aa4AhqR+ISDPgP8DHQEWgGrABWCEi1Z0yhYCvgBigE1ACaAYkAE28FbSIhHvru43JjiUCE9RU9SCwFFdCSPUy8I6qvq6qx1X1D1V9DlgJDHPK3AtUAW5T1S2qmqKqh1T1n6q6JKt9iUiMiHwhIn+IyO8i8oyzfqaIjHArd4OI7HVb3ikiT4vIRuCk835hhu9+XUTGO+9LisgMETkgIvtEZISIhOXuf8qEMksEJqiJSCWgMxDvLBcFmgPvZ1F8AXCj87498LmqnvBwP8WBL4HPcdUyauKqUXjqLqALcDkwD7jJ+U6ck/zfgfecsjOBJGcf1wAdgD452Jcx6VgiMMHqIxE5DuwBDgEvOOtL4/q9P5DFNgeA1Pb/MtmUyc7NwEFVfVVVzzg1jR9ysP14Vd2jqqdVdRewFrjN+awtcEpVV4pIBeAm4DFVPamqh4BxQI8c7MuYdCwRmGDVTVWLAzcAdTl/gj8KpABXZrHNlcAR531CNmWyUxn49ZIiddmTYfk9XLUEgLs5XxuoChQEDojInyLyJzAFKJ+LfZsQZ4nABDVV/Q5XU8oYZ/kk8D1wRxbF/8755pwvgY4iUszDXe0Bqmfz2UmgqNvyFVmFmmH5feAGp2nrNs4ngj3AWaCsql7uvEqoaoyHcRqTiSUCEwpeA24Ukaud5cFAbxF5VESKi0gppzO3GfCiU2Y2rpPuByJSV0QKiEgZEXlGRG7KYh+fAleKyGMiUtj53qbOZ+txtfmXFpErgMcuFrCqHga+Bd4GflPVrc76A7hGPL3qDG8tICI1RKR1Tv9TjEllicAEPeek+g4w1Fn+L9ARuB1XP8AuXJ2u16vqdqfMWVwdxj8DXwB/AatwNTFlavtX1eO4Opq7AgeB7UAb5+PZuIan7sR1Ep/vYejvOTG8l2H9vUAhYAuupq6F5KwZy5h0xB5MY4wxoc1qBMYYE+IsERhjTIizRGCMMSHOEoExxoS4gJvgqmzZshoZGenvMIwxJqCsWbPmiKqWy+qzgEsEkZGRrF692t9hGGNMQBGRXdl9Zk1DxhgT4iwRGGNMiLNEYIwxIc4SgTHGhDhLBMYYE+K8lghE5C0ROSQim7L5XERkvIjEi8hGEWnorViMMcZkz5s1gpm4Hvqdnc5ALefVD3jTi7EYY4zJhtfuI1DVZSISeYEit+J6gLgCK0XkchG50plv3RhjgpqqcjYphVPnkjl1LonT55Kd98mcTkw6//5cMsdOnibhz+Pcfl1trq58eZ7H4s8byq4i/eP59jrrMiUCEemHq9ZAlSpVfBKcMcYkp2iWJ+nT51Jc6xPd1p9Lf/I+lZh5XWr50+eSOJWYTE6fAlCzUvmgSwQeU9WpwFSARo0a2QMUjDGA66r6XHJK+hO1c4XtOhFncZJOTL0CT8l05Z120ndO2OeSUnIUT1gBoWjBMIoUCqNooTCKFAqnaKEwikeEU7544XTrXO/D0soXKRRO0YLn1yefPcVrY15m7ruzqFb5KqZNnkibZpFe+X/0ZyLYh+uB36kqOeuMMUEkJUXdroSTOeWcfM+fvJM4k5jxRJ65eeSUczJ3/67Tickkp+Ts2rBweAHnRBx+/oRdMIyylxWiaKGiRDgn47QTdRYn6aLOyTz186IFXd9VMEwQkVz/nyUnJ1OvXgu2bdvGoEGDGDZsGEWKFMn192bHn4lgMTBAROYBTYFj1j9gjH+cS0pJO0mfTnflnEWTR8ZybifpdCf0xNSTfM6uqgsIaSfpIgXTn5BLFyuc6SRdtFB4hnKu5fOfp18fViD3J2pvSUhIoHTp0oSFhfHSSy9RuXJlGjVq5PX9ei0RiMhc4AagrIjsBV4ACgKo6mRgCXATEA+cAu73VizGBDpV5UxiSqarYvf264xNIufbqd2upjOcpFPXJeXwqrpQWAG3q+XzV8WXFy1Excszn6Tdr7wzXU27XXEXKRRG4fACeXJVHUhUlTlz5jBw4EBGjRpF3759ue2223y2f2+OGrrrIp8r8LC39m+MryUlp6Rrl053ks5qfTYnafdOxtTmj9M57FgUIf1VcsHzzSAlixRMf6J2a6fO8iRd8HybdoRTNjzM7kXNK3v27OHBBx9kyZIlXHfddbRo0cLnMQREZ7ExeSF1uF52IzrSTtJZjATJdELP0M59+lwy55Jz1gRSMEzcrpbPN2+UiAjnihKF052k3TsZ3a+u0528C54/uUcUDL2r6kA0d+5cHnjgAZKTk3nttdcYMGAAYWFhPo/DEoHJV5LTOhYzDNnL4iR9JjGLq2m3Tsb027u2zWELCBEFC6Q7SaeeiMsXj0h3ko5wrrqzbPLIcJJO/bygXVWHvFKlStG0aVOmTp1KtWrV/BaHaE4HsvpZo0aN1B5M4z/ZDdfLdPJOzHw1ndVIkNMZmkfO5tFwvUydhdmdpC/QyVikYBgF8nHHogk8SUlJjBs3jnPnzvHss88Crr8pX9TeRGSNqmbZ82w1giCUkqKcScqiySOLk3TG4XoZR4xkah65hOF6hVKH67k3gxQKo8xlhTK1QWccrlfEbay1+9V0atlCYdYEYgLDhg0biIuLY82aNfz9739PSwD54ffXEoGfJCanXGB89PnljM0b50/kWd2WfmnD9URwTrqZT8ilihY631FYMHOTh/tIkEAcrmeMt509e5YRI0YwatQoSpcuzfvvv0/37t3zRQJIZYkgGxmH66UfH53F1XQWJ2n37TJ2RiYm5264Xmqbc8mihbiyZBZX09kM18s43rpoiA7XM8ZXtm/fzujRo7n77rsZO3YsZcqU8XdImYRMIjiTmMz05Ts4cuJc9qNGMjSN5LT7JOPNL6nNG1eUyH643vn26IxX425t2jZcz5iAcuLECT7++GN69uxJbGwsP//8M9WrV/d3WNkKmUSwdtdRxvznF4oVCuOyiPB0I0GKR4RToURhp6Mw8zwg6W5Fz2K4XpFCYUSEW8eiMQa++OIL+vXrx65du2jYsCFRUVH5OglACCWCZOfyftb/NaFRZGk/R2OMCTZHjx5l0KBBvPXWW9SuXZvvvvuOqKgof4flkZBJBMYY4y3Jycm0aNGCX375hSFDhjB06FAiIiL8HZbHLBEYY8wlOnLkSNokcSNHjqRKlSo0bBh4T90NmR7IALtvzhiTj6kq77zzDrVr12b69OkAdOvWLSCTAIRQIkhloySNMbmxa9cuOnfuTO/evYmKiqJVq1b+DinXQi4RGGPMpXr33XeJjY3lv//9LxMmTGD58uXUrVvX32HlWsj0EVjLkDEmt8qVK0eLFi2YMmUKVatW9Xc4eSZkEoExxuRUYmIir776KomJiTz//PN07NiRDh06BN2d+CHYNBRcP0BjjHesW7eOpk2bMmTIELZs2ULqTM3BlgQgJBOBMcZk78yZMzzzzDM0btyY/fv388EHHzB37tygTACpQiYRBNpzF4wx/hEfH8+YMWO499572bp1K7fffru/Q/I66yMwxoS8EydO8OGHH9KrVy9iY2PZtm2bX58Y5mshUyNIFcS1O2PMJVi6dCkxMTH07t2brVu3AoRUEoAQTATGGAOQkJBA79696dSpE0WLFmX58uUBM0lcXguZpiHrITDGpEqdJC4+Pp5nn32W5557LqAmictrIZMIjDHm8OHDlClThrCwMEaPHk3VqlVp0KCBv8Pyu5BrGrIuAmNCj6ry9ttvU7t2baZNmwbArbfeaknAEXKJwBgTWnbu3EnHjh35v//7P+rVq0ebNm38HVK+EzqJwDoJjAk5s2fPJjY2lu+//55Jkybx7bffUrt2bX+Hle+EXB9BMN8daIxJr0KFCrRq1YrJkydTpUoVf4eTb4VcIjDGBK/ExERefvllkpOTGTp0KB06dKBDhw7+DivfC5mmIbW2IWOC2tq1a2ncuDHPPfcc27Zts2llciBkEoExJjidPn2awYMH06RJE37//Xc+/PBD5syZY83AOeDVRCAinURkm4jEi8jgLD6vIiLfiMg6EdkoIjd5Mx6w4aPGBJsdO3YwduxY7rvvPrZs2UK3bt38HVLA8VoiEJEwYCLQGYgG7hKR6AzFngMWqOo1QA9gkrfiMcYEj7/++ouZM2cCEBMTw/bt25k+fTqlSpXyb2AByps1giZAvKruUNVzwDzg1gxlFCjhvC8J7PdWMNZcaExwWLJkCbGxscTFxaVNEhdMj430B28mgquAPW7Le5117oYB94jIXmAJ8EhWXyQi/URktYisPnz4sDdiNcbkc0eOHKFXr1506dKF4sWLs2LFipCdJC6v+buz+C5gpqpWAm4CZotIpphUdaqqNlLVRuXKlcvVDq3/yJjAkzpJ3Lx58xg6dChr167luuuu83dYQcOb9xHsAyq7LVdy1rmLAzoBqOr3IhIBlAUOeTEuY0yA+P333ylXrhxhYWGMGTOGqlWrUr9+fX+HFXS8WSP4EaglItVEpBCuzuDFGcrsBtoBiEgUEAF4pe3H+giMCRyqyowZM6hTpw5Tp04FoGvXrpYEvMRriUBVk4ABwFJgK67RQZtFZLiI3OIUexLoKyIbgLnAfWp3gRgT0nbs2EH79u3p06cPDRo0oH379v4OKeh5dYoJVV2CqxPYfd1Qt/dbgBbejCEjsTsJjMm3Zs2aRf/+/QkLC2Py5Mn07duXAgX83ZUZ/GyuIWNMvlGxYkXatm3Lm2++SaVKlfwdTsgImURg7U3G5D/nzp1j1KhRpKSkMGzYMG688UZuvPFGf4cVckKuzmXDR43JH3788UeuvfZaXnjhBXbs2GGTxPlRyCUCY4x/nTp1ikGDBnHddddx9OhRFi9ezDvvvGOTxPmRJQJjjE/99ttvTJgwgb59+7J582a6du3q75BCXuj0EVi10xi/OXbsGIsWLeL+++8nJiaG+Ph4KleufPENjU9YjcAY41WfffYZMTEx9OnTh59//hnAkkA+Y4nAGOMVhw8fpmfPntx8882UKlWK77//nrp16/o7LJOF0Gka8ncAxoSQ5ORkrr/+en777TdefPFFBg8eTKFChfwdlslGyCQCY4z3HTx4kPLlyxMWFsarr75KZGQksbGx/g7LXETINQ3ZCDVj8l5KSgpTpkyhdu3aTJkyBYCbb77ZkkCA8CgRiEgREanj7WCMMYEnPj6edu3a8eCDD9K4cWM6duzo75BMDl00EYhIV2A98Lmz3EBEMk4nne/Z6FFj8t7bb79NvXr1WLt2LdOmTePLL7+kevXq/g7L5JAnNYJhuJ4//CeAqq4HqnktImNMwKhSpQodO3Zky5Yt9OnTx+4ODlCedBYnquqxDD/ggL2+tmmojbl0Z8+e5V//+hcpKSkMHz6cdu3a0a5dO3+HZXLJkxrBZhG5GwgTkVoiMgH4n5fjMsbkMz/88APXXnstL774Irt377a79YOIJ4ngESAGOAu8BxwDBnozKO+wX1pjLsXJkyd54oknaNasGceOHePTTz9l5syZ1gwURDxJBF1U9VlVbey8ngNuuehW+ZT97hqTM7t27WLSpEk8+OCDbN68mS5duvg7JJPHPEkEQzxcZ4wJEn/++SfTp08HIDo6mvj4eCZNmkSJEiX8HJnxhmw7i0WkM3ATcJWIjHf7qASQ5O3AjDH+8fHHH/PQQw9x6NAhrr/+eurWrWuPjQxyF6oR7AdWA2eANW6vxUDA3TFi/VrGXNihQ4fo0aMH3bp1o1y5cqxcudImiQsR2dYIVHUDsEFE3lPVRB/G5FXWR2BMZsnJybRo0YLdu3czYsQInnrqKQoWLOjvsIyPeHIfQaSI/AuIBiJSV6qq3T5oTIDbv38/V1xxBWFhYbz++utERkYSHR3t77CMj3nSWfw28CaufoE2wDvAu94MyhusZciY81JSUnjzzTepW7cukydPBuCmm26yJBCiPEkERVT1K0BUdZeqDgNs/JgxAeqXX36hTZs29O/fn6ZNm9K5c2d/h2T8zJOmobMiUgDYLiIDgH3AZd4Ny3tsigkTymbMmMGAAQOIiIjgrbfe4r777rMbw4xHNYKBQFHgUeBa4B6gtzeDMsZ4R2RkJJ07d2bLli3cf//9lgQMcJEagYiEAXeq6iDgBHC/T6LyAhs+akLR2bNn+ec//wnAiBEjbJI4k6UL1ghUNRm43kexGGPy0P/+9z8aNGjASy+9xIEDB2ySOJMtT/oI1jkPonkfOJm6UlUXeS0qL7KasAl2J06c4Nlnn2XChAlUrlyZzz//3J4aZi7Ikz6CCCABaAt0dV43e/LlItJJRLaJSLyIDM6mzN9FZIuIbBaR9zwN3BiTtd27dzNlyhQefvhhNm3aZEnAXNRFawSqekn9Ak7/wkTgRmAv8KOILFbVLW5lauGawK6Fqh4VkfKXsi9PqN1JYILY0aNHef/99+nXrx/R0dHs2LGDihUr+jssEyA8enj9JWoCxKvqDlU9B8wDbs1Qpi8wUVWPAqjqIS/GY0xQ+vDDD4mOjqZ///5s27YNwJKAyRFvJoKrgD1uy3udde5qA7VFZIWIrBSRTll9kYj0E5HVIrL68OHDuQrKughMsDh48CB33HEHt99+O1dccQWrVq2iTp06/g7LBCBPOou9vf9awA1AJWCZiNRT1T/dC6nqVGAqQKNGjayNx4S85ORkWrZsyZ49exg5ciSDBg2ySeLMJbtoIhCRCsBIoKKqdhaRaKCZqs64yKb7gMpuy5Wcde72Aj84s5v+JiK/4EoMP3p6AJ6ykXMmGOzdu5eKFSsSFhbG+PHjqVatmk0VbXLNk6ahmcBSILXR8RfgMQ+2+xGoJSLVRKQQ0APXswzcfYSrNoCIlMXVVLTDg+++ZDZ81ASilJQUJkyYQN26dXnzzTcB6Ny5syUBkyc8SQRlVXUBkAKgqklA8sU2csoNwJVEtgILVHWziAwXkdRnHi8FEkRkC/AN8A9VTbiE4zAmaP3888+0atWKRx99lOuvv56bb/Zo9LYxHvOkj+CkiJTBmclZRK4Djnny5aq6BFiSYd1Qt/cKPOG8jDEZTJ8+nQEDBlC0aFFmzZpFr169bH4gk+c8SQRP4mrSqSEiK4BywN+8GpUXWBeBCUQ1atSga9euvPHGG1SoUMHf4Zgg5ckNZWtEpDVQB9foy22B/ehKu5oy+deZM2cYPnw4ACNHjqRNmza0adPGz1GZYHfRPgIR2Qg8BZxR1U2BnQSMyb9WrFhBgwYN+Ne//sXhw4dtkjjjM550FnfF9ZjKBSLyo4gMEpEqXo4rz9kflcmvjh8/ziOPPELLli05e/YsS5cuZdq0adYXYHzmoonAeTzly6p6LXA3UB/4zeuRGRMi9u7dy/Tp03nkkUf46aef6NChg79DMiHGozuLRaQqcKfzSsbVVBSQ7CLL5AcJCQksWLCAhx56iKioKHbs2MGVV17p77BMiPLkzuIfgIK4nkdwh6p69YYvY4KZqvLBBx/w8MMP88cff9C2bVvq1KljScD4lSd9BPeqakNV/ZclAWMu3YEDB+jevTt33HEHlStXZvXq1TZJnMkXsq0RiMg9qvou0EVEumT8XFXHejUyY4JI6iRx+/bt4+WXX+bxxx8nPNzfcz4a43Kh38Rizr/Fs/gsYIfgWBeB8aU9e/Zw1VVXERYWxsSJE6lWrRq1a9f2d1jGpJNt05CqTnHefqmqL7q/gK98E54xgSk5OZnx48enmySuY8eOlgRMvuRJH8EED9fla3YbgfGVrVu30rJlSwYOHEjr1q3p2rWrv0My5oIu1EfQDGgOlBMR90nhSgBh3g7MW+wmHeNNU6dO5ZFHHqF48eLMnj2bnj172u+cyfcu1EdQCLjMKePeT/AXATjpnDG+UKtWLW677TbGjx9P+fLl/R2OMR7JNhGo6nfAdyIyU1V3+TAmYwLG6dOnGTZsGCLCqFGjbJI4E5Au1DT0mqo+BrwhIpla2FX1lsxb5V8auAOdTD61bNky+vTpw/bt23nwwQdRVWsGMgHpQk1Ds51/x/giEF+xP1OTW3/99ReDBw/mzTffpHr16nz11Ve0bdvW32EZc8ku1DS0xvn3u9R1IlIKqKyqG30QmzH50v79+5k5cyZPPPEEw4cPp1ixYhffyJh8zJO5hr4FbnHKrgEOicgKVbXHS5qQceTIERYsWED//v2pW7cuv/32mz0xzAQNT+4jKKmqfwG3A++oalOgvXfDynt2H4G5FKrK/PnziY6O5rHHHuOXX34BsCRggooniSBcRK4E/g586uV4vM768oyn9u/fT7du3ejRowdVq1ZlzZo1dmewCUqezHo1HFgKrFDVH0WkOrDdu2EZ41/Jycm0atWKffv2MWbMGAYOHGiTxJmg5cnD69/H9SyC1OUdQHdvBuUN1jRkPLFr1y4qVapEWFgYkyZNonr16tSsWdPfYRnjVZ48vL6SiHwoIoec1wciUskXwRnjK8nJyYwdO5aoqKi0SeI6dOhgScCEBE/6CN4GFgMVndcnzrqAJHYngclg06ZNNG/enCeffJJ27drRrVs3f4dkjE95kgjKqerbqprkvGYC5bwclzE+MXnyZBo2bMiOHTt47733WLx4MZUqWYXXhBZPEkGCiNwjImHO6x4gwduB5TXrIjDu1Ok0ioqK4o477mDLli3cddddNkWECUmeDIP4P1zPHxjnLK8A7vdaRF5mf+eh7dSpUwwdOpSwsDBGjx5N69atad26tb/DMsavLlojUNVdqnqLqpZzXt1UdbcvgjMmL3377bfUr1+fV199lRMnTqTVCowJdZ6MGqouIp+IyGFn1NDHzr0ExgSEY8eO8cADD6RND/31118zceJEawYyxuFJH8F7wALgSlyjht4H5nozKG+wq7/QdeDAAd59910GDRrExo0b7XkBxmTgSSIoqqqz3UYNvQtEePLlItJJRLaJSLyIDL5Aue4ioiLSyNPAjbmQw4cPM2GC69HadevWZefOnbzyyisULVrUz5EZk/94kgj+LSKDRSRSRKqKyFPAEhEpLSKls9tIRMKAiUBnIBq4S0SisyhXHBgI/HBph2DMearKe++9R1RUFE8++WTaJHHlytmIZ2Oy40ki+DvwAPAN8C3wENAD15TUqy+wXRMgXlV3qOo5YB5waxbl/gmMBs54HrYxme3Zs4euXbvSs2dPatasybp162ySOGM84MlcQ9Uu8buvAva4Le8FmroXEJGGuB5085mI/CO7LxKRfkA/gCpVqlxSMNZDENySkpK44YYbOHjwIOPGjeORRx4hLCzM32EZExD8Np2iiBQAxgL3Xaysqk4FpgI0atQoV+d0GygSXHbu3EnlypUJDw9nypQpVK9enerVbVCbMTnhSdPQpdoHVHZbruSsS1UciAW+FZGdwHXAYuswNp5ISkpizJgxREVFMWnSJADat29vScCYS+DNGsGPQC0RqYYrAfQA7k79UFWPAWVTl51HYg5S1Qv1O1w6axsKGhs3biQuLo7Vq1dz66230r17wM2Kbky+4skNZeLMNTTUWa4iIk0utp2qJgEDcD3UZiuwQFU3i8hwEbklt4Gb0DRp0iSuvfZadu3axfz58/nwww+pWLGiv8MyJqB5UiOYBKQAbXE9rew48AHQ+GIbquoSYEmGdUOzKXuDB7Hkmt1NGphUFREhNjaWHj16MG7cOMqWLXvxDY0xF+VJImiqqg1FZB2Aqh4VkUJejssYAE6ePMlzzz1HeHg4r7zyCq1ataJVq1b+DsuYoOJJZ3Gic3OYAohIOVw1hICi1kkQcL766ivq1avHa6+9xtmzZ22aEGO8xJNEMB74ECgvIi8B/wVGejUqE9L+/PNP+vTpQ/v27QkPD2fZsmWMHz/emvWM8RJPbiibIyJrgHaAAN1UdavXI/MSO5Xkf7///jvz5s3j6aef5oUXXqBIkSL+DsmYoHbRRCAiVYBTuJ5VnLbOnklg8lLqyX/gwIHUqVOHnTt3WmewMT7iSWfxZ7j6BwTXrKPVgG1AjBfjynPWvJw/qSpz5sxh4MCBnDhxgptuuolatWpZEjDGhzx5Qlk9Va3v/FsL12Ry33s/NO+wZub8Y/fu3XTp0oVevXpRp04d1q9fT61atfwdljEhJ8d3FqvqWhFpevGSxmQvdZK4Q4cOMX78ePr372+TxBnjJ570ETzhtlgAaAjs91pEJqjt2LGDqlWrEh4ezrRp06hRowaRkZH+DsuYkObJ8NHibq/CuPoMsnquQL5mXQT+lZSUxOjRo4mOjmbixIkAtGvXzpKAMfnABWsEzo1kxVV1kI/i8TqxAaQ+t379euLi4li7di233XYbd9xxh79DMsa4ybZGICLhqpoMtPBhPCbIvPHGGzRu3Jh9+/axcOFCFi1axJVXXunvsIwxbi5UI1iFqz9gvYgsBt4HTqZ+qKqLvBybCWCpk8TVr1+fnj17MnbsWEqXzvYR18YYP/Jk1FAEkIBr9tHU+wkUCKhEYPcR+MaJEyd49tlnKViwIGPGjLFJ4owJABfqLC7vjBjaBPzk/LvZ+XeTD2LzCruPwHv+85//EBsby4QJE0hMTLRJ4owJEBeqEYQBl5H19Dz2F27SHD16lCeeeIKZM2dSp04dli1bxvXXX+/vsIwxHrpQIjigqsN9FomX2TTU3nPo0CEWLlzIkCFDGDp0KBEREf4OyRiTAxdKBNaIYrJ18OBB5s6dy+OPP542SVyZMmX8HZYx5hJcqI+gnc+i8CHLbrmjqsyaNYvo6GiGDBnC9u3bASwJGBPAsk0EqvqHLwMx+d/OnTvp1KkT9913H9HR0TZJnDFBIseTzgUqG8CSO0lJSbRp04YjR44wceJEHnzwQQoU8GSGEmNMfhcyiSCNtQ3lSHx8PNWqVSM8PJy33nqL6tWrU7VqVX+HZYzJQ3ZJZ7KUmJjIyJEjiYmJSZskrk2bNpYEjAlCoVcjMBe1du1a4uLiWL9+PXfccQd33nmnv0MyxnhRyNQIrIvAM+PHj6dJkyYcPHiQRYsWsWDBAipUqODvsIwxXhQyiSCVTUOdtdTpIK655hruvfdetmzZwm233ebnqIwxvmBNQyHu+PHjDBkyhMKFC/Pqq6/SsmVLWrZs6e+wjDE+FHI1AnPe559/TmxsLJMmTUJVbZI4Y0JU6CQCO8mlSUhIoHfv3nTu3JlixYqxYsUKxo4di9jUrMaEpNBJBA4717kSwYcffsjzzz/PunXraNasmb9DMsb4kVcTgYh0EpFtIhIvIoOz+PwJEdkiIhtF5CsRsUHqXnLgwAHGjBmDqlK7dm127drF8OHDKVy4sL9DM8b4mdcSgfPg+4lAZyAauEtEojMUWwc0UtX6wELgZW/FE6pUlbfeeouoqCief/554uPjAShVqpSfIzPG5BferBE0AeJVdYeqngPmAbe6F1DVb1T1lLO4EqjkrWBCsYfgt99+o0OHDsTFxXH11VezYcMGmyTOGJOJN4ePXgXscVveCzS9QPk44N9ZfSAi/YB+AFWqVMlVUKHSRZCUlETbtm1JSEjgzTffpF+/fjZJnDEmS/niPgIRuQdoBLTO6nNVnQpMBWjUqFEoXtx7bPv27VSvXp3w8HDefvttatSoQeXKlf0dljEmH/PmJeI+wP0MVMlZl46ItAeeBW5R1bPeCibYR48mJiYyYsQIYmNjeeONNwC44YYbLAkYYy7KmzWCH4FaIlINVwLoAdztXkBErgGmAJ1U9ZAXY3Hfpy9241OrV68mLi6OjRs30qNHD+666y5/h2SMCSBeqxGoahIwAFgKbAUWqOpmERkuIrc4xV4BLgPeF5H1IrLYW/EEq9dff52mTZty5MgRPv74Y+bOnUv58uX9HZYxJoB4tY9AVZcASzKsG+r2vr039x/MVBURoVGjRsTFxfHyyy9z+eWX+zssY0wAyhedxb4QLPPo/PXXXzz99NNEREQwbtw4WrRoQYsWLfwdljEmgIXceMJA7iFYsmQJMTExTJ06lfDw8KBJbsYY/wq5RBCIjhw5wj333EOXLl0oWbIk//vf/3jllVeCsuPbGON7lggCwNGjR/nkk0944YUXWLt2LU2bXui+PGOMyZnQ6SPwdwA5tG/fPubMmcM//vEPatWqxa5du6wz2BjjFSFXI8jvrSmqyrRp04iOjmbYsGH8+uuvAJYEjDFeE3KJID/79ddfadeuHf369aNhw4Zs3LiRmjVr+jssY0yQC5mmofwuKSmJdu3a8ccffzBlyhT69Oljk8QZY3wiZBJBfh1puW3bNmrUqEF4eDizZs2iRo0aVKrktdm4jTEmk5C75JR8cifBuXPnePHFF6lXrx4TJ04EoHXr1pYEjDE+FzI1gvxk1apVxMXFsWnTJu6++2569uzp75CMMSEsZGoE+aVl6LXXXqNZs2Zp9wbMmTOHsmXL+jssY0wIC5lE4G+p00E0adKEvn37snnzZm6++WY/R2WMMaHYNOTjLoJjx47x1FNPUaRIEV577TWaN29O8+bNfRuEMcZcgNUIvOiTTz4hOjqa6dOnU7hwYZskzhiTL4VMIvDlSfjw4cPcfffd3HLLLZQpU4aVK1cyevRomyTOGJMvhUwiSOWLc/GxY8dYsmQJL774IqtXr6Zx48be36kxxlyi0Osj8JI9e/bw7rvvMnjwYGrWrMmuXbsoWbKkv8MyxpiLCrkaQV5LSUlh8uTJxMTEMGLEiLRJ4iwJGGMChSWCXNi+fTtt27bloYceokmTJvz00082SZwxJuCEXNNQXnURJCUlceONN/Lnn38yY8YM7r//fusMNsYEpJBLBLm1detWatWqRXh4OLNnz6ZGjRpUrFjR32GZEJWYmMjevXs5c+aMv0Mx+URERASVKlWiYMGCHm9jicBDZ8+eZeTIkYwcOZJXXnmFxx57jJYtW/o7LBPi9u7dS/HixYmMjLQaqUFVSUhIYO/evVSrVs3j7UImEeTmNoKVK1cSFxfHli1b6NWrF7169cq7wIzJhTNnzlgSMGlEhDJlynD48OEcbRdyncU5/YN59dVXad68OcePH2fJkiW88847lClTxkvRGZNzlgSMu0v5fQi5ROCplJQUAJo1a8aDDz7Ipk2b6Ny5s5+jMsaYvGeJIIM///yTuLg4Bg4cCEDz5s2ZNGkSJUqU8HNkxuRPl112Wdr7JUuWULt2bXbt2uWTfa9bt464uDif7OtSnD17ljvvvJOaNWvStGlTdu7cmWW5119/ndjYWGJiYnjttdfS1t955500aNCABg0aEBkZSYMGDQD46aefuO+++/IszpBJBOrBEwk++ugjoqOjmTVrFsWLF7dJ4ozJga+++opHH32Uf//731StWtWjbZKTk3O1z5EjR/Loo496XD4pKSlX+8upGTNmUKpUKeLj43n88cd5+umnM5XZtGkT06ZNY9WqVWzYsIFPP/2U+Ph4AObPn8/69etZv3493bt35/bbbwegXr167N27l927d+dJnCHTWZwqq9azQ4cOMWDAAN5//30aNGjAp59+SsOGDX0emzG58eInm9my/688/c7oiiV4oWvMRcstW7aMvn37smTJEmrUqAHAu+++y/jx4zl37hxNmzZl0qRJhIWFcdlll/HAAw/w5ZdfMnHiRL7++ms++eQTTp8+TfPmzZkyZQoiwvjx45k8eTLh4eFER0czb968dPs8fvw4Gzdu5OqrrwZcT/4bOHAgZ86coUiRIrz99tvUqVOHmTNnsmjRIk6cOEFycjJLlizhkUceYdOmTSQmJjJs2DBuvfVWdu7cSa9evTh58iQAb7zxRq6njP/4448ZNmwYAH/7298YMGAAqpquHX/r1q00bdqUokWLAq5H1i5atIinnnoqrYyqsmDBAr7++uu0dV27dmXevHnpyl2qkKkRXMhff/3FF198wUsvvcSqVassCRiTA2fPnqVbt2589NFH1K1bF3Cd3ObPn8+KFStYv349YWFhzJkzB4CTJ0/StGlTNmzYwPXXX8+AAQP48ccf2bRpE6dPn+bTTz8FYNSoUaxbt46NGzcyefLkTPtdvXo1sbGxact169Zl+fLlrFu3juHDh/PMM8+kfbZ27VoWLlzId999x0svvUTbtm1ZtWoV33zzDf/4xz84efIk5cuX54svvmDt2rXMnz8/25pGy5Yt05pr3F9ffvllprL79u2jcuXKAISHh1OyZEkSEhLSlYmNjWX58uUkJCRw6tQplixZwp49e9KVWb58ORUqVKBWrVpp6xo1asTy5cuz/8HkQMjUCDK28uzevZvZs2fzzDPPULNmTXbv3k3x4sX9E5wxecCTK3dvKFiwIM2bN2fGjBm8/vrrgKuZaM2aNWkz754+fZry5csDEBYWRvfu3dO2/+abb3j55Zc5deoUf/zxBzExMXTt2pX69evTs2dPunXrRrdu3TLt98CBA5QrVy5t+dixY/Tu3Zvt27cjIiQmJqZ9duONN1K6dGkA/vOf/7B48WLGjBkDuIbg7t69m4oVKzJgwIC0xPXLL79kebx5dfJNFRUVxdNPP02HDh0oVqwYDRo0ICwsLF2ZuXPnctddd6VbV758efbv358nMXi1RiAinURkm4jEi8jgLD4vLCLznc9/EJFIb8YDoJrCpEmTiImJYeTIkWmTxFkSMObSFChQgAULFrBq1SpGjhwJuJoyevfunda+vW3btrQmkoiIiLQT3ZkzZ+jfvz8LFy7kp59+om/fvml3SX/22Wc8/PDDrF27lsaNG2dq3y9SpEi6O6qff/552rRpw6ZNm/jkk0/SfVasWLG096rKBx98kBbb7t27iYqKYty4cVSoUIENGzawevVqzp07l+Xx5qRGcNVVV6Vd3SclJXHs2LEsh5/HxcWxZs0ali1bRqlSpahdu3baZ0lJSSxatIg777wz3TapTWB5wWuJQETCgIlAZyAauEtEojMUiwOOqmpNYBww2lvxpOrUqTMPP/wwzZo1Y/PmzTZJnDF5oGjRonz22WfMmTOHGTNm0K5dOxYuXMihQ4cA+OOPP7IcSZR6si5btiwnTpxg4cKFgGv49p49e2jTpg2jR4/m2LFjnDhxIt22UVFRaZ2q4KoRXHXVVQDMnDkz21g7duzIhAkT0gaDrFu3Lm37K6+8kgIFCjB79uxsO7KXL1+elkTcX+3bt89U9pZbbmHWrFkALFy4kLZt22Y5zj/1/2n37t0sWrSIu+++O+2zL7/8krp161KpUqV02/zyyy/pmsZyw5s1giZAvKruUNVzwDzg1gxlbgVmOe8XAu3ES3fHpN4XsHnzZt5++22WLl1KZGSkN3ZlTEgqXbo0n3/+OSNGjCA+Pp4RI0bQoUMH6tevz4033siBAwcybXP55ZfTt29fYmNj6dixY1pTUnJyMvfccw/16tXjmmuu4dFHH+Xyyy9Pt23dunU5duwYx48fB+Cpp55iyJAhXHPNNRccHfT888+TmJhI/fr1iYmJ4fnnnwegf//+zJo1i6uvvpqff/45XS3iUsXFxZGQkEDNmjUZO3Yso0aNAmD//v3cdNNNaeW6d+9OdHQ0Xbt2ZeLEiemOdd68eZmahcDVpNalS5dcxwgg3hoiKSJ/Azqpah9nuRfQVFUHuJXZ5JTZ6yz/6pQ5kuG7+gH9AKpUqXLtpYxR/mLL70z7YgMv3x5DZOWrLvWwjMlXtm7dSlRUlL/D8Jtx48ZRvHhx+vTp4+9QfOrs2bO0bt2a//73v4SHZ+7qzer3QkTWqGqjrL4vIEYNqepUVW2kqo3cO4dy4sboCiwY2MGSgDFB5KGHHqJw4cL+DsPndu/ezahRo7JMApfCm6OG9gGV3ZYrOeuyKrNXRMKBkkACxhjjgYiIiJCcBLJWrVrphpLmljdrBD8CtUSkmogUAnoAizOUWQz0dt7/Dfha7XZeY3LE/mSMu0v5ffBaIlDVJGAAsBTYCixQ1c0iMlxEbnGKzQDKiEg88ASQaYipMSZ7ERERJCQkWDIwwPnnEURERORoO691FntLo0aNdPXq1f4Ow5h8wZ5QZjLK7gllF+osDpk7i40JRgULFszRk6iMyUpAjBoyxhjjPZYIjDEmxFkiMMaYEBdwncUichi41McflQWOXLRUcLFjDg12zKEhN8dcVVWzvCM34BJBbojI6ux6zYOVHXNosGMODd46ZmsaMsaYEGeJwBhjQlyoJYKp/g7AD+yYQ4Mdc2jwyjGHVB+BMcaYzEKtRmCMMSYDSwTGGBPigjIRiEgnEdkmIvEikmlGUxEpLCLznc9/EJFIP4SZpzw45idEZIuIbBSRr0Skqj/izEsXO2a3ct1FREUk4IcaenLMIvJ352e9WUTe83WMec2D3+0qIvKNiKxzfr9vyup7AoWIvCUih5wnOGb1uYjIeOf/Y6OINMz1TlU1qF5AGPArUB0oBGwAojOU6Q9Mdt73AOb7O24fHHMboKjz/qFQOGanXHFgGbASaOTvuH3wc64FrANKOcvl/R23D455KvCQ8z4a2OnvuHN5zK2AhsCmbD6/Cfg3IMB1wA+53Wcw1giaAPGqukNVzwHzgFszlLkVmOW8Xwi0ExHxYYx57aLHrKrfqOopZ3ElrifGBTJPfs4A/wRGA8EwT7Mnx9wXmKiqRwFU9ZCPY8xrnhyzAiWc9yWB/T6ML8+p6jLgjwsUuRV4R11WApeLyJW52WcwJoKrgD1uy3uddVmWUdcDdI4BZXwSnXd4cszu4nBdUQSyix6zU2WurKqf+TIwL/Lk51wbqC0iK0RkpYh08ll03uHJMQ8D7hGRvcAS4BHfhOY3Of17vyh7HkGIEZF7gEZAa3/H4k0iUgAYC9zn51B8LRxX89ANuGp9y0Sknqr+6c+gvOwuYKaqvioizYDZIhKrqin+DixQBGONYB9Q2W25krMuyzIiEo6rOpngk+i8w5NjRkTaA88Ct6jqWR/F5i0XO+biQCzwrYjsxNWWujjAO4w9+TnvBRaraqKq/gb8gisxBCpPjjkOWACgqt8DEbgmZwtWHv2950QwJoIfgVoiUk1ECuHqDF6cocxioLfz/m/A1+r0wgSoix6ziFwDTMGVBAK93RgucsyqekxVy6pqpKpG4uoXuUVVA/k5p578bn+EqzaAiJTF1VS0w4cx5jVPjnk30A5ARKJwJYLDPo3StxYD9zqjh64Djqnqgdx8YdA1DalqkogMAJbiGnHwlqpuFpHhwGpVXQzMwFV9jMfVKdPDfxHnnofH/ApwGfC+0y++W1Vv8VvQueThMQcVD495KdBBRLYAycA/VDVga7seHvOTwDQReRxXx/F9gXxhJyJzcSXzsk6/xwtAQQBVnYyrH+QmIB44Bdyf630G8P+XMcaYPBCMTUPGGGNywBKBMcaEOEsExhgT4iwRGGNMiLNEYIwxIc4Sgcm3RCRZRNa7vSIvUPaED0PLlohUFJGFzvsG7jNhisgtF5ol1QuxRIrI3b7anwlcNnzU5FsickJVL8vrsr4iIvfhmvF0gBf3Ee7Ml5XVZzcAg1T1Zm/t3wQHqxGYgCEilznPUlgrIj+JSKbZRkXkShFZ5tQgNolIS2d9BxH53tn2fRHJlDRE5FsRed1t2ybO+tIi8pEz9/tKEanvrG/tVltZJyLFnavwTc5dsMOBO53P7xSR+0TkDREpKSK7nPmQEJFiIrJHRAqKSA0R+VxE1ojIchGpm0Wcw0RktoiswHVjZKRTdq3zau4UHQW0dPb/uIiEicgrIvKjcywP5NGPxgQ6f8+9bS97ZffCdWfseuf1Ia474Us4n5XFdWdlaq32hPPvk8CzzvswXHMOlcX1TIJizvqngaFZ7O9bYJrzvhXOfPDABOAF531bYL3z/hOghfP+Mie+SLft7gPecPv+tGXgY6CN8/5OYLrz/iuglvO+Ka7pTzLGOQxYAxRxlosCEc77WrjuuAXX3amfum3XD3jOeV8YWA1U8/fP2V7+fwXdFBMmqJxW1QapCyJSEBgpIq2AFFxT71YADrpt8yPwllP2I1VdLyKtcT2wZIUzvUYh4Pts9jkXXHPCi0gJEbkcuB7o7qz/WkTKiEgJYAUwVkTmAItUda94/liL+bgSwDe4pjiZ5NRSmnN+GhBwnbCzslhVTzvvCwJviEgDXMmzdjbbdADqi8jfnOWSuBLHb54GbYKTJQITSHoC5YBrVTVRXLOKRrgXcE7grYAuwEwRGQscBb5Q1bs82EfGTrNsO9FUdZSIfIZr3pcVItIRzx+AsxhXUisNXAt8DRQD/nRPfhdw0u3948DvwNW4mnuzi0GAR1R1qYcxmhBhfQQmkJQEDjlJoA2Q6bnL4noW8++qOg2YjuuRfyuBFiJS0ylTTESyu2q+0ylzPa5ZHY8By3ElodQO2COq+peI1FDVn1R1NK6aSMb2/OO4mqYyUdUTzjav42q+SVbVv4DfROQOZ18iIld7+P9yQF3z7/fC1SSW1f6XAg85tSVEpLaIFPPg+02QsxqBCSRzgE9E5Cdc7ds/Z1HmBuAfIpIInADuVdXDzgieuSKS2tTyHK65+jM6IyLrcDW3/J+zbhiu5qaNuGZ7TJ3C/DEnIaUAm3E99c39kYHfAINFZD3wryz2NR9434k5VU/gTRF5zolhHq7n9F7IJOADEbkX+JzztYWNQLKIbABm4ko6kcBacbU9HQa6XeS7TQiw4aPGOETkW1zDLQP5mQXG5Jg1DRljTIizGoExxoQ4qxEYY0yIs0RgjDEhzhKBMcaEOEsExhgT4iwRGGNMiPt/cw6hk88BfukAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fpr_keras, tpr_keras, thresholds_keras = roc_curve(y_test, y_pred)\n",
    "auc_keras = auc(fpr_keras, tpr_keras)\n",
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(fpr_keras, tpr_keras, label='Keras (area = {:.2f})'.format(auc_keras))\n",
    "\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We successfully employed state of the art Generative Adversarial Networks for anomaly detection on high dimensional data such as the KDD dataset.\n",
    "- The GAN is particularly interesting  because it sets up a supervised learning problem in order to do unsupervised learning. While it generates fake data, and tries to determine if a sample is fake or real based on trivial labels, it really does not know what the different classes in the dataset are.\n",
    "- On the downside, GANs can be tough to train and suffer from convergence issues particularly because, the discriminator during training does not learn as much from the true dataset as it learns to distinguish between the probability distributions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a moment to consider how each of the three methods actually detected anomalies. If time, take a moment to attempt to articulate each to a partner or write in a notebook or the space below. Bonus. Reflect on how each responded to the rarity of anomalies and why."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<ol>\n",
    "<li>Zenati, H., Foo, C., Lecouat, B., Manek, G. and Chandrasekhar, V. (2018). Efficient GAN-Based Anomaly Detection. [online]   Arxiv.org. Available at: https://arxiv.org/abs/1802.06222</li>\n",
    "\n",
    "<li>Ben Poole Alex Lamb Martin Arjovsky Olivier Mastropietro Vincent Dumoulin, Ishmael Belghazi and Aaron Courville. Adversarially learned inference. International Conference on Learning Representations, 2017.</li>\n",
    "\n",
    "<li>Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A.Bharath. Generative adversarial networks: An overview. In the Proceedings of IEEE Signal Processing Magazine Special Issue on Deep Learning for Visual Understanding, accepted paper,2017.</li>\n",
    "\n",
    "<li>Martin Renqiang Min Wei Cheng Cristian Lumezanu Daeki Cho Haifeng Chen Bo Zong, Qi Song.Deep autoencoding gaussian mixture model for unsupervised anomaly detection. International Conference on Learning Representations, 2018.</li>\n",
    "\n",
    "<li>Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. International Conference on Machine Learning, pp. 1100-1109, 2016.</li>\n",
    "</ol>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/en-us/deep-learning-ai/education/\"> <img src=\"images/DLI Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}