Yi_Yuan_A2_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross sell predictive modeling for Halfway There \n",
    "\n",
    "I will build a model to predict which customer will subscribe to the Half Way There wine service. The goal is to develop a detailed understanding of the features and their relationship with the response variable. \n",
    "\n",
    "\n",
    "The first step would be to analyze the dataset for dirty data. \n",
    "\n",
    "Then I remove features that are irrelevant to cross selling and engineer a slew of new features based on the ones that remain. \n",
    "\n",
    "At this point we are ready to build classification models, perform hyper-parameter tuning to optimize the results, which we will then evaluate the performances and choose the best model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:30.639161Z",
     "start_time": "2020-02-06T15:33:26.417309Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yVette_1963/anaconda3/lib/python3.7/site-packages/sklearn/externals/six.py:31: FutureWarning: The module is deprecated in version 0.21 and will be removed in version 0.23 since we've dropped support for Python 2.7. Please rely on the official version of six (https://pypi.org/project/six/).\n",
      "  \"(https://pypi.org/project/six/).\", FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# importing libraries\n",
    "import numpy as np                                   # highlevel math functions\n",
    "import random            as rand                     # random number gen\n",
    "import pandas            as pd                       # data science essentials\n",
    "import matplotlib.pyplot as plt                      # data visualization\n",
    "import seaborn           as sns                      # enhanced data viz\n",
    "from sklearn.model_selection import train_test_split # train-test split\n",
    "import statsmodels.formula.api as smf                # regression modeling\n",
    "from sklearn.linear_model import LogisticRegression  # logistic regression\n",
    "from sklearn.metrics import confusion_matrix         # confusion matrix\n",
    "from sklearn.metrics import roc_auc_score            # auc score\n",
    "from sklearn.neighbors import KNeighborsClassifier   # KNN for classification\n",
    "from sklearn.neighbors import KNeighborsRegressor    # KNN for regression\n",
    "from sklearn.preprocessing import StandardScaler     # standard scaler\n",
    "\n",
    "from sklearn.tree import DecisionTreeClassifier      # classification trees\n",
    "from sklearn.tree import export_graphviz             # exports graphics\n",
    "from sklearn.externals.six import StringIO           # saves objects in memory\n",
    "from IPython.display import Image                    # displays on frontend\n",
    "import pydotplus                                     # interprets dot objects\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV     # hyperparameter tuning\n",
    "from sklearn.metrics import make_scorer              # customizable scorer\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier     # random forest\n",
    "from sklearn.ensemble import GradientBoostingClassifier # gbm\n",
    "\n",
    "# setting pandas print options\n",
    "pd.set_option('display.max_rows', 500)\n",
    "pd.set_option('display.max_columns', 500)\n",
    "pd.set_option('display.width', 1000)\n",
    "\n",
    "# specifying file name\n",
    "file = 'Apprentice_Chef_Dataset.xlsx'\n",
    "\n",
    "# reading file into python\n",
    "apchef = pd.read_excel(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis:\n",
    "\n",
    "   -remove irrelevant features if any\n",
    "   -missing value detection and imputation\n",
    "   -feature engineering\n",
    "   -encoding categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:30.652830Z",
     "start_time": "2020-02-06T15:33:30.645147Z"
    }
   },
   "outputs": [],
   "source": [
    "# checking the first 10 rows of the dataset\n",
    "# apchef.head(n = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since no features in the dataset are metrics that occur post cross sell, we \n",
    "don't need to remove any features. All of them could be of value to the analysis.\n",
    "\n",
    "### missing value detection and imputation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:30.689189Z",
     "start_time": "2020-02-06T15:33:30.657641Z"
    }
   },
   "outputs": [],
   "source": [
    "############################\n",
    "### create family groups ###\n",
    "\n",
    "# print(apchef.isnull().sum())\n",
    "\n",
    "# 47 family names are missing, however they are active users and we have their \n",
    "# contact information. Retain the accounts with missing family names.   \n",
    "\n",
    "# impute missing values\n",
    "apchef['FAMILY_NAME'] = apchef['FAMILY_NAME'].fillna('NA')\n",
    "\n",
    "# check to see result\n",
    "# print(apchef.isnull().sum().sum())\n",
    "\n",
    "# check the top largest families\n",
    "#lst = apchef['FAMILY_NAME'].value_counts()\n",
    "\n",
    "# placeholder list\n",
    "placeholder_lst = []\n",
    "\n",
    "# looping to family groups\n",
    "for name in apchef.loc[:, 'FAMILY_NAME']:\n",
    "    if 'Tully' in name:\n",
    "        placeholder_lst.append('Tully')\n",
    "    elif 'Frey' in name:\n",
    "        placeholder_lst.append('Tully')\n",
    "    elif 'Arryn' in name:\n",
    "        placeholder_lst.append('Tully')\n",
    "    elif 'Targaryen' in name:\n",
    "        placeholder_lst.append('Targaryen')\n",
    "    elif 'Lannister' in name:\n",
    "        placeholder_lst.append('Lannister')\n",
    "    elif 'Baratheon' in name:\n",
    "        placeholder_lst.append('Baratheon')\n",
    "    elif 'Tyrell' in name:\n",
    "        placeholder_lst.append('Tyrell')\n",
    "    elif 'Stark' in name:\n",
    "        placeholder_lst.append('Stark')\n",
    "    elif 'Snow' in name:\n",
    "        placeholder_lst.append('Stark') \n",
    "    elif 'Greyjoy' in name:\n",
    "        placeholder_lst.append('Greyjoy') \n",
    "    elif 'Martell' in name:\n",
    "        placeholder_lst.append('Martell')\n",
    "    elif 'Sand' in name:\n",
    "        placeholder_lst.append('Martell')\n",
    "    else: \n",
    "        placeholder_lst.append('Other')\n",
    "\n",
    "# converting list into a series\n",
    "apchef['FAMILY_GROUP'] = pd.Series(placeholder_lst)\n",
    "\n",
    "# checking results\n",
    "# apchef.loc[:,'FAMILY_GROUP'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature engineering\n",
    "\n",
    "Perhaps certain GOT families have similar tastes that make them more prone to \n",
    "adopting the wine service. We grouping families by lineage (complicated!)\n",
    "\n",
    "We continue to group email domains, hoping to find some commonalities among \n",
    "people from the same company. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:31.129672Z",
     "start_time": "2020-02-06T15:33:30.693129Z"
    }
   },
   "outputs": [],
   "source": [
    "#########################\n",
    "# split personal emails #\n",
    "\n",
    "# placeholder list\n",
    "placeholder_lst = []\n",
    "\n",
    "# looping over each email address\n",
    "for index, col in apchef.iterrows():\n",
    "    \n",
    "    # splitting email domain at '@'\n",
    "    split_email = apchef.loc[index, 'EMAIL'].split(sep = '@')\n",
    "    \n",
    "    # appending placeholder_lst with the results\n",
    "    placeholder_lst.append(split_email)\n",
    "    \n",
    "# converte placeholder_lst into a DataFrame \n",
    "email_df = pd.DataFrame(placeholder_lst)\n",
    "\n",
    "# displaying the results\n",
    "# email_df\n",
    "\n",
    "#######################################\n",
    "# concatenate with original DataFrame #\n",
    "\n",
    "# renaming column to concatenate\n",
    "email_df.columns = ['name' , 'EMAIL_DOMAIN']\n",
    "\n",
    "# concatenating personal_email_domain with friends DataFrame\n",
    "apchef = pd.concat([apchef, email_df.loc[:, 'EMAIL_DOMAIN']],\n",
    "                    axis = 1)  # 0 for row, 1 for col\n",
    "\n",
    "# printing value counts of personal_email_domain\n",
    "apchef.loc[: ,'EMAIL_DOMAIN'].value_counts()\n",
    "\n",
    "##################################################\n",
    "# aggregate domains into higher-level categories #\n",
    "\n",
    "# email domain types\n",
    "personal_email_domains = ['gmail.com', 'protonmail.com', 'yahoo.com',\n",
    "                          'msn.com','aol.com', 'passport.com',\n",
    "                          'hotmail.com', 'live.com', 'me.com']\n",
    "corporate_email_domains  = ['amex.com', 'cocacola.com', 'merck.com',\n",
    "                          'mcdonalds.com', 'jnj.com', 'nike.com', 'apple.com', \n",
    "                          'dupont.com', 'ge.org', 'ibm.com', 'chevron.com', \n",
    "                          'microsoft.com', 'unitedhealth.com','exxon.com', \n",
    "                          'travelers.com','boeing.com', 'caterpillar.com', \n",
    "                          'pg.com','verizon.com', 'mmm.com', 'disney.com', \n",
    "                          'walmart.com', 'pfizer.com', 'visa.com', \n",
    "                          'jpmorgan.com', 'goldmansacs.com', 'cisco.com',\n",
    "                          'unitedtech.com', 'intel.com', 'homedepot.com']\n",
    "\n",
    "# placeholder list\n",
    "placeholder_lst = []\n",
    "\n",
    "# looping to group observations by domain type\n",
    "for domain in apchef['EMAIL_DOMAIN']:\n",
    "        if domain in personal_email_domains:\n",
    "            placeholder_lst.append('personal')\n",
    "            \n",
    "        elif domain in corporate_email_domains:\n",
    "            placeholder_lst.append('corporate')\n",
    "            \n",
    "        else:\n",
    "            print('Unknown')\n",
    "\n",
    "# concatenating with original DataFrame\n",
    "apchef['DOMAIN_GROUP'] = pd.Series(placeholder_lst)\n",
    "\n",
    "# checking results\n",
    "# apchef['DOMAIN_GROUP'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outlier analysis\n",
    "\n",
    "In the section below, I identify outliers through visualization and develop \n",
    "features that reflect the outlier thresholds.\n",
    "The histograms plotting has been commented out to speed up processing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:35.920269Z",
     "start_time": "2020-02-06T15:33:31.133988Z"
    }
   },
   "outputs": [],
   "source": [
    "### Visual EDA (Histograms) ###\n",
    "\n",
    "# fig, ax = plt.subplots(figsize = (10,8))\n",
    "# plt.subplot(2,2,1)\n",
    "# sns.distplot(apchef['REVENUE'],\n",
    "#            bins = 'fd',\n",
    "#            color = 'g')\n",
    "# plt.xlabel('REVENUE')\n",
    "\n",
    "##############################\n",
    " \n",
    "# plt.subplot(2,2,2)\n",
    "# sns.distplot(apchef['CROSS_SELL_SUCCESS'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'y')\n",
    "# plt.xlabel('CROSS_SELL_SUCCESS')\n",
    "\n",
    "##############################\n",
    " \n",
    "# plt.subplot(2,2,3)\n",
    "# sns.distplot(apchef['TOTAL_MEALS_ORDERED'],\n",
    "#            bins = 'fd',\n",
    "#            color = 'orange')\n",
    "# plt.xlabel('TOTAL_MEALS_ORDERED')\n",
    "\n",
    "##############################\n",
    " \n",
    "# plt.subplot(2,2,4)\n",
    "# sns.distplot(apchef['UNIQUE_MEALS_PURCH'],\n",
    "#            bins = 'fd',\n",
    "#            color = 'r')\n",
    "# plt.xlabel('UNIQUE_MEALS_PURCH')\n",
    "# plt.tight_layout()\n",
    "# plt.savefig('Ap Chef Data Histograms 1 of 7.png')\n",
    "# plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "# fig, ax = plt.subplots(figsize = (10,8))\n",
    "# plt.subplot(2,2,1)\n",
    "# sns.distplot(apchef['CONTACTS_W_CUSTOMER_SERVICE'],\n",
    "#             bins = 'fd',\n",
    "#            color = 'g')\n",
    "# plt.xlabel('CONTACTS_W_CUSTOMER_SERVICE')\n",
    "\n",
    "##############################\n",
    " \n",
    "# plt.subplot(2,2,2)\n",
    "# sns.distplot(apchef['PRODUCT_CATEGORIES_VIEWED'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'y')\n",
    "#plt.xlabel('PRODUCT_CATEGORIES_VIEWED')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.distplot(apchef['AVG_TIME_PER_SITE_VISIT'],\n",
    "#            bins = 'fd',\n",
    "#            color = 'orange')\n",
    "#plt.xlabel('AVG_TIME_PER_SITE_VISIT')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.distplot(apchef['MOBILE_NUMBER'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'r')\n",
    "#plt.xlabel('MOBILE_NUMBER')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 2 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.distplot(apchef['CANCELLATIONS_BEFORE_NOON'],\n",
    "#            bins = 'fd',\n",
    "#           color = 'g')\n",
    "#plt.xlabel('CANCELLATIONS_BEFORE_NOON')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.distplot(apchef['CANCELLATIONS_AFTER_NOON'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'y')\n",
    "#plt.xlabel('CANCELLATIONS_AFTER_NOON')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.distplot(apchef['TASTES_AND_PREFERENCES'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'orange')\n",
    "#plt.xlabel('TASTES_AND_PREFERENCES')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.distplot(apchef['MOBILE_LOGINS'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'r')\n",
    "#plt.xlabel('MOBILE_LOGINS')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 3 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.distplot(apchef['PC_LOGINS'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'g')\n",
    "#plt.xlabel('PC_LOGINS')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.distplot(apchef['WEEKLY_PLAN'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'y')\n",
    "#plt.xlabel('WEEKLY_PLAN')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.distplot(apchef['EARLY_DELIVERIES'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'orange')\n",
    "#plt.xlabel('EARLY_DELIVERIES')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.distplot(apchef['LATE_DELIVERIES'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'r')\n",
    "#plt.xlabel('LATE_DELIVERIES')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 4 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.distplot(apchef['PACKAGE_LOCKER'],\n",
    "#            bins = 'fd',\n",
    "#            kde = False,\n",
    "#            rug = True,\n",
    "#            color = 'g')\n",
    "#plt.xlabel('PACKAGE_LOCKER')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.distplot(apchef['REFRIGERATED_LOCKER'],\n",
    " #           bins = 'fd',\n",
    " #           kde = False,\n",
    " #           rug = True,\n",
    " #           color = 'y')\n",
    "#plt.xlabel('REFRIGERATED_LOCKER')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.distplot(apchef['FOLLOWED_RECOMMENDATIONS_PCT'],\n",
    " #           bins = 'fd',\n",
    " #           kde = False,\n",
    " #           rug = True,\n",
    " #           color = 'orange')\n",
    "#plt.xlabel('FOLLOWED_RECOMMENDATIONS_PCT')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.distplot(apchef['AVG_PREP_VID_TIME'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'r')\n",
    "#plt.xlabel('AVG_PREP_VID_TIME')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 5 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.distplot(apchef['LARGEST_ORDER_SIZE'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'g')\n",
    "#plt.xlabel('LARGEST_ORDER_SIZE')\n",
    "\n",
    "##############################\n",
    "\n",
    "#plt.subplot(2,2,2)\n",
    "#sns.distplot(apchef['MASTER_CLASSES_ATTENDED'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'y')\n",
    "#plt.xlabel('MASTER_CLASSES_ATTENDED')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.distplot(apchef['MEDIAN_MEAL_RATING'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'orange')\n",
    "#plt.xlabel('MEDIAN_MEAL_RATING')\n",
    "\n",
    "##############################\n",
    "\n",
    "#plt.subplot(2,2,4)\n",
    "#sns.distplot(apchef['AVG_CLICKS_PER_VISIT'],\n",
    " #           bins = 'fd',\n",
    " #           color = 'r')\n",
    "#plt.xlabel('AVG_CLICKS_PER_VISIT')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 6 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.distplot(apchef['TOTAL_PHOTOS_VIEWED'],\n",
    " #           bins = 'fd',\n",
    " #           kde = False,\n",
    " #           rug = True,\n",
    " #           color = 'orange')\n",
    "#plt.xlabel('TOTAL_PHOTOS_VIEWED')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Histograms 7 of 7.png')\n",
    "#plt.show()\n",
    "\n",
    "# setting outlier thresholds\n",
    "REVENUE_hi      = 2400\n",
    "TOTAL_MEALS_ORDERED_lo = 15\n",
    "TOTAL_MEALS_ORDERED_hi  = 120\n",
    "CONTACTS_W_CUSTOMER_SERVICE_hi  = 10\n",
    "AVG_TIME_PER_SITE_VISIT_hi  = 180\n",
    "WEEKLY_PLAN_lo  = 1\n",
    "WEEKLY_PLAN_hi = 14\n",
    "CANCELLATIONS_BEFORE_NOON_hi  = 5\n",
    "EARLY_DELIVERIES_lo = 1\n",
    "EARLY_DELIVERIES_hi = 5\n",
    "LATE_DELIVERIES_lo = 10\n",
    "AVG_PREP_VID_TIME_lo    = 60\n",
    "AVG_PREP_VID_TIME_hi    = 190\n",
    "LARGEST_ORDER_SIZE_lo     = 1\n",
    "LARGEST_ORDER_SIZE_hi     = 9\n",
    "MASTER_CLASSES_ATTENDED_hi     = 1\n",
    "MEDIAN_MEAL_RATING_lo = 2\n",
    "MEDIAN_MEAL_RATING_hi  = 4\n",
    "AVG_CLICKS_PER_VISIT_lo  = 7.5\n",
    "AVG_CLICKS_PER_VISIT_hi  = 20\n",
    "TOTAL_PHOTOS_VIEWED_lo = 1\n",
    "\n",
    "##############################################################################\n",
    "## Feature Engineering (outlier thresholds)                                 ##\n",
    "##############################################################################\n",
    "\n",
    "# developing features (columns) for outliers\n",
    "\n",
    "# Revenue\n",
    "apchef['out_REVENUE'] = 0\n",
    "condition_hi = apchef.loc[0:,'out_REVENUE'][apchef['REVENUE'] > REVENUE_hi]\n",
    "\n",
    "apchef['out_REVENUE'].replace(to_replace = condition_hi,\n",
    "                            value      = 1,\n",
    "                            inplace    = True)\n",
    "\n",
    "# Total meals ordered\n",
    "apchef['out_TOTAL_MEALS_ORDERED'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_TOTAL_MEALS_ORDERED'][apchef['TOTAL_MEALS_ORDERED'] < TOTAL_MEALS_ORDERED_lo]\n",
    "condition_hi = apchef.loc[0:,'out_TOTAL_MEALS_ORDERED'][apchef['TOTAL_MEALS_ORDERED'] > TOTAL_MEALS_ORDERED_hi]\n",
    "\n",
    "apchef['out_TOTAL_MEALS_ORDERED'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_TOTAL_MEALS_ORDERED'].replace(to_replace = condition_hi,\n",
    "                                        value      = 1,\n",
    "                                        inplace    = True)\n",
    "\n",
    "# Contacts with customer service\n",
    "apchef['out_CONTACTS_W_CUSTOMER_SERVICE'] = 0\n",
    "condition_hi = apchef.loc[0:,'out_CONTACTS_W_CUSTOMER_SERVICE'][apchef['CONTACTS_W_CUSTOMER_SERVICE'] > CONTACTS_W_CUSTOMER_SERVICE_hi]\n",
    "\n",
    "apchef['out_CONTACTS_W_CUSTOMER_SERVICE'].replace(to_replace = condition_hi,\n",
    "                                                value      = 1,\n",
    "                                                inplace    = True)\n",
    "\n",
    "# Average time per site visit\n",
    "apchef['out_AVG_TIME_PER_SITE_VISIT'] = 0\n",
    "condition_hi = apchef.loc[0:,'out_AVG_TIME_PER_SITE_VISIT'][apchef['AVG_TIME_PER_SITE_VISIT'] > AVG_TIME_PER_SITE_VISIT_hi]\n",
    "\n",
    "apchef['out_AVG_TIME_PER_SITE_VISIT'].replace(to_replace = condition_hi,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "\n",
    "# Weekly plan\n",
    "apchef['out_WEEKLY_PLAN'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_WEEKLY_PLAN'][apchef['WEEKLY_PLAN'] < WEEKLY_PLAN_lo]\n",
    "condition_hi = apchef.loc[0:,'out_WEEKLY_PLAN'][apchef['WEEKLY_PLAN'] > WEEKLY_PLAN_hi]\n",
    "\n",
    "apchef['out_WEEKLY_PLAN'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_WEEKLY_PLAN'].replace(to_replace = condition_hi,\n",
    "                                value      = 1,\n",
    "                                inplace    = True)\n",
    "\n",
    "# Cancellations before noon\n",
    "apchef['out_CANCELLATIONS_BEFORE_NOON'] = 0\n",
    "condition_hi = apchef.loc[0:,'out_CANCELLATIONS_BEFORE_NOON'][apchef['CANCELLATIONS_BEFORE_NOON'] > CANCELLATIONS_BEFORE_NOON_hi]\n",
    "\n",
    "apchef['out_CANCELLATIONS_BEFORE_NOON'].replace(to_replace = condition_hi,\n",
    "                                                value      = 1,\n",
    "                                                inplace    = True)\n",
    "\n",
    "# Early deliveries\n",
    "apchef['out_EARLY_DELIVERIES'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_EARLY_DELIVERIES'][apchef['EARLY_DELIVERIES'] < EARLY_DELIVERIES_lo]\n",
    "condition_hi = apchef.loc[0:,'out_EARLY_DELIVERIES'][apchef['EARLY_DELIVERIES'] > EARLY_DELIVERIES_hi]\n",
    "\n",
    "apchef['out_EARLY_DELIVERIES'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_EARLY_DELIVERIES'].replace(to_replace = condition_hi,\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "\n",
    "# Late deliveries\n",
    "apchef['out_LATE_DELIVERIES'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_LATE_DELIVERIES'][apchef['LATE_DELIVERIES'] < LATE_DELIVERIES_lo]\n",
    "\n",
    "apchef['out_LATE_DELIVERIES'].replace(to_replace = condition_hi,\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "\n",
    "\n",
    "# Average prep video time\n",
    "apchef['out_AVG_PREP_VID_TIME'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_AVG_PREP_VID_TIME'][apchef['AVG_PREP_VID_TIME'] < AVG_PREP_VID_TIME_lo]\n",
    "condition_hi = apchef.loc[0:,'out_AVG_PREP_VID_TIME'][apchef['AVG_PREP_VID_TIME'] > AVG_PREP_VID_TIME_hi]\n",
    "\n",
    "apchef['out_AVG_PREP_VID_TIME'].replace(to_replace = condition_lo,\n",
    "                                     value      = 1,\n",
    "                                     inplace    = True)\n",
    "apchef['out_AVG_PREP_VID_TIME'].replace(to_replace = condition_hi,\n",
    "                                     value      = 1,\n",
    "                                     inplace    = True)\n",
    "\n",
    "# Largest order size\n",
    "apchef['out_LARGEST_ORDER_SIZE'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_LARGEST_ORDER_SIZE'][apchef['LARGEST_ORDER_SIZE'] < LARGEST_ORDER_SIZE_lo]\n",
    "condition_hi = apchef.loc[0:,'out_LARGEST_ORDER_SIZE'][apchef['LARGEST_ORDER_SIZE'] > LARGEST_ORDER_SIZE_hi]\n",
    "\n",
    "apchef['out_LARGEST_ORDER_SIZE'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_LARGEST_ORDER_SIZE'].replace(to_replace = condition_hi,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "\n",
    "# Median meal rating\n",
    "apchef['out_MEDIAN_MEAL_RATING'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_MEDIAN_MEAL_RATING'][apchef['MEDIAN_MEAL_RATING'] < MEDIAN_MEAL_RATING_lo]\n",
    "condition_hi = apchef.loc[0:,'out_MEDIAN_MEAL_RATING'][apchef['MEDIAN_MEAL_RATING'] > MEDIAN_MEAL_RATING_hi]\n",
    "\n",
    "apchef['out_MEDIAN_MEAL_RATING'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_MEDIAN_MEAL_RATING'].replace(to_replace = condition_hi,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "\n",
    "# Average clicks per visit\n",
    "apchef['out_AVG_CLICKS_PER_VISIT'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_AVG_CLICKS_PER_VISIT'][apchef['AVG_CLICKS_PER_VISIT'] < AVG_CLICKS_PER_VISIT_lo]\n",
    "condition_hi = apchef.loc[0:,'out_AVG_CLICKS_PER_VISIT'][apchef['AVG_CLICKS_PER_VISIT'] > AVG_CLICKS_PER_VISIT_hi]\n",
    "\n",
    "apchef['out_AVG_CLICKS_PER_VISIT'].replace(to_replace = condition_lo,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "apchef['out_AVG_CLICKS_PER_VISIT'].replace(to_replace = condition_hi,\n",
    "                                         value      = 1,\n",
    "                                         inplace    = True)\n",
    "\n",
    "# Master classes attended\n",
    "apchef['out_MASTER_CLASSES_ATTENDED'] = 0\n",
    "condition_hi = apchef.loc[0:,'out_MASTER_CLASSES_ATTENDED'][apchef['MASTER_CLASSES_ATTENDED'] > MASTER_CLASSES_ATTENDED_hi]\n",
    "\n",
    "apchef['out_MASTER_CLASSES_ATTENDED'].replace(to_replace = condition_hi,\n",
    "                                             value      = 1,\n",
    "                                             inplace    = True)\n",
    "\n",
    "# Total photos viewed\n",
    "apchef['out_TOTAL_PHOTOS_VIEWED'] = 0\n",
    "condition_lo = apchef.loc[0:,'out_TOTAL_PHOTOS_VIEWED'][apchef['TOTAL_PHOTOS_VIEWED'] < TOTAL_PHOTOS_VIEWED_lo]\n",
    "\n",
    "apchef['out_TOTAL_PHOTOS_VIEWED'].replace(to_replace = condition_lo,\n",
    "                                             value      = 1,\n",
    "                                             inplace    = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T06:28:33.765159Z",
     "start_time": "2020-02-06T06:28:29.339000Z"
    }
   },
   "source": [
    "### Trend analysis\n",
    "\n",
    "In the section below, I identify where a variable's trend changes in relationship\n",
    "with the revenue through visualization and build trend based features.\n",
    "The scatter plots have been commented out to speed up processing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:38.503126Z",
     "start_time": "2020-02-06T15:33:35.937135Z"
    }
   },
   "outputs": [],
   "source": [
    "### Visual EDA (Scatterplots) ###\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['CROSS_SELL_SUCCESS'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('CROSS_SELL_SUCCESS')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['TOTAL_MEALS_ORDERED'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('TOTAL_MEALS_ORDERED')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['UNIQUE_MEALS_PURCH'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('UNIQUE_MEALS_PURCH')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['CONTACTS_W_CUSTOMER_SERVICE'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('CONTACTS_W_CUSTOMER_SERVICE')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 1 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['PRODUCT_CATEGORIES_VIEWED'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('PRODUCT_CATEGORIES_VIEWED')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['AVG_TIME_PER_SITE_VISIT'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('AVG_TIME_PER_SITE_VISIT')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['MOBILE_NUMBER'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('MOBILE_NUMBER')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['CANCELLATIONS_BEFORE_NOON'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('CANCELLATIONS_BEFORE_NOON')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 2 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['CANCELLATIONS_AFTER_NOON'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('CANCELLATIONS_AFTER_NOON')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['TASTES_AND_PREFERENCES'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('TASTES_AND_PREFERENCES')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['MOBILE_LOGINS'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('MOBILE_LOGINS')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['PC_LOGINS'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('PC_LOGINS')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 3 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['WEEKLY_PLAN'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('WEEKLY_PLAN')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['EARLY_DELIVERIES'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('EARLY_DELIVERIES')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['LATE_DELIVERIES'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('LATE_DELIVERIES')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['PACKAGE_LOCKER'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('PACKAGE_LOCKER')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 4 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['REFRIGERATED_LOCKER'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('REFRIGERATED_LOCKER')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['FOLLOWED_RECOMMENDATIONS_PCT'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('FOLLOWED_RECOMMENDATIONS_PCT')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['AVG_PREP_VID_TIME'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('AVG_PREP_VID_TIME')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['LARGEST_ORDER_SIZE'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('LARGEST_ORDER_SIZE')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 5 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "##############################\n",
    "##############################\n",
    "\n",
    "#fig, ax = plt.subplots(figsize = (10,8))\n",
    "#plt.subplot(2,2,1)\n",
    "#sns.scatterplot(x = apchef['MASTER_CLASSES_ATTENDED'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'g')\n",
    "#plt.xlabel('MASTER_CLASSES_ATTENDED')\n",
    "\n",
    "##############################\n",
    "\n",
    "#plt.subplot(2,2,2)\n",
    "#sns.scatterplot(x = apchef['MEDIAN_MEAL_RATING'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'y')\n",
    "#plt.xlabel('MEDIAN_MEAL_RATING')\n",
    "\n",
    "##############################\n",
    " \n",
    "#plt.subplot(2,2,3)\n",
    "#sns.scatterplot(x = apchef['AVG_CLICKS_PER_VISIT'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'orange')\n",
    "#plt.xlabel('AVG_CLICKS_PER_VISIT')\n",
    "\n",
    "##############################\n",
    "\n",
    "#plt.subplot(2,2,4)\n",
    "#sns.scatterplot(x = apchef['TOTAL_PHOTOS_VIEWED'],\n",
    " #               y = apchef['REVENUE'],\n",
    " #               color = 'r')\n",
    "#plt.xlabel('TOTAL_PHOTOS_VIEWED')\n",
    "#plt.tight_layout()\n",
    "#plt.savefig('Ap Chef Data Scatterplot 6 of 6.png')\n",
    "#plt.show()\n",
    "\n",
    "# setting trend-based thresholds\n",
    "\n",
    "TOTAL_MEALS_ORDERED_change_hi  = 15 # trend changes above this point\n",
    "UNIQUE_MEALS_PURCH_change_hi = 9 # data scatters above this point\n",
    "CONTACTS_W_CUSTOMER_SERVICE_change_hi  = 10 # trend changes above this point\n",
    "AVG_TIME_PER_SITE_VISIT_change_hi  = 300 # data scatters above this point\n",
    "CANCELLATIONS_BEFORE_NOON_change_hi  = 5 # data scatters above this point\n",
    "CANCELLATIONS_AFTER_NOON_change_hi  = 2 # trend changes above this point\n",
    "MOBILE_LOGINS_change_hi = 6 # trend changes above this point\n",
    "PC_LOGINS_change_hi = 2 # trend changes above this point\n",
    "LATE_DELIVERIES_change_hi = 10 # data scatters above this point\n",
    "AVG_PREP_VID_TIME_change_hi = 300 # data scatters above this point\n",
    "LARGEST_ORDER_SIZE_change_hi = 8 # trend changes above this point\n",
    "MASTER_CLASSES_ATTENDED_change_hi  = 2 # trend changes above this point\n",
    "MEDIAN_MEAL_RATING_change_hi = 4 # trend changes above this point\n",
    "\n",
    "MOBILE_LOGINS_change_lo = 4 # data scatters below this point\n",
    "AVG_CLICKS_PER_VISIT_change_lo  = 10 # data scatters below this point\n",
    "PC_LOGINS_change_lo = 1 # data scatters below this point\n",
    "\n",
    "TOTAL_PHOTOS_VIEWED_change_at   = 0 # zero inflated\n",
    "\n",
    "# developing features (columns) for trend based thresholds\n",
    "\n",
    "########################################\n",
    "## change above threshold             ##\n",
    "########################################\n",
    "\n",
    "# greater than sign\n",
    " \n",
    "# Total meals ordered\n",
    "apchef['change_TOTAL_MEALS_ORDERED'] = 0\n",
    "condition = apchef.loc[0:,'change_TOTAL_MEALS_ORDERED'][apchef['TOTAL_MEALS_ORDERED'] > TOTAL_MEALS_ORDERED_change_hi]\n",
    "\n",
    "apchef['change_TOTAL_MEALS_ORDERED'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "# Unique meals purchased\n",
    "apchef['change_UNIQUE_MEALS_PURCH'] = 0\n",
    "condition = apchef.loc[0:,'change_UNIQUE_MEALS_PURCH'][apchef['UNIQUE_MEALS_PURCH'] > UNIQUE_MEALS_PURCH_change_hi]\n",
    "\n",
    "apchef['change_UNIQUE_MEALS_PURCH'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "\n",
    "# Contacts with customer service\n",
    "apchef['change_CONTACTS_W_CUSTOMER_SERVICE'] = 0\n",
    "condition = apchef.loc[0:,'change_CONTACTS_W_CUSTOMER_SERVICE'][apchef['CONTACTS_W_CUSTOMER_SERVICE'] > CONTACTS_W_CUSTOMER_SERVICE_change_hi]\n",
    "\n",
    "apchef['change_CONTACTS_W_CUSTOMER_SERVICE'].replace(to_replace = condition,\n",
    "                                                    value      = 1,\n",
    "                                                    inplace    = True)\n",
    "\n",
    "# Average time per site visit\n",
    "apchef['change_AVG_TIME_PER_SITE_VISIT'] = 0\n",
    "condition = apchef.loc[0:,'change_AVG_TIME_PER_SITE_VISIT'][apchef['AVG_TIME_PER_SITE_VISIT'] > AVG_TIME_PER_SITE_VISIT_change_hi]\n",
    "\n",
    "apchef['change_AVG_TIME_PER_SITE_VISIT'].replace(to_replace = condition,\n",
    "                                                value      = 1,\n",
    "                                                inplace    = True)\n",
    "\n",
    "# Cancellations before noon\n",
    "apchef['change_CANCELLATIONS_BEFORE_NOON'] = 0\n",
    "condition = apchef.loc[0:,'change_CANCELLATIONS_BEFORE_NOON'][apchef['CANCELLATIONS_BEFORE_NOON'] > CANCELLATIONS_BEFORE_NOON_change_hi]\n",
    "\n",
    "apchef['change_CANCELLATIONS_BEFORE_NOON'].replace(to_replace = condition,\n",
    "                                                value      = 1,\n",
    "                                                inplace    = True)\n",
    "\n",
    "# Cancellations after noon\n",
    "apchef['change_CANCELLATIONS_AFTER_NOON'] = 0\n",
    "condition = apchef.loc[0:,'change_CANCELLATIONS_AFTER_NOON'][apchef['CANCELLATIONS_AFTER_NOON'] > CANCELLATIONS_AFTER_NOON_change_hi]\n",
    "\n",
    "apchef['change_CANCELLATIONS_AFTER_NOON'].replace(to_replace = condition,\n",
    "                                                value      = 1,\n",
    "                                                inplace    = True)\n",
    "\n",
    "# Mobile logins\n",
    "apchef['change_MOBILE_LOGINS'] = 0\n",
    "condition = apchef.loc[0:,'change_MOBILE_LOGINS'][apchef['MOBILE_LOGINS'] > MOBILE_LOGINS_change_hi]\n",
    "\n",
    "apchef['change_MOBILE_LOGINS'].replace(to_replace = condition,\n",
    "                                        value      = 1,\n",
    "                                        inplace    = True)\n",
    "\n",
    "# PC logins\n",
    "apchef['change_PC_LOGINS'] = 0\n",
    "condition = apchef.loc[0:,'change_PC_LOGINS'][apchef['PC_LOGINS'] > PC_LOGINS_change_hi]\n",
    "\n",
    "apchef['change_PC_LOGINS'].replace(to_replace = condition,\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "\n",
    "# Late deliveries\n",
    "apchef['change_LATE_DELIVERIES'] = 0\n",
    "condition = apchef.loc[0:,'change_LATE_DELIVERIES'][apchef['LATE_DELIVERIES'] > LATE_DELIVERIES_change_hi]\n",
    "\n",
    "apchef['change_LATE_DELIVERIES'].replace(to_replace = condition,\n",
    "                                        value      = 1,\n",
    "                                        inplace    = True)\n",
    "\n",
    "# Average prep video time\n",
    "apchef['change_AVG_PREP_VID_TIME'] = 0\n",
    "condition = apchef.loc[0:,'change_AVG_PREP_VID_TIME'][apchef['AVG_PREP_VID_TIME'] > AVG_PREP_VID_TIME_change_hi]\n",
    "\n",
    "apchef['change_AVG_PREP_VID_TIME'].replace(to_replace = condition,\n",
    "                                        value      = 1,\n",
    "                                        inplace    = True)\n",
    "\n",
    "# Largest order size\n",
    "apchef['change_LARGEST_ORDER_SIZE'] = 0\n",
    "condition = apchef.loc[0:,'change_LARGEST_ORDER_SIZE'][apchef['LARGEST_ORDER_SIZE'] > LARGEST_ORDER_SIZE_change_hi]\n",
    "\n",
    "apchef['change_LARGEST_ORDER_SIZE'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "\n",
    "# Median meal rating\n",
    "apchef['change_MEDIAN_MEAL_RATING'] = 0\n",
    "condition = apchef.loc[0:,'change_MEDIAN_MEAL_RATING'][apchef['MEDIAN_MEAL_RATING'] > MEDIAN_MEAL_RATING_change_hi]\n",
    "\n",
    "apchef['change_MEDIAN_MEAL_RATING'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "\n",
    "########################################\n",
    "## change at threshold                ##\n",
    "########################################\n",
    "\n",
    "# double-equals sign\n",
    "\n",
    "# Total photos viewed\n",
    "apchef['change_TOTAL_PHOTOS_VIEWED'] = 0\n",
    "condition = apchef.loc[0:,'change_TOTAL_PHOTOS_VIEWED'][apchef['TOTAL_PHOTOS_VIEWED'] == TOTAL_PHOTOS_VIEWED_change_at]\n",
    "\n",
    "apchef['change_TOTAL_PHOTOS_VIEWED'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)\n",
    "\n",
    "# checking results\n",
    "# apchef.head(n = 10)\n",
    "\n",
    "########################################\n",
    "## change below threshold             ##\n",
    "########################################\n",
    "\n",
    "# less than sign\n",
    "\n",
    "# Mobile logins\n",
    "condition = apchef.loc[0:,'change_MOBILE_LOGINS'][apchef['MOBILE_LOGINS'] < MOBILE_LOGINS_change_lo]\n",
    "\n",
    "apchef['change_MOBILE_LOGINS'].replace(to_replace = condition,\n",
    "                                        value      = 1,\n",
    "                                        inplace    = True)\n",
    "\n",
    "# PC logins\n",
    "condition = apchef.loc[0:,'change_PC_LOGINS'][apchef['PC_LOGINS'] < PC_LOGINS_change_lo]\n",
    "\n",
    "apchef['change_PC_LOGINS'].replace(to_replace = condition,\n",
    "                                    value      = 1,\n",
    "                                    inplace    = True)\n",
    "\n",
    "# Average clicks per visit\n",
    "apchef['change_AVG_CLICKS_PER_VISIT'] = 0\n",
    "condition = apchef.loc[0:,'change_AVG_CLICKS_PER_VISIT'][apchef['AVG_CLICKS_PER_VISIT'] < AVG_CLICKS_PER_VISIT_change_lo]\n",
    "\n",
    "apchef['change_AVG_CLICKS_PER_VISIT'].replace(to_replace = condition,\n",
    "                                            value      = 1,\n",
    "                                            inplace    = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Encoding categorical variables\n",
    "\n",
    "In the section below, I identify all the categorical variables, conduct analysis\n",
    "through visualization and build dummy variables to replace object type data.\n",
    "The dummy variable columns are added to the original dataset where the \n",
    "categorical columns are dropped to prevent errors when running models.\n",
    "\n",
    "The box plots have been commented out to speed up processing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:38.545775Z",
     "start_time": "2020-02-06T15:33:38.505933Z"
    }
   },
   "outputs": [],
   "source": [
    "# printing columns where the variables for analysis are categorical\n",
    "\n",
    "# print(f\"\"\"\n",
    "# FAMILY_GROUP\n",
    "# {apchef['FAMILY_GROUP'].value_counts()}\n",
    "\n",
    "# EMAIL_DOMAIN\n",
    "# {apchef['EMAIL_DOMAIN'].value_counts()}\n",
    "\n",
    "# DOMAIN_GROUP\n",
    "# {apchef['DOMAIN_GROUP'].value_counts()}\n",
    "# \"\"\")\n",
    "\n",
    "# defining a function for categorical boxplots\n",
    "# def categorical_boxplots(response, cat_var, data):\n",
    "#     \"\"\"\n",
    "# \tThis function can be used for categorical variables\n",
    "\n",
    "# \tPARAMETERS\n",
    "# \t----------\n",
    "# \tresponse : str, response variable\n",
    "# \tcat_var  : str, categorical variable\n",
    "# \tdata     : DataFrame of the response and categorical variables\n",
    "# \t\"\"\"\n",
    "\n",
    "#    apchef.boxplot(column \t     = response,\n",
    "#    \t           by \t \t\t = cat_var,\n",
    "#        \t       vert \t     = False,\n",
    "#            \t   patch_artist = False,\n",
    "#                   meanline     = True,\n",
    "#               \t   showmeans    = True)\n",
    "\n",
    "#    plt.suptitle(\"\")\n",
    "#    plt.savefig(f'Ap Chef Data Boxplot {cat_var}.png')\n",
    "#    plt.show()\n",
    "\n",
    "# calling the function for each categorical variable\n",
    "#categorical_boxplots(response = 'REVENUE',\n",
    "#\t\t\t\t\t cat_var  = 'FAMILY_GROUP',\n",
    "#\t\t\t\t\t data     = apchef)\n",
    "#categorical_boxplots(response = 'REVENUE',\n",
    "#\t\t\t\t\t cat_var  = 'EMAIL_DOMAIN',\n",
    "#\t\t\t\t\t data     = apchef)\n",
    "#categorical_boxplots(response = 'REVENUE',\n",
    "#\t\t\t\t\t cat_var  = 'DOMAIN_GROUP',\n",
    "#\t\t\t\t\t data     = apchef)\n",
    "\n",
    "# one hot encoding categorical variables\n",
    "one_hot_FAMILY_GROUP       = pd.get_dummies(apchef['FAMILY_GROUP'])\n",
    "one_hot_EMAIL_DOMAIN       = pd.get_dummies(apchef['EMAIL_DOMAIN'])\n",
    "one_hot_DOMAIN_GROUP       = pd.get_dummies(apchef['DOMAIN_GROUP'])\n",
    "\n",
    "# dropping categorical variables after they've been encoded\n",
    "apchef = apchef.drop('FAMILY_GROUP', axis = 1)\n",
    "apchef = apchef.drop('EMAIL_DOMAIN', axis = 1)\n",
    "apchef = apchef.drop('DOMAIN_GROUP', axis = 1)\n",
    "\n",
    "# joining codings together\n",
    "apchef = apchef.join([one_hot_FAMILY_GROUP, one_hot_EMAIL_DOMAIN, one_hot_DOMAIN_GROUP])\n",
    "\n",
    "# saving new columns\n",
    "new_columns = apchef.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model building\n",
    "\n",
    "Now that I have finished developing features, we are ready to start the model \n",
    "building. As a first step, let's see how the explanatory variables correlate to \n",
    "the response variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:38.679672Z",
     "start_time": "2020-02-06T15:33:38.551700Z"
    }
   },
   "outputs": [],
   "source": [
    "# dropping categorical variables \n",
    "apchef = apchef.drop(labels = ['NAME','EMAIL','FIRST_NAME', 'FAMILY_NAME'], \n",
    "                     axis = 1)\n",
    "\n",
    "# checking the results\n",
    "# apchef.columns\n",
    "\n",
    "df_corr = apchef.corr().round(2)\n",
    "\n",
    "# printing (Pearson) correlations with Revenue\n",
    "# df_corr['CROSS_SELL_SUCCESS'].sort_values(ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train split test with Stratification\n",
    "\n",
    "It is important to prevent a response variable that is too imbalanced which can \n",
    "cause issues in our modeling. Therefore, I performed stratification in the \n",
    "training dataset in the train split test stage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:38.713291Z",
     "start_time": "2020-02-06T15:33:38.683219Z"
    }
   },
   "outputs": [],
   "source": [
    "# declaring explanatory variables\n",
    "apchef_explanatory = apchef.drop(labels='CROSS_SELL_SUCCESS', \n",
    "                            axis=1)\n",
    "\n",
    "# declaring response variable\n",
    "apchef_target = apchef.loc[:,'CROSS_SELL_SUCCESS']\n",
    "\n",
    "# train-test split with stratification\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "            apchef_explanatory,\n",
    "            apchef_target,\n",
    "            test_size = 0.25,\n",
    "            random_state = 219, \n",
    "            stratify = apchef_target)\n",
    "\n",
    "# merging training data for statsmodels\n",
    "apchef_train = pd.concat([X_train, y_train], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T06:32:57.398878Z",
     "start_time": "2020-02-06T06:32:57.295723Z"
    }
   },
   "source": [
    "### Logistic regression in statsmodels\n",
    "\n",
    "To start with, we build a logistic regression model in statsmodels using all of \n",
    "the explanatory variables. Based on the output above, we remove all variables \n",
    "that were deemed insignificant based on their p-values. \n",
    "\n",
    "Then, we check the p-values again to ensure that all variables are still \n",
    "significant (nan are insignificant and the corresponding variables are removed). \n",
    "\n",
    "The insignificant p-values are removed in an meticulous and incremental fashion \n",
    "due to high collinearity among the variables (i.e. outlier flags and trend based\n",
    "thresholds derived from the same column). The correlation matrix is also used \n",
    "for  choosing which variable to eliminate first when it comes to similar p-values: \n",
    "lower correlations are favored to be removed first. \n",
    "\n",
    "The summary yields valuable and interpretable insight on the variables that \n",
    "affect cross selling success.\n",
    "\n",
    "For productivity improvement, the following loop is written to print all \n",
    "explanatory variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T16:39:46.804651Z",
     "start_time": "2020-02-06T16:39:46.592655Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.465276\n",
      "         Iterations 7\n",
      "                           Logit Regression Results                           \n",
      "==============================================================================\n",
      "Dep. Variable:     CROSS_SELL_SUCCESS   No. Observations:                 1459\n",
      "Model:                          Logit   Df Residuals:                     1446\n",
      "Method:                           MLE   Df Model:                           12\n",
      "Date:                Thu, 06 Feb 2020   Pseudo R-squ.:                  0.2591\n",
      "Time:                        08:39:46   Log-Likelihood:                -678.84\n",
      "converged:                       True   LL-Null:                       -916.19\n",
      "Covariance Type:            nonrobust   LLR p-value:                 5.303e-94\n",
      "================================================================================================================\n",
      "                                                   coef    std err          z      P>|z|      [0.025      0.975]\n",
      "----------------------------------------------------------------------------------------------------------------\n",
      "Intercept                                       -1.6170      0.327     -4.945      0.000      -2.258      -0.976\n",
      "apchef_train['AVG_TIME_PER_SITE_VISIT']          0.0027      0.001      2.146      0.032       0.000       0.005\n",
      "apchef_train['MOBILE_NUMBER']                    0.6800      0.201      3.387      0.001       0.286       1.074\n",
      "apchef_train['CANCELLATIONS_AFTER_NOON']        -0.4759      0.149     -3.187      0.001      -0.769      -0.183\n",
      "apchef_train['TASTES_AND_PREFERENCES']           0.4937      0.146      3.385      0.001       0.208       0.780\n",
      "apchef_train['PC_LOGINS']                       -0.3195      0.126     -2.536      0.011      -0.566      -0.073\n",
      "apchef_train['REFRIGERATED_LOCKER']              0.5912      0.222      2.658      0.008       0.155       1.027\n",
      "apchef_train['FOLLOWED_RECOMMENDATIONS_PCT']     0.0592      0.004     15.079      0.000       0.051       0.067\n",
      "apchef_train['change_LATE_DELIVERIES']           1.6405      0.605      2.713      0.007       0.455       2.826\n",
      "apchef_train['Greyjoy']                         -1.5880      0.763     -2.081      0.037      -3.084      -0.092\n",
      "apchef_train['jpmorgan.com']                     2.6903      1.079      2.493      0.013       0.575       4.806\n",
      "apchef_train['microsoft.com']                    2.5974      1.072      2.424      0.015       0.497       4.698\n",
      "apchef_train['msn.com']                         -1.0830      0.303     -3.569      0.000      -1.678      -0.488\n",
      "================================================================================================================\n"
     ]
    }
   ],
   "source": [
    "# formatting each explanatory variable for statsmodels\n",
    "# for val in apchef_explanatory:\n",
    "#     print(f\"apchef_train['{val}'] +\")\n",
    "\n",
    "# instantiating a logistic regression model object\n",
    "logit_fit = smf.logit(formula = \"\"\" CROSS_SELL_SUCCESS ~ apchef_train['AVG_TIME_PER_SITE_VISIT'] +\n",
    "                                    apchef_train['MOBILE_NUMBER'] +\n",
    "                                    apchef_train['CANCELLATIONS_AFTER_NOON'] +\n",
    "                                    apchef_train['TASTES_AND_PREFERENCES'] +\n",
    "                                    apchef_train['PC_LOGINS'] +\n",
    "                                    apchef_train['REFRIGERATED_LOCKER'] +\n",
    "                                    apchef_train['FOLLOWED_RECOMMENDATIONS_PCT'] +\n",
    "                                    apchef_train['change_LATE_DELIVERIES'] +\n",
    "                                    apchef_train['Greyjoy'] +\n",
    "                                    apchef_train['jpmorgan.com'] +\n",
    "                                    apchef_train['microsoft.com'] +\n",
    "                                    apchef_train['msn.com']\"\"\",\n",
    "                                     data    = apchef_train)\n",
    "\n",
    "# fitting the model object\n",
    "logit_fit = logit_fit.fit()\n",
    "\n",
    "print(logit_fit.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:38.883014Z",
     "start_time": "2020-02-06T15:33:38.856562Z"
    }
   },
   "outputs": [],
   "source": [
    "# updating explanatory variables in scikit-learn \n",
    "x_variables = ['AVG_TIME_PER_SITE_VISIT', 'MOBILE_NUMBER', \n",
    "               'CANCELLATIONS_AFTER_NOON', 'TASTES_AND_PREFERENCES',\n",
    "               'PC_LOGINS', 'REFRIGERATED_LOCKER', \n",
    "               'FOLLOWED_RECOMMENDATIONS_PCT', 'change_LATE_DELIVERIES',\n",
    "               'Greyjoy', 'jpmorgan.com', 'microsoft.com', 'msn.com']\n",
    "\n",
    "# updating variable DataFrame based the the analysis above\n",
    "apchef_explanatory  = apchef_explanatory.loc[ : , x_variables]\n",
    "\n",
    "# updating train-test split with stratification\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "            apchef_explanatory,\n",
    "            apchef_target,\n",
    "            test_size = 0.25,\n",
    "            random_state = 219, \n",
    "            stratify = apchef_target)\n",
    "\n",
    "# updating training data \n",
    "apchef_train = pd.concat([X_train, y_train], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic regression in scikit-learn\n",
    "Build a logistic regression model in scikit-learn using the logit_sig \n",
    "explanatory variables and CROSS_SELL_SUCCESS as the  response variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:39.114651Z",
     "start_time": "2020-02-06T15:33:38.886322Z"
    }
   },
   "outputs": [],
   "source": [
    "# train/test split with the significant variables\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "            apchef_explanatory,\n",
    "            apchef_target,\n",
    "            random_state = 802,\n",
    "            test_size    = 0.25,\n",
    "            stratify     = apchef_target)\n",
    "\n",
    "\n",
    "# INSTANTIATING a logistic regression model\n",
    "logreg = LogisticRegression(solver = 'lbfgs',\n",
    "                            C = 1,\n",
    "                            max_iter = 1000,\n",
    "                            random_state = 219)\n",
    "\n",
    "\n",
    "# FITTING the training data\n",
    "logreg_fit = logreg.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "# PREDICTING based on the testing set\n",
    "logreg_pred = logreg_fit.predict(X_test)\n",
    "\n",
    "# creating an empty list\n",
    "model_performance = [['Model', 'Training Accuracy',\n",
    "                      'Testing Accuracy', 'AUC Value']]\n",
    "\n",
    "\n",
    "# train accuracy\n",
    "logreg_train_acc  = logreg_fit.score(X_train, y_train).round(4)\n",
    "\n",
    "\n",
    "# test accuracy\n",
    "logreg_test_acc   = logreg_fit.score(X_test, y_test).round(4)\n",
    "\n",
    "\n",
    "# auc value\n",
    "logreg_auc = roc_auc_score(y_true  = y_test,\n",
    "                           y_score = logreg_pred).round(4)\n",
    "\n",
    "# saving the results\n",
    "model_performance.append(['Logistic Regression',\n",
    "                          logreg_train_acc,\n",
    "                          logreg_test_acc,\n",
    "                          logreg_auc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KNN classification model\n",
    "\n",
    "In this section, I will scale the explanatory data and build a KNN \n",
    "classification model.\n",
    "\n",
    "#### User defined functions\n",
    "\n",
    "For productivity improvement, the following user defined functions is written \n",
    "to search for the optimal neighbors in KNN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:40.389644Z",
     "start_time": "2020-02-06T15:33:39.119019Z"
    }
   },
   "outputs": [],
   "source": [
    "########################################\n",
    "# optimal_neighbors\n",
    "########################################\n",
    "def optimal_neighbors(X_data,\n",
    "                      y_data,\n",
    "                      standardize = True,\n",
    "                      pct_test=0.25,\n",
    "                      seed=802,\n",
    "                      response_type='reg',\n",
    "                      max_neighbors=20):\n",
    "    \"\"\"\n",
    "Exhaustively compute training and testing results for KNN across\n",
    "[1, max_neighbors]. Outputs the maximum test score and (by default) a\n",
    "visualization of the results.\n",
    "PARAMETERS\n",
    "----------\n",
    "X_data        : explanatory variable data\n",
    "y_data        : response variable\n",
    "standardize   : whether or not to standardize the X data, default True\n",
    "pct_test      : test size for training and validation from (0,1), default 0.25\n",
    "seed          : random seed to be used in algorithm, default 802\n",
    "response_type : type of neighbors algorithm to use, default 'reg'\n",
    "    Use 'reg' for regression (KNeighborsRegressor)\n",
    "    Use 'class' for classification (KNeighborsClassifier)\n",
    "max_neighbors : maximum number of neighbors in exhaustive search, default 20\n",
    "\"\"\"    \n",
    "    \n",
    "    if standardize == True:\n",
    "        # optionally standardizing X_data\n",
    "        scaler             = StandardScaler()\n",
    "        scaler.fit(X_data)\n",
    "        X_scaled           = scaler.transform(X_data)\n",
    "        X_scaled_df        = pd.DataFrame(X_scaled)\n",
    "        X_data             = X_scaled_df\n",
    "\n",
    "    # train-test split\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X_data,\n",
    "                                                        y_data,\n",
    "                                                        test_size = pct_test,\n",
    "                                                        random_state = seed)\n",
    "\n",
    "    # creating lists for training set accuracy and test set accuracy\n",
    "    training_accuracy = []\n",
    "    test_accuracy = []\n",
    "    \n",
    "    # setting neighbor range\n",
    "    neighbors_settings = range(1, max_neighbors + 1)\n",
    "\n",
    "    for n_neighbors in neighbors_settings:\n",
    "        # building the model based on response variable type\n",
    "        if response_type == 'reg':\n",
    "            clf = KNeighborsRegressor(n_neighbors = n_neighbors)\n",
    "            clf.fit(X_train, y_train)\n",
    "            \n",
    "        elif response_type == 'class':\n",
    "            clf = KNeighborsClassifier(n_neighbors = n_neighbors)\n",
    "            clf.fit(X_train, y_train)            \n",
    "            \n",
    "        else:\n",
    "            print(\"Error: response_type must be 'reg' or 'class'\")\n",
    "        \n",
    "        # recording the training set accuracy\n",
    "        training_accuracy.append(clf.score(X_train, y_train))\n",
    "    \n",
    "        # recording the generalization accuracy\n",
    "        test_accuracy.append(clf.score(X_test, y_test))\n",
    "    \n",
    "    # returning optimal number of neighbors\n",
    "    # print(f\"The optimal number of neighbors is: {test_accuracy.index(max(test_accuracy))+1}\")\n",
    "    return test_accuracy.index(max(test_accuracy))+1\n",
    "\n",
    "########################################\n",
    "# visual_cm\n",
    "########################################\n",
    "def visual_cm(true_y, pred_y, labels = None):\n",
    "    \"\"\"\n",
    "Creates a visualization of a confusion matrix.\n",
    "\n",
    "PARAMETERS\n",
    "----------\n",
    "true_y : true values for the response variable\n",
    "pred_y : predicted values for the response variable\n",
    "labels : , default None\n",
    "    \"\"\"\n",
    "    # visualizing the confusion matrix\n",
    "\n",
    "    # setting labels\n",
    "    lbls = labels\n",
    "    \n",
    "\n",
    "    # declaring a confusion matrix object\n",
    "    cm = confusion_matrix(y_true = true_y,\n",
    "                          y_pred = pred_y)\n",
    "    print(cm)\n",
    "    \n",
    "\n",
    "# determining the optimal number of neighbors\n",
    "opt_neighbors = optimal_neighbors(X_data = X_train,\n",
    "                                 y_data = y_train)\n",
    "\n",
    "# INSTANTIATING StandardScaler()\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# FITTING the data\n",
    "scaler.fit(apchef_explanatory)\n",
    "\n",
    "# TRANSFORMING the data\n",
    "X_scaled     = scaler.transform(apchef_explanatory)\n",
    "\n",
    "# converting to a DataFrame\n",
    "X_scaled_df  = pd.DataFrame(X_scaled) \n",
    "\n",
    "\n",
    "# train-test split with the scaled data\n",
    "X_train_scaled, X_test_scaled, y_train_scaled, y_test_scaled = train_test_split(\n",
    "            X_scaled_df,\n",
    "            apchef_target,\n",
    "            random_state = 219,\n",
    "            test_size = 0.25,\n",
    "            stratify = apchef_target)\n",
    "\n",
    "# INSTANTIATING a KNN classification model with optimal neighbors\n",
    "knn_opt = KNeighborsClassifier(n_neighbors = opt_neighbors)\n",
    "\n",
    "# FITTING the training data\n",
    "knn_fit = knn_opt.fit(X_train_scaled, y_train_scaled)\n",
    "\n",
    "# PREDICTING based on the testing set\n",
    "knn_pred = knn_fit.predict(X_test_scaled)\n",
    "\n",
    "# train accuracy\n",
    "knn_train_acc = knn_fit.score(X_train_scaled, y_train_scaled).round(4)\n",
    "\n",
    "\n",
    "# test accuracy\n",
    "knn_test_acc  = knn_fit.score(X_test_scaled, y_test_scaled).round(4)\n",
    "\n",
    "\n",
    "# auc value\n",
    "knn_auc       = roc_auc_score(y_true  = y_test_scaled,\n",
    "                              y_score = knn_pred).round(4)\n",
    "\n",
    "# saving the results\n",
    "model_performance.append(['KNN Classification',\n",
    "                          knn_train_acc,\n",
    "                          knn_test_acc,\n",
    "                          knn_auc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The optimal number of neighbors is: 20. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CART  classification trees \n",
    "\n",
    "Instead of building a CART model that grows until it cannot grow further and \n",
    "overfits to training data, I am building a pruned tree with a maximum depth of 4 \n",
    "and a minimum number of samples per leaf of 25. \n",
    "\n",
    "The following user defined function is loaded for CART output to extract \n",
    "insight. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:40.806444Z",
     "start_time": "2020-02-06T15:33:40.392711Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yVette_1963/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:19: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAIWCAYAAADjzuSUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeZRlVX238ecLqIAoTqhogo0IGpmhnBAUcAi+OEBEsSUqRsUoipCgopgEByKKRsQJESPCQhxQVIwDBgERUKiGhm4QRGQIDrERBRlEaH7vH3eXuVxudd3qrurh8HzWqtX37rPP3r9zq1jLr3ufc1NVSJIkSZK6abUVXYAkSZIkafYY+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjpsjRVdgNR1D3vYw2rOnDkrugxJkiR13Lx5866vqvUG2w190iybM2cO4+PjK7oMSZIkdVySa4a1u71TkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbo00onyZwklyU5JsnCJCckeXaSs5NckeTJSZ6ZZH77uTDJA5LsmOSMJCe1809IkiHjPyLJyUkuaj/btfZ/avMtTLL/qLUs789HkiRJmo41VnQB0iQeB7wE2Ac4H3g5sD3wQuCdwOrAvlV1dpJ1gD+187YGNgV+BZwNPB340cDYRwJnVtXuSVYH1kmyLfBq4ClAgJ8kORP4/Qi17DZYfJJ9Wn822GCDZf0sJEmSpKXmSp9WVldV1YKqugu4BDitqgpYAMyhF+j+I8l+wIOq6s523nlVdV07b37rO2hn4FMAVbW4qm6kF+JOrqpbqupm4GvADiPWcg9VdXRVjVXV2Hrrrbdsn4QkSZK0DAx9Wlnd3vf6rr73dwFrVNVhwGuBtYAfJ3nCkPMWM/pq9j22gY5ay4jjS5IkSSuEoU+rpCQbtdW3DwDjwBOmOqfPacAb2jirJ3kg8ENgtyRrJ7k/sDtw1kzXLUmSJC1vhj6tqvZvD1a5CLgN+M6SOid5T5IXtrdvAXZKsgCYB2xaVRcAxwLnAT8BjqmqC2etekmSJGk5Se/WJEmzZWxsrMbHx1d0GZIkSeq4JPOqamyw3ZU+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6NOMS/LCJAetgHmfkGR+kguTbJvkjcu7BkmSJGllY+jTjKuqb1bVYaP0Tc9M/R3uBnyjqrYGfgcY+iRJknSvZ+jTtCSZk+SyJMckWZjkhCTPTnJ2kiuSPDnJ3kk+3vo/IsnJSS5qP9u1MX6a5JPABcBfJ5mbZEEb8wPt3NWTHNvaFiQ5oLVvleTHSS5uYz84yf8D9gdem+R04DBgo7byd/iQ69glyQWtptNa20OSfL2N++MkW7T2Q5J8PsmpSa5O8ndJPthq+m6S+yyPz16SJElaGmus6AK0Snoc8BJgH+B84OXA9sALgXcCX+/reyRwZlXtnmR1YB3gwcDjgVdX1RuTPAr4ALAt8Hvg1CS7Af8DPLqqNgNI8qA25nHAm6vqzCTvAf6tqvZPchRwc1V9KMkcYLOq2mqw+CTrAZ8BnlFVVyV5SDv0buDCqtotyc5tnonzNwJ2Ap4InAu8uKreluRkYNeBa5YkSZJWGq70aWlcVVULquou4BLgtKoqYAEwZ6DvzsCnAKpqcVXd2Nqvqaoft9dPAs6oqkVVdSdwAvAM4BfAY5N8LMkuwE1J1gUeVFVntnM/3/pOx1OBH1bVVa2uG1r79sDxre0HwEPbfADfqao72jWuDny3tQ+7ZpLsk2Q8yfiiRYumWZ4kSZI0cwx9Whq3972+q+/9XYy+enxL3+sM61BVvwe2BM4A9gWOmVaVkwtQk7Tfo4z27+2tpruAO1rIhUmuuaqOrqqxqhpbb731ZqBkSZIkaekY+jTbTgPeAH+5R++BQ/r8BHhmkoe1LaBzgTOTPAxYraq+CvwLsE1bKfx9kh3aua8Azhwy5h+BB0xS07ltvg1bXRPbO38I7NXadgSur6qbpnW1kiRJ0krGe/o0294CHJ3kNcBiegHw1/0dqurXSd4BnE5vte3bVfWNJFsCn+t7uuc72r+vAo5Ksja9LaCvHpy0qn7XHi6zkN7WzLcmmV9VW1XVoiT7AF9rY/8WeA5wSJvvYuDWNo8kSZK0Ssv/7VKTNBvGxsZqfHx8RZchSZKkjksyr6rGBtvd3ilJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5NS5JzVnQNkiRJkkZn6NO0VNV2y2OeJGssj3kkSZKkrjP0aVqS3JxkxyQ/THJykkuTHJVktb7jH0gyL8l/J3lykjOS/CLJC1ufNZN8LsmCJBcm2am1753kK0lOAU5NslqSTya5JMm3knw7yR6t778mOT/JwiRHJ0lrP6PNf16SnyXZYZLreGWSi5NclOT41vaYJKe19tOSbNDaj03yqSSnt+t4ZpL/TPLTJMfO9mcuSZIkLQtDn5bWk4F/BjYHNgL+rrXfHzijqrYF/gi8D3gOsDvwntZnX4Cq2hyYC3w+yZrt2NOAV1XVzm3MOW2O17ZjEz5eVU+qqs2AtYDn9x1bo6qeDOwP/Ntg4Uk2BQ4Gdq6qLYG3TIwJHFdVWwAnAEf2nfZgYGfgAOAU4CPApsDmSbYaMsc+ScaTjC9atOien54kSZK0nBj6tLTOq6pfVNVi4ERg+9b+Z+C77fUC4MyquqO9ntPatweOB6iqy4BrgE3ase9X1Q19/b5SVXdV1W+A0/vm3ynJT5IsoBfGNu079rX277y+OfvtDJxUVde3Gibmexrwhfb6+L5rAjilqqpdx/9W1YKqugu4ZNgcVXV0VY1V1dh66603pARJkiRp+fC+KS2tmuT9HS0cAdwF3A5QVXf13aeXJYx7S9/rof3aquAngbGq+p8khwBr9nW5vf27mOF/4xlS/zD9fSbGvKvv9cR7/zuSJEnSSsuVPi2tJyfZsN3Ltyfwo2mc+0NgL4AkmwAbAJcP6fcj4MXt3r5HADu29omAd32SdYA9pln7acBLkzy01fCQ1n4O8LL2ei+md02SJEnSSsnQp+maWP06FzgMWAhcBZw8jTE+CazetmZ+Cdi7qm4f0u+rwHVtjk8DPwFurKo/AJ+ht9Xy68D5U02Y5FFJvg1QVZcAhwJnJrkI+I/WbT/g1UkuBl7B/93rJ0mSJK2y8n878aQlaytjFwCvAg6squdPccpMzLlOVd3c5j4PeHq7v2+VMTY2VuPj4yu6DEmSJHVcknlVNTbY7r1IGkmSRwFnAB9azlN/K8mDgPsC713VAp8kSZK0ohn6NJKq+hX/94RN6AXA5THvjstjHkmSJKmrvKdPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RplZPkEUm+kOQXSeYlOTfJ7jMw7qOSnDQTNUqSJEkrC0OfVilJAnwd+GFVPbaqtgVeBvzVQL81pjt2Vf2qqvaYmUolSZKklYOhT6uanYE/V9VREw1VdU1VfSzJ3km+kuQU4FSAJG9Ncn6Si5O8u7W9N8lbJs5PcmiS/ZLMSbKwta2Z5HNJFiS5MMlOrf2sJFv1nXt2ki2Wz6VLkiRJ02fo06pmU+CCJRx/GvCqqto5yXOBjYEnA1sB2yZ5BvBZ4FUASVajt1J4wsA4+wJU1ebAXODzSdYEjgH2buduAtyvqi4eLCLJPknGk4wvWrRoaa9VkiRJWmaGPq3SknwiyUVJzm9N36+qG9rr57afC+kFxScAG1fV1cDvkmw9cbyqfjcw9PbA8QBVdRlwDbAJ8BXg+UnuA/wDcOywuqrq6Koaq6qx9dZbb2YuVpIkSVoK077vSVrBLgFePPGmqvZN8jBgvDXd0tc3wPur6tNDxplYsXsk8J9DjmfY5FV1a5LvAy8CXgqMTfcCJEmSpOXJlT6tan4ArJnkDX1ta0/S93vAPyRZByDJo5M8vB07GdgFeFLrN+iHwF7tvE2ADYDL27FjgCOB8/tWFSVJkqSVkit9WqVUVSXZDfhIkrcBi+it7r0dWGug76lJ/gY4t/fQT24G/h74bVX9OcnpwB+qanH/ae3fTwJHJVkA3AnsXVW3t3HnJbkJ+NysXagkSZI0Qwx9WuVU1a/pPXxlmGMH+n4U+Ohgp/YAl6cCL+lrfihwQzvvT7QHtgw591H0VslPnV7lkiRJ0vLn9k7d6yR5IvBz4LSquqK1jQEnMiQgDpz7SuAnwMFVddds1ypJkiQtK1f6dK9TVZcCjx1oG6f3dM6pzj0OOG6WSpMkSZJmnCt9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFvliQ5NskeK7qOCUnmJFk4ybE1klyf5P19bScnmZ/k50lubK/nJ9kuyRlJLu9rO2kJ8x6S5Jet3xVJvpbkiX3Hh47VzjtwyHg3t3+vSvL4gWNHJHlbkh0Hap6f5Nmtz+L2fmGSU5I8aPDzmeL8g5NckuTi1v6U0X8LkiRJ0vK3xoouQCuF5wKXAy9N8s7q2R16AQg4sKqeP9E5CcBeVTU+4vgfqaoPtXP3BH6QZPOqWtSOT2esCV8EXga8u427GrAH8HRgQ+Cs/pr73FZVW7VzPg/sCxw6pN89zk/yNOD5wDZVdXuShwH3nWbdkiRJ0nLlSt8MSfLKtvpzUZLjW/MzkpyT5BcTq35J1klyWpILkixI8qLWPifJT5N8pq0knZpkrXbsSW3sc5Mc3rcitXp7f347/vqlLH8u8FHgWuCpy/RBTKGqvgScCrx8GYc6kV7om/AM4OqqumYaY5wLPHoa/dcHrq+q2wGq6vqq+tU0zpckSZKWO0PfDEiyKXAwsHNVbQm8pR1aH9ie3urQYa3tT8DuVbUNsBPw4bSlM2Bj4BNVtSnwB+DFrf1zwD9W1dOAxX1Tvwa4saqeBDwJeF2SDadZ+1rAs4Bv0QtSc0c89YS+rY+HT2dO4ALgCcsyVlVdDNyVZMvW9DJ69U/YYWB75kb95ydZnd51f3OSKYadfyrw10l+luSTSZ45WX1J9kkynmR80aJFk3WTJEmSZp3bO2fGzsBJVXU9QFXd0HLc16vqLuDSJI9ofQP8e5JnAHfRW2maOHZVVc1vr+cBc9o9Zw+oqnNa+xfohUjobcvcou/ewXXpBcerplH784HTq+rWJF8F/iXJAVW1eIrzlmZL5oQMvF/asU4EXpbkEuBFwL/2HZtse+daSeYDc+h9xt+fZOyh5yfZFtiBXmD/UpKDqurYwX5VdTRwNMDY2FiNfEWSJEnSDDP0zYwAw/6H/e0DfQD2AtYDtq2qO5JcDaw5pP9iYC3uGZAG531zVX1vaYpu5gJPb3UAPJReoPnvZRhzKlsDSxsY+51Ib/XtTODiqvrtCOfcVlVbJVmX3urmvsCRo07YwvAZwBlJFgCvAo6dZt2SJEnScuP2zplxGr2HoDwUIMlDltB3XeC3LfDtBDxmSQNX1e+BPyaZuNeu/z627wFvSHKfNu8mSe4/atFJHkhv++kGVTWnqubQC0GjbvGctiQvprdCeeJUfadSVVcCv6O3dXZa41XVjcB+wIETn99Ukjw+ycZ9TVsB07mHUJIkSVruXOmbAVV1SZJDgTOTLAYuXEL3E4BTkowD84HLRpjiNcBnktxCb5XpxtZ+DL1tihe0+wIXAbstYZzHJ7mu7/1HgR9MPJik+QbwwST3G2i/x3Ukua29vr6qnr2Evgck+Xvg/sBCevc+9t/oNtlY70qy/0SnqvqrIWOfCLwfOHmgfYe2jXPC+6rqbl8tUVUXJrmIXpA+a6rz6W2b/Vjbcnsn8HNgn+GXLEmSJK0cUuXtRiu7JOtU1cT30x0ErF9Vb5niNK0kxsbGanx8JnazSpIkSZNLMq+qxgbbXelbNeya5B30fl/XAHuv2HIkSZIkrSoMfauA9t12Xxqlb5LNgeMHmm+vqqfMeGF3n/dg4CUDzV+pqmFffC5JkiRpOTH0dUxVLaD3gJHlPe+hgAFPkiRJWsn49E5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOW2NFFyB13YJf3sicg/5rRZchSZKkWXb1Ybuu6BKGcqVPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUofNWuhLsjjJ/L6fOa19+yTnJbms/ezTd84hSQ4cMtbNk8yxT9845yXZvrW/KMnX+/q9I8nP+96/IMk32+urkyzoq/PI1n5skquSXJTkZ0mOS/LoKa55YqyLk5yZ5DFL+DwOau33SXJYkiuSLGzX8bx2bN0275Xt57gk67Zjc5JUkvf2zfGwJHck+Xjf51lJHtfX54DWNjbC9f8yyf36xr66b+7bklyY5Ket5lcN+Ty+keTcvvcH983T/3ns1/+7T8+72mfysySnJ9l04HP+at/7PZIc214/Ism32u/t0iTfXsLva+I65re+RyVZrR3bJMm3k/y8XeOXk+zZV/PNSS5vr49b0t+FJEmStCKtMYtj31ZVW/U3JHkk8AVgt6q6IMnDgO8l+WVV/dd0Bk/yfOD1wPZVdX2SbYCvJ3kycA5wdF/3pwE3JXl4Vf0W2A44u+/4TlV1/ZBp3lpVJyUJsD9wepLNqurPSyhtp1bPu4F3Aa9r7ff4PJr3AusDm1XV7UkeATyzHfsssLCqXtmu+d3AMcBL2vFfAM8H/qW9fwlwycD4C4CXAe9r7/cALh1W85DaFgP/AHxqyLErq2rrVtdjga8lWa2qPtfaHgRsA9ycZMOquqqqDgUObcdv7v88khzSN/a+9H5HW1bVrUmeC3wzyaZV9afWZ6y9H7ze9wDfr6qPtnG3GFL74HVslWQN4AfAbi0o/hfwT1V1ShtnJ2DRRM1JzgAOrKrxKcaXJEmSVqjlvb1zX+DYqroAoAWNtwEHLcVYb6cXyq5vY10AfB7Yt6oWATf2rXA9GvgqvSBB+/ecUSeqno8AvwGeN+Jp57Z5J5VkbXqh8M1VdXub63+r6sut9m3phcIJ76EXdjZq728DfjqxagfsCXx5YJqvAy9q8z0WuBFYNOI1HAEc0ALRpKrqF8A/Afv1Nb8YOAX4Ir3QOR1vp/eZ3NrGP5Xe72uvvj4fAt455Nz1gev6art4lAmr6s42x+OAlwPnTgS+dvz0qlo46gWktwo9nmR88a03jnqaJEmSNONmM/St1bcV7uTWtikwb6DfeGufrqnGOgfYLsnjgSuAH7f3awBbAOf3nXd6X60HLGHOC4AnjFjfLvQC14T+z2N+kj3pBYxrq+qmIec/EZhfVYsnGtrr+dz98/oi8LIkf0VvZe5XA+PcBPxPks2AucCXhsw12fVfC/wIeMUI1zv42cwFTmw/c0c4H4AkDwTuX1VXDhwa/Dv5MrBN/9bV5hPAZ9uW0IOTPGrEedcGnkVvZXQz7vm3NS1VdXRVjVXV2Oprr7ssQ0mSJEnLZLlu7wQC1JC+w9qWRv/4Z9Nb0Vud3qrbecC/AlsDl/dtE4TJtzcOG38qp7ctmr+lt71zwrDtrkvaejjZZzXY/l16q4H/y/BAB/+32va39ILNqweOL+n6/x34Jr3tjkvyl8+mXf/jgB9VVSW5s22LHXmlbJLx+697MXA48A7gOxONVfW9tqK5C71V2Qvb3JOtbm6UZH4b+xtV9Z0kz1mGOiVJkqSVykgrfe2hFqclWdjeb5HkXVOdN8QlwNhA27bc8x6zUVzazu23Td9Y59ALfdvR26r3R2BNYEfufj/fdGwN/HSKPjsBj6F3re+Zou/PgQ2SPGDIsUuArSceLALQXm/ZX0O7v3Ae8M/0trAOcwq91brJVhUnVVU/p7e6+NIpuvZ/NnsCDwauSu/hL3MYcYtnq++WFtz69f9uJxwPPAPYYGCMG6rqC1X1Cnorus9YwpRXVtVWVbV1VR3S2i7hnn9bkiRJ0ipp1O2dn6G3onIH/OU+qenepwW9rXd7J5l4GMZDgQ8AH1yKsT4IfKCNQRtzb+CT7filwKOAHYALW9t84B+Zxv18bewk2Y/e/WLfnap/Vd1G78Evr0zykCX0u5Xew1qOTHLfNtf6Sf6+ha0Luftq4buAC9qxfh8G3l5Vv1tCPW+nPURlKRwK3OOpqhPSezLrh4CPtaa5wC5VNaeq5tALUNP5ezmc3meyVhv/2cD29B4C9BdVdQfwEXqf9UQtO7etmrQwvRG9barT8QV6W4F37Rt3lySbT3McSZIkaYUbdXvn2lV1Xu8hln9x53Qnq6pfJ/l74DPtf5AHOKL/gRnAu5Ls33fOXwFrJ7mur89/VNV/pPcVCuckKeCPwN9X1a/beZXkJ8C6LRxAb5vnPtwz9J2eZOLeuYsnnpYJHJ7kX4C16d0TuNMUT+4cvNYT6T285r20e/r6uny3qg6iF+TeB1ya5E/ALfS2oQK8BvhYel83kVb/a4bMdQn3fGrnYJ8vLuHwZNf/l/GTXEBvtW3CRkkupLd6+kfgY1X1uRYAN6D3eU2cf1WSm5I8pap+sqQ6m4/RWylc0Or6DfCiFl4HfZa7B+NtgY8nuZPe/6lxTFWdP+S8SVXVbe3psEckOYLe/9lxMfCW6YwjSZIkrQxSNfXtdEm+A7wJ+EpVbZNkD+A1VTXqkyyle637rb9xrf+qI1Z0GZIkSZplVx+269SdZlGSeVU1eDvdyCt9+9L73rsnJPklcBV3f3y+JEmSJGklNGXoaw8PGauqZye5P7BaeyjKvVbbNnq/geZXVNWCFVGPlqzdi3f8QPPtVfWUFVGPJEmStDxNGfqq6q4kbwK+XFW3LIeaVnqGhVVLC+ODXx8iSZIk3SuM+vTO7yc5MMlfJ3nIxM+sViZJkiRJWmaj3tP3D+3fffvaChj8LjVJkiRJ0kpkpNBXVRvOdiGSJEmSpJk3UuhL8sph7VV13MyWI0mSJEmaSaNu73xS3+s1gWcBFwCGPkmSJElaiY26vfPN/e+TrMs9H4EvaYjNH70u4yv4izolSZJ07zXq0zsH3QpsPJOFSJIkSZJm3qj39J1C72md0AuKTwS+MltFSZIkSZJmxqj39H2o7/WdwDVVdd0s1CNJkiRJmkGjbu/8f1V1Zvs5u6quS/KBWa1MkiRJkrTMRg19zxnS9ryZLESSJEmSNPOWuL0zyRuANwKPTXJx36EHAGfPZmGSJEmSpGU31T19XwC+A7wfOKiv/Y9VdcOsVSVJkiRJmhFLDH1VdSNwIzAXIMnD6X05+zpJ1qmqa2e/REmSJEnS0hrpnr4kL0hyBXAVcCZwNb0VQEmSJEnSSmzUB7m8D3gq8LOq2hB4Ft7TJ0mSJEkrvVFD3x1V9TtgtSSrVdXpwFazWJckSZIkaQaM+uXsf0iyDnAWcEKS39L7knZJkiRJ0kps1JW+FwG3AvsD3wWuBF4wW0VJkiRJkmbGSCt9VXVLkscAG1fV55OsDaw+u6VJkiRJkpbVqE/vfB1wEvDp1vRo4OuzVZQkSZIkaWaMur1zX+DpwE0AVXUF8PDZKkqSJEmSNDNGDX23V9WfJ94kWQOo2SlJkiRJkjRTRg19ZyZ5J7BWkucAXwFOmb2yJEmSJEkzYdTQdxCwCFgAvB74NvCu2SpKkiRJkjQzlvj0ziQbVNW1VXUX8Jn2I0mSJElaRSumek0AACAASURBVEy10veXJ3Qm+eos19JpSRYnmZ9kYZJTkjyotc9Jcls7NvHzynbs6iQL+tq3G+h/aZLjktyn9d8xybf65twlyXlJLmv9v5Rkg3bs2CRX9Y19TmvfO8mi1nZZkgOGXMtFSU7se/+Jvnr6r2WPEea5MMkVSb6XZLspPsNjk+wxpH3TJD9I8rM21r8kSd/x5yUZT/LTdk0fau2HJDmwvV4zyfeT/NvA72vi56DWfkaSy9tncH6SrUb7C5AkSZJWjKm+py99rx87m4XcC9xWVVsBJPk8vSeiHtqOXTlxbIidqur6iTdJ5kz0T7I68H3gpcAJ/Scl2Qz4GPDCqvppa3shMAe4tnV7a1WdNGTOL1XVm5I8FLg8yUlV9T9tjL+h938WPCPJ/avqlqrat6+2b/VfS5LnTzVP67cT8LUkO03UO4okawHfBN5QVae275D8KvBG4BPtc/g4sGtVXdYeQrTPwBj3befMq6p3t+bblvA72auqxpO8GjgceM6o9UqSJEnL21QrfTXJay2bc+l91+EyqarFwHmTjPV24N/7A1RVfbOqfjiN8X8H/BxYv6/55cDxwKnAC5em7knmOh04moFANoKXA2dX1altnFuBN9G7DxXgbcChVXVZO35nVX2y7/w1gC8CV1TVQUzPpL/HJPu01cXxRYsWTXNYSZIkaeZMFfq2THJTkj8CW7TXNyX5Y5KblkeBXdNW555Fb3VqwkYDWwl36Dt2emv7yZCx1gSeAnx3yFSbAhdMUc7hfXOeMHiwbQVdE7i4r3lP4EvAicDcKcYfaZ4+FwBPGHHMCZsC8/obqupKYJ0kDwQ2Gzw+4G3AnVW1/0D7WgO/kz2HnLsLfVugB2o4uqrGqmpsvfXWG/liJEmSpJm2xO2dVbX68irkXmCtJPPpba+cR29b5oSRt3c2G7WxNgZOqqqLh5z3F22b5mnA2sDRVfWhdmiybZd7tu2WjwdeV1V/auM8CVhUVdckuQ74zyQPrqrfL2n+Jcxzj1JH6DPsnMlWoUdZnf4R8LQkm1TVz/ral7S984Qk9wdWB7YZvVRJkiRp+Rv1Kxu07CZCxGOA+9K7p29pTYTExwFPbffqDbqEFkiq6net/9HAOiOM/6Wq2hTYAfhwkke29rnAE5JcDVwJPBB48TJcx6CtgZHv52suAcb6G5I8Fri5qv7Yjm+7hPN/COwPfCfJo0accy9gQ+ALwCemWa8kSZK0XBn6lrOquhHYDzgw7ambyzDWr+ndu/aOIYc/CBzcHrwyYe1pjn8uvfv33pJkNeAlwBZVNaeq5gAvYvQtnkuU5Jn07ueb7teCnABsn+TZbZy1gCPpXT/0HrTyziSbtOOrJfmn/gGq6qut33fTnqo6laq6g953VT514DOWJEmSViqGvhWgqi4ELgJe1poG7+nbbxrDfR1Ye+A+QKpqAfAW4Lj2NQVnA39Db3VqwuED8953yPgfAF4N7Ar8sqp+2Xfsh8ATk6w/5Lx+k82zZ3v/M+CdwItHeHLnp5Nc137Orarb6IXPdyW5HFgAnE/viZ20ra/7Aycm+SmwkLs/mIbW7yjga8A3272Sg/f0HTbknNuADwMHTlGzJEmStMKkyodySrNpbGysxsfHV3QZkiRJ6rgk86pqbLDdlT5JkiRJ6rCpvpxdWu6SfAJ4+kDzR6vqcyuiHkmSJGlVZujTSqeqluXJppIkSZL6uL1TkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpw9ZY0QVIXbfglzcy56D/ulvb1YftuoKqkSRJ0r2NK32SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU/LTZLFSeYnWZjkK0nWbu2PTPLFJFcmuTTJt5NsMskYc5IsHNKeJO9KckWSnyU5PcmmfcfXSfKpNseFSeYled3gmEl2TFJJXtB37reS7NheP7+df1Gr9fUz+iFJkiRJM8zQp+Xptqraqqo2A/4M/GOSACcDZ1TVRlX1ROCdwCOmOfa+wHbAllW1CfB+4JtJ1mzHjwF+D2xcVVsDuwAPmWSs64CDBxuT3Ac4GnhBVW0JbA2cMc06JUmSpOVqjRVdgO61zgK2AHYC7qiqoyYOVNX8pRjv7cCOVXVrG+PUJOcAeyU5A3gy8PKquqsdXwR8YJKxLgLuk+Q5VfX9vvYH0Ptv5ndtjNuBy5eiVkmSJGm5caVPy12SNYDnAQuAzYB5yzjeA4H7V9WVA4fGgU3bz0UTgW9E7wPe1d9QVTcA3wSuSXJikr2SDP1vKMk+ScaTjC++9cZpTCtJkiTNLEOflqe1ksynF8auBT47y/MFqHs0Jge3ewt/NdmJVXVW67vDQPtrgWcB5wEHAv85yflHV9VYVY2tvva6y3AJkiRJ0rIx9Gl5mrinb6uqenNV/Rm4BNh2WQatqpuAW5I8duDQNsCl7WfLiVW5qjq0qrYCHjjF0Icy5N6+qlpQVR8BngO8eFlqlyRJkmaboU8r2g+A+008SRMgyZOSPHOa4xwOHJlkrTbGs4HtgS9U1c/prS6+L8nq7fia9FYCJ1VVpwIPBrZs56wz8RTPZivgmmnWKUmSJC1XPshFK1RVVZLdgSOSHAT8Cbga2H8Jpz0+yXV97w8APkYvoC1Ishj4DfCiqrqt9XktvWD48yQ3ALfRe/jLVA4FvtFeB3hbkk+3828B9h5hDEmSJGmFSdU9bnmSNIPut/7Gtf6rjrhb29WH7bqCqpEkSVJXJZlXVWOD7W7vlCRJkqQOc3unVkpJNgeOH2i+vaqesiLqkSRJklZVhj6tlKpqAb0HpUiSJElaBm7vlCRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mF/ZIM2yzR+9LuOH7bqiy5AkSdK9lCt9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQO8ysbpFm24Jc3Mueg/xq5/9V+vYMkSZJmkCt9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0DdEkocmmd9+fpPkl33v75tk9ySV5Al956yW5MgkC5MsSHJ+kg2T/KSdd22SRX3jzElydes70XZkG+upfef9NMkhI9T8jSTnDrQdkuTWJA/va7u57/XiNsclSS5K8k9JJv2bSLJjkhuTXNjq+rch7Zcl+VDfOXsPXPf8JE9s13/bQPt9R+x/aZLjktxnYP7+c57djlWSD/fVc2D/55nkle13dkkb98DWfmySq/rGO6e1PyLJt9rndWmSb0/1u5EkSZJWpDVWdAEro6r6HbAV9IITcHNV9QeZucCPgJcBh7TmPYFHAVtU1V1J/gq4paqe0s7ZGxirqjf1jQOwU1VdP1DC54GXVtVFSVYHHr+kepM8CNgGuDnJhlV1Vd/h64F/Bt4+5NTbqmriOh8OfAFYF/i3JUx3VlU9P8n9gflJvjXQvhZwYZKTq+rsduxL/dfd5psDXDkxf1/7lP3bZ/J94KXACf3zD6n3duDvkrx/8HNO8jxgf+C5VfWrJGsCr+jr8taqOmlgvPcA36+qj7YxthgypyRJkrTScKVvmpKsAzwdeA290DdhfeDXVXUXQFVdV1W/X8ppHg78uo2zuKounaL/i4FTgC8O1ATwn8CeSR6ypAGq6rfAPsCb0pLXFP1vAeYBGw203wbMBx491RhLq6oWA+eNOMedwNHAAUOOvQM4sKp+1cb9U1V9Zorx1geu66vl4mGdkuyTZDzJ+OJbbxyhTEmSJGl2GPqmbzfgu1X1M+CGJNu09i8DL2hbAT+cZOsRxzu9bwvhRDD5CHB5kpOTvL6tQC3JXODE9jN34NjN9ILfW6YqpKp+Qe9v4uFT9U3yUOCpwCUD7Q8GNgZ+2Ne858DWy7Va+0Z9bZ8Yof/EHGsCTwG+29e8w8A5/WH0E8BeSdYduIzN6AXXyRzeN97EiuIngM8mOT3JwUkeNezEqjq6qsaqamz1tQenlSRJkpYft3dO31zgiPb6i+39BVV1XZLHAzu3n9OSvKSqTptivHts76yq97SQ8Vzg5W2OHYednOQRwOOAH1VVJbkzyWZVtbCv25H0tmJ+eNgYg0NOcXyHJBcCdwGHVdUlSXZs7RfT24p6WFX9pu+cYds1Ycj2zin6b5RkPr1QedLAKttk2zupqpuSHAfsB9w2xfX1u8f2zqr6XpLHArsAz6O3lXWzqlo0jXElSZKk5caVvmloq1s7A8ckuRp4K71VqQBU1e1V9Z2qeivw7/RWBZdKVV1ZVZ8CngVs2eYeZk/gwcBVraY5DGzxrKo/0Ltf741LmrOFmcXAb5fQ7ayq2rqqtq2qowbatwA2B96QZFiYW1YTIfFxwFOTvHAa5x5Bb0vu/fvaLgG2nW4RVXVDVX2hql4BnA88Y7pjSJIkScuLoW969gCOq6rHVNWcqvpr4Cpg+yTbTGz1S+8JmFsA1yzNJEl27buvbmN6QewPk3SfC+zS6plDL8QM3tcH8B/A65lkdTfJesBRwMerqpamboC27fX9DH9wzIyoql8DB9G7J2/Uc26gtwX3NX3N7wc+mOSRAEnul2S/JY2TZOcka7fXD6B3T+O107sCSZIkafkx9E3PXODkgbav0tuC+XDglCQLgYvpPUDk4yOM2X9P33Gt7RX07umbDxwP7NUeXnI37YmWGwA/nmhrT+68KclT+vu2LaQnA/fra16rzXsJ8N/AqcC7R6h5KkcBz0iyYXs/eI/edlOcP0r/rwNrJ9mhvR+8p2+PIed8GHjYxJuq+ja9e/T+u30G87h7KD58YMz70gvV420r67nAMVV1/hTXI0mSJK0wWYZFHUkjuN/6G9f6rzpi6o7N1YftOovVSJIkqauSzKuqscF2V/okSZIkqcN8eucqIsmruefXLpxdVfvOwlx/C3xgoPmqqtp9pueSJEmSNLsMfauIqvoc8LnlNNf3gO8tj7kkSZIkzS63d0qSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYT++UZtnmj16Xcb9wXZIkSSuIK32SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOuxeEfqSPDLJF5NcmeTSJN9Oskk7dkCSPyVZt6//jkkqyQv62r6VZMf2+j5JDktyRZKFSc5L8rx27OokC5LMbz9HtvZjk+wxUNecJAsnqXmNJNcneX9f28ltzJ8nubFvju2SnJFkrPVbN8lx7XqvbK/X7Zuzkry5b9yPJ9m7vX5qkp+0cX+a5JARPt9vJDl3oO2QJL/sq/GwEeq/vK/tpCHjXJpk7hS1HNv636+9f1iSq/uOb5rkB0l+1n5//5Ikfcd3S3Jxksva73G3UceWJEmSVkadD33tf9CfDJxRVRtV1ROBdwKPaF3mAucDuw+ceh1w8CTDvhdYH9isqjYDXgA8oO/4TlW1VfvZbylLfy5wOfDSiVBSVbtX1VbAa4Gz+uY4Z+DczwK/aNe7EXAVcEzf8d8Cb0ly3yHzfh7Yp82zGfDlJRWZ5EHANsCDkmw4cPgjfTUeNEL9e/W17TE4DvAi4NNJ7rOkmoDFwD8MqXUt4JvAYVW1CbAlsB3wxnZ8S+BDwIuq6gnAC4EPJdliqrElSZKklVXnQx+wE3BHVR010VBV86vqrCQbAesA76IX/vpdBNyY5Dn9jUnWBl4HvLmqbm/j/W9VLTEcLYW5wEeBa4GnjnpSkscB29ILphPeA4y16wVYBJwGvGrIEA8Hfg1QVYur6tIppnwxcArwReBlo9a5NKrqCuBW4MFTdD0COCDJGgPtLwfOrqpT23i3Am8CDmrHDwT+vaquasevAt4PvHWEse8myT5JxpOML1q0aOqLkyRJkmbJvSH0bQbMm+TYXOBE4Czg8UkePnD8ffQCYb/HAddW1U1LmPP0vm2KB0y34LYi9SzgW62+JW5pHPBEYH5VLZ5oaK/nA5v29TsM+Ockqw+c/xHg8rYV8/VJ1pxivonPcFidB/R9Dn87Qu0n9PU/fPBgkm2AK6rqt1OMcy3wI+AVA+2bMvC3UFVXAuvk/7d371GaVeWdx78/aQUVggjEMF5oh+ESRWmxEzJKIigmZHC4zOIquiDDqImXyaAgOjKGMANhAi6Cg3GBidxCuMpNVCBBYAQHYgPdQHMRtIlCkgFRUC7e4Jk/zinm9Mtb1W93V9VbnPp+1qpV591nn72f89ZeVf303vu8ya8MOw8sYeX3bbK2V1JVp1bV4qpavOmmm64iXEmSJGnmzIekbyr7A+dW1TPARcA+3ZNV9Q2AJL+9mu12l3eeuAZxvQu4pp2J+hKw15DkbDIBalXl7SzWP9DMftEpPxpYDFzVnrti0o6SV9AkwddX1beBXybZtlOlu7zzyhFi7y7v7M6uHZrkHuAm4KgR2gE4lmaGrjvGJ3tvaMuHnR9WNqxtSZIkaU6aD/9oXU6z3HEl7T6tLYG/ax/GsT/DZ9SOYeW9ffcBr0mywZC60+UAYJc2rpuBjWmWqY5iOfCmJM/+bNvj7YC7BuoeCxzBwDioqu9U1edpZhu3S7LxJH3tR7PUckUb60JmZonniVW1ddvfmSPMPlJV99HMbu7bKV5Ok9A+K8m/Bh6vqp8MO0+zX3GlJa6TtC1JkiTNSfMh6fs6sG6S900UJPkNmv1yR1XVwvbrXwGvTLJ59+J2/9dGNEnTxD6wvwY+O/EglCSbJXnPdATbLjPcEXjNRGzAhxhxiWebkNzKystSjwRuac91695Nk9C8q9P/bp2nWW5J8+CSRyfp7gBg106cb2YG9/VV1UU0yy2H7UUc5hiafXoTzgZ2TLILPLuM9rPAn7fnTwA+mWRhe34hzUN/PjNC25IkSdKc1Pukr6qK5smc72w/vmA5zRLBnWie6tl1McOTlmOAV3VeH0nzMJQ703zkwiXt6wndPX1ndspPSfJA+zXxEQdbd8oeAD4AfH3iITGtS4HdJz4qYASHAFu1H43wHWCrtmyYwXt7L82evqXAWTRLLp8evKhNiF4D3DhR1i4Z/XGSHUaMc1B3T9/fT1LnaOCj3ZnMyVTVcuCWzuunaJ4AemS7XPR2mie3ntyeX0oz8/nlJHfTPKDm4235lG1LkiRJc1WanEjSTFm8eHEtWbJk3GFIkiSp55LcXFWD25X6P9MnSZIkSfPZlJ81JgEk+QPgjweKb6iqD40jHoAknwPeOlB8UlWdNo54JEmSpLnKpE+r1CZScyqZGmfCKUmSJD2fuLxTkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpE+SJEmSesykT5IkSZJ6zKRPkiRJknrMpG8eSlJJzuq8XpDk4SSXd8r2THJbkruT3J5kz86505OsSLK0Pf8nnXPXJlncHt+fZJOBvg9u+1ra+XrdJHEubGP9SKfs5CQHD/bVqX9He7xTe+0hnfNvassOG/E+7unEeGFbflSSB9uyO5McsBpvvSRJkjTrTPrmpyeAbZO8uH39TuDBiZNJtgNOAPaoqm2A3YETkryx08bhVbUIWAQclOS1q9H/eVW1qPN15xR1HwL+OMmLVqP9CbcD+3Ve7w8sG6gz1X0c2Ilx7075ie01ewCnJHnhGsQmSZIkzQqTvvnra8Bu7fEBwDmdc4cBx1bVCoD2+58Bhw9pZ732+xMzFOfDwNXAQWtw7feA9ZK8IkmAXWnue5jVvo+quhd4EthoDWKTJEmSZoVJ3/x1LrB/kvWANwI3dc69Hrh5oP6StnzC8UmWAg8A51bVQ6vR934DyztfvIr6xwEfS7LOavQx4UJgH+AtwC3AzwbOT3UfZ3diPH6w4STbA/cOu/ck70+yJMmShx9+eA3CliRJkqaHSd88VVW3AQtpZvm+OnA6QK2ibGJZ5K8B70jyltXofnB551OriHUF8A/AuwdPDas+8Pp8mqRvcDZzwlT30V3e2Z3lPDTJPTSJ8lGTxHxqVS2uqsWbbrrpZLcmSZIkzTiTvvntMpq9e4PJ0HJg8UDZ9sBz9t5V1ePAtcCOMxBf17HAEaw8Zh9h5aWVLwd+MBDfvwC/oNm3ePVkja/mfZxYVVvT7Bc8s50tlSRJkuYkk7757YvA0VV1+0D5CcAnkyyE5qmYwH8FPjPYQJIFwA7Ad2YwTqrqbpqk812d4muB97T79aDZ93fNkMs/DRxRVU9P1v6a3EdVXUSz7HVN9htKkiRJs2LBuAPQ+FTVA8BJQ8qXJjkC+HL7ZMpfAB+vqqWdascnORJ4Ec0M2kWTdHNbkmfa4/OB22j29HVn1D5YVd8cIeRjgFs7r08FtgGWJSmaBOyTQ+5nqranuo+zk0wsPf1BVe0y5Pqjgb9N8oWqembIeUmSJGmsUjVsW5Sk6bJ48eJasmTJuMOQJElSzyW5uaoGt2m5vFOSJEmS+szlnRq7JG8Azhoo/llV7TCOeCRJkqQ+MenT2LUPklk07jgkSZKkPnJ5pyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT12IJxByD13e0PPsbCT3xlrdu5/7jdpiEaSZIkzTfO9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9E2DJHslqSTbtK9XJNl6oM5fJPl4e/ybSa5Ncm+SW5J8JckbJmn7U0mWtl9Pd47/c5KjkhzW1js9yZNJNuhce1Ib1ybt6+71S5N8Yop7ujbJPUmWJblh4n465RNtXNiWH5XkwbbsziQHrOI9+60kN7X170pyVFt+cJKTR7jvB7PyvbxsSB8vTfJIkg0Hyi9Jsu9EX23Z1u29TcRzalu+U5LLk/xBp6+fJ7m9PT5uqvuUJEmSxm3BuAPoiQOA64H9gaOAc9vjPwVI8gJgb+CtSV4BnA+8u6q+2Z7fEdgCuH2w4ao6Bjimrfd4VS2aODeRKHXcB+wB/E3b587Ag53zT3WvH8GBVbUkyfuB44Hdu+VD6p9YVSck2RK4OcmFVfWLSdo+A9i3qpYlWQdYKUke4b5PrKoTpgq+qp5IchWwZ9sfbQK4I/BuYN9O9c+2bV7a1nvDQFunAae15+4Hdq6qH0zVvyRJkjQXONO3lpKsD7wVOIQm0QM4p3MM8DvA/VX1j8CHgTMmEj6Aqrq+qi6ZhnDOAfZrj3cCbgB+OQ3t/m/g34xauaruBZ4ENpqi2q8C/9zWf7qq7lyrCCc3+LPYC7iiqp4cqLcZ8MDEi6p6TgK+OpK8P8mSJEuefvKxtWlKkiRJWismfWtvT5ok4tvAD5NsX1W3Ac8k2a6tsz9N8gHweuCWGYrlXmDTJBvRzD6eO3D+xQNLIvd7bhND/XtWnoU8u9PG8YOVk2wP3FtVD03R5onAPUkuTvKBJOuNGMuEQzsxXDNFvSuANyfZuH3d/VkMxvP1JF9Lcuiw5aKro6pOrarFVbV4nZdsuOoLJEmSpBli0rf2usnVue1raGeYkiygWXJ5wbCL231tdyU5aZriuYgmsdkB+MbAuaeqalHn67xVtHV2kqU0M5mHdcoP7LRxeKf80CT3ADfRLHOdVFUdDSwGrqJZannFqm5swImdGHaeop+fA5cBe7d7Gxe1fQ7WOw34dZqf007AjUnWXc2YJEmSpDnHPX1roZ09ejuwbZIC1gEqzQNbzqFJLq4DbuvMei0HtgcuBaiqHZLsDbxrmsI6l2Ym8YyqeibJ2rQ12d69yUzs6fsPwJlJtqiqn05Wuaq+A3w+yReAhzuzcdPtHOBIIMClk+0zrKp/Ar4IfDHJHcC2MxSPJEmSNGuc6Vs7ewNnVtXmVbWwql4NrAB2bBOaR4DjWHk54eeAg5O8pVP2kukKqKq+B3wK+MvpanMNYrgIWAIcNFmdJLvl/2ekWwJPA4/OUEjXtH18iOFLO0mya5IXtse/BmzMyg/BkSRJkp6XTPrWzgHAxQNlX6JZrghNgrFNt05V/QvNw1b+LMl9Sb5JkzyePF1BVdUpbdI5aHBP35p+3EB3T9/fT1LnaOCj7VNEh3kvzZ6+pcBZNLOKT69GDIcO3MvCySpW1TM0P5eNaR5KM8zvAnckWQZcCRze/qwkSZKk57VU1bhjkHpt3c22rM0O+ou1buf+43abhmgkSZLUV0lurqrFg+XO9EmSJElSj/kglzkkyaeAfQaKL2g/qHym+rwYeO1A8RFVuoX1jQAACONJREFUdeU0tf85mqd/dp3UPi1z2iT5PeB/DhSvqKq9prMfSZIk6fnGpG8OaZO7GUvwJulzRpOiqvrQTLbf6edKmr14kiRJkjpc3ilJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST3mRzZIM+wNr9yQJcftNu4wJEmSNE850ydJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST2Wqhp3DFKvJfkJcM+449CctAnwg3EHoTnHcaHJODY0jONCXZtX1aaDhQvGEYk0z9xTVYvHHYTmniRLHBsa5LjQZBwbGsZxoVG4vFOSJEmSesykT5IkSZJ6zKRPmnmnjjsAzVmODQ3juNBkHBsaxnGhVfJBLpIkSZLUY870SZIkSVKPmfRJ0yTJrknuSXJfkk8MOb9ukvPa8zclWTj7UWq2jTAuPprkziS3Jbk6yebjiFOzb1Vjo1Nv7ySVxKfzzQOjjIsk+7a/N5Yn+dvZjlHjMcLfk9ckuSbJre3flH83jjg1N7m8U5oGSdYBvg28E3gA+BZwQFXd2anzQeCNVfWHSfYH9qqq/cYSsGbFiONiZ+CmqnoyyR8BOzku+m+UsdHW2wD4CvAi4MNVtWS2Y9XsGfF3xpbA+cDbq+pHSX61qh4aS8CaNSOOjVOBW6vq80leB3y1qhaOI17NPc70SdPjN4H7quq7VfVz4Fxgj4E6ewBntMcXAu9IklmMUbNvleOiqq6pqifblzcCr5rlGDUeo/zOAPjvwJ8DP53N4DQ2o4yL9wGfq6ofAZjwzRujjI0CfqU93hD4p1mMT3OcSZ80PV4JfL/z+oG2bGidqvol8Biw8axEp3EZZVx0HQJ8bUYj0lyxyrGR5E3Aq6vq8tkMTGM1yu+MrYCtktyQ5MYku85adBqnUcbGUcB7kjwAfBX4yOyEpueDBeMOQOqJYTN2g2unR6mjfhn5Z57kPcBi4G0zGpHmiinHRpIXACcCB89WQJoTRvmdsQDYEtiJZmXAN5JsW1WPznBsGq9RxsYBwOlV9Zkk/xY4qx0bz8x8eJrrnOmTpscDwKs7r1/Fc5dVPFsnyQKapRc/nJXoNC6jjAuS7AJ8Cti9qn42S7FpvFY1NjYAtgWuTXI/8FvAZT7MpfdG/VtyaVX9oqpWAPfQJIHqt1HGxiE0+z2pqv8DrAdsMivRac4z6ZOmx7eALZO8NsmLgP2BywbqXAYc1B7vDXy9fJJS361yXLRL+E6hSfjcmzN/TDk2quqxqtqkqha2D2K4kWaM+CCXfhvlb8klwM4ASTahWe753VmNUuMwytj4HvAOgCS/TpP0PTyrUWrOMumTpkG7R+/DwJXAXcD5VbU8ydFJdm+r/TWwcZL7gI8Ckz6iXf0w4rg4HlgfuCDJ0iSDf8TVQyOODc0zI46LK4FHktwJXAMcXlWPjCdizZYRx8bHgPclWQacAxzsfy5rgh/ZIEmSJEk95kyfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJPZPk6fYjQCa+Fq5BGy9L8sHpj+7Z9ndPMqsfXZNkzySvm80+JWku8CMbJEnqmSSPV9X6a9nGQuDyqtp2Na9bp6qeXpu+Z0KSBcBf0dzTheOOR5JmkzN9kiTNA0nWSXJ8km8luS3JB9ry9ZNcneSWJLcn2aO95Dhgi3am8PgkOyW5vNPeyUkObo/vT/LpJNcD+yTZIskVSW5O8o0k2wyJ5+AkJ7fHpyf5fJJrknw3yduSfDHJXUlO71zzeJLPtLFenWTTtnxRkhvb+7o4yUZt+bVJjk1yHXAEsDtwfHtPWyR5X/t+LEvypSQv6cTz2STfbOPZuxPDx9v3aVmS49qyVd6vJI3TgnEHIEmSpt2Lkyxtj1dU1V7AIcBjVfUbSdYFbkhyFfB9YK+q+nGSTYAbk1wGfALYtqoWASTZaRV9/rSqdmzrXg38YVXdm2QH4C+Bt6/i+o3aOrsDXwbeCvwn4FtJFlXVUuClwC1V9bEknwb+BPgwcCbwkaq6LsnRbfl/adt9WVW9rY1rSzozfUkeraovtMf/o32P/ld73WbAjsA2wGXAhUl+H9gT2KGqnkzy8rbuqWtwv5I0a0z6JEnqn6cmkrWO3wXe2Jm12hDYEngAODbJ7wDPAK8EXrEGfZ4Hzcwh8BbggiQT59Yd4fovV1UluR34v1V1e9vecmAhsLSN77y2/t8AFyXZkCaxu64tPwO4YDCuSWzbJnsvA9YHruycu6SqngHuTDLxfuwCnFZVTwJU1Q/X4n4ladaY9EmSND+EZjbsypUKmyWamwJvrqpfJLkfWG/I9b9k5W0hg3WeaL+/AHh0SNK5Kj9rvz/TOZ54Pdm/V0Z5MMETU5w7Hdizqpa178NOQ+KB5r2b+D7Y55reryTNGvf0SZI0P1wJ/FGSFwIk2SrJS2lm/B5qE76dgc3b+j8BNuhc/4/A65Ks286uvWNYJ1X1Y2BFkn3afpJku2m6hxcAEzOV7waur6rHgB8l+e22/L3AdcMu5rn3tAHwz+17cuAI/V8F/MfO3r+Xz/D9StK0MOmTJGl++CvgTuCWJHcAp9DMoJ0NLE6yhCbxuRugqh6h2fd3R5Ljq+r7wPnAbe01t07R14HAIUmWAcuBPaaouzqeAF6f5GaaPXNHt+UH0Tyg5TZgUad80LnA4UluTbIF8N+Am4C/o73vqVTVFTT7+5a0eyYPa0/N1P1K0rTwIxskSdLzQqbhoygkaT5ypk+SJEmSesyZPkmSJEnqMWf6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx/4fqxCuvXOmWPkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_feature_importances(model, train, export = False):\n",
    "    \"\"\"\n",
    "    Plots the importance of features from a CART model.\n",
    "    \n",
    "    PARAMETERS\n",
    "    ----------\n",
    "    model  : CART model\n",
    "    train  : explanatory variable training data\n",
    "    export : whether or not to export as a .png image, default False\n",
    "    \"\"\"\n",
    "    \n",
    "    # declaring the number\n",
    "    n_features = X_train.shape[1]\n",
    "    \n",
    "    # setting plot window\n",
    "    fig, ax = plt.subplots(figsize=(12,9))\n",
    "    \n",
    "    plt.barh(range(n_features), model.feature_importances_, align='center')\n",
    "    plt.yticks(pd.np.arange(n_features), train.columns)\n",
    "    plt.xlabel(\"Feature importance\")\n",
    "    plt.ylabel(\"Feature\")\n",
    "    \n",
    "    if export == True:\n",
    "        plt.savefig('Tree_Leaf_50_Feature_Importance.png')\n",
    "\n",
    "def display_tree(tree, feature_df, height = 500, width = 800, export = False):\n",
    "    \"\"\"\n",
    "    PARAMETERS\n",
    "    ----------\n",
    "    tree       : fitted tree model object\n",
    "        fitted CART model to visualized\n",
    "    feature_df : DataFrame\n",
    "        DataFrame of explanatory features (used to generate labels)\n",
    "    height     : int, default 500\n",
    "        height in pixels to which to constrain image in html\n",
    "    width      : int, default 800\n",
    "        width in pixels to which to constrain image in html\n",
    "    export     : bool, defalut False\n",
    "        whether or not to export the tree as a .png file\n",
    "    \"\"\"\n",
    "    # visualizing the tree\n",
    "    dot_data = StringIO()\n",
    "    \n",
    "    # exporting tree to graphviz\n",
    "    export_graphviz(decision_tree      = tree,\n",
    "                    out_file           = dot_data,\n",
    "                    filled             = True,\n",
    "                    rounded            = True,\n",
    "                    special_characters = True,\n",
    "                    feature_names      = feature_df.columns)\n",
    "\n",
    "    # declaring a graph object\n",
    "    graph = pydotplus.graph_from_dot_data(dot_data.getvalue())\n",
    "\n",
    "    # creating image\n",
    "    img = Image(graph.create_png(),\n",
    "                height = height,\n",
    "                width  = width,\n",
    "                unconfined = True)\n",
    "\n",
    "    return img        \n",
    "\n",
    "\n",
    "# INSTANTIATING a classification tree object\n",
    "tree_pruned = DecisionTreeClassifier(max_depth = 4,\n",
    "                    min_samples_leaf=25,\n",
    "                    random_state=219)\n",
    "\n",
    "# FITTING the training data\n",
    "tree_pruned_fit = tree_pruned.fit(X_train, y_train)\n",
    "\n",
    "# PREDICTING on new data\n",
    "tree_pred = tree_pruned_fit.predict(X_test)\n",
    "\n",
    "# plotting feature importance\n",
    "plot_feature_importances(tree_pruned_fit,\n",
    "                         train = X_train,\n",
    "                         export = False)\n",
    "\n",
    "# train accuracy\n",
    "p_tree_train_acc = tree_pruned_fit.score(X_train, y_train).round(4)\n",
    "\n",
    "\n",
    "# test accuracy\n",
    "p_tree_test_acc  = tree_pruned_fit.score(X_test, y_test).round(4)\n",
    "\n",
    "\n",
    "# auc value\n",
    "p_tree_auc       = roc_auc_score(y_true  = y_test,\n",
    "                                 y_score = tree_pred).round(4)\n",
    "\n",
    "# saving the results\n",
    "model_performance.append(['Pruned Tree',\n",
    "                          p_tree_train_acc,\n",
    "                          p_tree_test_acc,\n",
    "                          p_tree_auc])\n",
    "\n",
    "# calling display_tree\n",
    "# display_tree(tree       = tree_pruned_fit,\n",
    "#             feature_df = X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter Tuning with GridSearchCV\n",
    "\n",
    "In the following section, I will automate this process using GridSearchCV to \n",
    "automate hyperparameter optimization on both Logistic Regression and CART model.\n",
    "\n",
    "#### Tuned Classification Trees\n",
    "\n",
    "In this section, I will develop a tuned classification tree model using GridSearchCV. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:40.833229Z",
     "start_time": "2020-02-06T15:33:40.811045Z"
    }
   },
   "outputs": [],
   "source": [
    "# INSTANTIATING a logistic regression model with tuned values\n",
    "\n",
    "tree_tuned = DecisionTreeClassifier(criterion = 'entropy',\n",
    "                                  max_depth = 4,\n",
    "                                  min_samples_leaf = 1,\n",
    "                                  splitter = 'best')\n",
    "\n",
    "#fit\n",
    "tree_tuned_fit = tree_tuned.fit(X_train,y_train)\n",
    "\n",
    "# PREDICTING based on the testing set\n",
    "tree_tuned_pred = tree_tuned.predict(X_test)\n",
    "\n",
    "# declaring model performance objects\n",
    "tree_train_acc = tree_tuned.score(X_train, y_train).round(4)\n",
    "tree_test_acc  = tree_tuned.score(X_test, y_test).round(4)\n",
    "tree_auc       = roc_auc_score(y_true  = y_test,y_score = tree_tuned_pred).round(4)\n",
    "\n",
    "# saving the results\n",
    "model_performance.append(['Tuned Tree',\n",
    "                          tree_train_acc, \n",
    "                          tree_test_acc, \n",
    "                          tree_auc])\n",
    "\n",
    "# calling display_tree\n",
    "# display_tree(tree       = tree_tuned_fit,\n",
    "#              feature_df = X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T12:15:27.251666Z",
     "start_time": "2020-02-06T12:10:51.159077Z"
    }
   },
   "source": [
    "### Random forest\n",
    "\n",
    "In this section, I will build a random forest model using the significant set of explanatory variables ( logit_sig ) and default values for the hyperparameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:41.437347Z",
     "start_time": "2020-02-06T15:33:40.835856Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yVette_1963/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:19: FutureWarning: The pandas.np module is deprecated and will be removed from pandas in a future version. Import numpy directly instead\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA30AAAIWCAYAAADjzuSUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzde7ym9bz/8de7iWrEOG2M9maSYutc45Sicvjll0NtkbQpG9lEau8Q8ds5tA2xJackW+qRHCLkmJ0KFTVTU1MpSZNd2EpMOkhNn98f93fZd3drzbrXzDrMXL2ej8d6zH1/r+/putbq8fD2/V7XlapCkiRJktRNa830BCRJkiRJU8fQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GFrz/QEpK576EMfWvPmzZvpaUiSJKnjFi1adENV/c1guaFPmmLz5s1j4cKFMz0NSZIkdVySa0Yrd3unJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA5be6YnIHXdkuuWMe+Qb61SH0sX7DpJs5EkSdK9jSt9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbo02onybwklyc5NsklSU5M8qwkZye5MsmTkjwjyeL2c2GS+yfZMcmZSU5u7U9MklH6f3iSU5Jc1H62a+X/0sa7JMmBw85luq+PJEmSNBFrz/QEpDE8FngxsB9wPvAyYHvgBcDbgVnA/lV1dpL1gT+3dlsDmwK/Bs4Gngb8eKDvo4Czqmr3JLOA9ZNsC7wSeDIQ4KdJzgL+MMRcdhucfJL9Wn1mPeBvVvVaSJIkSSvNlT6trq6uqiVVdRdwKXB6VRWwBJhHL9D9R5IDgAdW1Z2t3XlVdW1rt7jVHbQz8EmAqlpeVcvohbhTquqWqroZ+Cqww5BzuYeqOqaq5lfV/Fmz56zalZAkSZJWgaFPq6vb+z7f1ff9LmDtqloAvBpYD/hJkseP0m45w69m32Mb6LBzGbJ/SZIkaUYY+rRGSrJRW317P7AQePx4bfqcDryu9TMryQOAHwK7JZmd5H7A7sCPJnvekiRJ0nQz9GlNdWB7sMpFwG3Ad1ZUOcm7k7ygfX0TsFOSJcAiYNOqugA4DjgP+ClwbFVdOGWzlyRJkqZJercmSZoq68zduObuc+Qq9bF0wa6TNBtJkiR1VZJFVTV/sNyVPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqsLVnegJS122+wRwWLth1pqchSZKkeylX+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHeYrG6QptuS6Zcw75FsrrLPUVzpIkiRpirjSJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6NOmSvCDJITMw7uOTLE5yYZJtk7x+uucgSZIkrW4MfZp0VfWNqlowTN30TNbf4W7A16tqa+D3gKFPkiRJ93qGPk1IknlJLk9ybJJLkpyY5FlJzk5yZZInJdk3ycda/YcnOSXJRe1nu9bHz5J8ArgA+LskeyVZ0vp8f2s7K8lxrWxJkoNa+VZJfpLk4tb3g5L8X+BA4NVJzgAWABu1lb8jRjmPXZJc0OZ0eit7cJKvtX5/kmSLVn5Yks8lOS3J0iT/kOQDbU7fTXKf6bj2kiRJ0spYe6YnoDXSY4EXA/sB5wMvA7YHXgC8HfhaX92jgLOqavcks4D1gQcBjwNeWVWvT/JI4P3AtsAfgNOS7Ab8N7BBVW0GkOSBrc/jgTdW1VlJ3g38W1UdmORo4Oaq+mCSecBmVbXV4OST/A3waeDpVXV1kge3Q+8CLqyq3ZLs3MYZab8RsBPwBOBc4EVV9ZYkpwC7DpyzJEmStNpwpU8r4+qqWlJVdwGXAqdXVQFLgHkDdXcGPglQVcuralkrv6aqftI+PxE4s6qur6o7gROBpwO/BB6T5KNJdgFuSjIHeGBVndXafq7VnYinAD+sqqvbvG5s5dsDJ7SyHwAPaeMBfKeq7mjnOAv4bisf7ZxJsl+ShUkWLr912eBhSZIkadoY+rQybu/7fFff97sYfvX4lr7PGa1CVf0B2BI4E9gfOHZCsxxbgBqj/B7TaP/e3uZ0F3BHC7kwxjlX1TFVNb+q5s+aPWfwsCRJkjRtDH2aaqcDr4O/3qP3gFHq/BR4RpKHti2gewFnJXkosFZVfQV4J7BNWyn8Q5IdWtuXA2eN0uefgPuPMadz23gbtnmNbO/8IbB3K9sRuKGqbprQ2UqSJEmrGe/p01R7E3BMklcBy+kFwN/0V6iq3yR5G3AGvdW2b1fV15NsCXy27+meb2v/7gMcnWQ2vS2grxwctKp+3x4ucwm9rZlvTrK4qraqquuT7Ad8tfX9O+DZwGFtvIuBW9s4kiRJ0hot/7tLTdJUWGfuxjV3nyNXWGfpgl2naTaSJEnqqiSLqmr+YLnbOyVJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRha8/0BKSu23yDOSxcsOtMT0OSJEn3Uq70SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zFc2SFNsyXXLmHfIt+5WttRXOEiSJGmauNInSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfo0IUnOmek5SJIkSRqeoU8TUlXbTcc4SdaejnEkSZKkrjP0aUKS3JxkxyQ/THJKksuSHJ1krb7j70+yKMl/JXlSkjOT/DLJC1qddZN8NsmSJBcm2amV75vky0lOBU5LslaSTyS5NMk3k3w7yR6t7v9Lcn6SS5IckySt/Mw2/nlJfp5khzHO4xVJLk5yUZITWtmjk5zeyk9P8qhWflySTyY5o53HM5L8Z5KfJTluqq+5JEmStCoMfVpZTwL+Fdgc2Aj4h1Z+P+DMqtoW+BPwXuDZwO7Au1ud/QGqanNgL+BzSdZtx54K7FNVO7c+57UxXt2OjfhYVT2xqjYD1gOe13ds7ap6EnAg8G+DE0+yKXAosHNVbQm8aaRP4Piq2gI4ETiqr9mDgJ2Bg4BTgQ8DmwKbJ9lqlDH2S7IwycLlty6759WTJEmSpomhTyvrvKr6ZVUtB04Ctm/lfwG+2z4vAc6qqjva53mtfHvgBICquhy4BtikHft+Vd3YV+/LVXVXVf0WOKNv/J2S/DTJEnphbNO+Y19t/y7qG7PfzsDJVXVDm8PIeE8FPt8+n9B3TgCnVlW18/ifqlpSVXcBl442RlUdU1Xzq2r+rNlzRpmCJEmSND28b0orq8b4fkcLRwB3AbcDVNVdfffpZQX93tL3edR6bVXwE8D8qvrvJIcB6/ZVub39u5zR/8YzyvxH019npM+7+j6PfPe/I0mSJK22XOnTynpSkg3bvXx7Aj+eQNsfAnsDJNkEeBRwxSj1fgy8qN3b93Bgx1Y+EvBuSLI+sMcE53468JIkD2lzeHArPwd4afu8NxM7J0mSJGm1ZOjTRI2sfp0LLAAuAa4GTplAH58AZrWtmV8E9q2q20ep9xXg2jbGp4CfAsuq6o/Ap+lttfwacP54AyZ5ZJJvA1TVpcDhwFlJLgL+o1U7AHhlkouBl/O/9/pJkiRJa6z87048acXaytgFwD7AwVX1vHGaTMaY61fVzW3s84Cntfv71hjrzN245u5z5N3Kli7YdYZmI0mSpK5Ksqiq5g+Wey+ShpLkkcCZwAeneehvJnkgcF/gPWta4JMkSZJmmqFPQ6mqX/O/T9iEXgCcjnF3nI5xJEmSpK7ynj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpw3xPnzTFNt9gDgsX7DrT05AkSdK9lCt9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUof5nj5pii25bhnzDvnWPcqX+u4+SZIkTQNX+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDn9Y4SR6e5PNJfplkUZJzk+w+Cf0+MsnJkzFHSZIkaXVh6NMaJUmArwE/rKrHVNW2wEuBvx2ot/ZE+66qX1fVHpMzU0mSJGn1YOjTmmZn4C9VdfRIQVVdU1UfTbJvki8nORU4DSDJm5Ocn+TiJO9qZe9J8qaR9kkOT3JAknlJLmll6yb5bJIlSS5MslMr/1GSrfranp1ki+k5dUmSJGniDH1a02wKXLCC408F9qmqnZM8B9gYeBKwFbBtkqcDnwH2AUiyFr2VwhMH+tkfoKo2B/YCPpdkXeBYYN/WdhNgnaq6eHASSfZLsjDJwuW3LlvZc5UkSZJWmaFPa7QkH09yUZLzW9H3q+rG9vk57edCekHx8cDGVbUU+H2SrUeOV9XvB7reHjgBoKouB64BNgG+DDwvyX2AfwKOG21eVXVMVc2vqvmzZs+ZnJOVJEmSVsKE73uSZtilwItGvlTV/kkeCixsRbf01Q3wvqr61Cj9jKzYPQL4z1GOZ7TBq+rWJN8HXgi8BJg/0ROQJEmSppMrfVrT/ABYN8nr+spmj1H3e8A/JVkfIMkGSR7Wjp0C7AI8sdUb9ENg79ZuE+BRwBXt2LHAUcD5fauKkiRJ0mrJlT6tUaqqkuwGfDjJW4Dr6a3uvRVYb6DuaUn+Hji399BPbgb+EfhdVf0lyRnAH6tqeX+z9u8ngKOTLAHuBPatqttbv4uS3AR8dspOVJIkSZokhj6tcarqN/QevjKa4wbqfgT4yGCl9gCXpwAv7it+CHBja/dn2gNbRmn7SHqr5KdNbOaSJEnS9HN7p+51kjwB+AVwelVd2crmAycxSkAcaPsK4KfAoVV111TPVZIkSVpVrvTpXqeqLgMeM1C2kN7TOcdrezxw/BRNTZIkSZp0rvRJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHeZ7+qQptvkGc1i4YNeZnoYkSZLupVzpkyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM9/RJU2zJdcuYd8i3xjy+1Hf4SZIkaQq50idJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+qZIkuOS7DHT8xiRZF6SS8Y4tnaSG5K8r6/slCSLk/wiybL2eXGS7ZKcmeSKvrKTVzDuYUmua/WuTPLVJE/oOz5qX63dwaP0d3P79+okjxs4dmSStyTZcWDOi5M8q9VZ3r5fkuTUJA8cvD7jtD80yaVJLm7lTx7+tyBJkiRNv7VnegJaLTwHuAJ4SZK3V8/u0AtAwMFV9byRykkA9q6qhUP2/+Gq+mBruyfwgySbV9X17fhE+hrxBeClwLtav2sBewBPAzYEftQ/5z63VdVWrc3ngP2Bw0epd4/2SZ4KPA/YpqpuT/JQ4L4TnLckSZI0rVzpmyRJXtFWfy5KckIrfnqSc5L8cmTVL8n6SU5PckGSJUle2MrnJflZkk+3laTTkqzXjj2x9X1ukiP6VqRmte/nt+OvXcnp7wV8BPgV8JRVuhDjqKovAqcBL1vFrk6iF/pGPB1YWlXXTKCPc4ENJlB/LnBDVd0OUFU3VNWvJ9BekiRJmnaGvkmQZFPgUGDnqtoSeFM7NBfYnt7q0IJW9mdg96raBtgJ+FDa0hmwMfDxqtoU+CPwolb+WeCfq+qpwPK+oV8FLKuqJwJPBF6TZMMJzn094JnAN+kFqb2GbHpi39bHIyYyJnAB8PhV6auqLgbuSrJlK3opvfmP2GFge+ZG/e2TzKJ33t8YY4jR2p8G/F2Snyf5RJJnjDW/JPslWZhk4fJblw1zSpIkSdKUcHvn5NgZOLmqbgCoqhtbjvtaVd0FXJbk4a1ugH9P8nTgLnorTSPHrq6qxe3zImBeu+fs/lV1Tiv/PL0QCb1tmVv03Ts4h15wvHoCc38ecEZV3ZrkK8A7kxxUVcvHabcyWzJHZOD7yvZ1EvDSJJcCLwT+X9+xsbZ3rpdkMTCP3jX+/hh9j9o+ybbADvQC+xeTHFJVxw3Wq6pjgGMA1pm7cQ19RpIkSdIkM/RNjgCj/Q/72wfqAOwN/A2wbVXdkWQpsO4o9ZcD63HPgDQ47hur6nsrM+lmL+BpbR4AD6EXaP5rFfocz9bAygbGfifRW307C7i4qn43RJvbqmqrJHPorW7uDxw17IAtDJ8JnJlkCbAPcNwE5y1JkiRNG7d3To7T6T0E5SEASR68grpzgN+1wLcT8OgVdVxVfwD+lGTkXrv++9i+B7wuyX3auJskud+wk07yAHrbTx9VVfOqah69EDTsFs8JS/IieiuUJ41XdzxVdRXwe3pbZyfUX1UtAw4ADh65fuNJ8rgkG/cVbQVM5B5CSZIkadq50jcJqurSJIcDZyVZDly4guonAqcmWQgsBi4fYohXAZ9Ocgu9VaaRm8SOpbdN8YJ2X+D1wG4r6OdxSa7t+/4R4AcjDyZpvg58IMk6A+X3OI8kt7XPN1TVs1ZQ96Ak/wjcD7iE3r2P1/cdH6uvdyQ5cKRSVf3tKH2fBLwPOGWgfIe2jXPEe6vqbq+WqKoLk1xEL0j/aLz29LbNfrRtub0T+AWw3+inLEmSJK0eUuXtRqu7JOtX1cj76Q4B5lbVm8ZpptXEOnM3rrn7HDnm8aULdp3G2UiSJKmrkiyqqvmD5a70rRl2TfI2er+va4B9Z3Y6kiRJktYUhr41QHu33ReHqZtkc+CEgeLbq+rJkz6xu497KPDigeIvV9VoLz6XJEmSNE0MfR1TVUvoPWBkusc9HDDgSZIkSasZn94pSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowX9kgTbHNN5jDwgW7zvQ0JEmSdC/lSp8kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcN8ZYM0xZZct4x5h3xrpqchSZI6aqmvhtI4XOmTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GFTFvqSLE+yuO9nXivfPsl5SS5vP/v1tTksycGj9HXzGGPs19fPeUm2b+UvTPK1vnpvS/KLvu/PT/KN9nlpkiV98zyqlR+X5OokFyX5eZLjk2wwzjmP9HVxkrOSPHoF1+OQVn6fJAuSXJnkknYez23H5rRxr2o/xyeZ047NS1JJ3tM3xkOT3JHkY33Xs5I8tq/OQa1s/hDnf12Sdfr6Xto39m1JLkzyszbnfUa5Hl9Pcm7f90P7xum/Hgf0/+7T8452TX6e5Iwkmw5c56/0fd8jyXHt88OTfLP93i5L8u0V/L5GzmNxq3t0krXasU2SfDvJL9o5finJnn1zvjnJFe3z8Sv6u5AkSZJm0tpT2PdtVbVVf0GSRwCfB3arqguSPBT4XpLrqupbE+k8yfOA1wLbV9UNSbYBvpbkScA5wDF91Z8K3JTkYVX1O2A74Oy+4ztV1Q2jDPPmqjo5SYADgTOSbFZVf1nB1HZq83kX8A7gNa38HtejeQ8wF9isqm5P8nDgGe3YZ4BLquoV7ZzfBRwLvLgd/yXwPOCd7fuLgUsH+l8CvBR4b/u+B3DZaHMeZW7LgX8CPjnKsauqaus2r8cAX02yVlV9tpU9ENgGuDnJhlV1dVUdDhzejt/cfz2SHNbX9/70fkdbVtWtSZ4DfCPJplX151Znfvs+eL7vBr5fVR9p/W4xytwHz2OrJGsDPwB2a0HxW8C/VNWprZ+dgOtH5pzkTODgqlo4Tv+SJEnSjJru7Z37A8dV1QUALWi8BThkJfp6K71QdkPr6wLgc8D+VXU9sKxvhWsD4Cv0ggTt33OGHah6Pgz8FnjukM3ObeOOKclseqHwjVV1exvrf6rqS23u29ILhSPeTS/sbNS+3wb8bGTVDtgT+NLAMF8DXtjGewywDLh+yHM4EjioBaIxVdUvgX8BDugrfhFwKvAFeqFzIt5K75rc2vo/jd7va+++Oh8E3j5K27nAtX1zu3iYAavqzjbGY4GXAeeOBL52/IyqumTYE0hvFXphkoXLb102bDNJkiRp0k1l6FuvbyvcKa1sU2DRQL2FrXyixuvrHGC7JI8DrgR+0r6vDWwBnN/X7oy+uR60gjEvAB4/5Px2oRe4RvRfj8VJ9qQXMH5VVTeN0v4JwOKqWj5S0D4v5u7X6wvAS5P8Lb2VuV8P9HMT8N9JNgP2Ar44ylhjnf+vgB8DLx/ifAevzV7ASe1nryHaA5DkAcD9quqqgUODfydfArbp37rafBz4TNsSemiSRw457mzgmfRWRjfjnn9bE1JVx1TV/KqaP2v2nFXpSpIkSVol07q9EwhQo9QdrWxl9Pd/Nr0VvVn0Vt3OA/4fsDVwRd82QRh7e+No/Y/njLZF83f0tneOGG2764q2Ho51rQbLv0tvNfB/GD3Qwf+utv0fesHmlQPHV3T+/w58g952xxX567Vp5/9Y4MdVVUnubNtih14pG6P//vNeDhwBvA34zkhhVX2vrWjuQm9V9sI29lirmxslWdz6/npVfSfJs1dhnpIkSdJqZaiVvvZQi9OTXNK+b5HkHeO1G8WlwPyBsm255z1mw7iste23TV9f59ALfdvR26r3J2BdYEfufj/fRGwN/GycOjsBj6Z3ru8ep+4vgEcluf8oxy4Fth55sAhA+7xl/xza/YWLgH+lt4V1NKfSW60ba1VxTFX1C3qriy8Zp2r/tdkTeBBwdXoPf5nHkFs82/xuacGtX//vdsQJwNOBRw30cWNVfb6qXk5vRffpKxjyqqraqqq2rqrDWtml3PNvS5IkSVojDbu989P0VlTugL/eJzXR+7Sgt/Vu3yQjD8N4CPB+4AMr0dcHgPe3Pmh97gt8oh2/DHgksANwYStbDPwzE7ifr/WdJAfQu1/su+PVr6rb6D345RVJHryCerfSe1jLUUnu28aam+QfW9i6kLuvFr4DuKAd6/ch4K1V9fsVzOettIeorITDgXs8VXVEek9m/SDw0Va0F7BLVc2rqnn0AtRE/l6OoHdN1mv9PwvYnt5DgP6qqu4APkzvWo/MZee2VZMWpjeit011Ij5Pbyvwrn397pJk8wn2I0mSJM24Ybd3zq6q83oPsfyrOyc6WFX9Jsk/Ap9u/4M8wJH9D8wA3pHkwL42fwvMTnJtX53/qKr/SO8VCuckKeBPwD9W1W9au0ryU2BOCwfQ2+a5H/cMfWckGbl37uKRp2UCRyR5JzCb3j2BO43z5M7Bcz2J3sNr3kO7p6+vyner6hB6Qe69wGVJ/gzcQm8bKsCrgI+m97qJtPm/apSxLuWeT+0crPOFFRwe6/z/2n+SC+itto3YKMmF9FZP/wR8tKo+2wLgo+hdr5H2Vye5KcmTq+qnK5pn81F6K4VL2rx+C7ywhddBn+HuwXhb4GNJ7qT3f2ocW1Xnj9JuTFV1W3s67JFJjqT3f3ZcDLxpIv1IkiRJq4NUjX87XZLvAG8AvlxV2yTZA3hVVQ37JEvpXmuduRvX3H2OnOlpSJKkjlq6YNfxK+leIcmiqhq8nW7olb796b337vFJrgOu5u6Pz5ckSZIkrYbGDX3t4SHzq+pZSe4HrNUeinKv1baNrjNQ/PKqWjIT89GKtXvxThgovr2qnjwT85EkSZKm07ihr6ruSvIG4EtVdcs0zGm1Z1hYs7QwPvj6EEmSJOleYdind34/ycFJ/i7Jg0d+pnRmkiRJkqRVNuw9ff/U/t2/r6yAwXepSZIkSZJWI0OFvqracKonIkmSJEmafEOFviSvGK28qo6f3OlIkiRJkibTsNs7n9j3eV3gmcAFgKFPkiRJklZjw27vfGP/9yRzuOcj8CWNYvMN5rDQl6ZKkiRphgz79M5BtwIbT+ZEJEmSJEmTb9h7+k6l97RO6AXFJwBfnqpJSZIkSZImx7D39H2w7/OdwDVVde0UzEeSJEmSNImG3d75f6vqrPZzdlVdm+T9UzozSZIkSdIqGzb0PXuUsudO5kQkSZIkSZNvhds7k7wOeD3wmCQX9x26P3D2VE5MkiRJkrTqxrun7/PAd4D3AYf0lf+pqm6csllJkiRJkiZFqmr8WiOVk4fRezk7AFX1q6mYlNQl68zduObuc+SMjL3U9wNKkiTdayRZVFXzB8uHuqcvyfOTXAlcDZwFLKW3AihJkiRJWo0N+yCX9wJPAX5eVRsCz8R7+iRJkiRptTds6Lujqn4PrJVkrao6A9hqCuclSZIkSZoEw76c/Y9J1gd+BJyY5Hf0XtIuSZIkSZ7d9K8AACAASURBVFqNDbvS90LgVuBA4LvAVcDzp2pSkiRJkqTJMdRKX1XdkuTRwMZV9bkks4FZUzs1SZIkSdKqGvbpna8BTgY+1Yo2AL42VZOSJEmSJE2OYbd37g88DbgJoKquBB42VZOSJEmSJE2OYUPf7VX1l5EvSdYGhn+ruyRJkiRpRgwb+s5K8nZgvSTPBr4MnDp105IkSZIkTYZhQ98hwPXAEuC1wLeBd0zVpCRJkiRJk2OFT+9M8qiq+lVV3QV8uv1IkiRJktYQ4630/fUJnUm+MsVz6bQky5MsTnJJklOTPLCVz0tyWzs28vOKdmxpkiV95dsN1L8syfFJ7tPq75jkm31j7pLkvCSXt/pfTPKoduy4JFf39X1OK983yfWt7PIkB41yLhclOanv+8f75tN/LnsMMc6FSa5M8r0k241zDY9Lssco5Zsm+UGSn7e+3pkkfcefm2Rhkp+1c/pgKz8sycHt87pJvp/k3wZ+XyM/h7TyM5Nc0a7B+Um2Gu4vQJIkSZoZ472nL32fHzOVE7kXuK2qtgJI8jl6T0Q9vB27auTYKHaqqhtGviSZN1I/ySzg+8BLgBP7GyXZDPgo8IKq+lkrewEwD/hVq/bmqjp5lDG/WFVvSPIQ4IokJ1fVf7c+/p7e/1nw9CT3q6pbqmr/vrl9s/9ckjxvvHFavZ2ArybZaWS+w0iyHvAN4HVVdVp7h+RXgNcDH2/X4WPArlV1eXsI0X4Dfdy3tVlUVe9qxbet4Heyd1UtTPJK4Ajg2cPOV5IkSZpu46301RiftWrOpfeuw1VSVcuB88bo663Av/cHqKr6RlX9cAL9/x74BTC3r/hlwAnAacALVmbeY4x1BnAMA4FsCC8Dzq6q01o/twJvoHcfKsBbgMOr6vJ2/M6q+kRf+7WBLwBXVtUhTMyYv8ck+7XVxYXLb102wW4lSZKkyTNe6NsyyU1J/gRs0T7flORPSW6ajgl2TVudeya91akRGw1sJdyh79gZreyno/S1LvBk4LujDLUpcME40zmib8wTBw+2raDrAhf3Fe8JfBE4CdhrnP6HGqfPBcDjh+xzxKbAov6CqroKWD/JA4DNBo8PeAtwZ1UdOFC+3sDvZM9R2u5C3xbogTkcU1Xzq2r+rNlzhj4ZSZIkabKtcHtnVc2aroncC6yXZDG97ZWL6G3LHDH09s5mo9bXxsDJVXXxKO3+qm3TPB2YDRxTVR9sh8badrln2275OOA1VfXn1s8Tgeur6pok1wL/meRBVfWHFY2/gnHuMdUh6ozWZqxV6GFWp38MPDXJJlX1877yFW3vPDHJ/YBZwDbDT1WSJEmafsO+skGrbiREPBq4L717+lbWSEh8LPCUdq/eoEtpgaSqft/qHwOsP0T/X6yqTYEdgA8leUQr3wt4fJKlwFXAA4AXrcJ5DNoaGPp+vuZSYH5/QZLHADdX1Z/a8W1X0P6HwIHAd5I8csgx9wY2BD4PfHyC85UkSZKmlaFvmlXVMuAA4OC0p26uQl+/oXfv2ttGOfwB4ND24JURsyfY/7n07t97U5K1gBcDW1TVvKqaB7yQ4bd4rlCSZ9C7n2+irwU5Edg+ybNaP+sBR9E7f+g9aOXtSTZpx9dK8i/9HVTVV1q976Y9VXU8VXUHvXdVPmXgGkuSJEmrFUPfDKiqC4GLgJe2osF7+g6YQHdfA2YP3AdIVS0B3gQc315TcDbw9/RWp0YcMTDufUfp//3AK4Fdgeuq6rq+Yz8EnpBk7ijt+o01zp7t+8+BtwMvGuLJnZ9Kcm37ObeqbqMXPt+R5ApgCXA+vSd20ra+HgiclORnwCXc/cE0tHpHA18FvtHulRy8p2/BKG1uAz4EHDzOnCVJkqQZkyofyilNpXXmblxz9zlyRsZeumDXGRlXkiRJ0y/JoqqaP1juSp8kSZIkddh4L2eXpl2SjwNPGyj+SFV9dibmI0mSJK3JDH1a7VTVqjzZVJIkSVIft3dKkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mE/vlKbY5hvMYaEvSZckSdIMcaVPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jDf0ydNsSXXLWPeId+a6WlMiaW+f1CSJGm150qfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElShxn6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZujTtEmyPMniJJck+XKS2a38EUm+kOSqJJcl+XaSTcboY16SS0YpT5J3JLkyyc+TnJFk077j6yf5ZBvjwiSLkrxmsM8kOyapJM/va/vNJDu2z89r7S9qc33tpF4kSZIkaZIZ+jSdbquqrapqM+AvwD8nCXAKcGZVbVRVTwDeDjx8gn3vD2wHbFlVmwDvA76RZN12/FjgD8DGVbU1sAvw4DH6uhY4dLAwyX2AY4DnV9WWwNbAmROcpyRJkjSt1p7pCehe60fAFsBOwB1VdfTIgapavBL9vRXYsapubX2cluQcYO8kZwJPAl5WVXe149cD7x+jr4uA+yR5dlV9v6/8/vT+m/l96+N24IqVmKskSZI0bVzp07RLsjbwXGAJsBmwaBX7ewBwv6q6auDQQmDT9nPRSOAb0nuBd/QXVNWNwDeAa5KclGTvJKP+N5RkvyQLkyxcfuuyCQwrSZIkTS5Dn6bTekkW0wtjvwI+M8XjBah7FCaHtnsLfz1Ww6r6Uau7w0D5q4FnAucBBwP/OUb7Y6pqflXNnzV7ziqcgiRJkrRqDH2aTiP39G1VVW+sqr8AlwLbrkqnVXUTcEuSxwwc2ga4rP1sObIqV1WHV9VWwAPG6fpwRrm3r6qWVNWHgWcDL1qVuUuSJElTzdCnmfYDYJ2RJ2kCJHlikmdMsJ8jgKOSrNf6eBawPfD5qvoFvdXF9yaZ1Y6vS28lcExVdRrwIGDL1mb9kad4NlsB10xwnpIkSdK08kEumlFVVUl2B45McgjwZ2ApcOAKmj0uybV93w8CPkovoC1Jshz4LfDCqrqt1Xk1vWD4iyQ3ArfRe/jLeA4Hvt4+B3hLkk+19rcA+w7RhyRJkjRjUnWPW54kTaJ15m5cc/c5cqanMSWWLth1pqcgSZKkJsmiqpo/WO72TkmSJEnqMLd3arWUZHPghIHi26vqyTMxH0mSJGlNZejTaqmqltB7UIokSZKkVeD2TkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkdZuiTJEmSpA4z9EmSJElSh/nKBmmKbb7BHBYu2HWmpyFJkqR7KVf6JEmSJKnDDH2SJEmS1GGGPkmSJEnqMEOfJEmSJHWYoU+SJEmSOszQJ0mSJEkd5isbpCm25LplzDvkW5Pe71JfAyFJkqQhuNInSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfZIkSZLUYYY+SZIkSeowQ58kSZIkdZihT5IkSZI6zNAnSZIkSR1m6JMkSZKkDjP0SZIkSVKHGfokSZIkqcMMfaNI8pAki9vPb5Nc1/f9vkl2T1JJHt/XZq0kRyW5JMmSJOcn2TDJT1u7XyW5vq+feUmWtrojZUe1vp7S1+5nSQ4bYs5fT3LuQNlhSW5N8rC+spv7Pi9vY1ya5KIk/5JkzL+JJDsmWZbkwjavfxul/PIkH+xrs+/AeS9O8oR2/rcNlN93yPqXJTk+yX0Gxu9v86x2rJJ8qG8+B/dfzySvaL+zS1u/B7fy45Jc3dffOa384Um+2a7XZUm+Pd7vRpIkSZpJa8/0BFZHVfV7YCvoBSfg5qrqDzJ7AT8GXgoc1or3BB4JbFFVdyX5W+CWqnpya7MvML+q3tDXD8BOVXXDwBQ+B7ykqi5KMgt43Irmm+SBwDbAzUk2rKqr+w7fAPwr8NZRmt5WVSPn+TDg88Ac4N9WMNyPqup5Se4HLE7yzYHy9YALk5xSVWe3Y1/sP+823jzgqpHx+8rHrd+uyfeBlwAn9o8/ynxvB/4hyfsGr3OS5wIHAs+pql8nWRd4eV+VN1fVyQP9vRv4flV9pPWxxShjSpIkSasNV/omKMn6wNOAV9ELfSPmAr+pqrsAquraqvrDSg7zMOA3rZ/lVXXZOPVfBJwKfGFgTgD/CeyZ5MEr6qCqfgfsB7whLXmNU/8WYBGw0UD5bcBiYIPx+lhZVbUcOG/IMe4EjgEOGuXY24CDq+rXrd8/V9Wnx+lvLnBt31wuHq1Skv2SLEyycPmty4aYpiRJkjQ1DH0Ttxvw3ar6OXBjkm1a+ZeA57etgB9KsvWQ/Z3Rt4VwJJh8GLgiySlJXttWoFZkL+Ck9rPXwLGb6QW/N403kar6Jb2/iYeNVzfJQ4CnAJcOlD8I2Bj4YV/xngNbL9dr5Rv1lX18iPojY6wLPBn4bl/xDgNt+sPox4G9k8wZOI3N6AXXsRzR19/IiuLHgc8kOSPJoUkeOVrDqjqmquZX1fxZsweHlSRJkqaP2zsnbi/gyPb5C+37BVV1bZLHATu3n9OTvLiqTh+nv3ts76yqd7eQ8RzgZW2MHUdrnOThwGOBH1dVJbkzyWZVdUlftaPobcX80Gh9DHY5zvEdklwI3AUsqKpLk+zYyi+mtxV1QVX9tq/NaNs1YZTtnePU3yjJYnqh8uSBVbaxtndSVTclOR44ALhtnPPrd4/tnVX1vSSPAXYBnktvK+tmVXX9BPqVJEmSpo0rfRPQVrd2Bo5NshR4M71VqQBU1e1V9Z2qejPw7/RWBVdKVV1VVZ8Engls2cYezZ7Ag4Cr25zmMbDFs6r+SO9+vdevaMwWZpYDv1tBtR9V1dZVtW1VHT1QvgWwOfC6JKOFuVU1EhIfCzwlyQsm0PZIelty79dXdimw7UQnUVU3VtXnq+rlwPnA0yfahyRJkjRdDH0TswdwfFU9uqrmVdXfAVcD2yfZZmSrX3pPwNwCuGZlBkmya999dRvTC2J/HKP6XsAubT7z6IWYwfv6AP4DeC1jrO4m+RvgaOBjVVUrM2+Atu31fYz+4JhJUVW/AQ6hd0/esG1upLcF91V9xe8DPpDkEQBJ1klywIr6SbJzktnt8/3p3dP4q4mdgSRJkjR9DH0TsxdwykDZV+htwXwYcGqSS4CL6T1A5GND9Nl/T9/xrezl9O7pWwycAOzdHl5yN+2Jlo8CfjJS1p7ceVOSJ/fXbVtITwHW6Ster417KfBfwGnAu4aY83iOBp6eZMP2ffAeve3GaT9M/a8Bs5Ps0L4P3tO3xyhtPgQ8dORLVX2b3j16/9WuwSLuHoqPGOjzvvRC9cK2lfVc4NiqOn+c85EkSZJmTFZhUUfSENaZu3HN3efI8StO0NIFu056n5IkSVpzJVlUVfMHy13pkyRJkqQO8+mda4gkr+Ser104u6r2n4Kx/g/w/oHiq6tq98keS5IkSdLUMvStIarqs8Bnp2ms7wHfm46xJEmSJE0tt3dKkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mE/vlKbY5hvMYaEvUpckSdIMcaVPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jDf0ydNsSXXLWPeId+akbGX+n5ASZKkez1X+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR1mKFPkiRJkjrM0CdJkiRJHWbokyRJkqQOM/RJkiRJUocZ+iRJkiSpwwx9kiRJktRhhj5JkiRJ6jBDnyRJkiR12L0i9CV5RJIvJLkqyWVJvp1kk3bsoCR/TjKnr/6OSSrJ8/vKvplkx/b5PkkWJLkyySVJzkvy3HZsaZIlSRa3n6Na+XFJ9hiY17wkl4wx57WT3JDkfX1lp7Q+f5FkWd8Y2yU5M8n8Vm9OkuPb+V7VPs/pG7OSvLGv348l2bd9fkqSn7Z+f5bksCGu79eTnDtQdliS6/rmuGCI+V/RV3byKP1clmSvceZyXKu/Tvv+0CRL+45vmuQHSX7efn/vTJK+47sluTjJ5e33uNuwfUuSJEmro86HvvY/6E8BzqyqjarqCcDbgYe3KnsB5wO7DzS9Fjh0jG7fA8wFNquqzYDnA/fvO75TVW3Vfg5Yyak/B7gCeMlIKKmq3atqK+DVwI/6xjhnoO1ngF+2890IuBo4tu/474A3JbnvKON+DtivjbMZ8KUVTTLJA4FtgAcm2XDg8If75njIEPPfu69sj8F+gBcCn0pynxXNCVgO/NMoc10P+AawoKo2AbYEtgNe345vCXwQeGFVPR54AfDBJFuM17ckSZK0uup86AN2Au6oqqNHCqpqcVX9KMlGwPrAO+iFv34XAcuSPLu/MMls4DXAG6vq9tbf/1TVCsPRStgL+AjwK+ApwzZK8lhgW3rBdMS7gfntfAGuB04H9hmli4cBvwGoquVVddk4Q74IOBX4AvDSYee5MqrqSuBW4EHjVD0SOCjJ2gPlLwPOrqrTWn+3Am8ADmnHDwb+vaqubsevBt4HvHmIvu8myX5JFiZZuPzWZeOfnCRJkjRF7g2hbzNg0RjH9uL/t3f/0XpV9Z3H3x9BQYVBROtk8EcsRhgEiRhLl1KLlbZ24fBjFgiILugwaqdaO6gIjoxlmELTBifiQF1gKz8sAoqoiCPYIjCiC2rAkBiUAiVVaDv4o6L88AfJd/4459qTx+fePDe59z43575fa92V8+yzz97f89y9IN/svc+By4AvAXsm+aWB839MkxB2vQD4VlX9cIo+b+gsUzxpugG3M1KvBq5p45tySeOAvYHVVbVhoqA9Xg28qFNvOfDOJNsNXL8SuKtdivmWJDtupr+J73BYnCd1voffHiH2Szv1VwyeTLI/cHdVPbiZdr4F3Ay8caD8RQyMhaq6F9gpyb8Zdh5Yxabf22Rtb6KqLqiqZVW1bLun7DJVVUmSJGlWLYSkbyrHAJdX1UbgKuCo7smq+hJAkl+bZrvd5Z0rtyCu1wI3tDNRnwSOGJKcTSZAba68ncX6W5rZLzrlZwDLgC+0566dtKPkWTRJ8M1V9XfA40n26VTpLu+8boTYu8s7u7NrJyW5C7gVOH2EdgDOopmh647xyb4b2vJh54eVDWtbkiRJmpcWwl9a19Esd9xEu09rCfDX7cM4jmH4jNqZbLq37x7guUl2HlJ3phwLHNzGdRuwG80y1VGsA16S5Oe/2/Z4P+AbA3XPAk5hYBxU1b1V9SGa2cb9kuw2SV9H0yy1vK+NdTGzs8RzZVXt2fZ3yQizj1TVPTSzm6/rFK+jSWh/LskvAw9X1Y+GnafZr7jJEtdJ2pYkSZLmpYWQ9H0R2CHJmyYKkryMZr/c6VW1uP35d8DuSZ7Xvbjd/7UrTdI0sQ/sL4EPTjwIJcmiJG+YiWDbZYYHAs+diA14KyMu8WwTkq+x6bLU04Db23Pdut+kSWhe2+n/kM7TLJfQPLjkB5N0dyzwmk6cL2UW9/VV1VU0yy2H7UUc5kyafXoTLgUOTHIw/HwZ7QeBP2vPnw28J8ni9vximof+vH+EtiVJkqR5qfdJX1UVzZM5f7N9fcE6miWCB9E81bPrUwxPWs4Ent35fBrNw1DuTPPKhU+3nyd09/Rd0ik/P8n97c/EKw727JTdD7wF+OLEQ2JanwEOnXhVwAhOBF7YvhrhXuCFbdkwg/f2Rpo9fauBj9IsudwweFGbED0XuGWirF0y+sMkB4wY56Dunr6/maTOGcA7ujOZk6mqdcDtnc+P0TwB9LR2uehamie3ntueX00z8/nZJN+keUDNu9vyKduWJEmS5qs0OZGk2bLDoiW16PgPjKXv9csPGUu/kiRJmntJbquqwe1K/Z/pkyRJkqSFbMp3jUkASX4X+MOB4i9X1VvHEQ9AkvOAVwwUn1NVF44jHkmSJGm+MunTZrWJ1LxKpsaZcEqSJEnbEpd3SpIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo/59E5plu27+y6s8iXpkiRJGhNn+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx3xPnzTL1j7wEItP/dzY+l/vOwIlSZIWNGf6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+hagJJXko53P2yf5TpJrOmWHJ1mT5JtJ1iY5vHPuoiT3JVndnv+jzrkbkyxrj9cnecZA3ye0fa3u/Ow9SZyL21j/oFN2bpITBvvq1P96e3xQe+2JnfMvacveNeJ93NWJ8cq2/PQkD7RldyY5dhpfvSRJkjTnTPoWpkeAfZI8uf38m8ADEyeT7AecDRxWVXsBhwJnJ3lxp42Tq2opsBQ4Psnzp9H/FVW1tPNz5xR1HwT+MMmTptH+hLXA0Z3PxwB3DNSZ6j6O68R4ZKd8ZXvNYcD5SZ64BbFJkiRJc8Kkb+H6PHBIe3wscFnn3LuAs6rqPoD2zz8BTh7Szo7tn4/MUpzfAa4Hjt+Ca78F7JjkWUkCvIbmvoeZ9n1U1d3Ao8CuWxCbJEmSNCdM+hauy4FjkuwIvBi4tXPuRcBtA/VXteUTViRZDdwPXF5VD06j76MHlnc+eTP1lwPvTLLdNPqYcCVwFPBy4HbgJwPnp7qPSzsxrhhsOMn+wN3D7j3Jm5OsSrJqw6MPbUHYkiRJ0szYftwBaDyqak2SxTSzfP9n4HSA2kzZyVV1ZZKdgOuTvLyqvjJi91dU1dumEet9Sf4WeP3gqWHVBz5/HLgC2ItmNvPlA+enuo/jqmrVkD5OSvIm4JdpZg+HxXwBcAHADouWDItTkiRJmhPO9C1sV9Ps3btsoHwdsGygbH/gF/beVdXDwI3AgbMQX9dZwClsOma/x6ZLK58OfHcgvn8Gfkazb/H6yRqf5n2srKo9afYLXtLOlkqSJEnzkknfwvYR4IyqWjtQfjbwnnYmkPbP/wa8f7CBJNsDBwD3zmKcVNU3aZLO13aKbwTe0O7Xg2bf3w1DLn8fcEpVbZis/S25j6q6imbZ65bsN5QkSZLmhMs7F7Cquh84Z0j56iSnAJ9tn0z5M+DdVbW6U21FktOAJ9HMoF01STdrkmxsjz8OrKHZ09edUfv9EZeGngl8rfP5Applm3ckKZoE7D1D7meqtqe6j0uTPNYef7eqDh5y/RnAx5J8uKo2DjkvSZIkjVWq3G4kzaYdFi2pRcd/YGz9r19+yOYrSZIkaZuX5LaqGtym5fJOSZIkSeozl3dq7JLsC3x0oPgnVXXAOOKRJEmS+sSkT2PXPkhm6bjjkCRJkvrI5Z2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjvrJBmmX77r4Lq5YfMu4wJEmStEA50ydJkiRJPWbSJ0mSJEk9ZtInSZIkST1m0idJkiRJPWbSJ0mSJEk9ZtInSZIkST3mKxukWbb2gYdYfOrnxh2GJEmSZtn6efqaLmf6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6JEmSJKnHTPokSZIkqcdM+iRJkiSpx0z6ZkCSI5JUkr3az/cl2XOgzgeSvLs9/pUkNya5O8ntST6XZN9J2n5vktXtz4bO8duTnJ7kXW29i5I8mmTnzrXntHE9o/3cvX51klOnuKcbk9yV5I4kX564n075RBtXtuWnJ3mgLbszybGb+c5+Ncmtbf1vJDm9LT8hybkj3PcD2fRenjakj6cm+V6SXQbKP53kdRN9tWV7tvc2Ec8FbflBSa5J8rudvn6aZG17vHyq+5QkSZLGbftxB9ATxwI3A8cApwOXt8f/AyDJE4AjgVckeRbwceD1VfWV9vyBwB7A2sGGq+pM4My23sNVtXTi3ESi1HEPcBjwV22frwIe6Jx/rHv9CI6rqlVJ3gysAA7tlg+pv7Kqzk6yBLgtyZVV9bNJ2r4YeF1V3ZFkO2CTJHmE+15ZVWdPFXxVPZLkC8DhbX+0CeCBwOuB13Wqf7Bt8zNtvX0H2roQuLA9tx54VVV9d6r+JUmSpPnAmb6tlGQn4BXAiTSJHsBlnWOAVwLrq+ofgLcBF08kfABVdXNVfXoGwrkMOLo9Pgj4MvD4DLT7f4EXjFq5qu4GHgV2naLaLwH/1NbfUFV3blWEkxv8XRwBXFtVjw7UWwTcP/Ghqn4hAZ+OJG9OsirJqg2PPrQ1TUmSJElbxaRv6x1Ok0T8HfD9JPtX1RpgY5L92jrH0CQfAC8Cbp+lWO4GnplkV5rZx8sHzj95YEnk0b/YxFD/gU1nIS/ttLFisHKS/YG7q+rBKdpcCdyV5FNJ3pJkxxFjmXBSJ4Ybpqh3LfDSJLu1n7u/i8F4vpjk80lOGrZcdDqq6oKqWlZVy7Z7yi6bv0CSJEmaJSZ9W6+bXF3efoZ2hinJ9jRLLj8x7OJ2X9s3kpwzQ/FcRZPYHAB8aeDcY1W1tPNzxWbaujTJapqZzHd1yo/rtHFyp/ykJHcBt9Isc51UVZ0BLAO+QLPU8trN3diAlZ0YXjVFPz8FrgaObPc2Lm37HKx3IfDvaX5PBwG3JNlhmjFJkiRJ8457+rZCO3v0G8A+SQrYDqg0D2y5jCa5uAlY05n1WgfsD3wGoKoOSHIk8NoZCutympnEi6tqY5KtaWuyvXuTmdjT9x+BS5LsUVU/nqxyVd0LfCjJh4HvdGbjZtplwGlAgM9Mts+wqv4R+AjwkSRfB/aZpXgkSZKkOeNM39Y5Erikqp5XbPOAXQAACZdJREFUVYur6jnAfcCBbULzPWA5my4nPA84IcnLO2VPmamAqupbwHuBP5+pNrcghquAVcDxk9VJckj+NSNdAmwAfjBLId3Q9vFWhi/tJMlrkjyxPf63wG5s+hAcSZIkaZtk0rd1jgU+NVD2SZrlitAkGHt161TVP9M8bOVPktyT5Cs0yeO5MxVUVZ3fJp2DBvf0benrBrp7+v5mkjpnAO9onyI6zBtp9vStBj5KM6u4YRoxnDRwL4snq1hVG2l+L7vRPJRmmN8Cvp7kDuA64OT2dyVJkiRt01JV445B6rUdFi2pRcd/YNxhSJIkaZatX37IWPtPcltVLRssd6ZPkiRJknrMB7nMI0neCxw1UPyJ9kXls9Xnp4DnDxSfUlXXzVD759E8/bPrnPZpmTMmyW8DfzpQfF9VHTGT/UiSJEnbGpO+eaRN7mYtwZukz1lNiqrqrbPZfqef62j24kmSJEnqcHmnJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mK9skGbZvrvvwqrlh4w7DEmSJC1QzvRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPmfRJkiRJUo+Z9EmSJElSj5n0SZIkSVKPparGHYPUa0l+BNw17ji0TXgG8N1xB6FthuNFo3KsaDocL9u251XVMwcLtx9HJNICc1dVLRt3EJr/kqxyrGhUjheNyrGi6XC89JPLOyVJkiSpx0z6JEmSJKnHTPqk2XfBuAPQNsOxoulwvGhUjhVNh+Olh3yQiyRJkiT1mDN9kiRJktRjJn3SDEnymiR3JbknyalDzu+Q5Ir2/K1JFs99lJoPRhgrr0xye5LHkxw5jhg1P4wwVt6R5M4ka5Jcn+R544hT88MI4+X3kqxNsjrJzUn2HkecGr/NjZVOvSOTVBKf5rmNM+mTZkCS7YDzgN8B9gaOHfI/0xOBf6mqFwArgT+d2yg1H4w4Vr4FnAB8bG6j03wy4lj5GrCsql4MXAn82dxGqflixPHysarat6qW0oyV/zXHYWoeGHGskGRn4O3ArXMboWaDSZ80M34FuKeq/r6qfgpcDhw2UOcw4OL2+Erg1UkyhzFqftjsWKmq9VW1Btg4jgA1b4wyVm6oqkfbj7cAz57jGDV/jDJeftj5+FTABzssTKP8nQXgf9L848CP5zI4zQ6TPmlm7A58u/P5/rZsaJ2qehx4CNhtTqLTfDLKWJFg+mPlRODzsxqR5rORxkuStya5l+Yv82+fo9g0v2x2rCR5CfCcqrpmLgPT7DHpk2bGsBm7wX9BHaWO+s9xoFGNPFaSvAFYBqyY1Yg0n400XqrqvKraAzgFOG3Wo9J8NOVYSfIEmm0o75yziDTrTPqkmXE/8JzO52cD/zhZnSTbA7sA35+T6DSfjDJWJBhxrCQ5GHgvcGhV/WSOYtP8M93/tlwOHD6rEWm+2txY2RnYB7gxyXrgV4GrfZjLts2kT5oZXwWWJHl+kicBxwBXD9S5Gji+PT4S+GL5osyFaJSxIsEIY6VdgnU+TcL34Bhi1PwxynhZ0vl4CHD3HMan+WPKsVJVD1XVM6pqcVUtptkvfGhVrRpPuJoJJn3SDGj36L0NuA74BvDxqlqX5Iwkh7bV/hLYLck9wDuASR+RrP4aZawkeVmS+4GjgPOTrBtfxBqXEf+7sgLYCfhE+xh+/wFhgRpxvLwtybokq2n+P3T8JM2px0YcK+qZONEgSZIkSf3lTJ8kSZIk9ZhJnyRJkiT1mEmfJEmSJPWYSZ8kSZIk9ZhJnyRJkiT1mEmfJEk9k2RD+wqHiZ/FW9DG05L8/sxH9/P2D00yp6+uSXJ4kr3nsk9Jmg98ZYMkST2T5OGq2mkr21gMXFNV+0zzuu2qasPW9D0bkmwP/AXNPV057ngkaS450ydJ0gKQZLskK5J8NcmaJG9py3dKcn2S25OsTXJYe8lyYI92pnBFkoOSXNNp79wkJ7TH65O8L8nNwFFJ9khybZLbknwpyV5D4jkhybnt8UVJPpTkhiR/n+TXk3wkyTeSXNS55uEk729jvT7JM9vypUluae/rU0l2bctvTHJWkpuAU4BDgRXtPe2R5E3t93FHkk8meUonng8m+Uobz5GdGN7dfk93JFnelm32fiVpnLYfdwCSJGnGPTnJ6vb4vqo6AjgReKiqXpZkB+DLSb4AfBs4oqp+mOQZwC1JrgZOBfapqqUASQ7aTJ8/rqoD27rXA79XVXcnOQD4c+A3NnP9rm2dQ4HPAq8A/jPw1SRLq2o18FTg9qp6Z5L3AX8EvA24BPiDqropyRlt+X9t231aVf16G9cSOjN9SX5QVR9uj/+4/Y7+d3vdIuBAYC/gauDKJL8DHA4cUFWPJnl6W/eCLbhfSZozJn2SJPXPYxPJWsdvAS/uzFrtAiwB7gfOSvJKYCOwO/CsLejzCmhmDoGXA59IMnFuhxGu/2xVVZK1wP+rqrVte+uAxcDqNr4r2vp/BVyVZBeaxO6mtvxi4BODcU1inzbZexqwE3Bd59ynq2ojcGeSie/jYODCqnoUoKq+vxX3K0lzxqRPkqSFITSzYddtUtgs0Xwm8NKq+lmS9cCOQ65/nE23hQzWeaT98wnAD4YknZvzk/bPjZ3jic+T/X1llAcTPDLFuYuAw6vqjvZ7OGhIPNB8dxN/Dva5pfcrSXPGPX2SJC0M1wH/JckTAZK8MMlTaWb8HmwTvlcBz2vr/wjYuXP9PwB7J9mhnV179bBOquqHwH1Jjmr7SZL9ZugengBMzFS+Hri5qh4C/iXJr7XlbwRuGnYxv3hPOwP/1H4nx43Q/xeA/9TZ+/f0Wb5fSZoRJn2SJC0MfwHcCdye5OvA+TQzaJcCy5Ksokl8vglQVd+j2ff39SQrqurbwMeBNe01X5uir+OAE5PcAawDDpui7nQ8ArwoyW00e+bOaMuPp3lAyxpgaad80OXAyUm+lmQP4L8DtwJ/TXvfU6mqa2n2961q90y+qz01W/crSTPCVzZIkqRtQmbgVRSStBA50ydJkiRJPeZMnyRJkiT1mDN9kiRJktRjJn2SJEmS1GMmfZIkSZLUYyZ9kiRJktRjJn2SJEmS1GMmfZIkSZLUY/8fwTyOQ9lzhRIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# INSTANTIATING a random forest model with default values\n",
    "rf_default = RandomForestClassifier(n_estimators     = 10,\n",
    "                                    criterion        = 'gini',\n",
    "                                    max_depth        = None,\n",
    "                                    min_samples_leaf = 1,\n",
    "                                    bootstrap        = True,\n",
    "                                    warm_start       = False,\n",
    "                                    random_state     = 219)\n",
    "\n",
    "# FITTING the training data\n",
    "rf_default_fit = rf_default.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "# PREDICTING based on the testing set\n",
    "rf_default_fit_pred = rf_default_fit.predict(X_test)\n",
    "\n",
    "plot_feature_importances(rf_default_fit,\n",
    "                         train = X_train,\n",
    "                         export = False)\n",
    "\n",
    "# declaring model performance objects\n",
    "rf_train_acc = rf_default_fit.score(X_train, y_train).round(4)\n",
    "rf_test_acc  = rf_default_fit.score(X_test, y_test).round(4)\n",
    "rf_auc       = roc_auc_score(y_true  = y_test,\n",
    "                             y_score = rf_default_fit_pred).round(4)\n",
    "\n",
    "# appending to model_performance\n",
    "model_performance.append(['Random Forest',rf_train_acc, rf_test_acc, rf_auc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient boosted machines\n",
    "\n",
    "In this section, I develop a GradientBoostingClassifier model with default \n",
    "values for the hyperparameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:41.894035Z",
     "start_time": "2020-02-06T15:33:41.440083Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Model', 'Training Accuracy', 'Testing Accuracy', 'AUC Value'],\n",
       " ['Logistic Regression', 0.7019, 0.7248, 0.6637],\n",
       " ['KNN Classification', 0.7608, 0.7105, 0.6819],\n",
       " ['Pruned Tree', 0.7622, 0.7413, 0.7266],\n",
       " ['Tuned Tree', 0.767, 0.7433, 0.7298],\n",
       " ['Random Forest', 0.9829, 0.7166, 0.6899],\n",
       " ['GBM', 0.841, 0.7556, 0.7338]]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# INSTANTIATING the model object without hyperparameters\n",
    "gbm_default = GradientBoostingClassifier(loss          = 'deviance',\n",
    "                                              learning_rate = 0.1,\n",
    "                                              n_estimators  = 100,\n",
    "                                              criterion     = 'friedman_mse',\n",
    "                                              max_depth     = 3,\n",
    "                                              warm_start    = False,\n",
    "                                              random_state  = 802)\n",
    "\n",
    "# FIT step is needed as we are not using .best_estimator\n",
    "gbm_default_fit = gbm_default.fit(X_train, y_train)\n",
    "\n",
    "# PREDICTING based on the testing set\n",
    "gbm_default_pred = gbm_default_fit.predict(X_test)\n",
    "\n",
    "# declaring model performance objects\n",
    "gbm_train_acc = gbm_default_fit.score(X_train, y_train).round(4)\n",
    "gbm_test_acc  = gbm_default_fit.score(X_test, y_test).round(4)\n",
    "gbm_auc       = roc_auc_score(y_true  = y_test,\n",
    "                              y_score = gbm_default_pred).round(4)\n",
    "\n",
    "# appending to model_performance\n",
    "model_performance.append(['GBM', gbm_train_acc, gbm_test_acc, gbm_auc])\n",
    "\n",
    "# checking the results\n",
    "model_performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate models\n",
    "\n",
    "Now it's time to evaluate the performances and choose the best model by comparing\n",
    "AUC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-02-06T15:33:41.972550Z",
     "start_time": "2020-02-06T15:33:41.898171Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Model</th>\n",
       "      <th>Training Accuracy</th>\n",
       "      <th>Testing Accuracy</th>\n",
       "      <th>AUC Value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>GBM</td>\n",
       "      <td>0.8410</td>\n",
       "      <td>0.7556</td>\n",
       "      <td>0.7338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tuned Tree</td>\n",
       "      <td>0.7670</td>\n",
       "      <td>0.7433</td>\n",
       "      <td>0.7298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Pruned Tree</td>\n",
       "      <td>0.7622</td>\n",
       "      <td>0.7413</td>\n",
       "      <td>0.7266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Random Forest</td>\n",
       "      <td>0.9829</td>\n",
       "      <td>0.7166</td>\n",
       "      <td>0.6899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>KNN Classification</td>\n",
       "      <td>0.7608</td>\n",
       "      <td>0.7105</td>\n",
       "      <td>0.6819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Logistic Regression</td>\n",
       "      <td>0.7019</td>\n",
       "      <td>0.7248</td>\n",
       "      <td>0.6637</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Model  Training Accuracy  Testing Accuracy  AUC Value\n",
       "5                  GBM             0.8410            0.7556     0.7338\n",
       "3           Tuned Tree             0.7670            0.7433     0.7298\n",
       "2          Pruned Tree             0.7622            0.7413     0.7266\n",
       "4        Random Forest             0.9829            0.7166     0.6899\n",
       "1   KNN Classification             0.7608            0.7105     0.6819\n",
       "0  Logistic Regression             0.7019            0.7248     0.6637"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# converting to DataFrame and checking the results\n",
    "model_performance_df = pd.DataFrame(model_performance[1:], columns = model_performance[0])\n",
    "\n",
    "model_performance_df.sort_values(by = 'AUC Value', ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to AUC, GBM gives us the best prediction accuracy, even though it is \n",
    "overfitted (the difference between training and testing score is far beyond \n",
    "0.05). \n",
    "\n",
    "However, this result tells us that the gradient boosted machines model \n",
    "has 73.4% chance of telling whether a customer in the database is going to buy\n",
    "Halfway There. \n",
    "\n",
    "\n",
    "AUC primarily takes into account Sensitivity, which is how many times the model \n",
    "predicts that an existing customer will buy the new product over how many times \n",
    "they actually will buy the new product. On the other hand, the model also predicts \n",
    "the Specificity, which is how many times the it predicts that a customer won't \n",
    "adopt the new product over how many times she actually does not adopt the new product."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "192px",
    "width": "160px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "144px",
    "left": "807px",
    "right": "20px",
    "top": "119px",
    "width": "350px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}