credit-part3-modeling-4-imbalanced-cost-sensitive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08716952",
   "metadata": {
    "papermill": {
     "duration": 0.0431,
     "end_time": "2022-01-10T23:27:22.366306",
     "exception": false,
     "start_time": "2022-01-10T23:27:22.323206",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Cost-Sensitive-Learning\n",
    "## Misclassification cost as part of training\n",
    "\n",
    "\n",
    "There are 2 ways in which we can introduce cost into the learning function of the algorithm with Scikit-learn:\n",
    "\n",
    "Defining the class_weight parameter for those estimators that allow it, when we set the estimator\n",
    "Passing a sample_weight vector with the weights for every single observation, when we fit the estimator.\n",
    "With both the class_weight parameter or the sample_weight vector, we indicate that the loss function should be modified to accommodate the class imbalance and the cost attributed to each misclassification.\n",
    "\n",
    "**parameters**\n",
    "\n",
    "**class_weight:** can take 'balanced' as argument, in which case it will use the balance ratio as weight. Alternatively, it can take a dictionary with **{class: penalty}**, pairs. In this case, it penalizes mistakes in samples of class[i] with class_weight[i].\n",
    "\n",
    "So if **class_weight = {0:1, and 1:10}**, misclassification of observations of class 1 are penalized 10 times more than misclassification of observations of class 0.\n",
    "\n",
    "**sample_weight **is a vector of the same length as y, containing the weight or penalty for each individual observation. In principle, it is more flexible, because it allows us to set weights to the observations and not to the class as a whole. So in this case, for example we could set up higher penalties for fraudulent applications that are more costly (money-wise)than to those fraudulent applications that are of little money.\n",
    "\n",
    "**Important**\n",
    "\n",
    "If you use both class_weight and sample_weight, the final penalty will be the combination of the 2, so be very careful"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8d9ec4a2",
   "metadata": {
    "_cell_guid": "ddedf45c-cdd2-4013-973f-6ccf6c78347a",
    "_uuid": "20c9b28e-4c93-447d-8210-cad89f404354",
    "collapsed": false,
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:22.459322Z",
     "iopub.status.busy": "2022-01-10T23:27:22.457760Z",
     "iopub.status.idle": "2022-01-10T23:27:26.527474Z",
     "shell.execute_reply": "2022-01-10T23:27:26.526824Z",
     "shell.execute_reply.started": "2022-01-10T22:40:01.768090Z"
    },
    "jupyter": {
     "outputs_hidden": false
    },
    "papermill": {
     "duration": 4.11708,
     "end_time": "2022-01-10T23:27:26.527629",
     "exception": false,
     "start_time": "2022-01-10T23:27:22.410549",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type='text/css'>\n",
       ".datatable table.frame { margin-bottom: 0; }\n",
       ".datatable table.frame thead { border-bottom: none; }\n",
       ".datatable table.frame tr.coltypes td {  color: #FFFFFF;  line-height: 6px;  padding: 0 0.5em;}\n",
       ".datatable .bool    { background: #DDDD99; }\n",
       ".datatable .object  { background: #565656; }\n",
       ".datatable .int     { background: #5D9E5D; }\n",
       ".datatable .float   { background: #4040CC; }\n",
       ".datatable .str     { background: #CC4040; }\n",
       ".datatable .time    { background: #40CC40; }\n",
       ".datatable .row_index {  background: var(--jp-border-color3);  border-right: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  font-size: 9px;}\n",
       ".datatable .frame tbody td { text-align: left; }\n",
       ".datatable .frame tr.coltypes .row_index {  background: var(--jp-border-color0);}\n",
       ".datatable th:nth-child(2) { padding-left: 12px; }\n",
       ".datatable .hellipsis {  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .vellipsis {  background: var(--jp-layout-color0);  color: var(--jp-cell-editor-border-color);}\n",
       ".datatable .na {  color: var(--jp-cell-editor-border-color);  font-size: 80%;}\n",
       ".datatable .sp {  opacity: 0.25;}\n",
       ".datatable .footer { font-size: 9px; }\n",
       ".datatable .frame_dimensions {  background: var(--jp-border-color3);  border-top: 1px solid var(--jp-border-color0);  color: var(--jp-ui-font-color3);  display: inline-block;  opacity: 0.6;  padding: 1px 10px 1px 5px;}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Load the librarys\n",
    "import pandas as pd #To work with dataset\n",
    "import numpy as np #Math library\n",
    "import matplotlib.gridspec as gridspec\n",
    "import seaborn as sns #Graph library that use matplot in background\n",
    "import matplotlib.pyplot as plt #to plot some parameters in seaborn\n",
    "import warnings\n",
    "# Preparation  \n",
    "from sklearn.preprocessing import LabelEncoder, OrdinalEncoder\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import PowerTransformer, StandardScaler,Normalizer,RobustScaler,MaxAbsScaler,MinMaxScaler,QuantileTransformer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import KBinsDiscretizer\n",
    "# Import StandardScaler from scikit-learn\n",
    "from sklearn.experimental import enable_iterative_imputer\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.impute import KNNImputer,IterativeImputer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.compose import make_column_transformer,ColumnTransformer\n",
    "from sklearn.pipeline import make_pipeline, Pipeline,FeatureUnion\n",
    "from sklearn.manifold import TSNE\n",
    "# Import train_test_split()\n",
    "# Metrics\n",
    "from sklearn.metrics import roc_auc_score, average_precision_score,recall_score\n",
    "from sklearn.metrics import make_scorer,mean_absolute_error\n",
    "from sklearn.metrics import mean_squared_error,classification_report,f1_score\n",
    "from sklearn.metrics import roc_curve,confusion_matrix\n",
    "from yellowbrick.classifier import (\n",
    "    ClassificationReport,\n",
    "    DiscriminationThreshold,\n",
    ")\n",
    "from datetime import datetime, date\n",
    "from sklearn.linear_model import ElasticNet, Lasso,  BayesianRidge, LassoLarsIC\n",
    "from sklearn.linear_model import LinearRegression, RidgeCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#import tensorflow as tf \n",
    "#from tensorflow.keras import layers\n",
    "#from tensorflow.keras.callbacks import EarlyStopping\n",
    "#from tensorflow.keras.callbacks import LearningRateScheduler\n",
    "#import smogn\n",
    "from sklearn.experimental import enable_hist_gradient_boosting\n",
    "from sklearn.base import BaseEstimator, TransformerMixin, RegressorMixin, clone,ClassifierMixin\n",
    "from sklearn.kernel_ridge import KernelRidge\n",
    "from sklearn.ensemble import GradientBoostingRegressor,RandomForestRegressor\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "# For training random forest model\n",
    "import lightgbm as lgb\n",
    "from scipy import sparse\n",
    "from sklearn.neighbors import KNeighborsRegressor \n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.cluster import KMeans\n",
    "from yellowbrick.cluster import KElbowVisualizer\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import f_regression,f_classif,chi2\n",
    "from sklearn.feature_selection import mutual_info_regression\n",
    "from sklearn.feature_selection import SelectPercentile\n",
    "from sklearn.feature_selection import mutual_info_classif,VarianceThreshold\n",
    "\n",
    "from sklearn.model_selection import train_test_split,cross_validate\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "# Imbalanced data \n",
    "from imblearn.datasets import fetch_datasets\n",
    "# to correctly set up the cross-validation\n",
    "from imblearn.pipeline import make_pipeline\n",
    "from imblearn.under_sampling import (\n",
    "    RandomUnderSampler,\n",
    "    CondensedNearestNeighbour,\n",
    "    TomekLinks,\n",
    "    OneSidedSelection,\n",
    "    EditedNearestNeighbours,\n",
    "    RepeatedEditedNearestNeighbours,\n",
    "    AllKNN,\n",
    "    NeighbourhoodCleaningRule,\n",
    "    NearMiss,\n",
    "    InstanceHardnessThreshold\n",
    ")\n",
    "from imblearn.over_sampling import (\n",
    "    RandomOverSampler,\n",
    "    SMOTE,\n",
    "    ADASYN,\n",
    "    BorderlineSMOTE,\n",
    "    SVMSMOTE,\n",
    ")\n",
    "\n",
    "\n",
    "from lightgbm import LGBMClassifier\n",
    "import lightgbm as lgbm\n",
    "from catboost import CatBoostRegressor, CatBoostClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn import svm\n",
    "from xgboost import XGBClassifier,XGBRegressor\n",
    "from sklearn import set_config\n",
    "from itertools import combinations\n",
    "# Cluster :\n",
    "from sklearn.cluster import MiniBatchKMeans\n",
    "#from yellowbrick.cluster import KElbowVisualizer\n",
    "#import smong \n",
    "import category_encoders as ce\n",
    "import warnings\n",
    "#import optuna \n",
    "from joblib import Parallel, delayed\n",
    "import joblib \n",
    "from sklearn import set_config\n",
    "from typing import List, Optional, Union\n",
    "set_config(display='diagram')\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "160173b6",
   "metadata": {
    "_cell_guid": "38c6befb-892d-4ef3-9c63-13be5959b671",
    "_uuid": "2953f9f3-02f8-4fa9-a7fc-cb93565c2183",
    "papermill": {
     "duration": 0.044137,
     "end_time": "2022-01-10T23:27:26.617192",
     "exception": false,
     "start_time": "2022-01-10T23:27:26.573055",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "\n",
    "# Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bc8f6adf",
   "metadata": {
    "_cell_guid": "98a5b4a6-90e9-449d-96cc-0d85c9ff1c77",
    "_uuid": "e086592f-9f6a-4592-80b5-c1e7ad227e02",
    "collapsed": false,
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:26.713101Z",
     "iopub.status.busy": "2022-01-10T23:27:26.712502Z",
     "iopub.status.idle": "2022-01-10T23:27:27.111583Z",
     "shell.execute_reply": "2022-01-10T23:27:27.112003Z",
     "shell.execute_reply.started": "2022-01-10T22:40:05.698298Z"
    },
    "jupyter": {
     "outputs_hidden": false
    },
    "papermill": {
     "duration": 0.449087,
     "end_time": "2022-01-10T23:27:27.112151",
     "exception": false,
     "start_time": "2022-01-10T23:27:26.663064",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 199 ms, sys: 66.3 ms, total: 266 ms\n",
      "Wall time: 383 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>SeriousDlqin2yrs</th>\n",
       "      <th>RevolvingUtilizationOfUnsecuredLines</th>\n",
       "      <th>age</th>\n",
       "      <th>NumberOfTime30-59DaysPastDueNotWorse</th>\n",
       "      <th>DebtRatio</th>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <th>NumberOfOpenCreditLinesAndLoans</th>\n",
       "      <th>NumberOfTimes90DaysLate</th>\n",
       "      <th>NumberRealEstateLoansOrLines</th>\n",
       "      <th>NumberOfTime60-89DaysPastDueNotWorse</th>\n",
       "      <th>NumberOfDependents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.766127</td>\n",
       "      <td>45</td>\n",
       "      <td>2</td>\n",
       "      <td>0.802982</td>\n",
       "      <td>9120.0</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.957151</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "      <td>0.121876</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.658180</td>\n",
       "      <td>38</td>\n",
       "      <td>1</td>\n",
       "      <td>0.085113</td>\n",
       "      <td>3042.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  SeriousDlqin2yrs  RevolvingUtilizationOfUnsecuredLines  age  \\\n",
       "0           1                 1                              0.766127   45   \n",
       "1           2                 0                              0.957151   40   \n",
       "2           3                 0                              0.658180   38   \n",
       "\n",
       "   NumberOfTime30-59DaysPastDueNotWorse  DebtRatio  MonthlyIncome  \\\n",
       "0                                     2   0.802982         9120.0   \n",
       "1                                     0   0.121876         2600.0   \n",
       "2                                     1   0.085113         3042.0   \n",
       "\n",
       "   NumberOfOpenCreditLinesAndLoans  NumberOfTimes90DaysLate  \\\n",
       "0                               13                        0   \n",
       "1                                4                        0   \n",
       "2                                2                        1   \n",
       "\n",
       "   NumberRealEstateLoansOrLines  NumberOfTime60-89DaysPastDueNotWorse  \\\n",
       "0                             6                                     0   \n",
       "1                             0                                     0   \n",
       "2                             0                                     0   \n",
       "\n",
       "   NumberOfDependents  \n",
       "0                 2.0  \n",
       "1                 1.0  \n",
       "2                 0.0  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "train = pd.read_csv('../input/GiveMeSomeCredit/cs-training.csv')\n",
    "test = pd.read_csv('../input/GiveMeSomeCredit/cs-test.csv')\n",
    "train.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65acba55",
   "metadata": {
    "papermill": {
     "duration": 0.04303,
     "end_time": "2022-01-10T23:27:27.198825",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.155795",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1-Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815d0f3e",
   "metadata": {
    "papermill": {
     "duration": 0.042705,
     "end_time": "2022-01-10T23:27:27.284655",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.241950",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Convert Dtypes :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "49f98c7a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:27.398518Z",
     "iopub.status.busy": "2022-01-10T23:27:27.396902Z",
     "iopub.status.idle": "2022-01-10T23:27:27.450571Z",
     "shell.execute_reply": "2022-01-10T23:27:27.450013Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.220525Z"
    },
    "papermill": {
     "duration": 0.122608,
     "end_time": "2022-01-10T23:27:27.450731",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.328123",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Convert Dtypes :\n",
    "train[train.select_dtypes(['int64','int16','float16','float32','float64','int8']).columns] = train[train.select_dtypes(['int64','int16','float16','float32','float64','int8']).columns].apply(pd.to_numeric)\n",
    "train[train.select_dtypes(['object','category']).columns] = train.select_dtypes(['object','category']).apply(lambda x: x.astype('category'))\n",
    "# Convert Dtypes :\n",
    "test[test.select_dtypes(['int64','int16','float16','float32','float64','int8']).columns] = test[test.select_dtypes(['int64','int16','float16','float32','float64','int8']).columns].apply(pd.to_numeric)\n",
    "test[test.select_dtypes(['object','category']).columns] = test.select_dtypes(['object','category']).apply(lambda x: x.astype('category'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c0e3d8d",
   "metadata": {
    "papermill": {
     "duration": 0.04278,
     "end_time": "2022-01-10T23:27:27.538027",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.495247",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Imbalanced target\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "be3dff70",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:27.632198Z",
     "iopub.status.busy": "2022-01-10T23:27:27.631327Z",
     "iopub.status.idle": "2022-01-10T23:27:27.635070Z",
     "shell.execute_reply": "2022-01-10T23:27:27.634619Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.299690Z"
    },
    "papermill": {
     "duration": 0.054004,
     "end_time": "2022-01-10T23:27:27.635192",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.581188",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.93316\n",
       "1    0.06684\n",
       "Name: SeriousDlqin2yrs, dtype: float64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.SeriousDlqin2yrs.value_counts() / len(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e31c7a5c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:27.729278Z",
     "iopub.status.busy": "2022-01-10T23:27:27.728549Z",
     "iopub.status.idle": "2022-01-10T23:27:27.731276Z",
     "shell.execute_reply": "2022-01-10T23:27:27.731678Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.309184Z"
    },
    "papermill": {
     "duration": 0.052884,
     "end_time": "2022-01-10T23:27:27.731812",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.678928",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    139974\n",
       "1     10026\n",
       "Name: SeriousDlqin2yrs, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.SeriousDlqin2yrs.value_counts() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "60a39c3c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:27.824950Z",
     "iopub.status.busy": "2022-01-10T23:27:27.824113Z",
     "iopub.status.idle": "2022-01-10T23:27:28.024817Z",
     "shell.execute_reply": "2022-01-10T23:27:28.025317Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.322725Z"
    },
    "papermill": {
     "duration": 0.24991,
     "end_time": "2022-01-10T23:27:28.025487",
     "exception": false,
     "start_time": "2022-01-10T23:27:27.775577",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgcAAAFYCAYAAADHkV+EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAomUlEQVR4nO3dfXRV1Z3/8fclIUgFEmCRG51SWhV8QkilPmSgsQ3rJjxFQiB2ta5OA+NTZURgih10FREidiyOVKnWTHR8WFMrIIFKpkMgVENakBkUYxw7DmOjMDU3I0IAhYSE+/uDH/fXIwLRX5Ir6fu1lmtx9z13n+9mSc7n7rOzTygWi8WQJEn6v3okugBJkvT5YjiQJEkBhgNJkhRgOJAkSQGGA0mSFJCc6AI+D44ePcqHH35Iz549CYVCiS5HkqROFYvFOHLkCGeffTY9epw4T2A4AD788EPeeuutRJchSVKXGjZsGH379j2h3XAA9OzZEzj2l5SSkpLgaiRJ6lwtLS289dZb8evfxxkOIH4rISUlhV69eiW4GkmSusbJbqW7IFGSJAUYDiRJUoDhQJIkBRgOJElSgOFAkiQFGA4kSVKA4UCSJAV0WjiYP38+WVlZTJo06YT3nnjiCS688EI++OAD4Ng2jiUlJUQiEfLz83njjTfix5aXl5Obm0tubi7l5eXx9rq6OvLz84lEIpSUlBCLxQDYt28f06dPJzc3l+nTp9PU1NRZQ5QkqVvqtHBQWFhIWVnZCe3vvfcev/3tbzn33HPjbdXV1dTX11NZWcnixYtZuHAhcOxCv3z5clasWMHKlStZvnx5/GK/cOFCFi9eTGVlJfX19VRXVwNQWlpKVlYWlZWVZGVlUVpa2llDlCSpW+q0cHDFFVeQmpp6Qvt9993HvHnzArsyVVVVUVBQQCgUIjMzk/3799PY2EhNTQ2jR48mLS2N1NRURo8ezebNm2lsbOTgwYNkZmYSCoUoKCigqqoq0BdAQUEBGzdu7KwhSpLULXXpmoONGzeSnp7ORRddFGiPRqNkZGTEX2dkZBCNRk9oD4fDn9h+/HiAPXv2kJ6eDsCgQYPYs2dPZw5JkqRup8uerXDo0CEee+wxnnjiia46JaFQ6FM9grmurq7Da5jwRHWH9yklwr/MyE50CZK6SJeFg3fffZfdu3czefJkABoaGigsLGTlypWEw2EaGhrixzY0NBAOhwmHw2zbti3eHo1GufLKK096PMDAgQNpbGwkPT2dxsZGBgwY0O4ahw8f3vEPXjIcqJsYNWpUokuQ1EGam5tP+YW4y24rXHjhhWzZsoVNmzaxadMmMjIyWL16NYMGDSInJ4c1a9YQi8XYsWMHffv2JT09nTFjxlBTU0NTUxNNTU3U1NQwZswY0tPT6dOnDzt27CAWi7FmzRrGjh0LEO8LCLRLkqT26bSZg7lz57Jt2zb27t1LdnY2t912G0VFRZ947DXXXMNLL71EJBKhd+/eLFmyBIC0tDRuvfVWpk2bBsDMmTNJS0sD4O6772b+/PkcPnyY7OxssrOPTXnedNNNzJ49m1WrVnHuueeybNmyzhqiJEndUih2fIOAP2PHp1c647ZCeOaDHdqflCjRn81JdAmSOsjprnvukChJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKaDTwsH8+fPJyspi0qRJ8ba///u/Z9y4ceTn5zNz5kz2798ff++xxx4jEomQl5fH5s2b4+3V1dXk5eURiUQoLS2Nt+/atYuioiIikQizZ8+mpaUFgJaWFmbPnk0kEqGoqIjdu3d31hAlSeqWOi0cFBYWUlZWFmgbPXo069at44UXXuDLX/4yjz32GAA7d+6koqKCiooKysrKuOeee2hra6OtrY1FixZRVlZGRUUF69atY+fOnQAsXbqU4uJiNmzYQL9+/Vi1ahUAK1eupF+/fmzYsIHi4mKWLl3aWUOUJKlb6rRwcMUVV5CamhpoGzNmDMnJyQBkZmbS0NAAQFVVFRMnTiQlJYXBgwczZMgQamtrqa2tZciQIQwePJiUlBQmTpxIVVUVsViMrVu3kpeXB8CUKVOoqqoCYNOmTUyZMgWAvLw8tmzZQiwW66xhSpLU7SRszcHzzz9PdnY2ANFolIyMjPh74XCYaDR60va9e/fSr1+/eNDIyMggGo3G+zrnnHMASE5Opm/fvuzdu7erhiVJ0hkvOREnffTRR0lKSuLaa69NxOlPqq6uLtElSJ9b27dvT3QJkrpIl4eD1atX8+KLL/Lkk08SCoWAYzMCx28xwLFv/+FwGOAT2/v378/+/ftpbW0lOTmZhoaG+PHhcJj33nuPjIwMWltbOXDgAP37929XbcOHD6dXr14dNdRjnqju2P6kBBk1alSiS5DUQZqbm0/5hbhLbytUV1dTVlbGo48+Su/evePtOTk5VFRU0NLSwq5du6ivr2fEiBFcdtll1NfXs2vXLlpaWqioqCAnJ4dQKMRVV13F+vXrASgvLycnJyfeV3l5OQDr16/n6quvjocQSZJ0ep02czB37ly2bdvG3r17yc7O5rbbbqO0tJSWlhamT58OwMiRI1m0aBFDhw5l/PjxTJgwgaSkJBYsWEBSUhIACxYs4IYbbqCtrY2pU6cydOhQAObNm8ecOXNYtmwZF198MUVFRQBMmzaNefPmEYlESE1N5cEHH+ysIUqS1C2FYi7lj0+vdMZthfBMw4m6h+jP5iS6BEkd5HTXPXdIlCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIU0GnhYP78+WRlZTFp0qR42759+5g+fTq5ublMnz6dpqYmAGKxGCUlJUQiEfLz83njjTfinykvLyc3N5fc3FzKy8vj7XV1deTn5xOJRCgpKSEWi53yHJIkqX06LRwUFhZSVlYWaCstLSUrK4vKykqysrIoLS0FoLq6mvr6eiorK1m8eDELFy4Ejl3oly9fzooVK1i5ciXLly+PX+wXLlzI4sWLqayspL6+nurq6lOeQ5IktU+nhYMrrriC1NTUQFtVVRUFBQUAFBQUsHHjxkB7KBQiMzOT/fv309jYSE1NDaNHjyYtLY3U1FRGjx7N5s2baWxs5ODBg2RmZhIKhSgoKKCqquqU55AkSe2T3JUn27NnD+np6QAMGjSIPXv2ABCNRsnIyIgfl5GRQTQaPaE9HA5/Yvvx4091jvaoq6v77IOTurnt27cnugRJXaRLw8GfCoVChEKhz9U5hg8fTq9evTq2iCeqO7Y/KUFGjRqV6BIkdZDm5uZTfiHu0t9WGDhwII2NjQA0NjYyYMAA4NiMQENDQ/y4hoYGwuHwCe3RaPQT248ff6pzSJKk9unScJCTk8OaNWsAWLNmDWPHjg20x2IxduzYQd++fUlPT2fMmDHU1NTQ1NREU1MTNTU1jBkzhvT0dPr06cOOHTuIxWKf2NfHzyFJktqn024rzJ07l23btrF3716ys7O57bbbuOmmm5g9ezarVq3i3HPPZdmyZQBcc801vPTSS0QiEXr37s2SJUsASEtL49Zbb2XatGkAzJw5k7S0NADuvvtu5s+fz+HDh8nOziY7OxvgpOeQJEntE4od3yDgz9jxey+dseYgPPPBDu1PSpToz+YkugRJHeR01z13SJQkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQEJCQdPPvkkEydOZNKkScydO5fm5mZ27dpFUVERkUiE2bNn09LSAkBLSwuzZ88mEolQVFTE7t274/089thjRCIR8vLy2Lx5c7y9urqavLw8IpEIpaWlXT4+SZLOZF0eDqLRKE8//TTPP/8869ato62tjYqKCpYuXUpxcTEbNmygX79+rFq1CoCVK1fSr18/NmzYQHFxMUuXLgVg586dVFRUUFFRQVlZGffccw9tbW20tbWxaNEiysrKqKioYN26dezcubOrhylJ0hkrITMHbW1tHD58mNbWVg4fPsygQYPYunUreXl5AEyZMoWqqioANm3axJQpUwDIy8tjy5YtxGIxqqqqmDhxIikpKQwePJghQ4ZQW1tLbW0tQ4YMYfDgwaSkpDBx4sR4X5Ik6fS6PByEw2FmzJjBN7/5TcaMGUOfPn249NJL6devH8nJyQBkZGQQjUaBYzMN55xzDgDJycn07duXvXv3Eo1GycjICPQbjUZP2i5JktonuT0H3X777fz0pz89bVt7NDU1UVVVRVVVFX379uX2228PrBdIpLq6ukSXIH1ubd++PdElSOoi7QoH77777gltb7/99mc64e9+9zu++MUvMmDAAAByc3N55ZVX2L9/P62trSQnJ9PQ0EA4HAaOffN/7733yMjIoLW1lQMHDtC/f3/C4TANDQ3xfqPRaPwzJ2s/neHDh9OrV6/PNK6TeqK6Y/uTEmTUqFGJLkFSB2lubj7lF+JT3lZYsWIFU6dOpb6+nmnTpsX/y8vL4y/+4i8+U0Hnnnsur732GocOHSIWi7FlyxYuuOACrrrqKtavXw9AeXk5OTk5AOTk5FBeXg7A+vXrufrqqwmFQuTk5FBRUUFLSwu7du2ivr6eESNGcNlll1FfX8+uXbtoaWmhoqIi3pckSTq9U84cjB49miFDhrB48WLuuOOOeHufPn248MILP9MJR44cSV5eHlOmTCE5OZmLL76Yb33rW3zjG99gzpw5LFu2jIsvvpiioiIApk2bxrx584hEIqSmpvLggw8CMHToUMaPH8+ECRNISkpiwYIFJCUlAbBgwQJuuOEG2tramDp1KkOHDv1MtUqS9OcoFIvFYokuItGOT690xm2F8MwHO7Q/KVGiP5uT6BIkdZDTXffatebg7bff5tFHH2XXrl20trbG24/vRSBJkrqPdoWDuXPnMm7cOAoLC+NT95IkqXtqVzg4evQot9xyS2fXIkmSPgfatQlSZmYmv//97zu7FkmS9DnQrpmD2tpaVq9ezVe+8pXAwgXXHEiS1P20KxzceeednV2HJEn6nGhXOLjyyis7uw5JkvQ50a5wMHXqVEKh0Ant3laQJKn7aVc4+OEPfxj/c3NzMxUVFaSnp3daUZIkKXE+022FMWPG8O1vf7tTCpIkSYnVrl9l/LiDBw/y/vvvd3QtkiTpc+BTrzk4evQou3fvZvr06Z1amCRJSoxPveYgKSmJwYMHu+ZAkqRuqt1rDlpbW/nDH/4AwIABAzq1KEmSlDjtCgevv/46s2bNIiUlhVgsRmtrKw8//DCXXnppZ9cnSZK6WLvCwb333suSJUvIysoCYMuWLSxevJhf/vKXnVqcJEnqeu36bYVDhw7FgwFAVlYWhw4d6rSiJElS4rQrHPTu3ZuXX345/nrbtm307t2704qSJEmJ067bCnfddVd8zQHAkSNHeOihhzq1MEmSlBjtCgcHDhxg1apV7NmzB4CBAwfy1ltvdWphkiQpMdp1W+H+++9nwIABDBs2jGHDhtG/f3/uv//+zq5NkiQlQLvCQSwWCzyVsUePHrS1tXVaUZIkKXHaFQ7OPvtsXnvttfjr1157jS984QudVpQkSUqcdq05mDdvHjNnzuSCCy4AYOfOnSxfvrxTC5MkSYnRrnDw1a9+lYqKCnbs2AFAZmYmqampnVmXJElKkHaFA4DU1FSuueaazqxFkiR9DrRrzYEkSfrzYTiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAQkJB/v372fWrFmMGzeO8ePH8+qrr7Jv3z6mT59Obm4u06dPp6mpCTi2dXNJSQmRSIT8/HzeeOONeD/l5eXk5uaSm5tLeXl5vL2uro78/HwikQglJSXEYrEuH6MkSWeqhISDe++9l69//ev867/+K2vXruX888+ntLSUrKwsKisrycrKorS0FIDq6mrq6+uprKxk8eLFLFy4EIB9+/axfPlyVqxYwcqVK1m+fHk8UCxcuJDFixdTWVlJfX091dXViRimJElnpC4PBwcOHODf/u3fmDZtGgApKSn069ePqqoqCgoKACgoKGDjxo0A8fZQKERmZib79++nsbGRmpoaRo8eTVpaGqmpqYwePZrNmzfT2NjIwYMHyczMJBQKUVBQQFVVVVcPU5KkM1a7d0jsKLt372bAgAHMnz+f3//+91x66aXcdddd7Nmzh/T0dAAGDRrEnj17AIhGo2RkZMQ/n5GRQTQaPaE9HA5/Yvvx4yVJUvt0eThobW3lP/7jP/jRj37EyJEjKSkpid9COC4UCgUeEd1V6urquvyc0pli+/btiS5BUhfp8nCQkZFBRkYGI0eOBGDcuHGUlpYycOBAGhsbSU9Pp7GxkQEDBgDHZgQaGhrin29oaCAcDhMOh9m2bVu8PRqNcuWVV570+PYYPnw4vXr16ohh/j9PuN5B3cOoUaMSXYKkDtLc3HzKL8RdvuZg0KBBZGRk8PbbbwOwZcsWzj//fHJyclizZg0Aa9asYezYsQDx9lgsxo4dO+jbty/p6emMGTOGmpoampqaaGpqoqamhjFjxpCenk6fPn3YsWMHsVgs0JckSTq9Lp85APjRj37ED37wA44cOcLgwYO57777OHr0KLNnz2bVqlWce+65LFu2DIBrrrmGl156iUgkQu/evVmyZAkAaWlp3HrrrfGFjTNnziQtLQ2Au+++m/nz53P48GGys7PJzs5OxDAlSTojhWJuAhCfXumM2wrhmQ92aH9SokR/NifRJUjqIKe77rlDoiRJCjAcSJKkAMOBJEkKMBxIkqQAw4EkSQowHEiSpADDgSRJCjAcSJKkAMOBJEkKMBxIkqQAw4EkSQowHEiSpADDgSRJCjAcSJKkAMOBJEkKMBxIkqQAw4EkSQowHEiSpADDgSRJCjAcSJKkAMOBJEkKMBxIkqQAw4EkSQowHEiSpADDgSRJCjAcSJKkAMOBJEkKMBxIkqQAw4EkSQowHEiSpADDgSRJCjAcSJKkgISFg7a2NgoKCrj55psB2LVrF0VFRUQiEWbPnk1LSwsALS0tzJ49m0gkQlFREbt374738dhjjxGJRMjLy2Pz5s3x9urqavLy8ohEIpSWlnbtwCRJOsMlLBw8/fTTnH/++fHXS5cupbi4mA0bNtCvXz9WrVoFwMqVK+nXrx8bNmyguLiYpUuXArBz504qKiqoqKigrKyMe+65h7a2Ntra2li0aBFlZWVUVFSwbt06du7cmZAxSpJ0JkpIOGhoaODFF19k2rRpAMRiMbZu3UpeXh4AU6ZMoaqqCoBNmzYxZcoUAPLy8tiyZQuxWIyqqiomTpxISkoKgwcPZsiQIdTW1lJbW8uQIUMYPHgwKSkpTJw4Md6XJEk6veREnHTJkiXMmzePDz/8EIC9e/fSr18/kpOPlZORkUE0GgUgGo1yzjnnHCs2OZm+ffuyd+9eotEoI0eOjPcZDofjn8nIyAi019bWtquuurq6///BSd3U9u3bE12CpC7S5eHgN7/5DQMGDGD48OG8/PLLXX36Uxo+fDi9evXq2E6fqO7Y/qQEGTVqVKJLkNRBmpubT/mFuMvDwSuvvMKmTZuorq6mubmZgwcPcu+997J//35aW1tJTk6moaGBcDgMHPvm/95775GRkUFraysHDhygf//+hMNhGhoa4v1Go9H4Z07WLkmSTq/L1xz87d/+LdXV1WzatIl/+Id/4Oqrr+aBBx7gqquuYv369QCUl5eTk5MDQE5ODuXl5QCsX7+eq6++mlAoRE5ODhUVFbS0tLBr1y7q6+sZMWIEl112GfX19ezatYuWlhYqKirifUmSpNNLyJqDTzJv3jzmzJnDsmXLuPjiiykqKgJg2rRpzJs3j0gkQmpqKg8++CAAQ4cOZfz48UyYMIGkpCQWLFhAUlISAAsWLOCGG26gra2NqVOnMnTo0ISNS5KkM00oFovFEl1Eoh2/99IZaw7CMx/s0P6kRIn+bE6iS5DUQU533XOHREmSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhRgOJAkSQGGA0mSFNDl4eC9997ju9/9LhMmTGDixIk89dRTAOzbt4/p06eTm5vL9OnTaWpqAiAWi1FSUkIkEiE/P5833ngj3ld5eTm5ubnk5uZSXl4eb6+rqyM/P59IJEJJSQmxWKxrBylJ0hmsy8NBUlISf/d3f8e//Mu/8Nxzz/GLX/yCnTt3UlpaSlZWFpWVlWRlZVFaWgpAdXU19fX1VFZWsnjxYhYuXAgcCxPLly9nxYoVrFy5kuXLl8cDxcKFC1m8eDGVlZXU19dTXV3d1cOUJOmM1eXhID09nUsvvRSAPn36cN555xGNRqmqqqKgoACAgoICNm7cCBBvD4VCZGZmsn//fhobG6mpqWH06NGkpaWRmprK6NGj2bx5M42NjRw8eJDMzExCoRAFBQVUVVV19TAlSTpjJXTNwe7du3nzzTcZOXIke/bsIT09HYBBgwaxZ88eAKLRKBkZGfHPZGRkEI1GT2gPh8Of2H78eEmS1D7JiTrxhx9+yKxZs7jzzjvp06dP4L1QKEQoFOrymurq6rr8nNKZYvv27YkuQVIXSUg4OHLkCLNmzSI/P5/c3FwABg4cSGNjI+np6TQ2NjJgwADg2IxAQ0ND/LMNDQ2Ew2HC4TDbtm2Lt0ejUa688sqTHt8ew4cPp1evXh0xxP/nCdc7qHsYNWpUokuQ1EGam5tP+YW4y28rxGIx7rrrLs477zymT58eb8/JyWHNmjUArFmzhrFjxwbaY7EYO3bsoG/fvqSnpzNmzBhqampoamqiqamJmpoaxowZQ3p6On369GHHjh3EYrFAX5Ik6fS6fOZg+/btrF27lmHDhjF58mQA5s6dy0033cTs2bNZtWoV5557LsuWLQPgmmuu4aWXXiISidC7d2+WLFkCQFpaGrfeeivTpk0DYObMmaSlpQFw9913M3/+fA4fPkx2djbZ2dldPUxJks5YoZibAMSnVzrjtkJ45oMd2p+UKNGfzUl0CZI6yOmue+6QKEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICkhNdgCR1hic3LEp0CVKHKI4s6PJzOnMgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKcBwIEmSAgwHkiQpwHAgSZICDAeSJCnAcCBJkgIMB5IkKaDbhoPq6mry8vKIRCKUlpYmuhxJks4Y3TIctLW1sWjRIsrKyqioqGDdunXs3Lkz0WVJknRG6JbhoLa2liFDhjB48GBSUlKYOHEiVVVViS5LkqQzQnKiC+gM0WiUjIyM+OtwOExtbe1Jj4/FYgC0tLR0eC3pfXp1eJ9SIjQ3Nye6hE+lZ4/eiS5B6hCd8W/v+PXu+PXv47plOPi0jhw5AsBbb73V4X0/ed1VHd6nlAh1dXWJLuFTuSgtJ9ElSB2iM//tHTlyhLPOOuuE9m4ZDsLhMA0NDfHX0WiUcDh80uPPPvtshg0bRs+ePQmFQl1RoiRJCROLxThy5Ahnn332J77fLcPBZZddRn19Pbt27SIcDlNRUcEDDzxw0uN79OhB3759u7BCSZIS65NmDI7rluEgOTmZBQsWcMMNN9DW1sbUqVMZOnRoosuSJOmMEIqdbDWCJEn6s9Qtf5VRkiR9doYDSZIUYDjQGeHiiy9m8uTJTJo0iVmzZnHo0KFElySd0S688EJ+/OMfx18//vjjPPzww6f8zMaNG0+62+zDDz/M17/+9fi/UzeeO7MZDnRGOOuss1i7di3r1q2jZ8+e/PKXv0x0SdIZLSUlhcrKSj744IN2f+ZU4QCguLiYtWvX8tOf/pQ777yTo0ePdkSpSgDDgc44X/va13jnnXfYtGkTRUVFFBQUUFxczPvvv8/Ro0fJzc2N/8A7evQokUjkU/0AlP4cJCcn861vfYunnnrqhPd2797NX/3VX5Gfn8/3vvc9/vjHP/LKK6+wadMm7r//fiZPnsy777570r7PP/98kpOT2bt3L7feeiuFhYVMnDiR5557DoBVq1Zx7733xo9fsWIFS5Ys6fhB6jMzHOiM0traSnV1NcOGDWPUqFGsWLGCNWvWMHHiRMrKyujRowfXXnstv/rVrwD43e9+x0UXXcSAAQMSXLn0+XP99dfzwgsvcODAgUB7SUkJU6ZM4YUXXiA/P5+SkhIuv/xycnJyuOOOO1i7di1f+tKXTtrva6+9RigUYsCAASxZsoTVq1fz/PPP88wzz7B3717Gjx/Pb37zm/jutKtXr2bq1KmdOlZ9Ot1ynwN1P4cPH2by5MnAsZmDadOm8Yc//IE5c+bwv//7v7S0tPDFL34RgKlTp3LrrbdSXFzM888/T2FhYSJLlz63+vTpw+TJk3n66acDG+K8+uqr8fUHkydP5ic/+Um7+nvyySf51a9+xdlnn82yZcsIhUI888wzbNiwAYD33nuPd955h8zMTK6++mpefPFFzjvvPI4cOcKFF17Y8QPUZ2Y40Bnh+JqDP1VSUkJxcTFjx47l5ZdfZvny5QCcc845DBw4kC1btlBbW8vSpUsTUbJ0Rvje975HYWFhh4To4uJi/vqv/zr++uWXX+Z3v/sdzz33HL179+a73/1u/CFCRUVF/PznP+e8884zwH8OeVtBZ6wDBw7En5mxZs2awHtFRUXMmzePcePGkZSUlIDqpDNDWloa48aNY9WqVfG2r371q1RUVADwwgsv8LWvfQ049hyaDz/8sN19HzhwgNTUVHr37s1///d/s2PHjvh7I0eOpKGhgXXr1jFp0qSOGYw6jOFAZ6y/+Zu/4fbbb6ewsJC0tLTAezk5OXz00Ud+I5HaYcaMGezduzf++kc/+hGrV68mPz+ftWvXctdddwEwYcIEHn/8cQoKCk65IPG47OxsWltbGT9+PA888ACZmZmB98ePH8/ll19Oampqh45H///cPlnd0uuvv859993HL37xi0SXIukkbr75ZoqLi8nKykp0KfoYZw7U7ZSWljJr1izmzp2b6FIkfYL9+/eTl5dHr169DAafU84cSJKkAGcOJElSgOFAkiQFGA4kSVKAmyBJ3civf/1rHnvsMWKxGM3NzVx66aU88MADn6qPZ599lubmZoqLizu8vpycHFJSUkhJSeHQoUNccMEF3HjjjVx++eXAsSf7ffTRR/zwhz88ZT9VVVX8+7//+2mPe+WVV7j//vvZv38/ANdccw133HEHoVCoYwYkdVOGA6mbaGxs5J577qG8vJxzzjmHWCzGm2+++an6aG1t5dvf/nYnVXjMQw89xLBhwwCorKzkpptu4vHHH2fkyJHt7mPs2LGMHTv2tMf16dOHH//4x3z5y1+mpaWF733ve6xdu5aCgoJ2n6u1tZXkZH9U6s+L/8dL3cT7779PcnJyfEOoUCjEJZdcAhx7EM7SpUvju9vNmjWLb3zjG+zevZupU6dSWFjI1q1bue6663j//ffj397b2tpYunQpmzdvBuDrX/86P/jBD0hKSuK73/0uM2bM4Jvf/CZA4PXy5ctZt24dvXr1IhQK8fTTT9OvX78Tas7NzaW2tpbHH3+chx56KPBeS0sLJSUlbN26lf79+3PxxRfzwQcf8NBDD7F69WpefPFFHnroIV5++WWWLFnCyJEjefXVVwmFQjz44IOcf/758RACxx5RfMkll/DHP/4RgEmTJrFkyRJGjBgBwD/90z/x9ttvs3jxYnJycpgwYQJbt25l2LBh3HDDDcyfP59Dhw5x9OhRpkyZEtgmWOpuDAdSN3HRRRcxYsQIvvGNb3DVVVdx+eWXM3nyZJKSkrj77rspLS0lPT2dxsZGpk2bxrp16wDYt28fl112WXyK/vgDdwCee+453nzzTVavXg3AjTfeyHPPPcd3vvOdk9axb98+nnzySWpqajjrrLM4ePBg4KE+Hzdy5Eg2bdp0Qvtzzz3H7t27qaiooLW1leuvvz7+cK2P27lzJ/fddx+LFi3i0Ucf5ZFHHjnhdsqePXtYv349paWlwLEnEj777LOMGDGCWCzGs88+GwgoBw8ejG8pXFJSQk5ODjfffDMATU1NJx2P1B24IFHqJnr06MEjjzzCM888w1VXXcVLL73Etddey0svvcTu3bu58cYbmTx5MjfeeCOhUIh33nkHgF69ejF+/PhP7HPLli1MmTIlvk6gsLCQLVu2nLKOvn378qUvfYk77riDFStW8NFHH51yWv5kW628/PLLFBQU0LNnT3r37s2111570j6+8pWvxGdJMjMz2bVrV+D9gwcP8v3vf58ZM2bEj5s8eTKbN29m3759bN68mYEDB3LRRRfFP/Ontx6uuOIKVq5cybJly9iyZcsnzoJI3YkzB1I3M2zYMIYNG8b111/PhAkTiMViXHjhhfzzP//zCcfu3r2b3r17f6YFeklJSRw9ejT++vjT9pKSklixYgWvvPIKW7dupbCwkLKyssCF90+9/vrrDB069FOf/0+lpKTE/9yjRw9aW1vjrw8dOsQtt9zC6NGjmTFjRrz9C1/4Avn5+axevZpt27Zx/fXXB/r8whe+EP9zXl4emZmZ/Pa3v+Uf//Efef75533ap7o1Zw6kbiIajfLqq6/GXzc0NPDBBx9wwQUX8M4777B169b4e7W1tSf9xv6nsrKyWLNmDUeOHOHIkSOsWbOGv/zLvwTgS1/6Eq+//jpwbFr/+OLHgwcP8sEHH3DllVcya9Yshg0bxn/91399Yv8bN27k2WefDVy0j7v66qtZu3Ytra2tHD58OH4b5NNobm7mlltuYeTIkdx+++0nvP+d73yHp556irq6OnJzc0/azzvvvMOgQYMoLCxk5syZ8XFL3ZUzB1I30draysMPP8z//M//cNZZZ3H06FFmz57NJZdcwiOPPMJPfvITlixZwpEjRxg8eDA///nPT9vnt771Ld59912mTJkCwJgxY7juuuuAY+sPbr/9dqqqqrjkkkvi0/UHDx7ktttu4/Dhw8RiMS655JLAhXfWrFnxX2U8//zzKS0t/cTfVLjuuuv4z//8TyZMmED//v257LLL2LNnz6f6O1m1ahXbtm1j37591NTUADBu3Di+//3vAzB48GDOO+88RowYEZh9+Lhf//rXvPDCC/Ts2ZNQKMSdd975qeqQzjQ+W0HSGeFPf0Ohoxw8eJBx48bx/PPPEw6HO6xf6UznbQVJf5aeffZZJkyYwIwZMwwG0sc4cyBJkgKcOZAkSQGGA0mSFGA4kCRJAYYDSZIUYDiQJEkBhgNJkhTwfwA2IBwPwhWMrgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x396 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize the target variable\n",
    "g = sns.countplot(train['SeriousDlqin2yrs'])\n",
    "g.set_xticklabels(['Pay','Not Pay'])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8acd57e",
   "metadata": {
    "papermill": {
     "duration": 0.048334,
     "end_time": "2022-01-10T23:27:28.122473",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.074139",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Reduce Memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6c5b4ea3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:28.237732Z",
     "iopub.status.busy": "2022-01-10T23:27:28.236048Z",
     "iopub.status.idle": "2022-01-10T23:27:28.277573Z",
     "shell.execute_reply": "2022-01-10T23:27:28.278015Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.648820Z"
    },
    "papermill": {
     "duration": 0.105902,
     "end_time": "2022-01-10T23:27:28.278170",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.172268",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Memory usage of dataframe is 13.73 MB\n",
      "Memory usage after optimization is: 3.29 MB\n",
      "Decreased by 76.0%\n",
      "Memory usage of dataframe is 9.29 MB\n",
      "Memory usage after optimization is: 2.90 MB\n",
      "Decreased by 68.7%\n"
     ]
    }
   ],
   "source": [
    "# Author : https://www.kaggle.com/gemartin/load-data-reduce-memory-usage\n",
    "def reduce_mem_usage(df):\n",
    "    \"\"\" iterate through all the columns of a dataframe and modify the data type\n",
    "        to reduce memory usage.        \n",
    "    \"\"\"\n",
    "    start_mem = df.memory_usage().sum() / 1024**2\n",
    "    print('Memory usage of dataframe is {:.2f} MB'.format(start_mem))\n",
    "    \n",
    "    for col in df.columns:\n",
    "        col_type = df[col].dtype\n",
    "        name =df[col].dtype.name \n",
    "        \n",
    "        if col_type != object and col_type.name != 'category':\n",
    "        #if name != \"category\":    \n",
    "            c_min = df[col].min()\n",
    "            c_max = df[col].max()\n",
    "            if str(col_type)[:3] == 'int':\n",
    "                if c_min > np.iinfo(np.int8).min and c_max < np.iinfo(np.int8).max:\n",
    "                    df[col] = df[col].astype(np.int8)\n",
    "                elif c_min > np.iinfo(np.int16).min and c_max < np.iinfo(np.int16).max:\n",
    "                    df[col] = df[col].astype(np.int16)\n",
    "                elif c_min > np.iinfo(np.int32).min and c_max < np.iinfo(np.int32).max:\n",
    "                    df[col] = df[col].astype(np.int32)\n",
    "                elif c_min > np.iinfo(np.int64).min and c_max < np.iinfo(np.int64).max:\n",
    "                    df[col] = df[col].astype(np.int64)  \n",
    "            else:\n",
    "                if c_min > np.finfo(np.float16).min and c_max < np.finfo(np.float16).max:\n",
    "                    df[col] = df[col].astype(np.float16)\n",
    "                elif c_min > np.finfo(np.float32).min and c_max < np.finfo(np.float32).max:\n",
    "                    df[col] = df[col].astype(np.float32)\n",
    "                else:\n",
    "                    df[col] = df[col].astype(np.float64)\n",
    "        else:\n",
    "            df[col] = df[col].astype('category')\n",
    "\n",
    "    end_mem = df.memory_usage().sum() / 1024**2\n",
    "    print('Memory usage after optimization is: {:.2f} MB'.format(end_mem))\n",
    "    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem) / start_mem))\n",
    "    \n",
    "    return df\n",
    "train= reduce_mem_usage(train)\n",
    "test= reduce_mem_usage(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b588ce9",
   "metadata": {
    "papermill": {
     "duration": 0.044466,
     "end_time": "2022-01-10T23:27:28.367344",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.322878",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## X and Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8ec1c7d1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:28.460556Z",
     "iopub.status.busy": "2022-01-10T23:27:28.459731Z",
     "iopub.status.idle": "2022-01-10T23:27:28.465472Z",
     "shell.execute_reply": "2022-01-10T23:27:28.465058Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.709624Z"
    },
    "papermill": {
     "duration": 0.053445,
     "end_time": "2022-01-10T23:27:28.465585",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.412140",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cardinality : \n",
    "# - RevolvingUtilizationOfUnsecuredLines :125728, high Outlier\n",
    "# - DebtRatio :114194 , high Outlier \n",
    "# deal with outlier + bin \n",
    "PERCENTAGE = [\"RevolvingUtilizationOfUnsecuredLines\", \"DebtRatio\"]\n",
    "# MonthlyIncome:13594 , high outlier +bin \n",
    "REAL= [\"MonthlyIncome\"]\n",
    "# Can be considred as cat \n",
    "NUMERIC_DISCRET_low = [\"NumberOfDependents\",\n",
    "                       \"NumberOfTime60-89DaysPastDueNotWorse\",\n",
    "                       \"NumberRealEstateLoansOrLines\",\n",
    "                       \"NumberOfTimes90DaysLate\",\n",
    "                       \"NumberOfOpenCreditLinesAndLoans\",\n",
    "                       \"NumberOfTime30-59DaysPastDueNotWorse\",\n",
    "                       \"age\"]\n",
    "Late_Pay_Cols = ['NumberOfTime30-59DaysPastDueNotWorse',\n",
    "                 'NumberOfTimes90DaysLate',\n",
    "                 'NumberOfTime60-89DaysPastDueNotWorse']\n",
    "TARGET = [\"SeriousDlqin2yrs\"]\n",
    "\n",
    "#also change the type for TARGET to categorical\n",
    "#df[TARGET] = df[TARGET].astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0104e2e3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:28.559749Z",
     "iopub.status.busy": "2022-01-10T23:27:28.558953Z",
     "iopub.status.idle": "2022-01-10T23:27:28.563078Z",
     "shell.execute_reply": "2022-01-10T23:27:28.562618Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.718272Z"
    },
    "papermill": {
     "duration": 0.052252,
     "end_time": "2022-01-10T23:27:28.563195",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.510943",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "y = train['SeriousDlqin2yrs']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06078db",
   "metadata": {
    "papermill": {
     "duration": 0.047239,
     "end_time": "2022-01-10T23:27:28.655243",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.608004",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Imput before Descritaziation : \n",
    "\n",
    "### IterativeImputer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1091c60e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:28.757694Z",
     "iopub.status.busy": "2022-01-10T23:27:28.756039Z",
     "iopub.status.idle": "2022-01-10T23:27:28.760112Z",
     "shell.execute_reply": "2022-01-10T23:27:28.759601Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.730807Z"
    },
    "papermill": {
     "duration": 0.059444,
     "end_time": "2022-01-10T23:27:28.760237",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.700793",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 974 µs, sys: 0 ns, total: 974 µs\n",
      "Wall time: 807 µs\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "param =  {   \"verbosity\": 0,\n",
    "            #\"objective\": \"binary:logistic\",\n",
    "            #\"eval_metric\": \"auc\",\n",
    "            'random_state': 42,\n",
    "            # regression\n",
    "            'objective':'reg:squarederror', \n",
    "             'eval_metric': 'mae',\n",
    "            #early_stopping_rounds=100 ,\n",
    "            'gpu_id':0, \n",
    "            'predictor':\"gpu_predictor\",\n",
    "            # use exact for small dataset.\n",
    "            #\"tree_method\": \"exact\",\n",
    "            # big data :\n",
    "             'tree_method': 'gpu_hist',\n",
    "            # defines booster, gblinear for linear functions.\n",
    "             'booster': 'gbtree', \n",
    "            'lambda': 8.544792472633987e-07,\n",
    "            'alpha': 0.31141671752487043,\n",
    "            'subsample': 0.8779467596981366, \n",
    "            'colsample_bytree': 0.9759532762677546,\n",
    "            'learning_rate': 0.008686087328805853, \n",
    "            'n_estimators': 6988,\n",
    "            'max_depth': 9,\n",
    "            'min_child_weight': 2, \n",
    "            'eta': 3.7603213457541647e-06,\n",
    "            'gamma': 2.1478058456847449e-07,\n",
    "            'grow_policy': 'lossguide'}\n",
    "                \n",
    "\n",
    "#model_xgb = XGBRegressor(\n",
    "       #objective=\"mae\",\n",
    " #   **xgb_params2)\n",
    "\n",
    "numeric_transformer1 = Pipeline(\n",
    "                            steps=[\n",
    "                            ('imputer', SimpleImputer(strategy='median'\n",
    "                                                      ,add_indicator=True)),\n",
    "                            ('scaler', PowerTransformer()),#(Numerical Input, Numerical Output)\n",
    "                            # Create an SelectKBest object to select features with two best ANOVA F-Values\n",
    "                            #The F-value scores examine if, when we group the numerical feature by the target vector, the means for each group are significantly different\n",
    "                           # ('reducedim',  SelectPercentile(f_classif,percentile=90))\n",
    "                            ]\n",
    "                            )\n",
    "numeric_transformer2 = Pipeline(\n",
    "                            steps=[\n",
    "                            #('imputer', SimpleImputer(strategy='median'\n",
    "                             #                         ,add_indicator=True)),\n",
    "                            ('scaler', PowerTransformer()),#(Numerical Input, Numerical Output)\n",
    "                            # Create an SelectKBest object to select features with two best ANOVA F-Values\n",
    "                            #The F-value scores examine if, when we group the numerical feature by the target vector, the means for each group are significantly different\n",
    "                           # ('reducedim',  SelectPercentile(f_classif,percentile=90))\n",
    "                            ]\n",
    "                            )\n",
    "\n",
    "pipe_xgbr1 = Pipeline(\n",
    "                    steps=[\n",
    "                        ('preprocessor', numeric_transformer1),\n",
    "                        ('classifier', XGBRegressor(\n",
    "                      #objective=\"mae\",\n",
    "                       **param))\n",
    "                    ]\n",
    "                )\n",
    "pipe_xgbr2 = Pipeline(\n",
    "                    steps=[\n",
    "                        ('preprocessor', numeric_transformer2),\n",
    "                        ('classifier', XGBRegressor(\n",
    "                      #objective=\"mae\",\n",
    "                       **param))\n",
    "                    ]\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a51dfa2b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:27:28.862579Z",
     "iopub.status.busy": "2022-01-10T23:27:28.861665Z",
     "iopub.status.idle": "2022-01-10T23:29:50.570556Z",
     "shell.execute_reply": "2022-01-10T23:29:50.571185Z",
     "shell.execute_reply.started": "2022-01-10T22:40:06.743655Z"
    },
    "papermill": {
     "duration": 141.764057,
     "end_time": "2022-01-10T23:29:50.571392",
     "exception": false,
     "start_time": "2022-01-10T23:27:28.807335",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 17s, sys: 2.56 s, total: 2min 20s\n",
      "Wall time: 2min 21s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "train=train.drop(['Unnamed: 0','SeriousDlqin2yrs'], axis=1)\n",
    "test=test.drop(['Unnamed: 0','SeriousDlqin2yrs'], axis=1)\n",
    "train_final= pd.concat( [train, test], axis=0) \n",
    "#testdf_income= train_final[train_final['MonthlyIncome'].isnull()==True]\n",
    "traindf_income = train_final[train_final['MonthlyIncome'].isnull()==False]\n",
    "y_income = traindf_income['MonthlyIncome']\n",
    "X_income=traindf_income.drop([\"MonthlyIncome\"],axis=1)\n",
    "pipe_xgbr1.fit(X_income, y_income)\n",
    "train_income_missing=train[train['MonthlyIncome'].isnull()==True].drop([\"MonthlyIncome\"],axis=1)\n",
    "test_income_missing=test[test['MonthlyIncome'].isnull()==True].drop([\"MonthlyIncome\"],axis=1)\n",
    "train_predicted = pipe_xgbr1.predict(train_income_missing)\n",
    "test_predicted = pipe_xgbr1.predict(test_income_missing)\n",
    "train.loc[(train.MonthlyIncome.isnull()), 'MonthlyIncome'] = train_predicted\n",
    "test.loc[(test.MonthlyIncome.isnull()), 'MonthlyIncome'] = test_predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d0ab64a7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:29:50.669615Z",
     "iopub.status.busy": "2022-01-10T23:29:50.668694Z",
     "iopub.status.idle": "2022-01-10T23:32:37.664206Z",
     "shell.execute_reply": "2022-01-10T23:32:37.664655Z",
     "shell.execute_reply.started": "2022-01-10T22:42:19.168758Z"
    },
    "papermill": {
     "duration": 167.047502,
     "end_time": "2022-01-10T23:32:37.664809",
     "exception": false,
     "start_time": "2022-01-10T23:29:50.617307",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 45s, sys: 970 ms, total: 2min 46s\n",
      "Wall time: 2min 46s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "traindf_NumberOfDependents = train_final[train_final['NumberOfDependents'].isnull()==False]\n",
    "y_NumberOfDependents = traindf_NumberOfDependents['NumberOfDependents']\n",
    "X_NumberOfDependents=traindf_NumberOfDependents.drop([\"NumberOfDependents\"],axis=1)\n",
    "pipe_xgbr2.fit(X_NumberOfDependents, y_NumberOfDependents)\n",
    "train_NumberOfDependents_missing=train[train['NumberOfDependents'].isnull()==True].drop([\"NumberOfDependents\"],axis=1)\n",
    "test_NumberOfDependents_missing=test[test['NumberOfDependents'].isnull()==True].drop([\"NumberOfDependents\"],axis=1)\n",
    "train_predicted = pipe_xgbr2.predict(train_NumberOfDependents_missing)\n",
    "test_predicted = pipe_xgbr2.predict(test_NumberOfDependents_missing)\n",
    "train.loc[(train.NumberOfDependents.isnull()), 'NumberOfDependents'] = train_predicted\n",
    "test.loc[(test.NumberOfDependents.isnull()), 'NumberOfDependents'] = test_predicted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f82ec5a0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:37.771939Z",
     "iopub.status.busy": "2022-01-10T23:32:37.771038Z",
     "iopub.status.idle": "2022-01-10T23:32:38.340435Z",
     "shell.execute_reply": "2022-01-10T23:32:38.340905Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.118748Z"
    },
    "papermill": {
     "duration": 0.630741,
     "end_time": "2022-01-10T23:32:38.341082",
     "exception": false,
     "start_time": "2022-01-10T23:32:37.710341",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Add bin data \n",
    "# initializing append_str\n",
    "append_str = 'cat_'\n",
    "# Append suffix / prefix to strings in list\n",
    "num_features1=[\"RevolvingUtilizationOfUnsecuredLines\", \"DebtRatio\",\"MonthlyIncome\"]\n",
    "num_features2=[\"NumberOfDependents\",\n",
    "                       \"NumberOfTime60-89DaysPastDueNotWorse\",\n",
    "                       \"NumberRealEstateLoansOrLines\",\n",
    "                       \"NumberOfTimes90DaysLate\",\n",
    "                       \"NumberOfOpenCreditLinesAndLoans\",\n",
    "                       \"NumberOfTime30-59DaysPastDueNotWorse\",\n",
    "                       \"age\"]\n",
    "cat_features1 = [append_str + sub for sub in num_features1]\n",
    "cat_features2 = [append_str + sub for sub in num_features2]\n",
    "\n",
    "# create the discretizer object with strategy quantile and 1000 bins\n",
    "discretizer1 = KBinsDiscretizer(n_bins=40, encode='ordinal',strategy='quantile')\n",
    "discretizer2 = KBinsDiscretizer(n_bins=4, encode='ordinal',strategy='quantile')\n",
    "\n",
    "pipeline1 = Pipeline([\n",
    "        ('imputer', SimpleImputer( strategy='median')),\n",
    "        ('bin', discretizer1)\n",
    "    ])\n",
    "# fit the discretizer to the train set\n",
    "pipeline1.fit(train.loc[:,num_features1])\n",
    "# apply the discretisation\n",
    "train_cat1 = pipeline1.transform(train.loc[:,num_features1])\n",
    "test_cat1 = pipeline1.transform(test.loc[:,num_features1])\n",
    "train_df1=pd.DataFrame(train_cat1,columns=cat_features1).astype('category')\n",
    "test_df1=pd.DataFrame(test_cat1,columns=cat_features1).astype('category')\n",
    "train_final1= pd.concat( [train.loc[:,num_features1], train_df1], axis=1) \n",
    "test_final1= pd.concat( [test.loc[:,num_features1], test_df1], axis=1) \n",
    "\n",
    "pipeline2 = Pipeline([\n",
    "        ('imputer', SimpleImputer( strategy='median')),\n",
    "        ('bin', discretizer2)\n",
    "    ])\n",
    "# fit the discretizer to the train set\n",
    "pipeline2.fit(train.loc[:,num_features2])\n",
    "# apply the discretisation\n",
    "train_cat2 = pipeline2.transform(train.loc[:,num_features2])\n",
    "test_cat2 = pipeline2.transform(test.loc[:,num_features2])\n",
    "train_df2=pd.DataFrame(train_cat2,columns=cat_features2).astype('category')\n",
    "test_df2=pd.DataFrame(test_cat2,columns=cat_features2).astype('category')\n",
    "train_final2= pd.concat( [train.loc[:,num_features2], train_df2], axis=1) \n",
    "test_final2= pd.concat( [test.loc[:,num_features2], test_df2], axis=1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c5780c59",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:38.435342Z",
     "iopub.status.busy": "2022-01-10T23:32:38.434540Z",
     "iopub.status.idle": "2022-01-10T23:32:38.442354Z",
     "shell.execute_reply": "2022-01-10T23:32:38.441937Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.695325Z"
    },
    "papermill": {
     "duration": 0.055922,
     "end_time": "2022-01-10T23:32:38.442471",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.386549",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "train_final= pd.concat( [train_final1, train_final2], axis=1) \n",
    "test_final= pd.concat( [test_final1, test_final2], axis=1) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "306519a3",
   "metadata": {
    "papermill": {
     "duration": 0.045249,
     "end_time": "2022-01-10T23:32:38.532939",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.487690",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Final X and y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "13b5920c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:38.629296Z",
     "iopub.status.busy": "2022-01-10T23:32:38.628481Z",
     "iopub.status.idle": "2022-01-10T23:32:38.630899Z",
     "shell.execute_reply": "2022-01-10T23:32:38.630463Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.705656Z"
    },
    "papermill": {
     "duration": 0.052294,
     "end_time": "2022-01-10T23:32:38.631020",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.578726",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Pour le train test\n",
    "target= \"SeriousDlqin2yrs\"\n",
    "X = train_final# axis=1\n",
    "X_test_final =test_final# axis=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8ea54a30",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:38.727114Z",
     "iopub.status.busy": "2022-01-10T23:32:38.726157Z",
     "iopub.status.idle": "2022-01-10T23:32:38.748606Z",
     "shell.execute_reply": "2022-01-10T23:32:38.748158Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.712598Z"
    },
    "papermill": {
     "duration": 0.071968,
     "end_time": "2022-01-10T23:32:38.748732",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.676764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 101503 entries, 0 to 101502\n",
      "Data columns (total 20 columns):\n",
      " #   Column                                    Non-Null Count   Dtype   \n",
      "---  ------                                    --------------   -----   \n",
      " 0   RevolvingUtilizationOfUnsecuredLines      101503 non-null  float16 \n",
      " 1   DebtRatio                                 101503 non-null  float32 \n",
      " 2   MonthlyIncome                             101503 non-null  float32 \n",
      " 3   cat_RevolvingUtilizationOfUnsecuredLines  101503 non-null  category\n",
      " 4   cat_DebtRatio                             101503 non-null  category\n",
      " 5   cat_MonthlyIncome                         101503 non-null  category\n",
      " 6   NumberOfDependents                        101503 non-null  float16 \n",
      " 7   NumberOfTime60-89DaysPastDueNotWorse      101503 non-null  int8    \n",
      " 8   NumberRealEstateLoansOrLines              101503 non-null  int8    \n",
      " 9   NumberOfTimes90DaysLate                   101503 non-null  int8    \n",
      " 10  NumberOfOpenCreditLinesAndLoans           101503 non-null  int8    \n",
      " 11  NumberOfTime30-59DaysPastDueNotWorse      101503 non-null  int8    \n",
      " 12  age                                       101503 non-null  int8    \n",
      " 13  cat_NumberOfDependents                    101503 non-null  category\n",
      " 14  cat_NumberOfTime60-89DaysPastDueNotWorse  101503 non-null  category\n",
      " 15  cat_NumberRealEstateLoansOrLines          101503 non-null  category\n",
      " 16  cat_NumberOfTimes90DaysLate               101503 non-null  category\n",
      " 17  cat_NumberOfOpenCreditLinesAndLoans       101503 non-null  category\n",
      " 18  cat_NumberOfTime30-59DaysPastDueNotWorse  101503 non-null  category\n",
      " 19  cat_age                                   101503 non-null  category\n",
      "dtypes: category(10), float16(2), float32(2), int8(6)\n",
      "memory usage: 2.7 MB\n"
     ]
    }
   ],
   "source": [
    "X_test_final.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "11fd37e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:38.850691Z",
     "iopub.status.busy": "2022-01-10T23:32:38.849763Z",
     "iopub.status.idle": "2022-01-10T23:32:38.871298Z",
     "shell.execute_reply": "2022-01-10T23:32:38.872053Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.745499Z"
    },
    "papermill": {
     "duration": 0.072725,
     "end_time": "2022-01-10T23:32:38.872206",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.799481",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 150000 entries, 0 to 149999\n",
      "Data columns (total 20 columns):\n",
      " #   Column                                    Non-Null Count   Dtype   \n",
      "---  ------                                    --------------   -----   \n",
      " 0   RevolvingUtilizationOfUnsecuredLines      150000 non-null  float16 \n",
      " 1   DebtRatio                                 150000 non-null  float32 \n",
      " 2   MonthlyIncome                             150000 non-null  float32 \n",
      " 3   cat_RevolvingUtilizationOfUnsecuredLines  150000 non-null  category\n",
      " 4   cat_DebtRatio                             150000 non-null  category\n",
      " 5   cat_MonthlyIncome                         150000 non-null  category\n",
      " 6   NumberOfDependents                        150000 non-null  float16 \n",
      " 7   NumberOfTime60-89DaysPastDueNotWorse      150000 non-null  int8    \n",
      " 8   NumberRealEstateLoansOrLines              150000 non-null  int8    \n",
      " 9   NumberOfTimes90DaysLate                   150000 non-null  int8    \n",
      " 10  NumberOfOpenCreditLinesAndLoans           150000 non-null  int8    \n",
      " 11  NumberOfTime30-59DaysPastDueNotWorse      150000 non-null  int8    \n",
      " 12  age                                       150000 non-null  int8    \n",
      " 13  cat_NumberOfDependents                    150000 non-null  category\n",
      " 14  cat_NumberOfTime60-89DaysPastDueNotWorse  150000 non-null  category\n",
      " 15  cat_NumberRealEstateLoansOrLines          150000 non-null  category\n",
      " 16  cat_NumberOfTimes90DaysLate               150000 non-null  category\n",
      " 17  cat_NumberOfOpenCreditLinesAndLoans       150000 non-null  category\n",
      " 18  cat_NumberOfTime30-59DaysPastDueNotWorse  150000 non-null  category\n",
      " 19  cat_age                                   150000 non-null  category\n",
      "dtypes: category(10), float16(2), float32(2), int8(6)\n",
      "memory usage: 4.0 MB\n"
     ]
    }
   ],
   "source": [
    "X.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "09a4d5f7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:38.970865Z",
     "iopub.status.busy": "2022-01-10T23:32:38.969925Z",
     "iopub.status.idle": "2022-01-10T23:32:38.971899Z",
     "shell.execute_reply": "2022-01-10T23:32:38.972331Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.772351Z"
    },
    "papermill": {
     "duration": 0.053499,
     "end_time": "2022-01-10T23:32:38.972473",
     "exception": false,
     "start_time": "2022-01-10T23:32:38.918974",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "del train\n",
    "del test \n",
    "del train_final\n",
    "del test_final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "35ca6d33",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:39.074348Z",
     "iopub.status.busy": "2022-01-10T23:32:39.073548Z",
     "iopub.status.idle": "2022-01-10T23:32:39.079239Z",
     "shell.execute_reply": "2022-01-10T23:32:39.078761Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.778589Z"
    },
    "papermill": {
     "duration": 0.060212,
     "end_time": "2022-01-10T23:32:39.079353",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.019141",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['cat_RevolvingUtilizationOfUnsecuredLines', 'cat_DebtRatio',\n",
       "       'cat_MonthlyIncome', 'cat_NumberOfDependents',\n",
       "       'cat_NumberOfTime60-89DaysPastDueNotWorse',\n",
       "       'cat_NumberRealEstateLoansOrLines', 'cat_NumberOfTimes90DaysLate',\n",
       "       'cat_NumberOfOpenCreditLinesAndLoans',\n",
       "       'cat_NumberOfTime30-59DaysPastDueNotWorse', 'cat_age'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select non-numeric columns\n",
    "cat_columns = X.select_dtypes(exclude=['int64','int16','float16','float32','float64','int8']).columns\n",
    "cat_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7721d3f",
   "metadata": {
    "papermill": {
     "duration": 0.04594,
     "end_time": "2022-01-10T23:32:39.171229",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.125289",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Num Features\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "91b957af",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:39.272564Z",
     "iopub.status.busy": "2022-01-10T23:32:39.271783Z",
     "iopub.status.idle": "2022-01-10T23:32:39.277594Z",
     "shell.execute_reply": "2022-01-10T23:32:39.278077Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.794837Z"
    },
    "papermill": {
     "duration": 0.061065,
     "end_time": "2022-01-10T23:32:39.278215",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.217150",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['RevolvingUtilizationOfUnsecuredLines', 'DebtRatio', 'MonthlyIncome',\n",
       "       'NumberOfDependents', 'NumberOfTime60-89DaysPastDueNotWorse',\n",
       "       'NumberRealEstateLoansOrLines', 'NumberOfTimes90DaysLate',\n",
       "       'NumberOfOpenCreditLinesAndLoans',\n",
       "       'NumberOfTime30-59DaysPastDueNotWorse', 'age'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# select the float columns\n",
    "num_columns = X.select_dtypes(include=['int64','int16','float16','float32','float64','int8']).columns\n",
    "num_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bed11f6",
   "metadata": {
    "papermill": {
     "duration": 0.0462,
     "end_time": "2022-01-10T23:32:39.371344",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.325144",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# FE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "70f39b24",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:39.473288Z",
     "iopub.status.busy": "2022-01-10T23:32:39.472429Z",
     "iopub.status.idle": "2022-01-10T23:32:39.474288Z",
     "shell.execute_reply": "2022-01-10T23:32:39.474681Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.812865Z"
    },
    "papermill": {
     "duration": 0.057025,
     "end_time": "2022-01-10T23:32:39.474811",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.417786",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class MiniKmeansTransformerEncoder(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, num_clusters = 11, encoder=ce.woe.WOEEncoder()):\n",
    "        self.num_clusters = num_clusters\n",
    "        self.encoder= encoder\n",
    "        if self.num_clusters > 0:\n",
    "            self.kmeans = MiniBatchKMeans(n_clusters=self.num_clusters, random_state=0)\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        if self.num_clusters > 0:\n",
    "            self.kmeans.fit(X)\n",
    "            preds=self.kmeans.predict(X)\n",
    "            preds=pd.DataFrame(preds, columns=['kmeans']).astype('category')\n",
    "            self.encoder.fit(preds,y)\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X, y=None):\n",
    "        pred_classes = self.kmeans.predict(X)\n",
    "        pred_classes=pd.DataFrame(pred_classes, columns=['kmeans']).astype('category')\n",
    "        pred_encoded = self.encoder.transform(pred_classes)\n",
    "        return np.hstack((X, pred_encoded))\n",
    "        #return pred_encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94de3541",
   "metadata": {
    "papermill": {
     "duration": 0.046284,
     "end_time": "2022-01-10T23:32:39.567806",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.521522",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Complete Pipe \n",
    "\n",
    "1. Features Engineer\n",
    "\n",
    "1. Kmeans\n",
    "\n",
    "1. Sparse\n",
    "\n",
    "1. Poly\n",
    "\n",
    "1. Scaler/Transformer\n",
    "\n",
    "1. imput\n",
    "\n",
    "1. Add bin data \n",
    "\n",
    "## Pipe : \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3ea6c200",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:39.671893Z",
     "iopub.status.busy": "2022-01-10T23:32:39.670988Z",
     "iopub.status.idle": "2022-01-10T23:32:39.672938Z",
     "shell.execute_reply": "2022-01-10T23:32:39.673322Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.824858Z"
    },
    "papermill": {
     "duration": 0.05879,
     "end_time": "2022-01-10T23:32:39.673459",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.614669",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cat pipeline\n",
    "categorical_transformer = Pipeline(\n",
    "                    steps=[\n",
    "                        ('imputer', SimpleImputer(strategy='most_frequent',\n",
    "                                                  fill_value='missing',\n",
    "                                                  add_indicator=True)),\n",
    "                        ('encoder',  ce.woe.WOEEncoder()),#(Numerical Input, Categorical Output)\n",
    "                        #('sparse_features', SparseInteractions(degree=2)),\n",
    "                        #('reducedim',  SelectPercentile( mutual_info_classif, percentile=90))\n",
    "\n",
    "                    ]\n",
    "                    ) \n",
    "#Define vnum pipeline\n",
    "numeric_transformer = Pipeline(\n",
    "                            steps=[\n",
    "                            ('imputer', SimpleImputer(strategy='median'\n",
    "                                                      ,add_indicator=True)),\n",
    "                            #('general_features',FeaturesEngineer()),\n",
    "                            ('scaler', PowerTransformer()),#(Numerical Input, Numerical Output)\n",
    "                            # Create an SelectKBest object to select features with two best ANOVA F-Values\n",
    "                            #The F-value scores examine if, when we group the numerical feature by the target vector, the means for each group are significantly different\n",
    "                            ('kmeans',MiniKmeansTransformerEncoder()),\n",
    "                            ('polynominal_features', PolynomialFeatures(degree=2)),\n",
    "                            #('reducedim',  SelectPercentile(f_classif,percentile=90))\n",
    "                            ]\n",
    "                            )\n",
    "# Features union cat + num \n",
    "# WOE+PowerTransformer\n",
    "preprocessor_woe_powertransformer = ColumnTransformer(\n",
    "            transformers=[\n",
    "                ('numerical', numeric_transformer, num_columns),\n",
    "               # ('categorical', categorical_transformer, cat_columns)\n",
    "            ])\n",
    "#Define vnum pipeline\n",
    "numeric_transformer_final = Pipeline(\n",
    "                            steps=[\n",
    "                            ('imputer', SimpleImputer(strategy='median'\n",
    "                                                      ,add_indicator=True)),\n",
    "                            #('general_features',FeaturesEngineer()),\n",
    "                            ('scaler', PowerTransformer()),#(Numerical Input, Numerical Output)\n",
    "                            # Create an SelectKBest object to select features with two best ANOVA F-Values\n",
    "                            #The F-value scores examine if, when we group the numerical feature by the target vector, the means for each group are significantly different\n",
    "                            ('kmeans',MiniKmeansTransformerEncoder()),\n",
    "                            ('polynominal_features', PolynomialFeatures(degree=2)),\n",
    "                            ('reducedim',  SelectPercentile(f_classif,percentile=95))\n",
    "                            ]\n",
    "                            )\n",
    "# Features union cat + num \n",
    "# WOE+PowerTransformer\n",
    "preprocessor_woe_powertransformer_final = ColumnTransformer(\n",
    "            transformers=[\n",
    "                ('numerical', numeric_transformer_final, num_columns),\n",
    "               # ('categorical', categorical_transformer, cat_columns)\n",
    "            ])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deacd0d0",
   "metadata": {
    "papermill": {
     "duration": 0.048929,
     "end_time": "2022-01-10T23:32:39.769490",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.720561",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "810f0cd3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:39.875151Z",
     "iopub.status.busy": "2022-01-10T23:32:39.874167Z",
     "iopub.status.idle": "2022-01-10T23:32:40.220012Z",
     "shell.execute_reply": "2022-01-10T23:32:40.220447Z",
     "shell.execute_reply.started": "2022-01-10T22:44:55.841460Z"
    },
    "papermill": {
     "duration": 0.402317,
     "end_time": "2022-01-10T23:32:40.220594",
     "exception": false,
     "start_time": "2022-01-10T23:32:39.818277",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>div.sk-top-container {color: black;background-color: white;}div.sk-toggleable {background-color: white;}label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}div.sk-estimator:hover {background-color: #d4ebff;}div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}div.sk-item {z-index: 1;}div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}div.sk-parallel-item:only-child::after {width: 0;}div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}div.sk-label-container {position: relative;z-index: 2;text-align: center;}div.sk-container {display: inline-block;position: relative;}</style><div class=\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"03befd2f-b5a3-4684-b70c-30fc7c16ded0\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"03befd2f-b5a3-4684-b70c-30fc7c16ded0\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(transformers=[('numerical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 strategy='median')),\n",
       "                                                                  ('scaler',\n",
       "                                                                   PowerTransformer()),\n",
       "                                                                  ('kmeans',\n",
       "                                                                   MiniKmeansTransformerEncoder()),\n",
       "                                                                  ('polynominal_features',\n",
       "                                                                   PolynomialFeatures())]),\n",
       "                                                  Index(['RevolvingUtilizationOfUnsecuredLines', 'DebtRatio', 'MonthlyIncome'...\n",
       "       'NumberOfOpenCreditLinesAndLoans',\n",
       "       'NumberOfTime30-59DaysPastDueNotWorse', 'age'],\n",
       "      dtype='object'))])),\n",
       "                ('reducedim', SelectPercentile(percentile=95)),\n",
       "                ('classifier',\n",
       "                 LGBMClassifier(device_type='gpu',\n",
       "                                learning_rate=0.0018069834369607075,\n",
       "                                max_depth=8, metric='auc', min_samples_leaf=47,\n",
       "                                n_estimators=5000, random_state=42,\n",
       "                                subsample=0.8573598985000007, verbose=-1))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"ece51883-49f1-472b-8e66-e432acaad739\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"ece51883-49f1-472b-8e66-e432acaad739\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[('numerical',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(add_indicator=True,\n",
       "                                                                strategy='median')),\n",
       "                                                 ('scaler', PowerTransformer()),\n",
       "                                                 ('kmeans',\n",
       "                                                  MiniKmeansTransformerEncoder()),\n",
       "                                                 ('polynominal_features',\n",
       "                                                  PolynomialFeatures())]),\n",
       "                                 Index(['RevolvingUtilizationOfUnsecuredLines', 'DebtRatio', 'MonthlyIncome',\n",
       "       'NumberOfDependents', 'NumberOfTime60-89DaysPastDueNotWorse',\n",
       "       'NumberRealEstateLoansOrLines', 'NumberOfTimes90DaysLate',\n",
       "       'NumberOfOpenCreditLinesAndLoans',\n",
       "       'NumberOfTime30-59DaysPastDueNotWorse', 'age'],\n",
       "      dtype='object'))])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"340b6d2f-5c2b-411b-8ed6-f21a49e58c40\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"340b6d2f-5c2b-411b-8ed6-f21a49e58c40\">numerical</label><div class=\"sk-toggleable__content\"><pre>Index(['RevolvingUtilizationOfUnsecuredLines', 'DebtRatio', 'MonthlyIncome',\n",
       "       'NumberOfDependents', 'NumberOfTime60-89DaysPastDueNotWorse',\n",
       "       'NumberRealEstateLoansOrLines', 'NumberOfTimes90DaysLate',\n",
       "       'NumberOfOpenCreditLinesAndLoans',\n",
       "       'NumberOfTime30-59DaysPastDueNotWorse', 'age'],\n",
       "      dtype='object')</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"2444e4d8-fdcc-4720-8f38-cc90d7202ac9\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"2444e4d8-fdcc-4720-8f38-cc90d7202ac9\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(add_indicator=True, strategy='median')</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"b22188ab-cbf4-4810-9ad0-ce3bd4a06b7f\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"b22188ab-cbf4-4810-9ad0-ce3bd4a06b7f\">PowerTransformer</label><div class=\"sk-toggleable__content\"><pre>PowerTransformer()</pre></div></div></div><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"41dad6e7-984d-43c9-be29-410482b71adb\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"41dad6e7-984d-43c9-be29-410482b71adb\">kmeans: MiniKmeansTransformerEncoder</label><div class=\"sk-toggleable__content\"><pre>MiniKmeansTransformerEncoder()</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"328da9f3-9bc2-410c-baef-39db3ae0de09\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"328da9f3-9bc2-410c-baef-39db3ae0de09\">WOEEncoder</label><div class=\"sk-toggleable__content\"><pre>WOEEncoder()</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"db491481-b579-464e-afe3-309189ef27b8\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"db491481-b579-464e-afe3-309189ef27b8\">PolynomialFeatures</label><div class=\"sk-toggleable__content\"><pre>PolynomialFeatures()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"000ace87-eb4d-46b9-b071-3601020a0176\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"000ace87-eb4d-46b9-b071-3601020a0176\">SelectPercentile</label><div class=\"sk-toggleable__content\"><pre>SelectPercentile(percentile=95)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"241a19dc-3318-4e26-b9f0-be1451588c7d\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"241a19dc-3318-4e26-b9f0-be1451588c7d\">LGBMClassifier</label><div class=\"sk-toggleable__content\"><pre>LGBMClassifier(device_type='gpu', learning_rate=0.0018069834369607075,\n",
       "               max_depth=8, metric='auc', min_samples_leaf=47,\n",
       "               n_estimators=5000, random_state=42, subsample=0.8573598985000007,\n",
       "               verbose=-1)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(transformers=[('numerical',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=True,\n",
       "                                                                                 strategy='median')),\n",
       "                                                                  ('scaler',\n",
       "                                                                   PowerTransformer()),\n",
       "                                                                  ('kmeans',\n",
       "                                                                   MiniKmeansTransformerEncoder()),\n",
       "                                                                  ('polynominal_features',\n",
       "                                                                   PolynomialFeatures())]),\n",
       "                                                  Index(['RevolvingUtilizationOfUnsecuredLines', 'DebtRatio', 'MonthlyIncome'...\n",
       "       'NumberOfOpenCreditLinesAndLoans',\n",
       "       'NumberOfTime30-59DaysPastDueNotWorse', 'age'],\n",
       "      dtype='object'))])),\n",
       "                ('reducedim', SelectPercentile(percentile=95)),\n",
       "                ('classifier',\n",
       "                 LGBMClassifier(device_type='gpu',\n",
       "                                learning_rate=0.0018069834369607075,\n",
       "                                max_depth=8, metric='auc', min_samples_leaf=47,\n",
       "                                n_estimators=5000, random_state=42,\n",
       "                                subsample=0.8573598985000007, verbose=-1))])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lgbm_param2={'learning_rate': 0.0018069834369607075,\n",
    "                                 'max_depth': 8,\n",
    "                                 #'max_features': 4,\n",
    "                                 'min_samples_leaf': 47,\n",
    "                                 #'min_samples_split': 389,\n",
    "                                 'subsample': 0.8573598985000007,\n",
    "                                 #'n_iter_no_change': 300,\n",
    "                                 'n_estimators': 5000,\n",
    "                                 'verbose': -1,\n",
    "                                 'random_state': 42,\n",
    "                                 'metric': 'auc',\n",
    "                                \"device_type\" : \"gpu\",\n",
    "                                'boosting_type': 'gbdt',\n",
    "                                #'tree_method': \"gpu_hist\"\n",
    "                               }\n",
    "\n",
    "model_lgbm2 = lgbm.LGBMClassifier(**lgbm_param2,\n",
    "                              # use it  only on gpu\n",
    "                            # rsm=np.nan\n",
    "                              #cat_features=cat_features\n",
    "                             )\n",
    "pipe1_model_lgbm_woe_powertransformer2 = Pipeline(\n",
    "    steps=[\n",
    "        ('preprocessor', preprocessor_woe_powertransformer),\n",
    "         ('reducedim',  SelectPercentile( f_classif, percentile=95)),\n",
    "        # ('dim_red', SelectKBest(f_regression, k=20)),\n",
    "        #('Sparseinteractions', SparseInteractions(degree=2)),\n",
    "         #('PolynomialFeatures', PolynomialFeatures(degree=2)),\n",
    "        #('PCAreducer', PCA(n_components=0.8)),\n",
    "        ('classifier',model_lgbm2)\n",
    "    ]\n",
    ")\n",
    "pipe1_model_lgbm_woe_powertransformer2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "224a5e03",
   "metadata": {
    "papermill": {
     "duration": 0.049237,
     "end_time": "2022-01-10T23:32:40.317793",
     "exception": false,
     "start_time": "2022-01-10T23:32:40.268556",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "#  Train_Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8ab6d78a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:40.418754Z",
     "iopub.status.busy": "2022-01-10T23:32:40.417959Z",
     "iopub.status.idle": "2022-01-10T23:32:40.521020Z",
     "shell.execute_reply": "2022-01-10T23:32:40.520533Z",
     "shell.execute_reply.started": "2022-01-10T22:44:56.201018Z"
    },
    "papermill": {
     "duration": 0.156359,
     "end_time": "2022-01-10T23:32:40.521164",
     "exception": false,
     "start_time": "2022-01-10T23:32:40.364805",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "###############################################################################\n",
    "#                                  Get data                                 #\n",
    "###############################################################################\n",
    "X_train,X_test, y_train,  y_test = train_test_split(X, y,stratify=y, \n",
    "                                                       test_size=0.25,\n",
    "                                                        random_state=42,\n",
    "                                                        shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "17130491",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:40.623882Z",
     "iopub.status.busy": "2022-01-10T23:32:40.623073Z",
     "iopub.status.idle": "2022-01-10T23:32:44.208037Z",
     "shell.execute_reply": "2022-01-10T23:32:44.207513Z",
     "shell.execute_reply.started": "2022-01-10T22:44:56.307113Z"
    },
    "papermill": {
     "duration": 3.639147,
     "end_time": "2022-01-10T23:32:44.208179",
     "exception": false,
     "start_time": "2022-01-10T23:32:40.569032",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "preprocessor_woe_powertransformer_final.fit(X_train,y_train)\n",
    "X_train_pre = preprocessor_woe_powertransformer_final.transform(X_train)\n",
    "X_test_pre = preprocessor_woe_powertransformer_final.transform(X_test)\n",
    "#del x_final\n",
    "#del x_test_final"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c659d158",
   "metadata": {
    "papermill": {
     "duration": 0.047459,
     "end_time": "2022-01-10T23:32:44.303081",
     "exception": false,
     "start_time": "2022-01-10T23:32:44.255622",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "23b8c6b6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:44.403376Z",
     "iopub.status.busy": "2022-01-10T23:32:44.402593Z",
     "iopub.status.idle": "2022-01-10T23:32:44.662874Z",
     "shell.execute_reply": "2022-01-10T23:32:44.662188Z",
     "shell.execute_reply.started": "2022-01-10T22:44:59.585921Z"
    },
    "papermill": {
     "duration": 0.312806,
     "end_time": "2022-01-10T23:32:44.663013",
     "exception": false,
     "start_time": "2022-01-10T23:32:44.350207",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTreeClassifier\n",
      "ExtraTreeClassifier\n",
      "ExtraTreesClassifier\n",
      "LinearSVC\n",
      "LogisticRegression\n",
      "LogisticRegressionCV\n",
      "NuSVC\n",
      "PassiveAggressiveClassifier\n",
      "Perceptron\n",
      "RandomForestClassifier\n",
      "RidgeClassifier\n",
      "RidgeClassifierCV\n",
      "SGDClassifier\n",
      "SVC\n"
     ]
    }
   ],
   "source": [
    "# NOTE, THIS CELL WILL NOT WORK WITH SKLEARN VERSION > 0.24\n",
    "# JUST COMMENT IT OUT, OR DELETE THIS CELL TO CARRY ON WITH THE NOTEBOOK.\n",
    "\n",
    "# Let's find out which classifiers from sklearn support class_weight\n",
    "# as part of the __init__ method, that is, when we set the m up\n",
    "\n",
    "from sklearn.utils.testing import all_estimators\n",
    "\n",
    "estimators = all_estimators(type_filter='classifier')\n",
    "\n",
    "for name, class_ in estimators:\n",
    "    try:\n",
    "        if hasattr(class_(), 'class_weight'):\n",
    "            print(name)\n",
    "    except:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c891772",
   "metadata": {
    "papermill": {
     "duration": 0.047199,
     "end_time": "2022-01-10T23:32:44.758631",
     "exception": false,
     "start_time": "2022-01-10T23:32:44.711432",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Not all classifiers support class_weight. For those which don't, like GradientBoostingClassifier, we can still use sample_weight when we fit the estimator.\n",
    "# Using class_weight \n",
    "## LR /lgbm2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "7a9c2ac9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:44.865078Z",
     "iopub.status.busy": "2022-01-10T23:32:44.864195Z",
     "iopub.status.idle": "2022-01-10T23:32:44.866008Z",
     "shell.execute_reply": "2022-01-10T23:32:44.866415Z",
     "shell.execute_reply.started": "2022-01-10T22:44:59.849520Z"
    },
    "papermill": {
     "duration": 0.059908,
     "end_time": "2022-01-10T23:32:44.866548",
     "exception": false,
     "start_time": "2022-01-10T23:32:44.806640",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Logistic Regression with class_weight\n",
    "# we initialize the cost / weights when we set up the transformer\n",
    "def run_Logit1(X_train, X_test, y_train, y_test, class_weight):\n",
    "    \n",
    "    # weights introduced here\n",
    "    logit = lgbm.LGBMClassifier(**lgbm_param2,\n",
    "                              # use it  only on gpu\n",
    "                            # rsm=np.nan\n",
    "                              #cat_features=cat_features\n",
    "                                class_weight=class_weight # weights / cost\n",
    "                             )\n",
    "    \n",
    "    logit.fit(X_train, y_train)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = logit.predict_proba(X_train)\n",
    "    print(\n",
    "        'LGBM roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = logit.predict_proba(X_test)\n",
    "    print(\n",
    "        'LGBM roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))\n",
    "def run_Logit2(X_train, X_test, y_train, y_test, class_weight):\n",
    "    \n",
    "    # weights introduced here\n",
    "    logit = LogisticRegression(\n",
    "        penalty='l2',\n",
    "        solver='newton-cg',\n",
    "        random_state=0,\n",
    "        max_iter=10,\n",
    "        n_jobs=4,\n",
    "       class_weight=class_weight # weights / cost\n",
    "    )\n",
    "    \n",
    "    logit.fit(X_train, y_train)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = logit.predict_proba(X_train)\n",
    "    print(\n",
    "        'Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = logit.predict_proba(X_test)\n",
    "    print(\n",
    "        'Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c07f3abd",
   "metadata": {
    "papermill": {
     "duration": 0.047208,
     "end_time": "2022-01-10T23:32:44.961700",
     "exception": false,
     "start_time": "2022-01-10T23:32:44.914492",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## lgbm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "371a832d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:32:45.061333Z",
     "iopub.status.busy": "2022-01-10T23:32:45.060520Z",
     "iopub.status.idle": "2022-01-10T23:34:48.740643Z",
     "shell.execute_reply": "2022-01-10T23:34:48.740173Z",
     "shell.execute_reply.started": "2022-01-10T22:44:59.867203Z"
    },
    "papermill": {
     "duration": 123.731152,
     "end_time": "2022-01-10T23:34:48.740802",
     "exception": false,
     "start_time": "2022-01-10T23:32:45.009650",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n",
      "Train set\n",
      "LGBM roc-auc: 0.8978676276484274\n",
      "Test set\n",
      "LGBM roc-auc: 0.8661449077053045\n"
     ]
    }
   ],
   "source": [
    "# evaluate performance of algorithm built\n",
    "# using imbalanced dataset\n",
    "run_Logit1(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "737753a6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:34:48.842651Z",
     "iopub.status.busy": "2022-01-10T23:34:48.841593Z",
     "iopub.status.idle": "2022-01-10T23:34:57.181906Z",
     "shell.execute_reply": "2022-01-10T23:34:57.182593Z",
     "shell.execute_reply.started": "2022-01-10T22:46:58.225354Z"
    },
    "papermill": {
     "duration": 8.393006,
     "end_time": "2022-01-10T23:34:57.182834",
     "exception": false,
     "start_time": "2022-01-10T23:34:48.789828",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8600715713834044\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.8626891334375826\n"
     ]
    }
   ],
   "source": [
    "# evaluate performance of algorithm built\n",
    "# using imbalanced dataset\n",
    "run_Logit2(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6ee0c7d",
   "metadata": {
    "papermill": {
     "duration": 0.054267,
     "end_time": "2022-01-10T23:34:57.331690",
     "exception": false,
     "start_time": "2022-01-10T23:34:57.277423",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Balanced : \n",
    "By default, the value of class_weight=None, i.e. both the classes have been given equal weights. Other than that, we can either give it as ‘balanced’ or we can pass a dictionary that contains manual weights for both the classes.\n",
    "\n",
    "When the class_weights = ‘balanced’, the model automatically assigns the class weights inversely proportional to their respective frequencies.\n",
    "\n",
    "To be more precise, the formula to calculate this is:\n",
    "\n",
    "wj=n_samples / (n_classes * n_samplesj)\n",
    "\n",
    "Here,\n",
    "\n",
    "wj is the weight for each class(j signifies the class)\n",
    "n_samplesis the total number of samples or rows in the dataset\n",
    "n_classesis the total number of unique classes in the target\n",
    "n_samplesjis the total number of rows of the respective class\n",
    "For our heart stroke example:\n",
    "\n",
    "n_samples=  43400,  n_classes= 2(0&1), n_sample0= 42617, n_samples1= 783\n",
    "\n",
    "Weights for class 0:\n",
    "\n",
    "w0=  43400/(2*42617) = 0.509\n",
    "\n",
    "Weights for class 1:\n",
    "\n",
    "w1= 43400/(2*783) = 27.713\n",
    "\n",
    "I hope this makes things more clear that how class_weight = ‘balanced’ helps us to in giving higher weights to the minority class and lower weights to the majority class.\n",
    "\n",
    "Although passing value as ‘balanced’ gives good results in most cases but sometimes for extreme class imbalance, we can try giving weights manually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3d0f920f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:34:57.434651Z",
     "iopub.status.busy": "2022-01-10T23:34:57.433946Z",
     "iopub.status.idle": "2022-01-10T23:37:02.694492Z",
     "shell.execute_reply": "2022-01-10T23:37:02.695134Z",
     "shell.execute_reply.started": "2022-01-10T22:47:06.977782Z"
    },
    "papermill": {
     "duration": 125.314367,
     "end_time": "2022-01-10T23:37:02.695330",
     "exception": false,
     "start_time": "2022-01-10T23:34:57.380963",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n",
      "Train set\n",
      "LGBM roc-auc: 0.9018389571671199\n",
      "Test set\n",
      "LGBM roc-auc: 0.8648791850806848\n"
     ]
    }
   ],
   "source": [
    "# evaluate performance of algorithm built\n",
    "# cost estimated as imbalance ratio\n",
    "# 'balanced' indicates that we want same amount of \n",
    "# each observation, thus, imbalance ratio\n",
    "run_Logit1(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "664da922",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:37:02.801606Z",
     "iopub.status.busy": "2022-01-10T23:37:02.800642Z",
     "iopub.status.idle": "2022-01-10T23:37:09.172607Z",
     "shell.execute_reply": "2022-01-10T23:37:09.171419Z",
     "shell.execute_reply.started": "2022-01-10T22:49:06.925776Z"
    },
    "papermill": {
     "duration": 6.427086,
     "end_time": "2022-01-10T23:37:09.172817",
     "exception": false,
     "start_time": "2022-01-10T23:37:02.745731",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8608867393181274\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.8633146881729719\n"
     ]
    }
   ],
   "source": [
    "run_Logit2(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight='balanced')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "4b6519ba",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:37:09.331535Z",
     "iopub.status.busy": "2022-01-10T23:37:09.330722Z",
     "iopub.status.idle": "2022-01-10T23:39:11.902838Z",
     "shell.execute_reply": "2022-01-10T23:39:11.903406Z",
     "shell.execute_reply.started": "2022-01-10T22:49:13.052389Z"
    },
    "papermill": {
     "duration": 122.636426,
     "end_time": "2022-01-10T23:39:11.903584",
     "exception": false,
     "start_time": "2022-01-10T23:37:09.267158",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n",
      "Train set\n",
      "LGBM roc-auc: 0.9014341871856036\n",
      "Test set\n",
      "LGBM roc-auc: 0.8653286692722743\n"
     ]
    }
   ],
   "source": [
    "# evaluate performance of algorithm built\n",
    "# cost estimated as imbalance ratio\n",
    "\n",
    "# alternatively, we can pass a different cost\n",
    "# in a dictionary, if we know it already\n",
    "\n",
    "run_Logit1(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight={0:1, 1:10})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "9cd905e0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:12.021692Z",
     "iopub.status.busy": "2022-01-10T23:39:12.020750Z",
     "iopub.status.idle": "2022-01-10T23:39:18.264771Z",
     "shell.execute_reply": "2022-01-10T23:39:18.265551Z",
     "shell.execute_reply.started": "2022-01-10T22:51:11.717855Z"
    },
    "papermill": {
     "duration": 6.305362,
     "end_time": "2022-01-10T23:39:18.265816",
     "exception": false,
     "start_time": "2022-01-10T23:39:11.960454",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8605597710100809\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.863189903356366\n"
     ]
    }
   ],
   "source": [
    "run_Logit2(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          class_weight={0:1, 1:10})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d4123e1",
   "metadata": {
    "papermill": {
     "duration": 0.056832,
     "end_time": "2022-01-10T23:39:18.419730",
     "exception": false,
     "start_time": "2022-01-10T23:39:18.362898",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Using sample_weight\n",
    "\n",
    "All the documentation says that is should be:\n",
    "\n",
    "**scale_pos_weight = count(negative examples)/count(Positive examples)**\n",
    "\n",
    "In practice, that works pretty well, but if your dataset is extremely unbalanced I'd recommend using something more conservative like:\n",
    "\n",
    "**scale_pos_weight = sqrt(count(negative examples)/count(Positive examples))**\n",
    "\n",
    "credit : \n",
    "https://www.kaggle.com/bannourchaker/6-datacentricapprochfinal\n",
    "\n",
    "**Weight 1:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6e34f2dc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:18.531606Z",
     "iopub.status.busy": "2022-01-10T23:39:18.530938Z",
     "iopub.status.idle": "2022-01-10T23:39:18.533543Z",
     "shell.execute_reply": "2022-01-10T23:39:18.533970Z",
     "shell.execute_reply.started": "2022-01-10T22:51:18.073527Z"
    },
    "papermill": {
     "duration": 0.062393,
     "end_time": "2022-01-10T23:39:18.534105",
     "exception": false,
     "start_time": "2022-01-10T23:39:18.471712",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.961101137043686"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scale_pos_weight1 = 139974/(10026)\n",
    "scale_pos_weight1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e329508a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:18.642827Z",
     "iopub.status.busy": "2022-01-10T23:39:18.642204Z",
     "iopub.status.idle": "2022-01-10T23:39:18.644865Z",
     "shell.execute_reply": "2022-01-10T23:39:18.645273Z",
     "shell.execute_reply.started": "2022-01-10T22:51:18.093742Z"
    },
    "papermill": {
     "duration": 0.059976,
     "end_time": "2022-01-10T23:39:18.645409",
     "exception": false,
     "start_time": "2022-01-10T23:39:18.585433",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.7364556918346676"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scale_pos_weight2_smooth = np.sqrt(139974/(10026)) \n",
    "scale_pos_weight2_smooth "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "65731115",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:18.759309Z",
     "iopub.status.busy": "2022-01-10T23:39:18.758423Z",
     "iopub.status.idle": "2022-01-10T23:39:18.760335Z",
     "shell.execute_reply": "2022-01-10T23:39:18.760694Z",
     "shell.execute_reply.started": "2022-01-10T22:51:18.108974Z"
    },
    "papermill": {
     "duration": 0.063688,
     "end_time": "2022-01-10T23:39:18.760827",
     "exception": false,
     "start_time": "2022-01-10T23:39:18.697139",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "# Logistic Regression + sample_weight\n",
    "\n",
    "# we pass the weights / cost, when we train the algorithm\n",
    "\n",
    "def run_Logit3(X_train, X_test, y_train, y_test, sample_weight):\n",
    "    \n",
    "    logit = LogisticRegression(\n",
    "        penalty='l2',\n",
    "        solver='newton-cg',\n",
    "        random_state=0,\n",
    "        max_iter=10,\n",
    "        n_jobs=4,\n",
    "    )\n",
    "    \n",
    "    # costs are passed here\n",
    "    logit.fit(X_train, y_train, sample_weight=sample_weight)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = logit.predict_proba(X_train)\n",
    "    print(\n",
    "        'lR roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = logit.predict_proba(X_test)\n",
    "    print(\n",
    "        'LR roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))\n",
    "def run_Logit4(X_train, X_test, y_train, y_test, sample_weight):\n",
    "\n",
    "    logit = lgbm.LGBMClassifier(**lgbm_param2,\n",
    "                              # use it  only on gpu\n",
    "                            # rsm=np.nan\n",
    "                              #cat_features=cat_features\n",
    "                            #    class_weight=class_weight # weights / cost\n",
    "                             )\n",
    "    \n",
    "\n",
    "    # costs are passed here\n",
    "    logit.fit(X_train, y_train, sample_weight=sample_weight)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = logit.predict_proba(X_train)\n",
    "    print(\n",
    "        'LGBM roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = logit.predict_proba(X_test)\n",
    "    print(\n",
    "        'LGBM roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "8c08d938",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:18.868580Z",
     "iopub.status.busy": "2022-01-10T23:39:18.867721Z",
     "iopub.status.idle": "2022-01-10T23:39:26.280882Z",
     "shell.execute_reply": "2022-01-10T23:39:26.281808Z",
     "shell.execute_reply.started": "2022-01-10T22:51:18.132484Z"
    },
    "papermill": {
     "duration": 7.469203,
     "end_time": "2022-01-10T23:39:26.282065",
     "exception": false,
     "start_time": "2022-01-10T23:39:18.812862",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "lR roc-auc: 0.8600715713834044\n",
      "Test set\n",
      "LR roc-auc: 0.8626891334375826\n"
     ]
    }
   ],
   "source": [
    "# evaluate performance of algorithm built\n",
    "# using imbalanced dataset\n",
    "\n",
    "run_Logit3(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "55490fac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:39:26.445354Z",
     "iopub.status.busy": "2022-01-10T23:39:26.444504Z",
     "iopub.status.idle": "2022-01-10T23:41:24.589141Z",
     "shell.execute_reply": "2022-01-10T23:41:24.589665Z",
     "shell.execute_reply.started": "2022-01-10T22:51:25.204871Z"
    },
    "papermill": {
     "duration": 118.207808,
     "end_time": "2022-01-10T23:41:24.589819",
     "exception": false,
     "start_time": "2022-01-10T23:39:26.382011",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/utils/optimize.py:212: ConvergenceWarning: newton-cg failed to converge. Increase the number of iterations.\n",
      "  \"number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "LGBM roc-auc: 0.897707804272749\n",
      "Test set\n",
      "LGBM roc-auc: 0.8662377750676765\n"
     ]
    }
   ],
   "source": [
    "run_Logit4(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "05e72e6c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:41:24.703621Z",
     "iopub.status.busy": "2022-01-10T23:41:24.702575Z",
     "iopub.status.idle": "2022-01-10T23:41:31.561262Z",
     "shell.execute_reply": "2022-01-10T23:41:31.560477Z",
     "shell.execute_reply.started": "2022-01-10T22:53:19.979521Z"
    },
    "papermill": {
     "duration": 6.917849,
     "end_time": "2022-01-10T23:41:31.561479",
     "exception": false,
     "start_time": "2022-01-10T23:41:24.643630",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "lR roc-auc: 0.8606866125462601\n",
      "Test set\n",
      "LR roc-auc: 0.8632191866798644\n"
     ]
    }
   ],
   "source": [
    "run_Logit3(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=np.where(y_train==1,4,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "27f9d938",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:41:31.800790Z",
     "iopub.status.busy": "2022-01-10T23:41:31.800016Z",
     "iopub.status.idle": "2022-01-10T23:43:33.840651Z",
     "shell.execute_reply": "2022-01-10T23:43:33.841222Z",
     "shell.execute_reply.started": "2022-01-10T22:53:26.715453Z"
    },
    "papermill": {
     "duration": 122.141687,
     "end_time": "2022-01-10T23:43:33.841399",
     "exception": false,
     "start_time": "2022-01-10T23:41:31.699712",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n",
      "Train set\n",
      "LGBM roc-auc: 0.8994740835893767\n",
      "Test set\n",
      "LGBM roc-auc: 0.866644702653621\n"
     ]
    }
   ],
   "source": [
    "run_Logit4(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=np.where(y_train==1,4,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "41bf1042",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:43:33.956747Z",
     "iopub.status.busy": "2022-01-10T23:43:33.955830Z",
     "iopub.status.idle": "2022-01-10T23:43:41.338134Z",
     "shell.execute_reply": "2022-01-10T23:43:41.338794Z",
     "shell.execute_reply.started": "2022-01-10T22:55:25.489353Z"
    },
    "papermill": {
     "duration": 7.442597,
     "end_time": "2022-01-10T23:43:41.339049",
     "exception": false,
     "start_time": "2022-01-10T23:43:33.896452",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "lR roc-auc: 0.8611743472920881\n",
      "Test set\n",
      "LR roc-auc: 0.863419916564422\n"
     ]
    }
   ],
   "source": [
    "run_Logit3(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "         sample_weight=np.where(y_train==1,14,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "724ea169",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:43:41.506962Z",
     "iopub.status.busy": "2022-01-10T23:43:41.506211Z",
     "iopub.status.idle": "2022-01-10T23:45:45.185841Z",
     "shell.execute_reply": "2022-01-10T23:45:45.186860Z",
     "shell.execute_reply.started": "2022-01-10T22:55:33.014984Z"
    },
    "papermill": {
     "duration": 123.744075,
     "end_time": "2022-01-10T23:45:45.187086",
     "exception": false,
     "start_time": "2022-01-10T23:43:41.443011",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/utils/optimize.py:212: ConvergenceWarning: newton-cg failed to converge. Increase the number of iterations.\n",
      "  \"number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "LGBM roc-auc: 0.9020960464100556\n",
      "Test set\n",
      "LGBM roc-auc: 0.8648897501115344\n"
     ]
    }
   ],
   "source": [
    "run_Logit4(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=np.where(y_train==1,14,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "a73fd775",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:45:45.353368Z",
     "iopub.status.busy": "2022-01-10T23:45:45.352570Z",
     "iopub.status.idle": "2022-01-10T23:47:48.194146Z",
     "shell.execute_reply": "2022-01-10T23:47:48.194643Z",
     "shell.execute_reply.started": "2022-01-10T22:57:34.207731Z"
    },
    "papermill": {
     "duration": 122.916336,
     "end_time": "2022-01-10T23:47:48.194829",
     "exception": false,
     "start_time": "2022-01-10T23:45:45.278493",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[LightGBM] [Warning] min_data_in_leaf is set with min_child_samples=20, will be overridden by min_samples_leaf=47. Current value: min_data_in_leaf=47\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/sklearn/utils/optimize.py:212: ConvergenceWarning: newton-cg failed to converge. Increase the number of iterations.\n",
      "  \"number of iterations.\", ConvergenceWarning)\n",
      "/opt/conda/lib/python3.7/site-packages/sklearn/utils/optimize.py:212: ConvergenceWarning: newton-cg failed to converge. Increase the number of iterations.\n",
      "  \"number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "LGBM roc-auc: 0.9042314702547192\n",
      "Test set\n",
      "LGBM roc-auc: 0.8622588920841565\n"
     ]
    }
   ],
   "source": [
    "run_Logit4(X_train_pre,\n",
    "          X_test_pre,\n",
    "          y_train,\n",
    "          y_test,\n",
    "          sample_weight=np.where(y_train==1,50,1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "58360aac",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:47:48.319234Z",
     "iopub.status.busy": "2022-01-10T23:47:48.318618Z",
     "iopub.status.idle": "2022-01-10T23:47:48.321504Z",
     "shell.execute_reply": "2022-01-10T23:47:48.321914Z",
     "shell.execute_reply.started": "2022-01-10T22:59:33.262077Z"
    },
    "papermill": {
     "duration": 0.069381,
     "end_time": "2022-01-10T23:47:48.322053",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.252672",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 5 µs, total: 5 µs\n",
      "Wall time: 9.06 µs\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "#pipe_woe_powertransformer.fit(X,y)\n",
    "#x_pre = pipe_woe_powertransformer_red.transform(X)\n",
    "#x_test_final_pre = pipe_woe_powertransformer_red.transform(X_test_final)\n",
    "#del x_final\n",
    "#del x_test_final\n",
    "def plot_roc(y,y_pred):\n",
    "    from sklearn.metrics import auc\n",
    "    fpr, tpr, thresholds = roc_curve(y_true=y, y_score=y_pred)\n",
    "    auc = auc(fpr, tpr)\n",
    "    plt.figure(1)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.plot(fpr, tpr, label='Model(area = {:.3f})'.format(auc))\n",
    "    plt.xlabel('False positive rate')\n",
    "    plt.ylabel('True positive rate')\n",
    "    plt.title('ROC curve')\n",
    "    plt.legend(loc='best')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b24688e",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2021-12-31T12:33:50.384931Z",
     "iopub.status.idle": "2021-12-31T12:33:50.385528Z",
     "shell.execute_reply": "2021-12-31T12:33:50.385321Z",
     "shell.execute_reply.started": "2021-12-31T12:33:50.385297Z"
    },
    "papermill": {
     "duration": 0.058348,
     "end_time": "2022-01-10T23:47:48.439053",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.380705",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "    %%time \n",
    "    # evaluate each strategy on the dataset\n",
    "    results = list()\n",
    "    # Setting a 10-fold stratified cross-validation (note: shuffle=True)\n",
    "    SEED = 42\n",
    "    FOLDS = 10\n",
    "    skf = StratifiedKFold(n_splits=FOLDS, shuffle=True, random_state=SEED)\n",
    "    # CV interations\n",
    "    # Create arrays for the features and the response variable\n",
    "    roc_auc = list()\n",
    "    average_precision = list()\n",
    "    #X1=X.iloc[0:2000,:].copy()\n",
    "    #y1=y[0:2000].copy()\n",
    "    oof = np.empty((X.shape[0],))\n",
    "    #oof_bin = np.empty((X.shape[0],))\n",
    "    predictions=[]\n",
    "    mean_auc = 0\n",
    "    mean_ap=0\n",
    "    F1 = list()\n",
    "    Recall=list()\n",
    "    best_iteration = list()\n",
    "    for fold, (train_idx, test_idx) in enumerate(skf.split(X, y)):    \n",
    "        X_train, y_train = X.iloc[list(train_idx), :], y.iloc[list(train_idx)]\n",
    "        X_test, y_test = X.iloc[list(test_idx), :],y.iloc[list(test_idx)]\n",
    "        pipe1_model_lgbm_woe_powertransformer2.fit(X_train,y_train,\n",
    "                                                      #classifier__verbose=False\n",
    "                                                      #,verbose=False\n",
    "                                                     )\n",
    "        preds = pipe1_model_lgbm_woe_powertransformer2.predict_proba(X_test)[:,1]\n",
    "        oof[test_idx] = preds\n",
    "        auc_score= roc_auc_score(y_true=y_test, y_score=preds)\n",
    "        roc_auc.append(auc_score)\n",
    "        ap=average_precision_score(y_true=y_test, y_score=preds)\n",
    "        plot_roc(y_test,preds)\n",
    "        mean_auc += auc_score / FOLDS\n",
    "        mean_ap+=ap/FOLDS\n",
    "        predictions.append(pipe1_model_lgbm_woe_powertransformer2.predict_proba(X_test_final)[:,1]) \n",
    "        y_predicted = pipe1_model_lgbm_woe_powertransformer2.predict(X_test)\n",
    "        recall = recall_score(y_test, y_predicted)\n",
    "        f1= f1_score(y_test, y_predicted)\n",
    "        print(f\"Fold {fold} | AUC: {auc_score}\")\n",
    "        print(f\"Fold {fold} | Avergae_precesion: {ap}\")\n",
    "        print(f\"Fold {fold} | recall: {recall}\")\n",
    "        print(f\"Fold {fold} | F1: {f1}\")\n",
    "        print('Classification report:\\n',classification_report(y_test,y_predicted))\n",
    "        print('Confusion_matrix:\\n',confusion_matrix(y_test,y_predicted))\n",
    "        roc_auc.append(auc_score)\n",
    "        F1.append(f1)\n",
    "        Recall.append(recall)\n",
    "        #Mean of the predictions\n",
    "    print('-' * 80)  \n",
    "    print(f\"\\nOverall mean AUC score : {mean_auc}\")\n",
    "    print(f\"\\nOverall mean F1 score : {np.mean(F1)}\")\n",
    "    print(f\"\\nOverall mean average precision score : {mean_ap}\")\n",
    "    print(f\"\\nOverall mean recall score : {np.mean(Recall)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22a6959a",
   "metadata": {
    "papermill": {
     "duration": 0.058683,
     "end_time": "2022-01-10T23:47:48.557321",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.498638",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Prepare data + undersample in all data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "0fa5892d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:47:48.680799Z",
     "iopub.status.busy": "2022-01-10T23:47:48.680251Z",
     "iopub.status.idle": "2022-01-10T23:47:48.683706Z",
     "shell.execute_reply": "2022-01-10T23:47:48.684112Z",
     "shell.execute_reply.started": "2022-01-10T22:59:33.264189Z"
    },
    "papermill": {
     "duration": 0.068189,
     "end_time": "2022-01-10T23:47:48.684253",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.616064",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3 µs, sys: 1 µs, total: 4 µs\n",
      "Wall time: 6.44 µs\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "#pipe_woe_powertransformer.fit(X,y)\n",
    "#x_pre = pipe_woe_powertransformer_red.transform(X)\n",
    "#x_test_final_pre = pipe_woe_powertransformer_red.transform(X_test_final)\n",
    "#model = make_pipeline(\n",
    " #           scaler,\n",
    "  #          undersampler,\n",
    "   #         model_lgbm2,\n",
    "    #    )\n",
    "#rus = RandomUnderSampler(random_state=42, replacement=True)# fit predictor and target variable\n",
    "#x_rus, y_rus = rus.fit_resample(x, y)\n",
    "\n",
    "#print('original dataset shape:', Counter(y))\n",
    "#print('Resample dataset shape', Counter(y_rus))\n",
    "#del x_final\n",
    "#del x_test_final"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "814e73df",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:47:48.806383Z",
     "iopub.status.busy": "2022-01-10T23:47:48.805812Z",
     "iopub.status.idle": "2022-01-10T23:47:48.809686Z",
     "shell.execute_reply": "2022-01-10T23:47:48.809215Z",
     "shell.execute_reply.started": "2022-01-10T22:59:33.267039Z"
    },
    "papermill": {
     "duration": 0.066774,
     "end_time": "2022-01-10T23:47:48.809794",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.743020",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#predictions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "c053e854",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:47:48.932986Z",
     "iopub.status.busy": "2022-01-10T23:47:48.932421Z",
     "iopub.status.idle": "2022-01-10T23:47:48.936102Z",
     "shell.execute_reply": "2022-01-10T23:47:48.935586Z",
     "shell.execute_reply.started": "2022-01-10T22:59:33.268869Z"
    },
    "papermill": {
     "duration": 0.066926,
     "end_time": "2022-01-10T23:47:48.936246",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.869320",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "e9299564",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-01-10T23:47:49.059588Z",
     "iopub.status.busy": "2022-01-10T23:47:49.058662Z",
     "iopub.status.idle": "2022-01-10T23:47:49.061223Z",
     "shell.execute_reply": "2022-01-10T23:47:49.060729Z",
     "shell.execute_reply.started": "2022-01-10T22:59:33.271941Z"
    },
    "papermill": {
     "duration": 0.06597,
     "end_time": "2022-01-10T23:47:49.061344",
     "exception": false,
     "start_time": "2022-01-10T23:47:48.995374",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "#predictions = np.mean(np.column_stack(predictions), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a450af19",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2021-12-31T12:33:50.392054Z",
     "iopub.status.idle": "2021-12-31T12:33:50.392674Z",
     "shell.execute_reply": "2021-12-31T12:33:50.392439Z",
     "shell.execute_reply.started": "2021-12-31T12:33:50.392415Z"
    },
    "papermill": {
     "duration": 0.061227,
     "end_time": "2022-01-10T23:47:49.186581",
     "exception": false,
     "start_time": "2022-01-10T23:47:49.125354",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "    # Save the predictions to a CSV file\n",
    "    sub = pd.read_csv('../input/GiveMeSomeCredit/sampleEntry.csv')\n",
    "    sub['Probability']=predictions\n",
    "    sub.to_csv('pipe1_model_lgbm_woe_powertransformer2_simple.csv', index=False)\n",
    "    sub"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38850900",
   "metadata": {
    "papermill": {
     "duration": 0.059052,
     "end_time": "2022-01-10T23:47:49.304778",
     "exception": false,
     "start_time": "2022-01-10T23:47:49.245726",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusion: \n",
    "\n",
    "## Advantages and Disadvantage of over-sampling\n",
    "\n",
    "**Advantages**\n",
    "\n",
    "Unlike under-sampling, this method leads to no information loss.\n",
    "Outperforms under sampling\n",
    "\n",
    "**Disadvantages**\n",
    "\n",
    "It increases the likelihood of overfitting since it replicates the minority class events.\n",
    "\n",
    "credit: \n",
    "\n",
    "**Imbalanced:**\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2020/10/improve-class-imbalance-class-weights/\n",
    "\n",
    "https://www.kdnuggets.com/2021/09/imbalanced-classification-without-re-balancing-data.html?fbclid=IwAR0ZS0t-obo3EHV6WLpdd8Tj-ggmHYF5eucmYT_h1khnlQ0uaQBz2PbAvW4\n",
    "\n",
    "https://github.com/dataprofessor/machine-learning-for-imbalanced-data/blob/main/How_to_build_machine_learning_models_for_imbalanced_datasets.ipynb\n",
    "\n",
    "https://elitedatascience.com/imbalanced-classes\n",
    "\n",
    "https://towardsdatascience.com/methods-for-dealing-with-imbalanced-data-5b761be45a18\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2020/07/10-techniques-to-deal-with-class-imbalance-in-machine-learning/\n",
    "\n",
    "https://github.com/DeepSparkChaker/machine-learning-imbalanced-data/tree/master/Section-04-Undersampling\n",
    "\n",
    "https://github.com/DeepSparkChaker/machine-learning-imbalanced-data/blob/master/Section-04-Undersampling/04-12-Under-sampling-method-comparison-Precision-cross-val.ipynb\n",
    "\n",
    "https://github.com/DeepSparkChaker/machine-learning-imbalanced-data/blob/master/Section-05-Oversampling/05-11-Over-sampling-with-cross-validation.ipynb\n",
    "\n",
    "https://www.kaggle.com/shahules/tackling-class-imbalance#3\n",
    "\n",
    "**stacking:**\n",
    "\n",
    "https://towardsdatascience.com/stacking-classifiers-for-higher-predictive-performance-566f963e4840\n",
    "\n",
    "https://www.dataquest.io/blog/introduction-to-ensembles/\n",
    "\n",
    "https://scikit-learn.org/stable/modules/ensemble.html\n",
    "\n",
    "https://machinelearningmastery.com/stacking-ensemble-machine-learning-with-python/\n",
    "\n",
    "\n",
    "https://mlfromscratch.com/model-stacking-explained/#/\n",
    "\n",
    "http://rasbt.github.io/mlxtend/user_guide/classifier/StackingClassifier/\n",
    "\n",
    "http://rasbt.github.io/mlxtend/user_guide/classifier/StackingCVClassifier/\n",
    "\n",
    "\n",
    "https://www.kaggle.com/bannourchaker/7ensembling-stacking-averaging-scratchvssklearn/edit/run/73916270\n",
    "\n",
    "https://www.kaggle.com/bannourchaker/averagingmodels\n",
    "\n",
    "https://mlfromscratch.com/model-stacking-explained/#/\n",
    "\n",
    "https://neptune.ai/blog/ensemble-learning-guide\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2021/08/ensemble-stacking-for-machine-learning-and-deep-learning/\n",
    "\n",
    "https://www.kaggle.com/prashant111/adaboost-classifier-tutorial\n",
    "\n",
    "https://www.kaggle.com/bannourchaker/stacking/edit/run/75840835\n",
    "\n",
    "https://www.kaggle.com/bannourchaker/8-ensembling-blending-scratchvsoop\n",
    "\n",
    "**Bagging:**\n",
    "\n",
    "https://vitalflux.com/bagging-classifier-python-code-example/\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-for-ensemble-models/\n",
    "\n",
    "**complete guide Stacking :**\n",
    "\n",
    "https://neptune.ai/blog/ensemble-learning-guide\n",
    "\n",
    "https://www.analyticsvidhya.com/blog/2021/08/ensemble-stacking-for-machine-learning-and-deep-learning/\n",
    "\n",
    "https://www.kaggle.com/prashant111/adaboost-classifier-tutorial"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 1236.966986,
   "end_time": "2022-01-10T23:47:51.080914",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-01-10T23:27:14.113928",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}