titanic-xgboost-rf-ensemble.ipynb

{"cells":[{"source":"<a href=\"https://www.kaggle.com/code/yeemeitsang/titanic-xgboost-rf-ensemble?scriptVersionId=129648907\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","id":"a435270d","metadata":{"papermill":{"duration":0.010198,"end_time":"2023-05-15T13:25:31.506908","exception":false,"start_time":"2023-05-15T13:25:31.49671","status":"completed"},"tags":[]},"source":["**Introduction**<br>\n","Welcome to my Kaggle notebook, where I'll guide you through building a hybrid model of an XGBoost model and a Random Forest model.\n","\n","The XGBoost model requires no filling in of null values and only needs one-hot encoding for categorical features. On the other hand, the Random Forest model requires imputation of missing values by replacing them with the respective median values, in addition to one-hot encoding.\n","\n","To find the optimal hyperparameters for each model that would minimize the log loss, we conduct grid searches with cross-validation splitting. We use two different methods for grid searching: the XGBoost model uses itertools and a custom function, while the Random Forest model uses Scikit-learn's GridSearchCV.\n","\n","Both models are given equal weight in the hybrid model. With minimal feature engineering, the hybrid model achieves an accuracy of approximately 0.77 on the competition test dataset.\n","\n","If you're interested in exploring other machine learning techniques for this problem, I also have a TensorFlow Keras Sequential model and a Scikit-learn Logistic Regression model in [my GitHub repository](https://github.com/a-t-em/Kaggle-titanic-competition). Feel free to check it out and see how these models compare to the hybrid model presented here."]},{"cell_type":"code","execution_count":1,"id":"96bb5c81","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","execution":{"iopub.execute_input":"2023-05-15T13:25:31.529221Z","iopub.status.busy":"2023-05-15T13:25:31.52844Z","iopub.status.idle":"2023-05-15T13:25:33.42848Z","shell.execute_reply":"2023-05-15T13:25:33.427283Z"},"papermill":{"duration":1.914855,"end_time":"2023-05-15T13:25:33.431715","exception":false,"start_time":"2023-05-15T13:25:31.51686","status":"completed"},"tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["/opt/conda/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5\n","  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion}\"\n"]}],"source":["#import libraries\n","import pandas as pd\n","import numpy as np\n","import xgboost as xgb\n","from sklearn.ensemble import RandomForestClassifier\n","from sklearn.impute import SimpleImputer\n","from sklearn.preprocessing import MinMaxScaler\n","from sklearn.model_selection import train_test_split, KFold, GridSearchCV\n","from sklearn.metrics import make_scorer, log_loss\n","from sklearn.pipeline import Pipeline\n","from sklearn.compose import ColumnTransformer\n","from sklearn.preprocessing import OneHotEncoder\n","import itertools"]},{"cell_type":"markdown","id":"ee935197","metadata":{"papermill":{"duration":0.011152,"end_time":"2023-05-15T13:25:33.452799","exception":false,"start_time":"2023-05-15T13:25:33.441647","status":"completed"},"tags":[]},"source":["**Load and prepare training data**"]},{"cell_type":"code","execution_count":2,"id":"1e1be176","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.475303Z","iopub.status.busy":"2023-05-15T13:25:33.474913Z","iopub.status.idle":"2023-05-15T13:25:33.539763Z","shell.execute_reply":"2023-05-15T13:25:33.538689Z"},"papermill":{"duration":0.079489,"end_time":"2023-05-15T13:25:33.542423","exception":false,"start_time":"2023-05-15T13:25:33.462934","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>PassengerId</th>\n","      <th>Survived</th>\n","      <th>Pclass</th>\n","      <th>Name</th>\n","      <th>Sex</th>\n","      <th>Age</th>\n","      <th>SibSp</th>\n","      <th>Parch</th>\n","      <th>Ticket</th>\n","      <th>Fare</th>\n","      <th>Cabin</th>\n","      <th>Embarked</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>3</td>\n","      <td>Braund, Mr. Owen Harris</td>\n","      <td>male</td>\n","      <td>22.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>A/5 21171</td>\n","      <td>7.2500</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>2</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>Cumings, Mrs. John Bradley (Florence Briggs Th...</td>\n","      <td>female</td>\n","      <td>38.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>PC 17599</td>\n","      <td>71.2833</td>\n","      <td>C85</td>\n","      <td>C</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3</td>\n","      <td>1</td>\n","      <td>3</td>\n","      <td>Heikkinen, Miss. Laina</td>\n","      <td>female</td>\n","      <td>26.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>STON/O2. 3101282</td>\n","      <td>7.9250</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>4</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>Futrelle, Mrs. Jacques Heath (Lily May Peel)</td>\n","      <td>female</td>\n","      <td>35.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>113803</td>\n","      <td>53.1000</td>\n","      <td>C123</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>5</td>\n","      <td>0</td>\n","      <td>3</td>\n","      <td>Allen, Mr. William Henry</td>\n","      <td>male</td>\n","      <td>35.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>373450</td>\n","      <td>8.0500</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   PassengerId  Survived  Pclass  \\\n","0            1         0       3   \n","1            2         1       1   \n","2            3         1       3   \n","3            4         1       1   \n","4            5         0       3   \n","\n","                                                Name     Sex   Age  SibSp  \\\n","0                            Braund, Mr. Owen Harris    male  22.0      1   \n","1  Cumings, Mrs. John Bradley (Florence Briggs Th...  female  38.0      1   \n","2                             Heikkinen, Miss. Laina  female  26.0      0   \n","3       Futrelle, Mrs. Jacques Heath (Lily May Peel)  female  35.0      1   \n","4                           Allen, Mr. William Henry    male  35.0      0   \n","\n","   Parch            Ticket     Fare Cabin Embarked  \n","0      0         A/5 21171   7.2500   NaN        S  \n","1      0          PC 17599  71.2833   C85        C  \n","2      0  STON/O2. 3101282   7.9250   NaN        S  \n","3      0            113803  53.1000  C123        S  \n","4      0            373450   8.0500   NaN        S  "]},"execution_count":2,"metadata":{},"output_type":"execute_result"}],"source":["#load training dataset\n","df_train = pd.read_csv('/kaggle/input/titanic/train.csv')\n","df_train.head()"]},{"cell_type":"code","execution_count":3,"id":"7cde4ad3","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.565183Z","iopub.status.busy":"2023-05-15T13:25:33.56442Z","iopub.status.idle":"2023-05-15T13:25:33.590988Z","shell.execute_reply":"2023-05-15T13:25:33.589828Z"},"papermill":{"duration":0.041067,"end_time":"2023-05-15T13:25:33.593664","exception":false,"start_time":"2023-05-15T13:25:33.552597","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>Pclass</th>\n","      <th>Sex</th>\n","      <th>Age</th>\n","      <th>SibSp</th>\n","      <th>Parch</th>\n","      <th>Fare</th>\n","      <th>Embarked</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>3</td>\n","      <td>male</td>\n","      <td>22.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>7.2500</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>1</td>\n","      <td>female</td>\n","      <td>38.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>71.2833</td>\n","      <td>C</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>3</td>\n","      <td>female</td>\n","      <td>26.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>7.9250</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>1</td>\n","      <td>female</td>\n","      <td>35.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>53.1000</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>3</td>\n","      <td>male</td>\n","      <td>35.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>8.0500</td>\n","      <td>S</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   Pclass     Sex   Age  SibSp  Parch     Fare Embarked\n","0       3    male  22.0      1      0   7.2500        S\n","1       1  female  38.0      1      0  71.2833        C\n","2       3  female  26.0      0      0   7.9250        S\n","3       1  female  35.0      1      0  53.1000        S\n","4       3    male  35.0      0      0   8.0500        S"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["0    0\n","1    1\n","2    1\n","3    1\n","4    0\n","Name: Survived, dtype: int64"]},"execution_count":3,"metadata":{},"output_type":"execute_result"}],"source":["#split data into features and targets\n","X_train = df_train.drop(['Survived', 'PassengerId', 'Name', 'Ticket', 'Cabin'], axis=1)\n","y_train = df_train.Survived\n","#observe the first few rows of each\n","display(X_train.head())\n","y_train[:5]"]},{"cell_type":"code","execution_count":4,"id":"0c1073c7","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.61742Z","iopub.status.busy":"2023-05-15T13:25:33.616774Z","iopub.status.idle":"2023-05-15T13:25:33.621951Z","shell.execute_reply":"2023-05-15T13:25:33.621064Z"},"papermill":{"duration":0.019886,"end_time":"2023-05-15T13:25:33.624273","exception":false,"start_time":"2023-05-15T13:25:33.604387","status":"completed"},"tags":[]},"outputs":[],"source":["#define the categorical and numerical features for preprocessing\n","cat_features = ['Sex', 'Embarked']\n","num_features = ['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']"]},{"cell_type":"markdown","id":"c7c335d2","metadata":{"papermill":{"duration":0.010439,"end_time":"2023-05-15T13:25:33.645399","exception":false,"start_time":"2023-05-15T13:25:33.63496","status":"completed"},"tags":[]},"source":["**Build xgboost model**"]},{"cell_type":"markdown","id":"9d6aa26a","metadata":{"papermill":{"duration":0.01061,"end_time":"2023-05-15T13:25:33.666676","exception":false,"start_time":"2023-05-15T13:25:33.656066","status":"completed"},"tags":[]},"source":["- preprocessing"]},{"cell_type":"code","execution_count":5,"id":"0524163d","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.690216Z","iopub.status.busy":"2023-05-15T13:25:33.689586Z","iopub.status.idle":"2023-05-15T13:25:33.694913Z","shell.execute_reply":"2023-05-15T13:25:33.694069Z"},"papermill":{"duration":0.019873,"end_time":"2023-05-15T13:25:33.697243","exception":false,"start_time":"2023-05-15T13:25:33.67737","status":"completed"},"tags":[]},"outputs":[],"source":["#define the column transformer to one-hot encode categorical features\n","xgb_preprocessor = ColumnTransformer(\n","    transformers=[\n","        ('cat', OneHotEncoder(handle_unknown='ignore'), cat_features)],\n","    remainder='passthrough'\n",")"]},{"cell_type":"markdown","id":"418f5214","metadata":{"papermill":{"duration":0.010363,"end_time":"2023-05-15T13:25:33.718396","exception":false,"start_time":"2023-05-15T13:25:33.708033","status":"completed"},"tags":[]},"source":["- conduct grid search to find optimal hyperparameters"]},{"cell_type":"code","execution_count":6,"id":"b0241f7f","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.741983Z","iopub.status.busy":"2023-05-15T13:25:33.741341Z","iopub.status.idle":"2023-05-15T13:25:33.747765Z","shell.execute_reply":"2023-05-15T13:25:33.74693Z"},"papermill":{"duration":0.021128,"end_time":"2023-05-15T13:25:33.750131","exception":false,"start_time":"2023-05-15T13:25:33.729003","status":"completed"},"tags":[]},"outputs":[],"source":["#define parameter space\n","param_space = {\n","    'max_depth': [3, 5, 7],\n","    'min_child_weight': [1.0, 2.0, 4.0],\n","    'n_estimators': [10, 50, 100],\n","    'learning_rate': [0.01, 0.1],\n","}\n","\n","#get all possible combinations using itertools\n","param_combinations = itertools.product(param_space['max_depth'], \n","                                       param_space['min_child_weight'], \n","                                       param_space['n_estimators'], \n","                                       param_space['learning_rate'])"]},{"cell_type":"code","execution_count":7,"id":"165c7b11","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:33.773533Z","iopub.status.busy":"2023-05-15T13:25:33.772879Z","iopub.status.idle":"2023-05-15T13:25:51.107645Z","shell.execute_reply":"2023-05-15T13:25:51.1067Z"},"papermill":{"duration":17.3496,"end_time":"2023-05-15T13:25:51.110349","exception":false,"start_time":"2023-05-15T13:25:33.760749","status":"completed"},"tags":[]},"outputs":[],"source":["params = []\n","scores = []\n","\n","#loop over all combinations and calculate their log losses\n","for max_depth, min_child_weight, n_estimators, learning_rate in param_combinations:\n","    score_folds = []\n","    #perform cross validation\n","    kf = KFold(n_splits=5, shuffle=True)\n","    #split data into training and validation data for each fold\n","    for tr_idx, va_idx in kf.split(X_train):\n","        tr_x, va_x = X_train.iloc[tr_idx], X_train.iloc[va_idx]\n","        tr_y, va_y = y_train.iloc[tr_idx], y_train.iloc[va_idx]\n","        #use pipeline to facilitate processing\n","        pipeline = Pipeline([\n","                        ('preprocessor', xgb_preprocessor),\n","                        ('model', xgb.XGBClassifier(n_estimators=n_estimators, \\\n","                                                    max_depth=max_depth, \\\n","                                                    min_child_weight=min_child_weight, \\\n","                                                    learning_rate=learning_rate))\n","                   ])\n","        pipeline.fit(tr_x, tr_y)\n","        #predict on validation data\n","        va_pred = pipeline.predict_proba(va_x)[:, 1]\n","        #calculate log loss\n","        logloss = log_loss(va_y, va_pred)\n","        score_folds.append(logloss)\n","    \n","    #take the average score across all folds for each combination\n","    score_mean = np.mean(score_folds)\n","    #record the params and the average score\n","    params.append((max_depth, min_child_weight, n_estimators, learning_rate))\n","    scores.append(score_mean)"]},{"cell_type":"code","execution_count":8,"id":"1cf920f2","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.135438Z","iopub.status.busy":"2023-05-15T13:25:51.134966Z","iopub.status.idle":"2023-05-15T13:25:51.145887Z","shell.execute_reply":"2023-05-15T13:25:51.144641Z"},"papermill":{"duration":0.025988,"end_time":"2023-05-15T13:25:51.148701","exception":false,"start_time":"2023-05-15T13:25:51.122713","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["0.40799503066353837"]},"metadata":{},"output_type":"display_data"},{"data":{"text/plain":["(5, 4.0, 50, 0.1)"]},"metadata":{},"output_type":"display_data"}],"source":["#show the minimum log loss from all parameter combinations\n","display(np.array(scores).min())\n","best_idx = np.argsort(scores)[0]\n","#get the parameters that minimizes log loss\n","xgb_best_param = params[best_idx]\n","#view best parameters\n","display(xgb_best_param)\n","#get params stored in tuple as variables\n","max_depth, min_child_weight, n_estimators, learning_rate = xgb_best_param"]},{"cell_type":"markdown","id":"659cf828","metadata":{"papermill":{"duration":0.010898,"end_time":"2023-05-15T13:25:51.170721","exception":false,"start_time":"2023-05-15T13:25:51.159823","status":"completed"},"tags":[]},"source":["- finalize model"]},{"cell_type":"code","execution_count":9,"id":"0ca94bf7","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.194796Z","iopub.status.busy":"2023-05-15T13:25:51.194398Z","iopub.status.idle":"2023-05-15T13:25:51.294609Z","shell.execute_reply":"2023-05-15T13:25:51.293405Z"},"papermill":{"duration":0.11541,"end_time":"2023-05-15T13:25:51.297218","exception":false,"start_time":"2023-05-15T13:25:51.181808","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n","                                   transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;])])),\n","                (&#x27;model&#x27;,\n","                 XGBClassifier(base_score=None, booster=None, callbacks=None,\n","                               colsample_bylevel=None, colsample_bynode=None,\n","                               colsample_bytree=None,\n","                               early_stopping_rounds=None,\n","                               enable_categorical=False, eval_me...\n","                               feature_types=None, gamma=None, gpu_id=None,\n","                               grow_policy=None, importance_type=None,\n","                               interaction_constraints=None, learning_rate=0.1,\n","                               max_bin=None, max_cat_threshold=None,\n","                               max_cat_to_onehot=None, max_delta_step=None,\n","                               max_depth=5, max_leaves=None,\n","                               min_child_weight=4.0, missing=nan,\n","                               monotone_constraints=None, n_estimators=50,\n","                               n_jobs=None, num_parallel_tree=None,\n","                               predictor=None, random_state=None, ...))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n","                                   transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;])])),\n","                (&#x27;model&#x27;,\n","                 XGBClassifier(base_score=None, booster=None, callbacks=None,\n","                               colsample_bylevel=None, colsample_bynode=None,\n","                               colsample_bytree=None,\n","                               early_stopping_rounds=None,\n","                               enable_categorical=False, eval_me...\n","                               feature_types=None, gamma=None, gpu_id=None,\n","                               grow_policy=None, importance_type=None,\n","                               interaction_constraints=None, learning_rate=0.1,\n","                               max_bin=None, max_cat_threshold=None,\n","                               max_cat_to_onehot=None, max_delta_step=None,\n","                               max_depth=5, max_leaves=None,\n","                               min_child_weight=4.0, missing=nan,\n","                               monotone_constraints=None, n_estimators=50,\n","                               n_jobs=None, num_parallel_tree=None,\n","                               predictor=None, random_state=None, ...))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n","                  transformers=[(&#x27;cat&#x27;, OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                 [&#x27;Sex&#x27;, &#x27;Embarked&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">cat</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Sex&#x27;, &#x27;Embarked&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">remainder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;, &#x27;Parch&#x27;, &#x27;Fare&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">XGBClassifier</label><div class=\"sk-toggleable__content\"><pre>XGBClassifier(base_score=None, booster=None, callbacks=None,\n","              colsample_bylevel=None, colsample_bynode=None,\n","              colsample_bytree=None, early_stopping_rounds=None,\n","              enable_categorical=False, eval_metric=None, feature_types=None,\n","              gamma=None, gpu_id=None, grow_policy=None, importance_type=None,\n","              interaction_constraints=None, learning_rate=0.1, max_bin=None,\n","              max_cat_threshold=None, max_cat_to_onehot=None,\n","              max_delta_step=None, max_depth=5, max_leaves=None,\n","              min_child_weight=4.0, missing=nan, monotone_constraints=None,\n","              n_estimators=50, n_jobs=None, num_parallel_tree=None,\n","              predictor=None, random_state=None, ...)</pre></div></div></div></div></div></div></div>"],"text/plain":["Pipeline(steps=[('preprocessor',\n","                 ColumnTransformer(remainder='passthrough',\n","                                   transformers=[('cat',\n","                                                  OneHotEncoder(handle_unknown='ignore'),\n","                                                  ['Sex', 'Embarked'])])),\n","                ('model',\n","                 XGBClassifier(base_score=None, booster=None, callbacks=None,\n","                               colsample_bylevel=None, colsample_bynode=None,\n","                               colsample_bytree=None,\n","                               early_stopping_rounds=None,\n","                               enable_categorical=False, eval_me...\n","                               feature_types=None, gamma=None, gpu_id=None,\n","                               grow_policy=None, importance_type=None,\n","                               interaction_constraints=None, learning_rate=0.1,\n","                               max_bin=None, max_cat_threshold=None,\n","                               max_cat_to_onehot=None, max_delta_step=None,\n","                               max_depth=5, max_leaves=None,\n","                               min_child_weight=4.0, missing=nan,\n","                               monotone_constraints=None, n_estimators=50,\n","                               n_jobs=None, num_parallel_tree=None,\n","                               predictor=None, random_state=None, ...))])"]},"execution_count":9,"metadata":{},"output_type":"execute_result"}],"source":["# plug the best params values back into the model\n","xgb_pipeline = Pipeline([\n","                        ('preprocessor', xgb_preprocessor),\n","                        ('model', xgb.XGBClassifier(n_estimators=n_estimators, \\\n","                                                    max_depth=max_depth, \\\n","                                                    min_child_weight=min_child_weight, \n","                                                    learning_rate=learning_rate))\n","                   ])\n","#fit pipeline\n","xgb_pipeline.fit(X_train, y_train)"]},{"cell_type":"markdown","id":"ebca9e5c","metadata":{"papermill":{"duration":0.011119,"end_time":"2023-05-15T13:25:51.320227","exception":false,"start_time":"2023-05-15T13:25:51.309108","status":"completed"},"tags":[]},"source":["**Build random forest model**"]},{"cell_type":"markdown","id":"f5d92e20","metadata":{"papermill":{"duration":0.011228,"end_time":"2023-05-15T13:25:51.342944","exception":false,"start_time":"2023-05-15T13:25:51.331716","status":"completed"},"tags":[]},"source":["- preprocessing"]},{"cell_type":"code","execution_count":10,"id":"2bda61eb","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.371026Z","iopub.status.busy":"2023-05-15T13:25:51.370175Z","iopub.status.idle":"2023-05-15T13:25:51.380898Z","shell.execute_reply":"2023-05-15T13:25:51.379434Z"},"papermill":{"duration":0.030068,"end_time":"2023-05-15T13:25:51.384365","exception":false,"start_time":"2023-05-15T13:25:51.354297","status":"completed"},"tags":[]},"outputs":[],"source":["#fill in missing values and normalize them\n","minmax_transformer = Pipeline(steps=[\n","    ('imputer', SimpleImputer(strategy='median')),\n","    ('scaler', MinMaxScaler())\n","])\n","\n","#bundle together with one hot encoding for categorical values \n","rf_preprocessor = ColumnTransformer(\n","    transformers=[\n","        ('cat', OneHotEncoder(handle_unknown='ignore'), cat_features),\n","        ('minmax', minmax_transformer, num_features)]\n",")"]},{"cell_type":"markdown","id":"6aa89af8","metadata":{"papermill":{"duration":0.014844,"end_time":"2023-05-15T13:25:51.412814","exception":false,"start_time":"2023-05-15T13:25:51.39797","status":"completed"},"tags":[]},"source":["- conduct grid search to find optimal hyperparameters"]},{"cell_type":"code","execution_count":11,"id":"0a7605cc","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.446472Z","iopub.status.busy":"2023-05-15T13:25:51.4452Z","iopub.status.idle":"2023-05-15T13:25:51.451705Z","shell.execute_reply":"2023-05-15T13:25:51.450521Z"},"papermill":{"duration":0.026521,"end_time":"2023-05-15T13:25:51.454611","exception":false,"start_time":"2023-05-15T13:25:51.42809","status":"completed"},"tags":[]},"outputs":[],"source":["#define parameter space\n","param_grid = {\n","    'classifier__n_estimators': [50, 100, 200],\n","    'classifier__max_depth': [5, 10, 15]\n","}"]},{"cell_type":"code","execution_count":12,"id":"6f2704dd","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.484798Z","iopub.status.busy":"2023-05-15T13:25:51.48349Z","iopub.status.idle":"2023-05-15T13:25:51.854596Z","shell.execute_reply":"2023-05-15T13:25:51.85348Z"},"papermill":{"duration":0.386809,"end_time":"2023-05-15T13:25:51.857289","exception":false,"start_time":"2023-05-15T13:25:51.47048","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                                 (&#x27;minmax&#x27;,\n","                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                                  (&#x27;scaler&#x27;,\n","                                                                   MinMaxScaler())]),\n","                                                  [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;,\n","                                                   &#x27;Parch&#x27;, &#x27;Fare&#x27;])])),\n","                (&#x27;classifier&#x27;, RandomForestClassifier())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                                 (&#x27;minmax&#x27;,\n","                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                                  (&#x27;scaler&#x27;,\n","                                                                   MinMaxScaler())]),\n","                                                  [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;,\n","                                                   &#x27;Parch&#x27;, &#x27;Fare&#x27;])])),\n","                (&#x27;classifier&#x27;, RandomForestClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;cat&#x27;, OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                 [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                (&#x27;minmax&#x27;,\n","                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                 (&#x27;scaler&#x27;, MinMaxScaler())]),\n","                                 [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;, &#x27;Parch&#x27;, &#x27;Fare&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">cat</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Sex&#x27;, &#x27;Embarked&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">minmax</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;, &#x27;Parch&#x27;, &#x27;Fare&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier()</pre></div></div></div></div></div></div></div>"],"text/plain":["Pipeline(steps=[('preprocessor',\n","                 ColumnTransformer(transformers=[('cat',\n","                                                  OneHotEncoder(handle_unknown='ignore'),\n","                                                  ['Sex', 'Embarked']),\n","                                                 ('minmax',\n","                                                  Pipeline(steps=[('imputer',\n","                                                                   SimpleImputer(strategy='median')),\n","                                                                  ('scaler',\n","                                                                   MinMaxScaler())]),\n","                                                  ['Pclass', 'Age', 'SibSp',\n","                                                   'Parch', 'Fare'])])),\n","                ('classifier', RandomForestClassifier())])"]},"execution_count":12,"metadata":{},"output_type":"execute_result"}],"source":["#define model and pipeline\n","model_rf = RandomForestClassifier()\n","rf_pipeline = Pipeline(steps=[\n","    ('preprocessor', rf_preprocessor),\n","    ('classifier', model_rf)\n","])\n","rf_pipeline.fit(X_train, y_train)"]},{"cell_type":"code","execution_count":13,"id":"74dedf55","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:25:51.883638Z","iopub.status.busy":"2023-05-15T13:25:51.88319Z","iopub.status.idle":"2023-05-15T13:26:07.344949Z","shell.execute_reply":"2023-05-15T13:26:07.343742Z"},"papermill":{"duration":15.478424,"end_time":"2023-05-15T13:26:07.347932","exception":false,"start_time":"2023-05-15T13:25:51.869508","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["({'classifier__max_depth': 5, 'classifier__n_estimators': 50},\n"," -0.4181848361259025)"]},"execution_count":13,"metadata":{},"output_type":"execute_result"}],"source":["#create custom scorer for negative log loss\n","scorer = make_scorer(log_loss, greater_is_better=False, needs_proba=True)\n","#perform grid search with 5 fold cross validation\n","grid_search = GridSearchCV(rf_pipeline, param_grid=param_grid, cv=5, scoring=scorer)\n","grid_search.fit(X_train, y_train)\n","#view the best params and the best score \n","grid_search.best_params_, grid_search.best_score_"]},{"cell_type":"code","execution_count":14,"id":"5ee5c247","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:07.377882Z","iopub.status.busy":"2023-05-15T13:26:07.377482Z","iopub.status.idle":"2023-05-15T13:26:07.384844Z","shell.execute_reply":"2023-05-15T13:26:07.383818Z"},"papermill":{"duration":0.024787,"end_time":"2023-05-15T13:26:07.387219","exception":false,"start_time":"2023-05-15T13:26:07.362432","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["(50, 5)"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["#get best params as variables\n","n_estimators = grid_search.best_params_['classifier__n_estimators']\n","max_depth = grid_search.best_params_['classifier__max_depth']\n","n_estimators, max_depth"]},{"cell_type":"markdown","id":"9a806170","metadata":{"papermill":{"duration":0.011942,"end_time":"2023-05-15T13:26:07.411437","exception":false,"start_time":"2023-05-15T13:26:07.399495","status":"completed"},"tags":[]},"source":["- finalize model"]},{"cell_type":"code","execution_count":15,"id":"d3751fbf","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:07.437648Z","iopub.status.busy":"2023-05-15T13:26:07.437228Z","iopub.status.idle":"2023-05-15T13:26:07.795649Z","shell.execute_reply":"2023-05-15T13:26:07.794597Z"},"papermill":{"duration":0.37511,"end_time":"2023-05-15T13:26:07.798743","exception":false,"start_time":"2023-05-15T13:26:07.423633","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                                 (&#x27;minmax&#x27;,\n","                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                                  (&#x27;scaler&#x27;,\n","                                                                   MinMaxScaler())]),\n","                                                  [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;,\n","                                                   &#x27;Parch&#x27;, &#x27;Fare&#x27;])])),\n","                (&#x27;classifier&#x27;, RandomForestClassifier())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n","                 ColumnTransformer(transformers=[(&#x27;cat&#x27;,\n","                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                                  [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                                 (&#x27;minmax&#x27;,\n","                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                                  (&#x27;scaler&#x27;,\n","                                                                   MinMaxScaler())]),\n","                                                  [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;,\n","                                                   &#x27;Parch&#x27;, &#x27;Fare&#x27;])])),\n","                (&#x27;classifier&#x27;, RandomForestClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;cat&#x27;, OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n","                                 [&#x27;Sex&#x27;, &#x27;Embarked&#x27;]),\n","                                (&#x27;minmax&#x27;,\n","                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n","                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n","                                                 (&#x27;scaler&#x27;, MinMaxScaler())]),\n","                                 [&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;, &#x27;Parch&#x27;, &#x27;Fare&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">cat</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Sex&#x27;, &#x27;Embarked&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">minmax</label><div class=\"sk-toggleable__content\"><pre>[&#x27;Pclass&#x27;, &#x27;Age&#x27;, &#x27;SibSp&#x27;, &#x27;Parch&#x27;, &#x27;Fare&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">MinMaxScaler</label><div class=\"sk-toggleable__content\"><pre>MinMaxScaler()</pre></div></div></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier()</pre></div></div></div></div></div></div></div>"],"text/plain":["Pipeline(steps=[('preprocessor',\n","                 ColumnTransformer(transformers=[('cat',\n","                                                  OneHotEncoder(handle_unknown='ignore'),\n","                                                  ['Sex', 'Embarked']),\n","                                                 ('minmax',\n","                                                  Pipeline(steps=[('imputer',\n","                                                                   SimpleImputer(strategy='median')),\n","                                                                  ('scaler',\n","                                                                   MinMaxScaler())]),\n","                                                  ['Pclass', 'Age', 'SibSp',\n","                                                   'Parch', 'Fare'])])),\n","                ('classifier', RandomForestClassifier())])"]},"execution_count":15,"metadata":{},"output_type":"execute_result"}],"source":["#plug best params back into the model\n","model_rf = RandomForestClassifier(n_estimators=n_estimators, max_depth=max_depth)\n","rf_pipeline.fit(X_train, y_train)"]},{"cell_type":"markdown","id":"42875641","metadata":{"papermill":{"duration":0.01218,"end_time":"2023-05-15T13:26:07.823545","exception":false,"start_time":"2023-05-15T13:26:07.811365","status":"completed"},"tags":[]},"source":["**Predict on test dataset**"]},{"cell_type":"code","execution_count":16,"id":"5a089fab","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:07.851362Z","iopub.status.busy":"2023-05-15T13:26:07.850677Z","iopub.status.idle":"2023-05-15T13:26:07.876354Z","shell.execute_reply":"2023-05-15T13:26:07.875186Z"},"papermill":{"duration":0.042978,"end_time":"2023-05-15T13:26:07.879003","exception":false,"start_time":"2023-05-15T13:26:07.836025","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>PassengerId</th>\n","      <th>Pclass</th>\n","      <th>Name</th>\n","      <th>Sex</th>\n","      <th>Age</th>\n","      <th>SibSp</th>\n","      <th>Parch</th>\n","      <th>Ticket</th>\n","      <th>Fare</th>\n","      <th>Cabin</th>\n","      <th>Embarked</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>892</td>\n","      <td>3</td>\n","      <td>Kelly, Mr. James</td>\n","      <td>male</td>\n","      <td>34.5</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>330911</td>\n","      <td>7.8292</td>\n","      <td>NaN</td>\n","      <td>Q</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>893</td>\n","      <td>3</td>\n","      <td>Wilkes, Mrs. James (Ellen Needs)</td>\n","      <td>female</td>\n","      <td>47.0</td>\n","      <td>1</td>\n","      <td>0</td>\n","      <td>363272</td>\n","      <td>7.0000</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>894</td>\n","      <td>2</td>\n","      <td>Myles, Mr. Thomas Francis</td>\n","      <td>male</td>\n","      <td>62.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>240276</td>\n","      <td>9.6875</td>\n","      <td>NaN</td>\n","      <td>Q</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>895</td>\n","      <td>3</td>\n","      <td>Wirz, Mr. Albert</td>\n","      <td>male</td>\n","      <td>27.0</td>\n","      <td>0</td>\n","      <td>0</td>\n","      <td>315154</td>\n","      <td>8.6625</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>896</td>\n","      <td>3</td>\n","      <td>Hirvonen, Mrs. Alexander (Helga E Lindqvist)</td>\n","      <td>female</td>\n","      <td>22.0</td>\n","      <td>1</td>\n","      <td>1</td>\n","      <td>3101298</td>\n","      <td>12.2875</td>\n","      <td>NaN</td>\n","      <td>S</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   PassengerId  Pclass                                          Name     Sex  \\\n","0          892       3                              Kelly, Mr. James    male   \n","1          893       3              Wilkes, Mrs. James (Ellen Needs)  female   \n","2          894       2                     Myles, Mr. Thomas Francis    male   \n","3          895       3                              Wirz, Mr. Albert    male   \n","4          896       3  Hirvonen, Mrs. Alexander (Helga E Lindqvist)  female   \n","\n","    Age  SibSp  Parch   Ticket     Fare Cabin Embarked  \n","0  34.5      0      0   330911   7.8292   NaN        Q  \n","1  47.0      1      0   363272   7.0000   NaN        S  \n","2  62.0      0      0   240276   9.6875   NaN        Q  \n","3  27.0      0      0   315154   8.6625   NaN        S  \n","4  22.0      1      1  3101298  12.2875   NaN        S  "]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["#load test dataset\n","df_test = pd.read_csv('/kaggle/input/titanic/test.csv')\n","df_test.head()"]},{"cell_type":"code","execution_count":17,"id":"4ea3710f","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:07.906608Z","iopub.status.busy":"2023-05-15T13:26:07.906191Z","iopub.status.idle":"2023-05-15T13:26:07.924655Z","shell.execute_reply":"2023-05-15T13:26:07.923391Z"},"papermill":{"duration":0.035287,"end_time":"2023-05-15T13:26:07.927238","exception":false,"start_time":"2023-05-15T13:26:07.891951","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["array([0.06452899, 0.29458043, 0.08683947, 0.1409582 , 0.4075573 ],\n","      dtype=float32)"]},"execution_count":17,"metadata":{},"output_type":"execute_result"}],"source":["#predict with xgboost model\n","xgb_preds = xgb_pipeline.predict_proba(df_test)[:, 1]\n","#view first few prediction probabilities\n","xgb_preds[:5]"]},{"cell_type":"code","execution_count":18,"id":"1d4a1310","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:07.956304Z","iopub.status.busy":"2023-05-15T13:26:07.955916Z","iopub.status.idle":"2023-05-15T13:26:07.990122Z","shell.execute_reply":"2023-05-15T13:26:07.988911Z"},"papermill":{"duration":0.051233,"end_time":"2023-05-15T13:26:07.992859","exception":false,"start_time":"2023-05-15T13:26:07.941626","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["array([0.06, 0.21, 0.17, 0.67, 0.58])"]},"execution_count":18,"metadata":{},"output_type":"execute_result"}],"source":["#predict with random forest model\n","rf_preds = rf_pipeline.predict_proba(df_test)[:, 1]\n","#view first few prediction probabilities\n","rf_preds[:5]"]},{"cell_type":"code","execution_count":19,"id":"03ae9192","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:08.021487Z","iopub.status.busy":"2023-05-15T13:26:08.020507Z","iopub.status.idle":"2023-05-15T13:26:08.027855Z","shell.execute_reply":"2023-05-15T13:26:08.026834Z"},"papermill":{"duration":0.024174,"end_time":"2023-05-15T13:26:08.030193","exception":false,"start_time":"2023-05-15T13:26:08.006019","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["array([0, 0, 0, 0, 0])"]},"execution_count":19,"metadata":{},"output_type":"execute_result"}],"source":["#set weights for each model and combine results \n","pred = xgb_preds*0.5 + rf_preds*0.5\n","#set threshold and derive labels\n","pred_label = np.where(pred > 0.5, 1, 0)\n","#view first few labels\n","pred_label[:5]"]},{"cell_type":"markdown","id":"65fb50d7","metadata":{"papermill":{"duration":0.013045,"end_time":"2023-05-15T13:26:08.056687","exception":false,"start_time":"2023-05-15T13:26:08.043642","status":"completed"},"tags":[]},"source":["**Prepare submission file**"]},{"cell_type":"code","execution_count":20,"id":"8e577fb8","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:08.085297Z","iopub.status.busy":"2023-05-15T13:26:08.084657Z","iopub.status.idle":"2023-05-15T13:26:08.101096Z","shell.execute_reply":"2023-05-15T13:26:08.100038Z"},"papermill":{"duration":0.033834,"end_time":"2023-05-15T13:26:08.103736","exception":false,"start_time":"2023-05-15T13:26:08.069902","status":"completed"},"tags":[]},"outputs":[{"data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>PassengerId</th>\n","      <th>Survived</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>892</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>893</td>\n","      <td>1</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>894</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>895</td>\n","      <td>0</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>896</td>\n","      <td>1</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["   PassengerId  Survived\n","0          892         0\n","1          893         1\n","2          894         0\n","3          895         0\n","4          896         1"]},"execution_count":20,"metadata":{},"output_type":"execute_result"}],"source":["#load sample file\n","df_sub = pd.read_csv('/kaggle/input/titanic/gender_submission.csv')\n","df_sub.head()"]},{"cell_type":"code","execution_count":21,"id":"f4e872ad","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:08.132387Z","iopub.status.busy":"2023-05-15T13:26:08.131957Z","iopub.status.idle":"2023-05-15T13:26:08.144538Z","shell.execute_reply":"2023-05-15T13:26:08.143786Z"},"papermill":{"duration":0.029518,"end_time":"2023-05-15T13:26:08.146694","exception":false,"start_time":"2023-05-15T13:26:08.117176","status":"completed"},"tags":[]},"outputs":[{"data":{"text/plain":["count    418.000000\n","mean       0.337321\n","std        0.473362\n","min        0.000000\n","25%        0.000000\n","50%        0.000000\n","75%        1.000000\n","max        1.000000\n","Name: Survived, dtype: float64"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["#replace target column with predicted labels\n","df_sub['Survived'] = pred_label\n","#view stats to see if they make sense\n","df_sub.Survived.describe()"]},{"cell_type":"code","execution_count":22,"id":"fa3340c3","metadata":{"execution":{"iopub.execute_input":"2023-05-15T13:26:08.175954Z","iopub.status.busy":"2023-05-15T13:26:08.175262Z","iopub.status.idle":"2023-05-15T13:26:08.186288Z","shell.execute_reply":"2023-05-15T13:26:08.185401Z"},"papermill":{"duration":0.028557,"end_time":"2023-05-15T13:26:08.188748","exception":false,"start_time":"2023-05-15T13:26:08.160191","status":"completed"},"tags":[]},"outputs":[],"source":["#compile csv file for submission\n","df_sub.to_csv('submission.csv', index=False)"]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.10"},"papermill":{"default_parameters":{},"duration":50.533045,"end_time":"2023-05-15T13:26:09.224808","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2023-05-15T13:25:18.691763","version":"2.4.0"}},"nbformat":4,"nbformat_minor":5}