demo_titanic_data_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e732f27e",
   "metadata": {},
   "source": [
    "<h1 style=\"text-align:center\">Data Preparation</h1>\n",
    "\n",
    "<h4 style=\"text-align:center\"><a href = 'https://github.com/virchan' style='text-align:center'>https://github.com/virchan</a></h4> \n",
    "\n",
    "<h2>Abstract</h2>\n",
    "\n",
    "Welcome to this notebook, which serves as the first part of the author's portfolio project on predictive modelling. Here, we embark on the initial stages of the workflow, including data acquisition, exploratory data analysis, and data preprocessing. The primary outcome of this notebook is the production of a cleaned data set, which will be utilized in the subsequent stages of machine learning model training.\n",
    "\n",
    "<h2>Introduction</h2>\n",
    "\n",
    "We are utilizing the Kaggle Getting Started competition __Titanic - Machine Learning from Disaster ([Kaggle link](https://www.kaggle.com/competitions/titanic/))__ as a means to simulate the workflow of predictive modeling. With the provided data set and background information from Kaggle, we aim to illustrate the early stages of the predictive modeling workflow, focusing on data acquisition, data preprocessing, and particularly emphasizing exploratory data analysis. The final result of our work in this notebook is the cleaned version of the original data set, stored in the `csv/train_cleaned.csv` and `csv/test_cleaned.csv` files. These cleaned data sets will be used in the subsequent model training phase and model evaluation phase, as outlined in the `demo_ML_models.ipynb` and `demo_model_evaluation.ipynb` files, respectively.\n",
    "\n",
    "<h2>Data Acquisition</h2>\n",
    "\n",
    "The dataset can be accessed using the following shell command:\n",
    "\n",
    "<img src = \"images/[code_output][001][data_acquisition].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "Alternatively, it can also be accessed through the `csv/train.csv` and `csv/test.csv` files. The former contains samples for machine learning model training, while the latter is the testing set, containing the same information as the former except for the true labels. We load the training and testing sets into the dataframes `train_data` and `test_data`, respectively. Additionally, we concatenate them to form the `all_df` dataframe to facilitate further analysis.\n",
    "\n",
    "<h2>Exploratory Data Analysis (EDA)</h2>\n",
    "\n",
    "Performing exploratory data analysis (EDA) is crucial in ensuring the quality and reliability of a dataset. It allows us to identify and address potential data flaws, gain valuable insights, and prepare the dataset for subsequent data-driven decision-making and machine learning model training.  In this section, we conduct EDA on the `all_df` dataset with the objective of preparing it for the upcoming machine learning model training phase. Through EDA, we have identified several issues that require attention, including data sparsity, data quality, and data redundancy. To ensure accurate and optimal results, we provide recommendations to address or mitigate these issues.\n",
    "\n",
    "We begin with a quick inspection using the `.info()` method in the `pandas` library to gain an overview of the dataset. The dataset consists of 1309 samples (rows) and 11 features (columns). Here is a summary of the important features:\n",
    "\n",
    "<img src = \"images/[code_output][025][all_df_info].png\" width=\"300\" height=\"300\"/>\n",
    "\n",
    "The most crucial feature is the true label, indicated by the `Survived` column, which signifies whether a passenger survived or not. Out of the 1309 samples, 891 belong to the training set, with corresponding `Survived` labels. The dataset also includes passenger information such as name (`Name`), sex (`Sex`), age (`Age`), the number of siblings/spouses aboard the Titanic (`SibSp`), the number of parents/children aboard (`Parch`), the port of embarkation (`Embarked`), as well as transaction details such as the passenger's cabin number on the Titanic (`Cabin`), the fare price (`Fare`), the ticket number (`Ticket`), and the ticket class (`Pclass`).\n",
    "\n",
    "Upon inspection, we observe missing values in the dataset, particularly in the `Age`, `Fare`, and `Cabin` columns. One approach to handling missing data is to simply drop the rows with missing values. However, in our case, dismissing the missing values would significantly reduce the dataset size. Notably, the `Cabin` column exhibits a high sparsity of 78%, indicating that dropping missing values could lead to retaining only 22% of the original dataset. Sacrificing 78% of the data warrants careful consideration. Therefore, it is imperative to examine the sparsity of these columns in-depth before making a decision.\n",
    "\n",
    "<h3><code>Embarked</code> Column Missing Values</h3>\n",
    "\n",
    "The column `Embarked` has only two missing values:\n",
    "\n",
    "<img src=\"images/[code_output][008][Embarked_column_missing_values].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "After conducting further research on the passengers, it has been confirmed that both individuals boarded at Southampton (references: <a href = \"https://www.encyclopedia-titanica.org/titanic-survivor/martha-evelyn-stone.html\">source 1</a> and <a href = \"https://www.encyclopedia-titanica.org/titanic-survivor/amelia-icard.html\">source 2</a>). Therefore, we will replace the missing values with the information provided by these third-party sources.\n",
    "\n",
    "<h3>Cleaning the <code>Cabin</code> Column</h3>\n",
    "\n",
    "The`Cabin` column presents several issues:\n",
    "\n",
    "\"Sparsity in the Cabin Column\" | \"Dirty Values in the Cabin Column\" | \"Multiple Decks\"\n",
    ":-------------------------:|:-------------------------:|:-------------------------:|\n",
    "<img src=\"images/[code_output][009][Cabin_column_sparsity].png\" style=\"width:900px;\"/> |  <img src=\"images/[code_output][010][Cabin_column_dirty_data].png\" style=\"width:900px;\"/> | <img src=\"images/[code_output][011][Cabin_column_multi_deck].png\" style=\"width:900px;\"/>|\n",
    "\n",
    "Firstly, both the training and test sets exhibit a high sparsity of approximately 78%. Consequently, we cannot simply drop the missing values. Secondly, there is a dirty value, `T`, present in the column. However, it has been established that the G Deck was the lowest complete deck (reference: <a href = \"https://titanic.fandom.com/wiki/G_Deck\">source</a>). This knowledge will aid us in cleaning the `Cabin` column. We will treat all missing and dirty values as if they were located in the G Deck. Lastly, some samples contain multiple cabins. In such cases, we will select the cabin with the smallest alphabetical value.\n",
    "\n",
    "<h3>Sparsity in the <code>Age</code> Column</h3>\n",
    "\n",
    "The `Age` column exhibits an alarming level of sparsity.\n",
    "\n",
    "<img src=\"images/[code_output][014][age_sparsity].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "Dropping such a significant amount of data is not acceptable. Therefore, we need to find ways to fill in the missing values. There are two approaches we can take: replacing the missing data with the global mean or training a machine learning model to fill in the gaps. In this case, we will explore both methods, with a slight modification to the global mean approach.\n",
    "\n",
    "<h4>Age Group Means</h4>\n",
    "\n",
    "Replacing missing data with the global mean is likely to result in underfitting. To address this, we leverage the concept of \"groupby\" to introduce additional features for the mean-based method. We organize the dataset into several categories and compute the mean age for each category. Let's illustrate this with the `age_df` dataframe.\n",
    "\n",
    "<img src=\"images/[code_output][016][age_df].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "We group the `age_df` dataframe by the columns `Pclass`, `Sex`, and `Embarked`, and then calculate the means. It is important to note that grouping by more columns may result in some `NaN` values.\n",
    "\n",
    "<img src=\"images/[code_output][015][groupby_mean_age].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "For each triple (`Pclass`, `Sex`, `Embarked`), we obtain a mean value. We can view this as a regressor, where the inputs are the passenger's class, sex, and boarding location, and the output is the predicted age of the passenger. We store these results in the `age_groupby_mean` column. We will evaluate this method against the machine learning approach later.\n",
    "\n",
    "Our next task is to build a linear regressor to estimate the missing ages.\n",
    "\n",
    "<h4>Predicting Age with Linear Regression</h4>\n",
    "\n",
    "Recalling the `age_features` list mentioned earlier, we will construct a linear regressor to estimate the ages of the passengers using `Pclass`, `Sex`, `SibSp`, `Parch`, and `Embarked` as features. These features are the only ones from the original dataset that are meaningful (i.e., not randomly generated) and have no missing values. It is worth noting that all these features are categorical. Therefore, we apply the `pandas.get_dummies` function to transform them into binary features. The Python code snippet below demonstrates the creation of the linear regressor.\n",
    "\n",
    "<img src=\"images/[code_output][017][age_linear_regressor].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "<h4>Comparing the Group-Mean and Linear Regressor Methods</h4>\n",
    "\n",
    "A standard metric for evaluating age prediction is the coefficient of determination (<a href=\"https://en.wikipedia.org/wiki/Coefficient_of_determination\">Wikipedia link</a>). Given two lists:\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/svg.image?\\begin{align*}y_1,&space;\\cdots,&space;y_n&space;\\\\\\hat{y}_1,&space;\\cdots,&space;\\hat{y}_n\\end{align*}\" />\n",
    "\n",
    "representing the true values and predicted values, respectively, the coefficient of determination (`R-square`) is calculated using the formula:\n",
    "\n",
    "<img src=\"https://latex.codecogs.com/svg.image?R^2&space;=&space;1&space;-&space;\\frac{\\displaystyle&space;\\sum_i&space;(y_i&space;-&space;\\hat{y}_i)^2}{\\displaystyle&space;\\sum_i&space;(y_i&space;-&space;\\overline{y})^2}\" />\n",
    "\n",
    "where the denominator represents the mean of the true values. This metric allows us to compare a given predictor against the global mean predictor. In particular, we expect the group-mean method to have a non-zero `R-square`. \n",
    "\n",
    "Another appropriate metric for age prediction evaluation is the `l-infinity` norm, or the maximum residual error (`MaxError`):\n",
    "\n",
    "<img src = \"https://latex.codecogs.com/svg.image?\\operatorname{MaxError}&space;=&space;\\max_i&space;\\left\\vert&space;y_i&space;-&space;\\hat{y}_i&space;\\right\\vert\" />\n",
    "\n",
    "This metric considers the tuple of `n` ages to be naturally embedded in the Cartesian product\n",
    "\n",
    "<img src = \"https://latex.codecogs.com/svg.image?\\underbrace{(\\mathbb{R},&space;\\lvert&space;\\cdot&space;\\rvert)&space;\\times&space;\\cdots&space;\\times&space;(\\mathbb{R},&space;\\lvert&space;\\cdot&space;\\rvert)}_{n}\" />\n",
    "\n",
    "rather than the usual `n`-dimensional Euclidean space. Consequently, the `l-1` norm, or the mean absolute error (`MAE`)\n",
    "\n",
    "<img src = \"https://latex.codecogs.com/svg.image?\\operatorname{MAE}&space;=&space;\\sum_i&space;\\left\\vert&space;y_i&space;-&space;\\hat{y}_i&space;\\right\\vert\" />\n",
    "\n",
    "is also an appropriate choice in this case.\n",
    "\n",
    "The Python code snippet below calculates these metrics for each method, where the predicted outputs from linear regression and the group-mean method are stored in `y_pred_linear` and `y_pred_groupby` respectively. Additionally, scatter plots are provided to visualize the predicted values against the true values.\n",
    "\n",
    "\"Computational Results\" | \"Scatter Plot\"\n",
    ":-------------------------:|:-------------------------:|\n",
    "<img src=\"images/[code_output][018][age_linear_regressor_vs_groupby_mean].png\" width=\"800\" height=\"600\"/> |  <img src=\"images/[data_visual][scatter_plot][003][predicting_ages_with_linear_regression_and_groupby].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "\n",
    "The results are organized in the following table:\n",
    "\n",
    "<table>\n",
    "  <tr>\n",
    "    <th style=\"text-align: center\">Metric</th>\n",
    "    <th style=\"text-align: center\">Linear Regression</th>\n",
    "    <th style=\"text-align: center\">Group-Mean</th>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td style=\"text-align: center\"><code>R-square</code></td>\n",
    "    <td style=\"text-align: center\">0.30</td>\n",
    "    <td style=\"text-align: center\">0.20</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td style=\"text-align: center\"><code>MaxError</code></td>\n",
    "    <td style=\"text-align: center\">36.69</td>\n",
    "    <td style=\"text-align: center\">38.26</td>\n",
    "  </tr>\n",
    "  <tr>\n",
    "      <td style=\"text-align: center\"><code>MAE</code></td>\n",
    "    <td style=\"text-align: center\">9.62</td>\n",
    "    <td style=\"text-align: center\">10.36</td>\n",
    "  </tr>\n",
    "\n",
    "</table>\n",
    "\n",
    "Overall, the linear regressor outperforms the group-mean method in all metrics. It achieves 34.86% higher `R-square`, 4.28% lower `MaxError`, and 7.63% lower `MAE`. Furthermore, the scatter plots demonstrate that the linear regression method provides a better fit to the true values.\n",
    "\n",
    "However, despite the superior performance of the linear regressor, we have decided to use the group-mean method instead. The reason for this choice is that the linear regression method introduces data leakage for a later phase. Specifically, the `y_pred_linear` column is a linear combination of the columns used to train the regressor, leading to leakage. Therefore, the group-mean method is a more suitable approach in this scenario.\n",
    "\n",
    "<h3><code>group_size = SibSp + Parch</code></h3>\n",
    "\n",
    "The column `Parch` in the testing set exhibits outliers:\n",
    "\n",
    "<img src=\"images/[code_output][004][testing_set_outliers_Parch].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "The presence of these outliers in the testing set, which do not appear in the training set, can potentially cause issues during the model training phase. To address this problem, we propose a workaround by considering both the `Parch` and `SibSp` columns as indicators of travel group size. In other words, we will calculate the sum of these two features and create a new column called `group_size`.\n",
    "\n",
    "This approach offers two advantages. Firstly, combining the features pulls the outliers towards a similar range. To demonstrate this, we have created a dataframe called `temp_df`:\n",
    "\n",
    "<img src=\"images/[code_output][005][adding_Parch_and_SibSp].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "The newly created category, `group_size = 10`, contains samples from the training set as well as the outliers, reducing the potential impact of these outliers on the model.\n",
    "\n",
    "Secondly, combining the features reduces the dimensionality of the feature space. If we were to consider only `Parch` and `SibSp` individually, the feature space would have a dimension of 56. However, it is possible that not all combinations of these features are present in the dataset, leading to a high degree of sparsity. On the other hand, by using the combined feature `group_size`, the feature space is reduced to 9 dimensions, resulting in resource savings.\n",
    "\n",
    "<img src=\"images/[code_output][006][feature_space_dim_reduction].png\" width=\"800\" height=\"600\"/>\n",
    "\n",
    "Therefore, we recommend replacing the `Parch` and `SibSp` columns with their sum, `group_size`, which captures the overall travel group size.\n",
    "\n",
    "<h3>Dropping the <code>Fare</code>, <code>Name</code>, and <code>Ticket</code> Columns</h3>\n",
    "\n",
    "The `Fare` column displays an unusual phenomenon:\n",
    "\n",
    "<img src=\"images/[data_visual][histogram][001][fare_price_per_class].png\" width=\"800\" height=\"400\"/>\n",
    "\n",
    "For each passenger class, there is a significant variance in fare prices. This contradicts common sense, as ticket prices for each class should either be the same or have minimal variance. Additionally, this inconsistency is also contrary to other reliable sources, such as the BBC, which states that third-class tickets in 1912 cost around £7, second-class tickets around £13, and first-class tickets starting from £30 (reference: <a href = \"https://www.bbc.co.uk/bitesize/topics/z8mpfg8/articles/zng8jty\">source</a>). Even after accounting for currency conversion, there remains a substantial disparity between the dataset and external data sources.\n",
    "\n",
    "The dataset in question was prepared for the Kaggle Getting Started Competition and is primarily intended for educational purposes. However, there is no information regarding the data source, whether it is original or obtained from a third party. As a result, it is possible that the ticket prices were randomly generated. This hypothesis, brought to the author's attention by <a href = \"https://github.com/Skarlit\">Skarlit</a>, provides a plausible explanation for the observed large variances in fare prices across passenger classes. Given the principle of \"garbage in, garbage out,\" we recommend dropping the `Fare` column.\n",
    "\n",
    "Next, we propose dropping the `Name` column. There are two reasons for this decision. Firstly, we expect each passenger's name to be unique, rendering it an impractical feature for model training. Secondly, there is no scientific connection between a person's name and their survival on the Titanic, making the `Name` column irrelevant for predicting passenger survival. Therefore, we recommend removing the `Name` column.\n",
    "\n",
    "Similarly, we find that the `Ticket` column consists of alphanumeric strings that appear to be randomly generated. Consequently, we recommend dropping the `Ticket` column, as these strings do not provide meaningful information for our analysis.\n",
    "\n",
    "<h3>EDA Conclusion</h3>\n",
    "\n",
    "Based on the exploratory data analysis (EDA) conducted, we recommend the following data cleaning tasks to prepare the dataset for further analysis:\n",
    "\n",
    "1. Fill in the missing values in the `Embarked` column using the information provided by a reliable third-party source.\n",
    "2. Clean the `Cabin` column by replacing the values with the corresponding cabin codes.\n",
    "3. Address the missing values in the `Age` column by replacing them with the mean age of the respective age groups.\n",
    "4. Replace the `SibSp` and `Parch` columns with their sum, creating a new column called `group_size`, which represents the total number of people traveling with each passenger.\n",
    "5. Exclude the `Fare`, `Name`, and `Ticket` columns from further analysis, as they are deemed less relevant or contain redundant information.\n",
    "\n",
    "With these data cleaning tasks, we aim to improve the quality and consistency of the dataset, enabling more accurate and meaningful analyses. The next step involves implementing these tasks using Python programming language.\n",
    "\n",
    "<h2>Data Preprocessing</h2>\n",
    "\n",
    "In this section, we showcase the use of our powerful `titanic_data_cleaning` library for data preprocessing. This library streamlines the data preprocessing workflow, ensuring that the data is cleaned and transformed according to the recommendations derived from the previous exploratory data analysis stage. The processed data is then prepared in the correct format for subsequent machine learning model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "19497536",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from titanic_ml_classes.titanic_data_cleaning import titanic_data_cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "30d41934",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_csv_path: str = 'csv/train.csv'\n",
    "test_csv_path: str = 'csv/train.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c4082941",
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_cleaned: titanic_data_cleaning = titanic_data_cleaning(train_csv_path, test_csv_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2023a88",
   "metadata": {},
   "source": [
    "The `titanic_data_cleaning` instance performs various cleaning operations on the data and stores the cleaned versions in its `.train_cleaned_` and `.test_cleaned_` attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1c3e258d",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df: pd.DataFrame = titanic_cleaned.train_cleaned_\n",
    "test_df: pd.DataFrame = titanic_cleaned.test_cleaned_\n",
    "train_raw_df: pd.DataFrame = titanic_cleaned.train_og_\n",
    "test_raw_df: pd.DataFrame = titanic_cleaned.test_og_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f343c51",
   "metadata": {},
   "source": [
    "These operations include:\n",
    "\n",
    "* Filling in the missing values in the `Embarked` column based on the suggested third-party source.\n",
    "* Cleaning the `Cabin` column by replacing values with the corresponding cabin codes.\n",
    "* Addressing missing values in the `Age` column by replacing them with the mean age of the respective age groups.\n",
    "* Replacing the `SibSp` and `Parch` columns with their sum, creating a new column `group_size` that represents the total number of people traveling with each passenger.\n",
    "* Excluding the `Fare`, `Name`, and `Ticket` columns from further analysis as they are considered less relevant or contain redundant information.\n",
    "* Replacing the numerical column `Age` with the categorical column `age_group`.\n",
    "\n",
    "The `pandas.info()` method demonstrates the difference between the data before and after cleaning, highlighting the transformations applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ffb469e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 13 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   PassengerId  891 non-null    int64  \n",
      " 1   Survived     891 non-null    int64  \n",
      " 2   Pclass       891 non-null    int64  \n",
      " 3   Name         891 non-null    object \n",
      " 4   Sex          891 non-null    object \n",
      " 5   Age          714 non-null    float64\n",
      " 6   SibSp        891 non-null    int64  \n",
      " 7   Parch        891 non-null    int64  \n",
      " 8   Ticket       891 non-null    object \n",
      " 9   Fare         891 non-null    float64\n",
      " 10  Cabin        204 non-null    object \n",
      " 11  Embarked     891 non-null    object \n",
      " 12  cabin_code   891 non-null    object \n",
      "dtypes: float64(2), int64(5), object(6)\n",
      "memory usage: 90.6+ KB\n"
     ]
    }
   ],
   "source": [
    "train_raw_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3cc84283",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 891 entries, 0 to 890\n",
      "Data columns (total 8 columns):\n",
      " #   Column       Non-Null Count  Dtype   \n",
      "---  ------       --------------  -----   \n",
      " 0   PassengerId  891 non-null    int64   \n",
      " 1   Survived     891 non-null    int64   \n",
      " 2   Pclass       891 non-null    int64   \n",
      " 3   Sex          891 non-null    object  \n",
      " 4   Embarked     891 non-null    object  \n",
      " 5   Cabin        891 non-null    object  \n",
      " 6   age_group    891 non-null    category\n",
      " 7   group_size   891 non-null    int64   \n",
      "dtypes: category(1), int64(4), object(3)\n",
      "memory usage: 50.1+ KB\n"
     ]
    }
   ],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "68c7fe94",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 418 entries, 0 to 417\n",
      "Data columns (total 12 columns):\n",
      " #   Column       Non-Null Count  Dtype  \n",
      "---  ------       --------------  -----  \n",
      " 0   PassengerId  418 non-null    int64  \n",
      " 1   Pclass       418 non-null    int64  \n",
      " 2   Name         418 non-null    object \n",
      " 3   Sex          418 non-null    object \n",
      " 4   Age          332 non-null    float64\n",
      " 5   SibSp        418 non-null    int64  \n",
      " 6   Parch        418 non-null    int64  \n",
      " 7   Ticket       418 non-null    object \n",
      " 8   Fare         417 non-null    float64\n",
      " 9   Cabin        91 non-null     object \n",
      " 10  Embarked     418 non-null    object \n",
      " 11  cabin_code   418 non-null    object \n",
      "dtypes: float64(2), int64(4), object(6)\n",
      "memory usage: 39.3+ KB\n"
     ]
    }
   ],
   "source": [
    "test_raw_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "64b97b2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 418 entries, 0 to 417\n",
      "Data columns (total 7 columns):\n",
      " #   Column       Non-Null Count  Dtype   \n",
      "---  ------       --------------  -----   \n",
      " 0   PassengerId  418 non-null    int64   \n",
      " 1   Pclass       418 non-null    int64   \n",
      " 2   Sex          418 non-null    object  \n",
      " 3   Embarked     418 non-null    object  \n",
      " 4   Cabin        418 non-null    object  \n",
      " 5   age_group    418 non-null    category\n",
      " 6   group_size   418 non-null    int64   \n",
      "dtypes: category(1), int64(3), object(3)\n",
      "memory usage: 20.5+ KB\n"
     ]
    }
   ],
   "source": [
    "test_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "8ed068ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.to_csv('csv/train_cleaned.csv', index = False)\n",
    "test_df.to_csv('csv/test_cleaned.csv', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b3e9ef1",
   "metadata": {},
   "source": [
    "Finally, we save the cleaned dataset into `.csv` files, marking the completion of our data preparation stage. The next stage is model training, which can be accessed through the `demo_ML_models.ipynb file`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "testtest",
   "language": "python",
   "name": "testtest"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}