(4)automated_model_selection_and_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (4) Automated Model Selection & Analysis\n",
    "\n",
    "Now that we have preprocessed our data and hopefully addressed most of the initial problems we discussed in the first three notebooks, we can focus on **modelling** in this final notebook. See the below quote from `h2o`'s documentation to understand what it does:\n",
    "\n",
    "*We have designed an easy-to-use interface which automates the process of training a large selection of candidate models. H2O’s AutoML can also be a helpful tool for the advanced user, by providing a simple wrapper function that performs a large number of modeling-related tasks that would typically require many lines of code, and by freeing up their time to focus on other aspects of the data science pipeline tasks such as data-preprocessing, feature engineering and model deployment.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import h2o\n",
    "from h2o.automl import H2OAutoML\n",
    "h2o.init()\n",
    "SEED = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify Training & Testing Filepaths\n",
    "To experiment how models perform with previous, raw forms of data, simply change the `training_filepath` and `testing_filepath` variables to other data folders starting with `(0)`, `(1)`, or `(2)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_filepath = '(3)data_trimmed/label_encoded/train_users.csv'\n",
    "testing_filepath = '(3)data_trimmed/label_encoded/test_users.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Training Data as `H2OFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import training set as H20Frames\n",
    "X_train = h2o.import_file(training_filepath)\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Unrelated Columns for Training\n",
    "The response variable 'country_destination' will always be an unrelated column that should be seperated away from the training columns. For `(0)data`, make sure to remove columns 'id' and 'date_first_booking' as well. For `(1)data_manual_ops`, make sure to rem ove colum 'id'. For the rest of the data filepaths, what we have below is sufficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select/discard variables (columns) to base models on training set\n",
    "train_variables, response_variable = X_train.columns, 'country_destination'\n",
    "unrelated_variables = [response_variable]  # REMEMBER: 'id', 'date_first_booking'\n",
    "for variable in unrelated_variables:\n",
    "    train_variables.remove(variable)\n",
    "\n",
    "X_train[response_variable] = X_train[response_variable].asfactor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `h2o`'s Best Utility: `H2OAutoML`\n",
    "\n",
    "You can find a detailed documentation of the `H2OAutoML` module that does most of the magic [here](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html). Below, we outlined a few important parameters and explained what `H2OAutoML` does as well as what it doesn't do.\n",
    "\n",
    "You can stop *automated* model training with two parameters:\n",
    "* `@max_runtime_secs`: How long the AutoML will run before starting the training of final Stacked Ensemble models. Defaults to 3600 seconds (1 hour).\n",
    "* `@max_models`: Maximum number of models to build in an AutoML run excluding the Stacked Ensemble models. Defaults to None.\n",
    "\n",
    "You can enable either *downsampling* and *upsampling* with two parameters:\n",
    "* `@balance_classes`: Specify whether to oversample the minority classes to balance the class distribution. This option is not enabled by default and can increase the data frame size. Majority classes can be undersampled to satisfy the max_after_balance_size parameter.\n",
    "* `@max_after_balance_size`: Specify the maximum relative size of the training data after balancing class counts (balance_classes must be enabled). Defaults to 5.0. (The value can be less than 1.0).\n",
    "\n",
    "By default, this module trains & validates the following model architectures automatically:\n",
    "* **DRF**\n",
    "* **GLM**\n",
    "* **XGBoost (XGBoost GBM)**\n",
    "* **GBM (H2O GBM)**\n",
    "* **DeepLearning** (*Fully-connected multi-layer artificial neural network*)\n",
    "* **StackedEnsemble**\n",
    "\n",
    "You can either specify to *include* or *exclude* models with two parameters:\n",
    "* `@exclude_algos`: A list/vector of character strings naming the algorithms to skip during the model-building phase.\n",
    "* `@include_algos`: A list/vector of character strings naming the algorithms to include during the model-building phase. \n",
    "\n",
    "`H2OAutoML` performs *hyperparameter search* based on **Random Grid Search** over a variety of algorithms in order to deliver the best model. In `H2OAutoML`, the following hyperparameters are fully supported:\n",
    "* **GLM Hyperparameters**:  *alpha*, *missing_values_handling*\n",
    "* **XGBoost Hyperparameters**: *ntrees*, *max_depth*, *min_rows*, *min_sum_hessian_in_leaf*, *sample_rate*, *col_sample_rate*, *col_sample_rate_per_tree*, *booster*, *reg_lambda*, *reg_alpha*\n",
    "* **GBM Hyperparameters**: *histogram_type*, *ntrees*, *max_depth*, *min_rows*, *learn_rate*, *sample_rate*, *col_sample_rate*, *col_sample_rate_per_tree*, *min_split_improvement*\n",
    "* **Deep Learning Hyperparameters**: *epochs*, *adaptivate_rate*, *activation*, *rho*, *epsilon*, *input_dropout_ratio*, *hidden*, *hidden_dropout_ratios*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aml = H2OAutoML(nfolds=5,\n",
    "                #balance_classes=True,\n",
    "                #max_after_balance_size=1.0 if downsampling else 10000.0,\n",
    "                max_runtime_secs=10000,\n",
    "                max_models=None,\n",
    "                stopping_metric='AUTO',  # defaults to logloss for classification\n",
    "                sort_metric='AUTO',      # defaults to mean_per_class_error for classification\n",
    "                seed=SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform Training on `H2OFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aml.train(x=train_variables,\n",
    "          y=response_variable,\n",
    "          training_frame=X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View the AutoML Models Leaderboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lb = aml.leaderboard\n",
    "print(lb.head(rows=lb.nrows))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Best (Leader) `h2o` Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_path = h2o.save_model(model=aml.leader,\n",
    "                            path='saved_models/',\n",
    "                            force=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read Test Data as `H2OFrame`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = h2o.import_file(testing_filepath)\n",
    "X_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Best (Leader) `h2o` Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = h2o.load_model(model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = model.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save as Submission File & Submit [Here](https://www.kaggle.com/c/airbnb-recruiting-new-user-bookings/submit)\n",
    "\n",
    "**IMPORTANT**: Notice that we are assigning 'id' column in a sorted way, as `featuretools` automatically sorted our rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answers = pd.DataFrame()\n",
    "answers['id'] = pd.read_csv('(0)data/test_users.csv').sort_values('id')['id']\n",
    "answers['country'] = predictions.as_data_frame()['predict']\n",
    "answers.set_index('id', inplace=True)\n",
    "answers.to_csv('answers.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}