HW5-APM382-Townsend_Beckelman.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <p style=\"text-align: center;\">MIS 382N: Advanced Predictive Modeling</p>\n",
    "# <p style=\"text-align: center;\">Team: Davis Townsend & Brooks Beckelman</p>\n",
    "# <p style=\"text-align: center;\">Assignment 5</p>\n",
    "## <p style=\"text-align: center;\">Total points: 50 </p>\n",
    "## <p style=\"text-align: center;\">Due: Mon, November 28</p>\n",
    "\n",
    "\n",
    "\n",
    "Your homework should be written in a **Jupyter notebook**. Please submit **only one** ipynb file from each group, and include the names of all the group members. Also, please make sure your code runs and the graphics (and anything else) are displayed in your notebook before submitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 1 - Random Forest vs Boosting - Regression (15pts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this question, we will compare performance of different ensemble methods for regression problems: [Random Forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html), [Gradient Boosting Regressor](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) (GBR), and [AdaBoost](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html#sklearn.ensemble.AdaBoostRegressor). Board game data set from DataQuest will be used (you can download data from Canvas: 'games.csv').\n",
    "\n",
    "1. (1) Load the data, (2) remove duplicate rows, (3) remove features of type string (object in pandas), and (4) replace missing values by mean of each column. Then, partition data into features (X) and the target label (y) for regression task. We want to predict the *average_rating*. Use [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to split data into training and testing: test_size=0.33, random_state=42. (1pt)\n",
    "\n",
    "2. Use a [Random Forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html) to predict average_rating. Find the best parameters (including *n_estimators*) using [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html). Report the accuracy of your model in terms of RMSE. (4pts)\n",
    "\n",
    "3. Use [Gradient Boosting Regressor](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) (GBR), and [AdaBoost](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html#sklearn.ensemble.AdaBoostRegressor) for predicting the targets. Again, find the best parameters (including *n_estimators,* and* learning_rate*), and report corresponding RMSE for each algorithm. (8pts)\n",
    "\n",
    "4. Which model did you expect to be more accurate in predicting the targets? Why? Did your observation match this expectation? (2pts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import (train_test_split,GridSearchCV)\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.ensemble import (RandomForestRegressor,GradientBoostingRegressor,AdaBoostRegressor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Games = pd.read_csv('C:\\Users\\Owner\\Documents\\MSBA\\Datasets\\games.csv')\n",
    "#drop duplicates\n",
    "Games = Games.drop_duplicates()\n",
    "#exclude columns of type object i.e. strings\n",
    "Games = Games.select_dtypes(exclude=['object'])\n",
    "#replace missing values with mean of each column\n",
    "Games = Games.fillna(Games.mean())\n",
    "# split df into target and features\n",
    "Target = Games['average_rating']\n",
    "Features = Games.drop(['average_rating'], axis=1)\n",
    "# split into training and testing set\n",
    "x_train, x_test, y_train, y_test = train_test_split(Features, Target, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': [2, 6, 10, 14, 18], 'max_features': ['auto', 'sqrt', 'log2']},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RFR = RandomForestRegressor()\n",
    "# create paramter grid for exhaustive search\n",
    "param_grid = { 'n_estimators' : [2,6,10,14,18], 'max_features' : ['auto', 'sqrt', 'log2'] }\n",
    "#make model\n",
    "cv_rfv = GridSearchCV(estimator = RFR, param_grid=param_grid, cv=5)\n",
    "#fit model that does exhaustive search for best params\n",
    "cv_rfv.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters:{'max_features': 'sqrt', 'n_estimators': 18}\n",
      "Best score:0.942559933353\n"
     ]
    }
   ],
   "source": [
    "print \"{}:{}\".format(\"Best parameters\", cv_rfv.best_params_)\n",
    "print \"{}{}\".format(\"Best score:\", cv_rfv.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.01349835497\n"
     ]
    }
   ],
   "source": [
    "#make model with best parameters\n",
    "RFR = RandomForestRegressor(n_estimators = 18, max_features = 'sqrt')\n",
    "model = RFR.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "test_MSE = mean_squared_error(y_test, pred_y)\n",
    "print 'RMSE:', np.sqrt(test_MSE)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll do gradient boosting regressor and then Adaboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=GradientBoostingRegressor(alpha=0.9, criterion='friedman_mse', init=None,\n",
       "             learning_rate=0.1, loss='ls', max_depth=3, max_features=None,\n",
       "             max_leaf_nodes=None, min_impurity_split=1e-07,\n",
       "             min_samples_leaf=1, min_samples_split=2,\n",
       "             min_weight_fraction_leaf=0.0, n_estimators=100,\n",
       "             presort='auto', random_state=None, subsample=1.0, verbose=0,\n",
       "             warm_start=False),\n",
       "       fit_params={}, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': [100, 150, 200], 'loss': ['ls', 'lad', 'huber', 'quantile'], 'learning_rate': [0.001, 0.01, 0.1, 1], 'max_features': ['auto', 'sqrt', 'log2']},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GBR = GradientBoostingRegressor()\n",
    "# create paramter grid for exhaustive search\n",
    "param_grid_GBR = { 'learning_rate' : [.001,.01,.1,1], 'loss' : ['ls', 'lad', 'huber', 'quantile'], 'n_estimators' : [100,150,200], \n",
    "             'max_features' : ['auto', 'sqrt', 'log2']}\n",
    "#make model\n",
    "cv_GBR = GridSearchCV(estimator = GBR, param_grid=param_grid_GBR, cv=5)\n",
    "#fit model that does exhaustive search for best params\n",
    "cv_GBR.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters:{'max_features': 'auto', 'loss': 'ls', 'learning_rate': 0.1, 'n_estimators': 200}\n",
      "Best score:0.890112249897\n"
     ]
    }
   ],
   "source": [
    "print \"{}:{}\".format(\"Best parameters\", cv_GBR.best_params_)\n",
    "print \"{}{}\".format(\"Best score:\", cv_GBR.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.01565867416\n"
     ]
    }
   ],
   "source": [
    "#make model with best parameters\n",
    "GBR = GradientBoostingRegressor(max_features='auto', loss='ls', learning_rate=0.1, n_estimators=200)\n",
    "model = GBR.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "test_MSE = mean_squared_error(y_test, pred_y)\n",
    "print 'RMSE:', np.sqrt(test_MSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=AdaBoostRegressor(base_estimator=None, learning_rate=1.0, loss='linear',\n",
       "         n_estimators=50, random_state=None),\n",
       "       fit_params={}, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': [25, 50, 100], 'loss': ['linear', 'exponential', 'square'], 'learning_rate': [0.0001, 0.001, 0.01, 0.1, 1]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring=None, verbose=0)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ABR = AdaBoostRegressor()\n",
    "# create paramter grid for exhaustive search\n",
    "param_grid_ABR = { 'learning_rate' : [.0001,.001,.01,.1,1], 'loss' : ['linear', 'exponential', 'square'], 'n_estimators' : [25,50,100]}\n",
    "#make model\n",
    "cv_ABR = GridSearchCV(estimator = ABR, param_grid=param_grid_ABR, cv=5)\n",
    "#fit model that does exhaustive search for best params\n",
    "cv_ABR.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters:{'n_estimators': 100, 'loss': 'exponential', 'learning_rate': 0.1}\n",
      "Best score:0.86291001213\n"
     ]
    }
   ],
   "source": [
    "print \"{}:{}\".format(\"Best parameters\", cv_ABR.best_params_)\n",
    "print \"{}{}\".format(\"Best score:\", cv_ABR.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 1.13231974067\n"
     ]
    }
   ],
   "source": [
    "#make model with best parameters\n",
    "ABR = AdaBoostRegressor(n_estimators = 100, loss = 'exponential', learning_rate=0.1)\n",
    "model = ABR.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "test_MSE = mean_squared_error(y_test, pred_y)\n",
    "print 'RMSE:', np.sqrt(test_MSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I thought going into it that Gradient Boosting Regressor (GBR) would perform the best simply because it's a very powerful model.\n",
    "\n",
    "Interestingly,I didnt tune my parameters fully and Random Forest slightly outperformed Boosting but I suspect that if I tried more parameters and fully tuned my boosting model that it would outperform random forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem 2 - Random Forest vs Boosting - Classification (15 pts)\n",
    "In this question, we will compare performance of different ensemble methods for classification problems: [Random Forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html), [Gradient Boosting Decision Tree](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) (GBDT), and [AdaBoost](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier). [Spam Classification Data](https://archive.ics.uci.edu/ml/datasets/Spambase) of UCI will be used (you can download data from Canvas: 'spam_uci.csv'). Don't worry about column names. The last column represents target label, 1 if spam and zero otherwise.\n",
    "\n",
    "1. Load the data and partition it into features (X) and the target label (y) for classification task. Then, use [train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) to split data into training and testing: test_size=0.33, random_state=42. (1pt)\n",
    "\n",
    "2. Use a [Random Forest](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) to classify whether an email is spam. Find the best parameters (including *n_estimators* and *criterion*) using [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html). Report your testing accuracy ([accuracy_score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html)) and [roc_auc_score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#sklearn.metrics.roc_auc_score). You will need [predict_proba](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.predict_proba) for roc_auc_score. (4pts)\n",
    "\n",
    "3. Use [Gradient Boosting Decision Tree](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) (GBDT), and [AdaBoost](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostClassifier.html#sklearn.ensemble.AdaBoostClassifier) for the spam classification problem. Again, find the best parameters (including *n_estimators, learning_rate,* and *max_depth (GBDT only)*), and report corresponding accuracy_score and roc_auc_score on the test data for each algorithm. (8pts)\n",
    "\n",
    "4. Point out one advantage and one disadvantage of Random Forest compared to GBDT (2pts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import (train_test_split,GridSearchCV)\n",
    "from sklearn.metrics import (accuracy_score,roc_auc_score)\n",
    "from sklearn.ensemble import (RandomForestClassifier,GradientBoostingClassifier,AdaBoostClassifier)\n",
    "from sklearn.metrics import roc_curve, roc_auc_score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Spam_uci = pd.read_csv('C:\\Users\\Owner\\Documents\\MSBA\\Datasets\\spam_uci.csv')\n",
    "# split df into target and features\n",
    "Target = Spam_uci[[-1]]\n",
    "Features = Spam_uci.drop(['57'], axis=1)\n",
    "# split into training and testing set\n",
    "x_train, x_test, y_train, y_test = train_test_split(Features, Target, test_size=0.33, random_state=42)\n",
    "x_train = x_train.values\n",
    "x_test = x_test.values\n",
    "y_train = y_train.values\n",
    "#reshape for gridsearch cv's expected data shape\n",
    "c, r = y_train.shape\n",
    "y_train = y_train.reshape(c,)\n",
    "y_test = y_test.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, {'criterion': 'gini', 'n_estimators': 19})"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "RFC = RandomForestClassifier()\n",
    "# create paramter grid for exhaustive search\n",
    "param_grid = { 'n_estimators' : [10,15,17,19, 21], 'criterion' : ['gini', 'entropy'] }\n",
    "\n",
    "def hypertuning(estimator, param_grid, x_train, y_train):\n",
    "    '''this function takes as input an estimator, parameter grid to tune, and the features and target variable\n",
    "    \n",
    "        it return the best score and best parameters of this estimaor from given possible params'''\n",
    "    \n",
    "    cv_estimator = GridSearchCV(estimator = estimator, param_grid=param_grid, cv=5)\n",
    "    cv_estimator.fit(x_train, y_train)\n",
    "    return cv_estimator.best_score_, cv_estimator.best_params_\n",
    "\n",
    "hypertuning(RFC, param_grid, x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "using old function to get accuracy score and roc_auc score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "RFC = RandomForestClassifier(criterion = 'gini', n_estimators = 19)\n",
    "model = RFC.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "def BayesianClassifier(modeltype, train_X, train_y, test_X):\n",
    "    \n",
    "    '''takes bayesian models of Linear/Quadratic Discriminant Analysis, Gaussian Naiive Bayes\n",
    "    and return an array of predicted probabilities as an array\n",
    "    \n",
    "        Note: data must be in an array   '''\n",
    "    \n",
    "    fitted_model = modeltype.fit(train_X, train_y)\n",
    "    predicted_y = modeltype.predict(test_X)\n",
    "    predicted_probabilities = modeltype.predict_proba(test_X)\n",
    "    return predicted_probabilities\n",
    "\n",
    "def RocCurve(predprobs, actual_y):\n",
    "    '''predicted probabilities should be in form of an array\n",
    "    \n",
    "    function returns the fpr, tpr, threshold and the AUC of the ROC Curve as the score'''\n",
    "    \n",
    "    pos_probs = []\n",
    "    for i in predprobs:\n",
    "        pos_probs.append(i[1])\n",
    "    fpr, tpr, threshold = roc_curve(actual_y, pos_probs)\n",
    "    score = roc_auc_score(actual_y, pos_probs)\n",
    "    return fpr, tpr, threshold, score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC_score : 1.0\n",
      "accuracy score: 1.0\n"
     ]
    }
   ],
   "source": [
    "predictedprobsRFC = BayesianClassifier(RFC, x_train, y_train, x_test)\n",
    "score = RocCurve(predictedprobsRFC, y_test)[3]\n",
    "print \"AUC_score :\", score\n",
    "ACC = accuracy_score(y_test, pred_y) \n",
    "print \"accuracy score:\", ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gradient Boosted Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, {'learning_rate': 0.001, 'max_depth': 1, 'n_estimators': 250})"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "GBC = GradientBoostingClassifier()\n",
    "param_grid = { 'n_estimators' : [200, 250, 300], 'learning_rate' : [.0001, .001, .01, .1], 'max_depth' : [1,3,5,7] }\n",
    "hypertuning(GBC, param_grid, x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC_score : 0.998871331828\n",
      "accuracy score: 0.998683344305\n"
     ]
    }
   ],
   "source": [
    "GBC = GradientBoostingClassifier(learning_rate = .001, max_depth =1, n_estimators = 250)\n",
    "model = GBC.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "\n",
    "predictedprobsGBC = BayesianClassifier(GBC, x_train, y_train, x_test)\n",
    "score = RocCurve(predictedprobsGBC, y_test)[3]\n",
    "print \"AUC_score :\", score\n",
    "ACC = accuracy_score(y_test, pred_y) \n",
    "print \"accuracy score:\", ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adaboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0, {'learning_rate': 0.001, 'n_estimators': 25})"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ABC = AdaBoostClassifier()\n",
    "param_grid = { 'n_estimators' : [25,50, 75, 100], 'learning_rate' : [.001, .01, .1, 1,10]}\n",
    "hypertuning(ABC, param_grid, x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC_score : 0.998871331828\n",
      "accuracy score: 0.998683344305\n"
     ]
    }
   ],
   "source": [
    "ABC = AdaBoostClassifier(learning_rate = .001, n_estimators = 25)\n",
    "model = ABC.fit(x_train, y_train)\n",
    "pred_y = model.predict(x_test)\n",
    "\n",
    "predictedprobsABC = BayesianClassifier(ABC, x_train, y_train, x_test)\n",
    "score = RocCurve(predictedprobsABC, y_test)[3]\n",
    "print \"AUC_score :\", score\n",
    "ACC = accuracy_score(y_test, pred_y) \n",
    "print \"accuracy score:\", ACC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random forest is better for scaling up very large dataset for distributed cimputing for its ease of parallelization.\n",
    "\n",
    "Boosting in general usually outperforms random forest, but you have more parameters you need to hypertune."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Question 3 - Matrix Factorization for Rating Prediction (20pts)\n",
    "\n",
    "The movielens dataset contains 1 million movie ratings from several thousand users. We will be using *k*-rank matrix factorization to estimate this dataset as the product $X=UV^T$, where *U* and *V* have only $k$ columns.\n",
    "\n",
    "1) You can download the movielens 1M dataset from https://datahub.io/dataset/movielens, but for this problem use the data available on Canvas. It has been split into training and test sets, and converted to matrix format where the rows correspond to users and the columns to movies. Note that most of the entries are NaNs, indicating that these ratings are missing. An extra file, lens1m_361M_titles.csv, has been added so you can check out specific movies if you're curious.\n",
    "\n",
    "2) Scikit-learn is a little behind for recommender systems, and doesn't have any method to factorize matrices with missing data. Which means you get to code it! Slide 22 of the 'apa large scale learning' lecture notes has the equations for stochastic gradient descent on *U* and *V*. You will have to:\n",
    "* Set up initial guesses for the *U* and *V* matrices. I suggest small random values.\n",
    "* Find a suitable learning rate for the descent. A learning rate that is too large will probably blow up, like in HW3 problem 1.\n",
    "* Come up with a stopping policy\n",
    "* Code the descent algorithm (5 pts)\n",
    "\n",
    "3) Using your SGD algorithm, apply 2-rank matrix factorization on the filled training matrix. Calculate the RMSE of this model on the training data and on the test data (separately). The optimal score on the training data is around .86 RMSE; your version of gradient descent must go at least below .91 RMSE. (5 pts)\n",
    "\n",
    "4) You should notice some overfitting. Because matrix factorization learns separate scores for each movie, a movie with very few reviews may be easily overfit. You may want to only predict ratings when you have enough information to reach a good conclusion. Recalculate the RMSE on the test data, specifically for movies with at least 50 reviews (don't retrain the models). Also report the percent of movies that are still included (after cutting those with < 50 reviews), and the percent of test ratings that are still included. (5 pts)\n",
    "\n",
    "5) Repeat steps 3 and 4 with 5-rank factorization. Display training and test RMSE. (5 pts)\n",
    "\n",
    "Hints:  \n",
    "The numpy function *nanmean* is helpful for RMSE calculation.  \n",
    "The descent algorithm will probably run for at least several minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "titles = pd.read_csv('lens1m_361M_titles.csv')\n",
    "test_X = np.load('lens1m_361M_test.npy')\n",
    "train_X = np.load('lens1m_361M_train.npy')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}