Day 12/exercise-categorical-variables.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Intermediate Machine Learning](https://www.kaggle.com/learn/intermediate-machine-learning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/categorical-variables).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"By encoding **categorical variables**, you'll obtain your best results thus far!\n\n# Setup\n\nThe questions below will give you feedback on your work. Run the following cell to set up the feedback system.","metadata":{}},{"cell_type":"code","source":"# Set up code checking\nimport os\nif not os.path.exists(\"../input/train.csv\"):\n    os.symlink(\"../input/home-data-for-ml-course/train.csv\", \"../input/train.csv\")  \n    os.symlink(\"../input/home-data-for-ml-course/test.csv\", \"../input/test.csv\") \nfrom learntools.core import binder\nbinder.bind(globals())\nfrom learntools.ml_intermediate.ex3 import *\nprint(\"Setup Complete\")","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.101487Z","iopub.execute_input":"2021-08-14T11:43:31.101947Z","iopub.status.idle":"2021-08-14T11:43:31.110950Z","shell.execute_reply.started":"2021-08-14T11:43:31.101910Z","shell.execute_reply":"2021-08-14T11:43:31.109982Z"},"trusted":true},"execution_count":136,"outputs":[{"name":"stdout","text":"Setup Complete\n","output_type":"stream"}]},{"cell_type":"markdown","source":"In this exercise, you will work with data from the [Housing Prices Competition for Kaggle Learn Users](https://www.kaggle.com/c/home-data-for-ml-course). \n\n![Ames Housing dataset image](https://i.imgur.com/lTJVG4e.png)\n\nRun the next code cell without changes to load the training and validation sets in `X_train`, `X_valid`, `y_train`, and `y_valid`.  The test set is loaded in `X_test`.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\n# Read the data\nX = pd.read_csv('../input/train.csv', index_col='Id') \nX_test = pd.read_csv('../input/test.csv', index_col='Id')\n\n# Remove rows with missing target, separate target from predictors\nX.dropna(axis=0, subset=['SalePrice'], inplace=True)\ny = X.SalePrice\nX.drop(['SalePrice'], axis=1, inplace=True)\n\n# To keep things simple, we'll drop columns with missing values\ncols_with_missing = [col for col in X.columns if X[col].isnull().any()] \nX.drop(cols_with_missing, axis=1, inplace=True)\nX_test.drop(cols_with_missing, axis=1, inplace=True)\n\n# Break off validation set from training data\nX_train, X_valid, y_train, y_valid = train_test_split(X, y,\n                                                      train_size=0.8, test_size=0.2,\n                                                      random_state=0)\nX_test","metadata":{"execution":{"iopub.status.busy":"2021-08-14T12:11:20.966551Z","iopub.execute_input":"2021-08-14T12:11:20.967347Z","iopub.status.idle":"2021-08-14T12:11:21.116486Z","shell.execute_reply.started":"2021-08-14T12:11:20.967302Z","shell.execute_reply":"2021-08-14T12:11:21.115316Z"},"trusted":true},"execution_count":207,"outputs":[{"execution_count":207,"output_type":"execute_result","data":{"text/plain":"      MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\nId                                                                         \n1461          20       RH    11622   Pave      Reg         Lvl    AllPub   \n1462          20       RL    14267   Pave      IR1         Lvl    AllPub   \n1463          60       RL    13830   Pave      IR1         Lvl    AllPub   \n1464          60       RL     9978   Pave      IR1         Lvl    AllPub   \n1465         120       RL     5005   Pave      IR1         HLS    AllPub   \n...          ...      ...      ...    ...      ...         ...       ...   \n2915         160       RM     1936   Pave      Reg         Lvl    AllPub   \n2916         160       RM     1894   Pave      Reg         Lvl    AllPub   \n2917          20       RL    20000   Pave      Reg         Lvl    AllPub   \n2918          85       RL    10441   Pave      Reg         Lvl    AllPub   \n2919          60       RL     9627   Pave      Reg         Lvl    AllPub   \n\n     LotConfig LandSlope Neighborhood  ... OpenPorchSF EnclosedPorch  \\\nId                                     ...                             \n1461    Inside       Gtl        NAmes  ...           0             0   \n1462    Corner       Gtl        NAmes  ...          36             0   \n1463    Inside       Gtl      Gilbert  ...          34             0   \n1464    Inside       Gtl      Gilbert  ...          36             0   \n1465    Inside       Gtl      StoneBr  ...          82             0   \n...        ...       ...          ...  ...         ...           ...   \n2915    Inside       Gtl      MeadowV  ...           0             0   \n2916    Inside       Gtl      MeadowV  ...          24             0   \n2917    Inside       Gtl      Mitchel  ...           0             0   \n2918    Inside       Gtl      Mitchel  ...          32             0   \n2919    Inside       Mod      Mitchel  ...          48             0   \n\n     3SsnPorch ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType  \\\nId                                                                       \n1461         0         120         0        0       6    2010       WD   \n1462         0           0         0    12500       6    2010       WD   \n1463         0           0         0        0       3    2010       WD   \n1464         0           0         0        0       6    2010       WD   \n1465         0         144         0        0       1    2010       WD   \n...        ...         ...       ...      ...     ...     ...      ...   \n2915         0           0         0        0       6    2006       WD   \n2916         0           0         0        0       4    2006       WD   \n2917         0           0         0        0       9    2006       WD   \n2918         0           0         0      700       7    2006       WD   \n2919         0           0         0        0      11    2006       WD   \n\n     SaleCondition  \nId                  \n1461        Normal  \n1462        Normal  \n1463        Normal  \n1464        Normal  \n1465        Normal  \n...            ...  \n2915        Normal  \n2916       Abnorml  \n2917       Abnorml  \n2918        Normal  \n2919        Normal  \n\n[1459 rows x 60 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>MSSubClass</th>\n      <th>MSZoning</th>\n      <th>LotArea</th>\n      <th>Street</th>\n      <th>LotShape</th>\n      <th>LandContour</th>\n      <th>Utilities</th>\n      <th>LotConfig</th>\n      <th>LandSlope</th>\n      <th>Neighborhood</th>\n      <th>...</th>\n      <th>OpenPorchSF</th>\n      <th>EnclosedPorch</th>\n      <th>3SsnPorch</th>\n      <th>ScreenPorch</th>\n      <th>PoolArea</th>\n      <th>MiscVal</th>\n      <th>MoSold</th>\n      <th>YrSold</th>\n      <th>SaleType</th>\n      <th>SaleCondition</th>\n    </tr>\n    <tr>\n      <th>Id</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1461</th>\n      <td>20</td>\n      <td>RH</td>\n      <td>11622</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>NAmes</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>120</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>2010</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>1462</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>14267</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Corner</td>\n      <td>Gtl</td>\n      <td>NAmes</td>\n      <td>...</td>\n      <td>36</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>12500</td>\n      <td>6</td>\n      <td>2010</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>1463</th>\n      <td>60</td>\n      <td>RL</td>\n      <td>13830</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>Gilbert</td>\n      <td>...</td>\n      <td>34</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>3</td>\n      <td>2010</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>1464</th>\n      <td>60</td>\n      <td>RL</td>\n      <td>9978</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>Gilbert</td>\n      <td>...</td>\n      <td>36</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>2010</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>1465</th>\n      <td>120</td>\n      <td>RL</td>\n      <td>5005</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>HLS</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>StoneBr</td>\n      <td>...</td>\n      <td>82</td>\n      <td>0</td>\n      <td>0</td>\n      <td>144</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>2010</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>2915</th>\n      <td>160</td>\n      <td>RM</td>\n      <td>1936</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>MeadowV</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>6</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>2916</th>\n      <td>160</td>\n      <td>RM</td>\n      <td>1894</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>MeadowV</td>\n      <td>...</td>\n      <td>24</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>4</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Abnorml</td>\n    </tr>\n    <tr>\n      <th>2917</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>20000</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>Mitchel</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>9</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Abnorml</td>\n    </tr>\n    <tr>\n      <th>2918</th>\n      <td>85</td>\n      <td>RL</td>\n      <td>10441</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>Mitchel</td>\n      <td>...</td>\n      <td>32</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>700</td>\n      <td>7</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>2919</th>\n      <td>60</td>\n      <td>RL</td>\n      <td>9627</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Mod</td>\n      <td>Mitchel</td>\n      <td>...</td>\n      <td>48</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>11</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n  </tbody>\n</table>\n<p>1459 rows × 60 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"Use the next code cell to print the first five rows of the data.","metadata":{}},{"cell_type":"code","source":"X_train.head()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.232400Z","iopub.execute_input":"2021-08-14T11:43:31.233032Z","iopub.status.idle":"2021-08-14T11:43:31.262465Z","shell.execute_reply.started":"2021-08-14T11:43:31.232984Z","shell.execute_reply":"2021-08-14T11:43:31.260967Z"},"trusted":true},"execution_count":138,"outputs":[{"execution_count":138,"output_type":"execute_result","data":{"text/plain":"     MSSubClass MSZoning  LotArea Street LotShape LandContour Utilities  \\\nId                                                                        \n619          20       RL    11694   Pave      Reg         Lvl    AllPub   \n871          20       RL     6600   Pave      Reg         Lvl    AllPub   \n93           30       RL    13360   Pave      IR1         HLS    AllPub   \n818          20       RL    13265   Pave      IR1         Lvl    AllPub   \n303          20       RL    13704   Pave      IR1         Lvl    AllPub   \n\n    LotConfig LandSlope Neighborhood  ... OpenPorchSF EnclosedPorch 3SsnPorch  \\\nId                                    ...                                       \n619    Inside       Gtl      NridgHt  ...         108             0         0   \n871    Inside       Gtl        NAmes  ...           0             0         0   \n93     Inside       Gtl      Crawfor  ...           0            44         0   \n818   CulDSac       Gtl      Mitchel  ...          59             0         0   \n303    Corner       Gtl      CollgCr  ...          81             0         0   \n\n    ScreenPorch  PoolArea  MiscVal  MoSold  YrSold SaleType SaleCondition  \nId                                                                         \n619         260         0        0       7    2007      New       Partial  \n871           0         0        0       8    2009       WD        Normal  \n93            0         0        0       8    2009       WD        Normal  \n818           0         0        0       7    2008       WD        Normal  \n303           0         0        0       1    2006       WD        Normal  \n\n[5 rows x 60 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>MSSubClass</th>\n      <th>MSZoning</th>\n      <th>LotArea</th>\n      <th>Street</th>\n      <th>LotShape</th>\n      <th>LandContour</th>\n      <th>Utilities</th>\n      <th>LotConfig</th>\n      <th>LandSlope</th>\n      <th>Neighborhood</th>\n      <th>...</th>\n      <th>OpenPorchSF</th>\n      <th>EnclosedPorch</th>\n      <th>3SsnPorch</th>\n      <th>ScreenPorch</th>\n      <th>PoolArea</th>\n      <th>MiscVal</th>\n      <th>MoSold</th>\n      <th>YrSold</th>\n      <th>SaleType</th>\n      <th>SaleCondition</th>\n    </tr>\n    <tr>\n      <th>Id</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>619</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>11694</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>NridgHt</td>\n      <td>...</td>\n      <td>108</td>\n      <td>0</td>\n      <td>0</td>\n      <td>260</td>\n      <td>0</td>\n      <td>0</td>\n      <td>7</td>\n      <td>2007</td>\n      <td>New</td>\n      <td>Partial</td>\n    </tr>\n    <tr>\n      <th>871</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>6600</td>\n      <td>Pave</td>\n      <td>Reg</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>NAmes</td>\n      <td>...</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>8</td>\n      <td>2009</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>93</th>\n      <td>30</td>\n      <td>RL</td>\n      <td>13360</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>HLS</td>\n      <td>AllPub</td>\n      <td>Inside</td>\n      <td>Gtl</td>\n      <td>Crawfor</td>\n      <td>...</td>\n      <td>0</td>\n      <td>44</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>8</td>\n      <td>2009</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>818</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>13265</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>CulDSac</td>\n      <td>Gtl</td>\n      <td>Mitchel</td>\n      <td>...</td>\n      <td>59</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>7</td>\n      <td>2008</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n    <tr>\n      <th>303</th>\n      <td>20</td>\n      <td>RL</td>\n      <td>13704</td>\n      <td>Pave</td>\n      <td>IR1</td>\n      <td>Lvl</td>\n      <td>AllPub</td>\n      <td>Corner</td>\n      <td>Gtl</td>\n      <td>CollgCr</td>\n      <td>...</td>\n      <td>81</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>2006</td>\n      <td>WD</td>\n      <td>Normal</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 60 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":"Notice that the dataset contains both numerical and categorical variables.  You'll need to encode the categorical data before training a model.\n\nTo compare different models, you'll use the same `score_dataset()` function from the tutorial.  This function reports the [mean absolute error](https://en.wikipedia.org/wiki/Mean_absolute_error) (MAE) from a random forest model.","metadata":{}},{"cell_type":"code","source":"from sklearn.ensemble import RandomForestRegressor\nfrom sklearn.metrics import mean_absolute_error\n\n# function for comparing different approaches\ndef score_dataset(X_train, X_valid, y_train, y_valid):\n    model = RandomForestRegressor(n_estimators=100, random_state=0)\n    model.fit(X_train, y_train)\n    preds = model.predict(X_valid)\n    return mean_absolute_error(y_valid, preds)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.264365Z","iopub.execute_input":"2021-08-14T11:43:31.264917Z","iopub.status.idle":"2021-08-14T11:43:31.277720Z","shell.execute_reply.started":"2021-08-14T11:43:31.264865Z","shell.execute_reply":"2021-08-14T11:43:31.276341Z"},"trusted":true},"execution_count":139,"outputs":[]},{"cell_type":"markdown","source":"# Step 1: Drop columns with categorical data\n\nYou'll get started with the most straightforward approach.  Use the code cell below to preprocess the data in `X_train` and `X_valid` to remove columns with categorical data.  Set the preprocessed DataFrames to `drop_X_train` and `drop_X_valid`, respectively.  ","metadata":{}},{"cell_type":"code","source":"# Fill in the lines below: drop columns in training and validation data\n# Get list of categorical variables\ns = (X_train.dtypes == 'object')\n\ncategorical_cols = list(s[s].index)\n\nprint(\"Categorical variables:\")\nprint(categorical_cols)\n\ndrop_X_train = X_train.drop(categorical_cols, axis=1)\ndrop_X_valid = X_valid.drop(categorical_cols, axis=1)\n\n# Check your answers\nstep_1.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.279991Z","iopub.execute_input":"2021-08-14T11:43:31.280492Z","iopub.status.idle":"2021-08-14T11:43:31.310375Z","shell.execute_reply.started":"2021-08-14T11:43:31.280444Z","shell.execute_reply":"2021-08-14T11:43:31.309000Z"},"trusted":true},"execution_count":140,"outputs":[{"name":"stdout","text":"Categorical variables:\n['MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'Exterior1st', 'Exterior2nd', 'ExterQual', 'ExterCond', 'Foundation', 'Heating', 'HeatingQC', 'CentralAir', 'KitchenQual', 'Functional', 'PavedDrive', 'SaleType', 'SaleCondition']\n","output_type":"stream"},{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"1_Drop\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#step_1.hint()\n#step_1.solution()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.312545Z","iopub.execute_input":"2021-08-14T11:43:31.313106Z","iopub.status.idle":"2021-08-14T11:43:31.320250Z","shell.execute_reply.started":"2021-08-14T11:43:31.313059Z","shell.execute_reply":"2021-08-14T11:43:31.319397Z"},"trusted":true},"execution_count":141,"outputs":[]},{"cell_type":"markdown","source":"Run the next code cell to get the MAE for this approach.","metadata":{}},{"cell_type":"code","source":"print(\"MAE from Approach 1 (Drop categorical variables):\")\nprint(score_dataset(drop_X_train, drop_X_valid, y_train, y_valid))","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:31.322735Z","iopub.execute_input":"2021-08-14T11:43:31.323191Z","iopub.status.idle":"2021-08-14T11:43:32.596372Z","shell.execute_reply.started":"2021-08-14T11:43:31.323161Z","shell.execute_reply":"2021-08-14T11:43:32.595449Z"},"trusted":true},"execution_count":142,"outputs":[{"name":"stdout","text":"MAE from Approach 1 (Drop categorical variables):\n17837.82570776256\n","output_type":"stream"}]},{"cell_type":"markdown","source":"Before jumping into ordinal encoding, we'll investigate the dataset.  Specifically, we'll look at the `'Condition2'` column.  The code cell below prints the unique entries in both the training and validation sets.","metadata":{}},{"cell_type":"code","source":"print(\"Unique values in 'Condition2' column in training data:\", X_train['Condition2'].unique())\nprint(\"\\nUnique values in 'Condition2' column in validation data:\", X_valid['Condition2'].unique())","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.598006Z","iopub.execute_input":"2021-08-14T11:43:32.598503Z","iopub.status.idle":"2021-08-14T11:43:32.605747Z","shell.execute_reply.started":"2021-08-14T11:43:32.598418Z","shell.execute_reply":"2021-08-14T11:43:32.604630Z"},"trusted":true},"execution_count":143,"outputs":[{"name":"stdout","text":"Unique values in 'Condition2' column in training data: ['Norm' 'PosA' 'Feedr' 'PosN' 'Artery' 'RRAe']\n\nUnique values in 'Condition2' column in validation data: ['Norm' 'RRAn' 'RRNn' 'Artery' 'Feedr' 'PosN']\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# Step 2: Ordinal encoding\n\n### Part A\n\nIf you now write code to: \n- fit an ordinal encoder to the training data, and then \n- use it to transform both the training and validation data, \n\nyou'll get an error.  Can you see why this is the case?  (_You'll need  to use the above output to answer this question._)","metadata":{}},{"cell_type":"markdown","source":"Fitting an ordinal encoder to a column will give an integer value to each unique label. Now our validation data contains the values `'RRAn'` and `'RRNn'` which is not present in the training data.\nSo it'll give an error.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nstep_2.a.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.607319Z","iopub.execute_input":"2021-08-14T11:43:32.607635Z","iopub.status.idle":"2021-08-14T11:43:32.624978Z","shell.execute_reply.started":"2021-08-14T11:43:32.607607Z","shell.execute_reply":"2021-08-14T11:43:32.623759Z"},"trusted":true},"execution_count":144,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 4, \"questionId\": \"2.1_LabelA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct: \n\nFitting an ordinal encoder to a column in the training data creates a corresponding integer-valued label for each unique value **that appears in the training data**. In the case that the validation data contains values that don't also appear in the training data, the encoder will throw an error, because these values won't have an integer assigned to them.  Notice that the `'Condition2'` column in the validation data contains the values `'RRAn'` and `'RRNn'`, but these don't appear in the training data -- thus, if we try to use an ordinal encoder with scikit-learn, the code will throw an error.","text/markdown":"<span style=\"color:#33cc33\">Correct:</span> \n\nFitting an ordinal encoder to a column in the training data creates a corresponding integer-valued label for each unique value **that appears in the training data**. In the case that the validation data contains values that don't also appear in the training data, the encoder will throw an error, because these values won't have an integer assigned to them.  Notice that the `'Condition2'` column in the validation data contains the values `'RRAn'` and `'RRNn'`, but these don't appear in the training data -- thus, if we try to use an ordinal encoder with scikit-learn, the code will throw an error."},"metadata":{}}]},{"cell_type":"code","source":"#step_2.a.hint()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.626453Z","iopub.execute_input":"2021-08-14T11:43:32.626899Z","iopub.status.idle":"2021-08-14T11:43:32.635751Z","shell.execute_reply.started":"2021-08-14T11:43:32.626865Z","shell.execute_reply":"2021-08-14T11:43:32.634398Z"},"trusted":true},"execution_count":145,"outputs":[]},{"cell_type":"markdown","source":"This is a common problem that you'll encounter with real-world data, and there are many approaches to fixing this issue.  For instance, you can write a custom ordinal encoder to deal with new categories.  The simplest approach, however, is to drop the problematic categorical columns.  \n\nRun the code cell below to save the problematic columns to a Python list `bad_label_cols`.  Likewise, columns that can be safely ordinal encoded are stored in `good_label_cols`.","metadata":{}},{"cell_type":"code","source":"# All categorical columns\nobject_cols = [col for col in X_train.columns if X_train[col].dtype == \"object\"]\n\n# Columns that can be safely ordinal encoded\ngood_label_cols = [col for col in object_cols if \n                   set(X_valid[col]).issubset(set(X_train[col]))]\n        \n# Problematic columns that will be dropped from the dataset\nbad_label_cols = list(set(object_cols)-set(good_label_cols))\n        \nprint('Categorical columns that will be ordinal encoded:', good_label_cols)\nprint('\\nCategorical columns that will be dropped from the dataset:', bad_label_cols)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.637260Z","iopub.execute_input":"2021-08-14T11:43:32.637646Z","iopub.status.idle":"2021-08-14T11:43:32.662523Z","shell.execute_reply.started":"2021-08-14T11:43:32.637615Z","shell.execute_reply":"2021-08-14T11:43:32.661166Z"},"trusted":true},"execution_count":146,"outputs":[{"name":"stdout","text":"Categorical columns that will be ordinal encoded: ['MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Neighborhood', 'Condition1', 'BldgType', 'HouseStyle', 'RoofStyle', 'Exterior1st', 'Exterior2nd', 'ExterQual', 'ExterCond', 'Foundation', 'Heating', 'HeatingQC', 'CentralAir', 'KitchenQual', 'PavedDrive', 'SaleType', 'SaleCondition']\n\nCategorical columns that will be dropped from the dataset: ['RoofMatl', 'Functional', 'Condition2']\n","output_type":"stream"}]},{"cell_type":"markdown","source":"### Part B\n\nUse the next code cell to ordinal encode the data in `X_train` and `X_valid`.  Set the preprocessed DataFrames to `label_X_train` and `label_X_valid`, respectively.  \n- We have provided code below to drop the categorical columns in `bad_label_cols` from the dataset. \n- You should ordinal encode the categorical columns in `good_label_cols`.  ","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OrdinalEncoder\n\n# Drop categorical columns that will not be encoded\nlabel_X_train = X_train.drop(bad_label_cols, axis=1)\nlabel_X_valid = X_valid.drop(bad_label_cols, axis=1)\n\n# Apply ordinal encoder \nordinal_encoder = OrdinalEncoder()\nlabel_X_train[good_label_cols] = ordinal_encoder.fit_transform(X_train[good_label_cols])\nlabel_X_valid[good_label_cols] = ordinal_encoder.transform(X_valid[good_label_cols])\n\n# Check your answer\nstep_2.b.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.664220Z","iopub.execute_input":"2021-08-14T11:43:32.664713Z","iopub.status.idle":"2021-08-14T11:43:32.739697Z","shell.execute_reply.started":"2021-08-14T11:43:32.664607Z","shell.execute_reply":"2021-08-14T11:43:32.738364Z"},"trusted":true},"execution_count":147,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"2.2_LabelB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#step_2.b.hint()\n#step_2.b.solution()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.741238Z","iopub.execute_input":"2021-08-14T11:43:32.741701Z","iopub.status.idle":"2021-08-14T11:43:32.745943Z","shell.execute_reply.started":"2021-08-14T11:43:32.741656Z","shell.execute_reply":"2021-08-14T11:43:32.744877Z"},"trusted":true},"execution_count":148,"outputs":[]},{"cell_type":"markdown","source":"Run the next code cell to get the MAE for this approach.","metadata":{}},{"cell_type":"code","source":"print(\"MAE from Approach 2 (Ordinal Encoding):\") \nprint(score_dataset(label_X_train, label_X_valid, y_train, y_valid))","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:32.747444Z","iopub.execute_input":"2021-08-14T11:43:32.747867Z","iopub.status.idle":"2021-08-14T11:43:34.374559Z","shell.execute_reply.started":"2021-08-14T11:43:32.747803Z","shell.execute_reply":"2021-08-14T11:43:34.373486Z"},"trusted":true},"execution_count":149,"outputs":[{"name":"stdout","text":"MAE from Approach 2 (Ordinal Encoding):\n17098.01649543379\n","output_type":"stream"}]},{"cell_type":"markdown","source":"So far, you've tried two different approaches to dealing with categorical variables.  And, you've seen that encoding categorical data yields better results than removing columns from the dataset.\n\nSoon, you'll try one-hot encoding.  Before then, there's one additional topic we need to cover.  Begin by running the next code cell without changes.  ","metadata":{}},{"cell_type":"code","source":"# Get number of unique entries in each column with categorical data\nobject_nunique = list(map(lambda col: X_train[col].nunique(), object_cols))\nd = dict(zip(object_cols, object_nunique))\n\n# Print number of unique entries by column, in ascending order\nsorted(d.items(), key=lambda x: x[1])","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.376024Z","iopub.execute_input":"2021-08-14T11:43:34.376611Z","iopub.status.idle":"2021-08-14T11:43:34.399417Z","shell.execute_reply.started":"2021-08-14T11:43:34.376564Z","shell.execute_reply":"2021-08-14T11:43:34.398219Z"},"trusted":true},"execution_count":150,"outputs":[{"execution_count":150,"output_type":"execute_result","data":{"text/plain":"[('Street', 2),\n ('Utilities', 2),\n ('CentralAir', 2),\n ('LandSlope', 3),\n ('PavedDrive', 3),\n ('LotShape', 4),\n ('LandContour', 4),\n ('ExterQual', 4),\n ('KitchenQual', 4),\n ('MSZoning', 5),\n ('LotConfig', 5),\n ('BldgType', 5),\n ('ExterCond', 5),\n ('HeatingQC', 5),\n ('Condition2', 6),\n ('RoofStyle', 6),\n ('Foundation', 6),\n ('Heating', 6),\n ('Functional', 6),\n ('SaleCondition', 6),\n ('RoofMatl', 7),\n ('HouseStyle', 8),\n ('Condition1', 9),\n ('SaleType', 9),\n ('Exterior1st', 15),\n ('Exterior2nd', 16),\n ('Neighborhood', 25)]"},"metadata":{}}]},{"cell_type":"markdown","source":"# Step 3: Investigating cardinality\n\n### Part A\n\nThe output above shows, for each column with categorical data, the number of unique values in the column.  For instance, the `'Street'` column in the training data has two unique values: `'Grvl'` and `'Pave'`, corresponding to a gravel road and a paved road, respectively.\n\nWe refer to the number of unique entries of a categorical variable as the **cardinality** of that categorical variable.  For instance, the `'Street'` variable has cardinality 2.\n\nUse the output above to answer the questions below.","metadata":{}},{"cell_type":"code","source":"# Fill in the line below: How many categorical variables in the training data\n# have cardinality greater than 10?\n# To make the code more dynamic I've updated the code to implement generic function that will get all the columns with value greater than 10\ncols_gr_10 = [(k,v) for k, v in d.items() if v > 10]\nhigh_cardinality_numcols = len(cols_gr_10)\n\n# Fill in the line below: How many columns are needed to one-hot encode the \n# 'Neighborhood' variable in the training data?\nnum_cols_neighborhood = d['Neighborhood']\n\n# Check your answers\nstep_3.a.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.401254Z","iopub.execute_input":"2021-08-14T11:43:34.401664Z","iopub.status.idle":"2021-08-14T11:43:34.411289Z","shell.execute_reply.started":"2021-08-14T11:43:34.401622Z","shell.execute_reply":"2021-08-14T11:43:34.410182Z"},"trusted":true},"execution_count":151,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"3.1_CardinalityA\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#step_3.a.hint()\n#step_3.a.solution()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.412584Z","iopub.execute_input":"2021-08-14T11:43:34.412906Z","iopub.status.idle":"2021-08-14T11:43:34.423537Z","shell.execute_reply.started":"2021-08-14T11:43:34.412865Z","shell.execute_reply":"2021-08-14T11:43:34.422620Z"},"trusted":true},"execution_count":152,"outputs":[]},{"cell_type":"markdown","source":"### Part B\n\nFor large datasets with many rows, one-hot encoding can greatly expand the size of the dataset.  For this reason, we typically will only one-hot encode columns with relatively low cardinality.  Then, high cardinality columns can either be dropped from the dataset, or we can use ordinal encoding.\n\nAs an example, consider a dataset with 10,000 rows, and containing one categorical column with 100 unique entries.  \n- If this column is replaced with the corresponding one-hot encoding, how many entries are added to the dataset?  \n- If we instead replace the column with the ordinal encoding, how many entries are added?  \n\nUse your answers to fill in the lines below.","metadata":{}},{"cell_type":"code","source":"# Fill in the line below: How many entries are added to the dataset by \n# replacing the column with a one-hot encoding?\n# In the question it's asked about number of entries added to the dataset.\n# Now in case of one-hot encoding we will add number of columns for all unique entries and we will remove the actual column\n# So total number of entries added to the dataset will be (unique_entries - 1)*total_rows\nOH_entries_added = 99*10000\n\n# Fill in the line below: How many entries are added to the dataset by\n# replacing the column with an ordinal encoding?\n# Ordinal encoding does not add new column so there will be no new entries to the dataset.\n#  It'll just replace the values for the existing columns\nlabel_entries_added = 0\n\n# Check your answers\nstep_3.b.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.425303Z","iopub.execute_input":"2021-08-14T11:43:34.426038Z","iopub.status.idle":"2021-08-14T11:43:34.441403Z","shell.execute_reply.started":"2021-08-14T11:43:34.425990Z","shell.execute_reply":"2021-08-14T11:43:34.440177Z"},"trusted":true},"execution_count":153,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 1, \"questionId\": \"3.2_CardinalityB\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#step_3.b.hint()\n#step_3.b.solution()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.445328Z","iopub.execute_input":"2021-08-14T11:43:34.445801Z","iopub.status.idle":"2021-08-14T11:43:34.457261Z","shell.execute_reply.started":"2021-08-14T11:43:34.445766Z","shell.execute_reply":"2021-08-14T11:43:34.456099Z"},"trusted":true},"execution_count":154,"outputs":[]},{"cell_type":"markdown","source":"Next, you'll experiment with one-hot encoding.  But, instead of encoding all of the categorical variables in the dataset, you'll only create a one-hot encoding for columns with cardinality less than 10.\n\nRun the code cell below without changes to set `low_cardinality_cols` to a Python list containing the columns that will be one-hot encoded.  Likewise, `high_cardinality_cols` contains a list of categorical columns that will be dropped from the dataset.","metadata":{}},{"cell_type":"code","source":"# Columns that will be one-hot encoded\nlow_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]\n\n# Columns that will be dropped from the dataset\nhigh_cardinality_cols = list(set(object_cols)-set(low_cardinality_cols))\n\nprint('Categorical columns that will be one-hot encoded:', low_cardinality_cols)\nprint('\\nCategorical columns that will be dropped from the dataset:', high_cardinality_cols)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.459669Z","iopub.execute_input":"2021-08-14T11:43:34.460319Z","iopub.status.idle":"2021-08-14T11:43:34.489864Z","shell.execute_reply.started":"2021-08-14T11:43:34.460264Z","shell.execute_reply":"2021-08-14T11:43:34.488626Z"},"trusted":true},"execution_count":155,"outputs":[{"name":"stdout","text":"Categorical columns that will be one-hot encoded: ['MSZoning', 'Street', 'LotShape', 'LandContour', 'Utilities', 'LotConfig', 'LandSlope', 'Condition1', 'Condition2', 'BldgType', 'HouseStyle', 'RoofStyle', 'RoofMatl', 'ExterQual', 'ExterCond', 'Foundation', 'Heating', 'HeatingQC', 'CentralAir', 'KitchenQual', 'Functional', 'PavedDrive', 'SaleType', 'SaleCondition']\n\nCategorical columns that will be dropped from the dataset: ['Exterior2nd', 'Neighborhood', 'Exterior1st']\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# Step 4: One-hot encoding\n\nUse the next code cell to one-hot encode the data in `X_train` and `X_valid`.  Set the preprocessed DataFrames to `OH_X_train` and `OH_X_valid`, respectively.  \n- The full list of categorical columns in the dataset can be found in the Python list `object_cols`.\n- You should only one-hot encode the categorical columns in `low_cardinality_cols`.  All other categorical columns should be dropped from the dataset. ","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import OneHotEncoder\n\n# Use as many lines of code as you need!\n# Remove high_cadinality columns from the dataset\nX_train_copy = X_train.drop(high_cardinality_cols, axis=1)\nX_valid_copy = X_valid.drop(high_cardinality_cols, axis=1)\n\n# As a first step we will initialize the OneHotEncoder for our task\n# We will initialize the encoder with handle_unknow='ignore' and sparse=False to ignore the unknown values.\n# Make sure to wrap the encoder output in pd.DataFrame to get the numpy array instead of the 2d array\none_hot_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)\none_hot_cols_X_train = pd.DataFrame(one_hot_encoder.fit_transform(X_train_copy[low_cardinality_cols]))\none_hot_cols_X_valid = pd.DataFrame(one_hot_encoder.transform(X_valid_copy[low_cardinality_cols]))\n\n# Now the one-hot encoder will remove the index from the dataset so we will need to add it back\none_hot_cols_X_train.index = X_train_copy.index\none_hot_cols_X_valid.index = X_valid_copy.index\n\n# Now we will remove the low_cardinality_cols from the dataset\nnumarical_X_train = X_train_copy.drop(low_cardinality_cols, axis=1)\nnumarical_X_valid = X_valid_copy.drop(low_cardinality_cols, axis=1)\n\n# Now to get actual training set with one-hot encoded columns, we will merge the columns for numarical values and hot encoded data\nOH_X_train = pd.concat([numarical_X_train, one_hot_cols_X_train], axis=1)\nOH_X_valid = pd.concat([numarical_X_valid, one_hot_cols_X_valid], axis=1)\n\n# Check your answer\nstep_4.check()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.491061Z","iopub.execute_input":"2021-08-14T11:43:34.491381Z","iopub.status.idle":"2021-08-14T11:43:34.545270Z","shell.execute_reply.started":"2021-08-14T11:43:34.491351Z","shell.execute_reply":"2021-08-14T11:43:34.544197Z"},"trusted":true},"execution_count":156,"outputs":[{"output_type":"display_data","data":{"text/plain":"<IPython.core.display.Javascript object>","application/javascript":"parent.postMessage({\"jupyterEvent\": \"custom.exercise_interaction\", \"data\": {\"outcomeType\": 1, \"valueTowardsCompletion\": 0.5, \"interactionType\": 1, \"questionType\": 2, \"questionId\": \"4_OneHot\", \"learnToolsVersion\": \"0.3.4\", \"failureMessage\": \"\", \"exceptionClass\": \"\", \"trace\": \"\"}}, \"*\")"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Correct","text/markdown":"<span style=\"color:#33cc33\">Correct</span>"},"metadata":{}}]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n# step_4.hint()\n#step_4.solution()","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.546521Z","iopub.execute_input":"2021-08-14T11:43:34.546824Z","iopub.status.idle":"2021-08-14T11:43:34.550552Z","shell.execute_reply.started":"2021-08-14T11:43:34.546773Z","shell.execute_reply":"2021-08-14T11:43:34.549543Z"},"trusted":true},"execution_count":157,"outputs":[]},{"cell_type":"markdown","source":"Run the next code cell to get the MAE for this approach.","metadata":{}},{"cell_type":"code","source":"print(\"MAE from Approach 3 (One-Hot Encoding):\") \nprint(score_dataset(OH_X_train, OH_X_valid, y_train, y_valid))","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:43:34.552088Z","iopub.execute_input":"2021-08-14T11:43:34.552405Z","iopub.status.idle":"2021-08-14T11:43:36.573091Z","shell.execute_reply.started":"2021-08-14T11:43:34.552375Z","shell.execute_reply":"2021-08-14T11:43:36.571863Z"},"trusted":true},"execution_count":158,"outputs":[{"name":"stdout","text":"MAE from Approach 3 (One-Hot Encoding):\n17525.345719178084\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# Generate test predictions and submit your results\n\nAfter you complete Step 4, if you'd like to use what you've learned to submit your results to the leaderboard, you'll need to preprocess the test data before generating predictions.\n\n**This step is completely optional, and you do not need to submit results to the leaderboard to successfully complete the exercise.**\n\nCheck out the previous exercise if you need help with remembering how to [join the competition](https://www.kaggle.com/c/home-data-for-ml-course) or save your results to CSV.  Once you have generated a file with your results, follow the instructions below:\n1. Begin by clicking on the **Save Version** button in the top right corner of the window.  This will generate a pop-up window.  \n2. Ensure that the **Save and Run All** option is selected, and then click on the **Save** button.\n3. This generates a window in the bottom left corner of the notebook.  After it has finished running, click on the number to the right of the **Save Version** button.  This pulls up a list of versions on the right of the screen.  Click on the ellipsis **(...)** to the right of the most recent version, and select **Open in Viewer**.  This brings you into view mode of the same page. You will need to scroll down to get back to these instructions.\n4. Click on the **Output** tab on the right of the screen.  Then, click on the file you would like to submit, and click on the blue **Submit** button to submit your results to the leaderboard.\n\nYou have now successfully submitted to the competition!\n\nIf you want to keep working to improve your performance, select the **Edit** button in the top right of the screen. Then you can change your code and repeat the process. There's a lot of room to improve, and you will climb up the leaderboard as you work.\n","metadata":{}},{"cell_type":"markdown","source":"# Prepare test data for X_test\nAs we need to predict the house prices from the X_test dataset we need to first prepare the dataset before using it for the prediction.","metadata":{}},{"cell_type":"markdown","source":"## 1. Remove high cardinality columns from the data","metadata":{}},{"cell_type":"code","source":"# Drop all the columns with high_cardinality_cols. This columns have cardinality values more than 10\nX_test_copy = X_test.drop(high_cardinality_cols, axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:44:28.022577Z","iopub.execute_input":"2021-08-14T11:44:28.023028Z","iopub.status.idle":"2021-08-14T11:44:28.031551Z","shell.execute_reply.started":"2021-08-14T11:44:28.022995Z","shell.execute_reply":"2021-08-14T11:44:28.029790Z"},"trusted":true},"execution_count":166,"outputs":[]},{"cell_type":"markdown","source":"## 2. Fix all the NaN values in low_cardinality_cols\n\nWe need to impute the data for the `low_cardinality_cols`.\nAs all the columns in `low_cardinality_cols` have string data we will use simple imputer and add \"missing\" as in place of all `NaN` values.","metadata":{}},{"cell_type":"code","source":"# Import SimpleImputer. This will be used to impute the Nan value during data processing step for test dataset\nfrom sklearn.impute import SimpleImputer\n\n# we will impute the data with Simple Imputer with strategy=c\"onstant\" and fill_value=\"missing\"\nstring_data_imputer = SimpleImputer(strategy=\"constant\", fill_value=\"missing\")\n\n# assign all the required columns to temporary variable\nX_test_imputed_cols = X_test_copy[low_cardinality_cols]\n\n# Impute the data and assign it back to the same temporary variable\nX_test_imputed = pd.DataFrame(string_data_imputer.fit_transform(X_test_imputed_cols))\n\n# Imputer will remove the column name so we will assign the column names\nX_test_imputed.columns = X_test_imputed_cols.columns","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:44:31.631548Z","iopub.execute_input":"2021-08-14T11:44:31.632105Z","iopub.status.idle":"2021-08-14T11:44:31.651430Z","shell.execute_reply.started":"2021-08-14T11:44:31.632070Z","shell.execute_reply":"2021-08-14T11:44:31.649961Z"},"trusted":true},"execution_count":167,"outputs":[]},{"cell_type":"markdown","source":"## 3. Fix all the NaN values in numarical cols in the data\n\nWe need to impute the data for the columns with numarical values.\nAs all the data other then `low_cardinality_cols` are numarical we will take those data in a variable and will impute the data with simple imputer","metadata":{}},{"cell_type":"code","source":"# Create Imputer for numarical values.\nnum_data_imputer = SimpleImputer()\n\n# now we will remove low_cardinality_cols from the test set and get all numaric columns\nnumarical_X_test = X_test_copy.drop(low_cardinality_cols, axis=1)\n\n# Impute the numarical columns of the dataset \nnumarical_X_test_imputed = pd.DataFrame(num_data_imputer.fit_transform(numarical_X_test))\n\n# As imputer removes the column from the data we need to add columns back to the dataset\nnumarical_X_test_imputed.columns = numarical_X_test.columns","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:44:35.818940Z","iopub.execute_input":"2021-08-14T11:44:35.819385Z","iopub.status.idle":"2021-08-14T11:44:35.834930Z","shell.execute_reply.started":"2021-08-14T11:44:35.819344Z","shell.execute_reply":"2021-08-14T11:44:35.833691Z"},"trusted":true},"execution_count":168,"outputs":[]},{"cell_type":"markdown","source":"## 4. Use one-hot encoding and get final test dataset\n\nUse the `one_hot_encoder` defined in the previous step(step 4: One-hot encoding) to get one-hot encoded values for all the `low_cardinality_cols`.\n\nTo get actual test dataset we need to merge `imputed numarical` columns and `one hot encoded` low_cardinality_cols.","metadata":{}},{"cell_type":"code","source":"# now we will use the one_hot_encoder to get the ecoded columns with low_cardinality_cols\none_hot_cols_X_test = pd.DataFrame(one_hot_encoder.transform(X_test_imputed))\n\n# add the index back to the one hot encoded cols\none_hot_cols_X_test.index = X_test_imputed.index\n\n# Now we will merge the imputed numarical cols with one-hot encoded columns to get final test dataset\none_hot_X_test = pd.concat([numarical_X_test_imputed, one_hot_cols_X_test], axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:44:38.034895Z","iopub.execute_input":"2021-08-14T11:44:38.035275Z","iopub.status.idle":"2021-08-14T11:44:38.065983Z","shell.execute_reply.started":"2021-08-14T11:44:38.035244Z","shell.execute_reply":"2021-08-14T11:44:38.064962Z"},"trusted":true},"execution_count":169,"outputs":[]},{"cell_type":"markdown","source":"## 5. Train the model with RandomForestRegressor\n\nTrain the model using RandomForestRegressor with `n_estimators=100` and `random_state=0`.\n\nuse `OH_X_train` from the previous step (step 4: One-hot encoding) and train(fit) the model","metadata":{}},{"cell_type":"code","source":"# Create a model with the one-hot encoding columns\nmodel = RandomForestRegressor(n_estimators=100, random_state=0)\n\n# Fit the model using the data from the previous step\nmodel.fit(OH_X_train, y_train)","metadata":{"execution":{"iopub.status.busy":"2021-08-14T11:44:41.035803Z","iopub.execute_input":"2021-08-14T11:44:41.036616Z","iopub.status.idle":"2021-08-14T11:44:43.025129Z","shell.execute_reply.started":"2021-08-14T11:44:41.036570Z","shell.execute_reply":"2021-08-14T11:44:43.024109Z"},"trusted":true},"execution_count":170,"outputs":[{"execution_count":170,"output_type":"execute_result","data":{"text/plain":"RandomForestRegressor(random_state=0)"},"metadata":{}}]},{"cell_type":"markdown","source":"## 6. Predit the house prices and print the result","metadata":{}},{"cell_type":"code","source":"# Predict the test dataset using our trained model\npreds_test = model.predict(one_hot_X_test)\n\nprint(preds_test)\n","metadata":{"execution":{"iopub.status.busy":"2021-08-14T12:10:38.866920Z","iopub.execute_input":"2021-08-14T12:10:38.867343Z","iopub.status.idle":"2021-08-14T12:10:38.918490Z","shell.execute_reply.started":"2021-08-14T12:10:38.867310Z","shell.execute_reply":"2021-08-14T12:10:38.916918Z"},"trusted":true},"execution_count":205,"outputs":[{"name":"stdout","text":"[127654.5  157672.   182222.9  ... 153695.25 109710.85 223045.55]\n","output_type":"stream"}]},{"cell_type":"markdown","source":"## 7. Save the result for submission\n\nWe need to submit our csv file with the correct values for the Id. In this exercise the `Id` does not start from 0. to get the actual values of the Ids add `X_test.index[one_hot_X_test.index]` as 'Id' field in below dataFrame","metadata":{}},{"cell_type":"code","source":"# Save test predictions to file\noutput = pd.DataFrame({'Id': X_test.index[one_hot_X_test.index],\n                       'SalePrice': preds_test})\noutput.to_csv('submission_categorical_variables.csv', index=False)\nprint(\"Successully generated file for submission\")","metadata":{"execution":{"iopub.status.busy":"2021-08-14T12:11:52.487141Z","iopub.execute_input":"2021-08-14T12:11:52.487594Z","iopub.status.idle":"2021-08-14T12:11:52.502417Z","shell.execute_reply.started":"2021-08-14T12:11:52.487557Z","shell.execute_reply":"2021-08-14T12:11:52.501095Z"},"trusted":true},"execution_count":208,"outputs":[{"name":"stdout","text":"Successully generated file for submission\n","output_type":"stream"}]},{"cell_type":"markdown","source":"# Keep going\n\nWith missing value handling and categorical encoding, your modeling process is getting complex. This complexity gets worse when you want to save your model to use in the future. The key to managing this complexity is something called **pipelines**. \n\n**[Learn to use pipelines](https://www.kaggle.com/alexisbcook/pipelines)** to preprocess datasets with categorical variables, missing values and any other messiness your data throws at you.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/161289) to chat with other Learners.*","metadata":{}}]}