car-fuel-cost-prediction.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ce9cb1b9-292a-40eb-8cc3-9911212796b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "\n",
    "# surpress warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecbcd526-1932-43db-90f2-6e470af564ac",
   "metadata": {},
   "source": [
    "# Car Fuel Cost Prediction\n",
    "\n",
    "### Brief problem description\n",
    "This is a regression problem, where given various vehicle characteristics we predict the fuel cost at 12,000 miles.\n",
    "\n",
    "Dataset: https://github.com/amercader/car-fuel-and-emissions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a30d01e-520e-4504-9915-339c82c4ee6b",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-1:** Import the dataset and define the target column for prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a3bf888c-db56-43c3-959b-565b615e1fee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "# read dataset\n",
    "df = pd.read_csv(\"data.csv\")\n",
    "\n",
    "# target\n",
    "target = \"fuel_cost_12000_miles\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3273b32-d219-40a0-b4e3-4710a7972132",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-2:** Drop examples for which we don't have the target value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d90db355-8bbd-48a3-9449-d48f8bb6db5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# keep only examples where the target is not missing\n",
    "df = df[df[target].notna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e53617b-46b3-4ce8-a254-bbde5fefca4d",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-3:** Investigate the percentage of missing values for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "642468c8-7361-472d-81f3-86aa992b92c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage (%) of missing values per column:\n",
      "\n",
      "file                        0.000000\n",
      "year                        0.000000\n",
      "manufacturer                0.000000\n",
      "model                       0.000000\n",
      "description                 0.000000\n",
      "euro_standard               0.000000\n",
      "tax_band                   76.361057\n",
      "transmission                0.030179\n",
      "transmission_type           1.029092\n",
      "engine_capacity             0.021125\n",
      "fuel_type                   0.000000\n",
      "urban_metric                0.039232\n",
      "extra_urban_metric          0.039232\n",
      "combined_metric             0.021125\n",
      "urban_imperial              0.039232\n",
      "extra_urban_imperial        0.039232\n",
      "combined_imperial           0.021125\n",
      "noise_level                 0.027161\n",
      "co2                         0.000000\n",
      "thc_emissions              50.250483\n",
      "co_emissions                0.093554\n",
      "nox_emissions               0.238411\n",
      "thc_nox_emissions          83.612989\n",
      "particulates_emissions     60.179261\n",
      "fuel_cost_12000_miles       0.000000\n",
      "fuel_cost_6000_miles      100.000000\n",
      "standard_12_months         89.392202\n",
      "standard_6_months          91.175761\n",
      "first_year_12_months       89.392202\n",
      "first_year_6_months        95.723684\n",
      "date_of_change            100.000000\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# check for missing values in the data\n",
    "print(\"Percentage (%) of missing values per column:\\n\")\n",
    "print(df.isna().sum() * 100 / len(df))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f32c52eb-f5d7-4f7c-b490-d488ddc9858a",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-4:** Keep only features with small percentage of missing values.\n",
    "\n",
    "We are going to drop the following features, since they have an unacceptable percentage of missing values:\n",
    "* tax_band\n",
    "* thc_emissions\n",
    "* thc_nox_emissions\n",
    "* particulates_emissions\n",
    "* fuel_cost_6000_miles\n",
    "* standard_12_months\n",
    "* standard_6_months\n",
    "* first_year_12_months\n",
    "* first_year_6_months\n",
    "* date_of_change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "27ca07d8-494f-44ea-a34c-644d3966a315",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop features with too many missing values\n",
    "df = df.drop(columns=[\n",
    "    \"tax_band\",\n",
    "    \"thc_emissions\",\n",
    "    \"thc_nox_emissions\",\n",
    "    \"particulates_emissions\",\n",
    "    \"fuel_cost_6000_miles\",\n",
    "    \"standard_12_months\",\n",
    "    \"standard_6_months\",\n",
    "    \"first_year_12_months\",\n",
    "    \"first_year_6_months\",\n",
    "    \"date_of_change\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e830869d-98b9-4ae6-b13c-9de2ba9581f8",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-5:** Print all features of 10 random examples to examine how the data looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1b5795c5-040c-4000-8714-06f91b75091a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>file</th>\n",
       "      <th>year</th>\n",
       "      <th>manufacturer</th>\n",
       "      <th>model</th>\n",
       "      <th>description</th>\n",
       "      <th>euro_standard</th>\n",
       "      <th>transmission</th>\n",
       "      <th>transmission_type</th>\n",
       "      <th>engine_capacity</th>\n",
       "      <th>fuel_type</th>\n",
       "      <th>urban_metric</th>\n",
       "      <th>extra_urban_metric</th>\n",
       "      <th>combined_metric</th>\n",
       "      <th>urban_imperial</th>\n",
       "      <th>extra_urban_imperial</th>\n",
       "      <th>combined_imperial</th>\n",
       "      <th>noise_level</th>\n",
       "      <th>co2</th>\n",
       "      <th>co_emissions</th>\n",
       "      <th>nox_emissions</th>\n",
       "      <th>fuel_cost_12000_miles</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18814</th>\n",
       "      <td>Part_A_Euro_IV_may2006.csv</td>\n",
       "      <td>2006</td>\n",
       "      <td>Vauxhall</td>\n",
       "      <td>Corsa, MY2006</td>\n",
       "      <td>1.4i 16v 5 Door Hatchback with 14\"/15 tyre</td>\n",
       "      <td>4</td>\n",
       "      <td>A4</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>1389.0</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>10.4</td>\n",
       "      <td>5.8</td>\n",
       "      <td>7.5</td>\n",
       "      <td>27.1</td>\n",
       "      <td>48.7</td>\n",
       "      <td>37.6</td>\n",
       "      <td>72.0</td>\n",
       "      <td>180</td>\n",
       "      <td>287.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>1306.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30379</th>\n",
       "      <td>download-data-for-May-2010-Euro-4.csv</td>\n",
       "      <td>2010</td>\n",
       "      <td>Chrysler Jeep</td>\n",
       "      <td>Jeep Wrangler Model Year 2009</td>\n",
       "      <td>2.8 Sport (JK72)</td>\n",
       "      <td>4</td>\n",
       "      <td>6MT</td>\n",
       "      <td>Manual</td>\n",
       "      <td>2777.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>10.3</td>\n",
       "      <td>6.9</td>\n",
       "      <td>8.2</td>\n",
       "      <td>27.4</td>\n",
       "      <td>40.9</td>\n",
       "      <td>34.4</td>\n",
       "      <td>75.0</td>\n",
       "      <td>215</td>\n",
       "      <td>48.0</td>\n",
       "      <td>324.0</td>\n",
       "      <td>1792.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26855</th>\n",
       "      <td>download-data-for-May-2009-Euro-4.csv</td>\n",
       "      <td>2009</td>\n",
       "      <td>Ford</td>\n",
       "      <td>Galaxy, Post 2009ﾽ Model Year</td>\n",
       "      <td>2.0 Duratec (145PS)</td>\n",
       "      <td>4</td>\n",
       "      <td>M5</td>\n",
       "      <td>Manual</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>11.2</td>\n",
       "      <td>6.5</td>\n",
       "      <td>8.2</td>\n",
       "      <td>25.2</td>\n",
       "      <td>43.5</td>\n",
       "      <td>34.4</td>\n",
       "      <td>72.0</td>\n",
       "      <td>197</td>\n",
       "      <td>506.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>1396.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34978</th>\n",
       "      <td>Euro 5 snapshot 2011.csv</td>\n",
       "      <td>2011</td>\n",
       "      <td>Ford</td>\n",
       "      <td>New Focus, Model Year Post 2011½</td>\n",
       "      <td>2.0 Duratorq TDCi (163PS) Estate (+DPF)</td>\n",
       "      <td>5</td>\n",
       "      <td>A6</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>6.8</td>\n",
       "      <td>4.4</td>\n",
       "      <td>5.3</td>\n",
       "      <td>41.5</td>\n",
       "      <td>64.2</td>\n",
       "      <td>53.3</td>\n",
       "      <td>69.0</td>\n",
       "      <td>139</td>\n",
       "      <td>178.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>1423.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37004</th>\n",
       "      <td>Euro 5 snapshot 2011.csv</td>\n",
       "      <td>2011</td>\n",
       "      <td>Toyota</td>\n",
       "      <td>Auris, Model Year 2010</td>\n",
       "      <td>TR 1.4 D-4D 6-speed Multimode</td>\n",
       "      <td>5</td>\n",
       "      <td>Multi6</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>1364.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>5.8</td>\n",
       "      <td>4.4</td>\n",
       "      <td>4.9</td>\n",
       "      <td>48.7</td>\n",
       "      <td>64.2</td>\n",
       "      <td>57.6</td>\n",
       "      <td>67.0</td>\n",
       "      <td>130</td>\n",
       "      <td>149.0</td>\n",
       "      <td>152.0</td>\n",
       "      <td>1316.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39013</th>\n",
       "      <td>download-data-for-Aug-2012-Euro-5.csv</td>\n",
       "      <td>2012</td>\n",
       "      <td>Kia</td>\n",
       "      <td>cee'd SW</td>\n",
       "      <td>'4' 1.6 CRDi 126bhp 6-speed manual</td>\n",
       "      <td>5</td>\n",
       "      <td>M6</td>\n",
       "      <td>Manual</td>\n",
       "      <td>1582.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>5.4</td>\n",
       "      <td>4.3</td>\n",
       "      <td>4.7</td>\n",
       "      <td>52.3</td>\n",
       "      <td>65.7</td>\n",
       "      <td>60.1</td>\n",
       "      <td>72.0</td>\n",
       "      <td>124</td>\n",
       "      <td>172.0</td>\n",
       "      <td>128.0</td>\n",
       "      <td>1334.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43408</th>\n",
       "      <td>download-data-for-Aug-2013-Euro-5.csv</td>\n",
       "      <td>2013</td>\n",
       "      <td>Peugeot</td>\n",
       "      <td>2008</td>\n",
       "      <td>1.4 HDi 70</td>\n",
       "      <td>5</td>\n",
       "      <td>M5</td>\n",
       "      <td>Manual</td>\n",
       "      <td>1398.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>4.6</td>\n",
       "      <td>3.6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>61.4</td>\n",
       "      <td>78.5</td>\n",
       "      <td>70.6</td>\n",
       "      <td>68.9</td>\n",
       "      <td>104</td>\n",
       "      <td>368.0</td>\n",
       "      <td>172.0</td>\n",
       "      <td>1120.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42566</th>\n",
       "      <td>download-data-for-Aug-2013-Euro-5.csv</td>\n",
       "      <td>2013</td>\n",
       "      <td>Hyundai</td>\n",
       "      <td>ix35</td>\n",
       "      <td>2.0l CRDi 4x4 (17\" wheels)</td>\n",
       "      <td>5</td>\n",
       "      <td>A6</td>\n",
       "      <td>Automatic</td>\n",
       "      <td>1995.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>9.1</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>31.0</td>\n",
       "      <td>47.1</td>\n",
       "      <td>39.8</td>\n",
       "      <td>73.0</td>\n",
       "      <td>187</td>\n",
       "      <td>279.0</td>\n",
       "      <td>133.0</td>\n",
       "      <td>1987.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30451</th>\n",
       "      <td>download-data-for-May-2010-Euro-5.csv</td>\n",
       "      <td>2010</td>\n",
       "      <td>Citroen</td>\n",
       "      <td>DS3</td>\n",
       "      <td>1.6 THP 150hp</td>\n",
       "      <td>5</td>\n",
       "      <td>M6</td>\n",
       "      <td>Manual</td>\n",
       "      <td>1598.0</td>\n",
       "      <td>Petrol</td>\n",
       "      <td>9.4</td>\n",
       "      <td>5.1</td>\n",
       "      <td>6.7</td>\n",
       "      <td>30.1</td>\n",
       "      <td>55.4</td>\n",
       "      <td>42.2</td>\n",
       "      <td>70.8</td>\n",
       "      <td>155</td>\n",
       "      <td>256.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1448.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39785</th>\n",
       "      <td>download-data-for-Aug-2012-Euro-5.csv</td>\n",
       "      <td>2012</td>\n",
       "      <td>Peugeot</td>\n",
       "      <td>3008</td>\n",
       "      <td>1.6 HDi 112 Access/Active</td>\n",
       "      <td>5</td>\n",
       "      <td>M6</td>\n",
       "      <td>Manual</td>\n",
       "      <td>1560.0</td>\n",
       "      <td>Diesel</td>\n",
       "      <td>6.3</td>\n",
       "      <td>4.2</td>\n",
       "      <td>5.0</td>\n",
       "      <td>44.8</td>\n",
       "      <td>67.3</td>\n",
       "      <td>56.5</td>\n",
       "      <td>72.9</td>\n",
       "      <td>130</td>\n",
       "      <td>309.0</td>\n",
       "      <td>146.0</td>\n",
       "      <td>1419.41</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        file  year   manufacturer  \\\n",
       "18814             Part_A_Euro_IV_may2006.csv  2006       Vauxhall   \n",
       "30379  download-data-for-May-2010-Euro-4.csv  2010  Chrysler Jeep   \n",
       "26855  download-data-for-May-2009-Euro-4.csv  2009           Ford   \n",
       "34978               Euro 5 snapshot 2011.csv  2011           Ford   \n",
       "37004               Euro 5 snapshot 2011.csv  2011         Toyota   \n",
       "39013  download-data-for-Aug-2012-Euro-5.csv  2012            Kia   \n",
       "43408  download-data-for-Aug-2013-Euro-5.csv  2013        Peugeot   \n",
       "42566  download-data-for-Aug-2013-Euro-5.csv  2013        Hyundai   \n",
       "30451  download-data-for-May-2010-Euro-5.csv  2010        Citroen   \n",
       "39785  download-data-for-Aug-2012-Euro-5.csv  2012        Peugeot   \n",
       "\n",
       "                                  model  \\\n",
       "18814                     Corsa, MY2006   \n",
       "30379     Jeep Wrangler Model Year 2009   \n",
       "26855     Galaxy, Post 2009ﾽ Model Year   \n",
       "34978  New Focus, Model Year Post 2011½   \n",
       "37004            Auris, Model Year 2010   \n",
       "39013                          cee'd SW   \n",
       "43408                              2008   \n",
       "42566                              ix35   \n",
       "30451                               DS3   \n",
       "39785                              3008   \n",
       "\n",
       "                                      description  euro_standard transmission  \\\n",
       "18814  1.4i 16v 5 Door Hatchback with 14\"/15 tyre              4           A4   \n",
       "30379                            2.8 Sport (JK72)              4          6MT   \n",
       "26855                         2.0 Duratec (145PS)              4           M5   \n",
       "34978     2.0 Duratorq TDCi (163PS) Estate (+DPF)              5           A6   \n",
       "37004               TR 1.4 D-4D 6-speed Multimode              5       Multi6   \n",
       "39013          '4' 1.6 CRDi 126bhp 6-speed manual              5           M6   \n",
       "43408                                  1.4 HDi 70              5           M5   \n",
       "42566                  2.0l CRDi 4x4 (17\" wheels)              5           A6   \n",
       "30451                               1.6 THP 150hp              5           M6   \n",
       "39785                   1.6 HDi 112 Access/Active              5           M6   \n",
       "\n",
       "      transmission_type  engine_capacity fuel_type  urban_metric  \\\n",
       "18814         Automatic           1389.0    Petrol          10.4   \n",
       "30379            Manual           2777.0    Diesel          10.3   \n",
       "26855            Manual           1999.0    Petrol          11.2   \n",
       "34978         Automatic           1997.0    Diesel           6.8   \n",
       "37004         Automatic           1364.0    Diesel           5.8   \n",
       "39013            Manual           1582.0    Diesel           5.4   \n",
       "43408            Manual           1398.0    Diesel           4.6   \n",
       "42566         Automatic           1995.0    Diesel           9.1   \n",
       "30451            Manual           1598.0    Petrol           9.4   \n",
       "39785            Manual           1560.0    Diesel           6.3   \n",
       "\n",
       "       extra_urban_metric  combined_metric  urban_imperial  \\\n",
       "18814                 5.8              7.5            27.1   \n",
       "30379                 6.9              8.2            27.4   \n",
       "26855                 6.5              8.2            25.2   \n",
       "34978                 4.4              5.3            41.5   \n",
       "37004                 4.4              4.9            48.7   \n",
       "39013                 4.3              4.7            52.3   \n",
       "43408                 3.6              4.0            61.4   \n",
       "42566                 6.0              7.1            31.0   \n",
       "30451                 5.1              6.7            30.1   \n",
       "39785                 4.2              5.0            44.8   \n",
       "\n",
       "       extra_urban_imperial  combined_imperial  noise_level  co2  \\\n",
       "18814                  48.7               37.6         72.0  180   \n",
       "30379                  40.9               34.4         75.0  215   \n",
       "26855                  43.5               34.4         72.0  197   \n",
       "34978                  64.2               53.3         69.0  139   \n",
       "37004                  64.2               57.6         67.0  130   \n",
       "39013                  65.7               60.1         72.0  124   \n",
       "43408                  78.5               70.6         68.9  104   \n",
       "42566                  47.1               39.8         73.0  187   \n",
       "30451                  55.4               42.2         70.8  155   \n",
       "39785                  67.3               56.5         72.9  130   \n",
       "\n",
       "       co_emissions  nox_emissions  fuel_cost_12000_miles  \n",
       "18814         287.0           57.0                1306.00  \n",
       "30379          48.0          324.0                1792.00  \n",
       "26855         506.0           52.0                1396.00  \n",
       "34978         178.0          166.0                1423.00  \n",
       "37004         149.0          152.0                1316.00  \n",
       "39013         172.0          128.0                1334.30  \n",
       "43408         368.0          172.0                1120.00  \n",
       "42566         279.0          133.0                1987.00  \n",
       "30451         256.0           18.0                1448.00  \n",
       "39785         309.0          146.0                1419.41  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set option to display all columns\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "# print 10 random examples\n",
    "df.sample(10, random_state=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e9c1f06-5b78-4694-a8b1-d21a2d90a4c1",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-6:** Drop additional features for modelling and generalization reasons.\n",
    "\n",
    "We are going to drop the following features:\n",
    "* file -> Considered irrelevant for our modelling.\n",
    "* model, description -> Do not generalize well. We do not want to overfit according to the model name or description. For example some model lines are targeted as daily/family vehicles while others as sport/leisure cars.\n",
    "* transmission -> Does not generalize well. Many manufacturers follow different naming conventions for similar technology mainly for marketing purposes, we do not want to overfit these conventions. Instead, we are going to keep the 'transmission_type' feature, which provides part of the information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6131fe14-1497-441e-a800-76df1e2d5509",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop features that don't serve our modelling\n",
    "df = df.drop(columns=[\n",
    "    \"file\",\n",
    "    \"model\",\n",
    "    \"description\",\n",
    "    \"transmission\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d7af967-ffff-4cd2-baf6-4dfbc214fd92",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-7:** Organize features into numerical and categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "395790b0-7776-43ed-9283-4a7388950e34",
   "metadata": {},
   "outputs": [],
   "source": [
    "# organize features\n",
    "numerical_features = [\n",
    "    \"year\",\n",
    "    \"euro_standard\",\n",
    "    \"engine_capacity\",\n",
    "    \"urban_metric\",\n",
    "    \"extra_urban_metric\",\n",
    "    \"combined_metric\",\n",
    "    \"urban_imperial\",\n",
    "    \"extra_urban_imperial\",\n",
    "    \"combined_imperial\",\n",
    "    \"noise_level\",\n",
    "    \"co2\",\n",
    "    \"co_emissions\",\n",
    "    \"nox_emissions\"]\n",
    "\n",
    "categorical_features = [\n",
    "    \"manufacturer\",\n",
    "    \"transmission_type\",\n",
    "    \"fuel_type\"]\n",
    "\n",
    "# ensure correct column type for numerical features\n",
    "for column in numerical_features + [target]:\n",
    "    df[column] = pd.to_numeric(df[column])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c38e103-facf-4099-98be-e0035b3f5fd1",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-8:** Define separate preprocessing pipelines for numerical and categorical features.\n",
    "\n",
    "* For numerical features first fill the missing values using k-nearest neighbors imputation and then scale them to have zero mean and unit variance.\n",
    "\n",
    "* For categorical features first fill the missing values with the most frequent and then one-hot encode them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "923ec185-384e-44a6-872a-eda0a8dbd075",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import KNNImputer, SimpleImputer\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "\n",
    "\n",
    "# preprocessing for numerical features\n",
    "numerical_preprocessing_pipeline = make_pipeline(KNNImputer(), StandardScaler())\n",
    "\n",
    "# preprocessing for categorical features\n",
    "categorical_preprocessing_pipeline = make_pipeline(SimpleImputer(strategy=\"most_frequent\"), OneHotEncoder(handle_unknown=\"ignore\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408bf3bd-4706-491b-a9d4-e4ba03e08996",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-9:** Define column transformer that applies the appropriate preprocessing pipeline on each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "aea4ab7c-8c81-4f54-9ffc-5de8a4b37787",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "\n",
    "# transformation according to feature type\n",
    "transformer = ColumnTransformer(transformers=[\n",
    "    (\"categorical_transformer\", categorical_preprocessing_pipeline, categorical_features),\n",
    "    (\"numerical_transformer\", numerical_preprocessing_pipeline, numerical_features)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd510c08-dd41-4f59-b73f-cb700fe503f8",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-10:** Split data into X, y for training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "004a1f82-64be-46a9-bfcb-7de3b2125ec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "# get X, y\n",
    "X = df[numerical_features + categorical_features]\n",
    "y = df[target]\n",
    "\n",
    "# split for training and testing\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974e2a21-7f3a-47d1-aea1-e67765c7ac94",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-11:** Train different models and test their performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6f20d37c-0ced-4865-b0fb-e7af46d71f86",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.svm import LinearSVR\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "\n",
    "# models to examine\n",
    "models = [\n",
    "    LinearRegression(),\n",
    "    LinearSVR(random_state=0),\n",
    "    DecisionTreeRegressor(random_state=0),\n",
    "    RandomForestRegressor(random_state=0)]\n",
    "\n",
    "# initialize dict to keep test scores\n",
    "scores = {}\n",
    "\n",
    "# train and test models\n",
    "for model in models:\n",
    "    regressor = make_pipeline(transformer, model)\n",
    "    regressor.fit(X_train, y_train)\n",
    "    y_pred = regressor.predict(X_test)\n",
    "    \n",
    "    scores[str(model)] = [\n",
    "        metrics.mean_absolute_error(y_test, y_pred),\n",
    "        metrics.mean_squared_error(y_test, y_pred, squared=False),\n",
    "        metrics.r2_score(y_test, y_pred)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6817e2-e642-4704-a2b2-dd75572d6373",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-12:** Examine model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "b5ad2596-f2c2-4939-9574-0d4b2f300f74",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LinearRegression()</th>\n",
       "      <th>LinearSVR(random_state=0)</th>\n",
       "      <th>DecisionTreeRegressor(random_state=0)</th>\n",
       "      <th>RandomForestRegressor(random_state=0)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>MAE</th>\n",
       "      <td>124.90</td>\n",
       "      <td>121.95</td>\n",
       "      <td>4.26</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RMSE</th>\n",
       "      <td>1094.18</td>\n",
       "      <td>786.21</td>\n",
       "      <td>38.56</td>\n",
       "      <td>32.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R-squared</th>\n",
       "      <td>-3.55</td>\n",
       "      <td>-1.35</td>\n",
       "      <td>0.99</td>\n",
       "      <td>1.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           LinearRegression()  LinearSVR(random_state=0)  \\\n",
       "MAE                    124.90                     121.95   \n",
       "RMSE                  1094.18                     786.21   \n",
       "R-squared               -3.55                      -1.35   \n",
       "\n",
       "           DecisionTreeRegressor(random_state=0)  \\\n",
       "MAE                                         4.26   \n",
       "RMSE                                       38.56   \n",
       "R-squared                                   0.99   \n",
       "\n",
       "           RandomForestRegressor(random_state=0)  \n",
       "MAE                                         5.00  \n",
       "RMSE                                       32.18  \n",
       "R-squared                                   1.00  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create dataframe\n",
    "df_scores = pd.DataFrame.from_dict(scores)\n",
    "df_scores.index = [\"MAE\", \"RMSE\", \"R-squared\"]\n",
    "\n",
    "# display df\n",
    "df_scores.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f6fc57-1bcf-44d2-a467-828301059375",
   "metadata": {},
   "source": [
    "<br/><br/>\n",
    "**Step-13:** Selecting a model.\n",
    "\n",
    "* LinearRegression and LinearSVR achieve comparable results, but as indicated by the R-squared metric fail to capture the trend of the data.\n",
    "* DecisionTreeRegressor and RandomForestRegressor achieve comparable results and a good performance on all examined metrics.\n",
    "* DecisionTreeRegressor has worse RMSE but better MAE compared to RandomForestRegressor.\n",
    "* DecisionTreeRegressor is a simpler, more explainable and interpretable model compared to RandomForestRegressor.\n",
    "\n",
    "Considering the above:\n",
    "* if we need an interpretable model and/or MAE is more important than RMSE, then we choose the DecisionTreeRegressor\n",
    "* if we don't need and interpretable model and RMSE is more important than MAE, then we choose the RandomForestRegressor"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}