4.1_Handling_categorical_features .ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append('demo_datasets')\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handling categorical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this short notebook we discuss a simple yet effective way of dealing with categorical features. Many prediction tasks (especially in social sciences) involve features which do not take numbers as values, including gender, race, nationality, education level, etc., to just name a few examples. Many machine learning models however (including the ones we discuss in [Machine Learning Refined](http://www.mlrefined.com)) are not capable of incorporating these features in their original format. Therefore some pre-processing must be applied to categorical features before feeding them into these learning models, which we discuss in this notebook. \n",
    "\n",
    ">The content of this notebook is supplementary material for the textbook Machine Learning Refined (Cambridge University Press, 2016). Visit http://www.mlrefined.com for free chapter downloads and tutorials, and [our Amazon site here](https://www.amazon.com/Machine-Learning-Refined-Foundations-Applications/dp/1107123526/ref=sr_1_1?ie=UTF8&qid=1474491657&sr=8-1&keywords=machine+learning+refined) for details regarding a hard copy of the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 'Dummy' features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets assume we want to classify individuals into two classes, affluent and not-affluent, using data that has, among many other features, the *most commonly used method of transportation* as an input feature with four possible outcomes: **walking**, **biking**, **driving**, and **public transportation**. How can we translate these outcomes into numbers decipherable by computers? Well, the first approach anyone might guess is to assign a distinct number to each outcome, e.g., 1 to walking, 2 to biking, 3 to driving, and 4 to public transportation. Seems easy enough!\n",
    "\n",
    "<img src=\"demo_images/dummy_1.png\" width=650 height=450/>\n",
    "\n",
    "There is however one issue with this approach. Imagine there are two individuals in our dataset, both belonging to the not-affluent class, who differ from each other only in terms of their most commonly used method of transportation. Lets call them Trey and Matt. Trey walks to work everyday while Matt takes the bus. Recall from our discussion of histogram features for real data types in [Section 4.6 of the book](http://media.wix.com/ugd/f09e45_cc1cba3852eb40da8395636f34c755fa.pdf), that\n",
    "\n",
    "<img src=\"demo_images/quote.png\" width=550 height=450/>\n",
    "\n",
    "That is, we generally want that instances from the same class to stay close to each other in the feature space and far away from instances of the other class(es). The current encoding of the transportation feature does not satisfy this desire, at least not for Trey and Matt who are - with the current encoding - maximally distant from one another! But also remember we assigned numbers 1 through 4 to the four outcomes arbitrarily. We could have instead encoded the outcomes as follows\n",
    "\n",
    "<img src=\"demo_images/dummy_2.png\" width=650 height=450/>\n",
    "\n",
    "which, one could argue, better represents the data since *\"not-affluent individuals are more likely to walk or use public transportation and less likely to drive their own vehicle\"*. Regardless of how much this statement is true or can be trusted, one thing is clear: we need a better and more general way of encoding categorical features that does not rely on  our intuition or preconceived biases. \n",
    "\n",
    "Fortunately there is a simple way of fixing this issue. Instead of assigning a unique integer to each of the four outcomes, we can replace the transportation feature with four new 'dummy' features:\n",
    "\n",
    "* Is the most commonly used method of transportation, **walking**? 1 for yes, &nbsp; 0 for no\n",
    "\n",
    "* Is the most commonly used method of transportation, **biking**? &nbsp;&nbsp; 1 for yes, &nbsp; 0 for no\n",
    "\n",
    "* Is the most commonly used method of transportation, **driving**? &nbsp; 1 for yes, &nbsp; 0 for no\n",
    "\n",
    "* Is the most commonly used method of transportation, **public transportation**? &nbsp; 1 for yes,&nbsp; 0 for no\n",
    "\n",
    "This way the transportation feature will be encoded using these dummy features as a binary string of length four, with exactly one '1' and three '0's:  \n",
    "\n",
    "\n",
    "<img src=\"demo_images/Trey_Matt.png\" width=475 align=left height=450/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This method of encoding categorical features is sometimes referred to as *one-hot encoding*. Note that using this approach all possible outcomes will be equidistant from one another - at the cost of replacing the original feature with several dummy variables. Now that we know how to handle categorical features, lets use it to prepare a real dataset for classification.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Census income data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [census income dataset](https://archive.ics.uci.edu/ml/datasets/Census+Income) comprises of 6 numerical and 8 categorical features (listed below) on 32561 individuals, along with a binary label: '**<=50K**' indicating that the individual makes less than $50K annually, and '**>50K**' indicating otherwise.\n",
    "\n",
    "* feature 0: **age** (numerical)\n",
    "* feature 1: **type of work** (categorical)\n",
    "* feature 2: **final weight determined by census** (numerical)\n",
    "* feature 3: **education level** (categorical)\n",
    "* feature 4: **number of years of education** (numerical)\n",
    "* feature 5: **marital status** (categorical)\n",
    "* feature 6: **occupation** (categorical) \n",
    "* feature 7: **relationship** (categorical)\n",
    "* feature 8: **race** (categorical)\n",
    "* feature 9: **sex** (categorical)\n",
    "* feature 10: **capital gain** (numerical)\n",
    "* feature 11: **capital loss** (numerical)\n",
    "* feature 12: **work hours per week** (numerical) \n",
    "* feature 13: **native country** (categorical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                  1       2           3   4                    5   \\\n",
       "0  39          State-gov   77516   Bachelors  13        Never-married   \n",
       "1  50   Self-emp-not-inc   83311   Bachelors  13   Married-civ-spouse   \n",
       "2  38            Private  215646     HS-grad   9             Divorced   \n",
       "3  53            Private  234721        11th   7   Married-civ-spouse   \n",
       "4  28            Private  338409   Bachelors  13   Married-civ-spouse   \n",
       "\n",
       "                   6               7       8        9     10  11  12  \\\n",
       "0        Adm-clerical   Not-in-family   White     Male  2174   0  40   \n",
       "1     Exec-managerial         Husband   White     Male     0   0  13   \n",
       "2   Handlers-cleaners   Not-in-family   White     Male     0   0  40   \n",
       "3   Handlers-cleaners         Husband   Black     Male     0   0  40   \n",
       "4      Prof-specialty            Wife   Black   Female     0   0  40   \n",
       "\n",
       "               13      14  \n",
       "0   United-States   <=50K  \n",
       "1   United-States   <=50K  \n",
       "2   United-States   <=50K  \n",
       "3   United-States   <=50K  \n",
       "4            Cuba   <=50K  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv(\"demo_datasets/adult.data\", header=None)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last column in the data contains the labels which we can store as a binary variable in y:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = (data[14] == ' >50K')*1  # binary label \n",
    "data = data.drop([14], 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[pandas.get_dummies](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html) makes it very easy to create dummy variables from categorical features:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Amer-Indian-Eskimo</th>\n",
       "      <th>Asian-Pac-Islander</th>\n",
       "      <th>Black</th>\n",
       "      <th>Other</th>\n",
       "      <th>White</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Amer-Indian-Eskimo   Asian-Pac-Islander   Black   Other   White\n",
       "0                  0.0                  0.0     0.0     0.0     1.0\n",
       "1                  0.0                  0.0     0.0     0.0     1.0\n",
       "2                  0.0                  0.0     0.0     0.0     1.0\n",
       "3                  0.0                  0.0     1.0     0.0     0.0\n",
       "4                  0.0                  0.0     1.0     0.0     0.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies = pd.get_dummies(data[8]) # getting dummies for the race feature \n",
    "dummies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the one-hot encoding method and as can be seen from the dataframe above, the first 3 individuals in the data are white while the next two are black. We now repeat this procedure, this time for all categorical features in the data:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in [1, 3, 5, 6, 7, 8, 9, 13]:      # indicies of categorical features\n",
    "    dummies = pd.get_dummies(data[i]).rename(columns=lambda x: 'dummy_' + str(x)) \n",
    "    data = pd.concat([data, dummies], axis=1) \n",
    "    data = data.drop([i], 1)\n",
    "X = np.asarray(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 2-D array X now contains all the features and is ready to be called by e.g., logistic regression:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "model = LogisticRegression()\n",
    "model.fit(X, y)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}