who-voted_impute.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predict which voter-eligible Colorado citizens actually voted in the 2016 US presidential election.\n",
    "\n",
    "This is the Spring 2017, Harvard Statistics 149: Generalized Linear Models prediction contest/course project.\n",
    "\n",
    "The goal of this project is to use the modeling methods you learned in Statistics 149 (and possibly other related methods) to analyze a data set on whether a Colorado voting-eligible citizen ended up actually voting in the 2016 US election. These data were kindly provided by moveon.org. The competition can be found [here](inclass.kaggle.com/c/who-voted) and ended April 30, 2017, at 10pm EDT.\n",
    "\n",
    "## Imputation of missing values\n",
    "\n",
    "The goal of this notebook is to perform imputation of missing data as we learned in class. We will test k-Nearest Neighbor (kNN) and Multiple Imputation by Chained Equations (MICE) on the distance features in both the train and test sets. This is the third notebook for this competition. For initial exploration of the data see [part 1](who-voted_EDA.ipynb) and [part 2](who-voted_features.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load [local functions](https://github.com/yuwie10/who-voted/blob/master/who_voted_functions.py) for this project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import who_voted_functions as wv\n",
    "import importlib as imp\n",
    "imp.reload(wv);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>7.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>36</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>U</td>\n",
       "      <td>Hispanic</td>\n",
       "      <td>25.4</td>\n",
       "      <td>63.4</td>\n",
       "      <td>54.0</td>\n",
       "      <td>16.7</td>\n",
       "      <td>16.5</td>\n",
       "      <td>39.6</td>\n",
       "      <td>27.3</td>\n",
       "      <td>420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>55</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>U</td>\n",
       "      <td>Uncoded</td>\n",
       "      <td>7.9</td>\n",
       "      <td>97.8</td>\n",
       "      <td>59.8</td>\n",
       "      <td>16.7</td>\n",
       "      <td>15.5</td>\n",
       "      <td>30.9</td>\n",
       "      <td>36.9</td>\n",
       "      <td>307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>2.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>24</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>50.2</td>\n",
       "      <td>7.6</td>\n",
       "      <td>49.5</td>\n",
       "      <td>14.6</td>\n",
       "      <td>24.0</td>\n",
       "      <td>29.6</td>\n",
       "      <td>31.7</td>\n",
       "      <td>292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>7.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>25</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>38.0</td>\n",
       "      <td>8.5</td>\n",
       "      <td>47.4</td>\n",
       "      <td>13.1</td>\n",
       "      <td>22.3</td>\n",
       "      <td>33.3</td>\n",
       "      <td>31.4</td>\n",
       "      <td>316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>5.0</td>\n",
       "      <td>19.0</td>\n",
       "      <td>22</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>30.5</td>\n",
       "      <td>19.1</td>\n",
       "      <td>23.1</td>\n",
       "      <td>16.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>39.1</td>\n",
       "      <td>34.5</td>\n",
       "      <td>392</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender congress_district state_house  age  dist_ballot  dist_poll party  \\\n",
       "0      M               7.0        31.0   36          NaN        NaN     U   \n",
       "1      F               6.0        38.0   55          NaN        NaN     U   \n",
       "2      F               2.0        53.0   24          NaN        NaN     U   \n",
       "3      F               7.0        30.0   25          NaN        NaN     D   \n",
       "4      M               5.0        19.0   22          NaN        NaN     R   \n",
       "\n",
       "        race  hs_only  married  children  cath  evang  non_chrst  other_chrst  \\\n",
       "0   Hispanic     25.4     63.4      54.0  16.7   16.5       39.6         27.3   \n",
       "1    Uncoded      7.9     97.8      59.8  16.7   15.5       30.9         36.9   \n",
       "2  Caucasian     50.2      7.6      49.5  14.6   24.0       29.6         31.7   \n",
       "3  Caucasian     38.0      8.5      47.4  13.1   22.3       33.3         31.4   \n",
       "4  Caucasian     30.5     19.1      23.1  16.0   10.5       39.1         34.5   \n",
       "\n",
       "   days_reg  \n",
       "0       420  \n",
       "1       307  \n",
       "2       292  \n",
       "3       316  \n",
       "4       392  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = wv.load_data('train_renamed.csv')\n",
    "train.drop('voted', axis = 1, inplace = True)\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>52.0</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>L</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>19.5</td>\n",
       "      <td>21.2</td>\n",
       "      <td>25.3</td>\n",
       "      <td>9.8</td>\n",
       "      <td>16.6</td>\n",
       "      <td>45.2</td>\n",
       "      <td>28.4</td>\n",
       "      <td>393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>5</td>\n",
       "      <td>19.0</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>39.7</td>\n",
       "      <td>20.2</td>\n",
       "      <td>29.1</td>\n",
       "      <td>12.0</td>\n",
       "      <td>14.4</td>\n",
       "      <td>41.4</td>\n",
       "      <td>32.2</td>\n",
       "      <td>668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>44.0</td>\n",
       "      <td>56</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>11.3</td>\n",
       "      <td>62.7</td>\n",
       "      <td>41.3</td>\n",
       "      <td>14.8</td>\n",
       "      <td>14.7</td>\n",
       "      <td>36.0</td>\n",
       "      <td>34.6</td>\n",
       "      <td>606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>7</td>\n",
       "      <td>34.0</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>32.8</td>\n",
       "      <td>11.6</td>\n",
       "      <td>33.1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>10.3</td>\n",
       "      <td>44.6</td>\n",
       "      <td>30.6</td>\n",
       "      <td>565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>6</td>\n",
       "      <td>41.0</td>\n",
       "      <td>26</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>D</td>\n",
       "      <td>Uncoded</td>\n",
       "      <td>10.2</td>\n",
       "      <td>14.7</td>\n",
       "      <td>22.4</td>\n",
       "      <td>8.2</td>\n",
       "      <td>18.4</td>\n",
       "      <td>43.5</td>\n",
       "      <td>29.9</td>\n",
       "      <td>336</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender congress_district  state_house  age  dist_ballot  dist_poll party  \\\n",
       "0      M                 2         52.0   30          NaN        NaN     L   \n",
       "1      F                 5         19.0   20          NaN        NaN     U   \n",
       "2      M                 4         44.0   56          NaN        NaN     R   \n",
       "3      F                 7         34.0   20          NaN        NaN     R   \n",
       "4      F                 6         41.0   26          NaN        NaN     D   \n",
       "\n",
       "        race  hs_only  married  children  cath  evang  non_chrst  other_chrst  \\\n",
       "0  Caucasian     19.5     21.2      25.3   9.8   16.6       45.2         28.4   \n",
       "1  Caucasian     39.7     20.2      29.1  12.0   14.4       41.4         32.2   \n",
       "2  Caucasian     11.3     62.7      41.3  14.8   14.7       36.0         34.6   \n",
       "3  Caucasian     32.8     11.6      33.1  14.5   10.3       44.6         30.6   \n",
       "4    Uncoded     10.2     14.7      22.4   8.2   18.4       43.5         29.9   \n",
       "\n",
       "   days_reg  \n",
       "0       393  \n",
       "1       668  \n",
       "2       606  \n",
       "3       565  \n",
       "4       336  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test = wv.load_data('test.csv', str1 = 'cd', str2 = 'sh')\n",
    "id_col = test['Id']\n",
    "test.drop('Id', axis = 1, inplace = True)\n",
    "test.columns = train.columns.tolist()\n",
    "test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(118529, 16)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(39510, 16)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 39510 entries, 0 to 39509\n",
      "Data columns (total 16 columns):\n",
      "gender               39510 non-null object\n",
      "congress_district    39509 non-null object\n",
      "state_house          39509 non-null float64\n",
      "age                  39510 non-null int64\n",
      "dist_ballot          1752 non-null float64\n",
      "dist_poll            1752 non-null float64\n",
      "party                39510 non-null object\n",
      "race                 39510 non-null object\n",
      "hs_only              39510 non-null float64\n",
      "married              39510 non-null float64\n",
      "children             39510 non-null float64\n",
      "cath                 39510 non-null float64\n",
      "evang                39510 non-null float64\n",
      "non_chrst            39510 non-null float64\n",
      "other_chrst          39510 non-null float64\n",
      "days_reg             39510 non-null int64\n",
      "dtypes: float64(10), int64(2), object(4)\n",
      "memory usage: 4.8+ MB\n"
     ]
    }
   ],
   "source": [
    "train.shape\n",
    "test.shape\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(158039, 16)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined = train.append(test)\n",
    "combined.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's investigate the distance predictors we are going to impute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsgAAAH8CAYAAADMqFlQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucXWddL/5PmoTGQuivHMcLclXgQSsWKNggYAuWm4Dc\n9MBPYaCI2J9FQFEULFAQjgoFBUQRTm2Jyu+I5SLggfbITUACgiK01i+niKAinhQLBEpL0ub8sVbK\n08meyUwye88keb9fr746+1mX/d3PXpP57Gc/a60Ne/fuDQAAMDhmrQsAAID1REAGAICOgAwAAB0B\nGQAAOgIyAAB0BGQAAOhsWusCYC211m6T5NNJPjk2HZNkd5KXV9X2cZ0XJLl83+NF9vPcJH9fVX8+\nYdn127fW9iaZq6orVlDj3ZP8dFWd2Vq7W5JfraofX+72B6O1tjHJm5J8b5JXVNXvLrHu25NcWFUX\ntNY+nuS0qvrSIusen+TNVXXfadR9KFprt0zyjiTXJjmzqj40xec6LcnvVtX3r3C7vUnmkjwkyY9X\n1UMOsP5rk7y6qj52gPV+Kcn3V9UTVlLPoWqt3TRDn/8/SZ6b5I5JfjbJu6rqjG69J2QZr/cgnv+2\nSc6tqke11m6e4Tj+odV8jllorV2Q5JKqOrc7Rn4oyelV9dQpP/c5Sb61qp4yzeeBWROQIfl6Vd15\n34PW2q2TvKu19rWqemNVPXcZ+7hvkn+YtGCZ2y/lxCS3GPf10SRTDcej70rygCQ3rqprl7tR34+L\nOCHJDx5KYVN0nyRfqKrT17qQVXS/JH+w1kUs4c5Jvr2qbpckrbV/SvKTVfWBGT3/rZO0JKmqz2cI\nlUeEqnprkreudR1wuBKQYYGq+uw4IvzLSd64YHTm+UkekeQbSb6Y5AlJHpnkbkle0lq7NsnDktws\nyfckeXuSb9+3/fgULxpHhY9JcnZVvX3hCNm+x0n+vyQvSHJ8a+38JK/LOPI4jsa+KkPI2JthJO7Z\nVbWntXZ1kt/MEJBunmFE/HcWvtbW2r2TvCTJceNrOjvJB5O8M8nmJB9rrT2qqj7dbXPzsY6bJ/ls\nkm/rlu0bvdqUZHuSbx0X/UVVPSfJ+Um+ZRxpPjnJ4zOMGN5o7LPfrKrfH1//I5Jcl+T2Y23zVXVJ\na+07krw6w2jjdRlGSF8x9sfLk9xprP1dSX557I/93req+veu7vskeeHYz+9J8vxxX19LcuMMof7x\nSZ6aYYT5P5I8pao+NR4fX09y9yTfkeQNSXYmeej4+ElV9e6FfZ/kJq21C5PcLsmXkjx53N8dMryv\nNxn7+ONJHl1VV0/YR1prt0jy+0luk2RDktdV1Utaay8at/+T1tp8VX2422ZzkldkOD7+z/h6vjwu\ne2+S/xz79/eTvHmR/d8myfuSvDfJSeOyp1TV+yfU+PAkz0uyMclXkvzi+Hx/mOS7xuOhMnwQPK+1\n9tyq+tOVvN5x2UMyvI/HZHjvzqyqv2+tPTvJw5NsyfB+/lKG8Pjfx+e/KMNxeElV3WTsn5cl+ZEM\n7/eHk/xCVe1qrf1zkgvGZbdK8qdV9cwJdZ6Y5HeT/JcMv58vHb9FOi0Ljq2quqbb7p/HPr93hpH1\nl1bV74/LnpwJx+Ai/fSEjP+mjO/ph5Lcc6z5/UkeX1XXjev9aoZj+N1JnlZVmxbs6zZZxns99v+z\nM/w+f1uG9+Y547InJnnGWPsV4/P/S2vtoRn+3blRkquS/NI0v72B5TIHGSb7+wxB63rjV/BPT3L3\nqrpbkouTnFJVr0ry0Qxh7M3j6sdV1YlV9SsT9v1PVXXXJI9N8rrW2txiRVTVv2T46vn9/VfOo1dk\nCHt3yhDQT8rwhz9Jjk1yRVXdM0PQ/s3W2pYFr+e/JLkwwx/EH8gQAP84Q6j90Ywj6304Hr0qyY6q\nOjHDH+s7Tij9Z7rXee8ktx8D7Bn55oj9t4zr/WhV3SXJo5O8uNvHqUl+fpyG8MEMH1iS5PeSfKqq\n7pjkHkme3Fq7XZLfTvKxqjo5yV3G1/GLi71vC/r5PV0/32ds/v4k/29VnZQhWDwzyX3Gx69P8pbW\n2oZx3buMtdwtyS8k+er4Vf3LM4SPSW6Z5GVjX7w+yR91ffe6qrpHhvB82yQPXmQfSfInSd5TVXca\n63xsa+0xVfVrST6f5Kf6cDz6uSR3SPJ9GULyrRYsv7Kqvq+qXrnY/sf1bpXkovE1/GqSPx3D5fVa\na3fM8IHmUeNx9twkf57k35M8Kcmnx+Ps0V29E8PxUq+3tfbtGY7fJ4zP85IMx/2tk5ye5NSx/deS\nvGD8ZmTf8z9gwXOcneHDxUnjf8eM+9vnJlV17wwjzj8/TtXoX/OmDAH8leNzPijJf2ut3WNc5fpj\nqw/HneMyfOA6LckLWmt3aq3dN0sfgwfyPeP+7pThG69TW2vfl+S3MkzFuEuGDy8bF9l+yfd6rOMZ\nGYLv3ZJsS/Ks1tq3ttZOGp/ngWN/vDXJr7XWbp/kv+Wb/wY8OcmbWms3XuZrgqkRkGGyvRlGM3r/\nliE4/21r7dwkH6+qtyyy/VJfEb86SarqkgzTMu6xxLpLeVCG0eS94x/ZV49t++ybD/23GQLzwj86\np2SYG/3hsZ5LMwTR0w7wvKdnGEFLVV2eYdRpoXcmeVRr7X9mGJn71ar6cr9CVX01w1zaB7fWfj1D\ncLlJt8rHqupfu9dws+75XzPu48tV9f1jHQ9J8rPjaOTHMoz63ikre996/1JVnx1/fmCGkcKd4/Ne\nkGEaym3G5W+rqt1V9YUMI4PvHNs/3dW90Ceq6q/Hny9IcrfxQ8SvJNnZWntmhpHSmy/ol+uNQeKe\nGT60ZOzjC3LD42CS05O8vqq+UVVfyxA6e+9f5v6vrKrXj8v2zd/+gQX7um+GOcX/NK737gyj1icf\noMb9HKCee2YYAf74uOxNVfWg8T18fJKfaq39ZpIzs0h/dh6U4ZuJ3VV1XZJXZsLvVlX92/haFr7H\nd0iypareNK73+SRvzHAcJTc8tiZ51fh7/a8ZjqX758DH4IG8raquq6pdSS4fa35Akou737NXLrH9\nku91Ve3N8K3Jya2152UYgd+Q4d+dH8kQrv9lXPd3qurMDB/OvjPDlLaPZzgOr8vwwRDWlIAMk909\n3zxxL0ky/qE8NcO0ii8m+e3W2ssX2f6rS+y7n9O7IcNJgXvHn/e50TJqXPj7e0yGqQX7fD25/g/X\nvudaavtJ+5hkYa17Fq5QVX+TYeTzNRn+gH+ktXaD+Z3jV+UfzzAP9AMZRu16X1/kOfeMj/ft57vH\nk702JvmJcTTyzhk+ADxlhe9br38PJ/XVhnyzrxaOAu5exv4Xzu3eO273/2cYSftshlHxv83+711f\n16T39VDfw32v/UD7X7jdMdn/dR3scTbJUvUsPC42tNZ+oLV21yR/neSmGb49+K0J+5j0PEvVu9ix\nudj2C/ex1L8PyQ37dV+fHugYPJBJNe/JDWtf6nyDJd/r8cPL3yW5a4Zj9pczHM/7nqd/b75l/GZh\nY4YPT3fufm+3Jblkma8JpkZAhgXGOaDPSfLSBe0nZfiH+7Kq+o0M4eWkcfGeLP8P1RPG/d01w/za\nD2eYs/r9rbUt49ezD+3WX2zfFyU5awwCx2YIVf9rmTUkyY6hjPaDYz0nJvnhDPMMl/LO8bnSWrtV\nhpPbbmAcqXvOOFL7tCSXZhhV25Nk4/h17N0yvO4XVtVFGUaA911BYyl/mWGqxr6rYrwrQz9elOQX\nuv54a5KnHOB9W66Lkjx633SY1toZGcL25SvcT++k1tq+kxp/NskHquqqDKN6LxinGezNEPQn9sk4\nGrgjyVljXccnmc83j4PFjp13Jpkfj7ctGaa3HMz+51prDxyXPTRDIPrkgt28O8n9W2vfPa533wzT\nSxZO+zigA9Tz4STfOx7HyXAuwB9nOKY/WlUvyzCP9uH5Zn8u9bt1Zmttc2vtmPH5VvK7VUm+0Vp7\n5FjnzZM8agX7mB+3u1WG0eN3ZDrH4EVJTm+tfdf4+ElLrHug9/r2GT6EnF1Vb8vwofTYDH39nvF5\nvnNc92czTKfad2zccdzvjyb5RIa54rCmBGQYTxob//vbDF/ZPquq/qJfqar+PsMJWB9trX00yRMz\nzDdNkrclObe19vhlPN93t9b+LsMJQo+pqv/MMLL1viT/mOHr7f4Pz4eS3LG19uYF+3lqhhNhPjn+\nV0letMzXnBouNfcTSV7ZWvtkhjmNZ9QiJ/10zkryfa21y5Kcl2EUeKHfSXLn1tolGeZnfybDyOi/\nZxhduizJ3yT51yQ19setMgTmA329+pQMQegTGaaE/EYNlzF7aoavcz+Z4Y/sJ5O8+ADv27JU1f/K\nEKzf3Vq7NMNX9g8ZR6cP1mVJntda+/skPzbuMxlOcnrzWOurMxwXS/XJTyX5kfE9/EiGr/IvGJe9\nJcNc0fsv2OYPMrwvl4z7/8xB7v/qJI8bX8OvJXl4LbjqSVX9Q4Y5z28aj4ffTPLQhVNuVmBiPVX1\nH+Oy141f1/9iksdkOO6+tbX2Dxmm3nw1yc1aa1szfHC7trX2kdxwJPWFSb6Q4di+LEOIftpyC6yq\n3RmC+NPG4/QvM3zoec8yd3Hb1trHMnyQeWoNVv0YHH/XfyHJRePx9r3Zf2rZPgd6rz+R4aTkfxz/\nHf2xDFPIbldVn8wwovzOcfsHZjiB8tIMH7b/x9j+60l+bJz2A2tqw969ew+8FgB02nBlg0uq6kDz\neVmBNlzF4sdruKTjtJ/rthlGq3+9hitaPDLJr1TVKQvWu0281xxlXOYNAI5O/5rhJNBPttb2ZLj0\n3hPXtiRYH4wgAwBAxxxkAADoCMgAANARkAEAoHNYnqS3c+euQ544fcIJx+XKKxe7mg2rRT/Pjr6e\nHX09O/p6dvT17Ojr2Zmb27rc27HfwFE7grxp04HuRcBq0M+zo69nR1/Pjr6eHX09O/p6/TtqAzIA\nAEwiIAMAQEdABgCAjoAMAAAdARkAADoCMgAAdARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwA\nAB0BGQAAOpumufPW2rOS/FiSGyX5vSTvS3JBkr1JLklyVlVd11p7XpIHJ9mT5OlV9ZFp1gUAAIuZ\n2ghya+20JD+U5J5JTk1yyyQvS3J2Vd07yYYkD2ut3XVcfkqSxyR51bRqAgCAA5nmFIsHJPlkkjcn\neVuStyc5OcMocpK8I8npSe6V5OKq2ltVn0uyqbU2N8W6AABgUdOcYvGtSW6d5CFJbpvkrUmOqaq9\n4/JdSY5PctMkX+y229e+c4q1AQDARNMMyF9M8o9V9Y0k1Vq7OsM0i322JvlSkq+MPy9sX9QJJxyX\nTZs2HnKBc3NbD7wSh0w/z46+nh19PTv6enb09ezo6/VtmgH5A0me1lp7WZLvTHLjJO9qrZ1WVe9N\n8qAk70lyeZIXt9bOTXKLDKPMVyy14yuvvOqQi5ub25qdO3cd8n5Ymn6eHX09O/p6dvT17Ojr2dHX\ns3OwH0SmFpCr6u2ttR9O8pEMc53PSvKZJK9trd0oyWVJLqyqa1tr70/yoW49AABYE1O9zFtVPXNC\n86kT1jsnyTnTrGVatm/fPLF9fn73jCsBAGA1uFEIAAB0BGQAAOgIyAAA0BGQAQCgIyADAEBHQAYA\ngI6ADAAAHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKCzaa0LOJxs3755rUsAAGDKjCADAEBHQAYA\ngI6ADAAAHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAdARkAADoCMgAA\ndARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwAAB0BGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCg\nIyADAEBHQAYAgI6ADAAAHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAd\nARkAADoCMgAAdARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwAAJ1Na13AkWr79s0T2+fnd8+4\nEgAAVsIIMgAAdARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwAAJ2pXge5tfa3Sb4yPvxMkj9I\n8vIke5JcXFXPb60dk+T3kpyU5JokT6qqy6dZFwAALGZqAbm1tiXJhqo6rWv7eJJHJfmnJH/RWrtL\nktsm2VJV92itbUvy0iQPm1ZdAACwlGmOIJ+U5LjW2sXj85yT5Niq+nSStNYuSnJ6ku9M8s4kqaod\nrbW7TbEmAABY0jTnIF+V5NwkD0hyZpLzx7Z9diU5PslNk3y5a7+2teYW2AAArIlpBtFPJbm8qvYm\n+VRr7ctJbtYt35rkS0mOG3/e55iq2rPUjk844bhs2rTxkAucm9t64JU6W1e2+iLPueXQd3KYWWk/\nc/D09ezo69nR17Ojr2dHX69v0wzIT0xypyQ/11q7eYYg/LXW2vdkmIP8gCTPT3KLJA9N8oZxDvIn\nD7TjK6+86kCrHNDc3Nbs3LlrRdvs2rX5kJ93587dh7yPw8nB9DMHR1/Pjr6eHX09O/p6dvT17Bzs\nB5FpBuTzklzQWvtAkr0ZAvN1Sf4kycYMV7H4cGvtb5Lcr7X210k2JDljijUBAMCSphaQq+obSX5y\nwqJtC9a7LsMcZQAAWHNuFAIAAB0BGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCgIyADAEBHQAYAgI6A\nDAAAHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAdARkAADoCMgAAdARk\nAADoCMgAANARkAEAoLNprQs43O3Yfd7E9m2bf3rGlQAAsBqMIAMAQEdABgCAjoAMAAAdARkAADpO\n0pux7ds3T2yfn98940oAAJjECDIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAdARkA\nADoCMgAAdARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwAAB0BGQAAOpvWuoDDyY7d5611CQAA\nTJkRZAAA6AjIAADQMcVindi+ffPE9vn53TOuBADg6GYEGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCg\nIyADAEBHQAYAgI6ADAAAHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAACd\nTdPceWvt25J8LMn9kuxJckGSvUkuSXJWVV3XWntekgePy59eVR+ZZk0AALCUqQXk1trmJH+Q5Otj\n08uSnF1V722tvTrJw1prn01yapJTktwyyRuT3H1aNc3Sjt3nTWzftvmnZ1wJAAArMc0pFucmeXWS\nz4+PT07yvvHndyQ5Pcm9klxcVXur6nNJNrXW5qZYEwAALGkqI8ittSck2VlVF7XWnjU2b6iqvePP\nu5Icn+SmSb7YbbqvfedS+z/hhOOyadPGQ65zbm7ritY/dsvmQ37OrcdtWdH6c3MrW389Wmk/c/D0\n9ezo69nR17Ojr2dHX69v05pi8cQke1trpye5c5LtSb6tW741yZeSfGX8eWH7kq688qpDLnBubmt2\n7ty1om2uuXr3IT/vrmuvXtH6O3ce+nOupYPpZw6Ovp4dfT07+np29PXs6OvZOdgPIlOZYlFVP1xV\np1bVaUk+nmQ+yTtaa6eNqzwoyfuTfDDJA1prx7TWbpXkmKq6Yho1AQDAckz1KhYLPCPJa1trN0py\nWZILq+ra1tr7k3woQ1g/a4b1AADAfqYekMdR5H1OnbD8nCTnTLsOAABYDjcKAQCAjoAMAAAdARkA\nADoCMgAAdARkAADoCMgAANARkAEAoCMgAwBAR0AGAICOgAwAAJ2p32qaG9qx+7yJ7ds2//SMKwEA\nYBIjyAAA0DGCvE4sNrI8n/kZVwIAcHQzggwAAB0BGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCg4zJv\n69z27Zsnts/P755xJQAARwcjyAAA0BGQAQCgIyADAEBHQAYAgI6ADAAAHQEZAAA6AjIAAHQEZAAA\n6LhRyDq3Y/d5E9vnMz/jSgAAjg5GkAEAoCMgAwBAR0AGAICOgAwAAB0BGQAAOgIyAAB0BGQAAOgI\nyAAA0BGQAQCgIyADAEDHraYPU9svPX9i+/yJZ8y4EgCAI4sRZAAA6AjIAADQEZABAKAjIAMAQEdA\nBgCAjoAMAAAdARkAADrLCsittVdOaHvd6pcDAABra8kbhbTW/nuS705yt9baid2izUmOn2ZhAACw\nFg50J70XJrlNkpcneX7XvifJZVOqCQAA1sySAbmq/jnJPyc5qbV20wyjxhvGxTdJ8p/TLA4AAGbt\nQCPISZLW2rOSPCvJF7vmvRmmX7AGduzYOLF9/sSJzQAALNOyAnKSJyX5nqraOc1iAABgrS33Mm+f\ni+kUAAAcBZY7gvy/k3ygtfaeJFfva6yqF0ylKgAAWCPLDcj/Nv6XfPMkPQAAOOIsKyBX1fMPvBYA\nABz+lnsVi+syXLWi9/mquuXqlwQAAGtnuSPI15/M11rbnOThSe4xraIAAGCtLPcqFterqt1V9WdJ\n7juFegAAYE0td4rFfPdwQ5ITk3xjKhUBAMAaWu5VLO7T/bw3yRVJHr365QAAwNpa7hzkM8a5x23c\n5pKq2jPVygAAYA0saw5ya+3kDDcLeV2S85N8rrV2yjQLAwCAtbDcKRavSPLoqvpwkrTWtiV5ZZIf\nXGyD1trGJK/NMOq8N8mZGe7Cd8H4+JIkZ1XVda215yV5cJI9SZ5eVR85qFdDtm/fPLF9fn73jCsB\nADg8LfcqFjfZF46TpKp2JNlygG0eOq57zyRnJ3lRkpclObuq7p3hZL+HtdbumuTUJKckeUySV63o\nFQAAwCpabkD+z9baw/Y9aK09PMkXl9qgqt6S5Mnjw1sn+VKSk5O8b2x7R5LTk9wrycVVtbeqPpdk\nU2ttbvkvAQAAVs9yp1g8OcnbW2vnZRj53Zvkhw60UVXtaa29Lskjkvx4kvtV1b478u1KcnySm+aG\nYXtf+87F9nvCCcdl06aNyyx9cXNzW1e0/rFbJk9fWE+2Hjd5YH9u7kAD/tOz0n7m4Onr2dHXs6Ov\nZ0dfz46+Xt+WG5AflOSqJHdN8j1J/jTJaUk+daANq+rxrbVfSfLhJN/SLdqaYVT5K+PPC9sXdeWV\nVy2z7MXNzW3Nzp27VrTNNVev/3m8u669emL7zp1rU/vB9DMHR1/Pjr6eHX09O/p6dvT17BzsB5Hl\nTrF4cpJ7VtXXquoTGaZK/PxSG7TWHtdae9b48Kok1yX5aGvttLHtQUnen+SDSR7QWjumtXarJMdU\n1RUrfB0AALAqljuCvDk3vHPeNzJMs1jKm5Kc31r7q3H7pye5LMlrW2s3Gn++sKquba29P8mHMgT2\ns1ZQPwAArKrlBuS3JHl3a+0N4+NHJvnzpTaoqq8l+a8TFp06Yd1zkpyzzFoAAGBqljXFoqp+JcO1\nkFuS707yiqp6zjQLAwCAtbDcEeRU1YVJLpxiLQAAsOaWe5IeAAAcFQRkAADoCMgAANARkAEAoCMg\nAwBAR0AGAIDOsi/zxuFhx+7zJrbPZ37GlQAAHJ6MIAMAQEdABgCAjoAMAAAdARkAADoCMgAAdFzF\n4iixffvmie3z87tnXAkAwPpmBBkAADoCMgAAdEyxOEq4gQgAwPIYQQYAgI6ADAAAHQEZAAA6AjIA\nAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAdARkAADoCMgAAdARkAADoCMgAANARkAEA\noCMgAwBAR0AGAICOgAwAAB0BGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCgIyADAEBHQAYAgI6ADAAA\nHQEZAAA6AjIAAHQEZAAA6AjIAADQEZABAKCzaa0LYG1tv/T8ie3zJ54x40oAANYHI8gAANAxgsyK\nGHEGAI50RpABAKAjIAMAQEdABgCAjoAMAAAdJ+kd5Xbs2Dixff7EGRcCALBOGEEGAICOgAwAAB0B\nGQAAOgIyAAB0BGQAAOgIyAAA0BGQAQCgM5XrILfWNif5wyS3SXJskhcm+YckFyTZm+SSJGdV1XWt\ntecleXCSPUmeXlUfmUZNTNf27Zv3a5uf370GlQAAHJppjSA/NskXq+reSR6Y5HeTvCzJ2WPbhiQP\na63dNcmpSU5J8pgkr5pSPQAAsCzTupPenyW5cPx5Q4bR4ZOTvG9se0eS+yepJBdX1d4kn2utbWqt\nzVXVzinVxTJtv/T8tS4BAGBNTCUgV9VXk6S1tjVDUD47ybljEE6SXUmOT3LTJF/sNt3XvmRAPuGE\n47Jp0+RbJK/E3NzWFa1/7Jb9pxEcqbZuXeFr3bplv6a5uS3j/1fWzxw8fT07+np29PXs6OvZ0dfr\n27RGkNNau2WSNyf5vap6fWvtxd3irUm+lOQr488L25d05ZVXHXJ9c3Nbs3PnrhVtc83VR8+c2l27\nrl3hBlfv17Rz5+6D6mcOjr6eHX09O/p6dvT17Ojr2TnYDyLTOknv25NcnOQpVfWusfnvWmunVdV7\nkzwoyXuSXJ7kxa21c5PcIskxVXXFNGpiunbsPm//xkuvzTNOe+rsiwEAOATTGkF+dpITkjyntfac\nse1pSV7RWrtRksuSXFhV17bW3p/kQxlOGDxrSvUAAMCyTGsO8tMyBOKFTp2w7jlJzplGHQAAsFJT\nm4PM4W3HjkM/CRIA4HDkTnoAANAxgsy6stj1l+dPPGPGlQAARysjyAAA0BGQAQCgIyADAEDHHGSm\n6jUfe012TbjLnjnFAMB6ZQQZAAA6AjIAAHQEZAAA6AjIAADQEZABAKAjIAMAQEdABgCAjoAMAAAd\nARkAADrupMe6sX375uzYvXG/9m3brl2DagCAo5WAzJrYfun5+7VNCscAALNmigUAAHQEZAAA6AjI\nAADQEZABAKDjJD0OC5NO6ps/8Yw1qAQAONIZQQYAgI4RZKZmx46NOXZLcs3VN7x8m+saAwDrmYDM\nzO3Y4XrHAMD6ZYoFAAB0jCBz2Jp04l7i5D0A4NAIyBxxBGcA4FCYYgEAAB0jyKx7i53Ut9jVMBZb\nf/7EVSvj+ZSxAAANuElEQVQJADiCGUEGAICOgAwAAB0BGQAAOgIyAAB0nKTHYcsd+QCAaTCCDAAA\nHQEZAAA6AjIAAHQEZAAA6AjIAADQcRULjnrbLz1/Yvv8iWfMuBIAYD0wggwAAB0jyLAII8sAcHQy\nggwAAB0BGQAAOqZYcNRYbMoEAEDPCDIAAHQEZAAA6AjIAADQEZABAKDjJD2OGjt2bJzYvm3btSta\nP3F9ZAA4khlBBgCAjhFkjnqLjxQDAEcjI8gAANARkAEAoCMgAwBAR0AGAICOgAwAAB0BGQAAOgIy\nAAB0BGQAAOgIyAAA0JnqnfRaa6ck+a2qOq21drskFyTZm+SSJGdV1XWttecleXCSPUmeXlUfmWZN\nAACwlKkF5NbaM5M8LsnXxqaXJTm7qt7bWnt1koe11j6b5NQkpyS5ZZI3Jrn7tGqC1bDYrannT5xx\nIQDAVExzisWnkzyye3xykveNP78jyelJ7pXk4qraW1WfS7KptTY3xZoAAGBJUxtBrqo3ttZu0zVt\nqKq948+7khyf5KZJvtits69957TqgmnZfun5E9vnTzxjxpUAAIdiqnOQF7iu+3lrki8l+cr488L2\nJZ1wwnHZtGny19wrMTe39cArdY7dsvmQn/NodLT029atk1/nSo+zQzHL5zra6evZ0dezo69nR1+v\nb7MMyH/XWjutqt6b5EFJ3pPk8iQvbq2dm+QWSY6pqisOtKMrr7zqkIuZm9uanTt3rWiba67efcjP\ne7Q5dsvmo6bfdu26dmL7So+zg3UwxzQHR1/Pjr6eHX09O/p6dg72g8gsA/Izkry2tXajJJclubCq\nrm2tvT/JhzLMhz5rhvUAAMB+phqQq+qfk2wbf/5UhitWLFznnCTnTLMOAABYLjcKAQCAjoAMAACd\nWc5BhqOSy78BwOHFCDIAAHQEZAAA6AjIAADQEZABAKAjIAMAQMdVLGCV7NixcWL7tm2Tb0ENAKxP\nRpABAKAjIAMAQEdABgCAjoAMAAAdJ+nBOuPW1ACwtowgAwBAR0AGAICOKRYwZYtdH3n+xBkXAgAs\nixFkAADoGEGGNbLYyXgAwNoyggwAAB0BGQAAOgIyAAB0BGQAAOg4SQ/WyGKXf5tk27Zrp1gJANAT\nkOEw4RbUADAbplgAAEDHCDKQZPIItdFpAI5GRpABAKBjBBkOc/tGfrdu3ZJdu66+vt3oLwAcHCPI\nAADQEZABAKAjIAMAQMccZDhCuW4yABwcARnI9u2bs2P3/nf2mz9xDYoBgDVmigUAAHSMIMNRZtLU\ni0mjx6u178S0DgAOLwIysGYEagDWIwEZjiI7dqzOSPFKbd++efKCk2dbBwAsh4AMrNhiI78AcCQQ\nkIHDhikZAMyCq1gAAEDHCDIcBhabO7xt27UrWh8AODABGViUucYAHI0EZDiM9SPFx25Jrrl6bUeO\nFx+5Pm9i67bplQIAB80cZAAA6AjIAADQMcUCWDOLTcmYP3HGhXA9l9IDMIIMAAA3YAQZWHfW6uoZ\nRk8BSIwgAwDADRhBBlbMjUgAOJIJyMBhY1IwX+xugqtp4dSLrVu3ZNeuq029ADhCCcjAumOEGoC1\nJCADRywn3ekDgIMhIAMchB07NiYf27xf+/z87hXt53AJsCut83B5XQCTCMjAUWW1gi2DlV6ST3AG\nDgcCMnBYWyzwJsmO3ZPnMu/Yfd7+beclyfLmPh+7ZbnVHbzt2ye/JkF+5YRyYKUEZGBRh8vJcpMC\n75o976XXCl4zIPQC0yQgAxyF1upuhQCHAwEZYB1abFR8PvMzrgTg6CMgA8zIaozarqepBYtNwZnF\nzVvWwnrqe2C6BGSAVbTSq2Ss1jzvnztv+8T2bduWv4+1DLyLhc9nnPbU/dfdvnnREzAnnrB58sqe\nU+AFBGSAVbbYyXursu9VCNTr7eTLaQfzae9f0IYjz4a9e/eudQ0rtnPnrkMuem5ua3bu3LWibRYb\noWFxx27ZnGuudlmqWdDXs6OvD85igXSpALt165bs2nX1stZfLZPqXM2QvdLgPKsAfjB/Fzk4+np2\n5ua2bjiY7dbFCHJr7Zgkv5fkpCTXJHlSVV2+tlUBsJpWGmx37NiYY7ck11y9vka8p2VSEB76bP/X\nf6TO84b1Yl0E5CQPT7Klqu7RWtuW5KVJHrbGNQHATCw5r3qxbdbZ1I5J9ay3aSbrrc9Yv9ZLQL5X\nkncmSVXtaK3dbY3rAeAotZKR7oOa7rHInR9Xy0qvlrKScLjYHR6T/e9cuZqj3CsJtsOHjcVuHrTI\n3TV37D+Fctu2I/OmP9P+kHCkfAhZLwH5pkm+3D2+trW2qar2rFVBADAN07zz42rOlZ445WP3xmzb\n/NPLrmVS8FzMata4WlYj7E0K7Mdu2Zy73PnqmYfGxb6pWLUTVqe8/1laFyfptdZelmRHVb1hfPyv\nVXWLNS4LAICj0DFrXcDog0l+NEnGOcifXNtyAAA4Wq2XKRZvTnK/1tpfJ9mQ5PCaqAIAwBFjXUyx\nAACA9WK9TLEAAIB1QUAGAICOgAwAAJ31cpLeVBzoFtattZdnuEnJvhuiP6yqvrzfjli21topSX6r\nqk5b0P7QJM9NsifJH1bVa9egvCPKEn39C0melGTn2PSzVVUzLu+I0FrbnOQPk9wmybFJXlhVb+2W\nO65XyTL62nG9SlprG5O8NklLsjfJmVV1Sbfccb1KltHXjutV1Fr7tiQfS3K/qvrHrn3Fx/QRHZBz\n4FtYn5zkAVV1xZpUd4RprT0zyeOSfG1B++Ykv53k7uOyD7bW3lpV/zH7Ko8Mi/X16OQk81X1sdlW\ndUR6bJIvVtXjWms3S/LxJG9NHNdTsGhfjxzXq+ehSVJV92ytnZbkRRn/NjquV92ifT1yXK+S8dj9\ngyRfn9C+4mP6SJ9icYNbWCe5/hbW4+jy7ZO8prX2wdbaE9emxCPKp5M8ckL79ya5vKqurKpvJPlA\nkh+eaWVHnsX6Ohn+wX1Wa+0DrbVnzbCmI9GfJXnO+POGDKMP+ziuV9dSfZ04rldNVb0lyZPHh7dO\n8qVuseN6FR2grxPH9Wo6N8mrk3x+QftBHdNHekCeeAvr8ecbJ3llhlGLByb5udbaD8y4viNKVb0x\nye4Jixa+D7uSHD+Too5QS/R1kvyPJGcmuW+Se7XWHjKzwo4wVfXVqtrVWtua5MIkZ3eLHder6AB9\nnTiuV1VV7WmtvS7D38E/6RY5rlfZEn2dOK5XRWvtCUl2VtVFExYf1DF9pAfkryTZ2j0+pqr2jUpc\nleTlVXVVVe1K8u4Mc5VZfQvfh63Z/1M0q6C1tiHJ71TVFeMn5b9Icpc1Luuw1lq7ZZL3JPmjqnp9\nt8hxvcoW62vH9XRU1eOT3CHJa1trNx6bHddTMKmvHder6okZbjj33iR3TrK9tfYd47KDOqaP9DnI\nH8ww/+cNE25hfYckf9pau0uGDwr3SvK62Zd4VLgsye3HeYVfzfDVxrlrW9IR66ZJLmmtfW+GuVb3\nzXDiEwehtfbtSS5O8pSqeteCxY7rVXSAvnZcr6LW2uOS3KKqfiPDYNF143+J43pVHaCvHderpKqu\nnzIxhuQzq+oLY9NBHdNHekDe7xbWrbVfzDAX5a2ttT9KsiPDV9Xbq+rSNaz1iNNa+8kkN6mq14z9\nflGGDyN/WFX/trbVHVkW9PWzM4zCXZPkXVX1P9e2usPas5OckOQ5rbV982Nfm+TGjutVd6C+dlyv\nnjclOb+19ldJNid5epJHtNb8e736DtTXjuspOdQM4lbTAADQOdLnIAMAwIoIyAAA0BGQAQCgIyAD\nAEBHQAYAgM6Rfpk3gMNCa+2CJJ9Kcq+q+tFF1jk+yeuq6uGzrA3gaGMEGWD9+Pxi4Xh0Qoa7RAEw\nRa6DDLAGxtvMvjTJQ5J8PsnGJOclOaeqbjNe5P6ZSa5N8pkkj03yhiQPTPIXVfWI1tqLkvxIkpsl\nuSLJI6vqC621f09yYYY7hO5J8l+r6jOttdPH5zwmyWeT/GSGO3i9JMlpYw0XVNVvz6ALANYtI8gA\na+NRSe6S5MQkP5HkdguWvzDJ/avq5CT/mOSOSZ6aYZT5Ea21241tP1RVd0hyeZKfGrf9jgx35bpL\nkr9K8pTW2rFJ/iTJ46vqTkk+keTxSX4mSarqrkl+MMnDWmv3ntJrBjgsCMgAa+O0JG+qqt1VtTPJ\nwlvMvi3JB1trL0ny9qr6eL+wqi5P8owkT2qtvTTJPZLcpFvlneP/L8kwwnynJP+2bz9V9eyqemWS\n05P8WGvt40k+nOQW47oARy0BGWBt7M0N/w3e0y+sqqdlGGX+zyR/3Fp7bL+8tXZykovHfVyY5M1J\nNnTbX909z4Ykuxdsf3xr7RYZplU8s6ruXFV3TrItyfmH/OoADmMCMsDa+MskP9FaO7a1dkKGucVJ\nktbaptba/05yRVX9RpLtGaZj7Mk3rz50apL3VtWrk/xDkvtnCLuLqSRzrbXvGx8/M8mZSd6d5Gda\na5tbazdJ8oEkp6zWiwQ4HLnMG8AaqKo/b63dPcMUiC9kCLn7lu1prT03yV+21q5K8qUM84X/T5LP\ntdbek+GkvTe11j6RYXT4E0luu8TzXT2OQm9vrd0oyaeTPC7JNUlun+TvMvxNOL+q3rvarxfgcOIq\nFgAA0DHFAgAAOgIyAAB0BGQAAOgIyAAA0BGQAQCgIyADAEBHQAYAgI6ADAAAnf8LpB22MyK01LcA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10329a208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "no_nulls = combined.dropna()\n",
    "distances = ['dist_ballot', 'dist_poll']\n",
    "colors = ['blue', 'green']\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "fig.set_size_inches(11.7, 8.27);\n",
    "\n",
    "for dist, color in zip(distances, colors):\n",
    "    p = sns.distplot(np.log(no_nulls[dist]),\n",
    "                     color = color,\n",
    "                     kde = False, bins = 100);\n",
    "    _ = p.set_xlabel('distance');\n",
    "    _ = p.set_ylabel('count');\n",
    "    _ = p.set_title('Distribution of distances from ballot drop off location or polling place');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because the two variables are similar, we can test imputation on just one of the two distance variables. Let's pick dist_poll and use this column as our response variable for kNN regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 9.2502 ,  4.28483,  1.99877, ...,  2.0558 ,  1.82069,  2.40694])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = no_nulls['dist_poll'].as_matrix()\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our design matrix we want to 'train' on all columns except the distance predictors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.58256449, -1.11572631, -0.21804297, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.52783299,  0.22213267,  0.57435185, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.45192949, -0.58921407,  1.17312983, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       ..., \n",
       "       [ 1.05037298, -0.2008034 , -0.90645837, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.0600245 , -0.59784541,  0.0544548 , ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-1.69296197,  2.86332523, -1.81000359, ...,  0.        ,\n",
       "         0.        ,  0.        ]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quantitative = ['age', 'hs_only', 'married', 'children', 'cath',\n",
    "                'evang', 'non_chrst', 'other_chrst', 'days_reg']\n",
    "categorical = ['gender', 'congress_district', 'state_house', 'party', 'race']\n",
    "\n",
    "X = wv.design_Xmatrix(no_nulls, quantitative, categorical, standardize = True)\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will pick a cross-validation split of 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cv = KFold(n_splits=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[SigOpt](sigopt.com) provides an API that uses Bayesian optimization to tune hyperparameters. We can set up a SigOpt experiment to evaluate the best number of nearest neighbors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sigopt import Connection\n",
    "conn = Connection(client_token = 'ZGILQJTPPAWFCOOOZPALZQHRKNBANVQQUEVEJJSYYOUVYNYI')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created experiment: https://sigopt.com/experiment/17663\n"
     ]
    }
   ],
   "source": [
    "#create SigOpt experiment\n",
    "experiment = conn.experiments().create(\n",
    "  name='who voted: kNN imputation',\n",
    "  parameters=[\n",
    "    dict(name='n_neighbors', type='int', bounds=dict(min=3, max=100))\n",
    "  ]\n",
    ")\n",
    "print('Created experiment: https://sigopt.com/experiment/' + experiment.id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_model(assignments, X, y, cv, weights):\n",
    "    '''\n",
    "    Evaluates different kNN regression model with different number of nearest neighbors.\n",
    "    \n",
    "    Inputs:\n",
    "    assignments = suggested parameters from SigOpt\n",
    "    X and y matrices\n",
    "    cv = cross-validation scheme\n",
    "    \n",
    "    Returns: mean and std of cv accuracies\n",
    "    '''\n",
    "    model = KNeighborsRegressor(n_neighbors=assignments['n_neighbors'],\n",
    "                                metric = 'hamming',\n",
    "                                weights = weights)\n",
    "    cv_accuracies = cross_val_score(model, X, y, cv=cv)\n",
    "    return (np.mean(cv_accuracies), np.std(cv_accuracies))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for _ in range(20):\n",
    "    # Receive a Suggestion from SigOpt\n",
    "    suggestion = conn.experiments(experiment.id).suggestions().create()\n",
    "\n",
    "    # Evaluate the model locally\n",
    "    (value, std) = evaluate_model(suggestion.assignments, X, y, cv, weights = 'distance')\n",
    "\n",
    "    # Report an Observation (with standard deviation) back to SigOpt\n",
    "    _ = conn.experiments(experiment.id).observations().create(\n",
    "        suggestion=suggestion.id,\n",
    "        value=value,\n",
    "        value_stddev=std,\n",
    "    );"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Assignments({\n",
       "  \"n_neighbors\": 26\n",
       "})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-fetch the best observed value and assignments\n",
    "best_assignments1 = conn.experiments(experiment.id).best_assignments().fetch().data[0].assignments\n",
    "best_assignments1\n",
    "\n",
    "## To wrap up the Experiment, fit the model on the best assigments\n",
    "# and train on all available data\n",
    "model_knn1 = KNeighborsRegressor(n_neighbors = best_assignments1['n_neighbors'],\n",
    "                                 metric = 'hamming',\n",
    "                                 weights = 'distance')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The best k is 26. Let's test how accurate imputation with 26 nearest neighbors is. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8.8796838187964262"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get train/test splits\n",
    "for train_index, test_index in cv.split(X, y):\n",
    "    X_train_cv, X_test_cv = X[train_index], X[test_index]\n",
    "    y_train_cv, y_test_cv = y[train_index], y[test_index]\n",
    "    \n",
    "#train model for evaluation\n",
    "kNN1_fit = model_knn1.fit(X_train_cv, y_train_cv)\n",
    "\n",
    "#get predictions\n",
    "y_predict_kNN1 = kNN1_fit.predict(X_test_cv)\n",
    "\n",
    "#mean squared error of kNN\n",
    "kNN1_error = mean_squared_error(y_test_cv, y_predict_kNN1)\n",
    "kNN1_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will compare this MSE with that from MICE.\n",
    "\n",
    "**Note:** kNN imputation was not optimal here because the 'train' set included both quantitative and dummy encoded variables. A better method would be to transform the categorical variables to continuous ones and then to optimize not only number of nearest neighbors, but also the distance metric used to calculate similarity and the weights of the neighbors. The kNN model with the lowest MSE could then be compared to the MSE from MICE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MICE\n",
    "\n",
    "For MICE, we will use the 'mice' package in R. This library takes a dataframe with all variables, including the missing variables, and fills any NaNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>dist_poll_original</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>3.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>37</td>\n",
       "      <td>9.25020</td>\n",
       "      <td>9.25020</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>6.7</td>\n",
       "      <td>59.3</td>\n",
       "      <td>67.0</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.7</td>\n",
       "      <td>49.9</td>\n",
       "      <td>33.3</td>\n",
       "      <td>411</td>\n",
       "      <td>9.25020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>7.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>54</td>\n",
       "      <td>3.32300</td>\n",
       "      <td>4.28483</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>22.2</td>\n",
       "      <td>81.4</td>\n",
       "      <td>56.0</td>\n",
       "      <td>12.9</td>\n",
       "      <td>20.7</td>\n",
       "      <td>36.3</td>\n",
       "      <td>30.1</td>\n",
       "      <td>579</td>\n",
       "      <td>4.28483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>6.0</td>\n",
       "      <td>36.0</td>\n",
       "      <td>39</td>\n",
       "      <td>1.99877</td>\n",
       "      <td>1.99877</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>12.8</td>\n",
       "      <td>98.1</td>\n",
       "      <td>84.2</td>\n",
       "      <td>15.7</td>\n",
       "      <td>16.5</td>\n",
       "      <td>33.2</td>\n",
       "      <td>34.5</td>\n",
       "      <td>544</td>\n",
       "      <td>1.99877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M</td>\n",
       "      <td>7.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>56</td>\n",
       "      <td>2.05759</td>\n",
       "      <td>2.41070</td>\n",
       "      <td>D</td>\n",
       "      <td>Middle Eastern</td>\n",
       "      <td>9.8</td>\n",
       "      <td>97.7</td>\n",
       "      <td>59.6</td>\n",
       "      <td>11.0</td>\n",
       "      <td>11.1</td>\n",
       "      <td>49.3</td>\n",
       "      <td>28.6</td>\n",
       "      <td>392</td>\n",
       "      <td>2.41070</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M</td>\n",
       "      <td>2.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>31</td>\n",
       "      <td>3.67070</td>\n",
       "      <td>3.67070</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>7.2</td>\n",
       "      <td>72.7</td>\n",
       "      <td>13.1</td>\n",
       "      <td>7.4</td>\n",
       "      <td>7.5</td>\n",
       "      <td>59.5</td>\n",
       "      <td>25.6</td>\n",
       "      <td>676</td>\n",
       "      <td>3.67070</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender congress_district state_house  age  dist_ballot  dist_poll party  \\\n",
       "0      F               3.0        61.0   37      9.25020    9.25020     U   \n",
       "1      F               7.0        31.0   54      3.32300    4.28483     R   \n",
       "2      M               6.0        36.0   39      1.99877    1.99877     U   \n",
       "3      M               7.0        27.0   56      2.05759    2.41070     D   \n",
       "4      M               2.0        33.0   31      3.67070    3.67070     U   \n",
       "\n",
       "             race  hs_only  married  children  cath  evang  non_chrst  \\\n",
       "0       Caucasian      6.7     59.3      67.0   8.1    8.7       49.9   \n",
       "1       Caucasian     22.2     81.4      56.0  12.9   20.7       36.3   \n",
       "2       Caucasian     12.8     98.1      84.2  15.7   16.5       33.2   \n",
       "3  Middle Eastern      9.8     97.7      59.6  11.0   11.1       49.3   \n",
       "4       Caucasian      7.2     72.7      13.1   7.4    7.5       59.5   \n",
       "\n",
       "   other_chrst  days_reg  dist_poll_original  \n",
       "0         33.3       411             9.25020  \n",
       "1         30.1       579             4.28483  \n",
       "2         34.5       544             1.99877  \n",
       "3         28.6       392             2.41070  \n",
       "4         25.6       676             3.67070  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#create dataframe as input for MICE algorithm\n",
    "data_r = no_nulls.copy()\n",
    "data_r = data_r.reset_index(drop = True)\n",
    "\n",
    "#create a copy of 'dist_poll' so original values are preserved\n",
    "data_r['dist_poll_original'] = data_r['dist_poll']\n",
    "data_r.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to compare the accuracy of MICE relative to kNN imputation, let's replace the dist_poll values in data_r corresponding to dist_poll values in y_test_cv used to test kNN imputation with NaN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for i in y_test_cv:\n",
    "    data_r['dist_poll'].replace(i, np.nan, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "780"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "703"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_r.ix[data_r['dist_poll'].isnull()])\n",
    "len(y_test_cv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are slightly more missing values in the dataframe for MICE imputation compared to the number of values in the test set due to replicates. However, the numbers are not too different therefore the errors should be comparable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#test matrix for MICE\n",
    "y_test_index = data_r.ix[data_r['dist_poll'].isnull()].index\n",
    "y_test_mice = data_r.ix[data_r['dist_poll'].isnull()]['dist_poll_original'].as_matrix()\n",
    "del data_r['dist_poll_original']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Impute missing values via MICE (see [here](chrome-extension://oemmndcbldboiebfnladdacbdfmadadm/https://cran.r-project.org/web/packages/mice/mice.pdf))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " iter imp variable\n",
       "  1   1  dist_poll\n",
       "  1   2  dist_poll\n",
       "  1   3  dist_poll\n",
       "  1   4  dist_poll\n",
       "  1   5  dist_poll\n",
       "  2   1  dist_poll\n",
       "  2   2  dist_poll\n",
       "  2   3  dist_poll\n",
       "  2   4  dist_poll\n",
       "  2   5  dist_poll\n",
       "  3   1  dist_poll\n",
       "  3   2  dist_poll\n",
       "  3   3  dist_poll\n",
       "  3   4  dist_poll\n",
       "  3   5  dist_poll\n",
       "  4   1  dist_poll\n",
       "  4   2  dist_poll\n",
       "  4   3  dist_poll\n",
       "  4   4  dist_poll\n",
       "  4   5  dist_poll\n",
       "  5   1  dist_poll\n",
       "  5   2  dist_poll\n",
       "  5   3  dist_poll\n",
       "  5   4  dist_poll\n",
       "  5   5  dist_poll\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i data_r -o data_imp\n",
    "\n",
    "library('mice')\n",
    "\n",
    "# Stochastic regression imputation\n",
    "imp <- mice(data_r, method = \"norm.predict\")\n",
    "\n",
    "# Store data\n",
    "data_imp <- complete(imp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.3563621328785747"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_predict_mice = data_imp.iloc[y_test_index, :]['dist_poll'].as_matrix()\n",
    "\n",
    "mice_error = mean_squared_error(y_test_mice, y_predict_mice)\n",
    "mice_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MICE resulted in lower MSE compared to kNN, therefore we will impute the missing distance metrics using the former technique.\n",
    "\n",
    "## Fill missing district values\n",
    "\n",
    "There are two observations from the train data and one from the test data where values for congress_district and state_house are missing. MICE cannot be used for categorical variables with multiple levels, so let's fill the three values with kNN (mode imputation would likely be sufficient as well, as there are so few missing values)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18566</th>\n",
       "      <td>M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>44.1</td>\n",
       "      <td>5.7</td>\n",
       "      <td>34.0</td>\n",
       "      <td>10.8</td>\n",
       "      <td>25.3</td>\n",
       "      <td>33.2</td>\n",
       "      <td>30.8</td>\n",
       "      <td>292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67915</th>\n",
       "      <td>M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Uncoded</td>\n",
       "      <td>8.2</td>\n",
       "      <td>87.0</td>\n",
       "      <td>66.2</td>\n",
       "      <td>12.3</td>\n",
       "      <td>9.3</td>\n",
       "      <td>41.5</td>\n",
       "      <td>36.9</td>\n",
       "      <td>440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5679</th>\n",
       "      <td>M</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>34</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>R</td>\n",
       "      <td>Hispanic</td>\n",
       "      <td>14.5</td>\n",
       "      <td>61.9</td>\n",
       "      <td>40.7</td>\n",
       "      <td>8.2</td>\n",
       "      <td>12.0</td>\n",
       "      <td>47.4</td>\n",
       "      <td>32.4</td>\n",
       "      <td>488</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      gender congress_district state_house  age  dist_ballot  dist_poll party  \\\n",
       "18566      M               NaN         NaN   23          NaN        NaN     R   \n",
       "67915      M               NaN         NaN   41          NaN        NaN     R   \n",
       "5679       M               NaN         NaN   34          NaN        NaN     R   \n",
       "\n",
       "            race  hs_only  married  children  cath  evang  non_chrst  \\\n",
       "18566  Caucasian     44.1      5.7      34.0  10.8   25.3       33.2   \n",
       "67915    Uncoded      8.2     87.0      66.2  12.3    9.3       41.5   \n",
       "5679    Hispanic     14.5     61.9      40.7   8.2   12.0       47.4   \n",
       "\n",
       "       other_chrst  days_reg  \n",
       "18566         30.8       292  \n",
       "67915         36.9       440  \n",
       "5679          32.4       488  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined.ix[combined['state_house'].isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be imputing the missing distance variables using MICE, so let's drop those columns for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dist_vars =  combined[['dist_ballot', 'dist_poll']]\n",
    "combined.drop(['dist_ballot', 'dist_poll'], axis = 1, inplace = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train on all but the observations with missing values and 'test' (fill) the missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 242,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_fill = combined.ix[combined['state_house'].isnull()]\n",
    "\n",
    "#remove district variables as these are nulls; we will be 'predicting' these values\n",
    "X_test_fill = X_test_fill.drop('congress_district state_house'.split(), axis = 1)\n",
    "\n",
    "quant_knn = ['age', 'hs_only', 'married', 'children', 'cath', \n",
    "             'evang', 'non_chrst', 'other_chrst', 'days_reg']\n",
    "categ_knn = ['gender', 'party', 'race']\n",
    "\n",
    "#matrix with values to use for 'prediction'\n",
    "X_test_fill = wv.design_Xmatrix(X_test_fill, quant_knn, categ_knn, standardize = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 243,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/site-packages/ipykernel/__main__.py:7: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n"
     ]
    }
   ],
   "source": [
    "X_train_fill = combined.dropna(subset = ['state_house'])\n",
    "\n",
    "#target variable to train on\n",
    "y_train_fillcd = X_train_fill['congress_district'].astype(int).as_matrix()\n",
    "y_train_fillsh = X_train_fill['state_house'].astype(int).as_matrix()\n",
    "\n",
    "X_train_fill.drop(['congress_district', 'state_house'], axis = 1, inplace = True)\n",
    "\n",
    "#'predictor' variables to train on\n",
    "X_train_fill = wv.design_Xmatrix(X_train_fill, quant_knn, categ_knn, \n",
    "                                 standardize = True, matrix = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The X matrix for the 'test' set (the data to fill) is lacking most of the levels of the categorical variables. We therefore have to add columns to X_test_fill with 0s for the other levels and order the columns so they match the columns in X_train_fill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols_to_add = X_train_fill.drop(X_test_fill.columns, axis = 1).columns\n",
    "for col in cols_to_add:\n",
    "    X_test_fill[col] = 0\n",
    "\n",
    "#get columns to re-organize\n",
    "organize_cols = X_test_fill.columns.tolist()\n",
    "\n",
    "#re-order columns so that columns in the test and train sets match\n",
    "ordered_cols = organize_cols[0:9] + organize_cols[14:15] + organize_cols[9:10] + organize_cols[15:20]\\\n",
    "+ organize_cols[10:11] + organize_cols[20:22] + organize_cols[11:12] + organize_cols[22:24] + organize_cols[12:13] + \\\n",
    "organize_cols[24:] + organize_cols[13:14]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>gender_F</th>\n",
       "      <th>gender_M</th>\n",
       "      <th>gender_U</th>\n",
       "      <th>party_D</th>\n",
       "      <th>party_G</th>\n",
       "      <th>party_L</th>\n",
       "      <th>party_O</th>\n",
       "      <th>party_R</th>\n",
       "      <th>party_U</th>\n",
       "      <th>race_African-American</th>\n",
       "      <th>race_Caucasian</th>\n",
       "      <th>race_Central Asian</th>\n",
       "      <th>race_East Asian</th>\n",
       "      <th>race_Hispanic</th>\n",
       "      <th>race_Jewish</th>\n",
       "      <th>race_Middle Eastern</th>\n",
       "      <th>race_Native American</th>\n",
       "      <th>race_Pacific Islander</th>\n",
       "      <th>race_South Asian</th>\n",
       "      <th>race_Uncoded</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18566</th>\n",
       "      <td>-1.065342</td>\n",
       "      <td>1.139003</td>\n",
       "      <td>-1.100980</td>\n",
       "      <td>-0.763190</td>\n",
       "      <td>0.176754</td>\n",
       "      <td>1.140268</td>\n",
       "      <td>-1.051344</td>\n",
       "      <td>-0.811516</td>\n",
       "      <td>-1.122386</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67915</th>\n",
       "      <td>0.918398</td>\n",
       "      <td>-0.733831</td>\n",
       "      <td>0.851958</td>\n",
       "      <td>1.132032</td>\n",
       "      <td>0.899838</td>\n",
       "      <td>-0.727748</td>\n",
       "      <td>0.112143</td>\n",
       "      <td>1.117152</td>\n",
       "      <td>0.326275</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5679</th>\n",
       "      <td>0.146944</td>\n",
       "      <td>-0.405172</td>\n",
       "      <td>0.249022</td>\n",
       "      <td>-0.368842</td>\n",
       "      <td>-1.076592</td>\n",
       "      <td>-0.412520</td>\n",
       "      <td>0.939201</td>\n",
       "      <td>-0.305636</td>\n",
       "      <td>0.796111</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            age   hs_only   married  children      cath     evang  non_chrst  \\\n",
       "18566 -1.065342  1.139003 -1.100980 -0.763190  0.176754  1.140268  -1.051344   \n",
       "67915  0.918398 -0.733831  0.851958  1.132032  0.899838 -0.727748   0.112143   \n",
       "5679   0.146944 -0.405172  0.249022 -0.368842 -1.076592 -0.412520   0.939201   \n",
       "\n",
       "       other_chrst  days_reg  gender_F  gender_M  gender_U  party_D  party_G  \\\n",
       "18566    -0.811516 -1.122386         0         1         0        0        0   \n",
       "67915     1.117152  0.326275         0         1         0        0        0   \n",
       "5679     -0.305636  0.796111         0         1         0        0        0   \n",
       "\n",
       "       party_L  party_O  party_R  party_U  race_African-American  \\\n",
       "18566        0        0        1        0                      0   \n",
       "67915        0        0        1        0                      0   \n",
       "5679         0        0        1        0                      0   \n",
       "\n",
       "       race_Caucasian  race_Central Asian  race_East Asian  race_Hispanic  \\\n",
       "18566               1                   0                0              0   \n",
       "67915               0                   0                0              0   \n",
       "5679                0                   0                0              1   \n",
       "\n",
       "       race_Jewish  race_Middle Eastern  race_Native American  \\\n",
       "18566            0                    0                     0   \n",
       "67915            0                    0                     0   \n",
       "5679             0                    0                     0   \n",
       "\n",
       "       race_Pacific Islander  race_South Asian  race_Uncoded  \n",
       "18566                      0                 0             0  \n",
       "67915                      0                 0             1  \n",
       "5679                       0                 0             0  "
      ]
     },
     "execution_count": 245,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_fill = X_test_fill[ordered_cols]\n",
    "X_test_fill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>gender_F</th>\n",
       "      <th>gender_M</th>\n",
       "      <th>gender_U</th>\n",
       "      <th>party_D</th>\n",
       "      <th>party_G</th>\n",
       "      <th>party_L</th>\n",
       "      <th>party_O</th>\n",
       "      <th>party_R</th>\n",
       "      <th>party_U</th>\n",
       "      <th>race_African-American</th>\n",
       "      <th>race_Caucasian</th>\n",
       "      <th>race_Central Asian</th>\n",
       "      <th>race_East Asian</th>\n",
       "      <th>race_Hispanic</th>\n",
       "      <th>race_Jewish</th>\n",
       "      <th>race_Middle Eastern</th>\n",
       "      <th>race_Native American</th>\n",
       "      <th>race_Pacific Islander</th>\n",
       "      <th>race_South Asian</th>\n",
       "      <th>race_Uncoded</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.019491</td>\n",
       "      <td>0.164934</td>\n",
       "      <td>0.597777</td>\n",
       "      <td>1.007627</td>\n",
       "      <td>0.742142</td>\n",
       "      <td>0.110427</td>\n",
       "      <td>-0.110196</td>\n",
       "      <td>-1.023766</td>\n",
       "      <td>-0.357578</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.231606</td>\n",
       "      <td>-1.006899</td>\n",
       "      <td>1.684673</td>\n",
       "      <td>1.289149</td>\n",
       "      <td>0.742142</td>\n",
       "      <td>-0.044526</td>\n",
       "      <td>-0.937741</td>\n",
       "      <td>1.540320</td>\n",
       "      <td>-1.390249</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.746056</td>\n",
       "      <td>1.825590</td>\n",
       "      <td>-1.165269</td>\n",
       "      <td>0.789204</td>\n",
       "      <td>0.386237</td>\n",
       "      <td>1.272572</td>\n",
       "      <td>-1.061397</td>\n",
       "      <td>0.151440</td>\n",
       "      <td>-1.527329</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.682260</td>\n",
       "      <td>1.008654</td>\n",
       "      <td>-1.136833</td>\n",
       "      <td>0.687274</td>\n",
       "      <td>0.132019</td>\n",
       "      <td>1.009152</td>\n",
       "      <td>-0.709453</td>\n",
       "      <td>0.071312</td>\n",
       "      <td>-1.308001</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-0.873647</td>\n",
       "      <td>0.506440</td>\n",
       "      <td>-0.801917</td>\n",
       "      <td>-0.492208</td>\n",
       "      <td>0.623507</td>\n",
       "      <td>-0.819289</td>\n",
       "      <td>-0.157756</td>\n",
       "      <td>0.899299</td>\n",
       "      <td>-0.613461</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        age   hs_only   married  children      cath     evang  non_chrst  \\\n",
       "0  0.019491  0.164934  0.597777  1.007627  0.742142  0.110427  -0.110196   \n",
       "1  1.231606 -1.006899  1.684673  1.289149  0.742142 -0.044526  -0.937741   \n",
       "2 -0.746056  1.825590 -1.165269  0.789204  0.386237  1.272572  -1.061397   \n",
       "3 -0.682260  1.008654 -1.136833  0.687274  0.132019  1.009152  -0.709453   \n",
       "4 -0.873647  0.506440 -0.801917 -0.492208  0.623507 -0.819289  -0.157756   \n",
       "\n",
       "   other_chrst  days_reg  gender_F  gender_M  gender_U  party_D  party_G  \\\n",
       "0    -1.023766 -0.357578         0         1         0        0        0   \n",
       "1     1.540320 -1.390249         1         0         0        0        0   \n",
       "2     0.151440 -1.527329         1         0         0        0        0   \n",
       "3     0.071312 -1.308001         1         0         0        1        0   \n",
       "4     0.899299 -0.613461         0         1         0        0        0   \n",
       "\n",
       "   party_L  party_O  party_R  party_U  race_African-American  race_Caucasian  \\\n",
       "0        0        0        0        1                      0               0   \n",
       "1        0        0        0        1                      0               0   \n",
       "2        0        0        0        1                      0               1   \n",
       "3        0        0        0        0                      0               1   \n",
       "4        0        0        1        0                      0               1   \n",
       "\n",
       "   race_Central Asian  race_East Asian  race_Hispanic  race_Jewish  \\\n",
       "0                   0                0              1            0   \n",
       "1                   0                0              0            0   \n",
       "2                   0                0              0            0   \n",
       "3                   0                0              0            0   \n",
       "4                   0                0              0            0   \n",
       "\n",
       "   race_Middle Eastern  race_Native American  race_Pacific Islander  \\\n",
       "0                    0                     0                      0   \n",
       "1                    0                     0                      0   \n",
       "2                    0                     0                      0   \n",
       "3                    0                     0                      0   \n",
       "4                    0                     0                      0   \n",
       "\n",
       "   race_South Asian  race_Uncoded  \n",
       "0                 0             0  \n",
       "1                 0             1  \n",
       "2                 0             0  \n",
       "3                 0             0  \n",
       "4                 0             0  "
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_fill.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train_fill = X_train_fill.as_matrix()\n",
    "X_test_fill = X_test_fill.as_matrix()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Predict' (fill) missing congressional and state house districts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.92307692,  2.19230769,  2.26923077])"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#use model where weights are uniform\n",
    "fillcd_knn = model_knn2.fit(X_train_fill, y_train_fillcd)\n",
    "\n",
    "#get predictions\n",
    "y_predict_cd = fillcd_knn.predict(X_test_fill)\n",
    "y_predict_cd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 14.57692308,  14.07692308,  16.57692308])"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fillsh_knn = model_knn2.fit(X_train_fill, y_train_fillsh)\n",
    "y_predict_sh = fillsh_knn.predict(X_test_fill)\n",
    "y_predict_sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "kNN predicts that all 3 individuals are most likely in congressional district 2 and are most likely in state house districts 15, 14 and 17. Let's fill in these values in the original data frame.\n",
    "\n",
    "**Note:** As already mentioned, mode imputation would have been sufficient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#fill congressional district\n",
    "combined = combined.fillna(value = {'congress_district':'2'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 269,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "combined.loc[18566] = combined.loc[18566].fillna('15')\n",
    "combined.loc[67915] = combined.loc[67915].fillna('14')\n",
    "combined.loc[5679] = combined.loc[5679].fillna('17')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filling missing distance variables using MICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "combined_R = pd.concat([combined, dist_vars], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 158039 entries, 0 to 39509\n",
      "Data columns (total 16 columns):\n",
      "gender               158039 non-null object\n",
      "congress_district    158039 non-null object\n",
      "state_house          158039 non-null object\n",
      "age                  158039 non-null int64\n",
      "party                158039 non-null object\n",
      "race                 158039 non-null object\n",
      "hs_only              158039 non-null float64\n",
      "married              158039 non-null float64\n",
      "children             158039 non-null float64\n",
      "cath                 158039 non-null float64\n",
      "evang                158039 non-null float64\n",
      "non_chrst            158039 non-null float64\n",
      "other_chrst          158039 non-null float64\n",
      "days_reg             158039 non-null int64\n",
      "dist_ballot          7034 non-null float64\n",
      "dist_poll            7034 non-null float64\n",
      "dtypes: float64(9), int64(2), object(5)\n",
      "memory usage: 20.5+ MB\n"
     ]
    }
   ],
   "source": [
    "combined_R.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " iter imp variable\n",
       "  1   1  dist_ballot  dist_poll\n",
       "  1   2  dist_ballot  dist_poll\n",
       "  1   3  dist_ballot  dist_poll\n",
       "  1   4  dist_ballot  dist_poll\n",
       "  1   5  dist_ballot  dist_poll\n",
       "  2   1  dist_ballot  dist_poll\n",
       "  2   2  dist_ballot  dist_poll\n",
       "  2   3  dist_ballot  dist_poll\n",
       "  2   4  dist_ballot  dist_poll\n",
       "  2   5  dist_ballot  dist_poll\n",
       "  3   1  dist_ballot  dist_poll\n",
       "  3   2  dist_ballot  dist_poll\n",
       "  3   3  dist_ballot  dist_poll\n",
       "  3   4  dist_ballot  dist_poll\n",
       "  3   5  dist_ballot  dist_poll\n",
       "  4   1  dist_ballot  dist_poll\n",
       "  4   2  dist_ballot  dist_poll\n",
       "  4   3  dist_ballot  dist_poll\n",
       "  4   4  dist_ballot  dist_poll\n",
       "  4   5  dist_ballot  dist_poll\n",
       "  5   1  dist_ballot  dist_poll\n",
       "  5   2  dist_ballot  dist_poll\n",
       "  5   3  dist_ballot  dist_poll\n",
       "  5   4  dist_ballot  dist_poll\n",
       "  5   5  dist_ballot  dist_poll\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%R -i combined_R -o combined_imp\n",
    "\n",
    "library('mice')\n",
    "\n",
    "# Stochastic regression imputation\n",
    "imp <- mice(combined_R, method = \"norm.predict\")\n",
    "\n",
    "# Store data\n",
    "combined_imp <- complete(imp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>7</td>\n",
       "      <td>31</td>\n",
       "      <td>36</td>\n",
       "      <td>U</td>\n",
       "      <td>Hispanic</td>\n",
       "      <td>25.4</td>\n",
       "      <td>63.4</td>\n",
       "      <td>54.0</td>\n",
       "      <td>16.7</td>\n",
       "      <td>16.5</td>\n",
       "      <td>39.6</td>\n",
       "      <td>27.3</td>\n",
       "      <td>420</td>\n",
       "      <td>2.658148</td>\n",
       "      <td>3.456541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>6</td>\n",
       "      <td>38</td>\n",
       "      <td>55</td>\n",
       "      <td>U</td>\n",
       "      <td>Uncoded</td>\n",
       "      <td>7.9</td>\n",
       "      <td>97.8</td>\n",
       "      <td>59.8</td>\n",
       "      <td>16.7</td>\n",
       "      <td>15.5</td>\n",
       "      <td>30.9</td>\n",
       "      <td>36.9</td>\n",
       "      <td>307</td>\n",
       "      <td>2.240415</td>\n",
       "      <td>2.130636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>24</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>50.2</td>\n",
       "      <td>7.6</td>\n",
       "      <td>49.5</td>\n",
       "      <td>14.6</td>\n",
       "      <td>24.0</td>\n",
       "      <td>29.6</td>\n",
       "      <td>31.7</td>\n",
       "      <td>292</td>\n",
       "      <td>2.332767</td>\n",
       "      <td>2.872147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>25</td>\n",
       "      <td>D</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>38.0</td>\n",
       "      <td>8.5</td>\n",
       "      <td>47.4</td>\n",
       "      <td>13.1</td>\n",
       "      <td>22.3</td>\n",
       "      <td>33.3</td>\n",
       "      <td>31.4</td>\n",
       "      <td>316</td>\n",
       "      <td>2.983480</td>\n",
       "      <td>3.193841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>M</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>22</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>30.5</td>\n",
       "      <td>19.1</td>\n",
       "      <td>23.1</td>\n",
       "      <td>16.0</td>\n",
       "      <td>10.5</td>\n",
       "      <td>39.1</td>\n",
       "      <td>34.5</td>\n",
       "      <td>392</td>\n",
       "      <td>3.160107</td>\n",
       "      <td>3.841248</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender congress_district state_house  age party       race  hs_only  \\\n",
       "1      M                 7          31   36     U   Hispanic     25.4   \n",
       "2      F                 6          38   55     U    Uncoded      7.9   \n",
       "3      F                 2          53   24     U  Caucasian     50.2   \n",
       "4      F                 7          30   25     D  Caucasian     38.0   \n",
       "5      M                 5          19   22     R  Caucasian     30.5   \n",
       "\n",
       "   married  children  cath  evang  non_chrst  other_chrst  days_reg  \\\n",
       "1     63.4      54.0  16.7   16.5       39.6         27.3       420   \n",
       "2     97.8      59.8  16.7   15.5       30.9         36.9       307   \n",
       "3      7.6      49.5  14.6   24.0       29.6         31.7       292   \n",
       "4      8.5      47.4  13.1   22.3       33.3         31.4       316   \n",
       "5     19.1      23.1  16.0   10.5       39.1         34.5       392   \n",
       "\n",
       "   dist_ballot  dist_poll  \n",
       "1     2.658148   3.456541  \n",
       "2     2.240415   2.130636  \n",
       "3     2.332767   2.872147  \n",
       "4     2.983480   3.193841  \n",
       "5     3.160107   3.841248  "
      ]
     },
     "execution_count": 283,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "combined_imp.reset_index(drop = True, inplace = True)\n",
    "combined_imp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill NaNs in original test and train data sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 284,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(118529, 16)"
      ]
     },
     "execution_count": 284,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(39510, 16)"
      ]
     },
     "execution_count": 284,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape\n",
    "test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 312,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_imp = combined_imp.iloc[118529:, :]\n",
    "train_imp = combined_imp.iloc[:118529, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(118529, 16)"
      ]
     },
     "execution_count": 309,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(39510, 16)"
      ]
     },
     "execution_count": 309,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_imp.shape\n",
    "test_imp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>voted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>118524</th>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>48</td>\n",
       "      <td>41</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>14.0</td>\n",
       "      <td>76.5</td>\n",
       "      <td>67.3</td>\n",
       "      <td>9.1</td>\n",
       "      <td>19.6</td>\n",
       "      <td>37.3</td>\n",
       "      <td>33.9</td>\n",
       "      <td>673</td>\n",
       "      <td>3.291435</td>\n",
       "      <td>5.169437</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118525</th>\n",
       "      <td>M</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>22.0</td>\n",
       "      <td>49.5</td>\n",
       "      <td>10.8</td>\n",
       "      <td>13.3</td>\n",
       "      <td>15.7</td>\n",
       "      <td>40.2</td>\n",
       "      <td>30.7</td>\n",
       "      <td>665</td>\n",
       "      <td>2.441778</td>\n",
       "      <td>2.594092</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118526</th>\n",
       "      <td>F</td>\n",
       "      <td>2</td>\n",
       "      <td>53</td>\n",
       "      <td>22</td>\n",
       "      <td>D</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>26.6</td>\n",
       "      <td>7.4</td>\n",
       "      <td>23.0</td>\n",
       "      <td>12.6</td>\n",
       "      <td>13.6</td>\n",
       "      <td>42.8</td>\n",
       "      <td>31.0</td>\n",
       "      <td>431</td>\n",
       "      <td>2.219713</td>\n",
       "      <td>2.578724</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118527</th>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>28</td>\n",
       "      <td>D</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>6.2</td>\n",
       "      <td>8.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>6.5</td>\n",
       "      <td>6.3</td>\n",
       "      <td>60.9</td>\n",
       "      <td>26.2</td>\n",
       "      <td>412</td>\n",
       "      <td>2.114958</td>\n",
       "      <td>2.055566</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118528</th>\n",
       "      <td>F</td>\n",
       "      <td>5</td>\n",
       "      <td>39</td>\n",
       "      <td>56</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>15.1</td>\n",
       "      <td>62.6</td>\n",
       "      <td>49.4</td>\n",
       "      <td>12.1</td>\n",
       "      <td>24.4</td>\n",
       "      <td>29.3</td>\n",
       "      <td>34.2</td>\n",
       "      <td>498</td>\n",
       "      <td>6.521782</td>\n",
       "      <td>7.519992</td>\n",
       "      <td>N</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       gender congress_district state_house  age party       race  hs_only  \\\n",
       "118524      M                 4          48   41     U  Caucasian     14.0   \n",
       "118525      M                 1           5   60     U  Caucasian     22.0   \n",
       "118526      F                 2          53   22     D  Caucasian     26.6   \n",
       "118527      F                 1           2   28     D  Caucasian      6.2   \n",
       "118528      F                 5          39   56     U  Caucasian     15.1   \n",
       "\n",
       "        married  children  cath  evang  non_chrst  other_chrst  days_reg  \\\n",
       "118524     76.5      67.3   9.1   19.6       37.3         33.9       673   \n",
       "118525     49.5      10.8  13.3   15.7       40.2         30.7       665   \n",
       "118526      7.4      23.0  12.6   13.6       42.8         31.0       431   \n",
       "118527      8.7       7.2   6.5    6.3       60.9         26.2       412   \n",
       "118528     62.6      49.4  12.1   24.4       29.3         34.2       498   \n",
       "\n",
       "        dist_ballot  dist_poll voted  \n",
       "118524     3.291435   5.169437     Y  \n",
       "118525     2.441778   2.594092     Y  \n",
       "118526     2.219713   2.578724     Y  \n",
       "118527     2.114958   2.055566     Y  \n",
       "118528     6.521782   7.519992     N  "
      ]
     },
     "execution_count": 313,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_imp = pd.concat([train_imp, data_raw['voted']], axis = 1)\n",
    "train_imp.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 319,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gender</th>\n",
       "      <th>congress_district</th>\n",
       "      <th>state_house</th>\n",
       "      <th>age</th>\n",
       "      <th>party</th>\n",
       "      <th>race</th>\n",
       "      <th>hs_only</th>\n",
       "      <th>married</th>\n",
       "      <th>children</th>\n",
       "      <th>cath</th>\n",
       "      <th>evang</th>\n",
       "      <th>non_chrst</th>\n",
       "      <th>other_chrst</th>\n",
       "      <th>days_reg</th>\n",
       "      <th>dist_ballot</th>\n",
       "      <th>dist_poll</th>\n",
       "      <th>Id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M</td>\n",
       "      <td>2</td>\n",
       "      <td>52</td>\n",
       "      <td>30</td>\n",
       "      <td>L</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>19.5</td>\n",
       "      <td>21.2</td>\n",
       "      <td>25.3</td>\n",
       "      <td>9.8</td>\n",
       "      <td>16.6</td>\n",
       "      <td>45.2</td>\n",
       "      <td>28.4</td>\n",
       "      <td>393</td>\n",
       "      <td>2.521099</td>\n",
       "      <td>2.770852</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>F</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>20</td>\n",
       "      <td>U</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>39.7</td>\n",
       "      <td>20.2</td>\n",
       "      <td>29.1</td>\n",
       "      <td>12.0</td>\n",
       "      <td>14.4</td>\n",
       "      <td>41.4</td>\n",
       "      <td>32.2</td>\n",
       "      <td>668</td>\n",
       "      <td>3.143472</td>\n",
       "      <td>3.972788</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M</td>\n",
       "      <td>4</td>\n",
       "      <td>44</td>\n",
       "      <td>56</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>11.3</td>\n",
       "      <td>62.7</td>\n",
       "      <td>41.3</td>\n",
       "      <td>14.8</td>\n",
       "      <td>14.7</td>\n",
       "      <td>36.0</td>\n",
       "      <td>34.6</td>\n",
       "      <td>606</td>\n",
       "      <td>2.736109</td>\n",
       "      <td>2.859035</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>7</td>\n",
       "      <td>34</td>\n",
       "      <td>20</td>\n",
       "      <td>R</td>\n",
       "      <td>Caucasian</td>\n",
       "      <td>32.8</td>\n",
       "      <td>11.6</td>\n",
       "      <td>33.1</td>\n",
       "      <td>14.5</td>\n",
       "      <td>10.3</td>\n",
       "      <td>44.6</td>\n",
       "      <td>30.6</td>\n",
       "      <td>565</td>\n",
       "      <td>2.136312</td>\n",
       "      <td>2.329654</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>6</td>\n",
       "      <td>41</td>\n",
       "      <td>26</td>\n",
       "      <td>D</td>\n",
       "      <td>Uncoded</td>\n",
       "      <td>10.2</td>\n",
       "      <td>14.7</td>\n",
       "      <td>22.4</td>\n",
       "      <td>8.2</td>\n",
       "      <td>18.4</td>\n",
       "      <td>43.5</td>\n",
       "      <td>29.9</td>\n",
       "      <td>336</td>\n",
       "      <td>2.785245</td>\n",
       "      <td>2.904892</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  gender congress_district state_house  age party       race  hs_only  \\\n",
       "0      M                 2          52   30     L  Caucasian     19.5   \n",
       "1      F                 5          19   20     U  Caucasian     39.7   \n",
       "2      M                 4          44   56     R  Caucasian     11.3   \n",
       "3      F                 7          34   20     R  Caucasian     32.8   \n",
       "4      F                 6          41   26     D    Uncoded     10.2   \n",
       "\n",
       "   married  children  cath  evang  non_chrst  other_chrst  days_reg  \\\n",
       "0     21.2      25.3   9.8   16.6       45.2         28.4       393   \n",
       "1     20.2      29.1  12.0   14.4       41.4         32.2       668   \n",
       "2     62.7      41.3  14.8   14.7       36.0         34.6       606   \n",
       "3     11.6      33.1  14.5   10.3       44.6         30.6       565   \n",
       "4     14.7      22.4   8.2   18.4       43.5         29.9       336   \n",
       "\n",
       "   dist_ballot  dist_poll  Id  \n",
       "0     2.521099   2.770852   1  \n",
       "1     3.143472   3.972788   2  \n",
       "2     2.736109   2.859035   3  \n",
       "3     2.136312   2.329654   4  \n",
       "4     2.785245   2.904892   5  "
      ]
     },
     "execution_count": 319,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_imp.reset_index(drop = True, inplace = True)\n",
    "test_imp = pd.concat([test_imp, id_col], axis = 1)\n",
    "test_imp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the imputed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_imp.to_csv('train_imp.csv', index = False, compression = 'gzip')\n",
    "test_imp.to_csv('test_imp.csv', index = False, compression = 'gzip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training and testing predictive models, see [part 4](who-voted_modeling.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}