cross_validation_experiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cross Validation Experiment\n",
    "### By Tanay Trivedi and Jonathan Bair\n",
    "\n",
    "The fundamental difference in the implementation of the cross val in 7.10.2 is that feature selection needs be done inside the folds, rather than outside the folds.\n",
    "\n",
    "We are going to generate the dataset inside the example first."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import sklearn\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors=np.linspace(1,5000,5000).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample=np.linspace(0,49,50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=pd.DataFrame(columns=predictors.astype(int),index=sample.astype(int))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.loc[0:24,'target']=1\n",
    "data.loc[25:50,'target']=0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictors_data=np.random.randn(50,5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.loc[0:49,predictors]=predictors_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=data.sample(frac=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "data=data.reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bad Cross Val\n",
    "To perform the bad method, we conduct the correlation test before cross validation, and then train the classifier inside the folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs=pd.Series(index=predictors)\n",
    "for i in predictors:\n",
    "    corrs.loc[i]=data[i].corr(data['target'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_predictors=corrs.sort_values(ascending=False).iloc[0:100].index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=data[best_predictors]\n",
    "y=data['target']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above lines conduct the correlation of each predictor against the target, sorts them and takes the best 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "kf = KFold(n_splits=10)\n",
    "index_gen=kf.split(X,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This generates 10 folds for cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "accs=pd.Series(index=range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    train_index,test_index=next(index_gen)\n",
    "    \n",
    "    neigh = KNeighborsClassifier(n_neighbors=1)\n",
    "    neigh.fit(X.loc[train_index], y.loc[train_index]) \n",
    "    y_hat=neigh.predict(X.loc[test_index])\n",
    "    y_real=y.loc[test_index].values\n",
    "    accs.loc[i]=float(len(y_hat[y_hat==y_real]))/len(y_hat)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1.0\n",
       "1    1.0\n",
       "2    1.0\n",
       "3    1.0\n",
       "4    0.8\n",
       "5    1.0\n",
       "6    1.0\n",
       "7    1.0\n",
       "8    1.0\n",
       "9    1.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.9999999999999907"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1-accs.mean())*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above is the average error rate among folds. Nonsensically high for random data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Good Cross Val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this, we perform the correlation feature selection within the train side of each fold before fitting the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=data[predictors]\n",
    "y=data['target']\n",
    "\n",
    "kf = KFold(n_splits=10)\n",
    "index_gen=kf.split(X,y)\n",
    "accs=pd.Series(index=range(10))\n",
    "for i in range(10):\n",
    "    train_index,test_index=next(index_gen)\n",
    "    this_X=X.loc[train_index]\n",
    "    this_y=y.loc[train_index]\n",
    "    \n",
    "    corrs=pd.Series(index=predictors)\n",
    "    for j in predictors:\n",
    "        corrs.loc[j]=this_X[j].corr(this_y)\n",
    "    best_predictors=corrs.sort_values(ascending=False).iloc[0:100].index\n",
    "    \n",
    "    this_X=this_X[best_predictors]\n",
    "    \n",
    "    neigh = KNeighborsClassifier(n_neighbors=1)\n",
    "    neigh.fit(this_X, this_y) \n",
    "    y_hat=neigh.predict(X.loc[test_index,best_predictors])\n",
    "    y_real=y.loc[test_index].values\n",
    "    accs.loc[i]=float(len(y_hat[y_hat==y_real]))/len(y_hat)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.4\n",
       "1    0.2\n",
       "2    0.8\n",
       "3    0.8\n",
       "4    0.6\n",
       "5    0.4\n",
       "6    0.8\n",
       "7    0.6\n",
       "8    0.4\n",
       "9    0.8\n",
       "dtype: float64"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42.00000000000001"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(1-accs.mean())*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is much more reasonable, it should be around 50%. Taking 10 folds with so little data creates some issues, but the idea is correct."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}