1_student_logreg.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression\n",
    "\n",
    "This model attempts to classify whether students ultimately succeed (pass) or not (fail/withdraw) in a course, based on student demographics information.\n",
    "\n",
    "##### Summary:\n",
    "\n",
    "Using only the most informative features did improve the score from 54% to 60%, but overall logistic regression based on student demographic information is not a very good model for predicting student success. However, this model's interpretability does give us some interesting takeaways in terms of which student features were selected:\n",
    "\n",
    "*Not significant:*\n",
    "* Age\n",
    "* Gendere\n",
    "* Disability status\n",
    "\n",
    "*Significant:*\n",
    "* Highest education level completed (positive)\n",
    "* Number of previous attempts at the course (negative)\n",
    "* Poverty level at place of origin (negative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_info = pd.read_csv('data/studentInfo.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>code_module</th>\n",
       "      <th>code_presentation</th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>region</th>\n",
       "      <th>highest_education</th>\n",
       "      <th>imd_band</th>\n",
       "      <th>age_band</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AAA</td>\n",
       "      <td>2013J</td>\n",
       "      <td>11391</td>\n",
       "      <td>M</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>90-100%</td>\n",
       "      <td>55&lt;=</td>\n",
       "      <td>0</td>\n",
       "      <td>240</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AAA</td>\n",
       "      <td>2013J</td>\n",
       "      <td>28400</td>\n",
       "      <td>F</td>\n",
       "      <td>Scotland</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>20-30%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AAA</td>\n",
       "      <td>2013J</td>\n",
       "      <td>30268</td>\n",
       "      <td>F</td>\n",
       "      <td>North Western Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>30-40%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>Y</td>\n",
       "      <td>Withdrawn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AAA</td>\n",
       "      <td>2013J</td>\n",
       "      <td>31604</td>\n",
       "      <td>F</td>\n",
       "      <td>South East Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AAA</td>\n",
       "      <td>2013J</td>\n",
       "      <td>32885</td>\n",
       "      <td>F</td>\n",
       "      <td>West Midlands Region</td>\n",
       "      <td>Lower Than A Level</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  code_module code_presentation  id_student gender                region  \\\n",
       "0         AAA             2013J       11391      M   East Anglian Region   \n",
       "1         AAA             2013J       28400      F              Scotland   \n",
       "2         AAA             2013J       30268      F  North Western Region   \n",
       "3         AAA             2013J       31604      F     South East Region   \n",
       "4         AAA             2013J       32885      F  West Midlands Region   \n",
       "\n",
       "       highest_education imd_band age_band  num_of_prev_attempts  \\\n",
       "0       HE Qualification  90-100%     55<=                     0   \n",
       "1       HE Qualification   20-30%    35-55                     0   \n",
       "2  A Level or Equivalent   30-40%    35-55                     0   \n",
       "3  A Level or Equivalent   50-60%    35-55                     0   \n",
       "4     Lower Than A Level   50-60%     0-35                     0   \n",
       "\n",
       "   studied_credits disability final_result  \n",
       "0              240          N         Pass  \n",
       "1               60          N         Pass  \n",
       "2               60          Y    Withdrawn  \n",
       "3               60          N         Pass  \n",
       "4               60          N         Pass  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing for Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_info.drop(['code_module', 'code_presentation'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>region</th>\n",
       "      <th>highest_education</th>\n",
       "      <th>imd_band</th>\n",
       "      <th>age_band</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11391</td>\n",
       "      <td>M</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>90-100%</td>\n",
       "      <td>55&lt;=</td>\n",
       "      <td>0</td>\n",
       "      <td>240</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28400</td>\n",
       "      <td>F</td>\n",
       "      <td>Scotland</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>20-30%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30268</td>\n",
       "      <td>F</td>\n",
       "      <td>North Western Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>30-40%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>Y</td>\n",
       "      <td>Withdrawn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31604</td>\n",
       "      <td>F</td>\n",
       "      <td>South East Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32885</td>\n",
       "      <td>F</td>\n",
       "      <td>West Midlands Region</td>\n",
       "      <td>Lower Than A Level</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>N</td>\n",
       "      <td>Pass</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student gender                region      highest_education imd_band  \\\n",
       "0       11391      M   East Anglian Region       HE Qualification  90-100%   \n",
       "1       28400      F              Scotland       HE Qualification   20-30%   \n",
       "2       30268      F  North Western Region  A Level or Equivalent   30-40%   \n",
       "3       31604      F     South East Region  A Level or Equivalent   50-60%   \n",
       "4       32885      F  West Midlands Region     Lower Than A Level   50-60%   \n",
       "\n",
       "  age_band  num_of_prev_attempts  studied_credits disability final_result  \n",
       "0     55<=                     0              240          N         Pass  \n",
       "1    35-55                     0               60          N         Pass  \n",
       "2    35-55                     0               60          Y    Withdrawn  \n",
       "3    35-55                     0               60          N         Pass  \n",
       "4     0-35                     0               60          N         Pass  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_info.final_result.replace(to_replace=dict(Pass=1, Distinction=1, Fail=0, Withdrawn=0), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_info['gender'] = (stu_info['gender']=='M').astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "stu_info['disability'] = (stu_info['disability']=='Y').astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>region</th>\n",
       "      <th>highest_education</th>\n",
       "      <th>imd_band</th>\n",
       "      <th>age_band</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11391</td>\n",
       "      <td>1</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>90-100%</td>\n",
       "      <td>55&lt;=</td>\n",
       "      <td>0</td>\n",
       "      <td>240</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28400</td>\n",
       "      <td>0</td>\n",
       "      <td>Scotland</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>20-30%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30268</td>\n",
       "      <td>0</td>\n",
       "      <td>North Western Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>30-40%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31604</td>\n",
       "      <td>0</td>\n",
       "      <td>South East Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32885</td>\n",
       "      <td>0</td>\n",
       "      <td>West Midlands Region</td>\n",
       "      <td>Lower Than A Level</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student  gender                region      highest_education imd_band  \\\n",
       "0       11391       1   East Anglian Region       HE Qualification  90-100%   \n",
       "1       28400       0              Scotland       HE Qualification   20-30%   \n",
       "2       30268       0  North Western Region  A Level or Equivalent   30-40%   \n",
       "3       31604       0     South East Region  A Level or Equivalent   50-60%   \n",
       "4       32885       0  West Midlands Region     Lower Than A Level   50-60%   \n",
       "\n",
       "  age_band  num_of_prev_attempts  studied_credits  disability  final_result  \n",
       "0     55<=                     0              240           0             1  \n",
       "1    35-55                     0               60           0             1  \n",
       "2    35-55                     0               60           1             0  \n",
       "3    35-55                     0               60           0             1  \n",
       "4     0-35                     0               60           0             1  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['East Anglian Region', 'Scotland', 'North Western Region',\n",
       "       'South East Region', 'West Midlands Region', 'Wales',\n",
       "       'North Region', 'South Region', 'Ireland', 'South West Region',\n",
       "       'East Midlands Region', 'Yorkshire Region', 'London Region'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info['region'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['HE Qualification', 'A Level or Equivalent', 'Lower Than A Level',\n",
       "       'Post Graduate Qualification', 'No Formal quals'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info['highest_education'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['90-100%', '20-30%', '30-40%', '50-60%', '80-90%', '70-80%', nan,\n",
       "       '60-70%', '40-50%', '10-20', '0-10%'], dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info['imd_band'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['55<=', '35-55', '0-35'], dtype=object)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info['age_band'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dummy variables for the categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_vars = ['region','highest_education','imd_band','age_band']\n",
    "for var in cat_vars:\n",
    "    cat_list = 'var' + '_' + var\n",
    "    cat_list = pd.get_dummies(stu_info[var], prefix=var)\n",
    "    stu_info1 = stu_info.join(cat_list)\n",
    "    stu_info = stu_info1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>region</th>\n",
       "      <th>highest_education</th>\n",
       "      <th>imd_band</th>\n",
       "      <th>age_band</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "      <th>...</th>\n",
       "      <th>imd_band_30-40%</th>\n",
       "      <th>imd_band_40-50%</th>\n",
       "      <th>imd_band_50-60%</th>\n",
       "      <th>imd_band_60-70%</th>\n",
       "      <th>imd_band_70-80%</th>\n",
       "      <th>imd_band_80-90%</th>\n",
       "      <th>imd_band_90-100%</th>\n",
       "      <th>age_band_0-35</th>\n",
       "      <th>age_band_35-55</th>\n",
       "      <th>age_band_55&lt;=</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11391</td>\n",
       "      <td>1</td>\n",
       "      <td>East Anglian Region</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>90-100%</td>\n",
       "      <td>55&lt;=</td>\n",
       "      <td>0</td>\n",
       "      <td>240</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28400</td>\n",
       "      <td>0</td>\n",
       "      <td>Scotland</td>\n",
       "      <td>HE Qualification</td>\n",
       "      <td>20-30%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30268</td>\n",
       "      <td>0</td>\n",
       "      <td>North Western Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>30-40%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31604</td>\n",
       "      <td>0</td>\n",
       "      <td>South East Region</td>\n",
       "      <td>A Level or Equivalent</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>35-55</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32885</td>\n",
       "      <td>0</td>\n",
       "      <td>West Midlands Region</td>\n",
       "      <td>Lower Than A Level</td>\n",
       "      <td>50-60%</td>\n",
       "      <td>0-35</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student  gender                region      highest_education imd_band  \\\n",
       "0       11391       1   East Anglian Region       HE Qualification  90-100%   \n",
       "1       28400       0              Scotland       HE Qualification   20-30%   \n",
       "2       30268       0  North Western Region  A Level or Equivalent   30-40%   \n",
       "3       31604       0     South East Region  A Level or Equivalent   50-60%   \n",
       "4       32885       0  West Midlands Region     Lower Than A Level   50-60%   \n",
       "\n",
       "  age_band  num_of_prev_attempts  studied_credits  disability  final_result  \\\n",
       "0     55<=                     0              240           0             1   \n",
       "1    35-55                     0               60           0             1   \n",
       "2    35-55                     0               60           1             0   \n",
       "3    35-55                     0               60           0             1   \n",
       "4     0-35                     0               60           0             1   \n",
       "\n",
       "   ...  imd_band_30-40%  imd_band_40-50%  imd_band_50-60%  imd_band_60-70%  \\\n",
       "0  ...                0                0                0                0   \n",
       "1  ...                0                0                0                0   \n",
       "2  ...                1                0                0                0   \n",
       "3  ...                0                0                1                0   \n",
       "4  ...                0                0                1                0   \n",
       "\n",
       "   imd_band_70-80%  imd_band_80-90%  imd_band_90-100%  age_band_0-35  \\\n",
       "0                0                0                 1              0   \n",
       "1                0                0                 0              0   \n",
       "2                0                0                 0              0   \n",
       "3                0                0                 0              0   \n",
       "4                0                0                 0              1   \n",
       "\n",
       "   age_band_35-55  age_band_55<=  \n",
       "0               0              1  \n",
       "1               1              0  \n",
       "2               1              0  \n",
       "3               1              0  \n",
       "4               0              0  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stu_info.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "info_vars = stu_info.columns.values.tolist()\n",
    "to_keep = [i for i in info_vars if i not in cat_vars]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['id_student', 'gender', 'num_of_prev_attempts', 'studied_credits',\n",
       "       'disability', 'final_result', 'region_East Anglian Region',\n",
       "       'region_East Midlands Region', 'region_Ireland',\n",
       "       'region_London Region', 'region_North Region',\n",
       "       'region_North Western Region', 'region_Scotland',\n",
       "       'region_South East Region', 'region_South Region',\n",
       "       'region_South West Region', 'region_Wales',\n",
       "       'region_West Midlands Region', 'region_Yorkshire Region',\n",
       "       'highest_education_A Level or Equivalent',\n",
       "       'highest_education_HE Qualification',\n",
       "       'highest_education_Lower Than A Level',\n",
       "       'highest_education_No Formal quals',\n",
       "       'highest_education_Post Graduate Qualification', 'imd_band_0-10%',\n",
       "       'imd_band_10-20', 'imd_band_20-30%', 'imd_band_30-40%',\n",
       "       'imd_band_40-50%', 'imd_band_50-60%', 'imd_band_60-70%',\n",
       "       'imd_band_70-80%', 'imd_band_80-90%', 'imd_band_90-100%',\n",
       "       'age_band_0-35', 'age_band_35-55', 'age_band_55<='], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stuinfo_final = stu_info[to_keep]\n",
    "stuinfo_final.columns.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id_student</th>\n",
       "      <th>gender</th>\n",
       "      <th>num_of_prev_attempts</th>\n",
       "      <th>studied_credits</th>\n",
       "      <th>disability</th>\n",
       "      <th>final_result</th>\n",
       "      <th>region_East Anglian Region</th>\n",
       "      <th>region_East Midlands Region</th>\n",
       "      <th>region_Ireland</th>\n",
       "      <th>region_London Region</th>\n",
       "      <th>...</th>\n",
       "      <th>imd_band_30-40%</th>\n",
       "      <th>imd_band_40-50%</th>\n",
       "      <th>imd_band_50-60%</th>\n",
       "      <th>imd_band_60-70%</th>\n",
       "      <th>imd_band_70-80%</th>\n",
       "      <th>imd_band_80-90%</th>\n",
       "      <th>imd_band_90-100%</th>\n",
       "      <th>age_band_0-35</th>\n",
       "      <th>age_band_35-55</th>\n",
       "      <th>age_band_55&lt;=</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11391</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>240</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28400</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>30268</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>31604</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32885</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   id_student  gender  num_of_prev_attempts  studied_credits  disability  \\\n",
       "0       11391       1                     0              240           0   \n",
       "1       28400       0                     0               60           0   \n",
       "2       30268       0                     0               60           1   \n",
       "3       31604       0                     0               60           0   \n",
       "4       32885       0                     0               60           0   \n",
       "\n",
       "   final_result  region_East Anglian Region  region_East Midlands Region  \\\n",
       "0             1                           1                            0   \n",
       "1             1                           0                            0   \n",
       "2             0                           0                            0   \n",
       "3             1                           0                            0   \n",
       "4             1                           0                            0   \n",
       "\n",
       "   region_Ireland  region_London Region  ...  imd_band_30-40%  \\\n",
       "0               0                     0  ...                0   \n",
       "1               0                     0  ...                0   \n",
       "2               0                     0  ...                1   \n",
       "3               0                     0  ...                0   \n",
       "4               0                     0  ...                0   \n",
       "\n",
       "   imd_band_40-50%  imd_band_50-60%  imd_band_60-70%  imd_band_70-80%  \\\n",
       "0                0                0                0                0   \n",
       "1                0                0                0                0   \n",
       "2                0                0                0                0   \n",
       "3                0                1                0                0   \n",
       "4                0                1                0                0   \n",
       "\n",
       "   imd_band_80-90%  imd_band_90-100%  age_band_0-35  age_band_35-55  \\\n",
       "0                0                 1              0               0   \n",
       "1                0                 0              0               1   \n",
       "2                0                 0              0               1   \n",
       "3                0                 0              0               1   \n",
       "4                0                 0              1               0   \n",
       "\n",
       "   age_band_55<=  \n",
       "0              1  \n",
       "1              0  \n",
       "2              0  \n",
       "3              0  \n",
       "4              0  \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stuinfo_final.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "stuinfo_final_vars = stuinfo_final.columns.values.tolist()\n",
    "Y = ['final_result']\n",
    "X = [i for i in stuinfo_final_vars if i not in Y ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.linear_model import LogisticRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False  True False False False False False False False False False\n",
      " False False False False False False  True  True False  True  True  True\n",
      "  True  True  True  True  True  True  True  True  True False False False]\n",
      "[22  6  1 21  2 13 16  5 12 14 11 20  8 10  9 18 17 15  1  1  7  1  1  1\n",
      "  1  1  1  1  1  1  1  1  1 19  4  3]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "\n",
    "rfe = RFE(model, 15)\n",
    "rfe = rfe.fit(stuinfo_final[X], stuinfo_final[Y] )\n",
    "print(rfe.support_)\n",
    "print(rfe.ranking_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "select_cols = [col for i, col in enumerate(X) if rfe.support_[i] == True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['num_of_prev_attempts',\n",
       " 'highest_education_A Level or Equivalent',\n",
       " 'highest_education_HE Qualification',\n",
       " 'highest_education_No Formal quals',\n",
       " 'highest_education_Post Graduate Qualification',\n",
       " 'imd_band_0-10%',\n",
       " 'imd_band_10-20',\n",
       " 'imd_band_20-30%',\n",
       " 'imd_band_30-40%',\n",
       " 'imd_band_40-50%',\n",
       " 'imd_band_50-60%',\n",
       " 'imd_band_60-70%',\n",
       " 'imd_band_70-80%',\n",
       " 'imd_band_80-90%',\n",
       " 'imd_band_90-100%']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "select_cols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "Y = stuinfo_final['final_result']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stuinfo_final.drop(['final_result'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Maximum number of iterations has been exceeded.\n",
      "         Current function value: 0.654991\n",
      "         Iterations: 35\n",
      "                           Logit Regression Results                           \n",
      "==============================================================================\n",
      "Dep. Variable:           final_result   No. Observations:                32593\n",
      "Model:                          Logit   Df Residuals:                    32559\n",
      "Method:                           MLE   Df Model:                           33\n",
      "Date:                Tue, 05 Nov 2019   Pseudo R-squ.:                 0.05291\n",
      "Time:                        18:22:11   Log-Likelihood:                -21348.\n",
      "converged:                      False   LL-Null:                       -22541.\n",
      "Covariance Type:            nonrobust   LLR p-value:                     0.000\n",
      "=================================================================================================================\n",
      "                                                    coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------------------------------------\n",
      "id_student                                     2.738e-08   2.15e-08      1.272      0.203   -1.48e-08    6.96e-08\n",
      "gender                                           -0.1714      0.023     -7.315      0.000      -0.217      -0.125\n",
      "num_of_prev_attempts                             -0.3646      0.027    -13.407      0.000      -0.418      -0.311\n",
      "studied_credits                                  -0.0065      0.000    -21.041      0.000      -0.007      -0.006\n",
      "disability                                       -0.2616      0.040     -6.496      0.000      -0.340      -0.183\n",
      "region_East Anglian Region                        0.2035   1.45e+06   1.41e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_East Midlands Region                       0.1620   1.45e+06   1.12e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_Ireland                                    0.3249   1.45e+06   2.24e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_London Region                              0.0773   1.45e+06   5.34e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_North Region                               0.0824   1.45e+06   5.69e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_North Western Region                       0.0357   1.45e+06   2.46e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_Scotland                                   0.1199   1.45e+06   8.28e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_South East Region                          0.3112   1.45e+06   2.15e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_South Region                               0.2580   1.45e+06   1.78e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_South West Region                          0.2855   1.45e+06   1.97e-07      1.000   -2.84e+06    2.84e+06\n",
      "region_Wales                                      0.1406   1.45e+06   9.71e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_West Midlands Region                       0.1251   1.45e+06   8.64e-08      1.000   -2.84e+06    2.84e+06\n",
      "region_Yorkshire Region                           0.1687   1.45e+06   1.16e-07      1.000   -2.84e+06    2.84e+06\n",
      "highest_education_A Level or Equivalent           0.7105   1.09e+06   6.52e-07      1.000   -2.14e+06    2.14e+06\n",
      "highest_education_HE Qualification                0.7789   1.09e+06   7.14e-07      1.000   -2.14e+06    2.14e+06\n",
      "highest_education_Lower Than A Level              0.1583   1.09e+06   1.45e-07      1.000   -2.14e+06    2.14e+06\n",
      "highest_education_No Formal quals                -0.1272   1.09e+06  -1.17e-07      1.000   -2.14e+06    2.14e+06\n",
      "highest_education_Post Graduate Qualification     0.7744   1.09e+06    7.1e-07      1.000   -2.14e+06    2.14e+06\n",
      "imd_band_0-10%                                   -1.1526      0.087    -13.247      0.000      -1.323      -0.982\n",
      "imd_band_10-20                                   -1.0491      0.086    -12.244      0.000      -1.217      -0.881\n",
      "imd_band_20-30%                                  -0.9732      0.086    -11.313      0.000      -1.142      -0.805\n",
      "imd_band_30-40%                                  -0.7627      0.086     -8.850      0.000      -0.932      -0.594\n",
      "imd_band_40-50%                                  -0.8032      0.087     -9.244      0.000      -0.974      -0.633\n",
      "imd_band_50-60%                                  -0.7369      0.087     -8.441      0.000      -0.908      -0.566\n",
      "imd_band_60-70%                                  -0.6259      0.088     -7.125      0.000      -0.798      -0.454\n",
      "imd_band_70-80%                                  -0.6566      0.088     -7.477      0.000      -0.829      -0.484\n",
      "imd_band_80-90%                                  -0.5499      0.088     -6.231      0.000      -0.723      -0.377\n",
      "imd_band_90-100%                                 -0.4543      0.090     -5.063      0.000      -0.630      -0.278\n",
      "age_band_0-35                                     0.6010        nan        nan        nan         nan         nan\n",
      "age_band_35-55                                    0.8000        nan        nan        nan         nan         nan\n",
      "age_band_55<=                                     0.8938        nan        nan        nan         nan         nan\n",
      "=================================================================================================================\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/statsmodels/base/model.py:568: ConvergenceWarning: Maximum Likelihood optimization failed to converge. Check mle_retvals\n",
      "  \"Check mle_retvals\", ConvergenceWarning)\n",
      "//anaconda3/lib/python3.7/site-packages/statsmodels/base/model.py:1357: RuntimeWarning: invalid value encountered in sqrt\n",
      "  bse_ = np.sqrt(np.diag(self.cov_params()))\n",
      "//anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py:901: RuntimeWarning: invalid value encountered in greater\n",
      "  return (a < x) & (x < b)\n",
      "//anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py:901: RuntimeWarning: invalid value encountered in less\n",
      "  return (a < x) & (x < b)\n",
      "//anaconda3/lib/python3.7/site-packages/scipy/stats/_distn_infrastructure.py:1892: RuntimeWarning: invalid value encountered in less_equal\n",
      "  cond2 = cond0 & (x <= _a)\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "logit_model=sm.Logit(Y,X)\n",
    "result=logit_model.fit()\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "clf = linear_model.LogisticRegression()\n",
    "clf.fit(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5427238977694597"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X,Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### With Select Features Only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = stuinfo_final[select_cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.666800\n",
      "         Iterations 5\n",
      "                           Logit Regression Results                           \n",
      "==============================================================================\n",
      "Dep. Variable:           final_result   No. Observations:                32593\n",
      "Model:                          Logit   Df Residuals:                    32578\n",
      "Method:                           MLE   Df Model:                           14\n",
      "Date:                Thu, 07 Nov 2019   Pseudo R-squ.:                 0.03583\n",
      "Time:                        09:30:48   Log-Likelihood:                -21733.\n",
      "converged:                       True   LL-Null:                       -22541.\n",
      "Covariance Type:            nonrobust   LLR p-value:                     0.000\n",
      "=================================================================================================================\n",
      "                                                    coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------------------------------------\n",
      "num_of_prev_attempts                             -0.4542      0.027    -16.931      0.000      -0.507      -0.402\n",
      "highest_education_A Level or Equivalent           0.5288      0.025     21.489      0.000       0.481       0.577\n",
      "highest_education_HE Qualification                0.6495      0.034     18.956      0.000       0.582       0.717\n",
      "highest_education_No Formal quals                -0.2609      0.120     -2.170      0.030      -0.497      -0.025\n",
      "highest_education_Post Graduate Qualification     0.8457      0.121      6.993      0.000       0.609       1.083\n",
      "imd_band_0-10%                                   -0.8101      0.039    -20.614      0.000      -0.887      -0.733\n",
      "imd_band_10-20                                   -0.6838      0.038    -18.152      0.000      -0.758      -0.610\n",
      "imd_band_20-30%                                  -0.6110      0.037    -16.555      0.000      -0.683      -0.539\n",
      "imd_band_30-40%                                  -0.3766      0.037    -10.155      0.000      -0.449      -0.304\n",
      "imd_band_40-50%                                  -0.4042      0.038    -10.503      0.000      -0.480      -0.329\n",
      "imd_band_50-60%                                  -0.3116      0.039     -7.989      0.000      -0.388      -0.235\n",
      "imd_band_60-70%                                  -0.1940      0.040     -4.796      0.000      -0.273      -0.115\n",
      "imd_band_70-80%                                  -0.2286      0.041     -5.639      0.000      -0.308      -0.149\n",
      "imd_band_80-90%                                  -0.1288      0.042     -3.099      0.002      -0.210      -0.047\n",
      "imd_band_90-100%                                 -0.0257      0.044     -0.587      0.557      -0.111       0.060\n",
      "=================================================================================================================\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "logit_model=sm.Logit(Y,X)\n",
    "result=logit_model.fit()\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "//anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn import linear_model\n",
    "clf = linear_model.LogisticRegression()\n",
    "clf.fit(X, Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5960175497806277"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf.score(X,Y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}