Hypothesis Testing Example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-22T18:23:37.242473Z",
     "start_time": "2018-05-22T18:23:33.036585Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from scipy.stats import ttest_ind"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing: Mean Housing Prices During 2008 Recession\n",
    "\n",
    "The purpose of this notebook is to look at how a town's mean housing cost is effected by having a substantial university population during a recession.\n",
    "\n",
    "Definitions:\n",
    "* A _quarter_ is a specific three month period, Q1 is January through March, Q2 is April through June, Q3 is July through September, Q4 is October through December.\n",
    "* A _recession_ is defined as starting with two consecutive quarters of GDP decline, and ending with two consecutive quarters of GDP growth.\n",
    "* A _recession bottom_ is the quarter within a recession which had the lowest GDP.\n",
    "* A _university town_ is a city which has a high percentage of university students compared to the total population of the city.\n",
    "\n",
    "**Hypothesis**: University towns have their mean housing prices less effected by the 2008 recession. Run a t-test to compare the ratio of the mean price of houses in university towns the quarter before the recession starts compared to the recession bottom. (`price_ratio=quarter_before_recession/recession_bottom`)\n",
    "\n",
    "The following data files are publicly available via the following sources:\n",
    "* From the [Zillow research data site](http://www.zillow.com/research/data/) there is housing data for the United States. In particular the datafile for [all homes at a city level](http://files.zillowstatic.com/research/public/City/City_Zhvi_AllHomes.csv), ```City_Zhvi_AllHomes.csv```, has median home sale prices at a fine grained level.\n",
    "* From the Wikipedia page on college towns is a list of [university towns in the United States](https://en.wikipedia.org/wiki/List_of_college_towns#College_towns_in_the_United_States) which has been copy and pasted into the file ```university_towns.txt```.\n",
    "* From Bureau of Economic Analysis, US Department of Commerce, the [GDP over time](http://www.bea.gov/national/index.htm#gdp) of the United States in current dollars (use the chained value in 2009 dollars), in quarterly intervals, in the file ```gdplev.xls```. For this project, I will only look at GDP data from the first quarter of 2000 onward.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This dictionary will be used to map state names to two letter acronyms\n",
    "states = {'OH': 'Ohio', 'KY': 'Kentucky', 'AS': 'American Samoa', 'NV': 'Nevada', 'WY': 'Wyoming', 'NA': 'National', 'AL': 'Alabama', 'MD': 'Maryland', 'AK': 'Alaska', 'UT': 'Utah', 'OR': 'Oregon', 'MT': 'Montana', 'IL': 'Illinois', 'TN': 'Tennessee', 'DC': 'District of Columbia', 'VT': 'Vermont', 'ID': 'Idaho', 'AR': 'Arkansas', 'ME': 'Maine', 'WA': 'Washington', 'HI': 'Hawaii', 'WI': 'Wisconsin', 'MI': 'Michigan', 'IN': 'Indiana', 'NJ': 'New Jersey', 'AZ': 'Arizona', 'GU': 'Guam', 'MS': 'Mississippi', 'PR': 'Puerto Rico', 'NC': 'North Carolina', 'TX': 'Texas', 'SD': 'South Dakota', 'MP': 'Northern Mariana Islands', 'IA': 'Iowa', 'MO': 'Missouri', 'CT': 'Connecticut', 'WV': 'West Virginia', 'SC': 'South Carolina', 'LA': 'Louisiana', 'KS': 'Kansas', 'NY': 'New York', 'NE': 'Nebraska', 'OK': 'Oklahoma', 'FL': 'Florida', 'CA': 'California', 'CO': 'Colorado', 'PA': 'Pennsylvania', 'DE': 'Delaware', 'NM': 'New Mexico', 'RI': 'Rhode Island', 'MN': 'Minnesota', 'VI': 'Virgin Islands', 'NH': 'New Hampshire', 'MA': 'Massachusetts', 'GA': 'Georgia', 'ND': 'North Dakota', 'VA': 'Virginia'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "umich_part_id": "021",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>RegionName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>Auburn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>Florence</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>Jacksonville</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>Livingston</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>Montevallo</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     State    RegionName\n",
       "0  Alabama        Auburn\n",
       "1  Alabama      Florence\n",
       "2  Alabama  Jacksonville\n",
       "3  Alabama    Livingston\n",
       "4  Alabama    Montevallo"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_list_of_university_towns():\n",
    "    '''Returns a DataFrame of towns and the states they are in from the \n",
    "    university_towns.txt list. The format of the DataFrame should be:\n",
    "    DataFrame( [ [\"Michigan\", \"Ann Arbor\"], [\"Michigan\", \"Yipsilanti\"] ], \n",
    "    columns=[\"State\", \"RegionName\"]  )\n",
    "    \n",
    "    The following cleaning needed to be done:\n",
    "\n",
    "    1. For \"State\" column, removing characters from \"[\" to the end.\n",
    "    2. For \"RegionName\" column, when applicable, removing every character from \" (\" to the end.\n",
    "    3. I may need to also remove newline character '\\n'. '''\n",
    "    datalst = []\n",
    "    state = None\n",
    "    \n",
    "    fh = open('university_towns.txt', 'r')\n",
    "    for line in fh:\n",
    "        if line.strip('\\n')[-6:] == '[edit]':\n",
    "            state = line.strip('\\n')[:-6]\n",
    "            continue\n",
    "        if '(' in line:\n",
    "            index = line.find('(')\n",
    "            temp_town = line[:index].strip()\n",
    "            datalst.append([state, temp_town])\n",
    "        else:\n",
    "            datalst.append([state, line.strip()])\n",
    "    df = pd.DataFrame(datalst, columns=[\"State\", \"RegionName\"])\n",
    "    return df\n",
    "get_list_of_university_towns().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "umich_part_id": "022",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2008q3'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_start():\n",
    "    '''Returns the year and quarter of the recession start time as a \n",
    "    string value in a format such as 2005q3\n",
    "    From Bureau of Economic Analysis, US Department of Commerce, the GDP over time of the United States in current dollars \n",
    "    (use the chained value in 2009 dollars), in quarterly intervals, in the file gdplev.xls. For this project, I will \n",
    "    only look at GDP data from the first quarter of 2000 onward.\n",
    "    '''\n",
    "    df = pd.read_excel('gdplev.xls', skiprows=219)\n",
    "    df.drop(['Unnamed: 0', 'Unnamed: 1', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 7', 12323.3], axis=1, inplace=True)\n",
    "    df.rename(columns={'1999q4': 'Quarter', 9926.1: 'GDP'}, inplace=True)\n",
    "    for i in range(2, len(df)):\n",
    "        if (df.iloc[i-2][1] > df.iloc[i-1][1]) and (df.iloc[i-1][1] > df.iloc[i][1]):\n",
    "            return df.iloc[i-2][0]\n",
    "get_recession_start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "umich_part_id": "023",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q4'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_end():\n",
    "    '''Returns the year and quarter of the recession end time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    df = pd.read_excel('gdplev.xls', skiprows=219)\n",
    "    df.drop(['Unnamed: 0', 'Unnamed: 1', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 7', 12323.3], axis=1, inplace=True)\n",
    "    df.rename(columns={'1999q4': 'Quarter', 9926.1: 'GDP'}, inplace=True)\n",
    "    strt = get_recession_start()\n",
    "    strt_index = df.loc[df['Quarter'] == strt].index.tolist()[0]\n",
    "    df = df.loc[strt_index:]\n",
    "    for i in range(2, len(df)):\n",
    "        if (df.iloc[i-2][1] < df.iloc[i-1][1]) and (df.iloc[i-1][1] < df.iloc[i][1]):\n",
    "            return df.iloc[i][0]\n",
    "get_recession_end()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "umich_part_id": "024",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2009q2'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_recession_bottom():\n",
    "    '''Returns the year and quarter of the recession bottom time as a \n",
    "    string value in a format such as 2005q3'''\n",
    "    df = pd.read_excel('gdplev.xls', skiprows=219)\n",
    "    df.drop(['Unnamed: 0', 'Unnamed: 1', 'Unnamed: 2', 'Unnamed: 3', 'Unnamed: 7', 12323.3], axis=1, inplace=True)\n",
    "    df.rename(columns={'1999q4': 'Quarter', 9926.1: 'GDP'}, inplace=True)\n",
    "    strt = get_recession_start()\n",
    "    strt_index = df.loc[df['Quarter'] == strt].index.tolist()[0]\n",
    "    end = get_recession_end()\n",
    "    end_index = df.loc[df['Quarter'] == end].index.tolist()[0]\n",
    "    df = df.loc[strt_index:end_index]\n",
    "    bottom = df['GDP'].min()\n",
    "    df =  df.loc[df['GDP'] == bottom].reset_index()\n",
    "    return df.iloc[0]['Quarter']\n",
    "get_recession_bottom()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true,
    "umich_part_id": "025",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>2000q1</th>\n",
       "      <th>2000q2</th>\n",
       "      <th>2000q3</th>\n",
       "      <th>2000q4</th>\n",
       "      <th>2001q1</th>\n",
       "      <th>2001q2</th>\n",
       "      <th>2001q3</th>\n",
       "      <th>2001q4</th>\n",
       "      <th>2002q1</th>\n",
       "      <th>2002q2</th>\n",
       "      <th>...</th>\n",
       "      <th>2014q2</th>\n",
       "      <th>2014q3</th>\n",
       "      <th>2014q4</th>\n",
       "      <th>2015q1</th>\n",
       "      <th>2015q2</th>\n",
       "      <th>2015q3</th>\n",
       "      <th>2015q4</th>\n",
       "      <th>2016q1</th>\n",
       "      <th>2016q2</th>\n",
       "      <th>2016q3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>State</th>\n",
       "      <th>RegionName</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>New York</th>\n",
       "      <th>New York</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>515466.666667</td>\n",
       "      <td>522800.000000</td>\n",
       "      <td>528066.666667</td>\n",
       "      <td>532266.666667</td>\n",
       "      <td>540800.000000</td>\n",
       "      <td>557200.000000</td>\n",
       "      <td>572833.333333</td>\n",
       "      <td>582866.666667</td>\n",
       "      <td>591633.333333</td>\n",
       "      <td>587200.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>California</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <td>207066.666667</td>\n",
       "      <td>214466.666667</td>\n",
       "      <td>220966.666667</td>\n",
       "      <td>226166.666667</td>\n",
       "      <td>233000.000000</td>\n",
       "      <td>239100.000000</td>\n",
       "      <td>245066.666667</td>\n",
       "      <td>253033.333333</td>\n",
       "      <td>261966.666667</td>\n",
       "      <td>272700.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>498033.333333</td>\n",
       "      <td>509066.666667</td>\n",
       "      <td>518866.666667</td>\n",
       "      <td>528800.000000</td>\n",
       "      <td>538166.666667</td>\n",
       "      <td>547266.666667</td>\n",
       "      <td>557733.333333</td>\n",
       "      <td>566033.333333</td>\n",
       "      <td>577466.666667</td>\n",
       "      <td>584050.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Illinois</th>\n",
       "      <th>Chicago</th>\n",
       "      <td>138400.000000</td>\n",
       "      <td>143633.333333</td>\n",
       "      <td>147866.666667</td>\n",
       "      <td>152133.333333</td>\n",
       "      <td>156933.333333</td>\n",
       "      <td>161800.000000</td>\n",
       "      <td>166400.000000</td>\n",
       "      <td>170433.333333</td>\n",
       "      <td>175500.000000</td>\n",
       "      <td>177566.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>192633.333333</td>\n",
       "      <td>195766.666667</td>\n",
       "      <td>201266.666667</td>\n",
       "      <td>201066.666667</td>\n",
       "      <td>206033.333333</td>\n",
       "      <td>208300.000000</td>\n",
       "      <td>207900.000000</td>\n",
       "      <td>206066.666667</td>\n",
       "      <td>208200.000000</td>\n",
       "      <td>212000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pennsylvania</th>\n",
       "      <th>Philadelphia</th>\n",
       "      <td>53000.000000</td>\n",
       "      <td>53633.333333</td>\n",
       "      <td>54133.333333</td>\n",
       "      <td>54700.000000</td>\n",
       "      <td>55333.333333</td>\n",
       "      <td>55533.333333</td>\n",
       "      <td>56266.666667</td>\n",
       "      <td>57533.333333</td>\n",
       "      <td>59133.333333</td>\n",
       "      <td>60733.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>113733.333333</td>\n",
       "      <td>115300.000000</td>\n",
       "      <td>115666.666667</td>\n",
       "      <td>116200.000000</td>\n",
       "      <td>117966.666667</td>\n",
       "      <td>121233.333333</td>\n",
       "      <td>122200.000000</td>\n",
       "      <td>123433.333333</td>\n",
       "      <td>126933.333333</td>\n",
       "      <td>128700.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Arizona</th>\n",
       "      <th>Phoenix</th>\n",
       "      <td>111833.333333</td>\n",
       "      <td>114366.666667</td>\n",
       "      <td>116000.000000</td>\n",
       "      <td>117400.000000</td>\n",
       "      <td>119600.000000</td>\n",
       "      <td>121566.666667</td>\n",
       "      <td>122700.000000</td>\n",
       "      <td>124300.000000</td>\n",
       "      <td>126533.333333</td>\n",
       "      <td>128366.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>164266.666667</td>\n",
       "      <td>165366.666667</td>\n",
       "      <td>168500.000000</td>\n",
       "      <td>171533.333333</td>\n",
       "      <td>174166.666667</td>\n",
       "      <td>179066.666667</td>\n",
       "      <td>183833.333333</td>\n",
       "      <td>187900.000000</td>\n",
       "      <td>191433.333333</td>\n",
       "      <td>195200.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 67 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  2000q1         2000q2         2000q3  \\\n",
       "State        RegionName                                                  \n",
       "New York     New York                NaN            NaN            NaN   \n",
       "California   Los Angeles   207066.666667  214466.666667  220966.666667   \n",
       "Illinois     Chicago       138400.000000  143633.333333  147866.666667   \n",
       "Pennsylvania Philadelphia   53000.000000   53633.333333   54133.333333   \n",
       "Arizona      Phoenix       111833.333333  114366.666667  116000.000000   \n",
       "\n",
       "                                  2000q4         2001q1         2001q2  \\\n",
       "State        RegionName                                                  \n",
       "New York     New York                NaN            NaN            NaN   \n",
       "California   Los Angeles   226166.666667  233000.000000  239100.000000   \n",
       "Illinois     Chicago       152133.333333  156933.333333  161800.000000   \n",
       "Pennsylvania Philadelphia   54700.000000   55333.333333   55533.333333   \n",
       "Arizona      Phoenix       117400.000000  119600.000000  121566.666667   \n",
       "\n",
       "                                  2001q3         2001q4         2002q1  \\\n",
       "State        RegionName                                                  \n",
       "New York     New York                NaN            NaN            NaN   \n",
       "California   Los Angeles   245066.666667  253033.333333  261966.666667   \n",
       "Illinois     Chicago       166400.000000  170433.333333  175500.000000   \n",
       "Pennsylvania Philadelphia   56266.666667   57533.333333   59133.333333   \n",
       "Arizona      Phoenix       122700.000000  124300.000000  126533.333333   \n",
       "\n",
       "                                  2002q2    ...            2014q2  \\\n",
       "State        RegionName                     ...                     \n",
       "New York     New York                NaN    ...     515466.666667   \n",
       "California   Los Angeles   272700.000000    ...     498033.333333   \n",
       "Illinois     Chicago       177566.666667    ...     192633.333333   \n",
       "Pennsylvania Philadelphia   60733.333333    ...     113733.333333   \n",
       "Arizona      Phoenix       128366.666667    ...     164266.666667   \n",
       "\n",
       "                                  2014q3         2014q4         2015q1  \\\n",
       "State        RegionName                                                  \n",
       "New York     New York      522800.000000  528066.666667  532266.666667   \n",
       "California   Los Angeles   509066.666667  518866.666667  528800.000000   \n",
       "Illinois     Chicago       195766.666667  201266.666667  201066.666667   \n",
       "Pennsylvania Philadelphia  115300.000000  115666.666667  116200.000000   \n",
       "Arizona      Phoenix       165366.666667  168500.000000  171533.333333   \n",
       "\n",
       "                                  2015q2         2015q3         2015q4  \\\n",
       "State        RegionName                                                  \n",
       "New York     New York      540800.000000  557200.000000  572833.333333   \n",
       "California   Los Angeles   538166.666667  547266.666667  557733.333333   \n",
       "Illinois     Chicago       206033.333333  208300.000000  207900.000000   \n",
       "Pennsylvania Philadelphia  117966.666667  121233.333333  122200.000000   \n",
       "Arizona      Phoenix       174166.666667  179066.666667  183833.333333   \n",
       "\n",
       "                                  2016q1         2016q2    2016q3  \n",
       "State        RegionName                                            \n",
       "New York     New York      582866.666667  591633.333333  587200.0  \n",
       "California   Los Angeles   566033.333333  577466.666667  584050.0  \n",
       "Illinois     Chicago       206066.666667  208200.000000  212000.0  \n",
       "Pennsylvania Philadelphia  123433.333333  126933.333333  128700.0  \n",
       "Arizona      Phoenix       187900.000000  191433.333333  195200.0  \n",
       "\n",
       "[5 rows x 67 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def convert_housing_data_to_quarters():\n",
    "    '''Converts the housing data to quarters and returns it as mean \n",
    "    values in a dataframe. This dataframe should be a dataframe with\n",
    "    columns for 2000q1 through 2016q3, and have a multi-index\n",
    "    in the shape of [\"State\",\"RegionName\"].\n",
    "    \n",
    "    Note: Quarters are defined in the project description, they are\n",
    "    not arbitrary three month periods.\n",
    "    \n",
    "    Q1 is January through March,\n",
    "    Q2 is April through June, \n",
    "    Q3 is July through September, \n",
    "    Q4 is October through December\n",
    "    '''\n",
    "    quarters = ['q1', 'q2', 'q3', 'q4']\n",
    "    years = list(range(2000,2017))\n",
    "    combined = []\n",
    "    for i in years:\n",
    "        for j in quarters:\n",
    "            combined.append((str(i)+j))\n",
    "    \n",
    "    df = pd.read_csv('City_Zhvi_AllHomes.csv')\n",
    "    df.drop(['RegionID', 'Metro', 'CountyName', 'SizeRank'], axis=1, inplace=True)\n",
    "    df['State'] = df['State'].map(states)\n",
    "    df.set_index(['State', 'RegionName'], inplace=True)\n",
    "    \n",
    "    columns = df.columns.tolist()\n",
    "    index = columns.index('2000-01') \n",
    "    columns = columns[0:index]\n",
    "    df.drop(columns, axis=1, inplace=True)\n",
    "    \n",
    "    columns_2_quarters = [list(df.columns)[x:x+3] for x in range(0, len(list(df.columns)), 3)]\n",
    "    \n",
    "    for x,y in zip(combined, columns_2_quarters):\n",
    "        df[x] = df[y].mean(axis=1)\n",
    "        df.drop(y, axis=1, inplace=True)\n",
    "    \n",
    "    return df\n",
    "convert_housing_data_to_quarters().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "umich_part_id": "026",
    "umich_partlist_id": "004"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ttest_indResult(statistic=3.5643958297363705, pvalue=0.00036641601595526971)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(True, 0.00036641601595526971, 'university town')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def run_ttest():\n",
    "    '''First creates new data showing the decline or growth of housing prices\n",
    "    between the recession start and the recession bottom. Then runs a ttest\n",
    "    comparing the university town values to the non-university towns values, \n",
    "    return whether the alternative hypothesis (that the two groups are the same)\n",
    "    is true or not as well as the p-value of the confidence.  \n",
    "    \n",
    "    Returns a tuple (different, p, better) where different=True if the t-test is\n",
    "    True at a p<0.01 (we reject the null hypothesis), or different=False if \n",
    "    otherwise (we cannot reject the null hypothesis). The variable p should\n",
    "    be equal to the exact p value returned from scipy.stats.ttest_ind(). The\n",
    "    value for better should be either \"university town\" or \"non-university town\"\n",
    "    depending on which has a lower mean price ratio (which is equivilent to a\n",
    "    reduced market loss).'''\n",
    "    start = get_recession_start()\n",
    "    end = get_recession_end()\n",
    "    bottom = get_recession_bottom()\n",
    "    uni_towns = set(get_list_of_university_towns()['RegionName'])\n",
    "    all_housing = convert_housing_data_to_quarters()\n",
    "    \n",
    "    ## housing data prep\n",
    "    col = list(all_housing.columns)\n",
    "    index_start = col.index(start)\n",
    "    index_end = col.index(end)\n",
    "    col = col[index_start:index_end+1]\n",
    "    all_housing = all_housing[col].reset_index()\n",
    "    all_housing[\"Ratio\"] = (all_housing['2008q3'] - all_housing['2009q2'])/ all_housing['2008q3']\n",
    "    \n",
    "    different = None\n",
    "    p = None\n",
    "    better = None\n",
    "    \n",
    "    ### Sets binary(categorical) variable for university town or not\n",
    "    unitwn = []\n",
    "    for index, row in all_housing.iterrows():\n",
    "        if row['RegionName'] in uni_towns:\n",
    "            unitwn.append(1)\n",
    "        else:\n",
    "            unitwn.append(0)\n",
    "    all_housing['UniTown'] = unitwn\n",
    "    \n",
    "    \n",
    "    uni = all_housing[all_housing['UniTown']==1]['Ratio'].dropna()\n",
    "    norm  = all_housing[all_housing['UniTown']==0]['Ratio'].dropna()\n",
    "    ### better\n",
    "    if norm.mean() < uni.mean():\n",
    "        better = 'non-university town'\n",
    "    else:\n",
    "        better = 'university town'\n",
    "        \n",
    "    ### p val\n",
    "    print(ttest_ind(norm, uni))\n",
    "    p = list(ttest_ind(norm, uni))[1]\n",
    "    if p <= 0.01:\n",
    "        different = True\n",
    "    else:\n",
    "        different = False\n",
    "    \n",
    "    \n",
    "    final = (different, p, better)\n",
    "    return final\n",
    "run_ttest()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-data-analysis",
   "graded_item_id": "Il9Fx",
   "launcher_item_id": "TeDW0",
   "part_id": "WGlun"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "umich": {
   "id": "Assignment 4",
   "version": "1.1"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}