11_PovertyWealthHeatlh.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Hidden\n",
    "from datascience import Table\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "plt.style.use('fivethirtyeight')\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "## Just for reference, since datasience is evolving rapidly\n",
    "import datascience as ds\n",
    "ds.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothesis Testing and Thinking about Causation\n",
    "\n",
    "As we begin to speak about hypothesis, one question that comes up concerns causation. Today, we will work a bit with hypothesis testing and then we will begin to consider causation. We will use out elderly poor \"sample\" for hypothesis testing, and we will look the relationship between life expectancy and wealth in a similar \"sample.\" \n",
    "\n",
    "/I/ Permutation test\n",
    "\n",
    "/II/ A-B test\n",
    "\n",
    "/III/ Bivariate relationships"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## /I/ Permutation Test and the Poverty Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last time, we calculated a 95% CI for poverty rates among those identified as black and white in Census Bureau statistics. The procedure qualifies as one traditional way to test a hypothesis. This time, we want to do a permutation test on the small data-set we looked at last time, namely poverty rates between elderly people identified as belonging to different races\n",
    "\n",
    "First, take a look at what the data looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Hidden\n",
    "pov_long = Table.read_table(\"us65pov_long.csv\")\n",
    "pov_long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall from last time that we have population level data, BUT that we are treating it as a random sample for 36 consecutive years. Recall also that we remarked that stark differences appeared to exist in poverty rates, as suggested by several visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#How many years are we calculating?\n",
    "pov_long.where(pov_long[\"category\"],\"all\").num_rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Histogram of poverty rate of white people, several options\n",
    "pov_long.select(\"65pc\").where(pov_long[\"category\"], \"white\").hist(bins=np.arange(0, 60, 1))\n",
    "plt.xlabel(\"poverty rate of white\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Histogram of poverty rate of black people\n",
    "pov_long.select(\"65pc\").where(pov_long[\"category\"], \"black\").hist(bins=36, normed=True)\n",
    "plt.xlabel(\"poverty rate of black\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The two histograms suggested there is a difference between poverty rates. Is there really?\n",
    "\n",
    "Let us do a permutation test to help us with our hypothesis, and recall that we need several components in a hypothesis test:\n",
    "\n",
    "**Null Hypothesis**\n",
    "\n",
    "**Alternate Hypothesis**\n",
    "\n",
    "**Test Statistic**\n",
    "\n",
    "**P-Value**\n",
    "\n",
    "First, assume we have the null hypothesis: \n",
    "Between the 1960s and 2000s, there was no difference between the poverty rates (65pc) typical for elderly white and black \"samples\" in the US.\n",
    "\n",
    "Second, the alternative hypothesis:\n",
    "There is a difference between the typical poverty rates (65pc) of white and black elderly \"samples\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Permutation test for poverty rate between white and black people\n",
    "\n",
    "meanW = np.mean(pov_long.where(pov_long[\"category\"], \"white\")[\"65pc\"])\n",
    "meanB = np.mean(pov_long.where(pov_long[\"category\"], \"black\")[\"65pc\"])\n",
    "obs_diff = meanB - meanW\n",
    "\n",
    "repetitions=10000\n",
    "diffs = []\n",
    "\n",
    "for i in range(repetitions):\n",
    "    \n",
    "    # sample WITHOUT replacement, same number as original sample size\n",
    "    resample = pov_long.sample()\n",
    "    \n",
    "    # Compute the difference of the means of the resampled values, between elderly black and white\n",
    "    dd = np.mean(resample.where(pov_long[\"category\"], \"black\")[\"65pc\"]) \\\n",
    "    - np.mean(resample.where(pov_long[\"category\"], \"white\")[\"65pc\"])\n",
    "    diffs.append([dd])\n",
    "\n",
    "# Display results   \n",
    "diffs = Table([diffs],['diff_in_means'])\n",
    "diffs.hist(bins=20, normed=True)\n",
    "plots.xlabel('Approx null distribution of difference in means')\n",
    "plots.title('Permutation Test')\n",
    "print('Observed difference in means: ', obs_diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The figure shows that under the null hypothesis of equal distributions in the \"samples,\" the empirical distribution of the difference between the sample means of the two groups is roughly bell shaped, centered at 0, stretching from about −8 percent to 8 percent. \n",
    "\n",
    "The **observed difference** in the original sample is about 21.4 percent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn\n",
    "\n",
    "Write out a conclusion; would you incorporate a 95% CI? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## /II/ AB Test and the Poverty Rate\n",
    "\n",
    "Now, let us analyze the poverty rate between the white elderly \"sample\" and non-white elderly (including those identified as black, Asian and Hispanic). We will be using AB Testing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, do a little modification to the table where in category, white is 1, non-white is 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Since we only need white and non-white category, remove \"all\"\n",
    "pov_long_new = pov_long.where(pov_long[\"category\"]!=\"all\")\n",
    "new_races = []\n",
    "for i in pov_long_new[\"category\"]:\n",
    "    if (i == \"white\"):\n",
    "        new_races.append(1)\n",
    "    else:\n",
    "        new_races.append(0)\n",
    "pov_long_new.append_column(\"whiteIndicator\", new_races)\n",
    "pov_long_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us define a function _bootstrap_ab_test_ that performs A-B testing using the bootstrap method and the difference in sample means as the test statistic. The null hypothesis is that the two underlying distributions in the population are equal; the alternative is that they are not.\n",
    "\n",
    "The arguments of the function are:\n",
    "•\tthe name of the table that contains the data in the original sample\n",
    "\n",
    "•\tthe label of the column containing the response variable (that is, the variable whose distribution is of interest)\n",
    "\n",
    "•\tthe label of the column containing the code 0 for Category A and 1 for Category B\n",
    "\n",
    "•\tthe number of repetitions of the resampling procedure\n",
    "\n",
    "The function returns the observed difference in means, the bootstrap empirical distribution of the difference in means, and the bootstrap empirical P-value. Because the alternative simply says that the two underlying distributions are different, the P-value is computed as the proportion of sampled differences that are at least as large in absolute value as the absolute value of the observed difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"Bootstrap A/B test for the difference in the mean response\n",
    "Assumes A=0, B=1\"\"\"\n",
    "\n",
    "def bootstrap_AB_test(samp_table, response_label, ab_label, repetitions):\n",
    "    \n",
    "    # Sort the sample table according to the A/B column; \n",
    "    # then select only the column of effects.\n",
    "    response = samp_table.sort(ab_label).select(response_label)\n",
    "    \n",
    "    # Find the number of entries in Category A.\n",
    "    n_A = samp_table.where(samp_table[ab_label],0).num_rows\n",
    "      \n",
    "    # Calculate the observed value of the test statistic.\n",
    "    meanA = np.mean(response[response_label][:n_A])\n",
    "    meanB = np.mean(response[response_label][n_A:])\n",
    "    obs_diff = meanA - meanB\n",
    "    \n",
    "    # Run the bootstrap procedure and get a list of resampled differences in means\n",
    "    diffs = []\n",
    "    for i in range(repetitions):\n",
    "        resample = response.sample(with_replacement=True)\n",
    "        d = np.mean(resample[response_label][:n_A]) - np.mean(resample[response_label][n_A:])\n",
    "        diffs.append([d])\n",
    "    \n",
    "    # Compute the bootstrap empirical P-value\n",
    "    diff_array = np.array(diffs)\n",
    "    p_value = np.count_nonzero(abs(diff_array) >= abs(obs_diff))/repetitions\n",
    "    \n",
    "    # Display results\n",
    "    diffs = Table([diffs],['diff_in_means'])\n",
    "    diffs.hist(bins=20,normed=True)\n",
    "    plots.xlabel('Approx null distribution of difference in means')\n",
    "    plots.title('Bootstrap A-B Test')\n",
    "    print(\"Observed difference in means: \", obs_diff)\n",
    "    print(\"Bootstrap empirical P-value: \", p_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the function bootstrap_AB_test to compare the poverty rate of white and non-white. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn: \n",
    "\n",
    "From the result, we can we reject the null hypothesis? What conclusion can we come to about that poverty rate of white and non-white elderly \"samples\"?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# /III/ More than a Correlation? \n",
    "\n",
    "Raj Chetty provides compelling evidence that wealth and health are strongly associated, and the _JAMA_ paper stops short of invoking the \"Big-C\": causation. \n",
    "\n",
    "Next week, we will talk in the main class about interpreting regression in more detail, and from our reading today there are a few things to note:\n",
    "\n",
    "--Chetty and his co-authors confirm that there seems to be a very strong association between health and wealth\n",
    "(see the powerful visaualization at Gap Minder, Global Trends Wealth & Health of Nations)\n",
    "\n",
    "--The two-variable linear regression, very briefly: \n",
    "\n",
    "•\tA two-variable linear regression relates two variables by fitting a straight line to the data\n",
    "\n",
    "•\tThe model allows us to:\n",
    "\n",
    "/i/ quantify a theory about the relationship between two variables;\n",
    "\n",
    "/ii/ measure the statistical significance of that relationship; and\n",
    "\n",
    "/iii/ predict the value of the dependent variable for a given value of the independent variable.\n",
    "\n",
    "•\tNote: a “linear” model is an assumption that may not best represent the actual relationship\n",
    "\n",
    "--Deaton suggests that we need to be wary about over-interpreting this association; and here are some common biases:\n",
    "\n",
    "•\t**Omitted Variables Bias** – the relationship between two variables may be (at least partially) a result of a third variable that is related to both (and that is currently inside the error term)\n",
    "•\t**Selection Bias** – the process of selecting the sample or observations may select a non-random and non-representative sample\n",
    "•\t**Reverse Causality** – rather than X causing Y, Y may have a causal influence on X\n",
    "\n",
    "We will import some more software, and Ryan Edwards posed similar questions recently using data from the US."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "usagdple= Table.read_table (\"usagdple.csv\")\n",
    "usagdple"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with some visualizations.\n",
    "Create scatter plots of:\n",
    "\n",
    "--life expectancy at birth for both sexes, e0b, versus year\n",
    "\n",
    "--log real GDP per capita, logrgdppc, versus year\n",
    "\n",
    "--life expectancy at birth for both sexes, e0b, versus log real GDP per capita, logrgdppc "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "usagdple.scatter('year','logrgdppc', fit_line=True, s=40)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn: scatter plots of key variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a scatter of year vs. logrgdppc, and describe the trend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a scatter of logrgdppc vs. e0b, and describe the trend; does it look like anything you saw before?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#This code gets us to use Pandas\n",
    "\n",
    "usa = usagdple.to_df()\n",
    "type(usa)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Our x variable is log real GDP per capita, while 'ones' is the so-called constant term /we'll talk more next week!/\n",
    "x = usa[['ones','logrgdppc']]\n",
    "y = usa['e0b']\n",
    "multiple_regress = sm.OLS(y, x).fit()\n",
    "multiple_regress.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's discuss the results -- what looks familiar?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}