HW2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 2: More Exploratory Data Analysis\n",
    "## Gene Expression Data and Election Polls \n",
    "\n",
    "Due: Thursday, October 2, 2014 11:59 PM\n",
    "\n",
    "<a href=https://raw.githubusercontent.com/cs109/2014/master/homework/HW2.ipynb download=HW2.ipynb> Download this assignment</a>\n",
    "\n",
    "#### Submission Instructions\n",
    "To submit your homework, create a folder named lastname_firstinitial_hw# and place your IPython notebooks, data files, and any other files in this folder. Your IPython Notebooks should be completely executed with the results visible in the notebook. We should not have to run any code. Compress the folder (please use .zip compression) and submit to the CS109 dropbox in the appropriate folder. If we cannot access your work because these directions are not followed correctly, we will not grade your work.\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "John Tukey wrote in [Exploratory Data Analysis, 1977](http://www.amazon.com/Exploratory-Data-Analysis-Wilder-Tukey/dp/0201076160/ref=pd_bbs_sr_2/103-4466654-5303007?ie=UTF8&s=books&qid=1189739816&sr=8-2): \"The greatest value of a picture is when it forces us to notice what we never expected to see.\" In this assignment we will continue using our exploratory data analysis tools, but apply it to new sets of data: [gene expression](http://en.wikipedia.org/wiki/Gene_expression) and polls from the [2012 Presidental Election](http://en.wikipedia.org/wiki/United_States_presidential_election,_2012) and from the [2014 Senate Midterm Elections](http://en.wikipedia.org/wiki/United_States_Senate_elections,_2014).   \n",
    "\n",
    "**First**: You will use exploratory data analysis and apply the [singular value decomposition](http://en.wikipedia.org/wiki/Singular_value_decomposition) (SVD) to a gene expression data matrix to determine if the the date that the gene expression samples are processed has large effect on the variability seen in the data.  \n",
    "\n",
    "**Second**: You will use the polls from the 2012 Presidential Elections to determine (1) Is there a pollster bias in presidential election polls? and (2) Is the average of polls better than just one poll?\n",
    "\n",
    "**Finally**: You will use the [HuffPost Pollster API](http://elections.huffingtonpost.com/pollster/api) to extract the polls for the current 2014 Senate Midterm Elections and provide a preliminary prediction of the result of each state.\n",
    "\n",
    "#### Data\n",
    "\n",
    "We will use the following data sets: \n",
    "\n",
    "1. A gene expression data set called `exprs_GSE5859.csv` and sample annotation table called `sampleinfo_GSE5859.csv` which are both available on Github in the 2014_data repository: [expression data set](https://github.com/cs109/2014_data/blob/master/exprs_GSE5859.csv) and [sample annotation table](https://github.com/cs109/2014_data/blob/master/sampleinfo_GSE5859.csv).  \n",
    "\n",
    "2. Polls from the [2012 Presidential Election: Barack Obama vs Mitt Romney](http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama). The polls we will use are from the [Huffington Post Pollster](http://elections.huffingtonpost.com/pollster).  \n",
    "\n",
    "3. Polls from the [2014 Senate Midterm Elections](http://elections.huffingtonpost.com/pollster) from the [HuffPost Pollster API](http://elections.huffingtonpost.com/pollster/api). \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Python modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# special IPython command to prepare the notebook for matplotlib\n",
    "%matplotlib inline \n",
    "import shutil \n",
    "import urllib.request  \n",
    "import requests \n",
    "import numpy as np\n",
    "import pandas as pd # pandas\n",
    "import matplotlib.pyplot as plt # module for plotting \n",
    "import datetime as dt # module for manipulating dates and times\n",
    "import numpy.linalg as lin # module for performing linear algebra operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 1\n",
    "\n",
    "In this problem we will be using a [gene expression](http://en.wikipedia.org/wiki/Gene_expression) data set obtained from a [microarray](http://en.wikipedia.org/wiki/DNA_microarray) experiement [Read more about the specific experiment here](http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE5859).  There are two data sets we will use:  \n",
    "\n",
    "1. The gene expression intensities where the rows represent the features on the microarray (e.g. genes) and the columsns represent the different microarray samples.  \n",
    "\n",
    "2. A table that contains the information about each of the samples (columns in the gene expression data set) such as the sex, the age, the treatment status, the date the samples were processed.  Each row represents one sample. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 1(a) \n",
    "Read in the two files from Github: [exprs_GSE5859.csv](https://github.com/cs109/2014_data/blob/master/exprs_GSE5859.csv) and [sampleinfo_GSE5859.csv](https://github.com/cs109/2014_data/blob/master/sampleinfo_GSE5859.csv) as pandas DataFrames called `exprs` and `sampleinfo`. Use the gene names as the index of the `exprs` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    ethnicity        date          filename sex\n",
      "0         CEU  2003-02-04   GSM25349.CEL.gz   M\n",
      "1         CEU  2003-02-04   GSM25350.CEL.gz   M\n",
      "2         CEU  2002-12-17   GSM25356.CEL.gz   M\n",
      "3         CEU  2003-01-30   GSM25357.CEL.gz   M\n",
      "4         CEU  2003-01-03   GSM25358.CEL.gz   M\n",
      "5         CEU  2003-01-16   GSM25359.CEL.gz   M\n",
      "6         CEU  2003-01-03   GSM25360.CEL.gz   F\n",
      "7         CEU  2003-02-04   GSM25361.CEL.gz   M\n",
      "8         CEU  2002-12-17   GSM25377.CEL.gz   M\n",
      "9         CEU  2002-12-17   GSM25378.CEL.gz   M\n",
      "10        CEU  2002-12-17   GSM25385.CEL.gz   M\n",
      "11        CEU  2003-01-03   GSM25386.CEL.gz   M\n",
      "12        CEU  2002-12-17   GSM25399.CEL.gz   F\n",
      "13        CEU  2002-12-17   GSM25400.CEL.gz   F\n",
      "14        CEU  2002-11-22   GSM25401.CEL.gz   M\n",
      "15        CEU  2002-12-06   GSM25402.CEL.gz   F\n",
      "16        CEU  2003-01-03   GSM25409.CEL.gz   M\n",
      "17        CEU  2003-02-21   GSM25410.CEL.gz   F\n",
      "18        CEU  2003-02-21   GSM25426.CEL.gz   M\n",
      "19        CEU  2002-12-19   GSM25427.CEL.gz   M\n",
      "20        CEU  2002-11-19   GSM25479.CEL.gz   M\n",
      "21        CEU  2002-11-21   GSM25480.CEL.gz   M\n",
      "22        CEU  2002-11-19   GSM25481.CEL.gz   F\n",
      "23        CEU  2002-11-21   GSM25482.CEL.gz   F\n",
      "24        CEU  2002-11-21   GSM25524.CEL.gz   M\n",
      "25        CEU  2002-11-27   GSM25525.CEL.gz   M\n",
      "26        CEU  2002-12-06   GSM25526.CEL.gz   M\n",
      "27        CEU  2002-12-06   GSM25527.CEL.gz   M\n",
      "28        CEU  2002-11-21   GSM25528.CEL.gz   M\n",
      "29        CEU  2002-11-27   GSM25529.CEL.gz   M\n",
      "..        ...         ...               ...  ..\n",
      "178       ASN  2005-12-21  GSM136672.CEL.gz   F\n",
      "179       ASN  2005-12-21  GSM136673.CEL.gz   M\n",
      "180       HAN  2006-04-28  GSM136674.CEL.gz   F\n",
      "181       HAN  2006-04-28  GSM136675.CEL.gz   F\n",
      "182       HAN  2006-04-28  GSM136676.CEL.gz   F\n",
      "183       HAN  2006-04-28  GSM136677.CEL.gz   M\n",
      "184       HAN  2006-04-28  GSM136705.CEL.gz   M\n",
      "185       HAN  2006-04-28  GSM136706.CEL.gz   F\n",
      "186       HAN  2006-04-28  GSM136707.CEL.gz   F\n",
      "187       HAN  2006-04-28  GSM136708.CEL.gz   F\n",
      "188       HAN  2006-04-28  GSM136709.CEL.gz   M\n",
      "189       HAN  2006-04-28  GSM136710.CEL.gz   F\n",
      "190       HAN  2006-04-28  GSM136711.CEL.gz   M\n",
      "191       HAN  2006-04-28  GSM136712.CEL.gz   F\n",
      "192       HAN  2006-04-28  GSM136713.CEL.gz   F\n",
      "193       HAN  2006-04-28  GSM136714.CEL.gz   M\n",
      "194       HAN  2006-04-28  GSM136715.CEL.gz   M\n",
      "195       HAN  2006-04-28  GSM136716.CEL.gz   M\n",
      "196       HAN  2006-04-28  GSM136717.CEL.gz   F\n",
      "197       HAN  2006-04-28  GSM136718.CEL.gz   M\n",
      "198       HAN  2006-04-28  GSM136719.CEL.gz   F\n",
      "199       HAN  2006-04-28  GSM136720.CEL.gz   M\n",
      "200       HAN  2006-04-28  GSM136721.CEL.gz   F\n",
      "201       HAN  2006-04-28  GSM136722.CEL.gz   F\n",
      "202       HAN  2006-04-28  GSM136723.CEL.gz   M\n",
      "203       HAN  2006-04-28  GSM136724.CEL.gz   M\n",
      "204       CEU  2005-02-16  GSM136725.CEL.gz   F\n",
      "205       CEU  2005-02-16  GSM136726.CEL.gz   M\n",
      "206       CEU  2005-06-10  GSM136727.CEL.gz   M\n",
      "207       ASN  2005-05-13  GSM136729.CEL.gz   M\n",
      "\n",
      "[208 rows x 4 columns]\n"
     ]
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "#url_exprs = urllib.request.urlopen(\"https://raw.githubusercontent.com/cs109/2014_data/master/exprs_GSE5859.csv\")\n",
    "url2_sample = urllib.request.urlopen(\"https://raw.githubusercontent.com/cs109/2014_data/master/sampleinfo_GSE5859.csv\")\n",
    "\n",
    "#exprs_pd=pd.read_csv(url_exprs,index_col=0)\n",
    "\n",
    "\n",
    "sampleinfo_pd=pd.read_csv(url2_sample )\n",
    "print(sampleinfo_pd)\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure the order of the columns in the gene expression DataFrame match the order of file names in the sample annotation DataFrame. If the order of the columns the `exprs` DataFrame do not match the order of the file names in the `sampleinfo` DataFrame, reorder the columns in the `exprs` DataFrame. \n",
    "\n",
    "**Note**: The column names of the gene expression DataFrame are the filenames of the orignal files from which these data were obtained.  \n",
    "\n",
    "**Hint**: The method `list.index(x)` [[read here](https://docs.python.org/2/tutorial/datastructures.html)] can be used to return the index in the list of the first item whose value is x. It is an error if there is no such item. To check if the order of the columns in `exprs` matches the order of the rows in `sampleinfo`, you can check using the method `.all()` on a Boolean or list of Booleans: \n",
    "\n",
    "Example code: `(exprs.columns == sampleinfo.filename).all()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "#Number of rows in sampleinfo is same as number of columns \n",
    "#print(sampleinfo_pd.filename)\n",
    "\n",
    "#your code here\n",
    "temp_list_a = list(sampleinfo_pd.filename)\n",
    "temp_list_a=np.array(temp_list_a) \n",
    "temp_list_a=temp_list_a.transpose()\n",
    "exprs_pd = exprs_pd.reindex_axis(temp_list_a,axis=1)\n",
    "#print(exprs_pd1)\n",
    "# check if all the column names match the file names in sampleinfo\n",
    "(exprs_pd.columns == sampleinfo_pd.filename).all()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the head of the two tables: `exprs` and `sampleinfo`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GSM25349.CEL.gz</th>\n",
       "      <th>GSM25350.CEL.gz</th>\n",
       "      <th>GSM25356.CEL.gz</th>\n",
       "      <th>GSM25357.CEL.gz</th>\n",
       "      <th>GSM25358.CEL.gz</th>\n",
       "      <th>GSM25359.CEL.gz</th>\n",
       "      <th>GSM25360.CEL.gz</th>\n",
       "      <th>GSM25361.CEL.gz</th>\n",
       "      <th>GSM25377.CEL.gz</th>\n",
       "      <th>GSM25378.CEL.gz</th>\n",
       "      <th>...</th>\n",
       "      <th>GSM136719.CEL.gz</th>\n",
       "      <th>GSM136720.CEL.gz</th>\n",
       "      <th>GSM136721.CEL.gz</th>\n",
       "      <th>GSM136722.CEL.gz</th>\n",
       "      <th>GSM136723.CEL.gz</th>\n",
       "      <th>GSM136724.CEL.gz</th>\n",
       "      <th>GSM136725.CEL.gz</th>\n",
       "      <th>GSM136726.CEL.gz</th>\n",
       "      <th>GSM136727.CEL.gz</th>\n",
       "      <th>GSM136729.CEL.gz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1007_s_at</th>\n",
       "      <td>6.627014</td>\n",
       "      <td>6.249807</td>\n",
       "      <td>5.934128</td>\n",
       "      <td>6.145268</td>\n",
       "      <td>6.091270</td>\n",
       "      <td>6.041186</td>\n",
       "      <td>6.050375</td>\n",
       "      <td>6.063847</td>\n",
       "      <td>6.226106</td>\n",
       "      <td>6.358282</td>\n",
       "      <td>...</td>\n",
       "      <td>6.466445</td>\n",
       "      <td>6.533592</td>\n",
       "      <td>6.631492</td>\n",
       "      <td>6.513362</td>\n",
       "      <td>6.440706</td>\n",
       "      <td>6.704324</td>\n",
       "      <td>6.488579</td>\n",
       "      <td>6.809481</td>\n",
       "      <td>6.133068</td>\n",
       "      <td>6.155473</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1053_at</th>\n",
       "      <td>6.939184</td>\n",
       "      <td>6.818038</td>\n",
       "      <td>7.047962</td>\n",
       "      <td>7.422477</td>\n",
       "      <td>7.272361</td>\n",
       "      <td>7.128216</td>\n",
       "      <td>6.750719</td>\n",
       "      <td>6.836287</td>\n",
       "      <td>6.738022</td>\n",
       "      <td>7.367895</td>\n",
       "      <td>...</td>\n",
       "      <td>7.032885</td>\n",
       "      <td>7.156344</td>\n",
       "      <td>7.018025</td>\n",
       "      <td>6.973322</td>\n",
       "      <td>6.884738</td>\n",
       "      <td>7.120898</td>\n",
       "      <td>7.517410</td>\n",
       "      <td>7.200596</td>\n",
       "      <td>7.280781</td>\n",
       "      <td>7.111583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117_at</th>\n",
       "      <td>5.113570</td>\n",
       "      <td>5.074096</td>\n",
       "      <td>5.371201</td>\n",
       "      <td>5.266550</td>\n",
       "      <td>5.342047</td>\n",
       "      <td>5.063876</td>\n",
       "      <td>5.315898</td>\n",
       "      <td>5.483652</td>\n",
       "      <td>6.689444</td>\n",
       "      <td>6.482782</td>\n",
       "      <td>...</td>\n",
       "      <td>5.661112</td>\n",
       "      <td>5.127260</td>\n",
       "      <td>5.151840</td>\n",
       "      <td>5.505602</td>\n",
       "      <td>5.687689</td>\n",
       "      <td>4.942651</td>\n",
       "      <td>5.247190</td>\n",
       "      <td>5.237239</td>\n",
       "      <td>5.401876</td>\n",
       "      <td>5.302628</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121_at</th>\n",
       "      <td>7.833862</td>\n",
       "      <td>7.780682</td>\n",
       "      <td>7.458197</td>\n",
       "      <td>7.655948</td>\n",
       "      <td>7.546555</td>\n",
       "      <td>7.072670</td>\n",
       "      <td>7.092984</td>\n",
       "      <td>6.954225</td>\n",
       "      <td>7.489785</td>\n",
       "      <td>7.388539</td>\n",
       "      <td>...</td>\n",
       "      <td>7.769734</td>\n",
       "      <td>7.815864</td>\n",
       "      <td>7.683279</td>\n",
       "      <td>7.883231</td>\n",
       "      <td>7.913621</td>\n",
       "      <td>7.834196</td>\n",
       "      <td>7.331864</td>\n",
       "      <td>7.357102</td>\n",
       "      <td>7.607461</td>\n",
       "      <td>7.456453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1255_g_at</th>\n",
       "      <td>3.152269</td>\n",
       "      <td>3.111747</td>\n",
       "      <td>3.018932</td>\n",
       "      <td>3.154545</td>\n",
       "      <td>3.107954</td>\n",
       "      <td>3.224284</td>\n",
       "      <td>3.114241</td>\n",
       "      <td>3.044975</td>\n",
       "      <td>3.304038</td>\n",
       "      <td>2.887919</td>\n",
       "      <td>...</td>\n",
       "      <td>3.257484</td>\n",
       "      <td>3.339234</td>\n",
       "      <td>3.298384</td>\n",
       "      <td>3.150654</td>\n",
       "      <td>3.344501</td>\n",
       "      <td>3.230285</td>\n",
       "      <td>3.175846</td>\n",
       "      <td>3.105092</td>\n",
       "      <td>3.225123</td>\n",
       "      <td>3.090149</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 208 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           GSM25349.CEL.gz  GSM25350.CEL.gz  GSM25356.CEL.gz  GSM25357.CEL.gz  \\\n",
       "1007_s_at         6.627014         6.249807         5.934128         6.145268   \n",
       "1053_at           6.939184         6.818038         7.047962         7.422477   \n",
       "117_at            5.113570         5.074096         5.371201         5.266550   \n",
       "121_at            7.833862         7.780682         7.458197         7.655948   \n",
       "1255_g_at         3.152269         3.111747         3.018932         3.154545   \n",
       "\n",
       "           GSM25358.CEL.gz  GSM25359.CEL.gz  GSM25360.CEL.gz  GSM25361.CEL.gz  \\\n",
       "1007_s_at         6.091270         6.041186         6.050375         6.063847   \n",
       "1053_at           7.272361         7.128216         6.750719         6.836287   \n",
       "117_at            5.342047         5.063876         5.315898         5.483652   \n",
       "121_at            7.546555         7.072670         7.092984         6.954225   \n",
       "1255_g_at         3.107954         3.224284         3.114241         3.044975   \n",
       "\n",
       "           GSM25377.CEL.gz  GSM25378.CEL.gz        ...         \\\n",
       "1007_s_at         6.226106         6.358282        ...          \n",
       "1053_at           6.738022         7.367895        ...          \n",
       "117_at            6.689444         6.482782        ...          \n",
       "121_at            7.489785         7.388539        ...          \n",
       "1255_g_at         3.304038         2.887919        ...          \n",
       "\n",
       "           GSM136719.CEL.gz  GSM136720.CEL.gz  GSM136721.CEL.gz  \\\n",
       "1007_s_at          6.466445          6.533592          6.631492   \n",
       "1053_at            7.032885          7.156344          7.018025   \n",
       "117_at             5.661112          5.127260          5.151840   \n",
       "121_at             7.769734          7.815864          7.683279   \n",
       "1255_g_at          3.257484          3.339234          3.298384   \n",
       "\n",
       "           GSM136722.CEL.gz  GSM136723.CEL.gz  GSM136724.CEL.gz  \\\n",
       "1007_s_at          6.513362          6.440706          6.704324   \n",
       "1053_at            6.973322          6.884738          7.120898   \n",
       "117_at             5.505602          5.687689          4.942651   \n",
       "121_at             7.883231          7.913621          7.834196   \n",
       "1255_g_at          3.150654          3.344501          3.230285   \n",
       "\n",
       "           GSM136725.CEL.gz  GSM136726.CEL.gz  GSM136727.CEL.gz  \\\n",
       "1007_s_at          6.488579          6.809481          6.133068   \n",
       "1053_at            7.517410          7.200596          7.280781   \n",
       "117_at             5.247190          5.237239          5.401876   \n",
       "121_at             7.331864          7.357102          7.607461   \n",
       "1255_g_at          3.175846          3.105092          3.225123   \n",
       "\n",
       "           GSM136729.CEL.gz  \n",
       "1007_s_at          6.155473  \n",
       "1053_at            7.111583  \n",
       "117_at             5.302628  \n",
       "121_at             7.456453  \n",
       "1255_g_at          3.090149  \n",
       "\n",
       "[5 rows x 208 columns]"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "exprs_pd.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>date</th>\n",
       "      <th>filename</th>\n",
       "      <th>sex</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25349.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25350.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2002-12-17</td>\n",
       "      <td>GSM25356.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-30</td>\n",
       "      <td>GSM25357.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-03</td>\n",
       "      <td>GSM25358.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ethnicity        date         filename sex\n",
       "0       CEU  2003-02-04  GSM25349.CEL.gz   M\n",
       "1       CEU  2003-02-04  GSM25350.CEL.gz   M\n",
       "2       CEU  2002-12-17  GSM25356.CEL.gz   M\n",
       "3       CEU  2003-01-30  GSM25357.CEL.gz   M\n",
       "4       CEU  2003-01-03  GSM25358.CEL.gz   M"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampleinfo_pd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 1(b)\n",
    "\n",
    "Extract the year and month as integers from the `sampleinfo` table. \n",
    "\n",
    "**Hint**: To convert a Series or a column of a pandas DataFrame that contains a date-like object, you can use the `to_datetime` function [[read here](http://pandas.pydata.org/pandas-docs/stable/timeseries.html)].  This will create a `DatetimeIndex` which can be used to extract the month and year for each row in the DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>date</th>\n",
       "      <th>filename</th>\n",
       "      <th>sex</th>\n",
       "      <th>elapsedInDays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25349.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25350.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2002-12-17</td>\n",
       "      <td>GSM25356.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-30</td>\n",
       "      <td>GSM25357.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-03</td>\n",
       "      <td>GSM25358.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ethnicity       date         filename sex  elapsedInDays\n",
       "0       CEU 2003-02-04  GSM25349.CEL.gz   M             96\n",
       "1       CEU 2003-02-04  GSM25350.CEL.gz   M             96\n",
       "2       CEU 2002-12-17  GSM25356.CEL.gz   M             47\n",
       "3       CEU 2003-01-30  GSM25357.CEL.gz   M             91\n",
       "4       CEU 2003-01-03  GSM25358.CEL.gz   M             64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "\n",
    "sampleinfo_pd[\"date\"]= pd.to_datetime(sampleinfo_pd[\"date\"],yearfirst=True)\n",
    "basedate=pd.Timestamp('2002-10-31')\n",
    "sampleinfo_pd[\"elapsedInDays\"] = sampleinfo_pd.date.apply(lambda x: (x - basedate).days)\n",
    "sampleinfo_pd.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 1(c)\n",
    "\n",
    "Convert the dates in the `date` column from the `sampleinfo` table into days since October 31, 2002. Add a column to the `sampleinfo` DataFrame titled `elapsedInDays` containing the days since October 31, 2002.  Show the head of the `sampleinfo` DataFrame which includes the new column.  \n",
    "\n",
    "**Hint**: Use the `datetime` module to create a new `datetime` object for the specific date October 31, 2002. Then, subtract the October 31, 2002 date from each date from the `date` column in the `sampleinfo` DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here\n",
    "\n",
    "\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 1(d)\n",
    "\n",
    "Use exploratory analysis and the singular value decomposition (SVD) of the gene expression data matrix to determine if the date the samples were processed has large effect on the variability seen in the data or if it is just ethnicity (which is confounded with date). \n",
    "\n",
    "**Hint**: See the end of the [lecture from 9/23/2014 for help with SVD](http://nbviewer.ipython.org/github/cs109/2014/blob/master/lectures/lecture07/data_scraping_transcript.ipynb). \n",
    "\n",
    "First subset the the `sampleinfo` DataFrame to include only the CEU ethnicity.  Call this new subsetted DataFrame `sampleinfoCEU`.  Show the head of `sampleinfoCEU` DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>date</th>\n",
       "      <th>filename</th>\n",
       "      <th>sex</th>\n",
       "      <th>elapsedInDays</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25349.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-02-04</td>\n",
       "      <td>GSM25350.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2002-12-17</td>\n",
       "      <td>GSM25356.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-30</td>\n",
       "      <td>GSM25357.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>CEU</td>\n",
       "      <td>2003-01-03</td>\n",
       "      <td>GSM25358.CEL.gz</td>\n",
       "      <td>M</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  ethnicity       date         filename sex  elapsedInDays\n",
       "0       CEU 2003-02-04  GSM25349.CEL.gz   M             96\n",
       "1       CEU 2003-02-04  GSM25350.CEL.gz   M             96\n",
       "2       CEU 2002-12-17  GSM25356.CEL.gz   M             47\n",
       "3       CEU 2003-01-30  GSM25357.CEL.gz   M             91\n",
       "4       CEU 2003-01-03  GSM25358.CEL.gz   M             64"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "sampleinfoCEU = sampleinfo_pd[sampleinfo_pd.ethnicity==\"CEU\"]\n",
    "sampleinfoCEU.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, subset the `exprs` DataFrame to only include the samples with the CEU ethnicity. Name this new subsetted DataFrame `exprsCEU`. Show the head of the `exprsCEU` DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GSM25349.CEL.gz</th>\n",
       "      <th>GSM25350.CEL.gz</th>\n",
       "      <th>GSM25356.CEL.gz</th>\n",
       "      <th>GSM25357.CEL.gz</th>\n",
       "      <th>GSM25358.CEL.gz</th>\n",
       "      <th>GSM25359.CEL.gz</th>\n",
       "      <th>GSM25360.CEL.gz</th>\n",
       "      <th>GSM25361.CEL.gz</th>\n",
       "      <th>GSM25377.CEL.gz</th>\n",
       "      <th>GSM25378.CEL.gz</th>\n",
       "      <th>...</th>\n",
       "      <th>GSM48658.CEL.gz</th>\n",
       "      <th>GSM48660.CEL.gz</th>\n",
       "      <th>GSM48661.CEL.gz</th>\n",
       "      <th>GSM48662.CEL.gz</th>\n",
       "      <th>GSM48663.CEL.gz</th>\n",
       "      <th>GSM48664.CEL.gz</th>\n",
       "      <th>GSM48665.CEL.gz</th>\n",
       "      <th>GSM136725.CEL.gz</th>\n",
       "      <th>GSM136726.CEL.gz</th>\n",
       "      <th>GSM136727.CEL.gz</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1007_s_at</th>\n",
       "      <td>6.627014</td>\n",
       "      <td>6.249807</td>\n",
       "      <td>5.934128</td>\n",
       "      <td>6.145268</td>\n",
       "      <td>6.091270</td>\n",
       "      <td>6.041186</td>\n",
       "      <td>6.050375</td>\n",
       "      <td>6.063847</td>\n",
       "      <td>6.226106</td>\n",
       "      <td>6.358282</td>\n",
       "      <td>...</td>\n",
       "      <td>6.501510</td>\n",
       "      <td>6.558100</td>\n",
       "      <td>6.618286</td>\n",
       "      <td>6.869995</td>\n",
       "      <td>5.981000</td>\n",
       "      <td>6.403285</td>\n",
       "      <td>6.248702</td>\n",
       "      <td>6.488579</td>\n",
       "      <td>6.809481</td>\n",
       "      <td>6.133068</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1053_at</th>\n",
       "      <td>6.939184</td>\n",
       "      <td>6.818038</td>\n",
       "      <td>7.047962</td>\n",
       "      <td>7.422477</td>\n",
       "      <td>7.272361</td>\n",
       "      <td>7.128216</td>\n",
       "      <td>6.750719</td>\n",
       "      <td>6.836287</td>\n",
       "      <td>6.738022</td>\n",
       "      <td>7.367895</td>\n",
       "      <td>...</td>\n",
       "      <td>7.066686</td>\n",
       "      <td>7.386702</td>\n",
       "      <td>6.407958</td>\n",
       "      <td>7.089180</td>\n",
       "      <td>7.120923</td>\n",
       "      <td>7.018998</td>\n",
       "      <td>7.155419</td>\n",
       "      <td>7.517410</td>\n",
       "      <td>7.200596</td>\n",
       "      <td>7.280781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117_at</th>\n",
       "      <td>5.113570</td>\n",
       "      <td>5.074096</td>\n",
       "      <td>5.371201</td>\n",
       "      <td>5.266550</td>\n",
       "      <td>5.342047</td>\n",
       "      <td>5.063876</td>\n",
       "      <td>5.315898</td>\n",
       "      <td>5.483652</td>\n",
       "      <td>6.689444</td>\n",
       "      <td>6.482782</td>\n",
       "      <td>...</td>\n",
       "      <td>5.600514</td>\n",
       "      <td>5.232676</td>\n",
       "      <td>5.630720</td>\n",
       "      <td>4.944748</td>\n",
       "      <td>5.275577</td>\n",
       "      <td>5.770358</td>\n",
       "      <td>5.616842</td>\n",
       "      <td>5.247190</td>\n",
       "      <td>5.237239</td>\n",
       "      <td>5.401876</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121_at</th>\n",
       "      <td>7.833862</td>\n",
       "      <td>7.780682</td>\n",
       "      <td>7.458197</td>\n",
       "      <td>7.655948</td>\n",
       "      <td>7.546555</td>\n",
       "      <td>7.072670</td>\n",
       "      <td>7.092984</td>\n",
       "      <td>6.954225</td>\n",
       "      <td>7.489785</td>\n",
       "      <td>7.388539</td>\n",
       "      <td>...</td>\n",
       "      <td>7.437535</td>\n",
       "      <td>7.714650</td>\n",
       "      <td>7.416252</td>\n",
       "      <td>7.746448</td>\n",
       "      <td>8.001434</td>\n",
       "      <td>7.626723</td>\n",
       "      <td>7.452299</td>\n",
       "      <td>7.331864</td>\n",
       "      <td>7.357102</td>\n",
       "      <td>7.607461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1255_g_at</th>\n",
       "      <td>3.152269</td>\n",
       "      <td>3.111747</td>\n",
       "      <td>3.018932</td>\n",
       "      <td>3.154545</td>\n",
       "      <td>3.107954</td>\n",
       "      <td>3.224284</td>\n",
       "      <td>3.114241</td>\n",
       "      <td>3.044975</td>\n",
       "      <td>3.304038</td>\n",
       "      <td>2.887919</td>\n",
       "      <td>...</td>\n",
       "      <td>3.009983</td>\n",
       "      <td>3.151203</td>\n",
       "      <td>3.199709</td>\n",
       "      <td>3.159496</td>\n",
       "      <td>3.149710</td>\n",
       "      <td>3.242780</td>\n",
       "      <td>3.433125</td>\n",
       "      <td>3.175846</td>\n",
       "      <td>3.105092</td>\n",
       "      <td>3.225123</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 102 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           GSM25349.CEL.gz  GSM25350.CEL.gz  GSM25356.CEL.gz  GSM25357.CEL.gz  \\\n",
       "1007_s_at         6.627014         6.249807         5.934128         6.145268   \n",
       "1053_at           6.939184         6.818038         7.047962         7.422477   \n",
       "117_at            5.113570         5.074096         5.371201         5.266550   \n",
       "121_at            7.833862         7.780682         7.458197         7.655948   \n",
       "1255_g_at         3.152269         3.111747         3.018932         3.154545   \n",
       "\n",
       "           GSM25358.CEL.gz  GSM25359.CEL.gz  GSM25360.CEL.gz  GSM25361.CEL.gz  \\\n",
       "1007_s_at         6.091270         6.041186         6.050375         6.063847   \n",
       "1053_at           7.272361         7.128216         6.750719         6.836287   \n",
       "117_at            5.342047         5.063876         5.315898         5.483652   \n",
       "121_at            7.546555         7.072670         7.092984         6.954225   \n",
       "1255_g_at         3.107954         3.224284         3.114241         3.044975   \n",
       "\n",
       "           GSM25377.CEL.gz  GSM25378.CEL.gz        ...         \\\n",
       "1007_s_at         6.226106         6.358282        ...          \n",
       "1053_at           6.738022         7.367895        ...          \n",
       "117_at            6.689444         6.482782        ...          \n",
       "121_at            7.489785         7.388539        ...          \n",
       "1255_g_at         3.304038         2.887919        ...          \n",
       "\n",
       "           GSM48658.CEL.gz  GSM48660.CEL.gz  GSM48661.CEL.gz  GSM48662.CEL.gz  \\\n",
       "1007_s_at         6.501510         6.558100         6.618286         6.869995   \n",
       "1053_at           7.066686         7.386702         6.407958         7.089180   \n",
       "117_at            5.600514         5.232676         5.630720         4.944748   \n",
       "121_at            7.437535         7.714650         7.416252         7.746448   \n",
       "1255_g_at         3.009983         3.151203         3.199709         3.159496   \n",
       "\n",
       "           GSM48663.CEL.gz  GSM48664.CEL.gz  GSM48665.CEL.gz  \\\n",
       "1007_s_at         5.981000         6.403285         6.248702   \n",
       "1053_at           7.120923         7.018998         7.155419   \n",
       "117_at            5.275577         5.770358         5.616842   \n",
       "121_at            8.001434         7.626723         7.452299   \n",
       "1255_g_at         3.149710         3.242780         3.433125   \n",
       "\n",
       "           GSM136725.CEL.gz  GSM136726.CEL.gz  GSM136727.CEL.gz  \n",
       "1007_s_at          6.488579          6.809481          6.133068  \n",
       "1053_at            7.517410          7.200596          7.280781  \n",
       "117_at             5.247190          5.237239          5.401876  \n",
       "121_at             7.331864          7.357102          7.607461  \n",
       "1255_g_at          3.175846          3.105092          3.225123  \n",
       "\n",
       "[5 rows x 102 columns]"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "temp_list_b=np.array(sampleinfoCEU.filename).transpose()\n",
    "exprsCEU_pd= exprs_pd[temp_list_b]\n",
    "exprsCEU_pd.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check to make sure the order of the columns in the `exprsCEU` DataFrame matches the rows in the `sampleinfoCEU` DataFrame.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "(exprsCEU_pd.columns==sampleinfoCEU.filename).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the average gene expression intensity in the `exprsCEU` DataFrame across all the samples. For each sample in the `exprsCEU` DataFrame, subtract the average gene expression intensity from each of the samples. Show the head of the mean normalized gene expression data.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/oracle/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>GSM25349.CEL.gz</th>\n",
       "      <th>GSM25350.CEL.gz</th>\n",
       "      <th>GSM25356.CEL.gz</th>\n",
       "      <th>GSM25357.CEL.gz</th>\n",
       "      <th>GSM25358.CEL.gz</th>\n",
       "      <th>GSM25359.CEL.gz</th>\n",
       "      <th>GSM25360.CEL.gz</th>\n",
       "      <th>GSM25361.CEL.gz</th>\n",
       "      <th>GSM25377.CEL.gz</th>\n",
       "      <th>GSM25378.CEL.gz</th>\n",
       "      <th>...</th>\n",
       "      <th>GSM48660.CEL.gz</th>\n",
       "      <th>GSM48661.CEL.gz</th>\n",
       "      <th>GSM48662.CEL.gz</th>\n",
       "      <th>GSM48663.CEL.gz</th>\n",
       "      <th>GSM48664.CEL.gz</th>\n",
       "      <th>GSM48665.CEL.gz</th>\n",
       "      <th>GSM136725.CEL.gz</th>\n",
       "      <th>GSM136726.CEL.gz</th>\n",
       "      <th>GSM136727.CEL.gz</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1007_s_at</th>\n",
       "      <td>6.627014</td>\n",
       "      <td>6.249807</td>\n",
       "      <td>5.934128</td>\n",
       "      <td>6.145268</td>\n",
       "      <td>6.091270</td>\n",
       "      <td>6.041186</td>\n",
       "      <td>6.050375</td>\n",
       "      <td>6.063847</td>\n",
       "      <td>6.226106</td>\n",
       "      <td>6.358282</td>\n",
       "      <td>...</td>\n",
       "      <td>6.558100</td>\n",
       "      <td>6.618286</td>\n",
       "      <td>6.869995</td>\n",
       "      <td>5.981000</td>\n",
       "      <td>6.403285</td>\n",
       "      <td>6.248702</td>\n",
       "      <td>6.488579</td>\n",
       "      <td>6.809481</td>\n",
       "      <td>6.133068</td>\n",
       "      <td>6.261955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1053_at</th>\n",
       "      <td>6.939184</td>\n",
       "      <td>6.818038</td>\n",
       "      <td>7.047962</td>\n",
       "      <td>7.422477</td>\n",
       "      <td>7.272361</td>\n",
       "      <td>7.128216</td>\n",
       "      <td>6.750719</td>\n",
       "      <td>6.836287</td>\n",
       "      <td>6.738022</td>\n",
       "      <td>7.367895</td>\n",
       "      <td>...</td>\n",
       "      <td>7.386702</td>\n",
       "      <td>6.407958</td>\n",
       "      <td>7.089180</td>\n",
       "      <td>7.120923</td>\n",
       "      <td>7.018998</td>\n",
       "      <td>7.155419</td>\n",
       "      <td>7.517410</td>\n",
       "      <td>7.200596</td>\n",
       "      <td>7.280781</td>\n",
       "      <td>7.075217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117_at</th>\n",
       "      <td>5.113570</td>\n",
       "      <td>5.074096</td>\n",
       "      <td>5.371201</td>\n",
       "      <td>5.266550</td>\n",
       "      <td>5.342047</td>\n",
       "      <td>5.063876</td>\n",
       "      <td>5.315898</td>\n",
       "      <td>5.483652</td>\n",
       "      <td>6.689444</td>\n",
       "      <td>6.482782</td>\n",
       "      <td>...</td>\n",
       "      <td>5.232676</td>\n",
       "      <td>5.630720</td>\n",
       "      <td>4.944748</td>\n",
       "      <td>5.275577</td>\n",
       "      <td>5.770358</td>\n",
       "      <td>5.616842</td>\n",
       "      <td>5.247190</td>\n",
       "      <td>5.237239</td>\n",
       "      <td>5.401876</td>\n",
       "      <td>5.604127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>121_at</th>\n",
       "      <td>7.833862</td>\n",
       "      <td>7.780682</td>\n",
       "      <td>7.458197</td>\n",
       "      <td>7.655948</td>\n",
       "      <td>7.546555</td>\n",
       "      <td>7.072670</td>\n",
       "      <td>7.092984</td>\n",
       "      <td>6.954225</td>\n",
       "      <td>7.489785</td>\n",
       "      <td>7.388539</td>\n",
       "      <td>...</td>\n",
       "      <td>7.714650</td>\n",
       "      <td>7.416252</td>\n",
       "      <td>7.746448</td>\n",
       "      <td>8.001434</td>\n",
       "      <td>7.626723</td>\n",
       "      <td>7.452299</td>\n",
       "      <td>7.331864</td>\n",
       "      <td>7.357102</td>\n",
       "      <td>7.607461</td>\n",
       "      <td>7.415835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1255_g_at</th>\n",
       "      <td>3.152269</td>\n",
       "      <td>3.111747</td>\n",
       "      <td>3.018932</td>\n",
       "      <td>3.154545</td>\n",
       "      <td>3.107954</td>\n",
       "      <td>3.224284</td>\n",
       "      <td>3.114241</td>\n",
       "      <td>3.044975</td>\n",
       "      <td>3.304038</td>\n",
       "      <td>2.887919</td>\n",
       "      <td>...</td>\n",
       "      <td>3.151203</td>\n",
       "      <td>3.199709</td>\n",
       "      <td>3.159496</td>\n",
       "      <td>3.149710</td>\n",
       "      <td>3.242780</td>\n",
       "      <td>3.433125</td>\n",
       "      <td>3.175846</td>\n",
       "      <td>3.105092</td>\n",
       "      <td>3.225123</td>\n",
       "      <td>3.133934</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 103 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           GSM25349.CEL.gz  GSM25350.CEL.gz  GSM25356.CEL.gz  GSM25357.CEL.gz  \\\n",
       "1007_s_at         6.627014         6.249807         5.934128         6.145268   \n",
       "1053_at           6.939184         6.818038         7.047962         7.422477   \n",
       "117_at            5.113570         5.074096         5.371201         5.266550   \n",
       "121_at            7.833862         7.780682         7.458197         7.655948   \n",
       "1255_g_at         3.152269         3.111747         3.018932         3.154545   \n",
       "\n",
       "           GSM25358.CEL.gz  GSM25359.CEL.gz  GSM25360.CEL.gz  GSM25361.CEL.gz  \\\n",
       "1007_s_at         6.091270         6.041186         6.050375         6.063847   \n",
       "1053_at           7.272361         7.128216         6.750719         6.836287   \n",
       "117_at            5.342047         5.063876         5.315898         5.483652   \n",
       "121_at            7.546555         7.072670         7.092984         6.954225   \n",
       "1255_g_at         3.107954         3.224284         3.114241         3.044975   \n",
       "\n",
       "           GSM25377.CEL.gz  GSM25378.CEL.gz    ...     GSM48660.CEL.gz  \\\n",
       "1007_s_at         6.226106         6.358282    ...            6.558100   \n",
       "1053_at           6.738022         7.367895    ...            7.386702   \n",
       "117_at            6.689444         6.482782    ...            5.232676   \n",
       "121_at            7.489785         7.388539    ...            7.714650   \n",
       "1255_g_at         3.304038         2.887919    ...            3.151203   \n",
       "\n",
       "           GSM48661.CEL.gz  GSM48662.CEL.gz  GSM48663.CEL.gz  GSM48664.CEL.gz  \\\n",
       "1007_s_at         6.618286         6.869995         5.981000         6.403285   \n",
       "1053_at           6.407958         7.089180         7.120923         7.018998   \n",
       "117_at            5.630720         4.944748         5.275577         5.770358   \n",
       "121_at            7.416252         7.746448         8.001434         7.626723   \n",
       "1255_g_at         3.199709         3.159496         3.149710         3.242780   \n",
       "\n",
       "           GSM48665.CEL.gz  GSM136725.CEL.gz  GSM136726.CEL.gz  \\\n",
       "1007_s_at         6.248702          6.488579          6.809481   \n",
       "1053_at           7.155419          7.517410          7.200596   \n",
       "117_at            5.616842          5.247190          5.237239   \n",
       "121_at            7.452299          7.331864          7.357102   \n",
       "1255_g_at         3.433125          3.175846          3.105092   \n",
       "\n",
       "           GSM136727.CEL.gz      mean  \n",
       "1007_s_at          6.133068  6.261955  \n",
       "1053_at            7.280781  7.075217  \n",
       "117_at             5.401876  5.604127  \n",
       "121_at             7.607461  7.415835  \n",
       "1255_g_at          3.225123  3.133934  \n",
       "\n",
       "[5 rows x 103 columns]"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "exprsCEU_pd[\"mean\"] = np.mean(exprsCEU_pd,axis =1)\n",
    "exprsCEU_pd.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this mean normalized gene expression data, compute the projection to the first Principal Component (PC1).  \n",
    "\n",
    "**Hint**: Use the `numpy.linalg.svd()` function in the `numpy.linalg` module (or the `scipy.linalg.svd()` function in the `scipy.linalg` module) to apply an [singular value decomposition](http://en.wikipedia.org/wiki/Singular_value_decomposition) to a matrix.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "U\n",
      "[[-0.01100349  0.00610128 -0.00347624 ..., -0.009698   -0.00948823\n",
      "  -0.01145174]\n",
      " [-0.01243387  0.01231938 -0.00273229 ..., -0.02460923 -0.04204628\n",
      "  -0.01965097]\n",
      " [-0.00984616 -0.0183444  -0.00813099 ..., -0.02217508 -0.02111863\n",
      "  -0.01692594]\n",
      " ..., \n",
      " [-0.01711842  0.01928585 -0.01200345 ...,  0.94669278  0.03134413\n",
      "   0.03375682]\n",
      " [-0.01511171 -0.18602474 -0.022842   ...,  0.03070145  0.83539086\n",
      "  -0.16208723]\n",
      " [-0.01610244 -0.18833369 -0.02071295 ...,  0.0331666  -0.16274305\n",
      "   0.82912746]]\n",
      "V\n",
      "[  5.77553218e+03   1.31566141e+02   1.16262742e+02   1.04022007e+02\n",
      "   8.37253726e+01   7.35197884e+01   6.99954587e+01   6.59834653e+01\n",
      "   6.12754440e+01   5.64813008e+01   5.24860216e+01   5.03401297e+01\n",
      "   4.80094931e+01   4.39894903e+01   4.29360448e+01   4.10490890e+01\n",
      "   3.89438118e+01   3.57159101e+01   3.53498809e+01   3.51195987e+01\n",
      "   3.39770990e+01   3.26938291e+01   3.20643221e+01   3.15324323e+01\n",
      "   3.07257845e+01   3.00348271e+01   2.96372527e+01   2.92202092e+01\n",
      "   2.84210252e+01   2.80240861e+01   2.77140559e+01   2.69323285e+01\n",
      "   2.66736835e+01   2.63689688e+01   2.62882265e+01   2.58196112e+01\n",
      "   2.56907247e+01   2.54535481e+01   2.48486170e+01   2.44832383e+01\n",
      "   2.40166062e+01   2.39637340e+01   2.35671096e+01   2.33999810e+01\n",
      "   2.30490875e+01   2.26092714e+01   2.24401150e+01   2.22121375e+01\n",
      "   2.18985871e+01   2.15254394e+01   2.15080124e+01   2.10062955e+01\n",
      "   2.09562708e+01   2.06388205e+01   2.02254801e+01   1.99424178e+01\n",
      "   1.98468684e+01   1.97220813e+01   1.93692333e+01   1.90880312e+01\n",
      "   1.89716160e+01   1.86802026e+01   1.84290708e+01   1.82879328e+01\n",
      "   1.81017949e+01   1.80575195e+01   1.75062449e+01   1.74628919e+01\n",
      "   1.73468528e+01   1.69172305e+01   1.68999265e+01   1.66007878e+01\n",
      "   1.64532096e+01   1.62447886e+01   1.60768006e+01   1.59100680e+01\n",
      "   1.58118036e+01   1.56995281e+01   1.55230337e+01   1.52804622e+01\n",
      "   1.50414524e+01   1.49983254e+01   1.48654470e+01   1.46552589e+01\n",
      "   1.45490559e+01   1.44400912e+01   1.41552322e+01   1.40455748e+01\n",
      "   1.37818356e+01   1.36373396e+01   1.29173866e+01   1.27561893e+01\n",
      "   1.25744885e+01   1.21207807e+01   1.19448903e+01   1.16699812e+01\n",
      "   1.10345602e+01   1.06997629e+01   1.02880309e+01   9.95767777e+00\n",
      "   9.50026628e+00   8.13632727e+00   2.08444731e-13]\n",
      "d\n",
      "[[ -9.87260581e-02  -9.87551008e-02  -9.82561831e-02 ...,  -9.86678301e-02\n",
      "   -9.88356458e-02  -9.85310473e-02]\n",
      " [  4.93011659e-02   5.42980146e-02  -1.63631455e-02 ...,   1.90941917e-01\n",
      "    1.95314529e-01  -2.61593305e-04]\n",
      " [ -1.41114617e-01  -1.32337383e-01  -1.17523329e-01 ...,   1.26509782e-01\n",
      "   -1.02868445e-01  -3.29169900e-04]\n",
      " ..., \n",
      " [  4.36288739e-02  -3.93540723e-02  -2.52639450e-03 ...,   1.03721281e-02\n",
      "   -1.18645285e-02   2.63835296e-05]\n",
      " [ -2.13340914e-04  -7.26998656e-03   5.36830434e-03 ...,   8.99018986e-03\n",
      "   -5.35818689e-04   1.35237394e-05]\n",
      " [ -9.75621364e-03  -9.75621364e-03  -9.75621364e-03 ...,  -9.75621364e-03\n",
      "   -9.75621364e-03   9.95133791e-01]]\n"
     ]
    }
   ],
   "source": [
    "#your code here\n",
    "\n",
    "U,V,d = np.linalg.svd(exprsCEU_pd)\n",
    "print(\"U\")\n",
    "print(U)\n",
    "print(\"V\")\n",
    "print(V)\n",
    "print(\"d\")\n",
    "print(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a histogram using the values from PC1.  Use a bin size of 25.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a scatter plot with the days since October 31, 2002 on the x-axis and PC1 on the y-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Around what day do you notice a difference in the way the samples were processed?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion for Problem 1\n",
    "\n",
    "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 2: Is there a pollster bias in presidential election polls?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(a)\n",
    "\n",
    "The [HuffPost Pollster](http://elections.huffingtonpost.com/pollster) contains many political polls. You can access these polls from individual races as a CSV but you can also access polls through the [HuffPost Pollster API](http://elections.huffingtonpost.com/pollster/api) to access the data.  \n",
    "\n",
    "Read in the polls from the [2012 Presidential Election: Barack Obama vs Mitt Romney](http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama) into a pandas DataFrame called `election`. For this problem, you may read in the polls for this race directly using [the CSV file](http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama.csv) available from the HuffPost Pollster page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the head of the `election` DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many polls were conducted in November? Define this number as M.  \n",
    "\n",
    "**Hint**: Subset the `election` DataFrame for only dates in the `Start Date` column that are in November 2012.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What was the median of the number of observations in the November polls? Define this quantity as N. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(b)\n",
    "\n",
    "Using the median sample size $N$ from Problem 1(a), simulate the results from a single poll:  simulate the number of votes for Obama out of a sample size $N$ where $p$ = 0.53 is the percent of voters who are voting for Obama.  \n",
    "\n",
    "**Hint**: Use the binomial distribution with parameters $N$ and $p$ = 0.53. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, perform a Monte Carlo simulation to obtain the estimated percentage of Obama votes with a sample size $N$ where $N$ is the median sample size calculated in Problem 2(a). Let $p$=0.53 be the percent of voters are voting for Obama. \n",
    "\n",
    "**Hint**: You will repeat the simulation above 1,000 times and plot the distribution of the estimated *percent* of Obama votes from a single poll.  The results from the single poll you simulate is random variable and will be different every time you sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the distribution of the estimated percentage of Obama votes from your single poll. What is the distribution of the estimated percentage of Obama votes? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the standard error (SE) of the estimated percentage from the poll. \n",
    "\n",
    "**Hint**: Remember the SE is the standard deviation (SD) of the distribution of a random variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(c)\n",
    "\n",
    "Now suppose we run M polls where M is the number of polls that happened in November (calculated in Problem 2(a)). Run 1,000 simulations and compute the mean of the M polls for each simulation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the distribution of the average of polls?\n",
    "\n",
    "**Hint**: Show a plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the standard error (SE) of the average of polls? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the SE of the average of polls larger, the same, or smaller than that the SD of a single poll (calculated in Problem 2(b))? By how much?\n",
    "\n",
    "**Hint**: Compute a ratio of the two quantities.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(d) \n",
    "\n",
    "Repeat Problem 2(c) but now record the *across poll* standard deviation in each simulation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the distribution of the *across M polls* standard deviation?\n",
    "\n",
    "**Hint**: Show a plot. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(e) \n",
    "\n",
    "What is the standard deviation of M polls in our real (not simulated) 2012 presidential election data ? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this larger, the same, or smaller than what we expeced if polls were not biased."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 2(f)\n",
    "\n",
    "**For AC209 Students**: Learn about the normal approximation for the binomial distribution and derive the results of Problem 2(b) and 2(c) analytically (using this approximation). Compare the results obtained analytically to those obtained from simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion for Problem 2\n",
    "\n",
    "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 3: Is the average of polls better than just one poll?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 3(a)\n",
    "\n",
    "Most undecided voters vote for one of the two candidates at the election. Therefore, the reported percentages underestimate the final value of both candidates. However, if we assume the undecided will split evenly, then the observed difference should be an unbiased estimate of the final difference. \n",
    "\n",
    "Add a new column to the `election` DataFrame containg the difference between Obama and Romeny called `Diff`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 3(b)\n",
    "\n",
    "Make a plot of the differences for the week before the election (e.g. 5 days) where the days are on the x-axis and the differences are on the y-axis.  Add a horizontal line showing 3.9%: the difference between Obama and Romney on election day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 3(c) \n",
    "\n",
    "Make a plot showing the differences by pollster where the pollsters are on the x-axis and the differences on the y-axis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the *across poll* difference larger than the *between pollster* difference? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 3(d)\n",
    "\n",
    "Take the average for each pollster and then compute the average of that. Given this difference how confident would you have been of an Obama victory?\n",
    "\n",
    "**Hint**: Compute an estimate of the SE of this average based exclusively on the observed data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 3(e)\n",
    "\n",
    "**For AC209 Students**: Show the difference against time and see if you can detect a trend towards the end. Use this trend to see if it improves the final estimate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion for Problem 3\n",
    "\n",
    "*Write a brief discussion of your conclusions to the questions and tasks above in 100 words or less.*\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 4\n",
    "\n",
    "In this last problem, we will use the polls from the [2014 Senate Midterm Elections](http://elections.huffingtonpost.com/pollster) from the [HuffPost Pollster API](http://elections.huffingtonpost.com/pollster/api) to create a preliminary prediction of the result of each state. \n",
    "\n",
    "The HuffPost Pollster API allows you to access the data as a CSV or a JSON response by tacking \".csv\" or \".json\" at the end of the URLs. For example the 2012 Presidential Election could be accessed as a [.json](http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama.json) instead of a [.csv](http://elections.huffingtonpost.com/pollster/2012-general-election-romney-vs-obama.csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 4(a)\n",
    "\n",
    "Read in the polls for **all** of the 2014 Senate Elections using the HuffPost API. For example, we can consider the [2014 Senate race in Kentucky between Mitch McConnell and Alison Grimes](http://elections.huffingtonpost.com/pollster/2014-kentucky-senate-mcconnell-vs-grimes). \n",
    "\n",
    "To search for the 2014 Senate races, use the `topics` parameter in the API [[read more about topics here](http://elections.huffingtonpost.com/pollster/api)].  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_str = \"http://elections.huffingtonpost.com/pollster/api/charts/?topic=2014-senate\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To list all the URLs related to the 2014 Senate races using the pollster API, we can use a list comprehension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'http://elections.huffingtonpost.com/pollster/2014-kentucky-senate-mcconnell-vs-grimes.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-arkansas-senate-cotton-vs-pryor.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-michigan-senate-land-vs-peters.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-louisiana-senate-cassidy-vs-landrieu.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-new-hampshire-senate-brown-vs-shaheen.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-west-virginia-senate-capito-vs-tennant.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-new-hampshire-senate-bass-vs-shaheen.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-north-carolina-senate-tillis-vs-hagan.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-virginia-senate-gillespie-vs-warner.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-colorado-senate-gardner-vs-udall.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-illinois-senate-oberweis-vs-durbin.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-alaska-senate-sullivan-vs-begich.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-iowa-senate-ernst-vs-braley.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-mississippi-senate-cochran-vs-childers.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-oregon-senate-wehby-vs-merkley.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-georgia-senate-perdue-vs-nunn.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-louisiana-senate-sasse-vs-domina.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-south-dakota-senate-rounds-vs-weiland.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-maine-senate-collins-vs-bellows.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-minnesota-senate-mcfadden-vs-franken.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-texas-senate-cornyn-vs-alameel.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-south-carolina-senate-graham-vs-hutto.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-south-carolina-senate-scott-vs-dickerson.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-oklahoma-senate-inhofe-vs-silverstein.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-new-mexico-senate-weh-vs-udall.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-new-jersey-senate-bell-vs-booker.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-idaho-senate-risch-vs-mitchell.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-tennessee-senate-alexander-vs-ball.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-wyoming-senate.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-kansas-senate-roberts-vs-orman-vs-taylor.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-hawaii-senate-cavasso-vs-schatz.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-oklahoma-senate-lankford-vs-johnson.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-montana-senate-daines-vs-curtis.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-rhode-island-senate-zaccaria-vs-reed.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-massachusetts-senate-herr-vs-markey.csv',\n",
       " u'http://elections.huffingtonpost.com/pollster/2014-delaware-senate-wade-vs-coons.csv']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "election_urls = [election['url'] + '.csv' for election in requests.get(url_str).json()]\n",
    "election_urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because there so many Senate races, we can create a dictionary of pandas DataFrames that will be keyed by the name of the election (a string). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_frame(url):\n",
    "    \"\"\"\n",
    "    Returns a pandas DataFrame object containing\n",
    "    the data returned from the given url\n",
    "    \"\"\"\n",
    "    source = requests.get(url).text\n",
    "    \n",
    "    # Use StringIO because pd.DataFrame.from_csv requires .read() method\n",
    "    s = StringIO(source)\n",
    "    \n",
    "    return pd.DataFrame.from_csv(s, index_col=None).convert_objects(\n",
    "            convert_dates=\"coerce\", convert_numeric=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Makes a dictionary of pandas DataFrames keyed on election string.\n",
    "dfs = dict((election.split(\"/\")[-1][:-4], build_frame(election)) for election in election_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the head of the DataFrame containing the polls for the 2014 Senate race in Kentucky between McConnell and Grimes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Problem 4(b)\n",
    "\n",
    "For each 2014 Senate race, create a preliminary prediction of the result for that state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "#your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submission Instructions\n",
    "\n",
    "To submit your homework, create a folder named **lastname_firstinitial_hw#** and place your IPython notebooks, data files, and any other files in this folder. Your IPython Notebooks should be completely executed with the results visible in the notebook. We should not have to run any code.  Compress the folder (please use .zip compression) and submit to the CS109 dropbox in the appropriate folder. *If we cannot access your work because these directions are not followed correctly, we will not grade your work.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}