class3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python\n",
    "# Class 3: Data manipulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "In the last class, \n",
    "we learned to import and inspect spreadsheet data into Python,\n",
    "and then explored ways to extract particular columns/rows/slices.\n",
    "That means we're ready to dive deeper into additional data manipulation activities provided by `pandas`.\n",
    "\n",
    "By the end of this lesson, you should be able to:\n",
    "- subset data by idenitfying rows that meet particular conditions\n",
    "- grouping data to summarize by category\n",
    "- visualize data with `matplotlib`\n",
    "- accommodate missing data\n",
    "\n",
    "## Getting set up\n",
    "\n",
    "Create a new Python notebook called `class3` in your project directory.\n",
    "\n",
    "We'll first load our library for this lesson:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load pandas library\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can import and inspect our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>24477.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>371.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-24477.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.958904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1936.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3406</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>26615.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>136.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26615.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.191781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>TCGA-18-3407</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>28171.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>2304.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-28171.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>2099.0</td>\n",
       "      <td>1.643836</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1927.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3408</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>27154.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8083/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-27154.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3747.0</td>\n",
       "      <td>1.095890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3409</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage iib</td>\n",
       "      <td>29827.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>146.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-29827.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1923.0</td>\n",
       "      <td>not reported</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3410</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "0             C34.1    stage ia           24477.0         dead     8070/3   \n",
       "1             C34.1    stage ib           26615.0         dead     8070/3   \n",
       "2             C34.3    stage ib           28171.0         dead     8070/3   \n",
       "3             C34.1    stage ia           27154.0        alive     8083/3   \n",
       "4             C34.3   stage iib           29827.0         dead     8070/3   \n",
       "\n",
       "   days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "0          371.0  live                     C34.1       -24477.0   \n",
       "1          136.0  live                     C34.1       -26615.0   \n",
       "2         2304.0  live                     C34.3       -28171.0   \n",
       "3            NaN  live                     C34.1       -27154.0   \n",
       "4          146.0  live                     C34.3       -29827.0   \n",
       "\n",
       "  site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "0                       C34.1                     NaN           10.958904   \n",
       "1                       C34.1                     NaN            2.191781   \n",
       "2                       C34.3                  2099.0            1.643836   \n",
       "3                       C34.1                  3747.0            1.095890   \n",
       "4                       C34.3                     NaN                 NaN   \n",
       "\n",
       "   years_smoked  gender  year_of_birth          race               ethnicity  \\\n",
       "0           NaN    male         1936.0         white  not hispanic or latino   \n",
       "1           NaN    male         1931.0         asian  not hispanic or latino   \n",
       "2           NaN  female         1927.0         white  not hispanic or latino   \n",
       "3           NaN    male         1930.0         white  not hispanic or latino   \n",
       "4           NaN    male         1923.0  not reported            not reported   \n",
       "\n",
       "   year_of_death bcr_patient_barcode disease  \n",
       "0         2004.0        TCGA-18-3406    LUSC  \n",
       "1         2003.0        TCGA-18-3407    LUSC  \n",
       "2            NaN        TCGA-18-3408    LUSC  \n",
       "3            NaN        TCGA-18-3409    LUSC  \n",
       "4         2004.0        TCGA-18-3410    LUSC  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import data\n",
    "clinical_df = pd.read_csv(\"data/clinical.csv\") \n",
    "clinical_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditional subsetting\n",
    "\n",
    "Now that we're set up with data and tools,\n",
    "we're going to explore conditional subsetting.\n",
    "This means extracting particular rows based on a criteria.\n",
    "\n",
    "For example, \n",
    "we may want to find all data from patients born in 1930.\n",
    "We may be tempted to try something like `clinical_df.year_of_birth == 1930`,\n",
    "but the output we get isn't quite what we wanted: the results are logical data (true/false).\n",
    "\n",
    "> We used double equal signs (`==`) to indicate mathematical equivalency,\n",
    "and to differentiate from variable assignment and specifying parameters for arguments\n",
    "(like in the last lesson with `sep=` for loading data).\n",
    "\n",
    "However, we can combine this information with what we know about data subsetting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>27154.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8083/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-27154.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3747.0</td>\n",
       "      <td>1.095890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3409</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>24253.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-24253.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>3724.0</td>\n",
       "      <td>5.479452</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-21-1080</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage iiia</td>\n",
       "      <td>26937.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>1057.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26937.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>1057.0</td>\n",
       "      <td>3.287671</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>TCGA-22-4601</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage iia</td>\n",
       "      <td>26952.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>399.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26952.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.739726</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-22-4604</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>27749.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-27749.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>1845.0</td>\n",
       "      <td>2.739726</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not reported</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-22-5471</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "3              C34.1    stage ia           27154.0        alive     8083/3   \n",
       "26             C34.3    stage ib           24253.0        alive     8070/3   \n",
       "51             C34.1  stage iiia           26937.0         dead     8070/3   \n",
       "52             C34.1   stage iia           26952.0         dead     8070/3   \n",
       "57             C34.3    stage ib           27749.0        alive     8070/3   \n",
       "\n",
       "    days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "3             NaN  live                     C34.1       -27154.0   \n",
       "26            NaN  live                     C34.3       -24253.0   \n",
       "51         1057.0  live                     C34.1       -26937.0   \n",
       "52          399.0  live                     C34.1       -26952.0   \n",
       "57            NaN  live                     C34.3       -27749.0   \n",
       "\n",
       "   site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "3                        C34.1                  3747.0            1.095890   \n",
       "26                       C34.3                  3724.0            5.479452   \n",
       "51                       C34.1                  1057.0            3.287671   \n",
       "52                       C34.1                     NaN            2.739726   \n",
       "57                       C34.3                  1845.0            2.739726   \n",
       "\n",
       "    years_smoked  gender  year_of_birth   race               ethnicity  \\\n",
       "3            NaN    male         1930.0  white  not hispanic or latino   \n",
       "26           NaN    male         1930.0  white  not hispanic or latino   \n",
       "51           NaN  female         1930.0  white            not reported   \n",
       "52           NaN    male         1930.0  white            not reported   \n",
       "57           NaN    male         1930.0  white            not reported   \n",
       "\n",
       "    year_of_death bcr_patient_barcode disease  \n",
       "3             NaN        TCGA-18-3409    LUSC  \n",
       "26            NaN        TCGA-21-1080    LUSC  \n",
       "51         2005.0        TCGA-22-4601    LUSC  \n",
       "52         2004.0        TCGA-22-4604    LUSC  \n",
       "57            NaN        TCGA-22-5471    LUSC  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conditionally subset all patients born in 1930\n",
    "clinical_df[clinical_df.year_of_birth == 1930].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can invert the select (e.g., identify all patients *not* born in 1930)\n",
    "using `!=`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>24477.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>371.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-24477.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.958904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1936.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3406</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>26615.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>136.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26615.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.191781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>TCGA-18-3407</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>28171.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>2304.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-28171.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>2099.0</td>\n",
       "      <td>1.643836</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1927.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3408</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage iib</td>\n",
       "      <td>29827.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>146.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-29827.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1923.0</td>\n",
       "      <td>not reported</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3410</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage iiia</td>\n",
       "      <td>23370.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-23370.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3576.0</td>\n",
       "      <td>2.739726</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1942.0</td>\n",
       "      <td>not reported</td>\n",
       "      <td>not reported</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3411</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "0             C34.1    stage ia           24477.0         dead     8070/3   \n",
       "1             C34.1    stage ib           26615.0         dead     8070/3   \n",
       "2             C34.3    stage ib           28171.0         dead     8070/3   \n",
       "4             C34.3   stage iib           29827.0         dead     8070/3   \n",
       "5             C34.1  stage iiia           23370.0        alive     8070/3   \n",
       "\n",
       "   days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "0          371.0  live                     C34.1       -24477.0   \n",
       "1          136.0  live                     C34.1       -26615.0   \n",
       "2         2304.0  live                     C34.3       -28171.0   \n",
       "4          146.0  live                     C34.3       -29827.0   \n",
       "5            NaN  live                     C34.1       -23370.0   \n",
       "\n",
       "  site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "0                       C34.1                     NaN           10.958904   \n",
       "1                       C34.1                     NaN            2.191781   \n",
       "2                       C34.3                  2099.0            1.643836   \n",
       "4                       C34.3                     NaN                 NaN   \n",
       "5                       C34.1                  3576.0            2.739726   \n",
       "\n",
       "   years_smoked  gender  year_of_birth          race               ethnicity  \\\n",
       "0           NaN    male         1936.0         white  not hispanic or latino   \n",
       "1           NaN    male         1931.0         asian  not hispanic or latino   \n",
       "2           NaN  female         1927.0         white  not hispanic or latino   \n",
       "4           NaN    male         1923.0  not reported            not reported   \n",
       "5           NaN  female         1942.0  not reported            not reported   \n",
       "\n",
       "   year_of_death bcr_patient_barcode disease  \n",
       "0         2004.0        TCGA-18-3406    LUSC  \n",
       "1         2003.0        TCGA-18-3407    LUSC  \n",
       "2            NaN        TCGA-18-3408    LUSC  \n",
       "4         2004.0        TCGA-18-3410    LUSC  \n",
       "5            NaN        TCGA-18-3411    LUSC  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# conditionally subset all patients NOT born in 1930\n",
    "clinical_df[clinical_df.year_of_birth != 1930].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine multiple criteria into the same filter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>24477.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>371.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-24477.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.958904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1936.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3406</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>26615.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>136.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26615.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.191781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>TCGA-18-3407</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>27154.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8083/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-27154.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3747.0</td>\n",
       "      <td>1.095890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3409</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage iv</td>\n",
       "      <td>26938.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>716.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-26938.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.369863</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1932.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>TCGA-18-3414</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage iib</td>\n",
       "      <td>26813.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-26813.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>2811.0</td>\n",
       "      <td>2.739726</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1934.0</td>\n",
       "      <td>not reported</td>\n",
       "      <td>not reported</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3419</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "0              C34.1    stage ia           24477.0         dead     8070/3   \n",
       "1              C34.1    stage ib           26615.0         dead     8070/3   \n",
       "3              C34.1    stage ia           27154.0        alive     8083/3   \n",
       "7              C34.3    stage iv           26938.0         dead     8070/3   \n",
       "11             C34.3   stage iib           26813.0        alive     8070/3   \n",
       "\n",
       "    days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "0           371.0  live                     C34.1       -24477.0   \n",
       "1           136.0  live                     C34.1       -26615.0   \n",
       "3             NaN  live                     C34.1       -27154.0   \n",
       "7           716.0  live                     C34.3       -26938.0   \n",
       "11            NaN  live                     C34.3       -26813.0   \n",
       "\n",
       "   site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "0                        C34.1                     NaN           10.958904   \n",
       "1                        C34.1                     NaN            2.191781   \n",
       "3                        C34.1                  3747.0            1.095890   \n",
       "7                        C34.3                     NaN            1.369863   \n",
       "11                       C34.3                  2811.0            2.739726   \n",
       "\n",
       "    years_smoked gender  year_of_birth          race               ethnicity  \\\n",
       "0            NaN   male         1936.0         white  not hispanic or latino   \n",
       "1            NaN   male         1931.0         asian  not hispanic or latino   \n",
       "3            NaN   male         1930.0         white  not hispanic or latino   \n",
       "7            NaN   male         1932.0         asian  not hispanic or latino   \n",
       "11           NaN   male         1934.0  not reported            not reported   \n",
       "\n",
       "    year_of_death bcr_patient_barcode disease  \n",
       "0          2004.0        TCGA-18-3406    LUSC  \n",
       "1          2003.0        TCGA-18-3407    LUSC  \n",
       "3             NaN        TCGA-18-3409    LUSC  \n",
       "7          2006.0        TCGA-18-3414    LUSC  \n",
       "11            NaN        TCGA-18-3419    LUSC  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract all patients born between 1930 and 1940\n",
    "clinical_df[(clinical_df.year_of_birth >= 1930) & (clinical_df.year_of_birth <= 1940)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the example above, the ampersand (`&`) represents *AND*, \n",
    "meaning any value must meet *both* conditions.\n",
    "We can include a vertical pipe (`|`), representing *OR*,\n",
    "meaning a value must meet *at least one* condition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>26615.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>136.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26615.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.191781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>TCGA-18-3407</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>27154.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8083/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-27154.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3747.0</td>\n",
       "      <td>1.095890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3409</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>24253.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-24253.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>3724.0</td>\n",
       "      <td>5.479452</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-21-1080</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage iiia</td>\n",
       "      <td>26937.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>1057.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26937.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>1057.0</td>\n",
       "      <td>3.287671</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>TCGA-22-4601</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage iia</td>\n",
       "      <td>26952.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>399.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26952.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.739726</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-22-4604</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "1              C34.1    stage ib           26615.0         dead     8070/3   \n",
       "3              C34.1    stage ia           27154.0        alive     8083/3   \n",
       "26             C34.3    stage ib           24253.0        alive     8070/3   \n",
       "51             C34.1  stage iiia           26937.0         dead     8070/3   \n",
       "52             C34.1   stage iia           26952.0         dead     8070/3   \n",
       "\n",
       "    days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "1           136.0  live                     C34.1       -26615.0   \n",
       "3             NaN  live                     C34.1       -27154.0   \n",
       "26            NaN  live                     C34.3       -24253.0   \n",
       "51         1057.0  live                     C34.1       -26937.0   \n",
       "52          399.0  live                     C34.1       -26952.0   \n",
       "\n",
       "   site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "1                        C34.1                     NaN            2.191781   \n",
       "3                        C34.1                  3747.0            1.095890   \n",
       "26                       C34.3                  3724.0            5.479452   \n",
       "51                       C34.1                  1057.0            3.287671   \n",
       "52                       C34.1                     NaN            2.739726   \n",
       "\n",
       "    years_smoked  gender  year_of_birth   race               ethnicity  \\\n",
       "1            NaN    male         1931.0  asian  not hispanic or latino   \n",
       "3            NaN    male         1930.0  white  not hispanic or latino   \n",
       "26           NaN    male         1930.0  white  not hispanic or latino   \n",
       "51           NaN  female         1930.0  white            not reported   \n",
       "52           NaN    male         1930.0  white            not reported   \n",
       "\n",
       "    year_of_death bcr_patient_barcode disease  \n",
       "1          2003.0        TCGA-18-3407    LUSC  \n",
       "3             NaN        TCGA-18-3409    LUSC  \n",
       "26            NaN        TCGA-21-1080    LUSC  \n",
       "51         2005.0        TCGA-22-4601    LUSC  \n",
       "52         2004.0        TCGA-22-4604    LUSC  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract all data for cases born in 1930 or 1931\n",
    "clinical_df[(clinical_df.year_of_birth == 1930) | (clinical_df.year_of_birth == 1931)].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Challenge-conditionals\n",
    "Print to the screen all data from `clinical_df` for patients with stage ia tumors (column `tumor_stage`) who live more than 365 days (`days_to_death`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping data\n",
    "\n",
    "Another useful feature of `pandas` is the ability to group data by categories,\n",
    "so you can then summarize other quantitative variables in the dataset.\n",
    "\n",
    "First, we can explore what categories exist for our `race` column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['white', 'asian', 'not reported', 'black or african american', nan,\n",
       "       'american indian or alaska native',\n",
       "       'native hawaiian or other pacific islander'], dtype=object)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# identify unique elements in a column\n",
    "pd.unique(clinical_df[\"race\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the code above is synonymous to `pd.unique(clinical_df.race)`, \n",
    "which uses a different syntax to access the column.\n",
    "\n",
    "Now that we know the categories in `race`,\n",
    "we may be interested in summarizing quantitative variables based on these categories.\n",
    "We can assign our grouped data to a new variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# group data by race \n",
    "grouped_data = clinical_df.groupby(\"race\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`grouped_data` represents the same data, but with the data oriented according to categories\n",
    "(though it isn't interpretable by humans, so it's not useful to print it to the screen).\n",
    "\n",
    "> In the command above,\n",
    "we can't specify race using `.` because of the syntax required by `groupby`.\n",
    "\n",
    "We can then calculate summary statistics for our grouped data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"8\" halign=\"left\">age_at_diagnosis</th>\n",
       "      <th colspan=\"2\" halign=\"left\">days_to_death</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"2\" halign=\"left\">year_of_birth</th>\n",
       "      <th colspan=\"8\" halign=\"left\">year_of_death</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>...</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>american indian or alaska native</th>\n",
       "      <td>15.0</td>\n",
       "      <td>18653.400000</td>\n",
       "      <td>4803.399954</td>\n",
       "      <td>7855.0</td>\n",
       "      <td>16817.50</td>\n",
       "      <td>19423.0</td>\n",
       "      <td>21279.00</td>\n",
       "      <td>28472.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>277.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>1965.0</td>\n",
       "      <td>1977.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2006.000000</td>\n",
       "      <td>7.937254</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>2010.5</td>\n",
       "      <td>2012.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>asian</th>\n",
       "      <td>299.0</td>\n",
       "      <td>21743.906355</td>\n",
       "      <td>4925.857739</td>\n",
       "      <td>6572.0</td>\n",
       "      <td>18681.50</td>\n",
       "      <td>21428.0</td>\n",
       "      <td>25789.00</td>\n",
       "      <td>32872.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>514.880597</td>\n",
       "      <td>...</td>\n",
       "      <td>1961.0</td>\n",
       "      <td>1991.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>2006.703704</td>\n",
       "      <td>3.244105</td>\n",
       "      <td>1999.0</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>2006.0</td>\n",
       "      <td>2009.5</td>\n",
       "      <td>2012.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>black or african american</th>\n",
       "      <td>575.0</td>\n",
       "      <td>21970.570435</td>\n",
       "      <td>5037.715451</td>\n",
       "      <td>6377.0</td>\n",
       "      <td>18577.50</td>\n",
       "      <td>22372.0</td>\n",
       "      <td>25804.00</td>\n",
       "      <td>32872.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>893.726257</td>\n",
       "      <td>...</td>\n",
       "      <td>1959.0</td>\n",
       "      <td>1991.0</td>\n",
       "      <td>131.0</td>\n",
       "      <td>2006.809160</td>\n",
       "      <td>4.665940</td>\n",
       "      <td>1992.0</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>2014.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>native hawaiian or other pacific islander</th>\n",
       "      <td>12.0</td>\n",
       "      <td>19740.000000</td>\n",
       "      <td>3433.788181</td>\n",
       "      <td>14225.0</td>\n",
       "      <td>17718.00</td>\n",
       "      <td>20124.0</td>\n",
       "      <td>21709.75</td>\n",
       "      <td>24981.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>753.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1960.0</td>\n",
       "      <td>1969.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2006.000000</td>\n",
       "      <td>3.464102</td>\n",
       "      <td>2002.0</td>\n",
       "      <td>2005.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>2008.0</td>\n",
       "      <td>2008.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>not reported</th>\n",
       "      <td>1082.0</td>\n",
       "      <td>23870.847505</td>\n",
       "      <td>4128.186661</td>\n",
       "      <td>5303.0</td>\n",
       "      <td>21427.25</td>\n",
       "      <td>24134.5</td>\n",
       "      <td>26453.75</td>\n",
       "      <td>32872.0</td>\n",
       "      <td>204.0</td>\n",
       "      <td>715.009804</td>\n",
       "      <td>...</td>\n",
       "      <td>1951.0</td>\n",
       "      <td>1988.0</td>\n",
       "      <td>147.0</td>\n",
       "      <td>2006.115646</td>\n",
       "      <td>4.509529</td>\n",
       "      <td>1991.0</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>2013.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white</th>\n",
       "      <td>4735.0</td>\n",
       "      <td>22062.366631</td>\n",
       "      <td>5223.048363</td>\n",
       "      <td>3982.0</td>\n",
       "      <td>18771.00</td>\n",
       "      <td>22642.0</td>\n",
       "      <td>25929.00</td>\n",
       "      <td>32872.0</td>\n",
       "      <td>1730.0</td>\n",
       "      <td>911.302312</td>\n",
       "      <td>...</td>\n",
       "      <td>1957.0</td>\n",
       "      <td>1993.0</td>\n",
       "      <td>1255.0</td>\n",
       "      <td>2006.153785</td>\n",
       "      <td>4.485962</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>2007.0</td>\n",
       "      <td>2009.0</td>\n",
       "      <td>2014.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          age_at_diagnosis                \\\n",
       "                                                     count          mean   \n",
       "race                                                                       \n",
       "american indian or alaska native                      15.0  18653.400000   \n",
       "asian                                                299.0  21743.906355   \n",
       "black or african american                            575.0  21970.570435   \n",
       "native hawaiian or other pacific islander             12.0  19740.000000   \n",
       "not reported                                        1082.0  23870.847505   \n",
       "white                                               4735.0  22062.366631   \n",
       "\n",
       "                                                                           \\\n",
       "                                                   std      min       25%   \n",
       "race                                                                        \n",
       "american indian or alaska native           4803.399954   7855.0  16817.50   \n",
       "asian                                      4925.857739   6572.0  18681.50   \n",
       "black or african american                  5037.715451   6377.0  18577.50   \n",
       "native hawaiian or other pacific islander  3433.788181  14225.0  17718.00   \n",
       "not reported                               4128.186661   5303.0  21427.25   \n",
       "white                                      5223.048363   3982.0  18771.00   \n",
       "\n",
       "                                                                       \\\n",
       "                                               50%       75%      max   \n",
       "race                                                                    \n",
       "american indian or alaska native           19423.0  21279.00  28472.0   \n",
       "asian                                      21428.0  25789.00  32872.0   \n",
       "black or african american                  22372.0  25804.00  32872.0   \n",
       "native hawaiian or other pacific islander  20124.0  21709.75  24981.0   \n",
       "not reported                               24134.5  26453.75  32872.0   \n",
       "white                                      22642.0  25929.00  32872.0   \n",
       "\n",
       "                                          days_to_death              ...  \\\n",
       "                                                  count        mean  ...   \n",
       "race                                                                 ...   \n",
       "american indian or alaska native                    3.0  277.333333  ...   \n",
       "asian                                              67.0  514.880597  ...   \n",
       "black or african american                         179.0  893.726257  ...   \n",
       "native hawaiian or other pacific islander           4.0  753.000000  ...   \n",
       "not reported                                      204.0  715.009804  ...   \n",
       "white                                            1730.0  911.302312  ...   \n",
       "\n",
       "                                          year_of_birth         year_of_death  \\\n",
       "                                                    75%     max         count   \n",
       "race                                                                            \n",
       "american indian or alaska native                 1965.0  1977.0           3.0   \n",
       "asian                                            1961.0  1991.0          27.0   \n",
       "black or african american                        1959.0  1991.0         131.0   \n",
       "native hawaiian or other pacific islander        1960.0  1969.0           3.0   \n",
       "not reported                                     1951.0  1988.0         147.0   \n",
       "white                                            1957.0  1993.0        1255.0   \n",
       "\n",
       "                                                                          \\\n",
       "                                                  mean       std     min   \n",
       "race                                                                       \n",
       "american indian or alaska native           2006.000000  7.937254  1997.0   \n",
       "asian                                      2006.703704  3.244105  1999.0   \n",
       "black or african american                  2006.809160  4.665940  1992.0   \n",
       "native hawaiian or other pacific islander  2006.000000  3.464102  2002.0   \n",
       "not reported                               2006.115646  4.509529  1991.0   \n",
       "white                                      2006.153785  4.485962  1990.0   \n",
       "\n",
       "                                                                           \n",
       "                                              25%     50%     75%     max  \n",
       "race                                                                       \n",
       "american indian or alaska native           2003.0  2009.0  2010.5  2012.0  \n",
       "asian                                      2005.0  2006.0  2009.5  2012.0  \n",
       "black or african american                  2004.0  2008.0  2011.0  2014.0  \n",
       "native hawaiian or other pacific islander  2005.0  2008.0  2008.0  2008.0  \n",
       "not reported                               2004.0  2007.0  2009.0  2013.0  \n",
       "white                                      2004.0  2007.0  2009.0  2014.0  \n",
       "\n",
       "[6 rows x 64 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# summary stats for all columns by race\n",
    "grouped_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This output only summarizes quantitative variables \n",
    "(the names listed in the first line of the output, e.g., `age_at_diagnosis`).\n",
    "The subtitles in the output (e.g., `count`, `mean`, etc)\n",
    "represent the summary statistics for each variable, \n",
    "grouped by race (in the row labels).\n",
    "\n",
    "Only a subset of the output is included above;\n",
    "the ellipsis (`...`) in the middle of the dataset indicate it's been truncated.\n",
    "You can extract only the data for a single column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>american indian or alaska native</th>\n",
       "      <td>3.0</td>\n",
       "      <td>277.333333</td>\n",
       "      <td>153.968612</td>\n",
       "      <td>100.0</td>\n",
       "      <td>227.5</td>\n",
       "      <td>355.0</td>\n",
       "      <td>366.0</td>\n",
       "      <td>377.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>asian</th>\n",
       "      <td>67.0</td>\n",
       "      <td>514.880597</td>\n",
       "      <td>469.331682</td>\n",
       "      <td>3.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>385.0</td>\n",
       "      <td>709.5</td>\n",
       "      <td>2532.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>black or african american</th>\n",
       "      <td>179.0</td>\n",
       "      <td>893.726257</td>\n",
       "      <td>1032.724507</td>\n",
       "      <td>5.0</td>\n",
       "      <td>244.0</td>\n",
       "      <td>532.0</td>\n",
       "      <td>1114.0</td>\n",
       "      <td>6456.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>native hawaiian or other pacific islander</th>\n",
       "      <td>4.0</td>\n",
       "      <td>753.000000</td>\n",
       "      <td>491.974254</td>\n",
       "      <td>262.0</td>\n",
       "      <td>502.0</td>\n",
       "      <td>661.0</td>\n",
       "      <td>912.0</td>\n",
       "      <td>1428.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>not reported</th>\n",
       "      <td>204.0</td>\n",
       "      <td>715.009804</td>\n",
       "      <td>778.817671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>180.0</td>\n",
       "      <td>456.5</td>\n",
       "      <td>945.0</td>\n",
       "      <td>3838.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white</th>\n",
       "      <td>1730.0</td>\n",
       "      <td>911.302312</td>\n",
       "      <td>1095.713823</td>\n",
       "      <td>2.0</td>\n",
       "      <td>291.0</td>\n",
       "      <td>541.0</td>\n",
       "      <td>1066.5</td>\n",
       "      <td>10870.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            count        mean          std  \\\n",
       "race                                                                         \n",
       "american indian or alaska native              3.0  277.333333   153.968612   \n",
       "asian                                        67.0  514.880597   469.331682   \n",
       "black or african american                   179.0  893.726257  1032.724507   \n",
       "native hawaiian or other pacific islander     4.0  753.000000   491.974254   \n",
       "not reported                                204.0  715.009804   778.817671   \n",
       "white                                      1730.0  911.302312  1095.713823   \n",
       "\n",
       "                                             min    25%    50%     75%  \\\n",
       "race                                                                     \n",
       "american indian or alaska native           100.0  227.5  355.0   366.0   \n",
       "asian                                        3.0  196.0  385.0   709.5   \n",
       "black or african american                    5.0  244.0  532.0  1114.0   \n",
       "native hawaiian or other pacific islander  262.0  502.0  661.0   912.0   \n",
       "not reported                                 0.0  180.0  456.5   945.0   \n",
       "white                                        2.0  291.0  541.0  1066.5   \n",
       "\n",
       "                                               max  \n",
       "race                                                \n",
       "american indian or alaska native             377.0  \n",
       "asian                                       2532.0  \n",
       "black or african american                   6456.0  \n",
       "native hawaiian or other pacific islander   1428.0  \n",
       "not reported                                3838.0  \n",
       "white                                      10870.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# summary stats for race for only one column (days_to_death)\n",
    "grouped_data.days_to_death.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to these basic summary statistics for quantitative variables,\n",
    "we can also assess how missing data affects the data available for each category.\n",
    "We obtain this information by counting the nuber of data points for each category from our grouped data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>american indian or alaska native</th>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>asian</th>\n",
       "      <td>316</td>\n",
       "      <td>316</td>\n",
       "      <td>299</td>\n",
       "      <td>316</td>\n",
       "      <td>316</td>\n",
       "      <td>67</td>\n",
       "      <td>316</td>\n",
       "      <td>316</td>\n",
       "      <td>299</td>\n",
       "      <td>316</td>\n",
       "      <td>296</td>\n",
       "      <td>27</td>\n",
       "      <td>6</td>\n",
       "      <td>316</td>\n",
       "      <td>315</td>\n",
       "      <td>316</td>\n",
       "      <td>27</td>\n",
       "      <td>316</td>\n",
       "      <td>316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>black or african american</th>\n",
       "      <td>581</td>\n",
       "      <td>581</td>\n",
       "      <td>575</td>\n",
       "      <td>581</td>\n",
       "      <td>581</td>\n",
       "      <td>179</td>\n",
       "      <td>581</td>\n",
       "      <td>581</td>\n",
       "      <td>575</td>\n",
       "      <td>581</td>\n",
       "      <td>474</td>\n",
       "      <td>72</td>\n",
       "      <td>24</td>\n",
       "      <td>581</td>\n",
       "      <td>579</td>\n",
       "      <td>581</td>\n",
       "      <td>131</td>\n",
       "      <td>581</td>\n",
       "      <td>581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>native hawaiian or other pacific islander</th>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>not reported</th>\n",
       "      <td>1116</td>\n",
       "      <td>1116</td>\n",
       "      <td>1082</td>\n",
       "      <td>1116</td>\n",
       "      <td>1116</td>\n",
       "      <td>204</td>\n",
       "      <td>1116</td>\n",
       "      <td>1116</td>\n",
       "      <td>1082</td>\n",
       "      <td>1116</td>\n",
       "      <td>976</td>\n",
       "      <td>226</td>\n",
       "      <td>138</td>\n",
       "      <td>1116</td>\n",
       "      <td>1011</td>\n",
       "      <td>1116</td>\n",
       "      <td>147</td>\n",
       "      <td>1116</td>\n",
       "      <td>1116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>white</th>\n",
       "      <td>4753</td>\n",
       "      <td>4753</td>\n",
       "      <td>4735</td>\n",
       "      <td>4753</td>\n",
       "      <td>4753</td>\n",
       "      <td>1730</td>\n",
       "      <td>4753</td>\n",
       "      <td>4753</td>\n",
       "      <td>4735</td>\n",
       "      <td>4753</td>\n",
       "      <td>3947</td>\n",
       "      <td>843</td>\n",
       "      <td>280</td>\n",
       "      <td>4753</td>\n",
       "      <td>4730</td>\n",
       "      <td>4753</td>\n",
       "      <td>1255</td>\n",
       "      <td>4753</td>\n",
       "      <td>4753</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           primary_diagnosis  tumor_stage  \\\n",
       "race                                                                        \n",
       "american indian or alaska native                          15           15   \n",
       "asian                                                    316          316   \n",
       "black or african american                                581          581   \n",
       "native hawaiian or other pacific islander                 12           12   \n",
       "not reported                                            1116         1116   \n",
       "white                                                   4753         4753   \n",
       "\n",
       "                                           age_at_diagnosis  vital_status  \\\n",
       "race                                                                        \n",
       "american indian or alaska native                         15            15   \n",
       "asian                                                   299           316   \n",
       "black or african american                               575           581   \n",
       "native hawaiian or other pacific islander                12            12   \n",
       "not reported                                           1082          1116   \n",
       "white                                                  4735          4753   \n",
       "\n",
       "                                           morphology  days_to_death  state  \\\n",
       "race                                                                          \n",
       "american indian or alaska native                   15              3     15   \n",
       "asian                                             316             67    316   \n",
       "black or african american                         581            179    581   \n",
       "native hawaiian or other pacific islander          12              4     12   \n",
       "not reported                                     1116            204   1116   \n",
       "white                                            4753           1730   4753   \n",
       "\n",
       "                                           tissue_or_organ_of_origin  \\\n",
       "race                                                                   \n",
       "american indian or alaska native                                  15   \n",
       "asian                                                            316   \n",
       "black or african american                                        581   \n",
       "native hawaiian or other pacific islander                         12   \n",
       "not reported                                                    1116   \n",
       "white                                                           4753   \n",
       "\n",
       "                                           days_to_birth  \\\n",
       "race                                                       \n",
       "american indian or alaska native                      15   \n",
       "asian                                                299   \n",
       "black or african american                            575   \n",
       "native hawaiian or other pacific islander             12   \n",
       "not reported                                        1082   \n",
       "white                                               4735   \n",
       "\n",
       "                                           site_of_resection_or_biopsy  \\\n",
       "race                                                                     \n",
       "american indian or alaska native                                    15   \n",
       "asian                                                              316   \n",
       "black or african american                                          581   \n",
       "native hawaiian or other pacific islander                           12   \n",
       "not reported                                                      1116   \n",
       "white                                                             4753   \n",
       "\n",
       "                                           days_to_last_follow_up  \\\n",
       "race                                                                \n",
       "american indian or alaska native                               12   \n",
       "asian                                                         296   \n",
       "black or african american                                     474   \n",
       "native hawaiian or other pacific islander                       9   \n",
       "not reported                                                  976   \n",
       "white                                                        3947   \n",
       "\n",
       "                                           cigarettes_per_day  years_smoked  \\\n",
       "race                                                                          \n",
       "american indian or alaska native                            2             0   \n",
       "asian                                                      27             6   \n",
       "black or african american                                  72            24   \n",
       "native hawaiian or other pacific islander                   1             0   \n",
       "not reported                                              226           138   \n",
       "white                                                     843           280   \n",
       "\n",
       "                                           gender  year_of_birth  ethnicity  \\\n",
       "race                                                                          \n",
       "american indian or alaska native               15             15         15   \n",
       "asian                                         316            315        316   \n",
       "black or african american                     581            579        581   \n",
       "native hawaiian or other pacific islander      12             12         12   \n",
       "not reported                                 1116           1011       1116   \n",
       "white                                        4753           4730       4753   \n",
       "\n",
       "                                           year_of_death  bcr_patient_barcode  \\\n",
       "race                                                                            \n",
       "american indian or alaska native                       3                   15   \n",
       "asian                                                 27                  316   \n",
       "black or african american                            131                  581   \n",
       "native hawaiian or other pacific islander              3                   12   \n",
       "not reported                                         147                 1116   \n",
       "white                                               1255                 4753   \n",
       "\n",
       "                                           disease  \n",
       "race                                                \n",
       "american indian or alaska native                15  \n",
       "asian                                          316  \n",
       "black or african american                      581  \n",
       "native hawaiian or other pacific islander       12  \n",
       "not reported                                  1116  \n",
       "white                                         4753  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show the number of patients for each race available for each columns \n",
    "grouped_data.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the output above,\n",
    "we can also extract the data for only one column.\n",
    "This lets us know how many cases for each race category are in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "race\n",
       "american indian or alaska native               15\n",
       "asian                                         316\n",
       "black or african american                     581\n",
       "native hawaiian or other pacific islander      12\n",
       "not reported                                 1116\n",
       "white                                        4753\n",
       "Name: race, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# counts for only race\n",
    "grouped_data.race.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's useful to compare the output from a different column, \n",
    "which allows us to understand how missing data affects our ability to\n",
    "perform statistics tests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "race\n",
       "american indian or alaska native                3\n",
       "asian                                          67\n",
       "black or african american                     179\n",
       "native hawaiian or other pacific islander       4\n",
       "not reported                                  204\n",
       "white                                        1730\n",
       "Name: days_to_death, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# count the number of each race for which days to death data is available\n",
    "grouped_data.days_to_death.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we see that missing data for days to death drops the number of patients in each race category by a substantial amount.\n",
    "\n",
    "Moreover, we can extract a single race category from the output above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only display one race (asian), from days_to_death grouped by race\n",
    "grouped_data.days_to_death.count().asian"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example of how you can use various pieces of Python syntax\n",
    "to ask increasingly specific questions of the data.\n",
    "\n",
    "It's useful to remember that the command above is synonymous with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# another way: only display one race (asian), from days_to_death grouped by race\n",
    "clinical_df.groupby(\"race\")[\"days_to_death\"].count()[\"asian\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command differs from the previous example because \n",
    "it begins by grouping the original data object (`clinical_df`),\n",
    "and applies alternative syntax for identifying columns.\n",
    "\n",
    "> Parentheses vs square brackets: You may be wondering at this point when to use parentheses (`()`) and when to use square brackets (`[]`).\n",
    "Parentheses are used to encase options associated with a function.\n",
    "For example, `groupby` above is followed by parentheses,\n",
    "because `\"race\"` is being used as an option for the method.\n",
    "`count` is followed by empty parentheses, because it's a function,\n",
    "but there are no arguments.\n",
    "On the other hand, square brackets are associated with accessing (subsetting or extracting) parts of a larger data object;\n",
    "they are not explicitly part of a function.\n",
    "\n",
    "Once you've identified the specific reformatting of data that you need,\n",
    "you may want to assign the output to a new object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "race\n",
      "american indian or alaska native                3\n",
      "asian                                          67\n",
      "black or african american                     179\n",
      "native hawaiian or other pacific islander       4\n",
      "not reported                                  204\n",
      "white                                        1730\n",
      "Name: days_to_death, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# save output to object for later use\n",
    "race_counts = grouped_data.days_to_death.count()\n",
    "print(race_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Challenge-combine\n",
    "Write code that will display the number of patients in this dataset who are listed as alive.\n",
    "> \n",
    "> (Hint: you should first group by `vital_status`, \n",
    "then extract only the `vital_status` results,\n",
    "then tally the number in each category, \n",
    "and finally extract only the answer for living patients)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing data with pandas\n",
    "\n",
    "The object we assigned at the end of the last section has data grouped by race\n",
    "and assessed for the days to death column. \n",
    "We can use these data to visualize the counts.\n",
    "\n",
    "First, we'll execute a line of code that may be necessary for some notebooks to visualize plots:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# allow figures appear inline for notebooks\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above may not make a difference for your specific notebook installation\n",
    "(some new versions of notebooks are smarter than others!).\n",
    "If you end up in an environment where your plots fail to show and that code doesn't work,\n",
    "you can can also try running `plt.show()` after executing code for a plot.\n",
    "\n",
    "Our entire next class is dedicated to plotting with the package `plotnine`.\n",
    "In this lesson,\n",
    "we'll do a quick introduction to plotting with `matplotlib`.\n",
    "This collection of functions is commonly used for plotting in Python, \n",
    "and is useful for taking a quick look at common types of plots since it's \n",
    "integrated with `pandas.`\n",
    "\n",
    "We'll use the object we created at the end of the last section to demonstrate a bar plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAHGCAYAAACYdFvnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAvmElEQVR4nO3de9ylc73/8dfbiERTyiQZY4YmhZgYpZKkdmiXdEQnlZ1CpdrtNrstUfZud9od9k4Jhe2n2BI6YZdDTmnGYZxrQpkIESYxzHj//riuxZrbPffM3Pe67+9a1/V+Ph7rca/1XYf7s5j7va71vb4H2SYiItphldIFRETExEnoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREi6xauoDlWWeddTx9+vTSZUREDJS5c+f+2faUoe19H/rTp09nzpw5pcuIiBgokn4/XHu6dyIiWiShHxHRIssNfUnHSLpD0tVdbd+XdEV9uVnSFXX7dEkPdN33za7nbC3pKknzJX1NksblHUVExDKtSJ/+d4H/Ao7rNNjevXNd0peAe7se/zvbs4Z5nSOAfYBLgJ8AOwM/XemKIyJi1JZ7pG/7fODu4e6rj9bfCpw40mtIWg+YbPtiVyu8HQfsttLVRkTEmIy1T/9lwO22f9vVNkPS5ZLOk/Syum19YEHXYxbUbRERMYHGOmRzT5Y+yr8NmGb7LklbAz+UtBkwXP/9Mtd0lrQPVVcQ06ZNG2OJERHRMeojfUmrAm8Evt9ps73I9l319bnA74DnUB3ZT+16+lTg1mW9tu0jbc+2PXvKlMfNLYiIiFEay5H+q4DrbT/abSNpCnC37SWSNgJmAjfavlvSQknbAr8C3gV8fSyFR0SUMP3AH0/o77v5c3/f09dbkSGbJwIXA5tIWiBp7/quPXj8CdztgXmSrgT+F/iA7c5J4H2Bo4D5VN8AMnInImKCLfdI3/aey2h/9zBtpwCnLOPxc4DNV7K+iIjooczIjYhokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREiyT0IyJaJKEfEdEiCf2IiBZJ6EdEtEhCPyKiRRL6EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWWW7oSzpG0h2Sru5q+7SkP0q6or68puu+gyTNl3SDpJ262reWdFV939ckqfdvJyIiRrIiR/rfBXYepv0/bc+qLz8BkLQpsAewWf2cb0iaVD/+CGAfYGZ9Ge41IyJiHC039G2fD9y9gq/3euB7thfZvgmYD7xQ0nrAZNsX2zZwHLDbKGuOiIhRGkuf/gclzau7f9au29YHbul6zIK6bf36+tD2iIiYQKMN/SOAjYFZwG3Al+r24frpPUL7sCTtI2mOpDl33nnnKEuMiIihRhX6tm+3vcT2I8C3gRfWdy0ANuh66FTg1rp96jDty3r9I23Ptj17ypQpoykxIiKGMarQr/voO94AdEb2nA7sIWl1STOoTtheavs2YKGkbetRO+8CThtD3RERMQqrLu8Bkk4EdgDWkbQAOATYQdIsqi6am4H3A9i+RtJJwLXAYmB/20vql9qXaiTQGsBP60tEREyg5Ya+7T2HaT56hMcfDhw+TPscYPOVqi4iInoqM3IjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREiyT0IyJaJKEfEdEiCf2IiBZJ6EdEtEhCPyKiRRL6EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWSehHRLRIQj8iokWWG/qSjpF0h6Sru9q+IOl6SfMknSrpqXX7dEkPSLqivnyz6zlbS7pK0nxJX5OkcXlHERGxTCtypP9dYOchbWcDm9veAvgNcFDXfb+zPau+fKCr/QhgH2BmfRn6mhERMc6WG/q2zwfuHtJ2lu3F9c1LgKkjvYak9YDJti+2beA4YLdRVRwREaPWiz799wI/7bo9Q9Llks6T9LK6bX1gQddjFtRtERExgVYdy5MlfRJYDJxQN90GTLN9l6StgR9K2gwYrv/eI7zuPlRdQUybNm0sJUZERJdRH+lL2gt4LfD2ussG24ts31Vfnwv8DngO1ZF9dxfQVODWZb227SNtz7Y9e8qUKaMtMSIihhhV6EvaGfhnYFfbf+tqnyJpUn19I6oTtjfavg1YKGnbetTOu4DTxlx9RESslOV270g6EdgBWEfSAuAQqtE6qwNn1yMvL6lH6mwPHCZpMbAE+IDtzkngfalGAq1BdQ6g+zxARERMgOWGvu09h2k+ehmPPQU4ZRn3zQE2X6nqIiKipzIjNyKiRRL6EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWSehHRLRIQj8iokUS+hERLZLQj4hokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREiyT0IyJaZLmhL+kYSXdIurqr7WmSzpb02/rn2l33HSRpvqQbJO3U1b61pKvq+74mSb1/OxERMZIVOdL/LrDzkLYDgZ/bngn8vL6NpE2BPYDN6ud8Q9Kk+jlHAPsAM+vL0NeMiIhxttzQt30+cPeQ5tcDx9bXjwV262r/nu1Ftm8C5gMvlLQeMNn2xbYNHNf1nIiImCCj7dNf1/ZtAPXPZ9Tt6wO3dD1uQd22fn19aHtEREygXp/IHa6f3iO0D/8i0j6S5kiac+edd/asuIiIthtt6N9ed9lQ/7yjbl8AbND1uKnArXX71GHah2X7SNuzbc+eMmXKKEuMiIihRhv6pwN71df3Ak7rat9D0uqSZlCdsL207gJaKGnbetTOu7qeExERE2TV5T1A0onADsA6khYAhwCfA06StDfwB+AtALavkXQScC2wGNjf9pL6pfalGgm0BvDT+hIRERNouaFve89l3PXKZTz+cODwYdrnAJuvVHUREdFTmZEbEdEiCf2IiBZJ6EdEtEhCPyKiRRL6EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWSehHRLRIQj8iokUS+hERLZLQj4hokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES0y6tCXtImkK7ou90n6iKRPS/pjV/trup5zkKT5km6QtFNv3kJERKyoVUf7RNs3ALMAJE0C/gicCrwH+E/bX+x+vKRNgT2AzYBnAf8n6Tm2l4y2hoiIWDm96t55JfA7278f4TGvB75ne5Htm4D5wAt79PsjImIF9Cr09wBO7Lr9QUnzJB0jae26bX3glq7HLKjbIiJigow59CWtBuwKnFw3HQFsTNX1cxvwpc5Dh3m6l/Ga+0iaI2nOnXfeOdYSIyKi1osj/V2Ay2zfDmD7dttLbD8CfJvHunAWABt0PW8qcOtwL2j7SNuzbc+eMmVKD0qMiAjoTejvSVfXjqT1uu57A3B1ff10YA9Jq0uaAcwELu3B74+IiBU06tE7AJKeBPwd8P6u5s9LmkXVdXNz5z7b10g6CbgWWAzsn5E7ERETa0yhb/tvwNOHtL1zhMcfDhw+lt8ZERGjlxm5EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWSehHRLRIQj8iokUS+hERLZLQj4hokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREiyT0IyJaJKEfEdEiYwp9STdLukrSFZLm1G1Pk3S2pN/WP9fuevxBkuZLukHSTmMtPiIiVk4vjvRfYXuW7dn17QOBn9ueCfy8vo2kTYE9gM2AnYFvSJrUg98fEREraDy6d14PHFtfPxbYrav9e7YX2b4JmA+8cBx+f0RELMNYQ9/AWZLmStqnblvX9m0A9c9n1O3rA7d0PXdB3RYRERNk1TE+/6W2b5X0DOBsSdeP8FgN0+ZhH1h9gOwDMG3atDGWGBERHWM60rd9a/3zDuBUqu6a2yWtB1D/vKN++AJgg66nTwVuXcbrHml7tu3ZU6ZMGUuJERHRZdShL2lNSU/uXAdeDVwNnA7sVT9sL+C0+vrpwB6SVpc0A5gJXDra3x8REStvLN076wKnSuq8zv+z/TNJvwZOkrQ38AfgLQC2r5F0EnAtsBjY3/aSMVUfERErZdShb/tGYMth2u8CXrmM5xwOHD7a3xkREWOTGbkRES2S0I+IaJGEfkREiyT0IyJaJKEfEdEiCf2IiBZJ6EdEtEhCPyKiRRL6EREtktCPiGiRhH5ERIsk9CMiWiShHxHRIgn9iIgWSehHRLRIQj8iokUS+hERLZLQj4hokYR+RESLJPQjIlokoR8R0SKjDn1JG0g6R9J1kq6RdEDd/mlJf5R0RX15TddzDpI0X9INknbqxRuIiIgVt+oYnrsY+Efbl0l6MjBX0tn1ff9p+4vdD5a0KbAHsBnwLOD/JD3H9pIx1BARESth1Ef6tm+zfVl9fSFwHbD+CE95PfA924ts3wTMB1442t8fERErryd9+pKmAy8AflU3fVDSPEnHSFq7blsfuKXraQtYxoeEpH0kzZE058477+xFiRERQQ9CX9JawCnAR2zfBxwBbAzMAm4DvtR56DBP93CvaftI27Ntz54yZcpYS4yIiNqYQl/SE6gC/wTbPwCwfbvtJbYfAb7NY104C4ANup4+Fbh1LL8/IiJWzlhG7wg4GrjO9pe72tfretgbgKvr66cDe0haXdIMYCZw6Wh/f0RErLyxjN55KfBO4CpJV9Rt/wLsKWkWVdfNzcD7AWxfI+kk4FqqkT/7Z+RORMTEGnXo276A4fvpfzLCcw4HDh/t74yIiLEZy5F+RMTjTD/wxxP6+27+3N9P6O8bdFmGISKiRRL6EREtku6d6DvpHogYPznSj4hokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES2S0I+IaJGEfkREiyT0IyJaJKEfEdEiWXBtAGVBsogYrRzpR0S0SEI/IqJFEvoRES2S0I+IaJEJD31JO0u6QdJ8SQdO9O+PiGizCQ19SZOA/wZ2ATYF9pS06UTWEBHRZhN9pP9CYL7tG20/BHwPeP0E1xAR0VqyPXG/THozsLPtf6hvvxN4ke0PDnncPsA+9c1NgBsmrEhYB/jzBP6+idTk9wZ5f4Mu76+3NrQ9ZWjjRE/O0jBtj/vUsX0kcOT4l/N4kubYnl3id4+3Jr83yPsbdHl/E2Oiu3cWABt03Z4K3DrBNUREtNZEh/6vgZmSZkhaDdgDOH2Ca4iIaK0J7d6xvVjSB4EzgUnAMbavmcgaVkCRbqUJ0uT3Bnl/gy7vbwJM6InciIgoKzNyIyJaJKEfEdEiCf2IPiRpFUlvLV1HNE/69AFJGwIzbf+fpDWAVW0vLF1XjEzSc4B/Ajaka1CC7R2LFdVDks63vX3pOnpN0lYj3W/7somqpY1aH/qS3kc1+/dptjeWNBP4pu1XFi6tJyS9FPg0jwWjANveqGRdvSDpSuCbwFxgSafd9txiRfWQpIOBB4DvA/d32m3fXayoHpB0Tn31icBs4Eqqf5dbAL+yvV2p2npJ0rrAvwHPsr1Lvc7Yi20fXbSuhL6uoFoT6Fe2X1C3XWX7+UUL6xFJ1wMf5fHBeFexonpE0lzbW5euY7xIummY5kZ8YANI+h5wuO2r6tubAx+3/e6ihfWIpJ8C3wE+aXtLSasCl5fOluyRC4tsPyRVK0TU/2Oa9El4r+2fli5inJwhaT/gVGBRp3HQj4Q7bM8oXcM4e24n8AFsXy1pVsF6em0d2ydJOggenae0ZHlPGm8JfThP0r8Aa0j6O2A/4IzCNfXSOZK+APyApYOxCf2me9U//6mrzUBTjoSfBHwMmGZ7n7rrcRPbPypcWq9cJ+ko4H+o/r+9A7iubEk9db+kp1MfREraFri3bEnp3kHSKsDewKup+hXPBI5yQ/7DdPWfdnNTTnY2maTvU3XLvcv25vUgg4ttzypbWW9IeiKwL9A5WX0+cITtB8tV1Tv1CeuvA5sDVwNTgLfYvrJoXQ3JtlGT9AbgJ7YXLffB0XfqfuBNqU4KAmD7uHIV9U5nVUZJl3edb7rS9pala+uV+oNsmu2JXD59Qkhaneo82iZUB5Q3AKuUzpp078CuwFcknU+1qcuZthcXrqmnJP09sBlLB+Nh5SrqDUmHADtQhf5PqHZkuwBoROgDD9Wh2Oke2JiuLrpBJ2lX4AvAasCMuj//MNu7Fi2sdy62vRXw6Ppiki4DRhyyOt5aPznL9nuAZwMnA28Dflf3MzaCpG8CuwMfojraeAvV8M0meDPwSuBP9f/HLYHVy5bUU4cAPwM2kHQC8HPgE2VL6qlDqEbO3QNg+wpgerlyekPSMyVtTXWe8AWStqovOwBPKltdjvQBsP1wPbzKwBpUWzj+Q9mqeuYltreQNM/2oZK+RHVStwkesP2IpMWSJgN30JCTuAC2z66PDLel+sA+wHaTdpZabPvezsi5BtkJeDfVfiFf7mpfCPxLiYK6tT70Je1Mta7/K4BzgaOAJk1/f6D++TdJzwLuApoyFHCOpKcC36Y64flX4NKiFfXAMDNWb6t/TpM0rSEjrwCulvQ2YFI9MunDwEWFaxoz28cCx0p6k+1TStczVE7kVhNEvgf8tPQJlvFQz+r8OlU3yH9TfZs5yvbBRQvrMUnTgcm255WuZaxaNGP1ScAnqUbOQTVy7jOD/nco6R22/0fSPzL8drBfHuZpE6b1od8m9WiCJ9ouPla4F+qRV7/ovJ/6qH8H2z8sWVevtGDG6ltsn7y8tkEj6f22v1UPNHgc24dOdE3dWhv6ki6wvZ2khSz9adxZm2ZyodJ6QtKOtn8h6Y3D3W974Pv1JV0xdMx69/DGQbeM9/e4tkEl6bJ6dMuIbdFbre3T73xFtv3k0rWMk5cDvwBeN8x9phknc4cbfdakf9ONnLEqaRfgNcD6kr7WdddkoDHDpSVNAd5HNSKpexXY95aqCVp8pN8h6Xjb71xeW/QfScdQDffrnKv4ELB2g7o/GjljVdKWwCzgMOBTXXctBM6x/ZcSdfWapIuAX/L4xQ6LntxN6A/5OlkvuDbP9qYFy+oZSQdQrfS3kGqUy1bAgbbPKlpYD0haEzgYeBVVt9xZwGdt3z/iE6M4SZOA42y/vXQt46Vfu+JaOzlL0kF1f/4Wku6rLwuB24HTCpfXS++1fR/VCIlnAO8BPle2pN6wfb/tA23Ptr217YOaFPiSXirpbEm/kXRj51K6rl6wvQR4uqTVStcyjn4k6TWlixgqR/rSv9s+qHQd46WelLWFpK8C59o+ddBPdkr6iu2PSDqD4YfENWIavxq8FwKApG9RffM8naU3iSk6pHGshgwOWYtq6YzOuYrig0SadNJrVGwfJGltYCZLr01zfrmqemqupLOoJmQdJOnJwCOFaxqr4+ufXyxaxfhr8l4IALfWl1WAxgyo6AwOkXQ8VZ/+L233zQn4HOlL/wAcQDVl+gqqKe8XN2Xp4Xrp6FnAE6jWpVkHWN/210vWNVZ1n/Cxtt9RupbxIulzwCSauRfCo+oDEdv+a+laeknSjsB2wMuolge5nOoD4KtF60ro6ypgG+AS27MkPRc41PbuhUvriSZ/qEk6E3id7YdK1zIemr4XQj3Z7HjgaXXTn6n2Drhm2c8aLPXByTZUy7x8gGq9qOeWrKn13TvAg7YflISk1W1fL2mT0kX10AE89qH2is6HWuGaeuVm4EJJjeoT7rD9itI1jLMjgY/ZPgegXoXy28BLCtbUM5J+DqwJXEzVzbON7TvKVpXQB1hQT9//IXC2pL9Q9TM2RZM/1BrZJ9xNDd0LobZmJ/ABbJ9bD8NtinnA1lQ7Z90L3CPpYtsPjPy08dX67p1ukl4OPAX4WVO6DCSdSjVM8yPAjsBfgCfY7ruhZKMlac0mDdXsqPdCeBJV18BRVPsHXGp776KF9Uj9b/MyHjsx/w5gtu3dihU1DiStRfU3+HHgmbaL7vmQ0OfRfrd1WXqq9B/KVTQ+mvahJunFwNHAWran1TM93297v8Kl9UTXcNvOz7WAH9h+9XKfPADqUXOHUp3sFNWM4083aEbuB6lO4m4N/J7q/f3S9i9K1tX67h1JH6Lawed2HhvKaKplbBvF9nmla+ixr1BtWHE6gO0rJW0/4jMGS5P3QqAO9w9LegrwiO2FpWvqsTWoNlGZ6z7agrX1oU91onOTpkx4aRvbtwzZeWnJsh47gH5Un2/6AlU3iKm6eRpB0jbAMdTnYyTdSzWDfG7RwnrE9hdK1zCchD7cQnWSJQbPLZJeAriezv9hGrAKZYftz9RXT5H0Ixq0F0LtaGA/278EkLQd1TpRjfuW3U8S+nAjcK6kH7P0BJhGDPtruA8AXwXWBxZQLbi2f9GKemBZeyDU9zViL4Tawk7gA9i+oF7CIMZRQh/+UF9Wqy8xIFxtEt7EVRqH2wOhoyl7IQBcWq+/cyLV+9qd6gBsK2jezON+kdE7MbAkzaBaQ386S4+8asSCa023jBnHHY2ZedxvEvoxsCRdSdUvfBVdi8g1ZZRSk/dCiHIS+jGwJP3K9otK1zFeJF1pe0tJO1GdqzgY+E5T9pCVtC7wb8CzbO8iaVPgxbaPLlxao7V2E5VohK9KOkTSiyVt1bmULqqHOmNRX0MV9ld2tTXBd4EzgWfVt39DNXM8xlHrT+TW+5DuzePXNym6eXGskOcD76RaXqJ7Yl1T+oKbuBdCt3VsnyTpIADbiyU1aZ5FX2p96FOt+3E91czOw6hGgzRmrHfDvQHYqAlLSizD3lR7Idxo+2+Snk61hktT3F+/JwNI2pbMmRl3CX14tu23SHq97WMl/T+qr5zR/64EngoUX662lyQ91/b1VIEPsNGQWcdN8TGqJTQ2lnQhMIVqUbkYRwl9eLj+eU+9qcOfqIYARv9bF7he0q9ZemLdoA/Z/BiwD/ClYe5rRPdVvcjhy+vLJlTnKm6w/fCIT4wxa/3onXpnqVOo+oe/S7WR8cG2v1Wyrli+etXQx2nKkM2mk3Su7R1K19E2CX3pabbvHtI2w/ZNpWqKaANJh1Mt9f19lt75LDNxx1FCv+pL3MX2ffXt5wEn2968bGWxPPWJv68Dz6NaQmMScL/tyUULixXS9D2A+1X69KvJIWfU29JtAhxHM9dzaaL/AvYATgZmA+8CZhatKFZYC/YA7kutD33bP5b0BKoVGp8M7Gb7t4XLihVke76kSbaXAN+RdFHpmnpF0huAX3SWU67X1t/B9g9L1hWDrbXdO5K+Tj0+uLYj1TLLNwPY/nCBsmIlSDofeBXVxiJ/Am4D3m17y6KF9YikK2zPGtJ2ue0XFCopGqDNR/pzhtxuxG49LfNOqqVEPgh8FNgAeFPRinpruGVSGvM3K2l124uW1xa91doj/eHUGzVvYHte6VoiJB0D3AP8N9W30g8Ba9t+d8GyekbSZUMXjxuuLXqrMUcNoyXpXGBXqv8WVwB3SjrP9sdK1hVBFfIHUw1pFM3ZGeyZVLudrSHpBTy2iNxk4EnFCmuJ1h/pd/pI60laG9g+RNI829mnM2IcSNoLeDfViKvubtaFwHcbtB1kX2r9kT6wqqT1gLcCnyxdTISkr9j+iKQzWHqwATD4y0zYPhY4VtKbbJ9Sup62SehXK2ueCVxg+9eSNgIyZHMASHoO8E/Ahiy9XeKgT+45rv75xaJVjL+fS/oysH19+zzgsM4Q1Rgfre/eicFVb5f4TaqRV4+uw257oEdiSfq57VdK+g/b/1y6nvEi6RTgauDYuumdwJa231iuquZrfehnE5XBJWmu7a1L19Frkq4F9qX6QHsbQ3bLasraNMuYh/C4tuitdO9kE5VBdoak/YBTWXpp5buX/ZSB8CngQGAq8OUh9zViaeXaA5K2s30BgKSXAg8UrqnxcqT/2Oideba3qJdkOLMB/cKNJ2m4lVBte6MJL2YcSDrY9mdK1zFeJG1Jdf7iKVTfZu6mmlF9ZdHCGi5H+tlEZWDZnlG6hvHQtXPWj4fb6L0p3Tt1uG8paXJ9+77CJbVCQh+OrGfiHky1ddtaVF+vYwDUH9SbsvT5mOOW/YyB0Pids6BacoFq2YzpVEOnAbB9WMGyGq/13TsxuCQdAuxAFfo/AXahGnqbfVYHgKSfUW2EPnT01XAfdtEjrQ19SSMus2B76Am06DOSrgK2BC63vaWkdYGjbL+ucGk9IWl/4ATb99S31wb2tP2NooX1iKSrs1nRxBtuFb+2ePJyLtH/HrD9CLC47he+A2jESdza+zqBD2D7L8D7ypXTcxdJen7pItqmtX36tg8tXUOM2Zx6Y5FvU3UR/BW4tGhFvbWKJLn+Oi5pEtW2kE2xHfDuehTWIqoRPM66V+Ortd07HZmc1QySpgOTm7QstqQvUJ3k/CbVCdwPALfY/seSdfWKpA2Ha7f9+4mupU0S+tLJVJOz3kbX5CzbBxQtLJar6dsJSloFeD/wSh5bWvmoemvIiFFJ6Gdy1sDKdoIRK6+1ffpdMjlrcDV9O8GZwL/z+HkITTpZHROszaN3OjqTs/6VanLWtcB/lC0pVtAcSV+WtLGkjST9J83a6/g7wBHAYuAVVEsWHF+0ohh4re/eicElaU2qmdSv4rE+78/avr9oYT3SWUVU0lW2n1+3/dL2y0rXFoOrMV+Fo33qcD+wdB3j6MH6ZO5vJX0Q+CPwjMI1xYDLkX4MnKZvJ9ghaRuqZb6fCnyGajXKz9u+pGRdMdhaHfr1UdS2ti8qXUusOElb254r6eXD3W/7vImuaTzVs41te2HpWmLwtTr0ASRdbPvFpeuIlVPPTj3W9jtK1zJeJM2mOpnbWRbkXuC9g74dZJSV0TtwlqQ3qbOuawyEeoLSFElNWpZgqGOA/WxPtz0d2J/qQyBi1HIit1q7fE1giaQHeGz9j8lly4oVcDNwoaTTgUdH7DRohdSFtn/ZuWH7Aknp4okxaX3o286KmoPr1vqyCs1cGfVSSd8CTqQ6Yb07cG5nN62m7KAVE6v1ffoAknYFtq9vnmv7RyXriZFJOt72OyUdYPurpesZL5LOGeFuZ6mQGI3Wh76kzwHbACfUTXsCc203efz3QJN0LdUuWadT7Zy11PkY23cXKCtiICT0pXnArHozjs6okMuzpnf/kvRhYF+qDVP+yNKh76xNE7FsGb1TeWrX9aeUKiJWjO2v2X4ecIztjWzP6Lok8CNG0PoTuVSrGF5e95+Kqm//oLIlxYqwvS+ApGew9CqUfyhWVI9k4mCMl9Z37wBIWo+qX1/Ar2z/qXBJsQIkvQ74MvAsqv1xN6TaAGezooX1SCYOxnhI9w5g+zbbp9s+LYE/UD4LbAv8xvYMqh2mLixbUk9l4mD0XLp3YpA9bPsuSatIWsX2OZKatBdCJg5GzyX0Y5DdI2kt4HzgBEl3UG040giZOBjjIX36PDpMc126PgSbcDKw6epNVB6g6qZ8O9XIqxNs31W0sB6pu3XeDsyw/RlJGwDr2b60cGkxwFof+pI+BBwC3A48Ujc74/SjNElHUP2b3NH28+ptPc+yvU3h0mKApXsHDgA2acrRYTTKi2xvJelyANt/afiqojEBMnoHbqFapzyi3zxcdz0aQNIUHvs2GjEqOdKHG6lWLvwxsKjT2KDleRupDZuoAF8DTgWeIelw4M3Av5YtKQZdQh/+UF9Wqy8xAGwvkTRF0mq2Hypdz3iwfYKkuVTzDwTsZvu6wmXFgGv9idwYXPVa81tRrbbZxE1UMrIseq71R/p1P+kngM1Yev2WrFXe/xq9icqQkWVLqCdnARlZFqPW+iN9SWcB3wc+DnwA2Au40/Y/Fy0sVpikJ1MNs/1r6Vp6SdJ8qhE8GVkWPZPRO/B020dTTek/z/Z7qdZziT4nafN6OOPVwDWS5kpqxGJrtYwsi55rffcO8HD98zZJf0/VXTC1YD2x4o4EPmb7HABJOwDfBl5SsKYxk/Sx+mpGlkXPJfThs5KeAvwj8HVgMvDRsiXFClqzE/gAts+tl2YYdJ3zE8ONLGt3f2yMWev79GNwSToVuAw4vm56BzDb9m7FiuohSW+xffLy2iJWRmtDX9InbH9e0tcZ5ujJ9ocLlBUroV6L5lBgu7rpfOBQ238pV1XvSLrM9lbLa4tYGW3u3ulMcplTtIoYtTrcG/fhLGkX4DXA+pK+1nXXZBq0dHSU0doj/Yh+JWlLYBZwGPCprrsWAuc05ZtMlNHa0Jd0BiOcFLO96wSWE/E4kp5ANSHrOXXTDbYfHuEpEcvV5u6dL9Y/3wg8E/if+vaewM0lCoqVI+lptu8e0jbD9k2lauqxlwDHUf17FLCBpL1sn1+0qhhorT3S75B0vu3tl9cW/UfShcAutu+rb28KnGR787KV9Ua92NrbbN9Q334OcKLtrctWFoMsM3JhiqSNOjckzQCmFKwnVty/AWdIWkvS1sDJVMM2m+IJncAHsP0b4AkF64kGaHP3TsdHqWY93ljfng68v1w5saJs/7ju9z6LakLTbrZ/W7isXpoj6Wgem4fwdmBuwXqiAVrfvQMgaXXgufXN620vGunxUdYwcyt2pFqy4GZozhyL+t/l/lTzEEQ1D+Eb+fcZY5HQByS9hOoIv3vN8uOKFRQjkrTXSPfbPnaiaokYNK0PfUnHAxsDV1CtWQ7VMr2NOFpssnqdnQdtL6lvTwJWt/23spVF9K+EvnQdsKnb/h9iAEm6BHhVZx19SWsBZ9ke6FU2I8ZTRu9Ua7E/s3QRMSpP7N44pb7+pIL19IykSZK+ULqOaJ6M3oF1gGslXcrSa5ZnRm7/u1/SVrYvA6iHbT5QuKaeqDd+31qS8i00eimhD58uXUCM2keAkyXdWt9eD9i9XDk9dzlwmqSTWXrj9x+UKykGXev79GOw1eP0N6Ea0nh9k9amkfSdYZpdb+kZMSqtDX1JF9jeTtJClh7zLao/rMmFSouVIGlzYFPgiZ22DLeNWLbWhn4MPkmHADtQhf5PgF2AC2y/uWRdvSJpKtUWni+lOjC5ADjA9oKihcVAy+idGGRvBl4J/Mn2e4AtgdXLltRT3wFOB54FrA+cUbdFjFpCPwbZA7YfARZLmgzcAWy0nOcMkim2v2N7cX35LlkMMMYooR+DbI6kpwLfplqI7DLg0qIV9dafJb2jHrM/SdI7gLtKFxWDLX360QiSpgOTbc8rXUuvSJoG/BfwYqo+/Yuo+vR/X7SwGGitD31JbwT+A3gG1cidjN7pc5K2Gun+zmStiHi8hL40H3id7etK1xIrRtI5I9xt2ztOWDERAyahL11o+6Wl64iImAgJfemrVAuu/ZCl197JVPc+J+mJwH5Um4wY+CXwTdsPFi0soo8l9DPVfWBJOglYCPxP3bQnsLbtt5SrqnfqnbPexOM3+DmsVE0x+Fq/4Fo9qScG0ya2t+y6fY6kK4tV03unAfdSDUfNFonRE60P/bqLYG9gM5ZevyVH+v3vcknb2r4EQNKLgAsL19RLU23vXLqIaJZMzoLjqfr0dwLOA6ZSdRlEn5J0laR5wIuAiyTdLOkm4GJg+7LV9dRFkp5fuoholvTpS5fbfoGkeba3qJfqPTPD/vqXpA1Hur8pk5ckXQs8G7iJqnunM4dki6KFxUBrffcO0Fl//Z56md4/UZ04iz7VlFBfAbuULiCaJ6EPR0paG/hXqhUN1wI+VbakiMc+3CQ9g67zTRFj0frunYh+JWlX4EtUSyvfAWwIXGd7s6KFxUBr/YlcSf9Wr9TYub22pM8WLCmi4zPAtsBvbM+g2jugSaOTooDWhz6wi+17Ojds/wV4TblyIh71sO27gFUkrWL7HGBW4ZpiwKVPHyZJWt32IgBJa9Cs3ZdicN0jaS3gfOAESXcAiwvXFAOu9X36kj4B7Eq1DZ2B9wKn2/580cKi9SStCTxINVTz7cBTgBPqo/+IUWl96ANI2oWqv1TAWbbPLFxSRMS4SOhH9BlJF9jeTtJCqm+fj95FNviJMWpt6OcPKyLaqLWhH9GvJE22fZ+kpw13v+27J7qmaI5Wh76kVYB5tjcvXUtEh6Qf2X5tvYicqb59dtj2RoVKiwZo9ZBN249IulLSNNt/KF1PBIDt19Y/Z5SuJZqn1aFfWw+4RtKlwP2dRtu7lispolKvCzWTpfd6OL9cRTHoEvpwaOkCIoYj6R+AA6j2eLiCakmGi4Es+x2j1vplGGyfB9wMPKG+/mvgsqJFRVQOALYBfm/7FcALgDvLlhSDrvWhL+l9wP8C36qb1gd+WKygiMc8aPtBqDZJt309sEnhmmLApXsH9gdeCPwKwPZv6/XLI0pbUK8A+0PgbEl/AW4tWlEMvIQ+LLL9kFSNipO0KktP1ooowvYb6quflnQO1do7PytYUjRAQh/Ok/QvwBqS/g7YDzijcE0RSDoM+CVwUX2+KWLMWj05Cx6doLU38GqqSTBnAke57f9hojhJ7wW2A14MLKT6ADjf9mlFC4uB1vrQj+h3kp4JvBX4OLC27ScXLikGWEbvSK+VdLmkuyXdJ2mhpPtK1xUh6ShJFwFHUHXFvhlYu2xVMejSpw9fAd4IXJUunegzTwcmAfcAdwN/tp2ds2JMEvpwC3B1Aj/6TWf0jqTnATsB50iaZHtq2cpikCX04RPATySdByzqNNr+crmSIqquR+BlwPZU3Tq/oDqZGzFqCX04HPgr1YJWqxWuJaLbLlSbon/VdiZlRU+0fvSOpDm2Z5euIyJiIrR+9A7wf5JeXbqIiKEkbSvp15L+KukhSUsysizGKkf61R65a1L15z9M9siNPiFpDrAHcDIwG3gX8GzbnyxaWAy01vfpZ6JL9DPb8+sRO0uA79Tj9iNGrfWhD9mdKPrW3yStBlwh6fPAbVTfSiNGLd07y9idyHZ2J4qiJG0I3E41quyjVKtsfsP2/KKFxUBL6EtXUe1OdIntWZKeCxxqe/fCpUVE9Fy6d+rdiSQ9ujuRpOxOFMVJeinwaWBDuv5WbW9UqqYYfAn97E4U/etoqm6ducCSwrVEQ7S+e6ebpJdT705k+6HS9US7SfqV7ReVriOaJaEf0WckbVVffSvVKps/YOl1oS4rUVc0Q0I/os/U++EuizOyLMYioR8R0SJZeyciokUS+hERLZLQj4hokYR+RJ+S9CRJB0v6dn17Zr2bVsSoJfQj+td3qIZqvri+vQD4bLlyogkS+hH9a2Pbn6fa5wHbD1Dt9xAxagn9iP71kKQ1AANI2piuSVoRo5G1dyL616eBnwEbSDoBeCnw7pIFxeDL5KyIPibp6VR7PIhq+e8/Fy4pBlyO9CP6lKTTgROB023fX7qeaIb06Uf0ry8BLwOulXSypDdLeuLynhQxknTvRPQ5SZOAHYH3ATvbnly4pBhg6d6J6GP16J3XAbsDWwHHlq0oBl2O9CP6lKTvAy+iGsFzEnCu7UfKVhWDLqEf0ack7QycbTtbJUbPJPQj+oykHW3/QtIbh7vf9g8muqZojvTpR/SflwO/oOrLH8pU2ydGjEqO9CP6lKQZtm9aXlvEysg4/Yj+dcowbf874VVEo6R7J6LPSHousBnwlCH9+pOBTM6KMUnoR/SfTYDXAk9l6X79hVQTtCJGLX36EX1K0ottX1y6jmiWhH5En6rX2dmbqqvn0W4d2+8tVlQMvJzIjehfxwPPBHYCzgOmUnXxRIxajvQj+pSky22/QNI821tIegJwpu0dS9cWgytH+hH96+H65z2SNgeeAkwvV040QUbvRPSvIyWtDRwMnA6sBXyqbEkx6NK9ExHRIjnSj+hTklYH3kTVpfPo36rtw0rVFIMvoR/Rv04D7gXmAosK1xINke6diD4l6Wrbm5euI5olo3ci+tdFkp5fuoholhzpR/QpSdcCzwZuoureEWDbWxQtLAZaQj+iT0nacLh227+f6FqiORL6EREtkj79iIgWSehHRLRIQj8iokUS+hEjUCV/J9EY+cccMYSk6ZKuk/QN4DLgaElzJF0j6dCux20j6SJJV0q6VNKTJU2S9AVJv5Y0T9L7y72TiMfLMgwRw9sEeI/t/SQ9zfbdkiYBP5e0BXA98H1gd9u/ljQZeIBqp6t7bW9Tr51zoaSzbN9U7J1EdEnoRwzv97Yvqa+/VdI+VH8v6wGbAgZus/1rANv3AUh6NbCFpDfXz30KMJNqglVEcQn9iOHdDyBpBvBxYBvbf5H0Xar9akUV/EMJ+JDtMyeq0IiVkT79iJFNpvoAuFfSusAudfv1wLMkbQNQ9+evCpwJ7FtvbYik50has0DdEcPKkX7ECGxfKely4BrgRuDCuv0hSbsDX5e0BlV//quAo6jWv79MkoA7gd0KlB4xrCzDEBHRIuneiYhokYR+RESLJPQjIlokoR8R0SIJ/YiIFknoR0S0SEI/IqJFEvoRES3y/wEoqc3TsAgn0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# bar chart of number of patients of each race\n",
    "race_counts.plot(kind=\"bar\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The semicolon at the end of the command suppresses text output at the top of the plot.\n",
    "\n",
    "You can see the other types of plots available by looking at the help documentation for `.plot()`.\n",
    "A few more examples of this type of plotting approach is available [here](https://towardsdatascience.com/5-minute-guide-to-plotting-with-pandas-e8c0f40a1df4).\n",
    "\n",
    "> #### Challenge-total-count \n",
    "Create a new object called `total_count` that counts the number of samples for each cancer type (disease)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing data\n",
    "\n",
    "The final section in this lesson will help you compare a few ways to handle missing data across a data frame.\n",
    "\n",
    "We can use `isnull` to show us where missing data are found in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   primary_diagnosis  tumor_stage  age_at_diagnosis  vital_status  morphology  \\\n",
       "0              False        False             False         False       False   \n",
       "1              False        False             False         False       False   \n",
       "2              False        False             False         False       False   \n",
       "3              False        False             False         False       False   \n",
       "4              False        False             False         False       False   \n",
       "\n",
       "   days_to_death  state  tissue_or_organ_of_origin  days_to_birth  \\\n",
       "0          False  False                      False          False   \n",
       "1          False  False                      False          False   \n",
       "2          False  False                      False          False   \n",
       "3           True  False                      False          False   \n",
       "4          False  False                      False          False   \n",
       "\n",
       "   site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "0                        False                    True               False   \n",
       "1                        False                    True               False   \n",
       "2                        False                   False               False   \n",
       "3                        False                   False               False   \n",
       "4                        False                    True                True   \n",
       "\n",
       "   years_smoked  gender  year_of_birth   race  ethnicity  year_of_death  \\\n",
       "0          True   False          False  False      False          False   \n",
       "1          True   False          False  False      False          False   \n",
       "2          True   False          False  False      False           True   \n",
       "3          True   False          False  False      False           True   \n",
       "4          True   False          False  False      False          False   \n",
       "\n",
       "   bcr_patient_barcode  disease  \n",
       "0                False    False  \n",
       "1                False    False  \n",
       "2                False    False  \n",
       "3                False    False  \n",
       "4                False    False  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.isnull(clinical_df).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any field showing `True` represents a cell with missing data\n",
    "(missing data in the original data frame are shown as `NaN`).\n",
    "\n",
    "The effects of these missing data include multiple complications later in analysis,\n",
    "including code errors and misinterpreting statistical output.\n",
    "One basic example of this occurs from attempting to change the data type in year of birth \n",
    "(from float to integer\n",
    "\n",
    "`clinical_df.year_of_birth = clinical_df.year_of_birth.astype(\"int64\"))`\n",
    "\n",
    "That code would result in a `ValueError`:\n",
    "    \n",
    "`ValueError: Cannot convert non-finite values (NA or inf) to integer`\n",
    "\n",
    "This means `NA` values are prohibiting data conversion for this column.\n",
    "\n",
    "If we used `dropna` to remove all rows with missing data,\n",
    "and then use `len` to assess how many rows are remaining:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract all rows WITHOUT missing data\n",
    "len(clinical_df.dropna())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This leaves us with only 50 of our original 6832 cases!\n",
    "This is because most of our cases have missing data in one or more field.\n",
    "\n",
    "We'll explore two methods for handling missing data that allow us to retain the specific data we need.\n",
    "\n",
    "### Replacing data in copied data frame\n",
    "\n",
    "The first approach to handling missing data is to copy the existing data frame and then replacing missing data.\n",
    "\n",
    "As we discussed in the last lesson,\n",
    "you need to apply the `copy()` method to create another data frame \n",
    "(rather than referencing the prior object):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new copy of data frame\n",
    "clinical_copy = clinical_df.copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can use `pandas` functions to replace `NaN` with a different value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace missing values with 0\n",
    "clinical_copy.year_of_birth = clinical_copy.year_of_birth.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach allows us to retain all observations in the original data frame,\n",
    "and is one method sometimes used (for example, in machine learning)\n",
    "to retain as much data as possible.\n",
    "\n",
    "However, this approach does alter the statistical properties of the remaining data.\n",
    "If we compare the average year of birth for our original data frame and the copied frame with missing data replaced by 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "original = 1947.560942659862\n",
      "copied = 1899.0999707259953\n"
     ]
    }
   ],
   "source": [
    "print(\"original =\", clinical_df.year_of_birth.mean())\n",
    "print(\"copied =\", clinical_copy.year_of_birth.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find the average is vastly different.\n",
    "Therefore, this approach isn't good for projects that rely on summary statistics.\n",
    "\n",
    "A slight modification would be to replace all missing values with the mean year of birth instead (`clinical_copy.year_of_birth.fillna(clinical_copy.year_of_birth.mean())`).\n",
    "We won't try to do this here (since we've already replaced those missing data!),\n",
    "but can provide another option depending on your need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Masking missing data\n",
    "\n",
    "The second approach to handling missing data is masking,\n",
    "which excludes particular subsets of data that contain missing values.\n",
    "We'll explore a few approaches to masking\n",
    "while constructing the two data frames we'll be using next class for data visualization.\n",
    "\n",
    "The first dataset we'll create has missing data removed for cigarettes per day \n",
    "and age at diagnosis.\n",
    "\n",
    "> Because masking doesn't involve modifying the original data,\n",
    "only filtering out whole rows and columns,\n",
    "we don't need to copy a new object.\n",
    "\n",
    "We can use `dropna` with the `subset` argument to exclude \n",
    "missing values for only one column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# exclude missing data in only cigarettes per day\n",
    "smoke_complete = clinical_df.dropna(subset = [\"cigarettes_per_day\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assigned this to (referenced) a new object so we can continue with our filters.\n",
    "\n",
    "It's also possible to use `isnull` to subset data,\n",
    "though we don't do so here:\n",
    "\n",
    "`clinical_df[-pd.isnull(clinical_df.cigarettes_per_day)]`\n",
    "\n",
    "Next, we can apply a filter that will remove missing data for age at diagnosis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# apply additional filter for age at diagnosis\n",
    "smoke_complete = smoke_complete[smoke_complete.age_at_diagnosis > 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While we could've used `dropna`,\n",
    "it's useful to know you can apply additional filters depending on the specific data involved.\n",
    "Here, missing data do not pass the filter of being greater than 0,\n",
    "and there are no cases in the dataset with an age at diagnosis that is less than 0,\n",
    "so this effectively removes only missing values.\n",
    "\n",
    "Once we are satisfied with our filtering,\n",
    "we may want to retain our data for later.\n",
    "This could be so we can load it into another program,\n",
    "or share with another collaborator,\n",
    "both of which require us to access the data outside our notebook.\n",
    "\n",
    "We can save our filtered data frame as spreadsheet-style data to a csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save filtered data to file\n",
    "smoke_complete.to_csv(\"data/smoke_complete.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `index=False` argument prevents the row's index values from being printed before the first column.\n",
    "\n",
    "You can see the new file in your project directory (in the data subfolder).\n",
    "This is the first of two datasets we'll use in the next class on data visualization.\n",
    "\n",
    "Now, we'll move on and use our masking skills to create the second dataframe we'll use in the next lesson.\n",
    "\n",
    "First, we can reference the original data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reference original data\n",
    "birth_reduced = clinical_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### Challenge-filter\n",
    "Mask (filter out) missing data from `birth_reduced` for `year_of_birth` and `vital_status`. (Hint: check to see what categories exist for the latter column).\n",
    "\n",
    "There are several ways to approach the challenge above.\n",
    "First, you might try to use `dropna` to remove multiple columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "birth_reduced = birth_reduced.dropna(subset = [\"year_of_birth\", \"vital_status\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this executes without a problem, \n",
    "you can check to see what categories are present in `vital_status`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['dead', 'alive', 'not reported'], dtype=object)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show categories\n",
    "pd.unique(birth_reduced.vital_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This reveals there are some missing data encoded in a category called `not_reported`.\n",
    "This means we need to add an additional filter to remove that category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "birth_reduced = birth_reduced[birth_reduced.vital_status != \"not reported\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check to see that it worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['dead', 'alive'], dtype=object)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.unique(birth_reduced.vital_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one remaining manipulation we'll cover with our `birth_reduced` dataset.\n",
    "\n",
    "We have many different cancer types in this dataset, \n",
    "but we would like to retain only those cancers with a large number of cases represented.\n",
    "We can apply a filter on this criteria,\n",
    "but will need to start by counting how many cases for each cancer type exist in our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "disease\n",
       "BLCA     394\n",
       "BRCA    1094\n",
       "CESC     305\n",
       "COAD     459\n",
       "GBM      594\n",
       "LGG      515\n",
       "LUSC     958\n",
       "MESO      87\n",
       "PRAD     469\n",
       "READ     170\n",
       "SKCM     459\n",
       "STAD     437\n",
       "THYM     123\n",
       "UCEC     539\n",
       "UCS       57\n",
       "Name: disease, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# count number of samples for each cancer type\n",
    "dis_counts = birth_reduced.groupby(\"disease\").disease.count()\n",
    "dis_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output above doesn't render nicely because we've used `groupby`.\n",
    "The reason for that is because the grouped output doens't include the row labels of index values. \n",
    "we'll need to reset the index values so our `dis_counts` object is properly formatted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>disease</th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BLCA</td>\n",
       "      <td>394</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BRCA</td>\n",
       "      <td>1094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>CESC</td>\n",
       "      <td>305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>COAD</td>\n",
       "      <td>459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GBM</td>\n",
       "      <td>594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LGG</td>\n",
       "      <td>515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>LUSC</td>\n",
       "      <td>958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>MESO</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>PRAD</td>\n",
       "      <td>469</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>READ</td>\n",
       "      <td>170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>SKCM</td>\n",
       "      <td>459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>STAD</td>\n",
       "      <td>437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>THYM</td>\n",
       "      <td>123</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>UCEC</td>\n",
       "      <td>539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>UCS</td>\n",
       "      <td>57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   disease  counts\n",
       "0     BLCA     394\n",
       "1     BRCA    1094\n",
       "2     CESC     305\n",
       "3     COAD     459\n",
       "4      GBM     594\n",
       "5      LGG     515\n",
       "6     LUSC     958\n",
       "7     MESO      87\n",
       "8     PRAD     469\n",
       "9     READ     170\n",
       "10    SKCM     459\n",
       "11    STAD     437\n",
       "12    THYM     123\n",
       "13    UCEC     539\n",
       "14     UCS      57"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reset index to default\n",
    "dis_counts = dis_counts.reset_index(name=\"counts\")\n",
    "dis_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index values are now shown in bold to the left of the table.\n",
    "\n",
    "Next, we can filter the new object so it only includes cancer types with many cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>disease</th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BRCA</td>\n",
       "      <td>1094</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GBM</td>\n",
       "      <td>594</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LGG</td>\n",
       "      <td>515</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>LUSC</td>\n",
       "      <td>958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>UCEC</td>\n",
       "      <td>539</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   disease  counts\n",
       "1     BRCA    1094\n",
       "4      GBM     594\n",
       "5      LGG     515\n",
       "6     LUSC     958\n",
       "13    UCEC     539"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keep only diseases with many observations\n",
    "dis_counts = dis_counts[dis_counts.counts > 500]\n",
    "dis_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`dis_counts` is a small table listing only the cancer types we'd like to keep in the first column,\n",
    "with the number of observations for each cancer type in the second column.\n",
    "We'll now use a special filter, `isin`,\n",
    "to compare each item in the `disease` column in `birth_reduced` to the `disease` column in `dis_counts`,\n",
    "keeping only the observations for cancer types that match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>primary_diagnosis</th>\n",
       "      <th>tumor_stage</th>\n",
       "      <th>age_at_diagnosis</th>\n",
       "      <th>vital_status</th>\n",
       "      <th>morphology</th>\n",
       "      <th>days_to_death</th>\n",
       "      <th>state</th>\n",
       "      <th>tissue_or_organ_of_origin</th>\n",
       "      <th>days_to_birth</th>\n",
       "      <th>site_of_resection_or_biopsy</th>\n",
       "      <th>days_to_last_follow_up</th>\n",
       "      <th>cigarettes_per_day</th>\n",
       "      <th>years_smoked</th>\n",
       "      <th>gender</th>\n",
       "      <th>year_of_birth</th>\n",
       "      <th>race</th>\n",
       "      <th>ethnicity</th>\n",
       "      <th>year_of_death</th>\n",
       "      <th>bcr_patient_barcode</th>\n",
       "      <th>disease</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>24477.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>371.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-24477.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10.958904</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1936.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3406</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>26615.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>136.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-26615.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.191781</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1931.0</td>\n",
       "      <td>asian</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>TCGA-18-3407</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage ib</td>\n",
       "      <td>28171.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>2304.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-28171.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>2099.0</td>\n",
       "      <td>1.643836</td>\n",
       "      <td>NaN</td>\n",
       "      <td>female</td>\n",
       "      <td>1927.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3408</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C34.1</td>\n",
       "      <td>stage ia</td>\n",
       "      <td>27154.0</td>\n",
       "      <td>alive</td>\n",
       "      <td>8083/3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>-27154.0</td>\n",
       "      <td>C34.1</td>\n",
       "      <td>3747.0</td>\n",
       "      <td>1.095890</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>white</td>\n",
       "      <td>not hispanic or latino</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TCGA-18-3409</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C34.3</td>\n",
       "      <td>stage iib</td>\n",
       "      <td>29827.0</td>\n",
       "      <td>dead</td>\n",
       "      <td>8070/3</td>\n",
       "      <td>146.0</td>\n",
       "      <td>live</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>-29827.0</td>\n",
       "      <td>C34.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>male</td>\n",
       "      <td>1923.0</td>\n",
       "      <td>not reported</td>\n",
       "      <td>not reported</td>\n",
       "      <td>2004.0</td>\n",
       "      <td>TCGA-18-3410</td>\n",
       "      <td>LUSC</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  primary_diagnosis tumor_stage  age_at_diagnosis vital_status morphology  \\\n",
       "0             C34.1    stage ia           24477.0         dead     8070/3   \n",
       "1             C34.1    stage ib           26615.0         dead     8070/3   \n",
       "2             C34.3    stage ib           28171.0         dead     8070/3   \n",
       "3             C34.1    stage ia           27154.0        alive     8083/3   \n",
       "4             C34.3   stage iib           29827.0         dead     8070/3   \n",
       "\n",
       "   days_to_death state tissue_or_organ_of_origin  days_to_birth  \\\n",
       "0          371.0  live                     C34.1       -24477.0   \n",
       "1          136.0  live                     C34.1       -26615.0   \n",
       "2         2304.0  live                     C34.3       -28171.0   \n",
       "3            NaN  live                     C34.1       -27154.0   \n",
       "4          146.0  live                     C34.3       -29827.0   \n",
       "\n",
       "  site_of_resection_or_biopsy  days_to_last_follow_up  cigarettes_per_day  \\\n",
       "0                       C34.1                     NaN           10.958904   \n",
       "1                       C34.1                     NaN            2.191781   \n",
       "2                       C34.3                  2099.0            1.643836   \n",
       "3                       C34.1                  3747.0            1.095890   \n",
       "4                       C34.3                     NaN                 NaN   \n",
       "\n",
       "   years_smoked  gender  year_of_birth          race               ethnicity  \\\n",
       "0           NaN    male         1936.0         white  not hispanic or latino   \n",
       "1           NaN    male         1931.0         asian  not hispanic or latino   \n",
       "2           NaN  female         1927.0         white  not hispanic or latino   \n",
       "3           NaN    male         1930.0         white  not hispanic or latino   \n",
       "4           NaN    male         1923.0  not reported            not reported   \n",
       "\n",
       "   year_of_death bcr_patient_barcode disease  \n",
       "0         2004.0        TCGA-18-3406    LUSC  \n",
       "1         2003.0        TCGA-18-3407    LUSC  \n",
       "2            NaN        TCGA-18-3408    LUSC  \n",
       "3            NaN        TCGA-18-3409    LUSC  \n",
       "4         2004.0        TCGA-18-3410    LUSC  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# extract values for frequently occurring cancers\n",
    "birth_reduced = birth_reduced[birth_reduced[\"disease\"].isin(dis_counts.disease)]\n",
    "birth_reduced.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our resulting data table includes all observations from `birth_reduced` with cancer types that match those specified in `dis_counts`.\n",
    "\n",
    "Finally, we can also write these data to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write data to csv\n",
    "birth_reduced.to_csv(\"data/birth_reduced.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrapping up\n",
    "\n",
    "Today, learned some additional data manipulation tools available in `pandas`,\n",
    "including grouping data and dealing with missing data.\n",
    "\n",
    "We'll wrap up this course in our next lesson by learning how to create publication-quality images using the `plotnine` package.\n",
    "To prepare for next class,\n",
    "make sure you have `plotnine` installed.\n",
    "You can check by loading the package:\n",
    "\n",
    "`import plotnine`\n",
    "\n",
    "If you receive an error saying the package is not found,\n",
    "please see the instructions at the [beginning of Class 4](class4.ipynb) for installing it via your notebook.\n",
    "\n",
    "## Extra exercises\n",
    "\n",
    "Answers to all challenge exercises are available [here](https://fredhutchio.github.io/python_intro/solutions/). \n",
    "\n",
    "> #### Challenge-subset\n",
    "Assign a new object called `clinical_subset` containing data from patients who are alive and of Hispanic or Latino ethnicity.\n",
    "(Hint: use `unique()` to see a list of values in a column.\n",
    "\n",
    "> #### Challenge-subset2\n",
    "Use the `clinical_subset` object to create a bar graph showing the number of patients for each category of `primary_diagnosis`.\n",
    "Label the title and axes appropriately.\n",
    "(Hint: Look at the Matplotlib documentation for help in labeling)\n",
    "\n",
    "> #### Challenge-disease-plot\n",
    "Create a bar graph showing the number of patients for each disease represented in `clinical_df`.\n",
    "\n",
    "> #### Challenge-alive-yob\n",
    "Print the number of patients in the dataset listed as alive with a known year of birth."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}