1 - Data Structures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Introduction to Pandas Data Structures\n",
    "\n",
    "This lesson introduces [Pandas](https://pandas.pydata.org/) a library for data analysis and manipulation for the Python programming language. Pandas provides a set of data structures, *DataFrames* and *Series* for representing and manipulating data fast and efficiently.\n",
    "\n",
    "## Learning Objectives\n",
    "\n",
    "After completing this lesson you will:\n",
    "\n",
    "- Know the three Pandas data structures: Series, DataFrames, and Indexes.\n",
    "- Understand what kinds of data can be represented in each of the Pandas data structures.\n",
    "- Be able to create Pandas data structures from vanilla Python data structures.\n",
    "\n",
    "\n",
    "## Data used in this lesson\n",
    "\n",
    "While this lesson does not use any external data files, there will be data represented as Markdown tables in this lesson. Be ready to do some typing because you will need to copy these data tables into your code. Don't worry, the data tables are not large. 😊\n",
    "\n",
    "This lesson uses a snapshot of data from the [2010 Pittsburgh Neighborhood Profiles](https://ucsur.pitt.edu/files/census/UCSUR_SF1_NeighborhoodProfiles_July2011.pdf). The information represented in these data includes neighborhood name, the population, and the area in square miles.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Pandas library\n",
    "\n",
    "Run the code cell below to load the Pandas library into memory. Note the `as pd`, this loads Pandas with an *alias*, \"pd\", that is faster to write than the word \"pandas.\" This alias is a common convention used in the Python Data Science community."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Pandas Data Structures\n",
    "\n",
    "To understand Pandas, which takes time, it is helpful to start with the data structures it uses to represent data in memory:\n",
    "* Series - For one dimensional data (think lists) \n",
    "* Dataframe - For two dimensional data (think spreadsheets)\n",
    "* Index - For naming, selecting, and transforming data within a Pandas Series or Dataframe (think column and row names)\n",
    "\n",
    "The Pandas data structures are built on top of data structures from [Numpy](https://numpy.org/), a powerful numeric computing library for Python. We won't cover Numpy in this less, but it is helpful to know the library underpinning Pandas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series\n",
    "\n",
    "The first data structure you will encounter is the *Series*. Pandas Series is a:\n",
    "* A one-dimensional array of indexed data\n",
    "* A blend of a Python lists and dictionaries\n",
    "* A basic building block of the DataFrame\n",
    "\n",
    "## How to create a Series\n",
    "\n",
    "To create a Series you must use the `Series()` function and pass it your list-like data as an argument. Run the code cell below to create a Pandas Series from a Python List."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a regular Python list of floating point numbers\n",
    "my_list = [0.25, 0.5, 0.75, 1.0]\n",
    "\n",
    "# Transform that list into a Series\n",
    "data = pd.Series(my_list)\n",
    "\n",
    "# Display the data in the series\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Parts of a Series\n",
    "\n",
    "1. Look at the output above, you can see the data from the list but you can also see some additional information. Specifically, there are a set of numbers (0-3) and a line that says `dtype`\n",
    "    - What do you think these additional pieces of information represent?\n",
    "2. In the code cell below, copy the code above and modify the variable `my_list` to be a list of integers instead of floating point numbers.\n",
    "    - Look at the output, what has changed? What is the `dtype` or *data type* of the Series now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a series of integer numbers\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Parts of a Series\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "# Create a regular Python list of floating point numbers\n",
    "my_list = [25, 5, 75, 1]\n",
    "\n",
    "# Transform that list into a Series\n",
    "data = pd.Series(my_list)\n",
    "\n",
    "# Display the data in the series\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Task - Series Data Types\n",
    "\n",
    "1. Copy and modify the code above to make `my_list` a mixture of integers and floating point numbers.\n",
    "    - What happens to the data and the `dtype` of the Series?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Series Data Types\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "# Create a regular Python list of floating point numbers\n",
    "my_list = [2.5, 5, 7.5, 1]\n",
    "\n",
    "# Transform that list into a Series\n",
    "data = pd.Series(my_list)\n",
    "\n",
    "# Display the data in the series\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike Python lists, a Pandas Series must have all data be of the same type. When you mix integers and floating point numbers Pandas will convert your data to the  data type possible."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Task - Creating a Series\n",
    "\n",
    "| Neighborhood              | Population | Area |\n",
    "| ------------------------- | ---------- | ---- |\n",
    "| East Liberty              | 5869       | 0.58 |\n",
    "| Greenfield                | 7294       | 0.78 |\n",
    "| Squirrel Hill North       | 11363      | 1.22 |\n",
    "| Bloomfield                | 8442       | 0.70 |\n",
    "| Central Business District | 3629       | 0.65 |\n",
    "|Data source [2010 Pittsburgh Neighborhood Profiles](https://ucsur.pitt.edu/files/census/UCSUR_SF1_NeighborhoodProfiles_July2011.pdf)|\n",
    "\n",
    "\n",
    "1.  From the data table above, create a python dictionary representing the population of a few Pittsburgh neighborhoods. The dictionary values should be the population data and the keys should be the neighborhood name.\n",
    "2. Use the `Series()` function to create a series called `neighborhood_series`, using the Python dictionary as the data parameter.\n",
    "3. Look at the resulting output, consider how has the data from the dictionary, which is an unordered structure with keys and values, been transformed into an ordered, one-dimensional Series?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Creating a Series\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Answer\n",
    "\n",
    "neighborhood_dictionary = {\n",
    "    \"East Liberty\":5869,\n",
    "    \"Greenfield\":7294,\n",
    "    \"Squirrel Hill North\":11363,\n",
    "    \"Bloomfield\":8442,\n",
    "    \"Central Business District\":3629\n",
    "}\n",
    "\n",
    "neighborhood_series = pd.Series(neighborhood_dictionary)\n",
    "neighborhood_series\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Components of a Series\n",
    "\n",
    "A Series has several different parts for storing both data and metadata about the data being represented by the series. There are four components of a Series that are most relevant for this lesson:\n",
    "1. A sequence of ordered values.\n",
    "2. An explicit named index for each value.\n",
    "3. An implicit numerical index for each value.\n",
    "4. A data type of all the values.\n",
    "\n",
    "So if you look at the results of the previous task above you will see:\n",
    "1. A sequence of ordered values representing neighborhood populations.\n",
    "2. An explicit named index indicating the name of the neighborhood.\n",
    "3. An implicit numerical index, which isn't shown, but is there if we want to use it.\n",
    "3. A data type, `int64` indicating the population values are Integers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Values from a Series\n",
    "\n",
    "You can index into a series using the same syntax as with Python lists. Using Python's *indexing* and *slicing* notation you can extract a specific values or set of values from a Series, just as you would with a list. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Indexing and Slicing a Series\n",
    "\n",
    "1. From the `neighborhood_series` data you created in the previous, using Python's indexing syntax extract the first element of the Series, the population of the first neighborhood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Indexing and Slicing a Series\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Indexing and Slicing a Series Again\n",
    "\n",
    "1. From the `neighborhood_series`, using numerical indexing to select the value for the neighborhood \"Squirrel Hill North\" using its *numerical position* in the Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Indexing and Slicing a Series Again\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Slicing a Series\n",
    "* From the `neighborhood_series`, use slicing syntax to create a subset of the 2nd through 4th elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Slicing a Series\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[2:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Indexing the End of a Series\n",
    "\n",
    "1. If you didn't know how many values were in the series, how would you using indexing to get the last item?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Indexing the End of a Series\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Named Index \n",
    "\n",
    "Pandas series can also behave like a Python dictionary, that is, you can look up values by their names rather than position in the sequence.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tasks - Extracting Values by Name\n",
    "\n",
    "* Extract the value for \"Greenfield\" from the series `neighborhood_series`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Extracting Values by Name\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[\"Greenfield\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Extract a Slice\n",
    "\n",
    "1. Extract a subset of values, that is a slice, of from \"Greenfield\" to the \"Central Business District\" but use the names, not the numerical positions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Extract a Slice\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhood_series[\"Greenfield\":\"Central Business District\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how with named indexing the first and last are inclusive, but with numerical indexing the the first number is inclusive and the second number is exclusive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataframes\n",
    "\n",
    "Dataframes are a two-dimensional data structure, just like an Excel spreadsheet. They have rows and columns. Each column in a DataFrame is a Series under the hood.\n",
    "\n",
    "Let's start with the Pittsburgh neighborhood data table.\n",
    "\n",
    "| Neighborhood              | Population | Area |\n",
    "| ------------------------- | ---------- | ---- |\n",
    "| East Liberty              | 5869       | 0.58 |\n",
    "| Greenfield                | 7294       | 0.78 |\n",
    "| Squirrel Hill North       | 11363      | 1.22 |\n",
    "| Bloomfield                | 8442       | 0.70 |\n",
    "| Central Business District | 3629       | 0.65 |\n",
    "|Data source [2010 Pittsburgh Neighborhood Profiles](https://ucsur.pitt.edu/files/census/UCSUR_SF1_NeighborhoodProfiles_July2011.pdf)|\n",
    "\n",
    "And now lets create three lists from each of the columns in the data table above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the first list of neighborhoods\n",
    "neighborhoods = [\"East Liberty\", \"Greenfield\", \"Squirrel Hill North\", \"Bloomfield\", \"Central Business District\"]\n",
    "neighborhoods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Create Lists of Data\n",
    "\n",
    "1. Create two more lists, one for the population values (called `population` and one for the area values( called `area`). Make sure the order of the values matches the `neighborhoods` series above!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Create Lists of Data\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "population = [5869, 7294, 11363, 8442, 3629]\n",
    "area = [0.58, 0.78, 1.22, 0.70, 0.65]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating DataFrames using Python data structures\n",
    "\n",
    "Creating a DataFrame from Python means you need to compose your data into a two dimensional structure using a combination of Python lists and Dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a Dataframe from a dictionary of lists\n",
    "\n",
    "# create a dictionary with \n",
    "data = {\"neighborhoods\":neighborhoods,\n",
    "        \"population\": population,\n",
    "        \"area\": area}\n",
    "\n",
    "pgh_neighborhood_info = pd.DataFrame(data)\n",
    "pgh_neighborhood_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look! We have re-created the table above but now it is loaded as a Pandas DataFrame. Notice how the DataFrame has been rendered as a pretty HTML table, that is a benefit of Jupyter Notebooks.\n",
    "\n",
    "There are other ways to create DataFrames, for example maybe your data is more row-centric and you have a list of lists.\n",
    "\n",
    "```python\n",
    "[['East Liberty', 5869, 0.58],\n",
    " ['Greenfield', 7294, 0.78],\n",
    " ['Squirrel Hill North', 11363, 1.22],\n",
    " ['Bloomfield', 8442, 0.7],\n",
    " ['Central Business District', 3629, 0.65]]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Creating a Dataframe from a list of lists\n",
    "\n",
    "1. Copy the list of lists in the code above and save it to a variable called \"data\".\n",
    "2. Create a new list with three string values: \"neighborhood\", \"population\", and \"area\" and save it to a variable called \"column_names\".\n",
    "3. Recreate the `pgh_neighborhood_info` DataFrame using the `pd.Dataframe()` function. Put the `data` variable as the first positional argument and the `column_names` as the value for the `columns` keyword argument.  \n",
    "4. Display the DataFrame and see if it is the same as the output above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Creating a DataFrame from a list of lists\n",
    "\n",
    "Click on the ellipsis (...) below to see the answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# create the list of lists\n",
    "data = [['East Liberty', 5869, 0.58],\n",
    " ['Greenfield', 7294, 0.78],\n",
    " ['Squirrel Hill North', 11363, 1.22],\n",
    " ['Bloomfield', 8442, 0.7],\n",
    " ['Central Business District', 3629, 0.65]]\n",
    "\n",
    "# create a list of column names\n",
    "column_names = [\"neighborhood\", \"population\", \"area\"]\n",
    "\n",
    "# create the dataframe with positional and keyword argument\n",
    "pgh_neighborhood_info = pd.DataFrame(data, columns=column_names)\n",
    "pgh_neighborhood_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slicing a DataFrame\n",
    "\n",
    "You can use indexing notation to select individual or groups of columns from a DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pgh_neighborhood_info['area']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the output is not a list, but a Series. When we created the DataFrame from the list of lists the inner lists were converted to Series and the outer list structured the DataFrame. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Selecting Columns of a Dataframe\n",
    "\n",
    "1. Slice out the `population` column from the `pgh_neighborhood_info` DataFrame.\n",
    "2. Try to slice both the `neighborhood` and `population` columns, what happens?\n",
    "3. Create a python list with two strings representing the names of the columns and save that to a variable (you pick the name).\n",
    "4. Pass your newly created variable to DataFrame within the slicing notation (that is, put it in the square brackets). What happens now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Selecting Columns of a DataFrame\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer 1\n",
    "pgh_neighborhood_info['population']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer 2\n",
    "pgh_neighborhood_info['neighborhood','population']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# asnwer 3 & 4\n",
    "foo = [\"neighborhood\", \"population\"]\n",
    "pgh_neighborhood_info[foo]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why do we get a key error when we try to slice two column names?\n",
    "If we don't wrap our values in a list, pandas will try to find a single column with a complex name `('neighborhood', 'population')`, which is possible, rather than two columns with the names \"neighborhood\" and \"population.\"\n",
    "\n",
    "This wart is an artifact of the way in which column names are treated in Pandas. As an Index!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Index\n",
    "\n",
    "* Pandas `Series` and `DataFrames` are containers for data\n",
    "* The Index (and Indexing) is the mechanism to make that data retrievable\n",
    "* In a `Series` the index is the key to each value in the list\n",
    "* In a `DataFrame` the index is the column names, but there is also an index for each row\n",
    "* Indexing allows you to merge or join disparate datasets together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the Pittsburgh neighborhood data table.\n",
    "\n",
    "| Neighborhood              | Population | Area |\n",
    "| ------------------------- | ---------- | ---- |\n",
    "| East Liberty              | 5869       | 0.58 |\n",
    "| Greenfield                | 7294       | 0.78 |\n",
    "| Squirrel Hill North       | 11363      | 1.22 |\n",
    "| Bloomfield                | 8442       | 0.70 |\n",
    "| Central Business District | 3629       | 0.65 |\n",
    "|Data source [2010 Pittsburgh Neighborhood Profiles](https://ucsur.pitt.edu/files/census/UCSUR_SF1_NeighborhoodProfiles_July2011.pdf)|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Create a Population Series\n",
    "\n",
    "1. Create a Python dictionary with neighborhood names as keys and population as values\n",
    "2. Use the `pd.Series` function to create a Pandas Series called `population_series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Create a Population Series\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Answer\n",
    "\n",
    "# create a dictionary of population\n",
    "population_dictionary = {\n",
    "    \"East Liberty\":5869,\n",
    "    \"Greenfield\":7294,\n",
    "    \"Squirrel Hill North\":11363,\n",
    "    \"Bloomfield\":8442,\n",
    "    \"Central Business District\":3629\n",
    "}\n",
    "\n",
    "population_series = pd.Series(population_dictionary)\n",
    "population_series\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Create an Area Series\n",
    "\n",
    "1. Create a Python dictionary with neighborhood names as keys and area as values. This time, write the neighborhoods in a different order. Additionally, add the neighborhood \"Beechview\", which has an area of 1.46 square miles, to your dictionary.\n",
    "2. Use the `pd.Series` function to create a Pandas Series called `area_series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Create an Area Series\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# create a dictionary with beechview added\n",
    "area_dictionary = {\n",
    "    \"Central Business District\":0.65,\n",
    "    \"East Liberty\":0.58,\n",
    "    \"Squirrel Hill North\":1.22,\n",
    "    \"Greenfield\":0.78,\n",
    "    \"Bloomfield\":0.70,\n",
    "    \"Beechview\":1.46\n",
    "}\n",
    "\n",
    "area_series = pd.Series(area_dictionary)\n",
    "area_series\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Create a DataFrame from Differently Ordered Series\n",
    "\n",
    "1. Create a DataFrame called `neighborhoods` with two columns `population` and `area` using the two Series you created as the data. Display your DataFrame\n",
    "2. Check to make sure the populations and area line up with the data table above.\n",
    "3. What is the population value for \"Beechview\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Create a DataFrame from Differently Ordered Series\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# create a Dataframe for population and area\n",
    "neighborhoods = pd.DataFrame({\"population\":population_series, \n",
    "                              \"area\": area_series})\n",
    "\n",
    "neighborhoods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the data have been aligned and ordered based on the neighborhood names. Because population values for Beechview were missing, Pandas fills in a missing value `NaN` (which will be discussed in a later lesson)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Selecting Rows of a DataFrame\n",
    "\n",
    "1. Try using the indexing syntax to select the second row of the `neighborhoods` DataFrame using the neighborhood's name and position. \n",
    "2. Why do you think you get an error? What kind of error do you get?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Selecting Rows of a DataFrame\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# try to select the first row of the dataframe\n",
    "neighborhoods['Bloomfield']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# try to select the first row of the dataframe\n",
    "neighborhoods[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indexing a DataFrame will always select a column (or columns). To select rows we must use the `loc`[(docs)](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.loc.html) and `iloc`[(docs)](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html) properties of a Pandas DataFrame. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Selecting Rows of a DataFrame using `loc`\n",
    "\n",
    "1. Run the code cell below to select the row for the Bloomfield neighborhood\n",
    "2. What data structure do you get back?\n",
    "3. What do the index values correspond to?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using loc to index rows by name\n",
    "neighborhoods.loc[\"Bloomfield\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Selecting Rows of a DataFrame using `loc`\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "You get back a Pandas Series representing one row of the Dataframe. The column names become the named index for the values. Also note the Series actually has a `Name`, which is another metadata field of a Series, that corresponds to the named index value specified by `loc`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Selecging Rows of a DataFrame using `iloc`\n",
    "\n",
    "1. Use the `iloc` function to select the same row as the previous task, but using the numeric index instead of the named index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Task - Selecting Rows of a DataFrame using `iloc`\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "\n",
    "# select the row at index position 1 from the neighborhoods dataframe\n",
    "neighborhoods.iloc[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Rows and Columns\n",
    "\n",
    "The `loc` and `iloc` properties allow for the selection of rows and columns using the following syntax:\n",
    "```python\n",
    "df.loc[row_indexer,column_indexer]\n",
    "```\n",
    "\n",
    "The `indexer` can be a pointer to a specific value or even a slice. Let's try it with our `neighborhoods` DataFrame. Run the code cells below to see"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# display the neighborhoods dataframe\n",
    "neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the row for Greenfield and the column population\n",
    "neighborhoods.loc[\"Greenfield\", \"population\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the population for several neighborhood\n",
    "neighborhoods.loc[\"Bloomfield\":\"Greenfield\", \"population\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Select with `iloc`\n",
    "\n",
    "1. Recreate the results above using the positional, numeric index with `iloc` instead of label-based index with `loc`.\n",
    "2. Pay close attention to the inclusive and exclusive values in the range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Select with `iloc`\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# answer\n",
    "neighborhoods.iloc[1:5, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tasks - Slicing and Dicing a DataFrame\n",
    "\n",
    "* Using `iloc` and the slicing syntax, select specific values from the example DataFrame based on the highlighted blocks in the image.\n",
    "* It is helpful to think of the slicing syntax to grab the rows THEN think of the slicing syntax for the columns.\n",
    "* Put the row slices *before* the comma and the column slices *after* the comma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is our example Dataframe\n",
    "indexing_example = pd.DataFrame([[1,2,3],[4,5,6],[7,8,9]])\n",
    "indexing_example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing1.png)\n",
    "* Select the second two columns of the first two rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[???]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scratch space\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing2.png)\n",
    "* Select the third row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[???]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scratch space\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing3.png)\n",
    "* Select the first two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[???]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scratch space\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing4.png)\n",
    "* Select the first two columns of the second row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[???]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scratch space\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answers - Slicing and Dicing a Dataframe\n",
    "\n",
    "* Using the `iloc` and slicing syntax slice the following dataframe based on the highlighted blocks in the image\n",
    "* first think of the slicing syntax to grab just the rows you want THEN think of the slicing syntax for the columns you want\n",
    "* Put the row slices *before* the comma and the column slices *after* the comma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "![first slice exercise](images/indexing1.png)\n",
    "* Select the second two columns of the first two rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[:2, 1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing2.png)\n",
    "* Select the third row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[2, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing3.png)\n",
    "* Select the first two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[:,:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![first slice exercise](images/indexing4.png)\n",
    "* Select the first two columns of the second row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Put the slicing syntax in your answer here\n",
    "indexing_example.iloc[1,:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why `loc` and `iloc`?\n",
    "\n",
    "Because pandas Data Structures have both explicit named and implicit numeric indexes, it can sometimes be ambiguous which index is being referenced. This can be especially true if you are using numbers in the explicit named index!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Task - Which One is 1?\n",
    "\n",
    "1. Run the code below to create a Series with an explicit numeric index.\n",
    "2. Run the code `confusing_index[1]` and look at the result.\n",
    "3. Which index, the implicit, numeric positional index or the explicit, named/label index?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neighborhoods_list = [\"East Liberty\", \n",
    "                 \"Greenfield\", \n",
    "                 \"Squirrel Hill North\", \n",
    "                 \"Bloomfield\", \n",
    "                 \"Central Business District\"]\n",
    "which_index = pd.Series(neighborhoods_list, index=[1,2,3,4,5])\n",
    "which_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Which index is being called?\n",
    "which_index[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Answer - Which One is 1?\n",
    "\n",
    "Click on the ellipses (...) below to see the answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "source": [
    "When indexing a Series without using `loc` or `iloc` than Pandas will use the named or label based index, which is equivalent to using `loc`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}