Python for Data Scientists.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"color:SteelBlue; font-size:42px;\">Python for Data Scientists</span> \n",
    "<hr>\n",
    "*v1.0 Last updated April 2, 2018*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br id=\"TOC\">\n",
    "<span style=\"color:Black; font-size:20px;\">**Table of Contents**</span>\n",
    "\n",
    "1. [Python Basics](#l1)\n",
    "2. [Data Structures](#l2)\n",
    "3. [Flow and Functions](#l3)\n",
    "4. [NumPy](#l4)\n",
    "5. [Pandas](#l5)\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<hr><br id=\"l1\">\n",
    "# Python Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Types\n",
    "You can check an object's type using the `type()` built-in function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Integers\n",
    "Let's take a look at the `type()` of the integer from above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'int'>\n"
     ]
    }
   ],
   "source": [
    "# Print type() of 2\n",
    "print(type(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Truncations**\n",
    "\n",
    "The `int()` function creats integers. It always rounds down. That's because what it's really doing is **truncating** the decimals from the number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n",
      "5\n"
     ]
    }
   ],
   "source": [
    "print( int(5.3) )\n",
    "print( int(5.9999) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Floats\n",
    "Floats are numeric objects with decimals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 403,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'int'>\n",
      "<class 'float'>\n",
      "<class 'float'>\n"
     ]
    }
   ],
   "source": [
    "print(type(5))\n",
    "print(type(5.))\n",
    "print(type(5.3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also convert an integer into a float using the `float() function`. Python will tag on a `.0` to indicate that it's a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 404,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1337\n",
      "1337.0\n"
     ]
    }
   ],
   "source": [
    "print( 1337 )        # int\n",
    "print( float(1337) ) # convert int to float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strings\n",
    "Strings are text objects enclosed by single or double quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 405,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a string.\n",
      "<class 'str'>\n"
     ]
    }
   ],
   "source": [
    "print( 'This is a string.' )\n",
    "print( type('This is a string.') )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Operators\n",
    "The following are the basic operations. Here is a [full list fo perators.](https://www.tutorialspoint.com/python/python_basic_operators.htm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 406,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12\n",
      "8\n",
      "5.0\n",
      "20\n",
      "100\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "print( 10 + 2 ) # Add\n",
    "print( 10 - 2 ) # Subtract\n",
    "print( 10 / 2 ) # Divide\n",
    "print( 10 * 2 ) # Multiply\n",
    "print( 10**2 )  # Squared\n",
    "print( 10 % 2 ) # Modulo (i.e. what is the remainder when 10 is divided by 2?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison operators\n",
    "Comparison operators are often based on boolean logic. These are binary obeject types that are either `True` or `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 407,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "False\n",
      "False\n",
      "False\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "print(5>3)   # Greater than\n",
    "print(5>=3)  # Greater than or equal to\n",
    "print(5< 3)  # Less than\n",
    "print(5<= 3) # Less than or equal to\n",
    "print(5== 3) # Equal to\n",
    "print(5!= 3) # Not equal to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example string opeartions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 408,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'55555'"
      ]
     },
     "execution_count": 408,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'5' * 5 # Repetition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 409,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1337\n",
      "992 + 345\n"
     ]
    }
   ],
   "source": [
    "print( 992 + 345 )   # Performs calculation\n",
    "print( '992 + 345' ) # Does not perform calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 410,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'992345'"
      ]
     },
     "execution_count": 410,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'992' + '345' # Concatentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Calculating volume</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the start of a long series of connected analysis seen throughout the notebook. Each analysis build on previous analyses and concepts.\n",
    "\n",
    "**Here is our hypothetical setting**:\n",
    "- You are 007 James Bond. You are on your first training mission.\n",
    "- You will travel to different locations in London.\n",
    "- First, you will prepare for and plan the trip.\n",
    "- After you arrive at each location, you will meet with local business owners and offer your help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**First, calculate the total volume of water you can carry, in cm$^3$.**\n",
    "* The formula for the volume of a cylinder is $V = \\pi r^2 h$\n",
    "* The container cylinder's radius is 4cm and its height is 16cm\n",
    "* You have room in your backpack for 3 containers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 411,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2411.52\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "2412.743157956961"
      ]
     },
     "execution_count": 411,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Total volume of water you can carry\n",
    "import math # need the pi constant\n",
    "print(3 * 3.14 * 4**2 * 16)\n",
    "3 * math.pi * 4**2 * 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q estimates that you'll need at least 2000 cm$^3$ for the mission. Is our container large enough?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 412,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 412,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(3 * math.pi * 4**2 * 16) > 2000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Formatting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 + 2 = 3\n"
     ]
    }
   ],
   "source": [
    "# Insert dynamic value into string\n",
    "print( '1 + 2 = {}'.format(1 + 2) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 414,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 + 2 = 3\n"
     ]
    }
   ],
   "source": [
    "# Insert 3 dynamic values into string\n",
    "print( '{} + {} = {}'.format(1, 2, 1 + 2) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variables are named objects and used to store data or other information:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 415,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "4\n",
      "Hello from Canada\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "# Set variables\n",
    "a = 2\n",
    "b = 4\n",
    "c = \"Hello from Canada\"\n",
    "\n",
    "print( a )\n",
    "print( b )\n",
    "print( c )\n",
    "print( a * b )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naming conventions\n",
    "Best practice: use lower_case_with_underscores. The shorter the variable the better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's say we wanted to calculate total return on investment based on the formula for compound interest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 416,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set variables\n",
    "principal = 1000.0\n",
    "yearly_interest_rate = 0.03\n",
    "n_years = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "159.27407430000017\n"
     ]
    }
   ],
   "source": [
    "# Calculate total investment return\n",
    "total_return = principal * (1 + yearly_interest_rate)**n_years - principal\n",
    "\n",
    "# Print total investment return\n",
    "print( total_return )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interlude: Getting help on python commands**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 418,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on built-in function print in module builtins:\n",
      "\n",
      "print(...)\n",
      "    print(value, ..., sep=' ', end='\\n', file=sys.stdout, flush=False)\n",
      "    \n",
      "    Prints the values to a stream, or to sys.stdout by default.\n",
      "    Optional keyword arguments:\n",
      "    file:  a file-like object (stream); defaults to the current sys.stdout.\n",
      "    sep:   string inserted between values, default a space.\n",
      "    end:   string appended after the last value, default a newline.\n",
      "    flush: whether to forcibly flush the stream.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Get help on the print() function\n",
    "help(print)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; margin: 40px 0 40px 0;\">\n",
    "[Back to Table of Contents](#TOC)\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br id=\"l2\">\n",
    "# Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists are mutable sequences of objects, enclosed by square brackets: `[]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4]\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "# Create list of integers\n",
    "integer_list = [0, 1, 2, 3, 4]\n",
    "\n",
    "print( integer_list )\n",
    "print( type(integer_list) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 420,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello', 1, 'Canada', 2, 3.0]\n"
     ]
    }
   ],
   "source": [
    "# Create list of mixed types: strings, ints, and floats\n",
    "my_list = ['hello', 1, 'Canada', 2, 3.0]\n",
    "\n",
    "# Print entire list\n",
    "print( my_list )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing lists\n",
    "In Python, lists are zero-indexed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 421,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello\n",
      "Canada\n"
     ]
    }
   ],
   "source": [
    "print( my_list[0] ) # Print the first element\n",
    "print( my_list[2] ) # Print the third element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Slicing**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 422,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 'Canada']\n"
     ]
    }
   ],
   "source": [
    "#Selects all starting from the 2nd element, but BEFORE the 4th element\n",
    "print( my_list[1:3] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 423,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hello', 1, 'Canada']\n",
      "[1, 'Canada', 2, 3.0]\n"
     ]
    }
   ],
   "source": [
    "# Selects all BEFORE the 4th element\n",
    "print( my_list[:3] )\n",
    "\n",
    "# Selects all starting from the 2nd element\n",
    "print( my_list[1:] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Negative indices**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 424,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.0\n",
      "['hello', 1, 'Canada', 2]\n",
      "[1, 'Canada', 2]\n"
     ]
    }
   ],
   "source": [
    "# Select the last element\n",
    "print( my_list[-1] )\n",
    "\n",
    "# Select all BEFORE the last element\n",
    "print( my_list[:-1] )\n",
    "\n",
    "# Selects all starting from the 2nd element, but before the last element\n",
    "print( my_list[1:-1] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mutable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lists are mutable, meaning you can change individual elements of a list. Lets update `my_list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bonjour\n",
      "Canada\n"
     ]
    }
   ],
   "source": [
    "my_list[0] = 'bonjour' # Sets new value for the first element\n",
    "\n",
    "print( my_list[0] ) # Print the first element\n",
    "print( my_list[2] ) # Print the third element"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Appending** to and **removing** elements from lists are both easy to do. These are a special class of 'list' functions called methods. You can see all list methods by calling the help for lists!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on class list in module builtins:\n",
      "\n",
      "class list(object)\n",
      " |  list() -> new empty list\n",
      " |  list(iterable) -> new list initialized from iterable's items\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __add__(self, value, /)\n",
      " |      Return self+value.\n",
      " |  \n",
      " |  __contains__(self, key, /)\n",
      " |      Return key in self.\n",
      " |  \n",
      " |  __delitem__(self, key, /)\n",
      " |      Delete self[key].\n",
      " |  \n",
      " |  __eq__(self, value, /)\n",
      " |      Return self==value.\n",
      " |  \n",
      " |  __ge__(self, value, /)\n",
      " |      Return self>=value.\n",
      " |  \n",
      " |  __getattribute__(self, name, /)\n",
      " |      Return getattr(self, name).\n",
      " |  \n",
      " |  __getitem__(...)\n",
      " |      x.__getitem__(y) <==> x[y]\n",
      " |  \n",
      " |  __gt__(self, value, /)\n",
      " |      Return self>value.\n",
      " |  \n",
      " |  __iadd__(self, value, /)\n",
      " |      Implement self+=value.\n",
      " |  \n",
      " |  __imul__(self, value, /)\n",
      " |      Implement self*=value.\n",
      " |  \n",
      " |  __init__(self, /, *args, **kwargs)\n",
      " |      Initialize self.  See help(type(self)) for accurate signature.\n",
      " |  \n",
      " |  __iter__(self, /)\n",
      " |      Implement iter(self).\n",
      " |  \n",
      " |  __le__(self, value, /)\n",
      " |      Return self<=value.\n",
      " |  \n",
      " |  __len__(self, /)\n",
      " |      Return len(self).\n",
      " |  \n",
      " |  __lt__(self, value, /)\n",
      " |      Return self<value.\n",
      " |  \n",
      " |  __mul__(self, value, /)\n",
      " |      Return self*value.n\n",
      " |  \n",
      " |  __ne__(self, value, /)\n",
      " |      Return self!=value.\n",
      " |  \n",
      " |  __new__(*args, **kwargs) from builtins.type\n",
      " |      Create and return a new object.  See help(type) for accurate signature.\n",
      " |  \n",
      " |  __repr__(self, /)\n",
      " |      Return repr(self).\n",
      " |  \n",
      " |  __reversed__(...)\n",
      " |      L.__reversed__() -- return a reverse iterator over the list\n",
      " |  \n",
      " |  __rmul__(self, value, /)\n",
      " |      Return self*value.\n",
      " |  \n",
      " |  __setitem__(self, key, value, /)\n",
      " |      Set self[key] to value.\n",
      " |  \n",
      " |  __sizeof__(...)\n",
      " |      L.__sizeof__() -- size of L in memory, in bytes\n",
      " |  \n",
      " |  append(...)\n",
      " |      L.append(object) -> None -- append object to end\n",
      " |  \n",
      " |  clear(...)\n",
      " |      L.clear() -> None -- remove all items from L\n",
      " |  \n",
      " |  copy(...)\n",
      " |      L.copy() -> list -- a shallow copy of L\n",
      " |  \n",
      " |  count(...)\n",
      " |      L.count(value) -> integer -- return number of occurrences of value\n",
      " |  \n",
      " |  extend(...)\n",
      " |      L.extend(iterable) -> None -- extend list by appending elements from the iterable\n",
      " |  \n",
      " |  index(...)\n",
      " |      L.index(value, [start, [stop]]) -> integer -- return first index of value.\n",
      " |      Raises ValueError if the value is not present.\n",
      " |  \n",
      " |  insert(...)\n",
      " |      L.insert(index, object) -- insert object before index\n",
      " |  \n",
      " |  pop(...)\n",
      " |      L.pop([index]) -> item -- remove and return item at index (default last).\n",
      " |      Raises IndexError if list is empty or index is out of range.\n",
      " |  \n",
      " |  remove(...)\n",
      " |      L.remove(value) -> None -- remove first occurrence of value.\n",
      " |      Raises ValueError if the value is not present.\n",
      " |  \n",
      " |  reverse(...)\n",
      " |      L.reverse() -- reverse *IN PLACE*\n",
      " |  \n",
      " |  sort(...)\n",
      " |      L.sort(key=None, reverse=False) -> None -- stable sort *IN PLACE*\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data and other attributes defined here:\n",
      " |  \n",
      " |  __hash__ = None\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['bonjour', 1, 'Canada', 2, 3.0, 22]\n",
      "['bonjour', 1, 'Canada', 2, 22]\n"
     ]
    }
   ],
   "source": [
    "# Add to end of the list\n",
    "my_list.append(22)\n",
    "print(my_list)\n",
    "\n",
    "# Remove an element from the list\n",
    "my_list.remove(3.0)\n",
    "print(my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List Operations\n",
    "Lets create a couple of lists and do some operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 428,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = [1, 2, 3]\n",
    "b = [4, 5, 6]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 4, 5, 6]\n"
     ]
    }
   ],
   "source": [
    "print( a + b ) # Concatentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 2, 3, 1, 2, 3, 1, 2, 3]\n"
     ]
    }
   ],
   "source": [
    "print( a * 3 ) # Repetition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 431,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print( 3 in a ) # Membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 6\n"
     ]
    }
   ],
   "source": [
    "print( min(b), max(b) ) # Min, Max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 433,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "print( len(a) ) # Length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Reading in data and creating variables</span> \n",
    "\n",
    "In previous analysis, you learned about 007's training mission. You also made sure you had enough water for the trip by performing calculations. Now, it's time to start planning the locations in London to visit!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center; margin-top:30px; margin-bottom:30px\">\n",
    "<img src=\"https://elitedatascience.com/wp-content/uploads/2017/02/london-map-boroughs.png\" alt=\"London Boroughs Map\" width=640 style=\"box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);\">\n",
    "<br>\n",
    "<em>Boroughs of London.</em>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets analayze some London boroughs and each of there locations. First. lets create lists of the neccessary borough variables that contain the locations. Python has a variety of **input/output** methods. If you want to see all of the methods, including the one below, you can learn more about them in the [documentation](https://docs.python.org/2/tutorial/inputoutput.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 434,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read lists of locations\n",
    "with open('project_files/brent.txt', 'r') as f:\n",
    "    brent = f.read().splitlines()\n",
    "    \n",
    "with open('project_files/camden.txt', 'r') as f:\n",
    "    camden = f.read().splitlines()\n",
    "\n",
    "with open('project_files/redbridge.txt', 'r') as f:\n",
    "    redbridge = f.read().splitlines()\n",
    "\n",
    "with open('project_files/southwark.txt', 'r') as f:\n",
    "    southwark = f.read().splitlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that when text files are read using the <code style=\"color:steelblue\">splitlines()</code> function, the resulting object is a list.\n",
    "\n",
    "So the four objects you just created from the files - <code style=\"color:steelblue\">brent</code>, <code style=\"color:steelblue\">camden</code>, <code style=\"color:steelblue\">redbridge</code>, and <code style=\"color:steelblue\">southwark</code> - are all lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 435,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'list'>\n",
      "<class 'list'>\n",
      "<class 'list'>\n",
      "<class 'list'>\n"
     ]
    }
   ],
   "source": [
    "print( type(brent) )\n",
    "print( type(camden) )\n",
    "print( type(redbridge) )\n",
    "print( type(southwark) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the number of locations in each listed borough:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n",
      "22\n",
      "18\n",
      "15\n"
     ]
    }
   ],
   "source": [
    "# Print length of each list\n",
    "print(len(brent))\n",
    "print(len(camden))\n",
    "print(len(redbridge))\n",
    "print(len(southwark))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 437,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Brent Park', 'Church End', 'Dollis Hill', 'Harlesden', 'Kensal Green']\n",
      "['Chalk Farm', 'Dartmouth Park', 'Fitzrovia', 'Frognal', 'Bloomsbury']\n",
      "['Chadwell Heath', 'Gants Hill', 'Goodmayes', 'Snaresbrook', 'Hainault']\n",
      "['Denmark Hill', 'Dulwich', 'Surrey Quays', 'East Dulwich', 'Elephant and Castle']\n"
     ]
    }
   ],
   "source": [
    "# Print the first 5 locations of each Borough\n",
    "print(brent[:5])\n",
    "print(camden[:5])\n",
    "print(redbridge[:5])\n",
    "print(southwark[:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 438,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# Is 'Newbury Park' in Redbridge?\n",
    "print( 'Newbury Park' in redbridge ) # Membership\n",
    "\n",
    "# Is 'Peckham' in Brent?\n",
    "print( 'Peckham' in brent ) # Membership"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 439,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bankside\n",
      "Walworth\n"
     ]
    }
   ],
   "source": [
    "# Print minimum value in southwark\n",
    "print( min(southwark)) # Min in this case is alphabet-sorted\n",
    "\n",
    "# Print maximum value in southwark\n",
    "print( max(southwark))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sets are unordered collections of **unique** objects, enclosed by curly braces: `{ }`\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 440,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0, 1, 2, 3, 4}\n",
      "<class 'set'>\n"
     ]
    }
   ],
   "source": [
    "integer_set = {0, 1, 2, 3, 4}\n",
    "\n",
    "print( integer_set )\n",
    "print( type(integer_set) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing duplicates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because each element in a set must be unique, sets are a great tool for removing duplicates. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1, 1, 2, 3, 5, 8, 13]\n",
      "{1, 2, 3, 5, 8, 13}\n"
     ]
    }
   ],
   "source": [
    "fibonacci_list = [ 1, 1, 2, 3, 5, 8, 13 ] # Will keep both 1's will remain\n",
    "fibonacci_set = { 1, 1, 2, 3, 5, 8, 13 }  # Only one 1 will remain\n",
    "\n",
    "print( fibonacci_list )\n",
    "print( fibonacci_set )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting to sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1, 2, 3, 5, 8, 13}\n"
     ]
    }
   ],
   "source": [
    "# Create a list\n",
    "fibonacci_list = [ 1, 1, 2, 3, 5, 8, 13 ]\n",
    "\n",
    "# Convert it to a set\n",
    "fibonacci_set = set(fibonacci_list)\n",
    "\n",
    "print( fibonacci_set )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 443,
   "metadata": {},
   "outputs": [],
   "source": [
    "powers_of_two = { 1, 2, 4, 8, 16 }\n",
    "fibonacci_set = { 1, 1, 2, 3, 5, 8, 13 }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 444,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1, 2, 3, 4, 5, 8, 13, 16}\n",
      "{1, 2, 3, 4, 5, 8, 13, 16}\n"
     ]
    }
   ],
   "source": [
    "# Union: Elements in either set (two ways, both do same thing)\n",
    "print( powers_of_two.union( fibonacci_set ) )\n",
    "print( powers_of_two | fibonacci_set )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 445,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{8, 1, 2}\n",
      "{8, 1, 2}\n"
     ]
    }
   ],
   "source": [
    "# Intersection: Elements in both sets (two ways, both do same thing)\n",
    "print( powers_of_two.intersection( fibonacci_set ) )\n",
    "print( powers_of_two & fibonacci_set )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 446,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1, 2, 4, 8, 16}\n",
      "{1, 2, 3, 5, 8, 13}\n",
      "{16, 4}\n",
      "{13, 3, 5}\n"
     ]
    }
   ],
   "source": [
    "# Difference\n",
    "print( powers_of_two )\n",
    "print( fibonacci_set )\n",
    "print( powers_of_two - fibonacci_set )\n",
    "print( fibonacci_set - powers_of_two)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: remomving duplicates in our London data</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue 007's training. Before we continue, lets see if wwe have duplicates and if so, remove them from our lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 447,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n",
      "True\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Does Brent have duplicates?\n",
    "print(len(brent) != len(set(brent)))\n",
    "\n",
    "# Does Camden have duplicates?\n",
    "print(len(camden) != len(set(camden)))\n",
    "\n",
    "# Does Redbridge have duplicates?\n",
    "print(len(redbridge) != len(set(redbridge)))\n",
    "\n",
    "# Does Southwark have duplicates?\n",
    "print(len(southwark) != len(set(southwark)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the lists with duplicates, remove duplicates by converting them into sets. Then, convert them back into lists."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n",
      "19\n",
      "16\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "# Convert lists to sets to remove duplicates, then convert them back to lists \n",
    "brent = list(set(brent))\n",
    "camden = list(set(camden))\n",
    "redbridge = list(set(redbridge))\n",
    "southwark = list(set(southwark))\n",
    "\n",
    "print(len(brent))\n",
    "print(len(camden))\n",
    "print(len(redbridge))\n",
    "print(len(southwark))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionaries are unordered collections of **key-value pairs**, enclosed by curly braces, like `{}`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 449,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'zero': 0, 'one': 1, 'two': 2, 'three': 3, 'four': 4}\n",
      "<class 'dict'>\n"
     ]
    }
   ],
   "source": [
    "integer_dict = {\n",
    "    'zero' : 0,\n",
    "    'one' : 1,\n",
    "    'two' : 2,\n",
    "    'three' : 3,\n",
    "    'four' : 4\n",
    "}\n",
    "\n",
    "print( integer_dict )\n",
    "print( type(integer_dict) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keys and values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A dictionary, also called a \"dict\", is like a miniature database for storing and organizing data.\n",
    "\n",
    "Each element in a dict is actually a key-value pair, and it's called an **item**.\n",
    "\n",
    "- **Key**: A key is like a name for the value. You should make them descriptive.\n",
    "- **Value**: A value is some other Python object to store. They can be floats, lists, functions, and so on.\n",
    "\n",
    "Here's an example with descriptive keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_dict = {\n",
    "    'title' : \"The Foundation Trillogy\",\n",
    "    'author' : 'Issac Assimov',\n",
    "    100 : ['A number.', 'The answer to 60 + 40 = ?'] # Keys can be integers too! \n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access values using their Keys:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 451,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Foundation Trillogy\n",
      "Issac Assimov\n"
     ]
    }
   ],
   "source": [
    "# Print the value for the 'title' key\n",
    "print( my_dict['title'] )\n",
    "\n",
    "# Print the value for the 'author' key\n",
    "print( my_dict['author'] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Updating values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Isaac Asimov\n"
     ]
    }
   ],
   "source": [
    "# Updating existing key-value pair\n",
    "my_dict['author'] = 'Isaac Asimov'\n",
    "\n",
    "# Print the value for the 'author' key\n",
    "print( my_dict['author'] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 453,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['A number.', 'The answer to 60 + 40 = ?', 'Answer to the Ultimate Question of Life, the Universe, and Everything']\n"
     ]
    }
   ],
   "source": [
    "# Append element to list\n",
    "my_dict[100].append('Answer to the Ultimate Question of Life, the Universe, and Everything')\n",
    "\n",
    "# Print value for the key 42\n",
    "print( my_dict[100] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Creating new items**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 454,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a new key-value pair\n",
    "my_dict['year'] = 1951"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 455,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Foundation Trillogy was written by Isaac Asimov in 1951.\n"
     ]
    }
   ],
   "source": [
    "# Print summary of the book.\n",
    "print('{} was written by {} in {}.'.format(my_dict['title'], my_dict['author'], my_dict['year']) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Convenience functions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 456,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['title', 'author', 100, 'year'])\n",
      "dict_values(['The Foundation Trillogy', 'Isaac Asimov', ['A number.', 'The answer to 60 + 40 = ?', 'Answer to the Ultimate Question of Life, the Universe, and Everything'], 1951])\n"
     ]
    }
   ],
   "source": [
    "# Keys - get entire list\n",
    "print( my_dict.keys() )\n",
    "\n",
    "# Values - get all values\n",
    "print( my_dict.values() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 457,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_items([('title', 'The Foundation Trillogy'), ('author', 'Isaac Asimov'), (100, ['A number.', 'The answer to 60 + 40 = ?', 'Answer to the Ultimate Question of Life, the Universe, and Everything']), ('year', 1951)])\n"
     ]
    }
   ],
   "source": [
    "# All items (list of tuples)\n",
    "print( my_dict.items() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: creating dictionaries and aggregating borough data</span> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets create a locations dictionary for the London boroughs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 458,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create location_dict\n",
    "location_dict = {\n",
    "    'Brent' : brent,\n",
    "    'Camden' : camden,\n",
    "    'Redbridge' : redbridge,\n",
    "    'Southwark' : southwark\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets double check that the dictionary has the correct keys using a for loop. More on loops syntax in later section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 459,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "True\n",
      "True\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "for borough in ['Brent', 'Camden', 'Redbridge', 'Southwark']:\n",
    "    print( borough in location_dict )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the dictionary is correct, we do not need the original lists, so we can get rid of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "metadata": {},
   "outputs": [],
   "source": [
    "brent, camden, redbridge, southwark = None, None, None, None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 461,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NoneType"
      ]
     },
     "execution_count": 461,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(brent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nb: `None` is an object that denotes emptiness."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to split our visit to London into two trips: one for Inner London and one for Outer London.\n",
    "\n",
    "**Lets add two new items to our dictionary:**\n",
    "1. **Key:** <code style=\"color:steelblue\">'Inner London'</code>... **Value:** All locations in <code style=\"color:steelblue\">'Camden'</code> and <code style=\"color:steelblue\">'Southwark'</code>.\n",
    "2. **Key:** <code style=\"color:steelblue\">'Outer London'</code>... **Value:** All locations in <code style=\"color:steelblue\">'Brent'</code> and <code style=\"color:steelblue\">'Redbridge'</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new key-value pair for 'Inner London'\n",
    "location_dict['Inner London'] = location_dict['Camden'] + location_dict['Southwark']\n",
    "\n",
    "# Create a new key-value pair for 'Outer London'\n",
    "location_dict['Outer London'] = location_dict['Brent'] + location_dict['Redbridge']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 463,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 463,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Quick QA - Add the locations of boroughs\n",
    "len(location_dict['Camden']) + len(location_dict['Southwark']) + len(location_dict['Brent']) + len(location_dict['Redbridge']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 464,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "67"
      ]
     },
     "execution_count": 464,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Quick QA - Add innner and outer loactions\n",
    "len(location_dict['Inner London']) + len(location_dict['Outer London'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Save work using Pickle**\n",
    "* Use a Python built-in package called <code style=\"color:steelblue\">pickle</code> to save the `location_dict` object to use later.\n",
    "* Pickle saves an entire object in a file on your computer.\n",
    "* Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 465,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import pickle module\n",
    "import pickle\n",
    "\n",
    "# Save object to disk\n",
    "with open('project_files/location_dict.pkl', 'wb') as f: # Saving the file 'location_dict' to project_files folder\n",
    "    pickle.dump(location_dict, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; margin: 40px 0 40px 0;\">\n",
    "[Back to Table of Contents](#TOC)\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br id=\"l3\">\n",
    "# Flow and Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## if statements allow conditional logic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### if..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If statements check if a condition is met before running a block of code.\n",
    "\n",
    "Begin them with the `if` keyword. Then the statement has two parts:\n",
    "\n",
    "1. The **condition**, which must evaluate to a boolean. (Technically, it's fine as long as its \"truthiness\" can be evaluated.\n",
    "2. The **code block** to run if the condition is met (indented with 4 spaces).\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 466,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We have enough fuel to last the zombie apocalypse. \n"
     ]
    }
   ],
   "source": [
    "current_fuel = 85\n",
    "\n",
    "# Condition\n",
    "if current_fuel >= 80:\n",
    "    # Code block to run if condition is met\n",
    "    print( 'We have enough fuel to last the zombie apocalypse. ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### if... else..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 467,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Restock! We need at least 30 gallons.\n"
     ]
    }
   ],
   "source": [
    "current_fuel = 50\n",
    "\n",
    "# Condition\n",
    "if current_fuel >= 80:\n",
    "    # Do this when condition is met\n",
    "    print( 'We have enough fuel to last the zombie apocalypse. ')\n",
    "else:\n",
    "    # Do this when condition is not met\n",
    "    print( 'Restock! We need at least {} gallons.'.format(80 - current_fuel) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### if... elif... else..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 468,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Restock! We need at least 10 gallons.\n"
     ]
    }
   ],
   "source": [
    "current_fuel = 70\n",
    "\n",
    "# First condition\n",
    "if current_fuel >= 80:\n",
    "    print( 'We have enough fuel to last the zombie apocalypse. ')\n",
    "# If first condition is not met, check this condition\n",
    "elif current_fuel < 60:\n",
    "    print( 'ALERT: WE ARE WAY TOO LOW ON FUEL!' )\n",
    "# If no conditions were met, perform this\n",
    "else:\n",
    "    print( 'Restock! We need at least {} gallons.'.format(80 - current_fuel) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: importing data and simple flow</span> \n",
    "We have created our location dictionary containing the boroughs we are interested in visiting. Now, we want to determine which location to visit first.\n",
    "\n",
    "First, let's import <code style=\"color:steelblue\">location_dict</code> again using <code style=\"color:steelblue\">pickle</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 469,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import location_dict using pickle\n",
    "import pickle\n",
    "\n",
    "# Read object from disk\n",
    "with open('project_files/location_dict.pkl', 'rb') as f:\n",
    "    location_dict = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 470,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['Brent', 'Camden', 'Redbridge', 'Southwark', 'Inner London', 'Outer London'])\n"
     ]
    }
   ],
   "source": [
    "# Print the keys in location_dict\n",
    "print(location_dict.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Pseudocode for code below**:\n",
    "\n",
    "* **If** our Inner London list has more locations than our Outer London list, print the message:\n",
    "\n",
    "\n",
    "<pre style=\"color:steelblue\">Inner London is huge!</pre>\n",
    "\n",
    "\n",
    "* **Else if** our Outer London list has more locations than our Inner London list, print the message:\n",
    "\n",
    "    \n",
    "<pre style=\"color:steelblue\">Outer London is huge!</pre>\n",
    "\n",
    "\n",
    "* **Else** (i.e. they have the same number of locations), print the message:\n",
    "\n",
    "\n",
    "<pre style=\"color:steelblue\">Inner and outer London are huge!</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outer London is huge!\n"
     ]
    }
   ],
   "source": [
    "if len(location_dict['Inner London']) > len(location_dict['Outer London']):\n",
    "    print('Inner London is huge!')\n",
    "elif len(location_dict['Outer London']) > len(location_dict['Inner London']):\n",
    "    print('Outer London is huge!')\n",
    "else:\n",
    "    print('Inner and outer London are huge!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## for loops allow iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### for..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# Print list\n",
    "for number in [0, 1, 2, 3, 4]: \n",
    "    print( number )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "1\n",
      "2\n",
      "3\n",
      "4\n"
     ]
    }
   ],
   "source": [
    "# Print range\n",
    "for number in range(5):\n",
    "    print( number )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 474,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "3\n",
      "2\n",
      "1\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "# Print reversed range\n",
    "for number in reversed(range(5)):\n",
    "    print( number )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Nested control flow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 475,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Buzz\n",
      "8\n",
      "7\n",
      "Buzz\n",
      "5\n",
      "4\n",
      "Buzz\n",
      "2\n",
      "1\n",
      "Liftoff!\n"
     ]
    }
   ],
   "source": [
    "# Check if number divisible by another number\n",
    "range_list = range(10)\n",
    "\n",
    "for number in reversed(range_list):\n",
    "    if number == 0:\n",
    "        print( 'Liftoff!' )\n",
    "    elif number % 3 == 0:\n",
    "        print( 'Buzz' )\n",
    "    else:\n",
    "        print( number )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 476,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4 x 6 = 24\n",
      "4 x 3 = 12\n",
      "3 x 6 = 18\n",
      "3 x 3 = 9\n",
      "2 x 6 = 12\n",
      "2 x 3 = 6\n"
     ]
    }
   ],
   "source": [
    "# for loops within other for loops\n",
    "list_a = [4, 3, 2]\n",
    "list_b = [6, 3]\n",
    "\n",
    "for a in list_a:\n",
    "    for b in list_b:\n",
    "        print( a, 'x', b, '=', a * b )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building new lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Set a variable to an empty list, which is just `[]`.\n",
    "2. Then, as you loop through elements, `.append()` the ones you want to your list variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 477,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8]\n",
      "[1, 3, 5, 7, 9]\n"
     ]
    }
   ],
   "source": [
    "# Separate range_list into even_list and odd_list\n",
    "\n",
    "range_list = range(10)\n",
    "\n",
    "# Initialize empty lists\n",
    "evens_list = []\n",
    "odds_list = []\n",
    "\n",
    "# Iterate through each number in range_list\n",
    "for number in range_list:\n",
    "    # check for divisibility by 2\n",
    "    if number % 2 == 0:\n",
    "        # If divisible by 2\n",
    "        evens_list.append(number)\n",
    "    else:\n",
    "        # If not divisible by 2\n",
    "        odds_list.append(number)\n",
    "\n",
    "# Confirm our lists are correct\n",
    "print( evens_list )\n",
    "print( odds_list )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Counting Locations in London</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets count the number of locations in each of our boroughs. \n",
    "\n",
    "* **Tip:** You can iterate through keys and values of a dictionary at the same time using <code style=\"color:steelblue\">.items()</code>, like so:\n",
    "\n",
    "<pre style=\"color:#bbb\">\n",
    "for <strong style=\"color:steelblue\">key, value</strong> in location_dict<strong style=\"color:steelblue\">.items()</strong>:\n",
    "    <span style=\"color:dimgray\"># code block</span>\n",
    "</pre>      \n",
    "        \n",
    "* **Tip:** Remember, to insert multiple dynamic values into a string, you can just add more places to <code style=\"color:steelblue\">.format()</code>, like so:\n",
    "\n",
    "<pre style=\"color:#bbb\">\n",
    "'<strong style=\"color:steelblue\">{}</strong> has <strong style=\"color:steelblue\">{}</strong> locations.'.format(<strong style=\"color:steelblue\">first_value, second_value</strong>)\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 478,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Brent has 19 locations.\n",
      "Camden has 19 locations.\n",
      "Redbridge has 16 locations.\n",
      "Southwark has 13 locations.\n",
      "Inner London has 32 locations.\n",
      "Outer London has 35 locations.\n"
     ]
    }
   ],
   "source": [
    "# For each key in location_dict, print the number of locations in its list\n",
    "for key, value in location_dict.items():\n",
    "    print('{} has {} locations.'.format(key, len(value)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's give each location in Outer London a first impression based on its name. \n",
    "\n",
    "**Pseudocode**\n",
    "\n",
    "Combine <code style=\"color:steelblue\">if</code> and <code style=\"color:steelblue\">for</code> statements. For each location in Outer London...\n",
    "* **If** its name has <code style=\"color:steelblue\">'Farm'</code>, <code style=\"color:steelblue\">'Park'</code>, <code style=\"color:steelblue\">'Hill'</code>, or <code style=\"color:steelblue\">'Green'</code> in it, print:\n",
    "\n",
    "<pre style=\"color:steelblue\">{<strong>name</strong>} sounds pleasant.</pre>\n",
    "\n",
    "* **Else If** its name has <code style=\"color:steelblue\">'Royal'</code>, <code style=\"color:steelblue\">'Queen'</code>, or <code style=\"color:steelblue\">'King'</code> in it, print:\n",
    "\n",
    "<pre style=\"color:steelblue\">{<strong>name</strong>} sounds grand.</pre>\n",
    "    \n",
    "* If its name doesn't sound pleasant or grand, just ignore it.\n",
    "* **Tip:** If you want to check if any word from a list is found in a string, you can use <code style=\"color:steelblue\">any()</code>, like so:\n",
    "\n",
    "<pre style=\"color:steelblue\">\n",
    "any( word in name for word in <strong>list_of_words</strong> )\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 479,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kensal Green sounds pleasant.\n",
      "Wembley Park sounds pleasant.\n",
      "Queen's Park sounds pleasant.\n",
      "Dollis Hill sounds pleasant.\n",
      "Brent Park sounds pleasant.\n",
      "Park Royal sounds pleasant.\n",
      "Kingsbury sounds grand.\n",
      "Woodford Green sounds pleasant.\n",
      "Seven Kings sounds grand.\n",
      "Newbury Park sounds pleasant.\n",
      "Gants Hill sounds pleasant.\n"
     ]
    }
   ],
   "source": [
    "pleasant_sounding = ['Farm', 'Park', 'Hill', 'Green']\n",
    "royal_sounding = ['Royal', 'Queen', 'King']\n",
    "\n",
    "# Print first impression of each location in Outer London based on its names\n",
    "for name in location_dict['Outer London']:\n",
    "    if any(word in name for word in pleasant_sounding): # 'word' is part of any function\n",
    "        print( name, 'sounds pleasant.' )\n",
    "    elif any(word in name for word in royal_sounding):\n",
    "        print( name, 'sounds grand.' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List comprehensions construct new lists elegantly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List comprehensions\n",
    "List comprehensions aren't mandatory, but they help keep your code clean and concise.\n",
    "\n",
    "List comprehensions construct new lists out of existing ones after applying transformations or conditions to them.\n",
    "\n",
    "Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 480,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n"
     ]
    }
   ],
   "source": [
    "# Construct list of the squares in range(10) using list comprehension\n",
    "squares_list = [number**2 for number in range(10)]\n",
    "\n",
    "print( squares_list )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional inclusion\n",
    "You can even include conditional logic in list comprehensions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 481,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 2, 4, 6, 8]\n"
     ]
    }
   ],
   "source": [
    "# Conditional inclusion\n",
    "evens_list = [number for number in range(10) if number % 2 == 0]\n",
    "\n",
    "print( evens_list )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 482,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd', 'Even', 'Odd']\n"
     ]
    }
   ],
   "source": [
    "# Conditional outputs\n",
    "even_odd_labels = ['Even' if number % 2 == 0 else 'Odd' for number in range(10)]\n",
    "\n",
    "print( even_odd_labels )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other comprehensions\n",
    "Finally, list comprehensions are not limited to lists!\n",
    "\n",
    "- You can use them for other data structures too.\n",
    "- The syntax is the same, except you would enclose it with curly braces for sets and parentheses for tuples.\n",
    "\n",
    "For example, we can create a set like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 483,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0, 2, 4, 6, 8, 10, 12, 14, 16, 18}\n"
     ]
    }
   ],
   "source": [
    "# Construct set of doubles using set comprehension\n",
    "doubles_set = { number * 2 for number in range(10) }\n",
    "\n",
    "print( doubles_set )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: list comprehensions of London locations</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lets use a list comprehension, to create a new list called <code style=\"color:steelblue\">pleasant_locations</code>.**\n",
    "* It should contain locations in Outer London that are <code style=\"color:steelblue\">pleasant_sounding</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 484,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Kensal Green', 'Wembley Park', \"Queen's Park\", 'Dollis Hill', 'Brent Park', 'Park Royal', 'Woodford Green', 'Newbury Park', 'Gants Hill']\n"
     ]
    }
   ],
   "source": [
    "# Create pleasant_locations list using a list comprehension\n",
    "pleasant_locations = [name for name in location_dict['Outer London'] \n",
    "                      if any(word in name for word in pleasant_sounding)]\n",
    "\n",
    "# Print the pleasant-sounding locations\n",
    "print(pleasant_locations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 485,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print number of pleasant-sounding locations\n",
    "len(pleasant_locations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions are blocks of reusable code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions\n",
    "Functions are blocks of reusable code that can be called by name.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 486,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'function'>\n"
     ]
    }
   ],
   "source": [
    "def make_message_exciting(message='hello, world'):\n",
    "    text = message + '!'\n",
    "    return text\n",
    "\n",
    "print( type(make_message_exciting) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break that down:\n",
    "\n",
    "- Functions begin with the <code style=\"color:green\">**def**</code>  keyword, followed by the function name\n",
    "- They can take optional arguments, E.g., message = 'hello world'\n",
    "- They are then followed by an indented code block.\n",
    "- Finally, they return a value, which is also indented\n",
    "\n",
    "To call a function, simply type its name and a parentheses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 487,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello, world!'"
      ]
     },
     "execution_count": 487,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Call make_message_exciting() function\n",
    "make_message_exciting()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In practice\n",
    "In practice, functions are ideal for isolating functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 488,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\n",
      "8\n",
      "17\n"
     ]
    }
   ],
   "source": [
    "def square(x):\n",
    "    output = x*x\n",
    "    return output\n",
    "\n",
    "def cube(x):\n",
    "    output = x*x*x\n",
    "    return output\n",
    "\n",
    "print( square(3) )\n",
    "print( cube(2) )\n",
    "print( square(3) + cube(2) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional parts\n",
    "nb: code block in a function is actually optional, as long as you have a return statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 489,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello world'"
      ]
     },
     "execution_count": 489,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of function without a code block \n",
    "def hello_world():\n",
    "    return 'hello world'\n",
    "\n",
    "# Call hello_world() function\n",
    "hello_world()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And also... the return statement is optional, as long as you have a code block.\n",
    "\n",
    "- If no return statement is given, the function will return `None` by default.\n",
    "- Code blocks in the function will still run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n"
     ]
    }
   ],
   "source": [
    "# Example of function without a return statement\n",
    "def print_hello_world():\n",
    "    print( 'hello world' )\n",
    "\n",
    "# Call print_hello_world() function\n",
    "print_hello_world()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arguments\n",
    "Arguments can have also default values, set using the = operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 491,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, world.\n"
     ]
    }
   ],
   "source": [
    "def print_message( message='Hello, world', punctuation='.' ):\n",
    "    output = message + punctuation\n",
    "    print( output )\n",
    "\n",
    "# Print default message\n",
    "print_message()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To pass a new value for the argument, simply set it again when calling the function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 492,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nice to meet you...\n"
     ]
    }
   ],
   "source": [
    "# Print new message, but default punctuation\n",
    "print_message( message='Nice to meet you', punctuation='...' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When passing a value to an argument, you don't have to write the argument's name if the values are in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 493,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Where is everybody?\n"
     ]
    }
   ],
   "source": [
    "# Print new message without explicity setting the argument\n",
    "print_message( 'Where is everybody', '?' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Using functions</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lets write a function called <code style=\"color:steelblue\">filter_locations</code> that takes two arguments:**\n",
    "1. <code style=\"color:steelblue\">location_list</code>\n",
    "2. <code style=\"color:steelblue\">words_list</code>\n",
    "\n",
    "The function should return the list of names in <code style=\"color:steelblue\">location_list</code> that have any word in <code style=\"color:steelblue\">words_list</code>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 494,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_locations(location_list, words_list):\n",
    "    return [name for name in location_list \n",
    "            if any(word in name for word in words_list)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's test that function. \n",
    "\n",
    "<br>\n",
    "**Lets create a new <code style=\"color:steelblue\">pleasant_locations</code> list using the function we just wrote.**\n",
    "* Pass in the list of Outer London locations and the list of pleasant-sounding words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 495,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Kensal Green', 'Wembley Park', \"Queen's Park\", 'Dollis Hill', 'Brent Park', 'Park Royal', 'Woodford Green', 'Newbury Park', 'Gants Hill']\n"
     ]
    }
   ],
   "source": [
    "# Create pleasant_locations using filter_locations()\n",
    "pleasant_locations = filter_locations(location_dict['Outer London'], pleasant_sounding)\n",
    "\n",
    "# Print list of pleasant-sounding locations\n",
    "print( pleasant_locations )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, let's use this handy function to create a <code style=\"color:steelblue\">grand_locations</code> list for locations that sound grand.**\n",
    "* Pass in the list of Outer London locations and the list of grand-sounding words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 496,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[\"Queen's Park\", 'Park Royal', 'Kingsbury', 'Seven Kings']\n"
     ]
    }
   ],
   "source": [
    "# Create grand_locations using filter_locations()\n",
    "grand_locations = filter_locations(location_dict['Outer London'], royal_sounding)\n",
    "\n",
    "# Print list of grand-sounding locations\n",
    "print(grand_locations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome, are there any locations in both lists?\n",
    "\n",
    "<br>\n",
    "**Display the locations in both the <code style=\"color:steelblue\">pleasant_locations</code> and <code style=\"color:steelblue\">grand_locations</code> lists.**\n",
    "* You can compare these lists manually because they are short, but try using <code style=\"color:steelblue\">sets</code> and the <code style=\"color:steelblue\">.intersection()</code> function (or <code style=\"color:steelblue\">&</code> operator)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 497,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"Queen's Park\", 'Park Royal'}\n",
      "{\"Queen's Park\", 'Park Royal'}\n"
     ]
    }
   ],
   "source": [
    "# Display locations that sound pleasant and grand (two ways)\n",
    "\n",
    "# Using .intersection() function\n",
    "print(set(pleasant_locations).intersection( set(grand_locations) ))\n",
    "\n",
    "# Using & operator\n",
    "print(set(pleasant_locations) & set(grand_locations)) # I prefer this!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, we'll start with these for our visit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; margin: 40px 0 40px 0;\">\n",
    "[Back to Table of Contents](#TOC)\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br id=\"l4\">\n",
    "# NumPy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import NumPy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 498,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np  # In code to left, we set an alias for NumPy to np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arrays are homogeneous"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy arrays are tables of elements that all share the same data type (usually numeric). \n",
    "\n",
    "`numpy.ndarray` is its official type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 499,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3]\n",
      "<class 'numpy.ndarray'>\n"
     ]
    }
   ],
   "source": [
    "# Array of ints\n",
    "array_a = np.array([0, 1, 2, 3])\n",
    "\n",
    "print( array_a )\n",
    "print( type(array_a) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 500,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "int64\n"
     ]
    }
   ],
   "source": [
    "# Print data type of contained elements\n",
    "print( array_a.dtype )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The NumPy array *array_a* itself has type **numpy.ndarray**.\n",
    "- The elements contained inside the array have type **int64**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Homogenous\n",
    "NumPy arrays are homogenous, which means all of their elements must have **the same data type**.\n",
    "\n",
    "Because NumPy doesn't support mixed types, it converts all of the elements in a mixed array into strings. \n",
    "\n",
    "See following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 501,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['four' '5' '6']\n"
     ]
    }
   ],
   "source": [
    "# Mixed array with 1 string and 2 integers\n",
    "array_b = np.array(['four', 5, 6])\n",
    "\n",
    "# Print elements in array_b\n",
    "print( array_b )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Shape\n",
    "The two arrays we created above, array_a and array_b both have only 1 axis.\n",
    "\n",
    "- You can think an axis as a direction in the coordinate plane.\n",
    "- For example, lines have 1 axis, squares have 2 axes, cubes have 3 axes, etc.\n",
    "\n",
    "We can use the `.shape` attribute to see the axes for a NumPy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 502,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4,)\n",
      "(3,)\n"
     ]
    }
   ],
   "source": [
    "print( array_a.shape )\n",
    "print( array_b.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, `.shape` returns a tuple.\n",
    "\n",
    "- The number of elements in the tuple is the number of axes.\n",
    "- And each element's value is the length of that axis.\n",
    "\n",
    "Together, these two pieces of information make up the shape, or dimensions, of the array.\n",
    "\n",
    "- So array_a is a 4x1 array. It has 1 axis of length 4.\n",
    "- And array_b is a 3x1 array. It has 1 axis of length 3.\n",
    " - They are both considered \"1-dimensional\" arrays."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 503,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3]\n",
      "0\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "print(array_a)\n",
    "\n",
    "# First element of array_a\n",
    "print( array_a[0] )\n",
    "\n",
    "# Last element of array_a\n",
    "print( array_a[-1] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 504,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 3]\n"
     ]
    }
   ],
   "source": [
    "# Or can SLICE it: From second element of array_a up to the 4th\n",
    "print( array_a[2:4] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing data\n",
    "- NumPy has a special `np.nan` object for denoting missing values.\n",
    "- This object is called `NaN`, which stands for Not a Number.\n",
    "\n",
    "For example, let's create an array with a missing value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 505,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.2  8.8  4.   nan  6.1]\n"
     ]
    }
   ],
   "source": [
    "# Array with missing values\n",
    "array_with_missing_value = np.array([1.2, 8.8, 4.0, np.nan, 6.1])\n",
    "\n",
    "# Print array\n",
    "print( array_with_missing_value )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 506,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "float64\n"
     ]
    }
   ],
   "source": [
    "# Print array's dtype\n",
    "print( array_with_missing_value.dtype )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Building indices locations database</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***In the previous analysis...***\n",
    "\n",
    "> *In the previous analysis, we decided on Park Royal as our first stop in our training mission to London.*\n",
    "\n",
    "Now we've arrived at the tube station!\n",
    "\n",
    "<p style=\"text-align:center; margin-top:30px; margin-bottom:30px\">\n",
    "<img alt=\"Park Royal Tube Station\" style=\"width: 320px; box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.2), 0 6px 20px 0 rgba(0, 0, 0, 0.19);\"  src=\"https://elitedatascience.com/wp-content/uploads/2017/02/park-royal-tube.jpg\">\n",
    "<br>\n",
    "<em>Park Royal Tube Station</em>\n",
    "</p>\n",
    "\n",
    "Park Royal is home to London's largest business park. \n",
    "* The business park supports around 1,700 businesses.\n",
    "* Q hands you a manual with their names and locations. \n",
    "* *They are listed in order from smallest to largest* (this will be important later).\n",
    "\n",
    "<br>\n",
    "**First, let's create a NumPy array called <code style=\"color:steelblue\">business_ids</code>.**\n",
    "* The first business has ID <code>1</code>, the second one has ID <code>2</code> and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 507,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create array of business_ids with values ranging from 1 to 1700\n",
    "business_ids = np.array(range(1, 1701))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, print the shape of <code style=\"color:steelblue\">business_ids</code> to confirm it has shape <code style=\"color:steelblue\">(1700, )</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 508,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1700,)\n"
     ]
    }
   ],
   "source": [
    "# Print shape of business_ids\n",
    "print( business_ids.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally, print the last 10 business ID's to confirm the array is set up properly.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 509,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1691 1692 1693 1694 1695 1696 1697 1698 1699 1700]\n",
      "[1691 1692 1693 1694 1695 1696 1697 1698 1699]\n"
     ]
    }
   ],
   "source": [
    "# Print last 10 business ID's\n",
    "print( business_ids[-10:] )\n",
    "\n",
    "# Wrong wway to do it\n",
    "print( business_ids[-10:-1] ) # Remember, the end number not included!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arrays are multidimensional"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multidimensional array\n",
    "For creating matrices!\n",
    "\n",
    "For example, let's create a matrix with 2 rows and 3 columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 510,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n"
     ]
    }
   ],
   "source": [
    "array_c = np.array([[1, 2, 3], [4, 5, 6]])\n",
    "print( array_c )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 511,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2, 3)\n"
     ]
    }
   ],
   "source": [
    "# The NumPy array above is 2x3.\n",
    "print( array_c.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshape\n",
    "For example, we can reshape it from its original shape of 2x3 to the new shape of 3x2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 512,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2]\n",
      " [3 4]\n",
      " [5 6]]\n"
     ]
    }
   ],
   "source": [
    "# Reshape to 3x2\n",
    "print( array_c.reshape(3, 2) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nb: this is not the same as transposing array_c, because we are keeping the same order of the elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 513,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3 4 5 6]]\n"
     ]
    }
   ],
   "source": [
    "# Reshape to 1x6\n",
    "print( array_c.reshape(1,6) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 514,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "# Print number axes\n",
    "print(array_c)\n",
    "\n",
    "print( len(array_c.reshape(1,6).shape) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flatten\n",
    "If you want to reshape it to 1x6 and reduce the number of axes, then you actually need to reshape it to (6,) with an empty second axis.\n",
    "\n",
    "- There's a shortcut for reducing an array to 1 axis: the `.flatten()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 515,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3 4 5 6]\n",
      "[1 2 3 4 5 6]\n",
      "1\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "# Reshape and reduce axes\n",
    "print( array_c.reshape(6,) )\n",
    "\n",
    "# Flatten (reduce to 1 axis) # Shortcut\n",
    "print( array_c.flatten() )\n",
    "\n",
    "# Print number of axes\n",
    "print( len(array_c.reshape(6,).shape) )\n",
    "print( len(array_c.flatten().shape) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 516,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]]\n",
      "(2, 3)\n"
     ]
    }
   ],
   "source": [
    "# Nb: the original function is immutable!\n",
    "print(array_c)\n",
    "print(array_c.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 - Transpose\n",
    "If you do want to transpose the array, you can simply `.transpose()` it!\n",
    "\n",
    "- Transposing will flip the rows and columns of an array (not just the shape).\n",
    "- Reshaping will simply change the shape, but keep the same order of elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 517,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 4]\n",
      " [2 5]\n",
      " [3 6]]\n",
      "[[1 2 3]\n",
      " [4 5 6]]\n"
     ]
    }
   ],
   "source": [
    "# Transpose\n",
    "print( array_c.transpose() )\n",
    "\n",
    "# REMEMBER the original data is still the same\n",
    "print(array_c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing\n",
    "Let's look at an example with a 3x3 array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 518,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 2 3]\n",
      " [4 5 6]\n",
      " [7 8 9]]\n"
     ]
    }
   ],
   "source": [
    "# Create a 3x3 array\n",
    "array_d = np.array(range(1,10)).reshape(3,3)\n",
    "print(array_d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 519,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 2 3]\n"
     ]
    }
   ],
   "source": [
    "# Select all elements in the first row\n",
    "# Tip: A colon (:) means select all elements along that axis.\n",
    "print( array_d[0, :] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 520,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4 5 6]\n"
     ]
    }
   ],
   "source": [
    "# Select second row and all columns\n",
    "print( array_d[1, :] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 521,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 6 9]\n"
     ]
    }
   ],
   "source": [
    "# Select third column and all rows\n",
    "print( array_d[:, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 522,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 5 8]\n"
     ]
    }
   ],
   "source": [
    "# Select second column and all rows\n",
    "print( array_d[:, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 523,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[5 6]\n",
      " [8 9]]\n"
     ]
    }
   ],
   "source": [
    "# Select after second row and second column\n",
    "print( array_d[1:, 1:] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Sampling</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are **1700** businesses in Park Royal's business park. However, you only have **10** hours to stay in Park Royal before you need to move on to the next location. Assuming you could only visit **5** businesses per hour, which businesses should you visit?\n",
    "\n",
    "Well, 2 options immediately come to mind:\n",
    "1. You could just visit the first 50 businesses.\n",
    "2. You could randomly sample 50 businesses.\n",
    "\n",
    "While these appear fine at first glance, there are potential flaws with both of these approaches. Remember, the businesses were listed in order from smallest to largest! Therefore...\n",
    "1. Just visiting the first 50 would give us a biased sample of only the smallest businesses.\n",
    "2. Visiting a random sample of 50 is better, but there's still a chance that our sample ends up biased from pure chance.\n",
    "\n",
    "Instead, let's take a **stratified random sample** based on the size of the business.\n",
    "\n",
    "### WTF is stratified random sampling?\n",
    "\n",
    "This is a very important concept for practical machine learning. \n",
    "\n",
    "<br>\n",
    "> *<span style=\"color:tomato; font-weight:bold\">Stratified random sampling</span> is first grouping your observations based on a key variable, and then randomly sampling from those groups.* \n",
    "\n",
    "<br>\n",
    "\n",
    "This will ensure your sample is **representative** of the broader dataset along that key variable.\n",
    "\n",
    "<p style=\"text-align:center\">\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/f/fa/Stratified_sampling.PNG\" alt=\"Stratified Random Sampling\" style=\"width: 280px;\"/>\n",
    "<br>\n",
    "<em>Stratified Random Sampling</em>\n",
    "</p>\n",
    "\n",
    "For this analysis, it just means that...\n",
    "* We'll start by splitting our businesses into 10 groups of 170 businesses each.\n",
    "* The first group will have ID's from 1 to 170, the second will have ID's from 171 to 340, etc.\n",
    "* Then we'll randomly select 5 businesses from each group of 170.\n",
    "* Since the businesses are already ordered by size, this will ensure that small, medium, and big businesses are all represented in our sample.\n",
    "\n",
    "<br>\n",
    "**First, reshape the 1-dimensional <code style=\"color:steelblue\">business_ids</code> array into a new 2-dimensional <code style=\"color:steelblue\">id_matrix</code> array.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 524,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(170, 10)\n"
     ]
    }
   ],
   "source": [
    "# Create id_matrix by reshaping business_ids to have 10 columns\n",
    "id_matrix = business_ids.reshape(170,10)\n",
    "\n",
    "\n",
    "# Print shape\n",
    "print(id_matrix.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 525,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   1    2    3 ...,    8    9   10]\n",
      " [  11   12   13 ...,   18   19   20]\n",
      " [  21   22   23 ...,   28   29   30]\n",
      " ..., \n",
      " [1671 1672 1673 ..., 1678 1679 1680]\n",
      " [1681 1682 1683 ..., 1688 1689 1690]\n",
      " [1691 1692 1693 ..., 1698 1699 1700]]\n"
     ]
    }
   ],
   "source": [
    "print(id_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now we have a matrix with 10 columns representing 10 groups of businesses. But remember, our goal is to stratify our sample by size of the business, and because our businesses are ordered by size, the first group should be 1 to 170, the second group should be 171 to 340, and so on.\n",
    "\n",
    "<br>\n",
    "**Print the first column (group) of <code style=\"color:steelblue\">id_matrix</code>.**\n",
    "* Does it contain businesses 1 to 170?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 526,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   1   11   21   31   41   51   61   71   81   91  101  111  121  131  141\n",
      "  151  161  171  181  191  201  211  221  231  241  251  261  271  281  291\n",
      "  301  311  321  331  341  351  361  371  381  391  401  411  421  431  441\n",
      "  451  461  471  481  491  501  511  521  531  541  551  561  571  581  591\n",
      "  601  611  621  631  641  651  661  671  681  691  701  711  721  731  741\n",
      "  751  761  771  781  791  801  811  821  831  841  851  861  871  881  891\n",
      "  901  911  921  931  941  951  961  971  981  991 1001 1011 1021 1031 1041\n",
      " 1051 1061 1071 1081 1091 1101 1111 1121 1131 1141 1151 1161 1171 1181 1191\n",
      " 1201 1211 1221 1231 1241 1251 1261 1271 1281 1291 1301 1311 1321 1331 1341\n",
      " 1351 1361 1371 1381 1391 1401 1411 1421 1431 1441 1451 1461 1471 1481 1491\n",
      " 1501 1511 1521 1531 1541 1551 1561 1571 1581 1591 1601 1611 1621 1631 1641\n",
      " 1651 1661 1671 1681 1691]\n"
     ]
    }
   ],
   "source": [
    "# Print first column of id_matrix\n",
    "print(id_matrix[:,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Crap, that's not what we wanted.\n",
    "\n",
    "Remember, when you <code style=\"color:steelblue\">.reshape</code> an array, the new array keeps the order of the elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interlude: Toy Problem**\n",
    "\n",
    "Let's walk through a miniature example of what we just did in previous analysis, because it will be easier to understand by peeking under the hood.\n",
    "\n",
    "- Instead of 1700 businesses, let's say we only had 170 businesses to visit, but we still want to group them into 10 groups by ID.\n",
    "- Therefore, we want the first group to have businesses 1 to 17, the second group to have 18 to 34, the third group to have 35 to 51, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 527,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1   2   3   4   5   6   7   8   9  10]\n",
      " [ 11  12  13  14  15  16  17  18  19  20]\n",
      " [ 21  22  23  24  25  26  27  28  29  30]\n",
      " [ 31  32  33  34  35  36  37  38  39  40]\n",
      " [ 41  42  43  44  45  46  47  48  49  50]\n",
      " [ 51  52  53  54  55  56  57  58  59  60]\n",
      " [ 61  62  63  64  65  66  67  68  69  70]\n",
      " [ 71  72  73  74  75  76  77  78  79  80]\n",
      " [ 81  82  83  84  85  86  87  88  89  90]\n",
      " [ 91  92  93  94  95  96  97  98  99 100]\n",
      " [101 102 103 104 105 106 107 108 109 110]\n",
      " [111 112 113 114 115 116 117 118 119 120]\n",
      " [121 122 123 124 125 126 127 128 129 130]\n",
      " [131 132 133 134 135 136 137 138 139 140]\n",
      " [141 142 143 144 145 146 147 148 149 150]\n",
      " [151 152 153 154 155 156 157 158 159 160]\n",
      " [161 162 163 164 165 166 167 168 169 170]]\n"
     ]
    }
   ],
   "source": [
    "# Create array of 170 ID's\n",
    "mini_business_ids = np.array( range(1, 171) )\n",
    "\n",
    "# Reshape to have 10 columns\n",
    "mini_id_matrix = mini_business_ids.reshape(17, 10)\n",
    "\n",
    "# Display mini_id_matrix\n",
    "print( mini_id_matrix )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aha, now we're on to something... so instead of reshaping to 17 rows and 10 columns, what if we reshape to 10 rows and 17 columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 528,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17]\n",
      " [ 18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34]\n",
      " [ 35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51]\n",
      " [ 52  53  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68]\n",
      " [ 69  70  71  72  73  74  75  76  77  78  79  80  81  82  83  84  85]\n",
      " [ 86  87  88  89  90  91  92  93  94  95  96  97  98  99 100 101 102]\n",
      " [103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119]\n",
      " [120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136]\n",
      " [137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153]\n",
      " [154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170]]\n"
     ]
    }
   ],
   "source": [
    "# Reshape to have 10 rows\n",
    "mini_id_matrix = mini_business_ids.reshape(10, 17)\n",
    "\n",
    "# Display mini id matrix\n",
    "print( mini_id_matrix )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have 10 rows instead of 10 columns.\n",
    "\n",
    "- The first row has the businesses 1 to 17, which is what we want!\n",
    "- And if we want the first column to have businesses 1 to 17, we can simply transpose this matrix.\n",
    "\n",
    "Nb: **Toy problems** are miniature versions of your problem that are easier to break apart and understand conceptually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try building that <code style=\"color:steelblue\">id_matrix</code> again. \n",
    "\n",
    "<br>\n",
    "**This time, reshape it to have 10 rows, each representing 1 group of businesses.**\n",
    "* The first row should contain businesses 1 to 170."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 529,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reshape business_ids to have 10 rows, with 170 businesses each\n",
    "id_matrix = business_ids.reshape(10,170)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 530,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   1    2    3 ...,  168  169  170]\n",
      " [ 171  172  173 ...,  338  339  340]\n",
      " [ 341  342  343 ...,  508  509  510]\n",
      " ..., \n",
      " [1191 1192 1193 ..., 1358 1359 1360]\n",
      " [1361 1362 1363 ..., 1528 1529 1530]\n",
      " [1531 1532 1533 ..., 1698 1699 1700]]\n"
     ]
    }
   ],
   "source": [
    "print(id_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 531,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17  18\n",
      "  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36\n",
      "  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54\n",
      "  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71  72\n",
      "  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89  90\n",
      "  91  92  93  94  95  96  97  98  99 100 101 102 103 104 105 106 107 108\n",
      " 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126\n",
      " 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144\n",
      " 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162\n",
      " 163 164 165 166 167 168 169 170]\n"
     ]
    }
   ],
   "source": [
    "# Print the first row of id_matrix\n",
    "print(id_matrix[0,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have businesses with ID's from **1 to 170**.\n",
    "\n",
    "Now, what if we still want our groups to be grouped by column, instead of by row? Let's *flip our rows and columns* now that the rows have the correct groups.\n",
    "\n",
    "<br>\n",
    "**Overwrite <code style=\"color:steelblue\">id_matrix</code> with its transposed version.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 532,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   1  171  341 ..., 1191 1361 1531]\n",
      " [   2  172  342 ..., 1192 1362 1532]\n",
      " [   3  173  343 ..., 1193 1363 1533]\n",
      " ..., \n",
      " [ 168  338  508 ..., 1358 1528 1698]\n",
      " [ 169  339  509 ..., 1359 1529 1699]\n",
      " [ 170  340  510 ..., 1360 1530 1700]]\n",
      "(170, 10)\n"
     ]
    }
   ],
   "source": [
    "# Overwrite id_matrix with flipped version\n",
    "id_matrix = id_matrix.transpose()\n",
    "\n",
    "# Print first column\n",
    "print(id_matrix)\n",
    "\n",
    "# Print shape of new id_matrix\n",
    "print(id_matrix.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Math is elementwise\n",
    "- Functions are applied to each element in the array.\n",
    "- Operations are applied between corresponding elements in two arrays.\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 533,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]]\n",
      "[[  2.   5.]\n",
      " [ 10.   3.]]\n"
     ]
    }
   ],
   "source": [
    "# 2x2 matrix of floats\n",
    "x = np.array([[1.0, 2.0], [3.0, 4.0]])\n",
    "print( x )\n",
    "\n",
    "# 2x2 matrix of floats\n",
    "y = np.array([[2.0, 5.0], [10.0, 3.0]])\n",
    "print( y )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Between two arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 534,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  3.   7.]\n",
      " [ 13.   7.]]\n"
     ]
    }
   ],
   "source": [
    "# Addition\n",
    "print( x + y ) # This is matrices addition, from linear algebra!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 535,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-1. -3.]\n",
      " [-7.  1.]]\n"
     ]
    }
   ],
   "source": [
    "# Subtraction\n",
    "print( x - y )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 536,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  2.  10.]\n",
      " [ 30.  12.]]\n"
     ]
    }
   ],
   "source": [
    "# Multiplication\n",
    "print( x * y )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 537,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.5         0.4       ]\n",
      " [ 0.3         1.33333333]]\n"
     ]
    }
   ],
   "source": [
    "# Division\n",
    "print( x / y )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 538,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  1.]]\n"
     ]
    }
   ],
   "source": [
    "# Modulo\n",
    "print( x % y )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Between array and scalar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 539,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]]\n"
     ]
    }
   ],
   "source": [
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 540,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 3.  4.]\n",
      " [ 5.  6.]]\n"
     ]
    }
   ],
   "source": [
    "# Addition\n",
    "print( x + 2 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 541,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-1.  0.]\n",
      " [ 1.  2.]]\n"
     ]
    }
   ],
   "source": [
    "# Subtraction\n",
    "print( x - 2 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 542,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2.  4.]\n",
      " [ 6.  8.]]\n"
     ]
    }
   ],
   "source": [
    "# Multiplication\n",
    "print( x * 2 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 543,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  0.]\n",
      " [ 1.  0.]]\n"
     ]
    }
   ],
   "source": [
    "# Modulo\n",
    "print( x % 2 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Math functions\n",
    "\n",
    "A few of the most useful ones include:\n",
    "\n",
    "- `np.sqrt()` for square root\n",
    "- `np.abs()` for absolute value\n",
    "- `np.power()` for raising elements to a power\n",
    "- `np.exp()` for calculating exponentials\n",
    "- `np.log()` for calculating the natural log\n",
    "\n",
    "Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 544,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[  1.,   8.],\n",
       "       [ 27.,  64.]])"
      ]
     },
     "execution_count": 544,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Cubed\n",
    "print(x)\n",
    "np.power(x, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregation functions\n",
    "\n",
    "A few of the most useful ones include:\n",
    "\n",
    "- `np.sum()` for calculating the sum\n",
    "- `np.min()` for finding the smallest element\n",
    "- `np.max()` for finding the largest element\n",
    "- `np.median()` for finding the median\n",
    "- `np.mean()` for calculating the mean\n",
    "- `np.std()` for calculating the standard deviation\n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 545,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]]\n",
      "10.0\n"
     ]
    }
   ],
   "source": [
    "print( x )\n",
    "\n",
    "\n",
    "# Sum of all elements\n",
    "print( np.sum(x) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Axis\n",
    "Now, what if we want to aggregate across columns or rows, instead of across the entire array?\n",
    "\n",
    "We can pass in an axis argument:\n",
    "\n",
    "- `axis=0` to aggregate across columns\n",
    "- `axis=1` to aggregate across rows\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 546,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 1.  2.]\n",
      " [ 3.  4.]]\n",
      "[ 4.  6.]\n"
     ]
    }
   ],
   "source": [
    "print( x )\n",
    "\n",
    "# Sum of each column\n",
    "print( np.sum(x, axis=0) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 547,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3.  7.]\n"
     ]
    }
   ],
   "source": [
    "# Sum of each row\n",
    "print( np.sum(x, axis=1) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: finding min and max</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take another look at our <code style=\"color:steelblue\">id_matrix</code>. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 548,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[   1,  171,  341, ..., 1191, 1361, 1531],\n",
       "       [   2,  172,  342, ..., 1192, 1362, 1532],\n",
       "       [   3,  173,  343, ..., 1193, 1363, 1533],\n",
       "       ..., \n",
       "       [ 168,  338,  508, ..., 1358, 1528, 1698],\n",
       "       [ 169,  339,  509, ..., 1359, 1529, 1699],\n",
       "       [ 170,  340,  510, ..., 1360, 1530, 1700]])"
      ]
     },
     "execution_count": 548,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In <span style=\"color:royalblue\">previous analysis</span> we displayed the first group (column) and confirmed that the ID's were from **1 to 170**.\n",
    "* Now let's confirm the rest of the array is correct by finding the <code style=\"color:steelblue\">np.min()</code> and <code style=\"color:steelblue\">np.max()</code> of each group.\n",
    "\n",
    "<br>\n",
    "**First, create and print an object called <code style=\"color:steelblue\">group_min</code> with the minimum ID of each group (column) of <code style=\"color:steelblue\">id_matrix</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 549,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[   1  171  341  511  681  851 1021 1191 1361 1531]\n"
     ]
    }
   ],
   "source": [
    "# Create group_min of each column\n",
    "group_min = np.min(id_matrix, axis=0) \n",
    "\n",
    "# Print group_min\n",
    "print(group_min)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, lets create and print an object called <code style=\"color:steelblue\">group_max</code> with the maximum ID of each group (column) of <code style=\"color:steelblue\">id_matrix</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 550,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 170  340  510  680  850 1020 1190 1360 1530 1700]\n"
     ]
    }
   ],
   "source": [
    "# Create group_max of each column\n",
    "group_max = np.max(id_matrix, axis=0)\n",
    "\n",
    "# Print group_max\n",
    "print(group_max)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally, subtract <code style=\"color:steelblue\">group_min</code> from <code style=\"color:steelblue\">group_max</code> to confirm that each of the 10 groups has a <span style=\"tomato\">range</span> of 170.**\n",
    "* Remember to add 1 to the difference between max and min because the ends are inclusive \n",
    "    * i.e. 170 - 1 = 169"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 551,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[170 170 170 170 170 170 170 170 170 170]\n"
     ]
    }
   ],
   "source": [
    "# Print range of each group\n",
    "\n",
    "print(group_max - group_min +1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NumPy is reliably random\n",
    "\n",
    "NumPy comes with a submodule called `np.random`.\n",
    "\n",
    "- This submodule has many useful tools for random sampling from various distributions and generating randomized data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Interlude: Tab trick**\n",
    "\n",
    "Pressing tab after typeing `np.random.`, will show all the different methods! This is called tab completion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random array\n",
    "\n",
    "Let's try generating a 8x8 matrix of random digits from 0 to 9.\n",
    "\n",
    "We can use the `np.random.randomint()` function, which draws from a uniform distribution (i.e. each integer is equally likely to be drawn).\n",
    "\n",
    "It takes 3 arguments:\n",
    "\n",
    "- low: Lower boundary for sampling (inclusive)\n",
    "- high: Upper boundary for sampling (exclusive)\n",
    "- size: Number of samples to draw\n",
    "\n",
    "First, let's generate 64 random digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 552,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 8 8 3 0 1 3 5 4 3 9 2 2 8 0 3 4 3 7 9 0 5 8 2 7 4 6 5 4 4 3 7 2 8 2 1 8\n",
      " 4 8 2 6 7 9 0 0 2 3 5 9 6 1 5 5 5 3 0 6 5 8 8 6 6 4 8]\n"
     ]
    }
   ],
   "source": [
    "# Randomly draw 64 samples from a uniform distribution from [0, 10]\n",
    "sample = np.random.randint(low=0, high=10, size=64)\n",
    "\n",
    "print( sample )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can **reshape** that array into the **8x8** matrix we wanted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 553,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1 8 8 3 0 1 3 5]\n",
      " [4 3 9 2 2 8 0 3]\n",
      " [4 3 7 9 0 5 8 2]\n",
      " [7 4 6 5 4 4 3 7]\n",
      " [2 8 2 1 8 4 8 2]\n",
      " [6 7 9 0 0 2 3 5]\n",
      " [9 6 1 5 5 5 3 0]\n",
      " [6 5 8 8 6 6 4 8]]\n"
     ]
    }
   ],
   "source": [
    "print(sample.reshape(8,8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random multidimensional array\n",
    "However, there's an easier way to create a 8x8 matrix right from the start.\n",
    "\n",
    "The size argument of `np.random.randomint()` can actually accept a tuple with one value per axis, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 554,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[3 2 8 2 4 1 0 8]\n",
      " [0 5 1 8 6 4 3 0]\n",
      " [8 8 7 9 7 8 7 6]\n",
      " [4 0 5 9 4 7 1 2]\n",
      " [6 8 4 9 4 5 2 6]\n",
      " [4 2 3 2 5 2 9 1]\n",
      " [7 6 2 5 9 4 2 9]\n",
      " [4 6 3 5 1 4 9 1]]\n"
     ]
    }
   ],
   "source": [
    "# Randomly draw 8x8 samples from a uniform distribution from [0, 10]\n",
    "generated_matrix = np.random.randint(0, 10, (8,8))\n",
    "print( generated_matrix )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nb: every time you run the code above, will generate new sample values!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random seed\n",
    "\n",
    "In machine learning, you'll often want to be able to reproduce your random samples. For example, if you need to close your file and come back to it later, you'll want to draw the same random samples so you can get consistent results.\n",
    "\n",
    "That's where `np.random.seed()` comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 555,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[7 8 7 9 7 2 2 4]\n",
      " [8 9 6 6 7 8 1 6]\n",
      " [6 2 2 9 8 1 7 3]\n",
      " [1 3 9 3 4 8 8 7]\n",
      " [9 4 1 6 4 4 2 5]\n",
      " [7 4 2 3 9 2 4 8]\n",
      " [6 3 8 8 7 4 3 2]\n",
      " [9 1 2 0 3 8 2 1]]\n"
     ]
    }
   ],
   "source": [
    "# Set seed for reproducible results\n",
    "np.random.seed(1337)\n",
    "\n",
    "# Randomly draw 8x8 samples from a uniform distribution from [0, 10)\n",
    "generated_matrix = np.random.randint(0, 10, (8,8))\n",
    "\n",
    "print( generated_matrix )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random choice\n",
    "Next, what if we want to randomly select 5 elements from the first column of generated_matrix?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 556,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7 8 6 1 9 7 6 9]\n"
     ]
    }
   ],
   "source": [
    "# Select first column of generated_matrix\n",
    "print( generated_matrix[:,0] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, if we want to randomly select 5 elements from that column, we can use `np.random.choice()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 557,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7 6 9 7 7]\n"
     ]
    }
   ],
   "source": [
    "# Set seed for reproducible results\n",
    "np.random.seed(55)\n",
    "\n",
    "# Randomly select 5 elements from first column of generated_matrix\n",
    "print( np.random.choice(generated_matrix[:,0], 5) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Replacement\n",
    "\n",
    "By default, `np.random.choice()` samples with replacement.\n",
    "\n",
    "- This just means that after each element is selected, it is replenished before selecting the next one.\n",
    "- The third argument to `np.random.choice()` tells it to sample with replacement or not.\n",
    "\n",
    "Example (to ensure no elements are repeated):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 558,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9 9 6 8 1]\n"
     ]
    }
   ],
   "source": [
    "# Set seed for reproducible results\n",
    "np.random.seed(55)\n",
    "\n",
    "# Randomly select 5 elements from first column of generated_matrix\n",
    "print( np.random.choice(generated_matrix[:,0], 5, replace=False) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Random selection</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to select 5 businesses from each group in <code style=\"color:steelblue\">id_matrix</code>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Pseudocode for code below**\n",
    "\n",
    "**Lets write a loop that chooses 5 businesses from column of <code style=\"color:steelblue\">id_matrix</code>.**\n",
    "* Set the random seed to 9001.\n",
    "* Sample without replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 559,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  7  37 124  41  17]\n"
     ]
    }
   ],
   "source": [
    "# Seed random seed\n",
    "np.random.seed(9001)\n",
    "\n",
    "# Print selected businesses from each group\n",
    "print( np.random.choice(id_matrix[:,0], 5, replace=False) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 560,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Group 1: [  7  37 124  41  17]\n",
      "Group 2: [302 261 257 323 234]\n",
      "Group 3: [464 486 463 440 474]\n",
      "Group 4: [645 582 666 631 553]\n",
      "Group 5: [699 738 705 760 792]\n",
      "Group 6: [ 995 1000  909  869  899]\n",
      "Group 7: [1042 1058 1172 1122 1153]\n",
      "Group 8: [1304 1275 1343 1344 1236]\n",
      "Group 9: [1401 1417 1383 1387 1474]\n",
      "Group 10: [1642 1687 1545 1644 1549]\n"
     ]
    }
   ],
   "source": [
    "# Seed random seed\n",
    "np.random.seed(9001)\n",
    "\n",
    "# Print selected businesses from each group\n",
    "for group in range(id_matrix.shape[1]):\n",
    "    print('Group {}: {}'.format( group + 1, np.random.choice( id_matrix[:,group], 5, replace=False ) ) ) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome, now we're ready to start visiting the businesses... Let's start with Group 1 and the business with ID 7. Lets call this **Casey's Flower Shop**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align:center; margin: 40px 0 40px 0;\">\n",
    "[Back to Table of Contents](#TOC)\n",
    "</div>\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br id=\"l5\">\n",
    "# Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fisrt import Pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 561,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pandas DataFrames\n",
    "DataFrames are like highly optimized spreadsheets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most common ways to create a DataFrame is to pass a dictionary into the `pd.DataFrame()` function, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 562,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>column_1</th>\n",
       "      <th>column_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>b</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>c</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   column_1 column_2\n",
       "0         5        a\n",
       "1         4        b\n",
       "2         3        c"
      ]
     },
     "execution_count": 562,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_dataframe = pd.DataFrame({\n",
    "        'column_1' : [5, 4, 3],\n",
    "        'column_2' : ['a', 'b', 'c']\n",
    "    })\n",
    "\n",
    "example_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing\n",
    "- The one most useful way for us is by importing CSV (comma-separated) files.\n",
    "- Pandas has a handy `pd.read_csv()` function just for this purpose.\n",
    "- Pandas can also read from Excel, JSON, SQL, and many other formats. \n",
    "\n",
    "Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 563,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "# Read the iris dataset from a CSV file in our project_files folder\n",
    "df = pd.read_csv('project_files/iris.csv')\n",
    "\n",
    "# Print data type for df\n",
    "print( type(df) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 564,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal_length  sepal_width  petal_length  petal_width species\n",
       "0           5.1          3.5           1.4          0.2  setosa\n",
       "1           4.9          3.0           1.4          0.2  setosa\n",
       "2           4.7          3.2           1.3          0.2  setosa\n",
       "3           4.6          3.1           1.5          0.2  setosa\n",
       "4           5.0          3.6           1.4          0.2  setosa"
      ]
     },
     "execution_count": 564,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display the first 5 rows of the dataframe\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shape\n",
    "- Pandas DataFrames always have 2 axes (i.e. like a spreadsheet).\n",
    "- The first value in the shape tuple is the number of rows, and the second is the number of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 565,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(150, 5)\n"
     ]
    }
   ],
   "source": [
    "# Shape of dataframe\n",
    "print( df.shape )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 566,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "150\n",
      "150\n"
     ]
    }
   ],
   "source": [
    "# Number of rows (both of below are equivalent)\n",
    "print( len(df) )\n",
    "print( df.shape[0] )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 567,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sepal_length       4.3\n",
      "sepal_width          2\n",
      "petal_length         1\n",
      "petal_width        0.1\n",
      "species         setosa\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Print min value of each column in data frame df\n",
    "print( df.min() )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 568,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sepal_length          7.9\n",
      "sepal_width           4.4\n",
      "petal_length          6.9\n",
      "petal_width           2.5\n",
      "species         virginica\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Print max value of each column in data frame df\n",
    "print( df.max() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, one very quick way to summarize your data is with the `.describe()` function.\n",
    "\n",
    "It will display many statistics at once, including:\n",
    "\n",
    "- mean\n",
    "- standard deviation\n",
    "- mininum\n",
    "- 25th percentile\n",
    "- median\n",
    "- 75th percentile\n",
    "- maximum\n",
    "\n",
    "of each column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 569,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "      <td>150.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.843333</td>\n",
       "      <td>3.057333</td>\n",
       "      <td>3.758000</td>\n",
       "      <td>1.199333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.828066</td>\n",
       "      <td>0.435866</td>\n",
       "      <td>1.765298</td>\n",
       "      <td>0.762238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.300000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>5.100000</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>0.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.800000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.350000</td>\n",
       "      <td>1.300000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.400000</td>\n",
       "      <td>3.300000</td>\n",
       "      <td>5.100000</td>\n",
       "      <td>1.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>7.900000</td>\n",
       "      <td>4.400000</td>\n",
       "      <td>6.900000</td>\n",
       "      <td>2.500000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sepal_length  sepal_width  petal_length  petal_width\n",
       "count    150.000000   150.000000    150.000000   150.000000\n",
       "mean       5.843333     3.057333      3.758000     1.199333\n",
       "std        0.828066     0.435866      1.765298     0.762238\n",
       "min        4.300000     2.000000      1.000000     0.100000\n",
       "25%        5.100000     2.800000      1.600000     0.300000\n",
       "50%        5.800000     3.000000      4.350000     1.300000\n",
       "75%        6.400000     3.300000      5.100000     1.800000\n",
       "max        7.900000     4.400000      6.900000     2.500000"
      ]
     },
     "execution_count": 569,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display summary statistics for each variable\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Exploring the data</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***In the previous analysis...***\n",
    "\n",
    "> * we discovered that **Casey's Flower Shop** is the first local business we need to visit in Park Royal.\n",
    "\n",
    "\n",
    "When we arrive, we find out that the owner, Casey, urgently needs our help!\n",
    "> * They just received a new shipment of **Iris flowers**, but they've never stocked these flowers before.\n",
    "* Casey asks us to share what we know about these new flowers.\n",
    "* Luckily, we have the Iris dataset to learn more about them ourselves!\n",
    "\n",
    "While 150 observations is not exactly \"big data,\" it's still too large to fit on our screen at once. So let's use another toy problem to practice the concepts.\n",
    "\n",
    "<br>\n",
    "First, lets create a new DataFrame called <code style=\"color:steelblue\">toy_df</code>. It will contain the first 5 rows plus the last 5 rows from our original Iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 570,
   "metadata": {},
   "outputs": [],
   "source": [
    "toy_df = pd.concat([df.head(), df.tail()]) #pd.concat combines or concatinates the top five and last five"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, display <code style=\"color:steelblue\">toy_df</code>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 571,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>virginica</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species\n",
       "0             5.1          3.5           1.4          0.2     setosa\n",
       "1             4.9          3.0           1.4          0.2     setosa\n",
       "2             4.7          3.2           1.3          0.2     setosa\n",
       "3             4.6          3.1           1.5          0.2     setosa\n",
       "4             5.0          3.6           1.4          0.2     setosa\n",
       "145           6.7          3.0           5.2          2.3  virginica\n",
       "146           6.3          2.5           5.0          1.9  virginica\n",
       "147           6.5          3.0           5.2          2.0  virginica\n",
       "148           6.2          3.4           5.4          2.3  virginica\n",
       "149           5.9          3.0           5.1          1.8  virginica"
      ]
     },
     "execution_count": 571,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display toy_df\n",
    "toy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, display a summary table for <code style=\"color:steelblue\">toy_df</code>.**\n",
    "* It will show the mean, standard deviation, and quartiles for each of the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 572,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>5.590000</td>\n",
       "      <td>3.130000</td>\n",
       "      <td>3.290000</td>\n",
       "      <td>1.130000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.807534</td>\n",
       "      <td>0.316403</td>\n",
       "      <td>1.995244</td>\n",
       "      <td>0.992248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.600000</td>\n",
       "      <td>2.500000</td>\n",
       "      <td>1.300000</td>\n",
       "      <td>0.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>4.925000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.400000</td>\n",
       "      <td>0.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>5.500000</td>\n",
       "      <td>3.050000</td>\n",
       "      <td>3.250000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.275000</td>\n",
       "      <td>3.350000</td>\n",
       "      <td>5.175000</td>\n",
       "      <td>1.975000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.700000</td>\n",
       "      <td>3.600000</td>\n",
       "      <td>5.400000</td>\n",
       "      <td>2.300000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       sepal_length  sepal_width  petal_length  petal_width\n",
       "count     10.000000    10.000000     10.000000    10.000000\n",
       "mean       5.590000     3.130000      3.290000     1.130000\n",
       "std        0.807534     0.316403      1.995244     0.992248\n",
       "min        4.600000     2.500000      1.300000     0.200000\n",
       "25%        4.925000     3.000000      1.400000     0.200000\n",
       "50%        5.500000     3.050000      3.250000     1.000000\n",
       "75%        6.275000     3.350000      5.175000     1.975000\n",
       "max        6.700000     3.600000      5.400000     2.300000"
      ]
     },
     "execution_count": 572,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Describe toy_df\n",
    "toy_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Series are single columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas Series\n",
    "- Series are single columns of data from DataFrames.\n",
    "- `pandas.core.series.Series` is the official type.\n",
    "- You can directly create a Series from a list. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 573,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    0\n",
      "1    1\n",
      "2    2\n",
      "3    3\n",
      "4    4\n",
      "dtype: int64\n",
      "<class 'pandas.core.series.Series'>\n"
     ]
    }
   ],
   "source": [
    "integer_series = pd.Series([0, 1, 2, 3, 4])\n",
    "\n",
    "print( integer_series )\n",
    "print( type(integer_series) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting from DataFrames\n",
    "\n",
    "To select `petal_length` (two ways):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 574,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.series.Series'>\n",
      "<class 'pandas.core.series.Series'>\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "# Way 1\n",
    "print( type(df.petal_length) )\n",
    "\n",
    "# Way 2\n",
    "print( type(df['petal_length']) )\n",
    "\n",
    "# Check that both ways are identical\n",
    "print( all(df.petal_length == df['petal_length']) ) #Setting to all returns True if all true, or False otherwise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 575,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    1.4\n",
      "1    1.4\n",
      "2    1.3\n",
      "3    1.5\n",
      "4    1.4\n",
      "Name: petal_length, dtype: float64\n",
      "The minimum petal length is 1.0\n",
      "The maximum petal length is 6.9\n"
     ]
    }
   ],
   "source": [
    "# First 5 values of petal length\n",
    "print( df.petal_length.head() )\n",
    "\n",
    "# Minimum sepal length\n",
    "print( 'The minimum petal length is', df.petal_length.min() )\n",
    "\n",
    "# Maximum sepal length\n",
    "print( 'The maximum petal length is', df.petal_length.max() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unique values\n",
    "For categorical (i.e. non-numeric) variables, it's useful to know which unique values they have.\n",
    "\n",
    "- In the Iris dataset, the only categorical variable is 'species'\n",
    "- Each unique value is also called a **class**.\n",
    "- By the way, building a model to predict a categorical variable is called classification.\n",
    "\n",
    "To find the unique classes, you can use the `.unique()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 576,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['setosa' 'versicolor' 'virginica']\n"
     ]
    }
   ],
   "source": [
    "# Print unique species\n",
    "print( df.species.unique() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series math\n",
    "Series behave very similarly to 1-dimensional arrays from NumPy.\n",
    "\n",
    "For example, you can directly apply many NumPy math functions to Pandas Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 577,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.758\n",
      "3.758\n"
     ]
    }
   ],
   "source": [
    "# Pandas\n",
    "print( df.petal_length.mean() )\n",
    "\n",
    "# NumPy\n",
    "import numpy as np\n",
    "print( np.mean( df.petal_length ) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 578,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new petal area feature\n",
    "df['petal_area'] = df.petal_width * df.petal_length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: Feature engineering</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Elementwise operations are very useful in machine learning, especially for feature engineering.\n",
    "\n",
    "<br>\n",
    "\n",
    "> *<span style=\"color:tomato; font-weight:bold\">Feature engineering</span> is the process of creating new features (model input variables) from existing ones.*\n",
    "\n",
    "<br>\n",
    "\n",
    "Let's first use our <code style=\"color:steelblue\">toy_df</code> to illustrate the concept.\n",
    "\n",
    "In the Iris dataset, we have petal width and length, but what if we wanted to know petal area? Well, we can create a new <code style=\"color:steelblue\">petal_area</code> feature.\n",
    "\n",
    "<br>\n",
    "**First, display the two columns of <code style=\"color:steelblue\">petal_width</code> and <code style=\"color:steelblue\">petal_length</code> in <code style=\"color:steelblue\">toy_df</code>.**\n",
    "* **Tip:** You can index a DataFrame using a list of column names too, like so:\n",
    "\n",
    "\n",
    "<pre style=\"color:steelblue\">df[['column_1', 'column_2']]</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 579,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>petal_width</th>\n",
       "      <th>petal_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.2</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.2</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.2</td>\n",
       "      <td>1.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.2</td>\n",
       "      <td>1.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.2</td>\n",
       "      <td>1.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>2.3</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>1.9</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>2.0</td>\n",
       "      <td>5.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>2.3</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>1.8</td>\n",
       "      <td>5.1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     petal_width  petal_length\n",
       "0            0.2           1.4\n",
       "1            0.2           1.4\n",
       "2            0.2           1.3\n",
       "3            0.2           1.5\n",
       "4            0.2           1.4\n",
       "145          2.3           5.2\n",
       "146          1.9           5.0\n",
       "147          2.0           5.2\n",
       "148          2.3           5.4\n",
       "149          1.8           5.1"
      ]
     },
     "execution_count": 579,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display petal_width and petal_length\n",
    "toy_df[['petal_width', 'petal_length']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, create a new <code style=\"color:steelblue\">petal_area</code> feature in <code style=\"color:steelblue\">toy_df</code>.**\n",
    "We multiply the <code style=\"color:steelblue\">petal_width</code> column by the <code style=\"color:steelblue\">petal_length</code> column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 580,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>11.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>virginica</td>\n",
       "      <td>10.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species  \\\n",
       "0             5.1          3.5           1.4          0.2     setosa   \n",
       "1             4.9          3.0           1.4          0.2     setosa   \n",
       "2             4.7          3.2           1.3          0.2     setosa   \n",
       "3             4.6          3.1           1.5          0.2     setosa   \n",
       "4             5.0          3.6           1.4          0.2     setosa   \n",
       "145           6.7          3.0           5.2          2.3  virginica   \n",
       "146           6.3          2.5           5.0          1.9  virginica   \n",
       "147           6.5          3.0           5.2          2.0  virginica   \n",
       "148           6.2          3.4           5.4          2.3  virginica   \n",
       "149           5.9          3.0           5.1          1.8  virginica   \n",
       "\n",
       "     petal_area  \n",
       "0          0.28  \n",
       "1          0.28  \n",
       "2          0.26  \n",
       "3          0.30  \n",
       "4          0.28  \n",
       "145       11.96  \n",
       "146        9.50  \n",
       "147       10.40  \n",
       "148       12.42  \n",
       "149        9.18  "
      ]
     },
     "execution_count": 580,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new petal_area column\n",
    "toy_df['petal_area'] = toy_df.petal_width * toy_df.petal_length\n",
    "\n",
    "# Display toy_df\n",
    "toy_df\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By creating a <code style=\"color:steelblue\">petal_area</code> feature, it's now much easier to see that virginica flowers have significantly larger petals than setosa flowers do!\n",
    "\n",
    "Often, by creating new features, you can learn more about the data (and improve your machine learning models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Boolean masks allow filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Boolean masks\n",
    "\n",
    "Boolean masks are list-like sequences of True/False (boolean) values. These allow you to filter DataFrames in enlightening ways.\n",
    "\n",
    "\"List-like\" sequences include:\n",
    "\n",
    "- lists\n",
    "- NumPy arrays\n",
    "- Pandas Series\n",
    "\n",
    "For example, both of these are valid boolean masks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 581,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_mask = [True, True, False, True]\n",
    "series_mask = pd.Series([True, True, False, True])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 582,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     True\n",
      "1     True\n",
      "2    False\n",
      "3     True\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "print( series_mask )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional boolean masks\n",
    "Often, the more useful way to create a boolean mask is by directly applying a conditional statement to another Pandas Series.\n",
    "\n",
    "For example, let's say we had the following series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 583,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    10\n",
      "1     5\n",
      "2    -3\n",
      "3     2\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "example_series = pd.Series([10, 5, -3, 2])\n",
    "\n",
    "print( example_series )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to create a boolean mask for all of the positive values in example_series, we can do so like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 584,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     True\n",
      "1     True\n",
      "2    False\n",
      "3     True\n",
      "dtype: bool\n"
     ]
    }
   ],
   "source": [
    "# Create boolean mask from a condition\n",
    "series_mask = example_series > 0\n",
    "\n",
    "print( series_mask )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering\n",
    "Now, we can actually use that boolean mask to filter our Series and keep only the positive observations.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 585,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    10\n",
       "1     5\n",
       "3     2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 585,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Keep only True values from the boolean mask\n",
    "example_series[series_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inverted filtering\n",
    "Or, by using the tilde `~` operator (called the invert operator), we can filter our Series and keep only non-positive observations.\n",
    "\n",
    "This is equivalent to keeping only the False values from our boolean mask.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 586,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2   -3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 586,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Keep only False values from the boolean mask\n",
    "example_series[~series_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 587,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>15.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.6</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>15.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>7.7</td>\n",
       "      <td>3.8</td>\n",
       "      <td>6.7</td>\n",
       "      <td>2.2</td>\n",
       "      <td>virginica</td>\n",
       "      <td>14.74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>7.7</td>\n",
       "      <td>2.6</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>15.87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>135</th>\n",
       "      <td>7.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>14.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>14.25</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species  \\\n",
       "100           6.3          3.3           6.0          2.5  virginica   \n",
       "109           7.2          3.6           6.1          2.5  virginica   \n",
       "117           7.7          3.8           6.7          2.2  virginica   \n",
       "118           7.7          2.6           6.9          2.3  virginica   \n",
       "135           7.7          3.0           6.1          2.3  virginica   \n",
       "144           6.7          3.3           5.7          2.5  virginica   \n",
       "\n",
       "     petal_area  \n",
       "100       15.00  \n",
       "109       15.25  \n",
       "117       14.74  \n",
       "118       15.87  \n",
       "135       14.03  \n",
       "144       14.25  "
      ]
     },
     "execution_count": 587,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display [i.e., index] observations where petal_area > 14\n",
    "df[df.petal_area > 14]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indicator variables\n",
    "Indicator variables are variables that can take on one of two values:\n",
    "- 1 if a condition is met.\n",
    "- 0 if a condition is not met.\n",
    "\n",
    "In machine learning, we want the values to be 1/0 instead of True/False because our algorithms will require numeric inputs.\n",
    "\n",
    "Fortunately, you can convert a boolean mask into 1/0's using `.astype(int)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 588,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    1\n",
      "1    0\n",
      "2    0\n",
      "3    1\n",
      "4    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Example boolean Series\n",
    "example_mask = pd.Series([True, False, False, True, False])\n",
    "\n",
    "# Convert boolean Series into 1/0\n",
    "print( example_mask.astype(int) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, if we want to create an indicator variable for petal_area > 14, we can do so in one line of code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 589,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "      <th>giant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.26</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal_length  sepal_width  petal_length  petal_width species  petal_area  \\\n",
       "0           5.1          3.5           1.4          0.2  setosa        0.28   \n",
       "1           4.9          3.0           1.4          0.2  setosa        0.28   \n",
       "2           4.7          3.2           1.3          0.2  setosa        0.26   \n",
       "3           4.6          3.1           1.5          0.2  setosa        0.30   \n",
       "4           5.0          3.6           1.4          0.2  setosa        0.28   \n",
       "\n",
       "   giant  \n",
       "0      0  \n",
       "1      0  \n",
       "2      0  \n",
       "3      0  \n",
       "4      0  "
      ]
     },
     "execution_count": 589,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create indicator variable for petal_area > 14\n",
    "df['giant'] = (df.petal_area > 14).astype(int)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Multiple masks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "DataFrames can be indexed by any number of masks, separated by an & operator.\n",
    "\n",
    "Let's say we wanted to see only versicolor and virginica flowers with sepal_length > 3.2.\n",
    "\n",
    "- We can use the `.isin()` function to see if species is either versicolor or virginica.\n",
    "- To keep our code clean, if we have multiple masks, we might set them to separate variables.\n",
    "- We'll then combine the masks using the & operator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 590,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "      <th>giant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>4.7</td>\n",
       "      <td>1.6</td>\n",
       "      <td>versicolor</td>\n",
       "      <td>7.52</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>6.0</td>\n",
       "      <td>3.4</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1.6</td>\n",
       "      <td>versicolor</td>\n",
       "      <td>7.20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>100</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.3</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>15.00</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>109</th>\n",
       "      <td>7.2</td>\n",
       "      <td>3.6</td>\n",
       "      <td>6.1</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>15.25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>7.7</td>\n",
       "      <td>3.8</td>\n",
       "      <td>6.7</td>\n",
       "      <td>2.2</td>\n",
       "      <td>virginica</td>\n",
       "      <td>14.74</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>124</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.1</td>\n",
       "      <td>virginica</td>\n",
       "      <td>11.97</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>131</th>\n",
       "      <td>7.9</td>\n",
       "      <td>3.8</td>\n",
       "      <td>6.4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.80</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>136</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.6</td>\n",
       "      <td>2.4</td>\n",
       "      <td>virginica</td>\n",
       "      <td>13.44</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>144</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.3</td>\n",
       "      <td>5.7</td>\n",
       "      <td>2.5</td>\n",
       "      <td>virginica</td>\n",
       "      <td>14.25</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.42</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width     species  \\\n",
       "56            6.3          3.3           4.7          1.6  versicolor   \n",
       "85            6.0          3.4           4.5          1.6  versicolor   \n",
       "100           6.3          3.3           6.0          2.5   virginica   \n",
       "109           7.2          3.6           6.1          2.5   virginica   \n",
       "117           7.7          3.8           6.7          2.2   virginica   \n",
       "124           6.7          3.3           5.7          2.1   virginica   \n",
       "131           7.9          3.8           6.4          2.0   virginica   \n",
       "136           6.3          3.4           5.6          2.4   virginica   \n",
       "144           6.7          3.3           5.7          2.5   virginica   \n",
       "148           6.2          3.4           5.4          2.3   virginica   \n",
       "\n",
       "     petal_area  giant  \n",
       "56         7.52      0  \n",
       "85         7.20      0  \n",
       "100       15.00      1  \n",
       "109       15.25      1  \n",
       "117       14.74      1  \n",
       "124       11.97      0  \n",
       "131       12.80      0  \n",
       "136       13.44      0  \n",
       "144       14.25      1  \n",
       "148       12.42      0  "
      ]
     },
     "execution_count": 590,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Versicolor or virginica\n",
    "species_mask = df.species.isin(['versicolor', 'virginica'])\n",
    "\n",
    "# Sepal width > 3.2\n",
    "sepal_width_mask = df.sepal_width > 3.2\n",
    "\n",
    "# Index with both masks\n",
    "df[species_mask & sepal_width_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analaysis: Advanced filtering with boolean masks</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we'll use the <code style=\"color:steelblue\">toy_df</code> to really drive home these concepts.\n",
    "\n",
    "Let's say we wanted to display observations where <code style=\"color:steelblue\">petal_area > 10</code> and <code style=\"color:steelblue\">sepal_width > 3</code>.\n",
    "\n",
    "<br>\n",
    "**First, display <code style=\"color:steelblue\">toy_df</code> again.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 591,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>11.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>virginica</td>\n",
       "      <td>10.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species  \\\n",
       "0             5.1          3.5           1.4          0.2     setosa   \n",
       "1             4.9          3.0           1.4          0.2     setosa   \n",
       "2             4.7          3.2           1.3          0.2     setosa   \n",
       "3             4.6          3.1           1.5          0.2     setosa   \n",
       "4             5.0          3.6           1.4          0.2     setosa   \n",
       "145           6.7          3.0           5.2          2.3  virginica   \n",
       "146           6.3          2.5           5.0          1.9  virginica   \n",
       "147           6.5          3.0           5.2          2.0  virginica   \n",
       "148           6.2          3.4           5.4          2.3  virginica   \n",
       "149           5.9          3.0           5.1          1.8  virginica   \n",
       "\n",
       "     petal_area  \n",
       "0          0.28  \n",
       "1          0.28  \n",
       "2          0.26  \n",
       "3          0.30  \n",
       "4          0.28  \n",
       "145       11.96  \n",
       "146        9.50  \n",
       "147       10.40  \n",
       "148       12.42  \n",
       "149        9.18  "
      ]
     },
     "execution_count": 591,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display toy_df\n",
    "toy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Create a boolean mask for <code style=\"color:steelblue\">petal_area > 10</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 592,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      False\n",
       "1      False\n",
       "2      False\n",
       "3      False\n",
       "4      False\n",
       "145     True\n",
       "146    False\n",
       "147     True\n",
       "148     True\n",
       "149    False\n",
       "Name: petal_area, dtype: bool"
      ]
     },
     "execution_count": 592,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mask for petal_area > 10\n",
    "petal_area_mask = toy_df.petal_area > 10\n",
    "\n",
    "# Display petal_area_mask\n",
    "petal_area_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, create a boolean mask for <code style=\"color:steelblue\">sepal_width > 3</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 593,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       True\n",
       "1      False\n",
       "2       True\n",
       "3       True\n",
       "4       True\n",
       "145    False\n",
       "146    False\n",
       "147    False\n",
       "148     True\n",
       "149    False\n",
       "Name: sepal_width, dtype: bool"
      ]
     },
     "execution_count": 593,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Mask for sepal_width > 3\n",
    "sepal_width_mask = toy_df.sepal_width > 3\n",
    "\n",
    "# Display sepal_width_mask\n",
    "sepal_width_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Next, display the two masks combined using the <code style=\"color:steelblue\">&</code> operator.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 594,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      False\n",
       "1      False\n",
       "2      False\n",
       "3      False\n",
       "4      False\n",
       "145    False\n",
       "146    False\n",
       "147    False\n",
       "148     True\n",
       "149    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 594,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display both masks, combined\n",
    "petal_area_mask & sepal_width_mask\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally, select the observations from <code style=\"color:steelblue\">toy_df</code> where both conditions are met.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 595,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species  \\\n",
       "148           6.2          3.4           5.4          2.3  virginica   \n",
       "\n",
       "     petal_area  \n",
       "148       12.42  "
      ]
     },
     "execution_count": 595,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Index with both masks\n",
    "toy_df[petal_area_mask & sepal_width_mask]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby allow segmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Groupby\n",
    "The `.groupby()` function allows you to segment and summarize data across different classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, let's say we wanted to find the average measurements for each of the 3 species of Iris flowers.\n",
    "\n",
    "- First, take our DataFrame and group by species: `df.groupby('species')`\n",
    "- Then, simply take the mean by column: `df.groupby('species').mean()`\n",
    "\n",
    "Here's how the output looks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 596,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>petal_area</th>\n",
       "      <th>giant</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>species</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>setosa</th>\n",
       "      <td>5.006</td>\n",
       "      <td>3.428</td>\n",
       "      <td>1.462</td>\n",
       "      <td>0.246</td>\n",
       "      <td>0.3656</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>versicolor</th>\n",
       "      <td>5.936</td>\n",
       "      <td>2.770</td>\n",
       "      <td>4.260</td>\n",
       "      <td>1.326</td>\n",
       "      <td>5.7204</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>virginica</th>\n",
       "      <td>6.588</td>\n",
       "      <td>2.974</td>\n",
       "      <td>5.552</td>\n",
       "      <td>2.026</td>\n",
       "      <td>11.2962</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            sepal_length  sepal_width  petal_length  petal_width  petal_area  \\\n",
       "species                                                                        \n",
       "setosa             5.006        3.428         1.462        0.246      0.3656   \n",
       "versicolor         5.936        2.770         4.260        1.326      5.7204   \n",
       "virginica          6.588        2.974         5.552        2.026     11.2962   \n",
       "\n",
       "            giant  \n",
       "species            \n",
       "setosa       0.00  \n",
       "versicolor   0.00  \n",
       "virginica    0.12  "
      ]
     },
     "execution_count": 596,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display average measurements for each species\n",
    "df.groupby('species').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Agg\n",
    "Finally, what if we wanted to display multiple pieces of information (or **aggregations**)?\n",
    "\n",
    "Well, we can use the `.agg()` function and pass in a list of aggregations, like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 597,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">sepal_length</th>\n",
       "      <th colspan=\"3\" halign=\"left\">sepal_width</th>\n",
       "      <th colspan=\"3\" halign=\"left\">petal_length</th>\n",
       "      <th colspan=\"3\" halign=\"left\">petal_width</th>\n",
       "      <th colspan=\"3\" halign=\"left\">petal_area</th>\n",
       "      <th colspan=\"3\" halign=\"left\">giant</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>median</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>species</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>setosa</th>\n",
       "      <td>4.3</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.8</td>\n",
       "      <td>2.3</td>\n",
       "      <td>3.4</td>\n",
       "      <td>4.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.50</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.300</td>\n",
       "      <td>0.96</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>versicolor</th>\n",
       "      <td>4.9</td>\n",
       "      <td>5.9</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.8</td>\n",
       "      <td>3.4</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.35</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.3</td>\n",
       "      <td>1.8</td>\n",
       "      <td>3.30</td>\n",
       "      <td>5.615</td>\n",
       "      <td>8.64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>virginica</th>\n",
       "      <td>4.9</td>\n",
       "      <td>6.5</td>\n",
       "      <td>7.9</td>\n",
       "      <td>2.2</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.8</td>\n",
       "      <td>4.5</td>\n",
       "      <td>5.55</td>\n",
       "      <td>6.9</td>\n",
       "      <td>1.4</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>7.50</td>\n",
       "      <td>11.445</td>\n",
       "      <td>15.87</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           sepal_length             sepal_width             petal_length  \\\n",
       "                    min median  max         min median  max          min   \n",
       "species                                                                    \n",
       "setosa              4.3    5.0  5.8         2.3    3.4  4.4          1.0   \n",
       "versicolor          4.9    5.9  7.0         2.0    2.8  3.4          3.0   \n",
       "virginica           4.9    6.5  7.9         2.2    3.0  3.8          4.5   \n",
       "\n",
       "                       petal_width             petal_area                 \\\n",
       "           median  max         min median  max        min  median    max   \n",
       "species                                                                    \n",
       "setosa       1.50  1.9         0.1    0.2  0.6       0.11   0.300   0.96   \n",
       "versicolor   4.35  5.1         1.0    1.3  1.8       3.30   5.615   8.64   \n",
       "virginica    5.55  6.9         1.4    2.0  2.5       7.50  11.445  15.87   \n",
       "\n",
       "           giant             \n",
       "             min median max  \n",
       "species                      \n",
       "setosa         0      0   0  \n",
       "versicolor     0      0   0  \n",
       "virginica      0      0   1  "
      ]
     },
     "execution_count": 597,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display min, median, max measurements for each species\n",
    "df.groupby('species').agg(['min', 'median', 'max'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:SteelBlue;;\">Analysis: aggregating using groupby</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, armed with the power of **groupby**, we're almost ready to return to Casey and share what we've learned about Iris flowers! But before we do, let's just bring back our <code style=\"color:steelblue\">toy_df</code> for one last hoorah, just to make sure we know what's going on under the hood.\n",
    "\n",
    "<br>\n",
    "Let's calculate the median <code style=\"color:steelblue\">petal_area</code> for each species. \n",
    "* Since <code style=\"color:steelblue\">toy_df</code> is small, we can do this manually as well and check to make sure the values are correct.\n",
    "\n",
    "<br>\n",
    "**First, let's manually calculate the median <code style=\"color:steelblue\">petal_area</code> for the virginica flowers in our <code style=\"steelblue\">toy_df</code>.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 598,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>149</th>\n",
       "      <td>5.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.1</td>\n",
       "      <td>1.8</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>146</th>\n",
       "      <td>6.3</td>\n",
       "      <td>2.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>virginica</td>\n",
       "      <td>9.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147</th>\n",
       "      <td>6.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>virginica</td>\n",
       "      <td>10.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>6.7</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>11.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>148</th>\n",
       "      <td>6.2</td>\n",
       "      <td>3.4</td>\n",
       "      <td>5.4</td>\n",
       "      <td>2.3</td>\n",
       "      <td>virginica</td>\n",
       "      <td>12.42</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     sepal_length  sepal_width  petal_length  petal_width    species  \\\n",
       "149           5.9          3.0           5.1          1.8  virginica   \n",
       "146           6.3          2.5           5.0          1.9  virginica   \n",
       "147           6.5          3.0           5.2          2.0  virginica   \n",
       "145           6.7          3.0           5.2          2.3  virginica   \n",
       "148           6.2          3.4           5.4          2.3  virginica   \n",
       "\n",
       "     petal_area  \n",
       "149        9.18  \n",
       "146        9.50  \n",
       "147       10.40  \n",
       "145       11.96  \n",
       "148       12.42  "
      ]
     },
     "execution_count": 598,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display all 'virginica' species; values sorted by petal_area (both work)\n",
    "\n",
    "# Way 1\n",
    "toy_df[toy_df.species.isin(['virginica'])].sort_values('petal_area')\n",
    "\n",
    "# Way 2\n",
    "toy_df[toy_df.species == 'virginica'].sort_values(by='petal_area')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the output above, what's median <code style=\"color:steelblue\">petal_area</code> for the virginica species?\n",
    "\n",
    "<br>\n",
    "**Next, let's manually calculate the median <code style=\"color:steelblue\">petal_area</code> for the setosa flowers in our <code style=\"steelblue\">toy_df</code>.**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 599,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>species</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.7</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.3</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.1</td>\n",
       "      <td>3.5</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5.0</td>\n",
       "      <td>3.6</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.6</td>\n",
       "      <td>3.1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>setosa</td>\n",
       "      <td>0.30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sepal_length  sepal_width  petal_length  petal_width species  petal_area\n",
       "2           4.7          3.2           1.3          0.2  setosa        0.26\n",
       "0           5.1          3.5           1.4          0.2  setosa        0.28\n",
       "1           4.9          3.0           1.4          0.2  setosa        0.28\n",
       "4           5.0          3.6           1.4          0.2  setosa        0.28\n",
       "3           4.6          3.1           1.5          0.2  setosa        0.30"
      ]
     },
     "execution_count": 599,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display all 'setosa' species (both ways work the same)\n",
    "\n",
    "# Way 1\n",
    "toy_df[toy_df.species.isin(['setosa'])].sort_values('petal_area')\n",
    "\n",
    "# Way 2\n",
    "toy_df[toy_df.species == 'setosa'].sort_values(by='petal_area')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the output above, what's median <code style=\"color:steelblue\">petal_area</code> for the setosa species?\n",
    "\n",
    "<br>\n",
    "**Finally let's calculate the median values using a <code style=\"color:steelblue\">.groupby()</code>.**\n",
    "* Should get the same result!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 600,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sepal_length</th>\n",
       "      <th>sepal_width</th>\n",
       "      <th>petal_length</th>\n",
       "      <th>petal_width</th>\n",
       "      <th>petal_area</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>species</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>setosa</th>\n",
       "      <td>4.9</td>\n",
       "      <td>3.2</td>\n",
       "      <td>1.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>virginica</th>\n",
       "      <td>6.3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           sepal_length  sepal_width  petal_length  petal_width  petal_area\n",
       "species                                                                    \n",
       "setosa              4.9          3.2           1.4          0.2        0.28\n",
       "virginica           6.3          3.0           5.2          2.0       10.40"
      ]
     },
     "execution_count": 600,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Median petal_area in toy_df of all features\n",
    "toy_df.groupby('species').median()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The End"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "toc": {
   "nav_menu": {
    "height": "610px",
    "width": "330px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "409px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}