Day1/Day1-Activity.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UCLA ITS Data Camp, Day 1\n",
    "## Retrieving Data via API Calls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An increasingly common way to retrieve data is via an application programming interface (API), which refers to a framework of communication between you (client) and some server (computer) somewhere else in the world ([here](https://medium.com/@perrysetgo/what-exactly-is-an-api-69f36968a41f) for a high-level overview). You don't need to know how to program to make a basic API call; in fact, you are making API calls every day as you browse the internet and request webpages from the servers that host the web content. The most common way we make an API call is using the HTTP protocol (which you can see in any URL) and abiding by the [REST design principles](https://en.wikipedia.org/wiki/Representational_state_transfer). \n",
    "  \n",
    "Beyond serving webpages, HTTP REST APIs have become an increasingly popular method for exchanging data across the internet, especially for cases where the data is granular and is being refreshed constantly. When you are consuming data via a REST API, there are four main types of actions you can perform:\n",
    "\n",
    "GET: For reading/obtaining a resource from the server. This is the most common action and is the one we will be making for all of our calls since we will be _consuming_ data in this course.\n",
    "\n",
    "POST: For creating a resource on the server\n",
    "\n",
    "PUT: For updating a resource on the server\n",
    "\n",
    "DELETE: For deleting a resource on the server\n",
    "\n",
    "### Exercise 1: Accessing Open Data Portal APIs\n",
    "\n",
    "##### Notebook Setup\n",
    "As you did with the Pre-Course exercise, create a new project directory with the structure below and place this workbook within it.\n",
    "```\n",
    "day-1-prj/                     \n",
    "├── data/                        \n",
    "├── output/                      \n",
    "└── Day_1.ipynb    \n",
    "```\n",
    "##### Setting up an API call\n",
    "For today's lesson, we will be learning how to make REST API calls within Python, using the `requests` package, which you can read more about [here](https://2.python-requests.org/en/master/). If the package is not yet installed on your machine, make sure you do so. Data from an API is typically delivered as [XML or JSON](https://github.com/black-tea/ucla-its-data-camp-2019/blob/master/Pre-Course/Programming-Landscape.md#some-common-data-formats). \n",
    "\n",
    "In the last lesson we loaded data by reading in a CSV file. Today we are going to download the same set of data via an API call and ingesting the JSON-formatted response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import requests\n",
    "\n",
    "# Store the url in a variable\n",
    "url = 'https://opendata.arcgis.com/datasets/bed43aa2945a47b18ae888246712ccb1_0.geojson'\n",
    "\n",
    "# TODO: Make the request & store in a variable\n",
    "resp = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response object includes a lot of different information. For example, we can get the status of the request (successful or unsuccessful) by calling `resp.status_code`. You will enounter several different [response codes](https://www.restapitutorial.com/httpstatuscodes.html) when working with APIs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Print the status code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the response was successful (`200`), we want to take a look at the output. For JSON-formatted data, we can access the body of the response by calling `resp.json()`. Let's take a look at the JSON output of our response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine the response content\n",
    "resp.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Write out to disk\n",
    "Especially if you are requesting dynamic data (data that is sensitive to the time at which you queried it), it is good practice to move the data out of memory (your variable) and into persistent disk storage (a file) before you do anything else with it. Although this process seems a bit redundant, it will make more sense in a bit...espeically when we are doing multiple API calls.\n",
    "  \n",
    "Let's write some code to save the JSON data from our API call into data/. We are going to be using the [json](https://docs.python.org/3/library/json.html) python library for reading & writing json data. _Hint: Make sure you setup the project directory structure correctly or you will get an error!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Write out JSON to data/ or data/raw \n",
    "with open('data/raw/collisions_2009to2013.json', 'w') as outfile:\n",
    "    json.dump(resp.json(), outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Read back in the JSON data & parse to a dataframe\n",
    "Now that we have the stored to disk, let's read it back into memory and wrangle it into a more 'tidy' format for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Read the file into memory, store JSON as `collisions` variable\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you remember from the lecture earlier with the sf data portal, you can construct a dataframe directly from JSON-formatted data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas package\n",
    "import pandas as pd\n",
    "\n",
    "# Convert the JSON to a dataframe format\n",
    "collisions_df = pd.DataFrame(collisions['features'])\n",
    "\n",
    "# Show the head of the dataframe\n",
    "collisions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will see that the `geometry` column holds a JSON-formatted object with our lat/lon values and the `properties` column contains a JSON-formatted object with the rest of our data related to each collision. Following the example from the SF Data Portal earlier, create a separate column for `lat` and a separate column for `lon` that holds each value for each collision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create empty lat/lon lists, \n",
    "latitudes = []\n",
    "longitudes = []\n",
    "\n",
    "# loop through DF geom objects, then join lists back to dataframe\n",
    "for label, row in collisions_df.iterrows():\n",
    "    if pd.notnull(row['geometry']):\n",
    "        # TODO: append the first list item to `longitudes` and the second list item to `latitudes`\n",
    "\n",
    "        \n",
    "    else:\n",
    "        latitudes.append(None)\n",
    "        longitudes.append(None)\n",
    "        \n",
    "collisions_df['lat'] = latitudes\n",
    "collisions_df['lon'] = longitudes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Examine the new dataframe\n",
    "collisions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll still notice that all of our properties are within JSON objects as well. Pick any two column names within `properties` and write the loop again below to create two separate columns for those features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Loop through the DF, pick any two features\n",
    "#       and separate them into two new columns, \n",
    "#       then examine the head of collisions_df\n",
    "\n",
    "\n",
    "# View the head\n",
    "collisions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice work! We've actually taken the long way to access data from a GeoJSON object, but getting familiar with loops and dictionaries is absolutely critical for working with data in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2: Getting data from LA Metro\n",
    "Most of the time that you are querying APIs for data it will not be so straightforward to get it into a tidy format. Instead, what you will usually want to do is inspect the response content first before deciding how to proceed. Let's take a look at data from [LA Metro's Developer Portal](https://developer.metro.net/). Going to the [Metro Bus & Rail Real-time Arrivals](https://developer.metro.net/portfolio-item/real-time-arrivals/) page, we can see a variety of APIs that are publicly available. Take a look at all the [feeds](https://developer.metro.net/introduction/realtime-api-overview/realtime-api-returning-json/) returning JSON-formatted content, including route information, stop information, and realtime vehicle location information.\n",
    "\n",
    "You will notice that instead of GeoJSON, it is in a slightly different format that requires just a bit of wrangling to get it in the right format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create an API Call\n",
    "Pick any of the Metro routes and, following the structure in the example, make a call to get all the current vehicles on that route. Once we get the response (assuming it is successful), let's take a look at the content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Write the statement to call the Metro API and get all vehicles for a particular route\n",
    "#       and store the response as resp\n",
    "# (No need to import the requests package again)\n",
    "resp = \n",
    "\n",
    "# TODO: Store the JSON content as `data`\n",
    "data = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will notice that instead of GeoJSON, it is in a slightly different format. We can convert a list of key, value pairs into a Pandas dataframe easily by `df = pd.DataFrame(dict)`. Let's go ahead and convert the json output into a dict. _Hint: Make sure you access the list part of the JSON output!_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Convert the JSON output to a dataframe\n",
    "metro_df = \n",
    "\n",
    "# Examine the head of the dataframe\n",
    "metro_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Add a Column to the DataFrame\n",
    "One thing you will notice is that when we made the dataframe above, we are missing the timestamp of the query. If we plan to write out the data for analysis later, we need to add the time of the query as a column value. The easiest way to get the current time in Python is through the [datetime](https://docs.python.org/2/library/datetime.html) package. Take a little bit of time to look through the documentation with a particular focus on the `now()` method.\n",
    "\n",
    "Once we get the value of the current time, we can add it as a new column value to our current dataframe. Create an additonal column `call_time`. In the function, get the current timestamp of the call and add it as the value for that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the datetime module\n",
    "from datetime import datetime\n",
    "\n",
    "# TODO: Get the current time\n",
    "now = \n",
    "\n",
    "# TODO: Add the current time as a value to the dataframe column `call_time`\n",
    "metro_df['call_time'] = "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Wrap the API Call in a Function\n",
    "Let's create a function to take a Route ID and make the API call for all realtime vehicle locations on that route. Add in the code we used in the block above to also create a column with the time we called the API.\n",
    "\n",
    "_Function Input:_ Route ID  \n",
    "_Function Output:_ Response Dataframe with the content response "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Create the function\n",
    "def get_vehicles_byroute(routenum):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look to make sure our function is working correctly. Run the cell below to confirm that you are getting the desired result. Go ahead and try changing the input and see how the output changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Call the function for one of the routes\n",
    "routedata = get_vehicles_byroute(720)\n",
    "\n",
    "# Examine the head of the dataframe\n",
    "routedata.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Add Functionality\n",
    "Great! Now we are able to change the route number and get a dataframe with the current location of all vehicles on the route. One of the next things we might want to do would be to get data from the route throughout the day and store it for later analysis. To do that we are going to need to add the following functionality into our function:\n",
    "\n",
    "1. Write out the csv to the a file in our `data/processed` folder. Let's set the filename to the format `lametro_[routenum]_[timestamp].csv` (Eg. `lametro_720_2019-09-10-22-26-52.csv`). To write out the file, go ahead and use [Panda's method](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_csv.html) for writing out a csv file.\n",
    "2. Add conditional logic to only write out the file if the call was a success. If the call was not successful, print out the error message. Take a look [here](https://2.python-requests.org/en/master/user/quickstart/#response-status-codes) for some guidance.\n",
    "\n",
    "As the function gets a bit more complex, please add appropriate code comments inside to quickly convey the purpose of each code block.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Re-write the function with the requested features\n",
    "def get_vehicles_byroute(routenum):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and call the function again for one of the routes to ensure that it was written out correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_vehicles_byroute(704)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check your data folder - if everything was successful, you should see a CSV file with the data from the call.\n",
    "##### Introduction to variable-length arguments \n",
    "We've now built a function that, for a given route, will get current vehicle location data, format it into a dataframe, and write it out to a CSV file with the current datetime. What if we were interested in 2 routes? or 3 routes? Let's build another function that takes as input a _variable number of route numbers_ and then gets the vehicle data for each of them.\n",
    "  \n",
    "We will do this through the [_*args_ syntax](https://www.geeksforgeeks.org/args-kwargs-python/). Following that syntax, create a function called `get_vehicles_byroutes` that takes in a variable number of route numbers. For each route number, the function should call our other function `get_vehicles_byroute`. Between each call to our original function, add a 5 second pause to reduce the load on the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# TODO: Finish composing the function\n",
    "def get_vehicles_byroutes(*routes):\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create a Loop to run the Function \n",
    "Great! We now have a function that calls the Metro's API, records the location of all vehicles for a particular route(s), logs the current timestamp, and saves the file in a location of our choosing. Let's (1) pick a few routes we want to get data from and (2) create a loop that runs the `get_vehicles_byroutes` function 1x per minute, for 5 minutes with those route numbers as the input.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: Execute the function 5x, each time separated by a minute\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}