RandomForest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Big Data Toolkit and Machine Learning in ArcGIS Pro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <font size = \"3\">This demo is inspired by [this](https://www.kaggle.com/c/nyc-taxi-trip-duration/) Kaggle competition, and the data can be download from [here](https://www.kaggle.com/c/nyc-taxi-trip-duration/data).<font>\n",
    "- <font size = \"3\">We will be predicting the duration of a NYC taxi trip. We will approach this problem by leveraging arcpy to run a Forest-based Classification and Regression (Spatial Statistics) GeoProcessing task.<font>\n",
    "- <font size = \"3\">Topics covered in this demo:<font>\n",
    "    - Pyspark SQL\n",
    "    - BDT Spatial Type Functions\n",
    "    - BDT Processors and Sources\n",
    "    - ArcPy Feature Classes and Table Views\n",
    "    - ArcPy ML: Forest-based Classification and Regression\n",
    "    - Arcgis Spatial Dataframes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (1) Setup the environment with BDT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size = \"3\">Set filepaths, start a spark instance, create a helper function to track the execution time of certain tasks<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <font size = \"3\">ArcGIS Notebook needs access to 4 items to run BDT:<font>\n",
    "    - <font size = \"3\">[spark-esri](https://github.com/mraad/spark-esri) installed on the active python environment<font>\n",
    "    - <font size = \"3\">Path to a BDT .jar file<font>\n",
    "    - <font size = \"3\">Path to a BDT Python Bindings .egg file<font>\n",
    "    - <font size = \"3\">Path to a valid BDT license .lic file<font>\n",
    "- <font size = \"3\">Ensure that ArcGIS Pro is version 2.6<font>\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from spark_esri import spark_start, spark_stop\n",
    "from pyspark.sql.functions import pandas_udf, hour, cos, sin, radians, col, when\n",
    "from pyspark.sql.types import DoubleType, StringType\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "from arcgis.features import GeoAccessor\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "plt.style.use('dark_background')\n",
    "\n",
    "bdt_jarPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"ws\", \"DevSummit\", \"bdt-2.2.0-2.4.5-2.11-SNAPSHOT.jar\")\n",
    "mssql_jarPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"ws\", \"DevSummit\", \"mssql-jdbc-6.4.0.jre8.jar\")\n",
    "bdt_eggPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"ws\", \"DevSummit\", \"bdt-2.2.0-py3.6.egg\")\n",
    "bdt_licPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"ws\", \"DevSummit\", \"ale10305-2021.lic\")\n",
    "\n",
    "spark_stop()\n",
    "\n",
    "config = {\n",
    "    \"spark.driver.memory\": \"2G\",\n",
    "    \"spark.submit.pyFiles\": bdt_eggPath,\n",
    "    \"spark.jars\": bdt_jarPath + \",\" + mssql_jarPath,\n",
    "    \"spark.kryoserializer.buffer.max\": \"128\"\n",
    "}\n",
    "spark = spark_start(config=config)\n",
    "\n",
    "import time\n",
    "def timeElapsed(startTime):\n",
    "    endTime = time.perf_counter()\n",
    "    timeElapsed = endTime - startTime\n",
    "    if (timeElapsed < 60):\n",
    "        seconds = endTime - startTime\n",
    "        print(f\"Time Elapsed: {seconds:0.4f} seconds\")\n",
    "    else:\n",
    "        minutes = (endTime - startTime)/60\n",
    "        print(f\"Time Elapsed: {minutes:0.4f} minutes\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size = \"3\">Patch the active sparksession with BDT. Confirm Spatial Type Functions have been registered.<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|                 wkt|\n",
      "+--------------------+\n",
      "|[\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000�?\u0000\u0000\u0000...|\n",
      "+--------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import bdt\n",
    "\n",
    "spark.withBDT(bdt_licPath)\n",
    "\n",
    "spark.sql(\"SELECT ST_FromText('POINT(1.0 1.0)') AS wkt\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Prepare the Taxi Data For Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size = \"3\">Load the taxi kaggle [data](https://www.kaggle.com/c/nyc-taxi-trip-duration/data)<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxiPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"mnt\", \"data\", \"DevSummit\", \"nyc-taxi-kaggle\", \"train.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Record count: 1,458,644\n",
      "+---------+---------+-------------------+-------------------+---------------+-------------------+------------------+-------------------+------------------+------------------+-------------+\n",
      "|       id|vendor_id|    pickup_datetime|   dropoff_datetime|passenger_count|   pickup_longitude|   pickup_latitude|  dropoff_longitude|  dropoff_latitude|store_and_fwd_flag|trip_duration|\n",
      "+---------+---------+-------------------+-------------------+---------------+-------------------+------------------+-------------------+------------------+------------------+-------------+\n",
      "|id2875421|        2|2016-03-14 17:24:55|2016-03-14 17:32:30|              1|-73.982154846191406|40.767936706542969|-73.964630126953125|40.765602111816406|                 N|          455|\n",
      "|id2377394|        1|2016-06-12 00:43:35|2016-06-12 00:54:38|              1|-73.980415344238281|40.738563537597656|-73.999481201171875|40.731151580810547|                 N|          663|\n",
      "|id3858529|        2|2016-01-19 11:35:24|2016-01-19 12:10:48|              1|-73.979026794433594|40.763938903808594|-74.005332946777344|40.710086822509766|                 N|         2124|\n",
      "|id3504673|        2|2016-04-06 19:32:31|2016-04-06 19:39:40|              1|-74.010040283203125|   40.719970703125| -74.01226806640625|40.706718444824219|                 N|          429|\n",
      "|id2181028|        2|2016-03-26 13:30:55|2016-03-26 13:38:10|              1|-73.973052978515625|40.793209075927734|-73.972923278808594|40.782520294189453|                 N|          435|\n",
      "+---------+---------+-------------------+-------------------+---------------+-------------------+------------------+-------------------+------------------+------------------+-------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "taxi = spark.read.csv(taxiPath,header=True,sep=\",\")\n",
    "print(f\"Record count: {taxi.count():,}\")\n",
    "taxi.show(5, truncate=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">Enrich the Dataframe with the cosine and sine values of the pickup hour. This captures the cyclic nature of the time.<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "|   pickup_longitude|   pickup_latitude|  dropoff_longitude|  dropoff_latitude|trip_duration|pickup_hour|            hour_cos|           hour_sin|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "|-73.982154846191406|40.767936706542969|-73.964630126953125|40.765602111816406|          455|         17|-0.06824241336467135|-0.9976687691905392|\n",
      "|-73.980415344238281|40.738563537597656|-73.999481201171875|40.731151580810547|          663|          0|                 1.0|                0.0|\n",
      "|-73.979026794433594|40.763938903808594|-74.005332946777344|40.710086822509766|         2124|         11| -0.9906859460363306| 0.1361666490962471|\n",
      "|-74.010040283203125|   40.719970703125| -74.01226806640625|40.706718444824219|          429|         19|  0.4600650377311516|-0.8878852184023756|\n",
      "|-73.973052978515625|40.793209075927734|-73.972923278808594|40.782520294189453|          435|         13| -0.9172113015054532|-0.3984010898462412|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "taxi = taxi.withColumn(\"pickup_hour\", hour(\"pickup_datetime\")) \\\n",
    "           .withColumn(\"hour_rad\", radians(360.0*col(\"pickup_hour\")/23.0)) \\\n",
    "           .withColumn(\"hour_cos\", cos(\"hour_rad\")) \\\n",
    "           .withColumn(\"hour_sin\", sin(\"hour_rad\")).drop(\"id\", \"vendor_id\",\"passenger_count\",\"store_and_fwd_flag\",\"hour_rad\",\"pickup_datetime\",\"dropoff_datetime\")\n",
    "taxi.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- <font size=\"3\">Filter the dataframe to only the New York Area<font>\n",
    "- <font size=\"3\">Filter the trip duration to exclude extreme values. These values are arbitarily defined as trips between 2 and 100 minutes long.<font>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "|   pickup_longitude|   pickup_latitude|  dropoff_longitude|  dropoff_latitude|trip_duration|pickup_hour|            hour_cos|           hour_sin|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "|-73.982154846191406|40.767936706542969|-73.964630126953125|40.765602111816406|          455|         17|-0.06824241336467135|-0.9976687691905392|\n",
      "|-73.980415344238281|40.738563537597656|-73.999481201171875|40.731151580810547|          663|          0|                 1.0|                0.0|\n",
      "|-73.979026794433594|40.763938903808594|-74.005332946777344|40.710086822509766|         2124|         11| -0.9906859460363306| 0.1361666490962471|\n",
      "|-74.010040283203125|   40.719970703125| -74.01226806640625|40.706718444824219|          429|         19|  0.4600650377311516|-0.8878852184023756|\n",
      "|-73.973052978515625|40.793209075927734|-73.972923278808594|40.782520294189453|          435|         13| -0.9172113015054532|-0.3984010898462412|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# NYC Extent \n",
    "xmin,ymin,xmax,ymax = (-74.2555913638106, 40.496115395209344, -73.70000906387119, 40.91553277600007)\n",
    "\n",
    "where = \" AND \".join([\n",
    "    f\"pickup_longitude between {xmin} and {xmax}\",\n",
    "    f\"pickup_latitude between {ymin} and {ymax}\",\n",
    "    f\"dropoff_longitude between {xmin} and {xmax}\",\n",
    "    f\"dropoff_latitude between {ymin} and {ymax}\",\n",
    "    \"trip_duration between 120 and 6000\"\n",
    "])\n",
    "\n",
    "taxi = taxi.where(where)\n",
    "taxi.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">\n",
    "Use BDT Spatial Type Function ST_HaversineDistance to enrich the data with the Haversine Distance between the pickup and dropoff locations.\n",
    "</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+------------------+\n",
      "|   pickup_longitude|   pickup_latitude|  dropoff_longitude|  dropoff_latitude|trip_duration|pickup_hour|            hour_cos|           hour_sin|            meters|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+------------------+\n",
      "|-73.982154846191406|40.767936706542969|-73.964630126953125|40.765602111816406|          455|         17|-0.06824241336467135|-0.9976687691905392|1498.5207796474772|\n",
      "|-73.980415344238281|40.738563537597656|-73.999481201171875|40.731151580810547|          663|          0|                 1.0|                0.0|1805.5071687958239|\n",
      "|-73.979026794433594|40.763938903808594|-74.005332946777344|40.710086822509766|         2124|         11| -0.9906859460363306| 0.1361666490962471| 6385.098495252941|\n",
      "|-74.010040283203125|   40.719970703125| -74.01226806640625|40.706718444824219|          429|         19|  0.4600650377311516|-0.8878852184023756| 1485.498422771006|\n",
      "|-73.973052978515625|40.793209075927734|-73.972923278808594|40.782520294189453|          435|         13| -0.9172113015054532|-0.3984010898462412|1188.5884593342212|\n",
      "+-------------------+------------------+-------------------+------------------+-------------+-----------+--------------------+-------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "taxi.createOrReplaceTempView(\"taxi\")\n",
    "taxi = spark.sql(\"\"\" \n",
    "    SELECT \n",
    "        *, \n",
    "        ST_HaversineDistance(pickup_longitude, pickup_latitude, dropoff_longitude, dropoff_latitude)*1000 as meters\n",
    "    FROM\n",
    "        taxi\"\"\") \\\n",
    "\n",
    "taxi.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Exploration & Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxi.createOrReplaceTempView(\"taxi\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot 1: Average meters traveled for a taxi trip, binned by pickup hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABIoAAAJQCAYAAADhZI+GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdf5BV9X3/8RfLqqw/UDKY3WaXin+sBilViICMGW2qkjW0RWsG12rAKd2kA1FpmUk2jlPTH4OmHVudtJqGOMMSY+k2hrImbrIWHdNxDKzICLogEIWy6yzU+oM0NSJwv394spN8AVmQ5XLl8Zg5M/LZc859n/XHH0/POXdYklIAAAAAOOFVlXsAAAAAAI4PQhEAAAAASYQiAAAAAApCEQAAAABJhCIAAAAACkIRAAAAAEmS6nIPcCg7d+7Mtm3byj0GAAAAwIfGOeeck49+9KP7rR/3oWjbtm2ZPHlyuccAAAAA+NDo7u4+4LpHzwAAAABIIhQBAAAAUBCKAAAAAEhSAe8oAgAAACrfqFGjsmDBgowdOzbDhg0r9zgnhFKplK1bt+bee+/NG2+8MahjhCIAAABgyC1YsCDPPvts/uqv/ip79+4t9zgnhOHDh2fGjBlZsGBB7rzzzkEd49EzAAAAYMiNHTs2jz32mEh0DO3duzc/+MEPMnbs2EEfIxQBAAAAQ27YsGEiURns3bv3sB71E4oAAAAAyuwrX/lKuUdI4h1FAAAAQBncs/6Zo3q+hROmHdXzHWu333577rrrrsM6pqqqKvv27Tuqc7ijCAAAADhhLF++PM8++2xeeOGFtLS05Atf+EL+9m//duDnc+bMyde//vUkyR133JENGzakq6srDz/8cBYuXHjQ8z755JP5+7//+zz11FPp6enJxRdfnEceeSSbNm3KX//1Xw/sd+ONN2bVqlVZu3ZtvvGNb6Sqqip33XVXampqsnbt2jz00EMH3S9Jfvazn+Uv//Iv85Of/CTTpk3LXXfdlRdffDHPP/98/u7v/u4D/36EIgAAAOCE8cd//Me5+OKLc/HFF+fWW2/N8uXLc+211w78/Prrr8+yZcty8cUX57rrrstFF12UP/zDP8zFF198yHPv3r07l19+eb7xjW9kxYoVmT9/fn7rt34rN998cz7ykY/k4x//eK6//vpceumlmThxYvbu3Zsbb7wxX/nKV/L2229n4sSJuemmmw66X5KcfvrpeeGFF3LJJZdkw4YNufbaazN+/PhceOGF+Zu/+ZsP/Pvx6BkAAABwwrj11lsHwtCYMWNy7rnn5uWXX87UqVOzefPmfPzjH8/TTz+d2267LStWrMg777yTd955J48++ughz93R0ZEkWb9+fV588cX09/cnSV5++eWMGTMmn/zkJ/OJT3wi3d3dSZKamprs3Llzv/NcccUVB91vz549eeSRR5Ikb731Vn7xi1/kwQcfzPe///18//vf/4C/HaEIAAAAOEFcfvnlufLKKzNt2rS8/fbbefLJJzNixIgsW7Yss2bNysaNG7N8+fIjPv8777yTJNm3b9/AX//yz9XV1Rk2bFja2tpy++23v+953m+/X/ziFwPvJdq7d2+mTJmSK664Is3NzfniF7+YK6644ojnTzx6BgAAAJwgzjzzzLzxxht5++23c/755+eSSy5J8t57i2bOnJkbbrghy5YtS5I8/fTT+f3f//2ccsopOe200zJjxowP/PkrV67MZz/72Zx99tlJklGjRuU3f/M3kyTvvvtuqqurD7nfrzrttNNy5plnprOzMwsWLMhFF130gWd0RxEAAABwQvjhD3+YP/3TP83zzz+fl156KT/5yU+SJG+++WY2bNiQCy64YOBxr2effTYdHR15/vnns3Xr1jz77LN56623PtDnb9iwIXfccUe6urpSVVWVd999N/Pnz89//dd/5Zvf/GbWrVuX5557LjfddNNB9/tVZ5xxRlasWJERI0Zk2LBh+bM/+7MPNN8vlY7nrbu7u+wz2Gw2m81ms9lsNpvNZvtg29KlS8s+w+Fup512WilJqaamptTd3V2aOHFi2Wc6Wr/7g/UWdxQBAAAAHMA3v/nNXHDBBRkxYkTa2tqydu3aco805IQiAAAAgAP45VfS/6p//Md/zKWXXvpra/fdd1+WLFlyjKYaWkIRAAAAwCB98YtfLPcIQ8q3ngEAAABDrlQqZfjw4eUe44QzfPjwlEqlQe8vFAEAAABDbuvWrZkxY4ZYdAwNHz48M2bMyNatWwd9jEfPAAAAgCF37733ZsGCBbnuuusybNiwco9zQiiVStm6dWvuvffeQR8jFAEAAABD7o033sidd95Z7jE4BKEIOGbuWf/MkJ174YRpQ3ZuAACAE4V3FAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQRCgCAAAAoCAUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKAgFAEAAACQJKku9wDA4bln/TNDev6FE6YN6fkBAAA4fg3qjqJXXnkl69aty9q1a9Pd3Z0kGTVqVLq6urJp06Z0dXXlrLPOGti/tbU1mzdvzsaNGzN9+vSB9UmTJmXdunXZvHlz7rvvvqN8KQAAAAB8EIN+9OxTn/pUJk6cmMmTJyd5LwatXLky5513XlauXJnW1tYkybhx49Lc3Jzx48enqakp999/f6qq3vuYBx54IC0tLWlsbExjY2OampqG4JIAAAAAOBJH/I6imTNnpq2tLUnS1taWa665ZmB92bJl2b17d7Zu3ZotW7ZkypQpqaury8iRI7Nq1aokydKlSweOAQAAAKD8BhWKSqVSurq68uyzz6alpSVJUltbm/7+/iRJf39/amtrkyT19fXZvn37wLG9vb2pr69PfX19ent791s/kJaWlnR3d6e7uzujR48+sisDAAAA4LAM6mXWn/zkJ/Pqq6/m7LPPzuOPP56NGzfut0+pVDpqQy1evDiLFy9OkoF3IgEAAAAwtAZ1R9Grr76aJPnv//7vLF++PFOmTMmOHTtSV1eXJKmrq8vOnTuTJH19fRkzZszAsQ0NDenr60tfX18aGhr2WwcAAADg+HDIUHTqqafm9NNPH/jr6dOn54UXXkhHR0fmzJmTJJkzZ05WrFiRJOno6Ehzc3NOPvnkjB07No2NjVm9enX6+/uza9euTJ06NUkye/bsgWMAAAAAKL9DPnpWW1ub5cuXv7dzdXUefvjh/OhHP0p3d3fa29szd+7cbNu2LbNmzUqS9PT0pL29PT09PdmzZ0/mz5+fffv2JUnmzZuXJUuWpKamJp2dnens7BzCSwMAAADgcAxLcvReLjQEuru7M3ny5HKPAceNe9Y/M6TnXzhh2pCdeyhnH8q5AQAAPmwO1lsG9Y4iAAAAAD78hCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUhCIAAAAAkghFAAAAABSEIgAAAACSCEUAAAAAFIQiAAAAAJIIRQAAAAAUqss9AMDx7p71zwzp+RdOmDak5wcAABgsdxQBAAAAkMQdRZzAhvIuEXeIAAAAUIncUQQAAABAEqEIAAAAgIJQBAAAAEASoQgAAACAglAEAAAAQBKhCAAAAICCUAQAAABAEqEIAAAAgIJQBAAAAEASoQgAAACAglAEAAAAQBKhCAAAAIBCdbkHOJruWf/MkJ5/4YRpQ3p+AAAAgHIa9B1FVVVVee655/Loo48mSUaNGpWurq5s2rQpXV1dOeusswb2bW1tzebNm7Nx48ZMnz59YH3SpElZt25dNm/enPvuu+8oXgYAAAAAH9Sg7yi67bbbsmHDhowcOTLJezFo5cqV+drXvpYvf/nLaW1tTWtra8aNG5fm5uaMHz8+H/vYx/If//EfOe+887Jv37488MADaWlpyapVq/LYY4+lqakpP/zhD4fs4gAAAACOF0P5JNTRegpqUHcU1dfXZ8aMGfnWt741sDZz5sy0tbUlSdra2nLNNdcMrC9btiy7d+/O1q1bs2XLlkyZMiV1dXUZOXJkVq1alSRZunTpwDEAAAAAlN+gQtG9996bL33pS9m3b9/AWm1tbfr7+5Mk/f39qa2tTfJeVNq+ffvAfr29vamvr099fX16e3v3WwcAAADg+HDIR89mzJiRnTt35rnnnsvll19+0P1KpdJRG6qlpSWf//znkySjR48+auc9nlXC7WcAAADAh9shQ9Gll16aP/iDP8hnPvOZjBgxIiNHjsy3v/3t7NixI3V1denv709dXV127tyZJOnr68uYMWMGjm9oaEhfX1/6+vrS0NCw3/qBLF68OIsXL06SdHd3f6ALBAAAAGBwDvno2e23354xY8bk3HPPTXNzc5544ol87nOfS0dHR+bMmZMkmTNnTlasWJEk6ejoSHNzc04++eSMHTs2jY2NWb16dfr7+7Nr165MnTo1STJ79uyBYwAAAAAov0F/69n/7+677057e3vmzp2bbdu2ZdasWUmSnp6etLe3p6enJ3v27Mn8+fMH3m00b968LFmyJDU1Nens7ExnZ+fRuQoAAAAAPrDDCkVPPfVUnnrqqSTJ66+/niuvvPKA+y1atCiLFi3ab33NmjWZMGHCEYwJAAAAwFAb1LeeAQAAAPDhd8SPnkEytN/WlvjGNgAAADiW3FEEAAAAQBKhCAAAAICCUAQAAABAEqEIAAAAgIJQBAAAAEASoQgAAACAglAEAAAAQBKhCAAAAICCUAQAAABAEqEIAAAAgIJQBAAAAEASoQgAAACAglAEAAAAQJKkutwDAAAAAAzWPeufGdLzL5wwbUjPf7xzRxEAAAAASYQiAAAAAApCEQAAAABJhCIAAAAACkIRAAAAAEmEIgAAAAAKQhEAAAAASYQiAAAAAApCEQAAAABJhCIAAAAACkIRAAAAAEmEIgAAAAAKQhEAAAAASZLqcg8AAAAAHHv3rH9myM69cMK0ITs3Q8sdRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoVJd7AAAA4MPpnvXPDOn5F06YNqTnBzgRuaMIAAAAgCRCEQAAAAAFj54BAADAEfKIJR827igCAAAAIIlQBAAAAEBBKAIAAAAgiXcUAQBwgqjk94gM5ezefwLAr3JHEQAAAABJ3FEE8KHm/0ADAACHwx1FAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKFSXewAAAIDjzT3rnxmycy+cMG3Izg3wQbmjCAAAAIAkQhEAAAAABaEIAAAAgCRCEQAAAAAFoQgAAACAJL71DACAw+TboADgw8sdRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJEmqyz0AAAAA3LP+mSE798IJ04bs3PBh444iAAAAAJIIRQAAAAAUPHoGAADwITGUj28lHuGCE4FQBABUNO+0AAA4eoQiAIAy8H/9AYDjkXcUAQAAAJBEKAIAAACgIBQBAAAAkEQoAgAAAKBwyJdZn3LKKfnxj3+cU045JdXV1fnud7+br371qxk1alT+9V//NWPHjs3WrVsza9asvPnmm0mS1tbWzJ07N3v37s2tt96arq6uJMmkSZOyZMmS1NTU5LHHHsttt902tFcHQEXykt9jz+8cAIBkEHcUvfPOO/nd3/3dXHTRRbnooovS1NSUqVOnprW1NStXrsx5552XlStXprW1NUkybty4NDc3Z/z48Wlqasr999+fqqr3PuaBBx5IS0tLGhsb09jYmKampqG9OgAAAAAGbVCPnv385z9Pkpx00kk56aSTUiqVMnPmzLS1tSVJ2tracs011yRJZs6cmWXLlmX37t3ZunVrtmzZkilTpqSuri4jR47MqlWrkiRLly4dOAYAAACA8htUKKqqqsratWuzc+fOPP7441m9enVqa2vT39+fJOnv709tbW2SpL6+Ptu3bx84tre3N/X19amvr09vb+9+6wAAAAAcHwYVivbt25eJEyemoaEhU6ZMyfjx4/fbp1QqHbWhWlpa0t3dne7u7owePfqonRcAAACAgzusbz1766238uSTT6apqSk7duxIXV1dkqSuri47d+5MkvT19WXMmDEDxzQ0NKSvry99fX1paGjYb/1AFi9enMmTJ2fy5Ml57bXXDvuiAAAAADh8hwxFo0ePzplnnpkkGTFiRK666qps3LgxHR0dmTNnTpJkzpw5WbFiRZKko6Mjzc3NOfnkkzN27Ng0NjZm9erV6e/vz65duzJ16tQkyezZsweOAQAAAKD8qg+1w2/8xm+kra0tw4cPT1VVVdrb2/ODH/wgzzzzTNrb2zN37txs27Yts2bNSpL09PSkvb09PT092bNnT+bPn599+/YlSebNm5clS5akpqYmnZ2d6ezsHNqrA4BjbCi/Zt5XzAMAMNQOGYrWr1+fSZMm7bf++uuv58orrzzgMYsWLcqiRYv2W1+zZk0mTJhwBGMCAAAAMNQO6x1FAAAAAHx4CUUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQZRChqaGjIE088kRdffDEvvPBCbr311iTJqFGj0tXVlU2bNqWrqytnnXXWwDGtra3ZvHlzNm7cmOnTpw+sT5o0KevWrcvmzZtz3333DcHlAAAAAHCkDhmK9uzZk4ULF2b8+PG55JJLMn/+/IwbNy6tra1ZuXJlzjvvvKxcuTKtra1JknHjxqW5uTnjx49PU1NT7r///lRVvfcxDzzwQFpaWtLY2JjGxsY0NTUN7dUBAAAAMGiHDEX9/f1Zu3ZtkuR///d/s2HDhtTX12fmzJlpa2tLkrS1teWaa65JksycOTPLli3L7t27s3Xr1mzZsiVTpkxJXV1dRo4cmVWrViVJli5dOnAMAAAAAOV3WO8oOuecczJx4sSsWrUqtbW16e/vT/JeTKqtrU2S1NfXZ/v27QPH9Pb2pr6+PvX19ent7d1v/UBaWlrS3d2d7u7ujB49+rAvCgAAAIDDN+hQdNppp+WRRx7JggUL8rOf/Wy/n5dKpaM21OLFizN58uRMnjw5r7322lE7LwAAAAAHN6hQVF1dnUceeSTf+c53snz58iTJjh07UldXlySpq6vLzp07kyR9fX0ZM2bMwLENDQ3p6+tLX19fGhoa9lsHAAAA4PgwqFD04IMPZsOGDfmHf/iHgbWOjo7MmTMnSTJnzpysWLFiYL25uTknn3xyxo4dm8bGxqxevTr9/f3ZtWtXpk6dmiSZPXv2wDEAAAAAlF/1oXa49NJLM3v27Kxbt27gpda333577r777rS3t2fu3LnZtm1bZs2alSTp6elJe3t7enp6smfPnsyfPz/79u1LksybNy9LlixJTU1NOjs709nZOYSXBgAAAMDhOGQoevrppzNs2LAD/uzKK6884PqiRYuyaNGi/dbXrFmTCRMmHOaIAAAAABwLh/WtZwAAAAB8eAlFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkEYoAAAAAKAhFAAAAACQRigAAAAAoCEUAAAAAJBGKAAAAACgIRQAAAAAkGUQoevDBB7Njx46sX79+YG3UqFHp6urKpk2b0tXVlbPOOmvgZ62trdm8eXM2btyY6dOnD6xPmjQp69aty+bNm3Pfffcd5csAAAAA4IM6ZChasmRJmpqafm2ttbU1K1euzHnnnZeVK1emtbU1STJu3Lg0Nzdn/PjxaWpqyv3335+qqvc+4oEHHkhLS0saGxvT2Ni43zkBAAAAKK9DhqL//M//zOuvv/5razNnzkxbW1uSpK2tLddcc83A+rJly7J79+5s3bo1W7ZsyZQpU1JXV5eRI0dm1apVSZKlS5cOHAMAAADA8eGI3lFUW1ub/v7+JEl/f39qa2uTJPX19dm+ffvAfr29vamvr099fX16e3v3Wz+YlpaWdHd3p7u7O6NHjz6SEQEAAAA4TEflZdalUulonGbA4sWLM3ny5EyePDmvvfbaUT03AAAAAAd2RKFox44dqaurS5LU1dVl586dSZK+vr6MGTNmYL+Ghob09fWlr68vDQ0N+60DAAAAcPw4olDU0dGROXPmJEnmzJmTFStWDKw3Nzfn5JNPztixY9PY2JjVq1env78/u3btytSpU5Mks2fPHjgGAAAAgOND9aF2ePjhh/M7v/M7GT16dLZv354777wzd999d9rb2zN37txs27Yts2bNSpL09PSkvb09PT092bNnT+bPn599+/YlSebNm5clS5akpqYmnZ2d6ezsHNorAwAAAOCwHDIU/dEf/dEB16+88soDri9atCiLFi3ab33NmjWZMGHCYY4HAAAAwLFyVF5mDQAAAEDlE4oAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEUtUG0AABSgSURBVEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEgiFAEAAABQEIoAAAAASCIUAQAAAFAQigAAAABIIhQBAAAAUBCKAAAAAEhShlD06U9/Ohs3bszmzZvz5S9/+Vh/PAAAAAAHcUxDUVVVVf7pn/4pV199dS644ILccMMNGTdu3LEcAQAAAICDOKahaMqUKdmyZUteeeWVvPvuu1m2bFlmzpx5LEcAAAAA4CCGJSkdqw+77rrr0tTUlJaWliTJTTfdlKlTp+aWW275tf1aWlry+c9/Pkly/vnn56WXXhqSeUaPHp3XXnttSM49lCp17qRyZ6/UuZPKnb1S504qd/ZKnTup3Nkrde6kcmev1LmTyp29UudOKnf2Sp07qdzZK3XupHJnr9S5k8qdvVLnTip39qGe+5xzzslHP/rRA/6sdKy26667rrR48eKBP990002lr3/968fs8///rbu7u2yffSLOXcmzV+rclTx7pc5dybNX6tyVPHulzl3Js1fq3JU8e6XOXcmzV+rclTx7pc5dybNX6tyVPHulzl3Js5dr7mP66FlfX1/GjBkz8OeGhob09fUdyxEAAAAAOIhjGoq6u7vT2NiYsWPH5qSTTkpzc3M6OjqO5QgAAAAAHMTwJF89Vh9WKpWyefPmfOc738ktt9yShx56KN/73veO1ccf0HPPPVfWzz9SlTp3UrmzV+rcSeXOXqlzJ5U7e6XOnVTu7JU6d1K5s1fq3Enlzl6pcyeVO3ulzp1U7uyVOndSubNX6txJ5c5eqXMnlTt7OeY+pi+zBgAAAOD4dUwfPQMAAADg+CUUAQAAAJBEKAIAAACgcExfZl1u559/fubOnZtZs2bl6quvzm//9m/nf/7nf/Laa6+Ve7QPrfPPPz8XXnhhdu7cmXfffXdg/dOf/nR++tOflnGy9zd58uR87GMfy6uvvppx48blc5/7XEaNGpUtW7aUe7TD0tbWln//938v9xiH7dJLL81nP/vZnHHGGXn55ZfLPc77mjJlSnbt2pXdu3dnxIgRueOOO/Lnf/7nufDCC7N69eq888475R7xgG655Za8+uqr2bVrV7lHOWwnnXRSbrzxxpx99tl55ZVXcsMNN+QLX/hCzj333Kxduzb79u0r94gHde6552bu3Lm5/vrrc9VVV+Xcc8/Npk2bsnv37nKPBgAASU6gl1l/6Utfyg033JBly5alt7c3SdLQ0JDm5uYsW7YsX/va18o84ZG5+eabs2TJknKPcUC33HJL5s+fnw0bNuSiiy7Kbbfdlo6OjiTJmjVr8olPfKLMEx7YX/zFX+Tqq69OdXV1Hn/88UydOjVPPvlkrrrqqvzoRz/KokWLyj3iAa1YseLX/jxs2LB86lOfyhNPPJEkmTlzZjnGGpRVq1Zl6tSpSZI/+ZM/yfz587N8+fJMnz49jz766HH97+cLL7yQCy+8MHv37s0///M/5//+7//y3e9+N1dccUUuvPDCXHfddeUe8YDefPPN/PznP89Pf/rT/Mu//Ev+7d/+rWKi+UMPPZTq6uqceuqpefPNN3P66afne9/7Xq644ooMGzYsN998c7lHPKBbbrklv/d7v5cf//jH+cxnPpO1a9fmzTffzLXXXpt58+blqaeeKveIAACQ5L1Q9KHfXnrppVJ1dfV+6yeddFJp06ZNZZ/vSLdt27aVfYaDbevWrSuddtpppSSlc845p9Td3V269dZbS0lKzz33XNnne7+5q6qqSjU1NaW33nqrdMYZZ5SSlEaMGFF6/vnnyz7fwbY1a9aUvv3tb5cuv/zy0mWXXVa6/PLLS6+++mrpsssuK1122WVln+/9tl/952H16tWl0aNHl5KUTj311NK6devKPt/7bT09Pb/29+BXf7Z27dqyz/d+v/Nhw4aVrrrqqtK3vvWt0s6dO0udnZ2l2bNnl04//fSyz/d+2y//PRw+fHipv7+/VFVVtd/Pjsftl/9tSVKqqakpPfnkk6UkpTFjxhzX/0202U7k7eyzzy77DCfa9pGPfKTsM9hsR3MbOXJk6a677ipt2LCh9P/au/eYKus/DuDvc1CTFOU2jijzMEShC3hBp5XDW3bQMktsg/6QtqRaqUMrctUk5xJZc3PzlpcEKVci5mUIonJVuTnKc2QHSUGUFdpQLIemAz+/P4wzTnKO6C/9Po+9X9t7k+fh8Lz37Mj5nu/5Pg+XL1+WlpYWsdvtkpqaKgMHDlTe70GSm5urvIO7eHl5ycqVKyUzM1Pi4+Od9q1fv155P1cxmUyyYcMGWbdunfj6+kpKSorYbDbZuXOnDBo06JF2+c/co+j27dsYPHjwXdsDAwM1fZkCAFit1m5js9lgMplU13PJaDSira0NAHD+/HlMnjwZM2bMwOrVq2EwGBS3c629vR23b9/GjRs3UF9fj2vXrgEA/vrrL00/V8aOHYvq6mp89tln+OOPP1BSUoIbN26gtLQUpaWlquu5ZTQa4e3tDV9fXxgMBsfKluvXr6O9vV1xO/dqamocK1isVqtjpdzw4cOdLrfUGhGBiODw4cOYP38+Bg8ejA0bNiAmJkbzl/sZjUb07t0bXl5eePLJJzFw4EAAwBNPPIHevXsrbuder169ANzp2r9/fwBAU1OT5nsPGDAAqampqK2tdVyybbfbkZqa6jj/epSbm6u6gkteXl5YuXIlMjMzER8f77Rv/fr1ilr1jMlkwoYNG7Bu3Tr4+voiJSUFNpsNO3fuxKBBg1TXc8nHx8cpvr6+qKqqgre3N3x8fFTXc8tisTj+PWDAAGzduhVWqxU7duxAQECAwmbupaamws/PDwAQFRWF+vp6VFZWorGxEdHR0Yrbudc55goJCVFd5b5ERUWhsLAQ3377LYKCgnDo0CFcvXoVVVVVGDVqlOp6bvXr1w/Lly9HTU0Nrl69it9//x3l5eVISEhQXc2trKwstLa2YvLkyfDz84O/vz+mTJmC1tZWZGVlqa7n0ujRo7vNmDFjNP9cSU9Ph8FgwO7duxEXF4fs7Gz06dMHADBhwgTF7VzLyMiA3W5HU1MTioqKcOPGDcycORNHjx7F119//cj7KJ85exSxWCxy5swZyc3NlU2bNsmmTZskLy9Pzpw5IxaLRXk/d7l48aKMHDlShg4d6hSz2Sy//vqr8n6uUlBQICNHjnTa5uHhIdu3b5f29nbl/VyloqJCPD09BYAYDAbH9gEDBty1YkSLGTJkiGRlZcnatWs1veKsa86dOyf19fXS0NAg9fX1jhnzfv36aXpVTufzIj09Xc6ePSsVFRVy69Ytqa+vl+LiYomMjFTez1XcrWDpfP5rNUlJSVJfXy+NjY2ycOFCOXLkiGzevFlsNpssW7ZMeT9XWbRokVitVtm8ebPU1tbKW2+9JQDE399fSkpKlPdzl4MHD0pycrKYTCbHNpPJJMnJyZKfn6+8n7uMHj2624wZM0Z+++035f1cJTs7W1JTU2X27Nmyb98+yc7Olj59+ghw9+pFrSUvL08WLFggn3zyiVitVklOTpagoCBZsGCB7N27V3k/V+no6JCGhgan3Lp1y/HapLqfu3R9TmzZskVWrFghQ4cOlaSkJNmzZ4/yfq7SddVwYWGhjB07VgDI8OHD5cSJE8r7uUtDQ4N89dVXcv78eamsrJSkpCQJDAxU3uteqayslJiYGImLi5MLFy5IbGysAJCpU6dKWVmZ8n7usnfvXklISJAhQ4bI4sWL5fPPP5fQ0FDJyMiQL7/8Unk/Vzl9+vQD7VOd9vZ2KSgokMLCwrty/fp15f3c5Z/vHz799FM5duyY+Pr6avo1tOv4/J/v4xS8J1J/Qh5VDAaDjB8/XubMmSNz5syR8ePHO12yoNVs3bpVXnjhhW737dixQ3k/VxkyZIjTm4quef7555X3c5XOgfg/4+fnJ88++6zyfj3NzJkzNf2i2ZN4enpKcHCw8h49iZeXl0RGRsqYMWMkICBAeZ97Zfjw4co7/D8JDAx0DMgHDhwosbGxMm7cOOW97pWnn35aYmNjJSwsTHmX+4leB7mAfge6eh3kApob6PY4S5Yskby8PKfX+oaGBuW9epKuz4l/nmMtn3O73S4eHh4CQMrLy532af3S867nfOLEibJ+/Xppbm6WwsJCSUxMVN7PVdz9/9T6ZdAnT550+rqqqkqAO+/xamtrlfdzlfz8fPn444+dxocBAQGSnJwshw8fVt7PVU6dOiWhoaHd7rtw4YLyfu5it9udPvAHIAkJCVJTUyONjY3K+7lK1+f4ihUrnPYp+J2o/oQwDMMwDKPd6HWQC+h3oKvXQS6guYHufaVzVe7q1aulf//+ml9J1JmmpiZZvHixLFmy5K7OWr5324IFCyQ/P1+mTJkiKSkpsmbNGomOjpYvvvhCMjMzlfdzl+4mbI1Go1gsFtm2bZvyfq5SVlYm06dPl7lz50pjY6PMnj1bAEh0dLTmV3EdP37c8eH5rFmz5ODBg459Wv7QwtvbW1atWiW1tbVy5coVuXz5stjtdlm1apX4+Pgo7+cqsbGxMmLEiG73dT5vtJq0tDSZNm3aXdstFoum70+8fPlyxz1+u2bYsGGya9euR91H/QlhGIZhGEa76TrIvXz5stMg19vbW3k/d9HrQFevg1xAcwPdB8qsWbOkvLxcmpublXfpSZYtW+aUzj8KYTKZZPv27cr7ucukSZPkhx9+kJ9++klsNpscOHBAEhMTHSuNtJrvv/9eeYcHSWRkpBw8eFByc3MlLCxM1qxZI62trVJTUyPPPfec8n7uEhERIZWVlXLlyhU5evSoY3W0v7+/LFy4UHk/dwkLC5Np06bd9btR67dACQsLk6lTp+qut7vuMTExyrvp5JyrPxkMwzAMw+gznfda0mP02l2vvfXWvW/fvvLMM8/orreez/nj0FvP3fXaW+vdFy5cKKdPn5Y9e/bIuXPn5NVXX3Xs0/KlxHrtDdxZrajH7hrrrf6EMAzDMAyjz+jlpvmPU3e99tZzd7321nN3vfbWc3e99tZ6d5vN5lgdYjab5cSJE7Jo0SIBtH1fKL321nN3LfW+83d6iYiIiFywWq3dbjcYDDCZTI+4zf3Ra3e99gb0212vvQH9dtdrb0C/3fXaG9Bvd6PRiLa2NgDA+fPnMXnyZGRnZ8NsNsNgMChu55peewP67a6l3pwoIiIiIrdMJhMsFgtaW1udthsMBpSVlSlq1TN67a7X3oB+u+u1N6Df7nrtDei3u157A/rtfunSJYwcOdIx0dXW1oZXXnkF27ZtQ0REhOJ2rum1N6Df7lrqzYkiIiIicisnJwf9+/fv9tPc4uLiR1/oPui1u157A/rtrtfegH6767U3oN/ueu0N6Lf7vHnz0N7e7rSto6MDCQkJ2LRpk6JW96bX3oB+u2uptwF3rkEjIiIiIiIiIqL/OKPqAkREREREREREpA2cKCIiIiIiIiIiIgCcKCIiIqL/kC1btuCpp55yuT8lJQUffvjhQzl2QkIC1q5d+1B+NhEREdG/hTezJiIiov+MxMRE1RX+dR4eHujo6FBdg4iIiB4TXFFEREREjx2z2Yza2lpkZGTAarVi165d8PT0RFFREaKiogAAFosF1dXVOHnyJI4cOXLXz5g/fz5yc3PRt29fXLt2zbE9NjYW6enpAID09HRs3LgRpaWlqKurw8svv+y21+DBg5GXl4dffvkFaWlpju1xcXGw2Ww4deoUVq1a5dju7rirV69GYWGh088hIiIi+n9xRRERERE9lsLDw/H222+jrKwM33zzDd5//33HPn9/f2zZsgXR0dFobGyEj4+P02M/+OADTJ8+Ha+99hpu3brl9jjBwcGYNGkShg0bhqKiIoSGhuLmzZvdfu+oUaMwevRo3Lx5E3V1dVi7di06OjqQlpaGqKgotLa24tChQ5g9ezb27dvn9rgjRozAiy++iNu3b/fwjBARERHdG1cUERER0WPpwoULKCsrAwB89913mDhxomPfhAkTUFpaisbGRgBAa2urY9+8efMwY8YMzJ07956TRACQlZUFEcHZs2fR0NCA8PBwl99bUFCAP//8Ezdv3oTdbofZbMa4ceNQXFyMlpYWdHR0YMeOHYiOjr7ncXft2sVJIiIiIvrXcaKIiIiIHksi4vZrV06dOoXg4GAEBQV1+9i+ffs+8HG6rjTq6OhAr17uF3e7O25bW5vbxxIRERE9CE4UERER0WPJbDZjwoQJAIA333wTx44dc+yrqKhAdHQ0goODAcDp0rOff/4Z7777Lvbv34/AwEAAwKVLlxAeHg6DwYDXX3/d6ThvvPEGDAYDQkJCEBISgrq6uvvqWVVVhUmTJsHPzw9GoxHx8fEoKSm553GJiIiIHgZOFBEREdFjqba2FgkJCbBarfDx8cHGjRsd+1paWvDOO+/gxx9/xMmTJ7Fz506nxx4/fhwfffQRDhw4AD8/PyxduhQ5OTkoKChAc3Oz0/fW1dWhpKQEeXl5eO+991zen8iVixcvYunSpSgqKoLVakV1dTX2798PAG6PS0RERPQwGAD0bB02ERERkU6YzWbk5OQgIiLioR4nPT0dOTk52L1790M9DhEREdGjwhVFREREREREREQEgCuKiIiIiP5VL730EtLS0py2nTt3DnPmzFHUiIiIiKjnOFFEREREREREREQAeOkZERERERERERH9jRNFREREREREREQEgBNFRERERERERET0N04UERERERERERERAE4UERERERERERHR3/4HiPNc2eLPDhwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rows = spark.sql(f\"\"\"\n",
    "    SELECT \n",
    "        pickup_hour, AVG(meters) as avg_meters\n",
    "    FROM\n",
    "        taxi\n",
    "    GROUP BY\n",
    "        pickup_hour\n",
    "    ORDER BY \n",
    "        pickup_hour\n",
    "\"\"\")\\\n",
    "    .collect()\n",
    "\n",
    "pdf = pd.DataFrame(rows,columns=['pickup_hour','avg_meters'])\n",
    "\n",
    "plt.clf(); # Clear figure\n",
    "pdf.plot(x='pickup_hour',y='avg_meters',kind='bar',figsize=(20,10));\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot 2: Total count of taxi trips, binned by pickup hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJEAAAJQCAYAAAA+KvWZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdfayX9X3/8RecIy3ecKO058xz+AFbjh0zlJYWkTWx7bCKdRkuGntcW+lC6TIplpTEkm6pXf/AmoyoacWqNQWpK0V6I82kpYPOLht3BQO4g/WAQuG4I0MRurYTgev3h/OkFOEDyOn3fPXxSD7Rc53r+pz3RcCYZ67rS78kVQAAAADgBPrXegAAAAAA+j4RCQAAAIAiEQkAAACAIhEJAAAAgCIRCQAAAIAiEQkAAACAosZaD3C69uzZk507d9Z6DAAAAIA3jBEjRuTtb3/7a36vbiPSzp07M378+FqPAQAAAPCGsX79+uN+z+tsAAAAABSJSAAAAAAUiUgAAAAAFNXtZyIBAAAA/D4MHTo0s2bNysiRI9OvX79aj/O6VVWVHTt25M4778y+fftO+joRCQAAAOAEZs2alZ/97Gf50pe+lMOHD9d6nNetoaEhV199dWbNmpVbb731pK/zOhsAAADACYwcOTKPPvroGyIgJcnhw4fzz//8zxk5cuQpXSciAQAAAJxAv3793jAB6VWHDx8+5VfzRCQAAAAAinwmEgAAAMApmLdl9Rndb/aYiWd0v97iSSQAAACAPmzEiBHZunVrFixYkE2bNuXhhx/OwIED82d/9mfZuHFjNm/enAceeCADBgxIkjzzzDP58pe/nLVr12bt2rX5oz/6ozMyh4gEAAAA0Mf98R//ce67776MHTs2Bw4cyGc/+9ksWLAgH/nIR/LOd74zjY2N+du//due8w8cOJAJEybkq1/9au68884zMoOIBAAAANDH/eIXv8h//Md/JEm++c1vZtKkSXnmmWfS2dmZJFm4cGEuu+yynvO/9a1v9fxz4sQz87qciAQAAADQx1VVddTXL7744kmf/7vXni4RCQAAAKCPGzFiRC699NIkyV/91V/lZz/7WUaOHNnzeUcf//jH89hjj/Wc/5GPfKTnn6tXn5kPAve3swEAAAD0cVu3bs3UqVNz7733prOzMzfffHPWrFmThx9+OI2NjVm/fn2+9rWv9Zz/lre8JWvWrEn//v1zww03nJEZRCQAAACAUzB7zJn5jKFTcfjw4aM+ODtJVq1alXHjxr3m+XfffXe+9KUvndEZvM4GAAAAQJEnkQAAAAD6sJ07d2bMmDEnff6oUaN6ZQ5PIgEAAABQJCIBAAAAnEBVVWloaKj1GGdUQ0NDqqo6pWtEJAAAAIAT2LFjR66++uo3TEhqaGjI1VdfnR07dpzSdT4TCQAAAOAE7rzzzsyaNSvXXntt+vXrV+txXreqqrJjx47ceeedp3SdiAQAAABwAvv27cutt95a6zFqTkQCAAB4E5i3ZXWv7T17zMRe2xvoO3wmEgAAAABFnkQCAACgz/IEFfQdnkQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoKix1gMAAAAAfce8Lat7be/ZYyb22t70PhEJAAAAeoEYwxuN19kAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKDqpiDRr1qw88cQT2bJlS/7pn/4pb3nLWzJ06NCsWLEiTz31VFasWJEhQ4b0nD9nzpx0dnbmySefzBVXXNFzfNy4cdm8eXM6Oztz11139RwfMGBAFi9enM7OzqxZsyYjRow4g7cIAAAAwOtVjEgXXnhhbr755rz3ve/NmDFj0tDQkPb29syZMycrV67MRRddlJUrV2bOnDlJktGjR6e9vT0XX3xxJk+enPnz56d//1d+zD333JPp06enra0tbW1tmTx5cpJk2rRp2bdvX9ra2nLHHXfk9ttv78VbBgAAAOBUndSTSI2NjRk4cGAaGhpy9tln59lnn82UKVOycOHCJMnChQtzzTXXJEmmTJmSxYsX5+DBg9mxY0e2bduWSy65JM3NzRk0aFDWrl2bJHnwwQePuubVvZYuXZpJkyad8RsFAAAA4PQVI9Kzzz6bf/zHf8wvfvGL/Nd//Vf279+fH//4x2lqakp3d3eSpLu7O01NTUmSlpaW7Nq1q+f63bt3p6WlJS0tLdm9e/cxx3/3msOHD2f//v254IILjpll+vTpWb9+fdavX59hw4a9jtsGAAAA4FQUI9KQIUMyZcqUjBo1KhdeeGHOOeecfPSjHz3mvKqqemXA33b//fdn/PjxGT9+fPbu3dvrPw8AAACAVxQj0uWXX55nnnkme/fuzaFDh/Ld7343f/qnf5rnnnsuzc3NSZLm5ubs2bMnSdLV1ZXhw4f3XN/a2pqurq50dXWltbX1mOO/e01DQ0MGDx6c559//szdJQAAAACvSzEi/eIXv8ill16agQMHJkkmTZqUrVu3ZtmyZZk6dWqSZOrUqXnkkUeSJMuWLUt7e3sGDBiQkSNHpq2tLevWrUt3d3cOHDiQCRMmJEluvPHGo655da/rrrsuq1atOvN3CgAAAMBpayydsG7duixdujQbN27MoUOH8vjjj+e+++7LueeemyVLlmTatGnZuXNnrr/++iRJR0dHlixZko6Ojhw6dCgzZszIkSNHkiQ33XRTFixYkIEDB2b58uVZvnx5kuSBBx7IokWL0tnZmRdeeCHt7e29eMsAAAAAnKpiREqSL37xi/niF7941LEXXnghl19++WueP3fu3MydO/eY4xs2bMiYMWOOOf7SSy/1RCgAAAAA+p7i62wAAAAAcFJPIgEAAAD0dfO2rO61vWePmdhre9cLTyIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUNRY6wEAAADqxbwtq3tt79ljJvba3gBngieRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoaqz1AAAAUGvztqzutb1nj5nYa3sDwO+TJ5EAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKGqs9QAAABxt3pbVvbb37DETe21vAOCNTUQCAIA6JTgC8PskIgEAcEYIGgDwxuYzkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAAChqrPUAAAAAAG9m87as7rW9Z4+ZeMb28iQSAAAAAEUiEgAAAABFIhIAAAAARSISAAAAAEUiEgAAAABFIhIAAAAARY21HgAAoDfUy1+VCwBQLzyJBAAAAECRiAQAAABAkYgEAAAAQJGIBAAAAECRD9YGgN8DH/IMAEC98yQSAAAAAEUiEgAAAABFXmcDoK54LQwAAGrDk0gAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUNdZ6AACgb5u3ZXWv7T17zMRe2xsAgDPLk0gAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUNdZ6AAAA4M1n3pbVvbb37DETe21vgDczTyIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQdFIRafDgwXn44YezdevWdHR05NJLL83QoUOzYsWKPPXUU1mxYkWGDBnSc/6cOXPS2dmZJ598MldccUXP8XHjxmXz5s3p7OzMXXfd1XN8wIABWbx4cTo7O7NmzZqMGDHiDN4iAAAAAK/XSUWku+66Kz/84Q8zevTojB07Nlu3bs2cOXOycuXKXHTRRVm5cmXmzJmTJBk9enTa29tz8cUXZ/LkyZk/f37693/lx9xzzz2ZPn162tra0tbWlsmTJydJpk2bln379qWtrS133HFHbr/99l66XQAAAABORzEiDRo0KJdddlkeeOCBJMnLL7+c/fv3Z8qUKVm4cGGSZOHChbnmmmuSJFOmTMnixYtz8ODB7NixI9u2bcsll1yS5ubmDBo0KGvXrk2SPPjgg0dd8+peS5cuzaRJk878nQIAAABw2ooRadSoUfnv//7vfOMb38jGjRtz//335+yzz05TU1O6u7uTJN3d3WlqakqStLS0ZNeuXT3X7969Oy0tLWlpacnu3buPOf671xw+fDj79+/PBRdccObuEgAAAIDXpRiRGhsbM27cuNxzzz0ZN25cfvWrX/W8uvbbqqrqlQF/2/Tp07N+/fqsX78+w4YN6/WfBwAAAMArihFp9+7d2b17d9atW5fkldfNxo0bl+eeey7Nzc1Jkubm5uzZsydJ0tXVleHDh/dc39ramq6urnR1daW1tfWY4797TUNDQwYPHpznn3/+mFnuv//+jB8/PuPHj8/evXtP954BAAAAOEXFiPTcc89l165dueiii5IkkyZNSkdHR5YtW5apU6cmSaZOnZpHHnkkSbJs2bK0t7dnwIABGTlyZNra2rJu3bp0d3fnwIEDmTBhQpLkxhtvPOqaV/e67rrrsmrVqjN/pwAAAACctsaTOWnmzJl56KGHMmDAgDz99NP567/+6/Tv3z9LlizJtGnTsnPnzlx//fVJko6OjixZsiQdHR05dOhQZsyYkSNHjiRJbrrppixYsCADBw7M8uXLs3z58iTJAw88kEWLFqWzszMvvPBC2tvbe+l2AQAAADgdJxWRNm3alPHjxx9z/PLLL3/N8+fOnZu5c+cec3zDhg0ZM2bMMcdfeumlnggFAAAAQN9TfJ0NAAAAAEQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAihprPQAAv3/ztqzutb1nj5nYa3sDAAC140kkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIpEJAAAAACKRCQAAAAAikQkAAAAAIoaaz3A78u8Lat7be/ZYyb22t4AAAAAfYEnkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKDrpiNS/f/9s3LgxP/jBD5IkQ4cOzYoVK/LUU09lxYoVGTJkSM+5c+bMSWdnZ5588slcccUVPcfHjRuXzZs3p7OzM3fddVfP8QEDBmTx4sXp7OzMmjVrMmLEiDNxbwAAAACcIScdkT7zmc9k69atPV/PmTMnK1euzEUXXZSVK1dmzpw5SZLRo0envb09F198cSZPnpz58+enf/9Xfsw999yT6dOnp62tLW1tbZk8eXKSZNq0adm3b1/a2tpyxx135Pbbbz+T9wgAAADA63RSEamlpSVXX311vv71r/ccmzJlShYuXJgkWbhwYa655pqe44sXL87BgwezY8eObNu2LZdcckmam5szaNCgrF27Nkny4IMPHnXNq3stXbo0kyZNOnN3CAAAAMDrdlIR6c4778wtt9ySI0eO9BxrampKd3d3kqS7uztNTU1JXglOu3bt6jlv9+7daWlpSUtLS3bv3n3M8d+95vDhw9m/f38uuOCCY+aYPn161q9fn/Xr12fYsGGneq8AAAAAnKZiRLr66quzZ8+ebNy48YTnVVV1xoY6nvvvvz/jx4/P+PHjs3fv3l7/eQAAAAC8orF0wvve9778xV/8RT784Q/nrW99awYNGpRFixblueeeS3Nzc7q7u9Pc3Jw9e/YkSbq6ujJ8+PCe61tbW9PV1ZWurq60trYec/y3r+nq6kpDQ0MGDx6c559//kzfKwAAAACnqfgk0uc///kMHz48o0aNSnt7e1atWpWPf/zjWbZsWaZOnZokmTp1ah555JEkybJly9Le3p4BAwZk5MiRaWtry7p169Ld3Z0DBw5kwoQJSZIbb7zxqGte3eu6667LqlWreuVmAQAAADg9xSeRjufLX/5ylixZkmnTpmXnzp25/vrrkyQdHR1ZsmRJOjo6cujQocyYMaPns5RuuummLFiwIAMHDszy5cuzfPnyJMkDDzyQRYsWpbOzMy+88ELa29vPwK0BAAAAcKacUkR67LHH8thjjyVJXnjhhVx++eWved7cuXMzd+7cY45v2LAhY8aMOeb4Sy+91BOhAAAAAOh7TupvZwMAAADgzU1EAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgqLHWA3Bi87as7rW9Z4+Z2Gt7AwAAAG8snkQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgqLHWAwDUs3lbVvfa3rPHTOy1vQEAAE6VJ5EAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAACgSkQAAAAAoEpEAAAAAKBKRAAAAAChqrPUAvHHN27K61/aePWZir+0NAAAAHMuTSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABSJSAAAAAAUiUgAAAAAFIlIAAAAABQVI1Jra2tWrVqV//zP/8wTTzyRm2++OUkydOjQrFixIk899VRWrFiRIUOG9FwzZ86cdHZ25sknn8wVV1zRc3zcuHHZvHlzOjs7c9ddd/UcHzBgQBYvXpzOzs6sWbMmI0aMOJP3CAAAAMDrVIxIhw4dyuzZs3PxxRfn0ksvzYwZMzJ69OjMmTMnK1euzEUXXZSVK1dmzpw5SZLRo0envb09F198cSZPnpz58+enf/9Xfsw999yT6dOnp62tLW1tbZk8eXKSZNq0adm3b1/a2tpyxx135Pbbb+/FWwYAAADgVBUjUnd3dx5//PEkyf/8z/9k69ataWlpyZQpU7Jw4cIkycKFC3PNNdckSaZMmZLFixfn4MGD2bFjR7Zt25ZLLrkkzc3NGTRoUNauXZskefDBB4+65tW9li5dmkmTJp35OwUAAADgtDWeyskjRozIu9/97qxduzZNTU3p7u5O8kpoampqSpK0tLRkzZo1Pdfs3r07LS0tefnll7N79+5jjr96za5du5Ikhw8fzv79+3PBBRfk+eeff313B6dh3pbVvbb37DETe21vAAAA6E0nHZHOOeecfOc738msWbPyy1/+8pjvV1V1Rgd7LdOnT8+nPvWpJMmwYcN6/ecBAAAA8IqT+tvZGhsb853vfCcPPfRQvve97yVJnnvuuTQ3NydJmpubs2fPniRJV1dXhg8f3nNta2trurq60tXVldbW1mOO/+41DQ0NGTx48Gs+hXT//fdn/PjxGT9+fPbu3Xs69wsAAADAaTipiPTAAw9k69atueOOO3qOLVu2LFOnTk2STJ06NY888kjP8fb29gwYMCAjR45MW1tb1q1bl+7u7hw4cCATJkxIktx4441HXfPqXtddd11WrVp15u4QAAAAgNet+Drb+973vtx4443ZvHlzzwdsf/7zn8+Xv/zlLFmyJNOmTcvOnTtz/fXXJ0k6OjqyZMmSdHR05NChQ5kxY0aOHDmSJLnpppuyYMGCDBw4MMuXL8/y5cuTvBKpFi1alM7Ozrzwwgtpb2/vrfsFAAAA4DQUI9K///u/p1+/fq/5vcsvv/w1j8+dOzdz58495viGDRsyZsyYY46/9NJLPREKAAAAgL7npF5nAwAAAODNTUQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKBIRAIAAACgSEQCAAAAoKix1gMAZ868Lat7be/ZYyb22t4AAAD0fZ5EAgAAAKBIRAIAAACgSEQCAAAAoEhEAgAAAKDIB2sDNecDwQEAAPo+TyIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUCQiAQAAAFAkIgEAAABQJCIBAAAAUNRnItKVV16ZJ598Mp2dnfnc5z5X63EAAAAA+C19IiL1798/d999d6666qr8yZ/8SW644YaMHj261mMBAAAA8H/6RES65JJLsm3btjzzzDN5+eWXs3jx4kyZMqXWYwEAAADwf/olqWo9xLXXXpvJkydn+vTpSZKPfexjmTBhQmbOnHnUedOnT8+nPvWpJMk73vGO/PznP++VeYYNG5a9e/f2yt69rV5nr9e5k/qdvV7nTup39nqdO6nf2et17qR+Z6/XuZP6nb1e507qd/Z6nTup3wS4EcEAABMQSURBVNnrde6kfmev17mT+p29XudO6nf2ep076d3ZR4wYkbe//e3H/X5V63XttddW999/f8/XH/vYx6qvfOUrNZtn/fr1Nf81ebPNXq9z1/Ps9Tp3Pc9er3PX8+z1Onc9z16vc9fz7PU6dz3PXq9z1/Ps9Tp3Pc9er3PX8+z1Onc9z16vc9dy9j7xOltXV1eGDx/e83Vra2u6urpqOBEAAAAAv61PRKT169enra0tI0eOzFlnnZX29vYsW7as1mMBAAAA8H8aknyx1kNUVZXOzs489NBDmTlzZr75zW/mu9/9bk1n2rhxY01//utRr7PX69xJ/c5er3Mn9Tt7vc6d1O/s9Tp3Ur+z1+vcSf3OXq9zJ/U7e73OndTv7PU6d1K/s9fr3En9zl6vcyf1O3u9zp3UZvY+8cHaAAAAAPRtfeJ1NgAAAAD6NhEJAAAAgCIRCQAAAICiPvHB2rX2jne8I9OmTcv111+fq666Ku985zvz/PPPZ+/evbUe7Q3rHe94R8aOHZs9e/bk5Zdf7jl+5ZVXZvv27TWcrGz8+PG58MIL8+yzz2b06NH5+Mc/nqFDh2bbtm21Hu2ULFy4MN///vdrPcYpe9/73pfrrrsu5513Xp5++ulaj3Ncl1xySQ4cOJCDBw/mrW99a/7+7/8+n/3sZzN27NisW7cuL730Uq1HPK6ZM2fm2WefzYEDB2o9yik566yz8tGPfjRve9vb8swzz+SGG27I3/zN32TUqFF5/PHHc+TIkVqPeFyjRo3KtGnT8pGPfCQf+tCHMmrUqDz11FM5ePBgrUcDAIAeb/oP1r7llltyww03ZPHixdm9e3eSpLW1Ne3t7Vm8eHFuv/32Gk94ej7xiU9kwYIFtR7jNc2cOTMzZszI1q1b8653vSuf+cxnsmzZsiTJhg0b8p73vKfGEx7fF77whVx11VVpbGzMj3/840yYMCE/+clP8qEPfSg/+tGPMnfu3FqP+JoeeeSRo77u169fPvjBD2bVqlVJkilTptRirJOydu3aTJgwIUnyyU9+MjNmzMj3vve9XHHFFfnBD37QZ/+MPvHEExk7dmwOHz6ce++9N7/+9a+zdOnSTJo0KWPHjs21115b6xGP68UXX8yvfvWrbN++Pd/61rfy8MMP10VU/+Y3v5nGxsacffbZefHFF3Puuefmu9/9biZNmpR+/frlE5/4RK1HfE0zZ87Mn//5n+enP/1pPvzhD+fxxx/Piy++mL/8y7/MTTfdlMcee6zWIwIAQI/qzbx+/vOfV42NjcccP+uss6qnnnqq5vOd7tq5c2fNZzje2rx5c3XOOedUSaoRI0ZU69evr26++eYqSbVx48aaz1eavX///tXAgQOr/fv3V+edd16VpHrrW99abdq0qebzHW9t2LChWrRoUfX+97+/uuyyy6r3v//91bPPPltddtll1WWXXVbz+U60fvv3xLp166phw4ZVSaqzzz672rx5c83nO97q6Og46tf/t7/3+OOP13y+0q95v379qg996EPV17/+9WrPnj3V8uXLqxtvvLE699xzaz7f8darfwYbGhqq7u7uqn///sd8ry+uV/+7kqQaOHBg9ZOf/KRKUg0fPrzP/zfRst6s621ve1vNZ3izrfPPP7/mM1jWmVyDBg2qbrvttmrr1q3V888/X+3du7fq6Oiobrvttmrw4ME1n+9016OPPlrzGY63zjvvvGru3LnVgw8+WN1www1Hfe/uu++u+XwnWk1NTdX8+fOrr371q9X5559f3XrrrdXmzZurb3/721Vzc/PvdZY3/WciHTlyJBdeeOExx//gD/6gT7/6kCSbNm16zbV58+Y0NTXVerzj6t+/f371q18lSXbu3JkPfOADueqqqzJv3rz069evxtOd2KFDh3LkyJH85je/yfbt2/PLX/4ySfK///u/ffr3y3vf+95s2LAhf/d3f5f9+/fnsccey29+85v89Kc/zU9/+tNaj3dC/fv3z5AhQ3L++eenX79+PU/E/PrXv86hQ4dqPN3xPfHEEz1PvmzatKnnCbu2trajXuHsi6qqSlVV+fGPf5xPfvKTufDCCzN//vxMnjy5T79C2L9//5x11lk577zzcvbZZ2fw4MFJkre85S0566yzajzdiTU2NiZ5ZdZzzz03SbJr164+P/egQYNy2223ZevWrT2vgXd0dOS2227r+fWvR48++mitRziu8847L3Pnzs2DDz6YG2644ajv3X333TWa6uQ0NTVl/vz5+epXv5rzzz8/t956azZv3pxvf/vbaW5urvV4xzV06NCj1vnnn59169ZlyJAhGTp0aK3HO6Err7yy598HDRqUr3/969m0aVMeeuihvP3tb6/hZCd222235YILLkiSvOc978n27duzdu3a7NixI5dddlmNpzuxV/9/6w//8A9rPcopec973pNVq1Zl0aJFaW1tzYoVK/Liiy9m3bp1ede73lXr8U7onHPOyT/8wz/kiSeeyIsvvpg9e/Zk9erVmTp1aq1HO6ElS5Zk3759+cAHPpALLrggw4YNywc/+MHs27cvS5YsqfV4J/Tud7/7Nde4ceP69O+Xb3zjG+nXr1++853vpL29PUuXLs2AAQOSJJdeemmNpzuxBQsWpKOjI7t27cpPfvKT/OY3v8mHP/zh/Nu//Vu+9rWv/d7nqXlVq+W68sorq87OzurRRx+t7r333uree++tli9fXnV2dlZXXnllzec70eru7q7Gjh1b/b//9/+OWiNGjKi6urpqPt/x1sqVK6uxY8cedayhoaFauHBhdejQoZrPd6K1Zs2aauDAgVWSql+/fj3HBw0adMzTJn1xtbS0VEuWLKm+8pWv9Omn1X57PfPMM9X27durp59+utq+fXtPaT/nnHP69BM9gwYNqr7xjW9U27Ztq9asWVMdPHiw2r59e/Wv//qv1Tvf+c6az3eidaKnX179/d8X16xZs6rt27dXO3bsqGbOnFn9y7/8S3XfffdVmzdvrr7whS/UfL7jrZtvvrnatGlTdd9991Vbt26tPvGJT1RJqmHDhlWPPfZYzec70frhD39Y3XLLLVVTU1PPsaampuqWW26pfvSjH9V8vhOtd7/73a+5xo0bVz377LM1n+94a+nSpdVtt91WTZkypXrkkUeqpUuXVgMGDKiSY5967Gtr+fLl1ac//enqc5/7XLVp06bqlltuqVpbW6tPf/rT1fe///2az3e8dfjw4erpp58+ah08ePD/t3f/oVXVfxzHX3f2xwQj18SxjO7Q6daP2Q8TreSuNJv9MGvXwPrD+0etojTUfiARWUiwEYIgIbFyFknZSk2m02qbml6bYu2HbC5izvXHEtRBsWDh7f39I3bZ1d175765cz76fMDrj53P7j0vDne75372OWfx9yWv+6XK4NdEZWWlrV271m666SZbsWKFbd++3fN+yTJ4pXF9fb3dfffdJsmmTp1qR48e9bxfqnR2dtr7779vp06dssbGRluxYoXl5uZ63itdGhsbbcGCBbZkyRLr7u62cDhskmzu3LkWjUY975cqO3bssEgkYpMmTbKVK1faW2+9Zfn5+bZ582Z77733PO+XLCdOnBjRmB9y/vx5q6urs/r6+ovy119/ed4vWS787PDmm2/awYMH7frrr/f9e+jg8/MLP8d58JnI+wPidQKBgM2aNctKS0uttLTUZs2alXAZhF/z0Ucf2X333Tfk2JYtWzzvlyyTJk1K+MAxOPfee6/n/VJl4ET9wmRnZ9ttt93meb/h5pFHHvH1m+pwMnbsWMvLy/O8R7pce+21Nn36dLvrrrts4sSJnvcZTqZOnep5h5EmNzc3frJ+3XXXWTgctpkzZ3reK11uueUWC4fDVlBQ4HmXSwknwKMfToBHP6tWrbLa2tqE9/nOzk7Pew0ng18TFx5jPx/ztrY2GzNmjEmyw4cPJ4z5+VL2C4/5nDlz7IMPPrCenh6rr6+3srIyz/slS6qfT79fWt3U1JTw9ZEjR0z69zNee3u75/2SZe/evfb6668nnB9OnDjR3njjDfvuu+8875cqra2tlp+fP+RYd3e35/2Spa2tLWEhgCSLRCJ2/Phx6+rq8rxfqgx+na9duzZhzIPfi94fEEIIIYS4F06ARz+cAHuTgZW869ats3Hjxvl+BdJAfvvtN1u5cqWtWrXqos5+vlfcsmXLbO/evfbAAw/YmjVrbP369RYKheydd96xTz/91PN+qTLUZG5GRoaVlJTYpk2bPO+XLNFo1ObPn2+LFy+2rq4uW7RokUmyUCjk+9Vfhw4div9hfeHChbZnz574mJ//oDF+/HgrLy+39vZ2O3funJ09e9ba2tqsvLzcsrKyPO+XKuFw2KZNmzbk2MBrx4+pqKiwefPmXbS9pKTE9/dDfvfdd+P3FR6cKVOmWHV19Wj38f6AEEIIIcS9DD4BPnv2bMIJ8Pjx4z3vlyqcAI9+fHYCPKIsXLjQDh8+bD09PZ53GU7efvvthAz8c4qcnBz75JNPPO+XKsXFxfbFF1/YTz/9ZC0tLbZr1y4rKyuLr1Dyaz7//HPPO4wk06dPtz179tju3butoKDA1q9fb729vXb8+HG75557PO+XKkVFRdbY2Gjnzp2zH374Ib6iesKECbZ8+XLP+6VKQUGBzZs376LfjX6/rcpA97lz5zrXPVnvBQsWeN7NoWPu/cEghBBCyJWVgXs7uRhXu7va27XumZmZduuttzrX2+VjfiX0drm7q7393n358uV24sQJ2759u508edIef/zx+JjfL092tfuyZcuc7O3D7t4fEEIIIYRcWXHl5v1XUndXe7vc3dXeLnd3tbfL3V3t7ffuLS0t8RUlwWDQjh49aq+88opJ/r8PlavdXe3tt+7//k9hAACAS9Tc3Dzk9kAgoJycnFFuc2lc7e5qb8nd7q72ltzt7mpvyd3urvaW3O2ekZGhvr4+SdKpU6d0//3366uvvlIwGFQgEPC4XWqudne1t+Sv7kwiAQCAEcnJyVFJSYl6e3sTtgcCAUWjUY9aDY+r3V3tLbnb3dXekrvdXe0tudvd1d6Su91Pnz6t22+/PT4J1tfXp8cee0ybNm1SUVGRx+1Sc7W7q70lf3VnEgkAAIxITU2Nxo0bN+Rfgfft2zf6hS6Bq91d7S25293V3pK73V3tLbnb3dXekrvdly5dqvPnzydsi8ViikQi+vDDDz1qNTyudne1t+Sv7gH9e10bAAAAAAAAkFSG1wUAAAAAAADgf0wiAQAAAAAAIC0mkQAAACRVVlbq5ptvTjq+Zs0avfrqq5dl35FIRBs2bLgszw0AAPBf4cbaAAAAksrKyryu8J8bM2aMYrGY1zUAAMAVgpVIAADgqhIMBtXe3q7NmzerublZ1dXVGjt2rBoaGjRjxgxJUklJiY4dO6ampiZ9//33Fz3Hc889p927dyszM1N//vlnfHs4HFZVVZUkqaqqShs3btSBAwfU0dGhRx99NGWvG264QbW1tfrll19UUVER375kyRK1tLSotbVV5eXl8e2p9rtu3TrV19cnPA8AAMD/i5VIAADgqlNYWKhnn31W0WhUH3/8sV566aX42IQJE1RZWalQKKSuri5lZWUlPPbll1/W/Pnz9cQTT+jvv/9OuZ+8vDwVFxdrypQpamhoUH5+vvr7+4f83jvuuEN33nmn+vv71dHRoQ0bNigWi6miokIzZsxQb2+vvv32Wy1atEjffPNNyv1OmzZNDz74oP75559hHhEAAID0WIkEAACuOt3d3YpGo5Kkzz77THPmzImPzZ49WwcOHFBXV5ckqbe3Nz62dOlSPfzww1q8eHHaCSRJ+vLLL2Vm+vXXX9XZ2anCwsKk31tXV6c//vhD/f39amtrUzAY1MyZM7Vv3z6dOXNGsVhMW7ZsUSgUSrvf6upqJpAAAMB/jkkkAABw1TGzlF8n09raqry8PN14441DPjYzM3PE+xm8QikWi+maa1IvGE+1376+vpSPBQAAGAkmkQAAwFUnGAxq9uzZkqRnnnlGBw8ejI/9+OOPCoVCysvLk6SEy9l+/vlnvfDCC9q5c6dyc3MlSadPn1ZhYaECgYCefPLJhP089dRTCgQCmjx5siZPnqyOjo5L6nnkyBEVFxcrOztbGRkZevrpp7V///60+wUAALgcmEQCAABXnfb2dkUiETU3NysrK0sbN26Mj505c0bPP/+8tm3bpqamJm3dujXhsYcOHdJrr72mXbt2KTs7W6tXr1ZNTY3q6urU09OT8L0dHR3av3+/amtr9eKLLya9H1Iyv//+u1avXq2GhgY1Nzfr2LFj2rlzpySl3C8AAMDlEJA0vPXbAAAAV4BgMKiamhoVFRVd1v1UVVWppqZGX3/99WXdDwAAwGhhJRIAAAAAAADSYiUSAADAKHnooYdUUVGRsO3kyZMqLS31qBEAAMDwMYkEAAAAAACAtLicDQAAAAAAAGkxiQQAAAAAAIC0mEQCAAAAAABAWkwiAQAAAAAAIC0mkQAAAAAAAJDW/wAxyUeM616BTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rows = spark.sql(f\"\"\"\n",
    "    SELECT \n",
    "        pickup_hour, COUNT(1) as pop\n",
    "    FROM \n",
    "        taxi\n",
    "    GROUP BY \n",
    "        pickup_hour\n",
    "    ORDER BY \n",
    "        pickup_hour\n",
    "\"\"\")\\\n",
    "    .collect()\n",
    "\n",
    "pdf = pd.DataFrame(rows,columns=['pickup_hour','pop'])\n",
    "\n",
    "plt.clf()\n",
    "ax = pdf.plot(x='pickup_hour',y='pop',kind='bar',figsize=(20,10))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot 3: Total count of taxi trips, binned by log(duration)\n",
    "- <font size=\"3\">The distribution is roughly log-normal</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABJcAAAJaCAYAAACSr6a4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdf6yWd2H//1eBHYtWOAwcx55DgC13N8bQFAu0M85+Qnd6tmohGaEnOkFL2NIyta6bO3Zb8MfCirGjzK0kIvJrNkdETVkmlhOwrjEFjtAW3IFx0IKcUw+1cAraNEXg/v7Rb++spRV9n1No6+ORvJPyvq/3db2v80+TZ677ui9JUg0AAAAAFBhysTcAAAAAwGuXuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQbdr4DVq1alfe85z154oknMmXKlNr8X/3VX2XRokU5c+ZM/uu//it/93d/lyRpa2vLggULcubMmXzkIx/Jli1bkiRTp07NmjVrMnz48Hzzm9/MRz/60SRJXV1d1q1bl3e84x05duxYbrrpphw+fDhJMm/evPzDP/xDkuSf/umfsm7duvPe0BNPPFFbDwAAAMDAjR8/Pr/1W7/1sp9Xf9F417veVb3yyiure/furc1de+211Y6OjmpdXV01SfUtb3lLNUl10qRJ1UceeaRaV1dXnTBhQvXgwYPVIUOGVJNUd+zYUZ0xY0Y1SfWb3/xmtaWlpZqkesstt1RXrFhRTVK96aabqu3t7dUk1VGjRlV/8IMfVEeNGlWtr6+v/uAHP6jW19f/wr0mqXZ2dp73GMMwDMMwDMMwDMMwDOOXH7+ot5z3a3EPPvhgjh8//oK5W265JXfeeWdOnTqVJPnJT36SJJk1a1ba29tz6tSpHDp0KAcPHsz06dPT0NCQESNGZMeOHUmSdevWZfbs2bU1a9euTZJs3LgxM2fOTJJcf/316ejoSH9/f5566ql0dHSkpaXlfNsFAAAA4AIqeufSFVdckXe9613Zvn17HnjggVx11VVJksbGxhw5cqR2XE9PTxobG9PY2Jienp5z5l+85syZMzlx4kRGjx79sud6KQsXLkxnZ2c6OzszZsyYklsCAAAAoMB537n0kouGDctv/uZv5uqrr860adOyYcOG/PZv//Zg7+2XtnLlyqxcuTJJ0tnZedH2AQAAAPDrpigu9fT05Otf/3qS52LO2bNnM2bMmPT29mbcuHG145qamtLb25ve3t40NTWdM5+ktqa3tzdDhw7NyJEjc+zYsfT29ubaa699wZoHHnigZLsAAADA68yoUaNy2223ZcKECbnkkksu9nZeF6rVag4dOpS77747/f39v9ra843x48e/4IXef/mXf1n91Kc+VU1SrVQq1R/96EfVJNXf//3ff8ELvX/wgx+87Au9/+RP/qSapHrrrbe+4IXeX/nKV6rJcy/0/uEPf1itr6+v1tfXV3/4wx9WR40aNaAXTBmGYRiGYRiGYRiG8foYn/rUp6rvfe97q0OHDr3oe3m9jKFDh1ZvvPHGWvP5v+M8veUXn/jee++tPv7449VTp05Vjxw5Ur355purv/Ebv1Fdv359de/evdVdu3ZV/9//+3+14++4447qwYMHq/v376/9IlyS6jve8Y7q3r17qwcPHqx+/vOfr82/4Q1vqG7YsKHa3d1d3bFjR3XixIm1zz70oQ9Vu7u7q93d3dUPfvCDv9QfQlwyDMMwDMMwDMMwjNf/WLt2rbD0CoyhQ4dW165de878L+otl/z///G60dnZmWnTpl3sbQAAAACvoHXr1mXevHkXexuvSy/1t/1FvaXo1+IAAAAAICl8oTcAAADAq8ldex8a1PPdPuWa8x5z+vTp7N27t/bv2bNn5/Dhw4O6j9cCcQkAAACgwDPPPJMrr7zyZT+/5JJLUq2+rt5G9JJ8LQ4AAABgEIwfPz5dXV3593//9+zevTvjxo3L3/zN32Tnzp159NFH88lPfrJ27B133JH9+/eno6Mj9957b26//fYkybe//e284x3vSJKMHj06jz32WJJkyJAh+exnP1s711/8xV8kSd797nfn29/+dr761a9m3759+Y//+I/aNa666qp897vfzSOPPJIdO3bksssuy3e+8528/e1vrx3z4IMP5m1ve9uA7tuTSwAAAAAFhg8fnocffjhJ8thjj+VjH/tYfvd3fzcf+tCHsmjRovzxH/9xKpVKpk+fnksuuSSbNm3Ku971rjz99NNpbW3NlVdemWHDhmX37t3ZtWvXL7zWggULcuLEiUyfPj11dXX57ne/my1btiRJrrzyykyePDmPP/54vvvd7+ad73xndu7cma985Su56aab8r3vfS9vfvOb88wzz2TVqlX54Ac/mI997GOpVCq59NJLs2fPngH9HcQlAAAAgAIv/lrc+PHjc/jw4ezYsSNJ0tzcnObm5lqAuuyyy1KpVPLmN7853/jGN/LMM88kSTZt2nTeazU3N+dtb3tb5syZkyQZOXJkKpVKTp06lZ07d6a3tzdJ8sgjj2TChAk5ceJEfvzjH+d73/tekuSnP/1pkuSrX/1q/vEf/zF/+7d/m5tvvjlr1qwZ8N9BXAIAAAAYJE8//XTtvy+55JL88z//c77whS+84JiPfvSjL7v+9OnTGTLkubcYXXrppS8414c//OHa00rPe/e7351nn3229u8zZ85k2LCXzz3PPPNMOjo6MmvWrMydO7f2FbyB8M4lAAAAgFfA/fffn5tvvjlvetObkiSXX3553vKWt+S///u/M3v27Fx66aW57LLL8t73vre25tChQ7Xg8/xTSs+f65ZbbqmFo0qlkje+8Y0ve+3//d//zVvf+tZcddVVSZ57amro0KFJki9+8Yv513/913R2duapp54a8H16cgkAAAB4zbt9yjUXewvn6OjoyKRJk/LQQw8lSX72s5/lz//8z/Pwww/nK1/5Sh555JEcPnw4Dz74YG3N5z73uWzYsCEf+MAHsnXr1tr8F7/4xUyYMCG7d+/OJZdckp/85CeZPXv2y1775z//eW666aZ8/vOfz/Dhw/PMM8/kuuuuy9NPP53du3fn5MmTWb169aDc5yVJXle/idfZ2Zlp06Zd7G0AAAAAr6B169Zl3rx5F3sbg2Lx4sX52c9+lrvuuuuCXO+tb31rHnjggfze7/1eqtVzs9BL/W1/UW/xtTgAAACAXxMf+MAHsmPHjvz93//9S4alEr4WBwAAAHARfepTn7pg11q/fn3Wr18/qOf05BIAAADwmlOtVmsvqGbwDB069Fd+oklcAgAAAF5zDh06lBtuuEFgGkRDhw7NDTfckEOHDv1K63wtDgAAAHjNufvuu3Pbbbflz/7sz3LJJZdc7O28LlSr1Rw6dCh33333r7ROXAIAAABec/r7+7N48eKLvQ3ia3EAAAAADIAnlwCA16W79j50Qa93+5RrLuj1AABeLTy5BAAAAEAxcQkAAACAYr4WBwC/pnxtDACAweDJJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYsMu9gYAAPjV3LX3oQt6vdunXHNBrwcAvLZ4cgkAAACAYp5cAoCX4ekQAAA4P08uAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi541Lq1atytGjR7N3795zPvvrv/7rVKvVjB49ujbX1taW7u7u7N+/P83NzbX5qVOnZs+ePenu7s7y5ctr83V1dWlvb093d3e2b9+e8ePH1z6bN29eDhw4kAMHDmTevHnFNwkAAADAK+O8cWnNmjVpaWk5Z76pqSnNzc05fPhwbW7SpElpbW3N5MmT09LSknvuuSdDhjx3iRUrVmThwoWpVCqpVCq1cy5YsCD9/f2pVCpZtmxZli5dmiQZNWpUFi9enBkzZmT69OlZvHhx6uvrB+WmAQAAABgc541LDz74YI4fP37O/LJly/Lxj3881Wq1Njdr1qy0t7fn1KlTOXToUA4ePJjp06enoaEhI0aMyI4dO5Ik69aty+zZs2tr1q5dmyTZuHFjZs6cmSS5/vrr09HRkf7+/jz11FPp6Oh4ycgFAAAAwMVT9M6lG2+8Mb29vdmzZ88L5hsbG3PkyJHav3t6etLY2JjGxsb09PScM//iNWfOnMmJEycyevTolz3XS1m4cGE6OzvT2dmZMWPGlNwSAAAAAAWG/aoLhg8fnjvuuOMF71O62FauXJmVK1cmSTo7Oy/ybgAAAAB+ffzKTy79zu/8TiZOnJhHH300jz32WJqamrJ79+6MHTs2vb29GTduXO3Ypqam9Pb2pre3N01NTefMJ3nBmqFDh2bkyJE5duzYy54LAAAAgFePXzkuff/738/YsWMzceLETJw4MT09PZk6dWqOHj2aTZs2pbW1NXV1dZkwYUIqlUp27tyZvr6+nDx5MjNmzEjy3K/A3XfffUmSTZs2Zf78+UmSOXPmZNu2bUmS+++/P83Nzamvr099fX2am5tz//33D9Z9AwAAADAIzvu1uHvvvTfXXnttxowZkyNHjmTx4sX50pe+9JLHdnV1ZcOGDenq6srp06ezaNGinD17Nkly6623Zs2aNRk+fHg2b96czZs3J0lWrVqV9evXp7u7O8ePH09ra2uSpL+/P5/5zGdqX3P79Kc/nf7+/kG5aQAAAAAGx3nj0vve975f+PnEiRNf8O8lS5ZkyZIl5xy3a9euTJky5Zz5Z599NnPnzn3Jc69evTqrV68+3xYBAAAAuEiKfi0OAAAAABJxCQAAAIABEJcAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKDbsYm8AgNeuu/Y+dEGvd/uUay7o9QAAgPPz5BIAAAAAxTy5BADAq4qnIgHgtcWTSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACg2Hnj0qpVq3L06NHs3bu3NvfZz342+/bty6OPPpqvf/3rGTlyZO2ztra2dHd3Z//+/Wlubq7NT506NXv27El3d3eWL19em6+rq0t7e3u6u7uzffv2jB8/vvbZvHnzcuDAgRw4cCDz5s0b8M0CAAAAMLjOG5fWrFmTlpaWF8x1dHTkD/7gD/L2t789Bw4cyCc+8YkkyaRJk9La2prJkyenpaUl99xzT4YMee4SK1asyMKFC1OpVFKpVGrnXLBgQfr7+1OpVLJs2bIsXbo0STJq1KgsXrw4M2bMyPTp07N48eLU19cP6s0DAAAAMDDnjUsPPvhgjh8//oK5jo6OnDlzJkmyffv2NDU1JUlmzZqV9vb2nDp1KocOHcrBgwczffr0NDQ0ZMSIEdmxY0eSZN26dZk9e3Ztzdq1a5MkGzduzMyZM5Mk119/fTo6OtLf35+nnnoqHR0d50QuAAAAAC6uAb9z6eabb87mzZuTJI2NjTly5Ejts56enjQ2NqaxsTE9PT3nzL94zZkzZ3LixImMHj36Zc/1UhYuXJjOzs50dnZmzJgxA70lAAAAAH5JA4pLd9xxR06fPp0vf/nLg7WfIitXrsy0adMybdq0PPnkkxd1LwAAAAC/Torj0vz58/Oe97wn73//+2tzvb29GTduXO3fTU1N6e3tTW9vb+2rc/93/sVrhg4dmpEjR+bYsWMvey4AAAAAXj2K4tL111+fj3/847nxxhvzzDPP1OY3bdqU1tbW1NXVZcKECalUKtm5c2f6+vpy8uTJzJgxI8lzvwJ333331dbMnz8/STJnzpxs27YtSXL//fenubk59fX1qa+vT3Nzc+6///4B3SwAAAAAg2vY+Q649957c+2112bMmDE5cuRIFi9enE984hN5wxvekI6OjiTPvdT7lltuSVdXVzZs2JCurq6cPn06ixYtytmzZ5Mkt956a9asWZPhw4dn8+bNtfc0rVq1KuvXr093d3eOHz+e1tbWJEl/f38+85nPpLOzM0ny6U9/Ov39/a/IHwEAAACAMueNS+973/vOmfvSl770sscvWbIkS5YsOWd+165dmTJlyjnzzz77bObOnfuS51q9enVWr159vi0CAAAAcJEM+NfiAAAAAPj1JS4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAsWEXewMAr2d37X3ogl7v9inXXNDrAQAAeHIJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUOy8cWnVqlU5evRo9u7dW5sbNWpUtmzZkgMHDmTLli2pr6+vfdbW1pbu7u7s378/zc3NtfmpU6dmz5496e7uzvLly2vzdXV1aW9vT3d3d7Zv357x48fXPps3b14OHDiQAwcOZN68eQO+WQAAAAAG13nj0po1a9LS0vKCuba2tmzdujVXXHFFtm7dmra2tiTJpEmT0tramsmTJ6elpSX33HNPhgx57hIrVqzIwoULU6lUUqlUaudcsGBB+vv7U6lUsmzZsixdujTJcwFr8eLFmTFjRqZPn57Fixe/IGIBAAAAcPGdNy49+OCDOX78+AvmZs2albVr1yZJ1q5dm9mzZ9fm29vbc+rUqRw6dCgHDx7M9OnT09DQkBEjRmTHjh1JknXr1r1gzfPn2rhxY2bOnJkkuf7669PR0ZH+/v489dRT6ejoOCdyAQAAAHBxFb1zaezYsenr60uS9PX1ZezYsUmSxsbGHDlypHZcT09PGhsb09jYmJ6ennPmX7zmzJkzOXHiREaPHv2y53opCxcuTGdnZzo7OzNmzJiSWwIAAACgwKC80LtarQ7GaYqtXLky06ZNy7Rp0/Lkk09e1L0AAAAA/DopiktHjx5NQ0NDkqShoSFPPPFEkqS3tzfjxo2rHdfU1JTe3t709vamqanpnPkXrxk6dGhGjhyZY8eOvey5AAAAAHj1KIpLmzZtyvz585Mk8+fPz3333Vebb21tTV1dXSZMmJBKpZKdO3emr68vJ0+ezIwZM5I89ytw/3fN8+eaM2dOtm3bliS5//7709zcnPr6+tTX16e5uTn333//wO4WAAAAgEE17HwH3Hvvvbn22mszZsyYHDlyJIsXL86dd96ZDRs2ZMGCBTl8+HDmzp2bJOnq6sqGDRvS1dWV06dPZ9GiRTl79myS5NZbb82aNWsyfPjwbN68OZs3b06SrFq1KuvXr093d3eOHz+e1tbWJEl/f38+85nPpLOzM0ny6U9/Ov39/a/IHwEAAACAMueNS+973/tecv666657yfklS5ZkyZIl58zv2rUrU6ZMOWf+2WefrcWpF1u9enVWr159vi0CAAAAcJEMygu9AQAAAPj1JC4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLn/bU4AABg8Ny196ELer3bp1xzQa8HwK8fTy4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFBhSXbrvttnz/+9/P3r17c++99+YNb3hDRo0alS1btuTAgQPZsmVL6uvra8e3tbWlu7s7+/fvT3Nzc21+6tSp2bNnT7q7u7N8+fLafF1dXdrb29Pd3Z3t27dn/PjxA9kuAAAAAIOsOC5dfvnl+chHPpKrrroqU6ZMydChQ9Pa2pq2trZs3bo1V1xxRbZu3Zq2trYkyaRJk9La2prJkyenpaUl99xzT4YMee7yK1asyMKFC1OpVFKpVNLS0pIkWbBgQfr7+1OpVLJs2bIsXbp0EG4ZAAAAgMEybECLhw3L8OHD8/Of/zxvfOMb8/jjj+cTn/hErr322iTJ2rVr88ADD6StrS2zZs1Ke3t7Tp06lUOHDuXgwYOZPn16Dh06lBEjRmTHjh1JknXr1mX27Nn51re+lVmzZuWTn/xkkmTjxo35t3/7twHdLPDqdNfehy7o9W6fcs0FvR4AAMDrWfGTS48//ng+97nP5Uc/+lF+/OMf58SJE+no6MjYsWPT19eXJOnr68vYsWOTJI2NjTly5EhtfU9PTxobG9PY2Jienp5z5l+85syZMzlx4kRGjx59zl4WLlyYzs7OdHZ2ZsyYMaW3BAAAAMCvqDgu1dfXZ9asWZk4cWIuv/zyvOlNb8r73//+c46rVqsD2uAvY+XKlZk2bVqmTZuWJ5988hW/HgAAAADPKY5L1113XR577LE8+eSTOX36dL7+9a/nD//wD3P06NE0NDQkSRoaGvLEE08kSXp7ezNu3Lja+qampvT29qa3tzdNTU3nzL94zdChQzNy5MgcO3asdMsAAAAADLLiuPSjH/0oV199dYYPH54kmTlzZvbt25dNmzZl/vz5SZL58+fnvvvuS5Js2rQpra2tqaury4QJE1KpVLJz58709fXl5MmTmTFjRpJk3rx5L1jz/LnmzJmTbdu2ld8pAAAAAIOu+IXeO3fuzMaNG7N79+6cPn06Dz/8cL7whS/ksssuy4YNG7JgwYIcPnw4c+fOTZJ0dXVlw4YN6erqyunTp7No0aKcPXs2SXLrrbdmzZo1GT58eDZv3pzNmzcnSVatWpX169enu7s7x48fT2tr6yDcMgAAAACDZUC/FvfJT36y9mtuzzt+/Hiuu+66lzx+yZIlWbJkyTnzu3btypQpU86Zf/bZZ2txCgAAAIBXn+KvxQEAAACAuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoNuxibwAAAHj9uGvvQxf0erdPueaCXg+Ac3lyCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxQYUl0aOHJmvfvWr2bdvX7q6unL11Vdn1KhR2bJlSw4cOJAtW7akvr6+dnxbW1u6u7uzf//+NDc31+anTp2aPXv2pLu7O8uXL6/N19XVpb29Pd3d3dm+fXvGjx8/kO0CAAAAMMgGFJeWL1+eb33rW5k0aVLe/va3Z9++fWlra8vWrVtzxRVXZOvWrWlra0uSTJo0Ka2trZk8eXJaWlpyzz33ZMiQ5y6/YsWKLFy4MJVKJZVKJS0tLUmSBQsWpL+/P5VKJcuWLcvSpUsHeLsAAAAADKbiuDRixIj80R/9UVatWpUk+fnPf54TJ05k1qxZWbt2bZJk7dq1mT17dpJk1qxZaW9vz6lTp3Lo0KEcPHgw06dPT0NDQ0aMGJEdO/90ibIAACAASURBVHYkSdatW/eCNc+fa+PGjZk5c2b5nQIAAAAw6Irj0sSJE/OTn/wkq1evzu7du7Ny5cq88Y1vzNixY9PX15ck6evry9ixY5MkjY2NOXLkSG19T09PGhsb09jYmJ6ennPmX7zmzJkzOXHiREaPHn3OXhYuXJjOzs50dnZmzJgxpbcEAAAAwK+oOC4NGzYsU6dOzYoVKzJ16tQ8/fTTta/A/V/VanVAG/xlrFy5MtOmTcu0adPy5JNPvuLXAwAAAOA5xXGpp6cnPT092blzZ5LnvrY2derUHD16NA0NDUmShoaGPPHEE0mS3t7ejBs3rra+qakpvb296e3tTVNT0znzL14zdOjQjBw5MseOHSvdMgAAAACDbFjpwqNHj+bIkSO54oorcuDAgcycOTNdXV3p6urK/Pnzs3Tp0syfPz/33XdfkmTTpk2599578y//8i+5/PLLU6lUsnPnzpw9ezYnT57MjBkzsmPHjsybNy+f//zna2vmz5+f7du3Z86cOdm2bdvg3DW8xty196ELer3bp1xzQa8HAADAa1dxXEqSD3/4w/nyl7+curq6/PCHP8yHPvShDBkyJBs2bMiCBQty+PDhzJ07N0nS1dWVDRs2pKurK6dPn86iRYty9uzZJMmtt96aNWvWZPjw4dm8eXM2b96cJFm1alXWr1+f7u7uHD9+PK2trQO8XQAAAAAG04Di0qOPPppp06adM3/ddde95PFLlizJkiVLzpnftWtXpkyZcs78s88+W4tTAAAAALz6FL9zCQAAAADEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAECxYRd7AwAAAK8Vd+196IJe7/Yp11zQ6wGU8OQSAAAAAMXEJQAAAACKiUsAAAAAFBOXAAAAACgmLgEAAABQTFwCAAAAoJi4BAAAAEAxcQkAAACAYuISAAAAAMXEJQAAAACKiUsAAAAAFBtwXBoyZEh2796d//zP/0ySjBo1Klu2bMmBAweyZcuW1NfX145ta2tLd3d39u/fn+bm5tr81KlTs2fPnnR3d2f58uW1+bq6urS3t6e7uzvbt2/P+PHjB7pdAAAAAAbRgOPSRz/60ezbt6/277a2tmzdujVXXHFFtm7dmra2tiTJpEmT0tramsmTJ6elpSX33HNPhgx57vIrVqzIwoULU6lUUqlU0tLSkiRZsGBB+vv7U6lUsmzZsixdunSg2wUAAABgEA0oLjU2NuaGG27IF7/4xdrcrFmzsnbt2iTJ2rVrM3v27Np8e3t7Tp06lUOHDuXgwYOZPn16GhoaMmLEiOzYsSNJsm7duhesef5cGzduzMyZMweyXQAAAAAG2YDi0t13352Pf/zjOXv2bG1u7Nix6evrS5L09fVl7NixSZ4LUUeOHKkd19PTk8bGxjQ2Nqanp+ec+RevOXPmTE6cOJHRo0efs4+FCxems7MznZ2dGTNmzEBuCQAAAIBfwbDShTfccEOeeOKJ7N69O+9+97tf9rhqtVp6iV/aypUrs3LlyiRJZ2fnK349Xn3u2vvQBb3e7VOuuaDXAwAAgFer4rj0zne+MzfeeGP+9E//NJdeemlGjBiR9evX5+jRo2loaEhfX18aGhryxBNPJEl6e3szbty42vqmpqb09vamt7c3TU1N58z/3zW9vb0ZOnRoRo4cmWPHjpVuGQAAAIBBVvy1uDvuuCPjxo3LxIkT09ramm3btuUDH/hANm3alPnz5ydJ5s+fn/vuuy9JsmnTprS2tqauri4TJkxIpVLJzp0709fXl5MnT2bGjBlJknnz5r1gzfPnmjNnTrZt2zagmwUAAABgcBU/ufRy7rzzzmzYsCELFizI4cOHM3fu3CRJV1dXNmzYkK6urpw+fTqLFi2qvavp1ltvzZo1azJ8+PBs3rw5mzdvTpKsWrUq69evT3d3d44fP57W1tbB3i4AAAAAAzAocek73/lOvvOd7yRJjh8/nuuuu+4lj1uyZEmWLFlyzvyuXbsyZcqUc+afffbZWpwCAAAA4NVnQL8WBwAAAMCvN3EJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoJi4BAAAAUExcAgAAAKCYuAQAAABAMXEJAAAAgGLiEgAAAADFxCUAAAAAiolLAAAAABQTlwAAAAAoNuxibwAAAICL7669D13Q690+5ZoLej3glSMu/ZrwPwoAAADgleBrcQAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKCYuAQAAAFBMXAIAAACgmLgEAAAAQDFxCQAAAIBi4hIAAAAAxcQlAAAAAIqJSwAAAAAUE5cAAAAAKFYcl5qamrJt27b8z//8T77//e/nIx/5SJJk1KhR2bJlSw4cOJAtW7akvr6+tqatrS3d3d3Zv39/mpuba/NTp07Nnj170t3dneXLl9fm6+rq0t7enu7u7mzfvj3jx48v3S4AAAAAr4DiuHT69OncfvvtmTx5cq6++uosWrQokyZNSltbW7Zu3ZorrrgiW7duTVtbW5Jk0qRJaW1tzeTJk9PS0pJ77rknQ4Y8d/kVK1Zk4cKFqVQqqVQqaWlpSZIsWLAg/f39qVQqWbZsWZYuXToItwwAAADAYCmOS319fXn44YeTJD/72c+yb9++NDY2ZtasWVm7dm2SZO3atZk9e3aSZNasWWlvb8+pU6dy6NChHDx4MNOnT09DQ0NGjBiRHTt2JEnWrVv3gjXPn2vjxo2ZOXNm+Z0CAAAAMOgG5Z1L48ePz5VXXpkdO3Zk7Nix6evrS/JcgBo7dmySpLGxMUeOHKmt6enpSWNjYxobG9PT03PO/IvXnDlzJidOnMjo0aPPuf7ChQvT2dmZzs7OjBkzZjBuCQAAAIBfwoDj0pve9KZ87Wtfy2233Zaf/vSn53xerVYHeonzWrlyZaZNm5Zp06blySeffMWvBwAAAMBzBhSXhg0blq997Wv58pe/nG984xtJkqNHj6ahoSFJ0tDQkCeeeCJJ0tvbm3HjxtXWNjU1pbe3N729vWlqajpn/sVrhg4dmpEjR+bYsWMD2TIAAAAAg2hAcWnVqlXZt29fli1bVpvbtGlT5s+fnySZP39+7rvvvtp8a2tr6urqMmHChFQqlezcuTN9fX05efJkZsyYkSSZN2/eC9Y8f645c+Zk27ZtA9kuAAAAAINsWOnCd77znZk3b1727NlTe7H3HXfckTvvvDMbNmzIggULcvjw4cydOzdJ0tXVlQ0bNqSrqyunT5/OokWLcvb/a+/eo6Ku8z+Ov2dAVFBQQ8tAQc37LQwwwdQCBG/rBhTgqhseS7eW2nJTT5dtLc1TbZkS7m6WYtnRMqOzpuTZykpRU1IRzRt5K0tbM0stWcn37w9/M6sIODOCXz9fn49z3kfBOc77dRjefOfNd75z5oyIiNx7772Sn58vDRs2lMLCQiksLBSRs8ur119/XXbv3i1Hjx6VzMzMS80LAAAAAACAWuTzcqmoqEgcDkeV/5aYmFjl559++ml5+umnL/j8559/Lt27d7/g8+Xl5e7lFAAAAAAAAK48tfJucQAAAAAAALg6sVwCAAAAAACAz1guAQAAAAAAwGcslwAAAAAAAOAzlksAAAAAAADwGcslAAAAAAAA+IzlEgAAAAAAAHzGcgkAAAAAAAA+87e6AQAAAAAA6trzpWsv6/1N6N7nst4fYCWWS/+PQQMAAAAAAOA9XhYHAAAAAAAAn7FcAgAAAAAAgM9YLgEAAAAAAMBnLJcAAAAAAADgM5ZLAAAAAAAA8BnLJQAAAAAAAPiM5RIAAAAAAAB8xnIJAAAAAAAAPmO5BAAAAAAAAJ+xXAIAAAAAAIDPWC4BAAAAAADAZyyXAAAAAAAA4DOWSwAAAAAAAPAZyyUAAAAAAAD4jOUSAAAAAAAAfMZyCQAAAAAAAD5juQQAAAAAAACfsVwCAAAAAACAz1guAQAAAAAAwGcslwAAAAAAAOAzlksAAAAAAADwGcslAAAAAAAA+IzlEgAAAAAAAHzGcgkAAAAAAAA+87e6AQAAAAAAcGmeL117We9vQvc+l/X+cGXjzCUAAAAAAAD4jOUSAAAAAAAAfMZyCQAAAAAAAD5juQQAAAAAAACfsVwCAAAAAACAz1guAQAAAAAAwGcslwAAAAAAAOAzlksAAAAAAADwGcslAAAAAAAA+IzlEgAAAAAAAHzGcgkAAAAAAAA+Y7kEAAAAAAAAn7FcAgAAAAAAgM9YLgEAAAAAAMBn/lY3AAAAAAAAUJPnS9de1vub0L3PZb0/03HmEgAAAAAAAHzGcgkAAAAAAAA+Y7kEAAAAAAAAn7FcAgAAAAAAgM9YLgEAAAAAAMBnLJcAAAAAAADgM5ZLAAAAAAAA8Jm/1Q0AAAAAAABczZ4vXXtZ729C9z61+v9x5hIAAAAAAAB8ZsRyKTk5WXbs2CG7d++WSZMmWd0OAAAAAAAA/t8Vv1xyOp2Sl5cngwYNki5dukhWVpZ07tzZ6rYAAAAAAAAgBiyXYmNjpaysTPbu3SunT5+WRYsWyfDhw61uCwAAAAAAACLiEBG1uomapKWlSUpKitx9990iIjJy5Ejp3bu35OTkuG9z9913yz333CMiIh07dpSdO3d6fT+hoaFy5MiR2mn6CkQ+s5HPXHbOJkI+05HPXHbOJkI+05HPXHbOJkI+05HPehEREdKiRYtq/12v5EpLS9M5c+a4Px45cqTm5ubW+v1s2LDB8qx1WeQzu8hnbtk5G/nML/KZW3bORj7zi3zmlp2zkc/8It+VXVf8y+IOHjworVq1cn8cHh4uBw8etLAjAAAAAAAAuFzxy6UNGzZI+/btJTIyUurVqyeZmZnyr3/9y+q2AAAAAAAAICJ+IvJXq5uoiarK7t275Y033pCcnBxZsGCBvPPOO3VyXxs3bqyT//dKQT6zkc9cds4mQj7Tkc9cds4mQj7Tkc9cds4mQj7Tke/KdcVf0BsAAAAAAABXriv+ZXEAAAAAAAC4crFcAgAAAAAAgM9YLgGGqVevntUt1Ck757NzNtiDw+GwuoU6Zfd8AAAAVmG5VElQUJDVLdSp8PBwCQgIkIYNG1rdSp2we774+HgZN26c1W3UGTvns3M2EZE+ffpITEyM+Pn5Wd1KnbB7vqioKBE5+yYadmTnfJGRkVa3UKfsni8kJMTqFuoU+cxl12NpF7vng7lM30codbaGDBmi+fn5Onv2bE1MTNSwsDDLe6rNGjx4sK5Zs0YXLFigY8aMUT8/P3U4HJb3VZf5rO6pNis5OVmPHTumu3fvtl02u+ezczYR0ZSUFD148KD269ePfAZWUlKS7tu3T7t27Wp5L+TzrqKjo/Wrr77SqVOnWt4L+byvQYMG6TvvvKPz58/X/v37W94P+cjnqsGDB+v8+fN10aJFGhcXp/7+/pb3RD7Pq3///jps2DDL+yCf9zVs2DBdtGiRLly4UAcNGqStWrWyvCcvy/IGrojq0aOH7tq1S2NiYjQ7O1tzc3P1hRde0MjISMt7q43q2bOn7ty5U2NjY3XkyJE6f/58y3sin+c1dOhQXbdunXbr1k1ffvllnTFjhuU9kY9sIqJNmzbVDz74QJOTk1VEtGHDhuf9aXrZPV9KSoquX79e+/btqyJiu+WZ3fNFREToihUrdP78+friiy9a3g/5PK8hQ4bopk2btG/fvvrEE0/o4sWLbfULP/KZW8nJybpt2za95ZZbdMqUKfrWW29ps2bNLO+rtmrgwIG2zpeQkKDHjh3T/fv36+9//3vL+yGf59W+fXv9+uuvNS4uTu+//36dOnWqzpgxQzt06GB5b56Wn4j8VSBxcXHSsGFD+fvf/y6bN2+Wrl27Sp8+fSQsLEy2bNkiJ0+etLrFSxIfHy+BgYEye/Zs+e677yQnJ0d69uwpnTp1kqNHj8r3339vdYuXJC4uzrb5goODZcqUKfLMM8/IunXr5Oeff5aoqChZvXq1nDp1yur2LllwcLA8+eSTtsxn52wup06dkujoaHnppZekRYsWsmDBAklISJCBAwfKN998I99++63VLV4Su+ebOXOmVFRUyLRp06Rly5YyadIkGT58uAQEBMiPP/4oJ06csLrFS2LnfE6nUwICAuTWW2+Vl19+WaKioiQ6Olr+85//SOPGjeWHH36wusVL4nA4bJvPz89PHnnkEcnNzZUPP/xQ9u/fLxkZGdKmTRtxOBxSXl5u9GPzasj36KOP2jJfgwYNZNKkSZKfny/Lli2Tjz/+WIYPHy7NmzeXdevWWd3eJWvQoIFMnDhR5s2bJ8uXL7ddPhGR9PR0KSgokJkzZ8r06dPl1KlTUlJSYnVbteaOO+6Qd955x5b5wsLCJDIyUmbMmCGfffaZ/PDDDxIWFib9+vWTrVu3yvHjx61u8aKu+uVSeHi4/PTTT/Lrr7/KiBEj5MyZM1JaWiopKSny/fffS0BAgGzdutXo5YSIyPHjx+WZZ56RLl26SF5enrzyyitSWloq119/vURFRUlRUZH8+uuvVrfps+PHj8v06dNtma+8vFyWL18uO3bsEJGzWR988EGpX7++rF271uLuLk1gYKCcOHFCli1bZrt8YWFhcuTIEdt+7VycTqeMHz9eoqOjZcCAAfLpp5/K8uXLJSAgQNLS0uSjjz6S8vJyq9v0icPhEIfDIePHj5eYmBjb5RMReffddyU7O1sGDhwo2dnZsnHjRjlz5ox07txZGjZsKFu2bLG6RZ8EBQXJ6dOnpaCgQEaPHi3Jycm2yRcWFibHjx8XVZWTJ09K586dpaysTAoKCiQnJ0cmT54sS5cula+//trqVi/ZyZMnpUuXLrJ7925b5VNVee+992Tnzp0SEhIiRUVF8v7778uxY8ckJiZGgoODZfPmzVa36TNVdf9ct1u+oKAgKS8vl6VLl9ru6xceHi5Hjx6VLVu2SElJiZw+fVpUVTp27ChBQUHyySefiMjZn/umXr+uoqJCSktLZcuWLVJRUWG7fCIi69atkwMHDsiePXukrKxM/vKXv8h///tf92Oyfv36xj4nEjmb7+uvv5aysjLb5AsMDJTTp0/L0aNH5c9//rP4+flJcXGxHDp0yP1z/ocffpB9+/ZZ3apHLD99yqpKSUnRX3/9Ve+44w4VEc3KytL169frkiVLtLCwUEVEp0+fbuxr/aOjozU6OlpjYmJURDQyMlJTU1P1tddec98mKSlJX3/9dct79aVCQ0PPOwU5IiJCU1NTz3tJnMn5YmNjNSkpSfv16+f+nNPpVBHRuLg4/fDDD7Vbt26W9+lrJSQk6NSpU7VBgwbuz7m+nqbnc82W9PR022UT+d9siY2NVRHR8PBwfffdd3XNmjXu29xwww06b948rV+/vuX9eluVZ4srX1FRkS3yxcbGamJioiYkJKiIaGBgoK5cuVInT57svs0f/vAHnTt3ruW9+lIJCQn61FNPaePGjVXk7EsY7ZKv8nGLiOhTTz2lt99+u0ZHR+uBAwd0xYoV+txzz1neqy/lmi033XST+3NPPvmkpqWl2SLfubPl3Blz4403uv8+btw4Ix+bIv87bql8/SG75HMdt1R+WbQd8rlmS1pa2gX/duedd+q0adNURPS3v/2tJiUlWd6vt+WaLVFRUbbMV/m45dxKTEzUTZs26eDBgzUpKUlHjx5t3Es4Xcctt956q+3yVT5mGThwoM6bN++8n/OTJk3SV1991fJePSzLG7CkkpOTtaioSGfPnq2zZ8/WwMBAFRFt1qyZtm/f3n1ht5ycHJ0wYYLl/XpbSUlJumvXLn344Ye1pKRE//SnP6nI2eVEQUGBZmVlqYhoenq6rlixQoODgy3v2ZsaPny4rly5UkeMGOFeuIicvZ5GQUGBZmZmGp0vOTlZv/jiC3300Uf1+PHj5/2gczqdes011+isWbN0+PDhlvfqS6WkpGhxcfF5izO75HPNlry8PM3Ly9OmTZue9+8mZxO5cLY8+OCDGhAQoP369dODBw+6n/RlZGToJ598Ytx1DKqaLU6nU/v376/ffPONPvvss0bnO3e2nDhxQgcNGqQiZ2fnuRc0HTlypM6bN08DAgIs79mbqjxbXAeY/v7+Wq9ePaPzVZ4trsdep06dtKCgQL/99ltNTU3VVq1aaW5urrZo0cLynr2p6o5bOnbsaIt8Vc2Wqq4B9rvf/U7z8/ONW1xXni2Vj1tMz1fVcYtrvpie79znRFUdt2RmZur06dM1PT1dd+zYoW3btrW8Z2+q8mx54IEHzvt5Z3q+qmZL5eVKt27d9KefftKjR49qp06dLO/Zm6pptrhympqvqrnStGlTHTVqlL722muak5OjIsYds1jewGWv3r17a0lJicbFxem1116rhYWF2r59+wtud//99+uWLVu0S5culvfsTQUHB+uKFSt08ODBKnJ223vq1Cl99NFH1c/PT0eMGKFFRUW6ePFi3b59u3HvoNO6dWvdunWrLlq0SKdOnaoZGRnn/WDPysrSVatWGZuve/fuum3bNvdv/h5++OEq3y1g4sSJunnz5vOym1AdOnTQX375RUeMGKEios2bN9eIiIgLvk4m5qs8W5YvX+6eLef+oDcxm0j1s2XixInqdDq1Xbt2+vHHH2teXp6WlJQY9713sdnStm1b/eijj4zNV91siYiIOO9248eP1w0bNhiXr7rZ0qNHD+Pz1TRbwsPDdcaMGe4z0fz8/Ix6YitS/WyZNGmStmnTRmfMmOF+QmFivovNFleNGzdOi4uLjXpsinh+3GJqPk+PW0zMV9NscT1GExMT9bvvvtOVK1dq586dLe/Zm6putkyePNk9RwYOHGhsPk9nS0ZGhh44cMC4fJ7OFhPzVTdX2rVrpyKit912m27cuFEXL16s+/btu+BY5gouyxu47DVo0CDt2bOn++NZs2bpBx98cN4W2+l06ksvvWTUD4hz64UXXtDbbrvN/YT2tdde0w0bNuioUaNU5OwD+vbbb9fWrVtb3qu3FRQUpDExMdq0aVO999579cUXX9SMjAz3b6UbN26s7du319TUVCPzde7c2f34bNmypf700086b9483bhxo44bN+6821577bWW9+tttWzZUp977jmdOXOm9unTR//973/rnDlz9NChQzp+/Hij83kyW0zN5qrKs+X111/X4uJivfvuu1Xk7OwMDg6+4DefJlRVsyUzM/O8M16cTqeGhIQYmc+T2XL99dfrW2+9ZeTLNj2ZLabmq262uB6brrOvTXopQOWqbrZkZGS4b2Nqvosdt9SvX187dOigixcvNu6xKeLZbOnYsaOx+TyZLabm8+S4pV27dlpSUmLcL9tdVd1zorFjxxqf72Kzxd/fXx0Ohz7++ONG5vNktjgcDn3ssceMy1fVXHnllVf08OHD7mNqf39/bdu2rTZv3tzyfr0oyxuwrFynI4eGhurcuXPdp6TZ4a2K//rXv+rcuXP1gQce0NzcXJ01a5bGxsZqUVGRhoWFWd7fpZbrtw3169d3D1PXS/1MXChVVU6nU7OysnTMmDEqItqrVy89duyY3nzzzZb3dqkVHh6u06ZN0/Lycv3jH/+oIqI33XSTHjlyRHv37m15f5da1c0WU58YnVvVzZbVq1fb4nuvqtniepltmzZtLO+vNqq62eL63jPktOsqq6bZ4rpGmMn5Ks8W129zTTsLsqqqarb07t1bV69ereHh4Zb3d6lV02xx5Tv3GoQmVnWzxfW9Z3K+mmZLdHS08fmqO25xOp3q5+enjRo1srxHb8t1zFXTcUt4eLjWq1fPyHyuqmm2REZGWt5fbVR1syUuLs7y3i6lqpor0dHReuTIEY2Pj7e8Px/L8gYua1V1AFavXj3Ny8vTWbNmWd7fpda5i7Hx48frQw89pNOnT3cPnn/+85/GXX/oYhUYGKj33nuvTpkyRd944w09ePCgNmnSxPK+aqNcF410/YCcOXOm+wLtpldYWJgmJyef97nc3Fz3QZodyk6z5dzFmJ1mi+uMj+r+zTVbFixYYORsqS5fVbPF9QTQpKoqX1Wz5aWXXjJuttT02LTDbKkqn51mS1hY2HlnPVbObvpxS3X57DJbqspnl9lS02PTDrOlqnx2mi01VVXHLSaeaV1V2fU5kQ2fD1neQJ1XfHy8jhw50v1xVWcPhIaG6t69e92vyTWpKuer6iU4IqJ33XWXrlmzxrRT63To0KF6//33X/R27777ru7Zs8ek16R6lS8zM1M3bdp0weuMr/SqKd+5y9CsrCzdvHmzUfk8+dqZPFsq56vunVQcsgAACVdJREFU7AhTZ8tvfvMbzc3NvWjfps4WT/OZOltqymf6bPHka2fybKmcz26zZeDAgbpmzRr32Y7VnbVq6mzxNJ9rtph2VmvlfJXfOMb1dxNniydfO5NnS+V81S3RTJ0tvXv31pEjR2rv3r2rzSZi7mzxNJ+Jxy01ZTN9rlQqyxuos3I4HBoUFKRbt27Vbdu2nXe9mnO/iK6/jx8/Xq+77jrL+66NfK7X2IqcPU0yISFBy8rKtHv37pb37U0lJSXppk2bNDExsdrbOJ1O7dChgx4+fNiW+Ro3bqyjRo0y8uLynuSrV6+epqena2lpqVH5PMlm6my5WD47zJZ+/frp9u3ba8xn8mzxJJ/Js8WTfKbOFk+ymTxbasrnymXybHHNzr179+qLL75Y5W1Mni2e5DN5tniSz9TZ4kk2k2eLJ/lMni3Dhg3TkpISzc/P18WLF+sNN9zg/jc7HLd4ks/U2VJTNleZOleqKMsbqPN6+OGH9aGHHtL58+e739q2qqppQ3ollyf5QkJCjLuAcJ8+ffTQoUPuUx6Dg4O1devWGhgYWOV1sUJDQy3vua7yDRgwwLjXTXuT7+abb77gHauu5PL2sWnabPEmn4mzRUT0wQcf1AkTJqjI2YsqJiYmamxsrIaEhFxwW9Nmizf5TJwt3uQzbbZ4k03EvNniTT4TZ0tCQoLu3r1bu3Tpov7+/rpixQq95ZZbqr29abPFm3wmzhZv8pk2W7x9bJo2W7zJZ+Jsadasmb7//vvuN5p69dVXNT09XZs3b17ltb5Mmy3e5DNttniTzbS5UlX5y1WgoqJCWrduLfPnz5exY8fK888/L+Xl5fLII49IXFychISESGFhoZw+fdrqVn1SU774+Hhp1KiRrFixQn788UerW/XK999/L6dPn5aWLVtKs2bN5O2335ZffvlFTpw4IYWFhZKfny/R0dHSvHlzKSwslCNHjljdslc8yRcTEyONGzeWjz76yOp2veZpvuDgYPnwww+tbtcr3j42TZstnua75pprjJwtImfnZkBAgIiIvP3227J//36pqKgQh8Mh2dnZ0qVLFwkLCzNytohcPF+3bt3kmmuuMe57z+Vi+bp27SqhoaFG5vPmsWnabBHx7Gt33XXXGTlb/Pz8ZPTo0fLFF19ISEiI7Ny5U7p27SqrVq1y3yYmJkZCQ0ONnC2e5IuNjZVGjRoZedziab7GjRsbN1u8fWyaNls8zdesWTMjZ0tFRYU0bNhQOnXqJAcOHJABAwZI8+bN5fbbb5c9e/bI448/LlFRUXLdddcZOVs8yderVy9p0qSJcbPF02xNmzY1bq5Ux/INV11X27ZtddKkSSoi+tBDD+nPP/+seXl5KiKalpZm3GmfV1O+Hj166JdffqlfffWVjh07Vh0Oh2ZnZ+vChQv1+uuv1zvuuMP2+Vq2bGl5n+TzLZvdH5sm5+vWrZvu2LFDFy5cqHfddZeKnH03uH/84x86ZMiQqyKfqd97ds/HY9Pcr52rXC/hSE5O1m+//db99vT+/v6amZlp9NePfGbns3M2u+dLS0vT4uJiXbt2rT722GMqInrrrbdqfn6+RkdHa3p6uu3zmfqzwc7ZqijLG6jzatmypc6dO1fHjh2ru3bt0scff1yXLl2qo0aNsrw38l28OnfurPfdd995nyssLNQOHTpY3hv5ru58ds52NeQbOnSo7tmzR6dMmeL+3Jw5czQ1NdXy3sh3deezc7arId+5NWXKFJ08eXK1F702vchnbtk5m13zNWnSRJ999lkdMmSI+3NLliyp8fqfJpWd89k5W6WyvIHLUlOmTNH9+/fr0KFDVeTs6zXDw8Mt74t83ldqaqoWFxcb93pp8tk/n52z2TGfn5+fjho1Sr/88ksdM2aMjhkzRjds2KDt2rWzvDfyXd357Jztash3bqWmpuqqVauqvB6fHYp85pads9k5X0pKis6dO1eTkpJ02LBh+vnnnxt/nZ6rJZ+ds51TljdwWSo8PFx79erl/thOW+yrIZ+rsrOzddu2baZfRZ98Nsxn52x2zxcVFaXTpk3Tv/3tb+5T6O1U5DO37JztasjnqjfffNOOTyDIZ4Oycza75gsJCdGcnBz9+OOP9f3339cePXpY3hP5yOYqx///BTBC//795dChQ7Jz506rW6kT5DOXnbOJ2D8fAACAKRo1aiQOh0OOHz9udSt1ws757JyN5RIAAAAAAAB85rS6AQAAAAAAAJiL5RIAAAAAAAB8xnIJAAAAAAAAPmO5BAAAAAAAAJ+xXAIAAAAAAIDPWC4BAABcgtp6O+H+/fvL0qVLa+X/AgAAuJxYLgEAABjK4XBY3QIAAADLJQAAgNry7LPPSmlpqWzZskXuvPNOETm7AMrLy5OtW7fK0qVLZdmyZZKWliYiIsnJybJ9+3ZZtWqVpKamuv+fJ554QiZMmOD+uLS0VCIiIiQiIkK++OILycvLk40bN0qrVq0ub0AAAIAq+FvdAAAAgB2kpqbKjTfeKD179pTQ0FDZsGGDfPrppxIfHy+RkZHSvXt3adGihWzfvl3mzp0r9evXlzlz5shtt90mZWVl8uabb3p0Px07dpTs7Gy577776jgRAACAZzhzCQAAoBb07dtXFi5cKGfOnJHvvvtOPvnkE4mJiZG+ffvK4sWLRVXl8OHDsnLlShER6dSpk+zdu1fKyspERGTBggUe3c/+/fvls88+q7McAAAA3mK5BAAAcIWpqKgQp/N/h2kNGjRw//3kyZNWtAQAAFAtlksAAAC1YNWqVZKRkSFOp1NCQ0OlX79+sn79eikqKpK0tDRxOBzSokULGTBggIiI7NixQyIjI6Vt27YiIpKVleX+v/bt2ye9evUSEZGoqChp06bNZc8DAADgKa65BAAAUAsKCgqkT58+UlJSIqoqEydOlMOHD8uSJUskISFBtm7dKjt37pTPPvtMfvzxRykvL5d77rlHli1bJkeOHJHVq1dLt27dRERkyZIlMnr0aNm4caMUFxfLrl27LE4HAABQPYeIqNVNAAAA2FlQUJCcPHlSmjVrJuvXr5f4+Hg5fPiw1W0BAADUCs5cAgAAqGPvvfeeNGnSRAICAuSpp55isQQAAGyFM5cAAAAAAADgMy7oDQAAAAAAAJ+xXAIAAAAAAIDPWC4BAAAAAADAZyyXAAAAAAAA4DOWSwAAAAAAAPDZ/wFzo3cJse6QMAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "values,counts = taxi \\\n",
    "    .selectExpr('log(trip_duration)')\\\n",
    "    .rdd\\\n",
    "    .flatMap(lambda x:x)\\\n",
    "    .histogram(20)\n",
    "\n",
    "labels = [format(v,'.2f') for v in values]\n",
    "\n",
    "plt.clf()\n",
    "ax = pd.DataFrame(list(zip(labels,counts)),columns=['logdur','Frequency'])\\\n",
    "    .set_index('logdur')\\\n",
    "    .plot(kind='bar', figsize=(20,10), align='edge', rot=45)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Engineering, Cont."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - <font size=\"3\">Rename the columns for convenience<font>\n",
    " - <font size=\"3\">Convert location data to double values<font>\n",
    " - <font size=\"3\">Filter the data by the newly added meters per second column<font> \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- pLon: double (nullable = true)\n",
      " |-- pLat: double (nullable = true)\n",
      " |-- dLon: double (nullable = true)\n",
      " |-- dLat: double (nullable = true)\n",
      " |-- duration: double (nullable = true)\n",
      " |-- hour: double (nullable = true)\n",
      " |-- hourCos: double (nullable = true)\n",
      " |-- hourSin: double (nullable = true)\n",
      " |-- meters: double (nullable = true)\n",
      " |-- mps: double (nullable = true)\n",
      "\n",
      "+------------------+------------------+------------------+------------------+--------+----+--------------------+-------------------+------------------+------------------+\n",
      "|              pLon|              pLat|              dLon|              dLat|duration|hour|             hourCos|            hourSin|            meters|               mps|\n",
      "+------------------+------------------+------------------+------------------+--------+----+--------------------+-------------------+------------------+------------------+\n",
      "| -73.9821548461914| 40.76793670654297|-73.96463012695312|40.765602111816406|   455.0|17.0|-0.06824241336467135|-0.9976687691905392|1498.5207796474772|3.2934522629614884|\n",
      "|-73.98041534423828|40.738563537597656|-73.99948120117188| 40.73115158081055|   663.0| 0.0|                 1.0|                0.0|1805.5071687958239|2.7232385653029016|\n",
      "| -73.9790267944336|40.763938903808594|-74.00533294677734|40.710086822509766|  2124.0|11.0| -0.9906859460363306| 0.1361666490962471| 6385.098495252941| 3.006166899836601|\n",
      "|-74.01004028320312|   40.719970703125|-74.01226806640625| 40.70671844482422|   429.0|19.0|  0.4600650377311516|-0.8878852184023756| 1485.498422771006|3.4627002861795013|\n",
      "|-73.97305297851562|40.793209075927734| -73.9729232788086| 40.78252029418945|   435.0|13.0| -0.9172113015054532|-0.3984010898462412|1188.5884593342212|  2.73238726283729|\n",
      "+------------------+------------------+------------------+------------------+--------+----+--------------------+-------------------+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import monotonically_increasing_id\n",
    "\n",
    "newNames = ['pLon', 'pLat', 'dLon', 'dLat', 'duration', 'hour', 'hourCos', 'hourSin', 'meters']\n",
    "taxi = taxi.toDF(*newNames)\n",
    "\n",
    "taxi = taxi.withColumn(\"pLon\", col(\"pLon\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"pLat\", col(\"pLat\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"dLon\", col(\"dLon\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"dLat\", col(\"dLat\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"duration\", col(\"duration\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"hour\", col(\"hour\").cast(DoubleType())) \\\n",
    "                    .withColumn(\"mps\", col(\"meters\")/col(\"duration\")) \\\n",
    "                    .filter(\"mps BETWEEN 0.5 AND 20\")\n",
    "taxi.printSchema()\n",
    "taxi.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - <font size =\"3\">We will augment the kaggle taxi data with NYC Zipcode data for dropoff and pickup locations. The goal is to add region-level data (less granularity) for the model to train on.<font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load in NYC Zipcode Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <font size=\"3\">Load in WKT data using BDT SourceGeneric<font>\n",
    "- <font size=\"3\">ST_FromText will convert the WKT values into Shape Structs, the internal geometry objects of BDT<font>   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Elapsed: 0.5422 seconds\n",
      "Record count: 246\n",
      "+-----+--------+--------+--------------------+\n",
      "|   ID|    NAME| RG_NAME|           __SHAPE__|\n",
      "+-----+--------+--------+--------------------+\n",
      "|10001|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0002\u0000...|\n",
      "|10002|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10003|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10004|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0002\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10005|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10006|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10007|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10009|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u000e\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10010|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0003\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "|10011|New York|New York|[\u0001\u0006\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\u0003\u0000\u0000\u0000\u0001\u0000...|\n",
      "+-----+--------+--------+--------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "countiesWKTPath = os.path.join(\"C:\", os.sep, \"Users\", \"bdt2\", \"mnt\", \"data\", \"DevSummit\", \"zipcodes\", \"zip.wkt\")\n",
    "\n",
    "formaty = \"csv\"\n",
    "options = {\n",
    "   \"path\": countiesWKTPath,\n",
    "   \"header\": \"true\",\n",
    "   \"delimiter\": \"\\t\",\n",
    "   \"inferSchema\": \"true\"}\n",
    "\n",
    "startTime = time.perf_counter()\n",
    "zipcode = spark.bdt.sourceGeneric(formaty, options, selectExpr=[\"ID\", \"NAME\", \"RG_NAME\", \"ST_FromText(wkt) as __SHAPE__\"], geometryType = \"Polygon\", wkid=4326, shapeField = \"__SHAPE__\")\n",
    "timeElapsed(startTime)\n",
    "\n",
    "zipcode = zipcode.where(\"RG_NAME == 'New York'\")\n",
    "print(f\"Record count: {zipcode.count():,}\")\n",
    "zipcode.show(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Joining the Taxi Data with the Zipcode Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <font size=\"3\">We will use ProcessorAddShapeFromXY to add a point Shape Struct to the taxi data<font>\n",
    "- <font size=\"3\">We will use ProcessorPointInPolygon twice: First to enrich each taxi record with the NYC Zipcode information of the dropoff location, and then the pickup location<font>\n",
    "- <font size=\"3\">Uncessary intermediate columns are removed, and column data types are changed<font>\n",
    "- <font size=\"3\">Three new columns are added: <font>\n",
    "    - <font size=\"3\">pRegionID: ID of pickup Zipcode <font>\n",
    "    - <font size=\"3\">dRegionID: ID of dropoff Zipcode <font>\n",
    "    - <font size=\"3\">oneRegion: Whether the entire taxi trip was in one region or not <font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------+------------------+------------------+------------------+--------+----+-------------------+-------------------+-----------------+------------------+---------+\n",
      "|              pLon|              pLat|              dLon|              dLat|duration|hour|            hourCos|            hourSin|           meters|               mps|pRegionID|\n",
      "+------------------+------------------+------------------+------------------+--------+----+-------------------+-------------------+-----------------+------------------+---------+\n",
      "|-74.01004028320312|   40.719970703125|-74.01226806640625| 40.70671844482422|   429.0|19.0| 0.4600650377311516|-0.8878852184023756|1485.498422771006|3.4627002861795013|    10013|\n",
      "|-74.00398254394531|  40.7130126953125|-73.97919464111328| 40.74992370605469|  1317.0| 9.0|-0.7757112907044199| 0.6310879443260526|4605.201075572882| 3.496735820480548|    10007|\n",
      "|-74.00550079345703| 40.70658874511719| -73.9858169555664| 40.74420166015625|  1022.0|19.0| 0.4600650377311516|-0.8878852184023756|4499.284303994801| 4.402430825826616|    10038|\n",
      "|-74.00899505615234|40.712589263916016| -73.9986343383789|40.722808837890625|   707.0|12.0|-0.9906859460363308|-0.1361666490962464|1433.099593298032|2.0270149834484186|    10007|\n",
      "|-74.01632690429688| 40.70499038696289|-73.98210906982422| 40.77375030517578|  1478.0|15.0|-0.5766803221148672| -0.816969893010442|8171.202963147671| 5.528554102265001|    10004|\n",
      "+------------------+------------------+------------------+------------------+--------+----+-------------------+-------------------+-----------------+------------------+---------+\n",
      "only showing top 5 rows\n",
      "\n",
      "Time Elapsed: 6.0880 seconds\n"
     ]
    }
   ],
   "source": [
    "startTime = time.perf_counter()\n",
    "\n",
    "taxiShp = spark.bdt.processorAddShapeFromXY(taxi, \"pLon\", \"pLat\", keep=True)\n",
    "\n",
    "pipped = spark.bdt.processorPointInPolygon(taxiShp, zipcode, cellSize=1.0, emitPippedOnly=True, pointShapeField=\"__SHAPE__\", polygonShapeField=\"__SHAPE__\")\n",
    "\n",
    "taxi = pipped.withColumnRenamed(\"ID\", \"pRegionID\") \\\n",
    "                 .drop(\"RG_NAME\") \\\n",
    "                 .drop(\"NAME\") \\\n",
    "                 .drop(\"__SHAPE__\")\n",
    "taxi.show(5)\n",
    "timeElapsed(startTime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+\n",
      "|duration|              pLon|             pLat|              dLon|              dLat|            hourCos|            hourSin|            meters|pRegionID|dRegionID|oneRegion|\n",
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+\n",
      "|   429.0|-74.01004028320312|  40.719970703125|-74.01226806640625| 40.70671844482422| 0.4600650377311516|-0.8878852184023756| 1485.498422771006|    10013|    10004|    false|\n",
      "|   512.0|-74.00180053710938|40.72613525390625|-74.00715637207031|40.740806579589844|-0.9906859460363308|-0.1361666490962464| 1692.642538820824|    10012|    10014|    false|\n",
      "|   174.0| -74.0063247680664|40.73339080810547|-74.00000762939453|40.743255615234375|-0.3348796121709864|-0.9422609221188204|1219.2200067751526|    10014|    10011|    false|\n",
      "|   215.0|-74.00298309326172|40.72331237792969|-74.00155639648438|40.728824615478516| 0.6825531432186536|-0.7308359642781246| 624.6123563331846|    10013|    10012|    false|\n",
      "|   674.0|-74.01116943359375|40.70360565185547|-74.00569152832031| 40.74135971069336|                1.0|                0.0|4223.3651561629185|    10004|    10014|    false|\n",
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+\n",
      "only showing top 5 rows\n",
      "\n",
      "Time Elapsed: 18.8419 seconds\n"
     ]
    }
   ],
   "source": [
    "startTime = time.perf_counter()\n",
    "\n",
    "taxiShp2 = spark.bdt.processorAddShapeFromXY(taxi, \"dLon\", \"dLat\", keep=True)\n",
    "\n",
    "pipped2 = spark.bdt.processorPointInPolygon(taxiShp2, zipcode, cellSize=1.0, emitPippedOnly=True, pointShapeField=\"__SHAPE__\", polygonShapeField=\"__SHAPE__\")\n",
    "\n",
    "taxi = pipped2.withColumnRenamed(\"ID\", \"dRegionID\") \\\n",
    "                 .drop(\"RG_NAME\") \\\n",
    "                 .drop(\"NAME\") \\\n",
    "                 .drop(\"__SHAPE__\") \\\n",
    "                 .withColumn(\"oneRegion\", when(col(\"pRegionID\") == col(\"dRegionID\"), \"true\").otherwise(\"false\")) \\\n",
    "                 .withColumn(\"pRegionID\", col(\"pRegionID\").cast(StringType())) \\\n",
    "                 .withColumn(\"dRegionID\", col(\"dRegionID\").cast(StringType())) \\\n",
    "                 .select(*[\"duration\", \"pLon\", \"pLat\", \"dLon\", \"dLat\", \"hourCos\", \"hourSin\", \"meters\", \"pRegionID\", \"dRegionID\", \"oneRegion\"])\n",
    "\n",
    "taxi.show(5)\n",
    "timeElapsed(startTime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">Use BDT ProcessorWebMercator to convert WGS84 Lat/Lon values to WebMercator X/Y values in meters<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- duration: double (nullable = true)\n",
      " |-- pLon: double (nullable = true)\n",
      " |-- pLat: double (nullable = true)\n",
      " |-- dLon: double (nullable = true)\n",
      " |-- dLat: double (nullable = true)\n",
      " |-- hourCos: double (nullable = true)\n",
      " |-- hourSin: double (nullable = true)\n",
      " |-- meters: double (nullable = true)\n",
      " |-- pRegionID: string (nullable = true)\n",
      " |-- dRegionID: string (nullable = true)\n",
      " |-- oneRegion: string (nullable = false)\n",
      " |-- SHAPE@X: double (nullable = true)\n",
      " |-- SHAPE@Y: double (nullable = true)\n",
      "\n",
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+------------------+------------------+\n",
      "|duration|              pLon|             pLat|              dLon|              dLat|            hourCos|            hourSin|            meters|pRegionID|dRegionID|oneRegion|           SHAPE@X|           SHAPE@Y|\n",
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+------------------+------------------+\n",
      "|   429.0|-74.01004028320312|  40.719970703125|-74.01226806640625| 40.70671844482422| 0.4600650377311516|-0.8878852184023756| 1485.498422771006|    10013|    10004|    false|-8238759.997915836| 4971124.737422713|\n",
      "|   512.0|-74.00180053710938|40.72613525390625|-74.00715637207031|40.740806579589844|-0.9906859460363308|-0.1361666490962464| 1692.642538820824|    10012|    10014|    false|-8237842.753576414|4972030.2132059205|\n",
      "|   174.0| -74.0063247680664|40.73339080810547|-74.00000762939453|40.743255615234375|-0.3348796121709864|-0.9422609221188204|1219.2200067751526|    10014|    10011|    false|-8238346.388662782| 4973096.047732088|\n",
      "|   215.0|-74.00298309326172|40.72331237792969|-74.00155639648438|40.728824615478516| 0.6825531432186536|-0.7308359642781246| 624.6123563331846|    10013|    10012|    false|-8237974.395125127| 4971615.566609255|\n",
      "|   674.0|-74.01116943359375|40.70360565185547|-74.00569152832031| 40.74135971069336|                1.0|                0.0|4223.3651561629185|    10004|    10014|    false| -8238885.69436235| 4968721.374869845|\n",
      "+--------+------------------+-----------------+------------------+------------------+-------------------+-------------------+------------------+---------+---------+---------+------------------+------------------+\n",
      "only showing top 5 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "taxi = spark.bdt.processorWebMercator(taxi, \"pLat\", \"pLon\") \\\n",
    "                   .withColumnRenamed(\"X\", \"SHAPE@X\") \\\n",
    "                    .withColumnRenamed(\"Y\", \"SHAPE@Y\") \n",
    "taxi.printSchema()\n",
    "taxi.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"3\">Split the data into Train and Test datasets. The validation dataset will be taken from the Train dataset in the model creation step.<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "taxiTrain, taxiTest = taxi.randomSplit([0.85, 0.15])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Put all of the data from Train and Test Pyspark Dataframes into Feature Classes\n",
    "- <font size=\"3\">Collect all the dataframe rows and insert them into a new Feature Class<font>\n",
    "- <font size=\"3\">Table Views for both the Train and Test Feature Classes will be automatically created in the environment<font>\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Elapsed: 53.8707 seconds\n"
     ]
    }
   ],
   "source": [
    "trainRows = taxiTrain.collect()\n",
    "\n",
    "# The Scratch GDB environment setting is the location of a file geodatabase you can use to write temporary data.\n",
    "# C:\\Users\\<user_name>\\AppData\\Local\\Temp\\1\\scratch.gdb.\n",
    "startTime = time.perf_counter()\n",
    "ws = arcpy.env.scratchGDB  # Forest GP cannot accept memory based ws :-(\n",
    "nm = \"Train\"\n",
    "\n",
    "\n",
    "fields = [f.name for f in taxiTrain.schema.fields]\n",
    "fieldsTypes = [(f.name, f.dataType.simpleString().upper()) for f in taxiTrain.schema.fields]\n",
    "\n",
    "fc = os.path.join(ws, nm)\n",
    "\n",
    "arcpy.management.Delete(fc)\n",
    "\n",
    "sp_ref = arcpy.SpatialReference(3857)\n",
    "arcpy.management.CreateFeatureclass(ws, nm, \"POINT\", spatial_reference=sp_ref)\n",
    "for name, dType in fieldsTypes[:-2]:\n",
    "    arcpy.management.AddField(fc, name, dType)\n",
    "\n",
    "with arcpy.da.InsertCursor(fc, fields) as cursor:\n",
    "    for row in trainRows:\n",
    "        cursor.insertRow(row)\n",
    "timeElapsed(startTime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Elapsed: 14.2621 seconds\n"
     ]
    }
   ],
   "source": [
    "testRows = taxiTest.collect()\n",
    "\n",
    "# The Scratch GDB environment setting is the location of a file geodatabase you can use to write temporary data.\n",
    "# C:\\Users\\<user_name>\\AppData\\Local\\Temp\\1\\scratch.gdb.\n",
    "startTime = time.perf_counter()\n",
    "ws = arcpy.env.scratchGDB  # Forest GP cannot accept memory based ws :-(\n",
    "nm = \"Test\"\n",
    "\n",
    "\n",
    "fields = [f.name for f in taxiTest.schema.fields]\n",
    "fieldsTypes = [(f.name, f.dataType.simpleString().upper()) for f in taxiTest.schema.fields]\n",
    "\n",
    "fc = os.path.join(ws, nm)\n",
    "\n",
    "arcpy.management.Delete(fc)\n",
    "\n",
    "sp_ref = arcpy.SpatialReference(3857)\n",
    "arcpy.management.CreateFeatureclass(ws, nm, \"POINT\", spatial_reference=sp_ref)\n",
    "for name, dType in fieldsTypes[:-2]:\n",
    "    arcpy.management.AddField(fc, name, dType)\n",
    "\n",
    "with arcpy.da.InsertCursor(fc, fields) as cursor:\n",
    "    for row in testRows:\n",
    "        cursor.insertRow(row)\n",
    "timeElapsed(startTime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Forest Based Regression on the Feature Class\n",
    "- <font size=\"3\">Details about the Random Forest tool can be found [here](https://pro.arcgis.com/en/pro-app/latest/tool-reference/geoanalytics-desktop/forest-based-classification-and-regression.htm)<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time Elapsed: 2.2869 minutes\n",
      "None\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<h2>Output</h2><table><tr><th>id</th><th>value</th></tr><tr><td>0</td><td>C:\\Users\\bdt2\\Documents\\ArcGIS\\Projects\\DevSummit\\scratch.gdb\\Trained</td></tr><tr><td>1</td><td>C:\\Users\\bdt2\\Documents\\ArcGIS\\Projects\\DevSummit\\scratch.gdb\\Importance</td></tr><tr><td>2</td><td>C:\\Users\\bdt2\\Documents\\ArcGIS\\Projects\\DevSummit\\scratch.gdb\\Predictions</td></tr></table><h2>Messages</h2>Start Time: Friday, February 12, 2021 8:48:02 PM<br/>Started Spark Job 1 with 233 tasks.<br/>Spark Job 1 completed in 12.8 seconds.<br/>Started Spark Job 2 with 64 tasks.<br/>Spark Job 2 completed in 2.4 seconds.<br/>Started Spark Job 3 with 1 tasks.<br/>Spark Job 3 completed in 0.4 seconds.<br/>Started Spark Job 4 with 32 tasks.<br/>Spark Job 4 completed in 2.2 seconds.<br/>Started Spark Job 5 with 39 tasks.<br/>Spark Job 5 completed in 2.7 seconds.<br/>Started Spark Job 6 with 64 tasks.<br/>Spark Job 6 completed in 5.5 seconds.<br/>Started Spark Job 7 with 64 tasks.<br/>Spark Job 7 completed in 3.4 seconds.<br/>Started Spark Job 8 with 64 tasks.<br/>Spark Job 8 completed in 2.3 seconds.<br/>Started Spark Job 9 with 64 tasks.<br/>Spark Job 9 completed in 3.4 seconds.<br/>Started Spark Job 10 with 64 tasks.<br/>Spark Job 10 completed in 3.7 seconds.<br/>Started Spark Job 11 with 1 tasks.<br/>Spark Job 11 completed in 0.3 seconds.<br/>Started Spark Job 12 with 37 tasks.<br/>Spark Job 12 completed in 5.5 seconds.<br/>Started Spark Job 13 with 32 tasks.<br/>Spark Job 13 completed in 4.4 seconds.<br/>Started Spark Job 14 with 33 tasks.<br/>Spark Job 14 completed in 2.1 seconds.<br/>Started Spark Job 15 with 37 tasks.<br/>Spark Job 15 completed in 2.3 seconds.<br/>Started Spark Job 16 with 32 tasks.<br/>Spark Job 16 completed in 2.2 seconds.<br/>Started Spark Job 17 with 65 tasks.<br/>Spark Job 17 completed in 6.1 seconds.<br/>Started Spark Job 18 with 33 tasks.<br/>Spark Job 18 completed in 1.3 seconds.<br/>Model Characteristics<br/>Number of Trees: 200<br/>Leaf Size: 5<br/>Tree Depth: 5<br/>% of Training Available per Tree: 100<br/>Number of Randomly Sampled Variables: 3<br/>% of Training Data Excluded for Validation: 18<br/>Training Data: Regression Diagnostics<br/>metricName, score<br/>R-Squared, 0.68<br/>Mean Squared Error, 131763.58<br/>Root Mean Squared Error, 362.99<br/>Mean Absolute Error, 253.48<br/>Explained Variance, 218400.23<br/>*Predictions for the data used to train the model compared to the observed categories for those features<br/>Validation Data: Regression Diagnostics<br/>metricName, score<br/>R-Squared, 0.68<br/>Mean Squared Error, 132324.47<br/>Root Mean Squared Error, 363.76<br/>Mean Absolute Error, 253.28<br/>Explained Variance, 222154.22<br/>*Predictions for the validation data (excluded from model training) compared to the observed values for those test features<br/>Started Spark Job 19 with 64 tasks.<br/>Spark Job 19 completed in 5.3 seconds.<br/>Top Variable of Importance<br/>Variable, Importance, Percentage<br/>hourSin, 0.01<br/>oneRegion.false, 0.02<br/>hourCos, 0.03<br/>dLat, 0.04<br/>pLat, 0.05<br/>dLon, 0.07<br/>pLon, 0.14<br/>meters, 0.63<br/>Started Spark Job 20 with 2 tasks.<br/>Spark Job 20 completed in 0.0 seconds.<br/>Started Spark Job 21 with 32 tasks.<br/>Spark Job 21 completed in 1.3 seconds.<br/>------------------------------------------------------------<br/>-------------------Model Characteristics--------------------<br/>Number of Trees                                          200<br/>Leaf Size                                                  5<br/>Tree Depth                                                 5<br/>% of Training Available per Tree                         100<br/>Number of Randomly Sampled Variables                       3<br/>% of Training Data Excluded for Validation                18<br/>                                                                                                <br/>-----------Training Data: Regression Diagnostics------------<br/>metricName                         score<br/>R-Squared                           0.68<br/>Mean Squared Error             131763.58<br/>Root Mean Squared Error           362.99<br/>Mean Absolute Error               253.48<br/>Explained Variance             218400.23<br/>*Predictions for the data used to train the model compared to the observed categories for those features<br/>                                                                                                <br/>----------Validation Data: Regression Diagnostics-----------<br/>metricName                         score<br/>R-Squared                           0.68<br/>Mean Squared Error             132324.47<br/>Root Mean Squared Error           363.76<br/>Mean Absolute Error               253.28<br/>Explained Variance             222154.22<br/>*Predictions for the validation data (excluded from model training) compared to the observed values for those test features<br/>                                                                                                <br/>-----------------Top Variable of Importance-----------------<br/>Variable                                           Importance Percentage<br/>hourSin                                                      0.01<br/>oneRegion.false                                              0.02<br/>hourCos                                                      0.03<br/>dLat                                                         0.04<br/>pLat                                                         0.05<br/>dLon                                                         0.07<br/>pLon                                                         0.14<br/>meters                                                       0.63<br/><br/>Succeeded at Friday, February 12, 2021 8:50:14 PM (Elapsed Time: 2 minutes 12 seconds)<br/><style>.rendered_html td, .rendered_html th {text-align: left;}.rendered_html tbody {border: 1px solid black;}</style>"
      ],
      "text/plain": [
       "<Result 'C:\\\\Users\\\\bdt2\\\\Documents\\\\ArcGIS\\\\Projects\\\\DevSummit\\\\scratch.gdb\\\\Trained'>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "startTime = time.perf_counter()\n",
    "\n",
    "predictionType = \"TRAIN_AND_PREDICT\"\n",
    "trainingTV = \"Train\"\n",
    "trainedFC = os.path.join(arcpy.env.scratchGDB,\"Trained\")\n",
    "variablePredict = \"duration\"\n",
    "categorical = False\n",
    "explanatoryVariables = [[\"pLon\", \"false\"], [\"pLat\", \"false\"], [\"dLon\", \"false\"], [\"dLat\", \"false\"], [\"hourCos\", \"false\"], [\"hourSin\", \"false\"], [\"meters\", \"false\"], [\"oneRegion\", \"true\"]]\n",
    "importanceFC = os.path.join(arcpy.env.scratchGDB, \"Importance\")\n",
    "predictionTV = \"Test\"\n",
    "predictionFC = os.path.join(arcpy.env.scratchGDB, \"Predictions\")\n",
    "predictionMatches = [[\"pLon\", \"pLon\"], [\"pLat\", \"pLat\"], [\"dLon\", \"dLon\"], [\"dLat\", \"dLat\"], [\"hourCos\", \"hourCos\"], [\"hourSin\", \"hourSin\"], [\"meters\", \"meters\"], [\"oneRegion\", \"oneRegion\"]]\n",
    "numTrees=200\n",
    "minLeafSize=5\n",
    "maxTreeDepth=5\n",
    "validationPercentage=18\n",
    "\n",
    "results = arcpy.gapro.Forest(\n",
    "    prediction_type=predictionType,\n",
    "    in_features=trainingTV,\n",
    "    output_trained_features=trainedFC,\n",
    "    variable_predict=variablePredict,\n",
    "    treat_variable_as_categorical=categorical,\n",
    "    explanatory_variables=explanatoryVariables,\n",
    "    features_to_predict=predictionTV,\n",
    "    variable_of_importance=importanceFC,\n",
    "    output_predicted=predictionFC,\n",
    "    explanatory_variable_matching=predictionMatches,\n",
    "    number_of_trees=numTrees,\n",
    "    minimum_leaf_size=minLeafSize,\n",
    "    maximum_tree_depth=maxTreeDepth,\n",
    "    percentage_for_validation=validationPercentage)\n",
    "print(timeElapsed(startTime))\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Performance on Test Data (R^2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- <font size= \"3\">Use the arcgis library to load the Predicitons Feature Class into a [Spatial Dataframe](https://github.com/mraad/spark-esri)<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>OBJECTID1</th>\n",
       "      <th>OBJECTID</th>\n",
       "      <th>duration</th>\n",
       "      <th>pLon</th>\n",
       "      <th>pLat</th>\n",
       "      <th>dLon</th>\n",
       "      <th>dLat</th>\n",
       "      <th>hourCos</th>\n",
       "      <th>hourSin</th>\n",
       "      <th>meters</th>\n",
       "      <th>pRegionID</th>\n",
       "      <th>dRegionID</th>\n",
       "      <th>oneRegion</th>\n",
       "      <th>duration_predicted</th>\n",
       "      <th>SHAPE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>120.0</td>\n",
       "      <td>-74.015007</td>\n",
       "      <td>40.709900</td>\n",
       "      <td>-74.010040</td>\n",
       "      <td>40.721409</td>\n",
       "      <td>-0.068242</td>\n",
       "      <td>0.997669</td>\n",
       "      <td>1346.458579</td>\n",
       "      <td>10280</td>\n",
       "      <td>10013</td>\n",
       "      <td>false</td>\n",
       "      <td>516.824928</td>\n",
       "      <td>{\"x\": -8239312.8924, \"y\": 4969645.6752, \"spati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>120.0</td>\n",
       "      <td>-74.014732</td>\n",
       "      <td>40.709846</td>\n",
       "      <td>-74.013367</td>\n",
       "      <td>40.714924</td>\n",
       "      <td>0.682553</td>\n",
       "      <td>-0.730836</td>\n",
       "      <td>576.191110</td>\n",
       "      <td>10280</td>\n",
       "      <td>10007</td>\n",
       "      <td>false</td>\n",
       "      <td>498.041582</td>\n",
       "      <td>{\"x\": -8239282.317600001, \"y\": 4969637.8323, \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>120.0</td>\n",
       "      <td>-74.009598</td>\n",
       "      <td>40.714790</td>\n",
       "      <td>-74.005226</td>\n",
       "      <td>40.720016</td>\n",
       "      <td>-0.990686</td>\n",
       "      <td>-0.136167</td>\n",
       "      <td>688.073684</td>\n",
       "      <td>10007</td>\n",
       "      <td>10013</td>\n",
       "      <td>false</td>\n",
       "      <td>525.973882</td>\n",
       "      <td>{\"x\": -8238710.738500001, \"y\": 4970363.8888000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>120.0</td>\n",
       "      <td>-74.006554</td>\n",
       "      <td>40.714939</td>\n",
       "      <td>-74.013580</td>\n",
       "      <td>40.715218</td>\n",
       "      <td>-0.334880</td>\n",
       "      <td>0.942261</td>\n",
       "      <td>593.028196</td>\n",
       "      <td>10007</td>\n",
       "      <td>10282</td>\n",
       "      <td>false</td>\n",
       "      <td>476.930782</td>\n",
       "      <td>{\"x\": -8238371.867699999, \"y\": 4970385.7386000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>120.0</td>\n",
       "      <td>-74.006325</td>\n",
       "      <td>40.733295</td>\n",
       "      <td>-74.005234</td>\n",
       "      <td>40.736675</td>\n",
       "      <td>-0.576680</td>\n",
       "      <td>0.816970</td>\n",
       "      <td>386.897884</td>\n",
       "      <td>10014</td>\n",
       "      <td>10014</td>\n",
       "      <td>true</td>\n",
       "      <td>385.513468</td>\n",
       "      <td>{\"x\": -8238346.388699999, \"y\": 4973082.0376000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212870</th>\n",
       "      <td>212871</td>\n",
       "      <td>212871</td>\n",
       "      <td>5927.0</td>\n",
       "      <td>-73.985901</td>\n",
       "      <td>40.739441</td>\n",
       "      <td>-73.870750</td>\n",
       "      <td>40.774033</td>\n",
       "      <td>-0.775711</td>\n",
       "      <td>-0.631088</td>\n",
       "      <td>10433.854481</td>\n",
       "      <td>10010</td>\n",
       "      <td>11371</td>\n",
       "      <td>false</td>\n",
       "      <td>1819.501329</td>\n",
       "      <td>{\"x\": -8236072.8116999995, \"y\": 4973984.892399...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212871</th>\n",
       "      <td>212872</td>\n",
       "      <td>212872</td>\n",
       "      <td>5971.0</td>\n",
       "      <td>-73.776749</td>\n",
       "      <td>40.645580</td>\n",
       "      <td>-73.985580</td>\n",
       "      <td>40.759472</td>\n",
       "      <td>-0.576680</td>\n",
       "      <td>-0.816970</td>\n",
       "      <td>21685.971900</td>\n",
       "      <td>11430</td>\n",
       "      <td>10036</td>\n",
       "      <td>false</td>\n",
       "      <td>3231.197310</td>\n",
       "      <td>{\"x\": -8212790.0929000005, \"y\": 4960204.556900...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212872</th>\n",
       "      <td>212873</td>\n",
       "      <td>212873</td>\n",
       "      <td>5972.0</td>\n",
       "      <td>-73.716507</td>\n",
       "      <td>40.719639</td>\n",
       "      <td>-73.984329</td>\n",
       "      <td>40.760029</td>\n",
       "      <td>0.962917</td>\n",
       "      <td>-0.269797</td>\n",
       "      <td>23006.705701</td>\n",
       "      <td>11001</td>\n",
       "      <td>10019</td>\n",
       "      <td>false</td>\n",
       "      <td>2033.290291</td>\n",
       "      <td>{\"x\": -8206084.0176, \"y\": 4971075.991999999, \"...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212873</th>\n",
       "      <td>212874</td>\n",
       "      <td>212874</td>\n",
       "      <td>5983.0</td>\n",
       "      <td>-73.986122</td>\n",
       "      <td>40.757004</td>\n",
       "      <td>-73.790894</td>\n",
       "      <td>40.645523</td>\n",
       "      <td>-0.775711</td>\n",
       "      <td>-0.631088</td>\n",
       "      <td>20603.761436</td>\n",
       "      <td>10036</td>\n",
       "      <td>11430</td>\n",
       "      <td>false</td>\n",
       "      <td>3098.860477</td>\n",
       "      <td>{\"x\": -8236097.441400001, \"y\": 4976565.5781000...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>212874</th>\n",
       "      <td>212875</td>\n",
       "      <td>212875</td>\n",
       "      <td>5996.0</td>\n",
       "      <td>-74.005600</td>\n",
       "      <td>40.750923</td>\n",
       "      <td>-73.782471</td>\n",
       "      <td>40.648777</td>\n",
       "      <td>-0.576680</td>\n",
       "      <td>-0.816970</td>\n",
       "      <td>21973.224473</td>\n",
       "      <td>10001</td>\n",
       "      <td>11430</td>\n",
       "      <td>false</td>\n",
       "      <td>3163.817910</td>\n",
       "      <td>{\"x\": -8238265.7051, \"y\": 4975672.014200002, \"...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>212875 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        OBJECTID1  ...                                              SHAPE\n",
       "0               1  ...  {\"x\": -8239312.8924, \"y\": 4969645.6752, \"spati...\n",
       "1               2  ...  {\"x\": -8239282.317600001, \"y\": 4969637.8323, \"...\n",
       "2               3  ...  {\"x\": -8238710.738500001, \"y\": 4970363.8888000...\n",
       "3               4  ...  {\"x\": -8238371.867699999, \"y\": 4970385.7386000...\n",
       "4               5  ...  {\"x\": -8238346.388699999, \"y\": 4973082.0376000...\n",
       "...           ...  ...                                                ...\n",
       "212870     212871  ...  {\"x\": -8236072.8116999995, \"y\": 4973984.892399...\n",
       "212871     212872  ...  {\"x\": -8212790.0929000005, \"y\": 4960204.556900...\n",
       "212872     212873  ...  {\"x\": -8206084.0176, \"y\": 4971075.991999999, \"...\n",
       "212873     212874  ...  {\"x\": -8236097.441400001, \"y\": 4976565.5781000...\n",
       "212874     212875  ...  {\"x\": -8238265.7051, \"y\": 4975672.014200002, \"...\n",
       "\n",
       "[212875 rows x 15 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6837737295134211\n"
     ]
    }
   ],
   "source": [
    "sdf = GeoAccessor.from_featureclass(os.path.join(arcpy.env.scratchGDB, \"Predictions\"))\n",
    "display(sdf)\n",
    "actual = sdf[\"duration\"].to_numpy()\n",
    "prediction = sdf[\"duration_predicted\"].to_numpy()\n",
    "\n",
    "correlation_matrix = np.corrcoef(actual, prediction)\n",
    "correlation_xy = correlation_matrix[0,1]\n",
    "r_squared = correlation_xy**2\n",
    "\n",
    "print(r_squared)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ArcGISPro",
   "language": "Python",
   "name": "python3"
  },
  "language_info": {
   "file_extension": ".py",
   "name": "python",
   "version": "3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}