1_backfill_feature_groups.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "73ee3ec9",
   "metadata": {},
   "source": [
    "# <span style=\"font-width:bold; font-size: 3rem; color:#1EB182;\"><img src=\"../../images/icon102.png\" width=\"38px\"></img> **Hopsworks Feature Store** </span>\n",
    "\n",
    "<span style=\"font-width:bold; font-size: 3rem; color:#333;\">- Part 01: Backfill Features to the Feature Store</span>\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/logicalclocks/hopsworks-tutorials/blob/master/advanced_tutorials/air_quality/1_backfill_feature_groups.ipynb)\n",
    "\n",
    "\n",
    "## 🗒️ This notebook is divided into the following sections:\n",
    "1. Fetch historical data\n",
    "2. Connect to the Hopsworks feature store\n",
    "3. Create feature groups and insert them to the feature store\n",
    "\n",
    "![tutorial-flow](../../images/01_featuregroups.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f04d5c5e",
   "metadata": {},
   "source": [
    "### <span style='color:#ff5f27'> 📝 Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f65f0db4-1e4b-4f28-a17c-eadcb0d0f016",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "!pip install geopy folium streamlit-folium geopy --q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd165941",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import time\n",
    "import requests\n",
    "from urllib.request import urlopen\n",
    "import json\n",
    "import pandas as pd\n",
    "import folium\n",
    "from functions import *\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba9903fc",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7a1965a-0da7-4263-a68a-8b2e8cb753f1",
   "metadata": {},
   "source": [
    "## <span style='color:#ff5f27'> 🌍 Representing the Target cities </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd578db1-69e7-4230-b3f2-807b8056283a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_url='https://repo.hops.works/dev/jdowling/target_cities.json'\n",
    "response = urlopen(target_url)\n",
    "target_cities = json.loads(response.read())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2246ca9d",
   "metadata": {},
   "source": [
    "## <span style='color:#ff5f27'> 🌫 Processing Air Quality data</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4a1c5d1",
   "metadata": {},
   "source": [
    "### [🇪🇺 EEA](https://discomap.eea.europa.eu/map/fme/AirQualityExport.htm)\n",
    "#### EEA means European Environmental Agency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96b8be01-6286-4886-8043-56e0e49b314e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_cities[\"EU\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bb2a868-5f3a-4065-b651-318c24826b97",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_eu = pd.read_csv(\"data/backfill_pm2_5_eu.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5620df22-f744-4550-a81a-7e5d71aae542",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_eu.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0e23728-a01d-45bc-bf25-4a9c77f21d66",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Size of this dataframe:\", df_eu.shape)\n",
    "\n",
    "df_eu.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e45567-dd6b-4e5e-a153-82a2f4f32fbc",
   "metadata": {},
   "source": [
    "### [🇺🇸 USEPA](https://aqs.epa.gov/aqsweb/documents/data_api.html#daily)\n",
    "#### USEPA means United States Environmental Protection Agency\n",
    "[Manual downloading](https://www.epa.gov/outdoor-air-quality-data/download-daily-data)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4952759-0fb9-4229-8b78-2e37cffb144d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_cities[\"US\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6aceaee-9431-48fd-818a-41fbdd07575c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_us = pd.read_csv(\"data/backfill_pm2_5_us.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e7ff20e-8a1a-4fa3-b801-71beead7b5f2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_us.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3818e3e1-8674-4634-9023-92be8410fba5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Size of this dataframe:\", df_us.shape)\n",
    "\n",
    "df_us.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25557752-31c8-4da9-a52c-4415c4d20ae3",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ff5f27;\">🏢 Processing special city - `Seattle`</span>\n",
    "#### We need different stations across the Seattle. \n",
    "I downloaded daily `PM2.5` data manually [here](https://www.epa.gov/outdoor-air-quality-data/download-daily-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f54d2cb-991c-47cb-a686-76c9f7a87170",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "target_cities[\"Seattle\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31c8505d-68bc-40b6-be0f-42d8532dbd48",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_seattle = pd.read_csv(\"data/backfill_pm2_5_seattle.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f6583c9-3b2a-41c6-a020-aeede88c4867",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_seattle.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "065a5b03-28f7-475c-9c6a-4340388157d8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Size of this dataframe:\", df_seattle.shape)\n",
    "\n",
    "df_seattle.sample(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3b17ca4-0e9d-4207-ad62-90ea9c157def",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_seattle.city_name.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c278a55d-f083-4f95-b292-92e545b9c408",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ff5f27;\">🌟 All together</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d55ae92-4bf9-43ae-8841-6767f5f68bec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality = pd.concat([df_eu, df_us, df_seattle]).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5df39e2-2ce6-48df-9063-9827da8e7317",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality.sample(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "794c30fe-fb54-4fa0-a34c-5cef68f52473",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed9bc7f1-d62e-4b1f-97af-6ecd30fe4b67",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88a9e0ef-e9d2-4e3c-91af-c4e619b8c906",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4687e802",
   "metadata": {
    "tags": []
   },
   "source": [
    "## <span style='color:#ff5f27'> 🌦 Loading Weather Data from [Open Meteo](https://open-meteo.com/en/docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c46283b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_weather = pd.read_csv(\"data/backfill_weather.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1921b61c-d002-417e-88a6-9fe1cad0a7d4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_weather.city_name.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5dcd0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_weather.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc9b7ad6",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8f886c3-a5ac-4370-a6a2-22838ab7409e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality.date = pd.to_datetime(df_air_quality.date)\n",
    "df_weather.date = pd.to_datetime(df_weather.date)\n",
    "\n",
    "df_air_quality[\"unix_time\"] = df_air_quality[\"date\"].apply(convert_date_to_unix)\n",
    "df_weather[\"unix_time\"] = df_weather[\"date\"].apply(convert_date_to_unix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b6af890-87a3-4468-8eda-576c2dd75464",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_air_quality.date = df_air_quality.date.astype(str)\n",
    "df_weather.date = df_weather.date.astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ad5ea08",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_air_quality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ebd846-0420-4e4c-8a5b-0827fa91c693",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb6f83ba",
   "metadata": {},
   "source": [
    "### <span style=\"color:#ff5f27;\"> 🔮 Connecting to Hopsworks Feature Store </span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd068240",
   "metadata": {},
   "outputs": [],
   "source": [
    "import hopsworks\n",
    "\n",
    "project = hopsworks.login()\n",
    "\n",
    "fs = project.get_feature_store() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71db5ac1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from great_expectations.core import ExpectationSuite, ExpectationConfiguration\n",
    "\n",
    "expectation_suite = ExpectationSuite(expectation_suite_name=\"pmi_data\")\n",
    "\n",
    "expectation_suite.add_expectation(\n",
    "    ExpectationConfiguration(\n",
    "        expectation_type=\"expect_column_values_to_be_between\",\n",
    "        kwargs={\n",
    "            \"column\": \"pm2_5\", \n",
    "            \"min_value\": 0.0,\n",
    "            \"max_value\": 1000.0,\n",
    "        }\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63d8c3b9",
   "metadata": {},
   "source": [
    "## <span style=\"color:#ff5f27;\">🪄 Creating Feature Groups</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a2515c4",
   "metadata": {},
   "source": [
    "### <span style='color:#ff5f27'> 🌫 Air Quality Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d7088a8",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "air_quality_fg = fs.get_or_create_feature_group(\n",
    "    name='air_quality',\n",
    "    description='Air Quality characteristics of each day',\n",
    "    version=1,\n",
    "    primary_key=['city_name'], #'unix_time',\n",
    "    online_enabled=False,\n",
    "    expectation_suite = expectation_suite,\n",
    "    event_time=\"unix_time\"\n",
    ")    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e04a975-bb58-42e2-9abd-90e68ae37864",
   "metadata": {},
   "outputs": [],
   "source": [
    "air_quality_fg.insert(df_air_quality, write_options={\"wait_for_job\": False})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a73a9029",
   "metadata": {},
   "source": [
    "### <span style='color:#ff5f27'> 🌦 Weather Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acc2b799",
   "metadata": {},
   "outputs": [],
   "source": [
    "weather_fg = fs.get_or_create_feature_group(\n",
    "    name='weather',\n",
    "    description='Weather characteristics of each day',\n",
    "    version=1,\n",
    "    primary_key=['city_name'], #'unix_time'\n",
    "    online_enabled=False,\n",
    "    event_time=\"unix_time\"\n",
    ") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9583b4d1-e2e3-4f56-9e5d-23caa0c49457",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "weather_fg.insert(df_weather, write_options={\"wait_for_job\": False})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b087a12f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "38",
   "language": "python",
   "name": "38"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}