unit05_correlations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "666e4f61-fd34-46cb-b28a-8a2d5f9bc5f6",
   "metadata": {},
   "source": [
    "# Unit 5 - Correlations \n",
    "\n",
    "\n",
    "1. [Correlation](#section1)\n",
    "2. [Heatmaps](#section2)\n",
    "3. [Get Dummies](#section3)\n",
    "4. [Summary - bringing it all together](#section4)\n",
    "5. [Correlation is NOT causation](#section5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4be1b52-98fb-4e1f-b85b-90c564c44a1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt  #for reshaping graph size\n",
    "import seaborn as sns  # for creating the graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdd13e32-8ffa-4315-ae8c-3bbdab82d61a",
   "metadata": {},
   "source": [
    "## 1. Correlation "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93dddf3c-9722-4fac-891c-ef527a1b3fab",
   "metadata": {},
   "source": [
    "#### Correlation vs. Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22f247c5-a81b-4d18-b97e-ccb79ab409cb",
   "metadata": {},
   "source": [
    "**Correlation** -the degree of relationship between two random variables: `x` and `y`. \n",
    "* Purpose: descriptive statistics\n",
    "*  `x` and `y` can be interchanged\n",
    "*  Random variables: rerunning the experiment can change both `x` and `y`\n",
    "\n",
    "**Regression** - the affect of an independent variable (`y`) on random (dependent) variables (`x_1`...`x_n`)\n",
    "* Purpose: prediction, estimation\n",
    "* `x` and `y` cannot be interchanged\n",
    "* Fixed `x`s , Random `y`. Re-running the experiment will not change `x`s, but might change `y`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f329e801-1c42-40ea-8bc4-6caedf4524fa",
   "metadata": {},
   "source": [
    "Examples:\n",
    "\n",
    "* Math & Physics tests?\n",
    "* Temperature & Electricity bill?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "47d877f7-f9ca-4d18-9060-a0eb2cc9a2b2",
   "metadata": {},
   "source": [
    "### Pearson  \n",
    "\n",
    "For linear correlation (normality assumption)\n",
    "\n",
    "$$\n",
    "r = \\frac{\\text{cov}(x, y)}{\\sigma_x \\sigma_y}\n",
    "$$\n",
    "\n",
    "where:\n",
    "\n",
    "$$\n",
    "\\text{cov} = \\frac{\\sum_{i=1}^n (x_i - \\bar{x})(y_i - \\bar{y})}{n}\n",
    "$$\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0c90439-84ec-4031-a2c7-63c4ae7802b7",
   "metadata": {},
   "source": [
    "<div>\n",
    "    <center>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/correlation slopes.png?raw=true\" width=\"700\"/>\n",
    "    </center>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11ec9c36-0d5a-4994-8d7a-1af54d7e76e1",
   "metadata": {},
   "source": [
    "### Spearman and Kendal \n",
    "\n",
    "can capture non-linear correlations to some extent\n",
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/Spearman.png?raw=true\" width=\"500\"/>\n",
    "    <p style=\"text-align: center;\"><em>By Skbkekas - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=8778554</em></p></center>\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a581c4e9-ab6c-4a20-882e-a0ef8ecce0e0",
   "metadata": {},
   "source": [
    "## 2. Heatmaps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b761a11d-8d5a-47a6-b6b5-d4d2d295bcd2",
   "metadata": {},
   "source": [
    "Read the pickled file you saved on your PC at the end of unit 3:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c506bc92-a78e-4307-a6b9-0f92237b51a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df = pd.read_pickle(\"pickled_strike\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9f2a68a-70d1-45c9-94c0-0002772d3776",
   "metadata": {},
   "source": [
    "If you haven't, then un-comment the below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13822132-ea3d-4ee7-9ed7-04cbba8dd76e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# url1 = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/aircraft%20wildlife%20strikes%202018-2020.csv'\n",
    "# url2 = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/aircraft%20wildlife%20strikes%202021-2023.csv'\n",
    "# strike_df_18_20 = pd.read_csv(url1)\n",
    "# strike_df_21_23 = pd.read_csv(url2)\n",
    "# strike_df = pd.concat([strike_df_18_20 ,strike_df_21_23]).reset_index(drop = True)\n",
    "# strike_df['date'] = pd.to_datetime(strike_df['INCIDENT_DATE'],format='%d/%m/%Y')\n",
    "# strike_df['month'] = pd.DatetimeIndex(strike_df['date']).month\n",
    "# strike_df['year'] = pd.DatetimeIndex(strike_df['date']).year\n",
    "# strike_df[\"people_impact\"] = strike_df[['NR_INJURIES', 'NR_FATALITIES']].sum(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c031769f-f374-4521-b8cc-32c7c6a28e44",
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_features = ['HEIGHT', 'SPEED', 'AC_MASS']\n",
    "target_features = ['AircraftOutOfService','people_impact','struck_parts', 'damaged_parts']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7eb7fedc-2316-411b-bc89-4dd21a839706",
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df_num = strike_df[numeric_features + target_features]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "895cde75-3322-4d44-9c5e-9e19f718823d",
   "metadata": {},
   "source": [
    "If the distribution of the varaibles is not normal, better use Spearman or Kendall correlation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2152355e-e955-4712-aaa5-98780612b2c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "correlation_matrix = strike_df_num.corr(numeric_only = True, method = 'spearman' ).round(2)\n",
    "correlation_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd1906b6-0bb2-4af8-af68-c33640c8c7ca",
   "metadata": {},
   "source": [
    "Make it look a bit nicer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68545451-c4c9-46d1-84ee-413e6efba94c",
   "metadata": {},
   "outputs": [],
   "source": [
    "subset_correlation_matrix = correlation_matrix[target_features].transpose()\n",
    "subset_correlation_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2615e88-e743-4890-8ae0-e556ad668566",
   "metadata": {},
   "source": [
    "Turn into a heatmap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e38fad8d-a24b-494b-81c3-1e438d267e50",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(17,2))\n",
    "sns.heatmap(data=subset_correlation_matrix,cmap='coolwarm', annot=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "177dffea-d3f9-4ad5-b334-61dce2c247d3",
   "metadata": {},
   "source": [
    "## 3. Get Dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a965b4a-9ddb-400e-980a-d399abd41897",
   "metadata": {},
   "source": [
    "Convert categorical variable into dummy/indicator variables.\n",
    "\n",
    "Each variable is converted in as many 0/1 variables as there are different values.  \n",
    "Columns in the output are each named after a value; the name of the original variable is prepended to the value.  \n",
    "\n",
    "[pandas.get_dummies](https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a6c6f56-40b2-4d4d-8fce-844856cf7b38",
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = ['WARNED','PHASE_OF_FLIGHT','SKY','TIME_OF_DAY']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dea0111-5a1d-424b-b2f3-5f89d1d91c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_features = target_features + numeric_features + categorical_features \n",
    "all_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cc980c6-ce96-464f-b035-028480191b50",
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df_dum = pd.get_dummies(strike_df[all_features])\n",
    "strike_df_dum.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d82fc2df-6bfb-457f-ac6d-d6f335bfa73c",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_correlation_matrix = strike_df_dum.corr( numeric_only = False,method = 'spearman' ).round(2)\n",
    "subset_all_correlation_matrix = all_correlation_matrix[target_features].transpose()\n",
    "subset_all_correlation_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a51ef6-cee4-41f1-8c44-aac372989515",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Create a heatmap for `subset_all_correlation_matrix`\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad4292e-41ae-4fa0-a6c0-c0a2dcdf5312",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(17,2))\n",
    "# YOUR CODE HERE\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00721a93-bee3-4767-a21e-ab8e36823a15",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/heatmap.png\" width=\"900\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d47cdd9-1f9b-496e-a0df-4e1704563873",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Study the correlations:\n",
    "\n",
    "* `HEIGHT` and `damaged_parts`\n",
    "* `AC_MASS` & `damaged_parts`\n",
    "* `AircraftOutOfService` and `damaged_parts`\n",
    "* `PHASE_OF_FLIGHT` and `struck_parts` / `damaged_parts`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf11f1f-fa6b-48a0-b1aa-34031b4be6af",
   "metadata": {},
   "source": [
    "### `HEIGHT` and `damaged_parts`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cd0e7efb-4431-4a39-b930-d1fd5a057e4b",
   "metadata": {},
   "source": [
    "##### Note that the y-axis is on a **log** scale here!\n",
    "\n",
    "[For an intuitive understanding of log scales](https://www.youtube.com/watch?v=0fKBhvDjuy0&ab_channel=EamesOffice)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be209ce7-8323-4e55-ac7a-38c5fa873129",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(10, 4), ncols=2)\n",
    "\n",
    "sns.scatterplot(data = strike_df, y = 'HEIGHT', x = 'damaged_parts', ax = axes[0])\n",
    "axes[0].set_yscale('log')\n",
    "\n",
    "sns.stripplot(data = strike_df, y = 'HEIGHT', x = 'damaged_parts', ax = axes[1])\n",
    "axes[1].set_yscale('log')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d3a5255-b485-434d-b6c1-045bba1d1b88",
   "metadata": {},
   "source": [
    "### `AC_MASS` & `damaged_parts`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dad62f10-d10b-48bd-bf08-d6e13ebb20dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(10, 4), ncols=3)\n",
    "\n",
    "sns.barplot(data = strike_df, x = 'AC_MASS', y = 'damaged_parts', ax = axes[0])\n",
    "sns.stripplot(data = strike_df, x = 'AC_MASS', y = 'damaged_parts', ax = axes[1])\n",
    "sns.scatterplot(data = strike_df, x = 'AC_MASS', y = 'damaged_parts', ax = axes[2])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4b91717-e812-4b78-ab9d-2af6b4217e5e",
   "metadata": {},
   "source": [
    "### `AircraftOutOfService` and `damaged_parts`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5eaf6c56-b5b7-42b8-908a-ff92e640461a",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Create a scatterplot and a stripplot for `AircraftOutOfService` and `damaged_parts`  \n",
    "> Don't forget to use log scale\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41473acf-0c02-4abd-ab6d-6cb6b2d1f402",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(10, 4), ncols=2)\n",
    "\n",
    "#YOUR CODE HERE\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11d57b30-e9c0-41f0-952b-711e409c4381",
   "metadata": {},
   "source": [
    "### PHASE_OF_FLIGHT and `struck_parts` / `damaged_parts`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677ced5f-eb3b-4cc7-93fb-dfebe44a80dc",
   "metadata": {},
   "source": [
    "We will first order the parts by their phase of flight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af0d017a-c3e3-4a4f-8c85-1e09d450fcd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "struck_counts = strike_df.groupby('PHASE_OF_FLIGHT')['struck_parts'].sum().sort_values(ascending=False)\n",
    "struck_order = struck_counts.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d09c3c4-eed3-463a-9e5f-405ce437cc22",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(12, 5), ncols=2)\n",
    "\n",
    "sns.barplot(data = strike_df, y = 'struck_parts', x = 'PHASE_OF_FLIGHT', ax = axes[0], order = struck_order, estimator = sum)\n",
    "axes[0].set_xticklabels(axes[0].get_xticklabels(), rotation=45, ha='right')\n",
    "\n",
    "sns.barplot(data = strike_df, y = 'damaged_parts', x = 'PHASE_OF_FLIGHT', ax = axes[1], order = struck_order, estimator = sum)\n",
    "axes[1].set_xticklabels(axes[1].get_xticklabels(), rotation=45, ha='right')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "898d6c6a-ee25-4f6c-830c-0adcad090902",
   "metadata": {},
   "source": [
    "## 4. Summary - bringing it all together:\n",
    ">\n",
    "> * Load your data to GitHub, then read it\n",
    "> * Look at it using: `len`, `shape`, `info`, `describe`\n",
    "> * Describe the variables:  \n",
    ">   * Categorical variables: `countplot`\n",
    ">   * Numerical variables:  `histplot`, `boxplot`\n",
    ">   * Statistics: `barplot`\n",
    ">   * Relationships: `lineplot`,  `stripplot`\n",
    ">   * Tables for averages, max, min etc.\n",
    ">   * Do you need to remove outliers? Change labels? Fill missing values?\n",
    "> * Look for correlations: `corr`, `heatmap`  \n",
    "> * Visualize the correlations: `scatterplot`\n",
    ">\n",
    ">  **Note 1:** These are just examples! There are more ways to visualize  \n",
    ">  **Note 2:** Sometimes you need to group or filter the data first (or to melt it.. in the next lessons)  \n",
    ">  **Note 3:** Sometimes presenting is in a table gives the best effect  \n",
    "\n",
    "\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19d23777-51e3-45c0-9654-2ba99940eb9b",
   "metadata": {},
   "source": [
    "## 5. Correlation is NOT causation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a1efd7c8-0984-448e-bd7c-5cb929a1d761",
   "metadata": {},
   "source": [
    "Sometimes correlation indicates causation. But not always.  \n",
    "Sometimes there is a hidden variable\n",
    "Sometimes it just isn't causation: [spurios (meaning false/fake) correlations website](https://www.tylervigen.com/spurious-correlations)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "258440e7-6ada-4746-935a-045c8b9efd98",
   "metadata": {},
   "source": [
    "Be Humble. We see things in the data. We report them. We don't know their cause."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d758e66-e95a-4088-b6ae-f42ca17fbefa",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}