unit04_visualize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unit 4 - Descriptive visualization of the data\n",
    "---\n",
    "\n",
    "1. [Boxplots](#section1)\n",
    "2. [Histograms](#section2)\n",
    "3. [What to use when](#section3)\n",
    "4. [Same stats, different graphs](#section4)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Introducing an additional library: [seaborn](https://seaborn.pydata.org/) - for statistical data visualization\\\n",
    "Behind the scenes, seaborn uses matplotlib to draw its plots.\\\n",
    "[matplotlib.pyplot](https://matplotlib.org/stable/api/pyplot_summary.html#module-matplotlib.pyplot) is the GUI manager of the figure. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt  #for reshaping graph size\n",
    "import seaborn as sns  # for creating the graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1'></a>\n",
    "## 1. Boxplots \n",
    "\n",
    "What are they good for? Let's look at an example with the Titanic dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df = sns.load_dataset('titanic')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We would like to vizualize the passengers `age`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### Attempt #1: With `scatterplot`\n",
    "\n",
    "The figure size is set using matplotlib, but there are other ways. See [this](https://stackoverflow.com/questions/31594549/how-to-change-the-figure-size-of-a-seaborn-axes-or-figure-level-plot) highly voted question on stackoverflow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))  #figure size\n",
    "sns.scatterplot(data = titanic_df[['age']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the raw data:\n",
    "\n",
    "axis x - the 891 passangers \n",
    "\n",
    "axis y - the age of each passenger\n",
    "\n",
    "This is not informative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### Attempt #2: With `lineplot`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(6,3))\n",
    "g = sns.lineplot(data = titanic_df[['age']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Attempt #3:`boxplot`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(2,5))\n",
    "g = sns.boxplot(data = titanic_df[['age' ]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the figure using [savefig](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.savefig.html) if we want to use it later.\\\n",
    "`bbox_inches` - only the given portion of the figure is saved. If 'tight', create a tight box around the figure. Try removing tight and see the difference. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g.figure.savefig(\"boxplot.png\", bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/boxplot.png\" width=\"600\"/>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data seems fine. What would we think if we had the outliers under the bottom whisker?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data from a project in 2022. Israel vs. the world."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/graze%20%20footprints.png\" width=\"800\"/>\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section2'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vaccination data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://raw.githubusercontent.com/owid/covid-19-data/master/public/data/vaccinations/vaccinations.csv'\n",
    "vacc_df = pd.read_csv(url) \n",
    "vacc_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "use a groupby and look at part of the data, by location:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_df = vacc_df.groupby('location')[['daily_vaccinations','people_fully_vaccinated_per_hundred'\\\n",
    "                                          ,'total_boosters_per_hundred']].max()\n",
    "grouped_df = grouped_df.reset_index()\n",
    "grouped_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### sort the values using 'sort_values()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped_df.sort_values('people_fully_vaccinated_per_hundred', ascending = False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> For the data in `grouped_df`:\n",
    ">\n",
    "> display a scatterplot for `total_boosters_per_hundred`\n",
    ">\n",
    "> display a boxplot for two `total_boosters_per_hundred` and `people_fully_vaccinated_per_hundred` in the **same** boxplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**It's not perfect. Or else we would have had outliers for any number over a 100.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section2'></a>\n",
    "## 2. Histograms"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why use histograms? \\\n",
    "Boxplots display summary statistics, but they don't tell us much about the distribution shape. \\\n",
    "We use histograms to show the shape. \n",
    "\n",
    "<div>\n",
    "<center><img src=\"https://github.com/nlihin/data-analytics/blob/main/images/distribution%20shapes.png?raw=true\" width=\"500\"/>\n",
    "    <p style=\"text-align: center;\"><em>O'Niel and Schutt. Doing data science: Straight talk from the frontline. O'Rielly Meida, Inc. 2013. </em></p></center>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Histograms can show the number (count), percentage, probability or density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2,2, figsize = (10,5))\n",
    "plt.subplots_adjust(wspace = 0.5)\n",
    "\n",
    "sns.histplot(titanic_df, x ='age', ax = ax[0,0])\n",
    "sns.histplot(data=titanic_df, x='age', stat='percent', ax = ax[0,1])\n",
    "sns.histplot(data=titanic_df, x='age', stat='probability', ax = ax[1,0])\n",
    "sns.histplot(data=titanic_df, x='age', stat='density', ax = ax[1,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape won't change as long as the number of bins doesn't change. \n",
    "Change the number of bins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.histplot(data=titanic_df, x='age', stat='percent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Histograms of males and females:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.histplot(data=titanic_df[titanic_df.sex == 'male'], x='age', stat='percent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    ">\n",
    "> Create a histogram for the age of female passengers on the Titanic\n",
    ">\n",
    "> This histogram doesn't have the same number of bins as the histogram of males. Any idea why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first - filter so you have the females:\n",
    "female_df = titanic_df[titanic_df.sex == 'female']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.histplot(data= female_df, x='age', stat='percent')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(female_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(titanic_df[titanic_df.sex == 'male'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Create two histograms, one for males and one for females, with the **same** number of bins\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqoAAAEYCAYAAACUb9SJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAcdklEQVR4nO3db2zV5f3/8Vdb7ClGWnBdT0t3tAOHqCDFVrqCxLic2URTx43FTgztGv9M7YxysgkVbEUmZU5JEyk2ok5v6IoaIUaaOjxKjNqFWGiiEzBYtJ3xHOgcPaxoCz3X94Y/j7+uLZzPoae9zuH5SM6Nfryuc94XxRcvzuk5pBhjjAAAAADLpE72AAAAAMBoKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALCS46L67rvvqry8XDNnzlRKSop27Nhxxj27d+/WVVddJZfLpUsuuUTPP/98DKMCQHIgRwEgOo6Lan9/vxYsWKCmpqao1h8+fFg33nijrrvuOnV2dur+++/X7bffrjfffNPxsACQDMhRAIhOijHGxLw5JUXbt2/XsmXLxlyzatUq7dy5Ux9//HHk2m9+8xsdO3ZMbW1tsT40ACQFchQAxjYl3g/Q3t4ur9c77FpZWZnuv//+MfcMDAxoYGAg8nU4HNbXX3+tH/3oR0pJSYnXqADOYcYYHT9+XDNnzlRqql0/vk+OAkgE8cjRuBfVQCAgt9s97Jrb7VYoFNI333yjqVOnjtjT0NCgdevWxXs0ABihp6dHP/nJTyZ7jGHIUQCJZDxzNO5FNRa1tbXy+XyRr/v6+nTRRRepp6dHmZmZkzgZgGQVCoXk8Xg0bdq0yR5lXJCjACZaPHI07kU1NzdXwWBw2LVgMKjMzMxRnwWQJJfLJZfLNeJ6ZmYmAQsgrmx8WZwcBZBIxjNH4/6DWKWlpfL7/cOu7dq1S6WlpfF+aABICuQogHOV46L63//+V52dners7JT03cemdHZ2qru7W9J3LzdVVlZG1t91113q6urSAw88oAMHDmjLli16+eWXtXLlyvE5AQAkGHIUAKLjuKh++OGHWrhwoRYuXChJ8vl8Wrhwoerq6iRJX331VSRsJemnP/2pdu7cqV27dmnBggV64okn9Mwzz6isrGycjgAAiYUcBYDonNXnqE6UUCikrKws9fX18bNVAOIi2XMm2c8HYPLFI2fs+rBAAAAA4P+hqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFgppqLa1NSkgoICZWRkqKSkRHv27Dnt+sbGRl166aWaOnWqPB6PVq5cqW+//TamgQEgGZCjAHBmjovqtm3b5PP5VF9fr71792rBggUqKyvTkSNHRl3/0ksvafXq1aqvr9f+/fv17LPPatu2bXrwwQfPengASETkKABEx3FR3bRpk+644w5VV1fr8ssvV3Nzs84//3w999xzo67/4IMPtGTJEi1fvlwFBQW6/vrrdcstt5zx2QMASFbkKABEx1FRHRwcVEdHh7xe7w93kJoqr9er9vb2UfcsXrxYHR0dkUDt6upSa2urbrjhhjEfZ2BgQKFQaNgNAJIBOQoA0ZviZHFvb6+GhobkdruHXXe73Tpw4MCoe5YvX67e3l5dc801Msbo1KlTuuuuu077klVDQ4PWrVvnZDQASAjkKABEL+7v+t+9e7c2bNigLVu2aO/evXrttde0c+dOrV+/fsw9tbW16uvri9x6enriPSYAWIscBXCucvSManZ2ttLS0hQMBoddDwaDys3NHXXPQw89pBUrVuj222+XJM2fP1/9/f268847tWbNGqWmjuzKLpdLLpfLyWgAkBDIUQCInqNnVNPT01VUVCS/3x+5Fg6H5ff7VVpaOuqeEydOjAjRtLQ0SZIxxum8AJDQyFEAiJ6jZ1QlyefzqaqqSsXFxVq0aJEaGxvV39+v6upqSVJlZaXy8/PV0NAgSSovL9emTZu0cOFClZSU6NChQ3rooYdUXl4eCVoAOJeQowAQHcdFtaKiQkePHlVdXZ0CgYAKCwvV1tYWeWNAd3f3sL/5r127VikpKVq7dq2+/PJL/fjHP1Z5ebkeffTR8TsFACQQchQAopNiEuB1o1AopKysLPX19SkzM3OyxwGQhJI9Z5L9fAAmXzxyJu7v+gcAAABiQVEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwUkxFtampSQUFBcrIyFBJSYn27Nlz2vXHjh1TTU2N8vLy5HK5NGfOHLW2tsY0MAAkA3IUAM5sitMN27Ztk8/nU3Nzs0pKStTY2KiysjIdPHhQOTk5I9YPDg7ql7/8pXJycvTqq68qPz9fX3zxhaZPnz4e8wNAwiFHASA6KcYY42RDSUmJrr76am3evFmSFA6H5fF4dO+992r16tUj1jc3N+svf/mLDhw4oPPOOy+mIUOhkLKystTX16fMzMyY7gMATmcic4YcBZCM4pEzjl76HxwcVEdHh7xe7w93kJoqr9er9vb2Ufe8/vrrKi0tVU1Njdxut+bNm6cNGzZoaGjo7CYHgAREjgJA9By99N/b26uhoSG53e5h191utw4cODDqnq6uLr399tu69dZb1draqkOHDumee+7RyZMnVV9fP+qegYEBDQwMRL4OhUJOxgQAa5GjABC9uL/rPxwOKycnR08//bSKiopUUVGhNWvWqLm5ecw9DQ0NysrKitw8Hk+8xwQAa5GjAM5Vjopqdna20tLSFAwGh10PBoPKzc0ddU9eXp7mzJmjtLS0yLXLLrtMgUBAg4ODo+6pra1VX19f5NbT0+NkTACwFjkKANFzVFTT09NVVFQkv98fuRYOh+X3+1VaWjrqniVLlujQoUMKh8ORa59++qny8vKUnp4+6h6Xy6XMzMxhNwBIBuQoAETP8Uv/Pp9PW7du1QsvvKD9+/fr7rvvVn9/v6qrqyVJlZWVqq2tjay/++679fXXX+u+++7Tp59+qp07d2rDhg2qqakZv1MAQAIhRwEgOo4/R7WiokJHjx5VXV2dAoGACgsL1dbWFnljQHd3t1JTf+i/Ho9Hb775plauXKkrr7xS+fn5uu+++7Rq1arxOwUAJBByFACi4/hzVCcDn/8HIN6SPWeS/XwAJt+kf44qAAAAMFEoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKMRXVpqYmFRQUKCMjQyUlJdqzZ09U+1paWpSSkqJly5bF8rAAkDTIUQA4M8dFddu2bfL5fKqvr9fevXu1YMEClZWV6ciRI6fd9/nnn+sPf/iDli5dGvOwAJAMyFEAiI7jorpp0ybdcccdqq6u1uWXX67m5madf/75eu6558bcMzQ0pFtvvVXr1q3TrFmzzmpgAEh05CgARMdRUR0cHFRHR4e8Xu8Pd5CaKq/Xq/b29jH3PfLII8rJydFtt90W1eMMDAwoFAoNuwFAMiBHASB6jopqb2+vhoaG5Ha7h113u90KBAKj7nnvvff07LPPauvWrVE/TkNDg7KysiI3j8fjZEwAsBY5CgDRi+u7/o8fP64VK1Zo69atys7OjnpfbW2t+vr6Ireenp44TgkA9iJHAZzLpjhZnJ2drbS0NAWDwWHXg8GgcnNzR6z/7LPP9Pnnn6u8vDxyLRwOf/fAU6bo4MGDmj179oh9LpdLLpfLyWgAkBDIUQCInqNnVNPT01VUVCS/3x+5Fg6H5ff7VVpaOmL93Llz9dFHH6mzszNyu+mmm3Tdddeps7OTl6IAnHPIUQCInqNnVCXJ5/OpqqpKxcXFWrRokRobG9Xf36/q6mpJUmVlpfLz89XQ0KCMjAzNmzdv2P7p06dL0ojrAHCuIEcBIDqOi2pFRYWOHj2quro6BQIBFRYWqq2tLfLGgO7ubqWm8g9eAcBYyFEAiE6KMcZM9hBnEgqFlJWVpb6+PmVmZk72OACSULLnTLKfD8Dki0fO8Fd2AAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYKWYimpTU5MKCgqUkZGhkpIS7dmzZ8y1W7du1dKlSzVjxgzNmDFDXq/3tOsB4FxAjgLAmTkuqtu2bZPP51N9fb327t2rBQsWqKysTEeOHBl1/e7du3XLLbfonXfeUXt7uzwej66//np9+eWXZz08ACQichQAopNijDFONpSUlOjqq6/W5s2bJUnhcFgej0f33nuvVq9efcb9Q0NDmjFjhjZv3qzKysqoHjMUCikrK0t9fX3KzMx0Mi4ARGUic4YcBZCM4pEzjp5RHRwcVEdHh7xe7w93kJoqr9er9vb2qO7jxIkTOnnypC688MIx1wwMDCgUCg27AUAyIEcBIHqOimpvb6+GhobkdruHXXe73QoEAlHdx6pVqzRz5sxhIf2/GhoalJWVFbl5PB4nYwKAtchRAIjehL7rf+PGjWppadH27duVkZEx5rra2lr19fVFbj09PRM4JQDYixwFcC6Z4mRxdna20tLSFAwGh10PBoPKzc097d7HH39cGzdu1FtvvaUrr7zytGtdLpdcLpeT0QAgIZCjABA9R8+opqenq6ioSH6/P3ItHA7L7/ertLR0zH2PPfaY1q9fr7a2NhUXF8c+LQAkOHIUAKLn6BlVSfL5fKqqqlJxcbEWLVqkxsZG9ff3q7q6WpJUWVmp/Px8NTQ0SJL+/Oc/q66uTi+99JIKCgoiP4N1wQUX6IILLhjHowBAYiBHASA6jotqRUWFjh49qrq6OgUCARUWFqqtrS3yxoDu7m6lpv7wRO1TTz2lwcFB/frXvx52P/X19Xr44YfPbnoASEDkKABEx/HnqE4GPv8PQLwle84k+/kATL5J/xxVAAAAYKJQVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVqKoAgAAwEoUVQAAAFiJogoAAAArUVQBAABgJYoqAAAArERRBQAAgJUoqgAAALASRRUAAABWoqgCAADAShRVAAAAWImiCgAAACtRVAEAAGAliioAAACsRFEFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKwUU1FtampSQUGBMjIyVFJSoj179px2/SuvvKK5c+cqIyND8+fPV2tra0zDAkCyIEcB4MwcF9Vt27bJ5/Opvr5ee/fu1YIFC1RWVqYjR46Muv6DDz7QLbfcottuu0379u3TsmXLtGzZMn388cdnPTwAJCJyFACik2KMMU42lJSU6Oqrr9bmzZslSeFwWB6PR/fee69Wr149Yn1FRYX6+/v1xhtvRK79/Oc/V2FhoZqbm6N6zFAopKysLPX19SkzM9PJuAAQlYnMGXIUQDKKR85McbJ4cHBQHR0dqq2tjVxLTU2V1+tVe3v7qHva29vl8/mGXSsrK9OOHTvGfJyBgQENDAxEvu7r65P03S8AAMTD9/ni8O/ujpGjAJJVPHLUUVHt7e3V0NCQ3G73sOtut1sHDhwYdU8gEBh1fSAQGPNxGhoatG7duhHXPR6Pk3EBwLF///vfysrKitv9k6MAkt145qijojpRamtrhz17cOzYMV188cXq7u6O6x8gkyUUCsnj8ainpycpX5JL9vNJyX/GZD+f9N0zjhdddJEuvPDCyR5lXJCjySfZz8j5El88ctRRUc3OzlZaWpqCweCw68FgULm5uaPuyc3NdbReklwul1wu14jrWVlZSfvNlaTMzEzOl+CS/YzJfj7pu5fh44kcja9z4fdosp+R8yW+8cxRR/eUnp6uoqIi+f3+yLVwOCy/36/S0tJR95SWlg5bL0m7du0acz0AJDNyFACi5/ilf5/Pp6qqKhUXF2vRokVqbGxUf3+/qqurJUmVlZXKz89XQ0ODJOm+++7TtddeqyeeeEI33nijWlpa9OGHH+rpp58e35MAQIIgRwEgOo6LakVFhY4ePaq6ujoFAgEVFhaqra0t8oP+3d3dw57yXbx4sV566SWtXbtWDz74oH72s59px44dmjdvXtSP6XK5VF9fP+rLWMmA8yW+ZD9jsp9PmtgzkqPjL9nPJyX/GTlf4ovHGR1/jioAAAAwEeL7rgEAAAAgRhRVAAAAWImiCgAAACtRVAEAAGAla4pqU1OTCgoKlJGRoZKSEu3Zs+e061955RXNnTtXGRkZmj9/vlpbWydo0tg4Od/WrVu1dOlSzZgxQzNmzJDX6z3jr8dkc/r9+15LS4tSUlK0bNmy+A44Dpye8dixY6qpqVFeXp5cLpfmzJlj9e9Tp+drbGzUpZdeqqlTp8rj8WjlypX69ttvJ2haZ959912Vl5dr5syZSklJ0Y4dO864Z/fu3brqqqvkcrl0ySWX6Pnnn4/7nGeLHP1BIuaolPxZSo4OR45GwVigpaXFpKenm+eee87885//NHfccYeZPn26CQaDo65///33TVpamnnsscfMJ598YtauXWvOO+8889FHH03w5NFxer7ly5ebpqYms2/fPrN//37z29/+1mRlZZl//etfEzx5dJye73uHDx82+fn5ZunSpeZXv/rVxAwbI6dnHBgYMMXFxeaGG24w7733njl8+LDZvXu36ezsnODJo+P0fC+++KJxuVzmxRdfNIcPHzZvvvmmycvLMytXrpzgyaPT2tpq1qxZY1577TUjyWzfvv2067u6usz5559vfD6f+eSTT8yTTz5p0tLSTFtb28QMHANydLhEy1Fjkj9LydHhyNHoWFFUFy1aZGpqaiJfDw0NmZkzZ5qGhoZR1998883mxhtvHHatpKTE/O53v4vrnLFyer7/derUKTNt2jTzwgsvxGvEsxLL+U6dOmUWL15snnnmGVNVVWV1uBrj/IxPPfWUmTVrlhkcHJyoEc+K0/PV1NSYX/ziF8Ou+Xw+s2TJkrjOOR6iCdgHHnjAXHHFFcOuVVRUmLKysjhOdnbI0dOzPUeNSf4sJUeHI0ejM+kv/Q8ODqqjo0NerzdyLTU1VV6vV+3t7aPuaW9vH7ZeksrKysZcP5liOd//OnHihE6ePKkLL7wwXmPGLNbzPfLII8rJydFtt902EWOelVjO+Prrr6u0tFQ1NTVyu92aN2+eNmzYoKGhoYkaO2qxnG/x4sXq6OiIvKzV1dWl1tZW3XDDDRMyc7wlUsZI5Gg0bM5RKfmzlBwdiRyNjuN/mWq89fb2amhoKPIvsnzP7XbrwIEDo+4JBAKjrg8EAnGbM1axnO9/rVq1SjNnzhzxDbdBLOd777339Oyzz6qzs3MCJjx7sZyxq6tLb7/9tm699Va1trbq0KFDuueee3Ty5EnV19dPxNhRi+V8y5cvV29vr6655hoZY3Tq1CndddddevDBBydi5LgbK2NCoZC++eYbTZ06dZImGx05emY256iU/FlKjo5EjkaXo5P+jCpOb+PGjWppadH27duVkZEx2eOctePHj2vFihXaunWrsrOzJ3ucuAmHw8rJydHTTz+toqIiVVRUaM2aNWpubp7s0cbF7t27tWHDBm3ZskV79+7Va6+9pp07d2r9+vWTPRowQrLlqHRuZCk5CsmCZ1Szs7OVlpamYDA47HowGFRubu6oe3Jzcx2tn0yxnO97jz/+uDZu3Ki33npLV155ZTzHjJnT83322Wf6/PPPVV5eHrkWDoclSVOmTNHBgwc1e/bs+A7tUCzfw7y8PJ133nlKS0uLXLvssssUCAQ0ODio9PT0uM7sRCzne+ihh7RixQrdfvvtkqT58+erv79fd955p9asWTPs36lPRGNlTGZmpnXPpkrk6OkkQo5KyZ+l5OhI5Gh0Jv1XIT09XUVFRfL7/ZFr4XBYfr9fpaWlo+4pLS0dtl6Sdu3aNeb6yRTL+STpscce0/r169XW1qbi4uKJGDUmTs83d+5cffTRR+rs7IzcbrrpJl133XXq7OyUx+OZyPGjEsv3cMmSJTp06FDkDw5J+vTTT5WXl2dVuEqxne/EiRMjQvT7P0y++zn7xJZIGSORo2NJlByVkj9LydGRyNEoOXrrVZy0tLQYl8tlnn/+efPJJ5+YO++800yfPt0EAgFjjDErVqwwq1evjqx///33zZQpU8zjjz9u9u/fb+rr663/WBUn59u4caNJT083r776qvnqq68it+PHj0/WEU7L6fn+l+3vVDXG+Rm7u7vNtGnTzO9//3tz8OBB88Ybb5icnBzzpz/9abKOcFpOz1dfX2+mTZtm/va3v5muri7z97//3cyePdvcfPPNk3WE0zp+/LjZt2+f2bdvn5FkNm3aZPbt22e++OILY4wxq1evNitWrIis//5jVf74xz+a/fv3m6ampoT4eCpyNHFz1Jjkz1JylBxN2I+nMsaYJ5980lx00UUmPT3dLFq0yPzjH/+I/Ldrr73WVFVVDVv/8ssvmzlz5pj09HRzxRVXmJ07d07wxM44Od/FF19sJI241dfXT/zgUXL6/fv/2R6u33N6xg8++MCUlJQYl8tlZs2aZR599FFz6tSpCZ46ek7Od/LkSfPwww+b2bNnm4yMDOPxeMw999xj/vOf/0z84FF45513Rv1/6vszVVVVmWuvvXbEnsLCQpOenm5mzZpl/vrXv0743E6Ro1WRrxMxR41J/iwlR6siX5Oj0UkxJgmeXwYAAEDSmfSfUQUAAABGQ1EFAACAlSiqAAAAsBJFFQAAAFaiqAIAAMBKFFUAAABYiaIKAAAAK1FUAQAAYCWKKgAAAKxEUQUAAICVKKoAAACwEkUVAAAAVvo/3dQpUfomHTcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1,2, figsize = (8,3))\n",
    "#YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both sexs on the same graph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.histplot(data=titanic_df, x='age', stat='percent', hue='sex', multiple = 'layer' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    ">\n",
    "> try other options:\n",
    ">\n",
    "> `multiple{“layer”, “dodge”, “stack”, “fill”}`\n",
    ">\n",
    "> what is the default?\n",
    ">\n",
    "> create a histogram for `total_boosters_per_hundred` for our `grouped_df` dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.histplot(data=titanic_df, x='age', stat='percent', hue='sex', multiple = '...' )\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section3'></a>\n",
    "## 3. What to use when?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the pickled file you saved on your PC at the end of unit 3 (last lesson):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df = pd.read_pickle(\"pickled_strike\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you haven't, then un-comment the below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# url1 = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/aircraft%20wildlife%20strikes%202018-2020.csv'\n",
    "# url2 = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/aircraft%20wildlife%20strikes%202021-2023.csv'\n",
    "# strike_df_18_20 = pd.read_csv(url1)\n",
    "# strike_df_21_23 = pd.read_csv(url2)\n",
    "# strike_df = pd.concat([strike_df_18_20 ,strike_df_21_23]).reset_index(drop = True)\n",
    "# strike_df['date'] = pd.to_datetime(strike_df['INCIDENT_DATE'],format='%d/%m/%Y')\n",
    "# strike_df['month'] = pd.DatetimeIndex(strike_df['date']).month\n",
    "# strike_df['year'] = pd.DatetimeIndex(strike_df['date']).year\n",
    "# strike_df[\"people_impact\"] = strike_df[['NR_INJURIES', 'NR_FATALITIES']].sum(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "strike_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### numeric features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_features = ['HEIGHT', 'SPEED', 'AC_MASS']\n",
    "target_features = ['AircraftOutOfService','people_impact','struck_parts', 'damaged_parts']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### categorical features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_features = ['WARNED','PHASE_OF_FLIGHT','SKY','TIME_OF_DAY']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What to use when?\n",
    "* **Countplots** are good for describing **categorical features**  \n",
    "* **Hitograms** and **boxplots** are good for describing **numeric features**\n",
    "* **Scatterplots** are good for describing the interaction between two **numeric features**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(ncols=3, figsize=(15, 6))\n",
    "sns.countplot(data=strike_df, x = \"WARNED\", ax = axes[0])\n",
    "sns.countplot(data=strike_df, x = \"TIME_OF_DAY\", ax = axes[1])\n",
    "sns.countplot(data=strike_df, x = \"SKY\", ax = axes[2])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lets look at some of the numeric features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(12, 3), ncols=3)\n",
    "sns.boxplot(data=strike_df[['HEIGHT']], ax = axes[0])\n",
    "sns.boxplot(data=strike_df[['SPEED']], ax = axes[1])\n",
    "sns.boxplot(data=strike_df[['AC_MASS']], ax = axes[2])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(ncols=3, figsize=(12, 3))\n",
    "for i, feature in enumerate(numeric_features):\n",
    "    sns.histplot(data=strike_df[feature], ax=axs[i], bins=20)\n",
    "   \n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.regplot(data = strike_df, y = \"SPEED\", x = \"HEIGHT\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Remove outliers so that:  `SPEED<600` & `HEIGHT<20000`\n",
    ">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section4'></a>\n",
    "## 4. Same stats, different graphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/DatasaurusDozen.tsv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(url, sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['dataset'].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataset statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('dataset').agg(['count', 'mean', 'std'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can you say about the mean, std, and number of points in each dataset?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    ">\n",
    "> Create 3 boxplot figures. For : `dataset == 'slant_down'` `dataset == 'star'` `dataset == 'circle'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,3, figsize = (12,4))\n",
    "plt.subplots_adjust(wspace = 0.5)\n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets display the histograms, but do it in a better way, using FacetGrid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FacetGrid is designed to split your data in several categories and plot the same relationship with the same plotting function across all categories for easy comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(df, col=\"dataset\", hue=\"dataset\", col_wrap=4, height = 2)\n",
    "g.map(sns.histplot, 'x')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The same, but with scatterplots instead of histplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(df, col=\"dataset\", hue=\"dataset\", col_wrap=4, height =2)\n",
    "g.map(sns.scatterplot, \"x\", \"y\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    ">### Functions covered in this unit:\n",
    ">\n",
    "> `scatterplot` - (x,y) points on the graphs\n",
    ">\n",
    "> `lineplot` - simple lineplot\n",
    ">\n",
    "> `plt.figure(fixsize(m,n))` - set the size of the graph\\figure to (m,n)\n",
    ">\n",
    "> `boxplot` - create a boxplot\n",
    ">\n",
    "> `reset_index` - reset index to a numerical index beginning at 0\n",
    ">\n",
    "> `sort_values()` - sorts values \n",
    ">\n",
    "> `histplot` - create a histogram\n",
    ">\n",
    "> `std()` - standard deviation\n",
    ">\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}