unit07_EDA_on marketing data.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EDA example - Marketing analytics\n",
    "\n",
    "<a id='top'></a>\n",
    "\n",
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/marketing.PNG\" width=\"400\"/>\n",
    "</div>\n",
    "\n",
    "\n",
    "1. [Wrangling data](#section1)\n",
    "\n",
    "   1.1 [Fix columns](#section1.1)   \n",
    "   1.2 [Missing values](#section1.2)  \n",
    "   1.3 [Melt the data](#melting)\\\n",
    "   1.4 [Outliers](#section1.4)    \n",
    "   1.5 [Transformations](#section1.5)\n",
    "  \n",
    "2. [Understand the data](#section2)\n",
    "3. [Correlations](#section3)\n",
    "4. [Categorical data](#section4)\n",
    "\n",
    "## New concepts:\n",
    "[Melting data](#melting)\\\n",
    "[One-hot encoding](#onehot)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Information on the data\n",
    "We'll work with a small marketing analytics dataset, taken from [iFood](https://www.crunchbase.com/organization/ifood)\n",
    "\n",
    "Based on the example [here](https://www.kaggle.com/jennifercrockett/marketing-analytics-eda-task-final)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://raw.githubusercontent.com/nlihin/data-analytics/main/datasets/marketing_data.csv'\n",
    "mrkt_df = pd.read_csv(url)\n",
    "mrkt_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data description is [here](https://www.kaggle.com/jackdaoud/marketing-data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1'></a>\n",
    "### 1. Wrangling data\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1.1'></a>\n",
    "#### 1.1 Fix columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may already notice something is strange with the Income column, is it aligned to the right??\n",
    "\n",
    "Let's look at the types of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So here is the problem: the 'Income' column contains extra whitespace, clean it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.columns = mrkt_df.columns.str.replace(' ', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And: the 'Income' column should be turned to numeric (float is better)\\\n",
    "regex: regular expression. We want the `$` sign treated as a string so regex should be set to False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.Income"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "---\n",
    "#### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Turn `Income` to float. You will need to get rid of the `$` and the `,` first. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sanity check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.dtypes['Income']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.Income"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1.2'></a>\n",
    "#### 1.2 Missing values\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature `Income` contains 24 null values\n",
    "\n",
    "Plot this feature to identify best strategy for imputation:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['Income'] = mrkt_df['Income'].str.replace('$', '', regex = False)\n",
    "mrkt_df['Income'] = mrkt_df['Income'].str.replace(',', '')\n",
    "mrkt_df['Income'] = mrkt_df['Income'].astype(float)\n",
    "\n",
    "sns.histplot(mrkt_df, x = 'Income')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see most of the incomes are between 0-10000. We can confirm this with a box plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(2,4))\n",
    "sns.boxplot(data = mrkt_df, y= 'Income')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, the best will be to fill the missing values with the median value and not the mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['Income'] = mrkt_df['Income'].fillna(mrkt_df['Income'].median())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use a log scale to display the income, so that the display is centered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.histplot(mrkt_df.Income, log_scale=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='melting'></a>\n",
    "#### 1.3 Melt the data\n",
    "Melt the data into long format, so that we can explore all of it at once\\\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['ID','Year_Birth', 'Income', 'Kidhome', 'Teenhome','Recency', 'MntWines', 'MntFruits', 'MntMeatProducts', 'MntFishProducts', 'MntSweetProducts', 'MntGoldProds', 'NumDealsPurchases', 'NumWebPurchases', 'NumCatalogPurchases','NumStorePurchases', 'NumWebVisitsMonth' ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "melted_mrkt_df = pd.melt(mrkt_df[features], id_vars = 'ID', value_vars = features)\n",
    "melted_mrkt_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1.4'></a>\n",
    "#### 1.4 Noisy data - outliers\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to skim through the data and see if it looks ok. \n",
    "\n",
    "So plot boxplots for data that is numeric, except binary data and the ID's"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select only numeric columns:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are interested in creating a different boxplot for each `variable`, which is why `col=\"variable\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = sns.FacetGrid(data = melted_mrkt_df, col=\"variable\",  col_wrap=4, height=2.5)\n",
    "g.map_dataframe(sns.boxplot, y = 'value')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### <span style=\"color:blue\"> Exercise:</span>\n",
    ">\n",
    "> It's hard to read this, because axis y is shared\n",
    ">\n",
    "> What can we do??\n",
    ">\n",
    "> Hint: google `FacetGrid`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Treat the outliers\n",
    "> Do something with the birth years\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(3,3))\n",
    "sns.boxplot( y = 'Year_Birth', data = mrkt_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.Year_Birth.sort_values(ascending=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df = mrkt_df[mrkt_df.Year_Birth>1922].copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(3,3))\n",
    "sns.boxplot( y = 'Year_Birth', data = mrkt_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section1.5'></a>\n",
    "#### 1.5 Transformations\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change the date column to a date type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['Dt_Customer'] = pd.to_datetime(mrkt_df['Dt_Customer'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section2'></a>\n",
    "### 2. Understand the data\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find patterns in the data, we need to find correlations in the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs = mrkt_df.corr(method = 'kendall', numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs = mrkt_df.corr(method = 'kendall', numeric_only=True)\n",
    "plt.figure(figsize=(15,8))  #figure size\n",
    "sns.heatmap(corrs, cmap='coolwarm', center=0, annot = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Difficult to look for correlations in this way. So we need to first transform the data, aggregate some of the fields"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total number of dependents in the home ('Dependents') can be engineered from the sum of 'Kidhome' and 'Teenhome'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['Dependents'] = mrkt_df['Kidhome'] + mrkt_df['Teenhome']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The year of becoming a customer ('Year_Customer') can be engineered from 'Dt_Customer'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['Year_Customer'] = pd.DatetimeIndex(mrkt_df['Dt_Customer']).year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total amount spent ('TotalMnt') can be engineered from the sum of all features containing the keyword 'Mnt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnt_cols = ['MntWines', 'MntFruits','MntMeatProducts', 'MntFishProducts', 'MntSweetProducts','MntGoldProds']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have used `.sum()` to sum by columns. Now we want to sum rows. So `axis=1`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df['TotalMnt'] = mrkt_df[mnt_cols].sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total purchases ('TotalPurchases') can be engineered from the sum of all features containing the keyword 'Purchases'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "purchases_cols = ['NumDealsPurchases', 'NumWebPurchases','NumCatalogPurchases', 'NumStorePurchases']\n",
    "mrkt_df['TotalPurchases'] = mrkt_df[purchases_cols].sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The total number of campains accepted ('TotalCampaignsAcc') can be engineered from the sum of all features containing the keywords 'Cmp' and 'Response' (the latest campaign)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "campaigns_cols = ['AcceptedCmp3', 'AcceptedCmp4', 'AcceptedCmp5', 'AcceptedCmp1','AcceptedCmp2'] + ['Response'] # 'Response' is for the latest campaign\n",
    "mrkt_df['TotalCampaignsAcc'] = mrkt_df[campaigns_cols].sum(axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at our new columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df[['ID', 'Dependents', 'Year_Customer', 'TotalMnt', 'TotalPurchases', 'TotalCampaignsAcc', 'NumDealsPurchases']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section3'></a>\n",
    "### 3. Correlations - patterns in the data\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now we are ready to search for correlations again"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To understand what type of correlation, plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ['ID', 'Income', 'Dependents','TotalMnt','TotalPurchases', 'TotalCampaignsAcc', 'NumWebVisitsMonth', 'NumWebPurchases', 'NumDealsPurchases']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "melted_mrkt_df = pd.melt(mrkt_df[features], id_vars = 'ID', value_vars = features)\n",
    "melted_mrkt_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand what type of correlation, plot the histograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ggg = sns.FacetGrid(data = melted_mrkt_df, col=\"variable\",  col_wrap=4, sharex = False, sharey= False, height = 2.5)\n",
    "ggg.map_dataframe(sns.histplot, x = 'value')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not even close to a normal distribution, so use Kendall-tau or Spearman instead of the default Pearson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### New correlation heatmap:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs = mrkt_df[features].corr(method = 'kendall')\n",
    "\n",
    "plt.figure(figsize=(10,5))  #figure size\n",
    "sns.heatmap(corrs, cmap='coolwarm', center=0, annot = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found some correlations :-)\n",
    "\n",
    "Let's begin with income and spendings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.regplot(x='Income', y='TotalMnt', data=mrkt_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the same, but remove the outliers:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Plot the same graph, but remove the outlier\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:green\"> Observation 1:</span>\n",
    ">\n",
    ">The higher the income is, the more you spend\n",
    ">\n",
    "> (a bit trivial, not a very good observation)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A plot illustrating negative effect of having dependents (kids & teens) on spending:\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,3))\n",
    "sns.regplot(x='Dependents', y='TotalMnt', data=mrkt_df);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A linear plot doesn't look good here since data is discrete (same for ordinal data - e.g. - none, few, many)\n",
    "\n",
    "---\n",
    "\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Try `boxplot` instead. What looks better?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,4))\n",
    "#YOUR CODE HERE\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A plot illustrating positive effect of having dependents (kids & teens) on number of deals purchased:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(4,4))\n",
    "sns.boxplot(x='Dependents', y='NumDealsPurchases', data=mrkt_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "plot side-by-side so it will be easier to see"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,2, figsize = (10,5))\n",
    "plt.subplots_adjust(wspace = 0.3)\n",
    "sns.boxplot(x='Dependents', y='TotalMnt', data=mrkt_df, ax = ax[0])\n",
    "sns.boxplot(x='Dependents', y='NumDealsPurchases', data=mrkt_df, ax = ax[1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:green\"> Observation 2:</span>\n",
    "\n",
    ">\n",
    ">People with more kids spend less\n",
    ">\n",
    ">People with more kids buy more deals\n",
    "\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plots illustrating the positive effect of campaigns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5.5,4))\n",
    "sns.boxplot(x='TotalCampaignsAcc', y='TotalMnt', data=mrkt_df);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### <span style=\"color:green\"> Observation 3:</span>\n",
    ">\n",
    ">Campaigns seem to be working\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### <span style=\"color:green\"> Observation???</span>\n",
    ">\n",
    "> Campigns don't seem to corrolate with deals, even for different countries (this may be a good thing)\n",
    ">\n",
    "> This is not really an observation, since it doesn't focus on finding a new connection. It is only an observation if the company currently believes that there is a connection and you're proving them wrong.\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about web visits and web purchases?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(1,2,figsize=(8, 4))\n",
    "sns.regplot(x='NumWebVisitsMonth', y='NumWebPurchases', data=mrkt_df, ax=ax[0])\n",
    "sns.regplot(x='NumWebVisitsMonth', y='NumDealsPurchases', data=mrkt_df, ax = ax[1])\n",
    "plt.subplots_adjust(wspace = 0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### <span style=\"color:green\"> Observation 4:</span>\n",
    ">\n",
    "> Number of web visits in the last month is not positively correlated with number of web purchases\n",
    ">\n",
    "> Instead, it is positively correlated with the number of deals purchased, suggesting that the website is effective in stimulating purchases\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='section4'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a id='section4'></a>\n",
    "### 4. Categorical data\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<div>\n",
    "<img src=\"https://raw.githubusercontent.com/nlihin/data-analytics/main/images/categories.jpeg\" width=\"500\"/>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets look at our non-numeric data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_numeric_columns = mrkt_df.select_dtypes(exclude=['int64', 'float64'])\n",
    "non_numeric_columns.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three of these features are categorical: `Education`, `Marital_Status`, `Country`\n",
    "\n",
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "> Visualize these features, using `sns.countplot`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(figsize=(20, 5), ncols=3)\n",
    "\n",
    "#YOUR CODE HERE\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the marital status:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mrkt_df[[\"Marital_Status\"]].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Is there a correlation between marital status and spendings?\n",
    "\n",
    "Create a df that holds only the status's we're interested in - the four biggest categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status_mrkt = mrkt_df.loc[(mrkt_df.Marital_Status  == 'Single')|(mrkt_df.Marital_Status == 'Married')|(mrkt_df.Marital_Status == 'Together')|\\\n",
    "                          (mrkt_df.Marital_Status == 'Divorced')].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='onehot'></a>\n",
    "#### One hot encoding\n",
    "[back to top](#top)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create one-hot encodings for the categorical variables\n",
    "\n",
    "Note: use smartly, so as not to add too many dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features2 = ['Income', 'Dependents','TotalMnt','TotalPurchases', 'TotalCampaignsAcc',\\\n",
    "             'NumDealsPurchases', 'NumWebVisitsMonth', 'NumWebPurchases', 'Marital_Status', 'Response']\n",
    "\n",
    "status_mrkt_with_dummies = pd.get_dummies(status_mrkt[features2])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status_mrkt_with_dummies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One-hot encoding doesn't affect variables that are not categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrs = status_mrkt_with_dummies.corr(method = 'kendall')\n",
    "\n",
    "plt.figure(figsize=(10,7))  #figure size\n",
    "sns.heatmap(corrs, cmap='coolwarm', center=0, annot = True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### <span style=\"color:green\"> Observation???</span>\n",
    ">\n",
    "> There isn't any correlation between martial staus and spendings\n",
    ">\n",
    ">##### Observations should focus on what there IS, not what there isn't\n",
    ">\n",
    ">Let's try to look at the response to campgains\n",
    "\n",
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <span style=\"color:blue\"> Exercise:</span>\n",
    "Is there a connection between the status, dependents and response?\n",
    "> groupby `Marital_Status` and `Dependents` to find out\n",
    "> use `unstack()` on the result to create a table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can use a pivot table to obtain the same results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "status_mrkt.pivot_table('Response', index='Marital_Status', columns='Dependents', aggfunc='mean')  #aggfunc = 'mean' is the default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why would we want a table? Because with a table it's easier to figure out what's going on and what we should plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Single & Divorced with no kids are more likly to respond to a campaign"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(5,4))\n",
    "#plt.xticks(rotation=90)\n",
    "ax = sns.barplot(data = status_mrkt, x='Marital_Status', y='Response', errorbar=None, hue = 'Dependents')\n",
    "ax.set(ylabel='average response')\n",
    "plt.legend(title='Dependents', loc=('upper right')) #the legend position\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### <span style=\"color:green\"> Observation 5:</span>\n",
    ">\n",
    "> Single & Divorced with no kids are more likly to respond to a campaign - average response is higher than 30%\n",
    ">\n",
    "> Married & Together with no kids respond at around 20%\n",
    ">\n",
    "\n",
    " \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is this it? No!! There is always more to do. We haven't touched country, education, or campaign responses and much more. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "> A summary of new functions:\n",
    ">\n",
    ">* `drop()` - remove labels or columns [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.drop.html?highlight=drop#pandas.DataFrame.drop)\n",
    ">\n",
    ">* `str.replace('x','y')` - replace string x with string y\n",
    ">\n",
    ">* `reset_index(drop=true)` - resets the index. As defult, the old index is added as a column. Use drop=True to avoid this. [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html)\n",
    ">\n",
    ">* `plt.subplots_adjust` - a matplotlib function. adjusts space between the plots. [documentation](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.subplots_adjust.html)\n",
    ">\n",
    ">* `sum(axis=1)` - sums the values according to the rows, instead of the default sum by columns\n",
    ">\n",
    ">* create a list using [list comprehension](https://docs.python.org/3.4/tutorial/datastructures.html#list-comprehensions) - \"do something for data in dataframe if condition holds\"\n",
    ">\n",
    ">* `catplot()` - plot categorical variables onto a grid. [documentation](https://seaborn.pydata.org/generated/seaborn.catplot.html)\n",
    ">\n",
    ">* `countplot()` - counts observations for each category (this is like a histogram, but for categorical data)\n",
    ">\n",
    ">* `pd.get_dummies()` - creates one-hot encodings for categorical variables. [documentation](https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html)\n",
    ">\n",
    ">* `unstack()` - change the hiearachy of data. [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.unstack.html)\n",
    ">\n",
    ">* `pivot()` - create a pivot table to group and summarize data. [documentation](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html)\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}