Bayes Not BAEs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Bayes Not BAEs.ipynb",
      "version": "0.3.2",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "xIYlY4W1J9WD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Bayes or BAEs?\n",
        "\n",
        "Rev. Thomas Bayes           |  Salt Bae \n",
        ":-------------------------:|:-------------------------:\n",
        "![](https://upload.wikimedia.org/wikipedia/commons/d/d4/Thomas_Bayes.gif) | ![](https://media.giphy.com/media/l4Jz3a8jO92crUlWM/giphy.gif)\n",
        "\n",
        "## Where your bae at? Use bayes' theorem. \n",
        "### Seriously though. The army uses it to find [nuclear bombs](https://translatingnerd.com/2018/02/08/searching-for-lost-nuclear-bombs-bayes-rule-in-action/nuclear)\n",
        "\n",
        "Learning Objects\n",
        "1. Learn how to calculate **conditional probabilities** using Bayes' theorem. \n",
        "2. Frame a **classification problem** as a serries of conditional probabilities\n",
        "3. Make a **prediction** by choosing the species with the higest conditional probability.\n",
        "4. Illustrate the limitations of bayes' conditional probabilities and introduce the assumption of **independence** via naive bayes.\n",
        "5. *Optional:* Find bae"
      ]
    },
    {
      "metadata": {
        "id": "w-3tHZAVJ9WG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Calculating a probability\n",
        "\n",
        "We learn new information each day. In essence, we update the knowledge that we already have on a daily basis from our past experiences. Each new day that passes we update our prior beliefs. We assign a probability of events occurring in the future based on these prior beliefs. \n",
        "\n",
        "#### Bayes' Rule\n",
        "$$P(A \\ | \\ B) = \\frac {P(B \\ | \\ A) \\times P(A)} {P(B)}$$\n",
        "\n",
        "##### Components\n",
        "- $P(A \\ | \\ B)$ **Posterior**: How probrable is our hypothesis given the observed evidence? (*Note: Not directly computable*)\n",
        "- $P(B \\ | \\ A)$ **Likelihood**: How probable is the evidence given that our hypothesis is true?\n",
        "- $P(A)$ **Prior**: How probable is the hypothesis *before* observing evidence?\n",
        "- $P(B)$ **Marginal**: How probable is the new evidence under all possible hypotheses? \n",
        "\n",
        "----------------------------------------------"
      ]
    },
    {
      "metadata": {
        "id": "nn7od6dVJ9WH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Applying Bayes' Theorem to Classification\n",
        "\n",
        "![Iris Flower](https://images.unsplash.com/photo-1549719073-96ba59673da9?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=2748&q=80)\n",
        "\n",
        "Can **Bayes' theorem** help us to solve a **classification problem**, namely predicting the species of an iris?\n",
        "\n",
        "### Preparing the data\n",
        "\n",
        "We'll read the iris data into a DataFrame, and **round up** all of the measurements to the next integer. \n",
        "\n",
        "*Note:* This step is make subsequent calculations easier - do not do for other applications."
      ]
    },
    {
      "metadata": {
        "id": "pc5FjyXLJ9WI",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "gw-M3qcCJ9WR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# read the iris data into a DataFrame\n",
        "url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'\n",
        "col_names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']\n",
        "iris = pd.read_csv(url, header=None, names=col_names)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tIoMaWtjJ9WV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "#Inspect the Data\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wgGGbZIcJ9WZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# apply the ceiling function to the numeric columns\n",
        "iris.loc[:, 'sepal_length':'petal_width'] = \\\n",
        "    iris.loc[:, 'sepal_length':'petal_width'].apply(np.ceil)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "m0muBXVRJ9Wl",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "iris.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "utgkS0bfJ9Wo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Deciding how to make a prediction\n",
        "\n",
        "Let's say that I have an **out-of-sample iris** with the following measurements: **7, 3, 5, 2**. How might I predict the species?"
      ]
    },
    {
      "metadata": {
        "id": "t3E0wgd7J9Wp",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# show all the observations with features 7, 3, 5, 2\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "apfFPty_J9Wx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# count all the species for these observations\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "0cwuIpkwJ9W2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# count the species for all the observations\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "OwJBNA_vJ9XJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Let's frame this as a **conditional probability problem**: What is the probability of some particular species, given the measurements 7, 3, 5, and 2?\n",
        "\n",
        "$$P(species \\ | \\ 7352)$$\n",
        "\n",
        "We could calculate the conditional probability for **each of the three species**, and then predict the species with the **highest probability**:\n",
        "\n",
        "$$P(setosa \\ | \\ 7352)$$\n",
        "$$P(versicolor \\ | \\ 7352)$$\n",
        "$$P(virginica \\ | \\ 7352)$$"
      ]
    },
    {
      "metadata": {
        "id": "IqFANnzlJ9XL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Calculating the probability of each species\n",
        "\n",
        "**Bayes' theorem** gives us a way to calculate these conditional probabilities.\n",
        "\n",
        "Let's start with **versicolor**:\n",
        "\n",
        "$$P(versicolor \\ | \\ 7352) = \\frac {P(7352 \\ | \\ versicolor) \\times P(versicolor)} {P(7352)}$$\n",
        "\n",
        "We can calculate each of the terms on the right side of the equation:\n",
        "\n",
        "$$P(7352 \\ | \\ versicolor) = \\frac {13} {50} = 0.26$$\n",
        "\n",
        "$$P(versicolor) = \\frac {50} {150} = 0.33$$\n",
        "\n",
        "$$P(7352) = \\frac {17} {150} = 0.11$$\n",
        "\n",
        "Therefore, Bayes' theorem says the **probability of versicolor given these measurements** is:\n",
        "\n",
        "$$P(versicolor \\ | \\ 7352) = \\frac {0.26 \\times 0.33} {0.11} = 0.76$$\n",
        "\n",
        "Let's repeat this process for **virginica** and **setosa**:\n",
        "\n",
        "$$P(virginica \\ | \\ 7352) = \\frac {0.08 \\times 0.33} {0.11} = 0.24$$\n",
        "\n",
        "$$P(setosa \\ | \\ 7352) = \\frac {0 \\times 0.33} {0.11} = 0$$\n",
        "\n",
        "We predict that the iris is a versicolor, since that species had the **highest conditional probability**.\n",
        "\n",
        "### Section Summary\n",
        "\n",
        "1. We framed a **classification problem** as three conditional probability problems.\n",
        "2. We used **Bayes' theorem** to calculate those conditional probabilities.\n",
        "3. We made a **prediction** by choosing the species with the highest conditional probability.\n",
        "\n",
        "------------------------------------------"
      ]
    },
    {
      "metadata": {
        "id": "WDquuP7LJ9XT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Naive Bayes\n",
        "\n",
        "![Woman Texting](https://images.unsplash.com/photo-1525771576046-15ba04b2693b?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1650&q=80)\n",
        "\n",
        "**Business Case**: Your managers at Smartphone Inc. have asked to develop a system to bucket text messages into two categories: spam and not spam (ham). The system will be implemented on your companies products to help users identify suspicious texts. \n",
        "\n",
        "Let's begin breaking down this problem by looking at only one sample text message. We will use Bayes Theorem with a \"Naive\" assumption to make the calculation. \n",
        "\n",
        "$$P(spam \\ | \\ \\text{send money now}) = \\frac {P(\\text{send money now} \\ | \\ spam) \\times P(spam)} {P(\\text{send money now})}$$\n",
        "\n",
        "By assuming that the features (the words) are **conditionally independent**, we can simplify the likelihood function:\n",
        "\n",
        "$$P(spam \\ | \\ \\text{send money now}) \\approx \\frac {P(\\text{send} \\ | \\ spam) \\times P(\\text{money} \\ | \\ spam) \\times P(\\text{now} \\ | \\ spam) \\times P(spam)} {P(\\text{send money now})}$$\n",
        "\n",
        "We can calculate all of the values in the numerator by examining a corpus of **spam email**:\n",
        "\n",
        "$$P(spam \\ | \\ \\text{send money now}) \\approx \\frac {0.2 \\times 0.1 \\times 0.1 \\times 0.9} {P(\\text{send money now})} = \\frac {0.0018} {P(\\text{send money now})}$$\n",
        "\n",
        "We would repeat this process with a corpus of **ham email**:\n",
        "\n",
        "$$P(ham \\ | \\ \\text{send money now}) \\approx \\frac {0.05 \\times 0.01 \\times 0.1 \\times 0.1} {P(\\text{send money now})} = \\frac {0.000005} {P(\\text{send money now})}$$\n",
        "\n",
        "All we care about is whether spam or ham has the **higher probability**, and so we predict that the email is **spam**.\n"
      ]
    },
    {
      "metadata": {
        "id": "eOM580DkJ9XV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Importing Libraries\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.feature_extraction.text import CountVectorizer"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "B-8MmjihJ9XZ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "url = \"https://raw.githubusercontent.com/sokjc/BayesNotBaes/master/sms.tsv\"\n",
        "\n",
        "df = pd.read_csv(url, sep='\\t', header=None, names=['label', 'msg'])\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(df.msg, df.label, random_state=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ok9kKhMZJ9Xj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# How frequent are the labels?\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8hMcEIJVJ9Xq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# What are the most frequent texts?\n",
        "df.msg.describe()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "5pii-apJJ9Xx",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# convert label to a binary variable\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "y0llLxfKJ9X8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### Sample Feature Extraction Steps\n",
        "1. Learn the vocabulary of the training data\n",
        "2. Tranform the training data into a 'document-term-matrix'\n",
        "3. Transform the testing data into a 'document-term-matrix' based on vocabulary of training data.\n"
      ]
    },
    {
      "metadata": {
        "id": "P6vpkCTVJ9X-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# start with a simple example\n",
        "train_simple = ['call you tonight',\n",
        "                'Call me a cab',\n",
        "                'please call me... PLEASE 44!']\n",
        "\n",
        "# learn the 'vocabulary' of the training data\n",
        "vect = CountVectorizer()\n",
        "vect.fit(train_simple)\n",
        "vect.get_feature_names()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "CHImKpv-J9YE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# transform training data into a 'document-term matrix'\n",
        "train_simple_dtm = vect.transform(train_simple)\n",
        "\n",
        "print('Document-Term Matrix')\n",
        "print(train_simple_dtm.toarray())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Vl4ijFIRJ9YX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# examine the vocabulary and document-term matrix together\n",
        "pd.DataFrame(train_simple_dtm.toarray(), columns=vect.get_feature_names())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "QnJtqDK0J9Y7",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# transform testing data into a document-term matrix (using existing vocabulary)\n",
        "test_simple = [\"please don't call me\"]\n",
        "test_simple_dtm = vect.transform(test_simple)\n",
        "test_simple_dtm.toarray()\n",
        "pd.DataFrame(test_simple_dtm.toarray(), columns=vect.get_feature_names())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Z1qc3t90J9ZJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Guided Exercise\n",
        "Repeat the sample feature extraction steps above to create a document term matrix (dtm) for the training and test datasets of our sms data.\n",
        "\n",
        "Remember, we've already read in our data and created our X_train and X_test matrices."
      ]
    },
    {
      "metadata": {
        "id": "kKYygR83J9ZL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# instantiate the vectorizer\n",
        "vect = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pe4oIL28J9ZQ",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# learn vocabulary and create document-term matrix in a single step\n",
        "# fit_transofmr accomplishses both the \"fit\" and \"transform\" funciton in one line\n",
        "X_train_dtm = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1IpMgpJdJ9ZX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# transform testing data into a document-term matrix\n",
        "X_test_dtm = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "DOMpma8XJ9Zd",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# store feature names and examine them\n",
        "train_features = ..."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iX7mz8BpJ9Zg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Fit Naive Bayes Model\n",
        "Train a Naive Bayes model using our X_train_dtm and our y_train"
      ]
    },
    {
      "metadata": {
        "id": "oumYTe3OJ9Zh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Step 1: Import Model\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "\n",
        "# Step 2: Instiante Model\n",
        "nb = MultinomialNB()\n",
        "\n",
        "# Step 3: Fit Model\n",
        "nb.fit(X_train_dtm, y_train)\n",
        "\n",
        "# Step 4: Predict - make predictions on test data using test_dtm\n",
        "preds = nb.predict(X_test_dtm)\n",
        "preds"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "fAEQ34ILJ9Zo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Evaluate Naive Bayes Model"
      ]
    },
    {
      "metadata": {
        "id": "4r43QL6HJ9Zq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Confusion Matrix\n",
        "# compare predictions to true labels\n",
        "from sklearn import metrics\n",
        "acc = metrics.accuracy_score(y_test, preds)\n",
        "matrix = metrics.confusion_matrix(y_test, preds)\n",
        "\n",
        "print(\"Confusion Matrix\")\n",
        "print(matrix)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Z-ln0WPRJ9Zv",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Bonus: sklearn Pipelines"
      ]
    },
    {
      "metadata": {
        "id": "zwQEMffKJ9Zw",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "vect = CountVectorizer()\n",
        "nb = MultinomialNB()\n",
        "\n",
        "pipe = Pipeline([('vect',vect), ('nb', nb)])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "N-DeuebJJ9Z0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# We can train it on the raw data\n",
        "pipe.fit(X_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "7YTBR2C8J9Z5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# And then use the pipe to make predictions on raw data\n",
        "pipe.predict(['Send me lots of money now', 'you won the lottery in Nigeria'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "s9lPOlz_J9Z8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Section Summary\n",
        "- The **\"naive\" assumption** of Naive Bayes (that the features are conditionally independent) is critical to making these calculations simple.\n",
        "- The **normalization constant** (the denominator) can be ignored since it's the same for all classes.\n",
        "- The **prior probability** is much less relevant once you have a lot of features."
      ]
    },
    {
      "metadata": {
        "id": "od7OP7TKJ9Z9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Resources\n",
        "\n",
        "- The Theory That Would Not Die, Sharon Bertsch McGrayne\n",
        "- [How Bayesian Inference Works](https://brohrer.github.io/how_bayesian_inference_works.html)\n",
        "- [Naive Bayes Unfolded](https://medium.com/data-science-group-iitr/naive-bayes-unfolded-b2ab036b42b1)\n",
        "- [Count Vectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n",
        "- Data Source for this Exercise - https://archive.ics.uci.edu/ml/datasets/SMS+Spam+Collection "
      ]
    }
  ]
}