14 - Hypothesis Testing.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "name": "14 - Hypothesis Testing.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/wcj365/python-stats-dataviz/blob/master/14%20-%20Hypothesis%20Testing.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oa4c8ATJFdW7",
        "colab_type": "text"
      },
      "source": [
        "## 14 - Hypothesis Testing\n",
        "\n",
        "**Null Hypothesis:** Status Quo\n",
        "\n",
        "**Alternative Hypothesis:** Challenging the status quo\n",
        "\n",
        "### One-Sample t-Test\n",
        "A one-sample t-test checks whether a sample mean differs from the population mean.\n",
        "\n",
        "### Two-Sample t-Test\n",
        "A two-sample t-test investigates whether the means of two independent data samples differ from one another. In a two-sample test, the null hypothesis is that the means of both groups are the same."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lRMLVL2SFdW-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "import scipy.stats as stats\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yMZDCdtfFdXL",
        "colab_type": "code",
        "outputId": "605b97be-f16d-498f-cf2d-64da025633c2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 728
        }
      },
      "source": [
        "# Generate three random samples from normal distribution.\n",
        "# rvs stands for \"random variable sample\"\n",
        "# The inputs are (population mean, population standard deviation, sample size)\n",
        "\n",
        "sample_A = stats.norm.rvs(10000, 1000, 50)\n",
        "sample_B = stats.norm.rvs(1000, 100, 50)\n",
        "sample_C = stats.norm.rvs(1000, 100, 50)\n",
        "\n",
        "print(\"Sample A:\\n\\n\", sample_A, end=\"\\n\\n\")\n",
        "print(\"Sample B: =\\n\\n\", sample_B, end=\"\\n\\n\")\n",
        "print(\"Sample C: =\\n\\n\", sample_C)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sample A:\n",
            "\n",
            " [11160.80984163  9897.15605455  9917.85807547  7207.45721356\n",
            "  9044.63900747  9708.01468765  9814.66261502 11212.4509415\n",
            "  9781.42686519 11230.99268175  9683.88885264 11407.7875564\n",
            "  9611.69452933  9572.65961423 10942.26233215 12019.97828666\n",
            " 10560.91872098  8792.77569039  9467.34957794 10635.89734731\n",
            " 10401.78021359 11378.53257134 11077.45005989  9446.35531629\n",
            " 11885.4180422  11199.01227669  9827.50478455  9006.84111528\n",
            "  9475.23153781  9863.81254383  9833.30848997 10470.77754384\n",
            "  9569.99617606  9423.63220264  9045.16925518  9606.65397477\n",
            " 10293.65403821 11276.30980596 10526.23972395 10773.0072756\n",
            " 10971.59482502 11198.34729122  9262.69635057 11886.74650527\n",
            "  8779.90343026  8703.77020232 10279.87608205  9742.93844366\n",
            " 10748.94363695 10633.7913478 ]\n",
            "\n",
            "Sample B =\n",
            "\n",
            " [ 986.33899723  919.60244233 1200.47068717  889.0902896  1087.63889928\n",
            "  822.22279957  952.68589368 1010.73056979 1142.9187592  1096.48661537\n",
            " 1271.97007861  902.84377869 1166.08454648  843.29867575  890.32356969\n",
            "  978.77198084  954.71647625 1008.48455951  968.87882929 1150.58890347\n",
            "  970.1523125  1019.59713997 1056.08100861 1026.22720067  902.41272648\n",
            " 1081.21481183  935.40735703  917.87607526  894.71865778  830.71855704\n",
            "  940.96842889  844.22014759 1018.96223769  921.33719649  945.82100402\n",
            " 1024.44712696 1171.14056294 1095.84695427 1155.10438525 1078.23725316\n",
            " 1068.45879302 1100.50356516 1103.80484434  898.72305296  925.1506372\n",
            "  852.3225874  1009.53501444  926.14413526  978.91327944 1074.15873582]\n",
            "\n",
            "Sample C =\n",
            "\n",
            " [ 749.6451396   750.76703108  887.93495106 1077.9065379  1021.58877028\n",
            "  851.42467834  876.93539016  867.66091971  892.03160627  943.14585046\n",
            " 1003.78149424 1071.90193191  911.91973343  812.3718779   940.65948924\n",
            " 1110.62715001 1044.74648764 1057.65439663 1036.68503004 1073.70163322\n",
            " 1020.86957318 1033.40611325  911.17510464 1052.33268148 1093.0409637\n",
            " 1007.94471043 1020.53655415 1142.33588416 1063.60588245  935.51824607\n",
            " 1129.47412781 1158.71932506 1059.43615861  778.84132841  942.69616289\n",
            " 1026.25050788 1056.99891832 1124.21839713  837.58471718 1192.49324717\n",
            "  902.38762606 1191.6616748  1036.66327648  955.20790491  922.20264413\n",
            " 1102.78469832  861.72688617 1078.93859513 1153.65322902 1061.63589436]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jBUetD5zFdXW",
        "colab_type": "code",
        "outputId": "6c0da059-b5cb-4a5b-b115-b6f3f7fa85bb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        }
      },
      "source": [
        "print(\"Sample mean of sample A =\", round(sample_A.mean(),0))\n",
        "print(\"Sample mean of sample B =\", round(sample_B.mean(),0))\n",
        "print(\"Sample mean of sample C =\", round(sample_C.mean(),0))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sample mean of sample A = 10185.0\n",
            "Sample mean of sample B = 1006.0\n",
            "Sample mean of sample C = 1012.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mD3OT-MNFdXd",
        "colab_type": "text"
      },
      "source": [
        "### One Sample t-Test\n",
        "\n",
        "H0: the mean of population that sample A drawn from is 0\n",
        "\n",
        "Ha: the mean of population that sample B drawn from is not 0\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V8tPHMoQFdXe",
        "colab_type": "code",
        "outputId": "974c76df-af5c-4502-dba7-7f23c669fbd4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "stats.ttest_1samp(sample_A, 0)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_1sampResult(statistic=73.61093736981869, pvalue=7.768664898302252e-52)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WiqpIbZFdXl",
        "colab_type": "text"
      },
      "source": [
        "p-value is the probability of having the sameple under the hull hypothesis.\n",
        "If the population mean is 0 (null hypothesis), then the chance for the sample to have a mean of 10185 is very very slim - almost impossible."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uwT_PbNzHzQp",
        "colab_type": "text"
      },
      "source": [
        "H0: the mean of population that sample A sampled from is 9500\n",
        "\n",
        "Ha: the mean of population that sample A sampled from is not 9500"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KGDLn7QdHllN",
        "colab_type": "code",
        "outputId": "f8977c32-d997-4365-976e-2f3c8c3e90a8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "stats.ttest_1samp(sample_A, 9500)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_1sampResult(statistic=4.81701903598267, pvalue=1.4446299243790565e-05)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vYTPPfMuICSw",
        "colab_type": "text"
      },
      "source": [
        "p-value is the probability of having the sameple under the hull hypothesis. If the population mean is 9500 (null hypothesis), then the chance for the sample to have a mean of 10185 is very slim."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bhtmZXyUdr0f",
        "colab_type": "text"
      },
      "source": [
        "H0: the mean of population that sample A sampled from is 9950\n",
        "\n",
        "Ha: the mean of population that sample A sampled from is not 9950"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZiOFqdgJIBYj",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "4a1fb030-7877-447a-91c3-5c9368ab05b8"
      },
      "source": [
        "stats.ttest_1samp(sample_A, 9950)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_1sampResult(statistic=1.558359746485175, pvalue=0.12558402033014057)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMpjBJKUdieW",
        "colab_type": "text"
      },
      "source": [
        "p-value is the probability of having the sameple under the hull hypothesis. If the population mean is 9950 (null hypothesis), then the chance for the sample to have a mean of 10185 is about 12.6%, not slim. So we are unable to reject the null hypothesis. So we would conclude that given the evidence (sample data) the population mean is likely to be 9995 (we are still not 100% certain)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U7K0BHJSFdXm",
        "colab_type": "text"
      },
      "source": [
        "### Two Sample Test - sample A vs sample B\n",
        "\n",
        "H0: The populations that sample A and B were sampled from have the same mean\n",
        "\n",
        "Ha: The populations that sample A and B were sampled from have different means"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1_tAewhhFdXn",
        "colab_type": "code",
        "outputId": "e0d71d94-012e-4a47-bec6-b7cfd8718390",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "stats.ttest_ind(sample_A,sample_B)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=65.9752975108562, pvalue=5.091613568499395e-83)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RYXUGMhhFdXr",
        "colab_type": "text"
      },
      "source": [
        "With very small tiny p-value, we reject the null hypothesis and accept the alternative hypothesis."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnmnctInFdXs",
        "colab_type": "text"
      },
      "source": [
        "### Two Sample Test - sample B vs C\n",
        "\n",
        "H0: The populations that sample B and C were sampled from have the same mean\n",
        "\n",
        "Ha: The populations that sample B and C were sampled from have different means"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dms6WRlQFdXt",
        "colab_type": "code",
        "outputId": "b5c6ae55-7bf6-417c-913a-5fe81be0dd0b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "stats.ttest_ind(sample_B,sample_C)"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=0.15836381504601296, pvalue=0.8744959998730909)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oHmrR_SaFdXx",
        "colab_type": "text"
      },
      "source": [
        "With very large p-value, we are unable to reject the null hypothesis. So we accept the null hypothesis. We have enough evidence to believe sample B and C were drawn from populations with the same mean."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eOYq73gpFdXy",
        "colab_type": "text"
      },
      "source": [
        "### Two Sample Test - sample D vs E \n",
        "\n",
        "Let's make the population means different (1000 vs 1005)\n",
        "\n",
        "H0: The populations that sample D and E were sampled from have the same mean\n",
        "Ha: The populations that sample D and E were sampled from have different means"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XwqyXgTmFdXz",
        "colab_type": "code",
        "outputId": "04790ffd-aec9-421f-842d-a97dd4e9775a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "sample_D = stats.norm.rvs(1000, 100, 50)\n",
        "sample_E = stats.norm.rvs(1005, 100, 50)\n",
        "\n",
        "stats.ttest_ind(sample_D, sample_E)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=0.3957814119673394, pvalue=0.6931266064232611)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F8l4lsL6FdX2",
        "colab_type": "text"
      },
      "source": [
        "Still relatively large p-value, so unable to reject the null hypothesis. \n",
        "Even though the population means are different but the difference is not statistically significant.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yGLtsuoIFdX3",
        "colab_type": "text"
      },
      "source": [
        "### Two Sample Test - sample D vs E \n",
        "\n",
        "Let's make the population means somewhat more different (1000 vs 1010)\n",
        "\n",
        "H0: The populations that sample D and E were sampled from have the same mean\n",
        "Ha: The populations that sample D and E were sampled from have different means"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgBe09SoFdX3",
        "colab_type": "code",
        "outputId": "47049573-70ed-436b-a2b4-a6981f0992e1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "sample_D = stats.norm.rvs(1000, 100, 50)\n",
        "sample_E = stats.norm.rvs(1015, 100, 50)\n",
        "\n",
        "stats.ttest_ind(sample_D, sample_E)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=-0.9016450519626695, pvalue=0.3694560773602692)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nX5k9y7MFdX8",
        "colab_type": "text"
      },
      "source": [
        "The p-value is smaller but still greater than 0.05 (the standard used in social science research). We will not reject the null hypothesis. We conclude the populations are not significantly different."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tj2D2tLuFdX9",
        "colab_type": "text"
      },
      "source": [
        "### Two Sample Test - sample D vs E \n",
        "\n",
        "Let's make the population means even more different (1000 vs 1030)\n",
        "\n",
        "H0: The populations that sample D and E were sampled from have the same mean\n",
        "Ha: The populations that sample D and E were sampled from have different means"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZEWy5azZFdX9",
        "colab_type": "code",
        "outputId": "9d9e64b5-e693-45df-d895-6a99870bcf6d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "sample_D = stats.norm.rvs(1000, 100, 50)\n",
        "sample_E = stats.norm.rvs(1030, 100, 50)\n",
        "\n",
        "stats.ttest_ind(sample_D, sample_E)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Ttest_indResult(statistic=-2.0660955236732907, pvalue=0.04145771923675248)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sIw3ACs8FdYB",
        "colab_type": "text"
      },
      "source": [
        "Now, we have a p-value that is less than standard 0.05.\n",
        "We can reject the null hypothesis and state that the means of the two populations are not the same."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4vcuA6zFdYB",
        "colab_type": "text"
      },
      "source": [
        "### The End"
      ]
    }
  ]
}