Class Notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Mean\n",
    "The sample mean from a group of observations is an estimate of the mean of the population. For a number, m, of discrete elements means, z.\n",
    "$$ \\bar{z} = {1\\over m} \\sum^{m}_{i=1}z_{i} = {1\\over m}[z_{i}+...+z_{m}] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Covariance\n",
    "$$ cov(z_{i}, z_{j}) = E[(z_{i}-\\mu_{i})(z_{j}, \\mu_{j}) $$\n",
    "##### Correlation Coefficient\n",
    "Otherwise known as the Pearson coefficient:\n",
    "$$ -1 \\leq \\rho_{y} = {{cov(z_{i},z_{j})}\\over{\\sigma_{i}\\sigma_{j}}} \\leq 1 $$\n",
    "Keep in mind that independence does represent non-correlation BUT non-correlation does not necessarily mean independence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Law of Large #s\n",
    "#### Proof: Chebyshev Inequality\n",
    "Theorem: Let [X<sub>1</sub>, X<sub>2</sub>,..., X<sub>m</sub>] be m independently and identically distributed (iid) random variables with a finite mean, mu, and finite variance sigma^2:\n",
    "\n",
    "Proof:\n",
    "$$ \\forall \\epsilon > 0 \\ \\ P[|X-\\mu_{x}| \\geq \\sigma \\epsilon] \\leq {1 \\over \\epsilon^{2}} $$\n",
    "Let x be the sample mean x-bar.\n",
    "$$ E[\\bar{X}_{m}) = \\mu_{x} $$\n",
    "$$ var(\\bar{x}_{m})= {\\sigma^{2}_{x}\\over m} = {\\sigma^{2}\\over x_{m}} $$\n",
    "$$ \\forall \\epsilon > 0 \\ \\ P[|\\bar{x}_{m}-\\mu_{x}| \\geq \\sigma \\epsilon] \\leq {1 \\over \\epsilon^{2}} $$\n",
    "Finish Later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Least Squares\n",
    "The sample mean is a leas-square estimation of the true meal of the ???. Given a sample of size m consider the following model:\n",
    "$$ z = \\theta_{t} + e $$\n",
    "Where theta_t is the true value of a parameter of location, for example the true mean, mu_z. e is a random error assumed to be a random variable:\n",
    "$$ z_{1} = \\theta_{t} + e_{1} $$\n",
    "$$ z_{2} = \\theta_{t} + e_{2} $$\n",
    "$$ ... $$\n",
    "$$ z_{k} = \\theta_{t} + e_{k} $$\n",
    "Repalce theta_t with a parameter of location, theta. Then, the model comes down to:\n",
    "$$ z = \\theta + r $$\n",
    "Where\n",
    "$$ r = z - \\theta $$\n",
    "is called the residual. Consider the residuals as essentially the difference between the true value, theta, and the random variable z's that occur.\n",
    "$$ \\theta \\in \\mathbb{R} $$\n",
    "theta is the unknown parameter of location.\n",
    "We will formulate the problem as follows: finding theta that minimizes:\n",
    "$$ J(\\theta) = \\frac{1}{m}\\sum^{m}_{i=1}r_{i}^{2} $$\n",
    "The least squares estimator minimizes the sum of the residuals.\n",
    "A necessary condition of optimality is:\n",
    "$$ \\frac{dJ(\\theta)}{d\\theta} = 0 $$\n",
    "$$ \\frac{dJ(\\theta)}{d\\theta} = -2\\sum^{m}_{i=1}(z_{i}-\\theta) = 0 $$\n",
    "???\n",
    "$$ \\hat{\\theta} = \\frac{1}{m} \\sum^{m}_{i=1}z_{i} = \\bar{z}_{m} $$\n",
    "Property: The least-square estimator is the maximum likelihood estimator at the Gaussian distribution.\n",
    "The Gaussian (normal) pdf:\n",
    "$$ f(x) = \\frac{1}{\\sqrt{2\\pi}\\sigma_{x}} e^{\\frac{(x-\\mu_{x})^{2}}{d\\sigma^{2}_{x}}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample Median\n",
    "Consider m observations of random variables of z. Order these m observations by increasing value to get:\n",
    "$$ z_{1} \\leq z_{2} \\leq ... \\leq z_{m} $$\n",
    "$$ v = \\frac{m}{2} + 1 $$\n",
    "Given these parameters, then:\n",
    "$$ \\hat{z}_{med} = z_{(v)} \\ for \\ odd \\ m $$\n",
    "$$ \\hat{z}_{med} = \\frac{z_{(v-1)}+z_{(v)}}{2} \\ for \\ odd \\ m $$\n",
    "**Note** The Property of *consistency* for an estimator is how well it converges to the true value.\n",
    "Property 1: By definition (construction), the sample median is a conistent estimator of the true median of the population.\n",
    "Property 2: The sample median is the Least-Absolute Value estimator on the L1-norm estimator:\n",
    "$$ J(\\theta) = \\sum^{m}_{i=1}|r_{i}| $$\n",
    "The sample median is the ML estimator at the Laplacian distribution with probability density function given by:\n",
    "$$ f(z) = \\frac{1}{2a}e^{-\\frac{|z-b|}{a}} \\, \\ \\ ; \\sigma^{2} = 2a^{2} $$"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}