Lecture 2 Model Linear Classification and Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2 Model Linear Classification and Regression\n",
    "ChiHackNight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear models make a prediction using a linear function of the input features.\n",
    "\n",
    "#### Linear Models for Regression\n",
    "The output is called the Y hat and is the prediction the model makes. The features are X[0]...X[p] where 0 and p are the features of a single data point W and B are parameters of the model that are learned. \n",
    "\n",
    "If you have only one feature the prediction formula looks like: \n",
    "Yhat=w[0]\\*x[0]=b\n",
    "Which is the equation for a line. w[0] = the slope and b is the y intercept or the y-axis offset.\n",
    "As we add more features, w contains the slopes along each feature axis. \n",
    "\n",
    "The predicted response is the weighted sum of the input features, with weights (which can be negative) given by the enteries of w.\n",
    "\n",
    "Lets learn the parameters for w[0] and b for the simple one dimensional synthetic wave dataset (we will get a line)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We are importing and also printing out the versions of these main libraries:\n",
      "scikit-learn: 0.19.1\n",
      "numpy: 1.14.2\n",
      "pandas: 0.23.0\n",
      "can;t seem to grab the matplotlib version: matplotlib.__version__\n",
      "keras: 2.1.5\n",
      "tensorflow: 1.7.0\n",
      "seaborn: 0.8.1\n",
      "mglearn: 0.1.6\n"
     ]
    }
   ],
   "source": [
    "print('We are importing and also printing out the versions of these main libraries:')\n",
    "import sklearn\n",
    "print('scikit-learn: ' + sklearn.__version__)\n",
    "import numpy as np\n",
    "print('numpy: '+ np.__version__)\n",
    "import pandas as pd\n",
    "print('pandas: ' + pd.__version__)\n",
    "import matplotlib.pyplot as plt\n",
    "print('can;t seem to grab the matplotlib version: ' + 'matplotlib.__version__')\n",
    "import keras\n",
    "print('keras: ' + keras.__version__)\n",
    "import tensorflow\n",
    "print('tensorflow: ' + tensorflow.__version__) \n",
    "import seaborn as sns\n",
    "print('seaborn: ' + sns.__version__)\n",
    "import mglearn\n",
    "print('mglearn: ' + mglearn.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w[0]: 0.393906  b: -0.031804\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdMAAAGuCAYAAADLQuyyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8VNW99/Hv5AI4EC4NcpHcUDFAAgYwkR4RogJawJ5in0o0rTc8WFutbZVWy6OIj1GPtU+Pnp7z4tA+VlvTBvBSBawXlNCCaAJyEZEI1twAuV8SQkIu+/kjTSBkJpmZPTN775nP+/Xqq2SyZ2ZlZdzfrL3X+i2XYRgCAACBi7G6AQAAOB1hCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpEGYul6uXy+UqcblcW10u16cul2uR1W0CYI6LdaZAeLlcLpek3oZh1LpcrnhJ6yTdZxjGhxY3DUCA4qxuABBtjNa/YGv/+WX8P//HX7WAg3GZF7CAy+WKdblcWyQdkPSuYRgfWd0mAIHz6zLvhg0bjIaGhqC9+cmTJ9W7d++gvZ7T0R9nREtf1NbW6uGHH9aPfvQjDR8+vMP3VqxYoZUrV0qS6uvr9eKLL1rRRFuKls+HL+iLjoLdH7m5uS5fjvP3nmlQL0UVFxcrNzc3mC/paPTHGdHUF4sWLVLv3r31wAMPeD0mPT1dZWVlYWyVvUXT56M79EVHIegPn8KUy7xAmB08eFDHjh2TJJ06dUqrV6/WyJEjLW4VADOYgASE2b59+3TrrbequblZLS0tuvHGGzVr1iyrmwXABMIUCLOxY8dq8+bNVjcDQBARpgBggcbGRlVXV6u+vt7U6/Tr10+fffZZkFrlfIH2R69evZSUlKT4+PiA3pcwBQALVFdXKyEhQWlpaWqt4xGYmpoaJSQkBLFlzhZIfxiGocOHD6u6urrTrHpfMQEJACxQX1+vxMREU0GK4HC5XEpMTDR1lYAwBQCLEKT2YfZ3QZgCAExLS0vToUOHTB/jVIQpAAAmEaYAEKXKy8s1cuRI3XnnncrMzFR+fr5Wr16tK664QiNGjFBJSYmOHDmib33rWxo7dqwmTpyobdu2SZIOHz6s6dOna9y4cbrrrrt0djW9l156STk5OcrKytJdd92l5uZmq37EsGE2LwBYbNGKT7Vj74mAntvc3KzY2NhOj4++oK8WXp/R7fN3796t5cuXa8mSJcrOztaf/vQnrVu3Tm+88YaeeOIJJScna9y4cfrLX/6i999/X7fccou2bNmiRYsWadKkSXrkkUe0atUqLVmyRJL02WefaenSpVq/fr3i4+P1gx/8QIWFhbrlllsC+vmcgjAFgCg2fPhwjRkzRpKUkZGha665Ri6XS2PGjFF5ebkqKir0yiuvSJKuvvpqHT58WMePH9ff/vY3vfrqq5KkmTNnasCAAZKk9957T5s2bVJ2drak1pKZgwYNsuAnCy/CFAAs5ssI0huz60x79uzZ/u+YmJj2r2NiYtTU1KS4uM4x0Tbz1dMMWMMwdOutt+rJJ58MuE1OxD1TAIBXkydPVmFhoaTWHVkGDhyovn37dnj8r3/9q44ePSpJuuaaa/Tyyy/rwIEDkqQjR46ooqLCmsaHESNTAIBXjz76qG6//XaNHTtWbre7fV/dhQsX6qabbtL48eM1ZcoUpaSkSJJGjx6txx9/XNOnT1dLS4vi4+P1X//1X0pNTbXyxwg5whQAolRaWpq2b9/e/vULL7zg8Xuvv/56p+cmJibqnXfeaf/617/+dfu/58yZozlz5nR6Tnl5eRBabU9c5gUAwCTCFAAAkwhTAABMIkwBADCJMAUAwCTCFAAAkwhTAIhCx44d03//938H9NwZM2bo2LFjXR7zyCOPaPXq1QG9fldeeOEF3XPPPV0eU1xcrA8++CDo790VwhQAHKCwsEJpaasUE7NcaWmrVFhorqpQV2Ha3S4vb775pvr379/lMY899pimTp0acPvMIEwBAJ0UFlZo3rxNqqiok2FIFRV1mjdvk6lAffDBB/XFF18oKytL8+fPV3Fxsa666irdfPPN7YXvv/Wtb2nChAnKyMho3xVGOrPJd3l5uUaNGqV/+7d/U0ZGhqZPn65Tp05Jkm677Ta9/PLL7ccvXLhQ48eP15gxY7Rz505J0sGDBzVt2jSNHz9ed911l1JTUz1uHv773/9el1xyiaZMmaL169e3P75ixQpdfvnlGjdunKZOnar9+/eroqJCixcv1q9//WtlZWXp73//u8fjgo0wBQCbW7Bgu+rqOo4W6+qatWDBdi/P6N5TTz2liy66SFu2bNEvf/lLSVJJSYkKCgq0Y8cOSdLzzz+vTZs2aePGjXruued0+PDhTq+za9cu/fCHP9Snn36q/v37t+8wc66BAwfq448/1t13361nnnlGkrRo0SJdffXV+vjjjzV79mxVVlZ2et6+ffu0cOFCrV+/Xu+++2572yRp0qRJ+vDDD7V582bl5eXp6aefVmpqqr7//e/rJz/5ibZs2aIrr7zS43HBRjlBALC5yso6vx4PVE5OjoYPH97+9XPPPafXXntNklRVVaVdu3YpMTGxw3OGDx+urKwsSdKECRO8lgy84YYb2o9p27pt3bp17a9/3XXXtW/jdraPPvpIubm5Ov/88yW1lir8/PPPJUnV1dWaM2eO9u3bp9OnT3do+9l8Pc4MRqYAYHMpKW6/Hg9U79692/9dXFys1atXa8OGDdq6davGjRun+vr6Ts85ewu32NhYNTU1eXzttuPOPsYwDJ/a5WmrN0m69957dc899+iTTz7R//zP/3hsnz/HmUGYAoDNFRRkyu2O7fCY2x2rgoLMgF8zISFBNTU1Xr9//PhxDRgwQG63Wzt37tSHH34Y8Ht5M2nSJC1btkyS9M4777Rv43a2yy+/XMXFxTp8+LAaGxu1fPnyDm0cNmyYJLXvZiN1/tm8HRdMhCkA2Fx+fqqWLJmg1FS3XC4pNdWtJUsmKD8/8G3NEhMTdcUVVygzM1Pz58/v9P3rrrtOTU1NGjt2rB5++GFNnDjRzI/g0cKFC/XOO+9o/Pjx+utf/6qhQ4d22uh86NChevTRR/X1r39dU6dO1fjx49u/9+ijj+o73/mOrrzySg0cOLD98euvv16vvfZa+wQkb8cFk8vXYfY/+XVwd4qLi5WbmxvMl3Q0+uMM+qKj9PR0lZWVWd0M24iEz8dnn32mUaNGmX6dmpqaTgHkFA0NDYqNjVVcXJw2bNigu+++W1u2bDH1mmb6w8vvxPM15nMwAQkAYInKykrdeOONamlpUY8ePfTb3/7W6iYFjDAFAFhixIgR2rx5s9XNCArumQIAYBJhCgAW8XPOCkLI7O+CMAUAC/Tq1UuHDx8mUG3AMAwdPnxYvXr1Cvg1uGcKABZISkpSdXW1Dh48aOp16uvrTYVApAm0P3r16qWkpKSA35cwBQALxMfHB6WsXXFxscaNGxeEFkUGq/qDy7wAAJhEmAIAYBJhCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpAAAmEaYAAJhEmAIAYBJhCgCASYQpAAAmEaZAmFVVVemqq67SqFGjlJGRoWeffdbqJgEwKc7qBgDRJi4uTr/61a80fvx41dTUaMKECZo2bZpGjx5tddMABIiRKRBmQ4cO1fjx4yVJCQkJGjVqlPbs2WNxqwCYQZgCFiovL9fmzZt1+eWXW90UACa4DMPw+eANGzYYDQ0NQXvz2tpa9enTJ2iv53T0xxnR0BenTp3Sfffdp+9+97uaPHlyp++vWLFCK1eulCQdPXpUy5YtC3cTbSsaPh++oi86CnZ/5Obmunw5zq8wleTXwd0pLi5Wbm5uMF/S0eiPMyK9LxobGzVr1ixde+21+ulPf9rt8enp6SorKwtDy5wh0j8f/qAvOgpBf/gUplzmBcLMMAzNnTtXo0aN8ilIAdgfYQqE2fr16/XHP/5R77//vrKyspSVlaU333zT6mYBMIGlMUCYTZo0SX7eXgFgc4xMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFAMAkwhQAAJMIUwAATCJMAQAwiTAFANhOYWGF0tJWKSZmudLSVqmwsMLqJnUpzuoGAABwtsLCCs2bt0l1dc2SpIqKOs2bt0mSlJ+famXTvGJkCgCwlQULtrcHaZu6umYtWLDdohZ1jzAFANhKZWWdX4/bAWEKALCVlBS3X4/bAWEKALCVgoJMud2xHR5zu2NVUJBpUYu6R5gCAGwlPz9VS5ZMUGqqWy6XlJrq1pIlE2w7+UhiNi8AwIvCwgotWLBdlZV1Sklxq6AgM2yBlp+fauvwPBdhCgDoxInLU6zEZV4AQCdOXJ5iJcIUANCJE5enWIkwBQB04sTlKVYiTAEAnThxeYqVCFMAiCK+FpDvanmK04rQhwOzeQEEzMqlE/CfvzN0PS1PYZavZ4xMAQSk7aRaUVEnwzhzUmWUYl/BmKHLLF/PCFMAAeGk6jzBmKHLLF/PCFMAAeGk6jzBmKHLLF/PCFMAAeGk6jzBmKFr11m+O/ae0MLXt6tkX5Ml70+YAgiIXU+q8C4YBeTtVIS+tqFJfy6p1L/+Zp1mPPd3/bmkSl/VtYS9HRKzeQEEqO3kyWxeZwlGAXkri9AbhqGt1cdVVFKpFVv36uTpZl0yuI8emTVas8cN09bSDyxpF2EKIGBO29kjknRclrQq4v+QOX6qUX/ZvEd/LqnUzq9qdF58rGaNHaq8nBSNT+kvl8tlafsIUwBwmGhZ62kYhkrLj6qopFKrPtmnhqYWZQ7rq8e/lal/zbpACb3irW5iO8IUABymq2VJkRCmh2sb9MrH1SoqrdI/Dp5UQs84feeyJOVlpyhzWD+rm+cRYQoADhOJy5JaWgyt231IS0ur9M6Or9TYbGhC6gD98n9dpJljh8rdw95xZe/WARHqjjvu0MqVKzVo0CBt306RA/gnJcWtiorOwWnXZUldlZ386ni9lm+s0tKNVao+ekr93fH63sQ05eUk65LBCRa33HeEKWCB2267Tffcc49uueUWq5sCByooyOxwz1Sy77Ikb/d3t+85roODG/T+zgNqMaR/uShR869N17UZQ9QrPrabV7UfwhSwwOTJk1VeXm51M+BQTlqW5O3+7i8LPlfW/P66a8pFmnNZstIG9raohcFB0QbAgdgCC/n5qSovn6n33z9f5eUzbRmkkvf7uC01hjY8dLV+ft1IxwepJLkMw/D54A0bNhgNDQ1Be/Pa2lr16dMnaK/ndPTHGdHQF1999ZUeeugh/f73v/f4/RUrVmjlypWSpKNHj2rZsmWSpNWr6/XMMzU6+z/Fnj2lBx5I0NSpvULe7u6sXl2v3/3upA4caNGgQTG6887eQW9XNHw+fGXXvthX26K11Y16/sEaNR7vnDODB8eoqCgx6O8b7P7Izc31aQGrX2Eqya+Du1NcXKzc3NxgvqSj0R9nRENflJeXa9asWT5NQEpPT1dZWZkkKS1tlcfJJ6mpbpWXzwx6O/1x7v0xqfVeXrDLzUXD58NXduqL+sZmvfnJPhWVVKmk/IjiYlxKOXqePnjhiBrqz5T56+ozYXaP3BD0h09hyj1TwGHsvCwi0tc/wrPP9p1QUUmlXtu8Ryfqm5SW6NbPrxupb08YpkEJvVQ4ybeAdHIxCsIUsMBNN92k4uJiHTp0SElJSVq0aJHmzp3r03PtvCzCzkGP4KptaNKKrXtVVFqlrVXH1CM2RtdlDlFeTrImDk9UTMyZAZ2vZSed/McYYQpY4M9//nPAz7Xzsgg7Bz3MO7vI/Btb96runCLzA3r3MPX6Tv5jjDAFHMbOyyLsHPQI3PG6Rr22ubW8XyiLzDv5jzHCFHAgu+7WYuegh38Mw1DJl0dUVFqlN8NUZN7Jf4wRpgCCyq5BD99YWWTeyX+MEaYAEOXaiswXlVbq3R37LS0y79Q/xghTAIhS5xaZH+CO1y1fT1NedrJG+Flk3uz6UKcjTAFEhGg/mfuqqblFa8oOqqikUmvKzhSZ/9l1I3VtxmD1jPO/yLyT14cGC2EKwPGccjK3MvCrjtRpaWmVlm+q0v4TDRrYp6fumnKR8rKTlZporjauk9eHBgthCjgYo7FWTjiZWxH4p5ta9O6O/SoqrdS63YfkkjTlkvP12L+m6OqRgxQfG5y9Tpy8PjRYCFPAoZwyGgsHJ5zMwxn4XxysVVFJpV75eI+OnDytYf3P033XjNCNlyXrgv7nBfW9JGevDw0WwhRwKCeMxsLFCSfzUAe+pyLzU0cNVl5Osq4ccb5iY4JTWMETJ68PDRb2MwUcygmjsXApKMiU291x4ozdTubegt1s4FfVtGjh69uVU7BaP122VQdq6vXz60bqg4eu1uLvTVBu+qCQBqnUeiVkyZIJSk11y+Vq3cHIzE5BTtyvl5Ep4FBOGI2FixMW+wdz9NaxyPwp9Yir0nUZrUXmv35hYtDK+/kjWOtDnXr7gjAFHIpLax3ZfbG/2cD3VmT+5pE9NP87U0wXmQ+nribOOfX2BWEKOJQTRmPoKJDA767I/Nq1ax0XpF2NPJ16+4IwBRzM7qMxBMaKIvPh0t3I06m3LwhTAH5hbWvoWFlkPly6G3k69fYFYQrAZ54u0X3veyX67ndLlJpKsAbCU5H5y1IH6G4LisyHQ3cjT6fevois3xKAkPJ0ic4wWv/fKbMu7SKYReadxJeRpxNvXxCmAHzW3SQQJ8y6tFIoisw7jVNHnt0hTAH4zNslurPZfdalFc4tMn9+QvCKzDuRE0ee3SFMAfjM0yW6c9l91mW4nG5q0Ts7vtLS0ir9fdchxbhCU2Qe9kCYAvDZ2ZfoKirq5HKduWcqOWPWZajtPlCrpaUdi8z/eGroiszDHghTAH45+xIdy2RanTrdWmR+aWn4i8zDHghTAAGLxHtf/tix94SKSiv12uY9qqlvUlqiWw9+Y6S+PT5J5yf0tLp5CCPCFAD80F5kvqRSW6uPq0dcjK7LGKKbclI08cKvWVJkHtYjTAGgG96KzD8ya7RmjxvmqNq4CA3CFAC88FRk/vpLh2pOdmuReUahgYu0++2EKQCcxVOR+THD+qlgdqa+eamzi8zbhVP3LO0KYQoAkg7VNujVc4rM33hZsvJykpVxQWQUmbcLp+5Z2hXCFEDUOPfS4uOPZyolxx01Rebtwql7lnaFTwqAqODp0uKtd5RowLXxSs52R0WRebtw6p6lXSFMAUSFX/zik06XFlsapZhNMfrwtWuiosi8XTh1z9KuUBwSQESrOlKnZ94uU2XlKY/fP/RVA0EaZvn5qVqyZIJSU91yuaTUVLeWLJng2PulEiNTAA5w7r3O7343Vrm53o/3VGS+d2KsTh7uXKDfyZcWnSzSqmcxMgVga233Oisq6mQYrfc6n3mmRoWFFZ2O3X2gVgWrdmjik+/pnj9t1j8OntRPpl6idT+/Wv/z7AS53R1HoHa8tFhYWKG0tFWKiVmutLRVHn9O2A8jUwC25mkZRUOD2pdRtBWZLyqtVGn5Ua9F5p2wKXUkrr+MFoQpAFvrahnFI69v96vIvN0vLUbi+stoQZgCsDVvyyhiElwqKq3SNzKHKC87MorMR+L6y2jBPVMAtlZQkKle53U8VcXES7ffl6aSX1yjZ/PG6esXJTo+SCXvk6GYJGV/hCkAWzpe16gX1n+pPx6sUO9rYhXXzyW5pCHDeunB+X3028cuU393ZO3WUlCQ6YhJUuiMy7wATAnm7h+GYeijL4+oqKRSb27/SqebWjQ2qZ+e+9/jdf2lQ9uLzBcXFwfxJ7APJ0ySgmeEKYCABWv26aHaBr2yqVpLS6v0j0OtRebnRGmRebtPkoJnhCkQAazaG9LM7NOWFkPrdh/qVGT+B1ddrJljhuq8HlQlgnMQpoDDWbk2MZDZp/uOn9Lyja2j0D3HTmmAOz5ii8xH2gbY8I4wBRzO19FhKE7svu7+0dTcovd3HtDS0iqtKTugFkO64uJEPfiNkZqeMTgia+NSgCG6EKaAw/kyOgzVib273T8qD9dp6cZKLd9YrQM1DRqU0FN3516kOZelKCUxspd7UIAhuhCmgMP5MjoM1Ynd0+zTRf9ntPqNidd3f/eR1u1uLTKfmz5IednJunrkIMXFRseKPAowRBfCFHA4X/aGDOWJvW326e4DtVpaWqlnP96tI5+e1rD+5+knUy/Rdy5L0gX9zzP9Pk4TiRtgwzvCFHA4X9YmhurE7qnI/LTRg5WXk6JJFw9sLzIfjSJxA2x4R5gCEaC7tYnBPrHv2HtCRaWV7UXmhw/s3W2R+WhDAYboQpgCUSAYJ/bahiat2LpXRSWV2lp9XD3iYjQjc4jyclJ0+XDnF5kPBQowRA/CFBGB9XzdC+TEbhiGtlQdU1FJlVZs26u6081KH5yghdeP1uxxwyKuNm4bPk/wF2EKx2M9X/Adqzut1zbv0dLSKu38qkbnxcfq+kuHKi8nReOS+0f0KJTPEwJBmMLxWM8XHN6KzD8xe0yHIvORjs8TAkGYwvFYz2cOReY74vOEQBCmcDzW8/mPIvPe8XlCIAhTOB7r+Xznqcj8rV9PU15Osi4eFFlF5gPF5wmBIEzheKzn65qnIvOTLh4Y0UXmzeDzhEAQpogIrOfrLJqLzJvF5wn+IkwBC7z11lu677771NzcrDvvvFMPPvhgUF63oalZ7+7Yr6KSqqguMg+EG2EKhFlzc7N++MMf6t1331VSUpKys7P1zW9+U6NHjw74NduKzL/y8R4dOUmReSDcCFMgzEpKSnTxxRfrwgsvlCTl5eXp9ddf9ztMKTIP2AdhirChRFurPXv2KDk5uf3rpKQkffTRRz4//9O9x1VUUqW/bDlTZP6hb4zUtyckaWAfiswDVnAZhuHzwRMnTjSOHj0atDdvbGxUfHx0VFXxRST3x4kThvbvb9bZHzeXSxo8OFZ9+3YeQUVyX9TU1OjkyZMaMmSIJOnEiROqr6/XoEGDOhx3/PhxHTt2TJLU1NSsQcnDVXPa0OkWySXJHe9SQg+XekXhZNxI/nz4i77oKNj98fnnn79tGMZ13R3nV5hK8uvg7qSnp6usrCyYL+lokdwfaWmrPC6ET011q7x8ZqfHI7kvNmzYoEcffVRvv/22JOnJJ5+UJD300EMdjjMMQ5urjmlpSZX+454bNPS2Z5U+OEF5OckRXWTeF5H8+fAXfdFRCPrDp/sljr/My6VDZ6BE2xnZ2dnatWuXvvzySw0bNkxFRUX605/+1P79c4vMu3vESqfr9NoP/kVZEV5kHnAqR4cpuzs4ByXazoiLi9NvfvMbXXvttWpubtYdd9yh0aNH68N/HPZaZP78p49oXMoAq5sOwAtLw3TWrFmmnh9puzuY7Q8787dEWyT3hSTNmDFDM2bMaC8yf/Wv1urLQyeV0CtOednJmpPdsch8//79LWyt/UT658Mf9EVHVvWHpWF6/fXXm3p+pF06NNsfduZvibZI7gtPReaz0wbonqsu1gwvReb79Yu+3Vu6EsmfD3/RFx1Z1R+OvszLpUNnifYSbRSZByKX5bXFHn74YY0dO1ZZWVmaPn269u7d6/NzCwoy5XZ3/Cveybs7LF68WCNHjtTYsWM1e/bs9mUR0Wr58uXKyMhQTEyMNm7caHVzAtLU3KJ3Pv1Kd7xQqiueel//993PNXxgb/3m5nH68BfX6H/PGt1lkL711ltKT0/Xl19+qaeeeiqMLbefO+64Q4MGDVJmpjP/+w62qqoqXXXVVbr11luVkZGhZ5991uomWaa+vl45OTm69NJLddttt2nhwoXhb4RhGP78L6jWrFljHD9+vP3rZ5991rjrrrv8eo2XXio3UlNXGi7XMiM1daXx0kvlwW5m2Dz99NNGY2OjYRiG8bOf/cz42c9+ZnGLrLNmzRpjx44dxs6dO40pU6YYpaWlVjfJLxWHThpPv/WZkf34u0bqz1ca2Y+/azz91mdGxaGTPr9GU1OTceGFFxpffPGFMWLECGPs2LHGp59+GsJW29vatWuNTZs2GRkZGcaaNWusbo7l9u7da2zatMlYs2aNceLECWPEiBFR+/loaWkxampqDMMwjHfffdfIyckxNmzYEKyX9ykfLb/M27dv3/Z/nzx50u9p/5F06TA7O1txca2/kokTJ+rll1+2uEX+C+ZSpVGjRgW5daHV0NSsdz7dr6LSSq3ffdh0kfmzyw66XK6Ayw5GismTJ6u8vNzqZtjG0KFDNXToUBUXFyshIUGjRo3Snj17ovLz4XK51KdPH0lSU1OTGhsbw76EzPIwlaQFCxboD3/4g/r166c1a9ZY3RxbeP755zVnzhyrm+EXb0uV1q8/pDff/Cpi1wLvPlCropJKvbo5uEXmzZYdRPQoLy/X5s2bdfnll1vdFMs0NzdrwoQJKisr049+9KOw90VYwnTq1Kn66quvOj2el5en3NxcFRQUqKCgQE8++aR+85vfaNGiReFoliW89UVBQUH7jM2CggLFxcUpPz8/3M0zxdtSpcWL/9FeRvDctcDdfTbsKhxF5g0P1cko2IBznTp1St/+9rf1H//xHx2u9EWb2NhYbdmyRStXrtSvfvUrbd++Paz318MSpqtXr/b4eHFxcYevb775Zs2cOTOiw9RbX0it/fHiiy9q5cqVeu+99xx34vS2JOncTDh7LbCvnw27CGeR+aSkJFVVVbV/XV1drQsuuCCo7wFna2xs1COPPKL8/HzdcMMNVjfHFvr06aPc3Fy99dZbkRemXdm1a5dGjBghSXrjjTc0cuRIi1tknZKSEr3wwgtau3at3G7nLe/xtlTJEyetBa6pb9SKra2j0G3Vx9UjLkYGQiNXAAASVElEQVQzMocoLydFlw//Wsj+6Dm77KBhGJ3KDiK6GYahuXPnKjU1VT/96U+tbo6lDh48qPj4ePXv318NDQ1avXq1fv7zn4e1DZYWui8uLtZ//ud/qqysTDExMUpNTdXixYs1bNiwYL6NYwwbNkwxMTFKTEyU1DoJafHixRa3ynfn3jOVWneG8fQR81bgvk1xcbGOHj2qe++9VwcPHlT//v2VlZXVXhw+1Ix/FpkvKqnUym37VHe62ZIi82+++aZ+/OMfq7y8XAsXLtSCBQvC8r52dNNNN6m4uFiHDh1S//799dRTT2nu3LlWN8sy69at05VXXqkLL7xQCQmty6ueeOIJzZgxw+KWhd+2bdt06623qrm5WTU1Nbr99tv1yCOPBOvlffpr2fIwtfN9sXCLhP44dzbvjBlD9OKLFZ3KCC5ZMqHLSUhW9YWnIvPXj71AeTnJlhaZZ2eQjiLhv5VgoS86CkF/RMeuMbAXT0uVrrhioK139jEMQx99ecRrkfmEXuwVCaBrhClCzq5rgQ/WNOiVj1vL+3VVZB4AukOYIqo0txj6+66DKiqp0urP9quppfsi8wDQHcIUIWWXzdv3HT+lZaXVWrbxTJH52/7FvkXmz+23hoagTlcAEGSEKULG6s3bm5pb9P7OAyoqrVJx2QG1GNKkiwfqoRkjNW30YPWMs+co1FO/uVzNKiyssOXlcgCEKULIqs3bKw/XaenGSi3fWK0DNQ0alNBTd+depDmXpSgl0f7rdz31m2HIsZveA9GAMEXIhHPzdk9F5q9KH6S8nBRdlX6+30XmrRRpm94D0YAwRciEY/P23Qdq9OeSKr36cbWO1jVqWP/z9NNprUXmh/YLvMi8WWbuFbPpPeA8hClCpqAgs1NFpGBs3n7qdLNWfbJPRSWV2ljRscj8lRcPVEwQisybYfZesad+c7nk2E3vgWhAmCJk2oIjWLN5w1lk3gyz94o99VtDQyz3SwEbI0wRUmYLNngqMj9zzFDNyU4OaZF5M4Jxz/PcfktPj+5C5oDdOSpM7bJmEaFlGIZ2H2vWmy9vbS8yP3JIgh69frRmj0tSP7e9y/txzxOIPo4JU6vXLCL02orMF5VUqWx/vdw99un6sRfopstTdGlSP1uOQj0J1b1iAPblmDC1as0iQsswDH34jyNaWnqmyPylSf10W0YPPXDjVerT0zEf0XbBvlcMwP5seabydDmXtXeR5ewi85+sPaYTf29S4wlDQy/opRv/PVnDhn3pyCBtY9fi/gBCw3ZnK2+Xc7/2tR46fPh0p+O5D+UcnorMDz7QU7Wrm9VY31p7dt+ees2bt0k/+YlbbNEIwClsF6beLueed16M3O5Y7kM5UFdF5qfm/E0N9S0djq+ra9bvfndSjz9uUYMBwE+2C1Nvl22PHGnUH/+Yw30oh/C1yLy33/eBAy0eHwcAO7JdmHa1rID7UPZXcfiklpZW6eVNZ4rM/yD3Yt14WbLHIvPeft+DBjmnli4A2C5MWVbgPA1NzXr70/1aGkCReW+/7zvv5F44AOewXZiyrMA5glFk3tvve9iwL0PZdAAIKtuFqcSyAjvzVGR+esZgzckOvMi8p993cTFhCsA5bBmmsJ/te45raemZIvMX2rTIPABYgTCFVzX1jXpj614VlVTpkz3H1TMuRjNsXmQeAKxAmKIDwzD0ceUxFZVUauW2fTrV6Kwi8wBgBcIUklqLzL/68R4VlVbq8/21cveI1TcvdV6ReQCwAmEaxdqKzBeVVuqvZxWZf2L2GH0z6wJH18YFgHDibBmFzi4y/+Whk0roFae87GTlZado9AV9rW4eADhORIYpm4h35qnIfE7a13Tv1Rdrxpih6hUfa3UTAcCxIi5M2US8o3OLzH+tdw/dfkWa5mSn6OJBfaxuHgBEhIgLUzYRlxqbW7TGhyLzQLBxVQjRKuLCNJo3EW8rMr98U7UO+lBkPtg4kUY3rgohmkXc1hzeNgsP9ybihYUVSktbpZiY5UpLW6XCwoqQvE9DU7Pe2LpXN//2Q035ZbEWr/1CY4f1029vuUwfPHi1Hrg2PaAg9bf9bSfSioo6GcaZE2mofm7YT1dXhYBIF3EjUzvsOhOOv9A9FZm/f9ol+s5lyRrSr5ep1w6k/VxeRzRfFQIiLkztsOtMqIIlFEXmPQmk/ZxI0dVexECki7gwlazfdSbYwRLuIvOBtJ8TKexwVQiwSkSGqdWCESzeisznZScrJ8RF5gNpPydS2OGqEGCViA5Tq2aXBhoshmFoU8VRy4vMB9J+TqSQrL8qBFglYsPUymn6/gZLW5H5/7f+lPa8/YHcPWL1r1kXKC/HmiLzgQYjJ1IA0Spiw9Tq2aXdBYunIvPD+8XoidmZtigyTzACgO8iNkztOrv0YE2DXt5UraWllSo/XNehyPyBzz9W7uUplrYPAOC/iA1TO80u9VZk/kfXjOhQZP7A52FvGgAgCCI2TO0wu3TvsVNatrFKyzdWU2QeACJYxIapmdmlZmYBNza36P2dB1RUUqm1nx9UiyFdOWKgfjFjlKaNHqwecRFXwREAol7EhqkU2CSaQGcBn1tkfnDf1iLzc7KTlfw1ChdEKor7A5AiPEwD4c8s4IamZr396X4VlVTqgy8OK8YlXT1ykOZkp+iq9PMVF2tuFMqJ2t7YJQVAG8L0HL7MAt61v0ZFpWeKzCcNCF6R+TacqO3P6uVXAOyDMD2Ht1nAScnnafnGKhWVVmlTxVHFx7o0bfRg5WWnaFIQi8y34URtf3ZdfgUg/JgNc46Cgky53bEdHovr6ZIxwdD8l7fp6MnTeugbI7XhoWv03/kTNPmS84MepBInaicIZO/c5cuXKyMjQzExMdq4cWOomgYgzAjTc+Tnp+q5/8pS4pAekqTYvi4NvK6H/tecZC2dN1Hv3T9Fd025KCS7tZzNLpucwztPf3h1t/wqMzNTr776qiZPnhzq5gEIIy7z/pNhGPq48lhrkfkv9qnPrXG6bMgA5WUnh7XIfBs7rJNF1wJZfjVq1KhwNQ9AGDk6TIMx27WtyHxRaaU+319reZH5NuzC4gzUMAYgSS7DMHw+eMOGDUZDQ0PQ3ry2tlZ9+gRWCWj16no980yNzm5Oz57SAw8kaOrUrmfUGoahnUdatLa6URv3N6upRbqwX4ymJMUpZ2iczouzJkDN9EekcXJf3H///Tpy5Einx+fOnatJkyZJkn784x/r7rvvVnp6utfXWbFihVauXClJOnr0qJYtWxaaBjuQkz8fwUZfdBTs/sjNzfUpEPwKU0l+Hdyd4uJi5ebmBvTctLRVHmfdpqa6VV4+0+Nzzi0y37dXnGaPG6a8nBSNGto3oHYEk5n+iDSR3he5ubl65plndNlll/l0fHp6usrKykLcKueI9M+HP+iLjkLQHz6FqWMv8/o627W5xdDfdh3U0m6KzFutsLBC999/WAcOLOeSLgA4jGPDtLtdYZxUZP5MgYYWSRRoiGSvvfaa7r33Xh08eFAzZ85UVlaW3n77baubBcAkx4apt9muc+5O1u2/L3FUkXkKNESP2bNna/bs2VY3A0CQOTZMz53t2u/8eA2YHK+lR6s0uNlZReYDKdBA3V4AsA/HhmlDU7MSMuP1LwsSpS8U9CLz4eTvRubU7QUAe3FcmO7aX6M/l1Tp1c3VOhaiIvPh5m+BBi4LA4C9OCJM6043adW2fR2KzE8fPURzspNDUmQ+3NoC8P77N+rAgZZuL9tStxcA7MXWYbp9z3EVlVbq9c17VdPQpAsH9tYvZozUDeOTQl4bN9zy81M1bNiXPq2P8veycLThfjKAcLNdmNbUN+qNrXtVVFKlT/YcV8+4GM0YM1R52cnKGf41y8r72Ql1e73jfjIAK9giTDsUmd+2T6camzVySIIWfTND38oaFvYi83ZH3V7vuJ8MwAqWhmntaUPPr/vSdkXmnYAC655xPxmAFSwL04de/UTLN9apqWWHLk3ur6duGKNZl16gPj1tMViGQ3E/GYAVLFuM2adnrKYkxemv912p1394hfJyUghSmBbIht3+KCysUFraKsXELFda2ioVFlYE5XUBOJtlYbpg5mh9b3RPW+zWYlecuP2Xn5+qJUsmKDXVLZerdRehJUsmBOWSeNvkpoqKOhnGmclN/F4AMBS0KWalBi5U95OZ3ATAG+fU3IsyXZ24vWEkG1pMbgLgDWEaZr4Gnr8nbi5Bhp63SUxMbgJAmIaRP4Hn74k7kJEs/BPqyU0AnIswDSN/As/fEzeXIEMvlJObADgbE5DCyJ/A87fKEesrw4NiGQA8IUzDyN/A8+fETb1eALAOl3nDKJT33LgECQDWYWQaRqEuUM8lSACwBmEaZgQeAEQeLvMCAGASYQoAgEmEKQAAJhGmAACYRJgCAGASYQoAgEmEKQAAJhGmAACYRJg6EJuAA4C9UAHJYdr2RG0raN+2J6okKisBgEUYmdrcuaPQ++7bwibgAGAzjExtzNMo1Bs2AQcA6zAytbEFC7Z3GoV6wybgAGAdwtTGfB1tsgk4AFiLMLUxb6PNxMR4NgEHABvhnqmNFRRkdrhnKrWOQp99dhzhCQA2wsjUxvLzU7VkyQRGoQBgc4xMbS4/P5XwBACbY2QKAIBJhKkJlPUDAEiEqaTAQrGtoEJFRZ0M40xZPwIVAKJP1IdpoKHoqaACZf0AIDpFfZgGGoreCipQ1g8Aok/Uh2mgoeitoAJl/QAg+kR9mAYaigUFmXK7Yzs8Rlk/AIhOUR+mgYYiBRUAAG2ivmhDW/gtWLBdlZV1Sklxq6Ag06dQpKACAEAiTCURigAAc6L+Mi8AAGYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpkAYzZ8/XyNHjtTYsWM1e/ZsHTt2zOomAQgCwhQIo2nTpmn79u3atm2bLrnkEj355JNWNwlAEBCmQBhNnz5dcXGtJbEnTpyo6upqi1sEIBgIU8Aizz//vL7xjW9Y3QwAQeAyDMPngzds2GA0NDQE7c1ra2vVp0+foL2e09EfZzi5L+6//34dOXKk0+Nz587VpEmTJEkvvfSSysrK9Nhjj8nlcnl8nRUrVmjlypWSpKNHj2rZsmWha7TDOPnzEWz0RUfB7o/c3FzP/4Gew68wleTXwd0pLi5Wbm5uMF/S0eiPMyK5L1588UUtXrxY7733ntxut0/PSU9PV1lZWYhb5hyR/PnwF33RUQj6w6cwZT9TIIzeeust/fu//7vWrl3rc5ACsD/umQJhdM8996impkbTpk1TVlaWvv/971vdJABBwMgUCKPdu3db3QQAIcDIFAAAkwhTAABMIkwBADCJMAUAwCTCFAAAkwhTAABMIkwBADCJMAUAwCTCFAAAkwhTAABMIkwBADCJMAUAwCTCFAAAkwhTAABMIkwBADCJMAUAwCTCFAAAkwjTMCssrFBa2irFxCxXWtoqFRZWWN0kAIBJcVY3IJoUFlZo3rxNqqtrliRVVNRp3rxNkqT8/FQrmwYAMIGRaRgtWLC9PUjb1NU1a8GC7Ra1CAAQDIRpGFVW1vn1OADAGQjTMEpJcfv1OADAGQjTMCooyJTbHdvhMbc7VgUFmRa1CAAQDIRpGOXnp2rJkglKTXXL5ZJSU91asmQCk48AwOGYzRtm+fmphCcARBhGpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKhNHDDz+ssWPHKisrS9OnT9fevXutbhKAICBMgTCaP3++tm3bpi1btmjWrFl67LHHrG4SgCAgTIEw6tu3b/u/T548KZfLZWFrAARLnNUNAKLNggUL9Ic//EH9+vXTmjVrrG4OgCBwGYZhdRuAiOJyuVZLGuLhWwsMw3j9rOMektTLMIyFXl5nnqR5//yyl2EYmUFvLICgIEwBi7hcrlRJqwhJwPm4ZwqEkcvlGnHWl9+UtNOqtgAIHkamQBi5XK5XJKVLapFUIen7hmHssbZVAMwiTAEAMInLvAAAmESYAgBgEmEKAIBJhCkAACYRpgAAmESYAgBgEmEKAIBJhCkAACb9f4sajTI3zczHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mglearn.plots.plot_linear_regression_wave()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "from the plot:\n",
    "- w[0] the slope should be about 0.4\n",
    "- b the intercept or where the prediction line crosses the y-axis is slighly below 0.\n",
    "\n",
    "we can calculate this looking for the COEFFICIENT and the INTERCEPT (see below for the code)\n",
    "\n",
    "Linear models for regression can be characterized as regression models for which the prediction is a line for a single feature, a plane for 2 features, and a hyperplane for high dimensiality when we use many features.\n",
    "\n",
    "While this looks similar to the KNeighborsRegressor and makes it seem like we are limited with this straight line predictions and we seem to lose finer details, it actually is just this way with one feature and becomes powerful when we have many features using the linear models. In fact with more features than training data samples any target Y can be perfectly modeled on the training set as a linear function. \n",
    "\n",
    "There are many linear models for regression. The way to differentiate them is how the model params w and b are learned from the training data and how model complexity can be controlled.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Linear Regression Ordinary Least Squares (OLS)\n",
    "\n",
    "- The simplist and most classic example of linear methods for regression. \n",
    "- Linear regression finds the params w and b that **minimize the mean squared error** between predictions and the true regression targets Y on the training set. \n",
    "- The mean square error is the sum of the squared differences between the predictions and the true values. \n",
    "- Linear regression has no parameters, which is a beneft, but....it also has no way to control model complexity. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#produces the graph from earlier\n",
    "from sklearn.linear_model import LinearRegression\n",
    "X, y = mglearn.datasets.make_wave(n_samples=60)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)\n",
    "\n",
    "lr = LinearRegression().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SLOPE params w, are also called the weights or coefficients and are stored in the coef_ attribute while the offset or intercept b is stored in the intercept_ attribute\n",
    "\n",
    "The underscore at the end is just indicating that the model creates this data and saves it\n",
    "\n",
    "The intercept_ attribute is always a single float number while the coef_ is a Numpy array with one entry per input feature. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note these numbers are the actual numbers we saw on the graph earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr.coef_ : [0.39390555]\n",
      "lr.intercept_ : -0.03180434302675976\n"
     ]
    }
   ],
   "source": [
    "print(\"lr.coef_ : {}\".format(lr.coef_))\n",
    "print(\"lr.intercept_ : {}\".format(lr.intercept_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score (note this is an R^2 score bc we are doing regression): 0.67\n",
      "Test set Score (note this is an R^2 score bc we are doing regression): 0.66\n"
     ]
    }
   ],
   "source": [
    "print(\"Training set score (note this is an R^2 score bc we are doing regression): {:.2f}\".format(lr.score(X_train, y_train)))\n",
    "print(\"Test set Score (note this is an R^2 score bc we are doing regression): {:.2f}\".format(lr.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:my_env]",
   "language": "python",
   "name": "conda-env-my_env-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}