linear_regression_exercise.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "linear-regression-exercise.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nG2GQRD78oOd",
        "colab_type": "text"
      },
      "source": [
        "### y = m * x + b\n",
        "\n",
        "A minimal example of linear regression in TensorFlow 2.0, written from scratch without using any built-in layers, optimizers, or loss functions. We'll create a few points on a scatter plot, then find the best fit line using the equation `y = m * x + b`.\n",
        "\n",
        "### Bonus\n",
        "When we're finished, we'll plot the error surface as a function of m and b, to visualize the gradient descent process. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Z7ZBuk95jCb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x8doDs0i5oOU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V2n5vd8i5yCg",
        "colab_type": "text"
      },
      "source": [
        "Create a noisy dataset that's roughly linear, according to the equation y = m * x + b + noise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mekZMKUW5xQj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def make_noisy_data(m=0.1, b=0.3, n=100):\n",
        "  x = tf.random.uniform(shape=(n,))\n",
        "  noise = tf.random.normal(shape=(len(x),), stddev=0.01)\n",
        "  y = m * x + b + noise\n",
        "  return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Jsb2XBE5TPNY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train, y_train = make_noisy_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2tSNaq8W52A2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.plot(x_train, y_train, 'b.')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zpzVrkNhI3y5",
        "colab_type": "text"
      },
      "source": [
        "Define variables for our model. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mwoChE9N5-aE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "m = tf.Variable(0.)\n",
        "b = tf.Variable(0.)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0SEZq4GhJV5d",
        "colab_type": "text"
      },
      "source": [
        "Predict y given x."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TslVztFY7uQT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict(x):\n",
        "  y = m * x + b\n",
        "  return y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cJW-r9JEJi1i",
        "colab_type": "text"
      },
      "source": [
        "Our loss will be the squared difference between the predicted values and the true values."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTmFCjZX7JGV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def squared_error(y_pred, y_true):\n",
        "  # Your code here\n",
        "  # Calculate the mean squared error between y_pred and y_true\n",
        "  # Note: both are arrays\n",
        "  # You can either write code in Python using a for-loop\n",
        "  # Or use tf2 operations like tf.reduce_mean\n",
        "  # and tf.square\n",
        "  return 0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WVDFLZBrBikA",
        "colab_type": "text"
      },
      "source": [
        "Calculate loss before training."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "25EhTz5QBkYS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "loss = squared_error(predict(x_train), y_train)\n",
        "print(\"Starting loss\", loss.numpy())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WjHGdac1EWj6",
        "colab_type": "text"
      },
      "source": [
        "Use gradient descent to gradually improve our guess for `m` and `b`. At each step, we'll nudge them a little bit in the right direction to reduce the loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qp6QFVQD6xB_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learning_rate = 0.05\n",
        "steps = 200\n",
        "\n",
        "for i in range(steps):\n",
        "  \n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = predict(x_train)\n",
        "    loss = squared_error(predictions, y_train)\n",
        "    \n",
        "  gradients = tape.gradient(loss, [m, b])\n",
        "  \n",
        "  m.assign_sub(gradients[0] * learning_rate)\n",
        "  b.assign_sub(gradients[1] * learning_rate)\n",
        "  \n",
        "  if i % 20 == 0:    \n",
        "    print(\"Step %d, Loss %f\" % (i, loss.numpy()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J4lPk7ZC87kU",
        "colab_type": "text"
      },
      "source": [
        "The learned values for m and b."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVsdtuyp6-aP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print (\"m: %f, b: %f\" % (m.numpy(), b.numpy()))\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ra6bq2rV_19Z",
        "colab_type": "text"
      },
      "source": [
        "Plot the best fit line."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rBvBWPgO_3wy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.plot(x_train, y_train, 'b.')\n",
        "plt.plot(x_train, predict(x_train))\n",
        "plt.grid('off')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SIikRd5kDQSp",
        "colab_type": "text"
      },
      "source": [
        "A couple things you can explore:\n",
        "\n",
        "* To understand gradient descent, try printing out the `gradients` calculated below. See how they're used to adjust the variables (`m` and `b`).\n",
        "\n",
        "* You can use TF 2.0 a lot like NumPy.  Try printing out the training data we created (`x_train`, `y_train`) and understand the format. Next, do the same for the variables (m and b). Notice both of these can be converted to NumPy format (with `.numpy()`)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zJ17StfK6-pC",
        "colab_type": "text"
      },
      "source": [
        "### Bonus\n",
        "Let's visualize the error surface. This section is included purely for fun, you can skip it without missing anything."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jEQ_D8iP7fkf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Warning: hacky code ahead\n",
        "\n",
        "import numpy as np\n",
        "from mpl_toolkits.mplot3d import Axes3D\n",
        "\n",
        "# To plot the error surface, we'll need to get the loss\n",
        "# for a bunch of different values for m and b.\n",
        "\n",
        "ms = np.linspace(-1, 1)\n",
        "bs = np.linspace(-1, 1)\n",
        "m_mesh, b_mesh = np.meshgrid(ms, bs)\n",
        "\n",
        "def loss_for_values(m, b):\n",
        "  y = m * x_train + b\n",
        "  loss = squared_error(y, y_train)\n",
        "  return loss\n",
        "\n",
        "zs = np.array([loss_for_values(m, b) for (m,b) in zip(np.ravel(m_mesh), \n",
        "                                                      np.ravel(b_mesh))])\n",
        "z_mesh = zs.reshape(m_mesh.shape)\n",
        "\n",
        "fig = plt.figure(figsize=(12, 12))\n",
        "ax = fig.add_subplot(111, projection='3d')\n",
        "ax.plot_surface(m_mesh, b_mesh, z_mesh, color='b', alpha=0.06)\n",
        "\n",
        "# At this point we have an error surface. \n",
        "# Now we'll need a history of the gradient descent steps.\n",
        "# So as not to complicate the above code,\n",
        "# let's retrain the model here, keeping\n",
        "# track of m, b, and loss at each step.\n",
        "\n",
        "# Intentionally start with this guess to \n",
        "# make the plot nicer\n",
        "m = tf.Variable(-.5)\n",
        "b = tf.Variable(-.75)\n",
        "\n",
        "history = []\n",
        "\n",
        "for i in range(steps):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = predict(x_train)\n",
        "    loss = squared_error(predictions, y_train)\n",
        "  gradients = tape.gradient(loss, [m, b])\n",
        "  history.append((m.numpy(), b.numpy(), loss.numpy()))\n",
        "  m.assign_sub(gradients[0] * learning_rate)\n",
        "  b.assign_sub(gradients[1] * learning_rate)\n",
        "\n",
        "# Plot the trajectory\n",
        "ax.plot([h[0] for h in history], \n",
        "        [h[1] for h in history], \n",
        "        [h[2] for h in history],\n",
        "        marker='o')\n",
        "\n",
        "ax.set_xlabel('m', fontsize=18, labelpad=20)\n",
        "ax.set_ylabel('b', fontsize=18, labelpad=20)\n",
        "ax.set_zlabel('loss', fontsize=18, labelpad=20)\n",
        "\n",
        "ax.view_init(elev=22, azim=28)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}