code/02-gradient.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [A Simple Neuron Structure (Perceptron)](#toc2_)    \n",
    "  - [How to estimate **y**?](#toc2_1_)    \n",
    "- [Gradient](#toc3_)    \n",
    "  - [autograd](#toc3_1_)    \n",
    "    - [AutoGrad in Details : Example 1](#toc3_1_1_)    \n",
    "    - [AutoGrad in Details : Example 2](#toc3_1_2_)    \n",
    "    - [PyTorch Automatic Derivatives](#toc3_1_3_)    \n",
    "      - [Example 1: $f(x) = 2x + 3 \\rightarrow \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 2$](#toc3_1_3_1_)    \n",
    "      - [Example 2: $f(x) = 3x^2 - 2x + 5 \\quad\\rightarrow\\quad \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 6x - 2$](#toc3_1_3_2_)    \n",
    "      - [Example 3: $f(w_1, w_2) = w_1x_1 + w_2x_2 \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2} \\right) = (x_1, x_2)$](#toc3_1_3_3_)    \n",
    "    - [In-place Operations with `requires_grad=True` on Leaf Nodes](#toc3_1_4_)    \n",
    "  - [Gradient Descent](#toc3_2_)    \n",
    "    - [Example 4: $f(w_1, w_2, b) = w_1x_1 + w_2x_2 + b \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2}, \\frac{\\partial f}{\\partial b} \\right) = (x_1, x_2, 1)$](#toc3_2_1_)    \n",
    "      - [Chain Rule](#toc3_2_1_1_)    \n",
    "      - [Updating Weights](#toc3_2_1_2_)    \n",
    "    - [Gradient Descent Optimization Example](#toc3_2_2_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.autograd import Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[A Simple Neuron Structure (Perceptron)](#toc0_)\n",
    "\n",
    "- In many contexts, the terms **Neuron** and **Perceptron** are used interchangeably\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"display:flex; margin-top:50px;\">\n",
    "  <div style=\"width:20%; margin-right:auto; margin-left:auto;\">\n",
    "    <table style=\"margin:0 auto; width:80%; text-align:center\">\n",
    "      <caption style=\"font-weight:bold;\">Dataset</caption>\n",
    "      <thead>\n",
    "        <tr>\n",
    "          <th style=\"width:25%; text-align:center\"><span style=\"color:magenta;\">#</span></th>\n",
    "          <th style=\"width:25%; text-align:center\"><span style=\"color:#9090ff;\">x<sub>1</sub></span></th>\n",
    "          <th style=\"width:25%; text-align:center\"><span style=\"color:#9090ff;\">x<sub>2</sub></span></th>\n",
    "          <th style=\"width:25%; text-align:center\"><span style=\"color:red;\">y</span></th>\n",
    "        </tr>\n",
    "      </thead>\n",
    "      <tbody>\n",
    "        <tr>\n",
    "          <th>1</th>\n",
    "          <td>1</td>\n",
    "          <td>1</td>\n",
    "          <td>2</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>2</th>\n",
    "          <td>2</td>\n",
    "          <td>3</td>\n",
    "          <td>5</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>3</th>\n",
    "          <td>1</td>\n",
    "          <td>2</td>\n",
    "          <td>3</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>4</th>\n",
    "          <td>3</td>\n",
    "          <td>1</td>\n",
    "          <td>4</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>5</th>\n",
    "          <td>2</td>\n",
    "          <td>4</td>\n",
    "          <td>6</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>6</th>\n",
    "          <td>3</td>\n",
    "          <td>2</td>\n",
    "          <td>5</td>\n",
    "        </tr>\n",
    "        <tr>\n",
    "          <th>7</th>\n",
    "          <td>4</td>\n",
    "          <td>1</td>\n",
    "          <td>5</td>\n",
    "        </tr>\n",
    "      </tbody>\n",
    "    </table>\n",
    "  </div>\n",
    "  <div style=\"width:80%; padding:10px;\">\n",
    "    <figure style=\"text-align:center; margin:0;\">\n",
    "      <img src=\"../assets/images/original/perceptron/perceptron-1.svg\" alt=\"perceptron-1.svg\" style=\"max-width:80%; height:auto;\">\n",
    "      <figcaption style=\"text-align:center;\">A simple Neuron (Perceptron)</figcaption>\n",
    "    </figure>\n",
    "  </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_1_'></a>[How to estimate **y**?](#toc0_)\n",
    "\n",
    "1. **System of Equations**\n",
    "    $$\n",
    "    \\left\\{\n",
    "    \\begin{aligned}\n",
    "    1w_1 + 1w_2 &= 2 \\\\\n",
    "    2w_1 + 3w_2 &= 5 \\\\\n",
    "    1w_1 + 2w_2 &= 3 \\\\\n",
    "    3w_1 + 1w_2 &= 4 \\\\\n",
    "    2w_1 + 4w_2 &= 6 \\\\\n",
    "    3w_1 + 2w_2 &= 5 \\\\\n",
    "    4w_1 + 1w_2 &= 5 \\\\\n",
    "    \\end{aligned}\n",
    "    \\right.\n",
    "    $$\n",
    "\n",
    "    - **Disadvantages**\n",
    "      - `Complexity`: Neural networks are highly complex systems with millions of parameters ([GPT-4 has 1.76 trillion parameters](https://en.wikipedia.org/wiki/GPT-4#:~:text=Rumors%20claim%20that%20GPT%2D4,running%20and%20by%20George%20Hotz.)).\n",
    "      - `Non-linearity`: Neural networks use activation functions like Sigmoid, which introduce non-linearity into the network.\n",
    "    - **Critical issue: Overdetermined system**\n",
    "      - The number of equations are more than the number of unknowns.\n",
    "      - The system becomes inconsistent and cannot be solved exactly.\n",
    "      - It may lead to either \"No solution\" or \"An infinite number of solutions\".\n",
    "\n",
    "1. **Delta Rule**\n",
    "    - The delta rule, also known as the Widrow-Hoff rule or the LMS (least mean squares) rule.\n",
    "    - The delta rule is commonly associated with the AdaLiNe (Adaptive Linear Neuron) model.\n",
    "    - It is a simple supervised learning rule used for training single-layer neural networks (perceptrons).\n",
    "\n",
    "1. **Backpropagation**\n",
    "    - Backpropagation is an extended version of Delta Rule for multi-layer neural networks.\n",
    "    - It allows the network to learn from its mistakes by updating the weights iteratively using **Gradient Descent** (aka Steepest Descent).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[Gradient](#toc0_)\n",
    "\n",
    "- **Definition**:\n",
    "  - The gradient represents the rate of change of the output of a function with respect to its inputs.\n",
    "  - For functions with multiple variables, it generalizes the concept of a derivative, forming a vector of partial derivatives.\n",
    "- **Intuition**:\n",
    "  - In one-dimensional functions, the gradient (or derivative) corresponds to the slope of the function.\n",
    "  - In multi-dimensional functions, the gradient points in the direction of the steepest ascent of the function, with its magnitude indicating the rate of change.\n",
    "- **Applications**:\n",
    "  - Crucial for optimization techniques like **Gradient Descent**, where gradients guide the updates to minimize loss functions in machine learning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_1_'></a>[autograd](#toc0_)\n",
    "\n",
    "- **Overview**:\n",
    "  - PyTorch's **automatic differentiation engine**, which computes gradients efficiently for tensor operations.\n",
    "  - It enables dynamic computation graphs, making it flexible for building and training complex neural networks.\n",
    "- **How it Works**:\n",
    "  1. **Backward Pass**:\n",
    "      - Calling `torch.Tensor.backward()` computes the gradients for all tensors in the computation graph with `requires_grad=True`. These gradients are accumulated in the `grad` attribute of the respective tensors.\n",
    "  1. **Accessing Gradients**:\n",
    "      - Gradients are stored in `torch.Tensor.grad` after the backward pass.\n",
    "      - Optimizers (e.g., `torch.optim.SGD`, `torch.optim.Adam`) use these gradients to update model parameters during training.\n",
    "\n",
    "📚 **Tutorials**:\n",
    "\n",
    "- A Gentle Introduction to `torch.autograd`: [pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_1_'></a>[AutoGrad in Details : Example 1](#toc0_)\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../assets/images/original/gradient/autograd.svg\" alt=\"autograd.svg\" style=\"width: 80%;\">\n",
    "  <figcaption style=\"text-align: center;\">Lower-Level AutoGrad Mechanism</figcaption>\n",
    "</figure>\n",
    "\n",
    "©️ **Credits**:\n",
    "\n",
    "- Detailed info about **autograd**: [youtube.com/@elliotwaite](https://www.youtube.com/watch?v=MswxJw-8PvE)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomMul(Function):\n",
    "    @staticmethod\n",
    "    def forward(\n",
    "        ctx: torch.autograd.function.FunctionCtx,\n",
    "        input1: torch.Tensor,\n",
    "        input2: torch.Tensor,\n",
    "    ) -> torch.Tensor:\n",
    "        ctx.save_for_backward(input1, input2)\n",
    "        return input1 * input2\n",
    "\n",
    "    @staticmethod\n",
    "    def backward(\n",
    "        ctx: torch.autograd.function.FunctionCtx,\n",
    "        grad_output: torch.Tensor,\n",
    "    ) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        input1, input2 = ctx.saved_tensors\n",
    "        grad_input1 = grad_output * input2\n",
    "        grad_input2 = grad_output * input1\n",
    "        return grad_input1, grad_input2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# leaf nodes\n",
    "t_1 = torch.tensor(2.0)\n",
    "t_2 = torch.tensor(3.0, requires_grad=True)\n",
    "\n",
    "# perform a multiplication operation\n",
    "t_3 = CustomMul.apply(t_1, t_2)\n",
    "\n",
    "# backward\n",
    "t_3.backward()\n",
    "\n",
    "# log\n",
    "print(f\"t_1.grad: {t_1.grad}\")\n",
    "print(f\"t_2.grad: {t_2.grad}\")\n",
    "print(f\"t_3.grad_fn.next_functions : {t_3.grad_fn.next_functions}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_2_'></a>[AutoGrad in Details : Example 2](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# grad_fn\n",
    "x = torch.tensor(2.0, requires_grad=True)\n",
    "\n",
    "# perform operations\n",
    "y = x + 1\n",
    "z = y**2 * 3\n",
    "out = z.mean()\n",
    "\n",
    "\n",
    "# function to traverse the graph\n",
    "def print_computation_graph(grad_fn: torch.autograd.Function, level: int = 0):\n",
    "    if grad_fn is not None:\n",
    "        print(\" \" * level, grad_fn)\n",
    "        if hasattr(grad_fn, \"next_functions\"):\n",
    "            for fn in grad_fn.next_functions:\n",
    "                print_computation_graph(fn[0], level + 4)\n",
    "\n",
    "\n",
    "# start from the output node (out) and traverse backward\n",
    "print(\"computation graph:\")\n",
    "print_computation_graph(out.grad_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_3_'></a>[PyTorch Automatic Derivatives](#toc0_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <a id='toc3_1_3_1_'></a>[Example 1: $f(x) = 2x + 3 \\rightarrow \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 2$](#toc0_)\n",
    "\n",
    "- $\\nabla f(4) = 2$\n",
    "- $\\nabla f(0) = 2$\n",
    "- $\\nabla f(1) = 2$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x: torch.Tensor) -> torch.Tensor:\n",
    "    return 2 * x + 3  # torch.add(torch.multiply(2, x), 3)\n",
    "\n",
    "\n",
    "# x: independent variable\n",
    "x = torch.tensor(1, dtype=torch.float32, requires_grad=True)\n",
    "\n",
    "# f(x) or y : dependent variable\n",
    "y = f(x)\n",
    "\n",
    "# compute the gradients with respect to all Tensors that have `requires_grad=True`\n",
    "y.backward()\n",
    "\n",
    "# access computed gradients\n",
    "# if x at 1 moves by ε, then y moves by 2ε\n",
    "gradients = x.grad\n",
    "\n",
    "# log\n",
    "print(\"x     :\", x)\n",
    "print(\"y     :\", y)\n",
    "print(\"x.grad:\", gradients)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "_ = np.linspace(-4, 6, 100)\n",
    "plt.figure(figsize=(6, 4))\n",
    "plt.title(f\"x.grad: {x.grad}\")\n",
    "plt.plot(_, f(_), label=\"f(x) = 2x + 3\", color=\"blue\")\n",
    "plt.axvline(x=x.item(), color=\"red\", linestyle=\"--\", label=f\"x = {x}\")\n",
    "plt.axhline(y=f(x).item(), color=\"green\", linestyle=\"--\", label=f\"y = {f(x)}\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"f(x)\")\n",
    "plt.xticks(range(-10, 16, 2))\n",
    "plt.yticks(range(-10, 16, 2))\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <a id='toc3_1_3_2_'></a>[Example 2: $f(x) = 3x^2 - 2x + 5 \\quad\\rightarrow\\quad \\nabla f(x) = \\frac{\\partial f}{\\partial x} = 6x - 2$](#toc0_)\n",
    "\n",
    "- $\\nabla f(3) = 16$\n",
    "- $\\nabla f(0) = -2$\n",
    "- $\\nabla f(1) = 4$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x: torch.Tensor) -> torch.Tensor:\n",
    "    # torch.add(torch.sub(torch.mul(3, torch.pow(x, 2)), torch.mul(2, x)), 5)\n",
    "    return 3 * x**2 - 2 * x + 5\n",
    "\n",
    "\n",
    "x = torch.tensor(3, dtype=torch.float32, requires_grad=True)\n",
    "y = f(x)\n",
    "\n",
    "# compute the gradients with respect to all Tensors that have `requires_grad=True`\n",
    "y.backward()\n",
    "\n",
    "# access computed gradients\n",
    "# if x at 3 moves by ε, then y moves by (6 * 3 - 2)ε\n",
    "gradients = x.grad\n",
    "\n",
    "# log\n",
    "print(\"x     :\", x)\n",
    "print(\"y     :\", y)\n",
    "print(f\"x.grad: {gradients} [at x={x}]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "_ = np.linspace(-5, 5, 100)\n",
    "plt.figure(figsize=(6, 4))\n",
    "plt.title(f\"x.grad: {x.grad}\")\n",
    "plt.plot(_, f(_), label=\"f(x) = 3x^2 - 2x + 5\", color=\"blue\")\n",
    "plt.axvline(x=x.item(), color=\"red\", linestyle=\"--\", label=f\"x = {x}\")\n",
    "plt.axhline(y=f(x).item(), color=\"green\", linestyle=\"--\", label=f\"y = {f(x).item()}\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"f(x)\")\n",
    "plt.xticks(range(-5, 6))\n",
    "plt.yticks(range(0, 101, 10))\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### <a id='toc3_1_3_3_'></a>[Example 3: $f(w_1, w_2) = w_1x_1 + w_2x_2 \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2} \\right) = (x_1, x_2)$](#toc0_)\n",
    "\n",
    "- **Magnitude**: $|\\nabla f(w_1, w_2)| = \\sqrt{x_1^2 + x_2^2}$\n",
    "\n",
    "- **Direction**: $\\nabla f(w_1, w_2) = \\frac{x_1}{\\sqrt{x_1^2 + x_2^2}} \\hat{i} + \\frac{x_2}{\\sqrt{x_1^2 + x_2^2}} \\hat{j}$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(X: torch.Tensor, W: torch.Tensor) -> torch.Tensor:\n",
    "    return torch.dot(X, W)\n",
    "\n",
    "\n",
    "W = torch.tensor([1, 2], dtype=torch.float32, requires_grad=True)\n",
    "X = torch.tensor([2, 3], dtype=torch.float32)\n",
    "y = f(X, W)\n",
    "\n",
    "# compute the gradients\n",
    "y.backward()\n",
    "\n",
    "# access the gradients\n",
    "gradients = W.grad\n",
    "\n",
    "magnitude_grad = torch.norm(gradients)  # same as (grad ** 2).sum().sqrt()\n",
    "direction_grad = gradients / magnitude_grad  # normalized (unit vector)\n",
    "\n",
    "# log\n",
    "print(\"W:\", W)\n",
    "print(\"X:\", X)\n",
    "print(\"y:\", y)\n",
    "print(\"-\" * 50)\n",
    "print(\"magnitude of gradients:\", magnitude_grad.item())\n",
    "print(\"direction of gradients:\", direction_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot\n",
    "w1 = np.linspace(-10, 10, 100)\n",
    "w2 = np.linspace(-10, 10, 100)\n",
    "X1, X2 = np.meshgrid(w1, w2)\n",
    "_ = X1 * W[0].detach().numpy() + X2 * W[1].detach().numpy()\n",
    "\n",
    "fig = plt.figure(figsize=(12, 4), layout=\"compressed\")\n",
    "\n",
    "ax1 = fig.add_subplot(121, projection=\"3d\")\n",
    "ax1.plot_surface(X1, X2, _, cmap=\"viridis\")\n",
    "ax1.set_xlabel(\"w1\")\n",
    "ax1.set_ylabel(\"w2\")\n",
    "ax1.set_zlabel(\"f(w1, w2)\")\n",
    "ax1.set_title(\"f(w1, w2) = 2w1 + 3w2\")\n",
    "ax2 = fig.add_subplot(122)\n",
    "ax2.quiver(0, 0, direction_grad[0], direction_grad[1], angles=\"xy\", scale_units=\"xy\", scale=1, color=\"red\")\n",
    "ax2.set_xlim(-2, 2)\n",
    "ax2.set_ylim(-2, 2)\n",
    "ax2.set_xlabel(\"w1\")\n",
    "ax2.set_ylabel(\"w2\")\n",
    "ax2.set_title(\"Direction of gradients\")\n",
    "ax2.grid(\"on\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_1_4_'></a>[In-place Operations with `requires_grad=True` on Leaf Nodes](#toc0_)\n",
    "\n",
    "- **In-place operations** modify the content of a tensor **directly** without creating a new tensor.\n",
    "- Examples include operations like `+=`, `-=` or using functions with an underscore like `.add_()`, `.mul_()`, etc.\n",
    "\n",
    "**Why In-place Operations are Problematic for Gradients?**\n",
    "\n",
    "- **Loss of Original Data:**  \n",
    "  - When you perform an in-place operation on a tensor that requires gradients, PyTorch **loses track** of the original tensor values, which is essential for correctly calculating the gradient during the backward pass.\n",
    "  - This happens because, during the backward pass, PyTorch needs the original values to compute the gradients. If the tensor is modified in place, the **original value is overwritten** and cannot be accessed later for the backward calculation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = torch.tensor(0, dtype=torch.float64)\n",
    "x2 = torch.tensor(0, dtype=torch.float64, requires_grad=True)\n",
    "\n",
    "# out-of-place assignment\n",
    "x1 = x1 + 1  # x1 = x1.add(1)\n",
    "x2 = x2 + 1  # x2 = x2.add(1)\n",
    "\n",
    "# log\n",
    "print(\"x1:\", x1)\n",
    "print(\"x2:\", x2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = torch.tensor(0, dtype=torch.float64)\n",
    "x2 = torch.tensor(0, dtype=torch.float64, requires_grad=True)\n",
    "\n",
    "# in-place assignment\n",
    "x1 += 1  # x1.add_(1)\n",
    "\n",
    "try:\n",
    "    x2 += 1  # x2.add_(1)\n",
    "except RuntimeError as e:\n",
    "    print(e)\n",
    "\n",
    "# log\n",
    "print(\"x1:\", x1)\n",
    "print(\"x2:\", x2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_2_'></a>[Gradient Descent](#toc0_)\n",
    "\n",
    "- The gradient direction is indeed the direction in which a function increases most rapidly\n",
    "- To minimize the loss function, we shall move in the opposite of the gradient direction.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_2_1_'></a>[Example 4: $f(w_1, w_2, b) = w_1x_1 + w_2x_2 + b \\quad\\rightarrow\\quad \\nabla f(W) = \\left( \\frac{\\partial f}{\\partial w_1}, \\frac{\\partial f}{\\partial w_2}, \\frac{\\partial f}{\\partial b} \\right) = (x_1, x_2, 1)$](#toc0_)\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../assets/images/original/perceptron/adaline.svg\" alt=\"adaline.svg\" style=\"width: 80%;\">\n",
    "  <figcaption style=\"text-align: center;\">ADAptive LInear NEuron (ADALINE)</figcaption>\n",
    "</figure>\n",
    "\n",
    "$\n",
    "    W = \\begin{bmatrix} w_0 \\\\ w_1 \\\\ w_2 \\end{bmatrix}\\quad,\\quad\n",
    "    X = \\begin{bmatrix} 1 \\\\ x_1 \\\\ x_2 \\end{bmatrix}\\quad,\\quad\n",
    "    output = W^T X = \\begin{bmatrix} w_0 \\ w_1 \\ w_2 \\end{bmatrix}.\\begin{bmatrix} 1 \\\\ x_1 \\\\ x_2 \\end{bmatrix}= w_0 + w_1x_1 + w_2x_2\n",
    "$\n",
    "\n",
    "#### <a id='toc3_2_1_1_'></a>[Chain Rule](#toc0_)\n",
    "\n",
    "- Activation function must be differentiable\n",
    "- Loss (error) function must be differentiable\n",
    "\n",
    "$$\n",
    "\\nabla L(W) = (\\frac{\\partial \\text{loss}}{\\partial y_{\\text{pred}}} \\cdot \\frac{\\partial y_{\\text{pred}}}{\\partial \\text{output}} \\cdot \\frac{\\partial \\text{output}}{\\partial W})\n",
    "$$\n",
    "\n",
    "#### <a id='toc3_2_1_2_'></a>[Updating Weights](#toc0_)\n",
    "\n",
    "$$\n",
    "W_{new} = W_{old} - \\alpha \\nabla L(W_{old})\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc3_2_2_'></a>[Gradient Descent Optimization Example](#toc0_)\n",
    "\n",
    "- $x = [2, 3] \\quad,\\quad y = 0$\n",
    "- Note: $x$ is a single sample with two features\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y = 0\n",
    "y_true = torch.tensor(0, dtype=torch.int64)\n",
    "\n",
    "# 1 is the multiplication for bias\n",
    "X = torch.tensor([1, 2, 3], dtype=torch.float32)\n",
    "\n",
    "# initial weights [bias = .3]\n",
    "W = torch.tensor([0.3, 0.7, 0.5], dtype=torch.float32, requires_grad=True)\n",
    "\n",
    "# hyper parameters\n",
    "epochs = 10\n",
    "learning_rate = 0.5\n",
    "\n",
    "for epoch in range(epochs):\n",
    "    print(f\"epoch      : {epoch}\")\n",
    "\n",
    "    # feed-forward\n",
    "    output = torch.dot(X, W)\n",
    "    y_pred = torch.sigmoid(output)\n",
    "    print(f\"y_true     : {y_true.item()} (label)\")\n",
    "    print(f\"y_pred     : {y_pred.item()}\")\n",
    "    print(f\"prediction : {torch.where(y_pred < .5, 0, 1)} (label)\")\n",
    "\n",
    "    # loss\n",
    "    loss = (y_pred - y_true) ** 2\n",
    "    print(f\"loss       : {loss.item()}\")\n",
    "\n",
    "    # backward\n",
    "    loss.backward()\n",
    "    dW = W.grad\n",
    "    step = learning_rate * dW\n",
    "    print(f\"grad       : {dW}\")\n",
    "    print(f\"step       : {step}\")\n",
    "\n",
    "    # update weights\n",
    "    with torch.no_grad():\n",
    "        W -= step\n",
    "        W.grad.zero_()\n",
    "\n",
    "    print(f\"W_new      : {W}\")\n",
    "    print(\"-\" * 50)"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}