solutions/win2022/A3/convolutional_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "DDJwQPZcupab",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# EECS 498-007/598-005 Assignment 3-2: Convolutional Neural Networks and Batch Normalization\n",
    "\n",
    "Before we start, please put your name and UMID in following format\n",
    "\n",
    ": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "tt7vq1h6mRto",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Your Answer:**   \n",
    "Your NAME, #XXXXXXXX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "FrfeHl_-m4V-",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup Code\n",
    "Before getting started, we need to run some boilerplate code to set up our environment, same as Assignment 1. You'll need to rerun this setup code each time you start the notebook.\n",
    "\n",
    "First, run this cell load the autoreload extension. This allows us to edit .py source files, and re-import them into the notebook for a seamless editing and debugging experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 1564,
     "status": "ok",
     "timestamp": 1601239154125,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "VyQblYp0nEZq",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Q7ymI0aZ2W1b",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Google Colab Setup\n",
    "Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n",
    "\n",
    "Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1879,
     "status": "ok",
     "timestamp": 1601239154473,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "c_LLpLyC2eau",
    "new_sheet": false,
    "outputId": "7557c26f-13a9-4e0b-f693-efa4b18976b6",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "mbq-UT8J2mnv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['convolutional_networks.ipynb', 'fully_connected_networks.ipynb', 'eecs598', 'convolutional_networks.py', 'fully_connected_networks.py', 'a3_helper.py']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 2007,
     "status": "ok",
     "timestamp": 1601239154632,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "WcrhTOZW243H",
    "new_sheet": false,
    "outputId": "d90deddb-c9f9-43eb-ec00-6d9855d1cb4c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# TODO: Fill in the Google Drive path where you uploaded the assignment\n",
    "# Example: If you create a 2022WI folder and put all the files under A3 folder, then '2022WI/A3'\n",
    "GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = None\n",
    "GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n",
    "print(os.listdir(GOOGLE_DRIVE_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "xegb0uDA232J",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n",
    "\n",
    "```\n",
    "Hello from convolutional_networks.py!\n",
    "Hello from a3_helper.py!\n",
    "```\n",
    "\n",
    "as well as the last edit time for the file `convolutional_networks.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 2203,
     "status": "ok",
     "timestamp": 1601239154855,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "AhGQF5sw3Fas",
    "new_sheet": false,
    "outputId": "f96ecd03-4d84-4380-cadf-0d89236c4f20",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(GOOGLE_DRIVE_PATH)\n",
    "\n",
    "import time, os\n",
    "os.environ[\"TZ\"] = \"US/Eastern\"\n",
    "time.tzset()\n",
    "\n",
    "from convolutional_networks import hello_convolutional_networks\n",
    "hello_convolutional_networks()\n",
    "\n",
    "from a3_helper import hello_helper\n",
    "hello_helper()\n",
    "\n",
    "convolutional_networks_path = os.path.join(GOOGLE_DRIVE_PATH, 'convolutional_networks.py')\n",
    "convolutional_networks_edit_time = time.ctime(os.path.getmtime(convolutional_networks_path))\n",
    "print('convolutional_networks.py last edited on %s' % convolutional_networks_edit_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ynKS05gJ4iBo",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "fN1SShPR4lJV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup code\n",
    "Run some setup code for this notebook: Import some useful packages and increase the default figure size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "executionInfo": {
     "elapsed": 2179,
     "status": "ok",
     "timestamp": 1601239154856,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "VUCKw4Tl1ddj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "import torch\n",
    "import torchvision\n",
    "import matplotlib.pyplot as plt\n",
    "import statistics\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "%matplotlib inline\n",
    "\n",
    "from eecs598 import reset_seed, Solver\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0)\n",
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "lhqpd2IN2O-K",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Starting in this assignment, we will use the GPU to accelerate our computation. Run this cell to make sure you are using a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 2163,
     "status": "ok",
     "timestamp": 1601239154857,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "SGDxdBIMRX6b",
    "new_sheet": false,
    "outputId": "db57c504-cb4d-4a78-9967-eb15a5383ca7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available:\n",
    "  print('Good to go!')\n",
    "else:\n",
    "  print('Please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "-Yv3zQYw5B3s",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Load the CIFAR-10 dataset\n",
    "Then, we will first load the CIFAR-10 dataset, same as knn. The utility function `get_CIFAR10_data()` in `helper_functions` returns the entire CIFAR-10 dataset as a set of six **Torch tensors** while also preprocessing the RGB images:\n",
    "\n",
    "- `X_train` contains all training images (real numbers in the range $[0, 1]$)\n",
    "- `y_train` contains all training labels (integers in the range $[0, 9]$)\n",
    "- `X_val` contains all validation images\n",
    "- `y_val` contains all validation labels\n",
    "- `X_test` contains all test images\n",
    "- `y_test` contains all test labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 568
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 9744,
     "status": "ok",
     "timestamp": 1601239162469,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "V2mFlFmQ1ddm",
    "new_sheet": false,
    "outputId": "4a038a26-904b-4572-df02-c81b02edfb01",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Invoke the above function to get our data. \n",
    "import eecs598\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "data_dict = eecs598.data.preprocess_cifar10(cuda=True, dtype=torch.float64, flatten=False)\n",
    "print('Train data shape: ', data_dict['X_train'].shape)\n",
    "print('Train labels shape: ', data_dict['y_train'].shape)\n",
    "print('Validation data shape: ', data_dict['X_val'].shape)\n",
    "print('Validation labels shape: ', data_dict['y_val'].shape)\n",
    "print('Test data shape: ', data_dict['X_test'].shape)\n",
    "print('Test labels shape: ', data_dict['y_test'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "aQW_w1Wzw72f",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Convolutional networks\n",
    "So far we have worked with deep fully-connected networks, using them to explore different optimization strategies and network architectures. Fully-connected networks are a good testbed for experimentation because they are very computationally efficient, but in practice all state-of-the-art results use convolutional networks instead.\n",
    "\n",
    "First you will implement several layer types that are used in convolutional networks. You will then use these layers to train a convolutional network on the CIFAR-10 dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "CJInAlccoI5e",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Convolutional layer\n",
    "As in the previous notebook, we will package each new neural network operator in a class that defines a `forward` and `backward` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "x07DS91iw72o",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Convolutional layer: forward\n",
    "The core of a convolutional network is the convolution operation. Implement the forward pass for the convolution layer in the function `Conv.forward`. \n",
    "\n",
    "You don't have to worry too much about efficiency at this point; just write the code in whatever way you find most clear.\n",
    "\n",
    "After implementing the forward pass of the convolution operation, run the following to check your implementation. You should get a relative error less than `1e-7`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 9717,
     "status": "ok",
     "timestamp": 1601239162470,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "F5R_WY1Iw72p",
    "new_sheet": false,
    "outputId": "e9ac2acd-866c-4d9f-b6a5-d54de988fd7d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import Conv\n",
    "\n",
    "x_shape = torch.tensor((2, 3, 4, 4))\n",
    "w_shape = torch.tensor((3, 3, 4, 4))\n",
    "x = torch.linspace(-0.1, 0.5, steps=torch.prod(x_shape), dtype=torch.float64, device='cuda').reshape(*x_shape)\n",
    "w = torch.linspace(-0.2, 0.3, steps=torch.prod(w_shape), dtype=torch.float64, device='cuda').reshape(*w_shape)\n",
    "b = torch.linspace(-0.1, 0.2, steps=3, dtype=torch.float64, device='cuda')\n",
    "\n",
    "conv_param = {'stride': 2, 'pad': 1}\n",
    "out, _ = Conv.forward(x, w, b, conv_param)\n",
    "correct_out = torch.tensor([[[[-0.08759809, -0.10987781],\n",
    "                              [-0.18387192, -0.2109216 ]],\n",
    "                             [[ 0.21027089,  0.21661097],\n",
    "                              [ 0.22847626,  0.23004637]],\n",
    "                             [[ 0.50813986,  0.54309974],\n",
    "                              [ 0.64082444,  0.67101435]]],\n",
    "                            [[[-0.98053589, -1.03143541],\n",
    "                              [-1.19128892, -1.24695841]],\n",
    "                             [[ 0.69108355,  0.66880383],\n",
    "                              [ 0.59480972,  0.56776003]],\n",
    "                             [[ 2.36270298,  2.36904306],\n",
    "                              [ 2.38090835,  2.38247847]]]], \n",
    "                          dtype=torch.float64, device='cuda',\n",
    "            )\n",
    "\n",
    "# Compare your output to ours; difference should be around e-8\n",
    "print('Testing Conv.forward')\n",
    "print('difference: ', eecs598.grad.rel_error(out, correct_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "N5bKrl7Uw72t",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Aside: Image processing via convolutions\n",
    "\n",
    "As fun way to both check your implementation and gain a better understanding of the type of operation that convolutional layers can perform, we will set up an input containing two images and manually set up filters that perform common image processing operations (grayscale conversion and edge detection). The convolution forward pass will apply these operations to each of the input images. We can then visualize the results as a sanity check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 453
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 15519,
     "status": "ok",
     "timestamp": 1601239168300,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "k8BffZxdw72u",
    "new_sheet": false,
    "outputId": "46210ff8-4069-4085-b136-7f62b090d0d6",
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "from imageio import imread\n",
    "from PIL import Image\n",
    "from torchvision.transforms import ToTensor\n",
    "\n",
    "kitten_url = 'https://web.eecs.umich.edu/~justincj/teaching/eecs498/assets/a3/kitten.jpg'\n",
    "puppy_url = 'https://web.eecs.umich.edu/~justincj/teaching/eecs498/assets/a3/puppy.jpg'\n",
    "\n",
    "kitten = imread(kitten_url)\n",
    "puppy = imread(puppy_url)\n",
    "# kitten is wide, and puppy is already square\n",
    "d = kitten.shape[1] - kitten.shape[0]\n",
    "kitten_cropped = kitten[:, d//2:-d//2, :]\n",
    "\n",
    "img_size = 200   # Make this smaller if it runs too slow\n",
    "resized_puppy = ToTensor()(Image.fromarray(puppy).resize((img_size, img_size)))\n",
    "resized_kitten = ToTensor()(Image.fromarray(kitten_cropped).resize((img_size, img_size)))\n",
    "x = torch.stack([resized_puppy, resized_kitten])\n",
    "\n",
    "# Set up a convolutional weights holding 2 filters, each 3x3\n",
    "w = torch.zeros(2, 3, 3, 3, dtype=x.dtype)\n",
    "\n",
    "# The first filter converts the image to grayscale.\n",
    "# Set up the red, green, and blue channels of the filter.\n",
    "w[0, 0, :, :] = torch.tensor([[0, 0, 0], [0, 0.3, 0], [0, 0, 0]])\n",
    "w[0, 1, :, :] = torch.tensor([[0, 0, 0], [0, 0.6, 0], [0, 0, 0]])\n",
    "w[0, 2, :, :] = torch.tensor([[0, 0, 0], [0, 0.1, 0], [0, 0, 0]])\n",
    "\n",
    "# Second filter detects horizontal edges in the blue channel.\n",
    "w[1, 2, :, :] = torch.tensor([[1, 2, 1], [0, 0, 0], [-1, -2, -1]])\n",
    "\n",
    "# Vector of biases. We don't need any bias for the grayscale\n",
    "# filter, but for the edge detection filter we want to add 128\n",
    "# to each output so that nothing is negative.\n",
    "b = torch.tensor([0, 128], dtype=x.dtype)\n",
    "\n",
    "# Compute the result of convolving each input in x with each filter in w,\n",
    "# offsetting by b, and storing the results in out.\n",
    "out, _ = Conv.forward(x, w, b, {'stride': 1, 'pad': 1})\n",
    "\n",
    "def imshow_no_ax(img, normalize=True):\n",
    "  \"\"\" Tiny helper to show images as uint8 and remove axis labels \"\"\"\n",
    "  if normalize:\n",
    "    img_max, img_min = img.max(), img.min()\n",
    "    img = 255.0 * (img - img_min) / (img_max - img_min)\n",
    "  plt.imshow(img)\n",
    "  plt.gca().axis('off')\n",
    "\n",
    "# Show the original images and the results of the conv operation\n",
    "plt.subplot(2, 3, 1)\n",
    "imshow_no_ax(puppy, normalize=False)\n",
    "plt.title('Original image')\n",
    "plt.subplot(2, 3, 2)\n",
    "imshow_no_ax(out[0, 0])\n",
    "plt.title('Grayscale')\n",
    "plt.subplot(2, 3, 3)\n",
    "imshow_no_ax(out[0, 1])\n",
    "plt.title('Edges')\n",
    "plt.subplot(2, 3, 4)\n",
    "imshow_no_ax(kitten_cropped, normalize=False)\n",
    "plt.subplot(2, 3, 5)\n",
    "imshow_no_ax(out[1, 0])\n",
    "plt.subplot(2, 3, 6)\n",
    "imshow_no_ax(out[1, 1])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Lqsg-NxHw72y",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Convolutional layer: backward\n",
    "Implement the backward pass for the convolution operation in the function `Conv.backward`. Again, you don't need to worry too much about computational efficiency.\n",
    "\n",
    "After implementing the convolution backward pass, run the following to test your implementation. You should get errors less than `1e-8`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 25061,
     "status": "ok",
     "timestamp": 1601239177870,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "77NxvIGpw720",
    "new_sheet": false,
    "outputId": "70d97b6c-269e-4ad1-8457-b754a45d63c4",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import Conv\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(4, 3, 5, 5, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(2, 3, 3, 3, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(2, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(4, 2, 5, 5, dtype=torch.float64, device='cuda')\n",
    "conv_param = {'stride': 1, 'pad': 1}\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: Conv.forward(x, w, b, conv_param)[0], x, dout)\n",
    "dw_num = eecs598.grad.compute_numeric_gradient(lambda w: Conv.forward(x, w, b, conv_param)[0], w, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(lambda b: Conv.forward(x, w, b, conv_param)[0], b, dout)\n",
    "\n",
    "out, cache = Conv.forward(x, w, b, conv_param)\n",
    "dx, dw, db = Conv.backward(dout, cache)\n",
    "\n",
    "print('Testing Conv.backward function')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx, dx_num))\n",
    "print('dw error: ', eecs598.grad.rel_error(dw, dw_num))\n",
    "print('db error: ', eecs598.grad.rel_error(db, db_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "CS8EsPacrpG8",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Max-pooling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "elO_ys-8w723",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Max-pooling: forward\n",
    "Implement the forward pass for the max-pooling operation. Again, don't worry too much about computational efficiency.\n",
    "\n",
    "After implementing the forward pass for max-pooling, run the following to check your implementation. You should get errors less than `1e-7`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 25034,
     "status": "ok",
     "timestamp": 1601239177871,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "qmNJY6E7w724",
    "new_sheet": false,
    "outputId": "68a0f801-999e-4d6b-98b1-51151de7d8cf",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import MaxPool\n",
    "\n",
    "reset_seed(0)\n",
    "x_shape = torch.tensor((2, 3, 4, 4))\n",
    "x = torch.linspace(-0.3, 0.4, steps=torch.prod(x_shape), dtype=torch.float64, device='cuda').reshape(*x_shape)\n",
    "pool_param = {'pool_width': 2, 'pool_height': 2, 'stride': 2}\n",
    "\n",
    "out, _ = MaxPool.forward(x, pool_param)\n",
    "\n",
    "correct_out = torch.tensor([[[[-0.26315789, -0.24842105],\n",
    "                              [-0.20421053, -0.18947368]],\n",
    "                             [[-0.14526316, -0.13052632],\n",
    "                              [-0.08631579, -0.07157895]],\n",
    "                             [[-0.02736842, -0.01263158],\n",
    "                              [ 0.03157895,  0.04631579]]],\n",
    "                            [[[ 0.09052632,  0.10526316],\n",
    "                              [ 0.14947368,  0.16421053]],\n",
    "                             [[ 0.20842105,  0.22315789],\n",
    "                              [ 0.26736842,  0.28210526]],\n",
    "                             [[ 0.32631579,  0.34105263],\n",
    "                              [ 0.38526316,  0.4       ]]]],\n",
    "                           dtype=torch.float64, device='cuda')\n",
    "\n",
    "# Compare your output with ours. Difference should be on the order of e-8.\n",
    "print('Testing MaxPool.forward function:')\n",
    "print('difference: ', eecs598.grad.rel_error(out, correct_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "lLjMhG4nw728",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Max-pooling: backward\n",
    "Implement the backward pass for the max-pooling operation. You don't need to worry about computational efficiency.\n",
    "\n",
    "Check your implementation of the max pooling backward pass with numeric gradient checking by running the following. You should get errors less than `1e-10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 28868,
     "status": "ok",
     "timestamp": 1601239181736,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "9FTLuSsIw729",
    "new_sheet": false,
    "outputId": "73bd6bc8-cb9d-4669-9c67-98571ae50c1b",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import MaxPool\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(3, 2, 8, 8, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(3, 2, 4, 4, dtype=torch.float64, device='cuda')\n",
    "pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: MaxPool.forward(x, pool_param)[0], x, dout)\n",
    "\n",
    "out, cache = MaxPool.forward(x, pool_param)\n",
    "dx = MaxPool.backward(dout, cache)\n",
    "\n",
    "print('Testing MaxPool.backward function:')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "0-GA5mHcw73A",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Fast layers\n",
    "Making convolution and pooling layers fast can be challenging. To spare you the pain, we've provided fast implementations of the forward and backward passes for convolution and pooling layers. Those can be found at the bottom of `convolutional_networks.py`\n",
    "\n",
    "The fast convolution implementation depends on `torch.nn`\n",
    "\n",
    "The API for the fast versions of the convolution and pooling layers is exactly the same as the naive versions that you implemented above: the forward pass receives data, weights, and parameters and produces outputs and a cache object; the backward pass recieves upstream derivatives and the cache object and produces gradients with respect to the data and weights.\n",
    "\n",
    "```python\n",
    "class FastConv(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, conv_param):\n",
    "    N, C, H, W = x.shape\n",
    "    F, _, HH, WW = w.shape\n",
    "    stride, pad = conv_param['stride'], conv_param['pad']\n",
    "    layer = torch.nn.Conv2d(C, F, (HH, WW), stride=stride, padding=pad)\n",
    "    layer.weight = torch.nn.Parameter(w)\n",
    "    layer.bias = torch.nn.Parameter(b)\n",
    "    tx = x.detach()\n",
    "    tx.requires_grad = True\n",
    "    out = layer(tx)\n",
    "    cache = (x, w, b, conv_param, tx, out, layer)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    try:\n",
    "      x, _, _, _, tx, out, layer = cache\n",
    "      out.backward(dout)\n",
    "      dx = tx.grad.detach()\n",
    "      dw = layer.weight.grad.detach()\n",
    "      db = layer.bias.grad.detach()\n",
    "      layer.weight.grad = layer.bias.grad = None\n",
    "    except RuntimeError:\n",
    "      dx, dw, db = torch.zeros_like(tx), torch.zeros_like(layer.weight), torch.zeros_like(layer.bias)\n",
    "    return dx, dw, db\n",
    "\n",
    "\n",
    "class FastMaxPool(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, pool_param):\n",
    "    N, C, H, W = x.shape\n",
    "    pool_height, pool_width = pool_param['pool_height'], pool_param['pool_width']\n",
    "    stride = pool_param['stride']\n",
    "    layer = torch.nn.MaxPool2d(kernel_size=(pool_height, pool_width), stride=stride)\n",
    "    tx = x.detach()\n",
    "    tx.requires_grad = True\n",
    "    out = layer(tx)\n",
    "    cache = (x, pool_param, tx, out, layer)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    try:\n",
    "      x, _, tx, out, layer = cache\n",
    "      out.backward(dout)\n",
    "      dx = tx.grad.detach()\n",
    "    except RuntimeError:\n",
    "      dx = torch.zeros_like(tx)\n",
    "    return dx\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Z7B_QYt206g4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "\n",
    "We will now compare three different implementations of convolution (both forward and backward):\n",
    "\n",
    "1. Your naive, non-vectorized implementation on CPU\n",
    "2. The fast, vectorized implementation on CPU\n",
    "3. The fast, vectorized implementation on GPU\n",
    "\n",
    "The differences between your implementation and FastConv should be less than `1e-10`. When moving from your implementation to FastConv CPU, you will likely see speedups of at least 100x. When comparing your implementation to FastConv CUDA, you will likely see speedups of more than 500x. (These speedups are not hard requirements for this assignment since we are not asking you to write any vectorized implementations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 40611,
     "status": "ok",
     "timestamp": 1601239193509,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "mSbc5Ttvw73C",
    "new_sheet": false,
    "outputId": "2cd775f5-0b01-48bc-b17d-6f4058e1c3df",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Rel errors should be around e-11 or less\n",
    "from convolutional_networks import Conv, FastConv\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(10, 3, 31, 31, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(25, 3, 3, 3, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(25, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(10, 25, 16, 16, dtype=torch.float64, device='cuda')\n",
    "x_cuda, w_cuda, b_cuda, dout_cuda = x.to('cuda'), w.to('cuda'), b.to('cuda'), dout.to('cuda')\n",
    "conv_param = {'stride': 2, 'pad': 1}\n",
    "\n",
    "t0 = time.time()\n",
    "out_naive, cache_naive = Conv.forward(x, w, b, conv_param)\n",
    "t1 = time.time()\n",
    "out_fast, cache_fast = FastConv.forward(x, w, b, conv_param)\n",
    "t2 = time.time()\n",
    "out_fast_cuda, cache_fast_cuda = FastConv.forward(x_cuda, w_cuda, b_cuda, conv_param)\n",
    "t3 = time.time()\n",
    "\n",
    "print('Testing FastConv.forward:')\n",
    "print('Naive: %fs' % (t1 - t0))\n",
    "print('Fast: %fs' % (t2 - t1))\n",
    "print('Fast CUDA: %fs' % (t3 - t2))\n",
    "print('Speedup: %fx' % ((t1 - t0) / (t2 - t1)))\n",
    "print('Speedup CUDA: %fx' % ((t1 - t0) / (t3 - t2)))\n",
    "print('Difference: ', eecs598.grad.rel_error(out_naive, out_fast))\n",
    "print('Difference CUDA: ', eecs598.grad.rel_error(out_naive, out_fast_cuda.to(out_naive.device)))\n",
    "\n",
    "t0 = time.time()\n",
    "dx_naive, dw_naive, db_naive = Conv.backward(dout, cache_naive)\n",
    "t1 = time.time()\n",
    "dx_fast, dw_fast, db_fast = FastConv.backward(dout, cache_fast)\n",
    "t2 = time.time()\n",
    "dx_fast_cuda, dw_fast_cuda, db_fast_cuda = FastConv.backward(dout_cuda, cache_fast_cuda)\n",
    "t3 = time.time()\n",
    "\n",
    "print('\\nTesting FastConv.backward:')\n",
    "print('Naive: %fs' % (t1 - t0))\n",
    "print('Fast: %fs' % (t2 - t1))\n",
    "print('Fast CUDA: %fs' % (t3 - t2))\n",
    "print('Speedup: %fx' % ((t1 - t0) / (t2 - t1)))\n",
    "print('Speedup CUDA: %fx' % ((t1 - t0) / (t3 - t2)))\n",
    "print('dx difference: ', eecs598.grad.rel_error(dx_naive, dx_fast))\n",
    "print('dw difference: ', eecs598.grad.rel_error(dw_naive, dw_fast))\n",
    "print('db difference: ', eecs598.grad.rel_error(db_naive, db_fast))\n",
    "print('dx difference CUDA: ', eecs598.grad.rel_error(dx_naive, dx_fast_cuda.to(dx_naive.device)))\n",
    "print('dw difference CUDA: ', eecs598.grad.rel_error(dw_naive, dw_fast_cuda.to(dw_naive.device)))\n",
    "print('db difference CUDA: ', eecs598.grad.rel_error(db_naive, db_fast_cuda.to(db_naive.device)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "HdvR1krO2P_q",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will now similarly compare your naive implementation of max pooling against the fast implementation. You should see differences of 0 between your implementation and the fast implementation.\n",
    "\n",
    "When comparing your implementation against FastMaxPool on CPU, you will likely see speedups of more than 100x. When comparing your implementation against FastMaxPool on GPU, you will likely see speedups of more than 500x."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 46724,
     "status": "ok",
     "timestamp": 1601239199650,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "R0fykwCiw73F",
    "new_sheet": false,
    "outputId": "f6514ddc-063f-469e-b739-3d661d873f89",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Relative errors should be close to 0.0\n",
    "from convolutional_networks import Conv, MaxPool, FastConv, FastMaxPool\n",
    "\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(40, 3, 32, 32, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(40, 3, 16, 16, dtype=torch.float64, device='cuda')\n",
    "x_cuda, dout_cuda = x.to('cuda'), dout.to('cuda')\n",
    "pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}\n",
    "\n",
    "t0 = time.time()\n",
    "out_naive, cache_naive = MaxPool.forward(x, pool_param)\n",
    "t1 = time.time()\n",
    "out_fast, cache_fast = FastMaxPool.forward(x, pool_param)\n",
    "t2 = time.time()\n",
    "out_fast_cuda, cache_fast_cuda = FastMaxPool.forward(x_cuda, pool_param)\n",
    "t3 = time.time()\n",
    "\n",
    "print('Testing FastMaxPool.forward:')\n",
    "print('Naive: %fs' % (t1 - t0))\n",
    "print('Fast: %fs' % (t2 - t1))\n",
    "print('Fast CUDA: %fs' % (t3 - t2))\n",
    "print('Speedup: %fx' % ((t1 - t0) / (t2 - t1)))\n",
    "print('Speedup CUDA: %fx' % ((t1 - t0) / (t3 - t2)))\n",
    "print('Difference: ', eecs598.grad.rel_error(out_naive, out_fast))\n",
    "print('Difference CUDA: ', eecs598.grad.rel_error(out_naive, out_fast_cuda.to(out_naive.device)))\n",
    "\n",
    "t0 = time.time()\n",
    "dx_naive = MaxPool.backward(dout, cache_naive)\n",
    "t1 = time.time()\n",
    "dx_fast = FastMaxPool.backward(dout, cache_fast)\n",
    "t2 = time.time()\n",
    "dx_fast_cuda = FastMaxPool.backward(dout_cuda, cache_fast_cuda)\n",
    "t3 = time.time()\n",
    "\n",
    "print('\\nTesting FastMaxPool.backward:')\n",
    "print('Naive: %fs' % (t1 - t0))\n",
    "print('Fast: %fs' % (t2 - t1))\n",
    "print('Fast CUDA: %fs' % (t3 - t2))\n",
    "print('Speedup: %fx' % ((t1 - t0) / (t2 - t1)))\n",
    "print('Speedup CUDA: %fx' % ((t1 - t0) / (t3 - t2)))\n",
    "print('dx difference: ', eecs598.grad.rel_error(dx_naive, dx_fast))\n",
    "print('dx difference CUDA: ', eecs598.grad.rel_error(dx_naive, dx_fast_cuda.to(dx_naive.device)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Ix9ih6sKw73I",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Convolutional \"sandwich\" layers\n",
    "Previously we introduced the concept of \"sandwich\" layers that combine multiple operations into commonly used patterns. Below you will find sandwich layers that implement a few commonly used patterns for convolutional networks. We've included them at the bottom of `covolutional_networks.py` Run the cells below to sanity check they're working.\n",
    "\n",
    "**Note:** This will be using the ReLU function you implemented in the previous notebook. Make sure to implement it first. \n",
    "\n",
    "```python\n",
    "class Conv_ReLU(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, conv_param):\n",
    "    \"\"\"\n",
    "    A convenience layer that performs a convolution followed by a ReLU.\n",
    "    Inputs:\n",
    "    - x: Input to the convolutional layer\n",
    "    - w, b, conv_param: Weights and parameters for the convolutional layer\n",
    "    Returns a tuple of:\n",
    "    - out: Output from the ReLU\n",
    "    - cache: Object to give to the backward pass\n",
    "    \"\"\"\n",
    "    a, conv_cache = FastConv.forward(x, w, b, conv_param)\n",
    "    out, relu_cache = ReLU.forward(a)\n",
    "    cache = (conv_cache, relu_cache)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    \"\"\"\n",
    "    Backward pass for the conv-relu convenience layer.\n",
    "    \"\"\"\n",
    "    conv_cache, relu_cache = cache\n",
    "    da = ReLU.backward(dout, relu_cache)\n",
    "    dx, dw, db = FastConv.backward(da, conv_cache)\n",
    "    return dx, dw, db\n",
    "\n",
    "\n",
    "class Conv_ReLU_Pool(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, conv_param, pool_param):\n",
    "    \"\"\"\n",
    "    A convenience layer that performs a convolution, a ReLU, and a pool.\n",
    "    Inputs:\n",
    "    - x: Input to the convolutional layer\n",
    "    - w, b, conv_param: Weights and parameters for the convolutional layer\n",
    "    - pool_param: Parameters for the pooling layer\n",
    "    Returns a tuple of:\n",
    "    - out: Output from the pooling layer\n",
    "    - cache: Object to give to the backward pass\n",
    "    \"\"\"\n",
    "    a, conv_cache = FastConv.forward(x, w, b, conv_param)\n",
    "    s, relu_cache = ReLU.forward(a)\n",
    "    out, pool_cache = FastMaxPool.forward(s, pool_param)\n",
    "    cache = (conv_cache, relu_cache, pool_cache)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    \"\"\"\n",
    "    Backward pass for the conv-relu-pool convenience layer\n",
    "    \"\"\"\n",
    "    conv_cache, relu_cache, pool_cache = cache\n",
    "    ds = FastMaxPool.backward(dout, pool_cache)\n",
    "    da = ReLU.backward(ds, relu_cache)\n",
    "    dx, dw, db = FastConv.backward(da, conv_cache)\n",
    "    return dx, dw, db\n",
    "```\n",
    "\n",
    "Test the implementations of the sandwich layers by running the following. You should see errors less than `1e-7`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 170
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 48795,
     "status": "ok",
     "timestamp": 1601239201749,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "B-0DpEY8w73J",
    "new_sheet": false,
    "outputId": "e0031ce6-c181-462a-fb29-caabddd5dca9",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import Conv_ReLU, Conv_ReLU_Pool\n",
    "reset_seed(0)\n",
    "\n",
    "# Test Conv ReLU\n",
    "x = torch.randn(2, 3, 8, 8, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(3, 3, 3, 3, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(3, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(2, 3, 8, 8, dtype=torch.float64, device='cuda')\n",
    "conv_param = {'stride': 1, 'pad': 1}\n",
    "\n",
    "out, cache = Conv_ReLU.forward(x, w, b, conv_param)\n",
    "dx, dw, db = Conv_ReLU.backward(dout, cache)\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: Conv_ReLU.forward(x, w, b, conv_param)[0], x, dout)\n",
    "dw_num = eecs598.grad.compute_numeric_gradient(lambda w: Conv_ReLU.forward(x, w, b, conv_param)[0], w, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(lambda b: Conv_ReLU.forward(x, w, b, conv_param)[0], b, dout)\n",
    "\n",
    "# Relative errors should be around e-8 or less\n",
    "print('Testing Conv_ReLU:')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dw error: ', eecs598.grad.rel_error(dw_num, dw))\n",
    "print('db error: ', eecs598.grad.rel_error(db_num, db))\n",
    "\n",
    "# Test Conv ReLU Pool\n",
    "x = torch.randn(2, 3, 16, 16, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(3, 3, 3, 3, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(3, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(2, 3, 8, 8, dtype=torch.float64, device='cuda')\n",
    "conv_param = {'stride': 1, 'pad': 1}\n",
    "pool_param = {'pool_height': 2, 'pool_width': 2, 'stride': 2}\n",
    "\n",
    "out, cache = Conv_ReLU_Pool.forward(x, w, b, conv_param, pool_param)\n",
    "dx, dw, db = Conv_ReLU_Pool.backward(dout, cache)\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: Conv_ReLU_Pool.forward(x, w, b, conv_param, pool_param)[0], x, dout)\n",
    "dw_num = eecs598.grad.compute_numeric_gradient(lambda w: Conv_ReLU_Pool.forward(x, w, b, conv_param, pool_param)[0], w, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(lambda b: Conv_ReLU_Pool.forward(x, w, b, conv_param, pool_param)[0], b, dout)\n",
    "\n",
    "# Relative errors should be around e-8 or less\n",
    "print()\n",
    "print('Testing Conv_ReLU_Pool')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dw error: ', eecs598.grad.rel_error(dw_num, dw))\n",
    "print('db error: ', eecs598.grad.rel_error(db_num, db))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Kgp7ymihw73P",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Three-layer convolutional network\n",
    "Now that you have implemented all the necessary layers, we can put them together into a simple convolutional network.\n",
    "\n",
    "Complete the implementation of the `ThreeLayerConvNet` class. We STRONGLY recommend you to use the fast/sandwich layers (already imported for you) in your implementation. Run the following cells to help you debug:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "e_MfqAQXw73Q",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Sanity check loss\n",
    "After you build a new network, one of the first things you should do is sanity check the loss. When we use the softmax loss, we expect the loss for random weights (and no regularization) to be about `log(C)` for `C` classes. When we add regularization the loss should go up slightly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 48767,
     "status": "ok",
     "timestamp": 1601239201750,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "h-XEWaw2w73R",
    "new_sheet": false,
    "outputId": "5be30390-3579-4067-c10b-5471c1ed0b69",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import ThreeLayerConvNet\n",
    "\n",
    "reset_seed(0)\n",
    "model = ThreeLayerConvNet(dtype=torch.float64, device='cuda')\n",
    "\n",
    "N = 50\n",
    "X = torch.randn(N, 3, 32, 32, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(10, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "loss, grads = model.loss(X, y)\n",
    "print('Initial loss (no regularization): ', loss.item())\n",
    "\n",
    "model.reg = 0.5\n",
    "loss, grads = model.loss(X, y)\n",
    "print('Initial loss (with regularization): ', loss.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "QEIViSCjw73U",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Gradient check\n",
    "After the loss looks reasonable, use numeric gradient checking to make sure that your backward pass is correct. When you use numeric gradient checking you should use a small amount of artificial data and a small number of neurons at each layer.\n",
    "\n",
    "You should see errors less than `1e-5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 55687,
     "status": "ok",
     "timestamp": 1601239208699,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "xPrOgIsJw73V",
    "new_sheet": false,
    "outputId": "5937fae9-9a84-49cb-8a0b-4d6263cfa0df",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import ThreeLayerConvNet\n",
    "\n",
    "num_inputs = 2\n",
    "input_dims = (3, 16, 16)\n",
    "reg = 0.0\n",
    "num_classes = 10\n",
    "reset_seed(0)\n",
    "X = torch.randn(num_inputs, *input_dims, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(num_classes, size=(num_inputs,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "model = ThreeLayerConvNet(num_filters=3, filter_size=3,\n",
    "                          input_dims=input_dims, hidden_dim=7,\n",
    "                          weight_scale=5e-2, dtype=torch.float64, device='cuda')\n",
    "loss, grads = model.loss(X, y)\n",
    "\n",
    "for param_name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    param_grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[param_name])\n",
    "    print('%s max relative error: %e' % (param_name, eecs598.grad.rel_error(param_grad_num, grads[param_name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dUPRjnzww73Y",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Overfit small data\n",
    "A nice trick is to train your model with just a few training samples. You should be able to overfit small datasets, which will result in very high training accuracy and comparatively low validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 61860,
     "status": "ok",
     "timestamp": 1601239214900,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "pwwQ0XB7w73Z",
    "new_sheet": false,
    "outputId": "8a74748d-3cd4-4242-cc4d-c3fbcb73f07f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import ThreeLayerConvNet\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "num_train = 100\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "\n",
    "model = ThreeLayerConvNet(weight_scale=1e-3, dtype=torch.float32, device='cuda')\n",
    "\n",
    "solver = Solver(model, small_data,\n",
    "                num_epochs=30, batch_size=50,\n",
    "                update_rule=adam,\n",
    "                optim_config={\n",
    "                  'learning_rate': 2e-3,\n",
    "                },\n",
    "                verbose=True, print_every=1,\n",
    "                device='cuda')\n",
    "solver.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "KTDOqqdLw73d",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Plotting the loss, training accuracy, and validation accuracy should show clear overfitting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 603
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 61977,
     "status": "ok",
     "timestamp": 1601239215045,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "fypbffqsw73f",
    "new_sheet": false,
    "outputId": "bfc092db-fb3e-4209-e904-eb892b6f4802",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "plt.title('Training losses')\n",
    "plt.plot(solver.loss_history, 'o')\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('loss')\n",
    "plt.gcf().set_size_inches(9, 4)\n",
    "plt.show()\n",
    "\n",
    "plt.title('Train and Val accuracies')\n",
    "plt.plot(solver.train_acc_history, '-o')\n",
    "plt.plot(solver.val_acc_history, '-o')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.xlabel('epoch')\n",
    "plt.ylabel('accuracy')\n",
    "plt.gcf().set_size_inches(9, 4)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "W2vnSbfjw73i",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Train the net\n",
    "By training the three-layer convolutional network for one epoch, you should achieve greater than 50% accuracy on the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 66496,
     "status": "ok",
     "timestamp": 1601239219592,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "nfArKG-Gw73j",
    "new_sheet": false,
    "outputId": "2d4f7f04-6d6b-42fc-cf17-49938bbe2ab4",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import ThreeLayerConvNet\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "model = ThreeLayerConvNet(weight_scale=0.001, hidden_dim=500, reg=0.001, dtype=torch.float, device='cuda')\n",
    "\n",
    "solver = Solver(model, data_dict,\n",
    "                num_epochs=1, batch_size=64,\n",
    "                update_rule=adam,\n",
    "                optim_config={\n",
    "                  'learning_rate': 2e-3,\n",
    "                },\n",
    "                verbose=True, print_every=50, device='cuda')\n",
    "solver.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "vIYQ0nm2w73n",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Visualize Filters\n",
    "You can visualize the first-layer convolutional filters from the trained network by running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 66469,
     "status": "ok",
     "timestamp": 1601239219593,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "n3FLipRY4NUv",
    "new_sheet": false,
    "outputId": "f22ed38d-581e-4b7a-802c-36f50665b497",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from torchvision.utils import make_grid\n",
    "nrow = math.ceil(math.sqrt(model.params['W1'].shape[0]))\n",
    "grid = make_grid(model.params['W1'], nrow=nrow, padding=1, normalize=True, scale_each=True)\n",
    "plt.imshow(grid.to(device='cpu').permute(1, 2, 0))\n",
    "plt.axis('off')\n",
    "plt.gcf().set_size_inches(5, 5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "JleotK9yDcyv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Deep convolutional network\n",
    "Next you will implement a deep convolutional network with an arbitrary number of conv layers in VGGNet style.\n",
    "\n",
    "Read through the `DeepConvNet` class.\n",
    "\n",
    "Implement the initialization, the forward pass, and the backward pass. For the moment don't worry about implementing batch normalization; we will add those features soon. Again, we STRONGLY recommend you to use the fast/sandwich layers (already imported for you) in your implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "0AC0R6dv059E",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Sanity check loss\n",
    "After you build a new network, one of the first things you should do is sanity check the loss. When we use the softmax loss, we expect the loss for random weights (and no regularization) to be about `log(C)` for `C` classes. When we add regularization the loss should go up slightly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 66550,
     "status": "ok",
     "timestamp": 1601239219701,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "MPfK4L5P059L",
    "new_sheet": false,
    "outputId": "85f4dbfd-cb96-4885-8419-510ab02e2bd5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "input_dims = (3, 32, 32)\n",
    "model = DeepConvNet(num_filters=[8, 64], max_pools=[0, 1], dtype=torch.float64, device='cuda')\n",
    "\n",
    "N = 50\n",
    "X = torch.randn(N, *input_dims, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(10, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "loss, grads = model.loss(X, y)\n",
    "print('Initial loss (no regularization): ', loss.item())\n",
    "\n",
    "model.reg = 1.\n",
    "loss, grads = model.loss(X, y)\n",
    "print('Initial loss (with regularization): ', loss.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "8BfRDkEj1TX8",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Gradient check\n",
    "After the loss looks reasonable, use numeric gradient checking to make sure that your backward pass is correct. When you use numeric gradient checking you should use a small amount of artifical data and a small number of neurons at each layer.\n",
    "\n",
    "You should see relative errors less than `1e-5`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 92569,
     "status": "ok",
     "timestamp": 1601239245748,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "qiPiP1X11TYA",
    "new_sheet": false,
    "outputId": "c28e7eb3-62d1-4679-c9b1-402f61abd471",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "num_inputs = 2\n",
    "input_dims = (3, 8, 8)\n",
    "num_classes = 10\n",
    "X = torch.randn(N, *input_dims, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(10, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "for reg in [0, 3.14]:\n",
    "  print('Running check with reg = ', reg)\n",
    "  model = DeepConvNet(input_dims=input_dims, num_classes=num_classes,\n",
    "                      num_filters=[8, 8, 8],\n",
    "                      max_pools=[0, 2],\n",
    "                      reg=reg,\n",
    "                      weight_scale=5e-2, dtype=torch.float64, device='cuda')\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  # The relative errors should be up to the order of e-6\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[name])\n",
    "    print('%s max relative error: %e' % (name, eecs598.grad.rel_error(grad_num, grads[name])))\n",
    "  if reg == 0: print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "1_njNfEh3cxs",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Overfit small data\n",
    "As another sanity check, make sure you can overfit a small dataset of 50 images. In the following cell, tweak the **learning rate** and **weight initialization scale** to overfit and achieve 100% training accuracy within 30 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 99296,
     "status": "ok",
     "timestamp": 1601239252504,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "2NccCDJ3e1DR",
    "new_sheet": false,
    "outputId": "5ae0edb6-2605-48c2-a5d2-2d04c9d4c01e",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# TODO: Use a DeepConvNet to overfit 50 training examples by \n",
    "# tweaking just the learning rate and initialization scale.\n",
    "from convolutional_networks import DeepConvNet, find_overfit_parameters\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "num_train = 50\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "input_dims = small_data['X_train'].shape[1:]\n",
    "\n",
    "\n",
    "# Update the parameters in find_overfit_parameters in convolutional_networks.py\n",
    "weight_scale, learning_rate = find_overfit_parameters()\n",
    "\n",
    "\n",
    "model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                    num_filters=[8, 16, 32, 64],\n",
    "                    max_pools=[0, 1, 2, 3],\n",
    "                    reg=1e-5, weight_scale=weight_scale, dtype=torch.float32, device='cuda')\n",
    "solver = Solver(model, small_data,\n",
    "                print_every=10, num_epochs=30, batch_size=10,\n",
    "                update_rule=adam,\n",
    "                optim_config={\n",
    "                  'learning_rate': learning_rate,\n",
    "                },\n",
    "                device='cuda',\n",
    "         )\n",
    "# Turn off keep_best_params to allow final weights to be saved, instead of best weights on validation set.\n",
    "solver.train(return_best_params=False)\n",
    "\n",
    "plt.plot(solver.loss_history, 'o')\n",
    "plt.title('Training loss history')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Training loss')\n",
    "plt.show()\n",
    "\n",
    "val_acc = solver.check_accuracy(\n",
    "                        solver.X_train, solver.y_train, num_samples=solver.num_train_samples\n",
    "                    )\n",
    "print(val_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "rkoW8ir6N0aP",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If you're happy with the model's perfromance, run the following cell to save it. \n",
    "\n",
    "We will also reload the model and run it on the training data to verify it's the right weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 99268,
     "status": "ok",
     "timestamp": 1601239252505,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "5p-ZeicrNz8j",
    "new_sheet": false,
    "outputId": "2b048fb6-621c-4389-d373-620ae3bbe9f1",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'overfit_deepconvnet.pth')\n",
    "solver.model.save(path)\n",
    "\n",
    "# Create a new instance\n",
    "model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                    num_filters=[8, 16, 32, 64],\n",
    "                    max_pools=[0, 1, 2, 3],\n",
    "                    reg=1e-5, weight_scale=weight_scale, dtype=torch.float32, device='cuda')\n",
    "solver = Solver(model, small_data,\n",
    "                print_every=10, num_epochs=30, batch_size=10,\n",
    "                update_rule=adam,\n",
    "                optim_config={\n",
    "                  'learning_rate': learning_rate,\n",
    "                },\n",
    "                device='cuda',\n",
    "         )\n",
    "\n",
    "\n",
    "# Load model\n",
    "solver.model.load(path, dtype=torch.float32, device='cuda')\n",
    "\n",
    "# Evaluate on validation set\n",
    "accuracy = solver.check_accuracy(small_data['X_train'], small_data['y_train'])\n",
    "print(f\"Saved model's accuracy on training is {accuracy}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "FI3LznMXRnad",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Kaiming initialization\n",
    "So far, you manually tuned the weight scale and for weight initialization.\n",
    "However, this is inefficient when it comes to training deep neural networks; practically, as your weight matrix is larger, the weight scale should be small.\n",
    "Below you will implement [Kaiming initialization](http://arxiv-web3.library.cornell.edu/abs/1502.01852). For more details, refer to [cs231n note](http://cs231n.github.io/neural-networks-2/#init) and [PyTorch documentation](https://pytorch.org/docs/stable/nn.init.html#torch.nn.init.kaiming_normal_)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "XkZL_lsufSVZ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Convolutional nets with Kaiming initialization\n",
    "Now that you have a working implementation for Kaiming initialization, go back to your [`DeepConvnet`](#scrollTo=Ah-_nwx2BSxl). Modify your implementation to add Kaiming initialization.\n",
    "\n",
    "Concretely, when the `weight_scale` is set to `'kaiming'` in the constructor, you should initialize weights of convolutional and linear layers using `kaiming_initializer`. Once you are done, run the following to see the effect of kaiming initialization in deep CNNs.\n",
    "\n",
    "In this experiment, we train a 31-layer network with four different weight initialization schemes. Among them, only the Kaiming initialization method should achieve a non-random accuracy after one epoch of training.\n",
    "\n",
    "You may see `nan` loss when `weight_scale` is large, this shows a catastrophe of inappropriate weight initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 493
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 167698,
     "status": "ok",
     "timestamp": 1601239320965,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "sQJKcH60jfTm",
    "new_sheet": false,
    "outputId": "86249b59-67d5-40e9-c393-bf0e20813181",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "from fully_connected_networks import sgd_momentum\n",
    "reset_seed(0)\n",
    "\n",
    "# Try training a deep convolutional net with different weight initialization methods\n",
    "num_train = 10000\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "input_dims = data_dict['X_train'].shape[1:]\n",
    "\n",
    "weight_scales = ['kaiming', 1e-1, 1e-2, 1e-3]\n",
    "\n",
    "solvers = []\n",
    "for weight_scale in weight_scales:\n",
    "  print('Solver with weight scale: ', weight_scale)\n",
    "  model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                      num_filters=([8] * 10) + ([32] * 10) + ([128] * 10),\n",
    "                      max_pools=[9, 19],\n",
    "                      weight_scale=weight_scale,\n",
    "                      reg=1e-5, \n",
    "                      dtype=torch.float32,\n",
    "                      device='cuda'\n",
    "                      )\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=1, batch_size=128,\n",
    "                  update_rule=sgd_momentum,\n",
    "                  optim_config={\n",
    "                    'learning_rate': 2e-3,\n",
    "                  },\n",
    "                  print_every=20, device='cuda')\n",
    "  solver.train()\n",
    "  solvers.append(solver)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 168401,
     "status": "ok",
     "timestamp": 1601239321696,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "uG-dQDRCH_XD",
    "new_sheet": false,
    "outputId": "b87dc2be-6671-4708-e65b-8e663870580f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def plot_training_history_init(title, xlabel, solvers, labels, plot_fn, marker='-o'):\n",
    "  plt.title(title)\n",
    "  plt.xlabel(xlabel)\n",
    "  for solver, label in zip(solvers, labels):    \n",
    "    data = plot_fn(solver)\n",
    "    label = 'weight_scale=' + str(label)\n",
    "    plt.plot(data, marker, label=label)\n",
    "  plt.legend(loc='lower center', ncol=len(solvers))\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "plot_training_history_init('Training loss','Iteration', solvers, weight_scales,\n",
    "                            lambda x: x.loss_history, marker='o')\n",
    "plt.subplot(3, 1, 2)\n",
    "plot_training_history_init('Training accuracy','Epoch', solvers, weight_scales,\n",
    "                           lambda x: x.train_acc_history)\n",
    "plt.subplot(3, 1, 3)\n",
    "plot_training_history_init('Validation accuracy','Epoch', solvers, weight_scales,\n",
    "                           lambda x: x.val_acc_history)\n",
    "plt.gcf().set_size_inches(15, 25)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ogx_-gJ1e1EO",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Train a good model!\n",
    "Train the best convolutional model that you can on CIFAR-10, storing your best model in the `best_model` variable. We require you to get at least 71% accuracy on the validation set using a convolutional net, within 60 seconds of training.\n",
    "\n",
    "You might find it useful to use batch normalization in your model. However, since we do not ask you to implement it CUDA-friendly, it might slow down training. \n",
    "\n",
    "**Implement** `create_convolutional_solver_instance` while making sure to use the initialize your model with the input `dtype` and `device`, as well as initializing the solver on the input `device`. \n",
    "\n",
    "Hint: Your model does not have to be too deep.\n",
    "\n",
    "Hint 2: We used `batch_size = 128` for training a model with 74% validation accuracy. You don't have to follow this, but it would save your time for hyperparameter search.\n",
    "\n",
    "Hint 3: Note that we import all the functions from fully_connected_networks, so feel free to use the optimizers you've already imolemented; e.g., adam. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 391
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 220401,
     "status": "ok",
     "timestamp": 1601239373726,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "UPRQvUDJe1EQ",
    "new_sheet": false,
    "outputId": "f0f4ad71-7804-4c75-c59f-4585475a3412",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet, create_convolutional_solver_instance\n",
    "\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = True\n",
    "\n",
    "solver = create_convolutional_solver_instance(data_dict, torch.float32, \"cuda\")\n",
    "\n",
    "solver.train(time_limit=60)\n",
    "\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "uZWeaN-6e1ET",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Test your model!\n",
    "Run your best model on the validation and test sets. You should achieve above 71% accuracy on the validation set and 70% accuracy on the test set.\n",
    "\n",
    "(Our best model gets 74.3% validation accuracy and 73.5% test accuracy -- can you beat ours?)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 222841,
     "status": "ok",
     "timestamp": 1601239376193,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "xdVs_GEse1EU",
    "new_sheet": false,
    "outputId": "3b09f083-e4b5-4928-82ec-b67e8c8bebc4",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "print('Validation set accuracy: ', solver.check_accuracy(data_dict['X_val'], data_dict['y_val']))\n",
    "print('Test set accuracy: ', solver.check_accuracy(data_dict['X_test'], data_dict['y_test']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "M6hnNF-pR235",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If you're happy with the model's perfromance, run the following cell to save it. \n",
    "\n",
    "We will also reload the model and run it on the training data to verify it's the right weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 225507,
     "status": "ok",
     "timestamp": 1601239378887,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "hL6X4C-yR4xZ",
    "new_sheet": false,
    "outputId": "db245102-106e-4fcc-c5d8-c08d2c955100",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'one_minute_deepconvnet.pth')\n",
    "solver.model.save(path)\n",
    "\n",
    "# Create a new instance\n",
    "from convolutional_networks import DeepConvNet, create_convolutional_solver_instance\n",
    "\n",
    "solver = create_convolutional_solver_instance(data_dict, torch.float32, \"cuda\")\n",
    "\n",
    "# Load model\n",
    "solver.model.load(path, dtype=torch.float32, device='cuda')\n",
    "\n",
    "# Evaluate on validation set\n",
    "print('Validation set accuracy: ', solver.check_accuracy(data_dict['X_val'], data_dict['y_val']))\n",
    "print('Test set accuracy: ', solver.check_accuracy(data_dict['X_test'], data_dict['y_test']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "KqsqNYOVwypM",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Batch Normalization\n",
    "One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. \n",
    "One idea along these lines is batch normalization which was proposed by [1] in 2015.\n",
    "\n",
    "The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.\n",
    "\n",
    "The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.\n",
    "\n",
    "It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.\n",
    "\n",
    "[1] [Sergey Ioffe and Christian Szegedy, \"Batch Normalization: Accelerating Deep Network Training by Reducing\n",
    "Internal Covariate Shift\", ICML 2015.](https://arxiv.org/abs/1502.03167)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "s0ELNJIlwypX",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Batch normalization: forward\n",
    "Implement the batch normalization forward pass in the function `BatchNorm.forward`. Once you have done so, run the following to test your implementation.\n",
    "\n",
    "Referencing the paper linked to above in [1] may be helpful!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "q9cFW4heSrpt",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After implementing the forward pass for batch normalization, you can run the following to sanity check your implementation. After running batch normalization with beta=0 and gamma=1, the data should have zero mean and unit variance.\n",
    "\n",
    "After running batch normalization with nontrivial beta and gamma, the output data should have mean approximately equal to beta, and std approximatly equal to gamma."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 221
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 225478,
     "status": "ok",
     "timestamp": 1601239378888,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "7iB1mAkLwypZ",
    "new_sheet": false,
    "outputId": "8d3f081a-a2a4-49bd-d3f9-306e0611907c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Check the training-time forward pass by checking means and variances\n",
    "# of features both before and after batch normalization\n",
    "from convolutional_networks import BatchNorm\n",
    "\n",
    "def print_mean_std(x,dim=0):\n",
    "  means = ['%.3f' % xx for xx in x.mean(dim=dim).tolist()]\n",
    "  stds = ['%.3f' % xx for xx in x.std(dim=dim).tolist()]\n",
    "  print('  means: ', means)\n",
    "  print('  stds:  ', stds)\n",
    "  print()\n",
    "\n",
    "# Simulate the forward pass for a two-layer network\n",
    "reset_seed(0)\n",
    "N, D1, D2, D3 = 200, 50, 60, 3\n",
    "X = torch.randn(N, D1, dtype=torch.float64, device='cuda')\n",
    "W1 = torch.randn(D1, D2, dtype=torch.float64, device='cuda')\n",
    "W2 = torch.randn(D2, D3, dtype=torch.float64, device='cuda')\n",
    "a = X.matmul(W1).clamp(min=0.).matmul(W2)\n",
    "\n",
    "print('Before batch normalization:')\n",
    "print_mean_std(a,dim=0)\n",
    "\n",
    "# Run with gamma=1, beta=0. Means should be close to zero and stds close to one\n",
    "gamma = torch.ones(D3, dtype=torch.float64, device='cuda')\n",
    "beta = torch.zeros(D3, dtype=torch.float64, device='cuda')\n",
    "print('After batch normalization (gamma=1, beta=0)')\n",
    "a_norm, _ = BatchNorm.forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,dim=0)\n",
    "\n",
    "# Run again with nontrivial gamma and beta. Now means should be close to beta\n",
    "# and std should be close to gamma.\n",
    "gamma = torch.tensor([1.0, 2.0, 3.0], dtype=torch.float64, device='cuda')\n",
    "beta = torch.tensor([11.0, 12.0, 13.0], dtype=torch.float64, device='cuda')\n",
    "print('After batch normalization (gamma=', gamma.tolist(), ', beta=', beta.tolist(), ')')\n",
    "a_norm, _ = BatchNorm.forward(a, gamma, beta, {'mode': 'train'})\n",
    "print_mean_std(a_norm,dim=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "qGUPz4t_Tam4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We can sanity-check the test-time forward pass of batch normalization by running the following. First we run the training-time forward pass many times to \"warm up\" the running averages. If we then run a test-time forward pass, the output should have approximately zero mean and unit variance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 225620,
     "status": "ok",
     "timestamp": 1601239379058,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "w8lPQyI9wype",
    "new_sheet": false,
    "outputId": "356d790b-669f-4ec4-9503-c66f178c3e21",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import BatchNorm\n",
    "\n",
    "reset_seed(0)\n",
    "N, D1, D2, D3 = 200, 50, 60, 3\n",
    "W1 = torch.randn(D1, D2, dtype=torch.float64, device='cuda')\n",
    "W2 = torch.randn(D2, D3, dtype=torch.float64, device='cuda')\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "gamma = torch.ones(D3, dtype=torch.float64, device='cuda')\n",
    "beta = torch.zeros(D3, dtype=torch.float64, device='cuda')\n",
    "\n",
    "for t in range(500):\n",
    "  X = torch.randn(N, D1, dtype=torch.float64, device='cuda')\n",
    "  a = X.matmul(W1).clamp(min=0.).matmul(W2)\n",
    "  BatchNorm.forward(a, gamma, beta, bn_param)\n",
    "\n",
    "bn_param['mode'] = 'test'\n",
    "X = torch.randn(N, D1, dtype=torch.float64, device='cuda')\n",
    "a = X.matmul(W1).clamp(min=0.).matmul(W2)\n",
    "a_norm, _ = BatchNorm.forward(a, gamma, beta, bn_param)\n",
    "\n",
    "# Means should be close to zero and stds close to one, but will be\n",
    "# noisier than training-time forward passes.\n",
    "print('After batch normalization (test-time):')\n",
    "print_mean_std(a_norm,dim=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "jt0hsHxIwypj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Batch normalization: backward\n",
    "Now implement the backward pass for batch normalization in the function `BatchNorm.backward`.\n",
    "\n",
    "To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.\n",
    "\n",
    "Please don't forget to implement the train and test mode separately.\n",
    "\n",
    "Once you have finished, run the following to numerically check your backward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 225592,
     "status": "ok",
     "timestamp": 1601239379058,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "n2pMzvTqwypk",
    "new_sheet": false,
    "outputId": "eecb5521-0930-45ee-9e70-fee586a60c66",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import BatchNorm\n",
    "\n",
    "# Gradient check batchnorm backward pass\n",
    "reset_seed(0)\n",
    "N, D = 4, 5\n",
    "x = 5 * torch.randn(N, D, dtype=torch.float64, device='cuda') + 12\n",
    "gamma = torch.randn(D, dtype=torch.float64, device='cuda')\n",
    "beta = torch.randn(D, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(N, D, dtype=torch.float64, device='cuda')\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "fx = lambda x: BatchNorm.forward(x, gamma, beta, bn_param)[0]\n",
    "fg = lambda a: BatchNorm.forward(x, a, beta, bn_param)[0]\n",
    "fb = lambda b: BatchNorm.forward(x, gamma, b, bn_param)[0]\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(fx, x, dout)\n",
    "da_num = eecs598.grad.compute_numeric_gradient(fg, gamma.clone(), dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(fb, beta.clone(), dout)\n",
    "\n",
    "_, cache = BatchNorm.forward(x, gamma, beta, bn_param)\n",
    "dx, dgamma, dbeta = BatchNorm.backward(dout, cache)\n",
    "# You should expect to see relative errors between 1e-12 and 1e-9\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dgamma error: ', eecs598.grad.rel_error(da_num, dgamma))\n",
    "print('dbeta error: ', eecs598.grad.rel_error(db_num, dbeta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "hFxuZMUAwypp",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## (Challenge Question) Batch Normalization: alternative backward\n",
    "In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.\n",
    "\n",
    "Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!  \n",
    "\n",
    "In the forward pass, given a set of inputs $X=\\begin{bmatrix}x_1\\\\x_2\\\\...\\\\x_N\\end{bmatrix}$, \n",
    "\n",
    "we first calculate the mean $\\mu$ and variance $v$.\n",
    "With $\\mu$ and $v$ calculated, we can calculate the standard deviation $\\sigma$  and normalized data $Y$.\n",
    "The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).\n",
    "\n",
    "\\begin{align}\n",
    "& \\mu=\\frac{1}{N}\\sum_{k=1}^N x_k  &  v=\\frac{1}{N}\\sum_{k=1}^N (x_k-\\mu)^2 \\\\\n",
    "& \\sigma=\\sqrt{v+\\epsilon}         &  y_i=\\frac{x_i-\\mu}{\\sigma}\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "gVoopiQ7wypr",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<img src=\"https://web.eecs.umich.edu/~justincj/teaching/eecs498/assets/a3/batchnorm_graph.png\" width=691 height=202>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "EDTJ6AXawypt",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "The meat of our problem during backpropagation is to compute $\\frac{\\partial L}{\\partial X}$, given the upstream gradient we receive, $\\frac{\\partial L}{\\partial Y}.$ To do this, recall the chain rule in calculus gives us $\\frac{\\partial L}{\\partial X} = \\frac{\\partial L}{\\partial Y} \\cdot \\frac{\\partial Y}{\\partial X}$.\n",
    "\n",
    "The unknown/hart part is $\\frac{\\partial Y}{\\partial X}$. We can find this by first deriving step-by-step our local gradients at \n",
    "$\\frac{\\partial v}{\\partial X}$, $\\frac{\\partial \\mu}{\\partial X}$,\n",
    "$\\frac{\\partial \\sigma}{\\partial v}$, \n",
    "$\\frac{\\partial Y}{\\partial \\sigma}$, and $\\frac{\\partial Y}{\\partial \\mu}$,\n",
    "and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\\frac{\\partial Y}{\\partial X}$.\n",
    "\n",
    "If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\\frac{\\partial L}{\\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\\frac{\\partial \\mu}{\\partial x_i}, \\frac{\\partial v}{\\partial x_i}, \\frac{\\partial \\sigma}{\\partial x_i},$ then assemble these pieces to calculate $\\frac{\\partial y_i}{\\partial x_i}$. \n",
    "\n",
    "You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation. \n",
    "\n",
    "After doing so, implement the simplified batch normalization backward pass in the function `BatchNorm.backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 226313,
     "status": "ok",
     "timestamp": 1601239379807,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "cJZp2i7ywypv",
    "new_sheet": false,
    "outputId": "1a1ec985-647e-4e54-ff2a-7b57210a6980",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import BatchNorm\n",
    "\n",
    "reset_seed(0)\n",
    "N, D = 128, 2048\n",
    "x = 5 * torch.randn(N, D, dtype=torch.float64, device='cuda') + 12\n",
    "gamma = torch.randn(D, dtype=torch.float64, device='cuda')\n",
    "beta = torch.randn(D, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(N, D, dtype=torch.float64, device='cuda')\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "out, cache = BatchNorm.forward(x, gamma, beta, bn_param)\n",
    "\n",
    "t1 = time.time()\n",
    "dx1, dgamma1, dbeta1 = BatchNorm.backward(dout, cache)\n",
    "t2 = time.time()\n",
    "dx2, dgamma2, dbeta2 = BatchNorm.backward_alt(dout, cache)\n",
    "t3 = time.time()\n",
    "\n",
    "print('dx difference: ', eecs598.grad.rel_error(dx1, dx2))\n",
    "print('dgamma difference: ', eecs598.grad.rel_error(dgamma1, dgamma2))\n",
    "print('dbeta difference: ', eecs598.grad.rel_error(dbeta1, dbeta2))\n",
    "print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "uIJWjzFZw73z",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Spatial Batch Normalization\n",
    "As proposed in the original paper, batch normalization can also be used for convolutional networks, but we need to tweak it a bit; the modification will be called \"spatial batch normalization.\"\n",
    "\n",
    "Normally batch-normalization accepts inputs of shape `(N, D)` and produces outputs of shape `(N, D)`, where we normalize across the minibatch dimension `N`. For data coming from convolutional layers, batch normalization needs to accept inputs of shape `(N, C, H, W)` and produce outputs of shape `(N, C, H, W)` where the `N` dimension gives the minibatch size and the `(H, W)` dimensions give the spatial size of the feature map.\n",
    "\n",
    "If the feature map was produced using convolutions, then we expect every feature channel's statistics e.g. mean, variance to be relatively consistent both between different images, and different locations within the same image -- after all, every feature channel is produced by the same convolutional filter! Therefore spatial batch normalization computes a mean and variance for each of the `C` feature channels by computing statistics over the minibatch dimension `N` as well the spatial dimensions `H` and `W`.\n",
    "\n",
    "\n",
    "[1] [Sergey Ioffe and Christian Szegedy, \"Batch Normalization: Accelerating Deep Network Training by Reducing\n",
    "Internal Covariate Shift\", ICML 2015.](https://arxiv.org/abs/1502.03167)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "oqCmKCw7w730",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Spatial batch normalization: forward\n",
    "\n",
    "Implement the forward pass for spatial batch normalization in the function `SpatialBatchNorm.forward`. Check your implementation by running the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "5ZHLPa6-UkY1",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "\n",
    "After implementing the forward pass for spatial batch normalization, you can run the following to sanity check your code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 226285,
     "status": "ok",
     "timestamp": 1601239379808,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "3x-vfMnIw732",
    "new_sheet": false,
    "outputId": "1239f771-5a06-4db9-8eb0-6a289d06102d",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import SpatialBatchNorm\n",
    "\n",
    "reset_seed(0)\n",
    "# Check the training-time forward pass by checking means and variances\n",
    "# of features both before and after spatial batch normalization\n",
    "\n",
    "N, C, H, W = 2, 3, 4, 5\n",
    "x = 4 * torch.randn(N, C, H, W, dtype=torch.float64, device='cuda') + 10\n",
    "\n",
    "print('Before spatial batch normalization:')\n",
    "print('  Shape: ', x.shape)\n",
    "print('  Means: ', x.mean(dim=(0, 2, 3)))\n",
    "print('  Stds: ', x.std(dim=(0, 2, 3)))\n",
    "\n",
    "# Means should be close to zero and stds close to one\n",
    "gamma = torch.ones(C, dtype=torch.float64, device='cuda')\n",
    "beta = torch.zeros(C,dtype=torch.float64, device='cuda')\n",
    "bn_param = {'mode': 'train'}\n",
    "out, _ = SpatialBatchNorm.forward(x, gamma, beta, bn_param)\n",
    "print('After spatial batch normalization:')\n",
    "print('  Shape: ', out.shape)\n",
    "print('  Means: ', out.mean(dim=(0, 2, 3)))\n",
    "print('  Stds: ', out.std(dim=(0, 2, 3)))\n",
    "\n",
    "# Means should be close to beta and stds close to gamma\n",
    "gamma = torch.tensor([3, 4, 5], dtype=torch.float64, device='cuda')\n",
    "beta = torch.tensor([6, 7, 8], dtype=torch.float64, device='cuda')\n",
    "out, _ = SpatialBatchNorm.forward(x, gamma, beta, bn_param)\n",
    "print('After spatial batch normalization (nontrivial gamma, beta):')\n",
    "print('  Shape: ', out.shape)\n",
    "print('  Means: ', out.mean(dim=(0, 2, 3)))\n",
    "print('  Stds: ', out.std(dim=(0, 2, 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "RxzqSGqZUr2J",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Similar to the vanilla batch normalization implementation, run the following to sanity-check the test-time forward pass of spatial batch normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 226258,
     "status": "ok",
     "timestamp": 1601239379809,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "7z2Eu2Xlw736",
    "new_sheet": false,
    "outputId": "2b5212fa-9d29-4abc-bf43-15c3fc1ac228",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "reset_seed(0)\n",
    "# Check the test-time forward pass by running the training-time\n",
    "# forward pass many times to warm up the running averages, and then\n",
    "# checking the means and variances of activations after a test-time\n",
    "# forward pass.\n",
    "N, C, H, W = 10, 4, 11, 12\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "gamma = torch.ones(C, dtype=torch.float64, device='cuda')\n",
    "beta = torch.zeros(C, dtype=torch.float64, device='cuda')\n",
    "for t in range(50):\n",
    "  x = 2.3 * torch.randn(N, C, H, W, dtype=torch.float64, device='cuda') + 13\n",
    "  SpatialBatchNorm.forward(x, gamma, beta, bn_param)\n",
    "bn_param['mode'] = 'test'\n",
    "x = 2.3 * torch.randn(N, C, H, W, dtype=torch.float64, device='cuda') + 13\n",
    "a_norm, _ = SpatialBatchNorm.forward(x, gamma, beta, bn_param)\n",
    "\n",
    "# Means should be close to zero and stds close to one, but will be\n",
    "# noisier than training-time forward passes.\n",
    "print('After spatial batch normalization (test-time):')\n",
    "print('  means: ', a_norm.mean(dim=(0, 2, 3)))\n",
    "print('  stds: ', a_norm.std(dim=(0, 2, 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dVdE3j3iw739",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Spatial batch normalization: backward\n",
    "Implement the backward pass for spatial batch normalization in the function `SpatialBatchNorm.backward`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "B5efGyfwU3rK",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "After implementing the backward pass for spatial batch normalization, run the following to perform numeric gradient checking on your implementation. You should see errors less than `1e-6`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 226230,
     "status": "ok",
     "timestamp": 1601239379809,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "5Z2j4mQlw73_",
    "new_sheet": false,
    "outputId": "ee0fa520-6c47-41c1-9d8b-7c523b5f12ce",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "reset_seed(0)\n",
    "N, C, H, W = 2, 3, 4, 5\n",
    "x = 5 * torch.randn(N, C, H, W, dtype=torch.float64, device='cuda') + 12\n",
    "gamma = torch.randn(C, dtype=torch.float64, device='cuda')\n",
    "beta = torch.randn(C, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(N, C, H, W, dtype=torch.float64, device='cuda')\n",
    "\n",
    "bn_param = {'mode': 'train'}\n",
    "fx = lambda x: SpatialBatchNorm.forward(x, gamma, beta, bn_param)[0]\n",
    "fg = lambda a: SpatialBatchNorm.forward(x, gamma, beta, bn_param)[0]\n",
    "fb = lambda b: SpatialBatchNorm.forward(x, gamma, beta, bn_param)[0]\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(fx, x, dout)\n",
    "da_num = eecs598.grad.compute_numeric_gradient(fg, gamma, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(fb, beta, dout)\n",
    "\n",
    "_, cache = SpatialBatchNorm.forward(x, gamma, beta, bn_param)\n",
    "dx, dgamma, dbeta = SpatialBatchNorm.backward(dout, cache)\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dgamma error: ', eecs598.grad.rel_error(da_num, dgamma))\n",
    "print('dbeta error: ', eecs598.grad.rel_error(db_num, dbeta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "9qS4seVFEkAN",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# \"Sandwich\" layers with batch normalization\n",
    "Again, below you will find sandwich layers that implement a few commonly used patterns for convolutional networks. We include the functions in `convolutional_networks.py` but you can see them here for your convenience. \n",
    "\n",
    "```python\n",
    "class Linear_BatchNorm_ReLU(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, gamma, beta, bn_param):\n",
    "    \"\"\"\n",
    "    Convenience layer that performs an linear transform, batch normalization,\n",
    "    and ReLU.\n",
    "    Inputs:\n",
    "    - x: Array of shape (N, D1); input to the linear layer\n",
    "    - w, b: Arrays of shape (D2, D2) and (D2,) giving the weight and bias for\n",
    "      the linear transform.\n",
    "    - gamma, beta: Arrays of shape (D2,) and (D2,) giving scale and shift\n",
    "      parameters for batch normalization.\n",
    "    - bn_param: Dictionary of parameters for batch normalization.\n",
    "    Returns:\n",
    "    - out: Output from ReLU, of shape (N, D2)\n",
    "    - cache: Object to give to the backward pass.\n",
    "    \"\"\"\n",
    "    a, fc_cache = Linear.forward(x, w, b)\n",
    "    a_bn, bn_cache = BatchNorm.forward(a, gamma, beta, bn_param)\n",
    "    out, relu_cache = ReLU.forward(a_bn)\n",
    "    cache = (fc_cache, bn_cache, relu_cache)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    \"\"\"\n",
    "    Backward pass for the linear-batchnorm-relu convenience layer.\n",
    "    \"\"\"\n",
    "    fc_cache, bn_cache, relu_cache = cache\n",
    "    da_bn = ReLU.backward(dout, relu_cache)\n",
    "    da, dgamma, dbeta = BatchNorm.backward(da_bn, bn_cache)\n",
    "    dx, dw, db = Linear.backward(da, fc_cache)\n",
    "    return dx, dw, db, dgamma, dbeta\n",
    "\n",
    "\n",
    "class Conv_BatchNorm_ReLU(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, gamma, beta, conv_param, bn_param):\n",
    "    a, conv_cache = FastConv.forward(x, w, b, conv_param)\n",
    "    an, bn_cache = SpatialBatchNorm.forward(a, gamma, beta, bn_param)\n",
    "    out, relu_cache = ReLU.forward(an)\n",
    "    cache = (conv_cache, bn_cache, relu_cache)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    conv_cache, bn_cache, relu_cache = cache\n",
    "    dan = ReLU.backward(dout, relu_cache)\n",
    "    da, dgamma, dbeta = SpatialBatchNorm.backward(dan, bn_cache)\n",
    "    dx, dw, db = FastConv.backward(da, conv_cache)\n",
    "    return dx, dw, db, dgamma, dbeta\n",
    "\n",
    "\n",
    "class Conv_BatchNorm_ReLU_Pool(object):\n",
    "\n",
    "  @staticmethod\n",
    "  def forward(x, w, b, gamma, beta, conv_param, bn_param, pool_param):\n",
    "    a, conv_cache = FastConv.forward(x, w, b, conv_param)\n",
    "    an, bn_cache = SpatialBatchNorm.forward(a, gamma, beta, bn_param)\n",
    "    s, relu_cache = ReLU.forward(an)\n",
    "    out, pool_cache = FastMaxPool.forward(s, pool_param)\n",
    "    cache = (conv_cache, bn_cache, relu_cache, pool_cache)\n",
    "    return out, cache\n",
    "\n",
    "  @staticmethod\n",
    "  def backward(dout, cache):\n",
    "    conv_cache, bn_cache, relu_cache, pool_cache = cache\n",
    "    ds = FastMaxPool.backward(dout, pool_cache)\n",
    "    dan = ReLU.backward(ds, relu_cache)\n",
    "    da, dgamma, dbeta = SpatialBatchNorm.backward(dan, bn_cache)\n",
    "    dx, dw, db = FastConv.backward(da, conv_cache)\n",
    "    return dx, dw, db, dgamma, dbeta\n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "1C8k4fRgwyp1",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Convolutional nets with batch normalization\n",
    "Now that you have a working implementation for batch normalization, go back to your [`DeepConvnet`](#scrollTo=Ah-_nwx2BSxl). Modify your implementation to add batch normalization.\n",
    "\n",
    "Concretely, when the `batchnorm` flag is set to `True` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last linear layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.\n",
    "\n",
    "In the reg=0 case, you should see errors less than `1e-6` for all weights and batchnorm parameters (beta and gamma); for biases you will see high relative errors due to the extremely small magnitude of both numeric and analytic gradients.\n",
    "\n",
    "In the reg=3.14 case, you should see errors less than `1e-6` for all parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 561
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 270783,
     "status": "ok",
     "timestamp": 1601239424389,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "E5sLZ6J1wyp6",
    "new_sheet": false,
    "outputId": "249cd273-d333-4089-8ace-808906e9cfa1",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "reset_seed(0)\n",
    "\n",
    "num_inputs = 2\n",
    "input_dims = (3, 8, 8)\n",
    "num_classes = 10\n",
    "X = torch.randn(num_inputs, *input_dims, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(num_classes, size=(num_inputs,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "for reg in [0, 3.14]:\n",
    "  print('Running check with reg = ', reg)\n",
    "  model = DeepConvNet(input_dims=input_dims, num_classes=num_classes,\n",
    "                      num_filters=[8, 8, 8],\n",
    "                      max_pools=[0, 2],\n",
    "                      reg=reg, batchnorm=True,\n",
    "                      weight_scale='kaiming',\n",
    "                      dtype=torch.float64, device='cuda')\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  # The relative errors should be up to the order of e-3\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[name])\n",
    "    print('%s max relative error: %e' % (name, eecs598.grad.rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "T-D9TV8qwyp9",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Batchnorm for deep convolutional networks\n",
    "Run the following to train a deep convolutional network on a subset of 500 training examples both with and without batch normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 279241,
     "status": "ok",
     "timestamp": 1601239432874,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "IrrKxsVGwyp_",
    "new_sheet": false,
    "outputId": "0e2f0a61-ca35-4dd1-808d-0a922b109201",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "reset_seed(0)\n",
    "\n",
    "# Try training a deep convolutional net with batchnorm\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "input_dims = data_dict['X_train'].shape[1:]\n",
    "\n",
    "bn_model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                       num_filters=[16, 32, 32, 64, 64],\n",
    "                       max_pools=[0, 1, 2, 3, 4],\n",
    "                       weight_scale='kaiming',\n",
    "                       batchnorm=True,\n",
    "                       reg=1e-5,  dtype=torch.float32, device='cuda')\n",
    "model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                    num_filters=[16, 32, 32, 64, 64],\n",
    "                    max_pools=[0, 1, 2, 3, 4],\n",
    "                    weight_scale='kaiming',\n",
    "                    batchnorm=False,\n",
    "                    reg=1e-5,  dtype=torch.float32, device='cuda')\n",
    "\n",
    "print('Solver with batch norm:')\n",
    "bn_solver = Solver(bn_model, small_data,\n",
    "                   num_epochs=10, batch_size=100,\n",
    "                   update_rule=adam,\n",
    "                   optim_config={\n",
    "                     'learning_rate': 1e-3,\n",
    "                   },\n",
    "                   print_every=20, device='cuda')\n",
    "bn_solver.train()\n",
    "\n",
    "print('\\nSolver without batch norm:')\n",
    "solver = Solver(model, small_data,\n",
    "                num_epochs=10, batch_size=100,\n",
    "                update_rule=adam,\n",
    "                optim_config={\n",
    "                  'learning_rate': 1e-3,\n",
    "                },\n",
    "                print_every=20, device='cuda')\n",
    "solver.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "4XZ7jZE9wyqE",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 279588,
     "status": "ok",
     "timestamp": 1601239433249,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "1oBBnMuGehfi",
    "new_sheet": false,
    "outputId": "8d4e4cd6-7ff7-4bdb-cb87-a786c4fa94ef",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "def plot_training_history_bn(title, label, solvers, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):\n",
    "  \"\"\"utility function for plotting training history\"\"\"\n",
    "  plt.title(title)\n",
    "  plt.xlabel(label)\n",
    "  bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]\n",
    "  bl_plots = [plot_fn(solver) for solver in solvers]\n",
    "  num_bn = len(bn_plots)\n",
    "  num_bl = len(bl_plots)\n",
    "  for i in range(num_bn):\n",
    "    label='w/ BN'\n",
    "    if labels is not None:\n",
    "      label += str(labels[i])\n",
    "    plt.plot(bn_plots[i], bn_marker, label=label)\n",
    "  for i in range(num_bl):\n",
    "    label='w/o BN'\n",
    "    if labels is not None:\n",
    "      label += str(labels[i])\n",
    "    plt.plot(bl_plots[i], bl_marker, label=label)\n",
    "  plt.legend(loc='lower center', ncol=num_bn+num_bl)\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "plot_training_history_bn('Training loss','Iteration', [solver], [bn_solver], \\\n",
    "                      lambda x: x.loss_history, bl_marker='-o', bn_marker='-o')\n",
    "plt.subplot(3, 1, 2)\n",
    "plot_training_history_bn('Training accuracy','Epoch', [solver], [bn_solver], \\\n",
    "                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')\n",
    "plt.subplot(3, 1, 3)\n",
    "plot_training_history_bn('Validation accuracy','Epoch', [solver], [bn_solver], \\\n",
    "                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 25)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "YV6-pJEdSXMh",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Batch normalization and learning rate\n",
    "We will now run a small experiment to study the interaction of batch normalization and learning rate.\n",
    "\n",
    "The first cell will train convolutional networks with different learning rates. The second layer will plot training accuracy and validation set accuracy over time. You should find that using batch normalization helps the network to be less dependent to the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 312770,
     "status": "ok",
     "timestamp": 1601239466458,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "yVSQ5SesSXMq",
    "new_sheet": false,
    "outputId": "6249c67a-655a-4e1b-c323-e6fe43172cf1",
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore-input"
    ]
   },
   "outputs": [],
   "source": [
    "from convolutional_networks import DeepConvNet\n",
    "from fully_connected_networks import sgd_momentum\n",
    "reset_seed(0)\n",
    "\n",
    "# Try training a very deep net with batchnorm\n",
    "num_train = 10000\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "input_dims = data_dict['X_train'].shape[1:]\n",
    "num_epochs = 5\n",
    "lrs = [2e-1, 1e-1, 5e-2]\n",
    "lrs = [5e-3, 1e-2, 2e-2]\n",
    "\n",
    "solvers = []\n",
    "for lr in lrs:\n",
    "  print('No normalization: learning rate = ', lr)\n",
    "  model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                      num_filters=[8, 8, 8],\n",
    "                      max_pools=[0, 1, 2],\n",
    "                      weight_scale='kaiming',\n",
    "                      batchnorm=False,\n",
    "                      reg=1e-5, dtype=torch.float32, device='cuda')\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=num_epochs, batch_size=100,\n",
    "                  update_rule=sgd_momentum,\n",
    "                  optim_config={\n",
    "                    'learning_rate': lr,\n",
    "                  },\n",
    "                  verbose=False, device='cuda')\n",
    "  solver.train()\n",
    "  solvers.append(solver)\n",
    "\n",
    "bn_solvers = []\n",
    "for lr in lrs:\n",
    "  print('Normalization: learning rate = ', lr)\n",
    "  bn_model = DeepConvNet(input_dims=input_dims, num_classes=10,\n",
    "                         num_filters=[8, 8, 16, 16, 32, 32],\n",
    "                         max_pools=[1, 3, 5],\n",
    "                         weight_scale='kaiming',\n",
    "                         batchnorm=True,\n",
    "                         reg=1e-5, dtype=torch.float32, device='cuda')\n",
    "  bn_solver = Solver(bn_model, small_data,\n",
    "                     num_epochs=num_epochs, batch_size=128,\n",
    "                     update_rule=sgd_momentum,\n",
    "                     optim_config={\n",
    "                       'learning_rate': lr,\n",
    "                     },\n",
    "                     verbose=False, device='cuda')\n",
    "  bn_solver.train()\n",
    "  bn_solvers.append(bn_solver)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 908
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 313299,
     "status": "ok",
     "timestamp": 1601239467014,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "kBE8AT5SSXMv",
    "new_sheet": false,
    "outputId": "8a4fd98a-e7ad-487c-d19c-706254eb061a",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "plt.subplot(2, 1, 1)\n",
    "plot_training_history_bn('Training accuracy (Batch Normalization)','Epoch', solvers, bn_solvers, \\\n",
    "                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=[' lr={:.0e}'.format(lr) for lr in lrs])\n",
    "plt.subplot(2, 1, 2)\n",
    "plot_training_history_bn('Validation accuracy (Batch Normalization)','Epoch', solvers, bn_solvers, \\\n",
    "                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=[' lr={:.0e}'.format(lr) for lr in lrs])\n",
    "\n",
    "plt.gcf().set_size_inches(10, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "1D-y1IwsmrXI",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Submit Your Work\n",
    "After completing both notebooks for this assignment (`fully_connected_networks.ipynb` and this notebook, `convolutional_networks.ipynb`), run the following cell to create a `.zip` file for you to download and turn in. \n",
    "\n",
    "**Please MANUALLY SAVE every `*.ipynb` and `*.py` files before executing the following cell:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 313853,
     "status": "ok",
     "timestamp": 1601239467589,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "Uh-1eQIVm0vL",
    "new_sheet": false,
    "outputId": "08d5f0be-bbee-4d4d-db56-03ebddf8757f",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from eecs598.submit import make_a3_submission\n",
    "\n",
    "# TODO: Replace these with your actual uniquename and umid\n",
    "uniquename = None\n",
    "umid = None\n",
    "\n",
    "make_a3_submission(GOOGLE_DRIVE_PATH, uniquename, umid)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "convolutional_networks.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}