XAI_Practical_Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "21f0d79a",
   "metadata": {},
   "source": [
    "# R255: Topics of Machine Learning - Explainable AI Practical"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f1b4c16",
   "metadata": {},
   "source": [
    "In this practical, we will explore some of the tools and methods that we discussed and presented in our lectures. In particular we will focus on:\n",
    "- Showing how to use popular model-agnostic feature attribution libraries (e.g., LIME) to explain and analyse a pre-trained classifier.\n",
    "- Implementing vanilla Saliency maps for DNNs and using it to debug a DNN that fails to perform well on its test set.\n",
    "- Extending vanilla Saliency maps to generate smoother heatmaps via SmoothGrad, a variation of Saliency that generates more roboust and smoother saliency maps than vanilla gradients.\n",
    "- If time allows, we will also work on further extension of saliency maps.\n",
    "\n",
    "Before getting started, a few general comments (please **read carefully**):\n",
    "- We will **focus on methodology rather than fair evaluation and fine-tuning**. This means that we will avoid trying to use very complex models and architectures for the sake of time. Nevertheless, all the methods we are discussing and workign with today should be perfectly applicable to larger, more complex models. Which means you can use them for your own work and research!\n",
    "- To get **ticked for participation**, please show that you have completed this notebook to one of the instructors before leaving the session. \n",
    "- As much as we would've loved to cover more advanced topics like those in Concept Learning, it is tricky to do so within an hour and thirty minutes. Nevertheless, if you are done with this worksheet early and are interested in more advanced exercises, feel free to look into the other notebook in this repo or do some of the optional exercises here. This notebook is the one we will ask students assigned to this topic as their main R255 topic to complete for their non-project marks.\n",
    "- **If you are stuck at any point**, and/or if anything is unclear, please do not hestitate to ask to any of the instructors! Chances are someone else will have the same question as you do and we are here to help as much as we can."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37d82900",
   "metadata": {},
   "source": [
    "## General Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "914447bf",
   "metadata": {},
   "source": [
    "Time to gets our hands dirty with some of the XAI methods we learned in class! For this, we will first install some libraries which will\n",
    "be useful to explore the different methods we will play with today:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdb06803",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install lime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92571bc1",
   "metadata": {},
   "source": [
    "We will also import a bunch of useful libraries which will enable us to train models that we can then dissect and explore. In this practical, we will use [TensorFlow](https://www.tensorflow.org/) as our main framework of choice. This has been chosen as TensorFlow allows easy construction of DNNs without much scaffolding and enable easy gradient manipulation in computational graphs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf20a8da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will use TensorFlow for constructing DNNs\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "\n",
    "# And do all plotting with matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a66874c",
   "metadata": {},
   "source": [
    "Before we move even further, let's double-check that you are connected to a GPU instance in Colab. For this, please run the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d07433d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Connected GPUs:\", tf.config.list_physical_devices('GPU'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5844596f",
   "metadata": {},
   "source": [
    "If you see a non-empty list, then you are good to go. Otherwise, double-check that your runtime in Colab has been setup to a GPU by going to \"Runtime\" -> \"Change runtime type\" -> \"Hardware Accelerator\" -> \"GPU\". If after making this change, and restarting the notebook's kernel via \"Runtime\" -> \"Restart runtime\", you are still not seeing any GPUs, please contact one of the instructors for help."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b94c5b4",
   "metadata": {},
   "source": [
    "## Utility Functions\n",
    "Last bit of setup, we will use some simple utility functions to encourage determinism in this work. As convention, we suggest that this call is included at the begining of every answer that involves running some potentially non-deterministic code (which with TF is almost everything!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6d9c55e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_seeds(seed=42):\n",
    "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
    "    random.seed(seed)\n",
    "    tf.random.set_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "set_seeds(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "143169d8",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28891fe2",
   "metadata": {},
   "source": [
    "# Part 1: Data Loading and Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f1029ac",
   "metadata": {},
   "source": [
    "## Part 1.1: Dataset Loading\n",
    "\n",
    "In this practical, we will work with a very simple task constructed from the [MNIST datset](https://en.wikipedia.org/wiki/MNIST_database). Specifically, we will work with 28 x 56 grayscale\n",
    "images which have 2 handwritten digit in them. Each digit in each sample is between 0 and 5 (inclusive)\n",
    "and they are placed next to each other concatenated in their x-axis (i.e., each handwritten digit was\n",
    "originally 28 x 28). Each sample is annotated with one out of 11 labels indicating the result of adding\n",
    "both of these numbers. Our task, therefore, is to learn to predict the sum of two handwritten digits given both numbers in a single image.\n",
    "\n",
    "\n",
    "To save some time, we have pre-processed the MNIST dataset for you and saved\n",
    "this dataset in this repo. We will begin by loading both its training and test splits by unpacking\n",
    "`train_sum_mnist.npz` and `test_sum_mnist.npz` respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7aef344",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's load our MNIST Sum Training Set\n",
    "train_1_data = np.load(\"train_sum_mnist.npz\")\n",
    "x_train = train_1_data['x_train']\n",
    "y_train = train_1_data['y_train']\n",
    "print(\"Training samples shape:\", x_train.shape)\n",
    "print(\"Training labels shape:\", y_train.shape)\n",
    "\n",
    "# And our MNIST Sum testing set\n",
    "test_data = np.load(\"test_sum_mnist.npz\")\n",
    "x_test = test_data['x_test']\n",
    "y_test = test_data['y_test']\n",
    "print(\"Testing samples shape:\", x_test.shape)\n",
    "print(\"Testing labels shape:\", y_test.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc14cff8",
   "metadata": {},
   "source": [
    "Notice that we have a total of 36k samples (30k for training and 6k for testing) and labels are NOT one-hot encoded\n",
    "but rather given in their categorical form.\n",
    "\n",
    "To get a sense of how the samples in this dataset look like, it is always helpful to visualise a few samples from its training and testing sets.\n",
    "\n",
    "Run the code below to plot 5 random samples from both datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6019cad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will plot 5 random samples of both the testing and training set\n",
    "# to visualise how it looks like before diving into training\n",
    "# our model\n",
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "num_samples = 5\n",
    "fig, axs = plt.subplots(2, num_samples, figsize=(12, 3))\n",
    "for j, (name, ds) in enumerate([\n",
    "    (\"Training\", x_train),\n",
    "    (\"Testing\", x_test),\n",
    "]):\n",
    "    idxs = np.random.choice(ds.shape[0], num_samples, replace=False)\n",
    "    for i, ax in enumerate(axs[j, :]):\n",
    "        X = ds[idxs[i], :, :, 0]\n",
    "        ax.imshow(X, cmap='viridis', vmin=0, vmax=255)\n",
    "        ax.grid(False)\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "        if not i:\n",
    "            ax.set_ylabel(name, fontsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea88b106",
   "metadata": {},
   "source": [
    "We can also explore how different lables are distributed in the training set. This can be important to determine how we should fairly evaluate a model trained to solve this task.\n",
    "\n",
    "Run the following code to visualise a histogram of the training class distribution in this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d7a6357",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_classes = len(np.unique(y_train))\n",
    "fig, axs = plt.subplots(1, 2, figsize=(12, 3))\n",
    "for ax, (ds_name, ds) in zip(axs, [(\"train\", y_train), (\"test\", y_test)]):\n",
    "    ax.bar(\n",
    "        list(range(n_classes)),\n",
    "        np.sum(tf.one_hot(ds, n_classes), axis=0),\n",
    "        align='center',\n",
    "        width=0.5,\n",
    "    )\n",
    "    ax.set_xticks(list(range(n_classes)))\n",
    "    ax.set_ylabel(\"Counts\", fontsize=15)\n",
    "    ax.set_xlabel(\"Class Label\", fontsize=15)\n",
    "    ax.set_title(f\"{ds_name} set class distribution\", fontsize=15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e88c02",
   "metadata": {},
   "source": [
    "We see that classes, both in the training and test sets, are roughly centered at 5. This makes sense as 5 is the most common value one gets from adding two numbers uniformly sampled from {0, 1, 2, 3, 4, 5}. Cool, all looks good so far.\n",
    "\n",
    "Let's move on to training a model on this dataset. Notice that in practise we may want to correct at some point for this imbalance by weighting our model's loss function in favor of less represented classes, but for now we will ignore such magic tricks for the sake of simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "814e0a11",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f0018a2",
   "metadata": {},
   "source": [
    "## Part 1.2: Model Training\n",
    "\n",
    "We will now train a simple Convolutional DNN to predict the result obtained when adding two the two numbers in the input image. Before doing so, and in order to save us some headaches when using some of the libraries out there, we will extend our samples to be RGB samples by replicating their values across 3 different channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9226a918",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For simplicity, we will treat all images as RGB images.\n",
    "# Notice that we will not normalise them for now even though this may help.\n",
    "x_train = np.stack((x_train[:, :, :, 0].astype(np.float32),)*3, axis=-1)\n",
    "x_test = np.stack((x_test[:, :, :, 0].astype(np.float32),)*3, axis=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2cf6be",
   "metadata": {},
   "source": [
    "### Warmup Exercise 1.2.1\n",
    "#### Time to do some training and actual coding! Please complete the missing parts in the code below:\n",
    "\n",
    "**Hint**: If you need a super quick refresher of TensorFlow, take a look at [this](https://www.tensorflow.org/datasets/keras_example) super quick and easy official example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e879561a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fix seed for determinism purposes\n",
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# Let's compute the number of classes in our dataset\n",
    "n_classes = len(np.unique(y_train))\n",
    "\n",
    "\n",
    "# And time to create our model's architecture. For the sake of simplicity, we will build a model that has:\n",
    "#     - one 3x3 convolution with 4 output filters\n",
    "#     - one flattening layer\n",
    "#     - one fully connected (i.e., dense) layer with 64 hidden units in it and a ReLU activation\n",
    "#     - one output fully connected (i.e., dense) layer with n_classes output units in it and no activation function\n",
    "dnn = tf.keras.models.Sequential([\n",
    "    tf.keras.layers.Conv2D(\n",
    "        input_shape=(28, 28*2, 3),\n",
    "        filters=4,\n",
    "        kernel_size=(3, 3),\n",
    "        strides=(1, 1),\n",
    "        padding='valid',\n",
    "        activation='relu'\n",
    "    ),\n",
    "    tf.keras.layers.Flatten(),\n",
    "    tf.keras.layers.Dense(64, activation='relu'),\n",
    "    tf.keras.layers.Dense(n_classes)\n",
    "])\n",
    "\n",
    "# Compile with:\n",
    "# (1) an Adam optimizer with learning rate of 0.001\n",
    "# (2) a sparse categorical cross entropy loss function (make sure it goes from the logits!)\n",
    "# (3) a sparse categorical accuracy metric to track for debugging purposes\n",
    "dnn.compile(\n",
    "    optimizer=...TODO..., # <--- Complete this!\n",
    "    loss=...TODO..., # <--- Complete this!\n",
    "    metrics=[...TODO...],  # <--- Complete this!\n",
    ")\n",
    "\n",
    "# And finally train this model for a few epochs, say 40, with a relatively large batch size, say 128\n",
    "_ = dnn.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    epochs=...TODO..., # <--- Complete this!,\n",
    "    batch_size=...TODO..., # <--- Complete this!\n",
    "    validation_split=0.1, # We will use 10% of our data for validation purposes\n",
    "    shuffle=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb7a7109",
   "metadata": {},
   "source": [
    "### Warmup Exercise 1.2.2\n",
    "#### Compute the accuracy of this model in its training and test set. How do the two compare?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb3ff279",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34041d4",
   "metadata": {},
   "source": [
    "### Warmup Exercise 1.2.3\n",
    "#### Before diving deeper into what may be happening here, do you have any guesses for the reason behind the observed generalisation gap?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5336c5a7",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af38c81f",
   "metadata": {},
   "source": [
    "### Warmup Exercise 1.2.4\n",
    "#### If you would have to design an experiment to test your hypothesis above, how would it look like?\n",
    "**No need to run the actual experiment** (although feel free to do so if you have time later!) but we are looking for you to rather highlight ways in which you can use the content taught in this course for solving real research problems!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1cc489e",
   "metadata": {},
   "source": [
    "**TODO**: write your answer answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c7c4c03",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3b6008a",
   "metadata": {},
   "source": [
    "# Part 2: DNN Debugging with Local Interpretable Model-agnostic Explanations (LIME)\n",
    "\n",
    "The gap that we observed above is a weird one, specially since the model's validation accuracy wasn't bad at all. Here is when we begin to dive into how XAI can help you better understand this scenarios.\n",
    "\n",
    "With that in mind, let's first try and understand how our train model is \"attending\" each of its input images when making a prediction. If we are able to notice some interesting difference in how it reasons about samples in its training set vs how it reasons about inputs test set, then we may get a big hint of what is happening here. The first and easiest tool one can use to begin such and exploration is **feature importance methods**!\n",
    "\n",
    "In class, we learned several method within the family of feature importance XAI. However, here we will focus on exploring LIME, as this is one of the most widely used XAI libraries out there and it offers a very nice pip-installable library. To begin using LIME in image domains, we will need to import the library we recently installed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9bc6b2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from lime import lime_image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83e9a1b3",
   "metadata": {},
   "source": [
    "This module includes the class `LimeImageExplainer` which will allow us to generate visual explanations highlighting which parts of an image were considered to be important for a given model (see [this](https://github.com/marcotcr/lime/blob/master/doc/notebooks/Tutorial%20-%20Image%20Classification%20Keras.ipynb) short tutorial if you want more details on this module). These visual explanations, represented as masks, explain which super-pixels (or \"features\") are important for the top predicted class and can show positive and negative contributions using different colors (green with yellow borders for positive contibution and red for a negative contribution).\n",
    "\n",
    "To simplify our use of LIME in the following section, **we have provided you with a function that abstracts LIME's APIs** to generate explanations for a given set of samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f2b81af",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_explanations(\n",
    "    model,\n",
    "    input_samples,\n",
    "    num_samples=5,\n",
    "):\n",
    "    \"\"\"\n",
    "    A list of output masks, one for each input sample, of LIME explanations\n",
    "    generated for `model`'s output when predicting a label for\n",
    "    `input_samples`.\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model\n",
    "        input_samples: a Tensor of testing samples with shape (B, H, W, C),\n",
    "        num_samples: the number of samples to generate when learning LIME's local classifiers.\n",
    "    Returns:\n",
    "        LIME's explanations for the given input samples\n",
    "    \"\"\"\n",
    "    # Select a number of examples to visualise\n",
    "    # And generate a LIME explanation for all of them:\n",
    "    explainer = lime_image.LimeImageExplainer()\n",
    "    explanations = []\n",
    "    for example in input_samples:\n",
    "        explanation = explainer.explain_instance(\n",
    "            example.astype('double'),\n",
    "            model.predict,\n",
    "            top_labels=1,  # We want an explanation only for the top predicted layer\n",
    "            hide_color=0,  # We will zero everything that the explanation is not looking at\n",
    "            num_samples=num_samples,  # How many samples will LIME generate to learn its linear classifier\n",
    "        )\n",
    "        explanations.append(explanation)\n",
    "    return explanations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41a30870",
   "metadata": {},
   "source": [
    "Let's use this to generate explanations for a few random input samples in our training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b10432d",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "# We will generate explanations for 8 random samples of the training set\n",
    "train_idxs = np.random.choice(x_train.shape[0], 8, replace=False)\n",
    "train_explanations = generate_explanations(\n",
    "    model=dnn,\n",
    "    input_samples=x_train[train_idxs, :, :, :],\n",
    "    num_samples=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2ffa5a",
   "metadata": {},
   "source": [
    "This will allow us to visually inspect which parts of the input image were considered most important for the class\n",
    "predicted for our model. For this, we also provide a helper function abstracting some of the details of plotting these masks (still, we would recommend going over it to make sure you fully understand it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56862e6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimage.segmentation import mark_boundaries\n",
    "def visualise_explanations(\n",
    "    model,\n",
    "    explanations,\n",
    "    input_samples,\n",
    "    input_labels,\n",
    "    visualise_samples=False,\n",
    "    num_features=1,\n",
    "    n_rows=1,\n",
    "):\n",
    "    \"\"\"\n",
    "    Visualises the given list of LIME explanations generated for corresponding\n",
    "    `input_samples` with labels `input_labels`\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model used to generate the input explanations\n",
    "        explanations: a list of B LIME explanations (e.g., the ouptut of generate_explanations)\n",
    "        input_samples: a Tensor of testing samples with shape (B, H, W, C),\n",
    "        input_labels: a Tensor of correspondong testing labels with shape (B,),\n",
    "        visualise_samples: a flag indicating whether we want to include the original\n",
    "                           samples as part of the visualisation.\n",
    "        n_rows: the number of rows to use to visualise all B samples. Must divide B.\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    n_examples = len(explanations)\n",
    "    n_cols = n_examples // n_rows\n",
    "    if visualise_samples:\n",
    "        n_rows = n_rows * 2\n",
    "        \n",
    "    width = n_cols * 4\n",
    "    height = n_rows * 2\n",
    "    fig, axs = plt.subplots(\n",
    "        n_rows,\n",
    "        n_cols,\n",
    "        figsize=(width, height),\n",
    "        squeeze=False,\n",
    "    )\n",
    "    for i, (example, label, explanation) in enumerate(zip(input_samples, input_labels, explanations)):\n",
    "        r = i // n_cols\n",
    "        c = i % n_cols\n",
    "        if visualise_samples:\n",
    "            # First plot example\n",
    "            ax = axs[2*r, c]\n",
    "            ax.imshow(example[:, :, 0], cmap='viridis')\n",
    "            ax.grid(False)\n",
    "            ax.set_xticks([])\n",
    "            ax.set_yticks([])\n",
    "            if c == 0:\n",
    "                ax.set_ylabel(\"Training Sample\")\n",
    "\n",
    "        # Then plot mask showing most important features!\n",
    "        if visualise_samples:\n",
    "            ax = axs[2*r + 1, c]\n",
    "        else:\n",
    "            ax = axs[r, c]\n",
    "            \n",
    "        temp, mask = explanation.get_image_and_mask(\n",
    "            explanation.top_labels[0],\n",
    "            positive_only=False,  # We only highlight areas that positively contribute to the prediction\n",
    "            num_features=num_features,  # This asks the explainer to show only the most important 'super-pixel'\n",
    "            hide_rest=False,  # For now, let's show the background image together with the part that\n",
    "                             # is considered most important for the input sample\n",
    "        )\n",
    "        boundaries = mark_boundaries(temp / 2 + 0.5, mask[:, :])\n",
    "        ax.imshow(boundaries)\n",
    "        ax.grid(False)\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "        color = 'green' if np.argmax(model(np.expand_dims(example, axis=0)), axis=-1) == label else 'red'\n",
    "        for spine in ax.spines.values():\n",
    "            spine.set_edgecolor(color)\n",
    "            spine.set_linewidth(6)\n",
    "            if color == 'green':\n",
    "                ax.set_xlabel(\"Correctly Predicted\")\n",
    "            else:\n",
    "                ax.set_xlabel(\"Mispredicted\")\n",
    "        if c == 0:\n",
    "            ax.set_ylabel(\"Explanation\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b080945",
   "metadata": {},
   "outputs": [],
   "source": [
    "visualise_explanations(\n",
    "    model=dnn,\n",
    "    explanations=train_explanations,\n",
    "    input_samples=x_train[train_idxs, :, :, :],\n",
    "    input_labels=y_train[train_idxs],\n",
    "    visualise_samples=False,\n",
    "    num_features=1,\n",
    "    n_rows=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff6790d2",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "327ddec3",
   "metadata": {},
   "source": [
    "### Exercise 2.1\n",
    "#### (a) These explanations seem a bit off.... at close inspection of the code, we can see that we are using a very small number of generated samples to train our linear interpretable models for LIME. Below, run an ablation showing how the explanation of a training sample changes as you vary the number of samples used for LIME between {2, 10, 100, 1000, 2500}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "591a2e37",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb953f48",
   "metadata": {},
   "source": [
    "#### (b) What things do you notice as the number of LIME samples increases? Why do you think this is the case?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de08e1bb",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e484d61",
   "metadata": {},
   "source": [
    "#### (c) Try also playing with the number of super-pixels (or features) we show in each image (controlled via the `num_features` argument).  What do you observe?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bac21ad",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b4e89d2",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5d88207",
   "metadata": {},
   "source": [
    "Part of understanding why our model performs well in its training set but not in its testing set is to see how it also perceives images from the testing set. Now that we know how to use LIME to do this, let's see if something clear pops up when looking at the testing set:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7a315b",
   "metadata": {},
   "source": [
    "### Exercise 2.2\n",
    "#### Visualise the explanations for this model but a few random samples from the testing dataset\n",
    "When calling LIME, use a \"reasonable\" number of LIME samples based on your insights obtained in the previous exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62afbf40",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d04dd85e",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9300da09",
   "metadata": {},
   "source": [
    "### Exercise 2.3\n",
    "#### Do you notice anything suspicious in the images above? What can this tell you of a possible source for the observed generalisation gap?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d04f920",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "796c82bb",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be348199",
   "metadata": {},
   "source": [
    "### Exercise 2.4\n",
    "#### Describe an experiment that would confirm your hypothesis if successful"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac8388c4",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a67187d5",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f730277",
   "metadata": {},
   "source": [
    "### Exercise 2.5 (Optional)\n",
    "#### If time allows, run the experiment proposed above to see if your hypothesis holds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f83c7cc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: write your answer here (do it only if time allows!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "824d7b91",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "158329e7",
   "metadata": {},
   "source": [
    "# Part 3: DNN Debugging Via Saliency Maps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9640cfe",
   "metadata": {},
   "source": [
    "So far, we have been able to analyse our misbehaving DNN using LIME and were able to come up with a testible hypothesis as to what may be happening. Nevertheless, we seen with our LIME experiments, some of the explanations that we were given were hard to interpret, making us wonder if there are better, more robust approaches we could use to debug our DNN.\n",
    "\n",
    "In this section we will explore DNN-specific feature attribution methods (i.e., **Saliency Methods** from lecture!) to see if the explanations generated by these methods also bring evidence in favor of the conjecture we made above regarding the source of the generalisation gap in our model. We will begin with *Vanilla Gradient* and then move to more sofisticated methods after we have analysed this method a bit more."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e7866bc",
   "metadata": {},
   "source": [
    "## Part 3.1: Vanilla Gradient\n",
    "\n",
    "If you recall from our lectures, the *Vanilla Gradient* saliency method measured how important each input feature is for a given output by looking at the derivative of the model's output with respect to its input features. This takes advantage of the fact that DNNs are differientable models and uses the definition of gradient as a measurement of first-order sensitivity to changes in the input. If you need an extra recap of what Vanilla Gradient does, please take a look at [this short but clear description](https://christophm.github.io/interpretable-ml-book/pixel-attribution.html#vanilla-gradient-saliency-maps) or, for more details, at the [original paper](https://arxiv.org/abs/1312.6034) proposing this method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc11d641",
   "metadata": {},
   "source": [
    "### Exercise 3.1.1\n",
    "#### Implement the \"vanilla gradient\" method\n",
    "\n",
    "Remember that *vanilla Gradient* generates a heatmap for the sample we are trying to explain by computing the gradient of the model's output prediction with respect to the input features. In this exercise, we ask you to implement this in TensorFlow by completing the function `vanilla_gradient` whose signature is described below:\n",
    "\n",
    "**Hint 1**: In TensorFlow you can compute gradients of a function `fun(x)` with respect to some variable of interest `x` using a `tf.GradientTape` as follows:\n",
    "\n",
    "```python\n",
    "input_var = tf.Variable(x, dtype=float)\n",
    "with tf.GradientTape() as tape:\n",
    "    out = fun(x)\n",
    "grads = tape.gradient(out, input_var)\n",
    "```\n",
    "\n",
    "The big caveat is that you need to make sure that the entire computation you are diffientating is done inside the scope of the tape you are using (i.e., within the code block below `with tf.GradientTape() as tape:`). Further documentation for `tf.GradientTape` can be found [here](https://www.tensorflow.org/api_docs/python/tf/GradientTape).\n",
    "\n",
    "\n",
    "\n",
    "**Hint 2**: You may use the following function to normalise a saliency map into a grayscale image:\n",
    "```python\n",
    "def saliency_to_grayscale(tensor):\n",
    "    \"\"\"\n",
    "    Transform tensor over RGB axis to grayscale.\n",
    "    Args:\n",
    "        tensor (tf.Tensor): 4D-Tensor with shape (batch_size, H, W, 3)\n",
    "    Returns:\n",
    "        tf.Tensor: 3D-Tensor of grayscale tensor, with shape (batch_size, H, W)\n",
    "    \"\"\"\n",
    "    grayscale_tensor = tf.reduce_sum(tensor, axis=-1)\n",
    "    return tf.image.per_image_standardization(grayscale_tensor)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "382bf65f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def vanilla_gradient(model, samples):\n",
    "    \"\"\"\n",
    "    Outputs a saliency map, using the Vanilla Gradient algorithm, indicating which input\n",
    "    pixels are important for the class predicted by `model` for inputs in `samples`\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model whose outputs we want to explain\n",
    "        samples: a Tensor of testing samples with shape (B, H, W, C) which we want to explain\n",
    "    Returns:\n",
    "        a 3D tensor with shape (B, H, W) with as many grayscale saliency maps as inputs\n",
    "        in `samples`\n",
    "    \"\"\"\n",
    "    # TODO: Implement me!\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52d303cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def saliency_to_grayscale(tensor):\n",
    "    \"\"\"\n",
    "    Transform tensor over RGB axis to grayscale.\n",
    "    Args:\n",
    "        tensor (tf.Tensor): 4D-Tensor with shape (batch_size, H, W, 3)\n",
    "    Returns:\n",
    "        tf.Tensor: 3D-Tensor of grayscale tensor, with shape (batch_size, H, W)\n",
    "    \"\"\"\n",
    "    grayscale_tensor = np.mean(tensor, axis=-1)\n",
    "    return tf.image.per_image_standardization(grayscale_tensor)\n",
    "\n",
    "def vanilla_gradient(model, samples):\n",
    "    \"\"\"\n",
    "    Outputs a saliency map, using the Vanilla Gradient algorithm, indicating which input\n",
    "    pixels are important for the class predicted by `model` for inputs in `samples`\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model whose outputs we want to explain\n",
    "        samples: a Tensor of testing samples with shape (B, H, W, C) which we want to explain\n",
    "    Returns:\n",
    "        a 3D tensor with shape (B, H, W) with as many grayscale saliency maps as inputs\n",
    "        in `samples`\n",
    "    \"\"\"\n",
    "    images = tf.Variable(samples, dtype=float)\n",
    "    with tf.GradientTape() as tape:\n",
    "        pred = tf.nn.softmax(model(images, training=False), axis=-1)\n",
    "        target_classes = np.argmax(pred.numpy(), axis=-1)\n",
    "        loss = []\n",
    "        for i, idx in enumerate(target_classes):\n",
    "            loss.append(pred[i:i+1, idx])\n",
    "        loss = tf.concat(loss, axis=0)\n",
    "    grads = tape.gradient(loss, images)\n",
    "    saliency = saliency_to_grayscale(np.abs(grads))\n",
    "\n",
    "    return saliency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3de62d54",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cb88e60",
   "metadata": {},
   "source": [
    "Now that we have implemented our vanilla gradient method, it is time to use it to explore the same network we explored with LIME. For this, as we did with LIME, we will first **provide you with a simple method to visualise saliency maps** generated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "547496e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def visualise_saliency_explanations(\n",
    "    model,\n",
    "    saliency_maps,\n",
    "    input_samples,\n",
    "    input_labels,\n",
    "    visualise_samples=True,\n",
    "    n_rows=1,\n",
    "):\n",
    "    \"\"\"\n",
    "    Visualises the given list of saliency maps generated for corresponding\n",
    "    `input_samples` with labels `input_labels`\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model used to generate the input explanations\n",
    "        saliency_maps: a tensor with shape (B, H, W) with saliency maps corresponding to input_samples\n",
    "        input_samples: a Tensor of testing samples with shape (B, H, W, C),\n",
    "        input_labels: a Tensor of correspondong testing labels with shape (B,),\n",
    "        visualise_samples: a flag indicating whether we want to include the original\n",
    "                           samples as part of the visualisation.\n",
    "        n_rows: the number of rows to use to visualise all B samples. Must divide B.\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    samples_idxs = list(range(input_samples.shape[0]))\n",
    "    n_cols = len(samples_idxs) // n_rows\n",
    "    if visualise_samples:\n",
    "        n_rows = n_rows * 2\n",
    "        \n",
    "    width = n_cols * 4\n",
    "    height = n_rows * 2\n",
    "    fig, axs = plt.subplots(\n",
    "        n_rows,\n",
    "        n_cols,\n",
    "        figsize=(width, height),\n",
    "        squeeze=False,\n",
    "    )\n",
    "    for i, idx in enumerate(samples_idxs):\n",
    "        r = i // n_cols\n",
    "        c = i % n_cols\n",
    "        if visualise_samples:\n",
    "            # First plot example\n",
    "            ax = axs[2*r, c]\n",
    "            ax.imshow(input_samples[idx, :, :, 0]/255.0, cmap='viridis')\n",
    "            ax.grid(False)\n",
    "            ax.set_xticks([])\n",
    "            ax.set_yticks([])\n",
    "            if c == 0:\n",
    "                ax.set_ylabel(\"Training Sample\")\n",
    "\n",
    "        # Then plot mask showing most important features!\n",
    "        if visualise_samples:\n",
    "            ax = axs[2*r + 1, c]\n",
    "        else:\n",
    "            ax = axs[r, c]\n",
    "            \n",
    "        explanation = saliency_maps[idx:idx+1, :, :]\n",
    "        ax.imshow(explanation[0], cmap='hot')\n",
    "        ax.grid(False)\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "        color = 'green' if np.argmax(model(input_samples[idx:idx+1, :, :, :]), axis=-1) == input_labels[idx] else 'red'\n",
    "        for spine in ax.spines.values():\n",
    "            spine.set_edgecolor(color)\n",
    "            spine.set_linewidth(6)\n",
    "            if color == 'green':\n",
    "                ax.set_xlabel(\"Correctly Predicted\")\n",
    "            else:\n",
    "                ax.set_xlabel(\"Mispredicted\")\n",
    "        if c == 0:\n",
    "            ax.set_ylabel(\"Explanation\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8657c56a",
   "metadata": {},
   "source": [
    "This should allow us to visualise the same 8 training samples we visualise for LIME but using our vanilla saliency method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14387a7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "samples_idxs = np.random.choice(x_train.shape[0], 8, replace=False)\n",
    "visualise_saliency_explanations(\n",
    "    model=dnn,\n",
    "    saliency_maps=vanilla_gradient(\n",
    "        model=dnn,\n",
    "        samples=x_train[samples_idxs, :, :, :],\n",
    "    ),\n",
    "    input_samples=x_train[samples_idxs, :, :, :],\n",
    "    input_labels=y_train[samples_idxs],\n",
    "    visualise_samples=True,\n",
    "    n_rows=2,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11aa419d",
   "metadata": {},
   "source": [
    "### Exercise 3.1.2\n",
    "#### Visualise the explanations generated by vanilla gradient for 8 TEST samples in our MNIST additive task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba827f43",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b2a511e",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Exercise 3.1.3\n",
    "#### (a) Are these results what you expected? Do any of these explanations look odd to you?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7513f3f9",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d778270a",
   "metadata": {},
   "source": [
    "#### (b) How do these results compare to LIME's results qualitatively? Do you have any intuition as to why they may differ the way they do?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3e2854",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fc5bd1",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dff74988",
   "metadata": {},
   "source": [
    "## Part 3.2: SmoothGrad\n",
    "\n",
    "One of the biggest issues you can observe when using Vanilla Gradient's saliency map is that they are very noisy (for lack of a better word). This is because derivatives change a lot over very small scales/values, making these maps very noisy and fragile. To make this better, we will implement a method originally proposed by [Smilkov et al.](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/1706.03825&hl=en&sa=T&oi=gsb&ct=res&cd=0&d=17711739766775469027&ei=TTnbY4mSG-zZsQKBmrTIDQ&scisig=AAGBfm30TezairLOpExTUOEubRAjAsLRBQ) where we smooth the maps generated by vanilla gradient through a simple but clever averaging. Specifically, we will averge the result of computing saliency maps for multiple versions of the sampe input but with different Gaussian noise added to them. This will allow us to capture the salient features more clearly over small changes coming from the noise that we add to the image before computing the saliency map. For a quick and nice explanation of the mathematics behind SmoothGrad, please read [this short](https://christophm.github.io/interpretable-ml-book/pixel-attribution.html#smoothgrad) section before continuing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dfb39a7",
   "metadata": {},
   "source": [
    "### Exercise 3.2.1\n",
    "#### (a) Implement the SmoothGrad function whose signature is given below:\n",
    "\n",
    "**Hint**: it may be useful ro reuse `vanilla_saliency` for this implementation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af94b3e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def smoothgrad_saliency(model, samples, n_trials=5, noise_level=1):\n",
    "    \"\"\"\n",
    "    Outputs a saliency map, using the SmoothGrad on top of Saliency Gradient, indicating which input\n",
    "    pixels are important for the class predicted by `model` for inputs in `samples`.\n",
    "    \n",
    "    SmoothGrad proceeds as follows:\n",
    "        1. For each sample we want to explain, we generate `n_trials` new sample such that\n",
    "           each sample is added some Gaussian noise with mean 0 and std `noise_level`.\n",
    "        2. We then compute the vanilla gradient saliency map for each of these samples.\n",
    "        3. We output the mean of all the saliency maps across all trials for the same sample.\n",
    "    Args:\n",
    "        model: a valid TensorFlow Model whose outputs we want to explain\n",
    "        samples: a Tensor of testing samples with shape (B, H, W, C) which we want to explain\n",
    "        n_trials: how many noisy samples we will generate for each of the B inputs in input_samples\n",
    "        noise_level: the std used when adding Gaussian noise to each input sample.\n",
    "    Returns:\n",
    "        a 3D tensor with shape (B, H, W) with as many grayscale saliency maps as inputs\n",
    "        in `samples`\n",
    "    \"\"\"\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d24b99e2",
   "metadata": {},
   "source": [
    "#### (b) Before getting any sense of SmoothGrad's hyperparameters,  run SmoothGrad with its default arguments on 8 random test samples and visualise the output masks\n",
    "\n",
    "**Hint**: use `visualise_saliency_explanations` to do the visualisation for you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d803926",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a93c48a",
   "metadata": {},
   "source": [
    "#### (c) How do these explanations qualitatively compare to those extracted by both LIME and Vanilla Saliency? Why do you think that is the case?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228e70dc",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd99563f",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f5d9068",
   "metadata": {},
   "source": [
    "### Exercise 3.2.2\n",
    "#### (a) To try and get a sense of how SmoothGrad behaves, we first explore its sensibility to the noise level. For this, run an ablation over the level of noise added for each trial.\n",
    "\n",
    "When running this ablation, try out values of `noise_percentile` in `{0.01, 0.1, 1, 10, 100, 1000}` using a fixed number of trials (say 50 for now). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abfd1605",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f79d2ec4",
   "metadata": {},
   "source": [
    "#### (b) What hypothesis/conclusions may you reach from the results you are observing? Do you have any intuition as to why the ablation results look like they do?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd313c0e",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61c7312c",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6890e56a",
   "metadata": {},
   "source": [
    "### Exercise 3.2.3\n",
    "#### (a) Similarly, we now explore a related question: how does the number of samples used to compute SmoothGrad aftect its output result?\n",
    "\n",
    "To answer this question, run a simple ablation while changing the number of trials from within {1, 2, 4, 16, 64, 256, 512} and visualise the explanations generated for a single sample with SmoothGrad. You can use your results from the previous exercise to set `noise_level` to a sensible value.\n",
    "\n",
    "**Hint**: the code used for the previous exercise should be easily adaptable to quickly run this ablation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d03ad1ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0c25fb",
   "metadata": {},
   "source": [
    "#### (b) What hypothesis/conclusions may you reach from the results you are observing? Do you have any intuition as to why the ablation results look like they do?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb6e92c2",
   "metadata": {},
   "source": [
    "**TODO**: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf90e2a0",
   "metadata": {},
   "source": [
    "### Exercise 3.3.4\n",
    "#### Finally, use the knowledge obtained from the last two ablations to select reasonable values for `n_trials` and `noise_level` to see the explanations generated by SmoothGrad for some random test samples\n",
    "\n",
    "**Hint**: You may find it useful to rerun the code from Exercise 3.2.1.b here with the \"correct\" parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44e4bb44",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c08e2d73",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "758e1736",
   "metadata": {},
   "source": [
    "### Exercise 3.3.5 (Optional and only if time allows!)\n",
    "#### There is good evidence that scaling the gradient by the actual input may generate more meaningful saliency maps (one can thing of it as a first-order Taylor approximation). Try running the experiments above but this time scale the output of Vanilla Gradient by its inputs.\n",
    "\n",
    "Try it out and you should get crisper heatmaps! Notice how you can apply SmoothGrad to this variation as well to get even better heatmaps. This is a good example of how SmoothGrad is a method for improving any given saliency method rather than a saliency method itself.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "342ac133",
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seeds(42)  # <---- Make sure things are deterministic for comparison purposes\n",
    "\n",
    "# TODO: write your answer here!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7196a98",
   "metadata": {},
   "source": [
    "---\n",
    "---\n",
    "---\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}