Chapter03/ch3_nb4_apply_regularization_methods_to_cnns.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<p style=\"border: 1px solid #e7692c; border-left: 15px solid #e7692c; padding: 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c\">Tip.</strong> <a style=\"color: #000000;\" href=\"https://nbviewer.jupyter.org/github/PacktPublishing/Hands-On-Computer-Vision-with-Tensorflow/blob/master/ch3/ch3_nb4_apply_regularization_methods_to_cnns.ipynb\" title=\"View with Jupyter Online\">Click here to view this notebook on <code>nbviewer.jupyter.org</code></a>. \n",
    "    <br/>These notebooks are better read there, as Github default viewer ignores some of the formatting and interactive content.\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"font-size: 1em; padding: 0; margin: 0;\">\n",
    "    <tr style=\"vertical-align: top; padding: 0; margin: 0;\">\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; padding-right: 15px;\">\n",
    "    <p style=\"background: #363636; color:#ffffff; text-align:justify; padding: 10px 25px;\">\n",
    "        <strong style=\"font-size: 1.0em;\"><span style=\"font-size: 1.2em;\"><span style=\"color: #e7692c;\">Hands-on</span> Computer Vision with TensorFlow 2</span><br/>by <em>Eliot Andres</em> & <em>Benjamin Planche</em> (Packt Pub.)</strong><br/><br/>\n",
    "        <strong>> Chapter 3: Modern Neural Networks</strong><br/>\n",
    "    </p>\n",
    "\n",
    "<h1 style=\"width: 100%; text-align: left; padding: 0px 25px;\"><small style=\"color: #e7692c;\">Notebook 4:</small><br/>Applying Regularization Methods to CNNs</h1>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #363636; text-align:justify; padding: 0 10px;\">\n",
    "    In this final notebook for Chapter 3, we will demonstrate the effects of the various <strong>regularization methods</strong> we presented in the book, applying them to our simple classification use-case.\n",
    "</p>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #e7692c; padding: 0 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c;\">Tip.</strong> The notebooks shared on this git repository illustrate some of notions from the book \"<em><strong>Hands-on Computer Vision with TensorFlow 2</strong></em>\" written by Eliot Andres and Benjamin Planche and published by Packt. If you enjoyed the insights shared here, <strong>please consider acquiring the book!</strong>\n",
    "<br/><br/>\n",
    "The book provides further guidance for those eager to learn about computer vision and to harness the power of TensorFlow 2 and Keras to build performant recognition systems for object detection, segmentation, video processing, smartphone applications, and more.</p>\n",
    "        </td>\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; width: 250px;\">\n",
    "    <a href=\"https://www.packtpub.com\" title=\"Buy on Packt!\">\n",
    "        <img src=\"../banner_images/book_cover.png\" width=250>\n",
    "    </a>\n",
    "    <p style=\"background: #e7692c; color:#ffffff; padding: 10px; text-align:justify;\"><strong>Leverage deep learning to create powerful image processing apps with TensorFlow 2 and Keras. <br/></strong>Get the book for more insights!</p>\n",
    "    <ul style=\"height: 32px; white-space: nowrap; text-align: center; margin: 0px; padding: 0px; padding-top: 10px;\">\n",
    "    <li style=\"display: inline-block; height: 100%; vertical-align: middle; float: left; margin: 5px; padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com\" title=\"Get the book on Amazon!\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_amazon.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    <li style=\"display: inline-block; height: 100%; vertical-align: middle; float: left; margin: 5px; padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com\" title=\"Get your Packt book!\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_packt.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    <li style=\"display: inline-block; height: 100%; vertical-align: middle; float: left; margin: 5px; padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com\" title=\"Get the book on O'Reilly Safari!\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_oreilly.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    </ul>\n",
    "        </td>\n",
    "        </tr>\n",
    "        </table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "import os\n",
    "# Choosing which GPU this notebook can access\n",
    "# (useful when running multiple experiments in parallel, on different GPUs):\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]= \"0\" \n",
    "\n",
    "# Defining the seed for some random operations:\n",
    "random_seed = 42\n",
    "\n",
    "# Setting some variables to format the logs:\n",
    "log_begin_red, log_begin_blue, log_begin_green = '\\033[91m', '\\033[94m', '\\033[92m'\n",
    "log_begin_bold, log_begin_underline = '\\033[1m', '\\033[4m'\n",
    "log_end_format = '\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once again, we will use the [MNIST](http://yann.lecun.com/exdb/mnist) dataset[$^1$](#ref) for our demonstration (Yann LeCun and Corinna Cortes hold all copyrights for this dataset). Therefore, we prepare the data as done in the previous notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_classes = 10\n",
    "img_rows, img_cols, img_ch = 28, 28, 1\n",
    "input_shape = (img_rows, img_cols, img_ch)\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "x_train, x_test = x_train / 255.0, x_test / 255.0\n",
    "\n",
    "x_train = x_train.reshape(x_train.shape[0], *input_shape)\n",
    "x_test = x_test.reshape(x_test.shape[0], *input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However this time, to highlight the advantages of regularization, we will make the recognition task harder by artificially reducing the number of samples available for training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data: (200, 28, 28, 1)\n",
      "Testing data: (10000, 28, 28, 1)\n"
     ]
    }
   ],
   "source": [
    "x_train, y_train = x_train[:200], y_train[:200] # ... 200 training samples instead of 60,000...\n",
    "\n",
    "print('Training data: {}'.format(x_train.shape))\n",
    "print('Testing data: {}'.format(x_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training a Model with Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying the code presented in the chapter, we will first demonstrate how to implement and apply a regularization loss ourselves. Then, we will show how one can directly use Keras API to train models with standard regularization solutions (*L1/L2*, *dropout*, *batch normalization*), comparing their effects. We will use the training of *LeNet-5*[$^2$](#ref) on MNIST to illustrate all this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Model, Sequential\n",
    "from tensorflow.keras.layers import (Input, Activation, Dense, Flatten, Conv2D, \n",
    "                                     MaxPooling2D, Dropout, BatchNormalization)\n",
    "\n",
    "epochs      = 200\n",
    "batch_size  = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manually applying regularization losses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To demonstrate how to add a regularization loss to any layer, we will start from the simple convolution layer we presented in the book and in a previous [notebook](./ch3_nb2_build_and_train_first_cnn_with_tf2.pynb) (i.e., a simpler, self-made `Conv2d` layer):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def conv_layer(x, kernels, bias, s):\n",
    "    z = tf.nn.conv2d(x, kernels, strides=[1,s,s,1], padding='VALID')\n",
    "    # Finally, applying the bias and activation function (e.g. ReLU):\n",
    "    return tf.nn.relu(z + bias)\n",
    "\n",
    "\n",
    "class SimpleConvolutionLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self, num_kernels=32, kernel_size=(3, 3), stride=1):\n",
    "        \"\"\" Initialize the layer.\n",
    "        :param num_kernels: Number of kernels for the convolution\n",
    "        :param kernel_size: Kernel size (H x W)\n",
    "        :param stride: Vertical/horizontal stride\n",
    "        \"\"\"\n",
    "        super().__init__() \n",
    "        self.num_kernels = num_kernels\n",
    "        self.kernel_size = kernel_size\n",
    "        self.stride = stride\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        \"\"\" Build the layer, initializing its parameters.\n",
    "        This will be internally called the 1st time the layer is used.\n",
    "        :param input_shape: Input shape for the layer (e.g. BxHxWxC)\n",
    "        \"\"\"\n",
    "        num_input_ch = input_shape[-1] # assuming shape format BHWC\n",
    "        # Now we know the shape of the kernel tensor we need:\n",
    "        kernels_shape = (*self.kernel_size, num_input_ch, self.num_kernels)\n",
    "        # We initialize the filter values e.g. from a Glorot distribution:\n",
    "        glorot_init = tf.initializers.GlorotUniform()\n",
    "        self.kernels = self.add_weight( # method to add Variables to layer\n",
    "            name='kernels', shape=kernels_shape, initializer=glorot_init,\n",
    "            trainable=True) # and we make it trainable.\n",
    "        # Same for the bias variable (e.g. from a normal distribution):\n",
    "        self.bias = self.add_weight(\n",
    "            name='bias', shape=(self.num_kernels,), \n",
    "            initializer='random_normal', trainable=True)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        \"\"\" Call the layer, apply its operations to the input tensor.\"\"\"\n",
    "        return conv_layer(inputs, self.kernels, self.bias, self.stride)\n",
    "\n",
    "    def get_config(self):\n",
    "        \"\"\"\n",
    "        Helper function to define the layer and its parameters.\n",
    "        :return:        Dictionary containing the layer's configuration\n",
    "        \"\"\"\n",
    "        return {'num_kernels': self.num_kernels,\n",
    "                'kernel_size': self.kernel_size,\n",
    "                'strides': self.strides,\n",
    "                'use_bias': self.use_bias}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will extend this layer class to add kernel/bias regularization. As explained in the book, the `Layer`'s method `.add_loss()` can be used for that purpose:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from functools import partial\n",
    "\n",
    "def l2_reg(coef=1e-2):\n",
    "    \"\"\"\n",
    "    Returns a function computing a weighed L2 norm of a given tensor.\n",
    "    (this is basically a reimplementation of f.keras.regularizers.l2())\n",
    "    :param coef:    Weight for the norm\n",
    "    :return:        Loss function\n",
    "    \"\"\"\n",
    "    return lambda x: tf.reduce_sum(x ** 2) * coef\n",
    "\n",
    "    \n",
    "class ConvWithRegularizers(SimpleConvolutionLayer):\n",
    "    def __init__(self, num_kernels=32, kernel_size=(3, 3), stride=1,\n",
    "                 kernel_regularizer=l2_reg(), bias_regularizer=None):\n",
    "        \"\"\" \n",
    "        Initialize the layer.\n",
    "        :param num_kernels:        Number of kernels for the convolution\n",
    "        :param kernel_size:        Kernel size (H x W)\n",
    "        :param stride:             Vertical/horizontal stride\n",
    "        :param kernel_regularizer: (opt.) Regularization loss for the kernel variable\n",
    "        :param bias_regularizer:   (opt.) Regularization loss for the bias variable\n",
    "        \"\"\"\n",
    "        super().__init__(num_kernels, kernel_size, stride)  \n",
    "        self.kernel_regularizer = kernel_regularizer\n",
    "        self.bias_regularizer = bias_regularizer\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        \"\"\"\n",
    "        Build the layer, initializing its components.\n",
    "        \"\"\"\n",
    "        super().build(input_shape)\n",
    "        # Attaching the regularization losses to the variables.\n",
    "        if self.kernel_regularizer is not None:\n",
    "            self.add_loss(partial(self.kernel_regularizer, self.kernels))\n",
    "        if self.bias_regularizer is not None:\n",
    "            self.add_loss(partial(self.bias_regularizer, self.bias))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When instantiating this layer, the regularizer passed as attributes will be attached to the layer. The loss values of these regularizers can be obtained whenever, simply calling the layer's property `.losses`. Let us check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Regularization losses over kernel and bias parameters: [1.9857061, 0.07974565]\n",
      "L2 norms of kernel and bias parameters: [1.9857061, 0.07974565]\n"
     ]
    }
   ],
   "source": [
    "conv = ConvWithRegularizers(num_kernels=32, kernel_size=(3, 3), stride=1,\n",
    "                            kernel_regularizer=l2_reg(1.), bias_regularizer=l2_reg(1.))\n",
    "conv.build(input_shape=tf.TensorShape((None, 28, 28, 1)))\n",
    "\n",
    "# Fetching the layer's losses:\n",
    "reg_losses = conv.losses\n",
    "print('Regularization losses over kernel and bias parameters: {}'.format(\n",
    "    [loss.numpy() for loss in reg_losses]))\n",
    "\n",
    "# Comparing with the L2 norms of its kernel and bias tensors:\n",
    "kernel_norm, bias_norm = tf.reduce_sum(conv.kernels ** 2).numpy(), tf.reduce_sum(conv.bias ** 2).numpy()\n",
    "print('L2 norms of kernel and bias parameters: {}'.format([kernel_norm, bias_norm]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The neat thing with the property `.losses` is that it also list the losses attached to all the layers and  models composing an instance. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Losses attached to the model and its layers:\n",
      "\r",
      "[1.966768, 0.10652195, 32.14253, 0.09317449, 32.549416, 0.078679726] (6 losses)\n"
     ]
    }
   ],
   "source": [
    "model = Sequential([\n",
    "    Input(shape=input_shape),\n",
    "    ConvWithRegularizers(kernel_regularizer=l2_reg(1.), bias_regularizer=l2_reg(1.)),\n",
    "    ConvWithRegularizers(kernel_regularizer=l2_reg(1.), bias_regularizer=l2_reg(1.)),\n",
    "    ConvWithRegularizers(kernel_regularizer=l2_reg(1.), bias_regularizer=l2_reg(1.))  \n",
    "])\n",
    "\n",
    "print('Losses attached to the model and its layers:\\n\\r{} ({} losses)'.format(\n",
    "    [loss.numpy() for loss in model.losses], len(model.losses)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us now build a *LeNet-5* with our custom layers, and train it taking into account the regularization terms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LeNet5(Model): # `Model` has the same API as `Layer` + extends it\n",
    "    \n",
    "    def __init__(self, num_classes,\n",
    "                 kernel_regularizer=l2_reg(), bias_regularizer=l2_reg()):\n",
    "        # Create the model and its layers:\n",
    "        super(LeNet5, self).__init__()\n",
    "        self.conv1 = ConvWithRegularizers(\n",
    "            6, kernel_size=(5, 5), \n",
    "            kernel_regularizer=kernel_regularizer, bias_regularizer=bias_regularizer)\n",
    "        self.conv2 = ConvWithRegularizers(\n",
    "            16, kernel_size=(5, 5),\n",
    "            kernel_regularizer=kernel_regularizer, bias_regularizer=bias_regularizer)\n",
    "        self.max_pool = MaxPooling2D(pool_size=(2, 2))\n",
    "        self.flatten = Flatten()\n",
    "        self.dense1 = Dense(120, activation='relu')\n",
    "        self.dense2 = Dense(84, activation='relu')\n",
    "        self.dense3 = Dense(num_classes, activation='softmax')\n",
    "        \n",
    "    def call(self, x): # Apply the layers in order to process the inputs\n",
    "        x = self.max_pool(self.conv1(x)) # 1st block\n",
    "        x = self.max_pool(self.conv2(x)) # 2nd block\n",
    "        x = self.flatten(x)\n",
    "        x = self.dense3(self.dense2(self.dense1(x))) # dense layers\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training: \u001b[91mstart\u001b[0m\n",
      "Epoch   0/200: main loss = \u001b[94m2.298\u001b[0m; reg loss = \u001b[94m0.105\u001b[0m; val acc = \u001b[94m12.500%\u001b[0m\n",
      "Epoch  10/200: main loss = \u001b[94m1.748\u001b[0m; reg loss = \u001b[94m0.107\u001b[0m; val acc = \u001b[94m48.300%\u001b[0m\n",
      "Epoch  20/200: main loss = \u001b[94m0.424\u001b[0m; reg loss = \u001b[94m0.129\u001b[0m; val acc = \u001b[94m74.020%\u001b[0m\n",
      "Epoch  30/200: main loss = \u001b[94m0.131\u001b[0m; reg loss = \u001b[94m0.137\u001b[0m; val acc = \u001b[94m78.750%\u001b[0m\n",
      "Epoch  40/200: main loss = \u001b[94m0.052\u001b[0m; reg loss = \u001b[94m0.139\u001b[0m; val acc = \u001b[94m79.830%\u001b[0m\n",
      "Epoch  50/200: main loss = \u001b[94m0.028\u001b[0m; reg loss = \u001b[94m0.136\u001b[0m; val acc = \u001b[94m79.810%\u001b[0m\n",
      "Epoch  60/200: main loss = \u001b[94m0.019\u001b[0m; reg loss = \u001b[94m0.131\u001b[0m; val acc = \u001b[94m79.830%\u001b[0m\n",
      "Epoch  70/200: main loss = \u001b[94m0.015\u001b[0m; reg loss = \u001b[94m0.126\u001b[0m; val acc = \u001b[94m79.800%\u001b[0m\n",
      "Epoch  80/200: main loss = \u001b[94m0.013\u001b[0m; reg loss = \u001b[94m0.121\u001b[0m; val acc = \u001b[94m79.990%\u001b[0m\n",
      "Epoch  90/200: main loss = \u001b[94m0.011\u001b[0m; reg loss = \u001b[94m0.115\u001b[0m; val acc = \u001b[94m80.040%\u001b[0m\n",
      "Epoch 100/200: main loss = \u001b[94m0.011\u001b[0m; reg loss = \u001b[94m0.110\u001b[0m; val acc = \u001b[94m80.080%\u001b[0m\n",
      "Epoch 110/200: main loss = \u001b[94m0.010\u001b[0m; reg loss = \u001b[94m0.105\u001b[0m; val acc = \u001b[94m80.070%\u001b[0m\n",
      "Epoch 120/200: main loss = \u001b[94m0.010\u001b[0m; reg loss = \u001b[94m0.100\u001b[0m; val acc = \u001b[94m80.120%\u001b[0m\n",
      "Epoch 130/200: main loss = \u001b[94m0.009\u001b[0m; reg loss = \u001b[94m0.096\u001b[0m; val acc = \u001b[94m80.220%\u001b[0m\n",
      "Epoch 140/200: main loss = \u001b[94m0.009\u001b[0m; reg loss = \u001b[94m0.092\u001b[0m; val acc = \u001b[94m80.270%\u001b[0m\n",
      "Epoch 150/200: main loss = \u001b[94m0.009\u001b[0m; reg loss = \u001b[94m0.088\u001b[0m; val acc = \u001b[94m80.310%\u001b[0m\n",
      "Epoch 160/200: main loss = \u001b[94m0.009\u001b[0m; reg loss = \u001b[94m0.084\u001b[0m; val acc = \u001b[94m80.350%\u001b[0m\n",
      "Epoch 170/200: main loss = \u001b[94m0.009\u001b[0m; reg loss = \u001b[94m0.081\u001b[0m; val acc = \u001b[94m80.390%\u001b[0m\n",
      "Epoch 180/200: main loss = \u001b[94m0.008\u001b[0m; reg loss = \u001b[94m0.078\u001b[0m; val acc = \u001b[94m80.430%\u001b[0m\n",
      "Epoch 190/200: main loss = \u001b[94m0.008\u001b[0m; reg loss = \u001b[94m0.075\u001b[0m; val acc = \u001b[94m80.530%\u001b[0m\n",
      "Epoch 199/200: main loss = \u001b[94m0.008\u001b[0m; reg loss = \u001b[94m0.073\u001b[0m; val acc = \u001b[94m80.560%\u001b[0m\n",
      "Training: \u001b[92mend\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "optimizer  = tf.optimizers.SGD()\n",
    "dataset    = tf.data.Dataset.from_tensor_slices((x_train, y_train)).batch(batch_size)\n",
    "log_string_template = 'Epoch {0:3}/{1}: main loss = {5}{2:5.3f}{6}; ' + \\\n",
    "                      'reg loss = {5}{3:5.3f}{6}; val acc = {5}{4:5.3f}%{6}'\n",
    "\n",
    "def train_classifier_on_mnist(model, log_frequency=10):\n",
    "\n",
    "    avg_main_loss = tf.keras.metrics.Mean(name='avg_main_loss', dtype=tf.float32)\n",
    "    avg_reg_loss  = tf.keras.metrics.Mean(name='avg_reg_loss', dtype=tf.float32)\n",
    "\n",
    "    print(\"Training: {}start{}\".format(log_begin_red, log_end_format))\n",
    "    for epoch in range(epochs):\n",
    "        for (batch_images, batch_gts) in dataset:    # For each batch of this epoch\n",
    "\n",
    "            with tf.GradientTape() as grad_tape:     # Tell TF to tape the gradients\n",
    "                y = model(batch_images)              # Feed forward\n",
    "                main_loss = tf.losses.sparse_categorical_crossentropy(\n",
    "                    batch_gts, y)                    # Compute loss\n",
    "                reg_loss = sum(model.losses)         # List and add other losses\n",
    "                loss = main_loss + reg_loss\n",
    "\n",
    "            # Get the gradients of combined losses and back-propagate:\n",
    "            grads = grad_tape.gradient(loss, model.trainable_variables)\n",
    "            optimizer.apply_gradients(zip(grads, model.trainable_variables))\n",
    "\n",
    "            # Keep track of losses for display:\n",
    "            avg_main_loss.update_state(main_loss)\n",
    "            avg_reg_loss.update_state(reg_loss)\n",
    "\n",
    "        if epoch % log_frequency == 0 or epoch == (epochs - 1): # Log some metrics\n",
    "            # Validate, computing the accuracy on test data:\n",
    "            acc = tf.reduce_mean(tf.metrics.sparse_categorical_accuracy(\n",
    "                tf.constant(y_test), model(x_test))).numpy() * 100\n",
    "\n",
    "            main_loss = avg_main_loss.result()\n",
    "            reg_loss = avg_reg_loss.result()\n",
    "\n",
    "            print(log_string_template.format(\n",
    "                epoch, epochs, main_loss, reg_loss, acc, log_begin_blue, log_end_format))\n",
    "\n",
    "        avg_main_loss.reset_states()\n",
    "        avg_reg_loss.reset_states()\n",
    "    print(\"Training: {}end{}\".format(log_begin_green, log_end_format))\n",
    "    return model\n",
    "    \n",
    "\n",
    "model = LeNet5(10, kernel_regularizer=l2_reg(), bias_regularizer=l2_reg())\n",
    "model = train_classifier_on_mnist(model, log_frequency=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is interesting to observe that, at first, the regularization losses increase while the classification loss decreases. As the value of the latter is much higher at first, the network basically focuses on minimizing it, regardless of the values its kernels/biases are taking. Once the classification dropped low enough, then the regularization losses start being taken into account too. \n",
    "\n",
    "Let us compare the accuracy of our regularized network with one without these terms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training: \u001b[91mstart\u001b[0m\n",
      "Epoch   0/200: main loss = \u001b[94m2.311\u001b[0m; reg loss = \u001b[94m0.000\u001b[0m; val acc = \u001b[94m17.000%\u001b[0m\n",
      "Epoch  50/200: main loss = \u001b[94m0.017\u001b[0m; reg loss = \u001b[94m0.000\u001b[0m; val acc = \u001b[94m77.400%\u001b[0m\n",
      "Epoch 100/200: main loss = \u001b[94m0.003\u001b[0m; reg loss = \u001b[94m0.000\u001b[0m; val acc = \u001b[94m78.090%\u001b[0m\n",
      "Epoch 150/200: main loss = \u001b[94m0.001\u001b[0m; reg loss = \u001b[94m0.000\u001b[0m; val acc = \u001b[94m78.100%\u001b[0m\n",
      "Epoch 199/200: main loss = \u001b[94m0.001\u001b[0m; reg loss = \u001b[94m0.000\u001b[0m; val acc = \u001b[94m78.130%\u001b[0m\n",
      "Training: \u001b[92mend\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<__main__.LeNet5 at 0x7fe1def8b9b0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LeNet5(10, kernel_regularizer=None, bias_regularizer=None)\n",
    "model = train_classifier_on_mnist(model, log_frequency=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A 2% accuracy gain on the test set, which is non negligeable!\n",
    "\n",
    "The usage of **`add_loss()`** and **`.losses`** is however the main take-away of this experiment, as they can become useful for more complex models, when we want to apply layer-specific losses for instance.\n",
    "\n",
    "When it comes to regularization, TensorFlow and Keras already provide simpler tools, as we will cover in the next section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying Various Pre-implemented Regularization Methods\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the L1/L2 regularization, other methods were presented in the chapter. Switching completely to the Keras API, we will experiment with those methods and quickly compare their effect on our toy use-case.\n",
    "\n",
    "For that, let us create another _LeNet-5_ factory function (using the Sequential API this time. just to illustrate the differences):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lenet(name='lenet', input_shape=input_shape,\n",
    "          use_dropout=False, use_batchnorm=False, regularizer=None):\n",
    "    \"\"\"\n",
    "    Create a LeNet-5 Keras model, with optional regularization schemes.\n",
    "    :param name:           Name for the model\n",
    "    :param input_shape:    Model's input shape\n",
    "    :param use_dropout:    Flag to add Dropout layers after key layers\n",
    "    :param use_batchnorm:  Flag to add BatchNormalization layers after key layers\n",
    "    :param regularizer:    Regularization function to be applied to layers' kernels\n",
    "    :return:               LeNet-5 Keras model\n",
    "    \"\"\"\n",
    "    \n",
    "    layers = []\n",
    "    \n",
    "    # 1st block:\n",
    "    layers += [Conv2D(6, kernel_size=(5, 5), padding='same', \n",
    "                      input_shape=input_shape, kernel_regularizer=regularizer)]\n",
    "    if use_batchnorm:\n",
    "        layers += [BatchNormalization()]\n",
    "    layers += [Activation('relu'),\n",
    "               MaxPooling2D(pool_size=(2, 2))]\n",
    "    if use_dropout:\n",
    "        layers += [Dropout(0.25)]\n",
    "        \n",
    "    # 2nd block:\n",
    "    layers += [ \n",
    "        Conv2D(16, kernel_size=(5, 5), kernel_regularizer=regularizer)]\n",
    "    if use_batchnorm:\n",
    "        layers += [BatchNormalization()]\n",
    "    layers += [Activation('relu'),\n",
    "               MaxPooling2D(pool_size=(2, 2))]\n",
    "    if use_dropout:\n",
    "        layers += [Dropout(0.25)]\n",
    " \n",
    "    # Dense layers:\n",
    "    layers += [Flatten()]\n",
    "    \n",
    "    layers += [Dense(120, kernel_regularizer=regularizer)]\n",
    "    if use_batchnorm:\n",
    "        layers += [BatchNormalization()]\n",
    "    layers += [Activation('relu')]\n",
    "    if use_dropout:\n",
    "        layers += [Dropout(0.25)]\n",
    "        \n",
    "    layers += [Dense(84, kernel_regularizer=regularizer)]\n",
    "    layers += [Activation('relu')]\n",
    "        \n",
    "    layers += [Dense(num_classes, activation='softmax')]\n",
    "    \n",
    "    model = Sequential(layers, name=name)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To showcase the effect that modern optimizers (available in Tensorflow and Keras) have on trainings, we will create several similar LeNet instance, and train each with a different combination of regularization techniques[$^{3,4,5}$](#ref)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we pre-define the various configurations we want to compare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "configurations = {\n",
    "    'none':         {'use_dropout': False, 'use_batchnorm': False, 'regularizer': None},\n",
    "    'l1':           {'use_dropout': False, 'use_batchnorm': False, 'regularizer': tf.keras.regularizers.l1(0.01)},\n",
    "    'l2':           {'use_dropout': False, 'use_batchnorm': False, 'regularizer': tf.keras.regularizers.l2(0.01)},\n",
    "    'dropout':      {'use_dropout': True,  'use_batchnorm': False, 'regularizer': None},\n",
    "    'bn':           {'use_dropout': False, 'use_batchnorm': True,  'regularizer': None},\n",
    "    'l1+dropout':   {'use_dropout': False, 'use_batchnorm': True,  'regularizer': tf.keras.regularizers.l1(0.01)},\n",
    "    'l1+bn':        {'use_dropout': False, 'use_batchnorm': True,  'regularizer': tf.keras.regularizers.l1(0.01)},\n",
    "    'l1+dropout+bn':{'use_dropout': False, 'use_batchnorm': True,  'regularizer': tf.keras.regularizers.l1(0.01)}\n",
    "    # ...\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each regularization configuration we are considering, we will instantiate a new LeNet model and train it with. We will save their training `history` (containing the losses and metrics history over training), for comparison *(this process takes time, especially on CPU!)*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history_per_instance = dict()\n",
    "\n",
    "print(\"Experiment: {0}start{1} (training logs = off)\".format(log_begin_red, log_end_format))\n",
    "for config_name in configurations:\n",
    "    # Resetting the seeds (for random number generation), to reduce the impact of randomness on the comparison:\n",
    "    tf.random.set_seed(random_seed)\n",
    "    np.random.seed(random_seed)\n",
    "    # Creating the model:\n",
    "    model = lenet(\"lenet_{}\".format(config_name), **configurations[config_name])\n",
    "    model.compile(optimizer='sgd', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "    # Launching the training (we set `verbose=0`, so the training won't generate any logs):\n",
    "    print(\"\\t> Training with {0}: {1}start{2}\".format(\n",
    "        config_name, log_begin_red, log_end_format))\n",
    "    history = model.fit(x_train, y_train,\n",
    "                        batch_size=32, epochs=300, validation_data=(x_test, y_test),\n",
    "                        verbose=0)\n",
    "    history_per_instance[config_name] = history\n",
    "    print('\\t> Training with {0}: {1}done{2}.'.format(\n",
    "        config_name, log_begin_green, log_end_format))\n",
    "print(\"Experiment: {0}done{1}\".format(log_begin_green, log_end_format))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We finally plot the training and validation losses + accuracies, for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2, 2, figsize=(10,10), sharex='col') # add parameter `sharey='row'` for a more direct comparison\n",
    "ax[0, 0].set_title(\"loss\")\n",
    "ax[0, 1].set_title(\"val-loss\")\n",
    "ax[1, 0].set_title(\"accuracy\")\n",
    "ax[1, 1].set_title(\"val-accuracy\")\n",
    "\n",
    "lines, labels = [], []\n",
    "for config_name in history_per_instance:\n",
    "    history = history_per_instance[config_name]\n",
    "    ax[0, 0].plot(history.history['loss'])\n",
    "    ax[0, 1].plot(history.history['val_loss'])\n",
    "    ax[1, 0].plot(history.history['accuracy'])\n",
    "    line = ax[1, 1].plot(history.history['val_accuracy'])\n",
    "    lines.append(line[0])\n",
    "    labels.append(config_name)\n",
    "\n",
    "fig.legend(lines,labels, loc='center right', borderaxespad=0.1)\n",
    "plt.subplots_adjust(right=0.84)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for config_name in history_per_instance:\n",
    "    best_val_acc = max(history_per_instance[config_name].history['val_accuracy']) * 100\n",
    "    print('Max val-accuracy for model \"{}\": {:2.2f}%'.format(config_name, best_val_acc))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note:*** As we can observe from the figure, it seems like some of the models could still further improve if trained longer. We leave it  to our readers to check."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Given how small and artificial this problem is, the idea is not to directly compare the performance of these regularization techniques, but rather to demonstrate their overall benefits. It is wise to keep these methods in mind when tackling real-life computer vision problems, as data scarcity and overfitting are common problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref\"></a>\n",
    "#### References\n",
    "\n",
    "1. LeCun, Y., Cortes, C., Burges, C., 2010. MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann.lecun.com/exdb/mnist 2, 18.\n",
    "2. LeCun, Yann. \"*LeNet-5, convolutional neural networks.*\" [http://yann.lecun.com/exdb/lenet](http://yann.lecun.com/exdb/lenet) (2015): 20.\n",
    "3. Girosi, F., Jones, M., Poggio, T., 1995. Regularization theory and neural networks architectures. Neural computation 7, 219269.\n",
    "4. Ioffe, S., Szegedy, C., 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.\n",
    "5. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 19291958.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}