cifar10_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "YMxTDfS1hxsB"
      },
      "outputs": [],
      "source": [
        "# For tips on running notebooks in Google Colab, see\n",
        "# https://pytorch.org/tutorials/beginner/colab\n",
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FvtgHAz1hxsF"
      },
      "source": [
        "\n",
        "# Training a Classifier\n",
        "\n",
        "This is it. You have seen how to define neural networks, compute loss and make\n",
        "updates to the weights of the network.\n",
        "\n",
        "Now you might be thinking,\n",
        "\n",
        "## What about data?\n",
        "\n",
        "Generally, when you have to deal with image, text, audio or video data,\n",
        "you can use standard python packages that load data into a numpy array.\n",
        "Then you can convert this array into a ``torch.*Tensor``.\n",
        "\n",
        "-  For images, packages such as Pillow, OpenCV are useful\n",
        "-  For audio, packages such as scipy and librosa\n",
        "-  For text, either raw Python or Cython based loading, or NLTK and\n",
        "   SpaCy are useful\n",
        "\n",
        "Specifically for vision, we have created a package called\n",
        "``torchvision``, that has data loaders for common datasets such as\n",
        "ImageNet, CIFAR10, MNIST, etc. and data transformers for images, viz.,\n",
        "``torchvision.datasets`` and ``torch.utils.data.DataLoader``.\n",
        "\n",
        "This provides a huge convenience and avoids writing boilerplate code.\n",
        "\n",
        "For this tutorial, we will use the CIFAR10 dataset.\n",
        "It has the classes: ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,\n",
        "‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. The images in CIFAR-10 are of\n",
        "size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n",
        "\n",
        ".. figure:: /_static/img/cifar10.png\n",
        "   :alt: cifar10\n",
        "\n",
        "   cifar10\n",
        "\n",
        "\n",
        "## Training an image classifier\n",
        "\n",
        "We will do the following steps in order:\n",
        "\n",
        "1. Load and normalize the CIFAR10 training and test datasets using\n",
        "   ``torchvision``\n",
        "2. Define a Convolutional Neural Network\n",
        "3. Define a loss function\n",
        "4. Train the network on the training data\n",
        "5. Test the network on the test data\n",
        "\n",
        "### 1. Load and normalize CIFAR10\n",
        "\n",
        "Using ``torchvision``, it’s extremely easy to load CIFAR10.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "PCRtrZ9phxsK"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I8V8faKXhxsN"
      },
      "source": [
        "The output of torchvision datasets are PILImage images of range [0, 1].\n",
        "We transform them to Tensors of normalized range [-1, 1].\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JdtedVFNhxsO"
      },
      "source": [
        "<div class=\"alert alert-info\"><h4>Note</h4><p>If running on Windows and you get a BrokenPipeError, try setting\n",
        "    the num_worker of torch.utils.data.DataLoader() to 0.</p></div>\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "PG0wIkcYhxsP",
        "outputId": "7c5f09ba-c37e-4a3a-e55d-81982f89114c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Files already downloaded and verified\n",
            "Files already downloaded and verified\n"
          ]
        }
      ],
      "source": [
        "transform = transforms.Compose(\n",
        "    [transforms.ToTensor(),\n",
        "     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
        "\n",
        "batch_size = 4\n",
        "\n",
        "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
        "                                        download=True, transform=transform)\n",
        "trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,\n",
        "                                          shuffle=True, num_workers=2)\n",
        "\n",
        "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
        "                                       download=True, transform=transform)\n",
        "testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size,\n",
        "                                         shuffle=False, num_workers=2)\n",
        "\n",
        "classes = ('plane', 'car', 'bird', 'cat',\n",
        "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y8hEdJY0hxsR"
      },
      "source": [
        "Let us show some of the training images, for fun.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "si_W2idZhxsS",
        "outputId": "28fe5a87-21e3-4c83-abe9-4bffc2394f8d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAACwCAYAAACviAzDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABNZ0lEQVR4nO29eZRc1XX/u++tW1N3dVXP3Wq1WmokgRgNlkAI/OJJDiZ+2Bh+ic0itjys+JFIjkFvxTZ27Kw4IWIl6xcPWRj/kkXAeTHBIT+DExzjHxYYgi0xyBI2Bk0g0NhzV1fXPNzz/iDU2ftbdKkbWtUa9metXuueOrfuPffcc2+dPnvv73aMMYYURVEURVEahLvQDVAURVEU5cxCJx+KoiiKojQUnXwoiqIoitJQdPKhKIqiKEpD0cmHoiiKoigNRScfiqIoiqI0FJ18KIqiKIrSUHTyoSiKoihKQ9HJh6IoiqIoDUUnH4qiKIqiNJQTNvm44447aNmyZRSJRGjt2rX09NNPn6hTKYqiKIpyCuGciNwu3//+9+njH/84fec736G1a9fSN77xDbr//vtpz5491N3dXfe7vu/T0aNHqaWlhRzHme+mKYqiKIpyAjDG0PT0NPX19ZHrHmdtw5wALrvsMrNx48ZquVKpmL6+PrNly5bjfvfQoUOGiPRP//RP//RP//TvFPw7dOjQcX/rPZpnisUi7dixg2699dbqZ67r0vr162nbtm01+xcKBSoUCtWy+e+FmFtuuYXC4fB8N09RFEVRlBNAoVCgr3/969TS0nLcfed98jE2NkaVSoV6enrE5z09PbR79+6a/bds2UJ//ud/XvN5OBzWyYeiKIqinGLMxmViwaNdbr31Vpqamqr+HTp0aKGbpCiKoijKCWTeVz46OzspEAjQ8PCw+Hx4eJh6e3tr9tcVDkVRFEU5s5j3lY9QKESrV6+mrVu3Vj/zfZ+2bt1K69atm+/TKYqiKIpyijHvKx9ERJs3b6YNGzbQmjVr6LLLLqNvfOMblMlk6JOf/ORbPvYl124Q5eZwoLodCcrLCQdCojyWmq5uH0tOi7rh8Ux1++jwlKjLV4woV4iV/Yqo89h0LhQMiLq2uHXC6WhLiLrpTFqUuRNuS5NcGYo3RarbnW1xUWf8smxr2bYv6EBbA9Yu95v9r4q6QxNZUe7q7Kput7PzExFF2XVOjk+Iup3PPS/KL+7ZX92+fnXtStjr3NTTIcq+I+9lORKtbhdCsn9C8ebqdqSjVdS58ZgsR+y1hALyOCZs93VCcp7uBX15HC9o93XlcXxm//QdOZaIRbpDTQ0e+1/BxcOQecPtNz6yM2PNm8VAxL7n2Wfxttu21P3u5Is/rW739C4SdWNjY9XtXvAjCwTk8zUxYcdeKCTHS6KtrbodaWoWdcFgUJS7uuxYb2XfIyIKhu1x+TXicTDM0MjhQoGA/W65Ip9ZflM8aBsKI1R8e+AIriC79hw4JrF9vC8jEfl8+46tw+9h391xz/+mmfjf//YvMx7nzWKgY40PHVTX9WD2o7++IIWtdFx5Qu774LrYGFn22b0kM/O+eBxsG38W6/le4PhFyuXyjHUfvu4jdb87G07I5OMjH/kIjY6O0le/+lUaGhqiiy++mB5++OEaJ1RFURRFUc48Tsjkg4ho06ZNtGnTphN1eEVRFEVRTlEWPNpFURRFUZQzixO28nGiMGArrDDb6dh0QdSNjgyJ8tHRZHV7KlMSdQ7zDymXpZ2sKSrtx+yUZCoV2Nf6IpRLsj1839RUStTlStK+NpW2PheFojxHNm/bnsnL64iArVvYHCuyPdxfpjXeKuqKRh6HmxkDAdk/rczHYnFXu6jrgbIrbLTSr4QTLslzVMBXwnPscYIE/hfpfHXbL4+LuvJURpQD7H65kSZoRJbtJ/uDImDTZ+PH8WBsRZgvAPiO8PFsoF8pAPvy7RobNLvPaC+usR/PD+jnwZmLTZ/7G2Qz8v60tbbOeEyMkuPRdOiL0BSLsW0pgITHibF9o+D/EIraMtrMefvQ1m58ua/nMd8RsK27rD/Kpbyoq/hybHGvFwf8H8RwcmSdF0CfD1YAvzHPs/c54Eg/m0B9pwoBHy/Cv+E4YF9KXx/4PSD5ruTnrB2vAZoZua8x/Liy7dLPY+b+qH1cZn5+avuH+WkZ9PnAts7s88HLeI6aMTv/mVcEuvKhKIqiKEpD0cmHoiiKoigN5ZQzuySnZUjq+JRdOirKFUmazMrlw7FpW55KwfJlwC4xBYNy+aktIZdpEwm79FopYpic/W6xKJd+S8VidduvyCWvXK4oytmcPW6pKPdNZ635JJWWy7KxZhlG2MKWkMOwZBpjoaWDAzISqXlcmiuOjdqQxwAsbUZDLEzPk323uFuGzK655KLqdnr/dpoJXBI0LiwpsyVLtyzvZZCZzSp5uXToYygeuw+mAAOIrGksD6up2aCct7ue7ctgSC7Vuzw0OSqX+J0QewRD8DhC2bCyA8vmDg+HhP8pnAb8j4FLtHPJSN23qK+6HQBTBg/7jDITGdYRSbMHml1cdtxAUJrQakJL2dirtzSNob78/K4r69Ds4jrsXnpoTrLtKUFMdakgnz1h6oFl/EqZmVld+Z4yvmxftsTeTfCMVFgoMJq+ikX53qpHPVNcPdMBUmHmaxxneI565h3+XQxP98H01Nxin9uWuBwvkxNWmgFeRXVNcYgwEeF7ag4WkHrn4eeoF0pLNDfT2JtBVz4URVEURWkoOvlQFEVRFKWh6ORDURRFUZSGcsr5fKBZPluwdqsCmLAKPvhcGFvOQwibn7NhlbFmaZcvgDOJMbY+EZfy5rmsPW42l5PHKVsbGpdPJyLKZGXZ86xdESWEC8zPpAKhvoakDdYN2Pa4EQiTY7LkXd2doq4dZNsTCes7kpqalOdk4X++kfPZqWkZTuvPMjSvEqzv8+E4tr8CEDLnMBt1ANrjgK27ULA+RGUM6SuzEEPw46AohOUG7TmNI8eWSdtx50MKAMPs/U4Y/BQi0jehyO5foFn6PzS1WLn+Gnl1tAELOzQtOB2dduyhzZ77Y6AtG0NkuQ8G+nx4bKw74KtRsy/zD4lEIZUA8zup8Sup4/PhV1BK226XwG8slbLPVy6TFHXFnPR5K5bss4fh+1PTtuyFIVQcnsOhIStLEA7J6+rssvcHJdzHmfz9a7TTTPD+QZ+OGh+vWYZ5Hs+Pot45efixgRDm9g75/lu+YqC6HYvJvhw6NlLdfvklmVCVh8Uer62BAB8/so7/BPjwe4D+g/MVIiuOcwLeE7ryoSiKoihKQ9HJh6IoiqIoDUUnH4qiKIqiNJRTzudjOgs+Do6dP6XT0sdielr6P6RS09VtF+LnXWZ798Fmns1JG/7YhLWx5cLSVyPA0li7kKI9m7f+D2MT0j6LRrXWhNXrKINNr+TzWG15lHJFtr2JSXuHwxDbz9qTBD+OREz6FCxfZrUYRkflsBkfHrXHzMq+Gh6fEuWRSXsPZHJ7Sbkij1Mu+7hDdTMIcsNc7wDTlRdKcvyUuH27WfpxRHx73Hi71EEpg72f3z+vRpPEYsC2XSkwKWTwLXLycmyVA8xnKCXPnxmz/Ww8Wdfd1y+PG5r//zneip2Z+1igXZzXNTXB/QF9DofrfIBeCO/1EjxP6DdVYn4UWDfN5N9R56PI/LiSU3LcZzPg/8X2xetKpey74dihV2VbIWVDczP3xZLndJi+ejQmn7bWREKUU1MT1e2WmNQKCgaszwdq/DRH4DmYOWNCXerJgOPI4o87vqtrE9EznwtMX+Dba2mGaz7vwotFOcQ0duLNoBPj2b4dHZkWdalpe99rfT7gncZ8UAIe6MTwtwik/0A/Ood3UM1jac/hOPW1RAIe90GZf6cPXflQFEVRFKWh6ORDURRFUZSGcsqZXV46cFiUo0y62viwHAXLU80sM2lbm1x25KGu5bJcWnQCcmkxzaTQMymZhTMStuaKlkSrPI7Hw/0gayxJuOmgApq9PlsKdgOQLROWgiNhW9/aKpdefRZunJqWy4UOhJ45ZJclO6HveGbLI0MToi4N0vBZZj6pZ3bJJ0dluSRNaA4LTyxDiKHHlggdWJI00NOBhA2pC3lSCt6UWObczm55DgiDLfOQXpBmFsuiKAfNth34nluW66BxtvR6+OBRUbdz3/7qdvvAUlHX1d0nyi5ruo+y6KIwh6XWt7AqO81MEhEINzasT9IpeZ8rIEFdYpmhi2CPLLGyC8vvOQiJr7DnP56QIZd5Zi6Jtci0C9wMM5VMiropGKMJZvZYf9Vvi7rxcfsMjYDZpQzXbJiZt61dhst7QduePJgxIyDzn2ixpp9gEMOdmSkZxkQTyBKQfPxnDZqwRAbcmhBZtnk8yXL2HnXAPMszFKxYvhwaJK9rOGnf8yvOvUTU5aZtKor4Sy+Juowwt2HWWHlKbtqoCZdn4bUGM/eCKTfAfq9cB39Z7HdrwpvhnME65tD5QFc+FEVRFEVpKDr5UBRFURSloejkQ1EURVGUhnLK+XyMjUvbaRML7WxvaxV14RDIMQetHa8JQsTKwqcApX8rULbb0TDIbLNwv0xGxp1VWAhfFOSOW1qkB0Tfoi57nJQMg52cTNqWOvIWhiANe5Kley5B+msequgb8DcISJ8Pz7P9bMAuH2cS85mi7KsI2LqdWdqETUbuWIQwVK9ibcRmekTUhZmt0vPk/QkEZQhxmYVOFrLS78Vn9vRSr/SjMOCPYVgIZN6R/ZxnadCDIWlLdgK8rbJfwxSCsh3PQfClibLw0QTKfpdlewx77GtCE9lxUNYfiuTycEAIXZ+LifiJn/+iuu2BPfucFcuq20XIn5CG8NUQS0WfSst7GYtZ/4zuLunbUwSfD4/1X3NUjheHG+pLckwG2LPf0yH9L5YuXiLKXCIgFpXPfrDbXsfgshWirgTvoiIbPyH4XzKVtL4IlTKEbUMIcbFo6wt52c/cJwbDgjFNBBHKuFvebDh2jb8BK3oYUl0nDbyBvmtjKSQ6O+WYePGVQ6LclLD304RkWG6807ahu0f6ho2O2HsArnvkefB8MymEMjyzXELdgbBXLhNBBM/tHPq8nuS9+nwoiqIoinLKo5MPRVEURVEayilndoHVQnLZ0ngCVPu8gFxGyqTtEntmWmaH5OGruFSVTUvzCVccdWKwrM+yWWazUnGQKydGIStqFywFd3W1Vbe72+UyX1vclrOQEdOBELF82po9RsbkUjRLhErFMmSGhWW2IDMJ5PNy/TAUtmVcql/ULa+LW7Qy4y/QTER8ueyYjco+aOqyZhDTIufQlcMvV7fBekQ+zLc91iDPATODZ8s1y9Q5eeDikA19DXXKpfpswS5bh8EkUvG4Iq6sCxh5TzLs3gZg6XX5gF3Wj8ak8mcpKc1SJd8u8zvwPDlZ29ZcEZbqS/K56FiyuLpt4vKcPqon1iHHFIRD8pJp2Vkrq9seqD4eOzYkyiLDNKwSc1NKPievC8c6N0di1moeyl4Cs0shz0LXC/LZJwjzjLfZ7K8vHzgo6jq7rJruRWuuEHXJjHyGx5LWJGvgXdAStw/41LQ0Y1aKMvQ2z0wtBuwDFSZZEAaTQzk0s5mjHmgewWzGnLmYXXBffo9QDTXEMlXnQMW6CdRzuzvt+zg1JsddX4/t574+aXZJp+zvzOGD8jnEsH8+7koF2daCb9uH0bOeI58LnlEZFa9zzPRez0RFVKvuO9/oyoeiKIqiKA1FJx+KoiiKojSUOU8+nnjiCbrmmmuor6+PHMehBx98UNQbY+irX/0qLVq0iKLRKK1fv5727ds3X+1VFEVRFOUUZ84+H5lMht72trfRpz71Kbruuutq6v/6r/+avvWtb9F3v/tdGhwcpK985St01VVX0QsvvFCThfLNUIGMlGEmZx6BsFe/Iv0GuO1yYlKGgBaZNDPayUpg+zLM6JjLyXPUy/7H7ZoBMKeNj8tw2mLB+qd0tkm7czOTdc7DdfhGHjjGbNYOhHb5LMNhsST7NQdhjWMpa3N0wXYaDlifmGJe+sc4IJt8zqDNsPrLXTQjBRZOTEQUWyFDFdtXvq26PR6Ww7gwam2rIZDc9yAUOcTC1DwIFy24LLtpScroh6Gfsynb3pY22c+8DwJgZy0wGyzKJpchbLkSZMeFfxuCzB+ixEIsiYhGi7LtlSZrt3dgrAeZjbwMxuUSpASIL7I+HyUXfD7mEOK3bHCwuh2D7KKJTuv/4IHsdyonxxrP0hwMyH2n09YHIz0NWarHpJQ/D3lGH4JploYAfT7CEeuT48N7qlySz1P/wLLqdi9kHebS3j9+7ElR9eyOp0U5xOQE4uBHdtYye44A+L8lwFct6LE0FQ74HrH25HPyOmKxNpKAr8sMzC10E8O4Z/5uM2SmTjM/P8yMHeS/R/A8DS6VKQmWDthyBd5xyaNj1W2/Iv2Jlg1a37RYs/R/Gx6W465c5ik+5NgKMqn8lhbp27ioR46ftnbrT/TSSy+Lun0vHaCZwFDbExFey5nz5OPqq6+mq6+++g3rjDH0jW98g/70T/+UPvShDxER0T/90z9RT08PPfjgg/TRj370rbVWURRFUZRTnnn1+Thw4AANDQ3R+vXrq58lEglau3Ytbdu27Q2/UygUKJVKiT9FURRFUU5f5nXyMTT0WghST0+P+Lynp6dah2zZsoUSiUT1b8mSJW+4n6IoiqIopwcLrvNx66230ubNm6vlVCpVdwJSAttpmmlwHD06LOp8kKjl6Y2TKWm3KwjpcTknc0H22mWx5dxvgkhK34ZRSpvZ5VFvYnpa2uVLzOcDXBGoqdnaKrNg90aBAy9mNR2izVJ/IpWy9usi2PvKIB+eY83FFM6+a++JXwBp8byUrnZKKMf8xmRT+0U5Oib7cvJ5277ihPSXCbI07CiNXwTp6AIxvw5far/kmK9GPCtTbsei0tbdxFKWe6ghw7RP0BfB4WVXPo41Eu5MO6PkS5sw9ysxssvJB6lxE2RlsC3zFPZeXF6jn2gX5XHmFzQyJq85DeOgHu9897uq22jPbmu3dvJUWq6KDp4tx/Pu55+rbh85IuWx+WPqBaV/CuoZcN8stIOXy/aacyDLzr/XBLLsPV1S/4Ef1QF9l+2/tNdx+//8lqi7+IKVonzD//gd1jj5XPLrLJWln0tLi/Stod5ee5iSvHetTD8pFJJ+P7FmOM6b9fmAfuYlFzRjAuwd54J+SgjSJ/DmZUE7Ixqz75RwWB6np0MeJzd1rLqNaRj44KqUIBUH85uKt0kZffhZody07TvTKcdoe6dNtzG49BxR19raJcpjY9bn7YU9UkuJv0Mc+K2o9fFg/VzHl/HNMq8rH73/PYCHh+UkYHh4uFqHhMNhisfj4k9RFEVRlNOXeZ18DA4OUm9vL23durX6WSqVoqeeeorWrVs3n6dSFEVRFOUUZc5ml3Q6Tfv32yXxAwcO0K5du6i9vZ0GBgbo5ptvpr/8y7+klStXVkNt+/r66Nprr52XBqPZZZJlbZ2eguUwWE7lsuiwOibq3ABI9kJYZZgtj0eC0hwQjthlyXBY1vGsly6EoDqwLGpYe7IgZ15k11UqQGbPoDxuPsfkw0NyKTEQsEtpAVhyMz4sg1aYKQOW6lPZZHV75KiUij74yquiHGHhiH11Rp+fGxPlzMHtopw7uqO67RXlgaJllo0WwgbLrlyaLoVYqGsLXHPQrsKZtGxPsSj72bBl2XxJjsMSD7uEbMYVkRkWzC6QuVaYaHyQ1WfLu56DZkN5XB4yW3Hl2A5wvXUMOY/IMMZjSWtqeTEHoeuB2f9fc+maNdXtSKxV1Dns+RoekxLhrRAuOjpk79Gr+6TZjsuie0E51nnGWyKZpbmjU2anLfLQaDAVcJNEAJ7vCNz3TnbcWLxV1P3qRauLND4tn++1l75dlC+9yJphUklpguUZANyQbGsxJ828+Sx7N4HUuePYsVaB98Lhw1IynAj08cVx6izdQ50MVYYsrkxOHNtTAImAphZ7L9u7pRmxr6e1ut3VLseAX5L9zs/T0btI1EWYiS1XkO/GCfaMpMC0HonI59Kr2LJx5XsrwUye3T3SipACF4Lnf/Ob6vYYPDPc1OLAfUap+hMtrz7nycezzz5L7373u6vl1/01NmzYQPfccw99/vOfp0wmQ5/5zGcomUzSO97xDnr44YfnReNDURRFUZRTnzlPPt71rnfVzPg5juPQ1772Nfra1772lhqmKIqiKMrpieZ2URRFURSloSx4qO1cKRalTa3MbGyYItiAxLHhAVxgk+U2PQe0z8Ngj+TpuWNN0u7sMbnuCvinVHi4EqYzhn0dJodcAulzbh8tlWVbHYM55K2NugK+GmRsW7k0NBERmMXJLVsb6LED0p7+4q92Vbf37Nkj6qampD2yl4VR910MstKM0qLzRDlQOibK7Z32/hWGpH02MGGvMxTAUGi5aucm7L3sWiPDaR3TWt1O70uKuumUDO8t563PQ3Fa+tb4LATSB3l3l42fIPgPlVC6n0vF4+IjKxv03wEfEJ/ZzPNh2T9RJlUfcWV7yp0ypG+S+T8cm5Ih1MG4DJmtxzGmAeSFpb9M12I7Xv7riZ+LuosvukiUYy3WLt7ZIe3i3AUlCP4XzRAuyp8FDCvnq77BEITgB7gvgnwuR0alb0TfUjvWeLoEIqLRMSuPj89hOCA/GD1ifawy4B/iNll/hyiEFycnpbT30aPWN6tYkP5nxHw+AuAPVysKuZxmgvt84Oo5+hfwagd9oVjKeExLP50BH7iYfRYXLZbvmxXLbBh3OS/H3Qiku3BY/41OyHDiaLPtnyD4TfBrRo+XBNz3eJf1SQk3tYq6CZbeYu9emSutUJDP3vi49X3C++UyWYTj3QNOPWvHm0VXPhRFURRFaSg6+VAURVEUpaHo5ENRFEVRlIZyyvl85PPSvuVySxrEfPugU0COtd86QbBvMTsimFUpCHb6cMTaekMol82ktPOQeplL1BqwxaFkr890NfLgDxJpsjZrtFWmJqRvhMN8PlBPIBK1cr/FJmn3zoBs/JEDu6vbv/rlU6Lu6OEj1e2mmEwbveIc6bvRu+QsVpKp3zlLrv6YKCef+7Eo93TY9hVyIB/u2GtG1YEKTLcLHtN+mQC7cyVpj5OCsTQl7eK+b23ETkX6fHg+s1GD2025wmzxnvSx8Avgz0PMV8GFK+O2ZairgIOI8ZlfEsi0F0tMUtmR11GCtAc+05TJgT9RqTDzPUAe2/pwdbsAKRHexcL6KSuVkyknNQyamUR2tAkl1G37vCD4CaSTopxnqQ0CICtdYNeZzcrnOxq1/Rxtkn2XLYJdftqeMwRjsplJYLeCBHdTELQYmI7N6NRRUdfB9jV58AOCd4pfsucMOPIcXFK+5h0L6QqojqIC9xuo9fEAXyz+PkbdJZ5KwJejK9wMEuYhlooiL8dWlok95bPyeU6m5DslFLb33ZBMJeCydyzB/QmQvT/N4B84Niq1g+Jx65OyfKX0Z+rN2nO8uPs5UecbeQ9c9pvkwTvFdVk6iQr4JJ5gXQ9EVz4URVEURWkoOvlQFEVRFKWhnHJmFwxfdZn0uY+hQ7Dg67Aw2BAsp7oss6QLoV0RMLs0s6XQSi4p6jJTNgwLQ9YCbDk1PS2X9YYOHxDlEAt5jLXLEMdMxi7xo7kmOSaX8kzBtgczUgaZXHY4LNdLMcRwcsyG5oWC8jhnn39Jdbtz0TJRF0nIthu+DFia2exy9OlnZVvHhkR5fNIufQby8hwVz4YYNkHoWwSGfICFbmdeBYn7oC1H8nIs+VkZYpirWBNAXbMLZOgs5pjEPWZPNvKcrmdNYwbCYF2Pn0NU1Zhhgsb2QbEszQF5tjSdLUKoLaQZKJMtO3hdLJux7I1aeDhpCJ61V/daqego2G9So9LMwLMH9/bKLLI5Jife3iFlthct6hFlHh5ZhPfNksHB6jaaA3g6heaYNGOmsnKpPsXCYjEEPsFk46Pwhi6ALPorh6yZdTIpQ0AjzTaUs1CQz3NqWt736QwzJ0Gm7DILzQ7A/6s5yODszpyQXLxTUGkd5ei56aAmoyp7P8cg83JPb58o+8zUnsnKMNxxJn2eS8t+Xdw/IMotLCP49FRS1PH7FQEbWjLHMlp7ckwUyvK6Dg3Z93pPj2zP2SvOrm6nUvIdv3uPDAuusMzhPphW6uWmRfl7bgqrK43/JtGVD0VRFEVRGopOPhRFURRFaSg6+VAURVEUpaGccj4f5SkpU5xPW78B15NzqUA0LsrBllZb50lbYdCzPg+YTr6YkX4Vwylrax4+sFfUjRy2csdBsFU6zGaeYaF2RETpKen/0NFj7dC9ywZF3YEXd1a325qlRT0IKeTHR20YbAX8OHyWtjkYlmFgrR3SZn7W2TZkdvHSFaIuErNS2iVH+oMUjbwnoHg/I5O/+JEoNzfJL6YD1n7ruSCFzOL98g6E4vlg1+RhpyBvzrNahx1Zl0xLn48xZrdPZyGeloVml0oomW7Pb6BvPAjb85kPSigs7cdBHh4JoZKOK/15Qj7zNwjL/phmMtIZOM7U/ldE+bn91t9gxJfjjg+1i6k+UeZvFY7K45Rz9pqLZflcZnLS3yDKUpvHWsC+XrTXjP4FNfLqzHdjfEKOrUjEnqMHUpuHmWx7AHyNKCmva3SU+YbBmOju7rQFMLW/cuhVUY612raHwjLMNNxsn8tly1aKun37pER3nvkBOSAfEGd9GfTAjw4e6H2yu+S+THI+AOcIQPizw/QOsC4Us74sXd1SMt0JgodRmY2fQhrqbL/3dLSKqnhMPjPZDPODMbKuWLb3Pe9LX5rJjD2H68l346KuRaKci9j2JcF/xw3ZMbls8GxRt3e/vJcu67tgSD4zoYq9z+UshEkD/DlReXVFURRFUU55dPKhKIqiKEpDOeXMLq/sfFyUyxkb4uh5kH0WVDubO20cWKR9sahr4SFbvlyOGjkkTSupUbv0WZqGMCcWXhsEqVSPqYaWS3J5DhPX5lj4lmvksmwpbUOtSmV5CztgGXKMhQM6IWk6aG23ppX+5XIp79zzpcJed6/tu2xOmhXSLITNQT1LWJYNzDJkqwxmDbe5U5QdtkwchBBQCtj+KqFqH0bthWw/l8HMEK7YnQuODMMtevK6ptLsnM1yWdZjIXVhA2G47BF0Yak+DWq+bYvsMm2stV3UHTl4qLrdA2a6ETADDbGMprEWmX12zLPt25uV4278pZdEORmxZs0jU9I0KcwOS2fOXkxE5PJsp5ilmQ2nEqiEFosQGs2UHkslec2+P7N6o1DwJLncXMZM0AxUiCwxddh8XratVJT78pDmHCilXn75pdXt4aGDoq6nVyoId3RZ8yxm9faC9v3X279UNt6TY7S9x75HszkZkjowYJ/9IJi2h48eEuV9k9IszuGh0JjU24UH02U2Tw9kAGIxO+6aIXtyOCJNT6Wi7dsQmB9b4rZ/wkHZoJFxaQYvsuzpxYJsa5tj70k4Kn9zIlFrIlq0WGb8Pffsc0W5iWVLz8GY5DK4bb3S3LfsnFWivPuAfU7L8A7hCa7RlFKvrGYXRVEURVFOeXTyoSiKoihKQ9HJh6IoiqIoDeWU8/no7JZSyNPjzM6bl6FU2WkZrpROWv8MY2RmQJdl/gwQ2NsKSVEMGmtHCwUgjSMzUrsgzUzMfhyBrLEuaGKbvG1rJCCP0xK3ds2RISk7bqLSJtzSvay63blYah+fdc4F1e2ORbIuCLbTDPOxKDswZ/VYtkqwtbtgK5ytTG/Rk74Re49AmBwLh26GcNF02e7rhWRd0JVtj8bscUbGR0VdG/vu0l4p4d7UKkOTmytWDjlH0lei1bft6YpIvxvHteF3OZBT9zzppxBiEubTUzDWmY34GGSGHY7KEL9SxNqMr/zwtaIu2Gn9gFZNSL+bYcgKOp6x5yzAUOcZMqd2v0j14FlKS5Chk4dy5sHHI5eXEtSGhUC6cJ+DLGw5COkBPAgfFbLS4IvAS8UavxLm0wDjPhAAqXoWNjw2LuWy29utH8PHNtwo6iIR2XZ+3Exa+mq8yvyAHt26VdSNT8iMwOedZ0PpIyEZFsz9uHIZ2efDx6TEfb2fFMOkB1wId/bBVyzg8HQX8lkz7Bw+ZGxOQMhsSxPzjyjJ/gmH7BidnJDXUYHQcZf1SQDOOZmyfiUxX/4erLrA+s4tAYmCcAT8v9jz3RaQvhqvvGzDaYdH5XuqGdIntLD+ypQhBJ5ndod38YmQUK+HrnwoiqIoitJQdPKhKIqiKEpD0cmHoiiKoigN5ZTz+WgZOE+UYyyFu0OgEQDy5gWmyVGYkPHoBSa9W8xIXxFfmgrJKdsPCiVp/+OaATX+Dqx5qHcR8eS+laxt6zD4dXjN1q+jY7nUK+kalP3T3Wc1Fto6pWS6x2LQC5D4PCvN60S+tW87oJ3BJcp99FuAFOnCrlhH3XekScrfJx15ziYmhV4C/xlj7P0JxqTNk0B7JcPs9IE2kI5m9tE0xN2HQXMixPwIQpAHPUDWdyIFmi1lJjOdA52PjCf9bkzQti8ekzoffUySOxeXdWVIp36EpUH3+6T+Q8fAsup2BHyWzqrRAWB968v/Ywyzi991HJ+P6YK1mQdAqyfIrrlUkT4WRbgHESaLHgA9Cj/P/TFkfzig78LTEIArFpWYtocBsYoCGyPTkKK9YsDvhul+5PPyBeMznSEPpMVdkNwPhayPwXRaji2uNfLS3hdnrCMietf/ta663dOzCPa17fnFrqdFXdCbvf6DYT4NFfga6gMFmYZMW7vUtQgyLY1sQY6BCvwGnLWC+VmUZF2M+XG4zoWibiqFeiX8XsNzOs3GL6RziDTZd6wBP6Qy6En5FXst6SNSMp2YtlPqmJTYJ0/6E3UyjajMFPiVsOcCZf1LRVnmj7tBLaV5QFc+FEVRFEVpKHOafGzZsoUuvfRSamlpoe7ubrr22mtpz549Yp98Pk8bN26kjo4OisVidP3119Pw8PC8NlpRFEVRlFOXOZldHn/8cdq4cSNdeumlVC6X6Utf+hL99m//Nr3wwgvVzJC33HIL/ehHP6L777+fEokEbdq0ia677jr6+c9/Pi8NzoI8tcuycAYhRKwpJqVum7rtXMsZhFBXtnxYzEmp6BxknM2zcnlahqzl09ZkU8pL2WTDMiyWCnLZs1yU5zQs02fRldl5O5ZZU0tr54Coi3ZAyCwzM0zD8rthy9bBCIS6QQZGw5a8DchKuyz0NgjLwo4n70mljlw156mcbOv4tFxedXPWlNEM2U7725k0fUouf7tgouHy71FXLi1mXHvcXBMs2cahv5gJqwTXnGuxpo0yhDAfHbPXMQxy5pHuPlHuH7T3Op2VYbBNLOvm9DG5ZFyJtIpyZw8ztYRaRF3ZZ6HiII0fBRMEL1bAlOEGZv9/zdikvZZiUY6PAEsPUIFs09G4HKO5STtGfCNNGRGWWiAmu5kKYM6pGFvOQR9E2CtzEkJbfWaONGCv8SC01GFmKR+Wvw0z9wUg23QEpMbzLOuv78vrCIVtW4tgNsxAioRt25+tbicS8r0ZYuG8GLLb3irHc2xwHc0Il9EntLtA2CcLHw1HZB/09Nt33OHhY6Lu6DFZvnS1larvaG8VdUMH7b59fdK0094ln70Ck2nPZOR994L23Y3Zyvfus6k5jo3I53L5oHx3x5vs/QpEIYNz0p6jEpHvop5+KT8xyUy7h1PytytSsP3a5sA5MvL3iqcoKBTmX159TpOPhx9+WJTvuece6u7uph07dtBv/dZv0dTUFN11111077330nve8x4iIrr77rvp3HPPpe3bt9Pll18+fy1XFEVRFOWU5C35fExNvfZffnv7aw5uO3bsoFKpROvXr6/us2rVKhoYGKBt27a94TEKhQKlUinxpyiKoijK6cubnnz4vk8333wzXXnllXTBBa8pZQ4NDVEoFKLW1laxb09PDw1BxMbrbNmyhRKJRPVvyZIlb7ifoiiKoiinB2861Hbjxo30/PPP05NPPvmWGnDrrbfS5s2bq+VUKlV3AhKE8Daf2QaNwZA5kE1mkr0Y2sWjcv2ItNmHXGnXjCZs+GoQ4kX9krXBlkEOusLKhayUx/ZBGr7MbMQdgytFXXsbC0N1IISvLK8rV2B2aOg7l/k0VCqyrZ4HdnDmq1GB9OShEGuDK31yAkHZHpTPnontaenj0dwsfROIpbhelpA24WTU2sXzOenz4RTkdS1O2HvrGnnOErOJDoENP1CE0MCYbd8kXGKWSTw3LZbp7g8HrFTyc/sPiLreSKcopwO2b7sC0u7cw0yy+3fvFXXd564W5fe/96rqdjwmQ5rL+Tqh4lDm0tYozRzAGNU6XHnFO6vbv/jFM6Lu8CH7T8urQ1KGfPuuV0S5ULC2blSKbmJhuN1g+w9G5HvD5Y8UHKc9Ye9JOCRDHKNs3LW1yXPEY3L8JqfsCm+sSfpYtLBQSS8sz+HCe6spastlTG3A/L2OjSZF3Z4X5VhL5+x9b03Ic8ZZaGtnp/SN8P3Zh2B6LKTZgH9VwAGp+pL1PzDgyxJmEvMdndLfYXRc+uAdPsL8Ot4u943G7f0amZS+EUvj0h+Dt72clg94tmjfq7FWmd6ijbWvkJXvokxKSjqUMvZ+hSF8drJgf5PaF58t6pywHD/J7CvV7UBYjrtKyY47D8Lam0GWIJ+zzzd/tuaLNzX52LRpEz300EP0xBNPUH+//SHu7e2lYrFIyWRSrH4MDw9Tb2/vGxyJKBwOUxgeMEVRFEVRTl/mZHYxxtCmTZvogQceoEcffZQGB+V/catXr6ZgMEhbmUf0nj176ODBg7RuXR0vaEVRFEVRzhjmtPKxceNGuvfee+mHP/whtbS0VP04EokERaNRSiQS9OlPf5o2b95M7e3tFI/H6bOf/SytW7du3iJdAqB0yZd7IyG5JBkFdU2xLOnLZSS3Yo+DYbAEoaUVtqTsB2XoWyBql/GDUWnmCDKTQxSWKyFClXwWYujBsmyZhYv6oNpXgcyNRZ+Ft0E4JF91c6DOL8oluTILR8SMnR4Lq/Sxr0AN1S/Pbpk2F5fLl3kI0e2Kt1a3iyVpLtnDwp2nIHQz5svrWsUUEFe0S/Maset6/mWpKrjnkNSuyfOsti1yqbMnYdvqVmRb8812CbnUIk1o+aC8J4l+O9n3huQSaZFloD1r+VmirnmxVLYN8ucCxqFhYdO+Xz+8jo+Dmj3nkCDz3f8dGUdEtOZS+U9KhYVR3/4/vyXqfvAf/ynKl66x5qWly6Ry6wu/sgqfP926XdT5qJArsq9iVmSW/ToAYeVs15qQcwi1vezSNdXt9111tairMLuPC+qrBVCdjUTY+wcUNCtMRjSVkqH8GchOm5xM2n2n5DMTCdj2oOm2v1+GpNaD96QxOGJAdZZdSg4kC6aYuaKzRyo8kyv76+hRa3aZHJRjorvHPhcHDx4WdUcgZLc1YTMNl+Cd5jF14/YOqS7MMwSXC/K9AMmURfi1C09UT6/tkJFRaX58hWUvJiLKpOz9CwelSdrn5lowG5oi2IvZPZiLeW22zGnyceeddxIR0bve9S7x+d13302f+MQniIjo61//OrmuS9dffz0VCgW66qqr6Nvf/va8NFZRFEVRlFOfOU0+amertUQiEbrjjjvojjvueNONUhRFURTl9EVzuyiKoiiK0lBOuay2JbCdhphNLdos/S+am1GKmGWjhYyYXLoZ/RYcsHNyuziGH3L/h2BQ2vDLTEa5DOf3wUju8HOANDJvTRlswD5mH3R5eLGkzOx4Th7C9CBWscIyMGIYo8sk74Ngd3bBVljKcZsjzciycy4Q5bFDMjQwc9jaaAswhTZtVo4+lZU+MBkIGRtj42kV2GujcWvnzY/IsLhXi1K3Js18ZCqQAqAjZI+7culyUZcas6GBu1+QfiVhX9pkl/db/5SKkRc9tt9mwVy6RIYJxpZKW3eahXwnyrI/vJC1Efsgv1wjgc18nwykKT2Ou4iA54eKQUhqW5vtO0j6Sxedv0qUuzrs/RofkffnPe9+R3W7CXzBnv7lDlGOxa3vjwcpG1z28FVAet1n74kKvDOy8AwfOHykuv2//vH/E3X83dDV2SrPAc+3y7Kool2e34MQ+KbFWqQvQJj5jvT2SD+OdpamAn0appJJ2XaaGe6f58J7AkO1K8w3LAfP8LGjR6vbnb39ou6882RW76NHDla3X375ZVF38YUXVbdRmwp9YrJZ6XfC4f1+DHxFWppt38Wi8h4EA3JsRdk9iEal/1nFty/LR7bKVCWZDMg0sL4rw/vOZf5MAXA6iYBMQjptrxnv13ygKx+KoiiKojQUnXwoiqIoitJQdPKhKIqiKEpDOeV8PsJNYKtk0t5OjV1K2hEjzOZWBFt3MW/trKiHQSgzzWKwMe6dKx4EwUjtMnttuQg2crCZc7n3MpyfRx2VQTcD+yAasnY89NUoFll7ymitleesMD8YDwPU/Znj0304bon7utTx+VgEirgJkte5fdfO6vbiZdLHYfGA9XEYPyptsFMpaR/9xYvW32DiJSlLHgxam+xwBtJYL5Z+FNGg9VWYAC2RcMX219JeqUuQbrL+KVvhXhqw1y7u7Kpuj4xJnZGRkh2HIUgr0GGkbTnG6ksw7kJiOMvrwGA37hfkoH9IrfLHjIyOWon5SZC5zmSsr01Xl5SCP3xU+sQcPvhKdXsYtBCKeXuci1dLeeojo9IXwGf+Vi1x6YMSqNhzxmLSLj/FJNOzOemnEAQ9ntEJ6+vz6M8eF3WxJvueaolJOzz6P/C05yWQV+cpCQaWyHHX2irb3hK379XhoVFRlwxY/xn0E3g9r9fr7Dokny8O9xswLrxTHRxrzAcP3rH5vPVZGhqSz8HAMqlxM7jMauOYktRv4uM5Bto86OPA+9YDv8PmZtuXIU9eV6zZ9mtLVN7LKKTx4P6CgaAc2ykmzT6dlpotqEtlmH+eD7pCPtMoKYF2UrkM6UCET9MchHtmia58KIqiKIrSUHTyoSiKoihKQzn1zC4gCctDh9B0UIDlKB6Wi6GkPLy2AsvfAdiXL7vBCrtY1qqAaYebLmoyhEKZy0pXwOTAlyFR+M3z5FKekDSG+EfDltVMBU1NEEbIr0vuKSTUa0J/MQvxLOXVF/fIDJT7j0r54zzry1FY4vbH7dJ9AbLYll3ZP0m2nDoCfZCbtEudSV/O01eukPLv8bBdeh0flkv+fKW6EzKqmgpfNpfjdXJSHmdsbKS6fQRCSceZ2TCclcfJsnBeIqKmjmR1uwOyr4bCdsk/AMvLCDdPGkLT5OzhWawrvlwmjkTt/Vr99otE3csvvyLKvV0242x3lwwJnZywfWlgBIdC0pSbZ+bIkCfDI89ZZcN7r7nmA6LuPx6ycu+/ef4FUZebkuaIDDP/Xbhqhag7l5VRlr0A6QK4ZMDI8Iio27vHmpOSky+KutVrzpXHKdjjxOPSvBX17H0+wkKEiYiWnyXNHHURZjoIyYcyf+eWIH2Ca2z4ajI5KepeeUWGq5+z0oa29y6RIcQR9lsScGU/+yHZz/x9XXHl6G5hof0tTXK88F1DAXmNXgD/9+dZxuW7KJ2zYa9FyPKby0kzjOfw51Ieh/8GFcGsm5yScgJFMdbU7KIoiqIoyimOTj4URVEURWkoOvlQFEVRFKWhnHI+HwGwk3F/AwyPKkLoWZnZu8BsRzxS0EDKZAJ7oAgxhAPx1PRFlFBnPikGpZAJ/THYNob6suvEa8bQ3zzzBaiRTGf+GQ7YwdEHhNv3HZiz+sLnQ/Y5tr0M4V0zkZ6Utty9e/eIss8cKY5OSp+GobS1Xfog1+2AbHG6bI9zxIewYGMfjyJIcr/IZJuJiIJMij1dkFLMpYr1wQgG5VjioaVTSXkdpbIcP49s/T+2PVNy31jC2p0jEB5KkBZ+gtnJR8Zkf/C07DEmDU1E5AVBapyHThIwBzlmPmbTEEZY8e05PThkHNpXKvMQQzl+gyy1eCopx3qipUuU25kU+WEI1f7YjW+vbq9bt1rUtbTYUMqJyfeIunRW2td/+awNFX/fe39L1A0str4JWbDLb39WSsHzcPAMSIAnJ/61uv3qK6+IukgEUz/YsTaw4hxRd9bAour22Kj0Kwmg5n2NR5iFp5DwXHl+14HjGPYs+tKni1hIaC4rfWn27pXh8gE2Djvi0mfICdlRGwAft0hAtsfxmNQAPAfRsC1H4LciUOddXfvuZhIKFfiBCtgxGYR32gRIuocc29aCA+9bdpnlMvoZ4nudpU+YRVLZuaIrH4qiKIqiNBSdfCiKoiiK0lBOObNLCDPFujzbKqjkwXe58huGwYrQ0prskBCuxJanvAB2IQuXQsVKsSwsl/grEAbLz4jLcyG21OnBsicqnpZYCF3Nsp/YBnMWZsdly25BRy4tGnbOAigwIuVisW796zz205+K8uiIVDKMMjNDGEwZwaDtE6dZLlFOg7JsjoXQjhRkXYDd20BIjrs8KKU67LqdoOzLkSEbnvjLnXLZfPfu3dXtQhGUCtNy3O1nCqyDfVIBdjFTsEwkWkVdKAGhkyycNpeX92uchSmj2bIZzBzhsO0TF5ab5xKZZ3joIER8V9g9CYEq5qJueW8PHrHKnEeOyPHSFLNmsUBIXkdXzyJRPnrELmO3xWUY7vJl1iQyPnJU1C3qaa1uL+nrFHV4XWsvsqaNEGTO5Uv10bDsyJgn3xu5rDVTLW6X4cW3/r//T3X7ySe3yeO0yJDQnh47nnbtfE7UNUds45evWCnqimVUeC7QbDBmZkXT18pcBkC+i4p5GwLvePBcwnleYc9MD4S5x862YdMuSDigaYUru6JFMcjCodG0zd+5+PvkwEPCzYYlfE/lbL+Wa0zicl9uupzMp0QdP2cY+g5lEXi4L4b+zge68qEoiqIoSkPRyYeiKIqiKA1FJx+KoiiKojSUU87nA23JkbC11aFPA8qtFwu2jL4HPPTVB58PjDIqczseyJm7hoUfog2NhZn6FWlPB5cPYRNFH4Iw80VAWWKcTvI+KIM0c4jHLkK/1oT3EpdGntleyzNOEtXaETH8eCYK4IuwuF9m5WyOWZu1qcwsOV00skMCKNXM/EVcyCQpElTCmPBckI1nWUt98Cc6+Oor1e1KBW259jqXDS4TdV2d0m9gCavv65Dy7h0s2zOmIAg1S7+FpogtB8C2XWa+PpkchgzL5ykSsfeA+38QEQVDdVIWA9lpGxpdysv7k06ye1mU7Yk2QfbeDusH1L9EyvP7LGy6VJD+Os/tfEaUJydsKPLV75chs0OHDlS3U+NS4p73B76LEP7eCsI94M+XDy+GlWfJbMoh9l30cWtmIZkDA/L52cN8jYiIelg6gyMsOzAR0fDRQ9XtRFyGZoebIKy7DvK6MCQXs9yynybYt5K3Pg1l+F4zjEPDnvfpaXnfS+xlWYb75YDPW4C/Z6GjufQAysZz8F2IcOnzXE6+QzIZ23Z8h+K7Os/8/LhsPpGUUCh7s5M9IHqDTObzgK58KIqiKIrSUHTyoSiKoihKQ9HJh6IoiqIoDeWU8/nAlO0Os4k6QXk5GHNdL728z48LMc1ox+O+HGWwvwm5YUwnz3wTKmA/NzgPZOfEfbn+ggty8xgDzm3GmPa8wmLp0R5pQNuEp1dH23KJtQfl1Ysgp16jrzIDq86XEs9F9CWp2ONks6ADwJWZobWxqPSHIMPj9+X4CXItAtAaqBBcF5ODNg4ahdkYBa2KfpbmOxaT9vREIiHKHczPIx6OirqmgLV1o/YLapTwdOUG9TjqSKaXwfbObcvFkryv4YjUkahHnukSoEQ41x0ZmZTaHUeGpax0scQ0DMKy9cWCHT/HDu6TDShKLYTuVtv2IwekrP8D9+9n55C+Gk3M7wbvgReW/RFpsvcviPeL60aADLkPstvcpwvt++GIPUdbu/QfKsM7JRq1+65YsVzUpZP2HpTy8v60tkoNGaLZPd+17xv0+eA+ZqC/w95NlWxS1GVqZMBt3w6Pjomalw8drm4v6ukWdYkYaNrwtuE5uERUzXXZbdetr20ifWLqaW7Abxe8m/hvJPaGSIkA50c/pXjc+lCh/+R8oCsfiqIoiqI0lDlNPu6880666KKLKB6PUzwep3Xr1tGPf/zjan0+n6eNGzdSR0cHxWIxuv7662l4eLjOERVFURRFOdOYk9mlv7+fbr/9dlq5ciUZY+i73/0ufehDH6KdO3fS+eefT7fccgv96Ec/ovvvv58SiQRt2rSJrrvuOvr5z38+bw2uwLK+WA730VxSEz9qN3F5jMuSg0Q5hrrytaxKBUJ2eTZGlAxm58Ssm5h5lMv54vJYkZlvUBYd55OOx5YvaxIT2g9QQh53rrC2FyFkl68momw8hp36Yvlu5uGXzmbkB5hpmPWf8aHv2BJ3E4adYggkM2+5IDccYGaYAMjoB2BMhJg5J9wsl9hbmDmlvU1KYPOlzRCaRwIyvK2ZXVdzVIbPRlx7fg8zLYNpjpvqcEhUeMh5CUxvvmxPxbP7opnBFFDoemYSra3VbW4qIJL9XnLlGEhBOPZL+1+ubmMkZ6lgr2VqLCnqBhbJsOUWlhUYs197ATv2czBGeYbVmucbUgAY9m7CEHhuHi3mMTQbw8rt84VL4yJcH8ZSOCyfiyuuuLK6ffbKFaJucX8/a5scS729Uub/V4dltueZqHk3A9zMEIvJtra22HKpgGZeaXqanrD//O7fLc85yjL0nnfeeaJu+bJBUe5i5qV4kzTJlNn7B7OB89cNptvA+5Wasua/yeSUqBtnYd35rDQTYvbrEksjUtvL7BN4+DFFAjfZHy9M+M0wp8nHNddcI8q33XYb3XnnnbR9+3bq7++nu+66i+699156z3tei42/++676dxzz6Xt27fT5ZdfPn+tVhRFURTllOVN+3xUKhW67777KJPJ0Lp162jHjh1UKpVo/fr11X1WrVpFAwMDtG3bthmPUygUKJVKiT9FURRFUU5f5jz5+PWvf02xWIzC4TDddNNN9MADD9B5551HQ0NDFAqFqJUtoRK9ppw3NDT0xgcjoi1btlAikaj+LVmyZM4XoSiKoijKqcOcQ23POecc2rVrF01NTdG//du/0YYNG+jxxx9/0w249dZbafPmzdVyKpWqOwGpQEify2zNJVM/fMtntjC0t/ncHlcjnyvhYahox6vwL9dIpps33Caq9SkIM5s+2kddZr/FMD20vfPWcynv1xrBr0NW+RDOVWH9ZWqkkXn6a+hXLIvvzjz8HEfaH4MR6Q8RCTF5dbjvORZqFgzJc4Sgf7i0NfobEOv3JvCxiIJkeZDJOkci0kadaLY+H81ReY4Q80mpMYPD+PGYf0rIlWGeXFI9AH4tAQ/+x+D2/zopwJEShLnz0M4gpBnw5iCv3r14wJ4Dnstu5uuzuCh9Ed4G/kTHDtsU96nJpKjLZWyIaBnSwGO3S5l0WesF2DODdnBTx4cK+pX71hQK8jr4M4zPZTY7sy9NJiN9UIrsfpXgmsfHZdgpX3FOpaZFXQvzS+rokrL1S1fIkHh6dnY+HzjOasNO7TaGcRfKdt+mmJR3jzpy3E1M2GvJj74q6opT9p/iHeOHRN2LO2SY++L+ZdXt8y64WNTxd5qBUGP+LGLwBQ8jJyJKp5mEekHeg4lxG1aeTU6IukJZvtcrZO+7B883d52r1IQlo0+KfWacN/AeeavMefIRCoVoxYrXXgKrV6+mZ555hr75zW/SRz7yESoWi5RMJsXqx/DwcI1TEiccDtc4PymKoiiKcvrylnU+fN+nQqFAq1evpmAwSFu3bq3W7dmzhw4ePEjr1q17q6dRFEVRFOU0YU4rH7feeitdffXVNDAwQNPT03TvvffSz372M/rJT35CiUSCPv3pT9PmzZupvb2d4vE4ffazn6V169ZppIuiKIqiKFXmNPkYGRmhj3/843Ts2DFKJBJ00UUX0U9+8hN63/veR0REX//618l1Xbr++uupUCjQVVddRd/+9rfntcGoz8F9N45nRywxG3EJ7IjcRuuCsR19CngM+vFkyTkOa5+DAgLg80FMc8ILSjumx44TApMVpj7m8iEexPpXmPR5qSjbUwSV5CIzFqI9m/t1oHy6AX2Oev3DibXERdmD/mlivhMuXFcTa08A+iMEfRllMtdhkMDmwwn7Dk2F3E8gAtLiTexeNoNfSa2PDj8/+G6whUrPqSfJDVovkB5AxPNjCgJWxuO4cByHvz4wBUGNpszMeE2t9jAoY8/KbpMcE3HQoulsZf4IoAfEfUlAFkacg4iErw2OdS71jf3D3wX4XgjC+OXnxPNzvRf096ppK7PF4ztN3BMYLyXoH/5cRkHHIthk/SqaW1pFXaxGXn1mjqftAXtXt8olec1jY9YXIRyW19wUlSkKKuK78N5y7JhITUi/icnxEVEeGrK+LHv2PC/qeN8FjHTS4UMkn5d6HCiHz31HcNwZw+5XzfsXxijX8amrz4G/c6hLNYeH+E0wp8nHXXfdVbc+EonQHXfcQXfcccdbapSiKIqiKKcvmttFURRFUZSGcspltcVQzhKTgMZlPVxGKjNZch52S0RkWLkma6CpE04LiGyMIGtdR9m2ZpmNL9FVYFmNL9WjyQGPzLO64qonz/qLoVSYEZgvLeI94KG3Fchi65dnlo6uB5pHUHqcm5vwPodZ/3CpbCKZvROPi3LmXKYc+yMclu0JBusch5XRzMLPj9eBBNnSeQCkkPl3j7e87Qrz38z/f+Bxaso8qhyXbOdgdwl49l76Ro6XAL9muCwHxxILg3WC8pmJsO+WjxOSz/unXugxRh+WmEw6HtMLosnK4oMdyGNhy2idrfhoxpz5vvNwdVx9x325GdGDlASG3R8XQqrdgCzPluOPF5Y5vAzvNNYF+ZwML05NyVDkIDN5oqk0yO+zgXQJATynbU9yQoblEjNPeChDwLOu480EuMx+CVIb8Pc4Sh2EQIaAv+NQXsFx+HFnv/YwN5PZ7NCVD0VRFEVRGopOPhRFURRFaSg6+VAURVEUpaGccj4fHzindPydZoTPtZpm3OvkIH38XRYM9DPh5dnLatfjvEVL5+U4hJGJaRw/M4+n8ow1BCLKylvhW/c8uNBNUBrA//2BDy90E5STCF35UBRFURSloejkQ1EURVGUhqKTD0VRFEVRGopOPhRFURRFaSg6+VAURVEUpaGcdNEuryvdYdIdRVEURVFOXl7/3Z6NwrFj5qKD3AAOHz5MS5YsWehmKIqiKIryJjh06BD19/fX3eekm3z4vk9Hjx4lYwwNDAzQoUOHKB6PH/+LZxipVIqWLFmi/TMD2j/10f6pj/ZPfbR/ZuZM7htjDE1PT1NfX1/9vEh0EppdXNel/v5+SqVSREQUj8fPuBs4F7R/6qP9Ux/tn/po/9RH+2dmztS+SSQSs9pPHU4VRVEURWkoOvlQFEVRFKWhnLSTj3A4TH/2Z39G4XB4oZtyUqL9Ux/tn/po/9RH+6c+2j8zo30zO046h1NFURRFUU5vTtqVD0VRFEVRTk908qEoiqIoSkPRyYeiKIqiKA1FJx+KoiiKojQUnXwoiqIoitJQTtrJxx133EHLli2jSCRCa9eupaeffnqhm9RwtmzZQpdeeim1tLRQd3c3XXvttbRnzx6xTz6fp40bN1JHRwfFYjG6/vrraXh4eIFavLDcfvvt5DgO3XzzzdXPzvT+OXLkCP3+7/8+dXR0UDQapQsvvJCeffbZar0xhr761a/SokWLKBqN0vr162nfvn0L2OLGUalU6Ctf+QoNDg5SNBql5cuX01/8xV+IpFhnUv888cQTdM0111BfXx85jkMPPvigqJ9NX0xMTNCNN95I8XicWltb6dOf/jSl0+kGXsWJo17/lEol+sIXvkAXXnghNTc3U19fH3384x+no0ePimOczv0zZ8xJyH333WdCoZD5x3/8R/Ob3/zG/MEf/IFpbW01w8PDC920hnLVVVeZu+++2zz//PNm165d5nd+53fMwMCASafT1X1uuukms2TJErN161bz7LPPmssvv9xcccUVC9jqheHpp582y5YtMxdddJH53Oc+V/38TO6fiYkJs3TpUvOJT3zCPPXUU+bll182P/nJT8z+/fur+9x+++0mkUiYBx980Dz33HPmgx/8oBkcHDS5XG4BW94YbrvtNtPR0WEeeughc+DAAXP//febWCxmvvnNb1b3OZP65z//8z/Nl7/8ZfODH/zAEJF54IEHRP1s+uL973+/edvb3ma2b99u/uu//susWLHC3HDDDQ2+khNDvf5JJpNm/fr15vvf/77ZvXu32bZtm7nsssvM6tWrxTFO5/6ZKyfl5OOyyy4zGzdurJYrlYrp6+szW7ZsWcBWLTwjIyOGiMzjjz9ujHltwAeDQXP//fdX93nxxRcNEZlt27YtVDMbzvT0tFm5cqV55JFHzDvf+c7q5ONM758vfOEL5h3veMeM9b7vm97eXvM3f/M31c+SyaQJh8PmX/7lXxrRxAXlAx/4gPnUpz4lPrvuuuvMjTfeaIw5s/sHf1xn0xcvvPCCISLzzDPPVPf58Y9/bBzHMUeOHGlY2xvBG03OkKefftoQkXn11VeNMWdW/8yGk87sUiwWaceOHbR+/frqZ67r0vr162nbtm0L2LKFZ2pqioiI2tvbiYhox44dVCqVRF+tWrWKBgYGzqi+2rhxI33gAx8Q/UCk/fPv//7vtGbNGvrd3/1d6u7upksuuYT+4R/+oVp/4MABGhoaEv2TSCRo7dq1Z0T/XHHFFbR161bau3cvERE999xz9OSTT9LVV19NRNo/nNn0xbZt26i1tZXWrFlT3Wf9+vXkui499dRTDW/zQjM1NUWO41BraysRaf8gJ11W27GxMapUKtTT0yM+7+npod27dy9QqxYe3/fp5ptvpiuvvJIuuOACIiIaGhqiUChUHdyv09PTQ0NDQwvQysZz33330S9/+Ut65plnaurO9P55+eWX6c4776TNmzfTl770JXrmmWfoj//4jykUCtGGDRuqffBGz9qZ0D9f/OIXKZVK0apVqygQCFClUqHbbruNbrzxRiKiM75/OLPpi6GhIeru7hb1nudRe3v7Gddf+XyevvCFL9ANN9xQzWyr/SM56SYfyhuzceNGev755+nJJ59c6KacNBw6dIg+97nP0SOPPEKRSGShm3PS4fs+rVmzhv7qr/6KiIguueQSev755+k73/kObdiwYYFbt/D867/+K33ve9+je++9l84//3zatWsX3XzzzdTX16f9o7xpSqUS/d7v/R4ZY+jOO+9c6OactJx0ZpfOzk4KBAI1EQnDw8PU29u7QK1aWDZt2kQPPfQQPfbYY9Tf31/9vLe3l4rFIiWTSbH/mdJXO3bsoJGREXr7299OnueR53n0+OOP07e+9S3yPI96enrO6P5ZtGgRnXfeeeKzc889lw4ePEhEVO2DM/VZ+5M/+RP64he/SB/96EfpwgsvpI997GN0yy230JYtW4hI+4czm77o7e2lkZERUV8ul2liYuKM6a/XJx6vvvoqPfLII9VVDyLtH+Skm3yEQiFavXo1bd26tfqZ7/u0detWWrdu3QK2rPEYY2jTpk30wAMP0KOPPkqDg4OifvXq1RQMBkVf7dmzhw4ePHhG9NV73/te+vWvf027du2q/q1Zs4ZuvPHG6vaZ3D9XXnllTWj23r17aenSpURENDg4SL29vaJ/UqkUPfXUU2dE/2SzWXJd+QoMBALk+z4Raf9wZtMX69ato2QySTt27Kju8+ijj5Lv+7R27dqGt7nRvD7x2LdvH/30pz+ljo4OUX+m908NC+3x+kbcd999JhwOm3vuuce88MIL5jOf+YxpbW01Q0NDC920hvKHf/iHJpFImJ/97Gfm2LFj1b9sNlvd56abbjIDAwPm0UcfNc8++6xZt26dWbdu3QK2emHh0S7GnNn98/TTTxvP88xtt91m9u3bZ773ve+ZpqYm88///M/VfW6//XbT2tpqfvjDH5pf/epX5kMf+tBpG0qKbNiwwSxevLgaavuDH/zAdHZ2ms9//vPVfc6k/pmenjY7d+40O3fuNERk/vZv/9bs3LmzGq0xm754//vfby655BLz1FNPmSeffNKsXLnytAklrdc/xWLRfPCDHzT9/f1m165d4n1dKBSqxzid+2eunJSTD2OM+bu/+zszMDBgQqGQueyyy8z27dsXukkNh4je8O/uu++u7pPL5cwf/dEfmba2NtPU1GQ+/OEPm2PHji1coxcYnHyc6f3zH//xH+aCCy4w4XDYrFq1yvz93/+9qPd933zlK18xPT09JhwOm/e+971mz549C9TaxpJKpcznPvc5MzAwYCKRiDnrrLPMl7/8ZfFjcSb1z2OPPfaG75sNGzYYY2bXF+Pj4+aGG24wsVjMxONx88lPftJMT08vwNXMP/X658CBAzO+rx977LHqMU7n/pkrjjFMzk9RFEVRFOUEc9L5fCiKoiiKcnqjkw9FURRFURqKTj4URVEURWkoOvlQFEVRFKWh6ORDURRFUZSGopMPRVEURVEaik4+FEVRFEVpKDr5UBRFURSloejkQ1EURVGUhqKTD0VRFEVRGopOPhRFURRFaSj/P3l9lB3/KlANAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "plane truck truck cat  \n"
          ]
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "# functions to show an image\n",
        "\n",
        "\n",
        "def imshow(img):\n",
        "    img = img / 2 + 0.5     # unnormalize\n",
        "    npimg = img.numpy()\n",
        "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
        "    plt.show()\n",
        "\n",
        "\n",
        "# get some random training images\n",
        "dataiter = iter(trainloader)\n",
        "images, labels = next(dataiter)\n",
        "\n",
        "# show images\n",
        "imshow(torchvision.utils.make_grid(images))\n",
        "# print labels\n",
        "print(' '.join(f'{classes[labels[j]]:5s}' for j in range(batch_size)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Izn-xQMBhxsU"
      },
      "source": [
        "### 2. Define a Convolutional Neural Network\n",
        "Copy the neural network from the Neural Networks section before and modify it to\n",
        "take 3-channel images (instead of 1-channel images as it was defined).\n",
        "\n",
        "Changes made - slightly larger network so that we can prune it"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "JLa_p1vuhxsW"
      },
      "outputs": [],
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "\n",
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
        "        self.pool = nn.MaxPool2d(2, 2)\n",
        "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
        "        self.fc1 = nn.Linear(16 * 5 * 5, 250)  # 250 instead of 120\n",
        "        self.fc2 = nn.Linear(250, 200)  # 200 instead of 84\n",
        "        self.fc3 = nn.Linear(200, 10)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = self.pool(F.relu(self.conv1(x)))\n",
        "        x = self.pool(F.relu(self.conv2(x)))\n",
        "        x = torch.flatten(x, 1) # flatten all dimensions except batch\n",
        "        x = F.relu(self.fc1(x))\n",
        "        x = F.relu(self.fc2(x))\n",
        "        x = self.fc3(x)\n",
        "        return x\n",
        "\n",
        "\n",
        "net = Net()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.5 We create a class to handle pruning the network\n",
        "\n",
        "During training we monitor the mean and variance, and every few minibatches we prune the network neurons that have low variance."
      ],
      "metadata": {
        "id": "16W1rZXDipCf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# https://www.johndcook.com/blog/standard_deviation/\n",
        "\n",
        "class RunningStat:\n",
        "\n",
        "    def __init__(self):\n",
        "        self.m_n = 0\n",
        "\n",
        "    def clear(self):\n",
        "        self.m_n = 0\n",
        "\n",
        "    def push(self, x):\n",
        "        self.m_n += 1\n",
        "        x = x.detach().clone()\n",
        "        x.requires_grad = False\n",
        "        if self.m_n == 1:\n",
        "            self.m_oldM = self.m_newM = x + 0\n",
        "            self.m_oldS = x * 0\n",
        "            self.m_newS = x * 0\n",
        "        else:\n",
        "            self.m_newM = self.m_oldM + (x - self.m_oldM) / self.m_n\n",
        "            self.m_newS = self.m_oldS + (x - self.m_oldM) * (x - self.m_newM)\n",
        "            self.m_oldM = self.m_newM\n",
        "            self.m_oldS = self.m_newS\n",
        "\n",
        "    def num_data_values(self):\n",
        "        return self.m_n\n",
        "\n",
        "    def mean(self):\n",
        "        return self.m_newM if self.m_n > 0 else 0.0\n",
        "\n",
        "    def variance(self):\n",
        "        return self.m_newS / (self.m_n - 1) if self.m_n > 1 else 0.0\n",
        "\n",
        "    def standard_deviation(self):\n",
        "        return self.variance() ** 0.5\n",
        "\n",
        "    def reshape(self, newshape):\n",
        "      self.m_oldS = self.m_oldS[newshape]\n",
        "      self.m_oldM = self.m_oldM[newshape]\n",
        "      self.m_newS = self.m_newS[newshape]\n",
        "      self.m_newM = self.m_newM[newshape]\n",
        "\n",
        "from functools import wraps\n",
        "import numpy as np\n",
        "\n",
        "class linear_pruner:\n",
        "\n",
        "    def __init__(self, layerprev, layernext, activation=lambda x: x, model_and_optimizer = None):\n",
        "        self.layerprev = layerprev\n",
        "        self.layernext = layernext\n",
        "        self.old_layerprev_forward = layerprev.forward\n",
        "        self.activation = activation\n",
        "        self.model_and_optimizer = model_and_optimizer # for momentum\n",
        "        self.stats = RunningStat()\n",
        "\n",
        "    def hook(self):\n",
        "        @wraps(self.old_layerprev_forward)\n",
        "        def new_layerprev_forward_observe(*_, **__):\n",
        "            x = self.old_layerprev_forward(*_, **__)\n",
        "            if len(x.shape) == 1:\n",
        "                self.stats.push(x)\n",
        "            elif len(x.shape) == 2:\n",
        "                for i in x:\n",
        "                    self.stats.push(i)\n",
        "            else:\n",
        "                raise NotImplementedError(f\"Cannot override forward pass for input shape {x.shape}\")\n",
        "            return x\n",
        "        self.layerprev.forward = new_layerprev_forward_observe\n",
        "\n",
        "    def unhook(self):\n",
        "        self.layerprev.forward = self.old_layerprev_forward\n",
        "\n",
        "    def test_prune(self, threshold_mean_fraction: float = 0.6, threshold_percentile: int = 0):\n",
        "        # Threshold of what needs pruning\n",
        "        assert (threshold_mean_fraction >= 0) and (100 >threshold_percentile >= 0), \"Need valid_vaues\"\n",
        "        if threshold_mean_fraction > 0:\n",
        "            threshold = torch.mean(self.stats.standard_deviation())*threshold_mean_fraction\n",
        "        else:\n",
        "            threshold = np.percentile(np.array(self.stats.standard_deviation().detach().clone().cpu()), threshold_percentile)\n",
        "        bad_locations = torch.where(self.stats.standard_deviation() < threshold)[0]\n",
        "        meanreplacement = self.activation(self.stats.mean()[bad_locations])\n",
        "        @wraps(self.old_layerprev_forward)\n",
        "        def new_layerprev_forward_test(*_, **__):\n",
        "            x = self.old_layerprev_forward(*_, **__)\n",
        "            if len(x.shape) == 1:\n",
        "               x[bad_locations] = meanreplacement\n",
        "            elif len(x.shape) == 2:\n",
        "               x[:, bad_locations] = meanreplacement\n",
        "            else:\n",
        "                raise NotImplementedError(f\"Cannot override forward pass for input shape {x.shape}\")\n",
        "            return x\n",
        "        self.layerprev.forward = new_layerprev_forward_test\n",
        "\n",
        "    def permanent_prune(self, threshold_mean_fraction: float = 0.6, threshold_percentile: int = 0):\n",
        "        # Threshold of what needs pruning\n",
        "        assert (threshold_mean_fraction >= 0) and (100 >threshold_percentile >= 0), \"Need valid_vaues\"\n",
        "        if threshold_mean_fraction > 0:\n",
        "            threshold = torch.mean(self.stats.standard_deviation())*threshold_mean_fraction\n",
        "        else:\n",
        "            threshold = np.percentile(np.array(self.stats.standard_deviation().detach().clone().cpu()), threshold_percentile)\n",
        "        # Get pruning locations\n",
        "        bad_locations = torch.where(self.stats.standard_deviation() < threshold)[0]\n",
        "        good_locations = torch.where(self.stats.standard_deviation() >= threshold)[0]\n",
        "\n",
        "        if self.model_and_optimizer:\n",
        "            state_dict = self.model_and_optimizer[1].state_dict()['state']\n",
        "        # Prune layerprev and state\n",
        "        if self.model_and_optimizer:\n",
        "            index = next(i for i, p in enumerate(self.model_and_optimizer[0].parameters()) if p is self.layerprev.weight)\n",
        "            state_dict[index][\"momentum_buffer\"].data = state_dict[index][\"momentum_buffer\"].data[good_locations, :]\n",
        "        self.layerprev.weight.data = self.layerprev.weight.data[good_locations, :]\n",
        "        if self.model_and_optimizer:\n",
        "            index = next(i for i, p in enumerate(self.model_and_optimizer[0].parameters()) if p is self.layerprev.bias)\n",
        "            state_dict[index][\"momentum_buffer\"].data = state_dict[index][\"momentum_buffer\"].data[good_locations]\n",
        "\n",
        "        self.layerprev.bias.data = self.layerprev.bias.data[good_locations]\n",
        "        self.layerprev.out_features = len(good_locations)\n",
        "\n",
        "        # Prune layerprev grad\n",
        "        if self.layerprev.weight.grad is not None:\n",
        "            self.layerprev.weight.grad.data = self.layerprev.weight.grad.data[good_locations, :]\n",
        "        if self.layerprev.bias.grad is not None:\n",
        "            self.layerprev.bias.grad.data = self.layerprev.bias.grad.data[good_locations]\n",
        "\n",
        "        # Edit layernext bias data\n",
        "        temp_bias = self.layernext.bias.data + 0\n",
        "        self.layernext.bias.data = temp_bias * 0\n",
        "        dummy = torch.zeros(self.layernext.weight.data.shape[1], device=self.layernext.weight.data.device)\n",
        "        dummy[bad_locations] = self.stats.mean()[bad_locations]\n",
        "        dummy = self.activation(dummy)\n",
        "        bias_edit = self.layernext.forward(dummy)\n",
        "        self.layernext.bias.data = temp_bias + bias_edit\n",
        "\n",
        "        # Prune layernext weight\n",
        "        if self.model_and_optimizer:\n",
        "            index = next(i for i, p in enumerate(self.model_and_optimizer[0].parameters()) if p is self.layernext.weight)\n",
        "            state_dict[index][\"momentum_buffer\"].data = state_dict[index][\"momentum_buffer\"].data[:, good_locations]\n",
        "        self.layernext.weight.data = self.layernext.weight.data[:, good_locations]\n",
        "        self.layernext.in_features = len(good_locations)\n",
        "        # Prune layernext grad\n",
        "        if self.layernext.weight.grad is not None:\n",
        "            self.layernext.weight.grad.data = self.layernext.weight.grad.data[:, good_locations]\n",
        "        # reshape stats\n",
        "        self.stats.reshape(good_locations)\n",
        "        # Remake parameters\n",
        "        self.layerprev.weight = nn.Parameter(self.layerprev.weight)\n",
        "        self.layerprev.bias = nn.Parameter(self.layerprev.bias)\n",
        "        self.layernext.weight = nn.Parameter(self.layernext.weight)\n",
        "\n",
        "        #Force update optimizer params\n",
        "        if self.model_and_optimizer:\n",
        "            new_state = dict(zip(self.model_and_optimizer[0].parameters(), self.model_and_optimizer[1].state.values()))\n",
        "            self.model_and_optimizer[1].state.clear()\n",
        "            self.model_and_optimizer[1].state.update(new_state)\n",
        "            self.model_and_optimizer[1].param_groups[0]['params'] = list(self.model_and_optimizer[0].parameters())"
      ],
      "metadata": {
        "id": "xn-GOsW3imZm"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Z1U2utyhxsX"
      },
      "source": [
        "### 3. Define a Loss function and optimizer\n",
        "Let's use a Classification Cross-Entropy loss and SGD with momentum.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "dQ1CHLgkhxsX"
      },
      "outputs": [],
      "source": [
        "import torch.optim as optim\n",
        "\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3.5. Hook into the network and define the layers we want to use.\n",
        "\n",
        "Using the custom linear_pruner class"
      ],
      "metadata": {
        "id": "blrklyuokDfS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pruner1 = linear_pruner(net.fc1, net.fc2, F.relu, (net, optimizer))\n",
        "pruner2 = linear_pruner(net.fc2, net.fc3, F.relu, (net, optimizer))\n",
        "pruner1.hook()\n",
        "pruner2.hook()"
      ],
      "metadata": {
        "id": "bsDMmUU1kAxo"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zeF-Jzr6hxsY"
      },
      "source": [
        "### 4. Train the network\n",
        "\n",
        "This is when things start to get very interesting.\n",
        "We simply have to loop over our data iterator, and feed the inputs to the\n",
        "network and optimize.\n",
        "And on every mini batch we prune the network\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "HT5jUTLShxsY",
        "outputId": "3e0e1c19-e433-4e75-b7d3-7348700c1d56",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1,  2000] loss: 2.169\n",
            "[1,  4000] loss: 1.773\n",
            "[1,  6000] loss: 1.632\n",
            "[1,  8000] loss: 1.528\n",
            "[1, 10000] loss: 1.491\n",
            "[1, 12000] loss: 1.433\n",
            "[2,  2000] loss: 1.343\n",
            "[2,  4000] loss: 1.337\n",
            "[2,  6000] loss: 1.291\n",
            "[2,  8000] loss: 1.284\n",
            "[2, 10000] loss: 1.236\n",
            "[2, 12000] loss: 1.226\n",
            "Finished Training\n"
          ]
        }
      ],
      "source": [
        "for epoch in range(2):  # loop over the dataset multiple times\n",
        "\n",
        "    running_loss = 0.0\n",
        "    for i, data in enumerate(trainloader, 0):\n",
        "        # get the inputs; data is a list of [inputs, labels]\n",
        "        inputs, labels = data\n",
        "\n",
        "        # zero the parameter gradients\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        # forward + backward + optimize\n",
        "        outputs = net(inputs)\n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        # print statistics\n",
        "        running_loss += loss.item()\n",
        "        if i % 2000 == 1999:    # print every 2000 mini-batches\n",
        "            print(f'[{epoch + 1}, {i + 1:5d}] loss: {running_loss / 2000:.3f}')\n",
        "            running_loss = 0.0\n",
        "            # Pruning\n",
        "            pruner1.permanent_prune()\n",
        "            pruner2.permanent_prune()\n",
        "\n",
        "# We can then unhook from the model\n",
        "pruner1.unhook()\n",
        "pruner2.unhook()\n",
        "print('Finished Training')"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "let's see what our new model looks like"
      ],
      "metadata": {
        "id": "uJT4Xg76lQqN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(net)"
      ],
      "metadata": {
        "id": "CFendmoHlVWb",
        "outputId": "19abeb59-9d76-4dd9-d5d2-9d25b6ab34d7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Net(\n",
            "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
            "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
            "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
            "  (fc1): Linear(in_features=400, out_features=181, bias=True)\n",
            "  (fc2): Linear(in_features=181, out_features=107, bias=True)\n",
            "  (fc3): Linear(in_features=107, out_features=10, bias=True)\n",
            ")\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dY4D070ChxsZ"
      },
      "source": [
        "That definitely worked - our architecture has changed\n",
        "Let's quickly save our trained model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "mCWGzHNMhxsZ"
      },
      "outputs": [],
      "source": [
        "PATH = './cifar_net.pth'\n",
        "torch.save(net.state_dict(), PATH)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pSEPOxTmhxsZ"
      },
      "source": [
        "See [here](https://pytorch.org/docs/stable/notes/serialization.html)\n",
        "for more details on saving PyTorch models.\n",
        "\n",
        "### 5. Test the network on the test data\n",
        "\n",
        "We have trained the network for 2 passes over the training dataset.\n",
        "But we need to check if the network has learnt anything at all.\n",
        "\n",
        "We will check this by predicting the class label that the neural network\n",
        "outputs, and checking it against the ground-truth. If the prediction is\n",
        "correct, we add the sample to the list of correct predictions.\n",
        "\n",
        "Okay, first step. Let us display an image from the test set to get familiar.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "mUNw9_TEhxsa",
        "outputId": "ab6f32b2-4e12-4bcf-bbfe-fe69cc02de8c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 210
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAACwCAYAAACviAzDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABPEElEQVR4nO29eXRd1Xn3/5zhzqPGK8mSbBnb2GAzeUKBNyGJWyBZJBTeNslLizP8mpXWTgNeq0lImnQ1LTW/dq1m6CJktYtA+msoCX0DaUlCSgxhSG08YDN5xvKswZJ8dXXne87Zvz9o7n6eR9ZFAvnKw/NZS2udrX11zj5777Pv0f4+g6GUUiAIgiAIglAnzNlugCAIgiAIFxfy8iEIgiAIQl2Rlw9BEARBEOqKvHwIgiAIglBX5OVDEARBEIS6Ii8fgiAIgiDUFXn5EARBEAShrsjLhyAIgiAIdUVePgRBEARBqCvy8iEIgiAIQl05ay8f999/P8ybNw+CwSCsXr0atm7derYuJQiCIAjCeYRxNnK7/OhHP4I777wTvve978Hq1avhW9/6Fjz22GOwb98+aG1trfm3nufByZMnIRaLgWEYM900QRAEQRDOAkopGB8fh46ODjDNt9nbUGeBVatWqXXr1lXLruuqjo4OtXHjxrf922PHjikAkB/5kR/5kR/5kZ/z8OfYsWNv+11vwwxTLpdhx44dcM8991R/Z5omrFmzBjZv3jzh86VSCUqlUrWs/mcj5u6774ZAIDDTzRMEQRAE4SxQKpXgm9/8JsRisbf97Iy/fAwPD4PrupBKpcjvU6kU7N27d8LnN27cCH/1V3814feBQEBePgRBEAThPGMqJhOz7u1yzz33wNjYWPXn2LFjs90kQRAEQRDOIjO+89Hc3AyWZcHg4CD5/eDgILS1tU34vOxwCIIgCMLFxYzvfPj9fli+fDls2rSp+jvP82DTpk3Q29s705cTBEEQBOE8Y8Z3PgAANmzYAGvXroUVK1bAqlWr4Fvf+hbkcjn41Kc+9a7PPXfsp6RsKK967PfR2zGYq0+5rA1bHbdC6vx+f/XY9TxSpzzFzutWj02Ltk9VIvpz4JI6n79YPbaAt5Vew/Wc6nHFoe3xPKSnGfQ8jku1thL6LFfhPNR3XKMrl2n/uK6+Du5zAAAT3WeZ9V3OIUXIl/VnI5ethclYv349KTsOPVG93bBn7Hpq8vKEKvavgUKfMCdWagw6BgYrK8Bzgp5HTcPzvlaf4PM88MADNc8z931oHrh0nEdODVSPS8UiqZt/yQJSTibi1WOfRe/L79MPqp/XsXXCNnTbXadA6qIRH7oGvX8blS22MJw+PUrK2CDP5/OROtvQf2uY9BqOVyblWt6MpqEr87k8vYZN141gMFg9LpfpNRy0boaCIVJnsPv89j/8v5O2p7NLh1mINi8idSHLT8rxWLR6PF6i62guM1I9Nk22NrKnyEYdFLLpDnvQQn3A1t8JiyWqdj130jqP1eH28D43Wd/Vep4MNCcNfs+8PTXOiVUGv8kUB0XLhl+3Lz+yh9Q9u+X1Sa85Vc7Ky8fHPvYxOHXqFHz961+HgYEBuOqqq+Cpp56aYIQqCIIgCMLFx1l5+QB46z9X/t+rIAiCIAjCrHu7CIIgCIJwcXHWdj7OFuUJGjXSZJm9QQAipGyC1rBsm+pkRDvl8p+PXrOENFHHo7qdjbR4i9mD2Og0hkdtKsApkSK2o/DYNcqG1mddi+p0Zf5ZV1/UYNqggexKgj6ue9OyaSMdvMLabujzKGbnoph4allTe9+1eOfNMmfLxgSPyQRrC6b3e7gvFTc2QnYcTL82gD4X9Epn3+bj7YiG9Rw2WdzDUk7XeWVqtxD00+tHQvpvbdY0/DwFbHrPIT+b66i/Si6dzwFbP3t+9szg4bJtOj7Y5uStzyINn41PANmf8ccll6fPHq7GdmsAAAqtdyabSz5mf4DtTioluhbhtSDEPROn8Vx4SvedYzWQuoqPrtWupW0+TB+z+Shkq8fKzZE6Zj4DJaX/tsJsJYpoHjBzEChXqH2RidajQp7aAeG1itvvYNs506Rjp7j9DhpsPpaOg9YJ9jgbBvsOQmPb0ED7ORDStkYmWyc8vm4E9L242SjMNLLzIQiCIAhCXZGXD0EQBEEQ6sp5J7soj/luKpQXhrnpGS7djvIqepvLCtH3Lrz1yXf8uSuTH22tOYpus3kV/cf87/DWmcG2pbnrpIFcz5QVJHUFV+8RDozQrbxcmZ43m9X1lqLtiQWR+yFzx4yHqUtdKKD71jPZdiGSA7hcwnZBoeJNbTueb9tPZxv/bPBurk/kCX4evIfKdrAVl1bQ/wqlCp3rNt7udelYWkattnNJZmaYTn/ZSLYzmWznt3T7fCaTQEzaB0H8WeYGWypoycZiUmXQpnO9UtJb7ibQayhH1ynm5u4iOcvvo+c0+RigZ5G7O7tIks3nqdQ0cuoUKaea9bY6d8u1/Lp9FhP1+JzACpLNzlNC66rN+rXC5mEtTKU/67K1yGXrj2vofg7GaD83zdVek+bYaVIXzWdJuVzU3w9ulK6jXiJZPY4xCQ+3FQBIhtZyia5/ODRDMMjcVbErPXsmuGyJyzwjrIP62eOPLFs3/LZeC0Ih5hoNWO6j3x0ecDdhbCcw87Kz7HwIgiAIglBX5OVDEARBEIS6Ii8fgiAIgiDUlfPO5sN2qRsYWCjkNHNfDVhMj8T+d0xTw25O3OfR4XYKSBP1+amm1jbv0upxJj1M6oZHtH7rs6krlQnMZdbRQ1NQYVK354jWfVWgidRVLOqyVkY6Z3aMhng+Maj10miQ6df9aVLubtPtbYpxzRyHXqd9zqTUCVrvZNTSQ88WdbErmdAf+prKo5UOE3cryGbowKFDpC7VpkNXeyw8dksjdbcLIhc67yzd83TGy49sOTyHtt1CurSPuUr6mGZtuvr58vuY9m7pa/iYzZLPpHPfM3S96dH1xikil132rBVRv4eZzZTF7CiIcM/GIIfCyO/Y8TKpqxSoDUhDfKVuT4Cuadg8g6dEAGaPZmJbAPaMesjOTrG/m2CDVwMHkJsn0PXPs2j7SsjeyWK2TxHkFxsPM5u7l7eRcnlY24C0L72U1Bmn9NpYMuhYRplty3hBu/QG2RdEANn9mU3UJdVErrbcbboUpjYodkWf16qw60f03AqMjdG/67qMlPPJRPXYc6jLsIvmYdCjYzDBDtFFLt/uzO9TyM6HIAiCIAh1RV4+BEEQBEGoK/LyIQiCIAhCXTnvbD64aG7YSX3MdGaHp35HcQHKTFv2I99/1+W6JrNTQNfhIZZXr/md6vGO/95M6k4iG5CcQ7vecalWeOT4UPW47/gJUhdoaK8ed6Z6aFsDMVIuI33UF22h1yxqPXRk6CSpCzdQW5LjWZ3avMhsEVIxrXmGWRhpt0I1ahzBt1aEibeL81EPG5DpXG/q9iIsFoNP66quonWFLLU3SI9p3XlwmNrvhGJas26K0TlgGjymDQq5b0wjzge3w5n6X9bEj2yxFLuGD08YZu9lAY/ro+t9QOdhBWnfLrOtseJc+0a2JCwEtueg/nKpXUk2k64eR5meb7L5gdPU2z66FqRRbI/RDH1+Qiw0fBl1QblCx9L2I3sitha6LrWXcdB6WC7TfvYjmy7Fnn3PnZoN11ugFAA8joai7XEd1LfMWMJANhZFg851n0dtN4xmbQuVH6djWenbXz12DGqj49HhgxwO8c76wF/RbS0fY7F50JjwMPpFFnfEKup6mzYVSm36ngsD9NmPGXRdNxLN1WOX242h58nH0zewOWIhWyzbnHnbMNn5EARBEAShrsjLhyAIgiAIdeW8k11KJt1mG8vrbTaXuRU1ROnWXhy529lsGxS7+E2IhMzcybBbbj5Pw/s+8+RPq8eDabp9OZjVf3fkBP27IyePkbIV1DKMa8VJXSSut9l8YSrX2EG6fRhAW+5Bk25JDpd1dsb2zm5SVyzQbJGHDmnZZTRN+9mao9swr4W2x8dCfRsoVDNzmibwLJzcDfWdovhpauwmknDHbyO7uGhL2WNbnTiTL85yCQBwaiRTPc7kaL8WSiybZ173mBmg7te5gp6/0TDb4mf3iEWGd6NezZT0FTD0fboGfdawey0Oew5whtDnHgqLzkKf2+bkIcItg2UbJfIO60vkzu8yV9/suB7Lo7ytTC7BMkhXnI4lDqH+yquvkrorLr+clD10LyWX7tUHkTzhMfmokGeys63b4zCp1LJ1+yoO7fNSiX62FljO9ti6oPj/wSi8QZlJNC5qa2KcjV1LipRDrXOrx46iLqqAws+r5jZSVfDRcbcHRnSBpZDIoTVXpahc7fP0fRWZfB+JsbAI47ovS2yO2iHk9srWCbuplZQNn+4fV1FpMIZOazEZyDGo27Jh4vLMZxmXnQ9BEARBEOqKvHwIgiAIglBX5OVDEARBEIS6ct7ZfJwqUO1ptJKsHj/3m1+TussWUU3t/ZdrF6QGi9l8ID3SZJqeaVItzEVuYcyLEfqO6LDXowWqt6lwY/XYijJ3yMYMKYeSyepxuUg1vjJyj4w30HuMR2l5aEDbamROMxctpHkGWerlo6dpaHhfXGupQ/1HSF10YLx63Ban5wkx7d1hIfAnI5cv0F+wEPc2GiPF6izbOuMxAIDBDHqwDYjpTf4ubnLHUmbvkEUaP3e7DSFXxSJLQd6PbD6GTtM54LFrVpDxRn6cpg4fQq63x0/0k7rLFs4n5UvmdVaPLRZKm7Rdsf7gJh4kfDetmtBfNbCQrZbHXbORLVZhjPYPMHsDZaJQ1iE67/xo3vn5nKhQ+yYXn9dlnyVuwdRuIpfTNgWDg7RtkTi1hVIovYOyaVvLWf23QRYm/lQ6Tcovv65tQiIB2tYF8/W428x2pZQfJ+WQreu9En32XORe7NKlEKDIxqQWaEq4Hg/hPmEC6c8yd14fshEKHDxAm7PjBVJ2ViL7HZOtxyhthZ/ZjhSBjl8UpZuwAvQ8XkS3x1DUbdut6PPGmpKkzndihJQhq59pX4p+P8Ax/VmbzaXiKWoXZCE7QG8RDb1e9Ov2mczN3u8wOxO03vDo/DOB7HwIgiAIglBX5OVDEARBEIS6ct7JLnaCbiHnR/T7U8VPI72N5uk2ZL6sI8rF/SxyIXbn4tv4FnWFK5a1tHCK+YsOj+stuHCSul01tGh31pxHtyubgWXBRO5bZR9tazGnt0yLWXqeuczVK4+klaEy3U410Jbu2ChzmWPbogW0JWj5aX8MZrTbcP8YlYjmNjMJa4rbd+kC7dhomMpJpq33f13mCk3UE7b7zzzYwES6i2HWeBd/mwirA/06Cm1jYyOpCwX1VmepSPs5HNB1bS3NpE6xxufyum8jfrq9Wy7qsbVYJ2dLLDMrarvBZDEqGfHMwkDLkxYmdFdNgkizmZBZE8kuASYRRZn7dQK5A5pjVEoJoPkc5Dv8TOIz0Rj52VY9uPqa5Qx9LmMR/dkGNgf6jg+Q8qFjurz/4CZSd3o4XT3OFuk18pU3SNkGFJk0R11Jl126qHr8kQ/fROrmsHWiFNT9U8zRvivndFvjikXTLFD5phY+C2V/Za6b3PXWQxE1bfY/cvS0bp9znEZmjjOZavykbns5mCB1CvT3gTEwROoiHcwNNo4kCKBrXAhFIvanaX8UkTu2M0zlUD8bWyejxy8wSsMrVApI7gvR78B0Hw3T4A9p2SXWPpfUWSioqjLp81TibuVobSh7M6+7yM6HIAiCIAh1RV4+BEEQBEGoK9N++Xj++efhlltugY6ODjAMA5544glSr5SCr3/969De3g6hUAjWrFkDBw4cOPPJBEEQBEG46Ji2zUcul4Mrr7wSPv3pT8Ntt902of7v/u7v4Dvf+Q784Ac/gJ6eHvja174GN954I+zevRuCweAZzjg9Lr1iFSkf37KvehxNUD1yVe9qUg5b2kW0nKPaHLYhMHzU/sJVDaQca+2qHu96lb5YRZNat58zl4ZCVkg/9jE7Dq9E3a7KZa2x4bYBAFhIi3vjlVdIXTxAPxuOaO0ywkKxnxwYrB473M6FaaeNKAR0+jR1Szs9qst9/VR37kjRsMU2s7WZDDtONWmX2WNUTKQZGyyzJg7XzWxXeHZRbGOgasRa52HZWfR3kqXUYLYJgGxSkiykcqWCrmmxsWPu2Njmw7Do+BjImCUQ4mGSWbZn5B8+wYUOux5P8Jal/YOvMvGjUzf6OHb4cPW4UqHzYzyjn1O3Qm1XTpyg2Z5Po7mfY7ZQrU3aBiMaYdlEbTpeZeQObfvpWmDa2tYmx+x3irjDFF1aj56krut9x7VrdK5M7XeCCR0u24jQAaJPMEDEr8ey/8h+UnfypH6+X3jhN6RuCXO/bklqG4NCNk3qchm9NlWWXErqsmM0TUQtAn7d74rNdfCY8Ryy5zGZbU8WZRLPrriS1MXt5aScH9fzp8LCKxgBNEZl5s4bonMkh0LX81QLFVe3x2dSW5YCGh8eoLzAXIjzWd3WCLt+EZ0nEKWzoDFGv59c9H2RZWsBoLDxoQpdUx12X7jbK9Mx4poi0375uPnmm+Hmm28+Y51SCr71rW/BX/zFX8BHP/pRAAD4l3/5F0ilUvDEE0/Axz/+8XfXWkEQBEEQzntm1Oajr68PBgYGYM2aNdXfJRIJWL16NWzevPmMf1MqlSCTyZAfQRAEQRAuXGb05WPgf6JpplI0s2AqlarWcTZu3AiJRKL609XVdcbPCYIgCIJwYTDrcT7uuece2LBhQ7WcyWRqvoCEE9QWYO587cteYJG7u3sWkHIz0tfTfYdJXQXF+XAdGsdi1Xtvpeedv6J63LOMnmfHTm2D0RCl9g4nh7Tua7MwvAEf0+aQxJZlfvfpUa3BNkbp33FlzkW2HM0t1CamhLTt4dPUVsOw6HtpDIVtty0WDhpp328eO07qWhqoZr6wk4UNnoTv/8u/0vYwmxQf0jWjMaqPLujR8VRWXkHDC7PM5iQ0Ow+LrrCGz/RQh8UWwXEd/AHaHhyvw++nthpNDShMPFOFbRbLw4/DcPuYJoxSnaczVIdPj9GxHR9LV48rPIw9irnRxMJBL1xA7QR8OCU5m3jczqQWL/z3Fv13Bov/gGx2CgX6HBweoDEe8CX5ODcktE1DJMiePdZUHwq/brNQ2qat+z3P4jTY6BqK2eQMjNJw+BUUjCYcS9IGgB5LHGodYGLY+mJR90k8RmNDXLt8WfU4N0ZTKxRZyoajR/WcefPNN0ldAYXZPjJC50shT8fEDtC1ExOJ6LXAYWNQcfk81OPusBgTBrLDCaVo7I5MjvbXqTHd7wZLm1HOo5D7LN5NOU3P4yDjqICfrrkZtIYEfewr1dRlj9mflfLczkW3b6xA1xdkUgZhm/ZHrJN+X1q42mR2Lni/YUL2BPYQo4faOwvx1Wd056Ot7a0v28HBQfL7wcHBah0nEAhAPB4nP4IgCIIgXLjM6MtHT08PtLW1waZNOmJfJpOBl156CXp7e2fyUoIgCIIgnKdMW3bJZrNw8ODBarmvrw927doFjY2N0N3dDXfddRf8zd/8DSxcuLDqatvR0QG33nrrjDTYCjB30cE91eOrlq8kdZEE3QK0xrVrnuvQLSYbbSEfOkbdcK9v6KGNCOusoLEI3Z4L2rp9IRaGPIi33NkW3JyOdlLejbY+/X66xZ5B7mM9XYtI3aLFVGYYHdXbqdF4ktSdRCGFDeYilmyg4aHH0Fa+xSSZUFiftzBO++PAUZY9E7mMpc68GfbWefJ0W7hcoGUfkiDGqaoAYVTnLllM6oqKbpWbaMs0wNwqsZTgckmGyTCJRi1pcVc8QG7CPEyxhaUVliKZb3R6aFv0MMqeDABwYkiP5egIddsuFFiW0hLa1i/Q/iihjK6dXdR2q7urk5Qjfrx8sP6ZRlbbXQf0vYRDVJZTSA4tOXRuJRqoBItdOctFKgecyur5Y7HxiQWp+7PjoqzVPjomFopPbdj07wI5vR1frlDD+dFRKnvg/uLTpezqPfbxHB27Mks70NWin9OmBvpA4Sy7o6dPkbqmJF1TVlypwwIc76cuzGMok/je43RumWzd6KFThmCjvgzF6NqYzVNZyka6mcukAxtlYzXZ8+wBLRsWcptmbcWlSpnOrRCTwW0kn/hYVmTsXus6TC4p6vFy2BPtCzHXVhS638/mnQ/JdD6HyUcsDoCBrhN0mZTiOviD9PrsFzRLxdSf56ky7ZeP7du3w/vf//5q+bf2GmvXroWHH34YvvjFL0Iul4PPfvazkE6n4frrr4ennnpqRmJ8CIIgCIJw/jPtl48bbrhhgmEexjAM+MY3vgHf+MY33lXDBEEQBEG4MJHcLoIgCIIg1JVZd7WdLr4g9YYpIne3Uon62vqYzUU4gt3tqL4fQNpg1Ka66sP/9CAp3/Kx9foaORq/xB/Q73OmSfW/nvlzqsdDo9RNsJilGnVbqw7TPpqhemSprO95/gLqTnzJAmoDMrbz5epxbpzqqtgtzWEprQvMxiKZ1C5trqJ2HIkGrY86ZXrPlkn78vhJbZuQugIm5Q9uu52US8wlNBLS48ddxELIFsFghhM8iJ3n6Dnjs6k0aKMQx4rpvAUWBlx5+pomCwWP3YJtrhf7UHp7s7ZdCQ5xXPToXI/Eta1RQzJJ6twy/WzQ0n2XHqEGM8dPHK4eL2Cu6pZJlwtsB8PtKKYTjTmD7K+UR/sujFIChCw6Pp1dl5ByBd3nKRZXaBjZwaRSraQu0ExtWXJp/VnPpBMo0aCNGgIBGta6iLo579B5FozQdcut6GfRYukB/MhN1+en86USpOVV12hbjUVzO2h7ynpN6XuT9t2b+3aTcu9K7Zbb1UXPc/RVnZaiwmwIPJc+77Xwo3vxB+lc8hR1TQ4hV3LHoNcYz+hnz2Xus8EEtVVLRZANEXMXxesGt2mw2P/lFrLHIi7vb4NC6yq3+XBZuHelsC0L/awfW6gw27AS+57B1TazMXNBzzWDPbOGR+8LZWyYYOc3E8jOhyAIgiAIdUVePgRBEARBqCvy8iEIgiAIQl0572w+DJaKOY9sJYrMLsDH0sKPjyBt1aL2ID5IV4/bk1RHPLDnACmfPK7jnECe2m4cOX64enx12ypSN2eu9sPvGKIO8bmDR0i5MZCsHseSzaTuzTf7dFs75pC6NLNpqCDNcfAU9dH3kH+4wUKm55nNh2EirRAoERR6HTwae8FvsDgFw2fO8cPxKiweBtdg0XHUT+MthIJ63AtF2h/5CtXXDx86rNvK4nx098ytHvcdo+P85FObSLli6nkZDNDQ0WHUHp4qO4Ei+iYTNMbF1VdTo5iWZm1jcEknHXcThSW3mCaMYw0A0JgFhVaqkXe0J/XxHBp7xuUpwFF4amyDAzBBlq6JD8XuaWml9gZBFBdmeJiG7s/lqO0RzgFerFAdPNGin705zJYllqC2G/FmbRMyguLkAAC4SBdnU4mEf8+zuBXlCgsfDii0t58+e8GAns8+FseilUWAbmnQ5SCLDdGC7FPiLCT4yNGjpHzkzcPV47ZGut6MDerw975GmqKhbE39K8RGa4hl0PsKsnU9PaTjooxm+0ndqX49DxpidL1ZetkyUvYh274Ssw2rIHsVk6Vv4OuNiWL3c5subDvBPUFdEpOEB9bghlH4GizdBrkGXRttdh68FvDz+LA9EV/IWXNMZE/jTiNdwlSRnQ9BEARBEOqKvHwIgiAIglBXzjvZhW9VWWgLqr2ZbsHh7W4AgGde1SHLGxy6dbWwEW+bM9c3m0oQp4YO6+aU6LZs9yU6FLvFrh+O6+3d5hR17xthWS/HkHst2+2G1la9LWwzaanIXF3LaPu5wLbfHXRih12kWKLboo6j31ObmqmromHovvMbtK8CzE3OVZNnvcQ88Z//RcpehbqLmiiMcpS5VMfQ1vS8hbSfW5poeP6mdp0Bt5HdVzCiJZL0HiqLvbbnGCkX0HYr86YFG+1nxiNUdlnQraWd3lXX0LZFqAwTQVvcfAe3jMbdcek451EWWwCACgofHgrT9iSTest/cIAmiBwepiHCQyhLaaqN9l04TOdlLRqQrGixbfxSSc8ng/2vNDqSJuVMBrmvsufCQhlDj5yg9xXPUEkkkUii9tD+KSHXfoPN7QDOaBqhczKkeHZcNIBsGz0S0n/rU3TedzZRiTGM3FdzmTSpc5D0Y7At9R4mPe3Zq0PcL1p0Kf0wkidOnqSh14MsDQMAL2uwPGEzF1mPSRnjKIXEqVNUqk2f1m3Y/+pWUrf3lc2kvGCBTjcxb8ESUtfQjKRvJiu4LGs1KN0+LkBYJGw7rcWu9dy11WNusB5Zg5nrLzoPF2smZOOu4edOXH/537HP4vnNv1dmAtn5EARBEAShrsjLhyAIgiAIdUVePgRBEARBqCvnnc0HT2eciGrdORlj7n5Mt8sorZcOn6aaWnNMd0WEuaW5JtVdD588XD1ONSRI3VykMRbpn8HWHXuqxyf6qa1ILErd/XwovPAbB6lbHH5n9Nj7Y4lpc1mUkjvZSPVYBxkO9A8OkbpIjN6XjUIBh8NUz/b7kZ5doe68bo7eZ6qV2jFMxradr5NyyEfdV0sl7ULr99M+WH3tyurxkRPUNmOEeu3B0st1eGo/c4PNI7sXH7PfueYa6gZbRKnO/T76WC2cr+2ALl9C9fSO5mT1OB6m89crUrubYwM6LfrQadqv/cO6LsdC9afTaVIuV3RbfczN0x/QfeA6zDWRua+Gk3osl8LlpC6RmNo4A1D7jHyB3rOFjBUsFv7edem427a25/EUrfMHdHuam6kLcTRK+z2I5kEiwELuo3nIw98rFHrccejDn4hTWyMThdL3XHrPNnKv9UrUFiwRYNd09Fi6zNanjFKvF9hcCrPn+8iAfm53v0ntrUolvYZUinQOKGa7MVUsto7zrOeLL11cPV6whLqV58e1DcgbL79M6nZu30LKLzyvbbX27KZryqIlV1WPF15K7UGSDUlSxu7Q1oR7xmPi1ahjz5NH7ew8NmdInavP4zKDL4+dd6pOsQa3+TDofZnIJd+Z4Bb87pGdD0EQBEEQ6oq8fAiCIAiCUFfOO9mFZ89sa9WRC232LuUx19L2Tr39vR1JJwAAaUNH7lMW3bZONNPtsURcyzK+IN1enodkl2iCuv4+9P3/r3qcZ23LFKgbYx5FS2S7+NCGssgWR6kLaC7A26qlpr37aKTWwUG9VZ9hGW+TSXrReERvG1vM/c+HsmdaeeqK1xJh289BPX485iPm1DEW8bWRylKdndq187IrFtL2oK3pN3ZRV7wU296NooyiQ8NUk4nE9dZ0U5z+3Uduei8pmyikZyJBt7Sbm/Q8GB2lslTfET0mY2kajTUzRiN4jiP363SOztHRjM5O6zC3ZJ+Pyoj+gC6bLFtlIq77Lsmy4zYwySyA5Dd/iEpxWRYhtxZNKPooj2wbDem2ei6LYGzSMWlF0VENm90zinTpZ1JKkGVYtWzdJ1xaMXCqT1aHI8vmc/R54llKsVuuYtmM82N6jpw4TJ/ZURaWMhnS50k1JUldMKjHhLtKKpvKiHZYu6efOk6j+Xa167UxVqb3kSlN3QUTu5aaJt3iVyx7MI4oarHop8mmrurx9TdQF+8FC3pI+cXnfl097uuja1Nup16DM8xNedkVV5JyV5e+ps3cwV1HryEud59F0r/izqxM9jCQxMimFhgmdvVl33M8Min67ISIq7h9E1xt+Xknl3pmAtn5EARBEAShrsjLhyAIgiAIdUVePgRBEARBqCvnnc0HcesEgHiD1osdl95OgOmai3p0KO3tO6h+nfHpcMOeQbX21ByqOe7eo0P4vud9nyJ1m/9bu3rlcizDbHm4ejw0QF1A+XtgtqLLNlANv8HU9iFzQvQaY6eoRuxY2lYi1UrtJlwUNrnANPpiIU/KOeQO6XhUz64UdZbJVh/V5Tui1Bag5Oj6WjYfJ/a/QcoZ5qp4y+/+SfX4pps+SOp+9Yx2FWxN0nFuDbMMuCjMddCgem0qoXXwWIJmEw2ysOQO0nO5TYGDQhoP7KO689EhHeq7XKEarB2kbY3FtKt0a5D2a6U8uZuej7mOW8jOw2I2H7GY7q94nPadZVHdN5vTc2RwcJjUFYt0/tQijOwNKswlNITC0SfjVN/3mCuw7ddusKEobTt2IzSZZu8p5mKIn0X27xn24FXMrdJBc9tx6f1nRmj/4Bb4mM1HdkzbYvWfpPYXqUY6D5MRHZo+z+wxPGS74rClHrsFAwDM6dQ2DZcunE/qrrpMl/cfouvWztf2wFQxkJ2HadD2mDa1gfMh136XuYAaqN9N5oK/cBF1gfdQWoj+/v9L6k4P6749UBojdYMn9pHyJQu16++Sy+k1WlPaddtm3zlORbev4vBUE9Q+D89Ro1YWWWY/ZNRwrlW8jowBPy0zHkGGJxOy7M4AsvMhCIIgCEJdkZcPQRAEQRDqirx8CIIgCIJQV847m49IlOrgDc1a83SYjlg0qR4YjGq9NJmksRiOHtMhe69fSUNFF7NUYwvHdCjy/hPHSd3B/ft1e1jYZOzanstQjTHWREM+j41pzTgRpTYEly5aVj3e9speUvfynj5Svv79H6oe+1jq+UMHtX1IOkM1ah62vVjQdh5zU1RPD6H04Y1Mk1Y21Tmd8tTC9BbzNI7FsiuXkfIHPviB6nFTksZTuW61jsFhMj09xlKtx9F8svwslLZfx4bgsRg8oGM7dlrHZogz3dcDPfDzL11K6lo7F1WPR09T+50Yi7NRQTq9wcKH+9Dk4qm6i0Vqz5NFMSgUC/GcRWnYj/XTuCfcDqiS1+d1XXqecIT2QS1yyN4oFuJ2JvqZHjpFY6RkxtKk7Hm6TxawtPDJRr1OWD5uQ0DL2EanXKa2CHkU06ZYov3hlPX4GS61wVEleh6cwiGZpGkPQn4dV8M26LxLMhuqREyXy+waedQf5RJtj2nQ57IB2TSFA3RuHUcxdyz2+F5+KY2xcwqF+eeYyIaAx2uy2H36UbXHYoLgwBY8NkWZ2T51ds2rHs+bN4/UbRvU89th9kOnhtK0jOxD9ux5ldT19Gh7wUsuof2RSunQ8DEW0h4MakdRLKN4IWyd9CF7Jh67g4dXx9XK4OHeySdpc1gsD1yyphy0ferIzocgCIIgCHVlWi8fGzduhJUrV0IsFoPW1la49dZbYd8+ahVcLBZh3bp10NTUBNFoFG6//XYYHByc5IyCIAiCIFxsTEt2ee6552DdunWwcuVKcBwHvvKVr8Dv/u7vwu7duyESeWv7+u6774af/exn8Nhjj0EikYD169fDbbfdBr/5zW9mpMGeQ7c6E43aBTNXoFu/eeZOht0Ku7s6Sd3+N1CY6zwL8RzpJuWuS/Txkf00DPgJ5BrX27uKtgdtacc6aKbGxg4aFvjoqJZTCiXaHn9Eb9PGW7pI3dUxel+n0Fb14SO7SF0ur6WD9Bh1n21taSHlhNL3NTdKZY7WuN4W9RlULilXqENtBG23UodmyvzFV5Hyx+/8f0g57+oty30H6cuth7Yzg8xFt8K2FkfTaM54dG65KJw3U/TAA7rFPZ7Rd2MN0q3fk0Napiux7W8PZQmNMDfgQweopNd3VGc35uHDG5v1mPDt97ExKvGNDGu3T8XkEhOFuTZYyOtIiGZ/TSJX4CDL+lvI1nKkpgRQ+PeRYZpd+c3Tuq08a2uygbqOt7enqsdlliG0UtbSjsdcHDNM4isgecl16DUtJL/5ffR/NyylBCO0r0IsR0IRrQUec9mNRFEqAyZP+FlGVbymcZfqInLtNKzJ3VUBACoVvRYcH6EZk/M5PX+4K2lbO11vamEhCcDicgBzQwUDjd+EMOD4b7m/KP0szpYbi1FJmLiz8gzFPPS50u0bP03n6M5hlGX3lW2krrFJz9G2NrpWt7XPY21F6RyYDN+S0iElDObyzuezg6RUh7nlkvDqPIS7R+ezQvKj8mrJN++Mab18PPXUU6T88MMPQ2trK+zYsQPe+973wtjYGDz44IPwyCOPwAc+8JYm/9BDD8GSJUtgy5YtcO21185cywVBEARBOC95VzYfv/2PqrHxrf/Ed+zYAZVKBdasWVP9zOLFi6G7uxs2b958xnOUSiXIZDLkRxAEQRCEC5d3/PLheR7cddddcN1118HSpW9Z8A8MDIDf75+QDTOVSsHAwMAZzvKWHUkikaj+4OyBgiAIgiBceLxjV9t169bB66+/Di+++OK7asA999wDGzZsqJYzmUzNF5DxEer+F0KukyUWmtnw6O3hlMXNjdRuYb95qHo8NEo14BGL6l2JqNbfFi+l7lOHDmtdvkKlOOLOunAhdcla2HMJKR/p1zrrG2+8RtszjFKZB6hNQwMLK338DW070j9Md5UM5IpsBenftXfREMtzkT7YHaN6dtDUemipyFNKUx2ahxiejP99x/8h5YY2qi2/8rq2h+DudWWkT7rMjVIxXRO7kBnM9czFmierMye8tuv6ikP7YHhE26TgENwAANisIhlPkjru5jk6guYl0/CHh7VNQ4nZ2TgsdL5b1s+J5afPSDio50SAhV63HHrNchH3O53sOCz625FGbsonT9Bw4hHkxr34Mupu3dhMw62Hw3peFgv0GT59WqckqFSYS6qi60YYhc5PxKmNQySgyyFmY2EjuwGXudo6Dr1GBS0ORZM+EzhcNk897zI7NhyR37ZoaAHl6XEvlugcGDlFw70Po/Dv4+PUGut0Ol095nZJgRhdR2thKGzzQeu4S6iB7BgMNXnYb26rgV1SAQAKWX0vAwP0u+PkSV0eC9O/87HnC7vkR4J0bodt/bfc5fxEv16nDhw+ROoKhU2k7Lj6ms0tHaRu2bLLqscLF9Dvx5YW+hzEE9qtPBBioQ8AtZ3ZcTjs+woM5Kp9Flxt39HLx/r16+HJJ5+E559/Hjo79ZdCW1sblMtlSKfTZPdjcHAQ2traznAmgEAgAIHA1GMCCIIgCIJwfjMt2UUpBevXr4fHH38cnnnmGejpoR4ay5cvB5/PB5s26Te6ffv2wdGjR6G3t3dmWiwIgiAIwnnNtHY+1q1bB4888gj89Kc/hVgsVrXjSCQSEAqFIJFIwGc+8xnYsGEDNDY2Qjweh89//vPQ29s7Y54uhw7SravuhUuqx0GTbm16Zbr9bKPtsiDbOovFtHwRjdOtqsWLabTEX/3Xz6vH+TFqyxJu0u5+B49Tl6yuTu2y23PpNaQuwLa/53frz6ZHqevb7j3aLdhTdMv2+GnaBxnkflx06Q5TJq1loFbmBnZkhLqdNnYlq8cjfKfKQy67TFZRNpVoSp7e8q6137Vz13ZSfvW1XaRsgD6vZbHtbyTFWTbf/ucZXvVWp+2n7+J4jvh89O/8rA9MFA3VUvSzcb92tzOZTFax8PiwaLBst9kf1hJEJc+kA5RBuczcQ40Ky3iLNKMy28Z3Uaba3Dg9T5jN0ZaEvhebZfnFisTbOd02tuhnpoFJKTYeH/bMjmepe3g2q/sgEGByH3Il9ZgbbkeKupUHkPRksci2ytNjlCvSOysid+s0knkAAEZGaeTPApKFliyh64sP7RrzzW6LpSLF7rSlHJVLjqPM2TzyaLlM14l8TrdnLE1ds/0oyizv803PPEPK7119NUwKiqrqsQyqymHZYJFEw5RSMJC8xF1ALeZC/MrLO6rH2dO0D5pQdNhj/bQuzrJY+9E65jHpNB5FkVtZ9Fy/ra/hC1DJyjKZvH86XT0+3EezeqdP67F8eTtbi1hk5i4kmXe00zAR7R16ne9I0bpIlLquGyHd8YY58+rEtF4+HnjgAQAAuOGGG8jvH3roIfjkJz8JAADf/OY3wTRNuP3226FUKsGNN94I3/3ud2eksYIgCIIgnP9M6+WDB145E8FgEO6//364//7733GjBEEQBEG4cJHcLoIgCIIg1JXzLqvtroPUjqJ7qQ5h7gHV0Azu1ol0xgxzJ0untatZU+NVpO5DN72flK+6cnH1+Mc/eZxe09CaXyJBNbQ5HdozKMrcKi2Htr2xTQ9New/VqMdCWuN7edcuUtefZWGCfdoVONFO3eKaF+g6bhvhsjDk+5TWKw8OUJ8sP/KbK7AMqjk2BI6n++dmKu8TXnjuaVLOZ9L0mj6tpYbC1E0YT2tL0SnOs2CaPmzzQe85GNA6Lw8f7g/S7KJ2RPdt0E/drwOm1mhtrl8Hkasvy+xZKVFdvohcZrENAwCAh10V2Xls5iZM0isz24hkRJcTEdp30RB1Rwz49DV9Bp2jBguFXosK2lHl/WyjMPIuCxXNM6HayDWYmUZAENlxFHK07wpjdC0ooCK3AzJRSHXFbHT27dldPT5y+DCp4xmuFXIl7WinnoCNCT1/Cnlqe8XLaWQnMIJclgEACsjmzWVtzfPzoOCOJpsvYVvPg/6T1BWax2+qZfNRQbZI3D3ecOhcw1l3eWBvBbqOu+xms3QsiwV9zUsXLSF111y1onq849XXSd2WbVtJOZ3V67PL3KZb27Vb7PXXX0/qbDSfDx+hqTi2bKGBN5deprOpxxN0DRlE/cxzpfG1oC2lQ7P39MwjdTh8QG6c2vbwcAI+W6/5RTZeM4HsfAiCIAiCUFfk5UMQBEEQhLoiLx+CIAiCINSV887mY/8YjRsx7Gq9X/movYFZZpoWsjfgYYs72rUBwv96D43BEfRRG4eeuXOqxx/+3x8ndf/++M902wbo9fvHtN5WLB4kdX6gmuxoQZcPHmF5cZD+ploWk6qGFLVF8JCOZxhU3/eQ3YJnUD2/wuI/jKEU9kEf/WzQ1sJrzqBacoXFx1Ae1g4n1xFTLdTPvr9A/fBdN109jv9PYsPfYqP7zAzTGCnjGWpbU3Fx/Admp1ArjbRJ78sX0vNH+WjbHUM/ZiYz+gj79RhEQnTs3MrkNksQoOcxkL1KkMXjCDE7isaY1nK7WDj+znYdmpmF7oBSkerpptLPm83E92RcP6d5aoowgf3791SPL7/8MlIXQrYafDhMFgXDQ6nEB4eobVguo5/FUoHGaXCZbRi2j5i/YB6pa2nV/eOyBvmQfUqSxYnAsUMAaHR8Hvp877591eNsjsbV4J/F6Qo85o2YQ3ZteXbP+Tx9DsrIvijgo/Pn6KB+9tIo1DoAgOu9vQfkb8Hekty+gBdxunsW5R88ZA/CA6GEwvQZ+l83fBB9lJ7IRvFLFl21itQtXb6SlHG4Fz7vmpu0vdf8+TRNho3Gfd7CK0hdRzeN7xIK6WcmwWw+cN+NjtIHCttxAAC0tmgboliMnsdC9jsmC6DienT9q6Ax8Iypj/NUkZ0PQRAEQRDqirx8CIIgCIJQV8472WVfmr4v/fRFnfH1qrnNpK7NT8PZhtF2YjtLdNferLdJL5lPM6gCy3rZf0pve33/0Z+Ruh27tLsdz7JLdncVvQ/FXPHcgG6Py7b4bRRa3DGofOSYLOMsHmHmPlssI7dB5ptoM9dbC20xqyILA46c4Xw8a6xBy+XK1LIjqgqVbxIRum09jlx6Ky7dml68ZKk+Twd1Lx5i2TyHUDbPbJrKa9gdkbsqKpduf0dsvb25+MoFpO4kcuU8laEyUKGs214o0nu22PZuAIWNj/i4i6we95aGJKlr76BzfcEcHc68NUDnTxaFaR9lIcEt5nYajmhX8ijLdNzUpOtO9lEXQ04FyTnFbJrUmei5mJBZ2KLLl4vCph84sJ/UjY/p8/qZrOAP0LmOQ7p7LNWniTMWM2myCcl/3NU3X6BztIDKx44dJ3X4b9njA4qlU86X9TzkkkhuWEtNPnbPDgu576BsrDkWXt1BoeB51tYJekkNCkj6sTJUwrMVy5iM1lyHZUx20Bjw9nhMCsNKlMOeYQOnGfDoeTq6ad4y8JBLvEcH10Rred9RGla/UNbtMdjYxRL0Grjtp8doW20kl0Ti82jb2Lo+Oqb7+eQgbQ8Oax8w6ZrKEgKDEdXXLJ6m691MIDsfgiAIgiDUFXn5EARBEAShrsjLhyAIgiAIdeW8s/nIMp3qVy9rbXf/m4dI3c3LqdveJR1al+87dIDUvXelthMIMj19vEz1yB8/ta16/PJuGm44j1NDM7sJHJqZp5TG4YQBqA2Gy/TIErKrqDDN02BhrksohTxPDGgjt0+L+bOFw0wPRLor8+wCF7mScrcvh7mL+mNJVKLukJiRk1QHdytUcywgrTl/7Cipa7T0PbcEqd2Pr0TtKkKmbm/BYmm+FW57ba07X9C2I+9deTmpu3zJsurx0aPU/mEkrW1ASiycOrA5YiP38BBL9d6M3GmTEXrPLmv7wLDur33D/aTOQK6B8VZqLxOKU7fcMHLZbWymn40yV8FahNA8LDPbCOzGbTD3eJPNWRPZNcTjUXoeFEY/GqHumBZzRQ4H9XPLbSMO7N1bPR4bpXr6GEpp7yra5z4/bTsOBR9gYruBxjZfpC6yQ8zNMo9cby3WPw2JZPW4zNIe5AvU5sKp6PZ6E+w6sBEKtS8wuFFKDZ5//tnq8ZjzKqmL2MzNHD2nFWbHgd3jXZeOD1/jKsgOiK+j2O20WKJ1LrPnMZBNis9mrutJbWsYjSZZW9Gaz92JJ/SlLpvMPgT3s8m+A22blk30WT4+uHsMto4bBvsuCaNrFpn9F51q7wjZ+RAEQRAEoa7Iy4cgCIIgCHXlvJNdmppbSHn0tN5H6kcZHgEA/vuVvaTsVuaiEt2qamnT7rWGRbfVtm6nGQ9/9ozORljy6HYhoC05vnVG2sK22BXbk8PRGvlWIs4467PpEBp8P8zS92mzOgu5KsZidJvaYm23FNq+ZG7CHpJ2uCbT3ka332NxVM5PLru0tdOopcePMhmmhKMcUmmnb7+OEDnmp+PDRySHIq7mHLqF6xHXPC6T0S3TcklvY7/84n+Ruhsium+Xsn4tJLSUwd06eVbmInKrHGNZY7HL8JG9NOvlcCFDykWfbnuolfZzQ1uyehyIM3mCZbUNoyiegTCVegxr6ksLjjbsOnT+4CzRvH9KJSodYFfbEHsuTCSlFnI0umdplEqnR/Na+vHYGBjoWfQxeRa7p/uCTCJi3VEu6/OOn6bSSrGYRcdUJuSO6kE0nyoFuqZUQLehwCKc8jJ28zSYn7CDxke5dP76fVNznQcACKJM1BWLzS2PdlAAhRrwDOZSjdpqsrZyd2zP0/08UYJAUpNiWXZZTyu05hosvAFWc0ygY2Bb+vqlEn1muestvqTjMPkIyddcIufRumvJN5gyywCsmERexMmvLSr3dXTMhXeL7HwIgiAIglBX5OVDEARBEIS6Ii8fgiAIgiDUlfPO5oPbLfhQyGmnSDXpvkGqdZdyOnvme69ZROpCyfbq8ViR6s7PvbSdlAvIBbPC7AQCKFQzD/WLw3VzLKZrEpMC5qIVQHq6wcVkVjYCWlvFWRMBaMjeCtP7xpkujrNXlpgun2jQrmZtKCsqAEA0SNtTQJk2a736di/qJuVMjo5l7jgOk87CxiNXwVHWVj/r5zIaS+4eWSt0tKEmrzvw6lZSPjaudeAWk2rd2J7HZfps1qRtH1Bapz/IXIaPo4y8+TC9x1h3BymnerReG0zS7Ktk/jBtORqldkFh5Hpr+qidlJqGC2YmrccyP54mdUMn9TNdLFLN3GVZiCuVMjpmruto/posA6+PZa2mLujMRRa57PIQ6hXk9lnIUe2/VKLP0zgKga1oUyES12sIt71SFTonSlk9DxyHXnMM2RhwGw/udoptHDw1eTZn26Z2LobnTPLJieCs0dkcTTMQtvj8QW1lCwXO5FtmaRgch4UBN/VnFbPrwPPFc1j4eeZq6yJ7I247grMJcxMLpfQ9l5jb9ITQ8DjrL7MBVMRd3mV1zC0YfXlwixx8DavM+4OOZb5BP9/tXdTNvgPE5kMQBEEQhPMMefkQBEEQBKGuyMuHIAiCIAh15byz+eC+/jg1vWfRcOZloHrtYFbrby/vo779H8prLWxcUf/nE6dpOYi0bydPr1FEOms4zGwsfPYZPwdwhtDRBg7nS4dJIV1esfdHH0sPnkVhk8sO1Z2xDQiPJcLtOnJFrY9Gk9Suo6FFp2wvM915714aa8WHtOblNWTDeAONP9GSaiXlfmTzMUHXRMclZsdRYaYaOPS4O4304BM+iRpRYfp6bliHJjYDSVJnofDYJ5mWuwvoHDlo6zvLRan2HunSKexbOuaQuqaWFCkHUHjxMrsThfT+gM3iwvAysoeweFyNacRfHjisUyQoZieFdXEef8IOMPsDC8dioJ/1I5uUMIv9wj+LbbUcFucjm9U6eblE6zxkqGCyUNWeS58Lf0DHRUnNoTY52axOaZ85TW0jnDKLD4Tax2NT5MvYHoTZwHCbJRxBnZ3Hh/rdAm7HRtfGWhw7puMlHein9xFhIeZtbIs14QnX4+64bAw8asfgD5iT1mHbERalfUIYeRxbwzBYzB88L/kcRfZ53AaQp1Pw3MljrZjIVs0w6LznqTrwM1xjmKECtO/cRvpczFmm05MkaBifWuZwU0Z2PgRBEARBqCvTevl44IEH4IorroB4PA7xeBx6e3vhF7/4RbW+WCzCunXroKmpCaLRKNx+++0wODhY44yCIAiCIFxsTEt26ezshPvuuw8WLlwISin4wQ9+AB/96Edh586dcPnll8Pdd98NP/vZz+Cxxx6DRCIB69evh9tuuw1+85vfzFyLeWpAtMVkWWw7StGtX9fU9X1DdLvw+z/+efX4AzesIHV9J2lGvxzOVMhlD5QV1GJbiWG0decPUXmkME4lEez2pJgE4kPuq3wrnLtL4a1xvj1XwGGkWR13MUwiGaQp1U7qTo3o7J7p4QFSlz5CswcvmN8DUyHEstEGWOZRn1/3pcvcD/GdOAbfH2RuhGqS47dhgjMi2qbNsr7ci7a/E34qxe0t6pfzN5gsNsLCmzd16b5r76HSShKFow9EqEus6dEt3Ap+ZlhGTAvJE/aEbKv0PEQSMfg28dT/r7E8LVN5LDw/Dm8+4frMrdxUeGuaXqOEwtE7FdrPWC4BmOgCicHu6T4/nZMWckO1eUoE9gwHA/o8gRA9z+iIbmtunK5TPibPWqify0zKdfD2ew13TAAahpu7kQfRGpPNpEldPjcGU8VUKPw8lwNcunZjWWhC5lwLhVdXk693ADSEAfekx/NFsZDpfAIpGkOdgOUUHgrCQW2vsLZ67PtKoWzGXC7BWc75jRgTxlZfU9m0sQ7KrB7vaCN1ncto+Anb0PMyvf812qBOKuW+E6b18nHLLbeQ8r333gsPPPAAbNmyBTo7O+HBBx+ERx55BD7wgQ8AAMBDDz0ES5YsgS1btsC11177rhsrCIIgCML5zzu2+XBdFx599FHI5XLQ29sLO3bsgEqlAmvWrKl+ZvHixdDd3Q2bN2+e9DylUgkymQz5EQRBEAThwmXaLx+vvfYaRKNRCAQC8LnPfQ4ef/xxuOyyy2BgYAD8fj8kk0ny+VQqBQMDA2c+GQBs3LgREolE9aerq2vaNyEIgiAIwvnDtF1tL730Uti1axeMjY3Bv//7v8PatWvhueeee8cNuOeee2DDhg3VciaTqfkC0sRebopFrYnmWEppv0X1dQfprjwc9HNbX60e952kbrjpHPXDGs1qjZp5lkIE6e0Oc60KBCbX04MhquNZSNu1ffSzONyww+wLjAluV8iVtELvo4zCC4eC1AaluamJlBubtZ1HWdF31pJfT6NCgLbVY2nHcyzE8GRUmAtdrkC171hSt7eYY2G3Ub+7TC92uV0H+oUxudQ/AcXsBBRyqcuZtO0vlLUufiRP60bCun12is779s4WUu5p0eWmBB0fE827HNOAi8zuxUYafpDZ0gTD2tbG9tM5EQxRG5QAmjM8vfx08JCfI3cBVUgnV8x2RTG/aWKDwq6B05e73C6APV/4ObW4Czz6Wz6VsF2AW6Fhvl3mfl326b4rFKgNCrbz8JiLrOFnrv0oZcOEvkNTn7eV23zgepuHdC/r5+v0CHUgqJSn9jwDADgovLrL/q7MUgmQUPEes+1BRY/ZP5isD8poTDxuc4HsizyP3rOffT/gZYSfB9sicfMUD4cwZ/ZM3LaG2Iuw8TGQnQtwd2J20Qr6DqhE6NxuvPSS6vGceXS9KTLnkDf36rQioUqW1EEnvGum/fLh9/thwYIFAACwfPly2LZtG3z729+Gj33sY1AulyGdTpPdj8HBQWhra5vkbG896PhhFwRBEAThwuZdx/nwPA9KpRIsX74cfD4fbNq0qVq3b98+OHr0KPT29r7bywiCIAiCcIEwrZ2Pe+65B26++Wbo7u6G8fFxeOSRR+DXv/41/PKXv4REIgGf+cxnYMOGDdDY2AjxeBw+//nPQ29vr3i6CIIgCIJQZVovH0NDQ3DnnXdCf38/JBIJuOKKK+CXv/wl/M7v/A4AAHzzm98E0zTh9ttvh1KpBDfeeCN897vfndEGF5nNAIqeCyUWI9dnUb3LQZKaYrqmGdKa+WEW18NksTQcpDU7zH+/WNRab46lpce+9FxqivipZh5CcUBMpofimBehMI3pUC5TPfLUqI7B4bFwujby+W6I07gabY1JWm7TcSTSzMYik9YhoLNjaVKXbKRh0odPDaMSDdOOqbj0Gpaf6qMNLbq9lSgbZxT3g4UAgQqzw1HI5oN1MwkzPUEj54EkcIwHm8XVCOn2lRK0Py5Jan/5hkaa3j4ap49nNKznYSBI64oo7UCZp9xm9hgWCvM/ISAGKvuYXRKPKeND5+HxFXhciVoUUchwm6cSQO2ZEMKdpXc3kd2NyZ5vbLsxIfQ7K2P7EB7uHYcpd1k6+QoaA4utU5UstVlyUXsiJWq/g+08TDY+pQJLGc/jHpGqyet4uHUbzRE+lqODQ9XjSomuaXz61ASd1vKxOCPs+fahtQlctkGPjFkslkKDN0chQy6D2WkFkf1MQ5w+lybw2C+Tj7uFwvoHmM2b4yCbMnZOHm7dRfYp4xk6X7Bpi8fm/ZhBz2M363uZu4jG7mho0Gvuib0HSd3wwUP0POg+g77pDPTUmNbLx4MPPlizPhgMwv333w/333//u2qUIAiCIAgXLpLbRRAEQRCEunLeZbXl244BtOUVZnfjVejWJ46g67EA2R4KReyxrTynzFzYXH3Nia6Busy31fBW8OlRmq1ylLU1HtOyQoJleI2jMO1BoO6QrkflChttO1oBel+lov5skEkFNvM7dfJj6JheI5seqR57Fep7HGSZR4tTzHbKt2WTTVReikaQ62SJjgGWXRyXh17nYaVRSG72Lo63vE3ucsnCFtto2zjM5IkYGstUNEnqogHtDh5hodf9rO/KqJj10+sX8LYwc70Lsm1av4VDhNNtYixJGNzlkrsxIjdCv5+5//mmntUWZ2Lm/exDbeBSimL3iUd2YlR9HLqabpuDO7mrNs+i7SB39TLLMFtAUotbyJM6h7naRtB5QwkqPzqoXytFeg0uw2C4NAjY5ZyH62ayWAStKbkMXZsyOKQ6O49pTv0rxMK6d5mtvyyDswLdBxbQ+Wuj8sSMxMwNFk0Eno3Wc/Q18jYNbsmzjAOSMnHWWAAAD2UOL1a4DISz4fIQ7uwSqHkusDS7qO3cVTzeyjKAL9JpGEz2Pbdv20u6rUPDpM5ic91Gc6KWhPdOkZ0PQRAEQRDqirx8CIIgCIJQV+TlQxAEQRCEumIoLuTOMplMBhKJBHz5y1+WyKeCIAiCcJ5QKpXgvvvug7GxMYjH4zU/KzsfgiAIgiDUFXn5EARBEAShrsjLhyAIgiAIdUVePgRBEARBqCvy8iEIgiAIQl055yKc/tb5plQqvc0nBUEQBEE4V/jt9/ZUnGjPOVfb48ePQ1dX12w3QxAEQRCEd8CxY8egs7Oz5mfOuZcPz/Pg5MmToJSC7u5uOHbs2Nv6C1+MZDIZ6Orqkv6ZBOmf2kj/1Eb6pzbSP5NzMfeNUgrGx8eho6NjQi4mzjknu5imCZ2dnZDJvJXoJx6PX3QDOB2kf2oj/VMb6Z/aSP/URvpnci7WvkkkElP6nBicCoIgCIJQV+TlQxAEQRCEunLOvnwEAgH4y7/8S8nvMgnSP7WR/qmN9E9tpH9qI/0zOdI3U+OcMzgVBEEQBOHC5pzd+RAEQRAE4cJEXj4EQRAEQagr8vIhCIIgCEJdkZcPQRAEQRDqirx8CIIgCIJQV87Zl4/7778f5s2bB8FgEFavXg1bt26d7SbVnY0bN8LKlSshFotBa2sr3HrrrbBv3z7ymWKxCOvWrYOmpiaIRqNw++23w+Dg4Cy1eHa57777wDAMuOuuu6q/u9j758SJE/CHf/iH0NTUBKFQCJYtWwbbt2+v1iul4Otf/zq0t7dDKBSCNWvWwIEDB2axxfXDdV342te+Bj09PRAKheCSSy6Bv/7rvyZJsS6m/nn++efhlltugY6ODjAMA5544glSP5W+GB0dhTvuuAPi8Tgkk0n4zGc+A9lsto53cfao1T+VSgW+9KUvwbJlyyASiUBHRwfceeedcPLkSXKOC7l/po06B3n00UeV3+9X3//+99Ubb7yh/viP/1glk0k1ODg4202rKzfeeKN66KGH1Ouvv6527dqlPvShD6nu7m6VzWarn/nc5z6nurq61KZNm9T27dvVtddeq97znvfMYqtnh61bt6p58+apK664Qn3hC1+o/v5i7p/R0VE1d+5c9clPflK99NJL6tChQ+qXv/ylOnjwYPUz9913n0okEuqJJ55Qr7zyivrIRz6ienp6VKFQmMWW14d7771XNTU1qSeffFL19fWpxx57TEWjUfXtb3+7+pmLqX9+/vOfq69+9avqJz/5iQIA9fjjj5P6qfTFTTfdpK688kq1ZcsW9cILL6gFCxaoT3ziE3W+k7NDrf5Jp9NqzZo16kc/+pHau3ev2rx5s1q1apVavnw5OceF3D/T5Zx8+Vi1apVat25dtey6ruro6FAbN26cxVbNPkNDQwoA1HPPPaeUemvC+3w+9dhjj1U/s2fPHgUAavPmzbPVzLozPj6uFi5cqJ5++mn1vve9r/rycbH3z5e+9CV1/fXXT1rveZ5qa2tTf//3f1/9XTqdVoFAQP3bv/1bPZo4q3z4wx9Wn/70p8nvbrvtNnXHHXcopS7u/uFfrlPpi927dysAUNu2bat+5he/+IUyDEOdOHGibm2vB2d6OeNs3bpVAYA6cuSIUuri6p+pcM7JLuVyGXbs2AFr1qyp/s40TVizZg1s3rx5Fls2+4yNjQEAQGNjIwAA7NixAyqVCumrxYsXQ3d390XVV+vWrYMPf/jDpB8ApH/+4z/+A1asWAG///u/D62trXD11VfDP//zP1fr+/r6YGBggPRPIpGA1atXXxT98573vAc2bdoE+/fvBwCAV155BV588UW4+eabAUD6BzOVvti8eTMkk0lYsWJF9TNr1qwB0zThpZdeqnubZ5uxsTEwDAOSySQASP9wzrmstsPDw+C6LqRSKfL7VCoFe/funaVWzT6e58Fdd90F1113HSxduhQAAAYGBsDv91cn929JpVIwMDAwC62sP48++ii8/PLLsG3btgl1F3v/HDp0CB544AHYsGEDfOUrX4Ft27bBn/3Zn4Hf74e1a9dW++BMz9rF0D9f/vKXIZPJwOLFi8GyLHBdF+6991644447AAAu+v7BTKUvBgYGoLW1ldTbtg2NjY0XXX8Vi0X40pe+BJ/4xCeqmW2lfyjn3MuHcGbWrVsHr7/+Orz44ouz3ZRzhmPHjsEXvvAFePrppyEYDM52c845PM+DFStWwN/+7d8CAMDVV18Nr7/+Onzve9+DtWvXznLrZp8f//jH8MMf/hAeeeQRuPzyy2HXrl1w1113QUdHh/SP8I6pVCrwB3/wB6CUggceeGC2m3POcs7JLs3NzWBZ1gSPhMHBQWhra5ulVs0u69evhyeffBKeffZZ6OzsrP6+ra0NyuUypNNp8vmLpa927NgBQ0NDcM0114Bt22DbNjz33HPwne98B2zbhlQqdVH3T3t7O1x22WXkd0uWLIGjR48CAFT74GJ91v78z/8cvvzlL8PHP/5xWLZsGfzRH/0R3H333bBx40YAkP7BTKUv2traYGhoiNQ7jgOjo6MXTX/99sXjyJEj8PTTT1d3PQCkfzjn3MuH3++H5cuXw6ZNm6q/8zwPNm3aBL29vbPYsvqjlIL169fD448/Ds888wz09PSQ+uXLl4PP5yN9tW/fPjh69OhF0Vcf/OAH4bXXXoNdu3ZVf1asWAF33HFH9fhi7p/rrrtugmv2/v37Ye7cuQAA0NPTA21tbaR/MpkMvPTSSxdF/+TzeTBNugRalgWe5wGA9A9mKn3R29sL6XQaduzYUf3MM888A57nwerVq+ve5nrz2xePAwcOwK9+9Stoamoi9Rd7/0xgti1ez8Sjjz6qAoGAevjhh9Xu3bvVZz/7WZVMJtXAwMBsN62u/Mmf/IlKJBLq17/+terv76/+5PP56mc+97nPqe7ubvXMM8+o7du3q97eXtXb2zuLrZ5dsLeLUhd3/2zdulXZtq3uvfdedeDAAfXDH/5QhcNh9a//+q/Vz9x3330qmUyqn/70p+rVV19VH/3oRy9YV1LO2rVr1Zw5c6qutj/5yU9Uc3Oz+uIXv1j9zMXUP+Pj42rnzp1q586dCgDUP/zDP6idO3dWvTWm0hc33XSTuvrqq9VLL72kXnzxRbVw4cILxpW0Vv+Uy2X1kY98RHV2dqpdu3aR9bpUKlXPcSH3z3Q5J18+lFLqH//xH1V3d7fy+/1q1apVasuWLbPdpLoDAGf8eeihh6qfKRQK6k//9E9VQ0ODCofD6vd+7/dUf3//7DV6luEvHxd7//znf/6nWrp0qQoEAmrx4sXqn/7pn0i953nqa1/7mkqlUioQCKgPfvCDat++fbPU2vqSyWTUF77wBdXd3a2CwaCaP3+++upXv0q+LC6m/nn22WfPuN6sXbtWKTW1vhgZGVGf+MQnVDQaVfF4XH3qU59S4+Pjs3A3M0+t/unr65t0vX722Wer57iQ+2e6GEqhcH6CIAiCIAhnmXPO5kMQBEEQhAsbefkQBEEQBKGuyMuHIAiCIAh1RV4+BEEQBEGoK/LyIQiCIAhCXZGXD0EQBEEQ6oq8fAiCIAiCUFfk5UMQBEEQhLoiLx+CIAiCINQVefkQBEEQBKGuyMuHIAiCIAh15f8HdxvpomgNdv8AAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GroundTruth:  cat   ship  ship  plane\n"
          ]
        }
      ],
      "source": [
        "dataiter = iter(testloader)\n",
        "images, labels = next(dataiter)\n",
        "\n",
        "# print images\n",
        "imshow(torchvision.utils.make_grid(images))\n",
        "print('GroundTruth: ', ' '.join(f'{classes[labels[j]]:5s}' for j in range(4)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IPQO7KOahxsa"
      },
      "source": [
        "Next, let's see what happense when we try to load back in our saved model (note: this breaks because of pruning)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "NEmQdJluhxsa",
        "outputId": "9b0717db-f282-4e51-f242-4aa9cd60c3d2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"<ipython-input-13-4ced9f71ea99>\", line 4, in <cell line: 2>\n",
            "    net_loaded.load_state_dict(torch.load(PATH))\n",
            "  File \"/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py\", line 2152, in load_state_dict\n",
            "    raise RuntimeError('Error(s) in loading state_dict for {}:\\n\\t{}'.format(\n",
            "RuntimeError: Error(s) in loading state_dict for Net:\n",
            "\tsize mismatch for fc1.weight: copying a param with shape torch.Size([181, 400]) from checkpoint, the shape in current model is torch.Size([250, 400]).\n",
            "\tsize mismatch for fc1.bias: copying a param with shape torch.Size([181]) from checkpoint, the shape in current model is torch.Size([250]).\n",
            "\tsize mismatch for fc2.weight: copying a param with shape torch.Size([107, 181]) from checkpoint, the shape in current model is torch.Size([200, 250]).\n",
            "\tsize mismatch for fc2.bias: copying a param with shape torch.Size([107]) from checkpoint, the shape in current model is torch.Size([200]).\n",
            "\tsize mismatch for fc3.weight: copying a param with shape torch.Size([10, 107]) from checkpoint, the shape in current model is torch.Size([10, 200]).\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import traceback\n",
        "try:\n",
        "    net_loaded = Net()\n",
        "    net_loaded.load_state_dict(torch.load(PATH))\n",
        "except RuntimeError:\n",
        "    print(traceback.format_exc())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6NU3lhYhhxsb"
      },
      "source": [
        "Okay, now let us see what the neural network thinks these examples above are:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "JNkknB2shxsb"
      },
      "outputs": [],
      "source": [
        "outputs = net(images)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uu-z4oKchxsb"
      },
      "source": [
        "The outputs are energies for the 10 classes.\n",
        "The higher the energy for a class, the more the network\n",
        "thinks that the image is of the particular class.\n",
        "So, let's get the index of the highest energy:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "7Sl7vxBghxsc",
        "outputId": "96e3bafb-628f-4755-eb58-f9a3d121dfd4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Predicted:  cat   car   ship  plane\n"
          ]
        }
      ],
      "source": [
        "_, predicted = torch.max(outputs, 1)\n",
        "\n",
        "print('Predicted: ', ' '.join(f'{classes[predicted[j]]:5s}'\n",
        "                              for j in range(4)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMjtZTd9hxsc"
      },
      "source": [
        "The results seem pretty good.\n",
        "\n",
        "Let us look at how the network performs on the whole dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "v357TLI_hxsc",
        "outputId": "1ce7d50b-f44e-466b-ee68-fb958304d10f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy of the network on the 10000 test images: 58 %\n"
          ]
        }
      ],
      "source": [
        "correct = 0\n",
        "total = 0\n",
        "# since we're not training, we don't need to calculate the gradients for our outputs\n",
        "with torch.no_grad():\n",
        "    for data in testloader:\n",
        "        images, labels = data\n",
        "        # calculate outputs by running images through the network\n",
        "        outputs = net(images)\n",
        "        # the class with the highest energy is what we choose as prediction\n",
        "        _, predicted = torch.max(outputs.data, 1)\n",
        "        total += labels.size(0)\n",
        "        correct += (predicted == labels).sum().item()\n",
        "\n",
        "print(f'Accuracy of the network on the 10000 test images: {100 * correct // total} %')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lAB7x-Ikhxsd"
      },
      "source": [
        "That looks way better than chance, which is 10% accuracy (randomly picking\n",
        "a class out of 10 classes).\n",
        "Seems like the network learnt something.\n",
        "\n",
        "Hmmm, what are the classes that performed well, and the classes that did\n",
        "not perform well:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "0xn0vBvAhxsd",
        "outputId": "9bdb5d1f-a91d-40f1-90fc-71d74cb15e04",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy for class: plane is 62.1 %\n",
            "Accuracy for class: car   is 75.9 %\n",
            "Accuracy for class: bird  is 39.7 %\n",
            "Accuracy for class: cat   is 39.5 %\n",
            "Accuracy for class: deer  is 49.0 %\n",
            "Accuracy for class: dog   is 38.3 %\n",
            "Accuracy for class: frog  is 68.1 %\n",
            "Accuracy for class: horse is 68.7 %\n",
            "Accuracy for class: ship  is 70.0 %\n",
            "Accuracy for class: truck is 75.3 %\n"
          ]
        }
      ],
      "source": [
        "# prepare to count predictions for each class\n",
        "correct_pred = {classname: 0 for classname in classes}\n",
        "total_pred = {classname: 0 for classname in classes}\n",
        "\n",
        "# again no gradients needed\n",
        "with torch.no_grad():\n",
        "    for data in testloader:\n",
        "        images, labels = data\n",
        "        outputs = net(images)\n",
        "        _, predictions = torch.max(outputs, 1)\n",
        "        # collect the correct predictions for each class\n",
        "        for label, prediction in zip(labels, predictions):\n",
        "            if label == prediction:\n",
        "                correct_pred[classes[label]] += 1\n",
        "            total_pred[classes[label]] += 1\n",
        "\n",
        "\n",
        "# print accuracy for each class\n",
        "for classname, correct_count in correct_pred.items():\n",
        "    accuracy = 100 * float(correct_count) / total_pred[classname]\n",
        "    print(f'Accuracy for class: {classname:5s} is {accuracy:.1f} %')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fwl-3qafhxsd"
      },
      "source": [
        "Okay, so what next?\n",
        "\n",
        "How do we run these neural networks on the GPU?\n",
        "\n",
        "## Training on GPU\n",
        "Just like how you transfer a Tensor onto the GPU, you transfer the neural\n",
        "net onto the GPU.\n",
        "\n",
        "Let's first define our device as the first visible cuda device if we have\n",
        "CUDA available:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "ZkLdDxekhxse",
        "outputId": "de709875-e828-4364-f36e-68e97b0741c2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "cpu\n"
          ]
        }
      ],
      "source": [
        "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "# Assuming that we are on a CUDA machine, this should print a CUDA device:\n",
        "\n",
        "print(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-OLQmFahxse"
      },
      "source": [
        "The rest of this section assumes that ``device`` is a CUDA device.\n",
        "\n",
        "Then these methods will recursively go over all modules and convert their\n",
        "parameters and buffers to CUDA tensors:\n",
        "\n",
        ".. code:: python\n",
        "\n",
        "    net.to(device)\n",
        "\n",
        "\n",
        "Remember that you will have to send the inputs and targets at every step\n",
        "to the GPU too:\n",
        "\n",
        ".. code:: python\n",
        "\n",
        "        inputs, labels = data[0].to(device), data[1].to(device)\n",
        "\n",
        "Why don't I notice MASSIVE speedup compared to CPU? Because your network\n",
        "is really small.\n",
        "\n",
        "**Exercise:** Try increasing the width of your network (argument 2 of\n",
        "the first ``nn.Conv2d``, and argument 1 of the second ``nn.Conv2d`` –\n",
        "they need to be the same number), see what kind of speedup you get.\n",
        "\n",
        "**Goals achieved**:\n",
        "\n",
        "- Understanding PyTorch's Tensor library and neural networks at a high level.\n",
        "- Train a small neural network to classify images\n",
        "\n",
        "## Training on multiple GPUs\n",
        "If you want to see even more MASSIVE speedup using all of your GPUs,\n",
        "please check out :doc:`data_parallel_tutorial`.\n",
        "\n",
        "## Where do I go next?\n",
        "\n",
        "-  :doc:`Train neural nets to play video games </intermediate/reinforcement_q_learning>`\n",
        "-  `Train a state-of-the-art ResNet network on imagenet`_\n",
        "-  `Train a face generator using Generative Adversarial Networks`_\n",
        "-  `Train a word-level language model using Recurrent LSTM networks`_\n",
        "-  `More examples`_\n",
        "-  `More tutorials`_\n",
        "-  `Discuss PyTorch on the Forums`_\n",
        "-  `Chat with other users on Slack`_\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "Pf5Ds3j8hxsf"
      },
      "outputs": [],
      "source": [
        "del dataiter"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}