Vanilla GAN (PyTorch).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ultimate beginner's guide to GANs 👨🏽‍💻\n",
    "\n",
    "In this notebook you'll learn:\n",
    "\n",
    "✅ What are GANs exactly? <br/>\n",
    "✅ How to train them? <br/>\n",
    "✅ How to use them? <br/>\n",
    "\n",
    "After you complete this one you'll have a much better understanding of GANs!\n",
    "\n",
    "So, let's start!\n",
    "\n",
    "---\n",
    "\n",
    "## What the heck are GANs and how they came to be?\n",
    "\n",
    "GANs were originally proposed by Ian Goodfellow et al. in a seminal paper called [Generative Adversarial Nets](https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf).\n",
    "\n",
    "(`et al.` - fancy Latin phrase you'll be seing around it means `and others`)\n",
    "\n",
    "You DON'T need to understand the paper in order to understand this notebook. That being said:\n",
    "\n",
    "---\n",
    "\n",
    "GANs are a `framework` where 2 models (usually `neural networks`), called `generator (G)` and `discriminator (D)`, play a `minimax game` against each other. The generator is trying to learn the `distribution of real data` and is the network which we're usually interested in. During the game the goal of the generator is to trick the discriminator into \"thinking\" that the images it generates is real. The goal of the discriminator, on the other hand, is to correctly discriminate between the generated (fake) images and real images coming from some dataset (e.g. MNIST). \n",
    "\n",
    "At the equilibrium of the game the generator learns to generate images indistinguishable from the real images and the best that discriminator can do is output 0.5 - meaning it's 50% sure that what you gave him is a real image (and 50% sure that it's fake) - i.e. it doesn't have a clue of what's happening!\n",
    "\n",
    "Potentially confusing parts: <br/><br/>\n",
    "`minimax game` - basically they have some goal (objective function) and one is trying to minimize it, the other to maximize it, that's it. <br/><br/>\n",
    "`distribution of real data` - basically you can think of any data you use as a point in the `n-dimensional` space. For example, MNIST 28x28 image when flattened has 784 numbers. So 1 image is simply a point in the 784-dimensional space. That's it. when I say `n` in order to visualize it just think of `3` or `2` dimensions - that's how everybody does it. So you can think of your data as a 3D/2D cloud of points. Each point has some probability associated with it - how likely is it to appear - that's the `distribution` part. So if your model has internal representation of this 3D/2D point cloud there is nothing stopping it from generating more points from that cloud! And those are new images (be it human faces, digits or whatever) that never existed!\n",
    "\n",
    "<img src=\"data/examples/jupyter/data_distribution.PNG\" alt=\"example of a simple 2D data distribution\" align=\"center\" style=\"width: 550px;\"/> <br/>\n",
    "\n",
    "Here is an example of a simple data distribution. The data here is 2-dimensional and the height of the plot is the probability of certain datapoint appearing. You can see that points around (0, 0) have the highest probability of happening. Those datapoints could be your 784-dimensional images projected into 2-dimensional space via PCA, t-SNE, UMAP, etc. (you don't need to know what these are, they are just some dimensionality reduction methods out there).\n",
    "\n",
    "In reality this plot would have multiple peaks (`multi-modal`) and wouldn't be this nice.\n",
    "\n",
    "\n",
    "---\n",
    "\n",
    "That was everything you need to know for now as a beginner! Let's code!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I always like to structure my imports into Python's native libs,\n",
    "# stuff I installed via conda/pip and local file imports (we don't have those here)\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "import enum\n",
    "\n",
    "\n",
    "import cv2 as cv\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import git\n",
    "\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.optim import Adam\n",
    "from torchvision import transforms, datasets\n",
    "from torchvision.utils import make_grid, save_image\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.tensorboard import SummaryWriter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's create some constant to make stuff a bit easier\n",
    "BINARIES_PATH = os.path.join(os.getcwd(), 'models', 'binaries')  # location where trained models are located\n",
    "CHECKPOINTS_PATH = os.path.join(os.getcwd(), 'models', 'checkpoints')  # semi-trained models during training will be dumped here\n",
    "DATA_DIR_PATH = os.path.join(os.getcwd(), 'data')  # all data both input (MNIST) and generated will be stored here\n",
    "DEBUG_IMAGERY_PATH = os.path.join(DATA_DIR_PATH, 'debug_imagery')  # we'll be dumping images here during GAN training\n",
    "\n",
    "MNIST_IMG_SIZE = 28  # MNIST images have 28x28 resolution, it's just convinient to put this into a constant you'll see later why"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understand your data - Become One With Your Data!\n",
    "\n",
    "You should always invest time to understand your data. You should be able to answer questions like:\n",
    "1. How many images do I have?\n",
    "2. What's the shape of my image?\n",
    "3. How do my images look like?\n",
    "\n",
    "So let's first answer those questions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size: 60000 images.\n",
      "Image shape torch.Size([1, 28, 28])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAF1CAYAAADIswDXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOydebxUcx/H3197hDalVUm2ynqFxxZJ0uKxZcmWHiHLYwllC61Esif7kmRfIiIP2XOzVCqKUmlDRSWRfs8fM985Z+bO3Dt39nPn+3697mvO/M72O2fmnvn8vr/vIs45DMMwjOCwUb47YBiGYVQOe3AbhmEEDHtwG4ZhBAx7cBuGYQQMe3AbhmEEDHtwG4ZhBAx7cBtRiMiNIvJUho8pIvKoiKwQkcmZPHaqiMhjIjIw3/3IFyLSVESciGyS774Ylcce3AWCiBwsIh+LyG8islxEPhKR/fLdrwxxMNAeaOSca5Prk4vI2SLyYRaP/174IbhnTPvL4fa24fc3ht+f5Ntmk3Bb0/D7qB8UEekpIrNEZJWILBWR10VkaxEZLyKrw39/i8hfvvcjM3x9bUVkYSaPmc/zVAXswV0AiMg2wDjgbqAW0BC4CViXz35lkB2Aec65NfFWVhHV9x1wpr4RkdrAAcDPMdstB24WkY0rOqCIHAYMBk51zm0N7AY8C+Cc6+icq+6cqw6MBm7V98658zNyRUbBYg/uwmBnAOfcGOfcP865tc65Cc65qQAi0lxE3hWRX0XkFxEZLSI1dGcRmSciV4rIVBFZIyIPi0i9sCpbJSLviEjN8LY6RO4lIotEZLGIXJGoYyJyQHgksFJEvlb1GF53toj8ED7HXBHpHmf/nsBDwIFhNXiTKisRuVpElgCPhrc9V0TmhEccr4pIA99xnIj0FpHZ4fMNCN+XT0TkdxF5VkQ2i3P+3YCRvvOv9K2uGVawq0TkMxFp7ttvVxF5O9yXb0WkWwWf4WjgZN8D+VTgJeCvmO3eDLedXsHxAPYDPnHOfQngnFvunHvcObcqiX2jEJGNReS28PfnB6BTzPoeIjIzfC9+EJHzwu1bAeOBBj5F30BE2oTv/crwd+gevf8S4g4RWSahEeRUEWkVXrd5uB/zwyOIkSJSLdF5KnudRYNzzv7y/AdsA/wKPA50BGrGrN+JkKlhc2A7YBIwwrd+HvApUI+QWl8GfAHsHd7nXaB/eNumgAPGAFsBrQmpwiPD628EngovNwz36xhCP/Ltw++3C+/7O7BLeNv6QMsE13c28KHvfVtgPXBLuH/VgCOAX4B9wm13A5N8+zjg1fC9akloNDIR2BHYFpgBnJXM+cNtjxFSv22ATQg9eJ8Jr9sKWAD0CK/bJ9y3RNf3HvAfYALQMdw2GTgQWAi09d9boCvwA7Bp+PgOaOrr18Dw8iHAWkKjr4OAzROcP7JPOd+x84FZQGNCo7r/hc+7SXh9J6A5IMBhwB/APr7Pa2HM8fYlNKLYhNB3aiZwaXhdB2AKUCN8vN2A+uF1I8KfYy1ga+A1YEii89hf/D9T3AWAc+53QnZgBzwI/BxWnPXC6+c45952zq1zzv0MDCf0z+XnbufcUufcT8AHwGfOuS+dc+sIKb+9Y7a/yTm3xjk3jZDiPTVO104H3nDOveGc2+CcexsoJfQgB9gAtBKRas65xc65bypx2RsI/Zisc86tBboDjzjnvgj3uR8hldzUt88tzrnfw+eZDkxwzv3gnPuNkFqLvcaKeNE5N9k5t57Qg3uvcHtnQqadR51z651zXwAvACdWcLwngDNFZBeghnPuk3gbOedeJfRj+Z/yDuac+wA4ntAPx+vAryIyPBkzSxy6EfqxX+CcWw4MiTnX6865712I9wn9CB1STt+mOOc+Dd+fecADeN/Jvwk9lHcFxDk30zm3WEQEOBe4zIVGD6sImYJOSeF6ihp7cBcI4S/32c65RkAroAEhdYKI1BWRZ0TkJxH5nZBqqxNziKW+5bVx3leP2X6Bb/nH8Pli2QE4KTwcXhk2MxxMSD2tAU4mpOQWh00Ou1bikn92zv3pe98g3A8AnHOrCan7hr5tKnuNFbHEt/yHb/8dgP1jrrs7sH0Fx3uR0MjhYuDJCra9DrgW2KK8jZxz451zXQgp1GMJjR7KfeAnoAFlP/MIItJRRD4Nm4ZWEvpxjv2O+bffWUTGiciS8HdysG7vnHsXuAe4F1gqIqMkNI+zHbAlMMV3X98MtxuVwB7cBYhzbhah4W+rcNMQQmp8D+fcNoSUsKR5msa+5SbAojjbLACedM7V8P1t5ZwbGu7nW8659oTMJLMIjRaSJTYt5SJCD0wgYlutDfxUiWMme66KWAC8H3Pd1Z1zF5R7Euf+IKT8L6CCB3d49DIH6J1Mh8IjnomEzF6tKto+Dosp+5kDIbszoRHFbUA951wN4A2871i8+3c/oc+8Rfg7eY1ve5xzdznn9iVk1toZuJKQuWktIZOT3tdtXWiCNdF5jDjYg7sACE+EXSEijcLvGxMyXXwa3mRrYDWwUkQaEvonSJfrRWRLEWlJyJY7Ns42TwFdRKRDeHJri/DEYiMJTX52DT9g14X7908a/Xka6CEie4UfJIMJmXvmpXFMZSnQKN7kZQLGATuLyBkismn4b7/wRGdFXAMclmS/rwWuSrRSRI4VkVNEpGZ4wq8NIXPEp4n2KYdngUvCn11NoK9v3WaE5hV+BtaLSEfgKN/6pUBtEdnW17Y1oTmO1eGRVuRHLXyv9heRTYE1wJ/AP865DYR+3O8QkbrhbRuKSIdyzmPEwR7chcEqYH/gMxFZQ+gfczqg3h43EbJz/kbI1vliBs75PiHFNxG4zTk3IXYD59wCQsPzawj9Uy8g9KOxUfjvCkJKeTmhB0pS6jEeYTV5PSHlt5jQRFmmbJ/vAt8AS0TklyT6sorQg+sUQte3BG8itaJ9FznnkvIZd859RGgSMxErCNmEZxN6SD4FDHPOjU7m+DE8CLwFfE1o4jryHQpf7yWEHu4rgNMITSDq+lmEJrN/CJs4GgB9wtutCh/b/8O/TbhtBSGTzK+E1DzA1YS+d5+GTSzvALuUcx4jDuKcjU6KifBk31xg0/CknGEYAcMUt2EYRsCwB7dhGEbAyNqDW0SOllDE2RwR6VvxHkYucM7Nc86JmUkMI7hkxcYdDhD4jlCk3ULgc0L5FmZk/GSGYRhFRrYUdxtgTjiq7S/gGULeCYZhGEaaZCsrW0Oio7QWEnJ3i0udOnVc06ZNs9QVwzCM4DFv3jx++eWXuIF22XpwxztZlE1GRHoBvQCaNGlCaWlplrpiGIYRPEpKShKuy5apZCHR4bWNiAmpds6Ncs6VOOdKttvOUhUYhmEkS7YU9+dACxFpRijXxCmEoqySIpRErLiInSQutntQ7NcPdg+K/fqh7D1IRFYe3M659SJyEaEQ240JpeusTMpPwzAMIwFZKxnlnHuDUIYxwzAMI4NY5KRhGEbAsAe3YRhGwLAHt2EYRsCwB7dhGEbAsAe3YRhGwLAHt2EYRsDImjtgIfDvf/8bgOeffx6AhQsXAtC+fXsAZs+enZ+OGYZhpIEpbsMwjIBR5RT3LrvsElkePnw44IWRNmzYEIAxY8YA5SdxMSpG7/V7770HwPTp0wFvRFNVqF27NgBTpkwBoG3btkAoe1uQ2WyzUNH7mjVrAnD++ecD0L9/fwA2bNgAwG23her83nHHHWWOsXTp0qz3szL4w+TPOOMMAK677joAdtppJyD5sPJ4NG/eHMj/Z2+K2zAMI2BUOcX9wgsvRJYbN24cte7tt9+Ou0+bNm2AaLU+ePBgAD7++GMAxo0bB8D8+fMBeP/99zPU4+By7733AlC3bl0AFi9enM/uZJzNN98cgNdffx2ALbbYAoC//vorb32qLJp5U5WiqmqA+vXrA3DEEUdE7aNKW5XpFVdcEfXqp2fPngA88cQTmex2ytSpUyeyfPvttwOwbt06AM4991zAU8sHHnggAPXq1StznJUrVwLeaFJH6UcddRQAo0aNynDPK4cpbsMwjIBRZRR3s2bNAE/9+dFfzSFDhgBw8MEHA55NVn9xa9SoEdlHbWXHH3981Ovq1asBmDRpEuDZ0YOmwLfddlsABg4cCMCgQYMAWLJkSYX77rHHHoBn6/3zzz8B6NevX6a7mVdUce+3336A91kvWrQo4T75Zp999gHgsssuA2DfffcFYOeddwbSs+/GY+TIkYDnsfXuu+9m9PiV5eeff44s60jp2GNDVRM/+OADAL777jsAJk6cmPRxDzjgAAB++OGHjPQzXUxxG4ZhBAx7cBuGYQSMKmMq6dixI+C5NoFnIunRowcAd999NwCdO3dO+TzVq1cH4JhjjgHgsMMOAzyXI/ACfpIxO+SL5557DoB27doB8MUXXwDw6KOPVrjviBEjANhoo9Dv/pVXXgnATz/9lPF+5pM+ffpEvfdPfBcaLVq0ALzhv35Py0Mnk2NNBmomfOONUDr9GTNmlNlX91FXyauuugrIv6nEj07E6iTs5MmTAe9e+c0qFVEoJhLFFLdhGEbAqDKK+5xzzinT1rJlSwBeeuklAPbaa6+4+/74449AdDCBqg6d7Nxhhx3i7rvVVlsB0cEJqnaGDh2a/AXkgBtuuCGyrC5gOqn04osvlruvTmaC5za5Zs0aAF555ZWM9jOf7LbbbpFlHUkoOqlViFSrVg1IrLT/+OMPAO67775Im7q4ff3115U+3/r166PeF2LBb500v/DCCwFvVNmtWzfAc2cNIqa4DcMwAkbgFbe66am69qMqQF/Xrl0LQGlpKeDZaqdOnQrED2NVN8MzzzwTgLPPPhuARo0aJezTNddcA3g23yeffDLJq8kO+++/PxCtIFUt/+c//wHgt99+K/cYw4YNiyxvv/32AFxyySVA1bJtqwsgeCHhzzzzDACrVq3KS5+SQe3Vffv2jWr//PPPAc99NR0OOeSQyPLWW28NePMcqugLkTlz5gBeMJ26BZviNgzDMHJG4BW3BhZsskniS9HZYw0yqcwv7dy5cwG46aabAHjkkUcAz26+5557ltlH7Y133XVX1LYavJMrttlmG8CzX2+55ZaRdbfccguQOA2Aol46nTp1irTpdVQl27Zy6aWXRpb/+ecfAB588EEA/v7777z0KRn0O65h3plAU0bo3JB+98H7jmv4v6rYQka9vTQJmnqijR8/HohOUNWhQwcATjzxRMDzotHvgI7OdfSea0xxG4ZhBIzAK26/TTIW9dvUggqV8dtMxIIFCwAvjPa1114DoHXr1mW21Rn+3r17A3Drrbemff7KoDZ89WP1++smq5B0Rl6PAZ5KV7uqpsfVpD3qLx8k1LdXPQ4Ali9fDnjxAMWC2vYfeughoGwSKj86n6M+34WMjhDvuecewEsapikc/B43+l1W270+Q1SV6+hV5350RA7eSC2bmOI2DMMIGIFX3Ndffz0QP3mORkxmQmnHop4U5513HuDNWBcCqoBPOeUUwLs3atcGz8c1Eaq6Yo8Bnp1PS7+pj/uECROAYCpuHTH5R3BPP/10vrqTF3bffXcAxo4dC8Cuu+6acFv1XtHEW0FAnwP6vdXvqdrr1UMGvOvSUav+v+s2vXr1Arz5Mr/Hkd8DK1uY4jYMwwgYgVfc+guoyd9zjfrJvvrqq5E2tX8r/tnqXPDyyy8DZe3/b731VmRZvWViPSVUOWy88cZAdCShorPxK1asADz/cE3xGUQ0dav6+gM8/PDD+epOxlGb7AUXXBBpO+GEEwBvxKTbqI92eSlgNY+LpjPOl3dFKnz00UeAF22tqllLtEHi2AR9zqj9X4szdO/ePbKNKW7DMAyjDPbgNgzDCBiBN5XE1sfLF/7zx/YlV307+eSTgbI19L7//nsgOjGQukJpAFNFvPPOO5FlDWTSSZ4g15rUdAiaAvTbb7+NrPvmm2/y0qdsoOYP/wR1IvT7umzZMsALwtIJa/Dum7rc6v3Ldy3GZFDTngYr6fe4Muj/kpol1eyUK0xxG4ZhBIzAK+7y0Pp72UzHqS6HGiKbT7QIgk4s6qSLhnH7AwPUBUoDT5SmTZsCnpLQyUtNlA+ppQEtVDR5mKYH0ACrqoYmEfMXhzjjjDMAr1DCm2++CXiuolo4QgNw/N8BdR1s0KAB4I3C9H+tkIOWdAI+E0nDcp3GQjHFbRiGETBSVtwi0hh4Atge2ACMcs7dKSK1gLFAU2Ae0M05tyL9rsZH1UI8tzW1O2lazkyirkRaQEFtxn7U/qeuQ9lGQ+pr1aoFeJW+44XgqtubprRV1L1J7Zzjxo0DqpbKLo9CLk+WDqqiNaAkdrk8tByZvyyZuhCqzVz/1/S9lsTLlyLNFbHzSbkiHcW9HrjCObcbcABwoYjsDvQFJjrnWgATw+8NwzCMDJGy4nbOLQYWh5dXichMoCFwLNA2vNnjwHvA1Wn1shw0LHnAgAHZOgXgFQ/QWfNDDz0UiE6VqmhorSam0WRF2ebGG29MeV9Ni3vkkUcCXtDQ4MGD0+5XIaNh3f/73/8AGD16dD67Exi03J++KjqvpHMl06dPz2m/coV66egc1/3335/T82fExi0iTYG9gc+AeuGHuj7c62biHIZhGEaItL1KRKQ68AJwqXPu92TDu0WkF9ALoEmTJul2Iy4HHXQQ4KVrvPrqkPBPZjZZQ6C1wK7a9OLZ0iHaF1S9ObKR3CpbdO7cGfD8ujXk1+/XXJXQ4sc6wtCQ7XylTggq+n+iaAj8zJkz89GdSqFlCB977LFK79ulSxfASyuh6Z1zRVqKW0Q2JfTQHu2c0zLhS0Wkfnh9fWBZvH2dc6OccyXOuZJCrBBtGIZRqKTjVSLAw8BM55w/t+OrwFnA0PBrVutb/frrr4CnlPypGfUHQT0l1Pf0l19+qfC4ap+uKOpR07lq1CLAkiVLkup7IaElmhRNOl9VvQLUj129gbSgrJEcOhJt27Yt4P3/rVy5EshNMYFU0ZSsOn9TGcXdqlUrwCsArvdhypQpGexhxaRjKjkIOAOYJiJfhduuIfTAflZEegLzgZPS66JhGIbhJx2vkg+BRAbtdqket7JoIVdV0/60lRpBqKjNOx3U/1nTuZ522mkALF26NO1j55N999036n0Qck6kg46KdPS100475bM7GaNZs2aA5/Wg8yzp5JPREnw6DwJevg9V2vPnzwfgoosuSvk8uUI9wiozf6NzWxpRPG3aNCB5X/hMY5GThmEYAcMe3IZhGAGjyiSZ0vBuPz179gS8iahU+OuvvwD45JNPAK+6hb+aTFXi999/B7zqNkYwUBPJp59+CnhpDzRA5pFHHgE8E5+fRYsWAVCnTh3AS9969NFHA3DIIYcAsPfee5fZV/8/tMp7kCbm9To1wZh+9/1o8riXXnoJ8ILptDK8VoHPNaa4DcMwAkaVUdyKX3lr4I2mstTEUInwq+gPPvgA8Nx8/IUEqjKaGKuqpjeNRZWopkzQiTgIliukTj7q5Jl+1zVw7Oabbwbiu7eqC59e+6abbppw21jOO+88wHOPCwL6v3zqqacCXqGRCRMmlNn2pJNCTnGqtFu3bg3kT2krprgNwzAChuS75BdASUmJ81eJznVV9EIg9nPI9T3QEGXth7pX5op8Xb+qTFWqxx13XGRdJhLtV4ZM3AO122pIv1aqP+aYY+KeIx56Xt1W7eTPP/98ZBtNqhSbZCodcv0d6NixI+DNhWkwEXgpEDTAbvjwUIxhtkdh/ntQUlJCaWlp3JtgitswDCNgmOIuEPKtuPNNsV8/2D0o9usHU9yGYRhVFntwG4ZhBAx7cBuGYQQMe3AbhmEEDHtwG4ZhBAx7cBuGYQSMggx5LwQXxXxT7Peg2K8f7B4U+/WXhyluwzCMgGEPbsMwjIBhD27DMIyAUZA27mIPdYXiuwfFfv1g96DYrx+St+ub4jYMwwgY9uA2DMMIGPbgNgzDCBj24C5yWrVqRatWrVi/fj3r169njz32iBRCNQyjMLEHt2EYRsAoSK8SI3d06dIF8GaztTzVEUccAcDChQvz0zHDMBJiitswDCNgmOIucpo2bRr1vlmzZgDssssugCluwyhETHEbhmEEDHtwG4ZhBAwzlSSgY8eOANx0001AqOIyeJN4zz77LADLli0D4J577onsO3v27Jz1M1U23XRTwLuuWCZOnJjL7hQMrVu3jiwfc8wxAOy0004ANGnSBIClS5cC8OKLLwLw8ssv57KLhmGK2zAMI2iY4sZTVPfff3+k7bDDDgNg4403Bsomf+nWrVvU+5NPPjmy3Lt3b8BTZIVI165dAYo22Gb77bcH4IwzzgDg1FNPBaBFixaRbbbccksgceKf0047DYBHHnkk0nb11VcDsGLFigz32DA8THEbhmEEjLQVt4hsDJQCPznnOotILWAs0BSYB3RzzhWk/Nhhhx0AeOKJJwDYf//9E25bWloKeLZtpXv37gDsueeekbYhQ4YAha24TzrppLjtsdcXdKpXrw54AUX16tUDYNiwYVHr06Fnz56R5V9//RWAfv36pX3cbNC+ffvI8ptvvgnAiBEjALjiiivy0iej8mRCcf8XmOl73xeY6JxrAUwMvzcMwzAyRFqKW0QaAZ2AQcDl4eZjgbbh5ceB94Cr0zlPpmnTpg0A48ePB6BGjRplttHAk9GjRwNw1113AbBkyZKo7d5++20AJk+eHGmrW7cu4HkhzJ8/P2N9Twf/qEBt+LEMHDgwV93JCQMGDADg4osvBrzk/Ins1j/++GNk+Z133gHghRdeADwPohtuuAHw5gn8qGr98ssvgcIbwfTv3z+yrPfg8MMPB7wRqP8eJEK9b/T/YtWqVQAMHToUgI8//jhDPY5m0KBBQNnPTz8b/3xDItasWRP3GEEiXcU9ArgK2OBrq+ecWwwQfq2b5jkMwzAMH5Lqr46IdAaOcc71FpG2QJ+wjXulc66Gb7sVzrmacfbvBfQCaNKkyb7+X/lslSzaZpttAPjhhx8AqFkzulsfffRRZPmCCy4AYPHixQAsX7683GNPnTo1styyZUsABg8eDMD1119fYd9yUbbJf3066lBUMbVq1QqARYsWZfz85ZHJ69eRDnifi9qyEynuTp06AdFKUe9JLFtssQUA1113HRBtz9bjfvXVVwCcffbZAEyfPr3CfmfzO3DssccC0fMusedbt25d1KvO60ybNg2AvfbaK7LtwQcfDHheV9rXDz74AIC2bdtWuo/JXP8///wTd9vKoKr8jz/+iDrP6tWrI9vcfffdUfv89ddfQMXPgXTxX1dJSQmlpaVxvwTpKO6DgK4iMg94BjhCRJ4ClopIfYDw67IEHRzlnCtxzpVst912aXTDMAyjuEjZxu2c6wf0A/Ap7tNFZBhwFjA0/PpKBvqZEdRPO1Zp66/p5ZdfHmn75ptvKnVstaECvPvuuwDst99+KfUzH/z3v/8FyirtOnXqAHDggQdG2mbODM1Fz5kzJ0e9S45NNgl9nf0jnK233jpqG1VXqqzVF3vBggVJn+fPP/8EPMWtnioA55xzDgB77703AGeeeSYAV111VdLHzyQ6r/H000+XWae2XlWRjRs3BmDzzTcHoF27doDnkZMMiUYpmUK/c6pM9d7Hfs7loZ+REm8Upv74ikbL6rzYo48+CnhzW3///XfS588E2fDjHgq0F5HZQPvwe8MwDCNDZCRy0jn3HiHvEZxzvwLtMnFcwzAMoyxFEfKuiZROOeWUqHZ17dOh4LfffpvyOXQo5ae8gJ5copNKfndA5bvvvgPgueeei2rXSjgHHHAAEG0OWLlyJQA9evQAYNy4cRnucWroEFj7BWUnsfr06QN4ScEyMcT1TzzGnk8TVeXLVHL00UcDnvlj7dq1kXXqEqrfgdq1awOeO6ua+vSa/AnJ/PcYPHPLeeedl9kLiEHzxCvaJ3Vl9HP66acDnglI0Tm1hg0bJn1e/f7rZLO+ahCT38yq9zObWMi7YRhGwKjSirt58+YAPP7443HX669mOko7CKjCUDc2P6o4dcJNlfbxxx8PwIYNG8rso5O7I0eOBOBf//oXkP9AI+1zPDSY5o477sj4eT///POE6/Kd4ldHk/r5XnPNNZF16rKoqHucTtROmTIlav3OO+8cWY511dMka7l2I1WXRX31o4FTsai76G677ZbwuLVq1QI8Ja2TzbHoiMZ/rA4dOgDZnbw3xW0YhhEwqrTiVvW46667RrU/9NBDgBeuXlXZdtttAe964wUtqOuiJko66qijAE9plxfooHY/TZGaL8VdrVo1wBth+VFXT01dkA38dmM932abbQZEp4nNBxriroUzNECmMmjgmj91sX4vvv/+e8D7HgUB/Z4m83199dVXAW+eSO3mOl+mboh+G7uO7mLruWYSU9yGYRgBo8opbi1iAIltWNlIgBPPY0O9L/KFehI0aNCgzDpViToDrvbqWHRU4lcnZ511FuAFvOQbtT9qhXo/mt7gySefzNr5169fH1nWkOxC4dNPP037GBqMoiMr8K5TFWhVRYOUNF2Evqo3ic6f+dMDN2rUKOv9MsVtGIYRMApDMmUAVZcahgyeXW/u3LmAN2t+2WWXAV4BhUykdzzxxBPLtN1+++1pHzdbqFo+99xz465/+OGHAS/Z1j777BNZp4pb0XB4f2rbfBAvKVG2Epb50dJ34JU7Ux577LGsnz9bqLeQP52DoiH05XnUVGVeeSWUyUPjNzJRkKMymOI2DMMIGFVGcWvRV3+En9rhtEyXRkpOmjQJgI022ihqu1RQO7pGyIGXHlIT0hQiWmIrUbHg2Ag4f+RfrG07Fza9ZIg3cspFsnz/Zx97PvWfDiKaeExHEf7Uy4Vami1X6ChLPbdyjSluwzCMgFFlFHe8QqfqW6plpBRV4OkobbWdqv3PH5WokWfq41qIaEmtSy65pNztdAQTL0WtqvZEHinFwpFHHlmmTQtwBNHGrdGC+n+iowj//1hsCb9iYccddwTgxhtvBLy0x36yVbbNjyluwzCMgGEPbsMwjIBRZUwl9evXL9N22223xd02NrlOKmhlE53E81f+0HWFTEXVeXSory6V/kkYHTprUEu+TULz5s0D4Oeffwa8pGjNMNQAACAASURBVFrZZvjw4UB0jUu9Nxq45K9jWOh07twZKFtJXUO4Nfy7mOnVqxcAp556alS7Pz2wJp7KJqa4DcMwAkbgFbe6s8VLWVqZOoIVoZORDzzwAOClhP3tt9+A6HSh6m6YbzQUW/voV83+GpLxUNUaz51Ole2VV16ZkX6mi6YS1RGAP6m9Jv/RlK/+KuepovfR7waoaEBKvMnyQkWTdGlNVh1laUpaHVXGS/FbLJx//vkAdO/eHSj7fzFs2LDIsgb6ZRNT3IZhGAEj8Ipb7YuqEjKNBpfcfffdAHTt2hXwVOyIESMAGDBgQFbOnw5aTqpbt24AjB07NrKuRo0alTrWvffeG1nOd3GAROhIwB/mrsEjardNRXFrwFGbNm0Ar1SbKm//+dTNcsWKFZU+T75Q27bOE+l8jaZs0DmEYkSTx2nAUexcmrbr8yFXmOI2DMMIGIFX3Dprr/Y3DWMHLwF8svgT8fft2xeALl26AJ7NV9XIpZdeCniJqgoZDUTyh7dr//32YPA8bhYuXAh4Cbl++umnyDaZKLCbDbTg8U033RRpiy1oMGPGDMBLkqR23Xho8WFVXbHFptXO6Q+y0SLEQUJHjcp9990HwIMPPpiP7hQEmsJWPWliCwu//PLLgHevcp3awBS3YRhGwJBcJOGpiJKSEucv9plKKk5VlYcddlikTcNyVRHpOTQFrNqv1ePAXxBUlZoqelWi119/PZB5z5HYzyEX6UgLiUxe/wknnBBZVk8T/Tz1uMl87yvaVlPfDhw4MNKWjidTLr4Dfp9zHXWoh5Haslu2bAnkXkXm+3/AP0+mMSAXXnhh1DYvvfQS4I3G/PEbmcB/D0pKSigtLY17E0xxG4ZhBIzA27iV4447DoCvv/460ta4cWPAs1dXxLp16yLLWmR1zJgxQP6jA43keeGFFyLLOuehn2eicnblocnI1JtGk2rpq790WaGjcxYABxxwAFA2EjbIqWjTQed9IPdKu7KY4jYMwwgYVUZxq1+133NiyJAhANStWxfwbNlqn1bPCfXtnTp1amRfU9hVA/U00Ve153bq1AnwRlnt2rUDvOLJftT/feLEidntbA7we04pv//+O1CYsQi5QP30/fNjihb81nmMfCttxRS3YRhGwLAHt2EYRsCoMqYSRYd9UHaCwTA++eSTqFelWM0EAHPmzAHSqwgVZDRBXbx0rFrN3e/0UAiY4jYMwwgYVU5xG4aRHLNmzQK8tK3Fypo1a4DoEXqfPn0A6N27d176VBGmuA3DMAJGWiHvIlIDeAhoBTjgHOBbYCzQFJgHdHPOlZvjMhMh70En3+G++abYrx/sHuT7+v3nU7v32rVrc9qHXIW83wm86ZzbFdgTmAn0BSY651oAE8PvDcMwjAyRso1bRLYBDgXOBnDO/QX8JSLHAm3Dmz0OvAdcXZljF0Liq3xT7Peg2K8f7B4U+/WXRzqKe0fgZ+BREflSRB4Ska2Aes65xQDh17rxdhaRXiJSKiKlWrnEMAzDqJh0HtybAPsA9zvn9gbWUAmziHNulHOuxDlXokUKDMMwjIpJ58G9EFjonPss/P55Qg/ypSJSHyD8uiy9LhqGYRh+UrZxO+eWiMgCEdnFOfct0A6YEf47Cxgafn2lsscuttl0yP+Mer4p9usHuwfFfv2QvF0/3QCci4HRIrIZ8APQg5CKf1ZEegLzgZPSPIdhGIbhI60Ht3PuK6Akzqp26RzXMAzDSIxFThqGYQQMe3AbhmEEDEsyZUShrplao1ErXxdjIqLNN98c8O7BySefDMANN9wAeKlhTznllDz0zihmTHEbhmEEDFPcRU6zZs0AL33l+eefD0C1atWAwksgn23q168fWZ4wYQKQuDK8hWQb+cIUt2EYRsAwxZ0hvv3228iyKrFdd901X92pkMMPPxyAZ555BoBatWpFrddq5/fcc09uO5Yn1J6tKhsSK+1ioUWLFgA88MADALRu3Tqy7s033wTgjDPOyH3HDFPchmEYQaPKKe7rrrsusnz66acD0L59ewAWLFiQ8fOp4mjcuHGkbd68eRk/TzrsvvvuAIwePTrS1qhRIwBq1KgRte3zzz8PwMUXXwzAL7/8kosu5h0tlDt58uRIW7Epbh116WffsmVLwPMo2mOPPSLbXnXVVTnuXW7ZcsstgbKj5pkzZwK5L7AQiyluwzCMgFFlFHeDBg0AOPvssyNtO+ywAwBXXx2q46Aq4Y8//sjYefWY6utbSKjSfvvttwGoV69eZJ3a4X/88UcAunbtCniKotg8JtavXw/ABRdcEGl79tlnAWjevDkAd911V+47lgPUs+iRRx4B4MUXXwSgW7duUdsV8pxNOlxzzTUA/Pvf/4606ei0b99QpmqNb4hV3Pr+rLPOyk1nw5jiNgzDCBhVRnGrTUpVth+10b366qtAtOdAqpxwwgmAZwctRIV66aWXAp7SXrNmTWTdvffeC3hqwwjx119/RZbfeustALbaaisAunfvDsD++++f+45liD333DOyfP311wPQqlUrAC666CIA3nnnnbj7NmnSJLL8xRdfZKuLWeO4444DYODAgQDssssugJc+1v8/vO+++0a16Tb6/67v99lnn6h2gDZt2mTnAnyY4jYMwwgY9uA2DMMIGFXGVHLooYfm9HwHH3xwTs9XGXTCVIeGy5aFqse1a+elSddJFaNi1HVSJ3uDzJw5cyLLN998MwBz584FYNWqVeXue+6550aWX3rppSz0Ljs88cQTgDf5qGZVNYPMmjUL8EwnfmJNoIne+ydu+/XrB8CQIUPS7nsiTHEbhmEEjCqjuI8//vh8dyFCvtRs9erVATjxxBMBz2WpS5cuee1X0NGEW1tvvXWee5I+/gnqqVOnJrWPBmltv/32kTZ1HSxkdOJdJ5VVHev/hQbPvfzyy0B08JW6/5WUhAp8/frrrwC88cYbAHTo0CHqXPq/BzBgwADAC1578MEHM3I9fkxxG4ZhBIzAK261V+lrPFRlZCLwRgN9VMXGQ5Py5Jodd9wRgL333huAp556Cii+1KyZRlVXLFOmTMlxT/JDp06dAPjtt98ibdlIH5EJ/O6tGjyjSltf9fP0J4aD6BGpLqvSVlSlq+LWeSR99Z9H5wQ0oCn2WOlgitswDCNgBF5xqzfJIYccknCbSZMmAfDhhx+mfb7yAn3yTawyHDlyZJ56UjWoWbMm4CVdikVTCVR1NNjs/vvvz3NPEqOeI2rPhsQ27Vilrfi9QNTTJBZVzU8//TTgBWfFm2PTIB4NXDLFbRiGUcQEXnErGoIaDw39zsb5Ntoo9Ns3ffr0yLpEv9bZQme0L7vssqi+pWPT1+vS2fXy2LBhA5DZuYR84f8e3XnnnYAXEq6o7bSqzx2oF42mRdZkbYVEnTp1AC+uwu9nrcux3iOJSOX/Nl5StlykvzDFbRiGETACr7g1mi3bv3L6y642dT2fqk1/khn15shVNKfa2TR6S/vWsWNHAKZNm1bpY5555plAcj6o69atA2DGjBmA58eqNuA///yz0ufPFvo5brJJ9Fd/s802A7zESwCnnXYa4N1PTUD13nvvZbubBcEll1wCeEp00aJF+exOXLRYitqR/SMm9aOuSGmng86bxRvxl2cFSBdT3IZhGAEj8Iq7V69eFW6jdlr1PU3GI0Rn0lU1165dG/BmimPxR6Q9+eSTFR4/FxxxxBGAl1di9uzZSe9bmbJdW2yxBeD5j6vf6qBBgwC48cYbkz5WpmnYsCEAF154IeAVSoiNgoyX2jMWjTRMZQQTJDbeeGPA800eMWIEEP0dLxQ0v0i8z02/h7nA7wMem/NEfctPOumkjJ3PFLdhGEbAsAe3YRhGwAi8qSQZtH7cBx98AHgTGskMj5PZBrwgH8hOUplU0DSu2jd/WledSEzEHXfcAXhBA37TSefOnQFvskqrgcdy+OGHAzB06NBIWzYnKjUZ1JFHHhlpe+ihhwDP1JUOGuB0zDHHALkdimeK/fbbL7J8+eWXA/DTTz8Bnpuj1k+sW7cukJmKUdlCTZn6f5rNCcHy8Af3qZOATU4ahmEYEQKvuJP5pW3atGnUq6JBJurSFw/dRpPNq3pVp35dn8/E8j///DMA//nPfwDPpU0nYdUF7uOPP47s8/zzzwOe655We1eWLFkCwK233lrmfLVq1QI8N0Cd/NXzqiuhpgPVya5so0o73mehoy1V/zpZqeq5Mtx3332ANwmVzYT5maJ+/fqAV7kevKIKWmBA07a2bdsW8D5PLcRRiJQ3OZlLygvAGTx4cMbPZ4rbMAwjYKSluEXkMuA/gAOmAT2ALYGxQFNgHtDNObcirV6WQ2zKxsqgSru8fdXup0lsVNlrUitV8f60jo8++mil+5IOeh2PP/444LltqTte8+bNAdhmm20i+5x99tlRr0rPnj0BePPNNxOeTxW1qnYNcddADb1HX375ZVR/Mo0Gzdx+++0AdOvWrcw2n332GQD9+/ePav/Xv/4V95iff/55ZLlPnz6Ad716b3QE89///heAo48+OrKPqtcVK7L2lU8J7Ze/PJkmRtL5jHPOOQfwqtvn+nucCmpb1pB3/8g7l+UM/efNhb09ZcUtIg2BS4AS51wrYGPgFKAvMNE51wKYGH5vGIZhZIh0bdybANVE5G9CSnsR0A9oG17/OPAekLXsNGrfVVWZCvPmzQNg8eLFkbaBAwcCiWfUX3vtNSBxys98okpY01nq+0033TSyTaJRhnphJONNo7a72G31Ndu2UVXNGlSj+D8THSmpmrzlllsAT60rsWXewAuZVq8LLSigtmBV3voa77iFQo8ePQD43//+F2lT9a0jI8XvIVXoqGfM66+/DkQnRYtX/DfT6EhbRzTgff/1+6OvmSRlxe2c+wm4DZgPLAZ+c85NAOo55xaHt1kM1I23v4j0EpFSESnVh69hGIZRMSkrbhGpCRwLNANWAs+JyOnJ7u+cGwWMAigpKUl5SlgTp48bNw6IH6qt6jnRL5+GqP/++++pdqMgURVy7bXXAvE9RLKB+jfffPPNWT3P0qVL47b7y1dpSt/GjRsDZRWxhrFrioN43xH9Xug6f9HcoKCjBk3KBp5tW72RdK5E/eGDwBdffAF46SzU99yPjjjVE0xHV+mgzx2dV/Lbs1Vx60gwG2Xe0vEqORKY65z72Tn3N/Ai8C9gqYjUBwi/Fq4vkWEYRgBJx8Y9HzhARLYE1gLtgFJgDXAWMDT8+kq6nSy3E/PnA7DHHnsAnnIC75cvGxFuOputngX5ithKhuHDhwPRpbbUf1vttumgquebb74B4LbbbgNg9erVaR+7PLQE1dixYwE4+eSTAc9nuTwefvhhwPMaKs8LRBX3+PHjgbKFFfxRqIVaREJt+y+88EKkTecgDjroIACGDRsGJI6ELWR0vkVH1+DZuNX+PHnyZMBTwLH+/v7nhP4/axSk/r+rTVuVdrw5IE04lc3YjpQf3M65z0TkeeALYD3wJSHTR3XgWRHpSejhnrmUWIZhGEZ6XiXOuf5A/5jmdYTUt2EYhpEFJN+hohCanCwtLY28L2SzQyzr168H4I033oi0de3atdLHif0csn0PNBhHh5N+ExN4Yc86AejPN6xo1W/1CkrHTJDO9etkoU6GquufHw0OuummmwB47rnnKt1HTSHw/vvvA54pyJ+8K9GEaTLk+jsQi05ia477ZPLWZ5JMXL/fNbNfv36AV4tVJ19j3Vfjub5WtI2+1+++mgvBmwRNpaq7vw8lJSWUlpbGvQkW8m4YhhEwTHGnSVAVd6FR7NcP+b8HOhGnLpKa3nXMmDE5OX+2rv+oo44CvElKHVGko7hVTWtd19ggplQxxW0YhlFFCXxaV8MwMsMPP/wAwDPPPAOkZqMtRDRthb727t0b8Fz7NGjPH7Yei6YF1rmSfBdLMcVtGIYRMMzGXSDk276Zb4r9+sHuQbFfP5iN2zAMo8piD27DMIyAYQ9uwzCMgGEPbsMwjIBhD27DMIyAYQ9uwzCMgGEPbsMwjIBhD27DMIyAUZAh74UQFJRviv0eFPv1g92DYr/+8jDFbRiGETDswW0YhhEw7MFtGIYRMArSxl3syWWg+O5BsV8/2D0o9uuH5O36prgNwzAChj24DcMwAoY9uA3DMAKGPbgNwzAChj24DcMwAkZBepUYhpE/Fi1aBMD2228PeIVxzzvvvLz1yYjGFLdhGEbAsAe3YRhGwDBTiWEYUWgQyIYNG4BQtfGqxA477AB41zl//nwA9thjj8g2hx56aNx9TzjhBACGDx8OwGuvvZa1fpaHKW7DMIyAYYo7SWrVqgXAnDlzABg6dCgAt956a976lAk23XRTALp37w7AkCFDAKhbty4Af//9NwB9+vSJ7HP//fcD8M8//+Ssn0b2adeuHQA1atTIc0+yy+jRowFo0qQJAH/88QcQfd21a9eOu6+G4e+zzz4A3HjjjQC88MILkW1UwWcTU9yGYRgBo6gU9yabhC63f//+ANx8882ApyrLQ9XINttsA8BOO+2UjS7mjM033xyAsWPHAtC5c2cAlixZAsAHH3wAePfs4osvjuy73XbbAd59DCI6grjlllsAmDZtGuCpL4AFCxYA8PzzzwMwfvx4AFavXp2zfuaC+vXrA3D88ccDsNlmm0Wtf+KJJ3Lep2zQtWtXAFq3bg3AlltuGbX++++/jywvX7487jE22iikdVWt64i7R48ekW38tvJsYYrbMAwjYFSouEXkEaAzsMw51yrcVgsYCzQF5gHdnHMrwuv6AT2Bf4BLnHNvZaXnKXDNNdcA0K9fPwC23nprAC699NIK9z311FOj3k+cODHDvcstL774IgAdOnQAYP369QD06tULgDfeeCNqe7/Nb9iwYbnoYlbp2LEj4HkWtGrVqsw2+++/P+B5EqjSHjRoEFA17gN4166fvaIjjJEjR+a8T9lA/89jlbay6667Jn2sWbNmAdC8eXMAdtttt8g6/b747d6ZJhnF/RhwdExbX2Cic64FMDH8HhHZHTgFaBne5z4R2ThjvTUMwzAqVtzOuUki0jSm+VigbXj5ceA94Opw+zPOuXXAXBGZA7QBPslMd9OjcePG5b6Px1FHHQXA0UeHfrtmz54NZPfXNFuceeaZkeUjjzwSgD///BOA008/HSirtJVff/01sly9evVsdTHrqG2/adOmUe0jRowAYMyYMZG2Tp06AZ4S69atGwDXXnst4HnXBN3mrT7LsYUL9Luwbt26nPcpk+j1HXLIIXHXv//++5U+pu6jitvPwQcfDORfccejnnNuMUD4tW64vSGwwLfdwnBbGUSkl4iUikjpzz//nGI3DMMwio9Me5XEqzUUtxaPc24UMAqgpKQkuXo9abLzzjtHvU+kLv2oPVxn2tW2pTbhIFCzZk0AbrvttkjbxhuHLFjPPvssAC+//HLSxwtysqFGjRoBXvScqmW1Vy9dujSy7ZQpU6K2VcWtI4569epFHSNobLHFFgAcfvjhgGfvVyGlI4qgorbsRHNYa9asAbzRVmV46aWXADjnnHNS7F16pKq4l4pIfYDw67Jw+0LAb39oBCxKvXuGYRhGLKkq7leBs4Ch4ddXfO1Pi8hwoAHQApicbiczhdq4fv/9dwAmTJhQ4T6qqtT+99FHH2Wpd9njyiuvBKI9Q1RtXHLJJZU+3ooVKzLTsTzg90cHLxLWr7STZccddwSi/X+DhI5A1XtGGTBgQD66k3EaNGgAQJcuXeKunzp1KgDjxo2r9LFLS0sBb1S27777ptLFlEnGHXAMoYnIOiKyEOhP6IH9rIj0BOYDJwE4574RkWeBGcB64ELnnMVFG4ZhZJBkvEpOTbCqXYLtBwGD0umUYRiGkZiiCHnXUF6dfNEkMxrSHI8DDzwQ8NzGdN8gugGeeOKJgHcN4IVxB9nskQoaqqymr2QmmdWUoPtoWoAPP/wwG13MGbvvvnvcdv1uxG43Y8aMrPcplwwcODDlfTXtQ506dcqsO+ywwwDPNOl3pc0UFvJuGIYRMIpKcSu//fZbhftoukZ1A/zss88AWLhwYWY7l0VUXWqQgF9x+9O0xkMTD6nL5MMPPxxZd88992S0n7lEJ2X1Xtx9990V7nPQQQdF7TNz5kwA1q5dm40u5gwNKould+/egOf2Wa1aNSA6AVffvn0BePLJJ7PZxYwQG1hUUXsyqItobCAXwF577QXAtttuC5jiNgzDMKjiiltTr2rSFyVR4I2/RJOGhKvKuuOOO4DkUsAWCldddVXUe1WbUHEY80UXXQR4KTAT2UODhro/PvXUU0D54c4aaHPWWWdFtX/77bdZ6l1uUcUZqzyvv/56wEthqiXMNCkbwKOPPgp46X/nzZuX1b6mg3+kCd5nPmnSpJSPed1118U9NsBjjz0GZLeggiluwzCMgFGlFbeGdWt5LkW9Sn788ceodi3XFQ9NHKOq47vvvgO8ckevv/56BnqcXb788svIsl99+9EwaPWqUaqKylQvmrfeqjjbsI66VHn/9NNPgFeAI4hoIRDwPuN4qhHgq6++ArzEXLEpIwDOP/98wLN5BwG11acyRxFbjCEeOoeWzbQYprgNwzACRpVW3PqLumrVKsCb5dVEQ/oaj1i7X2yotPLxxx8Dham41U9dr8Wf1vKyyy4DvHukiYY0hWlsYQFNcQpw5513ZqnHhYXOayiafCmV8PhC4YwzzogsN2vWLO42WjRE/bm1FJeGdxcz6qmVqBhDrjDFbRiGETCqtOLWWV1VDOqfqvYptfdpMhr1VwXYfvvtAc/+pwUHYv2A1d+7ELnvvvsArxCw326dqOyWRgW+/fbbgGfn/de//hXZRu+NbluV8BfXUF9d/ayD7L+uqGIsj1QKCxQysaPnVPy3tRjDXXfdBXieNvHIRUStKW7DMIyAUaUVt6K2Xi2KEEvDhqEiPX6vC2X69OmA58ur+RqC4M+ttv0jjjgC8Mqwgefbrukply9fDngFFTRNpSpu9TaBsl46VQl/Un1V2pq29bnnnstLnzKJft6ZIgjFFmK9ZhJ50cQjthiDKu3YY/jzuOQin5EpbsMwjIBhD27DMIyAURSmkorQSUl/SK+iwRZff/11TvuUSdSs43dZrMh9UZNpaWi8BmFA/PtUVWjbtm2ZNq3VmShoKUi89957kWX9bP1mMPAm7zTkvUOHDlHtfoKeaCsRmoBL00YkqhCvz4XYRHbZxhS3YRhGwDDFTeLQePAmJ4sNTQcQT3FrYYYgh37HoiHb/tGE1iZ9/PHH89KnbKBBROAF2HTv3j1qG00ott9++wEwePBgIHpCTlMHBGGSPha9Pg02mzVrFhA9MX366acDXtBeLDoZqUo7mwml4mGK2zAMI2CY4gZ69OgBRNvw0kmyXpVQJeFPquNPf1tVOPfccwHPrgvw2muvARWnwA0qiVKxavCV/g/Ec58755xzgMIufbdy5UoApk2bBniBeFr8IHY07f/sEwXY6LHUTTYbRRKSwRS3YRhGwDDFjRfOHk9ZqNLUNK7FhioMf9KpRDPsQURHD/FKUMUWoqhqaDk6Vc9ari4WVdX33ntvpO2dd97Jcu/S55dffgGgffv2ALz77rtA4qIgfpWdKMBm5MiRQP6UtmKK2zAMI2CY4gbGjBkDeOWI/JRXXKEYiJcMXj0vNtlkk4TbBIVBgwYBnsKaMGFCZF1VTKLlR1NBdOnSBYABAwYA0LFjR8BLNnXDDTcA8NFHH+W6ixlB1bH64z/wwANA+akb1Gd/6tSpgOdlkmvvkUSY4jYMwwgYprgrYOLEifnuQl755JNPgOgE/IomrUpUfLmQUR9ef7paKOw0vdlCo/+0LFdV5cknnwS8z768OQxNKqWFfwsNU9yGYRgBwx7chmEYAcNMJXiO+v5agrNnzwa8is3Fioa+T5o0KdKm1UCWLVuWlz5lgiuuuAKIrnoEVaeavZGYa6+9Nuo1iJjiNgzDCBimuPGUtlbCMTzeeuutqNeqwo477hj1XoM1gpg0ySg+THEbhmEEDFPcRlHSrl27fHfBMFLGFLdhGEbAqPDBLSKPiMgyEZnuaxsmIrNEZKqIvCQiNXzr+onIHBH5VkQ6ZKvjhmEYxUoyivsx4OiYtreBVs65PYDvgH4AIrI7cArQMrzPfSKyccZ6axiGYVRs43bOTRKRpjFtE3xvPwVODC8fCzzjnFsHzBWROUAb4JPKdCpeetVio9jvQbFfP9g9KPbrL49M2LjPAcaHlxsCC3zrFobbyiAivUSkVERK/XXwDMMwjPJJ68EtItcC64HR2hRns7g/m865Uc65EudcyXbbbZdONwzDMIqKlN0BReQsoDPQznljmoVAY99mjYBFqXfPMAzDiCWlB7eIHA1cDRzmnPvDt+pV4GkRGQ40AFoAk1M4firdCjSx9rxiuwfFfv1g96DYrx+St+tX+OAWkTFAW6COiCwE+hPyItkceDt8cz91zp3vnPtGRJ4FZhAyoVzonPsnpSswDMMw4pKMV8mpcZofLmf7QcCgdDplGIZhJMYiJw3DMAKGPbgNwzAChj24DcMwAoY9uA3DMAKGPbgNwzAChj24DcMwAoYVUqgALWf29NNPA3DQQQcB8PDDIY/I8847Lz8dMwyjaDHFbRiGETBMcSdAlfb9998PwIEHHgjAhg0bAEs5aVRd2rdvD8CNN94IwAEHHJBw2wULQslAb775ZgAee+wxwPs/CSo1a9YEoH79+gA0b94cgKOOOipqu1NOOQWAWrVqRdpinw2LFoXSNbVp0waAJUuWpN0/U9yGYRgBwxR3AnbYYQcAjj46tviPUZXQkZSqoUsuuSSyrmnTpgCsWrUKgIsvvhiAJ598Moc9TI8ePXoAsOeeewIwc+bMyLoHHngAgDp16gBw2WWXAXDFFVcAsG7dOgAWL16c8PhbbrklAKNGjQK8kergwYMB+Oefwk9V1KVLCAN59AAAEH5JREFUl6hXgEMOOQSAFi1aABWPsP3rY7dV1a4q3hS3YRhGEWIPbsMwjIBhphIfTZo0iSyPGzeu3G07d+4MQMeOHQEYP358eZsXLNWqVQPg5JNPBuD4448HoFOnTgC88847AFx++eWRfebPnw94JoQgoSawnXbaCYAnnngCgC+//BLwzCMAc+fOBWD77bcHvIlqHfreeuut2e9wiuy6664AXHfddYB33ZMmTYpso20XXHABAFtvvXXUMfbbbz8Apk2blvA8++67LwAvvfQSAP379wc899nvv/8+javILGqqUHNY165dAWjdujVQfv7vlStXAvD3339Hteuk5Cab5PZRaorbMAwjYJji9uGfdNpmm22AxG5NqsiDqrRVcapqVPWxevVqAEaOHAl4AUd+pfbII48AcOWVV+amsxng0EMPBeCZZ54BoG7duoA3eaajiA4dOkT20SLWu+22GwAPPfQQ4LmAqQIvpJFHjRo1AHj99dcBT1Ureh9ilwGmT58OwC233AKUr7SVKVOmAN49eeuttwCoV68ekF/FvemmmwJw0kknAXDPPfcA3v+2TiL++OOPAMybNy+yr35PVqxYAcD7778PwC+//BJ1jqlTpwKw++67J+yHqvU1a9akeCVlMcVtGIYRMExxFxHHHXdcZFnVsqpJDd2fMGEC4AVW3HnnnQBceOGFkX233Xbb7Hc2Dfy2SnVXu+uuuwAYMWIE4LkB3nvvvYBny4/H119/DcD5558PwMEHHwzAa6+9BsC3334b2fbSSy8FYO3atWleRWpsscUWQFml/fHHHwPR8ziNGjUC4O233wbg2GOPBeCvv/6q9Hn1+N999x3guSFqez7Q0cfjjz8e1a4qWkcJ+vnGqul46P3V61O7eTx0lKrb6qguE5jiNgzDCBimuIE+ffoAsNdee1W4rXof3HDDDVntUyZR9aWqE6B69eoAnHjiiQBMnDgxah+dLVdF7kc9TgoN9Yro27dvpO2MM84APKWZjkdIaWkp4CluDdLQ9+AFbBxxxBGVPn4m0OuNRW20/fr1K7Nu6NChQGpKOwjEeouo50vsd7482rVrB8CgQaFyuiUlJVHr/fdO7fynnhoq1/vnn39WsscVY4rbMAwjYJjiBmrXrg14Ps3loSHwaicLAsOGDQOi7XFq706kOoYMGQLEV45qJy401JPBr7iVww8/HID//e9/lT7umWeeCXgh4Uo8v9/DDjsM8HygP//880qfLx005DyWX3/9FYi242qo+8KFCzPej2bNmgGw+eabR9o0hD5XqDeHpmA+55xzAM+PW0PPX3zxxYTH0Dme7t27A978TmxYe+/evSPLmmgrm5jiNgzDCBhFrbjV9zLWn7WqoLZotWOrXzJ4ngQ6yth7770BT62WZ8f+8MMPM9/ZDKDX6VdDqjBTUdqKqsdEiYb87XqP1Tc43yxfvhzwvGZ23HHHyDpViepRdNVVV2XsvG3btgU8zw6ApUuXZuz4yaBRjhdddBHg2aU14VZssjC/z/mYMWMA2HnnnQHvM9ZX9d9WO/bs2bMzfwHlYIrbMAwjYBS14tbk6GqPrGpo9NzkyZMBT1UDfPPNNwBsttlmgKcq169fD3gKVRWatoPnWVNoqOL2o0nsU0FtwP77VhFqV122bFnK500HjWRU1HdYoyLVgwpgq622AuCHH35I+7w6F9KyZcu0j5VpVHlrBOV7770HeB5GY8eOrfAY6uutUaXPPfdcprtZKUxxG4ZhBAx7cBuGYQSMojSVaApGDS7ZaKOyv1+xbVo5xG8yCApnn3024AUPxEMnW4YPHw7ArFmzAG9CSd9DdgIK0mGXXXYBvImkeOtS4YMPPgC8oJpE+L8TX331VcrnywQ66azcfvvtUe/9gSK9evUCMlMfUk1u+r+lE3+5npAsDzUJvfLKK4CXwqA8dFsNbPrjjz+y1LvKYYrbMAwjYBSl4tbE8QMHDgTKVxy6Tt3kCimFZ7KoWj7hhBOS3sdfe7HQ0SRPo0ePBuDcc8+NrHv++ecrdSz/BKe/qEJ53HbbbZFlLVyQLzSkXYODYhP/+8lEPUhNp3DWWWdFnffdd99N+9iZRp0R1P23vMIJ6uaX70nIRJjiNgzDCBhFqbiTsW0pGg6b69DlfOMPVYbcBxikwgsvvAB4tluA008/HfCUqCaz17SuWjhBlbY/ZLyiyt5vvvkmkH+V7Sc2UOTII48EKj/ySBadJ1JXOz1vPtO5Kvod1v/3m2++GfDcIMv7fI866ijAFLdhGIaRISpU3CLyCNAZWOacaxWzrg8wDNjOOfdLuK0f0BP4B7jEOfdWxnudIlr0VUsalcdnn30GeEnQ85UYP9+oHfCjjz7Kc08qRm35/nkITV+rXkFq19XyVbH4VVhFilvLuxUSscnPtPScFpTIlFeEho0/8MADUe1aiCNfAUj+kWKXLl2A6DkI8L4DGpimKRz8wUnqiTVq1Cig8EbcySjux4CjYxtFpDHQHpjva9sdOAVoGd7nPhHZOCM9NQzDMIAkFLdzbpKINI2z6g7gKuAVX9uxwDPOuXXAXBGZA7QBPkm/q+mjNq5kvAXUN7dYlbYSm1SnkNH0pOoRAJ5femyyoER88cUXkWUtw3XyySdHbaPeK1q6rJDQgrhaak5t0GrDf+KJJ1I+to5YAV599VWgbBpZnUvIl/eVqmzwEkXFon3U74b6oPsTcB1//PGAV0Q7iIq7DCLSFfjJOfd1zKqGwALf+4XhtnjH6CUipSJS6s9aZxiGYZRPpb1KRGRL4FrgqHir47TFlTjOuVHAKICSkpLyZVCaaGkptfclg5ZzKla0VJOiZbuCwPjx48ssayIxTe0ZG2WpiYf8KWufeuqpqG1+++03oLB93LVggipitdVqEYFUFLfOCWmqViirtNW2rl47uUaLhFx++eUJt7n++usBT2krGk167bXXRtq08IbOcd1///1AeknLMkkq7oDNgWbA1+GJq0bAFyLShpDCbuzbthFQGFdqGIZRRaj0g9s5Nw2oq+9FZB5Q4pz7RUReBZ4WkeFAA6AFMDlDfU0ZzTWRKOeEqgT9dYVo1VZMaCTcPvvsA3j3IUil2uKhNsqKbJVXXnllZNmvMMHLA6LKO0jsscceAPz73/+OtL388stJ7XvppZcCXjk7P/q90NJec+bMSaufqaLzV23atIm0rV69GvCiOjXvSCL8fdccK5qmdqeddgIKR3FXaOMWkTGEJhd3EZGFItIz0bbOuW+AZ4EZwJvAhc659ONqDcMwjAjJeJWcWsH6pjHvBwGJ09AZhmEYaVGlQ941+EKDBRIxYMAAIPmhY1VGJ51q1aoFeBM3FbnRVRVat24dWVY3MTWl3XDDDXnpUyrcddddgBfy3qhRIyA6PF+rHCUy/ahrYbzKQp98EvLw1f8dDbzJN/7vqQYbVWQiKe84mUh5mw0s5N0wDCNgVGnFrW5MdevWrWBLQ9EJWg11z1focq6pXbs24LkLgqe67rzzTsALyAkCGjClASQaLLTXXntFtvnvf/8LwIIFodCLGTNmAN5knlZ/jzfaGjFiBFA4SjteoJwmk4odKel3O/a6NLEUeO6iGmPy448/Zq6zGcAUt2EYRsCo0opbXZVUWWiinWOOOSZvfQoKmoinWOz+mmDIX+pMFWhsObAgocpbU92+/vrrkXWxSlRtwtWqVYvbrmHg4JV1KxTU1u6fo1D7vgbeKIkUdzw0CMsUt2EYhpEWVVpxK2qn8gcfGPFp0qQJAMuXLwfgrbcKJitvVvErNUXTtk6aNCnX3ck4OmrYe++9I239+/cHvP8LHZEqRx8dSgqqaQAKOeGaJrW65pprIm1a/ENt11q6LBlUpcemrS0UTHEbhmEEjKJQ3EbFqDeF2gWLLWPj3Llzy7QlWyw4CKg/8rRp0yJt8fyzg86XX34Zd7mqYYrbMAwjYJjiNgCYP39+1Hu/90ExcPHFF0e9GkYhY4rbMAwjYNiD2/h/e3cXIlUZx3H8+2MtSyPUzDJX0kIqk0qRsBciskhN9NZIEOoyyKIoRQi6Lnq56IWwFynRC7MSoVAs6CrTLM1Sc03RNUsjeqEglX5dnGd1GmZsJWaf57D/Dyw755wRvjs75+/MM7O7IYSaiaWSAJz+0faurvjbziGULh5xhxBCzRT5iHuw/ArRMxnst8Fg//ohboPB/vWfSTziDiGEmonBHUIINRODO4QQakYlrCNJOgb8AfyUu6WfRhOtnRCtnRGtndHp1sttX9zqQBGDG0DSVtvT//ua+UVrZ0RrZ0RrZ+RsjaWSEEKomRjcIYRQMyUN7ldzB5yFaO2MaO2MaO2MbK3FrHGHEELon5IecYcQQuiHIga3pFmS9kjqkbQkd08jSeMlfSxpl6SvJS1O+0dJ2ihpb/o8MncrgKQuSV9IWp+2i+wEkDRC0hpJu9Pte1OJvZIeSd/7nZJWSTqvpE5Jr0s6Kmlnw762fZKWpnNtj6S7M3c+nb7/OyS9K2lE7s52rQ3HHpNkSaNztWYf3JK6gBeB2cBk4F5Jk/NW/ctJ4FHb1wAzgAdT3xJgk+1JwKa0XYLFwK6G7VI7AV4APrR9NXA9VXdRvZLGAQ8B021PAbqABZTV+SYwq2lfy750310AXJv+zUvpHMzVuRGYYvs64FtgaQGd0LoVSeOBu4CDDfsGvDX74AZuBHpsf2f7OLAamJ+56RTbR2xvS5d/pxou46gaV6SrrQCy/wl5Sd3APcDyht3FdQJIuhC4DXgNwPZx279QZu8Q4HxJQ4BhwPcU1Gn7E+Dnpt3t+uYDq23/ZXs/0EN1DmbptL3B9sm0+SnQnbuzXWvyHPA40Pji4IC3ljC4xwGHGrZ7077iSJoATAU2A5fYPgLVcAfG5Cs75XmqO9XfDftK7AS4AjgGvJGWdpZLGk5hvbYPA89QPcI6AvxqewOFdbbQrq/k8+1+4IN0ubhOSfOAw7a3Nx0a8NYSBrda7CvurS6SLgDeAR62/VvunmaS5gJHbX+eu6WfhgDTgJdtT6X6lQclLeMAkNaG5wMTgcuA4ZIW5q36X4o83yQto1qWXNm3q8XVsnVKGgYsA55sdbjFvo62ljC4e4HxDdvdVE9FiyHpHKqhvdL22rT7R0lj0/GxwNFcfcktwDxJB6iWm+6Q9DbldfbpBXptb07ba6gGeWm9dwL7bR+zfQJYC9xMeZ3N2vUVd75JWgTMBe7z6fcnl9Z5JdV/3tvTOdYNbJN0KRlaSxjcW4BJkiZKOpdqkX9d5qZTJIlqHXaX7WcbDq0DFqXLi4D3B7qtke2ltrttT6C6DT+yvZDCOvvY/gE4JOmqtGsm8A3l9R4EZkgalu4LM6le5yits1m7vnXAAklDJU0EJgGfZegDqneUAU8A82z/2XCoqE7bX9keY3tCOsd6gWnpfjzwrbazfwBzqF5R3gcsy93T1HYr1dOeHcCX6WMOcBHVq/V70+dRuVsbmm8H1qfLJXfeAGxNt+17wMgSe4GngN3ATuAtYGhJncAqqvX3E1QD5YEz9VE95d8H7AFmZ+7soVof7ju3Xsnd2a616fgBYHSu1vjJyRBCqJkSlkpCCCGchRjcIYRQMzG4QwihZmJwhxBCzcTgDiGEmonBHUIINRODO4QQaiYGdwgh1Mw/qh1ppwUJm5kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "\n",
    "# Images are usually in the [0., 1.] or [0, 255] range, Normalize transform will bring them into [-1, 1] range\n",
    "# It's one of those things somebody figured out experimentally that it works (without special theoretical arguments)\n",
    "# https://github.com/soumith/ganhacks <- you can find more of those hacks here\n",
    "transform = transforms.Compose([\n",
    "        transforms.ToTensor(),\n",
    "        transforms.Normalize((.5,), (.5,))  \n",
    "    ])\n",
    "\n",
    "# MNIST is a super simple, \"hello world\" dataset so it's included in PyTorch.\n",
    "# First time you run this it will download the MNIST dataset and store it in DATA_DIR_PATH\n",
    "# the 'transform' (defined above) will be applied to every single image\n",
    "mnist_dataset = datasets.MNIST(root=DATA_DIR_PATH, train=True, download=True, transform=transform)\n",
    "\n",
    "# Nice wrapper class helps us load images in batches (suitable for GPUs)\n",
    "mnist_data_loader = DataLoader(mnist_dataset, batch_size=batch_size, shuffle=True, drop_last=True)\n",
    "\n",
    "# Let's answer our questions\n",
    "\n",
    "# Q1: How many images do I have?\n",
    "print(f'Dataset size: {len(mnist_dataset)} images.')\n",
    "\n",
    "num_imgs_to_visualize = 25  # number of images we'll display\n",
    "batch = next(iter(mnist_data_loader))  # take a single batch from the dataset\n",
    "img_batch = batch[0]  # extract only images and ignore the labels (batch[1])\n",
    "img_batch_subset = img_batch[:num_imgs_to_visualize]  # extract only a subset of images\n",
    "\n",
    "# Q2: What's the shape of my image?\n",
    "# format is (B,C,H,W), B - number of images in batch, C - number of channels, H - height, W - width\n",
    "print(f'Image shape {img_batch_subset.shape[1:]}')  # we ignore shape[0] - number of imgs in batch.\n",
    "\n",
    "# Q3: How do my images look like?\n",
    "# Creates a 5x5 grid of images, normalize will bring images from [-1, 1] range back into [0, 1] for display\n",
    "# pad_value is 1. (white) because it's 0. (black) by default but since our background is also black,\n",
    "# we wouldn't see the grid pattern so I set it to 1.\n",
    "grid = make_grid(img_batch_subset, nrow=int(np.sqrt(num_imgs_to_visualize)), normalize=True, pad_value=1.)\n",
    "grid = np.moveaxis(grid.numpy(), 0, 2)  # from CHW -> HWC format that's what matplotlib expects! Get used to this.\n",
    "plt.figure(figsize=(6, 6))\n",
    "plt.title(\"Samples from the MNIST dataset\")\n",
    "plt.imshow(grid)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understand your model (neural networks)!\n",
    "\n",
    "Let's define the generator and discriminator networks!\n",
    "\n",
    "The original paper used the maxout activation and dropout for regularization (you don't need to understand this). <br/>\n",
    "I'm using `LeakyReLU` instead and `batch normalization` which came after the original paper was published.\n",
    "\n",
    "Those design decisions are inspired by the DCGAN model which came later than the original GAN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Size of the generator's input vector. Generator will eventually learn how to map these into meaningful images!\n",
    "LATENT_SPACE_DIM = 100\n",
    "\n",
    "\n",
    "# This one will produce a batch of those vectors\n",
    "def get_gaussian_latent_batch(batch_size, device):\n",
    "    return torch.randn((batch_size, LATENT_SPACE_DIM), device=device)\n",
    "\n",
    "\n",
    "# It's cleaner if you define the block like this - bear with me\n",
    "def vanilla_block(in_feat, out_feat, normalize=True, activation=None):\n",
    "    layers = [nn.Linear(in_feat, out_feat)]\n",
    "    if normalize:\n",
    "        layers.append(nn.BatchNorm1d(out_feat))\n",
    "    # 0.2 was used in DCGAN, I experimented with other values like 0.5 didn't notice significant change\n",
    "    layers.append(nn.LeakyReLU(0.2) if activation is None else activation)\n",
    "    return layers\n",
    "\n",
    "\n",
    "class GeneratorNet(torch.nn.Module):\n",
    "    \"\"\"Simple 4-layer MLP generative neural network.\n",
    "\n",
    "    By default it works for MNIST size images (28x28).\n",
    "\n",
    "    There are many ways you can construct generator to work on MNIST.\n",
    "    Even without normalization layers it will work ok. Even with 5 layers it will work ok, etc.\n",
    "\n",
    "    It's generally an open-research question on how to evaluate GANs i.e. quantify that \"ok\" statement.\n",
    "\n",
    "    People tried to automate the task using IS (inception score, often used incorrectly), etc.\n",
    "    but so far it always ends up with some form of visual inspection (human in the loop).\n",
    "    \n",
    "    Fancy way of saying you'll have to take a look at the images from your generator and say hey this looks good!\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, img_shape=(MNIST_IMG_SIZE, MNIST_IMG_SIZE)):\n",
    "        super().__init__()\n",
    "        self.generated_img_shape = img_shape\n",
    "        num_neurons_per_layer = [LATENT_SPACE_DIM, 256, 512, 1024, img_shape[0] * img_shape[1]]\n",
    "\n",
    "        # Now you see why it's nice to define blocks - it's super concise!\n",
    "        # These are pretty much just linear layers followed by LeakyReLU and batch normalization\n",
    "        # Except for the last layer where we exclude batch normalization and we add Tanh (maps images into our [-1, 1] range!)\n",
    "        self.net = nn.Sequential(\n",
    "            *vanilla_block(num_neurons_per_layer[0], num_neurons_per_layer[1]),\n",
    "            *vanilla_block(num_neurons_per_layer[1], num_neurons_per_layer[2]),\n",
    "            *vanilla_block(num_neurons_per_layer[2], num_neurons_per_layer[3]),\n",
    "            *vanilla_block(num_neurons_per_layer[3], num_neurons_per_layer[4], normalize=False, activation=nn.Tanh())\n",
    "        )\n",
    "\n",
    "    def forward(self, latent_vector_batch):\n",
    "        img_batch_flattened = self.net(latent_vector_batch)\n",
    "        # just un-flatten using view into (N, 1, 28, 28) shape for MNIST\n",
    "        return img_batch_flattened.view(img_batch_flattened.shape[0], 1, *self.generated_img_shape)\n",
    "\n",
    "\n",
    "# You can interpret the output from the discriminator as a probability and the question it should\n",
    "# give an answer to is \"hey is this image real?\". If it outputs 1. it's 100% sure it's real. 0.5 - 50% sure, etc.\n",
    "class DiscriminatorNet(torch.nn.Module):\n",
    "    \"\"\"Simple 3-layer MLP discriminative neural network. It should output probability 1. for real images and 0. for fakes.\n",
    "\n",
    "    By default it works for MNIST size images (28x28).\n",
    "\n",
    "    Again there are many ways you can construct discriminator network that would work on MNIST.\n",
    "    You could use more or less layers, etc. Using normalization as in the DCGAN paper doesn't work well though.\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, img_shape=(MNIST_IMG_SIZE, MNIST_IMG_SIZE)):\n",
    "        super().__init__()\n",
    "        num_neurons_per_layer = [img_shape[0] * img_shape[1], 512, 256, 1]\n",
    "\n",
    "        # Last layer is Sigmoid function - basically the goal of the discriminator is to output 1.\n",
    "        # for real images and 0. for fake images and sigmoid is clamped between 0 and 1 so it's perfect.\n",
    "        self.net = nn.Sequential(\n",
    "            *vanilla_block(num_neurons_per_layer[0], num_neurons_per_layer[1], normalize=False),\n",
    "            *vanilla_block(num_neurons_per_layer[1], num_neurons_per_layer[2], normalize=False),\n",
    "            *vanilla_block(num_neurons_per_layer[2], num_neurons_per_layer[3], normalize=False, activation=nn.Sigmoid())\n",
    "        )\n",
    "\n",
    "    def forward(self, img_batch):\n",
    "        img_batch_flattened = img_batch.view(img_batch.shape[0], -1)  # flatten from (N,1,H,W) into (N, HxW)\n",
    "        return self.net(img_batch_flattened)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GAN Training\n",
    "\n",
    "**Feel free to skip this entire section** if you just want to use the pre-trained model to generate some new images - which don't exist in the original MNIST dataset and that's the whole magic of GANs!\n",
    "\n",
    "Phew, so far we got familiar with data and our models, awesome work! <br/>\n",
    "But brace yourselves as this is arguable the hardest part. How to actually train your GAN?\n",
    "\n",
    "Let's start with understanding the loss function! We'll be using `BCE (binary cross-entropy loss`), let's see why? <br/>\n",
    "\n",
    "If we input real images into the discriminator we expect it to output 1 (I'm 100% sure that this is a real image). <br/>\n",
    "The further away it is from 1 and the closer it is to 0 the more we should penalize it, as it is making wrong prediction. <br/>\n",
    "So this is how the loss should look like in that case (it's basically `-log(x)`):\n",
    "\n",
    "<img src=\"data/examples/jupyter/cross_entropy_loss.png\" alt=\"BCE loss when true label = 1.\" align=\"left\"/> <br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BCE loss basically becomes `-log(x)` when it's target (true) label is 1. <br/>\n",
    "\n",
    "Similarly for fake images, the target (true) label is 0 (as we want the discriminator to output 0 for fake images) and we want to penalize the generator if it starts outputing values close to 1. So we basically want to mirror the above loss function and that's just: `-log(1-x)`. <br/>\n",
    "\n",
    "BCE loss basically becomes `-log(1-x)` when it's target (true) label is 0. That's why it perfectly fits the task! <br/>\n",
    "\n",
    "\n",
    "### Training utility functions\n",
    "Let's define some useful utility functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tried SGD for the discriminator, had problems tweaking it - Adam simply works nicely but default lr 1e-3 won't work!\n",
    "# I had to train discriminator more (4 to 1 schedule worked) to get it working with default lr, still got worse results.\n",
    "# 0.0002 and 0.5, 0.999 are from the DCGAN paper it works here nicely!\n",
    "def get_optimizers(d_net, g_net):\n",
    "    d_opt = Adam(d_net.parameters(), lr=0.0002, betas=(0.5, 0.999))\n",
    "    g_opt = Adam(g_net.parameters(), lr=0.0002, betas=(0.5, 0.999))\n",
    "    return d_opt, g_opt\n",
    "\n",
    "\n",
    "# It's useful to add some metadata when saving your model, it should probably make sense to also add the number of epochs\n",
    "def get_training_state(generator_net, gan_type_name):\n",
    "    training_state = {\n",
    "        \"commit_hash\": git.Repo(search_parent_directories=True).head.object.hexsha,\n",
    "        \"state_dict\": generator_net.state_dict(),\n",
    "        \"gan_type\": gan_type_name\n",
    "    }\n",
    "    return training_state\n",
    "\n",
    "\n",
    "# Makes things useful when you have multiple models\n",
    "class GANType(enum.Enum):\n",
    "    VANILLA = 0\n",
    "\n",
    "\n",
    "# Feel free to ignore this one not important for GAN training. \n",
    "# It just figures out a good binary name so as not to overwrite your older models.\n",
    "def get_available_binary_name(gan_type_enum=GANType.VANILLA):\n",
    "    def valid_binary_name(binary_name):\n",
    "        # First time you see raw f-string? Don't worry the only trick is to double the brackets.\n",
    "        pattern = re.compile(rf'{gan_type_enum.name}_[0-9]{{6}}\\.pth')\n",
    "        return re.fullmatch(pattern, binary_name) is not None\n",
    "\n",
    "    prefix = gan_type_enum.name\n",
    "    # Just list the existing binaries so that we don't overwrite them but write to a new one\n",
    "    valid_binary_names = list(filter(valid_binary_name, os.listdir(BINARIES_PATH)))\n",
    "    if len(valid_binary_names) > 0:\n",
    "        last_binary_name = sorted(valid_binary_names)[-1]\n",
    "        new_suffix = int(last_binary_name.split('.')[0][-6:]) + 1  # increment by 1\n",
    "        return f'{prefix}_{str(new_suffix).zfill(6)}.pth'\n",
    "    else:\n",
    "        return f'{prefix}_000000.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tracking your model's progress during training\n",
    "You can track how your GAN training is progressing through:\n",
    "1. Console output\n",
    "2. Images dumped to: `data/debug_imagery`\n",
    "3. Tensorboard, just type in `tensorboard --logdir=runs` to your Anaconda console \n",
    "\n",
    "Note: to use tensorboard just navigate to project root first via `cd path_to_root` and open `http://localhost:6006/` (browser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "####################### constants #####################\n",
    "# For logging purpose\n",
    "ref_batch_size = 16\n",
    "ref_noise_batch = get_gaussian_latent_batch(ref_batch_size, device)  # Track G's quality during training on fixed noise vectors\n",
    "\n",
    "discriminator_loss_values = []\n",
    "generator_loss_values = []\n",
    "\n",
    "img_cnt = 0\n",
    "\n",
    "enable_tensorboard = True\n",
    "console_log_freq = 50\n",
    "debug_imagery_log_freq = 50\n",
    "checkpoint_freq = 2\n",
    "\n",
    "# For training purpose\n",
    "num_epochs = 10  # feel free to increase this\n",
    "\n",
    "########################################################\n",
    "\n",
    "writer = SummaryWriter()  # (tensorboard) writer will output to ./runs/ directory by default\n",
    "\n",
    "# Hopefully you have some GPU ^^\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Prepare feed-forward nets (place them on GPU if present) and optimizers which will tweak their weights\n",
    "discriminator_net = DiscriminatorNet().train().to(device)\n",
    "generator_net = GeneratorNet().train().to(device)\n",
    "\n",
    "discriminator_opt, generator_opt = get_optimizers(discriminator_net, generator_net)\n",
    "\n",
    "# 1s (real_images_gt) will configure BCELoss into -log(x) (check out the loss image above that's -log(x)) \n",
    "# whereas 0s (fake_images_gt) will configure it to -log(1-x)\n",
    "# So that means we can effectively use binary cross-entropy loss to achieve adversarial loss!\n",
    "adversarial_loss = nn.BCELoss()\n",
    "real_images_gt = torch.ones((batch_size, 1), device=device)\n",
    "fake_images_gt = torch.zeros((batch_size, 1), device=device)\n",
    "\n",
    "ts = time.time()  # start measuring time\n",
    "\n",
    "# GAN training loop, it's always smart to first train the discriminator so as to avoid mode collapse!\n",
    "# A mode collapse, for example, is when your generator learns to only generate a single digit instead of all 10 digits!\n",
    "for epoch in range(num_epochs):\n",
    "    for batch_idx, (real_images, _) in enumerate(mnist_data_loader):\n",
    "\n",
    "        real_images = real_images.to(device)  # Place imagery on GPU (if present)\n",
    "\n",
    "        #\n",
    "        # Train discriminator: maximize V = log(D(x)) + log(1-D(G(z))) or equivalently minimize -V\n",
    "        # Note: D = discriminator, x = real images, G = generator, z = latent Gaussian vectors, G(z) = fake images\n",
    "        #\n",
    "\n",
    "        # Zero out .grad variables in discriminator network,\n",
    "        # otherwise we would have corrupt results - leftover gradients from the previous training iteration\n",
    "        discriminator_opt.zero_grad()\n",
    "\n",
    "        # -log(D(x)) <- we minimize this by making D(x)/discriminator_net(real_images) as close to 1 as possible\n",
    "        real_discriminator_loss = adversarial_loss(discriminator_net(real_images), real_images_gt)\n",
    "\n",
    "        # G(z) | G == generator_net and z == get_gaussian_latent_batch(batch_size, device)\n",
    "        fake_images = generator_net(get_gaussian_latent_batch(batch_size, device))\n",
    "        # D(G(z)), we call detach() so that we don't calculate gradients for the generator during backward()\n",
    "        fake_images_predictions = discriminator_net(fake_images.detach())\n",
    "        # -log(1 - D(G(z))) <- we minimize this by making D(G(z)) as close to 0 as possible\n",
    "        fake_discriminator_loss = adversarial_loss(fake_images_predictions, fake_images_gt)\n",
    "\n",
    "        discriminator_loss = real_discriminator_loss + fake_discriminator_loss\n",
    "        discriminator_loss.backward()  # this will populate .grad vars in the discriminator net\n",
    "        discriminator_opt.step()  # perform D weights update according to optimizer's strategy\n",
    "\n",
    "        #\n",
    "        # Train generator: minimize V1 = log(1-D(G(z))) or equivalently maximize V2 = log(D(G(z))) (or min of -V2)\n",
    "        # The original expression (V1) had problems with diminishing gradients for G when D is too good.\n",
    "        #\n",
    "\n",
    "        # if you want to cause mode collapse probably the easiest way to do that would be to add \"for i in range(n)\"\n",
    "        # here (simply train G more frequent than D), n = 10 worked for me other values will also work - experiment.\n",
    "\n",
    "        # Zero out .grad variables in discriminator network (otherwise we would have corrupt results)\n",
    "        generator_opt.zero_grad()\n",
    "\n",
    "        # D(G(z)) (see above for explanations)\n",
    "        generated_images_predictions = discriminator_net(generator_net(get_gaussian_latent_batch(batch_size, device)))\n",
    "        # By placing real_images_gt here we minimize -log(D(G(z))) which happens when D approaches 1\n",
    "        # i.e. we're tricking D into thinking that these generated images are real!\n",
    "        generator_loss = adversarial_loss(generated_images_predictions, real_images_gt)\n",
    "\n",
    "        generator_loss.backward()  # this will populate .grad vars in the G net (also in D but we won't use those)\n",
    "        generator_opt.step()  # perform G weights update according to optimizer's strategy\n",
    "\n",
    "        #\n",
    "        # Logging and checkpoint creation\n",
    "        #\n",
    "\n",
    "        generator_loss_values.append(generator_loss.item())\n",
    "        discriminator_loss_values.append(discriminator_loss.item())\n",
    "        \n",
    "        if enable_tensorboard:\n",
    "            global_batch_idx = len(mnist_data_loader) * epoch + batch_idx + 1\n",
    "            writer.add_scalars('losses/g-and-d', {'g': generator_loss.item(), 'd': discriminator_loss.item()}, global_batch_idx)\n",
    "            # Save debug imagery to tensorboard also (some redundancy but it may be more beginner-friendly)\n",
    "            if batch_idx % debug_imagery_log_freq == 0:\n",
    "                with torch.no_grad():\n",
    "                    log_generated_images = generator_net(ref_noise_batch)\n",
    "                    log_generated_images = nn.Upsample(scale_factor=2, mode='nearest')(log_generated_images)\n",
    "                    intermediate_imagery_grid = make_grid(log_generated_images, nrow=int(np.sqrt(ref_batch_size)), normalize=True)\n",
    "                    writer.add_image('intermediate generated imagery', intermediate_imagery_grid, global_batch_idx)\n",
    "\n",
    "        if batch_idx % console_log_freq == 0:\n",
    "            prefix = 'GAN training: time elapsed'\n",
    "            print(f'{prefix} = {(time.time() - ts):.2f} [s] | epoch={epoch + 1} | batch= [{batch_idx + 1}/{len(mnist_data_loader)}]')\n",
    "\n",
    "        # Save intermediate generator images (more convenient like this than through tensorboard)\n",
    "        if batch_idx % debug_imagery_log_freq == 0:\n",
    "            with torch.no_grad():\n",
    "                log_generated_images = generator_net(ref_noise_batch)\n",
    "                log_generated_images_resized = nn.Upsample(scale_factor=2.5, mode='nearest')(log_generated_images)\n",
    "                out_path = os.path.join(DEBUG_IMAGERY_PATH, f'{str(img_cnt).zfill(6)}.jpg')\n",
    "                save_image(log_generated_images_resized, out_path, nrow=int(np.sqrt(ref_batch_size)), normalize=True)\n",
    "                img_cnt += 1\n",
    "\n",
    "        # Save generator checkpoint\n",
    "        if (epoch + 1) % checkpoint_freq == 0 and batch_idx == 0:\n",
    "            ckpt_model_name = f\"vanilla_ckpt_epoch_{epoch + 1}_batch_{batch_idx + 1}.pth\"\n",
    "            torch.save(get_training_state(generator_net, GANType.VANILLA.name), os.path.join(CHECKPOINTS_PATH, ckpt_model_name))\n",
    "\n",
    "# Save the latest generator in the binaries directory\n",
    "torch.save(get_training_state(generator_net, GANType.VANILLA.name), os.path.join(BINARIES_PATH, get_available_binary_name()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate images with your vanilla GAN\n",
    "\n",
    "Nice, finally we can use the generator we trained to generate some MNIST-like imagery!\n",
    "\n",
    "Let's define a couple of utility functions which will make things cleaner!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def postprocess_generated_img(generated_img_tensor):\n",
    "    assert isinstance(generated_img_tensor, torch.Tensor), f'Expected PyTorch tensor but got {type(generated_img_tensor)}.'\n",
    "\n",
    "    # Move the tensor from GPU to CPU, convert to numpy array, extract 0th batch, move the image channel\n",
    "    # from 0th to 2nd position (CHW -> HWC)\n",
    "    generated_img = np.moveaxis(generated_img_tensor.to('cpu').numpy()[0], 0, 2)\n",
    "\n",
    "    # Since MNIST images are grayscale (1-channel only) repeat 3 times to get RGB image\n",
    "    generated_img = np.repeat(generated_img,  3, axis=2)\n",
    "\n",
    "    # Imagery is in the range [-1, 1] (generator has tanh as the output activation) move it into [0, 1] range\n",
    "    generated_img -= np.min(generated_img)\n",
    "    generated_img /= np.max(generated_img)\n",
    "\n",
    "    return generated_img\n",
    "\n",
    "\n",
    "# This function will generate a random vector pass it to the generator which will generate a new image\n",
    "# which we will just post-process and return it\n",
    "def generate_from_random_latent_vector(generator):\n",
    "    with torch.no_grad():  # Tells PyTorch not to compute gradients which would have huge memory footprint\n",
    "        \n",
    "        # Generate a single random (latent) vector\n",
    "        latent_vector = get_gaussian_latent_batch(1, next(generator.parameters()).device)\n",
    "        \n",
    "        # Post process generator output (as it's in the [-1, 1] range, remember?)\n",
    "        generated_img = postprocess_generated_img(generator(latent_vector))\n",
    "\n",
    "    return generated_img\n",
    "\n",
    "\n",
    "# You don't need to get deep into this one - irrelevant for GANs - it will just figure out a good name for your generated\n",
    "# images so that you don't overwrite the old ones. They'll be stored with xxxxxx.jpg naming scheme.\n",
    "def get_available_file_name(input_dir): \n",
    "    def valid_frame_name(str):\n",
    "        pattern = re.compile(r'[0-9]{6}\\.jpg')  # regex, examples it covers: 000000.jpg or 923492.jpg, etc.\n",
    "        return re.fullmatch(pattern, str) is not None\n",
    "\n",
    "    # Filter out only images with xxxxxx.jpg format from the input_dir\n",
    "    valid_frames = list(filter(valid_frame_name, os.listdir(input_dir)))\n",
    "    if len(valid_frames) > 0:\n",
    "        # Images are saved in the <xxxxxx>.jpg format we find the biggest such <xxxxxx> number and increment by 1\n",
    "        last_img_name = sorted(valid_frames)[-1]\n",
    "        new_prefix = int(last_img_name.split('.')[0]) + 1  # increment by 1\n",
    "        return f'{str(new_prefix).zfill(6)}.jpg'\n",
    "    else:\n",
    "        return '000000.jpg'\n",
    "\n",
    "\n",
    "def save_and_maybe_display_image(dump_dir, dump_img, out_res=(256, 256), should_display=False):\n",
    "    assert isinstance(dump_img, np.ndarray), f'Expected numpy array got {type(dump_img)}.'\n",
    "\n",
    "    # step1: get next valid image name\n",
    "    dump_img_name = get_available_file_name(dump_dir)\n",
    "\n",
    "    # step2: convert to uint8 format <- OpenCV expects it otherwise your image will be completely black. Don't ask...\n",
    "    if dump_img.dtype != np.uint8:\n",
    "        dump_img = (dump_img*255).astype(np.uint8)\n",
    "\n",
    "    # step3: write image to the file system (::-1 because opencv expects BGR (and not RGB) format...)\n",
    "    cv.imwrite(os.path.join(dump_dir, dump_img_name), cv.resize(dump_img[:, :, ::-1], out_res, interpolation=cv.INTER_NEAREST))  \n",
    "\n",
    "    # step4: maybe display part of the function\n",
    "    if should_display:\n",
    "        plt.imshow(dump_img)\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We're now ready to generate some new digit images!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model states contains this data: dict_keys(['commit_hash', 'state_dict', 'gan_type'])\n",
      "Using VANILLA GAN!\n",
      "Generating new MNIST-like images!\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAQ30lEQVR4nO3db4xV9Z3H8c+XYUZkQIVFcALjgpUHiwTtCoSEZlNsllh9ANXUaMwGskT6oCSt4cEalqTGjQnZbIt9YBqniyndFJsmSsTErEXjnyUxzQCyCIsurGD5Mw4oQgGBgeG7D+awO8U5v99wz733HPi9X8nkztzvPff8OMNnzr33e875mbsLwPVvRNkDANAchB1IBGEHEkHYgUQQdiARI5u5MjMr7aN/MwvW6UrgeuHuQ/5nLxR2M7tf0s8ltUj6V3dfU+T5YlpaWnJr/f39wWVHjRoVrJ89ezZYD/2x4A9FvjK324gR+S9cL126FFx25MhwNC5evFjTmIbz/EWfO0/NL+PNrEXS85K+K2mGpMfMbEa9Bgagvoq8Z58raZ+7f+LufZJ+K2lRfYYFoN6KhH2ypIODfj6U3fdnzGy5mW01s60F1gWgoCLv2Yd6M/a1N2Hu3iWpSyr3AzogdUX27IckdQ76eYqkI8WGA6BRioS9W9J0M5tmZm2SHpW0qT7DAlBvNb+Md/eLZrZC0hsaaL296O676zayIcTaayGx1lqsNXfu3Lma1x0zbty4YP3LL79s2Lpjih6fUGbrLdZeC4m1v4q25hrVXgsp1Gd399clvV6nsQBoIA6XBRJB2IFEEHYgEYQdSARhBxJB2IFEWDNPzyzzcNnQ6Y5SvJ9cpMff6HPpy+xlt7W1Bet9fX0NXX9VLViwIFh/++23G7buvPPZ2bMDiSDsQCIIO5AIwg4kgrADiSDsQCKa3nprVJuoypeKrvLY0BiNvjptCK03IHGEHUgEYQcSQdiBRBB2IBGEHUgEYQcS0dQpmxupzF51a2trsH7hwoUmjQRVETu2ogzs2YFEEHYgEYQdSARhBxJB2IFEEHYgEYQdSEQyl5KOGTNmTLB++vTp3FrsMtWxPnzR38HSpUtrXnds7GfOnAnW9+7dG6yvXLkytzZ//vzgsrFzvvfv3x+sP/vss7m13t7e4LLbtm0L1ov+zm644Ybc2vnz5ws9d9757IUOqjGzA5JOSeqXdNHdZxd5PgCNU48j6Ba4++d1eB4ADcR7diARRcPukn5vZtvMbPlQDzCz5Wa21cy2FlwXgAKKvoyf7+5HzGyipM1m9pG7vzf4Ae7eJalLqvYHdMD1rtCe3d2PZLdHJW2UNLcegwJQfzWH3czazWzs5e8lLZS0q14DA1BfRV7GT5K0MTtvd6SkDe7+73UZVQlCffSYS5cuBeuxvmms1z1x4sRgvbOzM7e2ePHi4LIdHR3Beuz4g9iUzaFtE+tVx64DMHPmzGD90Ucfza1t3LgxuOz27duD9aJ99tD/iTlz5gSX7e7urmmdNYfd3T+RdHetywNoLlpvQCIIO5AIwg4kgrADiSDsQCI4xbUCYu2tJUuWBOurV6/Ord1yyy3BZfv6+oL1mLFjx9a8bH9/f7AeG1tsWuTQ6bmPPPJIcNk333wzWK8ypmwGEkfYgUQQdiARhB1IBGEHEkHYgUQQdiAR182UzVUW6wc//vjjwfqaNWuC9dGjR+fWYqffxi4V3d7eHqzHjtMITV187ty54LInTpwI1m+88cZgfefOnbm1kydPBpeNTblc5hThtWLPDiSCsAOJIOxAIgg7kAjCDiSCsAOJIOxAIuiz10HsUtCxSwOvXbs2WA9N7yuFpzY+ePBgcNnnnnsuWG9paQnWFyxYEKzv2bOnppokzZ0bnnNk4cKFwfqGDRtya7Gppsvso0+YMCFY//zz2uZRZc8OJIKwA4kg7EAiCDuQCMIOJIKwA4kg7EAirpvrxpd5/nHs2unr1q0L1h966KFC6+/t7c2tLVq0KLhsrNfd2toarH/11VfBeugYgNjxAzNmzAjWe3p6gvXPPvsstxY7z79MseM2YmOv+brxZvaimR01s12D7htvZpvNbG92Oy72PADKNZyX8b+SdP8V9z0l6S13ny7prexnABUWDbu7vyfp+BV3L5K0Pvt+vaTFdR4XgDqr9dj4Se7eI0nu3mNmE/MeaGbLJS2vcT0A6qThJ8K4e5ekLomJHYEy1dp66zWzDknKbo/Wb0gAGqHWsG+SdHke4SWSXq3PcAA0SvRlvJm9JOnbkiaY2SFJP5G0RtLvzGyZpD9K+v5wVxjqIcb6h6FeepnnH997773B+oMPPhisx44RiG2X/fv359ZCfW5JmjlzZrA+e/bsYH3Tpk3B+uHDh3Nr58+fDy770UcfBeuxHn+Vr+1+22235dZCxwcUEQ27uz+WU/pOnccCoIE4XBZIBGEHEkHYgUQQdiARhB1IRFNPcW1vb/e77rort97d3d20sdTTqFGjgvXjx688teDqlo8Jteb6+/uDy8amk4617t5///1gfdmyZbm1UFtOirfmqtxaiwlt99g2j6n5FFcA1wfCDiSCsAOJIOxAIgg7kAjCDiSCsAOJuG4uJT2MdQfrRbZD7Lm3bNkSrM+bN6/Q8xdRdLvE6qdOncqtLV26NLjsa6+9FqzHjiGosiKnesfQZwcSR9iBRBB2IBGEHUgEYQcSQdiBRBB2IBGV6rOXOe1yI82aNStYf+edd4L1m2++OVhvZB++kS5cuBCsx44/+OCDD+o5nOsGfXYgcYQdSARhBxJB2IFEEHYgEYQdSARhBxJRqT57qtra2oL1IteVnzNnTrA+efLkYD3W43/yySeD9dtvvz23FjqnW5LOnDkTrN9xxx3B+rFjx4L1MrW0tOTWip6nX3Of3cxeNLOjZrZr0H1Pm9lhM9uRfT1QaHQAGm44L+N/Jen+Ie5f6+73ZF+v13dYAOotGnZ3f09SeP4iAJVX5AO6FWa2M3uZPy7vQWa23My2mtnWAusCUFCtYf+FpG9IukdSj6Sf5j3Q3bvcfba7z65xXQDqoKawu3uvu/e7+yVJv5Q0t77DAlBvNYXdzDoG/fg9SbvyHgugGqJ9djN7SdK3JU2Q1CvpJ9nP90hySQck/cDde6Iro89+zYmdKz9+/Phgvbu7O7c2derU4LKx66fffffdwfru3buD9TKFtmvRY1/y+uz5M8L//4KPDXH3ukKjAdB0HC4LJIKwA4kg7EAiCDuQCMIOJCL6aXyVjB07NrcWmhoYtYu1gU6cOBGsb9iwIbe2atWq4LKxtt+UKVOC9Sq33kaOzI9e7BTXWqd0Zs8OJIKwA4kg7EAiCDuQCMIOJIKwA4kg7EAirqk+O7306on1fI8cOdKwdZ89e7Zhz91osemqG4E9O5AIwg4kgrADiSDsQCIIO5AIwg4kgrADibim+uzXqth52bF67JzyZk67faXYdNIrVqyo+blj/67YZayrrMilpGtdlj07kAjCDiSCsAOJIOxAIgg7kAjCDiSCsAOJoM9eB62trcH6ww8/HKx3dnYWWv/zzz+fW+vr6wsuG+vxx/5t9913X7Aem5Y55OTJk8H6u+++W/Nzly3UDy963EWe6J7dzDrN7G0z22Nmu83sR9n9481ss5ntzW7H1TQCAE0xnJfxFyWtdPe/kjRP0g/NbIakpyS95e7TJb2V/QygoqJhd/ced9+efX9K0h5JkyUtkrQ+e9h6SYsbNUgAxV3Ve3Yzmyrpm5L+IGmSu/dIA38QzGxizjLLJS0vNkwARQ077GY2RtLLkn7s7n+KfYhwmbt3SerKnqO8MzaAxA2r9WZmrRoI+m/c/ZXs7l4z68jqHZKONmaIAOohume3gV34Okl73P1ng0qbJC2RtCa7fbXoYO68885gfd++fUVX0RDt7e3B+gsvvFBo+VirJdT++vjjj4PLxi71HGsbTps2LVgPTU188eLF4LLPPPNMsH4tX1o89Ds/c+ZMQ9Y5nJfx8yX9naQPzWxHdt8qDYT8d2a2TNIfJX2/ISMEUBfRsLv7Fkl5b9C/U9/hAGgUDpcFEkHYgUQQdiARhB1IBGEHEmHNvAzx9XoEXewU1S1bthRaPiY0bXJsSuURI8J/74uebhla/xdffBFcdt68ecH6p59+GqxXWVtbW24tNp3zMC4tPuQvjT07kAjCDiSCsAOJIOxAIgg7kAjCDiSCsAOJ4FLSdXDw4MFgfc6cOcH6gQMHgvXYtMihXnmsj17UcK9YNJT+/v5gPXYZ7GtZ6N9WZJuGsGcHEkHYgUQQdiARhB1IBGEHEkHYgUQQdiAR18357I2a5rYZbrrppmB9+vTpwfoTTzyRW5s1a1Zw2di1+s+fPx+snzt3Llh/4403cmurV68OLhu7LnysT99ILS0twXqZY+N8diBxhB1IBGEHEkHYgUQQdiARhB1IBGEHEhHts5tZp6RfS7pN0iVJXe7+czN7WtITko5lD13l7q9Hnqu6zW6gjkLzr0vF5mAPHVPi7rl99uFcvOKipJXuvt3MxkraZmabs9pad/+Xqx4tgKYbzvzsPZJ6su9PmdkeSZMbPTAA9XVV79nNbKqkb0r6Q3bXCjPbaWYvmtm4nGWWm9lWM9taaKQAChn2sfFmNkbSu5KedfdXzGySpM8luaR/ktTh7n8feQ7esyMJVXzPPqw9u5m1SnpZ0m/c/ZXsSXvdvd/dL0n6paS5Vz1qAE0TDbsN/BlZJ2mPu/9s0P0dgx72PUm76j88APUynNbbtyT9h6QPNdB6k6RVkh6TdI8GXsYfkPSD7MO80HMFV3brrbcGx3Ls2LFgPbLuYL3Kp8AiLbHLf8em4a659ebuWyQNtXCwpw6gWjiCDkgEYQcSQdiBRBB2IBGEHUgEYQcS0fQpm0P97lh/MbRs0Uv7FunDjxwZ3oyxdRft8ccOnwwp85LIsamoY5epjmlra8utFZ0OevTo0cH62bNng/XQ7yXWR68Ve3YgEYQdSARhBxJB2IFEEHYgEYQdSARhBxLR7Cmbj0n6dNBdEzRwaasqqurYqjouibHVqp5j+0t3H/LCEE0N+9dWbrbV3WeXNoCAqo6tquOSGFutmjU2XsYDiSDsQCLKDntXyesPqerYqjouibHVqiljK/U9O4DmKXvPDqBJCDuQiFLCbmb3m9nHZrbPzJ4qYwx5zOyAmX1oZjvKnp8um0PvqJntGnTfeDPbbGZ7s9sh59graWxPm9nhbNvtMLMHShpbp5m9bWZ7zGy3mf0ou7/UbRcYV1O2W9Pfs5tZi6T/lvS3kg5J6pb0mLv/V1MHksPMDkia7e6lH4BhZn8j6bSkX7v7zOy+f5Z03N3XZH8ox7n7P1RkbE9LOl32NN7ZbEUdg6cZl7RY0lKVuO0C43pETdhuZezZ50ra5+6fuHufpN9KWlTCOCrP3d+TdPyKuxdJWp99v14D/1maLmdsleDuPe6+Pfv+lKTL04yXuu0C42qKMsI+WdLBQT8fUrXme3dJvzezbWa2vOzBDGHS5Wm2stuJJY/nStFpvJvpimnGK7Ptapn+vKgywj7UBdOq1P+b7+5/Lem7kn6YvVzF8PxC0jc0MAdgj6SfljmYbJrxlyX92N3/VOZYBhtiXE3ZbmWE/ZCkzkE/T5F0pIRxDMndj2S3RyVtVPWmou69PINudnu05PH8nypN4z3UNOOqwLYrc/rzMsLeLWm6mU0zszZJj0raVMI4vsbM2rMPTmRm7ZIWqnpTUW+StCT7fomkV0scy5+pyjTeedOMq+RtV/r05+7e9C9JD2jgE/n/kfSPZYwhZ1x3SPrP7Gt32WOT9JIGXtZd0MAromWS/kLSW5L2ZrfjKzS2f9PA1N47NRCsjpLG9i0NvDXcKWlH9vVA2dsuMK6mbDcOlwUSwRF0QCIIO5AIwg4kgrADiSDsQCIIO5AIwg4k4n8BIFvhXuFW7T4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# VANILLA_000000.pth is the model I pretrained for you, feel free to change it if you trained your own model (last section)!\n",
    "model_path = os.path.join(BINARIES_PATH, 'VANILLA_000000.pth')  \n",
    "assert os.path.exists(model_path), f'Could not find the model {model_path}. You first need to train your generator.'\n",
    "\n",
    "# Hopefully you have some GPU ^^\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# let's load the model, this is a dictionary containing model weights but also some metadata\n",
    "# commit_hash - simply tells me which version of my code generated this model (hey you have to learn git!)\n",
    "# gan_type - this one is \"VANILLA\" but I also have \"DCGAN\" and \"cGAN\" models\n",
    "# state_dict - contains the actuall neural network weights\n",
    "model_state = torch.load(model_path)  \n",
    "print(f'Model states contains this data: {model_state.keys()}')\n",
    "\n",
    "gan_type = model_state[\"gan_type\"]  # \n",
    "print(f'Using {gan_type} GAN!')\n",
    "\n",
    "# Let's instantiate a generator net and place it on GPU (if you have one)\n",
    "generator = GeneratorNet().to(device)\n",
    "# Load the weights, strict=True just makes sure that the architecture corresponds to the weights 100%\n",
    "generator.load_state_dict(model_state[\"state_dict\"], strict=True)\n",
    "generator.eval()  # puts some layers like batch norm in a good state so it's ready for inference <- fancy name right?\n",
    "    \n",
    "generated_imgs_path = os.path.join(DATA_DIR_PATH, 'generated_imagery')  # this is where we'll dump images\n",
    "os.makedirs(generated_imgs_path, exist_ok=True)\n",
    "\n",
    "#\n",
    "# This is where the magic happens!\n",
    "#\n",
    "\n",
    "print('Generating new MNIST-like images!')\n",
    "generated_img = generate_from_random_latent_vector(generator)\n",
    "save_and_maybe_display_image(generated_imgs_path, generated_img, should_display=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I'd love to hear your feedback\n",
    "\n",
    "If you found this notebook useful and would like me to add the same for cGAN and DCGAN please [open an issue](https://github.com/gordicaleksa/pytorch-gans/issues/new). <br/>\n",
    "\n",
    "I'm super not aware of how useful people find this, I usually do stuff through my IDE.\n",
    "\n",
    "# Connect with me\n",
    "\n",
    "I share lots of useful (I hope so at least!) content on LinkedIn, Twitter, YouTube and Medium. <br/>\n",
    "So feel free to connect with me there:\n",
    "1. My [LinkedIn](https://www.linkedin.com/in/aleksagordic) and [Twitter](https://twitter.com/gordic_aleksa) profiles\n",
    "2. My YouTube channel - [The AI Epiphany](https://www.youtube.com/c/TheAiEpiphany)\n",
    "3. My [Medium](https://gordicaleksa.medium.com/) profile\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}