mnist.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ufmHMbThnWEs"
   },
   "source": [
    "---\n",
    "# MNIST: Hand-written Digit Classification\n",
    "---\n",
    "\n",
    "[MNIST](https://en.wikipedia.org/wiki/MNIST_database) is a classic dataset in computer vision used for handwritten digit classification. Below, I experiment with several different machine learning approaches included, fully-connected neural networks, convolutional neural networks, and, most notably, vision transformers; I compare each architecture and their overall performance on the classification task. This project is done using PyTorch and GPU-accelerated using the Mac M1 chip."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "bSnMFngy9sPA"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "from einops import rearrange, repeat\n",
    "from einops.layers.torch import Rearrange"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zwocJbzfnZv5"
   },
   "source": [
    "## Loading MNIST\n",
    "The `torchvision` module contains links to many standard datasets. We can load the MNIST dataset into a `Dataset` object as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 431,
     "referenced_widgets": [
      "261c7f67235548a98e13f196c5fe8e92",
      "a6395dc9b98b4008a55acd20bfea8fa1",
      "b30f753332cf418ba24c3e3c66d6ede2",
      "f95b78bd4aeb45c38211fb8bbc44ac5e",
      "4c1630885bfc4fc1b56e1e7dd2c4d111",
      "76ceb0c5e55a4f649546bda2e517c9ef",
      "06a1ccecd6934249ba582aae46c21115",
      "dd646184d14b42ee9e7d23f837a0bc2e",
      "89e3360f282341f1a1c0d0f073a351af",
      "4d9c6a6b48ea4ae5ab04993762518a4b",
      "d80ee751e04246cd82d0dde0ff07adcc",
      "0d1eea56f56e416c98326a4dd45845e7",
      "3a73d6914c5a4cc3a8fcbca9346715fa",
      "8ce8ea4ebfad417fb6df5410741b0753",
      "ab25f081f93c49f2b137a57d1de5f630",
      "296842818dd846a5817c619792148201",
      "b9f40688fe474cde931a9886072c64d8",
      "c13181bfd2fd4e66b15f070062e5ffdd",
      "90d3ddc528904599b0b8d3c3e19b2819",
      "0d7ee23f17b047ce8865ed02fffc0cc6",
      "ece8364103b14437b16a6ff937427322",
      "59de2466b3814431a8e995637bed1efd",
      "92c3359d5c7d4cecad1a2179dd44c308",
      "34abb1e4a0c6477ba393377f785511a4",
      "ef2baa336dec49918e0b9f7691363015",
      "2cb003ba31134b39ac657a82280d5c31",
      "cb32a1674c9d41eaa15645ae9d34fa90",
      "34518066cc61449ca87d06356f6b25cd",
      "49c8182ddc6c4f7fb2de89e80151517a",
      "c4ca6d489fd947b092c7ff9cd6f8889b",
      "79ccf00bbc3a44cd80d401b1c8d333ed",
      "f98c27b1e9d24843a38f103bb4051968",
      "1ebbf2ad85f7446dadde649e7b0af20b",
      "34bbe2bb28a14ae9ae7e2efaa69ad9be",
      "4ebf2751d41241678b4629d6e1594529",
      "8e35093ad4874ba0b75c0157ac12b1b0",
      "aa3ab8c1a4944c8a803a4a510f331c91",
      "c5a8018df6d44808a00ff3c6289b2ef9",
      "2ee32878afc24d72b839eaa5f43ecfff",
      "67903fa5367e4464a2ee7aa21d1e33f4",
      "43404a2ab0ed4495a9c0c260072fbec7",
      "c4c646d0aa59467e9cbe21bfa7eecd20",
      "6ab64b1f3836474aabc9bce92ee3ff90",
      "f7a037b48d304440929b977043621481"
     ]
    },
    "id": "6NUqKdYrFpXA",
    "outputId": "75e319bb-807a-48d1-e195-7be3e3f3b5b9"
   },
   "outputs": [],
   "source": [
    "train_dataset = datasets.MNIST('./data', train=True, download=True,  # Downloads into a directory ../data\n",
    "                               transform=transforms.ToTensor())\n",
    "train_dataset, valid_dataset = torch.utils.data.random_split(train_dataset, \n",
    "                                                             [int(len(train_dataset)*0.8), int(len(train_dataset)*0.2)], \n",
    "                                                             generator=torch.Generator().manual_seed(42))\n",
    "test_dataset = datasets.MNIST('./data', train=False, download=False,  # No need to download again\n",
    "                              transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dPRWRI-TomP7"
   },
   "source": [
    "## Exploring our Data\n",
    "\n",
    "The `Dataset` object is an iterable where each element is a tuple of (input `Tensor`, target). Each image in the training set is a (1, 28, 28) image. In plain English, this means that all our training examples are 28 by 28 greyscale images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4Y1oIfzooiLa",
    "outputId": "4487ef17-e01b-45e7-8dd5-4f84effffb34"
   },
   "outputs": [],
   "source": [
    "first_train, first_target = train_dataset[0]\n",
    "\n",
    "print(f\"Data type of training data: {type(first_train)}\")\n",
    "print(f\"Size of training data point: {first_train.size()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also check how many data points we have in our training, validation, and test sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SX5hIU_4BdGk",
    "outputId": "120edc2d-35e4-4c2c-a091-9f586008df45"
   },
   "outputs": [],
   "source": [
    "print(f\"Number of training points: {len(train_dataset)}\")\n",
    "print(f\"Number of validation points: {len(valid_dataset)}\")\n",
    "print(f\"Number of test points: {len(test_dataset)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "q8Gf22fNovo_"
   },
   "source": [
    "We can convert images to numpy arrays and plot them with matplotlib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "id": "l7RZ_f05oohA",
    "outputId": "4e106d2f-034b-4c8f-b2c3-0437b5973af6"
   },
   "outputs": [],
   "source": [
    "plt.imshow(train_dataset[0][0][0].numpy(), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2mTmn0zWGBU2"
   },
   "source": [
    "## Training Loop\n",
    "\n",
    "We can define a function called `run_training_loop` which can be used to train our model, regardless of its architecture. It follows a standard training loop in PyTorch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "ZtFhoE5sGEb9"
   },
   "outputs": [],
   "source": [
    "def run_training_loop(model, batch_size=32, n_epochs=10, lr=1e-3):\n",
    "    \"\"\"\n",
    "    Run a training loop based on the input model and associated parameters\n",
    "    \n",
    "    Parameters:\n",
    "        model: The input model to be trained\n",
    "        batch_size: Number of training points to include in batch\n",
    "        n_epochs: Number of epochs to train the model for\n",
    "        lr: Learning rate used in Adam optimizer\n",
    "        \n",
    "    \"\"\"\n",
    "    train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "    valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size, shuffle=True)\n",
    "    \n",
    "    device = \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
    "    model.to(device)\n",
    "\n",
    "    # Choose Adam as the optimizer\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "\n",
    "    # Use the cross entropy loss function\n",
    "    loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "    # store metrics\n",
    "    train_loss_history = np.zeros([n_epochs, 1])\n",
    "    valid_accuracy_history = np.zeros([n_epochs, 1])\n",
    "    valid_loss_history = np.zeros([n_epochs, 1])\n",
    "\n",
    "    for epoch in range(n_epochs):\n",
    "\n",
    "        # Some layers, such as Dropout, behave differently during training\n",
    "        model.train()\n",
    "\n",
    "        train_loss = 0\n",
    "        for batch_idx, (data, target) in enumerate(train_loader):\n",
    "            \n",
    "            data, target = data.to(device), target.to(device)\n",
    "\n",
    "            # Erase accumulated gradients\n",
    "            optimizer.zero_grad()\n",
    "\n",
    "            # Forward pass\n",
    "            output = model(data)\n",
    "\n",
    "            # Calculate loss\n",
    "            loss = loss_fn(output, target)\n",
    "            train_loss += loss.item()\n",
    "\n",
    "            # Backward pass\n",
    "            loss.backward()\n",
    "            \n",
    "            # Weight update\n",
    "            optimizer.step()\n",
    "\n",
    "        train_loss_history[epoch] = train_loss / len(train_loader.dataset)\n",
    "\n",
    "        # Track loss each epoch\n",
    "        print('Train Epoch: %d  Average loss: %.4f' %\n",
    "              (epoch + 1,  train_loss_history[epoch]))\n",
    "\n",
    "        # Putting layers like Dropout into evaluation mode\n",
    "        model.eval()\n",
    "\n",
    "        valid_loss = 0\n",
    "        correct = 0\n",
    "\n",
    "        # Turning off automatic differentiation\n",
    "        with torch.no_grad():\n",
    "            for data, target in valid_loader:\n",
    "                data, target = data.to(device), target.to(device)\n",
    "                output = model(data)\n",
    "                valid_loss += loss_fn(output, target).item()  # Sum up batch loss\n",
    "                pred = output.argmax(dim=1, keepdim=True)  # Get the index of the max class score\n",
    "                correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "        valid_loss_history[epoch] = valid_loss / len(valid_loader.dataset)\n",
    "        valid_accuracy_history[epoch] = correct / len(valid_loader.dataset)\n",
    "\n",
    "        print('Valid set: Average loss: %.4f, Accuracy: %d/%d (%.4f)\\n' %\n",
    "              (valid_loss_history[epoch], correct, len(valid_loader.dataset),\n",
    "              100. * valid_accuracy_history[epoch]))\n",
    "    \n",
    "    return model, train_loss_history, valid_loss_history, valid_accuracy_history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Uk38yQyirQEW"
   },
   "source": [
    "## Testing\n",
    "After selecting hyperparameters from the validation set, evaluate on the held-out test set (distinct from the validation set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "E9rwgEpNGUyJ"
   },
   "outputs": [],
   "source": [
    "def test_performance(model, batch_size=32):\n",
    "    \"\"\"\n",
    "    Test model performance on test dataset\n",
    "    \n",
    "    Parameters:\n",
    "        model: The model to be tested\n",
    "        batch_size: Number of training points to include in batch\n",
    "    \"\"\"\n",
    "    test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True) \n",
    "\n",
    "    # Putting layers like Dropout into evaluation mode\n",
    "    model.eval()\n",
    "    # Use the cross entropy loss function\n",
    "    loss_fn = nn.CrossEntropyLoss()\n",
    "    \n",
    "    # Send model to appropriate device\n",
    "    device = \"mps\" if torch.backends.mps.is_available() else \"cpu\"\n",
    "    model.to(device)\n",
    "\n",
    "    test_loss = 0\n",
    "    correct = 0\n",
    "\n",
    "    # Turning off automatic differentiation\n",
    "    with torch.no_grad():\n",
    "        for data, target in test_loader:\n",
    "            data, target = data.to(device), target.to(device)\n",
    "            output = model(data)\n",
    "            test_loss += loss_fn(output, target).item()  # Sum up batch loss\n",
    "            pred = output.argmax(dim=1, keepdim=True)  # Get the index of the max class score\n",
    "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
    "\n",
    "    test_loss /= len(test_loader.dataset)\n",
    "    test_accuracy = correct / len(test_loader.dataset)\n",
    "\n",
    "    print('Test set: Average loss: %.4f, Accuracy: %d/%d (%.4f)' %\n",
    "          (test_loss, correct, len(test_loader.dataset),\n",
    "          100. * test_accuracy))\n",
    "    return test_loss, test_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Gp2rPfwVGvGG"
   },
   "source": [
    "# Fully-connected Neural Network\n",
    "\n",
    "We ignore the local structure of our 28 x 28 images in favor of their vector representation as 784 x 1 input vectors. Our first model has three linear layers and uses a ReLU activation between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TaH2mCHeGxQO"
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Flatten(),  \n",
    "    nn.Linear(784, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 62),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(62, 10),\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 838
    },
    "id": "R3qabsPEG0bx",
    "outputId": "7338eb8e-9fa8-4e82-e269-0fad96c8921e"
   },
   "outputs": [],
   "source": [
    "trained_model, train_loss_history, valid_loss_history, valid_accuracy_history = run_training_loop(model, n_epochs=10)\n",
    "test_performance(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "-YsfmwODG3-1",
    "outputId": "5b408b4c-94b5-4c29-a197-e858d9ee7186"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,6))\n",
    "ax.plot(train_loss_history, label='train')\n",
    "ax.plot(valid_loss_history, label='valid')\n",
    "ax.set_xlabel(\"Epoch Number\")\n",
    "ax.set_ylabel(\"Cross-Entropy Loss\")\n",
    "plt.title(\"Training and Validation Loss Curves\")\n",
    "plt.legend()\n",
    "plt.savefig(\"loss_curves_fcn.jpg\", dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OHI16r2D5DoJ"
   },
   "source": [
    "## Convolutional Neural Network\n",
    "\n",
    "The CNN below includes two convolutional layers with max pooling and batch normalization, followed by three fully-connected layers. Dropout layers are introduced to avoid overfitting the model to our training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ILTGTFff5Bh6",
    "outputId": "b131b8db-3f3d-49b5-c23c-e91b4902afab"
   },
   "outputs": [],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Conv2d(1, 6, 5), # (1, 28, 28) -> (6, 24, 24)\n",
    "    nn.BatchNorm2d(num_features=6),\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2), # (6, 24, 24) -> (6, 12, 12)\n",
    "    nn.Conv2d(6, 16, 5), # (6, 12, 12) -> (16, 8, 8)\n",
    "    nn.ReLU(),\n",
    "    nn.MaxPool2d(2, 2), # (16, 8, 8) -> (16, 4, 4)\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(16 * 4 * 4, 120),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(120, 84),\n",
    "    nn.Dropout(p=0.1),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(84, 10),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 837
    },
    "id": "j2WNXNZf5KKG",
    "outputId": "5b394a02-006b-42c7-9fac-83d9f7ed58ef"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Epoch: 1  Average loss: 0.0065\n",
      "Valid set: Average loss: 0.0024, Accuracy: 11742/12000 (97.8500)\n",
      "\n",
      "Train Epoch: 2  Average loss: 0.0021\n",
      "Valid set: Average loss: 0.0018, Accuracy: 11778/12000 (98.1500)\n",
      "\n",
      "Train Epoch: 3  Average loss: 0.0015\n",
      "Valid set: Average loss: 0.0015, Accuracy: 11825/12000 (98.5417)\n",
      "\n",
      "Train Epoch: 4  Average loss: 0.0012\n",
      "Valid set: Average loss: 0.0022, Accuracy: 11747/12000 (97.8917)\n",
      "\n",
      "Train Epoch: 5  Average loss: 0.0010\n",
      "Valid set: Average loss: 0.0013, Accuracy: 11862/12000 (98.8500)\n",
      "\n",
      "Train Epoch: 6  Average loss: 0.0008\n",
      "Valid set: Average loss: 0.0014, Accuracy: 11842/12000 (98.6833)\n",
      "\n",
      "Train Epoch: 7  Average loss: 0.0007\n",
      "Valid set: Average loss: 0.0014, Accuracy: 11862/12000 (98.8500)\n",
      "\n",
      "Train Epoch: 8  Average loss: 0.0007\n",
      "Valid set: Average loss: 0.0016, Accuracy: 11832/12000 (98.6000)\n",
      "\n",
      "Train Epoch: 9  Average loss: 0.0006\n",
      "Valid set: Average loss: 0.0014, Accuracy: 11865/12000 (98.8750)\n",
      "\n",
      "Train Epoch: 10  Average loss: 0.0004\n",
      "Valid set: Average loss: 0.0014, Accuracy: 11863/12000 (98.8583)\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt+0lEQVR4nO3de3yU9Zn38c+VyflIJgkJEGSiHCQgAiYp1mPFA9ZWbKuC1l2366NPW21td7tdbffZ7fapu+1ut6dn0dZWd1uroMXulrZYPIFalwJBEDkIBAgQTjmR83GS6/njniQDhmSASe7JzPV+vfLKzH2YuSYvuL9z/373/fuJqmKMMSb2xLldgDHGGHdYABhjTIyyADDGmBhlAWCMMTHKAsAYY2JUvNsFnI3c3Fz1+Xxul2GMMWPG5s2ba1U1b7B1YyoAfD4f5eXlbpdhjDFjhogcPNM6awIyxpgYZQFgjDExygLAGGNi1JjqAzDGmLPR3d1NVVUVHR0dbpcy4pKTkyksLCQhISHkfSwAjDFRq6qqioyMDHw+HyLidjkjRlWpq6ujqqqKoqKikPezJiBjTNTq6OggJycnqg/+ACJCTk7OWZ/pWAAYY6JatB/8+5zL54z6AOj09/DjN/bx1t4at0sxxpiIEvUBkOiJ48k39/PfW466XYoxJsY0NDTw+OOPn/V+H/3oR2loaAh/QaeJ+gAQEUqmZLOpst7tUowxMeZMAeD3+4fcb/Xq1YwbN26EqhoQ9QEAUFbk5VB9Gyeaov9SMGNM5HjkkUfYt28fc+fOpbS0lKuuuopbb72V4uJiAG677TYuu+wyZs2axZNPPtm/n8/no7a2lsrKSmbOnMn999/PrFmzuPHGG2lvbw9bfTFxGWipzwvApsp6PjZnosvVGGPc8I+/3cHOo01hfc3iiZn8w8dnnXH9t7/9bbZv387WrVtZt24dt9xyC9u3b++/VPPpp5/G6/XS3t5OaWkpn/rUp8jJyTnlNfbu3cvy5cv56U9/yp133smLL77IPffcE5b6Y+IMoHhiJikJHjYdsGYgY4x7ysrKTrlO/0c/+hGXXnopCxYs4PDhw+zdu/cD+xQVFTF37lwALrvsMiorK8NWT0ycASR44pg/ZRwbK0+6XYoxxiVDfVMfLWlpaf2P161bx6uvvsr69etJTU3l2muvHfQ6/qSkpP7HHo8nrE1AMXEGAE4z0PvHm2jq6Ha7FGNMjMjIyKC5uXnQdY2NjWRnZ5Oamsr777/Pn/70p1GuLkbOAMAJAFXYfPAkH5kx3u1yjDExICcnhyuuuILZs2eTkpJCfn5+/7pFixbx4x//mJkzZzJjxgwWLFgw6vXFTADMu2Ac8XHCpgP1FgDGmFHz3HPPDbo8KSmJl156adB1fe38ubm5bN++vX/5V77ylbDWFjNNQKmJ8cyalEW59QMYYwwQQwEAUDolm61VDXT6e9wuxRhjXBdbAVDkpcvfy7aqRrdLMcYY18VUAJRMyQawYSGMMYYQA0BEFonIbhGpEJFHBlmfJCLPB9ZvEBFf0LpHA8t3i8hNQcvHichKEXlfRHaJyOVh+URDyElP4qK8NLshzBhjCCEARMQDLANuBoqBu0Sk+LTN7gNOqupU4PvAdwL7FgNLgVnAIuDxwOsB/BD4g6peDFwK7Dr/jzO8siIv5QdP0tOro/F2xhgTsUI5AygDKlR1v6p2ASuAxadtsxj4eeDxSmChOLMTLAZWqGqnqh4AKoAyEckCrgaeAlDVLlVtOO9PE4JSn5fmDj97Tgx+c4YxxrgpPT0dgKNHj3L77bcPus21115LeXn5eb9XKAEwCTgc9LwqsGzQbVTVDzQCOUPsWwTUAP8hIltE5GciksYgROQBESkXkfKamvOf1CV4YDhjjIlUEydOZOXKlSP6Hm51AscD84EnVHUe0Ap8oG8BQFWfVNUSVS3Jy8s77zcuzE6hIDOZjdYPYIwZBY888gjLli3rf/6Nb3yDb33rWyxcuJD58+dzySWX8Jvf/OYD+1VWVjJ79mwA2tvbWbp0KTNnzuQTn/hE2MYDCuVO4CPA5KDnhYFlg21TJSLxQBZQN8S+VUCVqm4ILF/JGQIg3ESE0iIvGw/UoaoxM1+oMTHvpUfg+Hvhfc2CS+Dmbw+5yZIlS/jSl77Egw8+CMALL7zAmjVr+OIXv0hmZia1tbUsWLCAW2+99YzHoyeeeILU1FR27drFtm3bmD9/fljKD+UMYBMwTUSKRCQRp1N31WnbrALuDTy+HXhdVTWwfGngKqEiYBqwUVWPA4dFZEZgn4XAzvP8LCEr82VzoqmTqpPhG1XPGGMGM2/ePKqrqzl69Cjvvvsu2dnZFBQU8LWvfY05c+Zw/fXXc+TIEU6cOHHG13jzzTf75wCYM2cOc+bMCUttw54BqKpfRB4C1gAe4GlV3SEi3wTKVXUVTmfuMyJSAdTjhASB7V7AObj7gQdVte823C8AzwZCZT/wmbB8ohCUBPoBNh6oZ7I3dbTe1hjjpmG+qY+kO+64g5UrV3L8+HGWLFnCs88+S01NDZs3byYhIQGfzzfoUNAjLaTB4FR1NbD6tGV/H/S4A7jjDPs+Bjw2yPKtQMlZ1Bo2M/IzyEyOZ1NlPZ+6rNCNEowxMWTJkiXcf//91NbW8sYbb/DCCy8wfvx4EhISWLt2LQcPHhxy/6uvvprnnnuO6667ju3bt7Nt27aw1BUzo4EGi4sTSnxeuxLIGDMqZs2aRXNzM5MmTWLChAl8+tOf5uMf/ziXXHIJJSUlXHzxxUPu/7nPfY7PfOYzzJw5k5kzZ3LZZZeFpa6YDACAEl82r79fTV1LJznpScPvYIwx5+G99wY6oHNzc1m/fv2g27W0tADOxPB9Q0GnpKSwYsWKsNcUU2MBBSvrvx/Ahoc2xsSmmA2ASwqzSIyPo9yagYwxMSpmAyAp3sPcwnHWD2BMlHOuSI9+5/I5YzYAAEqLstl+tInWTr/bpRhjRkBycjJ1dXVRHwKqSl1dHcnJyWe1X8x2AoMzLtCytfvYeriBK6bmul2OMSbMCgsLqaqqIhzjiEW65ORkCgvP7rL2mA6A+VOyEXFuCLMAMCb6JCQkUFRU5HYZESumm4AykxOYWZBp/QDGmJgU0wEAzgQxWw410N3T63YpxhgzqmI+AEp9Xtq7e9hxtMntUowxZlRZAPgCE8Xb/ADGmBgT8wEwPjOZKTmp1g9gjIk5MR8A4DQDlR88GfXXChtjTDALAJxmoPrWLvbVtLhdijHGjBoLAAYmit94wAaGM8bEDgsAoCg3jdz0RBsYzhgTUywAcCaKL5niZaMFgDEmhlgABJQWeak62c6xRpso3hgTGywAAmyCGGNMrLEACJg5IYO0RI/dEGaMiRkWAAHxnjjmT8m2G8KMMTHDAiBIqc/L7hPNNLZ1u12KMcaMOAuAICW+bFRh8yE7CzDGRL+QAkBEFonIbhGpEJFHBlmfJCLPB9ZvEBFf0LpHA8t3i8hNQcsrReQ9EdkqIuVh+TTnad7kbBI8YjeEGWNiwrAzgomIB1gG3ABUAZtEZJWq7gza7D7gpKpOFZGlwHeAJSJSDCwFZgETgVdFZLqq9gT2+4iq1obx85yXlEQPsydl2Q1hxpiYEMoZQBlQoar7VbULWAEsPm2bxcDPA49XAgtFRALLV6hqp6oeACoCrxexynxetlU10tHdM/zGxhgzhoUSAJOAw0HPqwLLBt1GVf1AI5AzzL4KvCwim0XkgTO9uYg8ICLlIlI+GhM7l/i8dPX08u7hhhF/L2OMcZObncBXqup84GbgQRG5erCNVPVJVS1R1ZK8vLwRL6pkijNBTPlB6wcwxkS3UALgCDA56HlhYNmg24hIPJAF1A21r6r2/a4G/osIaRrKTktken46G+2GMGNMlAslADYB00SkSEQScTp1V522zSrg3sDj24HX1ZldZRWwNHCVUBEwDdgoImkikgEgImnAjcD28/844VHi8/LOwZP09NoEMcaY6DVsAATa9B8C1gC7gBdUdYeIfFNEbg1s9hSQIyIVwF8BjwT23QG8AOwE/gA8GLgCKB/4o4i8C2wEfq+qfwjvRzt3ZT4vzZ1+3j9uE8UbY6LXsJeBAqjqamD1acv+PuhxB3DHGfZ9DHjstGX7gUvPttjRUloUGBjuQD2zJma5XI0xxowMuxN4EJPGpTAxK9lGBjXGRDULgDMoLXImiLGJ4o0x0coC4AxKfV5qmjs5VN/mdinGGDMiLADOYGCieLsc1BgTnSwAzmDa+HSyUhJsfgBjTNSyADiDuDih1JdNuXUEG2OilAXAEEp8XvbXtlLT3Ol2KcYYE3YWAEPo6wew4aGNMdHIAmAIl0zKIjkhzu4HMMZEJQuAISTGxzF38jjrCDbGRCULgGGU+rzsONpIS6ff7VKMMSasLACGUerz0quw5ZA1AxljoosFwDDmT8kmTpyB4YwxJppYAAwjPSme4omZbLR+AGNMlLEACEGpz8vWww10+XvdLsUYY8LGAiAEZT4vHd29bD/a6HYpxhgTNhYAISjxDUwQY4wx0cICIAR5GUkU5abZDWHGmKhiARCiUl825Qfr6bWJ4o0xUcICIEQlPi8Nbd1U1LS4XYoxxoSFBUCIymyCGGNMlLEACNGUnFTyMpJsZFBjTNSwAAiRiFDm81pHsDEmaoQUACKySER2i0iFiDwyyPokEXk+sH6DiPiC1j0aWL5bRG46bT+PiGwRkd+d9ycZBSW+bI40tHOkod3tUowx5rwNGwAi4gGWATcDxcBdIlJ82mb3ASdVdSrwfeA7gX2LgaXALGAR8Hjg9fo8DOw63w8xWmyCGGNMNAnlDKAMqFDV/araBawAFp+2zWLg54HHK4GFIiKB5StUtVNVDwAVgddDRAqBW4Cfnf/HGB0zJ2SSnhRvHcHGmKgQSgBMAg4HPa8KLBt0G1X1A41AzjD7/gD4KjDkADsi8oCIlItIeU1NTQjljhxPnDB/SrZNEGOMiQqudAKLyMeAalXdPNy2qvqkqpaoakleXt4oVDe0Ml82e0600NDW5XYpxhhzXkIJgCPA5KDnhYFlg24jIvFAFlA3xL5XALeKSCVOk9J1IvLLc6h/1A30A9jVQMaYsS2UANgETBORIhFJxOnUXXXaNquAewOPbwdeV1UNLF8auEqoCJgGbFTVR1W1UFV9gdd7XVXvCcPnGXGXTh5HgkesGcgYM+bFD7eBqvpF5CFgDeABnlbVHSLyTaBcVVcBTwHPiEgFUI9zUCew3QvATsAPPKiqPSP0WUZFcoKHOYU2UbwxZuwbNgAAVHU1sPq0ZX8f9LgDuOMM+z4GPDbEa68D1oVSR6Qo9Xl56o/76ejuITnBM/wOxhgTgexO4HNQ6sumu0fZcqjB7VKMMeacWQCcg5IpXkTshjBjzNhmAXAOslITmJGfYRPFG2PGNAuAc1Tiy+adgyfx99hE8caYsckC4ByV+ry0dvXw/vFmt0sxxphzYgFwjsqKbIIYY8zYZgFwjiZkpVCYnWL3AxhjxiwLgPNQGpggxrnp2RhjxhYLgPNQ6vNS29JJZV2b26UYY8xZswA4D2VF2QBssn4AY8wYZAFwHi7KSyc7NcHuBzDGjEkWAOdBRCjxee2OYGPMmGQBcJ7KfF4q69qobu5wuxRjjDkrFgDnqcTX1w9gE8QYY8YWC4DzNHtSFikJHrsfwBgz5lgAnKcETxzzLrAJYowxY48FQBiU+LzsOtZEc0e326UYY0zILADCoMznpVfhHZsgxhgzhlgAhMG8C8bhiRO7IcwYM6ZYAIRBWlI8sydm2g1hxpgxxQIgTEp8Xt493ECnv8ftUowxJiQWAGFS6vPS6e9l+5FGt0sxxpiQWACESWnghrCNdkOYMWaMsAAIk5z0JC7MS7NxgYwxY0ZIASAii0Rkt4hUiMgjg6xPEpHnA+s3iIgvaN2jgeW7ReSmwLJkEdkoIu+KyA4R+cewfSIXlfm8lB88SW+vTRBjjIl8wwaAiHiAZcDNQDFwl4gUn7bZfcBJVZ0KfB/4TmDfYmApMAtYBDweeL1O4DpVvRSYCywSkQVh+UQuKvV5aWzvZk+1TRRvjIl8oZwBlAEVqrpfVbuAFcDi07ZZDPw88HglsFBEJLB8hap2quoBoAIoU0dLYPuEwM+Y/9pc6nMmit9Uaf0AxpjIF0oATAIOBz2vCiwbdBtV9QONQM5Q+4qIR0S2AtXAK6q6YbA3F5EHRKRcRMprampCKNc9k70p5Gcm2Q1hxpgxwbVOYFXtUdW5QCFQJiKzz7Ddk6paoqoleXl5o1rj2RKRwETx9TZRvDEm4oUSAEeAyUHPCwPLBt1GROKBLKAulH1VtQFYi9NHMOaV+rwca+yg6mS726UYY8yQQgmATcA0ESkSkUScTt1Vp22zCrg38Ph24HV1vgKvApYGrhIqAqYBG0UkT0TGAYhICnAD8P55f5oI0NcPUH7QmoGMMZFt2AAItOk/BKwBdgEvqOoOEfmmiNwa2OwpIEdEKoC/Ah4J7LsDeAHYCfwBeFBVe4AJwFoR2YYTMK+o6u/C+9HcMaMgg4zkeLshzBgT8eJD2UhVVwOrT1v290GPO4A7zrDvY8Bjpy3bBsw722LHAk+ccNmUbJsgxhgT8exO4BFQ6vNSUd1CfWuX26UYY8wZWQCMgLKiQD+AnQUYYyKYBcAImFOYRWJ8nDUDGWMimgXACEiK93BpYZbdEWyMiWgWACOk1Odl+5FG2rr8bpdijDGDsgAYIaVFXvy9ylabKN4YE6EsAEbI/AuyEbGB4YwxkcsCYIRkpSRwcUGmdQQbYyKWBcAIKvNl886hk/h7et0uxRhjPsACYASV+Ly0dfWw81iT26UYY8wHWACMoL4bwjba/ADGmAhkATCC8jOTucCbav0AxpiIZAEwwkp82ZRXnrQJYowxEccCYISV+bzUtXaxv7bV7VKMMeYU0R8AqrD8Llj/OPg7R/3tSwP9ADZPsDEm0kR/AHQ2QVcrrHkU/r0Utv0KekfvsswLc9PISUtko/UDGGMiTPQHQHIW/Plv4J5fQ1Im/Pp/wU+vhX1rR+XtRaS/H8AYYyJJ9AcAgAhMXQj/+034xJPQdhKeuQ2e+QQc2zbib1/q83Kovo0TTR0j/l7GGBOq2AiAPnFxcOkSeGgT3PgYHN0CP7kafv0AnDw4Ym9r9wMYYyJRbAVAn4Rk+PBD8MWtcMXDsPM38O8lsObr0Bb+g3TxhExSEz02Q5gxJqLEZgD0SRkHN/wjfOEdmHMn/Olx+OFceOt70N0etreJ98Qx/4JsNlo/gKO9AVZ9AR7/MNTsdrsaY2JWbAdAn6xJsHgZfPZtmHI5vPaP8KP58M4z0NsTlrco9Xl5/3gTje3dYXm9Mev938OyD8GWZ6HpCDx9Exze6HZVxsQkC4Bg+cVw9/PwF7+HzAmw6iF44grY/QfnfoLzUOrLRhV+/MY+Ov3hCZUxpbUWVv4lrLgb0nLh/tfggXWQ4oWf3wq7X3K7QmNijgXAYHxXwv96De74OfR0wfIl8J+3QFX5Ob9kaZGXm2bl88S6fVz/vTdY/d6x2BgeQhXeWwnLymDnKvjI1+H+tTBxHniL4L6XYfxMJxg2/9ztao2JKSEFgIgsEpHdIlIhIo8Msj5JRJ4PrN8gIr6gdY8Glu8WkZsCyyaLyFoR2SkiO0Tk4bB9onARgVm3wYMb4KPfhdo98LOF8MKfQ92+s365BE8cP/mzEp65r4y0xHg+/+w73PmT9Wyragh76RGj6ZhzYH/xPsj2OZfhXvNViE8c2CYtF+79LVy0EH77RXjjX877bMsYExoZ7luoiHiAPcANQBWwCbhLVXcGbfN5YI6qflZElgKfUNUlIlIMLAfKgInAq8B0YDwwQVXfEZEMYDNwW/BrDqakpETLy8/9W/h56WyG9cvg7R9BTyfMvxeufQTSx5/1S/X0Ks9vOsz3XtlNbUsXn5w3ib9ZNIMJWSkjULgLVGHLM7Dm75y/1XV/Bws+D3GeM+/T0w2rvgjvPgclf+mE7lDbG2NCIiKbVbVksHWhnAGUARWqul9Vu4AVwOLTtlkM9J2/rwQWiogElq9Q1U5VPQBUAGWqekxV3wFQ1WZgFzDpbD/YqErKcA74D2+Fy/4C3vm5c8XQ2n+GzpazeilPnHD3hy5g7Veu5XPXXsTv3jvGR767ju+/soe2Lv9IVD96TlY6N9mt+gIUXAKf+x/48BeGP5h7EuC2x+HKL0P5086ZVhivxDLGfFAoATAJOBz0vIoPHqz7t1FVP9AI5ISyb6C5aB6wYbA3F5EHRKRcRMprampCKHeEpY+HW/4NHtwI066HN74NP5oLG3/qfIs9CxnJCfztoot57a+uYeHMfH742l4+8t11rNxcRW/vGGsG6e2FDT9xLu2sKodbvuc07eRcFPpriMD134BF33GuFvrFbSNyX4YxxuFqJ7CIpAMvAl9S1UHnTVTVJ1W1RFVL8vLyRrfAoeRcBHf+wukszp0Oq7/iXN6447/Pug17sjeVZXfPZ+VnL6cgM5mv/OpdFi97e+zcOVy7F/7zo/DSV53LaD//Jyi9z7nz+lws+Czc/jQcfQf+42ZorApvvcYYILQAOAJMDnpeGFg26DYiEg9kAXVD7SsiCTgH/2dV9dfnUnxEKCxxLhu9+wXwJMKv7oWfXQ+Vb5/1S5X4vPzX56/gB0vmUtvSyZ0/Wc/nfrmZQ3VtI1B4GPT44Y/fdy6Vrd4Ftz0Bn14J4yYPv+9wZn8S7nkRmo7Cz25wXt8YE1ahdALH43QCL8Q5eG8C7lbVHUHbPAhcEtQJ/ElVvVNEZgHPMdAJ/BowDejF6TOoV9UvhVqsq53AoejtgXeXw9p/cm5ymr7IadIYP/OsX6q9q4efvrWfJ9bto6dX+YsrfDx03VQykxPCX/e5OL4dfvMgHNsKMz8OH/03yMgfgfd5D355O/jb4a4VMOXD4X8PY6LYUJ3AwwZA4AU+CvwA8ABPq+pjIvJNoFxVV4lIMvAMTlt+PbBUVfcH9v068JeAH6ep5yURuRJ4C3gPJwwAvqaqq4eqI+IDoE93O2z4Mbz1fehqhrl3w7Vfc+44Pksnmjr41zW7efGdKrJTE/nyDdO5q3Qy8R6XWu/8nfDmd+GP34OUbOdqnVm3jex7njwIv/wUNBxymoZmfmxk38+YKHLeARApxkwA9Gmrh7f+DTY+CRIHH/oszP9z8F7odHiehe1HGvm/v9vJhgP1TM9P5+u3FHPN9FHuE6na7Hzrr9kFc5bAom9Dqnd03ru1Dp670+kX+Oh3nT4GY8ywLADcdvIgrH0Mtr0AqHNT1NTrnR/fVZCUHtLLqCprdpzgn1/axcG6Nq6Znsff3TKTafkZI1o+XW2w7p+c+yDSC+DjP4DpN43sew5aRyv86jOwdw1c87dw7aNnHaTGxBoLgEhx8iBUvAIVr8H+N6C7FeISnCtn+gJhfPGwB7Uufy+/WF/JD1/bS1tXD3eXXcCXb5iONy1xyP3OSeXbzphI9fud+x9u+KYzy5pbevzwu4dhyy+ds6lbvg+eePfqMSbCWQBEIn8XHP4TVLzqBMKJ7c7yjAnO7GVTr4cLr3Xa2c+gvrWLH7y6h2c3HCI10cMXrpvKvR/2kRQfhjtoO5vhlX+A8qecM5Zb/x8UXX3+rxsOqs4Z1Zv/CtNvdvoFElPdrsqYiGQBMBY0HYV9rzuBsO916Gh0+g0KSwNnBwthwrxBr63fe6KZf1q9i7W7a5iSk8qjN1/MTbMKkHNtHql4FX77Jef6+wWfh+u+Dolp5/f5RsLGn8Lqv3H+Rnc/P3r9EcaMIRYAY02P3+nsrHjV+TnyDqCQmgMXXecEwkXXfWAcojf31PCt3+9kz4kWyoq8/J9birmk8Cyaa9rqnVnR3n0OcmfA4n+HyWXh/WzhtvM38OL9kD3FuW9g3AVuV2TCQdX6d8LEAmCsa62D/WsHAqE1MCTGhEsH+g4KS8GTgL+nlxWbDvP9V/ZQ39bFJ+cV8tVFM8jPTB76PXb9Fn7/1864/Vd+Ga7+G2fqzLGg8m1YfpfTDHTPi5A/y+2KzNnqaoVD6+HAm3DgLTi+zblabuI8mDjf+V1wiTX1nQMLgGjS2wsn3hvoOzj0J9AeSMqEC68JnB0spCm5gGVrK/iPP1biiRM+e81FPHD1haQkntY/0FLtNKPs/G/nP9jiZU6wjDUndjg3jHW1wl3POXM6mMjV3QFVGwcO+EfKodfvXBRRWOIc9E8ecM5+W447+4jHualy4tyBUMifBfFJrn6USGcBEM06Gp3/RBWvwt5XoSkwbk7exTD1eqrHX8m3to9j1Y56CjKT+eqiGdw2dxJxgnNZ6h/+1jloXvO3cMXDzqicY1XDYeeGsZMH4JM/Hfkb1Ezo/F1Os+aBN52fwxudocIlzjmQF13tXBJ9wYIP9jc1HYOjW5z9j25xQqE9ME6WJ9EJgYnzBs4W8i62K8OCWADEClVn4pq+pqLKt53/ZPEpNOQv4PmG6Syvn86U/By+Ff8Uk2vfgsIyp60/b4bb1YdHWz0sX+ocYG7+F/jQA25XFJt6/HD83YFv+IfWQ3cbIFAwG4qucQ74Uy4/+8uKVZ27wvsC4egWOLoVOgPjScanwIQ5pzYf5Uw998EJxzgLgFjV1QYH3x4IhLoKAHqIo0vj+Vf/Ejbn38H1syZy0+wCpo1PP/crhyJJd7sz//Du1XDVX8N1/8c6FEdaby9U7xg44B98e+CAnHfxwDd835Ujc7VWb69zr0rwWcLxbYHQARIzAk1HcwdCIdsXE/8uLACMo/4A7HsN6vZxcOqneelICmt2HGfLoQYAfDmp3DirgJtm5TNvcjZxcWP4P0ePH1b/NWz+T5h7j3P38lhu3oo0fWebfU06lX8caJbxXhh0wL9qZAYJDEWP36kxOBRObHfm+QbnHpv+pqPA2ULmxKgLBQsAM6Tqpg5e2XWCNTtOsH5fLd09Sm56EjcUj+fGWQV8+KKc8NxcNtpU4Y3vwLp/hmk3wh3/GZn3M4wFqk7fSt83/Mq3oOWEsy5r8sABv+gqyCp0t9ah+LucM5W+pqMjW6B6p3MhBUDaeJg0fyAU8mdB8jhISB2zTUgWACZkTR3drNtdw5odx1n3fjWtXT2kJXq49uLx3DSrgGtn5EXOkNShKn/aucR14jy4+1eQluN2RWNDY9XAAf/AmwMXGKQXOAd631XOgX+sN6V0tzvDjvedJRzd4pw5cNqxMTHd+QLR9zsp4yyepztjfgU/jx+BoVsGYQFgzkmnv4f/2VfHyzuO88rOE9S2dJHgES6/KJcbi/O5sTif8cPdXxApdv0OXrzP+XZ6z4vOQctNPX7n8sbGqoGf7vbAgVSCfvftIIOsO5vfhLZdr985CB540/nGD5DidQ74RVeD72rInTa2D/ih6GyGY+9CzW7oanGulOtsCTwe5vnpwXEmnsQzhMMg4ZHqPecRcC0AzHnr6VW2Hj7Jmh0nWLPjOAcDs5TNu2AcN80q4MbifC7MC21UU9ccXA/Ll0B8sjNz2YQ5I/M+qtDRcOrB/fSf5mMDzQ6RJikLfFcMfMMfXzxmmz9GnarT8dzV6oRIV2tQQAQ/bwkERgjPezqds66v7D6nkiwATFipKnurW1iz/Tgv7zzBe0caAZg6Pp2bZuVzY3EBcwqzIvOKoupdzr0CHU2w9Fnn5rmz5e90ZnxrPBJ0UD/s/G4KLOtqOXUfT6LTwZg12TkLySqEzElBzyc53/RU6f8G2ff4nH6f4/4iTl1xY7DPJ1r1dDuhco6j8FoAmBF1pKGdV3Y4YbDhQD09vUpBZjI3BsLgQxd6SXBrBrPBNB5xQqCuAj75E5j9qYF1qs5wGH0H9L6fpqDHfZ2fwdLyBg7sWZMDB/fCgQN8Wp59izausAAwo6ahrYvXdlXz8s7jvLGnho7uXjKT41k40+kzuHp6HmlJEXCXZvtJWH43HPofmHlrUJPNEeeUO1hC6hm+tQctGyvjJpmYYwFgXNHe1cNbe2t4eecJXtt1gpNt3STGx3HV1FxumlXARy4eT16Gi+O4dHfAbx92rmHPmvTBb+19PynZ0d/paaKWBYBxnb+nl02VJ3l553Fe3nGCIw3tAORnJjFzQibFEzKd3xMz8eWk4RnLN6EZE0EsAExEUVV2HG3if/bVsutYM7uONVFR3YK/1/m3mJLgYUZBRiAYMiiemMmMgkzSI6HpyJgxZqgAsP9RZtSJCLMnZTF70sBVDZ3+HvaeaGHXsSZ2Hmti17Emfr/tKMs3+vu38eWkMrPvTGFCJjMnZjIxKzkyrzYyZgywADARISne84FQUFWONnaw6+hAKOw81sRL24/3b5OVksDMCRmnNCNNy08fm0NXGDPKQgoAEVkE/BDwAD9T1W+ftj4J+AVwGVAHLFHVysC6R4H7gB7gi6q6JrD8aeBjQLWqzg7LpzFRRUSYNC6FSeNSuL54YECxlk4/u483sfNoEzsDTUjLNx6io7sXgPg4Yer49FNCYeaEDHLSbeIQY4IN2wcgIh5gD3ADUAVsAu5S1Z1B23wemKOqnxWRpcAnVHWJiBQDy4EyYCLwKjBdVXtE5GqgBfhFqAFgfQDmTHp6lcq6VnYedc4U+s4WTjQNXNKZn5kUFAjW4Wxiw/n2AZQBFaq6P/BiK4DFwM6gbRYD3wg8Xgn8uzgNs4uBFaraCRwQkYrA661X1TdFxHcOn8eYD/DECRflpXNRXjofv3Ri//K6ls7+jua+UHhrb+0pHc7TCzKYPj6dGQUZTM/PYEZBBuMzkqxvwUS9UAJgEnA46HkV8KEzbaOqfhFpBHICy/902r6TzrlaY85STnoSV05L4sppuf3LOv09VFS3BJqQmthzopm1u2v41eaq/m2yUhKYnp/eHwjT850fb9rojOBozGiI+E5gEXkAeADgggsucLkaEw2S4j3MmpjFrImnjq1S19LJnhMt7K1uZvfxZvacaOa37x7l2Q0DVyLlpicxoyAQDPkZTMvPYHp+OhljbYhsYwgtAI4Ak4OeFwaWDbZNlYjEA1k4ncGh7DskVX0SeBKcPoCz2deYs5GTnsTl6UlcftHAfAGqSnVzZ38g7D7ezJ7qFp7fdJi2roHRPCeNS+k/Y+g7a5g6Pp3kBLsayUSuUAJgEzBNRIpwDt5LgbtP22YVcC+wHrgdeF1VVURWAc+JyPdwOoGnARvDVbwxI01EyM9MJj8zmaun5/Uv7+1VjjS0BwKhmT3Hm9l9ooW3K+ro6ukN7Au+nDSmnda/4MtJIzHeBoYz7hs2AAJt+g8Ba3AuA31aVXeIyDeBclVdBTwFPBPo5K3HCQkC272A02HsBx5UdQZBF5HlwLVArohUAf+gqk+F/RMaMwLi4oTJ3lQme1NPuUTV39PLwfq2QCA4Zw17TrTw2vvV9AQ6nuPjhAvz0k5pRppRkEFhdkpkjZpqop4NBWHMKOj097C/pjUQCM3sPt7CnhPNHKpv699GBLypieRlJJGXkcT4jGTGZyYxPuhxXnoS4zOTSE2M+O47EyFsKAhjXJYU7+m//yBYW5efiuoWdh9v5khDO9XNndQ0d1Ld3Mm+6lpqWjrp7vngl7T0pHjG9wVFZnJ/MIw/LTiyUhLsclZzRhYAxrgoNTGeOYXjmFM4btD1vb1KQ3s31c0dVDcNhEN1c4cTFk2dvFfVQE1zJ61dH5xiMtETF3RG0RcSyQPPA2GRk5ZIvDU/xRwLAGMiWFyc4E1LxJuWyMUFQ2/b2ul3wqGpIxASfYHRQU1zJwfr2thUWc/Jtu4P7CsCOWlOUOSmJ/a/Z05aIt60pP7nfcuyUhKIszuoxzwLAGOiRFpSPEVJ8RTlpg25XZe/l5qWQDicEhbOWUZdaxcH69qob+2ipdM/6Gt44oTs1IRTgsEbCIuc4PBIT8Sbmkh2WqJ1cEcgCwBjYkxifFz/IHvD6fT3cLK1m7rWTupbu6hv7aKuJfC7tYv61k5Otnaz+3gz9a1dNLR3c6brSjKT48lJDzqbSHUCIueUs4uk/tBISbR7KEaaBYAx5oyS4j0UZHkoyAptzmN/Ty8N7d2nBEV9Wxf1LU5Y1AVC5HB9G1sPN3Cytat/XKbTJcbHMS4lgazAz7jUBDL7HqckkpUSz7hUpzkqM7C+b1s72wiNBYAxJmziPXHkpieRm54E+cNvr6o0dfgDZxedp5xdNLV309jeTUOb8/toQwe7jjXT2N59xqapPmmJHicMUp2g6A+NoJDIOi00xqUkkpEcH1N9GxYAxhjXiEj/AXi4votg3T29AwER+N0UFBbBwdHU3s2B2lYa2xtoaOum0987RD2QmTwQCt60RKbkpOLLSaMo1/kpzE6JmiumLACMMWNOgieOnPSkc5rkp6O7xwmLQFA0tgU/7hoIkPZuals6eefgSZqDzjjiA3eB+3JS8QVCoS8gJo5LGVPzS1gAGGNiSnKCh+QED+MzQ+vXUFXqWruorG3lQG0rlXXO7wO1bWw4UH/KoICJnjgme1P6zxZ8uWkU5Ti/CzKTI655yQLAGGOGICL9/RolPu8p6/pGiz1Q2+oERJ3zu7K2jbf21p7S3JQUH4cvJw1fbuopwVCUm+baBEQWAMYYc46CR4tdcGHOKet6e5VjTR0DZw6Bs4eK6hZef7/6lCE+UhM9/c1IvtzUoMdp5KQljlg4WAAYY8wIiIuT/vstrpiae8q6nl7laEP7KU1KlbWt7DjayB92HO8fORYgIymeGQUZ/Oqzl4c9CCwAjDFmlHmChhO/mrxT1nX39FJ1sv2UPocuf++InAVYABhjTARJ8MT1dyJ/ZITfKzouZjXGGHPWLACMMSZGWQAYY0yMsgAwxpgYZQFgjDExygLAGGNilAWAMcbEKAsAY4yJUaJnmr8tAolIDXDwHHfPBWrDWM5YZn+LU9nf41T29xgQDX+LKaqaN9iKMRUA50NEylW1xO06IoH9LU5lf49T2d9jQLT/LawJyBhjYpQFgDHGxKhYCoAn3S4ggtjf4lT29ziV/T0GRPXfImb6AIwxxpwqls4AjDHGBLEAMMaYGBX1ASAii0Rkt4hUiMgjbtfjJhGZLCJrRWSniOwQkYfdrsltIuIRkS0i8ju3a3GbiIwTkZUi8r6I7BKRy92uyU0i8uXA/5PtIrJcRJLdrincojoARMQDLANuBoqBu0Sk2N2qXOUH/lpVi4EFwIMx/vcAeBjY5XYREeKHwB9U9WLgUmL47yIik4AvAiWqOhvwAEvdrSr8ojoAgDKgQlX3q2oXsAJY7HJNrlHVY6r6TuBxM85/8EnuVuUeESkEbgF+5nYtbhORLOBq4CkAVe1S1QZXi3JfPJAiIvFAKnDU5XrCLtoDYBJwOOh5FTF8wAsmIj5gHrDB5VLc9APgq0Cvy3VEgiKgBviPQJPYz0Qkze2i3KKqR4DvAoeAY0Cjqr7sblXhF+0BYAYhIunAi8CXVLXJ7XrcICIfA6pVdbPbtUSIeGA+8ISqzgNagZjtMxORbJzWgiJgIpAmIve4W1X4RXsAHAEmBz0vDCyLWSKSgHPwf1ZVf+12PS66ArhVRCpxmgavE5FfuluSq6qAKlXtOyNciRMIsep64ICq1qhqN/Br4MMu1xR20R4Am4BpIlIkIok4nTirXK7JNSIiOG28u1T1e27X4yZVfVRVC1XVh/Pv4nVVjbpveKFS1ePAYRGZEVi0ENjpYkluOwQsEJHUwP+bhURhp3i82wWMJFX1i8hDwBqcXvynVXWHy2W56Qrgz4D3RGRrYNnXVHW1eyWZCPIF4NnAl6X9wGdcrsc1qrpBRFYC7+BcPbeFKBwWwoaCMMaYGBXtTUDGGGPOwALAGGNilAWAMcbEKAsAY4yJURYAxhgToywAjDEmRlkAGGNMjPr/ZW2DoVigl/cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set: Average loss: 0.0014, Accuracy: 9894/10000 (98.9400)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(0.0014301540810701227, 0.9894)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trained_model, train_loss_history, valid_loss_history, valid_accuracy_history = run_training_loop(model)\n",
    "test_performance(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "TxXD9bNfLYs0",
    "outputId": "cc115257-99ea-43b3-a9b6-7f46db0ccabf"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm0AAAGDCAYAAAB5rSfRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABN60lEQVR4nO3deXxddZ3/8dcne7O0WW4LXUmblH0pUKCksSKIgjoyKqvKKKI4iiIoKv7GcVzHZRwBFZlBwUEWASsMDIIgspRSoLRsbYG26UKbrmnStE2zJ5/fH+ekvaRJetPm5uQm7+fjkce995xz7/3cpG3e/a7m7oiIiIjI0JYWdQEiIiIisn8KbSIiIiIpQKFNREREJAUotImIiIikAIU2ERERkRSg0CYiIiKSAhTaRIYxM3vUzD410NdGyczWmtl7k/C6T5vZZ8P7nzCzxxO59gDeZ4qZNZhZ+oHWKiIjk0KbyBAT/kLv+uo0s6a4x5/oz2u5+7nufvtAXzsUmdl1Zjavh+MxM2s1s2MTfS13v8vd3zdAdb0jZLr7OnfPd/eOgXj9bu/lZlY+0K+b4Hu/38zmmdkuM6sxs2fM7MNR1CIyXCm0iQwx4S/0fHfPB9YB/xB37K6u68wsI7oqh6Q7gQozm9rt+MXAEndfGkFNI4KZnQ/8CfgDMAk4BPgO8A8H8FpmZvrdJNID/cUQSRFmdoaZVZvZN81sM/B7Mysys4fDlo3t4f1Jcc+J7/L7tJnNN7Ofh9euMbNzD/DaqXGtKk+Y2U1mdmcvdSdS4w/M7Lnw9R43s1jc+UvN7G0zqzWzf+nt++Pu1cCTwKXdTv0T8If91dGt5k+b2fy4x2eb2VtmtsPMfg1Y3LkyM3syrG+bmd1lZoXhuTuAKcD/hS2l3zCz0rBFLCO8ZoKZPWRmdWZWZWafi3vt75rZfWb2h/B7s8zMZvb2PeiNmY0JX6Mm/F5+uysYmVl52Cq2I6z/3vC4mdn1ZrbVzHaa2RLrobXSzAz4BfADd/+du+9w9053f8bdPxf3Oe6Me07378HTZvYjM3sOaAS+bmaLur3PNWb2UHg/O/yzuc7MtpjZf5nZqPBcLPzZ1off02dNIVCGCf1BFkkthwLFwGHAFQR/h38fPp4CNAG/7uP5pwHLgRjwM+DW8Jduf6+9G1gIlADfZd+gFC+RGj8OXAaMA7KAawHM7Gjg5vD1J4Tv12PQCt0eX4uZHQHMCOvt7/eq6zViwP3Atwm+F6uA2fGXAD8O6zsKmEzwPcHdL+WdraU/6+Et7gGqw+efD/y7mZ0Zd/7D4TWFwEOJ1NyDXwFjgGnAuwmC7GXhuR8AjwNFBN/bX4XH3wfMAQ4Pn3shUNvDax9B8JnnHkBd8S4l+DNdAPwXcISZTY87/3GCnyPAT8K6ZgDlwESClj2ArxF8P8cStPj9P0D7NcqwoNAmklo6gX9z9xZ3b3L3Wnf/s7s3uvsu4EcEv5R787a7/zYcT3U7MJ7gF1vC15rZFOAU4Dvu3uru8wnCRI8SrPH37r7C3ZuA+wh+GUMQYh5293nu3gL8a/g96M0DYY0V4eN/Ah5195oD+F51+QCwzN3nunsbcAOwOe7zVbn738KfSQ1Bq1Mir4uZTSYIgN9092Z3fxX4XVh3l/nu/kj4c7gDOCGR1457j3SCLuJvufsud18L/Cd7w20bQZCdENYwP+54AXAkYO7+prtv6uEtSsLbns71x/+4+zJ3b3f3HcCDwCXhZ5ge1vFQ+B+HK4Br3L0u/Fn+e/gZu+oeDxzm7m3u/qxrk20ZJhTaRFJLjbs3dz0ws1wz+++wy2snMA8otN5nJsaHjcbwbn4/r50A1MUdA1jfW8EJ1rg57n5jXE0T4l/b3XfTc2tPfJ1/Av4p/OX+CYJxVgfyverSvQaPf2xmh5jZPWa2IXzdOwla5BLR9b3cFXfsbYKWoy7dvzc51r/xjDEgM3zdnt7jGwSthQvD7tfPALj7kwStejcBW83sFjMb3cPrd/08xvejpp50/zN0N2FoI2hl+9/w5zsWyAUWh12g9cBfw+MA/wFUAY+b2Wozu+4g6xIZMhTaRFJL9xaDrxF0T53m7qMJurMgbsxVEmwCis0sN+7Y5D6uP5gaN8W/dvieJb1fDgStghcCZxO0FP3fQdbRvQbjnZ/33wl+LseFr/vJbq/ZVyvPRoLvZUHcsSnAhv3U1B/b2Nuats97uPtmd/+cu08APg/8xsIZqO7+S3c/GTiaoDvy6z28/nKCwPWxPmrYTRC0uhzawzXdv09/A8aa2QyC8NbVNbqNoGv7GHcvDL/GhBN3CFsTv+bu0wi6lr9qZmf1UZtIylBoE0ltBQS/wOrNrBj4t2S/obu/DSwCvmtmWWZ2On3PEjyYGucCHzKzSjPLAr7P/v/dehaoB24B7nH31oOs4y/AMWb20bCF6yreGToKgAZgh5lNZN9gs4VgLNk+3H09sAD4sZnlmNnxwOUErXUHKit8rRwzywmP3Qf8yMwKzOww4Ktd72FmF9jeCRnbCcJTp5mdYmanmVkmQehqpoeu6bDl8avAv5rZZWY22szSwp/ZLeFlrwJzLFijbgzwrf19iLAr+k8ELWfFBCEOd+8Efgtcb2bjws8w0czeH97/kAWTKwzYAXT0VLdIKlJoE0ltNwCjCFofXiDoJhoMnwBOJ+ga+yFwL9DSy7U3cIA1uvsy4EqCVpZNBKGiej/PcYIu0cPC24Oqw923ARcQDH6vBaYDz8Vd8j3gJIKA8BeCSQvxfgx8O+zKu7aHt7gEKCVodXuAYMziE4nU1otlBOG06+sy4MsEwWs1MJ/g+3lbeP0pwItm1kAwNvEr7r4aGE0QjrYTdKfWEgSofbj7XOAi4DPh59hC8OfiwfD83wj+jLwOLAYeTvCz3A28F/iTu7fHHf8mQRfoC2GX9BMEragQ/HyeIAjSzwO/cfenEnw/kSHNND5TRA6WBctEvOXuSW/pExEZqdTSJiL9FnadlYXdYOcA5wH/G3FZIiLDmlZUF5EDcShBN2AJQXflF9z9lWhLEhEZ3tQ9KiIiIpIC1D0qIiIikgIU2kRERERSwIgY0xaLxby0tDTqMkRERET2a/HixdvcfWz34yMitJWWlrJo0aKoyxARERHZLzN7u6fj6h4VERERSQEKbSIiIiIpQKFNREREJAWMiDFtIiIiMvS1tbVRXV1Nc3Nz1KUMipycHCZNmkRmZmZC1yu0iYiIyJBQXV1NQUEBpaWlmFnU5SSVu1NbW0t1dTVTp05N6DnqHhUREZEhobm5mZKSkmEf2ADMjJKSkn61Kiq0iYiIyJAxEgJbl/5+VoU2ERERkVB9fT2/+c1v+v28D3zgA9TX1w98QXEU2kRERERCvYW29vb2Pp/3yCOPUFhYmKSqApqIICIiIhK67rrrWLVqFTNmzCAzM5OcnByKiop46623WLFiBf/4j//I+vXraW5u5itf+QpXXHEFsHf3pYaGBs4991wqKytZsGABEydO5MEHH2TUqFEHXZtCm4iIiAw53/u/ZbyxceeAvubRE0bzb/9wTJ/X/OQnP2Hp0qW8+uqrPP3003zwgx9k6dKle2Z43nbbbRQXF9PU1MQpp5zCxz72MUpKSt7xGitXruSPf/wjv/3tb7nwwgv585//zCc/+cmDrl+hbQAs3bCD9k5nxuTCqEsRERGRAXTqqae+Y0mOX/7ylzzwwAMArF+/npUrV+4T2qZOncqMGTMAOPnkk1m7du2A1KLQNgC+et+rHDI6hzsuPy3qUkRERIaF/bWIDZa8vLw9959++mmeeOIJnn/+eXJzcznjjDN6XLIjOzt7z/309HSampoGpBZNRBgAFWUxXlpbR0t7R9SliIiIyEEoKChg165dPZ7bsWMHRUVF5Obm8tZbb/HCCy8Mam0KbQOgsjxGc1snL79dH3UpIiIichBKSkqYPXs2xx57LF//+tffce6cc86hvb2do446iuuuu45Zs2YNam3qHh0Ap00rJj3NWLBqG6eXlez/CSIiIjJk3X333T0ez87O5tFHH+3xXNe4tVgsxtKlS/ccv/baawesLrW0DYCCnEyOnzSG+VXboi5FREREhimFtgFSWR7j9eod7Gpui7oUERERGYYU2gZIRVmMjk7nxdV1UZciIiIiw5BC2wA56bBCcjLT1EUqIiIiSaHQNkCyM9I5pbSYBasU2kRERGTgKbQNoMryGCu2NLB1574L7YmIiIgcDIW2ATS7PAbAglW1EVciIiIigyE/Px+AjRs3cv755/d4zRlnnMGiRYsO+r0U2gbQ0eNHU5ibqXFtIiIiI8yECROYO3duUt9DoW0ApaUZFWUlLKjahrtHXY6IiIj003XXXcdNN9205/F3v/tdfvjDH3LWWWdx0kkncdxxx/Hggw/u87y1a9dy7LHHAtDU1MTFF1/MUUcdxUc+8pEB23tUOyIMsIqyGI8s2cza2kamxvL2/wQRERHZ16PXweYlA/uahx4H5/6kz0suuugirr76aq688koA7rvvPh577DGuuuoqRo8ezbZt25g1axYf/vCHMbMeX+Pmm28mNzeXN998k9dff52TTjppQMpXaBtgleG4tvlV2xTaREREUsyJJ57I1q1b2bhxIzU1NRQVFXHooYdyzTXXMG/ePNLS0tiwYQNbtmzh0EMP7fE15s2bx1VXXQXA8ccfz/HHHz8gtSm0DbDDSnKZWDiKBVXbuHTWYVGXIyIikpr20yKWTBdccAFz585l8+bNXHTRRdx1113U1NSwePFiMjMzKS0tpbl58FeK0Ji2AWZmzC4vYcGqWjo6Na5NREQk1Vx00UXcc889zJ07lwsuuIAdO3Ywbtw4MjMzeeqpp3j77bf7fP6cOXP2bDq/dOlSXn/99QGpS6EtCWaXx9jR1MYbG3dGXYqIiIj00zHHHMOuXbuYOHEi48eP5xOf+ASLFi3iuOOO4w9/+ANHHnlkn8//whe+QENDA0cddRTf+c53OPnkkwekLnWPJsHpZSVAMK7tuEljIq5GRERE+mvJkr2TIGKxGM8//3yP1zU0NABQWlrK0qVLARg1ahT33HPPgNeklrYkGFeQwxGHFGhLKxERERkwCm1JUlFewktr62hu64i6FBERERkGkhrazOwcM1tuZlVmdl0P57PN7N7w/ItmVhp37lvh8eVm9v6444VmNtfM3jKzN83s9GR+hgNVWR6jua2Tl9dtj7oUERERGQaSFtrMLB24CTgXOBq4xMyO7nbZ5cB2dy8Hrgd+Gj73aOBi4BjgHOA34esB3Aj81d2PBE4A3kzWZzgYp04tJj3NWFClfUhFREQSNZJ2FOrvZ01mS9upQJW7r3b3VuAe4Lxu15wH3B7enwucZcHywucB97h7i7uvAaqAU81sDDAHuBXA3VvdvT6Jn+GAFeRkMmNyofYhFRERSVBOTg61tbUjIri5O7W1teTk5CT8nGTOHp0IrI97XA2c1ts17t5uZjuAkvD4C92eOxFoAmqA35vZCcBi4Cvuvrv7m5vZFcAVAFOmTBmIz9Nvs8tK+PVTVexsbmN0TmYkNYiIiKSKSZMmUV1dTU1NTdSlDIqcnBwmTZqU8PWptuRHBnAS8GV3f9HMbgSuA/61+4XufgtwC8DMmTMjiewV5TF++WQVL6yq5X3H9LzVhYiIiAQyMzOZOnVq1GUMWcnsHt0ATI57PCk81uM1ZpYBjAFq+3huNVDt7i+Gx+cShLgh6cQphYzKTGfBKo1rExERkYOTzND2EjDdzKaaWRbBxIKHul3zEPCp8P75wJMedGQ/BFwczi6dCkwHFrr7ZmC9mR0RPucs4I0kfoaDkp2RzilTi3lO49pERETkICUttLl7O/Al4DGCGZ73ufsyM/u+mX04vOxWoMTMqoCvEnR14u7LgPsIAtlfgSvdvWvBsy8Dd5nZ68AM4N+T9RkGQmV5CSu3NrBl5+BvLCsiIiLDR1LHtLn7I8Aj3Y59J+5+M3BBL8/9EfCjHo6/Cswc0EKTqKIsBsCCVdv4yImJDzYUERERiacdEZLs6PGjKcrNZP5KjWsTERGRA6fQlmRpaUZFWYwFq7aNiHVnREREJDkU2gZBRXkJm3Y0s3rbPsvJiYiIiCREoW0QVJaH49o0i1REREQOkELbIJhSnMvEwlE8p31IRURE5AAptA0CM6OyPBjX1tGpcW0iIiLSfwptg6SivISdze0s27gj6lJEREQkBSm0DZKu9drma1ybiIiIHACFtkEytiCbIw8tYIHGtYmIiMgBUGgbRBVlMV5aW0dzW8f+LxYRERGJo9A2iCqnl9DS3snLb2+PuhQRERFJMQptg+jUqSVkpBnPrdK4NhEREekfhbZBlJ+dwYzJhczXuDYRERHpJ4W2QVZRHmNJdT07mtqiLkVERERSiELbIKssj9Hp8MJqtbaJiIhI4hTaBtmMyYWMykzXPqQiIiLSLwptgywrI41TpxZrkV0RERHpF4W2CFSWx1hVs5vNO5qjLkVERERShEJbBCrKSwBYoKU/REREJEEKbRE46tDRFOdlqYtUREREEqbQFoG0NOP0shIWVNXi7lGXIyIiIilAoS0ileUxNu9sZlXN7qhLERERkRSg0BaR2WUxQOPaREREJDEKbRGZUpLLpKJRzF+p0CYiIiL7p9AWocryGC+srqWjU+PaREREpG8KbRGqKI+xs7mdpRt2RF2KiIiIDHEKbRGqKAvWa9PSHyIiIrI/Cm0RiuVnc+ShBZqMICIiIvul0BaxyvIYL63dTnNbR9SliIiIyBCm0Bax2eUxWts7Wfz29qhLERERkSFMoS1ip04tJiPNNK5NRERE+qTQFrG87AxOnFLIAoU2ERER6YNC2xBQURZjyYYd7Ghsi7oUERERGaIU2oaAyukxOh2eX10bdSkiIiIyRCm0DQEnTCokNytdS3+IiIhIrxTahoCsjDROm1qsyQgiIiLSK4W2IWJ2eYzVNbvZtKMp6lJERERkCFJoGyIqymIAPFelcW0iIiKyL4W2IeLIQwsoycvS0h8iIiLSI4W2ISItzTi9rITnVm3D3aMuR0RERIaYpIY2MzvHzJabWZWZXdfD+Wwzuzc8/6KZlcad+1Z4fLmZvT/u+FozW2Jmr5rZomTWP9gqy2Ns2dnCqpqGqEsRERGRISZpoc3M0oGbgHOBo4FLzOzobpddDmx393LgeuCn4XOPBi4GjgHOAX4Tvl6X97j7DHefmaz6ozC7XOPaREREpGfJbGk7Fahy99Xu3grcA5zX7ZrzgNvD+3OBs8zMwuP3uHuLu68BqsLXG9YmF+cypThXS3+IiIjIPpIZ2iYC6+MeV4fHerzG3duBHUDJfp7rwONmttjMrujtzc3sCjNbZGaLampqDuqDDKbZ5SW8sLqW9o7OqEsRERGRISQVJyJUuvtJBN2uV5rZnJ4ucvdb3H2mu88cO3bs4FZ4ECrKYuxqbmfJhh1RlyIiIiJDSDJD2wZgctzjSeGxHq8xswxgDFDb13Pdvet2K/AAw6zbtKKsBIAFqzSuTURERPZKZmh7CZhuZlPNLItgYsFD3a55CPhUeP984EkP1rt4CLg4nF06FZgOLDSzPDMrADCzPOB9wNIkfoZBV5KfzVHjR/OcxrWJiIhInIxkvbC7t5vZl4DHgHTgNndfZmbfBxa5+0PArcAdZlYF1BEEO8Lr7gPeANqBK929w8wOAR4I5iqQAdzt7n9N1meISmV5Cbc//zbNbR3kZKbv/wkiIiIy7NlIWMh15syZvmhR6izp9tTyrVz2+5e48/LTqJwei7ocERERGURmtrinZc1ScSLCsHdqaTGZ6aalP0RERGQPhbYhKC87gxMnF7FglUKbiIiIBBTahqiK8hKWbNhBfWNr1KWIiIjIEKDQNkRVlsdwhxdWa+kPERERUWgbsk6YXEheVrr2IRURERFAoW3IykxP47RpJVqvTURERACFtiGtoqyE1dt2s7G+KepSREREJGIKbUNY1xptam0TERERhbYh7IhDCojlZ2kfUhEREVFoG8rMjNPLYsyv2sZI2LlCREREeqfQNsRVlpdQs6uFqq0NUZciIiIiEVJoG+IqyjSuTURERBTahrzJxbkcVpLLfK3XJiIiMqIptKWAirIYL66upb2jM+pSREREJCIKbSmgsjzGrpZ2Xt+wI+pSREREJCIKbSng9LISABZoXJuIiMiIpdCWAorzsjh6/GjmK7SJiIiMWAptKaJyeoyX366nqbUj6lJEREQkAgptKaKirITWjk4WvV0XdSkiIiISAYW2FHHq1GIy001dpCIiIiOUQluKyM3K4MQpRSzQem0iIiIjkkJbCqksj7F04w7qG1ujLkVEREQGmUJbCpldXoI7PL9KrW0iIiIjjUJbCjl+UiF5Weka1yYiIjIC7Te0mdlXzGy0BW41s5fN7H2DUZy8U2Z6GrOmlbBALW0iIiIjTiItbZ9x953A+4Ai4FLgJ0mtSnpVUR5jzbbdbKhviroUERERGUSJhDYLbz8A3OHuy+KOySCrLI8B8Jy6SEVEREaURELbYjN7nCC0PWZmBUBncsuS3hx+SD6x/GztQyoiIjLCZCRwzeXADGC1uzeaWTFwWVKrkl6ZGbPLS3huVS3ujpkaPUVEREaCRFraTgeWu3u9mX0S+DawI7llSV9ml8Wo2dXCyq0NUZciIiIigySR0HYz0GhmJwBfA1YBf0hqVdKnivISAOavVBepiIjISJFIaGt3dwfOA37t7jcBBcktS/oyqSiX0pJcFqxSaBMRERkpEgltu8zsWwRLffzFzNKAzOSWJftTUR7jhdV1tHdoToiIiMhIkEhouwhoIVivbTMwCfiPpFYl+1VZHqOhpZ3XqjW8UEREZCTYb2gLg9pdwBgz+xDQ7O4a0xax06eVYIaW/hARERkhEtnG6kJgIXABcCHwopmdn+zCpG9FeVkcM2G09iEVEREZIRJZp+1fgFPcfSuAmY0FngDmJrMw2b/ZZTF+/9xaGlvbyc1K5EcpIiIiqSqRMW1pXYEtVJvg8yTJKspjtHZ08tLa7VGXIiIiIkmWSPj6q5k9ZmafNrNPA38BHk1uWZKIU0qLyEpP07g2ERGREWC/fWru/nUz+yhQGR66xd0fSG5ZkojcrAxOnFLIc1qvTUREZNhLqJvT3e9396+GXw+Y2bpEnmdm55jZcjOrMrPrejifbWb3hudfNLPSuHPfCo8vN7P3d3teupm9YmYPJ1LHcFZZHmPZxp1s390adSkiIiKSRAc6Nm2/u5SbWTpwE3AucDRwiZkd3e2yy4Ht7l4OXA/8NHzu0cDFwDHAOcBvwtfr8hXgzQOsfVipKI/hDs+vro26FBEREUmiAw1tnsA1pwJV7r7a3VuBewi2wop3HnB7eH8ucJaZWXj8Hndvcfc1QFX4epjZJOCDwO8OsPZh5YRJY8jPztDSHyIiIsNcr2PazOyrvZ0C8hN47YnA+rjH1cBpvV3j7u1mtgMoCY+/0O25E8P7NwDfYD/7n5rZFcAVAFOmTEmg3NSUkZ7GrGnFmowgIiIyzPXV0lbQy1c+cGPyS9tXuCPDVndfvL9r3f0Wd5/p7jPHjh07CNVFp6IsxtraRqq3N0ZdioiIiCRJry1t7v69g3ztDcDkuMeTwmM9XVNtZhnAGIJ14Hp77oeBD5vZB4AcYLSZ3enunzzIWlNa5fQYAAuqarnwlNyIqxEREZFkSOYiuS8B081sqpllEUwseKjbNQ8Bnwrvnw886e4eHr84nF06FZgOLHT3b7n7JHcvDV/vyZEe2ACmj8tnbEG2lv4QEREZxpK291E4Ru1LwGNAOnCbuy8zs+8Di9z9IeBW4A4zqwLqCIIY4XX3AW8A7cCV7t6RrFpTnZkxu6yE+VW1uDvBXA4REREZTixo2OrjArP0VA9MM2fO9EWLFkVdRlLdt2g935j7Oo9dPYcjDu1zjoaIiIgMYWa22N1ndj+eSPfoSjP7jx7WWJMhZHZ5MK5NS3+IiIgMT4mEthOAFcDvzOwFM7vCzEYnuS7pp4mFo5gay9PSHyIiIsPUfkObu+9y99+6ewXwTeDfgE1mdruZlSe9QklYRVkJL6yupa2jM+pSREREZIDtN7SF+3x+2MweIFjY9j+BacD/AY8ktzzpj8ryGLtbO3i9uj7qUkRERGSAJTJ7dCXwFPAf7r4g7vhcM5uTnLLkQJxeVoIZPFdVy8mHFUddjoiIiAygRMa0He/ul3cLbAC4+1VJqEkOUGFuFsdOGKPJCCIiIsNQIqFtnJn9n5ltM7OtZvagmU1LemVyQCrKS3hl3XYaW9ujLkVEREQGUCKh7W7gPuBQYALwJ+CPySxKDlxleYy2DmfhmrqoSxEREZEBlEhoy3X3O9y9Pfy6k2DfTxmCZh5WTFZ6GgtW1UZdioiIiAygRCYiPGpm1wH3AA5cBDxiZsUA7q4mnSFkVFY6Jx1WyPyVGtcmIiIynCQS2i4Mbz/f7fjFBCFO49uGmMryGD9/fAV1u1spzsuKuhwREREZAIksrju1jy8FtiGoItzS6nl1kYqIiAwbiSyum2lmV5nZ3PDrS2aWORjFyYE5fuIYCrIztPSHiIjIMJJI9+jNQCbwm/DxpeGxzyarKDk4GelpnDathAWrFNpERESGi0RC2ynufkLc4yfN7LVkFSQDo7K8hCfe3ML6ukYmF+dGXY6IiIgcpESW/Ogws7KuB+HCuh3JK0kGwuxwXJta20RERIaHRELbtcBTZva0mT0DPAl8LbllycEqH5fPuIJs5ldpMoKIiMhw0Gf3qJmlAycA04EjwsPL3b0l2YXJwTEzZpfHmLeihs5OJy3Noi5JREREDkKfLW3u3gFc4u4t7v56+KXAliIqykqo3d3K8i27oi5FREREDlIi3aPPmdmvzexdZnZS11fSK5OD1jWu7Tkt/SEiIpLyEpk9OiO8/X7cMQfOHPBqZEBNKBzFtFgeC1bV8tl3aR1kERGRVJZIaLvc3VfHHwhnkEoKmF0e4/6Xq2nr6CQzPZGGVRERERmKEvktPreHY38a6EIkOWaXl7C7tYPX1tdHXYqIiIgchF5b2szsSOAYYIyZfTTu1GggJ9mFycCYNa0EM5hftY2ZpcVRlyMiIiIHqK/u0SOADwGFwD/EHd8FfC6JNckAKszN4riJY1hQVcvV7426GhERETlQvYY2d38QeNDMTnf35wexJhlgFWUxfvfsana3tJOXncgwRhERERlqEhnTVmVm/8/MbjGz27q+kl6ZDJjK8hjtnc7CtXVRlyIiIiIHKJFmlweBZ4En0J6jKWlmaRFZGWksqNrGe44YF3U5IiIicgASCW257v7NpFciSZOTmc7Mw4q0D6mIiEgKS6R79GEz+0DSK5Gkml0e481NO6lt0C5kIiIiqSiR0PYVguDWZGY7zWyXme1MdmEysCrKSgBYsEqtbSIiIqlov6HN3QvcPc3dR7n76PDx6MEoTgbOcRPHUJCTwYJV2odUREQkFfUa2szsk3H3Z3c796VkFiUDLyM9jVnTSpivzeNFRERSUl8tbV+Nu/+rbuc+k4RaJMkqy2Osr2tifV1j1KWIiIhIP/UV2qyX+z09lhQwuzwY1/acWttERERSTl+hzXu539NjSQFlY/M5ZHS2ukhFRERSUF/rtB1pZq8TtKqVhfcJH09LemUy4MyM2WUxnllRQ2enk5amBlMREZFU0VdoO2rQqpBBU1Ee4/5XNvDW5l0cPUGTgEVERFJFXxvGv939mJl9yN0fTm5Jkkxd49oWrNqm0CYiIpJCEllcN973k1KFDJrxY0YxbWyexrWJiIikmP6Gtn4NgjKzc8xsuZlVmdl1PZzPNrN7w/Mvmllp3LlvhceXm9n7w2M5ZrbQzF4zs2Vm9r1+1i8ES38sXFNHa3tn1KWIiIhIgvob2j6f6IVmlg7cBJwLHA1cYmZHd7vscmC7u5cD1wM/DZ97NHAxcAxwDvCb8PVagDPd/QRgBnCOmc3q52cY8SrKYjS2dvBadX3UpYiIiEiC9hvazOwCMysIH77fzO43s5MSeO1TgSp3X+3urcA9wHndrjkPuD28Pxc4y8wsPH6Pu7e4+xqgCjjVAw3h9Znhl5Yf6afTp5WQZjB/pbpIRUREUkUiLW3/6u67zKwSOBO4Fbg5gedNBNbHPa4Oj/V4jbu3AzuAkr6ea2bpZvYqsBX4m7u/2NObm9kVZrbIzBbV1NQkUO7IMSY3k+MmjtE+pCIiIikkkdDWEd5+EPitu/8FyEpeSX1z9w53nwFMAk41s2N7ue4Wd5/p7jPHjh07qDWmgoryGK+sq2d3S3vUpYiIiEgCEgltG8zsv4GLgEfMLDvR5wGT4x5PCo/1eI2ZZQBjgNpEnuvu9cBTBGPepJ8qy2O0dzoL19RFXYqIiIgkIJHwdSHwGPD+MCgVA19P4HkvAdPNbKqZZRFMLHio2zUPAZ8K758PPOnuHh6/OJxdOhWYDiw0s7FmVghgZqOAs4G3EqhFujn5sCKyMtK09IeIiEiK6GtHhC7jgb+4e4uZnQEcD/xhf09y93Yz+xJB4EsHbnP3ZWb2fWCRuz9EMD7uDjOrAuoIgh3hdfcBbwDtwJXu3mFm44Hbw5mkacB9Wuz3wORkpnNKaZE2jxcREUkRFjRs9XFBMOh/JlAKPAI8CBzj7h9IdnEDZebMmb5o0aKoyxhybnqqiv94bDmLvv1eYvnZUZcjIiIigJktdveZ3Y8n0j3aGc7s/CjwK3f/OkHrm6S4yvIYAAtW1UZciYiIiOxPIqGtzcwuAf4J6OqKzExeSTJYjp04htE5GSxQF6mIiMiQl0houww4HfiRu68JJwbckdyyZDCkpxmzppVoMoKIiEgK2G9oc/c3gGuBJeGaaNXu/tOkVyaDonJ6jOrtTayrbYy6FBEREelDIttYnQGsJNhH9DfACjObk9yyZLBUlAXj2tTaJiIiMrQl0j36n8D73P3d7j4HeD/B5u4yDJSNzePQ0Tk8py2tREREhrREQlumuy/veuDuK9BEhGHDzKgoL+H5VbV0dva9/IuIiIhEJ5HQttjMfmdmZ4RfvwW06NkwUlkeo253K29u3hl1KSIiItKLRELbPxPsTHBV+PUG8IVkFiWDa3bXem1VWq9NRERkqOpzG6twu6jX3P1I4BeDU5IMtkNG51A2No/5Vdv43JxpUZcjIiIiPeizpc3dO4DlZjZlkOqRiFSWx1i4po7W9s6oSxEREZEeJNI9WgQsM7O/m9lDXV/JLkwGV0V5jKa2Dl5Ztz3qUkRERKQHfXaPhv416VVI5GZNKyHN4LlVtZw2rSTqckRERKSbXlvazKzczGa7+zPxX0AHUD14JcpgGDMqk+MmFWofUhERkSGqr+7RG4Ce1oDYEZ6TYaayvIRX19fT0NIedSkiIiLSTV+h7RB3X9L9YHisNGkVSWRml8Vo73QWrtHSHyIiIkNNX6GtsI9zowa4DhkCTjqsiOyMNOavVGgTEREZavoKbYvM7HPdD5rZZ4HFyStJopKTmc4ppcUs0D6kIiIiQ05fs0evBh4ws0+wN6TNBLKAjyS5LolIRXkJP/vrcmp2tTC2IDvqckRERCTUa0ubu29x9wrge8Da8Ot77n66u28enPJksFV2bWml1jYREZEhZb/rtLn7U8BTg1CLDAHHTBjD6JwMFlTVct6MiVGXIyIiIqFEdkSQESQ9zagoizG/ahvuHnU5IiIiElJok33MLi9hQ30T6+oaoy5FREREQgptso+KcFzbfO2OICIiMmQotA2EujXQ3hp1FQNmWiyP8WNyWFCl9dpERESGCoW2g9XZCfd8An45Axb+Ftqao67ooJkF49oWrNpGZ6fGtYmIiAwFCm0Hywze9wMonAKPXAs3ngDP3wStqT0erHJ6Cdsb23hjU0/bz4qIiMhgU2g7WGZQfhZc9ih86mGITYfH/h/ccBzMvx5adkVd4QGpKNN6bSIiIkOJQttAMYOp74JPPwyfeQzGnwBPfBeuPxae/ik01UddYb8cMjqH6ePyma9xbSIiIkOCQlsyTJkFl94Pn30SppwOT/970PL25A+hsS7q6hI2uzzGS2vqaGnviLoUERGREU+hLZkmnQwfvwc+/yxMOwPm/UcQ3v72HWjYGnV1+1VRVkJTWwevrKuPuhQREZERT6FtMIw/Hi66A774Ahx+Diz4FdxwPDx6HezcFHV1vZpVVkKawQKt1yYiIhI5hbbBNO4oOP9WuPIlOOYjsPAWuPF4+MvXoH591NXtY3ROJsdPKtQiuyIiIkOAQlsUYuXwkZvhy4vhhEtg8e3wyxPhoS8HC/UOIZXlMV6r3sGu5raoS5GDsfa54EtERFKWQluUiqfCh38JV70CJ38aXrsXfnUyPPDPsG1l1NUBUFFeQkens3BN6kygkDjb34Z7Pwn/8wH4nw/CvJ+Da8FkEZFUpNA2FBROhg/+HK5+HU77Z1j2v/DrU+BPl8GWNyIt7aQpReRkpqmLNNW0NsJTP4abToWqv8OZ34bjzocnfwB/+hS0NERdoYiI9FNG1AVInIJD4Zx/h8pr4Plfw0u/g2X3w1H/AHO+Hqz9NshyMtM5pbSY56q24e6Y2aDXIP3gDm8+BI/9C+xYD8d+DM7+AYyZGJw79Hh44t+gdhVcfBcUlUZdsYiIJEgtbUNR/lg4+3tw9RKY8w1YPQ/+ew7cfRFULxr0cs48chwrtjTw0ZsX8OzKGlzda0PT1jfhD+fBff8E2aPh03+B828LAhsEC0DPvgo+8acg0N3yHlj9TLQ1i4hIwmwk/AKeOXOmL1o0+GFnwDTVB5vRv3ATNG2Hae+Bd38DDqsYlLfv6HTuW7SeX/19JRt3NHNKaRHXnH34nq2uJGJN9fDMT+HF/4bsgqAr9OTLIL2PhvTaVXDPx4Oxk+/7Icz6QhDqREQkcma22N1n7nNcoS2FtOyCl24Nuk5310Dpu4Ju06lzBuUXbkt7B/e9tJ5fP1XFlp0tnD6thK++73BOKS1O+ntLDzo74dU74YnvQWNtMJnlzH+FvJLEnt+yK5j08tbDcMLH4UPXQ2ZOUksWEZH9iyS0mdk5wI1AOvA7d/9Jt/PZwB+Ak4Fa4CJ3Xxue+xZwOdABXOXuj5nZ5PD6QwAHbnH3G/dXx7AJbV1aG2Hx/8BzN0LDZph8WtCNWn7WoIS35rYO/rhwHTc9tYptDS28a3qMa84+nJOmFCX9vSVUvQge+TpsfBkmz4IP/OzAxjx2dsK8n8HTP4aJJ8NFd8LoCQNfr4iIJGzQQ5uZpQMrgLOBauAl4BJ3fyPumi8Cx7v7P5vZxcBH3P0iMzsa+CNwKjABeAI4HBgHjHf3l82sAFgM/GP8a/Zk2IW2Lm3N8ModMP8G2FkNE04MwtsR5w5KeGtq7eDOF97m5mdWUbe7lfccMZZrzj6c4ycVJv29R6yGrfDEd+HVuyD/UHjfD+C4Cw7+5/3mw/DA5yErDy68A6acNiDliohI/0UR2k4Hvuvu7w8ffwvA3X8cd81j4TXPm1kGsBkYC1wXf238dd3e40Hg1+7+t75qGbahrUt7K7z2R5j/C9i+Fg45FuZcC0edB2nJn2uyu6Wd259fyy3zVlPf2MZ7jzqEa86ezjETxiT9vUeMjrZgzNozP4W2Jjj9yuBnnF0wcO+x9U344yWwoxo++J9w8qcG7rVFRCRhvYW2ZP5GnwjE781UHR7r8Rp3bwd2ACWJPNfMSoETgRd7enMzu8LMFpnZopqamgP/FKkgIyv4BfulxfCP/wXtzfCnT8PNp8Prf4LOjqS+fV52Bl88o5xnv/Eevnb24SxcU8sHfzmfL9y5mOWbdyX1vUeEVU/CzbPh8X8JusK/+EIwu3ggAxsE26xd8RRMfRf831XB9mod2glDRGSoSMklP8wsH/gzcLW77+zpGne/xd1nuvvMsWPHDm6BUUnPgBmXwJUL4WO3Agb3fzZYqPeVu5L+C7ggJ5MvnzWdZ795JledNZ1nV27jnBvn8aW7X6ZqqxZz7bfta+GeT8AdH4GOVrjk3mC5jlh58t5zVBF8Yi5UXBWsE/iH86BhmP+nR0QkRSQztG0AJsc9nhQe6/GasHt0DMGEhF6fa2aZBIHtLne/PymVp7q09GD1+y8sCMYnZeXCg1+EX50Ei26D9pakvv2YUZl89ezDmf/N9/DFM8p48q2tvO/6Z7jm3ldZs213Ut97WGhthKf+HW46LWhlO+s7QevaEecMzrIcaenBWLmP/hY2LIZbzoCNryb/fUVEpE/JHNOWQTAR4SyCwPUS8HF3XxZ3zZXAcXETET7q7hea2THA3eydiPB3YDrQCdwO1Ln71YnWMuzHtO2PO6x4LJgluGExjJ4Is6+Gky6FzFFJf/vahhZumbea259fS1uH89ETJ/LlM6czpSQ36e+dUtzhjQfh8W+HuxmcD2d/f+/iuFHY+Arc80lo3Abn3RT8Z0BERJIqqiU/PgDcQLDkx23u/iMz+z6wyN0fMrMc4A6CsWl1wMXuvjp87r8AnwHaCbpBHzWzSuBZYAlBgAP4f+7+SF91jPjQ1sU9aLmZ9x+w7nnIPwQqvgwzPxPMGkyyrbua+a+nV3Pni2/T2elcMHMSV76nnElFCm9sfRMe/QasmRdMJDn3Z1A6O+qqAg01wS4L6xYE3abv/W7QGiciIkmhxXUV2vZyh7Xzg5a3NfMgtySYjXjK5yBndNLffsvOZn7zVBV/XLgex7n4lClc+Z5yDh0zAhd2baqHp38CC29JfDeDKLS3wmPfCsa5lZ0F598ajH8TEZEBp9Cm0NazdS8G4a3qCcgpDLYzOu3zg/ILeUN9Ezc9VcV9L60nLc34+KlT+OJ7yhhXMALCW/fdDGZeFuxmkDvEd5dY/D/wl2uhcDJcfHcw41RERAaUQptCW982LIZ5P4fljwSbjZ96Bcz6YuJbIh2E9XWN/PrJKua+XE1munHprMP4/LvLiOVnJ/29I7H+JXj068F4sYPZzSAq616Aey+Ftkb46C1w5AejrkhEZFhRaFNoS8zmJcGYtzcegsxcOOXyYNxb/rikv/Xabbv55ZMr+d9XNpCdkc6nKkr5/JxpFOVlJf29B8WuLcFuBq/dPbC7GURhxwa495PBNlpnfCvYiWMQFnIWERkJFNoU2vpn61vw7M9h6Z8hPQtmfAKmvw8OOx1ykrvTwaqaBm58YiX/9/pGcjPT+UzlVD5bOY0xuZlJfd+k6drN4OmfBAsfJ2M3gyi0NcPDVwe7cRz5IfjIf6X+ZxIRGQIU2hTaDsy2qmB7rCV/ChZ4tTQYPwNKK2HqHJgyK2m/qFds2cWNT6zkL0s2UZCTwWcrp3FZZSmjc1IovK16Eh79JmxbEYTec34CJWVRVzVw3OGFm4NlSmKHw8V3Da/PJyISAYU2hbaD09YE6xcGs07XPgvVi6CzDSwdJp4Epe8KgtyUWQO+fMibm3Zy/d9W8PgbWxgzKpMr5kzjUxWl5GcPsRmW8bavhcf+Bd56GIqmBmHtiHOirip5Vj8dbJ3mnXD+76H8rKgrEhFJWQptCm0Dq3U3rH8R1jwbBLmNL0NnO6RlwsSTg/0rS98Fk08dsAV8l1Tv4IYnVvD3t7ZSnJfF5+dM49LTDyM3awiFt9ZGmH89PHdjsJbZnGvh9C9BxjCdVBGvbk2w7VbNm/De7wVjIVNxvJ6ISMQU2hTakqulIZhVuHZeGOJeCVpd0rNg0ilBgJv6ruD+QQaYV9Zt5/onVjJvRQ2x/Cz++d1lfHLWYeRkRrjgqzu88b/w2LdhZ/XQ2M0gCi0NwZZpbzwIx10IH/7loOy6ISIynCi0KbQNruadwa4La+YF3ambXgccMnKC4DZ1ThDkJp4MGQc2O3TR2jquf2IFz1XVMq4gmyvfU87Fp04mO2OQw9uWN4LdDNY+C4ccFyzhcVjF4NYwlLgHk1ie/BGMPx4uuitY101ERBKi0KbQFq2m7fD280GwWfMsbFkSHM8YBVNOC1vi5sCEEyG9fxMNXlhdyy8eX8HCtXWMH5PDl84s54KTJ5OVkeQlKJrq4ekfw8LfBjtJnPltOOnTQ283g6gsfxT+/LmgZfWiO0Z2kBUR6QeFNoW2oaWxDt5+LhwT9yxsfSM4npUfTGYoDcfEjT8hoRDk7ixYVct/Pr6cl9fVM7FwFFedVc5HT5pEZvoAh7fODnjlTvj794IwevJlQWAb6rsZRKFmBdxzSTAx49yfwszLNc5NRGQ/FNoU2oa2hhp4e/7eiQ3blgfHs0fDlNP3Tmw49Lg+Nyt3d55ZUcP1f1vBa9U7OKwkl6vOnM55MyaQMRDhbf1L8Mi1sOnVoK5zf5pauxlEoake7v8crHwcTvoUfODnB9wlLpJy3KF+XTBxq+YtKCqFQ44NtoDTeE/phUKbQltq2bUlaIFbG4a42qrgeM4YOGz23okN447pcSV+d+fJt7byi7+tYNnGnUyL5fGV907nQ8dPID3tAFp64nczKBgPZ/8AjjtfrUaJ6uyAJ38YrPk3+TS48A4oOCTqqkQGXkdbsLPM+heDyVnrX4Rdm8KTBoS/cy0NSqbDoccGIe7Q44P7+Yfo3xVRaFNoS3E7NwbhrWtiw/a1wfFRRUGI65rYMO6od/yD5+48tmwLNzyxgrc272L6uHyufu/hnHvsoaQlEt7aW2Hhf8PTPw12M6j4ErzrWsjOT87nHO6W3g8PXgk5hXDxncFEFJFU1lQP1S/tDWkbFgf78gKMmRz8J2XKrOB23FFBq9uWpUGw27w0uL9j/d7Xy43FBbnjgq/Y4f0e6yupTaFNoW14qV+/d6HfNc/CjnXB8dxYsMhv144NscPBjM5O55Glm7jhiZVUbW3gyEMLuPq9h/P+Yw7BevtfbdXfg90MalcOz90MorJ5Cfzx49CwBf7hRphxSdQViSTGPfgPY3wr2tY3AQ8WGj/0uDCgnQqTZyW+5E/TdtiyLAxxS4K/I1vfgo6W4Hx6Fow9IpidHh/oNI522FJoU2gb3ravDVviwi7VnRuC4/mHhCEuGBPXUTSNh5cE4W3Ntt0cM2E0Xz37cM48ctze8Fa3JtjNYPlfoHhaENYOf39kH21Y2l0Lf/pU8LOa9cWgu1mzbmWo6WgLlita/8LekNawJTiXPTpYvqirFW3iyQPbAt/RHvyHcU+QC1vndm/de83oiWGAiwtyxdP6HPcrqUGhTaFt5HCHutV7x8OteRYaNgfnCiZAaSUdh1XyeON0fvx8M+u2N3HC5EIunlHCB3bcw5jFv4G0jHA3gytHxm4GUehoC/YsffG/YOq74YL/UcuBRKtpezDZaP0LsO7FoKuzvSk4VzglaD2bclpwO+6oaMJRw9YgvG1Zurd7tWY5eEdwPjMXxh39ziB3yDFJ2yNakkOhTaFt5HIPJjJ0jYdbOx921wSnRk9kbcHJPLa1kH9o/QsTrZYnMuaw5KivcdJxx3Da1OJod1oYCV65Ex6+JpjgcfHdwS8bkWTr+s9dfFdnzVvBOUsPFoaOD2mjx0dbb1/amoPa44Pc5iXQXL/3mqKpYZCL62ItnKJJD0OUQptCm3RxD/5nuvbZMMjNh6Y6WkqO5m+l13Lv1km8uKaO1vZOcjLTmDWthHcfPpZ3Hz6WqbG83sfAyYFb/xLc+0lo2Qn/eDMc849RVyTDTXsrbHotrqtz4d6uxuwxwTi0roA28STIyou23oPlDjuq44Jc2MVat5o9M1izx8S1yB2rpUiGEIU2hTbpTWdnMHtrzKQ93R1NrR28sKaWZ5bX8MyKGtZs2w3AlOLcPQHu9LIS8rI1DmvA7NwE910azMR717Xwnn/pcTkXkYQ01gXBrKurc+PLwQxwCNZKi29FG3vkyPmz1tIQTJ7Y/HpcoFsGbcG/cXuXIjnunS1zWopkUCm0KbTJQXi7djfzVtTw9PIaFqyqpamtg6z0NE6ZWsS7Dx/LGUeMY/q4fLXCHaz2FvjLV4Mu08PPgY/eEqzNJ9IXd6hdFQS09S8GIa1rge60jGAB7D0h7TQoODTaeoeazk7YvmbfsXI9LUVy6HF7g5yWIkkahTaFNhkgLe0dLFq7naeXb+WZFTWs2NIAwPgxOWGAG0tFeYzROfrH7IC4B/u5/vW6YImVi++G2PSoq5KhpL0FNr66txVt/YvQuC04l1MYBLPJpwYzOyecBFm5UVabuuKXItm8JOhi3d9SJKMnBt/vzNygi1mh7oAotCm0SZJsrG/a0wr3XNU2drW0k55mnDyliHcfEXSlHj1+dGKL+cpea54NlgXpaIOP3QqHvy/qiiQqu2uDYLanq/OVvcGheNo7uzpjh4+crs4o7LMUSThWLn4pknjpWXsDXNftnvu5wX7TXfczw3N77nc9Ly8uCObvvT+MlzZRaFNok0HQ1tHJK+vq97TCLdu4E4BYfjZzDo9xxhHjeFd5jKI87b2ZkPp1cM/Hg18KZ30HKq/RuJrhrmu297oX9oa02pXBubRMmDDjnbsM5I+LtFwJNWwNulR3b4PW3cGuEK2N0Nqw937b7uDcnvuN4bmG4H5XEE9URk4PoS6RULif85mjIv93RqFNoU0isHVXM8+u2MbTK2p4dmUN9Y1tmMEJkwo5I2yFO35S4YHthzpStDYGW18tux+O+Qicd1Pqz+zrTXtLsE/lzk2wa2N42/W1GTpaCfavJPylYr3cksA1g/k6XdfQ9zW7NoVdnbXB9aOKwq7OMKRNOFEzG4ezjvYwxO3udtsQF/B293B+9/6DYmd7PwqxuFDXLRTmjIGL7kjat2BPBQptCm0SrY5O5/Xqep4OZ6S+Vl2POxTlZvKu6UGAm3P4WMYWaDHffbjDczfAE98Lxs1cfBcUHRZ1VYlzD4LIzo17Q1j3YLZzIzTV7fvcjJxgDbuC8XELPXvwmntu2ffxPtf08JwezxHd64wqfGdIK5murk4ZGO2tPbfuJRQE4+5jcPljSS9XoU2hTYaY7btbmbcyCHDzVtSwraEVgGMmjA5b4cZx4pRCMtP1S2uPlX+DuZcHY1kuvD3YXzZqbU1xIWzT3mC2c2PQOrZrY1wrWTyDvLHBoq0FE8Lb8Cv+WE5h5F01IjK4FNoU2mQI6+x03ti0k2dW1PDM8hoWr9tOR6dTkJNBZXksWBvuiLGMH6OuIbZVBePcaqvgnB/DqVckJ9R0dgYzEnsKYfEBLX7V+S6Zeb2HsK7b/EM0s05EeqTQptAmKWRncxvPrdzGM+Gs1M07g0VBjzikYM+M1JmlRWRnDN/ZU31q3gn3XwErHoUZn4QP/idk5iT+/Nbd7xwv9o5gFo4f27UZOtve+TxLg7xx+7aOjZ4QrP3VdSx7tFrHROSAKbQptEmKcndWbGngmRXBjNSFa+po63Bys9KpKCvZs7jv5OIRthZVZyc8/WOY9zOYOBMuujOYSbi7pucQtufYJmjZse/rZRX0HsK6bvPGQbp2wRCR5FJoU2iTYWJ3SzvPr6oNWuFWbGV9XRMA02J5zAkX9501rWTkbHT/xkPwwD+DdwRrunnHO89behjAxge3oyfEBbO47svsgmjqFxHpRqFNoU2GIXdnzbbdwVi4FTU8v6qWlvZOsjPSOG1aCWeEY+GmDfeN7re8AYtuC6bjdw9meWOH9SKcIjL8KLQptMkI0NzWwYtr6nhmedAKt7om2AR6UtEoTptawuGH5HP4IQWUj8tnYuEo7dIgIjIEKbQptMkItL6ucc9khter69m6a++K47lZ6Uwfl0/5uAKFORGRIUShTaFNhPrGVlZubWDllgZWbNnFyq27WLmlQWFORGQI6S20aRqUyAhSmJvFKaXFnFJa/I7jPYW5Z1fW8OeXq/dcozAnIhIthTYRUZgTEUkBCm0i0iuFORGRoUOhTUT6TWFORGTwKbSJyIBRmBMRSZ6khjYzOwe4EUgHfufuP+l2Phv4A3AyUAtc5O5rw3PfAi4HOoCr3P2x8PhtwIeAre5+bDLrF5GBoTAnInLwkrbkh5mlAyuAs4Fq4CXgEnd/I+6aLwLHu/s/m9nFwEfc/SIzOxr4I3AqMAF4Ajjc3TvMbA7QAPwh0dCmJT9EUsuBLk0ybWwek4pySVeYE5EUFsWSH6cCVe6+OizgHuA84I24a84Dvhvenwv82oK9ds4D7nH3FmCNmVWFr/e8u88zs9Ik1i0iETuYlrms9DRKY7lMi+VTNi6PabF8po3NY9rYfMaMyhzsjyIiMmCSGdomAuvjHlcDp/V2jbu3m9kOoCQ8/kK3505MXqkikgr6CnNVWxtYXbObVTUNrKrZzYqtu3jizS20d+7tTYjlZ1MWBriysXmUjc1X65yIpIxhOxHBzK4ArgCYMmVKxNWISDIV5mYxs7SYmd3CXFtHJ+vqGveEudVhoPvr0k1sb2zbc11869y0uDCn1jkRGUqSGdo2AJPjHk8Kj/V0TbWZZQBjCCYkJPLcPrn7LcAtEIxp61flIjIsZKanUTY2n7Kx+ZzNIe84V7e7ldU1ibXOdQU5tc6JSJSSGdpeAqab2VSCwHUx8PFu1zwEfAp4HjgfeNLd3cweAu42s18QTESYDixMYq0iMsIU52VRnKfWORFJHUkLbeEYtS8BjxEs+XGbuy8zs+8Di9z9IeBW4I5wokEdQbAjvO4+gkkL7cCV7t4BYGZ/BM4AYmZWDfybu9+arM8hIiNLMlrnuu6rdU5EDkbSlvwYSrTkh4gkU2+tc6trGvZpnTusJHdPq5xa50SkJ1Es+SEiMiIMbOtcXJiL5TOpaBQZ6WmD/ZFEZAhSaBMRSaL+j53b3GPr3LSxeYwfM4pYfhax/GxK8rMpyc9ibHibm6V/zkWGO/0tFxGJQMKtc9saWLV1Nyu3NrCgqpZdLe09vl5uVjol+VmU5GUTy88mlp9FSVzAi+VlESvIpiQvi6LcLG0BJpKCFNpERIaY3lrnAJrbOqjd3UptQwvbGlrY1tDKtoYWahu6jrVSvb2R16rrqdvdSkfnvuOW0wyK87LjWu3CsFeQRSy8LcnL3hP6cjLTB+Nji8h+KLSJiKSQnMx0JhaOYmLhqP1e29np1De1UdvQQk23YFe7u4WaXcHtunWNbGtoobG1o8fXyc/O2Ntql5dFSX42Y/OD267Q1xUAR+dkqhVPJEkU2kREhqm0NAtb7bKYfkjBfq9vbG2nNr7lbvfelrxtYeB7u7aRxW9vp66xlZ4WH8gI3zPWbczdOwJe2JpXnJdFdoZa8UQSpdAmIiIA5GZlkFucweTi3P1e29HpbG/cG/Dig92eY7tbWV2zm20NLbS0d/b4OgU5GXuCXXE43q4wN4vC3EyKcjMpzO06lhl8jcoiK0OzaWVkUmgTEZF+S0+zcMJD9n6vdXcaWzu6BbvgtnZ3a9h128Kabbt5ubGe+sZW2jp6X0M0PzuDMaMyKcrL3BvyRsWFvLzMuGNB6CvIyVC3raQ8hTYREUkqMyMvO4O87AwOK8nb7/VdIW97Yyv1jW17busbW9kePt4R3m5vbGN9XSP1TW3saGrrscsWgskXY0Zl7mm1K8rNYkxuV6jLjGvde+ftqMx0zBT2ZGhQaBMRkSElPuRNKkr8eR2dzs6mMOQ1hSFvdxjymvaGvPrGVjbvbOatzbvY3tja6wQMgKyMNIq6Ql5Xy123lrzC3EyK8oLHXeEvUwsiSxIotImIyLCQnmYU5WVRlJfVr+e1tHeELXddrXpdLXxdrXtdLX1trKppYPvbwfH2HpZT6VKQnbGnJS++5W50TiajR2UwOieTgnfcz2D0qOC8xuxJbxTaRERkRMvOSGfc6HTGjc5J+Dnuzu7WDrbvDgNd095WvO27g8f13bpwtze2sau5jT6yXlhPWhjgMsJgF38/CHmj40JeQbf7uVnq0h2uFNpERET6yczIz84gPzuDyfuugdyrrrC3s6mNXc3t7Gxu6+N+cLujqY3qukZ2hsdbe5mJ2yU9zfYJeQU5Ydgb1dv9rjCYSX5OBumatDEkKbSJiIgMkviwd6Ca2zr2BLtdze3sbGrr4347u5rbWLutcc/xhl62QouXn52xpzWvK9h1b9Hb934G40bnHNRnk77pOysiIpJCcjLTyclMZ2zB/pdb6Ul7RycNLe3sam5nRy8hr6uVr+v+ll3NrNy6Nyj2tD1al+K8LCYXjWJycS6Ti3OZUpzL5KJcJhePYkLhKE3SOAgKbSIiIiNIRnpaOMs1i8kH8PyuJVm6t+7tbGpn045m1m9vZH1dI0s37OCvSze/Y8JGmsH4MaOYXDxqT5ibUpLLpDDUjc3P1ni8Pii0iYiISMLil2QZP6bvazs6nc07m1lf18i6ukaq6xpZv72JdXWNPL28hq27Wt5x/ajMdCaFrXRTinOZVBSGu/BrpHe9juxPLyIiIkmTnmZMLBzFxMJRzJpWss/55rYOqrc3sr4uCHJd4W799iYWrqnbZ/xdV9frpLhu1yDUjYyuV4U2ERERiUROZjrl4wooH1ewzzl3p76xjfXbwyBX17Sn63XZhh08lkDX6+Q9rXTDo+tVoU1ERESGHLO9iyUfP6lwn/P97XrNyUzbE+S6ul73TJRIka7XoV+hiIiISDcD3fValJsZhLluM16nFOcOma5XhTYREREZdpLR9TqlOJc7P3taZIsPK7SJiIjIiHKgXa+7mtsi3S1CoU1EREQkzv66XqMSfQetiIiIiOyXQpuIiIhIClBoExEREUkBCm0iIiIiKUChTURERCQFKLSJiIiIpACFNhEREZEUoNAmIiIikgIU2kRERERSgEKbiIiISApQaBMRERFJAQptIiIiIilAoU1EREQkBZi7R11D0plZDfB2kt8mBmxL8ntIculnmNr080t9+hmmPv0MB8Zh7j62+8EREdoGg5ktcveZUdchB04/w9Smn1/q088w9elnmFzqHhURERFJAQptIiIiIilAoW3g3BJ1AXLQ9DNMbfr5pT79DFOffoZJpDFtIiIiIilALW0iIiIiKUCh7SCZ2TlmttzMqszsuqjrkf4xs8lm9pSZvWFmy8zsK1HXJAfGzNLN7BUzezjqWqT/zKzQzOaa2Vtm9qaZnR51TZI4M7sm/Dd0qZn90cxyoq5pOFJoOwhmlg7cBJwLHA1cYmZHR1uV9FM78DV3PxqYBVypn2HK+grwZtRFyAG7Efirux8JnIB+linDzCYCVwEz3f1YIB24ONqqhieFtoNzKlDl7qvdvRW4Bzgv4pqkH9x9k7u/HN7fRfCLYmK0VUl/mdkk4IPA76KuRfrPzMYAc4BbAdy91d3rIy1K+isDGGVmGUAusDHieoYlhbaDMxFYH/e4Gv3CT1lmVgqcCLwYcSnSfzcA3wA6I65DDsxUoAb4fdjF/Tszy4u6KEmMu28Afg6sAzYBO9z98WirGp4U2kQAM8sH/gxc7e47o65HEmdmHwK2uvviqGuRA5YBnATc7O4nArsBjRFOEWZWRNDLNBWYAOSZ2SejrWp4Umg7OBuAyXGPJ4XHJIWYWSZBYLvL3e+Puh7pt9nAh81sLcEQhTPN7M5oS5J+qgaq3b2rlXsuQYiT1PBeYI2717h7G3A/UBFxTcOSQtvBeQmYbmZTzSyLYODlQxHXJP1gZkYwjuZNd/9F1PVI/7n7t9x9kruXEvwdfNLd9b/8FOLum4H1ZnZEeOgs4I0IS5L+WQfMMrPc8N/Us9BEkqTIiLqAVObu7Wb2JeAxgtkyt7n7sojLkv6ZDVwKLDGzV8Nj/8/dH4muJJER6cvAXeF/gFcDl0VcjyTI3V80s7nAywQz8l9BOyMkhXZEEBEREUkB6h4VERERSQEKbSIiIiIpQKFNREREJAUotImIiIikAIU2ERERkRSg0CYiKcPMOszs1bivAVs138xKzWxpAtd918wazWxc3LGGwaxBREYmrdMmIqmkyd1nRF0EsA34GvDNqAuJZ2YZ7t4edR0ikhxqaRORlGdma83sZ2a2xMwWmll5eLzUzJ40s9fN7O9mNiU8foiZPWBmr4VfXVvupJvZb81smZk9bmajennL24CLzKy4Wx3vaCkzs2vN7Lvh/afN7HozW2Rmb5rZKWZ2v5mtNLMfxr1MhpndFV4z18xyw+efbGbPmNliM3vMzMbHve4NZrYI+MrBfzdFZKhSaBORVDKqW/foRXHndrj7ccCvgRvCY78Cbnf344G7gF+Gx38JPOPuJxDscdm1k8l04CZ3PwaoBz7WSx0NBMGtvyGp1d1nAv8FPAhcCRwLfNrMSsJrjgB+4+5HATuBL4b74/4KON/dTw7f+0dxr5vl7jPd/T/7WY+IpBB1j4pIKumre/SPcbfXh/dPBz4a3r8D+Fl4/0zgnwDcvQPYYWZFBJtevxpesxgo7aOWXwKvmtnP+1F/197ES4Bl7r4JwMxWA5MJguJ6d38uvO5O4CrgrwTh7m/B1o6kA5viXvfeftQgIilKoU1Ehgvv5X5/tMTd7wB66x7F3evN7G6C1rIu7byzByOnl9fv7PZenez997h77Q4YQcg7vZdydvdWp4gMH+oeFZHh4qK42+fD+wuAi8P7nwCeDe//HfgCgJmlm9mYA3zPXwCfZ2/g2gKMM7MSM8sGPnQArznFzLrC2ceB+cByYGzXcTPLNLNjDrBmEUlRCm0ikkq6j2n7Sdy5IjN7nWCc2TXhsS8Dl4XHL2XvGLSvAO8xsyUE3aBHH0gx7r4NeADIDh+3Ad8HFgJ/A946gJddDlxpZm8CRcDN7t4KnA/81MxeA14FKnp/CREZjsz9QHsRRESGBjNbC8wMQ5SIyLCkljYRERGRFKCWNhEREZEUoJY2ERERkRSg0CYiIiKSAhTaRERERFKAQpuIiIhIClBoExEREUkBCm0iIiIiKeD/A97wAgqTXViGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,6))\n",
    "ax.plot(train_loss_history, label='train')\n",
    "ax.plot(valid_loss_history, label='valid')\n",
    "ax.set_xlabel(\"Epoch Number\")\n",
    "ax.set_ylabel(\"Cross-Entropy Loss\")\n",
    "plt.title(\"Training and Validation Loss Curves\")\n",
    "plt.legend()\n",
    "plt.savefig(\"loss_curves_cnn.jpg\", dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oqgPrXHFy3H6"
   },
   "source": [
    "## Vision Transformer\n",
    "\n",
    "The Vision Transformer, oftentimes represented as ViT, is an achitecture introduced by the 2020 paper [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/abs/2010.11929). I wanted to see how such an architecture compares to the tried and true CNN at digit classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m_AgjdU36P9O"
   },
   "source": [
    "### Single-head Attention\n",
    "\n",
    "A key component of the Vision Transformer is the self-attention mechanism. Described by the infamous paper entitled [Attention is All You Need ](https://arxiv.org/abs/1706.03762), we use **scaled dot-product attention** here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vBgURvzV6Swk"
   },
   "outputs": [],
   "source": [
    "class SingleHeadAttention(nn.Module):\n",
    "    def __init__(self, input_dim, inner_dim, dropout = 0.):\n",
    "        super().__init__()\n",
    "        self.Q = nn.Linear(input_dim, inner_dim, bias=False)\n",
    "        self.K = nn.Linear(input_dim, inner_dim, bias=False)\n",
    "        self.V = nn.Linear(input_dim, inner_dim, bias=False)\n",
    "        self.softmax = nn.Softmax(dim=1)\n",
    "        self.scale = 1 / np.sqrt(inner_dim)\n",
    "        self.dropout = nn.Dropout(p=dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        Q_proj = self.Q(x)\n",
    "        K_proj = self.K(x)\n",
    "        V_proj = self.V(x)\n",
    "\n",
    "        K_T = K_proj.transpose(1, 2)\n",
    "        QK_T = torch.matmul(Q_proj, K_T) * self.scale\n",
    "        attention = self.softmax(QK_T)\n",
    "        self.attention_cache = attention\n",
    "        out = torch.matmul(attention, V_proj)\n",
    "        out = self.dropout(out)\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XkEQ0JDE6Ttn"
   },
   "source": [
    "### Multi-head Attention\n",
    "\n",
    "We can combine several attention-heads in parallel to create Multi-head Attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DRgiyIAl6X8k"
   },
   "outputs": [],
   "source": [
    "class Attention(nn.Module):\n",
    "    def __init__(self, dim, heads = 8, dim_head = 64, dropout = 0.):\n",
    "        super().__init__()\n",
    "        self.attention_heads = nn.ModuleList([\n",
    "            SingleHeadAttention(dim, dim_head, dropout=dropout)\n",
    "            for _ in range(heads)\n",
    "        ])\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = torch.cat([\n",
    "            head(x) for head in self.attention_heads\n",
    "        ], dim=-1)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "h9pbMlS46cgF"
   },
   "source": [
    "### Vision Transformer Layer\n",
    "\n",
    "We can abstract a single layer of the Vision Transformer as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zTJOcjl66e2C"
   },
   "outputs": [],
   "source": [
    "class ViTLayer(nn.Module):\n",
    "    def __init__(self, dim, heads, dim_head, mlp_dim, dropout = 0.):\n",
    "        super().__init__()\n",
    "        self.attn = nn.Sequential(\n",
    "            nn.LayerNorm(dim),\n",
    "            Attention(dim, heads = heads, dim_head = dim_head, dropout = dropout),\n",
    "            nn.Dropout(p=dropout),\n",
    "            nn.Linear(heads * dim_head, dim, bias=False)\n",
    "        )\n",
    "        self.feedforward = nn.Sequential(\n",
    "            nn.LayerNorm(dim),\n",
    "            nn.Linear(dim, mlp_dim),\n",
    "            nn.Dropout(p=dropout),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(mlp_dim, dim),\n",
    "            nn.Dropout(p=dropout),\n",
    "        )\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.attn(x)\n",
    "        x = x + self.feedforward(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B18OaSIn6iT8"
   },
   "source": [
    "### Putting it All Together: `ViT`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jwC4oDBty3tw"
   },
   "outputs": [],
   "source": [
    "class ViT(nn.Module):\n",
    "    def __init__(self, patch_size=7, dim=64, depth=6, heads=8, mlp_dim=128, \n",
    "                 dim_head = 64,\n",
    "                 dropout = 0., emb_dropout = 0.):\n",
    "        super().__init__()\n",
    "\n",
    "        image_height, image_width = 28, 28\n",
    "        num_classes = 10\n",
    "        channels = 1\n",
    "\n",
    "        patch_height = patch_size\n",
    "        patch_width = patch_size\n",
    "\n",
    "        assert image_height % patch_height == 0 and image_width % patch_width == 0, (\n",
    "            'Image dimensions must be divisible by the patch size.')\n",
    "\n",
    "        num_patches = (image_height // patch_height) * (image_width // patch_width)\n",
    "        patch_dim = channels * patch_height * patch_width\n",
    "\n",
    "        self.to_patch_embedding = nn.Sequential(\n",
    "            Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_height, p2 = patch_width),\n",
    "            nn.Linear(patch_dim, dim),\n",
    "        )\n",
    "\n",
    "        self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, dim))\n",
    "        self.cls_token = nn.Parameter(torch.randn(1, 1, dim))\n",
    "        self.emb_dropout = nn.Dropout(emb_dropout)\n",
    "\n",
    "        self.transformer = nn.Sequential(*nn.ModuleList([\n",
    "            ViTLayer(dim, heads, dim_head, mlp_dim, dropout)\n",
    "            for _ in range(depth)                       \n",
    "        ]))\n",
    "\n",
    "        self.mlp_head = nn.Sequential(\n",
    "            nn.LayerNorm(dim),\n",
    "            nn.Linear(dim, num_classes)\n",
    "        )\n",
    "\n",
    "    def forward(self, img):\n",
    "        x = self.to_patch_embedding(img)\n",
    "        b, n, _ = x.shape\n",
    "\n",
    "        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)\n",
    "        x = torch.cat((cls_tokens, x), dim=1)\n",
    "        x += self.pos_embedding[:, :(n + 1)]\n",
    "        x = self.emb_dropout(x)\n",
    "\n",
    "        x = self.transformer(x)\n",
    "\n",
    "        x = x[:, 0]\n",
    "        return self.mlp_head(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IGD2JfImzOx4"
   },
   "outputs": [],
   "source": [
    "model = ViT(patch_size=7, dim=64, depth=6, heads=8, mlp_dim=128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 837
    },
    "id": "jPqp5lxTzPA0",
    "outputId": "b6c97e5e-68ea-4c7b-ca11-956a08771ffe"
   },
   "outputs": [],
   "source": [
    "trained_model, train_loss_history, valid_loss_history, valid_accuracy_history = run_training_loop(model)\n",
    "test_performance(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,6))\n",
    "ax.plot(train_loss_history, label='train')\n",
    "ax.plot(valid_loss_history, label='valid')\n",
    "ax.set_xlabel(\"Epoch Number\")\n",
    "ax.set_ylabel(\"Cross-Entropy Loss\")\n",
    "plt.title(\"Training and Validation Loss Curves\")\n",
    "plt.legend()\n",
    "plt.savefig(\"loss_curves_vit.jpg\", dpi=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"ViT.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Looking at Attention Heads\n",
    "\n",
    "By passing through data through our trained model, we can look at the attention heads and thus see where our model is focusing most of its attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = valid_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(X.detach().numpy().squeeze(), cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model(X.unsqueeze(0).to(device))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heads = model.transformer[-1].attn[1].attention_heads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "heads[0].attention_cache.detach().cpu().numpy().squeeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(2, 4, figsize=(8,4))\n",
    "\n",
    "for i, (head, ax) in enumerate(zip(heads, axes.flatten()), 1):\n",
    "    ax.imshow(head.attention_cache.detach().cpu().numpy().squeeze(), cmap='gray')\n",
    "    ax.set_xticks([])\n",
    "    ax.set_yticks([])\n",
    "    ax.set_xlabel(f\"Head {i}\")\n",
    "    \n",
    "plt.suptitle(\"Attention Heads (First Valid)\");\n",
    "plt.savefig(\"attention_heads.jpg\", dpi=300, bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 404
    },
    "id": "eRvTJvZfzZ8H",
    "outputId": "343b3050-3634-4df8-aa03-88ef960b5a2a"
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10,6))\n",
    "ax.plot(train_loss_history, label='train')\n",
    "ax.plot(valid_loss_history, label='valid')\n",
    "ax.set_xlabel(\"Epoch Number\")\n",
    "ax.set_ylabel(\"Cross-Entropy Loss\")\n",
    "plt.title(\"Training and Validation Loss Curves\")\n",
    "plt.legend()\n",
    "plt.savefig(\"loss_curves_q3c.jpg\", dpi=300)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.11"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "06a1ccecd6934249ba582aae46c21115": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0d1eea56f56e416c98326a4dd45845e7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_3a73d6914c5a4cc3a8fcbca9346715fa",
       "IPY_MODEL_8ce8ea4ebfad417fb6df5410741b0753",
       "IPY_MODEL_ab25f081f93c49f2b137a57d1de5f630"
      ],
      "layout": "IPY_MODEL_296842818dd846a5817c619792148201"
     }
    },
    "0d7ee23f17b047ce8865ed02fffc0cc6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1ebbf2ad85f7446dadde649e7b0af20b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "261c7f67235548a98e13f196c5fe8e92": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a6395dc9b98b4008a55acd20bfea8fa1",
       "IPY_MODEL_b30f753332cf418ba24c3e3c66d6ede2",
       "IPY_MODEL_f95b78bd4aeb45c38211fb8bbc44ac5e"
      ],
      "layout": "IPY_MODEL_4c1630885bfc4fc1b56e1e7dd2c4d111"
     }
    },
    "296842818dd846a5817c619792148201": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2cb003ba31134b39ac657a82280d5c31": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f98c27b1e9d24843a38f103bb4051968",
      "placeholder": "​",
      "style": "IPY_MODEL_1ebbf2ad85f7446dadde649e7b0af20b",
      "value": " 1648877/1648877 [00:00&lt;00:00, 5956600.11it/s]"
     }
    },
    "2ee32878afc24d72b839eaa5f43ecfff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "34518066cc61449ca87d06356f6b25cd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "34abb1e4a0c6477ba393377f785511a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_34518066cc61449ca87d06356f6b25cd",
      "placeholder": "​",
      "style": "IPY_MODEL_49c8182ddc6c4f7fb2de89e80151517a",
      "value": "100%"
     }
    },
    "34bbe2bb28a14ae9ae7e2efaa69ad9be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4ebf2751d41241678b4629d6e1594529",
       "IPY_MODEL_8e35093ad4874ba0b75c0157ac12b1b0",
       "IPY_MODEL_aa3ab8c1a4944c8a803a4a510f331c91"
      ],
      "layout": "IPY_MODEL_c5a8018df6d44808a00ff3c6289b2ef9"
     }
    },
    "3a73d6914c5a4cc3a8fcbca9346715fa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b9f40688fe474cde931a9886072c64d8",
      "placeholder": "​",
      "style": "IPY_MODEL_c13181bfd2fd4e66b15f070062e5ffdd",
      "value": "100%"
     }
    },
    "43404a2ab0ed4495a9c0c260072fbec7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "49c8182ddc6c4f7fb2de89e80151517a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4c1630885bfc4fc1b56e1e7dd2c4d111": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d9c6a6b48ea4ae5ab04993762518a4b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4ebf2751d41241678b4629d6e1594529": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2ee32878afc24d72b839eaa5f43ecfff",
      "placeholder": "​",
      "style": "IPY_MODEL_67903fa5367e4464a2ee7aa21d1e33f4",
      "value": "100%"
     }
    },
    "59de2466b3814431a8e995637bed1efd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "67903fa5367e4464a2ee7aa21d1e33f4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6ab64b1f3836474aabc9bce92ee3ff90": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "76ceb0c5e55a4f649546bda2e517c9ef": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "79ccf00bbc3a44cd80d401b1c8d333ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "89e3360f282341f1a1c0d0f073a351af": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8ce8ea4ebfad417fb6df5410741b0753": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_90d3ddc528904599b0b8d3c3e19b2819",
      "max": 28881,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0d7ee23f17b047ce8865ed02fffc0cc6",
      "value": 28881
     }
    },
    "8e35093ad4874ba0b75c0157ac12b1b0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_43404a2ab0ed4495a9c0c260072fbec7",
      "max": 4542,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c4c646d0aa59467e9cbe21bfa7eecd20",
      "value": 4542
     }
    },
    "90d3ddc528904599b0b8d3c3e19b2819": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "92c3359d5c7d4cecad1a2179dd44c308": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_34abb1e4a0c6477ba393377f785511a4",
       "IPY_MODEL_ef2baa336dec49918e0b9f7691363015",
       "IPY_MODEL_2cb003ba31134b39ac657a82280d5c31"
      ],
      "layout": "IPY_MODEL_cb32a1674c9d41eaa15645ae9d34fa90"
     }
    },
    "a6395dc9b98b4008a55acd20bfea8fa1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_76ceb0c5e55a4f649546bda2e517c9ef",
      "placeholder": "​",
      "style": "IPY_MODEL_06a1ccecd6934249ba582aae46c21115",
      "value": "100%"
     }
    },
    "aa3ab8c1a4944c8a803a4a510f331c91": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6ab64b1f3836474aabc9bce92ee3ff90",
      "placeholder": "​",
      "style": "IPY_MODEL_f7a037b48d304440929b977043621481",
      "value": " 4542/4542 [00:00&lt;00:00, 65872.97it/s]"
     }
    },
    "ab25f081f93c49f2b137a57d1de5f630": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ece8364103b14437b16a6ff937427322",
      "placeholder": "​",
      "style": "IPY_MODEL_59de2466b3814431a8e995637bed1efd",
      "value": " 28881/28881 [00:00&lt;00:00, 147184.03it/s]"
     }
    },
    "b30f753332cf418ba24c3e3c66d6ede2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd646184d14b42ee9e7d23f837a0bc2e",
      "max": 9912422,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_89e3360f282341f1a1c0d0f073a351af",
      "value": 9912422
     }
    },
    "b9f40688fe474cde931a9886072c64d8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c13181bfd2fd4e66b15f070062e5ffdd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c4c646d0aa59467e9cbe21bfa7eecd20": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c4ca6d489fd947b092c7ff9cd6f8889b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c5a8018df6d44808a00ff3c6289b2ef9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cb32a1674c9d41eaa15645ae9d34fa90": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d80ee751e04246cd82d0dde0ff07adcc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dd646184d14b42ee9e7d23f837a0bc2e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ece8364103b14437b16a6ff937427322": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef2baa336dec49918e0b9f7691363015": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c4ca6d489fd947b092c7ff9cd6f8889b",
      "max": 1648877,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_79ccf00bbc3a44cd80d401b1c8d333ed",
      "value": 1648877
     }
    },
    "f7a037b48d304440929b977043621481": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f95b78bd4aeb45c38211fb8bbc44ac5e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4d9c6a6b48ea4ae5ab04993762518a4b",
      "placeholder": "​",
      "style": "IPY_MODEL_d80ee751e04246cd82d0dde0ff07adcc",
      "value": " 9912422/9912422 [00:00&lt;00:00, 18002137.58it/s]"
     }
    },
    "f98c27b1e9d24843a38f103bb4051968": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}