practical2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "y-8QoaJmXd8Z"
   },
   "source": [
    "# DL Indaba Practical 2\n",
    "# Feedforward Neural Networks on Real Data & Best Practices\n",
    "*Developed by Stephan Gouws, Avishkar Bhoopchand & Ulrich Paquet.*\n",
    "\n",
    "**Introduction**\n",
    "\n",
    "In this practical we will move on and discuss best practices for building and training models on real world data (the famous MNIST dataset of hand-written images of digits). We will develop a deep, fully-connected (\"feed-forward\") neural network model that can classify these images with around 98% accuracy (close to state-of-the-art for feedforward models on this dataset). \n",
    "\n",
    "**Learning objectives**:\n",
    "\n",
    "Understanding the issues involved in implementing and applying deep neural networks in practice (w/ a focus on TensorFlow). In particular:\n",
    "\n",
    "* Implementation sanity checking details & controlled overfitting on a small training set.\n",
    " \n",
    "* Training deep neural networks. Understand:\n",
    " * Conceptually how the backprop algorithm efficiently computes model gradients by applying the chain rule and applying dynamic programming (saving intermediate computations).\n",
    " * Overfitting and how to use regularization to avoid it (Weight decay/L2 and dropout).\n",
    " * Knowing when to stop training (early stopping).\n",
    " \n",
    "* Understand the need for hyperparameter tuning in finding good architectures and training settings. \n",
    "\n",
    "**What is expected of you:**\n",
    "\n",
    "* We have included rough time estimates of how long you should aim to spend on the important sections below.\n",
    "* Step through the notebook answering the questions by discussing them with your lab partner. \n",
    "* Execute each cell in turn & fill in the missing code by pair-programming with your lab partner. \n",
    "* 5 min before the end, pair up with someone else and make sure you both understand the concepts listed in \"Learning Objectives\" above. If not, please speak to the tutors!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5JQ4eAF7X2tK"
   },
   "source": [
    "# Setups and Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "G2tOik9MR8_O"
   },
   "source": [
    "For this practical, we will work with the [famous MNIST dataset of handwritten digits](http://yann.lecun.com/exdb/mnist/). The task is to classify which digit a particular image represents. Luckily, since MNIST is a very popular dataset, TensorFlow has some built-in functions to download it. \n",
    "\n",
    "The MNIST dataset consists of pairs of images (28x28 matrices) and labels. Each label\n",
    "is represented as a (sparse) binary    vector of length 10 with a 1 in position    `i` iff the image represents digit `i`, and 0 elsewhere. This is what the \"one_hot\" parameter you see below does. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 153,
     "output_extras": [
      {
       "item_id": 4
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8487,
     "status": "ok",
     "timestamp": 1503657242635,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "IYrNES0hD6Mi",
    "outputId": "258b73ac-670c-4b30-9b67-7fbe07be7c4b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Import TensorFlow and some other libraries we'll be using.\n",
    "import datetime\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "# Download the MNIST dataset onto the local machine.\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xdz38ZyZE3vW"
   },
   "source": [
    "# Visualizing the MNIST data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AmblsKmbl9sY"
   },
   "source": [
    "Let's visualize a few of the digits (from the training set): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 208,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 999,
     "status": "ok",
     "timestamp": 1503055879874,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "JpAbMODcG1se",
    "outputId": "76aa84c7-ee87-45ba-e73d-1476f54ec647"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABZgAAADBCAYAAABCFNVcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe4VNXZxuHnVQELYosiAYwSbNhiRBDBEiyY2BWNGo0a\nTCEC6qeoWGLBxB5jgkaNGtQoiBobFixYULFhRSWIFRCwINJEQdb3x95HZu09nJlZ08/53dd1Lnj3\n7LLmzMOcYbF5lznnBAAAAAAAAABAoVao9gAAAAAAAAAAAPWJCWYAAAAAAAAAQBAmmAEAAAAAAAAA\nQZhgBgAAAAAAAAAEYYIZAAAAAAAAABCECWYAAAAAAAAAQBAmmAEAANAoM3vSzJyZPVmh63UyszFm\n9lV83dcqcd1SMbPh8bg/rPZYimFmu8fPw5lZrwKPvSA+bkm5xgcAAIDawAQzAACoKWbWwsxON7O3\nzWyBmc01s/fM7F4z61bt8eViZhtmTMqlvqo9vjrxV0l7SlpV0suS3qjucLLLeF3PreA1d20sXxlf\nG5bgcl9JeiH+mlvgsVPj454vwTjyZmYtzWxI4v1jipndY2bbFXiulTK+n2eVa8wAAAD1bqVqDwAA\nACDhEkknxr+fIulrST+StJ+kuyW9WKVxhZguaVq1B1GHtoh/vcM5d0RjO5pZS+fctxUYU62Yq2ji\ntsG2klpKmifp7Yzt32Q72MxaSFrinMv5jx3OuZck7RAySOfctZKuDTm2SJdLGhD//l1JixS9f+wv\n6U5JE6owJgAAgCaNO5gBAECtOTz+dahzbmPn3NaS1pTUQ4nJZTM71swmmNnX8d2Kz5vZIRmPZ95N\nfIqZ/dfMFprZRDPrZWbbmtmL8bHPmNmmifPvaWZj47sgF5nZC2a2bwHP5Xrn3A6ZX/F5tzOzb+Nx\nHRdv2yQemzOzwfG2o+LxfW5mi83sy7h1xPd3cifuaP2NmT0Rfz9eMLPNzWyP+PnOM7MHzWz9jGO/\nb31hZgPN7OP42AfNrENjTyy+U/RsM/ufmX1jZl+Y2W2Zx5lZWzO7xcw+iff51MzGmdmRyznnhvFd\n3p3jTYfH4xueeC1Pje9IXajoHyRkZhuY2c1mNjP+Xk03s+vMbL2M83/fusLMjox/nWdmV5vZymb2\nFzObHY/3T408913Nvxv9nIbzZtl3v4y7aZ8uNmPOuVcSeZoRP/RKImszzOw/8bimmFk/M/tA0cTz\namZ2jJm9lJGt2Wb2kJl1zRhbqkWGZbS+MLPdzOzVOLcTzGz7jGNTLTIs+jPmzOwxMxtkZh/F3//7\nzKxtxn6tzOwqM5sTj+tvZnZx8nzL0fD+cY5zbpOM948dFd0Nn/m9/5VFf74Wxq/P42bWo+G5S1qc\nsfvQhu9ljusDAAA0O0wwAwCAWtPw+WQPM9vXzNZ3keedc9/foWnRf1m/UdJPJX2m6L/zd5c0ysx+\nn+W8F0j6iaSliu6QvUvSI5LWkNRCUs/4fA3n7yvpYUk/i889VVI3SffGjwVzzk2Q1DCBeZmZbSBp\nuKRVJI2VdFn8WHdJW0n6QtJb8eN7Snosc6I4w9WSfqjoe9hN0mhJ90haUdJqkn6u6A7PpB0kXSxp\nvqRW8X535Xgad0k6X9Fk8P8kmaLJvWfNbK2M8RwpqY2kiZIWKpro23U55/xG0d25DXckfx7X7yX2\nG6rodXlP0pJ4Enm8pKMUTSZOlrSupN9KesbMWieObyfpGklLJLWW1F/Rna3Hx9+DdpLOM7M9ljPO\n5F3E0+P61SzXGaXoe7OKpJ1UoYwldJT0L0Xf38/ibTtI2lLLsrWapL0kPW5m6+ZxzhUlPSBpZUV/\nfn4qaaSZrZjHsTtLuigeT2tJ+yr+h4LYhZL+qOjP5leKMnR8HueVlr1/7Glm+5hZ2/j9Y7xzblLD\nTmZ2mqT/SNpe0es3W1JvSU9a9A84De1BGkxT9tcYAACg2WOCGQAA1Jqr4193kHSfpBkW3SV7vpmt\nKklmtpqkM+L97pO0oaL/Bj8u3jbUzJKfc56S9GMta7+xnqR7nHObKppclaQdzWyV+PeXKJoYvE3S\nBs65jSVdH2/7S57PpeHO1oavezIeu0TS04om0V5UdIf2bEm/zmhf8HdJ6zjnNnXO/UTRhKAkrS5p\n7yzXuyV+PpfGdSdJf3HObS7p1njbblmOW1HST51zXST9X7ytm5n9LNuTMrOdJe0Tlz+P7xLtpGhC\neANFk4OStEn8a3/n3HbOuQ0ltZX0j2zndc7NSNyV+0B8N+7QxK7vS/qRc24rSacpmnz8oSQnaSfn\n3BaKWqpI0saSjk0c31JSn3h8UzP220ZSF0VtFaTs36vv7yLO2NRwp/qBWa5zSPz9/1u8rdQZy0dL\nSb91zm2maNJ7gaQrJK2dka1t4n3bSPpFnuc9KX5up8V1J0kb5XHcCpK6Oec2kXR/vG03STKz1bUs\nP/+Nz9lJ0qw8x9Tw/tEzPvdMM5tkZudmvH+0lnROvN/Q+Pv+I0X/uNNS0nlxe5DMhQ2vjV/jQwQA\nAAAPE8wAAKCmOOfOlXSQpHu1bGGxTSSdLenmuN5C0R2hknS7c26pc26xoh6rUnT36o8Sp34gnrj9\nMGNbw+TW+xnb1ovv4GyYKDtC0tK4JcJx8baNzWydPJ5Ow52tDV//y3ieSxXdcTtX0aSrJP3BOTc9\n4/g1Fd3NOtvMlirqKdvgh1mu1/B8PsyyreE5tlXaGxl3d96esX3LLPtK0Z3VDcbE35svJf0g3tYw\n+dpw7eEWLdT4oKTfS/pkOefN103OuTmS5Jz7TtFdqJI0JZ4YlHPu4XhMktQ1cfyXzrln49fg43jb\nROfch865+ZI+jbdl+14V4ivnXMP3ILM/cikzlo/5iu+cjv+sOElrSbrforYrSyW9k7F/tmxlc0v8\na+Zzy+d79ppzbmLi2IbjNlZ0F70U/dl2zrm5iu6Wzsk5d5akvoqy1/D+samiCeWGu8e30rL3j7Pj\n7/t3iu5glgL7TgMAADRXLPIHAABqjnPubkl3m5kp+q/318W/7pvlzuR8NUw2LcmyLbOfriWO+0DL\nJhwztcjjmtfHE+bLs56WTXRJy3oPN9xlOUbRJPMiRf81f7GWTe5ma0WQ73MspReznLth0vZMSc8q\nult4S0V3hP5c0iGK2pWEyvdu1uWZm/H7JVm2NTyfZBYKNSfLdbKdt5iM5ePTzEX9zKyNotYcayha\nRDOfbCV9F0/GS40/t2yyfV+yHReUWefcXZLuit8rfqqoPchPJO0Xv6dkekf+a585JgAAAOSBO5gB\nAEBNiRcH+4kkxXcvTpDUcHftvPiu07cUTYxJ0i/NbAUza6HozkUp6jP7UegYnHOfadldwBMVtV1o\nWFTtUEkXOudmhp5fkuL/rn+roknEhr6u55nZT+Pfb6poclmSfuOc207L2nuU2tYZi89ltgCYmG1n\nSS9l/P6vGd+bHpJOlXRt/FhPSU855wY553pL+l28fZsi785NTjw2jKdzw0JzZraXort0pcTibiXU\nkMHVCj2wEhnLvFyi3lzR5LIkHR1n65QSXatYkxX1Zpakg6XvJ8SztYRJsWihxm2k7+/WflnL/ufA\nvHii/U0ta4PyqKQeGd/7YyWdGz/2nZb1A/deYzPrG7femJS5QCEAAEBzxAQzAACoNcdJetXMPjOz\nCWb2kaIWApI0QpKccwu0rEftfoom6j5UtIiaJJ0dT0QX4/T4130V9YF+1cw+ia9zUp7nOM7Mnk98\ntYsfu0JR64+PJO2iqL1HC0m3xj1631fUK1eSbjCzNxQt2FcO30h6xcze0rJewS87557ItrNz7klJ\nD8XlSDObbGZvKloY7SlFd41K0UJuX5jZFDOboGUtCqYp6jddKlcp6ttsksaZ2URFvbklaYqkf5fw\nWpka/uFjkJm9ZGaF9k0uRcZCTFG04KIUtS95Q8vay1RVfFd0Qx/lX5rZ+4r+LGRb1DKb30l6zcw+\njd8/Ppb0y/ix2zKu0dDXe5Ck6fH3/lNFLTsOi/dzWjY5fZKZvWhmDcetqegfgTZV6e40BwAAqEtM\nMAMAgFpzlqL+y/MkbaZoYuldRRPK399l6Zy7QNJvJL2iqOfyWor6HP/SOXetiuScu11RO4eGhb82\nV3TX4x2SLsvzNO0VtR3I/GplZvspmghzkvo55+ZJ6q+o9cNmki5zzn2p6G7itxV9ZvtW0URkObys\naEKzdXydhxX1wW7MgYr62k5S1O+6g6KJwMslPRnvc7uiFhqrK+p7O0/Ra/vzzJYNxXLOfaqob+4t\nitovbKroLvbrJfXMaOVQaoMU3Q0rRX2eN2lk35QSZaxgzrkvFN0l/Y6idhiLJO1frusFGKJokvkr\nRX+uR0i6KX7s6+UdFDtD0T8uzFf0Z6mtoruiL9CyxQjlnPuLpF8res9YQ9Fr96Wk4Vr2DyGSNFDR\nHeYrKOr1vXHwswIAAGiirISf7QEAAFBHzOxJRXdPP+Wc27W6owEiZra+pIXx4n4N7WQmKJowfsY5\nt1NjxwMAAKCyWOQPAAAAQC3pJenfZvayorveuym6E3mJpD9Vc2AAAABIo0UGAAAAgFrynqKFL7eW\n9AtFf2e5V1Kv5fUFBwAAQPXQIgMAAAAAAAAAEIQ7mAEAAAAAAAAAQZhgBgAAAAAAAAAEYYIZAAAA\nAAAAABCECWYAAAAAAAAAQBAmmAEAAAAAAAAAQZhgBgAAAAAAAAAEYYIZAAAAAAAAABCECWYAAAAA\nAAAAQBAmmAEAAAAAAAAAQZhgBgAAAAAAAAAEYYIZAAAAAAAAABCECWYAAAAAAAAAQBAmmAEAAAAA\nAAAAQZhgBgAAAAAAAAAEYYIZAAAAAAAAABCECWYAAAAAAAAAQBAmmAEAAAAAAAAAQZhgBgAAAAAA\nAAAEYYIZAAAAAAAAABCECWYAAAAAAAAAQBAmmAEAAAAAAAAAQZhgBgAAAAAAAAAEYYIZAAAAAAAA\nABCECWYAAAAAAAAAQBAmmAEAAAAAAAAAQZhgBgAAAAAAAAAEYYIZAAAAAAAAABCkqAlmM9vLzP5n\nZlPM7PRSDQpNG7lBCHKDEOQGIcgNQpAbFIrMIAS5QQhygxDkBoUw51zYgWYrSposaQ9J0yS9JOlw\n59zbpRsemhpygxDkBiHIDUKQG4QgNygUmUEIcoMQ5AYhyA0KtVIRx3aTNMU5974kmdlISftLWm7Y\nzCxsNhs1zzlnee5KbvA9coMQ5AYhypUbMtOkfe6cWzfPfckNJPEzCmHIDUKQG4QgNwiRT26KaZHR\nXtLUjHpavA1oDLlBCHKDEOQGIcgNGnxUwL7kBoUiMwhBbhCC3CAEuUFBirmDOS9m9jtJvyv3ddC0\nkBuEIDcIQW5QKDKDEOQGIcgNQpAbhCA3CEFu0KCYCebpkjpm1B3ibR7n3HWSrpO4XR6SyA3CkBuE\nIDcIkTM3ZAZZkBsUip9RCEFuEILcIAS5QUGKaZHxkqSNzWwjM2sp6TBJ95VmWGjCyA1CkBuEIDcI\nQW4QgtygUGQGIcgNQpAbhCA3KEjwHczOuSVmNkDSGEkrSrrROfdWyUaGJoncIAS5QQhygxDkBiHI\nDQpFZhCC3CAEuUEIcoNCmXOVu4Od2+WbrgJWIi0YuWm6yA1CkBuEKFduyEyTNsE517UcJyY3TRc/\noxCC3CAEuUEIcoMQ+eSmmBYZAAAAAAAAAIBmjAlmAAAAAAAAAEAQJpgBAAAAAAAAAEGYYAYAAAAA\nAAAABGGCGQAAAAAAAAAQhAlmAAAAAAAAAEAQJpgBAAAAAAAAAEGYYAYAAAAAAAAABGGCGQAAAAAA\nAAAQZKVqD6A5a9eunVc//vjjXr3aaquljvnggw+8ul+/fo0+vnTp0mKGCAAAAABAzRsyZEhq26OP\nPurVL7/8cqWGgzoxatSonPsceuihFRgJqsnMvLpDhw5effzxx3v1wQcfnDpH586dvfrNN9/06jPO\nOMOrR48eXfA4axl3MAMAAAAAAAAAgjDBDAAAAAAAAAAIwgQzAAAAAAAAACCIOecqdzGzyl2sBq2/\n/vpeney5vPnmmxd9jV/96ldePWLEiKLPmQ/nnOXeK0y95GbVVVdNbdtnn328+r777vPqRYsWlXwc\nK62Ubq0+duxYr37rrbe8un///iUfRz7IDUKQG4QoV27qOTM77LCDV2+77bZeffLJJ3t1p06dUud4\n5plnvDr5ueOwww7z6pEjR+Yc1/z58736lltuyXlMmUxwznUtx4nrOTdoHD+jEILclMY777yT2jZj\nxgyv7t27d6WGU3bkpjSy9WA+5JBDvLpHjx5e/fzzz5d1TOVEbqQDDzwwtW3vvff26mOPPbbk1504\ncaJX77bbbql9Pv/885JftxTyyQ13MAMAAAAAAAAAgjDBDAAAAAAAAAAIwgQzAAAAAAAAACBIulkr\nSiLZb1mSHn30Ua8uRc/lpHPOOcerK9WDGdIVV1yR2nbcccd59XrrrefV5ejB3Llz59S2nj17evUX\nX3xR8usCqH8rr7yyV5966qlefcABB3h1smevlO65e/vtt5dodChG8rWVpOOPP96rjzjiiEbPkW3d\njl69enl18udNrv2zWbx4sVefcsopXv3ggw+mjhkyZEjO8wJoWlZZZRWvbteuXaOPS+nP5qUwd+5c\nr07+fQxAbRs/fnxqW7IH8wYbbODV9dyDuTlYY401vDr5d5hhw4aljsm2plapbbnlll6d7efFwIED\nyz6OcuEOZgAAAAAAAABAECaYAQAAAAAAAABBmGAGAAAAAAAAAARhghkAAAAAAAAAEIRF/kokuWhN\ntqbhW2yxRUHnnD17dmrbSiv5L1mbNm28epNNNinoGiidLl265NwnufhjORbb69u3b2pbcvGRkSNH\nlvy6yG7ttdf26ssvv9yrd9xxx9QxkydP9upvvvnGq19++WWvnjp1ajFDlCR99dVXqW2jR48u+ryo\nXfvuu29q23nnnefV22yzTaPnWLp0aWpbPu+FqLwf/ehHqW25FvWrlhYtWnh1ckGU1VdfPXXMHXfc\n4dVvvfWWVyffRxEuuZhwnz59vPqggw7y6t69e6fOkW3ByMZMmjQpte3000/36vvuu6+gc6J0kn9G\n27ZtW/A5NtpoI6/O9hn5mGOO8erkZ6wePXoUfN1S+OCDD7yaRf6A+pJtkT/Ul80228yrR4wY4dVb\nb711znMkP5s88sgjXp1cZDr5WVOSTjzxRK/eZ599Gr3mr3/969Q2FvkDAAAAAAAAADQ7TDADAAAA\nAAAAAIIwwQwAAAAAAAAACEIP5jytu+66Xt2/f3+vHjJkiFe3atUq5zkXLFjg1b/97W+9+uOPP04d\nc/PNN3t1sgczKifZU7tbt26pfe69916vfuedd0o+jnXWWcerzz777NQ+L7zwglfffvvtJR8Hsvvj\nH//o1UcffbRXZ3st5s+f79XJ3t277rqrV2fLXtKaa67Z6OPTpk1LbXv66ae9OtnLG9WT7DspSVtt\ntZVXn3HGGV6d7NGf7OkvST/84Q+LHtsvfvELrx43bpxXP/bYY0VfA7kl++TefffdZbnOq6++6tUP\nP/xwwec4/vjjvTpbj+VM2fpJv/TSS42e85prril4XM1RsnfuX/7yl9Q+Bx98sFfn+ixaaL/lbJK9\nFSXp1ltv9eoDDzzQq3mvKZ9BgwZ5dbLn8BprrFGW65qZV5ciW6WQ7B+N2jJmzJhqDwFACXXv3j21\n7ayzzvLqfHouJ/3f//2fV//9738v+BzJz7C5ejAn1x2R0p/hp0yZUvA4qoU7mAEAAAAAAAAAQZhg\nBgAAAAAAAAAEYYIZAAAAAAAAABCEHsxZHHbYYaltV155pVcnezLn45lnnmn0Op988olX/+xnP0ud\no1OnTo1e45///GfB40KYnXfe2auz9c956qmnvHrp0qUlH0eyF2W2cTz77LMlvy5KI9m/XZI+/PDD\nos+bzMEuu+zi1VdddZVXjx07NnUOei7XjmQf7gcffDC1zzbbbNPoOb788kuvTvaylKT333/fq3P9\nzMnm9ddf92r6oFZGsl/b6NGjvXrjjTcuy3Wvv/56rw75HJJ8Pxo8eLBXJ/u9onxGjhzp1cmfHfl4\n4oknvDpb/+9ca1Ik1w3I1kM7uQZF165dvZr3nvJ57733vLpcPZfrxXPPPVftIaARCxcurPYQUOOe\nf/75ag8BBejXr19qW3INmFySa5tJ0rBhw4LHFCrb2m177LGHV9ODGQAAAAAAAADQ5DHBDAAAAAAA\nAAAIwgQzAAAAAAAAACBIzh7MZnajpH0kfeqc2zLetrak2yVtKOlDSYc6575c3jlqXevWrb36lFNO\nSe2Tq+dysrfu+PHjU/vsueeeXr1o0aJGz9mrV69GH89mzpw5BR9TDs0hNwcffLBXf/TRR6l9brvt\ntoqPwzmX2mf48OFlH0cpNIfcVMrixYu9+q233vLqZD/WP//5z2UfU7k0hdysssoqXp1879h11129\nuk2bNqlzfPvtt1795JNPevV5553n1ZMmTUqd48c//rFXv/jii1nH2yDbz7GhQ4c2ekytaAq5ydS3\nb1+vLkXP5VmzZnl1tp8lpVj7IbkGxWWXXebVxxxzjFdny3+lNLXcXHzxxV6dXF8im+T7woknntjo\n46VYf+K0005LbUv2YE727k6+B1azx2ZTy83jjz/u1eeee26jdYgZM2aktk2dOtWrk595P/30U69O\nvodJ0nHHHVf02JKWLFlS8nM2tcyUy2abbebVm2yySZVGUhvITZhRo0ZVewhVVeu5Sf49aeutty74\nHG+88YZXJz8zSKX5vLLCCoXdw5v8+5skTZgwoehxVEs+z364pL0S206X9LhzbmNJj8c1kGm4yA0K\nN1zkBoUbLnKDwg0XuUHhhovcoHDDRW5QmOEiMyjccJEbFG64yA1KIOcEs3PuaUmzE5v3l3RT/Pub\nJB1Q4nGhzpEbhCA3CEFuEILcIAS5QQhyg0KRGYQgNwhBblAqOVtkLEdb51zD/1uaKant8nY0s99J\n+l3gddC0kBuEIDcIQW4QIq/ckBkkkBuEIDcoFJ9tEILcIAS5QcFCJ5i/55xzZpZu+rrs8eskXSdJ\nje2H5oXcIAS5QQhygxCN5YbMYHnIDUKQGxSKzzYIQW4QgtwgX6ETzLPMrJ1zboaZtZP0ac4jatj8\n+fO9+oUXXkjts9VWWzW6T3KBrDFjxhQ8jvbt23t1v379ch7z9ddfe/Wll15a8HUrqEnlpnfv3l59\nySWXpPb57LPPyj6OgQMH5rxmJcZRRk0qN7UqufBBE1BXudluu+28+uc//7lXt2jRwqsfeeSR1Dku\nuOACr3722WcLHkfyZ0ouJ598cmpbtgVP60jd5GbllVf26j59+hR9zuRCkDfeeKNXT5s2rehr5GP6\n9OlenVxIMNuCb0n777+/V19zzTXFD2z56iY3SSeddJJXm5lXZ3tPSH5vsy2kVqzk4trrr79+zmPW\nWmstr/7lL3/p1dVc5G856jY3yQVe//a3v3n1o48+mjrmr3/9q1d379690WtkWyhpxx13bPSY5KKV\n2X5GlcITTzzh1ck/R2VUt5mplOR7GCSRG4SpmdwkP4uMGzcutc8GG2zg1SNGjPDqiy66yKs///zz\nosfVsmXL1LYhQ4YUdI5si/zlWmS9lhW2xOEy90k6Ov790ZLuLc1w0MSRG4QgNwhBbhCC3CAEuUEI\ncoNCkRmEIDcIQW5QsJwTzGY2QtJ4SZua2TQz6yfpIkl7mNm7knaPa+B75AYhyA1CkBuEIDcIQW4Q\ngtygUGQGIcgNQpAblErOFhnOucOX89BuJR4LmhBygxDkBiHIDUKQG4QgNwhBblAoMoMQ5AYhyA1K\npehF/pqiM888M7Vt9OjRXv3QQw8VfZ1kT81BgwZ5dbKPTDZXXnmlV8+ZM6focSE/M2fO9Op8+gOW\nQrI3a+vWrb062W9Iyt7bB5UxYcIErx42bJhXf/LJJxUZxxprrOHVX375pVcn84zKeuaZZ7z68MP9\nz3nJ95d//etfqXMsWbKkoGu2atUqte2yyy5r9Ji5c+d69dixYwu6Jkpnp5128uqdd9656HMm+6gm\nX+9K6dixo1eH9DdN9uNFmFdeeSW1rRR9C3M566yzvLpDhw45j0nmNdm7G+WTz3o2e++9t1c/9dRT\nXr3FFlt4ddu2bVPnGDVqlFcnc5FczyDEwoULvXrkyJGpfQYMGODVfM6ubVOmTKn2EACU0ODBg/Pa\nVm7dunVLbevatWtB5/jiiy9KNZyaENqDGQAAAAAAAADQzDHBDAAAAAAAAAAIwgQzAAAAAAAAACAI\nPZizyNbHuBQ9l1dccUWv/vOf/+zVp5xySs5zJHu0XHXVVUWPC2GS3/uhQ4em9vnqq6+8+uKLL/bq\nkP67l1xyiVebmVdvu+22qWOS/VvpRVY5yfeOUryXhNhtN3+NhuT7HD2Ya8vdd99d9msceeSRqW19\n+vRp9Jg77rjDqydPnlzSMSG7bbbZJrXtP//5T9HnTf65X7p0adHnLIX27dt7dcuWLQs+x/Dhw0s0\nmqbt/vvv9+oDDzzQq3v27Jk65pFHHvHqSZMmFXzdLl26eHWyj+HKK69c8DkPPvhgr+b9qbYsWLDA\nq5944gmvTvZgzib5GjvnCh7H119/7dXvvfeeV/fu3durZ8+eXfA1UFvefPPNag8BNe6QQw5JbRs/\nfrxXJ3vAo/lZddVVvfq4444r+pznn39+0eeoJdzBDAAAAAAAAAAIwgQzAAAAAAAAACAIE8wAAAAA\nAAAAgCD0YK6ggw46yKvz6bmcdP3113v19OnTixoTwt18881e/atf/Sq1z4ABAxqtn3vuOa/O1hs5\n2SNzk0028epk/7nWrVunzkHPZWTr4YrmZYUV/H9TPuyww3Iek3x/Sa4dgMpYffXVU9t+8IMfFHSO\nbH1E9957b6+eP39+YQMrkS233NKrR4wYUZVxNEdHHXWUV0+YMMGrN91009QxP/vZzxqtyyFbr90T\nTjjBq5966qmyjwPhFi9e7NXvvvtu2a/57bffprZdffXVXn3aaaeVfRwon7XXXturJ06cmNpn1qxZ\nlRoO6sRK00Z8AAAUzklEQVShhx6ac59p06ZVYCSoJ0OGDPHq5GeofCT/LpWcU6p33MEMAAAAAAAA\nAAjCBDMAAAAAAAAAIAgTzAAAAAAAAACAIEwwAwAAAAAAAACCsMhfmVx66aWpbSeffHJB57jhhhtS\n284///zgMaG0pk6d6tV77bVXap9zzz3Xq7fffnuv7tmzp1fvuOOORY9j8ODBBZ8DTd+Pf/zjag8B\nVZb8GdS7d++cxwwbNsyrP/roo5KOCfn55z//mdpmZgWd46GHHkpte+2114LHVEoPPvigV3fo0KHR\n/bM99+QicIV+f5qrhQsXevW+++7r1QMHDkwd0759e6/efffdvbpNmzYlGt0ykyZNSm1Lvj+htnXr\n1s2rjzjiiJJf49Zbb/Xqe+65J7XP3XffXfLronr69u3r1YsWLUrtk1xgEujevXvOfe68884KjAS1\nLLmg9vHHH1/wOe69916vTs7nLV26tPCB1TDuYAYAAAAAAAAABGGCGQAAAAAAAAAQhAlmAAAAAAAA\nAEAQejCXyLhx47y6R48eBZ9j+PDhXj1gwIDUPt98803B50VlfPzxx6ltv/nNb7y6ZcuWXr3OOut4\n9SGHHJI6x9ChQ716/vz5Xr3FFls0+jiapzXXXNOrd9llF6++8sorKzkcVMGKK67o1cneqtl88MEH\nXn322WeXdEzIT7t27bx6lVVWSe2T7Dmcy4gRI4oaU6iVVvI/ap5zzjmpfdZbbz2vzvXcsj2e7ON8\n++235ztEZJgyZYpXn3DCCTmPad26tVevsIJ//0q2/CY/N3fu3LnRa/z2t7/NOQ7UjpNOOim1Ldl3\nctVVVy34vMlsJXtXPv/8815Nv+WmJ/l+ss8++3j1nDlzKjkc1KmOHTtWewioQcm/O910001evcYa\naxR8zkceecSrlyxZUvjA6gh3MAMAAAAAAAAAgjDBDAAAAAAAAAAIwgQzAAAAAAAAACBIk+vB3KJF\ni9S2Vq1aeXWyt8opp5zi1XvvvXfqHFtuuWWj1032GMzm22+/9erTTjvNq//xj394dbKvGOpfMgMz\nZszw6jZt2qSOWX311b06V09mQJLMzKuT73u8vzR9O++8s1f37Nkz5zHJ96R58+aVdEzIzwEHHODV\nG220UZVGUrjkz6z+/ft79RlnnFH0NR5//PHUtuSaB7Nnzy76OshPrs8hBx10UGpbrp7Ld911l1e/\n9tprhQ8MZTNkyBCv7t69u1f36tUrdUyyd26hfeSl9GeX119/3aufe+65gs+J+vKDH/zAqzfeeGOv\nHjZsWCWHgzrVoUMHr546dWpqn/Hjx1dqOKgRe+65p1fvtddeRZ8z+Z6UXAdp//33Tx2T/Jy7ePHi\nosdRKdzBDAAAAAAAAAAIwgQzAAAAAAAAACAIE8wAAAAAAAAAgCB114M52U+5R48eXp2tt9/uu+9e\n9HWT/UxD+oYle6fceOONXk1P1OYn2bs72XdTSmft3//+d1nHhKYpmSP6FDZ9Z555ZqOPv//++6lt\nhx9+eLmGgwL06dOn2kPIW7IX68CBA726bdu2BZ9z7ty5Xn311Vd7dXItAklatGhRwddBeeywww5e\nfe211+Y8ZubMmV6d7Km9cOHC4geGvCT7KUvSdddd59VdunTx6uTfkyolufbOxx9/XJVxoHqSn2/v\nvPPOKo0Etezyyy/36uQc0h133JE6JltfZtSv1VZbzauz/T3p5JNPLvl1kz8fk/M/DzzwQOqYcePG\nefWFF17o1e+8845Xz5o1K3WOb775pqBxlgp3MAMAAAAAAAAAgjDBDAAAAAAAAAAIwgQzAAAAAAAA\nACAIE8wAAAAAAAAAgCA1v8hfx44dvXrw4MFePWDAgJzneOWVV7x6q6228urkAhHZhCzql5RsLD52\n7FivvvXWW716/fXXT51j5MiRjV7j008/9epPPvmkkCGiwg466CCv3nbbbVP7/P3vf/fqOXPmlHVM\naBp22mknr16wYIFXv/rqq5UcDipgr7328upddtml0f2ffPLJ1LZp06aVckgI1KlTp5Kfs1u3bqlt\nK6zQ+H0G7du39+psCylvsMEGXl2Kz0vPPPOMV+dasBLVlVyw5txzz/XqbJ+zv/vuO6+++OKLvXre\nvHmlGRxySi7uev3116f2SS6ynku2xYUeeughrz7nnHO8OrnQUYcOHXJeJ5mjZBbR9HTu3NmrP/ro\nI69+6aWXKjkc1Inkon5J48ePr9BIUCnJ94pLL73Uq/fbb7+Cz5n82TZ8+PDUPsmFjjfbbDOvzufn\nafLv8ck6adiwYaltp556qldXatE/7mAGAAAAAAAAAARhghkAAAAAAAAAECTnBLOZdTSzJ8zsbTN7\ny8xOiLevbWaPmtm78a9rlX+4qBfkBiHIDUKQGxSKzCAEuUEIcoMQ5AYhyA1CkBuUiuXqlWdm7SS1\nc869YmarS5og6QBJx0ia7Zy7yMxOl7SWc+60HOdq9GJbbrllaluyX1eyH2ClJMeR7IubdNFFF6W2\nbbPNNiUdkyRNnz7dq88//3yv/te//lXya2bjnLPMupK5qWejRo3y6j59+qT26dKli1cnX/N6Rm7K\np1+/fl6dfG+o1ntpKZCb7D1NH3vsMa/u1auXV7/77rte3bdv39Q5Jk6cWILR1abM3NR6Zvr37+/V\n2Xqr1Qoz749jUA/mMWPGePUxxxzj1cn1JSpognOua0NR67mpluTrdeONN+Y85oUXXvDqXP0x60m9\n/YwaMWKEVx9yyCEFnyPZN/30009P7ZPsjXvnnXd69b777pvzOt9++61XJ3tM1vJ7ZS71lptqGTRo\nkFf/4Q9/8Ork35uaOnKTn+Rnk2TP5R133LGSw6m6ppibPffc06tvu+02r15rrcLnxl9//XWvHjhw\noFc/++yzOc+RfI866aSTvDrZK7pUjjzySK9O/qwPkcxNNjnvYHbOzXDOvRL/fp6kdyS1l7S/pJvi\n3W5SFEBAErlBGHKDEOQGhSIzCEFuEILcIAS5QQhygxDkBqVS0FK7ZrahpG0lvSCprXNuRvzQTElt\nl3PM7yT9LnyIqHfkBiHIDUKQGxSKzCAEuUEIcoMQ5AYhyA1CkBsUI+9F/systaS7JJ3onJub+ZiL\n/s9B1lvhnXPXOee6Zv4XQzQf5AYhyA1CkBsUiswgBLlBCHKDEOQGIcgNQpAbFCuvO5jNrIWioN3q\nnPtvvHmWmbVzzs2Ie7YU3SDv0UcfTW1r2zbrP5KU1BtvvOHVV155ZWqfu+++26vnzJnT6DmT/Vok\n6fDDD/fqZK+f7t27e3WrVq1S57juuuu8+oYbbvDqDz/8sNFxVVKlclNPkq9pss/dGWeckTpm9uzZ\nZR1TrSE3pdGmTZtqD6GimltuVltttdS2ZM/lpLFjx3p1U+63nI9azkyyP+2sWbNS+1Ti81E5PPnk\nk6lthx12mFfPnTs3tU+tqOXcVMvBBx9c8DEXX3xxGUZSu5p6bpK92DfaaKPUPhdeeKFX77TTTgVf\n55NPPvHqeu65nI+mnpsQe+yxR7WHUPPIjXTooYc2+vgdd9xRoZHUj3rKTceOHVPbStFzedKkSV79\npz/9yavz6bmcdM0113j1Pffc49V/+9vfUsdst912Xt2pUyevTq57kO3vCU888URB4yyVnHcwW/SJ\n4QZJ7zjn/prx0H2Sjo5/f7Ske0s/PNQrcoMQ5AYhyA0KRWYQgtwgBLlBCHKDEOQGIcgNSiWfO5h7\nSjpK0ptm9lq87QxJF0kaZWb9JH0kqfF/JkJzQ24QgtwgBLlBocgMQpAbhCA3CEFuEILcIAS5QUnk\nnGB2zj0jyZbz8G6lHQ6aCnKDEOQGIcgNCkVmEILcIAS5QQhygxDkBiHIDUolrx7MlfLII4+kth11\n1FFFn3fw4MFenex78sUXX3h1rv7K+Zg5c2Zq2xVXXNFoneynuOKKK6bOkew9hvoyYMAAr4565S9z\nwQUXpI5J9gJK9gMHsundu3e1h4Ay6tChQ8HHrL322mUYCcrhlVde8eoHH3wwtc+xxx5bqeE0Ktl7\ndfTo0V6d/Kwzfvz41DkWLVpU+oGhbE466SSv7tOnT6P7Z+v3/sADD5R0TAj3+eefF32Onj17Nlrn\n47vvvvPqbO8Lv//97ws+L5qWbGtQAEl9+/Zt9PFsn0VQP7KtVRbSczlpzz339Orp06cXfc6k5Dxh\nch0SSVp33XW9Orm20owZM7x64cKFJRpd8XL2YAYAAAAAAAAAIBsmmAEAAAAAAAAAQZhgBgAAAAAA\nAAAEYYIZAAAAAAAAABCkphb5y7ZgzcMPP+zV6623nlfffPPNOc/71VdfefXSpUsDRld+s2bNqvYQ\nUGabb755o49nyzML4QBIOuCAAwo+pmPHjmUYCSrh6quvTm3r2rWrVycXBFl//fWLvm5yka133303\ntc+4ceO8+vTTT/fqBQsWFD0OVE///v1T2y699FKvXmGFxu9XybY48eLFi4sbGErm66+/rsh1kn//\nevvtt736yiuv9Op///vfZR8T6k9y0a0XX3yxSiNBPXv++eerPQQU4csvv0xtmzx5sle3aNHCq88/\n/3yvvv/++1PnmDNnTglGV7zPPvus0bqWcQczAAAAAAAAACAIE8wAAAAAAAAAgCBMMAMAAAAAAAAA\ngphzrnIXM6vcxVBRzjkr17mbUm6GDh3q1YMHD/bqn/zkJ6ljJk2aVNYxVRO5KY2VV145tS3Zm2r2\n7Nle3b59+7KOqZzIjXTDDTekth1zzDFenfz5nuzbPHr06JKPq5aVKze1kpnkz49+/fp5dbbPe8ke\nqEnJ941Ro0YFjq5uTXDOdc29W+FqJTdJrVq18uqnn346tc/222/f6Dnuvfderz700ENT+zTlHsz1\n9jOqc+fOXt2zZ8/UPoMGDfLq5JoiY8aMyXmda6+91quT6+w0d/WWG9QGcoMQ5AYh8skNdzADAAAA\nAAAAAIIwwQwAAAAAAAAACMIEMwAAAAAAAAAgCD2YURL08UEIclMaq6yySmrbggULvDrZb3e//fYr\n65jKidxIHTp0SG37xz/+4dXDhw/36mRf1OamqfdgRlk0ux7M7dq18+rp06fnPGbWrFlevcsuu3j1\n5MmTix9YHWmKP6NWX311r15rrbW8+uOPP67kcJqkppgblB+5QQhygxD0YAYAAAAAAAAAlA0TzAAA\nAAAAAACAIEwwAwAAAAAAAACCrFTtAQAAivPdd9+lts2cOdOrFy9eXKnhoAKmTZuW2nbggQdWYSQA\nmpIvvvjCq1988cXUPt26dfPqW265xaubW8/l5mDevHmN1gAAANzBDAAAAAAAAAAIwgQzAAAAAAAA\nACAIE8wAAAAAAAAAgCBMMAMAAAAAAAAAgphzrnIXM6vcxVBRzjkr17nJTdNFbhCC3CBEuXJDZpq0\nCc65ruU4MblpuvgZhRDkBiHIDUKQG4TIJzfcwQwAAAAAAAAACMIEMwAAAAAAAAAgCBPMAAAAAAAA\nAIAgK1X4ep9L+kjSD+Lf17p6GadU3bH+qMznJzflUe1xkhsf48wPufExzvyUMzcNmZGq/zzzxTjz\nU4ncVPs5FqJexspn4tpRL+OUyE0tqZdxSuSmljDO/JAbH+PMT165qegif99f1Ozlci2aUkr1Mk6p\nvsYaql6eI+OsLfXyPBlnbamX58k4a0u9PE/GWTvq6TnWy1jrZZzFqJfnWC/jlOprrKHq5TnWyzil\n+hprqHp5joyzttTL82ScpUWLDAAAAAAAAABAECaYAQAAAAAAAABBqjXBfF2VrluoehmnVF9jDVUv\nz5Fx1pZ6eZ6Ms7bUy/NknLWlXp4n46wd9fQc62Ws9TLOYtTLc6yXcUr1NdZQ9fIc62WcUn2NNVS9\nPEfGWVvq5XkyzhKqSg9mAAAAAAAAAED9o0UGAAAAAAAAACBIRSeYzWwvM/ufmU0xs9Mree1czOxG\nM/vUzCZmbFvbzB41s3fjX9eq5hjjMXU0syfM7G0ze8vMTqjVsZYKuSkeuSE3IcgNuQkYY7PLjFS7\nuamHzMRjIjfkpmDkhtyEIDfkJkRzzE2tZkYiN7WM3BSvnnNTsQlmM1tR0lWSfi6pi6TDzaxLpa6f\nh+GS9kpsO13S4865jSU9HtfVtkTSyc65LpJ2kHR8/H2sxbEWjdyUDLkhNyHIDbkpVLPKjFTzuRmu\n2s+MRG7ITRhyQ25CkBtyE6JZ5abGMyORm5pEbkqmfnPjnKvIl6QeksZk1EMkDanU9fMc44aSJmbU\n/5PULv59O0n/q/YYs4z5Xkl71MNYyU3tfJGb6n+Rm9r7Ijdkpinmpt4yQ26qPzZyU7tf5IbckJva\n+Grquan1zJCb2vwiN+Smki0y2kuamlFPi7fVsrbOuRnx72dKalvNwSSZ2YaStpX0gmp8rEUgNyVG\nbmpWTb8W5KZm1exr0UwyI9Vfbmr6tSA3NaumXwtyU7Nq+rUgNzWrpl+LZpKbesuMVOOvBbmpWTX9\nWtRbbljkL08u+mcCV+1xNDCz1pLuknSic25u5mO1NtbmrNZeC3JTH2rttSA39aGWXgsyUx9q7bUg\nN/Wh1l4LclMfau21IDf1odZeC3JTH2rttSA39aHWXot6zE0lJ5inS+qYUXeIt9WyWWbWTpLiXz+t\n8ngkSWbWQlHQbnXO/TfeXJNjLQFyUyLkhtyEIDfkplDNLDNS/eWmJl8LckNuQpAbchOC3JCbEM0s\nN/WWGalGXwtyQ25C1GtuKjnB/JKkjc1sIzNrKekwSfdV8Poh7pN0dPz7oxX1PqkqMzNJN0h6xzn3\n14yHam6sJUJuSoDckJsQ5IbcFKoZZkaqv9zU3GtBbshNCHJDbkKQG3ITohnmpt4yI9Xga0FuyE2I\nus5NJRs+S/qFpMmS3pN0ZiWvncfYRkiaIWmxol4x/SSto2h1xnclPSZp7RoYZy9Ft8K/Iem1+OsX\ntThWclM7rwW5ITfkhtyQmeaXm3rIDLkhN+SG3JAbckNuau+rVjNDbmr7i9w079xY/AQAAAAAAAAA\nACgIi/wBAAAAAAAAAIIwwQwAAAAAAAAACMIEMwAAAAAAAAAgCBPMAAAAAAAAAIAgTDADAAAAAAAA\nAIIwwQwAAAAAAAAACMIEMwAAAAAAAAAgCBPMAAAAAAAAAIAg/w9ofSpzS8J16QAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f316b29c810>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "plt.ioff()\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# Helper plotting routine.\n",
    "def display_images(gens, title=\"\"):\n",
    "    fig, axs = plt.subplots(1, 10, figsize=(25, 3))\n",
    "    fig.suptitle(title, fontsize=14, fontweight='bold')\n",
    "    \n",
    "    for i in xrange(10):\n",
    "        reshaped_img = (gens[i].reshape(28, 28) * 255).astype(np.uint8)\n",
    "        axs.flat[i].imshow(reshaped_img)\n",
    "        #axs.flat[i].axis('off')        \n",
    "    return fig, axs\n",
    "\n",
    "\n",
    "batch_xs, batch_ys = mnist.train.next_batch(10)\n",
    "list_of_images = np.split(batch_xs, 10)\n",
    "_ = display_images(list_of_images, \"Some Examples from the Training Set.\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DrVjQKHlT2m4"
   },
   "source": [
    "# Building a Feed-Forward Neural Network\n",
    "\n",
    "In this section, we will build a neural network that takes the raw MNIST pixels as inputs and outputs 10 values, which we will interpret as the probability that the input image belongs to one of the 10 digit classes (0 through 9). Along the way, the data will pass through the hidden layers and activation functions we encountered in the first practical. \n",
    "\n",
    "**NOTE**: Standard feedforward neural network architectures can be summarised by chaining together the number of neurons in each layer, e.g. \"784-500-300-10\" would be a net with 784 input neurons, followed by a layer with 500 neurons, then 300, and finally 10 output classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T2L7D76zYZZv"
   },
   "source": [
    "## Build the model (30min)\n",
    "\n",
    "We want to explore the different choices and some of the best practices of training feedforward neural networks on the MNIST dataset. In this practical we will only be using \"fully connected\" (also referred to as \"FC\", \"affine\", or \"dense\") layers (i.e. no convolutional layers). So let's first write a little helper function to construct these:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "my3WeXTdEt34"
   },
   "outputs": [],
   "source": [
    "def _dense_linear_layer(inputs, layer_name, input_size, output_size):\n",
    "    \"\"\"\n",
    "    Builds a layer that takes a batch of inputs of size `input_size` and returns \n",
    "    a batch of outputs of size `output_size`.\n",
    "         \n",
    "    Args:\n",
    "        inputs: A `Tensor` of shape [batch_size, input_size].\n",
    "        layer_name: A string representing the name of the layer.\n",
    "        input_size: The size of the inputs\n",
    "        output_size: The size of the outputs\n",
    "        \n",
    "    Returns:\n",
    "        out, weights: tuple of layer outputs and weights.\n",
    "        \n",
    "    \"\"\"\n",
    "    # Name scopes allow us to logically group together related variables.\n",
    "    # Setting reuse=False avoids accidental reuse of variables between different runs.\n",
    "    with tf.variable_scope(layer_name, reuse=False):\n",
    "        # Create the weights for the layer\n",
    "        layer_weights = tf.get_variable(\"weights\",\n",
    "                                        shape=[input_size, output_size], \n",
    "                                        dtype=tf.float32, \n",
    "                                        initializer=tf.random_normal_initializer())\n",
    "        # Create the biases for the layer\n",
    "        layer_bias = tf.get_variable(\"biases\", \n",
    "                                     shape=[output_size], \n",
    "                                     dtype=tf.float32, \n",
    "                                     initializer=tf.random_normal_initializer())\n",
    "        \n",
    "        ## IMPLEMENT-ME: (1) \n",
    "        outputs = ...\n",
    "    \n",
    "    return (outputs, layer_weights)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LTFPjWj7FqJy"
   },
   "source": [
    "Now let's use this to construct a linear softmax classifier as before, which we will expand into a near state-of-the-art feed-forward model for MNIST. We first create an abstract `BaseSoftmaxClassifier` base class that houses common functionality between the models. Each specific model will then provide a `build_model` method that represents the logic of that specific model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "oaHvemlxVeIi"
   },
   "outputs": [],
   "source": [
    "class BaseSoftmaxClassifier(object):\n",
    "    def __init__(self, input_size, output_size, l2_lambda):        \n",
    "        # Define the input placeholders. The \"None\" dimension means that the \n",
    "        # placeholder can take any number of images as the batch size. \n",
    "        self.x = tf.placeholder(tf.float32, [None, input_size])\n",
    "        self.y = tf.placeholder(tf.float32, [None, output_size])    \n",
    "        self.input_size = input_size\n",
    "        self.output_size = output_size\n",
    "        self.l2_lambda = l2_lambda\n",
    "\n",
    "        self._all_weights = [] # Used to compute L2 regularization in compute_loss().\n",
    "        \n",
    "        # You should override these in your build_model() function.\n",
    "        self.logits = None\n",
    "        self.predictions = None\n",
    "        self.loss = None\n",
    "        \n",
    "        self.build_model()\n",
    "        \n",
    "    def get_logits(self):\n",
    "        return self.logits\n",
    "    \n",
    "    def build_model(self):\n",
    "        # OVERRIDE THIS FOR YOUR PARTICULAR MODEL.\n",
    "        raise NotImplementedError(\"Subclasses should implement this function!\")\n",
    "        \n",
    "    def compute_loss(self):\n",
    "        \"\"\"All models share the same softmax cross-entropy loss.\"\"\"\n",
    "        assert self.logits is not None    # Ensure that logits has been created! \n",
    "        \n",
    "        # IMPLEMENT-ME: (2)\n",
    "        # HINT: This time, use the TensorFlow function tf.nn.softmax_cross_entropy_with_logits  rather than \n",
    "        # implementing it manually like we did in Prac1\n",
    "        data_loss = ...\n",
    "\n",
    "        reg_loss = 0.\n",
    "        for w in self._all_weights:\n",
    "            # IMPLEMENT-ME: (3)\n",
    "            # HINT: TensorFlow has a built-in function for this too! tf.nn.l2_loss\n",
    "            reg_loss += ...\n",
    "            \n",
    "        return data_loss + self.l2_lambda * reg_loss\n",
    "    \n",
    "    def accuracy(self):\n",
    "        # Calculate accuracy.\n",
    "        assert self.predictions is not None    # Ensure that pred has been created!\n",
    "        \n",
    "        # IMPLEMENT-ME: (4)\n",
    "        # HINT: Look up the tf.equal and tf.argmax functions\n",
    "        correct_prediction = ...\n",
    "        \n",
    "        accuracy = tf.reduce_mean(tf.cast(correct_prediction, \"float\"))\n",
    "        return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JVqOqr8qKApu"
   },
   "source": [
    "If we wanted to reimplement the linear softmax classifier from before, we just need to override `build_model()` to perform one projection from the input to the output logits, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "jLVw-NWEKBI9"
   },
   "outputs": [],
   "source": [
    "class LinearSoftmaxClassifier(BaseSoftmaxClassifier):\n",
    "    def __init__(self, intput_size, output_size, l2_lambda):\n",
    "        super(LinearSoftmaxClassifier, self).__init__(input_size, output_size, l2_lambda)\n",
    "        \n",
    "    def build_model(self):\n",
    "        # The model takes x as input and produces output_size outputs.\n",
    "        self.logits, weights = _dense_linear_layer(\n",
    "                self.x, \"linear_layer\", self.input_size, self.output_size)\n",
    "        \n",
    "        self._all_weights.append(weights)\n",
    "        \n",
    "        self.predictions = tf.nn.softmax(self.logits)\n",
    "        \n",
    "        self.loss = self.compute_loss()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A1Lxtrw0KnmW"
   },
   "source": [
    "In order to build a ***deeper*** model, let's add several layers with multiple transformations and rectied linear units (relus):\n",
    "\n",
    "```python\n",
    "def build_model(self):\n",
    "     # The first layer takes x as input and has n_hidden_1 outputs.\n",
    "     layer1, weights1 = _build_linear_layer(self.x, \"layer1\", self.input_size, self.num_hidden_1)\n",
    "     self._all_weights.append(weights1)\n",
    "     layer1 = tf.nn.relu(layer1)\n",
    "        \n",
    "     # The second layer takes layer1's output as its input and has num_hidden_2 outputs.\n",
    "     layer2, weights2 = _build_linear_layer(layer1, \"layer2\", self.num_hidden_1, self.num_hidden_2)\n",
    "     self._all_weights.append(weights2)\n",
    "     layer2 = tf.nn.relu(layer2)\n",
    "     \n",
    "     # The final layer is our predictions and goes from num_hidden_2 inputs to \n",
    "     # num_classes outputs. The outputs are \"logits\" (un-normalised scores). \n",
    "     self.logits, weights3 = _build_linear_layer(layer2, \"output\", self.num_hidden_2, self.output_size)\n",
    "     self._all_weights.append(weights3)\n",
    "     \n",
    "     self.pred = tf.nn.softmax(self.logits)\n",
    "     self.loss = self.compute_loss(self.logits, self.y)\n",
    " ```\n",
    "\n",
    "**Note: Instead of writing special classes for linear nets (0 hidden layers), 1 hidden layer nets, 2 hidden layer nets, etc., we can generalize this as follows:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "QVCIXYUrHXai"
   },
   "outputs": [],
   "source": [
    "class DNNClassifier(BaseSoftmaxClassifier):\n",
    "    \"\"\"DNN = Deep Neural Network - now we're doing Deep Learning! :)\"\"\"\n",
    "    \n",
    "    def __init__(self, \n",
    "                 input_size=784,    # There are 28x28 = 784 pixels in MNIST images\n",
    "                 hidden_sizes=[],    # List of hidden layer dimensions, empty for linear model.\n",
    "                 output_size=10,    # There are 10 possible digit classes\n",
    "                 act_fn=tf.nn.relu,    # The activation function to use in the hidden layers\n",
    "                 l2_lambda=0.):    # The strength of regularisation, off by default.\n",
    "        \n",
    "        self.hidden_sizes = hidden_sizes\n",
    "        self.act_fn = act_fn\n",
    "        super(DNNClassifier, self).__init__(input_size, output_size, l2_lambda)\n",
    "        \n",
    "        \n",
    "    def build_model(self):\n",
    "        prev_layer = self.x\n",
    "        prev_size = self.input_size\n",
    "        \n",
    "        for layer_num, size in enumerate(self.hidden_sizes):        \n",
    "            layer_name = \"layer_\" + str(layer_num)\n",
    "            ## IMPLEMENT-ME: (5)\n",
    "            # HINT: Use the linear function we defined earlier!\n",
    "            layer, weights = ...\n",
    "\n",
    "            self._all_weights.append(weights)\n",
    "            ## IMPLEMENT-ME: (6)\n",
    "            # HINT: What do we still need to do after doing the \"linear\" part? \n",
    "            layer = ...\n",
    "            prev_layer, prev_size = layer, size\n",
    "\n",
    "        # The final layer is our predictions and goes from prev_size inputs to \n",
    "        # output_size outputs. The outputs are \"logits\", un-normalised scores. \n",
    "        self.logits, out_weights = _dense_linear_layer(prev_layer, \"output\", prev_size, self.output_size)\n",
    "        self._all_weights.append(out_weights)\n",
    "        \n",
    "        self.predictions = tf.nn.softmax(self.logits)\n",
    "        self.loss = self.compute_loss()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "aHkH_jEc9C3j"
   },
   "source": [
    "We can now create a linear model, i.e. a 784-10 architecture (note that there is only one possible linear model going from 784 inputs to 10 outputs), as follows:\n",
    "\n",
    "```python\n",
    "    tf_linear_model = DNNClassifier(input_size=784, hidden_sizes=[], output_size=10)\n",
    "```\n",
    "\n",
    "We can create a deep neural network (DNN, also called a \"multi-layer perceptron\") model, e.g. a 784-512-10 architecture (there can be many others...), as follows:\n",
    "\n",
    "```python\n",
    "    tf_784_512_10_model = DNNClassifier(input_size=784, hidden_sizes=[512], output_size=10)\n",
    "```\n",
    "\n",
    "and so forth.\n",
    "\n",
    "**NOTE: Make sure you understand how this works before you move on.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kVynk3SWzUg_"
   },
   "source": [
    "## Sanity Checks (5-10 min)\n",
    "\n",
    "### Dealing with randomness\n",
    "\n",
    "Pseudo-random number generators start from some value (the 'seed') and generate numbers which appear to be random. It is good practise to seed RNGs with a fixed value to encourage reproducibility of results. We do this in NumPy by using `np.random.seed(1234)`, where `1234` is your chosen seed value. \n",
    "\n",
    "In TensorFlow we can set a *graph-level* seed (global, using `tf.set_random_seed(1234)`) or an *op-level* seed (passed in to each op via the `seed` argument, to override the graph-level seed.)\n",
    "\n",
    "Next we need to initialize the parameters of our model. We do *not* want to initialize all weights to be the same. \n",
    "\n",
    "**QUESTION**: Can you think of why this might be a bad idea? (Think about a simple MLP with one input and 2 hidden units and one output. Look at the contributions via each weight connection to the activations on the hidden layer. Now let those weights be equal. How does that affect the contributions? How does that affect the update that backprop would propose for each weight? Do you see any problems?)\n",
    "\n",
    "The simplest approach is to **initialize weights (W's) to small random values**. For ReLUs specifically, it is recommended to initialize weights to `np.random.randn(n) * sqrt(2.0/n)`, where `n` is the number of inputs to that layer (neuron on the previous layer) (see [He et al., 2015](https://arxiv.org/pdf/1502.01852.pdf) for more information). This initialization encourages the distributions over ReLU activations to have roughly the same variance at each layer, which in turn ensures that useful gradient information gets sent back during backpropagation.\n",
    "\n",
    "Biases are not that sensitive to initialization and can be initialized to 0s or small random numbers.\n",
    "\n",
    "### Check the loss of a random model\n",
    "\n",
    "When we have a new model, it's always a good idea to do a quick sanity check. A random model that predicts C classes on random data should have no reason for preferring either class, i.e., on average its loss (negative log-likelihood) should be $-\\log(1/C) = -\\log(C^{-1}) = \\log(C)$. \n",
    "\n",
    "**NOTE**: TF actually computes the cross-entropy on the *logits* for numerical stability reasons (logs of small numbers blow up quickly..). This means we'll get a different value from `tf.nn.softmax_cross_entropy_with_logits`. So for this check we will just manually compute the cross-entropy (negative log-likelihood).\n",
    "\n",
    "The second thing to check is that adding the L2 loss (a strictly positive value), should *increase* total loss (here we'll just use the TF cross-entropy as provided)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 408,
     "output_extras": [
      {
       "item_id": 2
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1026,
     "status": "ok",
     "timestamp": 1503657319275,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "DUMQp9kw0Isx",
    "outputId": "b8d7a421-e489-4b54-a896-aa2da2435020"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sample of random data:\n",
      "[[ 0.47143516 -1.19097569  1.43270697 ..., -0.2453605  -1.26943186\n",
      "  -0.26232386]\n",
      " [ 2.33759848 -0.78171744  0.08009975 ..., -0.64055353  1.76256841\n",
      "  -0.08567302]\n",
      " [ 1.63617833 -0.54410827 -1.04999868 ..., -0.90640906  0.31915076\n",
      "  -0.49914386]\n",
      " [-0.66039926  0.0774697   0.38755182 ...,  0.31019053  1.87791254\n",
      "  -0.70183467]\n",
      " [-3.23350453  0.20024296 -0.13933709 ..., -0.72622006  0.50774695\n",
      "   0.18987136]]\n",
      "Shape:  (100, 784)\n",
      "Sample of random labels:\n",
      "[[ 0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  1.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  1.  0.]\n",
      " [ 0.  0.  0.  0.  0.  1.  0.  0.  0.  0.]\n",
      " [ 0.  0.  0.  0.  0.  0.  0.  0.  1.  0.]]\n",
      "Shape:  (100, 10)\n",
      "\n",
      "Sanity check manual avg cross entropy:  2.30259\n",
      "Model loss (no reg):  46.3142\n",
      "Sanity check loss (with regularization, should be higher):  4025.31\n"
     ]
    }
   ],
   "source": [
    "tf.set_random_seed(1234)\n",
    "np.random.seed(1234)\n",
    "\n",
    "# Generate a batch of 100 \"images\" of 784 pixels consisting of Gaussian noise.\n",
    "x_rnd = np.random.randn(100, 784)\n",
    "print \"Sample of random data:\\n\", x_rnd[:5,:]    # Print the first 5 \"images\"\n",
    "print \"Shape: \", x_rnd.shape\n",
    "# Generate some random one-hot labels.\n",
    "y_rnd = np.eye(10)[np.random.choice(10, 100)]\n",
    "print \"Sample of random labels:\\n\", y_rnd[:5,:]\n",
    "print \"Shape: \", y_rnd.shape\n",
    "\n",
    "# Model without regularization.\n",
    "tf.reset_default_graph()\n",
    "tf_linear_model = DNNClassifier(l2_lambda=0.0)\n",
    "x, y = tf_linear_model.x, tf_linear_model.y\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables.\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    \n",
    "    avg_cross_entropy = -tf.log(tf.reduce_mean(tf_linear_model.predictions))\n",
    "    loss_no_reg = tf_linear_model.loss\n",
    " \n",
    "    manual_avg_xent, loss_no_reg = sess.run([avg_cross_entropy, loss_no_reg],\n",
    "                                            feed_dict={x : x_rnd, y: y_rnd})\n",
    "    \n",
    "    \n",
    "# Sanity check: Loss should be about log(10) = 2.3026\n",
    "print '\\nSanity check manual avg cross entropy: ', manual_avg_xent\n",
    "print 'Model loss (no reg): ', loss_no_reg\n",
    "\n",
    "# Model with regularization.\n",
    "tf.reset_default_graph()\n",
    "tf_linear_model = DNNClassifier(l2_lambda=1.0)\n",
    "x, y = tf_linear_model.x, tf_linear_model.y\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables.\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "    loss_w_reg = tf_linear_model.loss.eval(feed_dict={x : x_rnd, y: y_rnd})\n",
    "\n",
    "# Sanity check: Loss should go up when you add regularization\n",
    "print 'Sanity check loss (with regularization, should be higher): ', loss_w_reg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X0YXZ-u284tK"
   },
   "source": [
    "## Compute the gradients: Backpropagation Review\n",
    "\n",
    "Once one has implemented the model and checked that the loss produces sensible outputs it is time to create the training loop. For this, we will need to obtain the gradients of the loss wrt each model parameter. We've already looked at this in detail in practical 1, where we performed this manually. One of the great benefits of using a library like TensorFlow, is that it can automatically derive the gradients wrt any parameter in the graph (using `tf.gradients()`).\n",
    "\n",
    "In the previous practical we saw that there is a common pattern to deriving the gradients in neural networks:\n",
    "\n",
    "1. propagate activations forward through the network (\"make a prediction\" $\\Rightarrow$ `fprop`),\n",
    "2. compute an error delta (\"see how far we're off\") , and\n",
    "3. propagate errors backwards to update the weights (\"update the weights to do better next time\" $\\Rightarrow$ `backprop`).\n",
    "\n",
    "It turns out that for deeper networks, ***this pattern repeats for every new layer***. Take a deep breath, and let's dive in, starting with fprop:\n",
    "\n",
    "### FPROP\n",
    "\n",
    "Conceptually, forward propagation is very simple: Starting with the input `x`, we repeatedly apply an affine function and a non-linearity to arrive at the output $a^L = \\sigma^L(W^{L-1}\\sigma^{L-1}( \\ldots \\sigma(W^1x + b^1) \\dots ) + b^{L-1})$. \n",
    "\n",
    "Mathematically, forward propagation comes down to a **composition of functions**. So for two layers, the output activation $a^2$ is the composition of the function $a^1 = f^1(x)$ and $a^2 = f^2(a^1)$ $\\Rightarrow$ $a^2 = f^2(f^1(x))$.\n",
    "\n",
    "**NOTE**: We save both the pre-activations (before the non-linearity) *and* the (post-)activations (after the nonlinearity). We'll need these for the backprop phase!\n",
    "\n",
    "In code it looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "BKxO41QtW0ka"
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-9-fe5acd516f4c>, line 17)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-9-fe5acd516f4c>\"\u001b[0;36m, line \u001b[0;32m17\u001b[0m\n\u001b[0;31m    return cache # Per-layer pre- and post-activations.\u001b[0m\n\u001b[0m         ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "# PSEUDOCODE:\n",
    "\n",
    "def fprop(x, weights, biases, per_layer_nonlinearities):\n",
    "\n",
    "    # Initialise the input. We pretend inputs are the first pre- and post-activations.\n",
    "    z = a = x\n",
    "    cache = [(z, a)] # We'll save z's and a's for the backprop phase.\n",
    "    \n",
    "    for W, b, act_fn in zip(weights, biases, per_layer_nonlinearities): \n",
    "        \n",
    "        z = np.dot(W, a) + b        # \"pre-activations\" / logits\n",
    "        a = act_fn(z)               # \"outputs\" / (post-)activations of the current layer\n",
    "        \n",
    "        # NOTE: We save both pre-activations and (post-)activations for the backwards phase!\n",
    "        cache.append((z, a)\n",
    "            \n",
    "    return cache # Per-layer pre- and post-activations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yeQ6_uE5asVK"
   },
   "source": [
    "### BACKPROP\n",
    "\n",
    "Given a loss or error function $E$ at the output (e.g. cross-entropy), we then need the derivative of the loss wrt each of the model parameters in order to train the network (decrease the loss). Mathematically, this comes down to the derivative of a composition of functions, and from calculus we know we have the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) for that: If $a = f(g(x))$, then $\\frac{\\partial a}{\\partial x} = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial x}$.\n",
    "\n",
    "In order to get the gradients on some intermediate parameter $\\theta^i = \\{W^i, b^i\\}$ in layer $i$, we just apply the chain rule over all $L$ 'layers' of this composition of functions (the neural network) to derive the intermediate gradients:\n",
    "\n",
    "\\begin{aligned}\n",
    "    \\frac{\\partial E}{\\partial \\theta^{(i)}} \n",
    "        &= \\underbrace{ \\frac{\\partial E}{\\partial z^{(L)}} \\frac{\\partial z^{(L)}}{\\partial z^{(L-1)}} \\ldots \\frac{\\partial z^{(i+2)}}{\\partial z^{(i+1)}}}_{\\triangleq \\delta^{(i+1)}} \\frac{\\partial z^{(i+1)}}{\\partial \\theta^{(i)}} \\\\\n",
    "    &= \\delta^{(i+1)} \\frac{\\partial z^{(i+1)}}{\\partial \\theta^{(i)}}\n",
    "\\end{aligned}\n",
    "\n",
    "We have glossed a little over the fact that we are dealing with matrices and vectors, for the sake of brevity. But please see these two great resources:\n",
    "* For a step-by-step walk-through of the mechanics of backpropagation, see [this great resource](http://cs224d.stanford.edu/lecture_notes/notes3.pdf).\n",
    "* For a great brush-up on the mechanics of vector calculus, see [this fantastic note](https://web.stanford.edu/class/cs224n/lecture_notes/cs224n-2017-gradient-notes.pdf) (for the same course).\n",
    "\n",
    "**NOTE**: \n",
    "* $E$ is the error/loss function at the output.\n",
    "* $W^{(i)}$ is an intermediate weight matrix mapping activations from layer $i$ to pre-activations $z^{(i+1)}$ at layer $i+1$.\n",
    "* The derivative with respect to the *inputs* of layer $i$ are called deltas and is defined as $\\delta^{(i)} \\triangleq \\frac{\\partial E}{\\partial z^{(i)}}$. \n",
    "    * $\\delta^i$ is a vector (the size of the layer $i$), and is the result of the deltas at the output $\\delta^L$ (dE/dlogits) multiplied by a product of [Jacobian matrices](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant). If you don't understand this statement, just think of $\\delta^i_j$ as the contribution that unit $j$ in layer $i$ made to the total error $E$. Also, read up on this in the resources linked above.\n",
    "    * **NB**: $z^{(i)}$ are the pre-activations (\"logits\" at the output layer)!\n",
    "* $\\frac{\\partial z^{(i+1)}}{\\partial W^{(i)}} = \\frac{\\partial (W^{(i)}a^i + b^i)}{\\partial W^{(i)}} = a^i$. \n",
    "* Likewise, $\\frac{\\partial z^{(i+1)}}{\\partial b^{(i)}} = 1$ (check this for yourself).\n",
    "\n",
    "\n",
    "**QUESTIONS**: \n",
    "1. What is the shape of $\\frac{\\partial E}{\\partial \\mathbb{W}^{(i)}}$?\n",
    "    * Convince yourself (now or later) that $\\frac{\\partial E}{\\partial W^{(i)}_{jk}} = \\delta^{(i+1)}_k {a^i_j}$ (a scalar), and therefore $\\frac{\\partial E}{\\partial \\mathbb{W}^{(i)}} = \\mathbb{\\delta}^{(i+1)} {\\mathbb{a}^i}^T$ (outer product of two vectors, therefore a matrix of the same shape as $W^i$). \n",
    "    * **In words: The gradient on weights $W^i$ at layer $i$ is the outer product of the deltas-vector from the layer above $\\delta^{(i+1)}$ and the activations vector from the layer below $a^i$.**\n",
    "\n",
    "2. What is the shape of $\\frac{\\partial E}{\\partial \\mathbb{b}^{(i)}}$?\n",
    "    * Convince yourself (now or later) that $\\frac{\\partial E}{\\partial \\mathbb{b}^{(i)}} = \\mathbb{\\delta}^{(i+1)}$ (a vector of the same length as $b^i$).\n",
    "    * **In words: The gradient on biases $b^i$ at layer $i$ is equal to the deltas at the layer above**.\n",
    "3. Make sure you understand why we save the `(z,a)`'s during fprop, and how we use them during backprop.\n",
    "\n",
    "\n",
    "Note that we need to compute the deltas at every layer, and $\\delta^i$ share all the terms of $\\delta^{(i+1)}$ (which we've already computed), except one. Backprop has one more trick up its sleeve, and that is to *reuse previously computed values to save computation* (this is called [dynamic programming](https://en.wikipedia.org/wiki/Dynamic_programming)). For the deltas, this comes down to computing $\\delta^{(i)}$ given $\\delta^{(i+1)}$. We won't show the full derivation (but we again use the chain rule, see the notes linked above) to arrive at:\n",
    "\n",
    "\\begin{align}\n",
    "    \\delta^{(i)} &= \\frac{\\partial E}{\\partial z^{(i)}} \\\\\n",
    "                             &= \\ldots \\\\\n",
    "                             &= \\underbrace{ \\left[ \\delta^{(i+1)} {\\mathbb{W}^{(i)}}^T \\right] }_\\textrm{Map the global delta 'backwards' through W up to layer $i$} \\circ \\underbrace{ \\sigma'(z^{(i)}) }_\\textrm{Correct for the local errors at this layer.}.\n",
    "\\end{align}\n",
    "\n",
    "**In words: The delta on the current layer $i$ is the delta on the layer above $\\delta^{(i+1)}$ multiplied by the transpose weights matrix between the two layers $W^i$ to yield a vector, scaled by the element-wise multiplication of $\\sigma'(z^{(i)})$ (the derivative of the non-linearity applied to the original pre-activations).**\n",
    "\n",
    "**QUESTIONs**: Look at the code below and make sure you understand (at least conceptually) how this works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "_tNoN3q4azpx"
   },
   "outputs": [],
   "source": [
    "# PSEUDOCODE\n",
    "\n",
    "def backprop(target, fprop_cache, weights):\n",
    "    \n",
    "    # Pop/remove the model prediction (last activation `a` we computed above) \n",
    "    # off the cache we created during the fprop phase.\n",
    "    (_, pred) = fprop_cache.pop() \n",
    "    \n",
    "    # Intialise delta^{L} (at the output layer) as dE/dz (cross-entropy).\n",
    "    delta_above = (target - pred)         \n",
    "    grads = []\n",
    "    \n",
    "    # Unroll backwards from the output:\n",
    "    for (z_below, a_below), W_between in reversed(zip(fprop_cache, weights)):\n",
    "        \n",
    "        # Compute dE/dW:\n",
    "        Wgrad = np.dot(delta_above, a_below.T) # Outer product\n",
    "        \n",
    "        # Compute dE/db:\n",
    "        bgrad = delta_above     \n",
    "        \n",
    "        # Save these:\n",
    "        grads.append((Wgrad, bgrad))\n",
    "        \n",
    "        # Update for the *next* iteration/layer. Note the elem-wise multiplication.\n",
    "        # Note the use of z_below, the preactivations in the layer below!\n",
    "        # delta^i = delta^{(i+1)}.(W^i)^T .* sigma'(z_i):\n",
    "        delta_above = np.dot(delta_above, W_between.T) * dsigmoid(z_below)\n",
    "    \n",
    "    grads.reverse()\n",
    "    return grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EvvygszheFJ1"
   },
   "source": [
    "Ppphhhhhheeeeewwwww, ok, come up for a breather. We know. This takes a while to wrap your head around. For now, just try to get the high-level picture! Depending on your background, this may well be the toughest part of the week. \n",
    "\n",
    "**BACKPROP SUMMARY NOTE**: It helps to have a good idea of what backprop does, and for this it helps to work through one example by hand (see the linked notes above). But you don't need to understand every detail above for the rest of this and the following lectures. The good news is that in practise you won't compute gradients by hand, and all of the above is done for us by `tf.gradients()` (or the similar function in other packages)!\n",
    "\n",
    "**BACKPROP FINAL QUESTION**:    Take a few minutes to explain the high-level details of the backprop algorithm to your neighbour. Try to understand how fprop is essentially just a composition of functions applied to the input, and backprop 'peels off' those compositions one by one by following the chain rule. In the process it avoids recomputation by saving activations during fprop, and computing deltas during backprop. Be sure to ask the tutors if you're stuck!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "id": "McPVC7ejezyZ"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n# PSEUDOCODE:\\n\\nbiases, weights = [b1, b2], [W1, W2]\\nx, y = ..., ...\\nnon_linearities = [relu, softmax]\\n\\nfprop_cache = fprop(x, weights, biases, non_linearities)\\ngrads = backprop(y, fprop_cache, weights)\\n'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"\n",
    "# PSEUDOCODE:\n",
    "\n",
    "biases, weights = [b1, b2], [W1, W2]\n",
    "x, y = ..., ...\n",
    "non_linearities = [relu, softmax]\n",
    "\n",
    "fprop_cache = fprop(x, weights, biases, non_linearities)\n",
    "grads = backprop(y, fprop_cache, weights)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lCu7TiQ7Lygf"
   },
   "source": [
    "## Training deep neural networks (10min)\n",
    "\n",
    "Now that we have a model with a loss and gradients, let's write a function to train it! This will be largely similar to the `train_tf_model()` function (same name, see below) from the previous practical. However, we are introducing several new concepts in this practical:\n",
    "\n",
    "* how to update parameters using different optimizers, \n",
    "* model complexity and how to match this to your data: \n",
    "    * recognizing overfitting \n",
    "    * adding regularization (L2, dropout)\n",
    "* knowing when to stop training: early stopping,\n",
    "\n",
    "As we go through these concepts, we will show how to use them in our training function.\n",
    "\n",
    "### Optimizers\n",
    "\n",
    "Training neural networks involves solving an optimization problem. **Stochastic gradient-based methods** are by far the most popular family of techniques that are being used for this. These methods evaluate the gradient of the loss on a small part of the data (called a **mini-batch**), and then propose a small change to the weights based on the current sample (and some maintain running averages over the previous steps), that reduces the loss, before moving on to another sample of the data:\n",
    "\n",
    "```\n",
    "step = optimizer(grad(cost), learning_rate, ...) \n",
    "new_weights = old_weights + step\n",
    "```\n",
    "\n",
    "The oldest algorithm is stochastic gradient descent, but there are many others (Adagrad, AdaDelta, RMSProp, ADAM, etc.). As a general rule of thumb, ADAM or SGD with Momentum tend to work quite well out of the box, but this depends on your model and your data! See these two great blog posts for more on this: \n",
    "\n",
    "* http://ruder.io/optimizing-gradient-descent/\n",
    "* https://medium.com/towards-data-science/types-of-optimization-algorithms-used-in-neural-networks-and-ways-to-optimize-gradient-95ae5d39529f\n",
    "\n",
    "In our code, we can select different optimization functions by passing in a different optimizer to `train_tf_model` (see the full list here: https://www.tensorflow.org/api_guides/python/train#Optimizers) as follows:\n",
    "\n",
    "```python\n",
    "\n",
    "optimizer = tf.train.RMSProp(...)\n",
    "results_tuple = train_tf_model(optimizer_fn=optimizer, ...)\n",
    "```\n",
    "\n",
    "### Training / validation / test splits\n",
    "\n",
    "When training supervised machine learning models, the goal is to build a model that will perform well on some test task with data that we'll obtain some time in the future. Unfortunately, until we solve time-travel, we don't yet have access to that data. So how do we train a model on data we have now, to perform well on some (unseen) data from the future? This, in a nutshell, is the statistical learning problem: we want to train a data on available data to **generalize** to unseen test data.\n",
    "\n",
    "The way we approach this is to take a dataset that we do have, and split it into a **training**, **validation**, and **test set** (split). Typically we will use ratios of 80/10/10 for example. We then train our model on the training set only, and use the validation set to make all kinds of decisions about architectural selection, hyperparameters, etc. When we're done, we evaluate our model and report its accuracy on the **test set only**. \n",
    "\n",
    "**NOTE**: We (typically) do not train on the validation set, and we **never** train on the test set. Think about why training on the test set might be a bad thing?\n",
    "\n",
    "### Model complexity, overfitting & regularization\n",
    "\n",
    "**Overfitting** occurs when improving the model's training loss (its performance on training data) comes at the expense of its **generalisation ability** (its performance on unseen test data). Generally it is a symptom of the **model complexity** increasing to fit the peculiarities (outliers) of the training data too accurately, causing it not to generalize well to new unseen (test) data. Overfitting is usually indicated when:\n",
    "\n",
    "* training & validation loss starts decreasing at different rates,\n",
    "* validation error starts increasing while training error still goes down,\n",
    "* training error reaches 0.\n",
    "\n",
    "**Underfitting** is the opposite: when a model cannot fit the training data well enough (usually a sign to train for longer or add more parameters to the model).\n",
    "\n",
    "You can think of complexity as how \"wiggly\" or \"wrinkly\" the decision boundary that the model can represent is. We can increase model complexity by adding more layers (i.e. more parameters). We can control or reduce the model complexity of an architecture using a family of techniques called **regularisers**. We've already encountered L2-regularisation (also called **weight-decay**), where we penalise the model for having very large weights. Another option is L1 regularization (which encourages sparsity of weights). Another very popular current technique is called **dropout** (we'll look at this in more detail in Practical 3). There are many others, but these two are the most popular.\n",
    "\n",
    "We will only be using L2 regularization in this practical. It can be set by passing a non-zero value to `l2_lambda` when constructing a `DNNClassifier` instance.\n",
    "\n",
    "### Early stopping\n",
    "\n",
    "Neural networks are nonlinear models and can have very complicated optimization landscapes. Stochastic gradient based methods for optimizing these loss functions do not proceed monotonically (i.e. does not just keep going up). Sometimes the loss can go down for a while before it goes up to reach a better part of parameter space later. How do we know when to stop training?\n",
    "\n",
    "**Early stopping** is one technique that helps with this. It is added to the training routine and means that we periodically evaluate the model's performance on the validation set (***Crucially! not the test set. Why?***). If the performance on the validation set starts becoming worse we know we have reached the point of overfitting (usually), so it usually makes sense to stop training and not waste any more computations. The `train_tf_model` function we build has an early-stopping feature that you can enable by passing the `stop_early=True` parameter. Have a look at the code to see how this is done. \n",
    "\n",
    "![early stopping](images/early_stopping.png \"Diagram illustrating early stopping using validation results\")\n",
    "\n",
    "For this practical, we just implemented the most basic idea of early stopping: stop training as soon as the model starts doing worse on validation data. However, there are different ways of implementing this idea. Two of the most popular are \n",
    "\n",
    "* \"early stopping with patience\": don't stop training immediately once validation accuracy degrades, but wait for P more epochs, and reset P if the model starts improving again within this timeframe,\n",
    "* training for T epochs, and simply selecting the best model based on validation score over the entire T epochs.\n",
    "\n",
    "**QUESTION**: What are the pros and cons of these different methods?\n",
    "\n",
    "\n",
    "\n",
    "## Wrapping these ideas into the training function (10-15min)\n",
    "\n",
    "The training function below implements all the ideas we discussed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "gCADIJteG816"
   },
   "outputs": [],
   "source": [
    "class MNISTFraction(object):\n",
    "    \"\"\"A helper class to extract only a fixed fraction of MNIST data.\"\"\"\n",
    "    def __init__(self, mnist, fraction):\n",
    "        self.mnist = mnist\n",
    "        self.num_images = int(mnist.num_examples * fraction)\n",
    "        self.image_data, self.label_data = mnist.images[:self.num_images], mnist.labels[:self.num_images]\n",
    "        self.start = 0\n",
    "        \n",
    "    def next_batch(self, batch_size):\n",
    "        start = self.start\n",
    "        end = min(start + batch_size, self.num_images)\n",
    "        self.start = 0 if end == self.num_images else end\n",
    "        return self.image_data[start:end], self.label_data[start:end]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "C52suFUDsami"
   },
   "outputs": [],
   "source": [
    "def train_tf_model(tf_model,                                     \n",
    "                   session,    # The active session.\n",
    "                   num_epochs,    # Max epochs/iterations to train for.\n",
    "                   batch_size=50,    # Number of examples per batch.\n",
    "                   keep_prob=1.0,    # (1. - dropout) probability, none by default.\n",
    "                   train_only_on_fraction=1.,    # Fraction of training data to use.\n",
    "                   optimizer_fn=None,    # The optimizer we want to use\n",
    "                   report_every=1, # Report training results every nr of epochs.\n",
    "                   eval_every=1,    # Evaluate on validation data every nr of epochs.\n",
    "                   stop_early=True,    # Use early stopping or not.\n",
    "                   verbose=True): \n",
    "\n",
    "    # Get the (symbolic) model input, output, loss and accuracy.\n",
    "    x, y = tf_model.x, tf_model.y\n",
    "    loss = tf_model.loss\n",
    "    accuracy = tf_model.accuracy()\n",
    "\n",
    "    # Compute the gradient of the loss with respect to the model parameters \n",
    "    # and create an op that will perform one parameter update using the specific\n",
    "    # optimizer's update rule in the direction of the gradients.\n",
    "    if optimizer_fn is None:\n",
    "        optimizer_fn = tf.train.AdamOptimizer()\n",
    "    optimizer_step = optimizer_fn.minimize(loss)\n",
    "\n",
    "    # Get the op which, when executed, will initialize the variables.\n",
    "    init = tf.global_variables_initializer()\n",
    "    # Actually initialize the variables (run the op).\n",
    "    session.run(init)\n",
    "\n",
    "    # Save the training loss and accuracies on training and validation data.\n",
    "    train_costs = []\n",
    "    train_accs = []\n",
    "    val_costs = []\n",
    "    val_accs = []\n",
    "\n",
    "    if train_only_on_fraction < 1:\n",
    "        mnist_train_data = MNISTFraction(mnist.train, train_only_on_fraction)\n",
    "    else:\n",
    "        mnist_train_data = mnist.train\n",
    "    \n",
    "    prev_c_eval = 1000000\n",
    "    \n",
    "    # Main training cycle.\n",
    "    for epoch in range(num_epochs):\n",
    "\n",
    "        avg_cost = 0.\n",
    "        avg_acc = 0.\n",
    "        total_batch = int(train_only_on_fraction * mnist.train.num_examples / batch_size)\n",
    "\n",
    "        # Loop over all batches.\n",
    "        for i in range(total_batch):\n",
    "            batch_x, batch_y = mnist_train_data.next_batch(batch_size)\n",
    "                        \n",
    "            # Run optimization op (backprop) and cost op (to get loss value),\n",
    "            # and compute the accuracy of the model.\n",
    "            feed_dict = {x: batch_x, y: batch_y}\n",
    "            if keep_prob < 1.:\n",
    "                feed_dict[\"keep_prob:0\"] = keep_prob\n",
    "                \n",
    "            _, c, a = session.run(\n",
    "                    [optimizer_step, loss, accuracy], feed_dict=feed_dict)\n",
    "                        \n",
    "            # Compute average loss/accuracy\n",
    "            avg_cost += c / total_batch\n",
    "            avg_acc += a / total_batch            \n",
    "        \n",
    "        train_costs.append((epoch, avg_cost))\n",
    "        train_accs.append((epoch, avg_acc))\n",
    "\n",
    "        # Display logs per epoch step\n",
    "        if epoch % report_every == 0 and verbose:\n",
    "            print \"Epoch:\", '%04d' % (epoch+1), \"Training cost=\", \\\n",
    "                        \"{:.9f}\".format(avg_cost)\n",
    "                \n",
    "        if epoch % eval_every == 0:\n",
    "            val_x, val_y = mnist.validation.images, mnist.validation.labels            \n",
    "            \n",
    "            feed_dict = {x : val_x, y : val_y}\n",
    "            if keep_prob < 1.:\n",
    "                feed_dict['keep_prob:0'] = 1.0\n",
    "                \n",
    "            c_eval, a_eval = session.run([loss, accuracy], feed_dict=feed_dict)\n",
    "            \n",
    "            if verbose:\n",
    "                print \"Epoch:\", '%04d' % (epoch+1), \"Validation acc=\", \\\n",
    "                            \"{:.9f}\".format(a_eval)\n",
    "                \n",
    "            if c_eval >= prev_c_eval and stop_early:\n",
    "                print \"Validation loss stopped improving, stopping training early after %d epochs!\" % (epoch + 1)\n",
    "                break\n",
    "                \n",
    "            prev_c_eval = c_eval\n",
    "                \n",
    "            val_costs.append((epoch, c_eval))\n",
    "            val_accs.append((epoch, a_eval))\n",
    "            \n",
    "    \n",
    "    print \"Optimization Finished!\"\n",
    "    return train_costs, train_accs, val_costs, val_accs\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     }
    },
    "colab_type": "code",
    "collapsed": true,
    "id": "AecVBHLe2USG"
   },
   "outputs": [],
   "source": [
    "# Helper functions to plot training progress.\n",
    "\n",
    "def my_plot(list_of_tuples):\n",
    "    \"\"\"Take a list of (epoch, value) and split these into lists of \n",
    "    epoch-only and value-only. Pass these to plot to make sure we\n",
    "    line up the values at the correct time-steps.\n",
    "    \"\"\"\n",
    "    plt.plot(*zip(*list_of_tuples))\n",
    "\n",
    "def plot_multi(values_lst, labels_lst, y_label, x_label='epoch'):\n",
    "    # Plot multiple curves.\n",
    "    assert len(values_lst) == len(labels_lst)\n",
    "    plt.subplot(2, 1, 2)\n",
    "    \n",
    "    for v in values_lst:\n",
    "        my_plot(v)\n",
    "    plt.legend(labels_lst, loc='upper left')\n",
    "    \n",
    "    plt.xlabel(x_label)\n",
    "    plt.ylabel(y_label)\n",
    "    plt.show()    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hyJG8AQPL_BH"
   },
   "source": [
    "## Wrapping everything together and verifying that it works (10min)\n",
    "\n",
    "Once we have a training function, it is usually a good idea to train on a small amount of your data first to verify that everything is indeed working. We can put all the pieces together to achieve this as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 289,
     "output_extras": [
      {
       "item_id": 4
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1198,
     "status": "ok",
     "timestamp": 1503071394643,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "guiSbTOIMAUm",
    "outputId": "32e73f8e-81de-4a8c-d7bb-1bd2286f8050"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 22.728044510\n",
      "Epoch: 0001 Validation acc= 0.075199999\n",
      "Epoch: 0002 Training cost= 21.128668196\n",
      "Epoch: 0003 Training cost= 19.666281579\n",
      "Epoch: 0003 Validation acc= 0.081200004\n",
      "Epoch: 0004 Training cost= 18.349199122\n",
      "Epoch: 0005 Training cost= 17.178852853\n",
      "Epoch: 0005 Validation acc= 0.090599999\n",
      "Epoch: 0006 Training cost= 16.147987530\n",
      "Epoch: 0007 Training cost= 15.243225618\n",
      "Epoch: 0007 Validation acc= 0.094599999\n",
      "Epoch: 0008 Training cost= 14.447481615\n",
      "Epoch: 0009 Training cost= 13.744003599\n",
      "Epoch: 0009 Validation acc= 0.101599999\n",
      "Epoch: 0010 Training cost= 13.119138050\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.1226\n"
     ]
    }
   ],
   "source": [
    "##### BUILD MODEL #####\n",
    "tf.reset_default_graph()    # Clear the graph.\n",
    "model = DNNClassifier()     # Choose model hyperparameters.\n",
    "\n",
    "with tf.Session() as sess:\n",
    "\n",
    "    ##### TRAIN MODEL #####\n",
    "\n",
    "    train_losses, train_accs, val_losses, val_accs = train_tf_model(\n",
    "            model,\n",
    "            session=sess,\n",
    "            num_epochs=10, \n",
    "            train_only_on_fraction=1e-1,\n",
    "            optimizer_fn=tf.train.GradientDescentOptimizer(learning_rate=1e-3),\n",
    "            report_every=1,\n",
    "            eval_every=2,\n",
    "            stop_early=False) \n",
    "\n",
    "    ##### EVALUATE MODEL ON TEST DATA #####\n",
    "\n",
    "    # Get the op which calculates model accuracy.\n",
    "    accuracy_op = model.accuracy()    # Get the symbolic accuracy operation\n",
    "    \n",
    "    # Connect the MNIST test images and labels to the model input/output\n",
    "    # placeholders, and compute the accuracy given the trained parameters.\n",
    "    accuracy = accuracy_op.eval(feed_dict = {model.x: mnist.test.images, \n",
    "                                             model.y: mnist.test.labels})\n",
    "    \n",
    "    print \"Accuracy on test set:\", accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kCJs9hNt9LFV"
   },
   "source": [
    "Instead of just training and checking that the loss goes down, it is usually a good idea to try to **overfit a small subset of your training data**. We will do this below by training a 1 hidden layer network on a subset of the MNIST training data, by setting the `train_only_on_fraction` training hyperparameter to 0.05 (i.e. 5%). We turn off early stopping for this. The following diagram illustrates the difference between under-fitting and over-fitting. Note that the diagram is idealised and it's not always this clear in practice!\n",
    "\n",
    "![overfitting](images/over_and_under_fitting.png \"Over and Under Fitting\")\n",
    "\n",
    "**QUESTION**: Why do we turn off early-stopping?\n",
    "\n",
    "In the rest of this practical, we will explore the effects of different model hyperparameters and different training choices, so let's wrap everything together to emphasize these different choices, and then train a simple model for a few epochs on 5% of the data to verify that everything works and that we can overfit a small portion of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 3109,
     "output_extras": [
      {
       "item_id": 14
      },
      {
       "item_id": 25
      },
      {
       "item_id": 26
      },
      {
       "item_id": 27
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6426,
     "status": "ok",
     "timestamp": 1503068187309,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "SvA0W9wTuFbD",
    "outputId": "c03d1f7d-550a-48d1-d94a-e337b03e902f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 86.671144555\n",
      "Epoch: 0001 Validation acc= 0.470400006\n",
      "Epoch: 0002 Training cost= 32.378755257\n",
      "Epoch: 0003 Training cost= 19.619923999\n",
      "Epoch: 0003 Validation acc= 0.705600023\n",
      "Epoch: 0004 Training cost= 14.065403336\n",
      "Epoch: 0005 Training cost= 10.729673871\n",
      "Epoch: 0005 Validation acc= 0.766799986\n",
      "Epoch: 0006 Training cost= 8.362369494\n",
      "Epoch: 0007 Training cost= 6.607117094\n",
      "Epoch: 0007 Validation acc= 0.790600002\n",
      "Epoch: 0008 Training cost= 5.190952872\n",
      "Epoch: 0009 Training cost= 4.023866670\n",
      "Epoch: 0009 Validation acc= 0.798600018\n",
      "Epoch: 0010 Training cost= 3.152098826\n",
      "Epoch: 0011 Training cost= 2.488907944\n",
      "Epoch: 0011 Validation acc= 0.804199994\n",
      "Epoch: 0012 Training cost= 1.918182273\n",
      "Epoch: 0013 Training cost= 1.529727884\n",
      "Epoch: 0013 Validation acc= 0.807200015\n",
      "Epoch: 0014 Training cost= 1.141758306\n",
      "Epoch: 0015 Training cost= 0.821120327\n",
      "Epoch: 0015 Validation acc= 0.809800029\n",
      "Epoch: 0016 Training cost= 0.596917208\n",
      "Epoch: 0017 Training cost= 0.448648420\n",
      "Epoch: 0017 Validation acc= 0.811800003\n",
      "Epoch: 0018 Training cost= 0.362648918\n",
      "Epoch: 0019 Training cost= 0.294025914\n",
      "Epoch: 0019 Validation acc= 0.815400004\n",
      "Epoch: 0020 Training cost= 0.211786659\n",
      "Epoch: 0021 Training cost= 0.199456693\n",
      "Epoch: 0021 Validation acc= 0.816600025\n",
      "Epoch: 0022 Training cost= 0.134907617\n",
      "Epoch: 0023 Training cost= 0.100976078\n",
      "Epoch: 0023 Validation acc= 0.816200018\n",
      "Epoch: 0024 Training cost= 0.082484495\n",
      "Epoch: 0025 Training cost= 0.110259895\n",
      "Epoch: 0025 Validation acc= 0.819199979\n",
      "Epoch: 0026 Training cost= 0.031071639\n",
      "Epoch: 0027 Training cost= 0.038540008\n",
      "Epoch: 0027 Validation acc= 0.820999980\n",
      "Epoch: 0028 Training cost= 0.009517248\n",
      "Epoch: 0029 Training cost= 0.010902141\n",
      "Epoch: 0029 Validation acc= 0.826399982\n",
      "Epoch: 0030 Training cost= 0.015957228\n",
      "Epoch: 0031 Training cost= 0.012406773\n",
      "Epoch: 0031 Validation acc= 0.827199996\n",
      "Epoch: 0032 Training cost= 0.002218569\n",
      "Epoch: 0033 Training cost= 0.001438031\n",
      "Epoch: 0033 Validation acc= 0.824999988\n",
      "Epoch: 0034 Training cost= 0.000026247\n",
      "Epoch: 0035 Training cost= 0.000015770\n",
      "Epoch: 0035 Validation acc= 0.824999988\n",
      "Epoch: 0036 Training cost= 0.000013471\n",
      "Epoch: 0037 Training cost= 0.000012097\n",
      "Epoch: 0037 Validation acc= 0.824800014\n",
      "Epoch: 0038 Training cost= 0.000011124\n",
      "Epoch: 0039 Training cost= 0.000010370\n",
      "Epoch: 0039 Validation acc= 0.824800014\n",
      "Epoch: 0040 Training cost= 0.000009756\n",
      "Epoch: 0041 Training cost= 0.000009236\n",
      "Epoch: 0041 Validation acc= 0.824999988\n",
      "Epoch: 0042 Training cost= 0.000008785\n",
      "Epoch: 0043 Training cost= 0.000008387\n",
      "Epoch: 0043 Validation acc= 0.824999988\n",
      "Epoch: 0044 Training cost= 0.000008030\n",
      "Epoch: 0045 Training cost= 0.000007706\n",
      "Epoch: 0045 Validation acc= 0.824999988\n",
      "Epoch: 0046 Training cost= 0.000007411\n",
      "Epoch: 0047 Training cost= 0.000007139\n",
      "Epoch: 0047 Validation acc= 0.824999988\n",
      "Epoch: 0048 Training cost= 0.000006887\n",
      "Epoch: 0049 Training cost= 0.000006652\n",
      "Epoch: 0049 Validation acc= 0.824999988\n",
      "Epoch: 0050 Training cost= 0.000006433\n",
      "Epoch: 0051 Training cost= 0.000006227\n",
      "Epoch: 0051 Validation acc= 0.824999988\n",
      "Epoch: 0052 Training cost= 0.000006033\n",
      "Epoch: 0053 Training cost= 0.000005850\n",
      "Epoch: 0053 Validation acc= 0.825200021\n",
      "Epoch: 0054 Training cost= 0.000005677\n",
      "Epoch: 0055 Training cost= 0.000005513\n",
      "Epoch: 0055 Validation acc= 0.825200021\n",
      "Epoch: 0056 Training cost= 0.000005356\n",
      "Epoch: 0057 Training cost= 0.000005207\n",
      "Epoch: 0057 Validation acc= 0.825200021\n",
      "Epoch: 0058 Training cost= 0.000005065\n",
      "Epoch: 0059 Training cost= 0.000004929\n",
      "Epoch: 0059 Validation acc= 0.825200021\n",
      "Epoch: 0060 Training cost= 0.000004799\n",
      "Epoch: 0061 Training cost= 0.000004675\n",
      "Epoch: 0061 Validation acc= 0.825200021\n",
      "Epoch: 0062 Training cost= 0.000004555\n",
      "Epoch: 0063 Training cost= 0.000004440\n",
      "Epoch: 0063 Validation acc= 0.825200021\n",
      "Epoch: 0064 Training cost= 0.000004330\n",
      "Epoch: 0065 Training cost= 0.000004224\n",
      "Epoch: 0065 Validation acc= 0.825200021\n",
      "Epoch: 0066 Training cost= 0.000004122\n",
      "Epoch: 0067 Training cost= 0.000004023\n",
      "Epoch: 0067 Validation acc= 0.825200021\n",
      "Epoch: 0068 Training cost= 0.000003928\n",
      "Epoch: 0069 Training cost= 0.000003836\n",
      "Epoch: 0069 Validation acc= 0.824999988\n",
      "Epoch: 0070 Training cost= 0.000003748\n",
      "Epoch: 0071 Training cost= 0.000003662\n",
      "Epoch: 0071 Validation acc= 0.824999988\n",
      "Epoch: 0072 Training cost= 0.000003579\n",
      "Epoch: 0073 Training cost= 0.000003499\n",
      "Epoch: 0073 Validation acc= 0.824999988\n",
      "Epoch: 0074 Training cost= 0.000003421\n",
      "Epoch: 0075 Training cost= 0.000003345\n",
      "Epoch: 0075 Validation acc= 0.825200021\n",
      "Epoch: 0076 Training cost= 0.000003272\n",
      "Epoch: 0077 Training cost= 0.000003202\n",
      "Epoch: 0077 Validation acc= 0.825200021\n",
      "Epoch: 0078 Training cost= 0.000003133\n",
      "Epoch: 0079 Training cost= 0.000003066\n",
      "Epoch: 0079 Validation acc= 0.825200021\n",
      "Epoch: 0080 Training cost= 0.000003001\n",
      "Epoch: 0081 Training cost= 0.000002938\n",
      "Epoch: 0081 Validation acc= 0.825399995\n",
      "Epoch: 0082 Training cost= 0.000002877\n",
      "Epoch: 0083 Training cost= 0.000002817\n",
      "Epoch: 0083 Validation acc= 0.825399995\n",
      "Epoch: 0084 Training cost= 0.000002759\n",
      "Epoch: 0085 Training cost= 0.000002702\n",
      "Epoch: 0085 Validation acc= 0.825399995\n",
      "Epoch: 0086 Training cost= 0.000002648\n",
      "Epoch: 0087 Training cost= 0.000002594\n",
      "Epoch: 0087 Validation acc= 0.824999988\n",
      "Epoch: 0088 Training cost= 0.000002542\n",
      "Epoch: 0089 Training cost= 0.000002491\n",
      "Epoch: 0089 Validation acc= 0.824999988\n",
      "Epoch: 0090 Training cost= 0.000002442\n",
      "Epoch: 0091 Training cost= 0.000002393\n",
      "Epoch: 0091 Validation acc= 0.824999988\n",
      "Epoch: 0092 Training cost= 0.000002346\n",
      "Epoch: 0093 Training cost= 0.000002299\n",
      "Epoch: 0093 Validation acc= 0.824800014\n",
      "Epoch: 0094 Training cost= 0.000002255\n",
      "Epoch: 0095 Training cost= 0.000002211\n",
      "Epoch: 0095 Validation acc= 0.824800014\n",
      "Epoch: 0096 Training cost= 0.000002168\n",
      "Epoch: 0097 Training cost= 0.000002126\n",
      "Epoch: 0097 Validation acc= 0.824800014\n",
      "Epoch: 0098 Training cost= 0.000002085\n",
      "Epoch: 0099 Training cost= 0.000002045\n",
      "Epoch: 0099 Validation acc= 0.824800014\n",
      "Epoch: 0100 Training cost= 0.000002006\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.8289\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAD2CAYAAACeCVCZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUHeV55/Hvc+/tvSW0ot1I7AiQhBEyZskwxvFgbAwz\nmMU2DrGNyUyYGJxkMjjLGSeTOYeTiZ3YiZcQ7Fi2CYRgOzA2XjHgYDBYLGY3YhGSQEtrb0m99zt/\n3JJoNS3Ukrq6ulvfzzn31K26VdXP1QvSr996661IKSFJkqThVSq6AEmSpEORIUySJKkAhjBJkqQC\nGMIkSZIKYAiTJEkqgCFMkiSpAIYwSZKkAhjCJEmSCmAIkyRJKkCl6AIGY8qUKWnu3LlFlyFJkrRP\njzzyyIaU0tR97TcqQtjcuXNZtmxZ0WVIkiTtU0S8Mpj9vBwpSZJUAEOYJElSAQxhkiRJBRgVY8IG\n0tXVxerVq2lvby+6lFzV19cze/Zsampqii5FkiQNoVEbwlavXs24ceOYO3cuEVF0OblIKbFx40ZW\nr17NvHnzii5HkiQNoVF7ObK9vZ3JkyeP2QAGEBFMnjx5zPf2SZJ0KBq1IQwYsgDW2t7Fyo076e1N\nQ3K+oTSWQ6YkSYeyUR3ChkpHdy9b2jrpTSMvhEmSpLHJEAaUst6m/QlhW7Zs4Ytf/OJ+/6zzzz+f\nLVu27PdxkiRpbDGEAaXsit/+XI3cWwjr7u5+0+PuuusuJkyYsD/lSZKkMWjU3h05lEpZCtufMWHX\nX389L774IosWLaKmpob6+nomTpzIc889x/PPP89FF13EqlWraG9v59prr+Xqq68GXn8E0/bt23n3\nu9/NWWedxQMPPMCsWbO44447aGhoyOU7SpKkkWVMhLA//39P88xr2w74+N6UaOvsob6mTDkLZPNn\njud/XXDiXo+54YYbeOqpp3j88ce59957ec973sNTTz21eyqJr371q0yaNIm2tjZOO+00Lr74YiZP\nnrzHOZYvX84tt9zCP/7jP3LppZfyrW99iyuuuOKAv4ckSRo9xkQIGwmWLFmyx1xen//85/nOd74D\nwKpVq1i+fPkbQti8efNYtGgRAKeeeiorVqwYtnolSVKxxkQIe7Meq8Ho6Orh1+taecukRiY01h7Q\nOZqamna/v/fee/nJT37Cgw8+SGNjI+ecc86Ac33V1dXtfl8ul2lrazugny1JkkYfB+bz+piwnv0Y\nEzZu3DhaW1sH/Gzr1q1MnDiRxsZGnnvuOX7xi18MSZ2SJGnsGBM9YQfrQO6OnDx5MmeeeSYnnXQS\nDQ0NTJs2bfdn5513Hl/+8pc54YQTOO644zj99NOHuGJJkjTaRRoFE5QuXrw4LVu2bI9tzz77LCec\ncMKQnD+lxJOvbmXa+Hqmja8fknMOpaH8rpIkKV8R8UhKafG+9vNyJNVHA5UinDFfkiQNm1xDWER8\nMiKejoinIuKWiKiPiEkR8eOIWJ4tJ+ZZw2CVAnp7i65CkiQdKnILYRExC/gEsDildBJQBi4Hrgfu\nTikdA9ydrRfOnjBJkjSc8r4cWQEaIqICNAKvARcCS7PPlwIX5VzDoJRKhjBJkjR8cgthKaVXgb8G\nVgJrgK0ppR8B01JKa7Ld1gLTBjo+Iq6OiGURsaylpSWvMner9oTl/mMkSZKAfC9HTqTa6zUPmAk0\nRcQez+RJ1VszB4w+KaUbU0qLU0qLp06dmleZu1XHhJnCJEnS8MjzcuQ7gZdTSi0ppS7g28AZwLqI\nmAGQLdfnWMOg5T0mrLm5ObdzS5Kk0SfPELYSOD0iGiMigHOBZ4E7gSuzfa4E7sixhkHzcqQkSRpO\nuc2Yn1J6KCJuBx4FuoHHgBuBZuC2iPgY8ApwaV417I9Sif3qCbv++uuZM2cO11xzDQCf/vSnqVQq\n3HPPPWzevJmuri7+8i//kgsvvDCvkiVJ0ig2NmbM//71sPbJg/oZHd09dPcmmmqzXDr9ZHj3DXvd\n/7HHHuO6667jvvvuA2D+/Pn88Ic/5LDDDmP8+PFs2LCB008/neXLlxMRNDc3s3379gOqzRnzJUka\nPQY7Y77PjsxEBCklEokg9rn/Kaecwvr163nttddoaWlh4sSJTJ8+nU9+8pP87Gc/o1Qq8eqrr7Ju\n3TqmT58+DN9AkiSNJmMjhL1Jj9VgbWltZ+3Wdk6aeRhR2ncIA7jkkku4/fbbWbt2LZdddhk333wz\nLS0tPPLII9TU1DB37lza29sPujZJkjT2jI0QNgRKUQ1evSlRGkRPGMBll13Gxz/+cTZs2MB9993H\nbbfdxuGHH05NTQ333HMPr7zySp4lS5KkUcwQlukbwgbrxBNPpLW1lVmzZjFjxgw+9KEPccEFF3Dy\nySezePFijj/++LzKlSRJo5whLFPOOr/2d5qKJ598/YaAKVOm8OCDDw6434EOypckSWNT3s+OHDV2\njQPrcbIwSZI0DAxhmXJ2OXI0TNkhSZJGv1EdwoYyMO26IbJnhGUwQ6EkSWPTqA1h9fX1bNy4cchC\nyoEMzM9bSomNGzdSX19fdCmSJGmIjdqB+bNnz2b16tW0tLQMyfl6ehPrtrbTuaGGprqR88dSX1/P\n7Nmziy5DkiQNsZGTNvZTTU0N8+bNG7LztbZ38d5P/4g/fc8JXLXoyCE7ryRJ0kBG7eXIodaYPTNy\nR0dPwZVIkqRDgSEsUy4FdZUSO7u6iy5FkiQdAgxhfTTVVdhpT5gkSRoGhrA+GmrK7Oi0J0ySJOXP\nENZHU12Ztk57wiRJUv4MYX001lbYYQiTJEnDwBDWR2NtmZ0dXo6UJEn5M4T10VhbYac9YZIkaRgY\nwvporC2z04H5kiRpGBjC+miqK9sTJkmShoUhrA8vR0qSpOFiCOujsbY6T1hKqehSJEnSGGcI66Ox\ntkJK0NHdW3QpkiRpjDOE9dFUVwZgh9NUSJKknBnC+mioqYYwx4VJkqS8GcL6aKqrAIYwSZKUP0NY\nH4212eVI5wqTJEk5M4T10Vib9YR12BMmSZLyZQjrY1dPmLPmS5KkvBnC+nBMmCRJGi6GsD4cEyZJ\nkoZLriEsIiZExO0R8VxEPBsRb4+ISRHx44hYni0n5lnD/tgVwtrsCZMkSTnLuyfsc8APUkrHAwuB\nZ4HrgbtTSscAd2frI8Kugfk7HJgvSZJyllsIi4jDgN8AvgKQUupMKW0BLgSWZrstBS7Kq4b9VS4F\ndZWSA/MlSVLu8uwJmwe0AP8UEY9FxE0R0QRMSymtyfZZC0wb6OCIuDoilkXEspaWlhzL3FNTXcWB\n+ZIkKXd5hrAK8FbgSymlU4Ad9Lv0mFJKQBro4JTSjSmlxSmlxVOnTs2xzD011JQdmC9JknKXZwhb\nDaxOKT2Urd9ONZSti4gZANlyfY417LemurKTtUqSpNzlFsJSSmuBVRFxXLbpXOAZ4E7gymzblcAd\nedVwIBprK+zsMoRJkqR8VXI+/+8BN0dELfAS8BGqwe+2iPgY8Apwac417JfG2jI7O7wcKUmS8pVr\nCEspPQ4sHuCjc/P8uQejsbbC5p1tRZchSZLGOGfM76eprkybA/MlSVLODGH9NNaW2eEUFZIkKWeG\nsH4aayuOCZMkSbkzhPXTVFtmZ1cP1SnMJEmS8mEI66ehtkJK0N7VW3QpkiRpDDOE9dNUVwbw+ZGS\nJClXhrB+Gmurs3b4/EhJkpQnQ1g/jbXVnjCfHylJkvJkCOtnVwizJ0ySJOXJENZPU112OdKHeEuS\npBwZwvppqPFypCRJyp8hrJ9dPWFtXo6UJEk5MoT10+TAfEmSNAwMYf007BqY75gwSZKUI0NYP84T\nJkmShoMhrJ9yKairlJwxX5Ik5coQNoCmuopjwiRJUq4MYQNorC17OVKSJOXKEDaAxtqyA/MlSVKu\nDGEDaKz1cqQkScqXIWwATXVlJ2uVJEm5MoQNoKGmwg5DmCRJytGgQlhEXBsR46PqKxHxaES8K+/i\nitJUV3aKCkmSlKvB9oR9NKW0DXgXMBH4MHBDblUNtxd/Ct/6OPT2AtUxYd4dKUmS8jTYEBbZ8nzg\nGymlp/tsG/22roYnb4PNLwO77o60J0ySJOVnsCHskYj4EdUQ9sOIGAf05lfWMJuxsLpc8yug+hDv\nnV09pJQKLEqSJI1lgw1hHwOuB05LKe0EaoCP5FbVcJt6ApRqdoewxroKKUF719jJmZIkaWQZbAh7\nO/DrlNKWiLgC+FNga35lDbNKLRx+PKx9AqhejgScK0ySJOVmsCHsS8DOiFgI/AHwIvD13KoqwoyF\nsOYJSInG2gqAs+ZLkqTcDDaEdafqAKkLgb9PKX0BGJdfWQWYvhB2boBtr9GU9YTt7LInTJIk5WOw\nIaw1Ij5FdWqK70VEieq4sLFjxoLqcu0TNOy6HGlPmCRJyslgQ9hlQAfV+cLWArOB/5tbVUWYdhIQ\nsOYJmuqyy5GOCZMkSTkZVAjLgtfNwGER8V6gPaU0qDFhEVGOiMci4rvZ+qSI+HFELM+WEw+4+qFU\n1wyTj4Y1v6KhJrsc6YStkiQpJ4N9bNGlwMPAJcClwEMR8f5B/oxrgWf7rF8P3J1SOga4O1sfGWYs\ngLX2hEmSpPwN9nLkn1CdI+zKlNJvAUuAP9vXQRExG3gPcFOfzRcCS7P3S4GLBl9uzmYshK2raO6p\nzr5hT5gkScrLYENYKaW0vs/6xkEe+7fAH7Hn7PrTUkprsvdrgWmDrCF/06uD85s2PwM4RYUkScrP\nYEPYDyLihxHx2xHx28D3gLve7IBs7Nj6lNIje9snm/ZiwGcDRcTVEbEsIpa1tLQMssyDlD2+qH7D\n04CTtUqSpPxUBrNTSul/RMTFwJnZphtTSt/Zx2FnAu+LiPOBemB8RHwTWBcRM1JKayJiBrB+oINT\nSjcCNwIsXrx4eB7i2DgJDptDae2vqK85ljYvR0qSpJwMtieMlNK3Ukq/n732FcBIKX0qpTQ7pTQX\nuBz4aUrpCuBO4MpstyuBOw6g7vxMXwBrnqCxtmJPmCRJys2b9oRFRCsDXy4MqlcTxx/Az7wBuC0i\nPga8QvVuy5FjxkL49V1Mru90TJgkScrNm4awlNKQPJoopXQvcG/2fiNw7lCcNxczFgCJk8qr2Nk5\nvehqJEnSGDXoy5GHjOwOyRNihZcjJUlSbgxh/Y2fCY1TOLb3JecJkyRJuTGE9RcBMxZwZPeLhjBJ\nkpQbQ9hApi9gZucKujraiq5EkiSNUYawgcxYSIVupnWsKLoSSZI0RhnCBpLNnH9k1wsFFyJJksYq\nQ9hAJs6jo9zEMeklenuHZ7J+SZJ0aDGEDaRUYmPzsZwYK2jvdnC+JEkaeoawvdh62AmcECvZ2d5Z\ndCmSJGkMMoTtxfZJJ9IYHXSuW150KZIkaQwyhO1Fx5STAOh57fGCK5EkSWORIWwvDj9yAR2phvZV\njxVdiiRJGoMMYXtx5LQJPM8cKuueLLoUSZI0BhnC9qJSLrGm4Vimbn8OktNUSJKkoWUIexNtk0+k\nOW0nbVlZdCmSJGmMMYS9iYYj3grAhuW/LLgSSZI01hjC3sTMYxfTk4ItLz1SdCmSJGmMMYS9iWNm\nT+WlNJNY90TRpUiSpDHGEPYm6iplVtYfy+RtzxZdiiRJGmMMYfuwY+J8JvZsJG1fX3QpkiRpDDGE\n7UPt7EUAjguTJElDyhC2D4cfuwSAjcsfLrgSSZI0lhjC9uHYubNZmaaS1jg4X5IkDR1D2D4011V4\nuXI047c4OF+SJA0dQ9ggbJs4n2ndr0L71qJLkSRJY4QhbBDKMxcC0LrisYIrkSRJY4UhbBAmH10d\nnN/i44skSdIQMYQNwrFHHcX6NIGuVx8vuhRJkjRGGMIGYWJTLS+Wj6R50zNFlyJJksYIQ9ggbRp/\nAtM7V0BXW9GlSJKkMcAQNljTF1Cml7bVTxZdiSRJGgMMYYM04ahTAVj3vDPnS5Kkg2cIG6Sjjz2J\nbamRjpVOUyFJkg5ebiEsIuZExD0R8UxEPB0R12bbJ0XEjyNiebacmFcNQ+nw8fU8X5pHw6aniy5F\nkiSNAXn2hHUDf5BSmg+cDlwTEfOB64G7U0rHAHdn6yNeRNDSfDzT2l6Enu6iy5EkSaNcbiEspbQm\npfRo9r4VeBaYBVwILM12WwpclFcNQ63n8JOpo5POdT5HUpIkHZxhGRMWEXOBU4CHgGkppTXZR2uB\naXs55uqIWBYRy1paWoajzH1qOuYsulKZrn/7PZ8jKUmSDkruISwimoFvAdellLb1/SyllIA00HEp\npRtTSotTSounTp2ad5mDMn/+yfxe97XUtzwJ3/jP0Lal6JIkSdIolWsIi4gaqgHs5pTSt7PN6yJi\nRvb5DGB9njUMpWnj66mceAHX9v4+ac0T8I2LoG1z0WVJkqRRKM+7IwP4CvBsSumzfT66E7gye38l\ncEdeNeThqrOP5Lsdi/jJgs/Auqfh6xfCzk1FlyVJkkaZPHvCzgQ+DLwjIh7PXucDNwC/GRHLgXdm\n66PGojkTWHzERP7i+Tn0XHozrH8Ovv4+g5gkSdoved4deX9KKVJKC1JKi7LXXSmljSmlc1NKx6SU\n3plSGnXp5aqz57FqUxs/7joZPvDP0PI8LH0f7NhYdGmSJGmUcMb8A/Cb86czZ1IDN/37y3D0O+GD\nt8LG5bD0AtixoejyJEnSKGAIOwDlUvDRM+ex7JXNPLZyMxz1Dvjgv8Cml+Br74XtI2NKDUmSNHIZ\nwg7QJYvnMK6+wlfuf7m64chz4EO3wZZXYOl7YfuouelTkiQVwBB2gJrrKnxwyVv4/lNrWb15Z3Xj\nvN+AD/0rbFkFX3sPbHq52CIlSdKIZQg7CFeeMReApQ+seH3j3LPgitth22vwd2+Fmy+B577n8yYl\nSdIeDGEHYeaEBs4/eQa3PryK7R19QtYRZ8Dv/gLO/kNY+yTc+kH4mxPh7v8Nm18prmBJkjRiGMIO\n0lVnzaO1o5sv3/vinh9MmAPv+BO47im4/BaYuQju/yx8biF847/AM3dAT1cxRUuSpMJVii5gtFs4\nZwLvP3U2X7j3Bc44ajJnHD1lzx3KFTj+/Opr62p47Jvw6Dfgtt+CpsNh0Qfh5Evg8PlQMhNLknSo\niOoztEe2xYsXp2XLlhVdxl7t7Ozmgr+7n23t3Xz/2rOZ0lz35gf09sALd8MjX4PnfwCpB+rGw6y3\nwuwlMGcJzDoVGicNS/2SJGnoRMQjKaXF+9zPEDY0nl2zjYu+8HOWzJvE0o8soVSKwR3YuhZeuhdW\nPQyrH64+jzL1Vj+bfEw1kM0+rbqcejyUyrl9B0mSdPAMYQX454dW8sffeZI/Ou84fvecow/sJB3b\n4bXHqoFs1S+ry53Z45Aq9TBuBoyfmS1nwLiZMG7669vGzYBK7dB9KUmStF8GG8IcEzaEPrBkDj9/\ncQOf+dHzLJk7icVzD+ByYl0zzDu7+gJIqToT/+plsPYJaF0D29bAq8vg2TXQ0/HGczROrr4aJkHD\nxD1fjf3W6w+rXgqtG294kyRpGNkTNsS2tXfx3s/fT3dPL3ddezYTGnMMNilB2+bqnGSta14PaK1r\noG1T9bO2zdC2pbrs3P7m5yvXQf14qBuXvbJwtmtbbXOfz/quN1f3q22uvq80VG9IkCTpEOTlyAI9\nsXoLF3/pARbMnsA/feQ0xtfXFF1SVXfH64GsbXM1qHW0Qvs26Nj12rXe+vp6x7bqts7t0DvISWfL\ntdUwVrPr1Qg19dmyoXppNbJxc2/4b7D/ekCUquPhogSRLUulPdejzzi8Pc7Z73xR6lNT4+s11Ta+\ncVulHso1UKnLvlNdNawaMiVJe2EIK9hdT67hE7c8xgkzxvP1jy5hYtMYuNSXEnS3V8etdWShrKM1\nW2+Fztbqsqsdutugqw26dmbLvq+d1fPsod+NDP0DVeqp3rDQmy13vfqv9z2u7zn7bu/tqdbR3Xbg\nfxZRqoaxSm0WymqqYbC061UZeD2iT927vtOu+vt8xzcE0b38OfX/bsOuyJ89xAr9cxxqY+m77Icx\n1YaDdSh+Zwbf1hffBNNOzLeWATgmrGDnnzyDukqJ/3bzo1x+4y/45lVvY+q4fUxdMdJFvN6z1Ty1\n6GoOXm9vNQx27ay+One+/r6rDTp3QE9ntQexpwO6O6vrb9jWUe0h7O3JXt3VQPWG9e5qoOzbo7f7\nfalfb98Ac8YN+AtTgb9EjYJf4AZvDH2XMdUu++MQ/N629b5V6vMrYwjYE5azn7+wgauWLmPGYfXc\n/PG3MeOwhqJLkiRJORpsT5hTtOfszKOn8PWPLWF9aweXfPlBVm7cWXRJkiRpBDCEDYPT5k7i5qve\nRmt7Nxf8/f1874k1RZckSZIKZggbJgvnTOCOa85k7pQmrvnnR/nDf/0V2zsGeaehJEkacwxhw2ju\nlCZu/69v5xPvOJpvP7qa8z/37zy6cnPRZUmSpAIYwoZZTbnE77/rOP7ld95OT2/iki8/yGd+9Gva\nOnuKLk2SJA0jQ1hBTps7ie9fdzYXLpzJ3/30Bd7xmXu54/FXGQ13q0qSpINnCCvQ+PoaPnvZIm77\nnbczubmWa299nIu/9ACPr9pSdGmSJClnhrARYMm8Sdx5zVn81cULWLmpjYu+8HOuvfUxlq9rLbo0\nSZKUEydrHWG2d3TzhXte4Gs/X0FbVw/vmj+N3/2PR7NozoSiS5MkSYPgsyNHuU07OvnaAytY+sAK\ntrZ1ccZRk/md/3AUZx09hXLpEH1WmCRJo4AhbIzY3tHNLQ+t5Kb7X2Ldtg6mja/jggUzuXDRLE6a\nNZ44JB9YK0nSyGUIG2M6unv48TPruOPx17j31+vp6kkcObWJCxbM5JzjprJg9gR7yCRJGgEMYWPY\nlp2dfP+ptdzx+Ks89PImUoLx9RXOOGoKZx4zhbOPnsIRkxvtJZMkqQCGsEPExu0d/PzFjdy/vIX7\nl2/gta3tAEwdV8dpcydy6hGTWHzERObPHE9N2ZthJUnKmyHsEJRS4uUNO/j5ixtZtmITy1Zs5tUt\nbQA01JQ5ZlozR01t5qipTRw1tZmjD2/miMlN1FYMZ5IkDZURHcIi4jzgc0AZuCmldMOb7W8IO3Br\nt7az7JVNPPrKFpavb+XF9dt395YBlEvBWyY1VoPZ4dWQNm9KE3MmNnL4uDpKjjOTJGm/jNgQFhFl\n4HngN4HVwC+BD6SUntnbMYawobWjo5uXN+zghfXbebEle63fwcsbdtDZ07t7v9pKidkTGpg9qZFZ\nE+qZ0lzX51XL5OZaxtfXMK6+hvqakmPQJEli8CGsMhzF9LMEeCGl9BJARNwKXAjsNYRpaDXVVThp\n1mGcNOuwPbZ39/SyenMbr2zaycpNO1m9aSerNlffP/PaNjbt6KB3L5m9phyMq6+hua5CfU2J+poy\n9ZUyddn72kqJunKJ2kr1VVcpUVcpU19TXdbVVLeVIihFEAERUIqgUipRU47dx9aWS1TKJSql6r6V\ncnVZLgWl7JhdxwL09CZ6U9q9BKo/s1Kirqa6rM3Gy+36ert+OSlF2BsoScpFESFsFrCqz/pq4G0F\n1KF+KuUSc6c0MXdK04Cf9/QmNu3oZMP2DjZs72DTjk5a27tpbe9mW3sXre1dbG/vpr2rl/buHtq7\nemht76altYPOnl46u7NX9r69q2evoW6kKZeCcgSlEgR7hrKURbddncoDfaW+R0RUz1FdQkT/Mw5w\n0N43SZL2w1+9fyHnnTS96DKAYkLYoETE1cDVAG95y1sKrkZQDSJTx9UxdVzdkJ2zq6eXjiyQtXf1\nkFI1zPSmarTpTYnunrR7v10hrqe3l55edi+7e3urxyTo3XV8tl7KAlS59HqvVkdXDx3dvdmrh87u\n6mXYXeFol96U6O1N9KRETy+7e9L6i35v+saq1DeWpWpI21Vb9f0bz5cGiHKj4B4aSRrxZk9sKLqE\n3YoIYa8Cc/qsz8627SGldCNwI1THhA1PaRpuNeUSNeUSzXUj9vcBSZJyUcTcBL8EjomIeRFRC1wO\n3FlAHZIkSYUZ9u6HlFJ3RPx34IdUp6j4akrp6eGuQ5IkqUiFXANKKd0F3FXEz5YkSRoJnCpdkiSp\nAIYwSZKkAhjCJEmSCmAIkyRJKkAhD/DeXxHRAryS84+ZAmzI+WfowNg2I5PtMnLZNiOT7TJyDXXb\nHJFSmrqvnUZFCBsOEbFsMA/b1PCzbUYm22Xksm1GJttl5CqqbbwcKUmSVABDmCRJUgEMYa+7segC\ntFe2zchku4xcts3IZLuMXIW0jWPCJEmSCmBPmCRJUgEMYUBEnBcRv46IFyLi+qLrOVRFxJyIuCci\nnomIpyPi2mz7pIj4cUQsz5YTi671UBQR5Yh4LCK+m63bLiNAREyIiNsj4rmIeDYi3m7bFC8iPpn9\nPfZURNwSEfW2SzEi4qsRsT4inuqzba9tERGfyvLAryPiP+VZ2yEfwiKiDHwBeDcwH/hARMwvtqpD\nVjfwByml+cDpwDVZW1wP3J1SOga4O1vX8LsWeLbPuu0yMnwO+EFK6XhgIdU2sm0KFBGzgE8Ai1NK\nJwFl4HJsl6J8DTiv37YB2yL7N+dy4MTsmC9mOSEXh3wIA5YAL6SUXkopdQK3AhcWXNMhKaW0JqX0\naPa+leo/JrOotsfSbLelwEXFVHjoiojZwHuAm/pstl0KFhGHAb8BfAUgpdSZUtqCbTMSVICGiKgA\njcBr2C6FSCn9DNjUb/Pe2uJC4NaUUkdK6WXgBao5IReGsOo/8qv6rK/OtqlAETEXOAV4CJiWUlqT\nfbQWmFZQWYeyvwX+COjts812Kd48oAX4p+xS8U0R0YRtU6iU0qvAXwMrgTXA1pTSj7BdRpK9tcWw\nZgJDmEaciGgGvgVcl1La1vezVL2d11t6h1FEvBdYn1J6ZG/72C6FqQBvBb6UUjoF2EG/S1y2zfDL\nxhddSDUkzwSaIuKKvvvYLiNHkW1hCINXgTl91mdn21SAiKihGsBuTil9O9u8LiJmZJ/PANYXVd8h\n6kzgfRGxgurl+ndExDexXUaC1cDqlNJD2frtVEOZbVOsdwIvp5RaUkpdwLeBM7BdRpK9tcWwZgJD\nGPwSOCayXU2OAAAC7ElEQVQi5kVELdUBeXcWXNMhKSKC6tiWZ1NKn+3z0Z3Aldn7K4E7hru2Q1lK\n6VMppdkppblU///4aUrpCmyXwqWU1gKrIuK4bNO5wDPYNkVbCZweEY3Z32vnUh3jaruMHHtrizuB\nyyOiLiLmAccAD+dVhJO1AhFxPtUxL2Xgqyml/1NwSYekiDgL+HfgSV4fe/THVMeF3Qa8BXgFuDSl\n1H+QpYZBRJwD/GFK6b0RMRnbpXARsYjqDRO1wEvAR6j+gm3bFCgi/hy4jOpd348BVwHN2C7DLiJu\nAc4BpgDrgP8F/Bt7aYuI+BPgo1Tb7rqU0vdzq80QJkmSNPy8HClJklQAQ5gkSVIBDGGSJEkFMIRJ\nkiQVwBAmSZJUAEOYJO1FRJwTEd8tug5JY5MhTJIkqQCGMEmjXkRcEREPR8TjEfEPEVGOiO0R8TcR\n8XRE3B0RU7N9F0XELyLiiYj4TvacPyLi6Ij4SUT8KiIejYijstM3R8TtEfFcRNyczYAuSQfNECZp\nVIuIE6jOTH5mSmkR0AN8CGgClqWUTgTuozpLNsDXgf+ZUlpA9ekMu7bfDHwhpbSQ6nP+1mTbTwGu\nA+YDR1J9lqYkHbRK0QVI0kE6FzgV+GXWSdVA9WG8vcC/ZPt8E/h2RBwGTEgp3ZdtXwr8a0SMA2al\nlL4DkFJqB8jO93BKaXW2/jgwF7g//68laawzhEka7QJYmlL61B4bI/6s334H+oy2jj7ve/DvTUlD\nxMuRkka7u4H3R8ThABExKSKOoPr32/uzfT4I3J9S2gpsjoizs+0fBu5LKbUCqyPiouwcdRHROKzf\nQtIhx9/oJI1qKaVnIuJPgR9FRAnoAq4BdgBLss/WUx03BnAl8OUsZL0EfCTb/mHgHyLiL7JzXDKM\nX0PSIShSOtAeekkauSJie0qpueg6JGlvvBwpSZJUAHvCJEmSCmBPmCRJUgEMYZIkSQUwhEmSJBXA\nECZJklQAQ5gkSVIBDGGSJEkF+P+PUN5rAXcA6gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f16873f4810>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAD2CAYAAAB4IJvdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XXWd//HXJzd7lyTdt7QpULqxFAm1LI4gMJbNiopF\nBQXF/hhBwd9vZuhsojP+RmacGQdHtCJTQGURBbTDr4CACChFmkKBrrR0IUmbJm2TNG325PP74562\ntyFJb0vOPTfJ+/l43Mc92z3nc3va9J3v+Z7vMXdHRERERKKTEXUBIiIiIoOdApmIiIhIxBTIRERE\nRCKmQCYiIiISMQUyERERkYgpkImIiIhETIFMREREJGIKZCIiIiIRUyATERERiVhm1AUcq1GjRnlJ\nSUnUZYiIiIgc1apVq3a7++ijbdfvAllJSQllZWVRlyEiIiJyVGa2PZntdMlSREREJGIKZCIiIiIR\nUyATERERiVi/60PWnba2NioqKmhubo66lNDl5uYyadIksrKyoi5FRERE+khogczMlgKXA9Xufko3\n6w24E7gUaASuc/fXjudYFRUVDBs2jJKSEuK7HZjcnT179lBRUcHUqVOjLkdERET6SJiXLO8D5vey\n/hJgWvBaBPzoeA/U3NzMyJEjB3QYAzAzRo4cOShaAkVERAaT0FrI3P1FMyvpZZMFwE/d3YFXzKzQ\nzMa7+87jOd5AD2MHDZbvOZi4O/VNbVTta6amoYXG1g6a2zpoaeukub2DptYOmoPp5rb4dEtbBx51\n4SIi/dw5J47kqtLiqMsAou1DNhEoT5ivCJa9J5CZ2SLirWhMnjw5JcWJdOXuvFNzgBVb9vDKO3v4\n09Y91Da2kZlh8Vcsg6yYkRXLIDcrFrwyyM2MkZWZQdco3dLewa59Leysb6K5rfOox8+KGbmZMXKy\nYuRkZhDLUDgXEXk/iovyoi7hkH7Rqd/d7wbuBigtLU27hoG6ujoefPBBvvKVrxzT5y699FIefPBB\nCgsLQ6pMjkV7RyfL11Txy7Jy9re0k5URDz2ZMcPMWL9zHzUNLQCMG57Lh6aNZkJhLu0dTnun097R\nSXun09LeebglK2jVamzqeM/xsmPG7AnDuXDGGMYV5DK+II/Rw3LIz04IcweDXWYGmTHdFC0iMlBF\nGcgqgcR2wknBsn6nrq6OH/7wh+8JZO3t7WRm9vxHvHz58rBLG1T2NbexprKetZX72LrnAG3tnXR0\nOm1BWIplGKdNKuCskhHMnlBAdmY84OxvaefhV9/l3j9uo7KuiZKR+RSPyKe9w+nodJrbO+jodM49\ncSTzToi/pozM1+VjERHpM1EGsmXAzWb2MPBBoP54+49FbfHixbzzzjvMmTOHrKwscnNzKSoqYsOG\nDbz99tt8/OMfp7y8nObmZm655RYWLVoEHH4M1P79+7nkkks477zzePnll5k4cSK/+c1vyMtLn6bU\nMLg7FbVNvFFRx/7m9vesj2UYedkxcjMPtxh1dDq1jW3UN7VS29hGXWMb5bWNrK2sZ9uexkOfLcrP\nIiczRmZwCTGWYTS1dvDEm/G/YrlZGcwpLmTyiHyefKuKhpZ25k4dwTc/NpsLZ4whQ5cDRUQkhcIc\n9uIh4HxglJlVALcDWQDuvgRYTnzIi83Eh724vi+O+63/Wcu6Hfv6YleHzJownNuvmN3j+jvuuIM1\na9awevVqfv/733PZZZexZs2aQ0NTLF26lBEjRtDU1MRZZ53FJz/5SUaOHHnEPjZt2sRDDz3ET37y\nEz796U/z6KOPcs011/Tp94ja7v0trN+5jzfK61hdXsfr79ax50Dr+9pnVswYOzyXUyYUcFVpMadM\nLGD2hOGMGprT7fbVDc2s2lbLym21lG3fy7I3dnDRzLF8+UMncHqxLh2LiEg0wrzL8jNHWe/ATWEd\nP0pz5849Ypyw73//+zz++OMAlJeXs2nTpvcEsqlTpzJnzhwAzjzzTLZt25ayesPQ2NrOM+t2sXbH\nPtbv3Mf6nQ3s3t9yaP2Jo4dwwYwxzCkuZE5xISOHZr9nH+0dTlPb4TsLm9s6yDCjMD+LoiHZFOZl\nkZ8dO6ZLh2OG5XLJqeO55NTxffI9RURE+kK/6NR/LHpryUqVIUOGHJr+/e9/z7PPPsuKFSvIz8/n\n/PPP73YcsZycwy06sViMpqamlNTa18r3NvKzV7bz8Kvvsq+5nezMDKaPHcYF00czY/xwZowbxikT\nCyjI05MGREREDhpwgSwKw4YNo6Ghodt19fX1FBUVkZ+fz4YNG3jllVdSXF343J0/bd3LvX/cyjPr\ndmFmzD9lHNedU8IZxYW6O1BEROQoFMj6wMiRIzn33HM55ZRTyMvLY+zYsYfWzZ8/nyVLljBz5kym\nT5/OvHnzIqy0b+1vaefx1yv5+YrtbNzVQFF+Fjd++ESuPXsK4wsG9g0JIiIifcniXbn6j9LSUi8r\nKzti2fr165k5c2ZEFaVe1N/37V0N/GzFdh57rYIDrR2cMnE4186bwoI5E8nNikVWl4iISLoxs1Xu\nXnq07dRCJklpbe/kt+uq+NmK7fxp616yMzO4/LTxXDtvCnOKCzUml4iIyPugQCa92lnfxEN/epeH\nVpZT09BC8Yg8Fl8yg0+XFjNiyHvvjBQREZFjp0Am3dq+5wDfe+Zt/ufNnXS685HpY7hm3hT+7OTR\neoaiiIhIH1MgkyNU1Tfz/d9t4pGV5WTGjC+eW8Lnzy6heER+1KWJiIgMWApkgrtTWdfE/S9v4/4V\n23F3PvfBydx0wUmMGZ4bdXkiIiIDngLZINTU2hF/dFF5LavfjT/GqLqhhQyDT3xgErdcOE0tYiIi\nIimkQBaBoUOHsn///kiO/bsNu7jt0beoaYg/xmjqqCGce9IozphcyIemjWbqqCFH2YOIiIj0NQWy\nQaKhuY1vP7GeX5SVM2PcML5z5amcOaWIIt0pKSIiEjkFsj6wePFiiouLuemm+LPSv/nNb5KZmcnz\nzz9PbW0tbW1tfPvb32bBggWR1PfKlj385S/fYEddE39x/oncetE0cjI1gKuIiEi6GHiB7MnFUPVW\n3+5z3KlwyR09rl64cCG33nrroUD2yCOP8PTTT/O1r32N4cOHs3v3bubNm8fHPvaxlA2g2tnprK6o\n41erKnjo1XeZMiKfX954NmdOGZGS44uIiEjyBl4gi8AZZ5xBdXU1O3bsoKamhqKiIsaNG8fXv/51\nXnzxRTIyMqisrGTXrl2MGzcutDo6Op2ybXt5ck0VT62pompfM1kx49p5U1h8yQzys3W6RURE0tHA\n+x+6l5asMF111VX86le/oqqqioULF/LAAw9QU1PDqlWryMrKoqSkhObm5tCOv6FqH1+6r4zKuiZy\nMjP48Mmjue3U6XxkxlgK8rJCO66IiIi8f6EGMjObD9wJxIB73P2OLuuLgKXAiUAz8EV3XxNmTWFZ\nuHAhX/7yl9m9ezcvvPACjzzyCGPGjCErK4vnn3+e7du3h3bsN8rr+PzSV8nLivGDz57BBdPHMCRn\n4GVtERGRgSq0/7XNLAbcBVwMVAArzWyZu69L2OxvgdXufqWZzQi2vzCsmsI0e/ZsGhoamDhxIuPH\nj+dzn/scV1xxBaeeeiqlpaXMmDEjlOOu3LaX6+9dSdGQLB68YZ7GDxMREemHwmxGmQtsdvctAGb2\nMLAASAxks4A7ANx9g5mVmNlYd98VYl2heeutwzcTjBo1ihUrVnS7XV+NQfbSphq+/NMyJhTm8cAN\nH2R8QV6f7FdERERSKyPEfU8EyhPmK4Jlid4APgFgZnOBKcCkEGsaMJ5dt4sv3VdGycgh/GLR2Qpj\nIiIi/ViYgSwZdwCFZrYa+CrwOtDRdSMzW2RmZWZWVlNTk+oa005jazs3/nwVM8cP4+FF8xg9LCfq\nkkREROR9CPOSZSVQnDA/KVh2iLvvA64HsPgAXVuBLV135O53A3cDlJaWencHc/eUjfEVFXenel8z\new608sETRrDkmjMZlqs7KEVERPq7MFvIVgLTzGyqmWUDVwPLEjcws8JgHcANwItBSDsmubm57Nmz\nB/dus9qA4O5U1DZSUVVNq2dy73VzFcZEREQGiNBayNy93cxuBp4mPuzFUndfa2Y3BuuXADOB+83M\ngbXAl47nWJMmTaKiooKBejmz053aA600tnVgsWz+/KyZZGdGfbVZRERE+or1t1al0tJSLysri7qM\nlKlpaOFL969kTWU931pwCtfOmxJ1SSIiIpIkM1vl7qVH206jh6axzdUNXHfvSnbvb+HH15Zy8ayx\nUZckIiIiIVAgS1Mvv7ObG3+2iuzMGL9YdDanFxdGXZKIiIiERIEsDT26qoLFj73JlJFDuPe6szT6\nvoiIyACnQJZG3J3/fHYTdz63iXNOHMmPrjlTDwYXEREZBBTI0oS7809PrGfpH7fyyQ9M4jufOFV3\nUoqIiAwSCmRp4kcvvMPSP27lunNKuP2KWQN+kFsRERE5TE0waeCRleX861MbWTBnAt+4XGFMRERk\nsFEgi9gz63ax+LE3+dC0UXz3U6eTkaEwJiIiMtgokEXo1a17ufnB1zh1YgFLrjlTfcZEREQGKSWA\niGyo2scN969kYmEeS687iyE56s4nIiIyWCmQRaCqvpnr711JXnaMn35pLiOH5kRdkoiIiERIgSzF\n9re088X7VrKvqY2l153FpCIN+ioiIjLY6TpZCrV3dPLVB19j464G7vlCKbMnFERdkrxf7nCgBmq3\nwd6t8femWuhohc426GgP3tugsz3+3tEanwbIHgo5w+Kv3OHB9PDDy3KGQfYQsFjyNZmBZQAWnyaY\nPzSd+J7E98PBOxOmg/kjpgcST/iuHPldu/5ZSBK6+ft36O8nh6f7lHf5+9rD390wzmG3/2Y6g0OF\n/G/m4J/twVdGLJiOJcxbSH/mJPGzJyO5nzthySuC/BHRHf8oFMhSxN25fdlant9Ywz9feSoXTB8T\ndUnSG3doa4K2RmjdD60HYN+Ow6Er8dV24MjP5hRALBMysiCWBRmZwXvWkcsB6iugZR+0NMTfDwY1\nERHpWxd9C867NeoqeqRAliI/eWkLD/zpXW788Il89oOToy6nf3GPB5X2ZujsOPK3vkPTXX7zco8H\nnMa90LQXGmuD971HvjfVxsNW4qutMf7e02/PmXlQVAIjpsIJH45PF5VA0VQonAxZucf/Pdtb4uGs\ntSEIaQ3Qsv8YfqPuoSUn8bfyY20ZsKBnQ6+/8Q6w4Vq6/a5089v/APvefa3blik44u9n2K1FXc9X\nKlprEo/bbYsRhPNvxqGzM/iz7Yi/d3YkzPvh+TCOfdSfPRG3po87LdrjH4UCWQo8tWYn/7x8A5ed\nNp6//uj0qMtJvfYW2F8dv7R3oCaYrob9NYeXtR6Ib9feDB0twXTL4WVJBQg7HNC8s5fWJoPcgnjT\ndV5R/LJg/qj4pcHsfMgacng6eyhk5cenh42Ph66hY8L5QW4WD3NZucDovt+/iIikrVADmZnNB+4E\nYsA97n5Hl/UFwM+ByUEt/+bu94ZZU6p1dMafUXnKxOH8+1X9cODXtmZoroPmemgK3pvr4y04rQfi\nrTetwaul63sDHNgNLfXd7zt7GAwZFQ84uQWQmQuZ2fH3WPaR85k5EMuJB65Dv/Ud/A2wm98ILSMI\nXCPe+55XGN+PiIhImggtkJlZDLgLuBioAFaa2TJ3X5ew2U3AOne/wsxGAxvN7AF3bw2rrlR7bv0u\nKuua+PvLZpKbFVEIcI+Hp/dcsqt973xz/ZEBrKPl6PvPGgI5Q4NWpaCT+tAxMOIEGDIaho6Ovw8Z\nE18+JJjP1h2mIiIiEG4L2Vxgs7tvATCzh4EFQGIgc2CYxR/eOBTYCwyoXs0/XbGd8QW5XDxrbLgH\nat4HtVvjnc73bjk8XbsN9u+K39nXk5wCyC+C3MJ469HwCfEWq7zC+HtuYcJ8MJ09NB7CsvLV2iQi\nIvI+hRnIJgLlCfMVwAe7bPMDYBmwAxgGLHSPutdf39lc3cAfNu/mrz46ncxYHwz55g4NO2HHaqh6\nC/a+cziANe4+ctsho+P9naacA8PGdX/p7mAfqoN3/ImIiEgkou7U/1FgNfAR4ETgGTN7yd33JW5k\nZouARQCTJ/efOxR/umI72bEMFp5VfOwfdod9lfHwtfMN2Lk6Pn2gOtjAYPjE+J1+My6NXx4smhp/\nHzE1ftlQRERE+oUwA1klkJhEJgXLEl0P3OHuDmw2s63ADODVxI3c/W7gboDS0tJ+MRpjQ3Mbj66q\n4PLTxjMq2Ucj7d0Cqx+EHa/Hw9fBVi/LgNEz4KSLYMIcGH86jDs13mdLRERE+r0wA9lKYJqZTSUe\nxK4GPttlm3eBC4GXzGwsMB3YEmJNKfPYa5UcaO3gC+eUHH3j1gPw0r/Dy/8Vv0NwzEw4ef7h8DX2\nFHWAFxERGcBCC2Tu3m5mNwNPEx/2Yqm7rzWzG4P1S4B/Au4zs7eIj5J3m7vv7nGn/URnp3P/im2c\nXlzI6cWFPW/oDmsfg9/+Q/zy5GlXw0XfhOHjU1SpiIiIpINQ+5C5+3JgeZdlSxKmdwB/HmYNUfjj\nO7vZUnOA7y08veeNdq2FJ2+DbS/FRw/+1FKYPC91RYqIiEjaiLpT/4B0/8vbGTkkm0tP7aalq6kW\nnv8OrLwn/jDpy78HH/iCho4QEREZxBTI+lj53kae27CLm84/iZzMhJDV2Qmv/wye+1Y8lJV+ES74\nu7R+8ryIiIikhgJZH/v5K9vJMONz8xKG56h6C5Z9NX735OSz4ZJ/hfHp/ZBTERERSR0Fsj7U0ek8\n+lolF80cw/iCvPjCpjp44Kr4MxY/cQ+c+qlwHkwtIiIi/ZYCWR967d1adu9v4fLTJhxe+Nu/h/3V\ncMOzMPED0RUnIiIiaasPnucjBz35VhXZsQwumDEmvmDzc/F+Y+d+TWFMREREeqRA1kfcnafXVvGh\naaMYmpMJLQ3wP7fAqJPhw4ujLk9ERETSmAJZH1lTuY/KuiY+esq4+IJnvgH1FbDgLsjKjbY4ERER\nSWsKZH3kqbU7iWUYF88cC1tegLKlcPZNUDw36tJEREQkzSmQ9ZGn1lQx74QRFGW2xoe4GHFCfJwx\nERERkaNQIOsDm6sbeKfmAPNnj4Pn/hHq3o1fqtQDwUVERCQJCmR94Kk1VQBcVrANXv0xzF0EU86J\ntigRERHpNxTI+sBTa6uYV5zLiGf/NxROgQu/EXVJIiIi0o9oYNj3qXxvI2sq9/GbaU9C+Tvw+WWQ\nMzTqskRERKQfSaqFzMweM7PLzEwtal08vbaKM2wTp1U8AGdeDyd8OOqSREREpJ9JNmD9EPgssMnM\n7jCz6SHW1K8899a73Jn3E2zYBLj4H6MuR0RERPqhpAKZuz/r7p8DPgBsA541s5fN7HozywqzwHRW\nva+ZD+34byZ3VsDH7oTc4VGXJCIiIv1Q0pcgzWwkcB1wA/A6cCfxgPZML5+Zb2YbzWyzmb3n+UFm\n9ldmtjp4rTGzDjMbcczfIiJlK37HotgT1M9YCCddFHU5IiIi0k8l1anfzB4HpgM/A65w953Bql+Y\nWVkPn4kBdwEXAxXASjNb5u7rDm7j7t8FvhtsfwXwdXffe7xfJtVmlH2DuoxCRi74l6hLERERkX4s\n2Ray77v7LHf/TkIYA8DdS3v4zFxgs7tvcfdW4GFgQS/H+AzwUJL1RK79QC0ntG3ijQkLsbyiqMsR\nERGRfizZQDbLzAoPzphZkZl95SifmQiUJ8xXBMvew8zygfnAoz2sX2RmZWZWVlNTk2TJ4areshqA\n7AmnRFyJiIiI9HfJBrIvu3vdwRl3rwW+3Id1XAH8safLle5+t7uXunvp6NGj+/Cwx69u25sAFE09\nPeJKREREpL9LNpDFzMwOzgT9w7KP8plKoDhhflKwrDtX048uVwJ0VK1jv+dSXKIRQEREROT9STaQ\nPUW8A/+FZnYh8fD01FE+sxKYZmZTzSybeOha1nUjMysAPgz8Jvmyo5dXt5EtGZMpyD9aLhURERHp\nXbKPTroN+F/AXwTzzwD39PYBd283s5uBp4EYsNTd15rZjcH6JcGmVwK/dfcDx1p8lEY3bmFV3tlR\nlyEiIiIDQFKBzN07gR8Fr6S5+3JgeZdlS7rM3wfcdyz7jZrvr6bA62kqOjnqUkRERGQASHYcsmnA\nd4BZQO7B5e5+Qkh1pbW6bW9QBGSOnR11KSIiIjIAJNuH7F7irWPtwAXAT4Gfh1VUuqvd9gYAhSVz\nIq5EREREBoJkA1meuz8HmLtvd/dvApeFV1Z666haxx4fxuTJk6MuRURERAaAZDv1t5hZBrAp6Khf\nCQwNr6z0llu7kXco5qyCvKhLERERkQEg2RayW4B84GvAmcA1wBfCKiqtuTOqcQvVeSeQMDSbiIiI\nyHE7agtZMAjsQnf/S2A/cH3oVaWz+gryvJGmQg0IKyIiIn3jqC1k7t4BnJeCWvqFpso1AGSMmxVx\nJSIiIjJQJNuH7HUzWwb8Ejg0gKu7PxZKVWmsdtsb5AEFk0+LuhQREREZIJINZLnAHuAjCcscGHSB\nrG3nWnb6CEomTYi6FBERERkgkh2pf3D3G0uQU7uRt30SZ4/Mj7oUERERGSCSHan/XuItYkdw9y/2\neUXprLODEY1bqcq5lKxYsjeoioiIiPQu2UuWTyRM5xJ/IPiOvi8nzdVuI9tbOVA4LepKREREZABJ\n9pLlo4nzZvYQ8IdQKkpj7VVryQRieoaliIiI9KHjve42DRjTl4X0B/Xb36DTjYLJCmQiIiLSd5Lt\nQ9bAkX3IqoDbQqkojbXuWMu7PoaS8YMui4qIiEiIkr1kOSzsQvqDnL0becsncc6YQfsYTxEREQlB\nUpcszexKMytImC80s48n8bn5ZrbRzDab2eIetjnfzFab2VozeyH50lOsvYWCxu1UZk1laE6y90KI\niIiIHF2yfchud/f6gzPuXgfc3tsHgmdg3gVcAswCPmNms7psUwj8EPiYu88GrjqG2lNrz2ZidOgO\nSxEREelzyQay7rY7WjPRXGCzu29x91bgYWBBl20+Czzm7u8CuHt1kvWknO9aB0DGWD3DUkRERPpW\nsoGszMz+w8xODF7/Aaw6ymcmAuUJ8xXBskQnA0Vm9nszW2Vmn0+ynpQ7UPEWbR6jYNLMqEsRERGR\nASbZQPZVoBX4BfGWrmbgpj44fiZwJnAZ8FHgH8zs5K4bmdkiMyszs7Kampo+OOyxa9mxlq0+jqnj\niiI5voiIiAxcyd5leQDotlN+LyqB4oT5ScGyRBXAnmD/B8zsReB04O0ux78buBugtLT0PY9wSoWs\nPRt424uZO1p3WIqIiEjfSvYuy2eCDvgH54vM7OmjfGwlMM3MpppZNnA1sKzLNr8BzjOzTDPLBz4I\nrE++/BRpPcDQpkq2xSYzelhO1NWIiIjIAJPs+A2jgjsrAXD3WjPrdXRUd283s5uBp4EYsNTd15rZ\njcH6Je6+3syeAt4EOoF73H3NcX2TMNVsJAOnYfjJmFnU1YiIiMgAk2wg6zSzyQfvhjSzEo4cub9b\n7r4cWN5l2ZIu898FvptkHdGojjfa2Rh16BcREZG+l2wg+zvgD8HArQZ8CFgUWlVppnXnWjo9i4KJ\nGoNMRERE+l6ynfqfMrNS4iHsdeDXQFOYhaWTjqq1bPaJjBqWH3UpIiIiMgAl+3DxG4BbiN8puRqY\nB6wAPhJeaekjtnsDb/s0CvOzoy5FREREBqBkxyG7BTgL2O7uFwBnAHW9f2SAaKolu7GKjZ2TKMrP\niroaERERGYCSDWTN7t4MYGY57r4BmB5eWWmkegMAb3sxhQpkIiIiEoJkO/VXBOOQ/Rp4xsxqge3h\nlZVGquPPsNzYWUxBni5ZioiISN9LtlP/lcHkN83seaAAeCq0qtJJ9XpaYkPYyQgK8tRCJiIiIn0v\n2RayQ9z9hTAKSVvV66nOncqQtkyyM5O9wisiIiKSPCWM3rhD9Toqskp0h6WIiIiERoGsN/uroWkv\nWzOmqEO/iIiIhEaBrDc18UcmbfRJCmQiIiISGgWy3gTPsFzTOoFC3WEpIiIiIVEg6031OsgfxfaW\nIRSohUxERERCokDWm+r1+JiZ1DW2UaghL0RERCQkCmQ9cYfq9bSNnEF7p6sPmYiIiIRGgawn9eXQ\nup/9BdMANOyFiIiIhEaBrCdBh/7aoScB6JKliIiIhCbUQGZm881so5ltNrPF3aw/38zqzWx18PpG\nmPUck+AZltU5JwBqIRMREZHwHPOjk5JlZjHgLuBioAJYaWbL3H1dl01fcvfLw6rjuFWvh+GT2NOR\nA6A+ZCIiIhKaMFvI5gKb3X2Lu7cCDwMLQjxe36peB8EdlqBLliIiIhKeMAPZRKA8Yb4iWNbVOWb2\nppk9aWazu9uRmS0yszIzK6upqQmj1iN1tEPN2zBmBnWNrQAah0xERERCE3Wn/teAye5+GvBfwK+7\n28jd73b3UncvHT16dPhV1W6FjhYYM4u6xjbys2PkZMbCP66IiIgMSmEGskqgOGF+UrDsEHff5+77\ng+nlQJaZjQqxpuQEHfoZM5O6Jg0KKyIiIuEKM5CtBKaZ2VQzywauBpYlbmBm48zMgum5QT17Qqwp\nOdUbAINR06lrbKNAd1iKiIhIiEK7y9Ld283sZuBpIAYsdfe1ZnZjsH4J8CngL8ysHWgCrnZ3D6um\npFWvgxFTITuf+qZWtZCJiIhIqEILZHDoMuTyLsuWJEz/APhBmDUcl+r1MGYWAHWNbZw0ZmjEBYmI\niMhAFnWn/vTT3gJ7NsOYmQDUNrZpDDIREREJlQJZV7s3gXfAmJm4e/ySpfqQiYiISIgUyLoaPgE+\ncQ9MPofG1g7aOlx9yERERCRUofYh65fyR8BpVwFQV9cE6LFJIiIiEi61kPXi0Cj9ebpkKSIiIuFR\nIOtF/cHnWKqFTEREREKkQNaLWgUyERERSQEFsl7UNcUvWRbpLksREREJkQJZL+qCFrIC3WUpIiIi\nIVIg60V9Uxu5WRnkZsWiLkVEREQGMAWyXtQ1tlKoOyxFREQkZApkvajTY5NEREQkBRTIelHX2Kb+\nYyIiIhKcJyY7AAAJ6UlEQVQ6BbJe1DW16g5LERERCZ0CWS90yVJERERSQYGsB+5OXVMbBQpkIiIi\nErJQA5mZzTezjWa22cwW97LdWWbWbmafCrOeY9Hc1klre6fushQREZHQhRbIzCwG3AVcAswCPmNm\ns3rY7l+A34ZVy/E4OEq/LlmKiIhI2MJsIZsLbHb3Le7eCjwMLOhmu68CjwLVIdZyzGoPBM+x1F2W\nIiIiErIwA9lEoDxhviJYdoiZTQSuBH7U247MbJGZlZlZWU1NTZ8X2p3DLWS6ZCkiIiLhirpT/38C\nt7l7Z28bufvd7l7q7qWjR49OSWH1wXMsdclSREREwpYZ4r4rgeKE+UnBskSlwMNmBjAKuNTM2t39\n1yHWlZS6JgUyERERSY0wA9lKYJqZTSUexK4GPpu4gbtPPThtZvcBT6RDGIP4GGSA7rIUERGR0IUW\nyNy93cxuBp4GYsBSd19rZjcG65eEdey+UNfUSnZmBrlZUV/VFRERkYEuzBYy3H05sLzLsm6DmLtf\nF2Ytx6ruQBuFeVkEl1NFREREQqPmnx7oOZYiIiKSKgpkPahr1GOTREREJDUUyHpQ39SmQWFFREQk\nJRTIelDX2KYhL0RERCQlFMh6UNfUqlH6RUREJCUUyLrR3NZBc1snBbpkKSIiIimgQNaNg4PC6i5L\nERERSQUFsm4cfrC4WshEREQkfApk3Tj82CQFMhEREQmfAlk3DgYyjUMmIiIiqaBA1o36Q5cs1YdM\nREREwqdA1o1aXbIUERGRFFIg60ZdYxvZsQzys2NRlyIiIiKDgAJZN+qbWinIz8LMoi5FREREBgEF\nsm7UNeo5liIiIpI6CmTd0HMsRUREJJVCDWRmNt/MNprZZjNb3M36BWb2ppmtNrMyMzsvzHqSVdfU\nRkGe7rAUERGR1AgtkJlZDLgLuASYBXzGzGZ12ew54HR3nwN8EbgnrHqORV1jq1rIREREJGXCbCGb\nC2x29y3u3go8DCxI3MDd97u7B7NDACcN1DW2UaRAJiIiIikSZiCbCJQnzFcEy45gZlea2Qbg/xFv\nJYtUc1sHTW0dGhRWREREUibyTv3u/ri7zwA+DvxTd9uY2aKgj1lZTU1NqPXsawoem6S7LEVERCRF\nwgxklUBxwvykYFm33P1F4AQzG9XNurvdvdTdS0ePHt33lSaoCwKZ+pCJiIhIqoQZyFYC08xsqpll\nA1cDyxI3MLOTLBh91cw+AOQAe0Ks6aiG5GRy/bklTBszLMoyREREZBDJDGvH7t5uZjcDTwMxYKm7\nrzWzG4P1S4BPAp83szagCViY0Mk/EhML87j9itlRliAiIiKDjEWcf45ZaWmpl5WVRV2GiIiIyFGZ\n2Sp3Lz3adpF36hcREREZ7BTIRERERCKmQCYiIiISMQUyERERkYgpkImIiIhETIFMREREJGL9btgL\nM6sBtqfgUKOA3Sk4jhwbnZf0pXOTnnRe0pPOS/rq63Mzxd2P+pihfhfIUsXMypIZN0RSS+clfenc\npCedl/Sk85K+ojo3umQpIiIiEjEFMhEREZGIKZD17O6oC5Bu6bykL52b9KTzkp50XtJXJOdGfchE\nREREIqYWMhEREZGIKZB1YWbzzWyjmW02s8VR1zNYmVmxmT1vZuvMbK2Z3RIsH2Fmz5jZpuC9KOpa\nByszi5nZ62b2RDCvcxMxMys0s1+Z2QYzW29mZ+u8pAcz+3rws2yNmT1kZrk6N9Ews6VmVm1maxKW\n9XguzOxvgkyw0cw+GlZdCmQJzCwG3AVcAswCPmNms6KtatBqB/6Pu88C5gE3BediMfCcu08Dngvm\nJRq3AOsT5nVuoncn8JS7zwBOJ35+dF4iZmYTga8Bpe5+ChADrkbnJir3AfO7LOv2XAT/71wNzA4+\n88MgK/Q5BbIjzQU2u/sWd28FHgYWRFzToOTuO939tWC6gfh/LBOJn4/7g83uBz4eTYWDm5lNAi4D\n7klYrHMTITMrAP4M+G8Ad2919zp0XtJFJpBnZplAPrADnZtIuPuLwN4ui3s6FwuAh929xd23ApuJ\nZ4U+p0B2pIlAecJ8RbBMImRmJcAZwJ+Ase6+M1hVBYyNqKzB7j+BvwY6E5bp3ERrKlAD3BtcSr7H\nzIag8xI5d68E/g14F9gJ1Lv7b9G5SSc9nYuU5QIFMklrZjYUeBS41d33Ja7z+C3Cuk04xczscqDa\n3Vf1tI3OTSQygQ8AP3L3M4ADdLkEpvMSjaA/0gLioXkCMMTMrkncRucmfUR1LhTIjlQJFCfMTwqW\nSQTMLIt4GHvA3R8LFu8ys/HB+vFAdVT1DWLnAh8zs23EL+t/xMx+js5N1CqACnf/UzD/K+IBTecl\nehcBW929xt3bgMeAc9C5SSc9nYuU5QIFsiOtBKaZ2VQzyybekW9ZxDUNSmZmxPvCrHf3/0hYtQz4\nQjD9BeA3qa5tsHP3v3H3Se5eQvzfyO/c/Rp0biLl7lVAuZlNDxZdCKxD5yUdvAvMM7P84GfbhcT7\nxercpI+ezsUy4GozyzGzqcA04NUwCtDAsF2Y2aXE+8fEgKXu/n8jLmlQMrPzgJeAtzjcT+lvifcj\newSYDGwHPu3uXTtnSoqY2fnAX7r75WY2Ep2bSJnZHOI3WmQDW4Drif/irfMSMTP7FrCQ+B3krwM3\nAEPRuUk5M3sIOB8YBewCbgd+TQ/nwsz+Dvgi8XN3q7s/GUpdCmQiIiIi0dIlSxEREZGIKZCJiIiI\nREyBTERERCRiCmQiIiIiEVMgExEREYmYApmISJLM7HwzeyLqOkRk4FEgExEREYmYApmIDDhmdo2Z\nvWpmq83sx2YWM7P9ZvY9M1trZs+Z2ehg2zlm9oqZvWlmjwfPHcTMTjKzZ83sDTN7zcxODHY/1Mx+\nZWYbzOyBYOR1EZH3RYFMRAYUM5tJfET0c919DtABfA4YApS5+2zgBeKjcwP8FLjN3U8j/mSIg8sf\nAO5y99OJP3dwZ7D8DOBWYBZwAvFne4qIvC+ZURcgItLHLgTOBFYGjVd5xB8U3An8Itjm58BjZlYA\nFLr7C8Hy+4FfmtkwYKK7Pw7g7s0Awf5edfeKYH41UAL8IfyvJSIDmQKZiAw0Btzv7n9zxEKzf+iy\n3fE+N64lYboD/RwVkT6gS5YiMtA8B3zKzMYAmNkIM5tC/Ofdp4JtPgv8wd3rgVoz+1Cw/FrgBXdv\nACrM7OPBPnLMLD+l30JEBhX9ZiciA4q7rzOzvwd+a2YZQBtwE3AAmBusqybezwzgC8CSIHBtAa4P\nll8L/NjM/jHYx1Up/BoiMsiY+/G22ouI9B9mtt/dh0Zdh4hId3TJUkRERCRiaiETERERiZhayERE\nREQipkAmIiIiEjEFMhEREZGIKZCJiIiIREyBTERERCRiCmQiIiIiEfv/RfDObqqTkjgAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f1687387a50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# NOTE THE CODE TEMPLATE BELOW WRAPS ALL OF THE ABOVE CODE, AND EXPOSES ONLY THE \n",
    "# DIFFERENT HYPERPARAMETER CHOICES. MAKE SURE YOU UNDERSTAND EXACTLY HOW\n",
    "# THE ABOVE CODE WORKS FIRST. TO SAVE SOME SPACE, WE WILL COPY AND MODIFY THE \n",
    "# CODE BELOW TO BUILD AND TRAIN DIFFERENT MODELS IN THE REST OF THIS PRACTICAL. \n",
    "# YOU CAN USE WHICHEVER VERSION YOU PREFER FOR YOUR OWN EXPERIMENTS.\n",
    "\n",
    "# Helper to wrap building, training, evaluating and plotting model accuracy.\n",
    "\n",
    "def build_train_eval_and_plot(build_params, train_params, verbose=True):\n",
    "    tf.reset_default_graph()\n",
    "    m = DNNClassifier(**build_params)\n",
    "\n",
    "    with tf.Session() as sess:\n",
    "        # Train model on the MNIST dataset.\n",
    "     \n",
    "        train_losses, train_accs, val_losses, val_accs = train_tf_model(\n",
    "                m, \n",
    "                sess,\n",
    "                verbose=verbose,\n",
    "                **train_params) \n",
    "        \n",
    "        # Now evaluate it on the test set:\n",
    "    \n",
    "        accuracy_op = m.accuracy()    # Get the symbolic accuracy operation\n",
    "        # Calculate the accuracy using the test images and labels.\n",
    "        accuracy = accuracy_op.eval({m.x: mnist.test.images, \n",
    "                                                                 m.y: mnist.test.labels})    \n",
    "        \n",
    "        if verbose: \n",
    "            print \"Accuracy on test set:\", accuracy\n",
    "            # Plot losses and accuracies.\n",
    "            plot_multi([train_losses, val_losses], ['train', 'val'], 'loss', 'epoch')\n",
    "            plot_multi([train_accs, val_accs], ['train', 'val'], 'accuracy', 'epoch')\n",
    "            \n",
    "        \n",
    "        ret = {'train_losses': train_losses, 'train_accs' : train_accs,\n",
    "                     'val_losses' : val_losses, 'val_accs' : val_accs,\n",
    "                     'test_acc' : accuracy}\n",
    "        \n",
    "        return m, ret\n",
    "\n",
    "#################################CODE TEMPLATE##################################\n",
    "# Specify the model hyperparameters (NOTE: All the defaults can be omitted):\n",
    "model_params = {\n",
    "        #'input_size' : 784,    # There are 28x28 = 784 pixels in MNIST images\n",
    "        'hidden_sizes' : [512], # List of hidden layer dimensions, empty for linear model.\n",
    "        #'output_size' : 10,    # There are 10 possible digit classes\n",
    "        #'act_fn' : tf.nn.relu,    # The activation function to use in the hidden layers\n",
    "        'l2_lambda' : 0.            # Strength of L2 regularization.\n",
    "}\n",
    "\n",
    "# Specify the training hyperparameters:\n",
    "training_params = {'num_epochs' : 100,     # Max epochs/iterations to train for.\n",
    "                        #'batch_size' : 100,    # Number of examples per batch, 100 default.\n",
    "                        #'keep_prob' : 1.0,    # (1. - dropout) probability, none by default.\n",
    "                        'train_only_on_fraction' : 5e-2,    # Fraction of training data to use, 1. for everything.\n",
    "                        'optimizer_fn' : None,    # Optimizer, None for Adam.\n",
    "                        'report_every' : 1, # Report training results every nr of epochs.\n",
    "                        'eval_every' : 2,     # Evaluate on validation data every nr of epochs.\n",
    "                        'stop_early' : False,    # Use early stopping or not.\n",
    "}\n",
    "\n",
    "# Build, train, evaluate and plot the results!\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True    # Modify as desired.\n",
    ")\n",
    "\n",
    "###############################END CODE TEMPLATE################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J-YKo75RW1iO"
   },
   "source": [
    "Above we plot the training loss vs the validation loss and the training accuracy vs the validation accuracy on only 5% (`train_only_on_fraction=5e-3`) of the training data (so that it doesn't take too long, and also so that our model can overfit easier). We see that the loss is coming down and the accuracies are going up, as expected! By training on a small subset of the training data, we established that \n",
    "\n",
    "* the data that the model is being trained on is hopefully not corrupt (this can happen during preprocessing, loading, etc), \n",
    "* our loss and gradients are likely correct, \n",
    "* our optimizer seems to do the right thing, \n",
    "* and generally, that our code probably works! \n",
    "\n",
    "**NOTE**: Notice the point where training loss/accuracy continues to improve, but validation accuracy starts to plateau? That is the point where the model starts to **overfit** the training data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "32oVqFziKYcv"
   },
   "source": [
    "# Architectural Choices (15min)\n",
    "\n",
    "## Depth\n",
    "\n",
    "### A Linear 784-10 Model\n",
    "\n",
    "Let's evaluate the simple linear model trained on the full dataset and then add layers to see what effect this will have on the accuracy.\n",
    "\n",
    "**NOTE**: If you're unsure what the \"784-10\" notation means, scroll up and re-read the section on \"Building a Feed-forward Neural Network\" where we explain that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 1528,
     "output_extras": [
      {
       "item_id": 27
      },
      {
       "item_id": 28
      },
      {
       "item_id": 29
      },
      {
       "item_id": 30
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 24080,
     "status": "ok",
     "timestamp": 1502999087933,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "mChxOFJnL_kt",
    "outputId": "ab2184b1-64b2-4166-e638-9ef35a99c663"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 4.470642067\n",
      "Epoch: 0001 Validation acc= 0.694800019\n"
     ]
    },
    {
     "ename": "MemoryError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m\u001b[0m",
      "\u001b[0;31mMemoryError\u001b[0mTraceback (most recent call last)",
      "\u001b[0;32m<ipython-input-21-dddf9dc2096e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mu'time'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mu''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mu\"# Train the linear model on the full dataset.\\n\\n################################################################################\\n# Specify the model hyperparameters.\\nmodel_params = {'l2_lambda' : 0.}\\n\\n# Specify the training hyperparameters:\\ntraining_params = {'num_epochs' : 50,     # Max epochs/iterations to train for.\\n                        'optimizer_fn' : None,            # Now we're using Adam.\\n                        'report_every' : 1, # Report training results every nr of epochs.\\n                        'eval_every' : 1,     # Evaluate on validation data every nr of epochs.\\n                        'stop_early' : True    \\n}\\n\\n# Build, train, evaluate and plot the results!\\ntrained_model, training_results = build_train_eval_and_plot(\\n        model_params, \\n        training_params, \\n        verbose=True    # Modify as desired.\\n)\\n\\n################################################################################\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/home/ubuntu/anaconda2/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2113\u001b[0m             \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvar_expand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2114\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2115\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2116\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-59>\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n",
      "\u001b[0;32m/home/ubuntu/anaconda2/lib/python2.7/site-packages/IPython/core/magic.pyc\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/ubuntu/anaconda2/lib/python2.7/site-packages/IPython/core/magics/execution.pyc\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1183\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1184\u001b[0m             \u001b[0mst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1185\u001b[0;31m             \u001b[0;32mexec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_ns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1186\u001b[0m             \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1187\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m<ipython-input-20-50784ea96410>\u001b[0m in \u001b[0;36mbuild_train_eval_and_plot\u001b[0;34m(build_params, train_params, verbose)\u001b[0m\n\u001b[1;32m     18\u001b[0m                 \u001b[0msess\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     19\u001b[0m                 \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mverbose\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m                 **train_params) \n\u001b[0m\u001b[1;32m     21\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     22\u001b[0m         \u001b[0;31m# Now evaluate it on the test set:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-18-5761bb70492a>\u001b[0m in \u001b[0;36mtrain_tf_model\u001b[0;34m(tf_model, session, num_epochs, batch_size, keep_prob, train_only_on_fraction, optimizer_fn, report_every, eval_every, stop_early, verbose)\u001b[0m\n\u001b[1;32m     50\u001b[0m         \u001b[0;31m# Loop over all batches.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal_batch\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 52\u001b[0;31m             \u001b[0mbatch_x\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbatch_y\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmnist_train_data\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnext_batch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     53\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m             \u001b[0;31m# Run optimization op (backprop) and cost op (to get loss value),\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/home/ubuntu/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.pyc\u001b[0m in \u001b[0;36mnext_batch\u001b[0;34m(self, batch_size, fake_data, shuffle)\u001b[0m\n\u001b[1;32m    194\u001b[0m         \u001b[0mperm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_examples\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    195\u001b[0m         \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshuffle\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mperm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 196\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_images\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mimages\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mperm\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    197\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_labels\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlabels\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mperm\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    198\u001b[0m       \u001b[0;31m# Start next epoch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mMemoryError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Train the linear model on the full dataset.\n",
    "\n",
    "################################################################################\n",
    "# Specify the model hyperparameters.\n",
    "model_params = {'l2_lambda' : 0.}\n",
    "\n",
    "# Specify the training hyperparameters:\n",
    "training_params = {'num_epochs' : 50,     # Max epochs/iterations to train for.\n",
    "                   'optimizer_fn' : None,            # Now we're using Adam.\n",
    "                   'report_every' : 1, # Report training results every nr of epochs.\n",
    "                   'eval_every' : 1,     # Evaluate on validation data every nr of epochs.\n",
    "                   'stop_early' : True    \n",
    "}\n",
    "\n",
    "# Build, train, evaluate and plot the results!\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True    # Modify as desired.\n",
    ")\n",
    "\n",
    "################################################################################"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dnaNJvLpYp9F"
   },
   "source": [
    "~91% is quite a bad score on MNIST! Now let's build a deeper model, to improve on that score. \n",
    "\n",
    "### 1 hidden layer: 784-512-10 Architecture w/ L2\n",
    "\n",
    "Notice that we add a bit of L2-regularization to our model.\n",
    "\n",
    "**QUESTION**: What does that do? What is the effect of removing it? Try it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 2293,
     "output_extras": [
      {
       "item_id": 49
      },
      {
       "item_id": 50
      },
      {
       "item_id": 51
      },
      {
       "item_id": 52
      },
      {
       "item_id": 53
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 64958,
     "status": "ok",
     "timestamp": 1503008137397,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "26XgusXy8OTs",
    "outputId": "6e4c8098-3ded-40ca-9281-ebbfc7862953"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 199.931141524\n",
      "Epoch: 0001 Validation acc= 0.870599747\n",
      "Epoch: 0002 Training cost= 141.988723103\n",
      "Epoch: 0002 Validation acc= 0.898999751\n",
      "Epoch: 0003 Training cost= 113.183370292\n",
      "Epoch: 0003 Validation acc= 0.912399769\n",
      "Epoch: 0004 Training cost= 93.298404125\n",
      "Epoch: 0004 Validation acc= 0.922599733\n",
      "Epoch: 0005 Training cost= 78.503151592\n",
      "Epoch: 0005 Validation acc= 0.921799719\n",
      "Epoch: 0006 Training cost= 66.877790590\n",
      "Epoch: 0006 Validation acc= 0.935599744\n",
      "Epoch: 0007 Training cost= 57.344941926\n",
      "Epoch: 0007 Validation acc= 0.935999691\n",
      "Epoch: 0008 Training cost= 49.325550454\n",
      "Epoch: 0008 Validation acc= 0.943399668\n",
      "Epoch: 0009 Training cost= 42.386559400\n",
      "Epoch: 0009 Validation acc= 0.940399766\n",
      "Epoch: 0010 Training cost= 36.450480402\n",
      "Epoch: 0010 Validation acc= 0.946999669\n",
      "Epoch: 0011 Training cost= 31.281487076\n",
      "Epoch: 0011 Validation acc= 0.944399714\n",
      "Epoch: 0012 Training cost= 26.775760117\n",
      "Epoch: 0012 Validation acc= 0.944599748\n",
      "Epoch: 0013 Training cost= 22.888392136\n",
      "Epoch: 0013 Validation acc= 0.950199783\n",
      "Epoch: 0014 Training cost= 19.539065340\n",
      "Epoch: 0014 Validation acc= 0.950399756\n",
      "Epoch: 0015 Training cost= 16.640647588\n",
      "Epoch: 0015 Validation acc= 0.948999763\n",
      "Epoch: 0016 Training cost= 14.154421536\n",
      "Epoch: 0016 Validation acc= 0.955999732\n",
      "Epoch: 0017 Training cost= 12.027777351\n",
      "Epoch: 0017 Validation acc= 0.957199812\n",
      "Epoch: 0018 Training cost= 10.202699273\n",
      "Epoch: 0018 Validation acc= 0.954399705\n",
      "Epoch: 0019 Training cost= 8.657644658\n",
      "Epoch: 0019 Validation acc= 0.953599632\n",
      "Epoch: 0020 Training cost= 7.334119753\n",
      "Epoch: 0020 Validation acc= 0.954599738\n",
      "Epoch: 0021 Training cost= 6.196740341\n",
      "Epoch: 0021 Validation acc= 0.961399794\n",
      "Epoch: 0022 Training cost= 5.227180183\n",
      "Epoch: 0022 Validation acc= 0.959199727\n",
      "Epoch: 0023 Training cost= 4.402065789\n",
      "Epoch: 0023 Validation acc= 0.956199765\n",
      "Epoch: 0024 Training cost= 3.707452070\n",
      "Epoch: 0024 Validation acc= 0.955999732\n",
      "Epoch: 0025 Training cost= 3.129813027\n",
      "Epoch: 0025 Validation acc= 0.964799762\n",
      "Epoch: 0026 Training cost= 2.621126144\n",
      "Epoch: 0026 Validation acc= 0.949599743\n",
      "Epoch: 0027 Training cost= 2.205886074\n",
      "Epoch: 0027 Validation acc= 0.958999693\n",
      "Epoch: 0028 Training cost= 1.853469856\n",
      "Epoch: 0028 Validation acc= 0.963399649\n",
      "Epoch: 0029 Training cost= 1.541123295\n",
      "Epoch: 0029 Validation acc= 0.964799762\n",
      "Epoch: 0030 Training cost= 1.291453608\n",
      "Epoch: 0030 Validation acc= 0.962999761\n",
      "Epoch: 0031 Training cost= 1.086839738\n",
      "Epoch: 0031 Validation acc= 0.963599741\n",
      "Epoch: 0032 Training cost= 0.914146203\n",
      "Epoch: 0032 Validation acc= 0.963399649\n",
      "Epoch: 0033 Training cost= 0.769378621\n",
      "Epoch: 0033 Validation acc= 0.965999722\n",
      "Epoch: 0034 Training cost= 0.650265682\n",
      "Epoch: 0034 Validation acc= 0.967199743\n",
      "Epoch: 0035 Training cost= 0.558896172\n",
      "Epoch: 0035 Validation acc= 0.969799757\n",
      "Epoch: 0036 Training cost= 0.482242939\n",
      "Epoch: 0036 Validation acc= 0.972199798\n",
      "Epoch: 0037 Training cost= 0.417804223\n",
      "Epoch: 0037 Validation acc= 0.966399789\n",
      "Epoch: 0038 Training cost= 0.370378067\n",
      "Epoch: 0038 Validation acc= 0.973199725\n",
      "Epoch: 0039 Training cost= 0.331554002\n",
      "Epoch: 0039 Validation acc= 0.973999739\n",
      "Epoch: 0040 Training cost= 0.299039026\n",
      "Epoch: 0040 Validation acc= 0.973399699\n",
      "Epoch: 0041 Training cost= 0.271972703\n",
      "Epoch: 0041 Validation acc= 0.975399673\n",
      "Epoch: 0042 Training cost= 0.252507508\n",
      "Epoch: 0042 Validation acc= 0.972199798\n",
      "Epoch: 0043 Training cost= 0.237628794\n",
      "Epoch: 0043 Validation acc= 0.975399673\n",
      "Epoch: 0044 Training cost= 0.222427252\n",
      "Epoch: 0044 Validation acc= 0.970399678\n",
      "Epoch: 0045 Training cost= 0.213050932\n",
      "Epoch: 0045 Validation acc= 0.973399699\n",
      "Epoch: 0046 Training cost= 0.206244858\n",
      "Epoch: 0046 Validation acc= 0.970799685\n",
      "Epoch: 0047 Training cost= 0.200895666\n",
      "Epoch: 0047 Validation acc= 0.975999713\n",
      "Epoch: 0048 Training cost= 0.193038337\n",
      "Epoch: 0048 Validation acc= 0.976599693\n",
      "Epoch: 0049 Training cost= 0.189712861\n",
      "Epoch: 0049 Validation acc= 0.973599732\n",
      "Epoch: 0050 Training cost= 0.186406397\n",
      "Epoch: 0050 Validation acc= 0.976599693\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.977\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAD+CAYAAABleEI9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPX97/HXmS2ZzGTPZCMkAmHfEUXUIotC1SKbW6t1\nwSvWR9W63f4qbfVR29rlZ/3pbW8V+tOf16VaUcGqWFAsKC6g7LvsIZB93yaTmTn3j0AAAQlImDnJ\n+/l4RDPfMzPnM/3YM++c7WuYpmkiIiIiIlHJFukCREREROTEFNZEREREopjCmoiIiEgUU1gTERER\niWIKayIiIiJRTGFNREREJIp1aFgrLi7mpptu4oorrmDy5Mm88MILANTU1DBz5kwmTZrEbbfdRl1d\nXdtrfvOb3zBx4kSmTJnCli1bOrI8ERERkajXoWHNbrfz0EMPsXDhQl599VVefvlldu7cydy5cxk9\nejSLFi1i1KhRzJkzB4Bly5ZRUFDA4sWLefTRR3nkkUc6sjwRERGRqNehYc3n89G/f38APB4PvXr1\noqSkhCVLljBt2jQApk2bxpIlSwBYsmQJU6dOBWDo0KHU1dVRXl7ekSWKiIiIRLWzds5aYWEhW7du\nZejQoVRUVJCWlga0BrrKykoASktLyczMbHtNRkYGJSUlZ6tEERERkahzVsJaQ0MD99xzD7Nnz8bj\n8WAYxnGfd7yZr070XBEREZGuoMPDWjAY5J577mHKlClceumlAKSmprYd3iwrKyMlJQVo3ZNWXFzc\n9tri4mLS09O/8f0nP/AWOwurO6h6ERERkchydPQKZs+eTX5+PjfffHPb2Pjx43nzzTeZNWsW8+fP\nZ8KECQBMmDCBl19+mSuuuIK1a9eSkJDQdrj0m6zbWkJCjL3DPoN0HJ8vnrKyupM/UaKOemdt6p+1\nqX/W5fPFn/JrOjSsrVq1irfffps+ffowdepUDMPgvvvu4/bbb+fee+/ljTfeIDs7m6eeegqASy65\nhGXLlnHZZZfhdrv53e9+1671FJTWd+THEBEREYmYDg1r55577gnvlfb8888fd/zhhx8+pXXYDCgo\n0V8XIiIi0jlZfgaDbunxFJTWEz7OxQkiIiIiVmf5sNarWyLNgRBl1U2RLkVERETkjLN8WOuRnQhA\nQYnOWxMREZHOx/JhrVe3Q2FN562JiIhI52P5sNajm/asiYiISOdl+bCW4HGRkhATVXvW6uvrmT//\n9VN+3U9/ei8NDQqdIiIicpjlwxpAbno8NQ0BauqbI10KAHV1tcyfP++Y8XA4/I2v++Mfn8Tj8XZU\nWSIiImJBHT6DwdmQm+Fl7Y5yCkrrGeyNiXQ5PPPMXzhwYD8zZ96A3W7H5YohPj6egoK9/P3vb/DQ\nQw9SVlZKINDMNdd8n8mTpwJwzTVX8eyzL9LY2MiDD97D4MHD2LhxHT5fBr///Z9wuVwR/mQiIiJy\ntnWSsNY6dUNBSR2De6Yetey1D3fwxdbSM7q+8/qlc+34/BMuv/POu9mzZxfPPfcya9as4qc/vY8X\nX3yNzMxMAGbPfoT4+Hiam5u5/fabuOSS8SQkJACHJ60vLNzHr371O/7jP37Oww8/xNKlHzJx4nfP\n6OcQERGR6Nc5wlp666HDaL3IYMCAgW1BDeC11/7Oxx8vA6C0tJTCwgIGDBgEHL6xb1ZWNr16tQbC\nvn37UVx84KzWLCIiItGhU4S11MRYPLGO415kcO34/G/cC3Y2xMbGtv2+Zs0qVq/+krlzn8flcnH3\n3XcQCASOec2RhzxtNvtxnyMiIiKdX6e4wMAwDLqneymtaqKpORjpcoiLi6OxsREA82vTYDU01BMf\nH4/L5WLv3j1s2rTxuO/x9deJiIhI19Qp9qxB63lrWwuqKSyrp3dOUkRrSUhIZPDgodx88/W4XDGk\npKS0LRs16kIWLHiDG2+8ltzcPAYNGnzEKw+fs2YYBiIiIiKG2Ql24ZSV1fHpxiL++50t3HBZHyac\nmxPpkqSdfL54ysqi5x550n7qnbWpf9am/lmXzxd/yq/pFIdBofVea6Bpp0RERKRz6TRhLTM1Dofd\nFrVXhIqIiIicjk4T1hx2Gzk+D/vL6wmGvnmmABERERGr6DRhDVpnMgiGTIoqGiNdioiIiMgZ0cnC\nms5bExERkc6lc4W1tosMdN6aiIiIdA6dKqzlpHswsN6etcsuGxPpEkRERCRKdaqwFutykJESR0Fp\nvaVmANANcEVEROREOs0MBofkZnhZuaWU8ho/viR3RGp4+uk/k5mZxbRpVwPw3HNzMQyDtWvXUF9f\nRzAY5Pbbf8TFF18SkfpERETEOjphWItn5ZZSCkrq8CW5eXPHO6wp3XBG1zE8fTDT8793wuWXXjqR\np576U1tY+/DDD3jiiT9z3XU3EBcXR01NNXfccavCmoiIiJxU5wtr6V6g9SKDc/umR6SG3r37Ul1d\nTUVFOVVVVSQkJJCamsZTTz3OunVrsdkMysvLqKqqJDk55eRvKCIiIl1W5wtrX7t9x/T8733jXrCO\nMm7cBP797w+oqKhgwoSJLF78HjU1NfzP/7yMzWbjmmuuork5cNbrEhEREWvpVBcYACR4XCR5XRSU\nRvb2HePHX8aSJYtZtuxDxo2bQH19PcnJKdhsNlav/pLi4qK251rpYggRERE5uzpdWIPWvWtVdc3U\nNUZuz1WPHj1pbGzE58sgJSWViRO/y9atm7n55u+zaNFC8vJ6tD1XV4OKiIjIiXS6w6DQekXo+p0V\nFJTUM7BH5M4J+3//79W23xMTk3jmmeeO+7zFi5edrZJERETEYjrnnrVDMxmUWuvmuCIiIiJf1znD\nWsbhK0JFRERErKxThrW0JDfuGLvlpp0SERER+bpOGdZshkF3n5fiykaaW0KRLkdERETktHXKsAat\nV4SaJhRG+BYeIiIiIt9Gpw1r3Q+dt6awJiIiIhZm+bD266VP0RIOHjOe97WZDERERESsyPJhbUPJ\nVr6q2nHMeHaaB7vN0BWhIiIiYmmWD2sAa0s3HjPmsNvoluahsKyeUDgcgapEREREvj3Lh7XEmHjW\nl28ibB4byHIz4mkJhimuaIxAZSIiIiLfnuXD2shuQ6lvaWBXzd5jlukiAxEREbE6y4e183OGArCu\n7NhDobrIQERERKzO8mFtUHpfYu0xrCvbiGmaRy3rnq5pp0RERMTaOjSszZ49mwsvvJDJkye3jf3l\nL39hzJgxTJs2jWnTpvHRRx+1LZszZw4TJ07k8ssvZ/ny5e1ah9PuZFBafyr8VRTWHzhqmTvGQXqS\nm4KSumOCnIiIiIgVdGhYmz59Os8+++wx47feeivz589n/vz5jBkzBoCdO3fy3nvvsXDhQv72t7/x\nq1/9qt0Ba6hvEHD8Q6G5GV4a/EEqa5u/xScRERERiYwODWsjR44kISHhmPHjhbAlS5ZwxRVX4HA4\nyMnJIS8vj/Xr17drPQNS+uKwOVhXtumYZd0PnbdWqvPWRERExHoics7ayy+/zJQpU/j5z39OXV1r\niCopKSErK6vtORkZGZSUlLTr/WIdMfRL7s2BhmJKG8uPWpaXofPWRERExLrOelj7wQ9+wAcffMBb\nb71FWloav//974Hj720zDKPd7zvsBIdCu6frilARERGxLsfZXmFKSkrb79deey0/+tGPAMjMzKSo\nqKhtWXFxMenp6e16T58vnrEJ5/PyttfZVL2FH4w8fEFDWpqXJG8M+8sb8Pniz9CnkDNJfbEu9c7a\n1D9rU/+6jg4Pa1/fY1ZWVobP5wPg/fffp0+fPgCMHz+eBx98kFtuuYWSkhIKCgoYMmRIu9ZRVta6\n1yw/sQfbK3axvbCQpJjEtuU5Pg8bd1eyu6ASr9t5Jj6WnCE+X3xb/8Ra1DtrU/+sTf2zrtMJ2R0a\n1h544AFWrFhBdXU1Y8eO5e6772bFihVs2bIFm81Gt27dePTRRwHIz8/n8ssv58orr8ThcPDII4+c\n0mFQaL0qdHv1LtaXbWZMzui28e4ZXjburmRfaT3985LP6GcUERER6UiG2QluQHbor4sqfzW/+PQx\n+iX35u7ht7ctX7mlhGfe2sR14/OZdH5upMqU49Bfh9al3lmb+mdt6p91nc6eNcvPYHCk5NgkcuNz\n+Kp6J40thydvPzyTgf7DFhEREWvpVGENWg+Fhs0wG8q3tI1lpMQR47RrQncRERGxnE4X1ob5BgKw\nrvzwDXJthkH3dC9F5Y0EWkKRKk1ERETklHW6sJbpySAjzsfmim0EQoG28e4ZXsKmyV4dChUREREL\n6XRhDVoPhbaEW9hS+dXhsV6pAHy+qX2zIoiIiIhEg04Z1g7NZrD2iNkMBvZIIdHrYsXmEh0KFRER\nEcvolGEtNz6HpJhENpRvIRRuDWZ2m42LBmXR2Bxk9fayCFcoIiIi0j6dMqwZhsFQ30Cagk1sr97V\nNn7xkNaJ4pevLzrRS0VERESiSqcMa3D8id0zU+LonZPIlj1VlNc0Rao0ERERkXbrtGGtV2IPPM44\n1pVtImyG28YvHpyFCXy6oThyxYmIiIi0U6cNa3abncGpA6gJ1LK3dl/b+Mh+6cQ47SzfUETY+jNt\niYiISCfXacMawNBDN8gtO3yDXHeMg/P6pVNe42fb3qpIlSYiIiLSLp06rPVL6YPL7mJd2UaOnK/+\n0IUGH2/QhQYiIiIS3Tp1WHPZnQxM6UtpUzlFDYdvhts7J5GMZDertpXR6A9GsEIRERGRb9apwxq0\nzmYARx8KNQyDi4dk0RIMs3KLZjQQERGR6NXpw9qgtH7YDTvryjYcNX7hoCwMAz7WPddEREQkinX6\nsOZ2uOmT3It99QeoaKpsG0+Oj2Fwz1R2F9Wyv6w+ghWKiIiInFinD2twxA1yyzcdNX7x4IMzGuhC\nAxEREYlSXSKsDfENxMA4ajYDgGG90/C6nXy6sZhgKHyCV4uIiIhETpcIawmueHok5rGzeg+1gbq2\ncYfdxgUDM6hrbGH9zooIVigiIiJyfF0irAGcmz4UE5OP939+1HjboVBdaCAiIiJRqMuEtQuyRuJx\nxrF033L8QX/beG5GPHmZ8azfWUFNfXMEKxQRERE5VpcJa7GOGMblXExjsInlB1YcteziwVmETZNP\nN2lydxEREYkuXSasAVyScyExdhcfFnxES6ilbXzUgAwcdhvL1xcdNS2ViIiISKR1qbAW54xjTLcL\nqQnU8XnxqrZxr9vJiD5pFFU0svNAbQQrFBERETlalwprAOO6fweHzcH7e5cSCofaxr8zJBvQhQYi\nIiISXbpcWEuMiefCrPOo8FeyqnRd23j/vGRSEmJYuaWE5kDoG95BRERE5OzpcmEN4NLcS7AZNhbv\n/Tdhs/VmuDabwUWDsvAHQny5rTTCFYqIiIi06pJhLdWdwnkZwylqKGFD+Za28YuG6J5rIiIiEl26\nZFgDmJg3FgODRXs/bLsCND3JTb/cJLbtq6a0qjHCFYqIiIh04bCW6clgqG8ge2v3sa1qR9v4xYf2\nrm3QPddEREQk8rpsWAOYmDcOgEV7/902dm7fdGJddj7ZUEQ4rHuuiYiISGR16bCWl9Cd/il9+Kpq\nB7tr9gIQ47QzakAGVXXNbNilyd1FREQksrp0WAOYdJy9a+OGdwNg/ke7CGtGAxEREYmgLh/W8pN6\n0jMxjw3lm9lf33oVaG5GPKMHZlBQWs9nG3XumoiIiEROlw9rhmEwKW88AIuP2Ls2fUwvHHYbb360\ni0CLbpIrIiIikdHlwxrAwNR+dPNmsapkHWWNreeppSbGctl5OVTVNfP+l/siXKGIiIh0VQprHNq7\nNg4Tk/cLlraNX3nBOXjdTt79bC+1jYHIFSgiIiJdVrvC2sKFC6mvrwfgqaee4rbbbmPjxo0dWtjZ\nNjx9CD53KiuKvqS6uQaAuFgHV110Dv5AiH8u3x3hCkVERKQraldYe/rpp/F6vaxfv57ly5czdepU\nfvOb33R0bWeVzbAxMW8cQTPEkoKP2sbHDu9GRrKbpWsOUFTREMEKRUREpCtqV1hzOBwAfPLJJ1xz\nzTVMnjyZ5ubmDi0sEs7PHEFSTCLL939OfUtrMHPYbVw9Np+wafL60p0RrlBERES6mnaFNcMwWLhw\nIQsXLmT06NEAtLS0dGhhkeCwObg09xIC4RaW7vukbXxEnzTycxJZs72cbQVVEaxQREREupp2hbVf\n/OIXvPPOO1x99dV0796dPXv2MGrUqJO+bvbs2Vx44YVMnjy5baympoaZM2cyadIkbrvtNurq6tqW\n/eY3v2HixIlMmTKFLVu2nMbH+fYuzD4fr9PD0sJP8Af9QGtYvW58PgCv/XuHbpQrIiIiZ027wtqI\nESP461//ys033wzAOeecwy9/+cuTvm769Ok8++yzR43NnTuX0aNHs2jRIkaNGsWcOXMAWLZsGQUF\nBSxevJhHH32URx555FQ/yxkRY3cxrvvFNAWbjprVoFd2Iuf3T2d3UR1fbCmNSG0iIiLS9bQrrP3+\n97+nrq6OYDDID37wA4YNG8Zbb7110teNHDmShISEo8aWLFnCtGnTAJg2bRpLlixpG586dSoAQ4cO\npa6ujvLy8lP6MGfK2JyLSY1N5oOCZRTUFraNT7+kF3abwRvLdtISDEekNhEREela2hXWPv30U+Lj\n41m+fDkZGRksWrSI55577rRWWFlZSVpaGgA+n4/KykoASktLyczMbHteRkYGJSUlp7WObyvWEcMP\n+l1N2Azz0tZ5BMNBANKT3Ew4N4fyGj9LVhWe5F1EREREvj3HqTz5iy++4LLLLiMjIwPDMM5oIeZx\nzgNr7zp8vvgzWkvre45gc+3FLNm1nE/KP+XqgVcCcMtVg/hkYzHvfr6XKeN6k+BxnfF1dzUd0T85\nO9Q7a1P/rE396zraFdZSU1N55JFH+Pjjj5k1axbBYJBQ6PTmy0xNTaW8vJy0tDTKyspISUkBWvek\nFRcfnjS9uLiY9PT0dr1nWVndyZ90Gi7PuYxV+zfwxqb3yI/rTTdvFgDfG53HPz7cwfP/3Mj3L+3d\nIevuKny++A7rn3Qs9c7a1D9rU/+s63RCdrsOg/7pT3+iR48ePPHEEyQmJlJcXMytt97arhV8fY/Z\n+PHjefPNNwGYP38+EyZMAGDChAksWLAAgLVr15KQkNB2uDRS3A433+87nZAZ4qUt8wiFWwPq+BE5\npCXG8uHqQkqrGiNao4iIiHRu7QprKSkp3HjjjXg8Hnbs2EFmZibTp08/6eseeOABrr/+enbv3s3Y\nsWN54403mDVrFp9++imTJk3is88+Y9asWQBccskl5OTkcNlll/Hwww9H7GrQrxuU1p/zM0dQUFfI\nh/s+BsDpsHH12F6EwiavL9sV4QpFRESkMzPM450s9jUbNmzgnnvuweVyYZomwWCQP//5zwwcOPBs\n1HhSHb0ruKGlkV+veJymoJ/Z591Lhicd0zT57Yur2HWgltk/PJf8bokdWkNnpV351qXeWZv6Z23q\nn3V12GHQ3/72tzz22GMsWrSIxYsX89hjj/HrX//6lFdmVR5nHNf3nU4wHOSlrfMIm2EMw+DacQdv\nlPvhjuNeICEiIiLybbUrrDU1NbVNMwVwwQUX0NTU1GFFRaNhvkGMSB/Crpq9LCv8FIA+3ZMY0cfH\njv01fLFVN8oVERGRM69dYc3tdvP555+3PV65ciVut7vDiopW1/aZiscZx1s736OssQKAa8b2wmG3\n8eKibVTW+iNcoYiIiHQ27Qprs2fP5qGHHmLSpElMmjSJn/3sZ/ziF7/o6NqiTrzLy7W9p9ASbuHv\nW18nbIbJSInj+5f2psEf5Jl/biIU1swGIiIicua06z5rQ4YMYfHixezevRvTNOnZsydOp7Oja4tK\n52YM48vSdWwo38wnB1bynW4XMHZYNlv2VvHl1lIWfLybGZf0inSZIiIi0kl84561pqamtp9gMEj3\n7t3Jzc0lGAx2uXPWDjEMg+v7TsPtiGXBjnep9FdhGAa3fLcfvqRY3v1sLxt3VUS6TBEREekkvjGs\nDR8+nBEjRjB8+PC23w89HjFixNmqMeokxSQyI38y/lAzr2x9E9M0iYt18KMpg7DbDP72zmaq65sj\nXaaIiIh0At94GHTr1q1nqw7LuSBrJKtK17G5chsrildxQdZIemQlcO24fF5Zsp25/9zEg9cPx2Y7\ns3OoioiISNfSrgsM5FiGYfCDfjOIsbt4ffvb1DTXAnDpyByG905ja0E1b3+6J7JFioiIiOUprH0L\nKbHJTMu/kqZgE89ufImWUAuGYXDrFf1JTYjhn8t3s2VvVaTLFBEREQtTWPuWLsoexbnpQ9lZs4cX\nt7xG2AzjdTu5Y8ogbDaDuW9vorYhEOkyRURExKIU1r4lm2Hjh/2vpWfiOawqXcfbuxYBkN8tkemX\n9KSmPsDf3tlMWNNRiYiIyGlQWDsDnHYndwy5mXR3Gov3/pvl+1tne5h0fi6De6ayaXcl732+N8JV\nioiIiBUprJ0hXqeHO4fOxOv08I+vFrCpYis2w+C27/Unyeti/ke7+WpfdaTLFBEREYtRWDuD0uPS\nuGPILdgNG89ufIl9dQdIiHNxx1UDMTGZ889N1De1RLpMERERsRCFtTOsZ2IeNw/4PoFQC0+ve44q\nfzV9c5OZenEPquqaefadzZg6f01ERETaSWGtAwxPH8zU/CuoCdTy13XP0RT0c+Xoc+ifl8y6nRX8\na0VBpEsUERERi1BY6yATuo9hTLcLOdBQzH9veBGTMLMmDyDR62Le0p18vP5ApEsUERERC1BY6yCG\nYXBNn6sYnNafrVXbeWXbmyR4XDxw3TA8sQ6ef28rX2wtjXSZIiIiEuUU1jqQzbBx68AbyI3vxmdF\nX7Bo74fk+Lzcf90wYpx25v5zE+t3lke6TBEREYliCmsdLMbu4kdDZpISm8zbuxaxsng1PbIS+MnV\nQ7DZDP7v/I1s1ZRUIiIicgIKa2dBYkw8dw65Fbcjlpe2zOOrqh30zU3mrumDCYdNnnpjPbsO1Ea6\nTBEREYlCCmtnSbY3k1mDbwLg6fXP81XVDgb3TOWOqwYSaAnxX6+tpbC0PsJVioiISLRRWDuL+iTn\n878G3Ug4HOKv655jc8U2RvZLZ+YV/WnwB3n8H2spqWyMdJkiIiISRRTWzrIhvoHMGnILAHPWP8+G\n8s1cNDiLGy7rQ21DgMdfXUNFjT+yRYqIiEjUUFiLgIGpffnRkFuxGTbmbniBNaUbmHBuDjMu6UlF\nbTOPv7qGmvrmSJcpIiIiUUBhLUL6pfTmx8P+F06bg+c2vcyXxWu4cvQ5XHFBHiVVTfzpH2s1j6iI\niIgorEVSflIP7h52OzF2F89vfpXPir5kxiU9GT+iG4VlDfzXa+toag5GukwRERGJIIW1COuRmMc9\nw2YR53Dz0pbX+OTACn5wWR8uHJTJ7qJanpy3TnvYREREujCFtSiQm5DDT0bcgdfp4ZVtb7Ks8BNu\nvaIf5/dPZ3thDY+9uIrS6qZIlykiIiIRoLAWJbp5s7hvxI9IdMXz+vZ/8uG+j5h11UC+OyqX4spG\nfvvCl+w8UBPpMkVEROQsU1iLIpmeDO4d8SOSYhJZsHMhi/Ys4dpx+fxwYh/qm1r449/XsGqbJn8X\nERHpShTWokx6nI/7RtxJamwy7+xezIIdC7lkeHbrXKKGwV/nb2TxygJM04x0qSIiInIWKKxFoTR3\nCveNuJN0dxrvFyzl6XX/Q6/cOH52wwgSvC5e/XAHf39/O+GwApuIiEhnp7AWpZJjk3hw5F0MSO3L\n5spt/OGLpzA8NfzyppHk+DwsWV3In99Yjz+gW3uIiIh0ZgprUczjjOPOIbdyRY/LqPRX86dVf2Vb\n/QYeuvFcBp6TzLqdFfzh5TVUa7YDERGRTkthLcrZDBtX9riMHw25BafNyUtb5zF/91v8eMZAxgzN\nYm9JHb994UsKy+ojXaqIiIh0AIU1ixiU1p+fnXcPOd5sPjmwgv+zdg6Tx2a0zSf6u5dWsWlPZaTL\nFBERkTNMYc1C0typPHDujxmVeS576/bxhy//D736tjDrqgG0BMP81z/W8fYnu3XhgYiISCeisGYx\nLruTH/a/luv7TscfbOYva/+b6rjN/O/rh5MU72L+x7t5/NU1VNXpPDYREZHOQGHNggzD4DvdLuC+\nEXeSGJPA27v+xYdVb/EfPxzCiD4+thZU8/CzK1izvSzSpYqIiMi3ZJgRurvq+PHj8Xq92Gw2HA4H\nr7/+OjU1Ndx3333s37+fnJwcnnzySeLj40/6XmVldWeh4uhUF6jnuU1/56uqHaS5U7mp/3UU7HLw\nypIdBENhJpybw7XjeuF02CNd6nH5fPFdun9Wpt5Zm/pnbeqfdfl8J881XxexPWuGYfDiiy+yYMEC\nXn/9dQDmzp3L6NGjWbRoEaNGjWLOnDmRKs8y4l1e7hp6GxPzxlHRVMl/rX6a2sQN/PyHw8lO87Bk\nVSG/eWEVRRUNkS5VRERETkPEwpppmoTD4aPGlixZwrRp0wCYNm0aH3zwQSRKsxy7zc6UXpfzk+F3\nkBKbxOK9/+blgue47eocLhmWzb7Sen71/Bd8tO6ApqkSERGxmIjuWbvtttuYMWMG8+bNA6CiooK0\ntDQAfD4fVVVVkSrPknon92T2+fdxYdb57K8v4sk1fyG7fzF3TBmA3Wbj+fe2Muefm2j0a9YDERER\nq3BEasWvvvoqPp+PyspKZs6cSY8ePTAMI1LldBqxjlhu6H81Q30DeWnrPBbsXEjPxM385IYpvL6o\nmJVbStl1oJY7rhpIr26JkS5XRERETiJiFxgc6S9/+QtxcXHMmzePF198kbS0NMrKyrjpppt47733\nIl2eZdU21/PfX77C54WriXHEcOOQaZRu9zHvw+0YhsHUMb34/qS+xLoiltlFRETkJCIS1pqamgiH\nw3g8HhobG5k5cyZ33XUXn332GYmJicyaNYu5c+dSW1vLgw8+eNL30xUxJ2aaJl+WrOUfXy2gKdjE\ngJS+nB9/GfMWF1Je4yctMZabJvVlUM/UiNSnK5qsS72zNvXP2tQ/6zqdq0EjEtb27dvHXXfdhWEY\nhEIhJk/O9g4mAAAX20lEQVSezKxZs6iurubee++lqKiI7OxsnnrqKRISEk76fvoP9uSq/NW8tGUe\nW6u2E+dwM6PXFAq2xbP4i0LCpskFAzK4fkJvEjyus1qXNjjWpd5Zm/pnbeqfdVkmrJ1p+g+2fUzT\n5OP9nzN/xzsEwi30Tc7nopTxLFxaxe6iOjyxDq4dl8/FQ7LO2vmD2uBYl3pnbeqftal/1qWwJu1S\n2ljGvK/+yebKbRgYXJQ9ivjaQby7vIjmQIh+uUn8cFJfslI9HV6LNjjWpd5Zm/pnbeqfdSmsySnZ\nWL6FN3e8Q0ljGW6Hm3FZY9mxLpl1Oypx2A2+N/ocLr8gD6ej4+7wog2Odal31qb+WZv6Z10Ka3LK\nQuEQy/Z/ysLd79MU9JMR52OoewxLP26hpj5AVmocN3+3H326J3XI+rXBsS71ztrUP2tT/6xLYU1O\nW12gnnd2L+aT/SswMemX1JeY8kGsWN2ACZzfP53pl/QiPcl9RterDY51qXfWpv5Zm/pnXQpr8q3t\nry/i9e1v81XVDmyGjWFJIylYl8W+4mbsNoOxw7sx+cJzzthVo9rgWJd6Z23qn7Wpf9alsCZnhGma\nrC/fxJvb36HcX4nX6aFf7HlsWR1PeVULMS47l4/KZeJ53b/1DXW1wbEu9c7a1D9rU/+sS2FNzqiW\ncJCl+5bzrz0f4g/5SXQl0MN2LhtWxVHfECLB42LKRefwnaHZOOyndxGCNjjWpd5Zm/pnbeqfdSms\nSYeob2lgScFHLN23nEC4hZSYZLKCw9jwZSzNLSbpyW5mXNKLkX19p3x/Nm1wrEu9szb1z9rUP+tS\nWJMOVdNcx/t7/83H+z8jaIZIi00juWEwm9bEEApDj6x4rhmbT7+85Ha/pzY41qXeWZv6Z23qn3Up\nrMlZUeWv5r09S/is6AvCZpiM2AxiKgewbaMLMOiXm8QVo/MYeE7KSfe0aYNjXeqdtal/1qb+WZfC\nmpxVZY0VLNzzPl8Ur8HEJDM2G6O4L7u+ag1tuRlerrggj5F907HZjh/atMGxLvXO2tQ/a1P/rEth\nTSKiqKGEd3ctZk3ZBgAyY7NwVPZix+Y4zLCN9GQ3l4/K5cJBWcfMhqANjnWpd9am/lmb+mddCmsS\nUfvq9vPe7g9YX74ZExOvw0uivw97NiURbHaR6HUx8bzujB3WDXdM6y0/tMGxLvXO2tQ/a1P/rEth\nTaJCeVMFywo/5dMDX+AP+XEYDtLMnhzYmkFzrYe4GAfjz+3Gped2p9c5qeqfRenLwtrUP2tT/6xL\nYU2iij/o5/OiVSwtXE5ZUwUAKbZsqndn01CSitNhZ/zI7lzQL528zFP/j1ciS18W1qb+WZv6Z10K\naxKVwmaYTRVbWbrvE7ZWbQcgzkggWJJLzb4MCDnpmZ3AuOHdOK9fOi6nPcIVS3voy8La1D9rU/+s\nS2FNot6B+mKWFi5nZfFqWsJBHIYDtz+Hil0+QnUpeGKdXDQ4i3HDu5GREhfpcuUb6MvC2tQ/a1P/\nrEthTSyjvqWBTw+sZGXJKorqSwGIJZ5ASTaNRZmYATcDzklm3PBuDM1PO+3prKTj6MvC2tQ/a1P/\nrEthTSwnLc3L5zs28NmBL1hduo5AuAWAWH8mtYUZhKoySPLEMmZoNt8Zkk1qYmyEK5ZD9GVhbeqf\ntal/1qWwJpZz5AbHH/SzunQ9nxV9wa6avQA4zBiC5Vn4i7MxmxLo0z2JCwZmMLJvOl63M5Kld3n6\nsrA29c/a1D/rUlgTyznRBqe4oYTPir5kRfEq6gL1ADhbkmgs9hGqzMLW4mFwz1QuGJjB0Pw0YnRR\nwlmnLwtrU/+sTf2zLoU1sZyTbXBC4RCbKrbyWdGXbKrYSsgMAeBoTqapJJ1QZSYuvIzo7WP0wAz6\nn5OM3abz284GfVlYm/pnbeqfdZ1OWHN0QB0iZ4zdZmeIbyBDfANpbGlkXdkmVpWuY1vVDpy5VThz\nt2FrSmFlaTqfzc8kwRnPef0yGDUgg57ZCSeck1RERMQqtGdNIup0/zqsDzSwtmwDq0rXs71qJyYm\nmEBDKoHyDEKVGSS4vAzJT2N4fhoDzkkhxqVDpWeS/rK3NvXP2tQ/69KeNekyvC4PF3e7gIu7XUBN\ncx1rytazumQdO409uLwVkLeZYFMSn1ek8slOH47mZAbkpTCsdxpD89NI8sZE+iOIiIi0i/asSUSd\n6b8Oq/zVrCldz/ryzeys2UPYDANghFy0VKUSrvERqkmjhy+NYb1b97p183kwDB0uPVX6y97a1D9r\nU/+sSxcYiOV05AanKehnW+V2NlVsY3PlNqqba1oXmBBuSCRU3Rrcku0ZDDwnhQHnpNA/L5kEj6tD\n6uls9GVhbeqftal/1qXDoCJHcDtiGZY+mGHpgzFNkwMNxWyq2Mrmim3sNPZg89bgzNlBU9DFitpk\nPl2RQvj9FLK9mQw8GNz6dE/CHaP/m4iISOToW0i6BMMw6ObNops3i4l542gKNrG1cgebK7aypfIr\nqhwl2FNKAKgIOllalcKSghSM+lTykrMZmNe6561ndoKmvhIRkbNKYU26JLfDzfD0wQw/uNetwl/F\n9qqdbK/exVdVO48Kb/uDTvaVpPDujmTsTWn0SMqmd04y+d0S6ZmdqJkURESkQymsSZdnGAZp7hTS\n3CmMzj4PgIqmSr6q3sX2qp3HhLe9ITu7KxL5V0ES4fokfK5MemdmkN8tkfxuiWSmxmHTBQsiInKG\nKKyJHEeqO4XR7hRGZ40EWsPb9upd7Kjeza7qvZTYS7EnVgJQC3zhd7NiWxLh1UnEBFLpmZxD727J\nnJOVQG5GPIm6aEFERE6TwppIO6S6U0h1p3DBwfDW2NLE3rp97KkpYFdNAbtq9uKPLYK0IkxgR9jG\nV+UJmAUJhBu9eEghNyGbHhkp5GbEk5fpJTUhVrcMERGRk1JYEzkNcU43/VP60D+lDwCmaVLWVM7u\nmgL21Bawo2ovRbYizPhqAFqAncD2+jjM0njCK+JxBRPJ9mbRKy2LvMx4cnxeMlPidAGDiIgcRWFN\n5AwwDIP0OB/pcT5GZZ0LQEuohaLGEvbXFbG/oYiCmgPstxXhj209/80E9gOFITvh7fGY673Q7CHR\nkUKmN528pAy6p8eTneohIyUOp0MhTkSkK1JYE+kgTruT3PgccuNz2sZM06QmUMv++mIO1Bext2Y/\nBbUHqLRXtO2Fa6B1L9yOFgNzVxzmJg9ms5d4WzLpcT7ykrLITUsmPdlNRnIcnliHDqeKiHRiCmsi\nZ5FhGCTFJJIUk8jA1L5t4y3hIOVNFZQ0llHcUEphTQkH6kuosJUTdJcCpTQBe4G9ITD3OzF3uTGb\n3diDHuIdiaTEJpMV7yM3OZ3slHjSk+NIiHMqyImIWJzCmkgUcNocZHkyyPJkgO/wuGma1LXUU9JQ\nRnFjKftqitlXU0JVcyX19lpMTy0AdQd/9obg83IwD7gwA26MFg9xRnxrmHMnk+FNISfRR0ZSPKkJ\nsSR4XLrNiIhIlFNYE4lihmGQ4IonwRVP7+Se0O3wsrAZpi5QT4W/ivKmCgqryyiqK6e8qZJaavA7\na8GooQloAkqBrQcfmPucmAE3BNzE4CXenkhyTBJpnmTSvclkJiSREu8myRuDN86pQCciEkEKayIW\nZTNsJMYkkBiTQM/EPM7PPHp52AxT3VxDRVMVxfUV7K8po7Shkkp/FfVmLX5HPaanlhZKqAQqgZ1B\noBrMKgNaXJiBWGiJwUUcbrsHryOepJhEUj2J9MzIxG3EkORp3UMXH+fEbtNFECIiZ5rCmkgnZTNs\npMQmkxKb3LpXrvvRyw8dYq30V1FaX0nhwTBX7a+lPlhHIw0EnHWYRg1BDh9qLQLww/K9YJpA0IkZ\ndEGLC5sZSwxuYu1uPA4P8c54Et1eUtwJpMTFk+KJJyEuFq/biSfWoduUiIi0g8KaSBd15CHWcxJy\nOT/72OeYpkljsIma5lqqm2soqauitL6KisYaGkL11Pjr8DsaCdj9hNwNADQf/Kk59CaBgz8HB8yQ\nHTPohJATW9iF3XThMmJx2WKJc7hxO9x4nG68LjfemDgSYuJIdHtIcntJdLvxuJ3Euuy6cEJEuoyo\nDGsfffQRjz32GKZpMmPGDGbNmhXpkkS6JMMw8Djj8DjjyPZmMiD18DKfL56ysrq2x6FwiIZgI3WB\neqr9dZTX11DRUEu1v5aa5noag000BZtoNvy0OP0EnX5MWx0haDuvri3gHTl4BNM0IOTADDqwmU5s\npgsHThy4cNpcuIwYYuwxxDpicNtjcTtjiXMdDH4uN94YN96YGDwxMcS5XMTGOHA57TonT0SiWtSF\ntXA4zK9//Wuef/550tPTufrqq5kwYQK9evWKdGki8g3sNnvbnrpu3ixIO/lrQuEQTUE/DS2N1Pjr\nqWioo7qpnjp/I/UtTTQEmmgK+vGH/DSH/LSYzbQQIOQMEDIaCdtq23bcHfvmB3/8x1+3aRoQtkHY\nDmE7hmnHhh0bDuw4sBsOHIYTOw6cNidOmwunzYnL5iTG7sJldxHriCHG6WwNhw4XMQ4nMU4XsQ4X\ncS4XbkcMbpeLGKcDp8OG3WZoj6CInLKoC2vr168nLy+Pbt1aL3u78sorWbJkicKaSCdkt9nxujx4\nXR4yPD5IPflrjhQ2w/iDzTQGm6htaqTO30RtcyP1za1BryHQRGPQjz/oxx9qJhAOEAy30BIOEjJb\nCBEkZIQI24OYRoCwESJshAh+U54KH/xpaX+dZtgA82AwNG0Ypg3DtGNgw8COYdqwGXZs2FoDo2HH\nfsRP62Nb62ObHYfhaH1sc+Cw2dt+7DY7TsOO3X5o7OByux2nzYbD5sBpd+C020mt89JQH8Bus+E8\n9B52Gw6bHaf94Gvsduw2G3abTSFTJIKiLqyVlJSQlZXV9jgjI4MNGzZEsCIRiVY2w0ac002c002a\nO+WMvKdpmrSEgzQG/DQGmmkINLf+3hKgqcVPU0sAf0szzcEA/mCA5nCAQDBAixkkGA7SEm4hGA4S\nMoOEzFBrIDSDhAkSJkTYFsIkDEbrY4ww2MxTKPDgD7TuOTwLTNOAQz+0/tvgOI8xMExb61jro4P/\nNE78u3H0+Nefd2i57dC4YRzx7oceH1p2+JUYBocuXzm8Do563uHfW9dw6LnG19djHFHZEe91aD1A\n2+MjQ23beo54zuE6Dr9X2+uNIz7ZEdnY+PorDYhzx9DUFIDjPe9rudrgyJqMry07/L/J15971PNO\nENaP+/wT5HrjqN+NowaPfYlxzO824/jLT/hnxHEW2DiFi5pO4e+TAZl55KX4Tv7E0xR1Yc00T2Gj\nJSJyhhmGgcvuxOV2kuSOPyvrNE2TkBmiJRQkEGrB39JCczCIvyVAIBQ8OB6kJRiiJdRCIBQkGA7R\nEgodDIitrw2FQwTNEKFwiJAZPvjv1p+wGSZshg8+DmOzQUswSJgwphkmjHnw32HMtt9NzIP/5ND4\nwTEO/oZhYhoHlxsHxwCMQ6ny0HJaHxvmMWHirDNP8LuV1Ea6ADnSv/Ym8dTlszvs/aMurGVmZnLg\nwIG2xyUlJaSnp3/ja3y+s7NBlY6h/lmXeici0vGi7iZHgwcPpqCggP379xMIBHj33XeZMGFCpMsS\nERERiYio27Nmt9v55S9/ycyZMzFNk6uvvloXF4iIiEiXZZg6SUxEREQkakXdYVAREREROUxhTURE\nRCSKKayJiIiIRLGou8DgVGgOUWuZPXs2S5cuJTU1lbfffhuAmpoa7rvvPvbv309OTg5PPvkk8fG6\nHUS0KS4u5qc//Snl5eXY7XauueYabrrpJvXPAgKBADfccAMtLS2EQiEmTZrEXXfdRWFhIffffz81\nNTUMHDiQP/7xjzgclv5K6NTC4TAzZswgIyODZ555Rv2zkPHjx+P1erHZbDgcDl5//fVT3nZads/a\noTlEn332Wd555x3effdddu7cGemy5BtMnz6dZ5999qixuXPnMnr0aBYtWsSoUaOYM2dOhKqTb2K3\n23nooYdYuHAhr776Ki+//DI7d+5U/yzA5XLxwgsvsGDBAhYsWMBHH33EunXrePzxx7n11ltZtGgR\n8fHxvP7665EuVb7BCy+8cNSdEdQ/6zAMgxdffJEFCxa09elUt52WDWtHziHqdDrb5hCV6DVy5EgS\nEhKOGluyZAnTpk0DYNq0aXzwwQeRKE1Owufz0b9/fwA8Hg+9evWipKRE/bMIt9sNtO5lCwaDGIbB\nihUrmDRpEtDau/fffz+SJco3KC4uZtmyZVxzzTVtY59//rn6ZxGmaRIOh48aO9Vtp2XD2vHmEC0t\nLY1gRXI6KisrSUtLA1oDQVVVVYQrkpMpLCxk69atDB06lIqKCvXPAsLhMFOnTuWiiy7ioosuonv3\n7iQkJGCztX4FZGZmavsZxR577DF++tOfts3PWVVVRWJiovpnEYZhcNtttzFjxgzmzZsHcMrbTsse\n4Nbt4UTOvoaGBu655x5mz56Nx+M54eTOEl1sNhsLFiygvr6eH//4x8c9ZUS9jE5Lly4lLS2N/v37\ns2LFCqD1++/r34HqX/R69dVX8fl8VFZWMnPmTHr06HHK/bJsWDudOUQl+qSmplJeXk5aWhplZWWk\npKREuiQ5gWAwyD333MOUKVO49NJLAfXParxeL+eddx7r1q2jtraWcDiMzWajuLhY288otXr1aj78\n8EOWLVtGc3MzDQ0NPPbYY9TV1al/FuHz+QBISUnh0ksvZf369ae87bTsYVDNIWpNX/9rcPz48bz5\n5psAzJ8/Xz2MYrNnzyY/P5+bb765bUz9i36VlZXU1dUB4Pf7+eyzz8jPz2fUqFH861//AtS7aHb/\n/fezdOlSlixZwhNPPMGoUaN4/PHH1T+LaGpqoqGhAYDGxkaWL19Onz59Tnnbaenppj766CN++9vf\nts0hqlt3RLcHHniAFStWUF1dTVpaGnfffTeXXnopP/nJTygqKiI7O5unnnrqmIsQJPJWrVrFjTfe\nSJ8+fTAMA8MwuO+++xgyZAj33nuv+hfFtm3bxs9+9jPC4TDhcJgrrriCO++8k3379nH//fdTW1tL\n//79+c///E+cTmeky5VvsHLlSp577jmeeeYZ9c8i9u3bx1133YVhGIRCISZPnsysWbOorq4+pW2n\npcOaiIiISGdn2cOgIiIiIl2BwpqIiIhIFFNYExEREYliCmsiIiIiUUxhTURERCSKKayJiIiIRDGF\nNRGRb2HlypXMmDEj0mWISCemsCYi8i1pXkYR6UiWnRtURORk1q9fz+OPP9423cs999xDfn4+M2bM\nYNq0aXzyyScAPPzww4wcORKABQsW8Oyzz2Kz2cjNzeVXv/pV27x9c+bM4Z133sFmsxEXF8crr7wC\ntM6b+vDDD7N27VpsNhtPPPEEPXv2jMAnFpFOyRQR6YRqa2vNqVOnmmVlZaZpmmZpaak5ZswYc8uW\nLWbfvn3Nt956yzRN01y5cqU5ZswYMxAImNu2bTMvvvhis7y83DRN03zyySfNe++91zRN03zzzTfN\n6667zmxoaDBN0zSrq6tN0zTNFStWmAMHDjS3bNlimqZpPv300+aDDz54Vj+riHRu2rMmIp3S6tWr\nKSws5Pbbb8c8OKue3W4nGAzicrm46qqrADjvvPOIjY1l9+7drFy5krFjx5KamgrA9ddfz5QpUwBY\nunQp3//+94mLiwMgMTGxbV09evSgX79+AAwdOpSlS5eerY8pIl2AwpqIdFr9+vXjxRdfPGps//79\nxzzPNE0Mw2j799fHD/1+IjExMW2/HwqEIiJnii4wEJFOafjw4ezZs4cVK1a0jW3YsAHTNAkEArz9\n9tsAfPnllwQCAXr06MHo0aNZtmwZFRUVALz22mtceOGFAIwfP55XXnml7fy36urqs/yJRKSr0p41\nEemUEhISePrpp/nDH/7A7373OwKBALm5ufz85z8nKSmJLVu28Le//Q2AJ554AofDQX5+Pvfffz+3\n3HILNpuN7t278+ijjwIwdepUSktLue6667Db7Xi9Xl5++eVIfkQR6SIM85v27YuIdDL79+9nxowZ\nfP7555EuRUSkXXQYVES6HN0XTUSsRHvWRERERKKY9qyJiIiIRDGFNREREZEoprAmIiIiEsUU1kRE\nRESimMKaiIiISBRTWBMRERGJYv8fxYRtNH2CY38AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55f690227f10>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAD+CAYAAACDUYl5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8U+XhP/DPycn90nsubSkFCgJiASf8QEERUFHwBq1f\ndZs3NnVT5vZ1Tiebut/c9KdzbEM3LxuT6TbdZMKUMsWBgIqiiFBQQChC6S29JG1zv53z+yNpaKBA\nWhrahM/79Qrn5OQkedKnNJ/znOc8jyDLsgwiIiIiGvQUA10AIiIiIkoOgxsRERFRmmBwIyIiIkoT\nDG5EREREaYLBjYiIiChNMLgRERERpYmUBrfFixfjggsuwFVXXXXcfX7xi1/gsssuwzXXXIPdu3fH\nt69cuRJz5szBnDlzsGrVqlQWk4iIiCgtpDS4LViwAMuWLTvu4xs3bkRtbS3Wrl2Ln//853jkkUcA\nAB0dHfj973+PFStW4LXXXsMzzzwDl8uVyqISERERDXopDW6TJk1CVlbWcR9ft24drr32WgDAhAkT\n4HK50Nraivfffx/Tpk2DyWRCVlYWpk2bhvfeey+VRSUiIiIa9Aa0j1tzczNsNlv8vs1mg91uh91u\nR2FhYXy71WqF3W4fiCISERERDRoDGtyOnm1LlmUIgnDMdgAQBOF0FYuIiIhoUBrQ4Ga1WtHU1BS/\n39TUBIvFApvNhoaGhmO2nwynXSUiIqJMpkz1G5woTM2ePRt/+9vfMHfuXGzfvh1ZWVkoKCjA9OnT\n8Zvf/AYulwuSJGHz5s247777TvpegiCgpYUXMaQrs9nE+ktjrL/0xbpLb6y/9GU2m3r9nJQGtx/+\n8IfYsmUL2tvbcfHFF+N73/seQqEQBEHA9ddfjxkzZmDjxo249NJLodPp8PjjjwMAsrOzcdddd6Gi\nogKCIGDRokUnvMiBiIiI6EwgyBl2fpFHHemLR43pjfWXvlh36Y31l7760uLGmROIiIiI0gSDGxER\nEVGaYHAjIiIiShMMbkRERERpgsGNiIiIKE0wuJ0GbrcbK1eu6PXz7r//B/B43CkoEREREaUjBrfT\nwOXqxMqVrx2zXZKkEz7vySd/C4PBmKpiERERUZpJ+cwJBDz33DNoaKjHwoXfgCiKUKs1MJlMqK09\nhL///V948MH70NLSjGAwgOuuuxFXXXUtAOC6667GsmUvw+v14r777kF5+UTs2rUDZrMV/+///Rpq\ntXqAPxkREVF6kWUZobAEWQZUKgUUaTYX+hkV3P65fj8+2dPcr685eYwF/zNr5An3+e53v4eDBw/g\nz3/+Gz777FPcf///4uWX/wmbzQYAWLz4EZhMJgQCAdx++82YMWNWbKaII79MdXWH8X//7+N44IGf\n4OGHH8SGDetx2WWX9+tnIaLBKSJJcHQGIAMw6VTQqkUIafZlcyKBUAROVwAKhQC9RgmtWoRSPD0n\nhGRZhj8YQbs7gE5PEP5gBFq1CJ1GCb1GCZ02Wh5RMTAnqGRZRkSK3SIyIpJ0zLpfAjo6vFAIQvSm\niN0EQFQIELq2xX5nJKnb60gyJElGOLaMRI5sj0jRgBO/RSIIhSWE4/ePLMPh2PNlGVLstaVurx9f\nyjJkOVoupaiAUuxaHn9dEKJTWgrAkXUh9g3ZbV0QBARCEfgCYXj94ejyqPWuxyLSkbkHVEoF1EoF\n1CoRapUIjVIBlUoBtVKERiVCrVJAVCigEAAIgBD9J7E8XWUBkJelwdyppSn7P3pGBbfB4uyzx8VD\nGwD8859/x3vvbQQANDc3o66uFmeffQ6AI79YhYVFKCuLBsTRo8egqanhtJaZzgwRSUIgGIE/dvP6\nw3D7Q/D4QvD4w/D4QvD6w/D4Q7HtYXj9IXgDYei1KujUIox6FYy6IzeTTgWDLnGbWiXGvmBw1BfN\n4Akj/mAYre1+tLT70O4OQKUUoVWL0GpEaNXRL/PorX+ChtcfQkvs/bpuzbFlW0cAUrdJbpSiEP3Z\n6tUwxX7eJr0aJp0KJv2R7VkGNUx6NQxa5YAFPVmW4fKG0NbpR1uHH45OP1o7/XB0BtDW4Udbpx9u\nX+iY56mVCug0yvhNrxET7qtVIlSiAKVSkfBlr+paVypi6wJCYQnt7iA6PAF0uINo9wTR4e5aDyAY\nOnG3FQDQqETojipDNNAJEBWK6FKM/h6L8ZsCCoUApSIaLkIRCYGghEAogmAo0m0Z3db9fvcARaeu\n6/fJqFPBnKODTqOEQhAQDEUQDEsIhqM/e38gjE5PBIGglPB/rjdEhYCLzy2GQavq508RdUYFt/+Z\nNfKkrWOng1arja9/9tmn2LZtK154YTnUajW+9707EQwGj3lO99OiCoXY4z50eoUjUizghI+EnVAE\ngWD01rXuD4YRCEUgCALUSgU0KjF6hBc7ktMoxfi6WhldCoIQPaqNHdGGI9FbKCwfWY9Ej3zDERmS\nfORIWZK7Hd1Guo6Au47Yo2UOhI6U0R+I3vcHIwhHTv4F1p1SFGDQqmDQqhCOSKhv9SMU7t1rHO1I\nC8GRo/L40XDs5xNdV8SPhtXdfoY9hapo4Iqu62IhSwbQ4Q5GA5IzFpY6ukKTH52e3v0fU4pC/P1E\nMfpFrhCiX+BdX+ZHLwUAHZ5oGTz+cI+vm2VQY3iRCeZsHUSFAJcvBJc3BJc3+rzDzSe/gElUCMgy\nqJGlV0eXBlXC/UKLB8527zEtJD2vSwiGYr+boegX3pF1CaFwtFUmGJbiLWnH+51QKRXIz9Ki1GpE\nrkkLGTJ8gUhC64jHH0Jrhw/hSP8GGEGI/mxteXrkGDXINqiRbdRApxbhC4bj5Thyi953+0Joae/f\n8igEARr1kRYeo04FpaiAKCaGQmW39XhgFAVotSp4vUFIsf//kgxEJBly978HcvQ+AIhi9PTgkdfp\n+p1UJN4XBaiV0b9Xqm5hWKXsdovdV3b9zvfwux5fj/1/EIRo+cKR2N+zsISwJMeW0b9p8fWwDFmW\nISN6ECDLOGo9uqFrXa0UodNGW0u7Wkz1GiVEhYCQFEZICiEkhRCMBKFX6mFUG45bL+FI9Hc9GI4g\nHJai7wsAsfLgmLIAwUgAeq2YstAGnGHBbaDo9Xp4vV4A0QruzuNxw2QyQa1W49Chg/j88109vkaG\nTSk7qEmxFoJ2VwBOdwDtrgDa3QE4XQG0u4OxZaDHVoJ0ohQFaFTRUJNlUMOcI8ZDjkYtQqsSodeq\nYNAp4+HMoFXCoDuyVCsV8ZacrvkSA6EI3N4Q3L4ebt5oS10oLB35komFAjm23vWl0/VY1x9Pty+E\nYCiAYCiCU/3f0PXl0dOXr6gQkJ+tRYklD+YcHcw5WuQaNQjFg3o0jPuD0dAbX+8W0kOBcMKpIalb\nADq2HhQw52hRVpwde7/oe5pzdDBn66BRiyf8LKFwBG5fGC5vMBbqgnB7Q+j0htDpCUZv3uiy0eHB\nIXvq57QUFUI8UBcVGJCfpY3dNMjP1iIvS4v8bC1MOlXSLYGhcATebmEqGIogHJG7HcBI8SAQih3c\ndB3YKEUB2UYNcoxqZBuiS5NeDYWi762QoXAEvmAk4QApEpESgu6RAygJEVlOOPWm6TotpxKhFIVe\ntYjKsgxv2AeH3wmHvx3F5hwYw8XQKrUnf3IS/GE/6tyNCEthlGUPgUrs/xDSn6/oCrqxs3U3drXt\nRoe7E6HOEEKREIJSt6XU89/rbLUJxcYiFBsL4zer3gxRIcZbcvU9RCVZluHwt6Pe3YB6d2P81uJr\ng6gQ8fi0n0Kv0vfjpzyCwe00yMrKRnn5BNxyyw1QqzXIy8uLPzZlygVYtepf+OY3/wdDh5binHPK\nuz3zyH/kTOrPMhCCoQhc3hA6vcF4a0WnNwiXp2s9FP9y6/QET3h6QqsWkWPUYIjZAL1WlRB0EpZq\nZcJ9WUasWT4SP4oLhqToqZFwdNm1XZZlqEQRSmW0xanraDe6LiScCup+VN79NM3RR75dR+iaWDhL\nVR8ijUqEJltEfnb/fIkcTZajR+oJP7/Y6Y5A6OgwFT42bMXWZRkoyNYeCUux9dwsTcr6M3UdmXdv\nydJqxFM6RaxSisg1icg1aZLa3x8Mx37PQ+iIhTpRKcLnCyb+zgjdT/0p4q0lYux0ZFcfIHW31peu\n1plTCUQn+pzZShHZhsFxUZZKKUKlPBKqI1IEgUgQgUggvgxFAvCHo/eDUhBKQQmVQgW1rIIqrIIq\ntlSLKqgVKqgUKqhEFZSCCFfIDYe/PRbOnEetOxGIJLYICxBg1uejxFiMElMxhpiKUGIsPmGLEgC4\nQx7UuRpw2FUfvbnr0eJtQ6xNCWpRjbF5Z6E8fyzOKRgLk3pwjHTQ5nNgR+vn2NGyCzXtB+PlVQoi\nVN1+njqNNvpzVXT7OYvR+66gC3XuRnzh2IsvHHvjr61UKFGot8QCnQ3FxiKoRTUa3I2o9zSiztWI\nBk8jfGF/Qpl0Sh1G5gzHqNyyfgvRPRHkDGvKaWlJ/dEkpUZXi013kizD0eFHk9OLpjYvmhxe2B1e\ntHuC8daZrpYaWY7uL8uJ28MRGYFQ5KTvr1YqYNKrkWNSI8eoQa5RgxxTbGlUI8ekQY5RA52Gxzs9\n6an+KD1kat1JsoSOQCdUCtVJA0xvNHrseK/+Q3zethf+sD8a0qSeT3X3N51SizxtLvK0OcjT5iJX\nk4OIMoi9zQdx2FUPX9iXsH+uJice5IaaigEAh131qHM1oNZVD2eg/ZjXLzFG9xcgYFfbbti9LQCi\n4XBY1lCMLzgb5xSMRaHBesJGBVmW0RHsTGiRqnc3otXnQJ42B1a9BVa9GTaDBVa9BTaDGTql7riv\n1eBpwo6WXdjR8jnq3A3xMg3PHooJ5nMwvmAcLPqCXv9M3SEPGtxN3crYgAaPHeHj1KkAARa9OR7q\nio02DDEWIUeT3etGFrPZ1OvyMrjRoOALhOGNyNhT04omx5GAZnf6euwfo9coE66cEmJXUwkC4p3c\nu9ZFhQCTXo2s7p229dFTJSZDdD1Lrz7pKSk6sUz98j8TpHPdybKMzqAbzd4WtPha0ext7bZsQ0gK\nQYCA0bkjMcl2Liaaxx03HJxIWApjR8suvFf/Efa1HwAA6JU6ZKlN0IgaaJQaaEQ1tGJ02X2bRtRA\nrVAhIkcST991W4akxHWj2hgPZ/na3HhY66nsXfUnyzLa/E7UxVrPat3RpSvYcz9Ik8qIElNxt1sR\n8rV5x4QPu6cZO9t2Y2frFwmtWwXaPJQXnI3ygrNRmlWCZl8L6t1NsdOH0aUn5E14LY2oRoEuH05/\nO7xHhUwgeurSqrfAarDAprcgV5uNmvaD2NGyC61+BwBAFESMzh2JCeZxKC8Yh2xN78PPyUSkCFp8\nrah3N6LO3YhQJIQiow3FxkIUGqxQi/3T8svgBga3dCDLMto6/dhf34H9ddHb4RY3jv5N1KhF2HL1\nsOXrYc3VwZavR2GeAZZcHVu9BqF0/vI/06VD3QUiQdi9zbB7WqJLbwtavK1o9rUec9oQiAYEi94M\ni64ADr8TX3XWAoieBjsnfywmWydiXP6Yk/bfcvid+KB+Cz5o/DgegMbkjsKFxVNRXnA2RMXAH/Cd\nrP46Ap3xU6Ey5HhQy1Zn9bqFyB3y4Iu2vahu/QK72/bCHwkcd98CXX6839iQ2DJPmwuFoIAsy3CH\nPGjyNMfrs2vd4W+Ph8MuGlGNcfljMMF8Dsblj4EuhaciTycGNzC4DUbhiITDze5oSKuP3pyuI//Z\nlaICwwtNGDsiHzk6Fax5+tiVXmr27Usj6fDlTz0bLHUnyzJcIfeRL3NPC5q8zWjyNB9zSg8AVAol\nzLoCWPQFsaU5vp6lNib8/Wj1tWGrfTs+afoMTd7oeJ5aUYuJlnMw2Xouzsotg0KI9m2UZAl7HPuw\nqf5D7GrdDRkydEodzi+chOnFU2HVm0/PDyRJA1V/YSmMfe0HsLP1CzS67bAYzLGAVoQig7XP/byC\nkSCava1o8jaj1efAEGMhRueOTMlFEgONwQ0MbgMtFI6gsc2L+hYP6lrd+KqhEwcaOxPGScrSqzBy\nSA5GFmdj1JBsDLWaoFIqBs2XB/UN6y99paruZFmGL+yHN+yFJ+SFO+SFNxRd98S2dd13hzxo8bUd\n00cLALLVWfF+UFaDGbZY36hsTVY8bPWmTPXuRmy1b8dW+/Z4IMxSm3CeZQKy1CZ80PgxWn1tAICh\npiG4qPh8nGed0G+nx/ob/++lr74EN55voj4JRyQ0ObxoaPWgrsWDhlYP6lvcaG73JZzyFAAUmQ0Y\nWZwdD2rmHB1b0mhARaQIvGHfoLlC7kRkWYY/4ocr6IYoKJGtMUGpGFx/uv1hP+zeloTTXU2eZrT4\n2hCRT35hEBA9hZmvzcNZuWXRDut6C2wGCyx6c7+eFhMEAUNMRRhiKsLVZZfjQMchfGL/DJ/Zq/Fu\n3fsAoi15Uwsn4aLi81GaVdJv703UHwbX/34atDo9Qew80IbPDzpQa3fD7vAeM2SGQavEyOJsFJuN\nKC4woLjAgBKrMaUDERL1hiRL2NZcjTcPvA2H34kbR1fggqLJA1aWRo8dDr8TrqAH7pAbrqA7vu4O\nuuEKeeAOuhHuFn4ECDCpjcjRZCNXk43s2DJHm40czZGbWlQhIkViA46GEYyEEJKCsQ7wYQSlYLwj\nvN6jhtsViF3Uo4BCUEABAQpBEbsvQIHoMixH0BI7jdV1KrM90HHM59OKWgwxFiFLY4ReqYdBpYdB\nZYBBpYsulXroVXoYVdGlWpH8mG79RSEoMDJnOEbmDMd1o67GHsc+dAZdmGA+B4YUjcFFdKoY3Aah\nSy+9CO+8s2lAyyDJMg42ulBd04qdB9pwsNEV7yqqVYsYZjOhqMBwJKSZDcg2sE8aDU6yLGO340u8\nUfMfHHY3QBREqBVq/G3Pa/CEPLi09OLTUo42nwN7nPuw17Efe5374Q55jruvWlTDpDJiiKkYJrUB\nRpURYSmM9kAHnIEONHqaUOuqO+7zFYICknxqs1gkI0eTjTG5o2JXAZpjwzpYkKU2pdXfA6VCiXMK\nxg50MYhOisFtEBqoP3ZuXwiff+VAdU0bdn3VBpc3OtK0qBAwemgOysvyUT4iH8UFhrT6g0yp0xl0\nYXPDJ9jWvAMGjQ7ZyhwU6PJit3wU6PKQpTb1uh9Sf/qqoxZv1PwHX7bXQICAydZzceWIyxCWwnh6\n+5+wqmYNPCEvrim7ot9/rz0hL7501mCPcx/2OPbF+00B0cAzxXYeCg1WGNVGmFQGmNRGGFVGmNSG\nk/ankmUZnrAX7f6OeJhrD3Sg3d8BZ6AdwUgoNhBpbNBXUQ2VQhkffFStUEMlKqFWqJCTZUCHywdZ\njs7PKEGCJEuxMRGPbJNlKTbQawGsejOsenNKBxolomMxuJ0Gzz77NGy2QsyfXwkA+POfX4AgCNi+\n/TO43S6Ew2Hcfvt3MH36jNNaLlmWcbjZjR01bdh5oA019R3x/mnZRjUuHF+I8WX5OHtYHoffoDhZ\nllHTcRCb6jZje8suROQIlAolJI/UYwtPV9+leKDT5sFqsGKIsTClrTJNnma8eeAtbG+JTiM3Ln8M\nrh5xOYaYiuL73Pu1u/DMjj/indoN8IQ8uGH0glMa3kGWZexv/wpfOPZir2M/al118WENtKIW4wvG\nYXTeyGgLld58Sp9dEAQYVQYYVYaEz9QX7NxOlD7OqG/j1/evxmfNO/v1Nc+1lGPByCtPuM8ll1yG\n3/3u1/Hgtn79f7FkydO4/vpvQK/Xo6OjHXfeedtpCW6hcAS7DzmxY38bdtS0wtEZHZZDEICy4myM\nH5GP8WX5KLEY2ao2QLrCz0C2UvXEF/bjk6Zt2FT/IRo9dgBAocGKC4vPx/+xfQ3F1jx8WVeHVl8b\n2nwOtPodaPW1odXnQJvPAXtsCIbujCpDwhyBxcZC2AxWqE6h873T3441X72DDxu3QoaM4VlDcU3Z\nXIzKHXHMvvm6XNz7tbvwhx3LsLnxE3jCPtx29o19Gnag1lWHf+17E/vbvwIQHSS0LGcYxuSOwui8\nUSg1DRkUY34RUXo7o4LbQBk1ajTa29vR1tYKp9OJrKws5OcX4He/ewo7dmyHQiGgtbUFTqcDubl5\nJ3/BXmp3B1Bd04bt+1rxxSFHfGgOg1aJqWdbMb4sH+eMyIdRx4sIUqXV14Zmb2vCMAjdh0Lovt0X\n9kGlUKHQYEGhwYZCgxWFBiuKjDbkanJOe6CudzdiU/2H+KRpGwKRIERBxHmWCbiw+HyMzBkeL4+o\nEOOtaj3xhnxo8zvQ4mtDY7fpZfY6o/29uigEBax6czzI5Wiyo53lYx3m453lBQUUiE5yL8Y60e9s\n/QIb6z5ASArDZrDi6hGXY3zB2Sf8mZnURtxz7p14ofov2NGyC3/Y8WfcMf6WpK9k7Ah04o0Db2FL\n46eQIaO84GxML5qCkTkjoFUmN38oEVGyUj6O26ZNm/DYY49BlmVUVFTgjjvuSHi8oaEBixcvhsPh\nQE5ODn71q1/BarUCAMaOHYsxY8ZAlmUUFRXhD3/4w0nfb7A29y9b9jyys7PR1taGggIz9Ho9Pvpo\nMx555BdQKBS47rqr8fTTL8Bms+Gyy2Zg7dqNfX4vWZZxyO7Cjv1t2L6/FYeajvxMCvP1mDCyABPK\n8jFySHbKJtPui0w7XSPJEna27sbGug8SgklPlIIYv+pOr9LBHw6gydt8zFx5WlETD3KFxmioM+vy\noRW10IhqKBXKPgU7WZYRkkLwhf3whX3whf1o9rbi/YYtONBxEEB0zsPpxVNxQdFkZKmPHXuor/Xn\nC/vR6GlCnSs6gXN9bALnnkbDT0auJgfzRlyGKbav9arVMhQJ4cUvXsGOll0oMRXj7gnfOuFwIcFI\nCOsPv4e3D61HMBJEkcGGilFXYUzeqD6VeyBl2v+9Mw3rL30NunHcJEnCo48+iuXLl8NisaCyshKz\nZ89GWVlZfJ8nnngC8+fPxzXXXIMtW7bg17/+NZ588kkAgE6nw8qVK1NZxNNm1qxL8eSTv0BHRwee\neeYFrFv3DnJz86BQKLBt21Y0NTXG9+1LlpZlGQcaO7F1TzO27mlBW6cfQPTCgrGluZg4sgDjR+bD\nmstL3FPNHfLgw4ZPsKn+Qzj8TgDAqJwRGJM36kg4U+oShkboaSiEiBRBq9+BRncTGj12NHiiy0Ou\nuvj0PUdTCIroHInxORNjcyXG5kxUCkr4I/6EgBYdINXXY/80AQLOzhuNi4acj3H5Y1Jy+lan1GJE\n9jCMyB4W3ybJEtp8TtS7G+AOebp1ju/qLB9dj8hSvNO8JMvI1eZgqu28Pp3qVIkqfGvcN/Dq3tex\nufETLNn2ByyacDvydbkJ+8myjG3NO7Cq5j9w+J0wqgyoGHklLij6P4Pu9DYRZZ6UBrfq6mqUlpai\nuLgYADBv3jysW7cuIbjV1NRg8eLFAIApU6bgrrvuij+WSZM6DB8+Al6vF2azFXl5+bjsssvxwAP3\n4pZbbsSYMWNRWjo8vm+yLSayLONAQyc+2dOMT/c2oy3WX02nEXH+OCvOHWXG2cPyoNfyjPjpcNjV\ngE11H+AT+2cISWGoFSpMK5qCGUMuQLGxsNevJyrE+JV7E1Ee3x6Wwmj2tqIxFuQc/nYEIkEEIoHY\nLQh/OAB3yIM2vwOho1rtuqgVKuiUWhhVBph1+dApddAptbGbDgaVHhPN5TDr8/v8M+krhaCAWZ9/\n2t9bVIj4+phKGFQGvFO7IRreJn4bhYboWYBDnYexYt+bONBxEEpBxKVDL8acYTP7NGk5EVFfpPQb\n3W63o7DwyBeW1WrFzp2JFweMGTMGa9euxU033YS1a9fC6/Wio6MD2dnZCIVCqKyshFKpxLe//W1c\ncsklqSxuyv3lL6/G17Ozc/Dcc3/ucb8TnSaVZBkH6mNh7cvm+MUFOo0SF5xjw6QxFowblgeVkkf+\nAOAOenCwsxZhKQwJia01Uqy1RurWigMABpUeJpURRnXX8AyG445UH5Ei2N6yCxvrPkBN7JRigTYP\nFw25AOcXToI+BYN4KhVKFBltKDLakto/IkUQlKJhLixF4uGMHeV7JggCrh05FwaVHqtq1uA3nz6L\nb469DttbdmFL06cAgInmc3Bt2bwBCbVEdGZLaXBLpsXs/vvvx6OPPoqVK1di0qRJsFqtEMXoF8q7\n774Ls9mMw4cP45ZbbsHo0aNRUnJmTj/S0OrBhu31+HRvS3yCdp1GiWmxsHY2w1qcN+TFjpbP8Wnz\nDux17u+XQUh1Sh1MKkN0vK3YmFsqhQrbmqvREewEAIzNOwsXD5mGs/NHD6pTZqJChE6hY6tQL11a\nejEMKgP+vmcFnt/5FwDAEGMRKkZdhbNyy07ybCKi1EhpcLPZbGhoaIjft9vtsFgsCftYLBY8/fTT\nAACv14u1a9fCaIx2CDabzQCAkpISTJkyBbt37z5pcOtLR7/BrNnhxd/X7sG7Ww9DkgGDToXZk0sw\nfUIxJowyD+qw5gv5IUOGXpV8YOhr/XlDPmytr8bm2q3YYd+NiBSdIqgsrxTnFZVDr9LFrkwUjlyh\neMwteoraHfSiM+BCh9+FjoALroALHX43OgIutHQeSjgg0Sm1uGLUTMwZNQNFJmufyp5JMu3/3zXm\nWSgqyMeq3W9j9ojpuHjYVCgG0QU9/SnT6u5Mw/o7c6Q0uJWXl6O2thb19fUwm82oqqrCkiVLEvZx\nOp3IyYkOcfD888+joqICANDZ2QmtVgu1Wg2Hw4Ft27bh29/+9knfM1OurOn0BLF680Fs2F6PcERG\nsdmAa6cPx4SRBVCK0S+Odufxp8sZKLIsY1/7Abxf/xF2tOxCWI4gW511ZDocgwU2vQVWvRk5muyE\n/ny9vTLKHw5gV9tubLPvwOeOvfErMIcYi3CeZQK+Zh2PAl0fTmWdYBQISZbgDfngDrnhCflQbLRF\nR473Ay3+zPjd66tMvbJtmHoEfjDhuwCAtrbB93+uP2Rq3Z0pWH/pa9BdVSqKIh566CEsXLgQsiyj\nsrISZWVlWLp0KcrLyzFz5kx8/PHHWLJkCQRBwOTJk/Hwww8DiF608PDDD0MURUiShDvvvDPhooZM\n5QuE8fZbPxaKAAAgAElEQVTHtXj748MIhCIoyNZi/oUjMOVsKxSKwTsgrjvowUdNW/FBwxY0e1sB\nADa9BbnaHNi9LfjSuR9fHjUkhkZUw6q3xOc2LHYVoKPTmzDlzvGm3WnytmBX626EpOi0XIUGazSs\nWcbDarAcU77+ohAUMKoNMKoNKXsPIiKi40n5OG6nW7oedYTCEaz7tB5rPjoEty+ELIMaV10wDDMm\nFsVb2Ppb1zhj25p3IEttwhBjEYqMhbAZLEmNXN9T65pSocS55vGYXjwFZdnD4i1qgUgQzd4W2D3N\naPI2oym23uxrPWassmRZ9AWxsDYh6Y76lDo86k9frLv0xvpLX4OuxY1OLiJJ+GBnE/79/ldwugLQ\naZRYcNEIXDqpBBp1aq76C0aC+KhxK9Yffg8t3Sa97qIQFLDpLUdNRVSELHV0Gqzjta5NK56CKbbz\nYOjhSkqNqEaJqRglpuKE7V3jdTV57RC1MtzuQGK/M0T7pMVHy0d0aVQbYdNbOC0XERGdURjcBtD2\nfa3457v70eTwQqVU4IopQ3HF1NKUTT3VEXBhU90HeK/+I3jCXigFERcUTsaFQ85HKBJGvbshPg1R\nvacJDZ4mfGL/LP58o8oAi74AtZ118da1ydavHdO61hvdx+viUSMREdGJMbgNALcvhL+98yW2fGGH\nQhBw8cQiXDVtOHJNqZnXsN7diPW172Gr/TOE5QgMKj2uGDYbFw25IGHqorKcYfF1SZbQ6nOgwd2I\nOndjfHmg49BJW9eIiIgoNRjcTrNP97bg5bf3oNMbwoiiLNw2dyyKC/q/o7ssy9jj2Id1hzdht+NL\nANE+YbNKLsIU29egFtUnfL5CUMCiL4BFX4CJliOj9ocioT7Ph0lERESnhsHtNHF5g/j7f/dhyxd2\nKEUFrptZhjmThyZ9pWibz4mdrV8gJIUQliKIyOHYMoKwFEFYCsfWo8smbwuaPHYA0XkyZw+9qF/m\nmuzLHJBERETUPxjcToPurWxlRVlYOG8sCvOTb2X70rkff9z5MrxhX9LPUQgKTLJOxKySC1GadWbO\nNkFERJRpGNxSyOUN4m/vfImPdzdDKSrwPzNH4rLJJb0aj+39+o/wjy9XQYCA+SPnwaa3QFSIUAoi\nlAplbF0Z3yYqotvVCjXUbB0jIiLKKAxuKfLp3ma8/PbePreyRaQIVu6vwrt178Og0uP2c27GqNwR\nKSwxERERDXYMbv2seyubStm3VjZf2Ic/f/53fNG2FzaDFd8df2vfpm4iIiKijMLg1o8a2zx46tXt\ncLoCKCvOwsK5vWtlA4BWXxuerV6OJo8dZ+eNxsJzvg6dMvlJ2omIiChzMbj1k/pWD5565TN0eIK4\n9sLhuPL8Yb2eW3Sf8wD+uOsleEJezCyZjvll8yAqUjN7AhEREaUfBrd+UNfsxq9e/Qwubwhfv2QU\nLpnU+6s4Nzd8glf3vg4ZMm4YvQAXFk9NQUmJiIgonTG4naJauwtPvbodbl8IN80ZjZnnFp/8Sd1I\nsoRVNWuwrnYT9Eodbi+/CWfljkxRaYmIiCidMbidgq8aO7HkH9vh9Ydx2xVjMOWcArx9cD1afW3Q\niBpolBpoRDU0ogZaMbau1EQfE9VQKZT4177V2NW2G1a9Gd8ZfyssevNAfywiIiIapBjc+qimvgNL\n/rkd/mAEC+eNRXZhO36x5S9o8zt7/VpjckfhW+d8E3oVL0IgIiKi42Nw64N9de34zT93IBiS8PUr\nSvAF/ovt1TuhEBS4dOjFmFo4CSEpBH84gECk6xZEIBI8ZptFb8alQ2fwIgQiIiI6KQa3Xtpb68Rv\nX6tGOBLG9NlBVDn/gkAkiBHZpbhh9AIUGwsHuohERESUoRjceuGLgw4sXVENSedE4fgafNLZDINS\nj8oxV2Nq4aRTnsCdiIiI6EQY3JK060Abnv73NghD9kJlqUVbCJhiOw/zR86DSW0c6OIRERHRGYDB\nLQmf7WvBcxvXQjx7DwRVAFa9BTeMno+zcssGumhERER0BmFwO4kvm5rxwq7lUI5ohSgoMXf4HFwy\ndAaUCv7oiIiI6PRi+jiB2s56PFP9RyiyvSjWDMPt514Ps56TvRMREdHAYHA7jk/t2/GXz/+JiDIM\ns28CHpz5dQhC7+YeJSIiIupPDG5HkWQJbx54G2sPvQtElJAPTcL3r7+GoY2IiIgGHINbN76wD8s/\nfwW72vZALZnQ+fkE3Dj9XOSaNANdNCIiIiKkfOCxTZs24fLLL8ecOXPwwgsvHPN4Q0MDbr31Vlx9\n9dW4+eabYbfb44+tXLkSc+bMwZw5c7Bq1aqUltPuacavtj6DXW17UKIbjo7P/g9Kcwox62tDUvq+\nRERERMlKaXCTJAmPPvooli1bhtWrV6Oqqgo1NTUJ+zzxxBOYP38+3njjDdx999349a9/DQDo6OjA\n73//e6xYsQKvvfYannnmGbhcrpSU8/O2PfjVp8/A7m3BzOIL4aweD0FS4ZbLx0Ch4ClSIiIiGhxS\nGtyqq6tRWlqK4uJiqFQqzJs3D+vWrUvYp6amBlOnTgUATJkyJf74+++/j2nTpsFkMiErKwvTpk3D\ne++916/lk2UZ7xzagGd3vIiQFMYtZ98AhX0cWpwBXDqpBKU2U7++HxEREdGpSGlws9vtKCw8Mnen\n1WpFc3Nzwj5jxozB2rVrAQBr166F1+tFR0dHj8/tfhr1VAUjQSz/4hWsqlmDbE0W7v3adzFEORr/\n+egQ8rI0uPbC4f32XkRERET9IaXBTZblk+5z//334+OPP8aCBQuwdetWWK1WiKLY43P768pOb8iL\nJduexVb7dozILsX9k+5BiWkIXnprDyKSjG9ccha0al63QURERINLStOJzWZDQ0ND/L7dbofFYknY\nx2Kx4OmnnwYAeL1erF27FkajETabDVu2bInv19TUFD+leiJm88lPby77tAqHXfWYMWwq7pj0dahE\nFd7Zcghf1nVg6jk2XDZtRLIfkfpZMvVHgxfrL32x7tIb6+/MkdLgVl5ejtraWtTX18NsNqOqqgpL\nlixJ2MfpdCInJweCIOD5559HRUUFAGD69On4zW9+A5fLBUmSsHnzZtx3330nfc+WlhNfwFDvbsTa\n/Ztg1ZuxYNjVaHf40enpxLI3dkGjFlF50YiTvgalhtls4s8+jbH+0hfrLr2x/tJXXwJ3SoObKIp4\n6KGHsHDhQsiyjMrKSpSVlWHp0qUoLy/HzJkz8fHHH2PJkiUQBAGTJ0/Gww8/DADIzs7GXXfdhYqK\nCgiCgEWLFiErK+uUyiPLMlZ8+QZkyKgYdVV8vtF/rN8Pjz+MG2ePQl6W9pQ/NxEREVEqCHIyHdHS\nyImOOrY378Qfd72McfljcNeEhQCA3Qcd+NWr21FqNeGhWyZx+I8BxKPG9Mb6S1+su/TG+ktffWlx\nS/kAvINFMBLC6/tXQxREVIy6CgAQCkfw0tt7IQjALVeMZmgjIiKiQe2MCW7rajehze/ExSXTYNWb\nAQBVHx6C3enD7POGYJjt1E7DEhEREaXaGRHcnP52rD20HiaVEVcMuwQA0NjmQdWHh5Br0mD+hbyK\nlIiIiAa/MyK4rapZg6AUwtVlV0Cn1EKWZbz01l5EJBlfv+Qs6DQcs42IiIgGv4wPbjXtB7HVvh1D\nTUMwtfA8AMC+ug7sPdyOCWX5+NpZBQNcQiIiIqLkZHRwk2QJr+37NwDgurOugUKIftyWdh8AYOKo\ngn6bjYGIiIgo1TI6uH3UuBWHXfWYbD0XI7JL49udrgAAINekGaiiEREREfVaUsGtqqoK4XA41WXp\nV76wD2/UvAW1qMa1I+cmPNbujga3HCODGxEREaWPpILb6tWrMWvWLPzud7+D3W5PdZn6xZqv/gtX\nyI05pbOQo8lOeKyrxS2HLW5ERESURpIKbs8++yxeeeUVhMNhVFRU4J577sFHH32U6rL1md3TjA11\nHyBfm4fZJRce83i7OwBRIcCkUw1A6YiIiIj6Juk+bsXFxfjhD3+IpUuXorq6Gt/97ndx1VVXYevW\nraksX5+s2P8mJFnCglFXQiUeG87a3UHkGDW8MIGIiIjSSlIDmAWDQaxZswavvPIKIpEIfvCDH2Du\n3Lmorq7G/fffj/Xr16e6nEnb1bobX7TtxejckZhQMO6YxyVJRoc7iBFFnCmBiIiI0ktSwW3WrFmY\nMmUKfvzjH+Pcc8+Nb580aRLOP//8lBWut8KRMP61700oBAUqR13dY4tahycISZbZv42IiIjSTlLB\n7fXXX4fFYunxsV/+8pf9WqBTsWbfu2j2tWLGkAtQZLT1uE/XFaW5vKKUiIiI0kxSfdxWrVqF9vb2\n+H2n04k//elPKStUX/3r8zUwqPSYN/yy4+7THr+iVH26ikVERETUL5Iexy0nJyd+Pzc3F6tXr05Z\nofrKF/bjqhGXw6DSH3cfJ1vciIiIKE0ldapUluVjtkUikX4vzKl68rLF0IeyT7gPZ00gIiKidJVU\ni9uwYcPw4osvQpZlSJKEP//5zxg6dGiqy9Zrw3JLTjrEB2dNICIionSVVHD7yU9+gnfffRfjx4/H\nxIkTsXHjRjz88MOpLltKtHPWBCIiIkpTSZ0qtVqteOmll+D1egEAev3x+5ANdk53EHqNEhqVONBF\nISIiIuqVpIIbALhcLnz11VcIBALxbZMnT05JoVKp3RVg/zYiIiJKS0kFtzVr1uCJJ55AZ2cnLBYL\namtrMWbMGKxcuTLV5etXgVAE3kAYwwtNA10UIiIiol5Lqo/bc889h9dffx2lpaV4++238ac//Qnl\n5eWpLlu/Y/82IiIiSmdJBTelUon8/Pz4ECDTpk3Dzp07U1qwVIjPmsDgRkRERGkoqVOlarUasiyj\ntLQUL7/8MoqLi+MXKqSTrjHcOBQIERERpaOkgtv3v/99uN1u3HffffjZz34Gl8uFRx55JKk32LRp\nEx577DHIsoyKigrccccdCY83NjbigQcegMvlgiRJuPfeezFjxgzU19dj7ty5GDFiBABgwoQJ+NnP\nfta7T3cUzppARERE6eykwS0SiaC2thbnn38+TCYTli9fnvSLS5KERx99FMuXL4fFYkFlZSVmz56N\nsrKy+D7PPvss5s6dixtuuAE1NTW4/fbbsX79egDA0KFD+/UCCCf7uBEREVEaO2kfN1EU8Y9//KNP\nL15dXY3S0lIUFxdDpVJh3rx5WLduXcI+giDA7XYDADo7O2G1Wvv0XslodwcBsI8bERERpaekLk6Y\nMmUK3nrrrV6/uN1uR2FhYfy+1WpFc3Nzwj6LFi3Cv//9b8yYMQPf+c538NBDD8Ufq6urw4IFC3DT\nTTdh69atvX7/o7W7AlAIArL06lN+LSIiIqLTLak+bitXrsSLL74IrVYLnU4HWZYhCAI+/PDDEz6v\np8npj1ZVVYWKigrceuut2L59O370ox+hqqoKZrMZGzZsQHZ2Nj7//HPcfffdqKqqgsFgOOHrmc3H\nH6Ot0xtEbpYGVmvWSctFA+NE9UeDH+svfbHu0hvr78yRVHD717/+1acXt9lsaGhoiN+32+2wWCwJ\n+6xYsQLLli0DAEycOBGBQAAOhwN5eXlQq6MtY+PGjUNJSQkOHjyIcePGnfA9W1pcPW6XZRltHX4M\ntZqOuw8NLLOZdZPOWH/pi3WX3lh/6asvgTupU6XFxcU93k6mvLwctbW1qK+vRzAYRFVVFWbPnp2w\nT1FRETZv3gwAqKmpQTAYRF5eHhwOByRJAgAcPnwYtbW1KCkp6e3ni3P5QohIMnKMPE1KRERE6Smp\nFrepU6dCEIRjtp/sVKkoinjooYewcOFCyLKMyspKlJWVYenSpSgvL8fMmTPxwAMP4Kc//SmWL18O\nhUKBJ554AgCwdetWLF26FEqlEgqFAj//+c+RldX3U5xdsybwwgQiIiJKV4KcREe0+vr6+HogEMCb\nb74JpVKJu+++O6WF64vjNRdX17Tit69Vo2LGCMw7f9jpLRQlhc396Y31l75Yd+mN9Ze+Tsup0hEj\nRuD73/8+Nm7c2Os3G0icNYGIiIjSXVLB7WiHDx9GW1tbf5clpTj4LhEREaW7XvdxkyQJ4XAYP/nJ\nT1JasP7WzumuiIiIKM31ejgQpVKJgoICiKKYskKlAmdNICIionSX1KlSj8eD3NxcFBcXw2q1IhAI\nYN++fakuW79yugLQqERo1ekVOImIiIi6JBXcfvzjH0OlUsXvK5VKPPDAAykrVCo4XQHkmDQ9DmtC\nRERElA6SCm6RSCQhuKnVakQikZQVqr+FwhLcvhByOfguERERpbGkgptSqcThw4fj92tra9Oqj1uH\nm1eUEhERUfpL6uKERYsW4cYbb8SMGTMAABs3bsQvfvGLlBasPzl5RSkRERFlgKSC28yZM/HXv/4V\nH3zwAQDgjjvuQGlpaUoL1p+6rihlixsRERGls6SCm8PhQFFREb7xjW8AAEKhEBwOB/Ly8lJauP7S\nNfguW9yIiIgonSXVx+3OO+9MuBghFArhO9/5TsoK1d/aOWsCERERZYCkglswGIROp4vf1+v1CAQC\nKStUf2MfNyIiIsoESc9V6nA44uttbW2QJCklBUqFrha3bA4HQkRERGksqT5uN910E2688UZcc801\nkGUZb7zxBm6//fZUl63fON0BZOlVUIpJ51QiIiKiQSep4FZZWYmhQ4diw4YNAIBf/vKXmDRpUirL\n1W9kWUa7KwBbvn6gi0JERER0SpIKbi6XC++99x727dsHv9+PXbt2AQBeeumllBauP/gCYQTDEvu3\nERERUdpL6tzh4sWLIYoiDh48iOuvvx6iKGL8+PGpLlu/iA8FwitKiYiIKM0lFdwOHTqEH/zgB9Bq\ntbjyyivx/PPPY+vWrakuW7/ouqI0hy1uRERElOaSCm5qdfRqTJVKhfb2dqhUqoSrTAezdhdnTSAi\nIqLMkFQft2HDhqG9vR1XXXUVrr/+ephMJowbNy7VZesX8THcGNyIiIgozSUV3J566ikAwG233Yby\n8nK4XC5cdNFFKS1Yf4nPmsBTpURERJTmkgpu3aXLMCBdeHECERERZYqMH5G23R2AUlTAoO11RiUi\nIiIaVFIe3DZt2oTLL78cc+bMwQsvvHDM442Njbj55psxf/58XHPNNdi4cWP8seeffx6XXXYZrrji\nCrz//vt9en+nO4AcoxqCIPT5MxARERENBilthpIkCY8++iiWL18Oi8WCyspKzJ49G2VlZfF9nn32\nWcydOxc33HADampqcPvtt2P9+vXYv38//vOf/2DNmjVoamrCbbfdhrVr1/YqgEUkCZ2eIEYWZ6fi\n4xERERGdViltcauurkZpaSmKi4uhUqkwb948rFu3LmEfQRDgdrsBAJ2dnbBarQCA9evXY+7cuVAq\nlRgyZAhKS0tRXV3dq/fv9IQgy+zfRkRERJkhpS1udrsdhYWF8ftWqxU7d+5M2GfRokVYuHAhXn75\nZfj9frz44ovx506cODHhuXa7vVfv7+QVpURERJRBUtriJsvySfepqqpCRUUFNm7ciOeffx4/+tGP\njvvc3vZTY3AjIiKiTJLSFjebzYaGhob4fbvdDovFkrDPihUrsGzZMgDAxIkTEQgE4HA4YLPZ0NjY\nGN+vqanpmOf2xGw2xdcje1sAAKXF2QnbafBiPaU31l/6Yt2lN9bfmSOlwa28vBy1tbWor6+H2WxG\nVVUVlixZkrBPUVERNm/ejPnz56OmpgbBYBB5eXmYNWsW7rvvPtx6662w2+2ora1NamL7lhZXfP1w\nUycAQJTlhO00OJnNJtZTGmP9pS/WXXpj/aWvvgTulAY3URTx0EMPYeHChZBlGZWVlSgrK8PSpUtR\nXl6OmTNn4oEHHsBPf/pTLF++HAqFAk888QQAYOTIkbjiiiswb948KJVKPPLII6dwqlTd75+NiIiI\n6HQT5GQ6oqWR7kcdv3rlM+w+5MRzP5wBtUocwFJRMnjUmN5Yf+mLdZfeWH/pqy8tbhk9c0K7OwCD\nVsnQRkRERBkh44Mbx3AjIiKiTJGxwc0fDMMXiHAoECIiIsoYGRvc2t1BAEAOW9yIiIgoQ2RscOu6\nojSXLW5ERESUITI2uLV3DQXCFjciIiLKEJkb3NxscSMiIqLMkrHBLX6qlC1uRERElCEyN7i5OWsC\nERERZZaMDW7trgBEhQCTgcGNiIiIMkPmBjd3ANlGNRS9nN+UiIiIaLDKyOAmyTLa3UFemEBEREQZ\nJSODm8sbQkSSOWsCERERZZSMDG4cw42IiIgyUUYGt64rSjkUCBEREWWSjAxu8RY3DgVCREREGSQz\ngxtnTSAiIqIMlJHBzck+bkRERJSBMjO4xWdNYHAjIiKizJGRwa3dFYBWLUKnUQ50UYiIiIj6TWYG\nN3eQV5QSERFRxsm44BYKR+D2hXialIiIiDJOxgU3pzsIgP3biIiIKPNkXHDrGsONp0qJiIgo02Re\ncOOsCURERJShUn7Z5aZNm/DYY49BlmVUVFTgjjvuSHj88ccfx5YtWyAIArxeL5xOJz7++GMAwNix\nYzFmzBjIsoyioiL84Q9/OOn7xcdw46lSIiIiyjApDW6SJOHRRx/F8uXLYbFYUFlZidmzZ6OsrCy+\nz4MPPhhf/+tf/4rdu3fH7+t0OqxcubJX73lk8F1Od0VERESZJaWnSqurq1FaWori4mKoVCrMmzcP\n69atO+7+q1evxpVXXhm/L8tyr9+T010RERFRpkppcLPb7SgsLIzft1qtaG5u7nHfhoYG1NfXY+rU\nqfFtoVAIlZWVuOGGG/Df//43qfdsdwUgAMjmBPNERESUYVJ6qrQ3LWZVVVWYM2cOBEGIb3v33Xdh\nNptx+PBh3HLLLRg9ejRKSkpO+DpOdwBZBjVERcZdd0FERERnuJQGN5vNhoaGhvh9u90Oi8XS475r\n1qzBI488krDNbDYDAEpKSjBlyhTs3r37hMFNlmV0uIMYajPBbDb1wyeg0431lt5Yf+mLdZfeWH9n\njpQGt/LyctTW1qK+vh5msxlVVVVYsmTJMfsdOHAAnZ2dmDhxYnxbZ2cntFot1Go1HA4Htm3bhm9/\n+9snfD+3L4RgWIJRq0JLi6vfPw+lltlsYr2lMdZf+mLdpTfWX/rqS+BOaXATRREPPfQQFi5cCFmW\nUVlZibKyMixduhTl5eWYOXMmgGhr27x58xKeW1NTg4cffhiiKEKSJNx5550JV6P2pK3DDwDI4Rhu\nRERElIEEuS+Xbg5S2/Y045E/foj5Fw7HVdOGD3RxqJd41JjeWH/pi3WX3lh/6asvLW4Z1YO/rcMH\ngC1uRERElJkyK7h1Rk+VcrorIiIiykSZFdy6+rhx8F0iIiLKQBkV3BwdbHEjIiKizJVRwa2t0we1\nUgG9JqUXyxIRERENiMwKbh1+5Bg1CbMvEBEREWWKjApuHe4AryglIiKijJVRwU2W2b+NiIiIMldG\nBTcAyOUVpURERJShMi645RjVA10EIiIiopTIuOBWkKMb6CIQERERpURGjZvxo2+eh1GFvZ/3i4iI\niCgdZFSL20XnDoFSzKiPRERERBTHlENERESUJhjciIiIiNIEgxsRERFRmmBwIyIiIkoTDG5ERERE\naYLBjYiIiChNMLgRERERpQkGNyIiIqI0weBGRERElCYY3IiIiIjSBIMbERERUZpgcCMiIiJKE8pU\nv8GmTZvw2GOPQZZlVFRU4I477kh4/PHHH8eWLVsgCAK8Xi+cTic+/vhjAMDKlSvx3HPPAQC++93v\n4tprr011cYmIiIgGrZQGN0mS8Oijj2L58uWwWCyorKzE7NmzUVZWFt/nwQcfjK//9a9/xe7duwEA\nHR0d+P3vf4+VK1dClmUsWLAAs2fPhslkSmWRiYiIiAatlJ4qra6uRmlpKYqLi6FSqTBv3jysW7fu\nuPuvXr0aV155JQDg/fffx7Rp02AymZCVlYVp06bhvffeS2VxiYiIiAa1lAY3u92OwsLC+H2r1Yrm\n5uYe921oaEB9fT2mTp163Ofa7fZUFpeIiIhoUEtpcJNlOel9q6qqMGfOHAiCcNzndj1GREREdCZK\naR83m82GhoaG+H273Q6LxdLjvmvWrMEjjzyS8NwtW7bE7zc1NcVb407EbGYfuHTG+ktvrL/0xbpL\nb6y/M0dKW9zKy8tRW1uL+vp6BINBVFVVYfbs2cfsd+DAAXR2dmLixInxbdOnT8fmzZvhcrnQ0dGB\nzZs3Y/r06aksLhEREdGgltIWN1EU8dBDD2HhwoWQZRmVlZUoKyvD0qVLUV5ejpkzZwKItrbNmzcv\n4bnZ2dm46667UFFRAUEQsGjRImRlZaWyuERERESDmiD3piMaEREREQ0YzpxARERElCYY3IiIiIjS\nBIMbERERUZpI+Vylp8vJ5kSlwWXx4sXYsGED8vPz8eabbwKITnP2v//7v6ivr8eQIUPw29/+llOc\nDUJNTU24//770draClEUcd111+Hmm29m/aWJYDCIb3zjGwiFQohEIpgzZw4WLVqEuro63Hvvvejo\n6MC4cePw5JNPQqnMmK+IjCJJEioqKmC1WvHcc8+x7tLIrFmzYDQaoVAooFQqsWLFil7/7cyIFreu\nOVGXLVuG1atXo6qqCjU1NQNdLDqBBQsWYNmyZQnbXnjhBZx//vl4++23MWXKFDz//PMDVDo6EVEU\n8eCDD2LNmjV49dVX8be//Q01NTWsvzShVqvx0ksvYdWqVVi1ahU2bdqEHTt24KmnnsJtt92Gt99+\nGyaTCStWrBjootJxvPTSSwlzfrPu0ocgCHj55ZexatWqeD319m9nRgS33s6JSgNv0qRJxwzvsm7d\nOsyfPx8AMH/+fPz3v/8diKLRSZjNZowdOxYAYDAYUFZWBrvdzvpLIzqdDkC09S0cDkMQBGzZsgVz\n5swBEK2/d955ZyCLSMfR1NSEjRs34rrrrotv++ijj1h3aUKWZUiSlLCtt387MyK49WZOVBq8HA4H\nCgWBwiMAAAVsSURBVAoKAETDgdPpHOAS0cnU1dVhz549mDBhAtra2lh/aUKSJFx77bWYNm0apk2b\nhpKSEmRlZUGhiH4l2Gw2/g0dpB577DHcf//98SkgnU4nsrOzWXdpQhAEfOtb30JFRQVee+01AOj1\n386MOAnOoeiITj+Px4N77rkHixcvhsFg4FzCaUShUGDVqlVwu924++67e+xawvocfDZs2ICCggKM\nHTs2PiWkLMvHfAey7gavV199FWazGQ6HAwsXLsTw4cN7XV8ZEdx6MycqDV75+flobW1FQUEBWlpa\nkJeXN9BFouMIh8O45557cM011+CSSy4BwPpLR0ajEZMnT8aOHTvQ2dkJSZKgUCjQ1NTEv6GD0LZt\n27B+/Xps3LgRgUAAHo8Hjz32GFwuF+suTZjNZgBAXl4eLrnkElRXV/f6b2dGnCpNdk5UGlyOPkqc\nNWsWXn/9dQDAypUrWYeD2OLFizFy5Ejccsst8W2sv/TgcDjgcrkAAH6/Hx9++CFGjhyJKVOm4K23\n3gLA+hus7r33XmzYsAHr1q3DkiVLMGXKFDz11FOsuzTh8/ng8XgAAF6vF++//z7OOuusXv/tzJgp\nrzZt2oRf/vKX8TlRORzI4PbDH/4QW7ZsQXt7OwoKCvC9730Pl1xyCb7//e+jsbERRUVF+N3vfsf5\naQehTz/9FN/85jdx1llnQRAECILw/9u7n1f49jiO46+Z8SsJZU3RFFKkWJAkS4XRLLCzoZTEZKGU\nYiOR7IbERpqyEPEHjIViksRimhU1ZjOoSdlMzPsu7v1O3Xv73h/lcs94PjZz+nRqPu9OnV69z8x5\na3p6Wo2NjZqamuL6/c/FYjHNzs4qk8kok8mop6dH4+PjisfjCgQCenl5UX19vVZWVpSfn//V28VP\nRCIR7ezsaGNjg2vnEPF4XBMTE3K5XHp/f1dvb6/GxsaUSqX+1b0zZ4IbAABArsuJR6UAAADfAcEN\nAADAIQhuAAAADkFwAwAAcAiCGwAAgEMQ3AAAAByC4AYAHyQSicjv93/1NgDkMIIbAHwg5kQC+C/l\nxKxSAPg7Nzc3Wl1dzY6cmZyclNfrld/v18DAgM7OziRJ8/PzamlpkSQdHh5qe3tbbrdbVVVVWlhY\nyM4R3Nzc1MnJidxut4qLixUKhST9Osd1fn5e19fXcrvdWltbU01NzRdUDCAnGQDkuJeXF/P5fPb4\n+GhmZslk0jo7Oy0ajVptba0dHR2ZmVkkErHOzk5Lp9MWi8Wso6PDnp6ezMxsfX3dpqamzMzs4ODA\nBgcH7fX11czMUqmUmZldXFxYQ0ODRaNRMzMLBoM2MzPzqbUCyG103ADkvKurKz08PGh0dFT225Q/\nj8ejt7c3FRQUqK+vT5LU2tqqoqIi3d3dKRKJqKurSxUVFZKkoaEh9ff3S5LC4bCGh4dVXFwsSSor\nK8t+V3V1terq6iRJTU1NCofDn1UmgG+A4AbgW6irq9Pu7u7v1hKJxJ/OMzO5XK7s5x/Xfxz/TGFh\nYfb4RzgEgI/CnxMA5Lzm5mbd39/r4uIiu3Z7eyszUzqd1vHxsSTp8vJS6XRa1dXVamtr0+npqZ6f\nnyVJ+/v7am9vlyR1d3crFAplfy+XSqU+uSIA3xUdNwA5r7S0VMFgUMvLy1paWlI6nVZVVZXm5uZU\nXl6uaDSqra0tSdLa2pry8vLk9XoVCAQ0MjIit9utyspKLS4uSpJ8Pp+SyaQGBwfl8XhUUlKivb29\nrywRwDfhsr/q+QNADkskEvL7/To/P//qrQDAP8KjUgDfGu9dA+AkdNwAAAAcgo4bAACAQxDcAAAA\nHILgBgAA4BAENwAAAIcguAEAADgEwQ0AAMAhfgFkVoulN+1SqAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55f696ced490>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 27s, sys: 12.2 s, total: 1min 39s\n",
      "Wall time: 1min 4s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "    \n",
    "# Specify the model hyperparameters (NOTE: All the defaults can be omitted):\n",
    "model_params = {\n",
    "        'hidden_sizes' : [512],    # List of hidden layer dimensions, empty for linear model.\n",
    "        'l2_lambda' : 1e-3            # Strength of L2 regularization.\n",
    "}\n",
    "\n",
    "# Specify the training hyperparameters:\n",
    "training_params = {\n",
    "        'num_epochs' : 50,        # Max epochs/iterations to train for.\n",
    "        'report_every' : 1,     # Report training results every nr of epochs.\n",
    "        'eval_every' : 1,         # Evaluate on validation data every nr of epochs.\n",
    "        'stop_early' : True    # Use early stopping or not.\n",
    "}\n",
    "\n",
    "# Build, train, evaluate and plot the results!\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True    # Modify as desired.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vhfzBJHw_ep6"
   },
   "source": [
    "**97.7%** is a much more decent score! The 1 hidden layer model gives a much better score than the linear model, so let's see if we can do better by adding another layer!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qJ_Ec2df8YNj"
   },
   "source": [
    "### Going deeper! A 784-512-512-10 architecture w/ L2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 4044,
     "output_extras": [
      {
       "item_id": 49
      },
      {
       "item_id": 98
      },
      {
       "item_id": 100
      },
      {
       "item_id": 101
      },
      {
       "item_id": 102
      },
      {
       "item_id": 103
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 166883,
     "status": "ok",
     "timestamp": 1502969316725,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "3jALpxhzC6yc",
    "outputId": "874ec485-034b-44e5-a771-65bbd620a817"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 576.249123591\n",
      "Epoch: 0001 Validation acc= 0.901599765\n",
      "Epoch: 0002 Training cost= 356.557729048\n",
      "Epoch: 0002 Validation acc= 0.921399713\n",
      "Epoch: 0003 Training cost= 317.592652199\n",
      "Epoch: 0003 Validation acc= 0.931799769\n",
      "Epoch: 0004 Training cost= 293.989943626\n",
      "Epoch: 0004 Validation acc= 0.937399745\n",
      "Epoch: 0005 Training cost= 276.594365179\n",
      "Epoch: 0005 Validation acc= 0.944599748\n",
      "Epoch: 0006 Training cost= 263.482251143\n",
      "Epoch: 0006 Validation acc= 0.946799755\n",
      "Epoch: 0007 Training cost= 251.240891252\n",
      "Epoch: 0007 Validation acc= 0.945599735\n",
      "Epoch: 0008 Training cost= 241.220871305\n",
      "Epoch: 0008 Validation acc= 0.949199736\n",
      "Epoch: 0009 Training cost= 231.527150657\n",
      "Epoch: 0009 Validation acc= 0.950399697\n",
      "Epoch: 0010 Training cost= 222.198544922\n",
      "Epoch: 0010 Validation acc= 0.953399718\n",
      "Epoch: 0011 Training cost= 213.551616044\n",
      "Epoch: 0011 Validation acc= 0.955799699\n",
      "Epoch: 0012 Training cost= 204.914471547\n",
      "Epoch: 0012 Validation acc= 0.957999706\n",
      "Epoch: 0013 Training cost= 196.558960072\n",
      "Epoch: 0013 Validation acc= 0.954799712\n",
      "Epoch: 0014 Training cost= 188.747769498\n",
      "Epoch: 0014 Validation acc= 0.959999740\n",
      "Epoch: 0015 Training cost= 180.734522983\n",
      "Epoch: 0015 Validation acc= 0.958799720\n",
      "Epoch: 0016 Training cost= 173.227038269\n",
      "Epoch: 0016 Validation acc= 0.963799715\n",
      "Epoch: 0017 Training cost= 166.165209351\n",
      "Epoch: 0017 Validation acc= 0.958999813\n",
      "Epoch: 0018 Training cost= 158.934809099\n",
      "Epoch: 0018 Validation acc= 0.962199748\n",
      "Epoch: 0019 Training cost= 152.528965426\n",
      "Epoch: 0019 Validation acc= 0.963199794\n",
      "Epoch: 0020 Training cost= 146.041184803\n",
      "Epoch: 0020 Validation acc= 0.960999727\n",
      "Epoch: 0021 Training cost= 139.855197560\n",
      "Epoch: 0021 Validation acc= 0.964799702\n",
      "Epoch: 0022 Training cost= 133.776780784\n",
      "Epoch: 0022 Validation acc= 0.967599690\n",
      "Epoch: 0023 Training cost= 128.551627003\n",
      "Epoch: 0023 Validation acc= 0.966199756\n",
      "Epoch: 0024 Training cost= 122.610672066\n",
      "Epoch: 0024 Validation acc= 0.968599677\n",
      "Epoch: 0025 Training cost= 117.313750860\n",
      "Epoch: 0025 Validation acc= 0.967199683\n",
      "Epoch: 0026 Training cost= 112.409426963\n",
      "Epoch: 0026 Validation acc= 0.966599703\n",
      "Epoch: 0027 Training cost= 107.463527194\n",
      "Epoch: 0027 Validation acc= 0.965999722\n",
      "Epoch: 0028 Training cost= 102.690966672\n",
      "Epoch: 0028 Validation acc= 0.966199636\n",
      "Epoch: 0029 Training cost= 97.922743253\n",
      "Epoch: 0029 Validation acc= 0.970599711\n",
      "Epoch: 0030 Training cost= 93.660167001\n",
      "Epoch: 0030 Validation acc= 0.968199790\n",
      "Epoch: 0031 Training cost= 89.266302893\n",
      "Epoch: 0031 Validation acc= 0.966199696\n",
      "Epoch: 0032 Training cost= 85.337237507\n",
      "Epoch: 0032 Validation acc= 0.972999752\n",
      "Epoch: 0033 Training cost= 81.284292270\n",
      "Epoch: 0033 Validation acc= 0.969199717\n",
      "Epoch: 0034 Training cost= 77.531017983\n",
      "Epoch: 0034 Validation acc= 0.966799676\n",
      "Epoch: 0035 Training cost= 73.750824987\n",
      "Epoch: 0035 Validation acc= 0.962999761\n",
      "Epoch: 0036 Training cost= 70.103137623\n",
      "Epoch: 0036 Validation acc= 0.969999731\n",
      "Epoch: 0037 Training cost= 66.711315141\n",
      "Epoch: 0037 Validation acc= 0.971399724\n",
      "Epoch: 0038 Training cost= 63.361611460\n",
      "Epoch: 0038 Validation acc= 0.968599737\n",
      "Epoch: 0039 Training cost= 60.011656321\n",
      "Epoch: 0039 Validation acc= 0.970199764\n",
      "Epoch: 0040 Training cost= 57.025321149\n",
      "Epoch: 0040 Validation acc= 0.969599724\n",
      "Epoch: 0041 Training cost= 53.959342832\n",
      "Epoch: 0041 Validation acc= 0.970399678\n",
      "Epoch: 0042 Training cost= 51.148343631\n",
      "Epoch: 0042 Validation acc= 0.970399737\n",
      "Epoch: 0043 Training cost= 48.408878347\n",
      "Epoch: 0043 Validation acc= 0.969199717\n",
      "Epoch: 0044 Training cost= 45.793727729\n",
      "Epoch: 0044 Validation acc= 0.969799697\n",
      "Epoch: 0045 Training cost= 43.255685619\n",
      "Epoch: 0045 Validation acc= 0.969599724\n",
      "Epoch: 0046 Training cost= 40.852977191\n",
      "Epoch: 0046 Validation acc= 0.971399784\n",
      "Epoch: 0047 Training cost= 38.543384718\n",
      "Epoch: 0047 Validation acc= 0.967999697\n",
      "Epoch: 0048 Training cost= 36.367765503\n",
      "Epoch: 0048 Validation acc= 0.972199738\n",
      "Epoch: 0049 Training cost= 34.265690106\n",
      "Epoch: 0049 Validation acc= 0.971599758\n",
      "Epoch: 0050 Training cost= 32.170752844\n",
      "Epoch: 0050 Validation acc= 0.966999650\n",
      "Epoch: 0051 Training cost= 30.278565372\n",
      "Epoch: 0051 Validation acc= 0.966999769\n",
      "Epoch: 0052 Training cost= 28.405725545\n",
      "Epoch: 0052 Validation acc= 0.969399750\n",
      "Epoch: 0053 Training cost= 26.525246187\n",
      "Epoch: 0053 Validation acc= 0.967999816\n",
      "Epoch: 0054 Training cost= 24.837529481\n",
      "Epoch: 0054 Validation acc= 0.970999718\n",
      "Epoch: 0055 Training cost= 23.152070309\n",
      "Epoch: 0055 Validation acc= 0.964599729\n",
      "Epoch: 0056 Training cost= 21.591280441\n",
      "Epoch: 0056 Validation acc= 0.971399724\n",
      "Epoch: 0057 Training cost= 20.122739747\n",
      "Epoch: 0057 Validation acc= 0.968599677\n",
      "Epoch: 0058 Training cost= 18.699150758\n",
      "Epoch: 0058 Validation acc= 0.969799638\n",
      "Epoch: 0059 Training cost= 17.338278760\n",
      "Epoch: 0059 Validation acc= 0.971799731\n",
      "Epoch: 0060 Training cost= 16.024516387\n",
      "Epoch: 0060 Validation acc= 0.962999761\n",
      "Epoch: 0061 Training cost= 14.899629550\n",
      "Epoch: 0061 Validation acc= 0.975199699\n",
      "Epoch: 0062 Training cost= 13.723355025\n",
      "Epoch: 0062 Validation acc= 0.968799710\n",
      "Epoch: 0063 Training cost= 12.608933298\n",
      "Epoch: 0063 Validation acc= 0.968399704\n",
      "Epoch: 0064 Training cost= 11.589327968\n",
      "Epoch: 0064 Validation acc= 0.967199743\n",
      "Epoch: 0065 Training cost= 10.676569299\n",
      "Epoch: 0065 Validation acc= 0.968399644\n",
      "Epoch: 0066 Training cost= 9.733573787\n",
      "Epoch: 0066 Validation acc= 0.969999731\n",
      "Epoch: 0067 Training cost= 8.906992539\n",
      "Epoch: 0067 Validation acc= 0.965599716\n",
      "Epoch: 0068 Training cost= 8.061578745\n",
      "Epoch: 0068 Validation acc= 0.970599711\n",
      "Epoch: 0069 Training cost= 7.316279169\n",
      "Epoch: 0069 Validation acc= 0.969999611\n",
      "Epoch: 0070 Training cost= 6.604641718\n",
      "Epoch: 0070 Validation acc= 0.970599711\n",
      "Epoch: 0071 Training cost= 5.932473201\n",
      "Epoch: 0071 Validation acc= 0.967399716\n",
      "Epoch: 0072 Training cost= 5.323358593\n",
      "Epoch: 0072 Validation acc= 0.968999684\n",
      "Epoch: 0073 Training cost= 4.746478335\n",
      "Epoch: 0073 Validation acc= 0.970599771\n",
      "Epoch: 0074 Training cost= 4.206226299\n",
      "Epoch: 0074 Validation acc= 0.967599809\n",
      "Epoch: 0075 Training cost= 3.727632008\n",
      "Epoch: 0075 Validation acc= 0.972599745\n",
      "Epoch: 0076 Training cost= 3.261459116\n",
      "Epoch: 0076 Validation acc= 0.969599724\n",
      "Epoch: 0077 Training cost= 2.828105771\n",
      "Epoch: 0077 Validation acc= 0.972399712\n",
      "Epoch: 0078 Training cost= 2.451501520\n",
      "Epoch: 0078 Validation acc= 0.966999710\n",
      "Epoch: 0079 Training cost= 2.098190145\n",
      "Epoch: 0079 Validation acc= 0.966599762\n",
      "Epoch: 0080 Training cost= 1.771351617\n",
      "Epoch: 0080 Validation acc= 0.970199704\n",
      "Epoch: 0081 Training cost= 1.476746987\n",
      "Epoch: 0081 Validation acc= 0.968999743\n",
      "Epoch: 0082 Training cost= 1.229572264\n",
      "Epoch: 0082 Validation acc= 0.968599737\n",
      "Epoch: 0083 Training cost= 1.003394099\n",
      "Epoch: 0083 Validation acc= 0.969799757\n",
      "Epoch: 0084 Training cost= 0.814870764\n",
      "Epoch: 0084 Validation acc= 0.972399652\n",
      "Epoch: 0085 Training cost= 0.665674133\n",
      "Epoch: 0085 Validation acc= 0.973599732\n",
      "Epoch: 0086 Training cost= 0.538791539\n",
      "Epoch: 0086 Validation acc= 0.975999713\n",
      "Epoch: 0087 Training cost= 0.444349860\n",
      "Epoch: 0087 Validation acc= 0.974999785\n",
      "Epoch: 0088 Training cost= 0.373609773\n",
      "Epoch: 0088 Validation acc= 0.978399694\n",
      "Epoch: 0089 Training cost= 0.316718706\n",
      "Epoch: 0089 Validation acc= 0.972999752\n",
      "Epoch: 0090 Training cost= 0.278378398\n",
      "Epoch: 0090 Validation acc= 0.978799701\n",
      "Epoch: 0091 Training cost= 0.247428952\n",
      "Epoch: 0091 Validation acc= 0.976799726\n",
      "Epoch: 0092 Training cost= 0.224561186\n",
      "Epoch: 0092 Validation acc= 0.975799739\n",
      "Epoch: 0093 Training cost= 0.206074331\n",
      "Epoch: 0093 Validation acc= 0.977399707\n",
      "Epoch: 0094 Training cost= 0.195836283\n",
      "Epoch: 0094 Validation acc= 0.978199720\n",
      "Epoch: 0095 Training cost= 0.184348083\n",
      "Epoch: 0095 Validation acc= 0.976799667\n",
      "Epoch: 0096 Training cost= 0.177935783\n",
      "Epoch: 0096 Validation acc= 0.974799752\n",
      "Epoch: 0097 Training cost= 0.170420442\n",
      "Epoch: 0097 Validation acc= 0.977799714\n",
      "Epoch: 0098 Training cost= 0.166264389\n",
      "Epoch: 0098 Validation acc= 0.979799628\n",
      "Epoch: 0099 Training cost= 0.162785649\n",
      "Epoch: 0099 Validation acc= 0.976399720\n",
      "Epoch: 0100 Training cost= 0.159062386\n",
      "Epoch: 0100 Validation acc= 0.978199780\n",
      "Epoch: 0101 Training cost= 0.156110233\n",
      "Epoch: 0101 Validation acc= 0.975999773\n",
      "Validation loss stopped improving, stopping training early after 101 epochs!\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.9742\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAD+CAYAAACDUYl5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8XPWd7//XmarRaNRHGstyL1iuQADb1GC5gImDBZgQ\nNpv8gF/Y5O7G1MteSCHL0n4XwsW7ubuJ94a0JeEmBMOGYhsEbuBu3HtXG/UyMxrNaMrvD9kyBhsL\nbFk64/fz8ZiH5dGZc77Hn5kzb3/P+Z6vkUwmk4iIiIhIv2fp6waIiIiISM8ouImIiIiYhIKbiIiI\niEkouImIiIiYhIKbiIiIiEkouImIiIiYRK8Ht0AgwPz587nxxhu56aab2LJlC62trdx9993MmjWL\ne+65h0Ag0L38k08+ycyZM7n55pvZtWtXbzdPRERExDR6Pbg99dRTXHfddbzzzju88cYbDB8+nIUL\nFzJ16lSWLFnC5MmT+eUvfwnA8uXLOXr0KEuXLuWJJ57g8ccf7+3miYiIiJhGrwa3YDDIhg0buPXW\nWwGw2Wx4PB7Ky8spKysDoKysjPLycgDKy8uZO3cuAJMmTSIQCNDQ0NCbTRQRERExjV4NbpWVleTk\n5PDoo49SVlbGj3/8Y8LhMI2NjeTn5wPg9XppamoCoK6uDp/P1/36wsJCamtre7OJIiIiIqbRq8Et\nFouxc+dO7rzzThYtWoTL5WLhwoUYhnHK5U81+9bplhURERG50PRqcPP5fPh8PiZMmADAzJkz2blz\nJ3l5ed2nQOvr68nNzQW6etj8fn/36/1+PwUFBZ+7DU21KiIiIhcKW2+uPD8/nwEDBnDo0CGGDRvG\nmjVrGDlyJCNHjuS1117j3nvvZdGiRZSWlgJQWlrKyy+/zOzZs9m8eTOZmZndp1RPxzAM6usDn7uM\n9F9er0f1MynVztxUP/NS7czN6/Wc1et7NbgB/OhHP+Lhhx8mFosxaNAgnnnmGeLxOPfffz9/+ctf\nKCoqYsGCBQBcd911LF++nBkzZuByuXjmmWd6u3kiIiIipmEkU+Bco/7nYV76n6N5qXbmpvqZl2pn\nbmfb46aZE0RERERMQsFNRERExCQU3ERERERMwvTB7alfr+3rJoiIiIicF6YPbut2+EmYf3yFiIiI\nyBmZPrglktDeEevrZpwkGAyyaNGrX/h1jzxyP6FQsBdaJCIiIqnA9MENIBju7OsmnCQQaGPRoj9/\n5vlEIvG5r/uf//NF3O6M3mqWiIiImFyv34D3fOhvwe0Xv/g51dVV3H3332C1WnE4nHg8Ho4ePcIf\n/vAXHn30Yerr64hGI8yb903mzJkLwLx5X+dXv/o97e3tPPzwfCZMuJjt27fg9Rby7LM/w+Fw9PGe\niYiISF9K+eD2p/f3s3533Tnd3uVjCrh92sjT/v773/8Bhw8f5KWXXubjjzfyyCMP8Pvf/wmfzwfA\nY489jsfjIRKJ8N3vfpvrrptGZmYmYHSvo7Kygn/6p2f4x3/8IT/5yaMsW/Y+M2fecE73Q0RERMwl\nJYJbqJ/1uH3a2LHjukMbwJ/+9AdWrlwOQF1dHZWVRxk7djxwYpDFgAFFjBjRFQ4vumgMfn/1eW2z\niIiI9D8pEdwC7acPbrdPG/m5vWPnQ1paWvfPH3+8kU2bNrBw4W9wOBz84Ad/RzQa/cxrPnla1GKx\nnnIZERERubCkxOCEUEf/6nFLT0+nvb0dgE9PBRsKBfF4PDgcDo4cOcyOHdtPuY4UmEJWREREzrGU\n6HHrb4MTMjOzmDBhEt/5zh04HE5yc3O7fzd58pW8/vpf+Na3bmfw4CGMHz/hE688cY2bYRiIiIiI\nfJKRNHnXzpyH3uArF3n5+7IJZ15Y+h2v10N9faCvmyFfgmpnbqqfeal25ub1es7q9alxqrSf9biJ\niIiI9AbTBzd3mo2AgpuIiIhcAEwf3DxuR7+7xk1ERESkN5g+uGW6HYTCnRqFKSIiIinP9MHNk+4g\nFk8S6Yz3dVNEREREepX5g5u760a1Ol0qIiIiqc70wS0z3fzBbcaMa/u6CSIiImIC5g9uKdDjppvt\nioiISE+YfuaE/niq9N///V/x+QZQVnYbAC+9tBDDMNi8+WOCwQCxWIzvfvd7XH31dX3cUhERETET\n8we3Y6dKQ+HYKX//2v43+bhu2znd5iUFE7hl5NdO+/vp02eyYMHPuoPb+++/xwsv/Cvf+MbfkJ6e\nTmtrC3/3d3cpuImIiMgXYvrgdvwat0B7tI9bcsKoURfR0tJCY2MDzc3NZGZmkpeXz4IFz7Nly2Ys\nFoOGhnqam5vIyck98wpFREREOA/Bbdq0aWRkZGCxWLDZbLz66qu0trbywAMPUFVVRXFxMS+++CIe\nT9fcXU8++SQrVqzA5XLx7LPPUlJS8rnrP36q9HQ9breM/Nrn9o71luuvL+WDD96jsbGR0tKZLF36\nDq2trfz61y9jsViYN+/rRCL9J2yKiIhI/9frgxMMw+D3v/89r7/+Oq+++ioACxcuZOrUqSxZsoTJ\nkyfzy1/+EoDly5dz9OhRli5dyhNPPMHjjz9+xvV3D07o6D/XuAFMmzaD8vKlLF/+PtdfX0owGCQn\nJxeLxcKmTRvw+2u6l9XNg0VERKQnej24JZNJEonESc+Vl5dTVlYGQFlZGeXl5d3Pz507F4BJkyYR\nCARoaGj43PX3x8EJAMOGDae9vR2vt5Dc3DxmzryB3bt38p3vfJMlS95myJBh3ctqVKmIiIj0RK+f\nKjUMg3vuuQfDMLjjjjuYN28ejY2N5OfnA+D1emlqagKgrq4On8/X/drCwkJqa2u7lz0Vp92Kw2bp\nd8EN4Le/faX756ysbH7xi5dOudzSpcvPV5NERETExHo9uL3yyivd4ezuu+9m2LBhp+1hOtUpw570\nRrlddoLt/S+4iYiIiJxLvR7cvF4vALm5uUyfPp2tW7eSl5dHQ0MD+fn51NfXk5vbNbKysLAQv9/f\n/Vq/309BQcEZt5HjSaOmMYjX6+mdnZBepbqZl2pnbqqfeal2F65eDW7hcJhEIoHb7aa9vZ1Vq1bx\nD//wD0ybNo3XXnuNe++9l0WLFlFaWgpAaWkpL7/8MrNnz2bz5s1kZmZ+7mnS45x2C+FInBp/Kzar\n6SeDuKB4vR7q6wN93Qz5ElQ7c1P9zEu1M7ezDd29GtwaGhr4h3/4BwzDIB6PM2fOHK6++mrGjx/P\n/fffz1/+8heKiopYsGABANdddx3Lly9nxowZuFwunnnmmR5tJ8NlB7oGKGRnOHttf0RERET6Uq8G\nt0GDBvHGG2985vns7Gx+85vfnPI1P/nJT77wdrqDW7uCm4iIiKSulDiv6P5Ej5uIiIhIqkqJ4OZR\ncBMREZELQEoEt+5Tpf1s9gQRERGRcyklgtvxU6Uh9biJiIhICkuJ4Ha8xy2gm/CKiIhICkuN4Jau\nHjcRERFJfakR3NI0OEFERERSX0oEN5fTitViaHCCiIiIpLSUCG6GYeBOsxEMx/q6KSIiIiK9JiWC\nG3SNLA22R/u6GSIiIiK9JmWCm8dlp70jRiKR7OumiIiIiPSKlAlubpedJNAe0elSERERSU0pE9wy\nNO2ViIiIpLjUC266Ca+IiIikqNQJbunqcRMREZHUljrBTTfhFRERkRSXOsFN17iJiIhIikuZ4OY+\nFtxCmj1BREREUlTKBLfjPW4BDU4QERGRFJU6we3Y4ISQTpWKiIhIikqZ4OZOswG6xk1ERERSV8oE\nN6vFQrrTRlDXuImIiEiKSpngBl3XuanHTURERFJVagW3dDvB9k6SSU00LyIiIqkntYKby048kaQj\nGu/rpoiIiIicc+cluCUSCcrKyvje974HQGVlJbfffjuzZs3iwQcfJBaLARCNRnnggQeYOXMm3/jG\nN6iurv5C23GnaWSpiIiIpK7zEtx+97vfMWLEiO6/P//889x1110sWbIEj8fDq6++CsCrr75KVlYW\nS5cu5Tvf+Q7PPffcF9pO9+wJGqAgIiIiKajXg5vf72f58uXMmzev+7k1a9Ywa9YsAMrKynjvvfcA\nKC8vp6ysDIBZs2axevXqL7StDNexW4LoJrwiIiKSgno9uD399NM88sgjGIYBQHNzM1lZWVgsXZv2\n+XzU1tYCUFdXh8/nA8BqtZKZmUlLS0uPt5WR7gB0LzcRERFJTb0a3JYtW0Z+fj4lJSXdIz2TyeRn\nRn0eD3Wffj6ZTHb/ric00byIiIikMltvrnzTpk28//77LF++nEgkQigU4umnnyYQCJBIJLBYLPj9\nfgoKCgAoLCzE7/dTWFhIPB4nGAySlZV1xu14vR4ABrZ0AJC0WLqfk/5PtTIv1c7cVD/zUu0uXL0a\n3B588EEefPBBANatW8dLL73E888/z/3338/ixYuZPXs2ixYtorS0FIBp06axaNEiJk2axOLFi5ky\nZUqPtlNfHwAgFunqaatrCHU/J/2b1+tRrUxKtTM31c+8VDtzO9vQ3Sf3cXvooYf49a9/zaxZs2ht\nbeW2224DYN68eTQ3NzNz5kx++9vf8tBDD32h9XqOXeMWCEfPeZtFRERE+lqv9rh90hVXXMEVV1wB\nwKBBg/jzn//8mWUcDgcLFiz40ts4PqpU93ETERGRVGT6mRP+edmLJJIJAOw2Kw67hWA41setEhER\nETn3TB/cttXuYX/Loe6/a6J5ERERSVWmD24AH1Wv7/45I03BTURERFKT6YPbgIwCNtdvpb0zDEBG\nup1IZ5zOWKKPWyYiIiJybpk+uF0//Eo6EzE21G4GdBNeERERSV2mD27XDZ2CxbCwumYdAO5jwU0j\nS0VERCTVmD645biyGJd3EUcDVVQGqslIU4+biIiIpCbTBzeAqQO67g/3Uc16MtIV3ERERCQ1pURw\nG583hkyHh/X+TbjSuialV3ATERGRVJMSwc1qsTLZ9xXaY2HqEl33dFNwExERkVSTEsENYGrR5QAc\n7NgOwJ6Klr5sjoiIiMg5lzLBrTDdy4isYRwKHmL4ECs7DjVR0xjq62aJiIiInDMpE9wArjzW61Yw\nohGA9zdV9WVzRERERM6plApulxRMJM3q5EjnLrI9Dj7cVkM4ognnRUREJDWkVHBzWh1cVngxLZFW\nxk+I0RGNs3qHv6+bJSIiInJOpFRwA7hq4GQAquwbsVoTlG+sJJlM9nGrRERERM5eygW3wZ5irh44\nBX+4luIJNdQ0trPrSHNfN0tERETkrKVccAMoG3ETeWm51Du2Y7hbKN9Y2ddNEhERETlrKRnc0mxO\n/rbkdgDco7ez+WAtDS3hPm6ViIiIyNlJyeAGMCpnONcPupq4PYht4F4++Fi3BhERERFzS9ngBjBn\n+A0UuLzYfEdYvn8r0c54XzdJRERE5EvrUXB7++23CQaDACxYsIB77rmH7du392rDzgWH1c63x34D\nMIgXb2HVjoq+bpKIiIjIl9aj4Pbv//7vZGRksHXrVlatWsXcuXN58skne7tt58SwrMFc67sGizPM\nW0fe0a1BRERExLR6FNxsNhsAH374IfPmzWPOnDlEIpFebdi5dMuYG3DGc+jwHOLf1vxF4U1ERERM\nqUfBzTAM3n77bd5++22mTp0KQGdnZ6827FyyW2x8f9J3IJLOzvA6/s/WV4gndL2biIiImEuPgtuP\nfvQj3nzzTW677TYGDRrE4cOHmTx58hlfF41GmTdvHnPnzmXOnDn8/Oc/B6CyspLbb7+dWbNm8eCD\nDxKLxbqXf+CBB5g5cybf+MY3qK6uPotdO9mogiJuHfi3JIJZbG78mF9s/S2RePScrV9ERESktxnJ\nXj5vGA6HcblcxONxvvnNb/LDH/6QX//618yaNYsbb7yRxx9/nJKSEu644w7+8Ic/sHfvXn7605/y\n9ttv8+677/K//tf/OuM26usDPWpLMpnk529sZkfiXazZDQzJHMT3J96Fx5FxtrspX5LX6+lx/aR/\nUe3MTfUzL9XO3Lxez1m9vkc9bs8++yyBQIBYLMadd97JxRdfzBtvvNGjDbhcLqCrNy0Wi2EYBmvX\nrmXWrFkAlJWV8d577wFQXl5OWVkZALNmzWL16tVfeIc+j2EY3DVrPGnVU4g3DORIWwUvbPw3GsKN\n53Q7IiIiIr2hR8Hto48+wuPxsGrVKgoLC1myZAkvvfRSjzaQSCSYO3cuV111FVdddRWDBg0iMzMT\ni6Vr0z6fj9raWgDq6urw+XwAWK1WMjMzaWlp+TL7dVoZLjv/703jiB4cj7N5NHXhBp7f+L+pCOgG\nvSIiItK/2b7IwuvXr2fGjBkUFhZiGEaPXmOxWHj99dcJBoP8/d//PQcOHPjMMsfX9emztslkskfb\n+aLdjtd7PeytauPNVQaTphawL/ohL378Cx6+6u+Y6Cv5QuuSs3e23cbSd1Q7c1P9zEu1u3D1KLjl\n5eXx+OOPs3LlSu69915isRjx+BcblZmRkcHll1/Oli1baGtrI5FIYLFY8Pv9FBQUAFBYWIjf76ew\nsJB4PE4wGCQrK+uM6/4y5/q/NnkwG3fVsmU1fP2mr/NB41s8veLn/G3J7Vzhu/QLr0++HF2rYV6q\nnbmpfual2pnbebnG7Wc/+xnDhg3jhRdeICsrC7/fz1133XXG1zU1NREIdL25Ojo6WL16NSNHjmTy\n5MksXrwYgEWLFlFaWgrAtGnTWLRoEQCLFy9mypQpX2qnesJht3LvnHFYLQbvvhfjtkHfxGl18Nud\nr/DukWW615uIiIj0Oz0eVRqLxTh06BCGYTB06NDum/J+nj179vA//sf/IJFIkEgkmD17Nt///vep\nqKjgwQcfpK2tjZKSEp577jnsdjvRaJT//t//O7t27SI7O5sXXniB4uLiM27nbP7nsXZnLQv/awdu\nl527bhnIq0f/SEukleuLr6Zs5E1YLdYvvW45M/3P0bxUO3NT/cxLtTO3s+1x61Fw27ZtG/Pnz8fh\ncJBMJonFYvzrv/4r48aNO6uNnytn+wZesaWa37yzm6wMB/9t3gheOfwH/KFaBnuK+VbJPAZmDDhH\nLZVP0wHIvFQ7c1P9zEu1M7fzEtzuuOMO7rvvvu5ZE9asWcOLL77IK6+8clYbP1fOxRv43fUV/LF8\nH3mZadx/Rwnv+Zewzr8Ji2Fh1pBp3DB0GjbLFxrLIT2gA5B5qXbmpvqZl2pnbuflGrdwONwd2gCm\nTJlCOBw+qw33NzMuH0TZNcNobOvg56/uYe6QW/j+xLvIdHh45/B7PLt+AYfbjvZ1M0VEROQC1qPg\n5nK5WLNmTfff161b131j3VTytSuHcuPkwdQ2tfOzVzYzyDWCH01+iKsHTqEmVMvzG/43f977BoFo\nsK+bKiIiIhegHp0q3bp1K/fddx8OhwPommD+X/7lXxg/fnyvN7AnzmWXcTKZ5D/f3csHm6rI8Tj5\nb2XjGVGUxd7mA/xh96vUhxtxWh1MG3QtpYOvwWVLvQB7PqnL37xUO3NT/cxLtTO383KNG3SFtUOH\nDpFMJhk+fDh2u/2sNnwunes3cDKZZPHao7y6/ABWi8G3Zl7EtZOK6EzE+LBqLYsPlxPoDOK2pTNz\n6PVcO/BKHNb+8+9hJjoAmZdqZ26qn3mpdubWq8HtTNex9ZfTpb31Bt5xqIlfvLGdUEeM6y4u4s7p\no7HbLETiUT6oWMV7R5cRjnWQ5cjkxmGlTB1wuQYwfEE6AJmXamduqp95qXbm1qvBbcyYMRiG0X0z\n2k9OTWUYBrt27TqrjZ8rvfkGrm8J8/PXtlFRF2REUSb/rWwCOR4nAKHOdt47upwPKlbRmegkLy2H\nG4fN4IrCS3T/tx7SAci8VDtzU/3MS7Uzt/N2qrQ/6+03cKQzzm8X72bNjlo86Xa+O2cs44fldf++\nNRJg6ZH3WVW1hlgyTkF6PjcNm8mlBROxGD0a/3HB0gHIvFQ7c1P9zEu1MzcFN3o/uEFXL+N7Gyv5\n0/v7SSSSzJ46hLnXDMNqORHMmjtaeOdwOatr1pNIJijOKOJvSm5jsOfMsz9cqHQAMi/VztxUP/NS\n7cxNwY3zE9yOO1TTxi/e2E59Swcji7P43tfHkZuZdtIyDeFG3jz4Lutrj9/A93puGFqq699OQQcg\n81LtzE31My/VztwU3Di/wQ2gvSPGbxfvZv3uOtxpNu65aSwXj8r/zHK7mvby8q5XaY60UOT28bcl\ntzM4U71vn6QDkHmpduam+pmXamduZxvcrD/96U9/em6a0nfa26PndXt2m4XLLvKSneFk8/5GVu/w\n09jawYjiLJz2E4MSvK48phZdTnsszI7G3ayuWU88EWN49lCsuvYNALfbed7rJ+eGamduqp95qXbm\n5nY7z+r16nE7SxV1Qf7PmzupqAviTrNx63UjuPbiIizHRuAet7tpHy/vfpWmjmZynNncOKyUKb7L\nLvjRp/qfo3mpduam+pmXamduOlVK3wY3gHgiwfubqli04iAd0TjDBmTyt7NGM9SXedJyHbEO3j70\nHiuqPqIzESM/LZfZw2Zwue+SC3b0qQ5A5qXamZvqZ16qnbkpuNH3we24lmCE//v+ftburMUAvnrp\nQMquGU6G6+RZFVoirSw5/AEfVq8lnoxTmO7tvn2I8ameulSnA5B5qXbmpvqZl2pnbgpu9J/gdtyu\nw03857t7qWlsJ8Nl55Zrh3PtpCIslpNDWWO4mcWHy1nj30AimWBMzijuuOgWvOl5p1lz6tEByLxU\nO3NT/cxLtTM3BTf6X3ADiMUTvLehkjc+PEQkGmdwQQZ3zhjN6EHZn1m2rr2BP+97g52Ne7Bb7Nw0\nbAbTBl1zQVz/pgOQeal25qb6mZdqZ24KbvTP4HZcSzDCq8sO8NF2PwBTxhUy76sju6fNOi6ZTLKx\ndjN/3vdfBDtDDMoo4s4L4Oa9OgCZl2pnbqqfeal25qbgRv8Obsftr2zl5Xf3cqQ2gMNu4cbJQ7jh\nisE4HSf3qgU7Qyza9xZr/BswMLiu+EpuHDadDLu7j1reu3QAMi/VztxUP/NS7cxNwQ1zBDeARCLJ\nqm01vLbiIG2hKDkeJ7dcO5yp432nvH3IH/e8RkO4EZctjZmDr+erg67GYbWfZu3mpAOQeal25qb6\nmZdqZ24KbpgnuB0XjsR4e80RlqyrIBZPMMTn4Y5pI7locM5Jy3UmYqysWs3iQ+WEYu1kO7P42vBZ\nTPZdmjK3D9EByLxUO3NT/cxLtTM3BTfMF9yOa2gN85flB1m7sxaAiSPyuOXa4QwuPLmo7Z1hlh75\ngGWVq+hMxChy+7h5xI2Myxtj+tuH6ABkXqqdual+5qXamZuCG+YNbscdqGrlz8sOsLeiBYArSgqY\ne81wfLnpJy3X3NHCmweXsta/kSRJRmQN5esjbmRk9rC+aPY5oQOQeal25qb6mZdqZ24Kbpg/uEHX\nqNIdh5r4y/KDHKkNYDEMrp7o4+tXDSM3M+2kZauCNfz14BK2NewEYHzeGOYMv4FiT1FfNP2s6ABk\nXqqdual+5qXamVu/Dm5+v59HHnmEhoYGrFYr8+bN49vf/jatra088MADVFVVUVxczIsvvojH07Uj\nTz75JCtWrMDlcvHss89SUlJyxu2k0hs4mUyycU89i1YepKaxHZvV4PpLirlp6hAy3Y6Tlj3YeoT/\nOvAO+1oOYmBwacFEbhhaSlGGr49a/8XpAGReqp25qX7mpdqZW78ObvX19TQ0NFBSUkIoFOKWW27h\n3/7t33jttdfIzs7mu9/9LgsXLqStrY2HH36Y5cuX8/LLL7Nw4UK2bNnCU089xZ/+9KcebCf13sCJ\nRJKPtvt5Y9UhGts6cNqtzLh8EDdcMYj0tBMjS5PJJLua9vJfB96hIlgNwCTveG4YMo3Bmf3/HnA6\nAJmXamduqp95qXbmdrbBrVeHJnq93u4eM7fbzYgRI6itraW8vJyysjIAysrKKC8vB6C8vJy5c+cC\nMGnSJAKBAA0NDb3ZxH7LYjG4euIAnr53Cn8zYzROh5U3PzrMP/5iNW+vOUIkGgfAMAzG5l3EP15+\nH9+b+P8wNHMwW+q38/9t+Bf+9+ZfcaDlcN/uiIiIiJwztvO1ocrKSnbv3s2kSZNobGwkPz8f6Ap3\nTU1NANTV1eHznTjNV1hYSG1tbfeyFyK7zULpV4q5esIAyjdV8s6aI7y67ABL1x3lxilD+OolA3Ha\nrRiGwYT8sYzPK2FP834WHy5nZ9MedjbtYVzeGMpG3sQAd2Ff746IiIichfMS3EKhEPPnz+exxx7D\n7Xaf9hYWpzpra/bbXZwrToeV2VOG8NWLi1iyroJ3N1Twf9/fz+K1R5k9tet5u60rwI3JHcWY3FEc\naDnMm4eWsqNxN7ua9nJV0WRuGjYDjyOjr3dHREREvoReD26xWIz58+dz8803M336dADy8vJoaGgg\nPz+f+vp6cnNzga4eNr/f3/1av99PQUHBGbdxtueLzebeQbl888YSFi3bz5urDvLH9/axZF0Ft00b\nxcwpQ3Dau6bR8nonMHnkeDbVbOf3m//CyqrVbKj9mLKxNzB79LR+MwvDhVa/VKLamZvqZ16q3YWr\n128H8sgjj5CTk8Ojjz7a/dxzzz1HVlYW995772kHJ2zevJmnn376gh2c0FOB9iiL1x2lfGMl0c4E\nnnQ7My4bxLRLB540iCGeiLOqei1vHVpKqLOdHGc2Nw4tZcqAy7BarJ+zhd6li2zNS7UzN9XPvFQ7\nc+vXo0o3btzIt771LUaPHo1hGBiGwQMPPMDEiRO5//77qampoaioiAULFpCZmQnAE088wcqVK3G5\nXDzzzDOMGzfujNvRGxja2qO8t6GC8o1VhCMxXE4r119SzIzLB5H1iduItHeGWXyknBWVH9GZiJGX\nlsMNQ0uZ7PtKnwQ4HYDMS7UzN9XPvFQ7c+vXwe180Rv4hHAkxgcfV7F0fQVtoSh2m4VrJg5g9pQh\nJ93ItyXSyrtHlrGqei2xRIz8tFxuGFrKFb5Lz2uA0wHIvFQ7c1P9zEu1MzcFNxTcTiXaGefDbTW8\ns/YoDa0dWC0GV03wMXvqUAqyXd3LtURaWXrkAz6sWkssGSfbmcVVRVdwVdFkspyZvd5OHYDMS7Uz\nN9XPvFRKpEs0AAAgAElEQVQ7c1NwQ8Ht88TiCdburOXN1UeobWrHYhhMHlvITVOHUJTv7l6uuaOF\n944uZ03NBjriESyGhUu8E7hm4FRGZg/rtdG9OgCZl2pnbqqfeal25qbghoJbTyQSSdbvruPN1Yep\nqg8BcPHIfG6YPJhRxVndwawjFmF97SZWVK6mOtQ1wndgxgBuGjaDifnjznmA0wHIvFQ7c1P9zEu1\nMzcFNxTcvohEMsnHext4Z+0RDla3ATC8KJMbrhjMpaO9WCxdwSyZTHKg9TArKj/i4/ptJJIJhmcN\npWzkbIZnDT1n7dEByLxUO3NT/cxLtTM3BTcU3L6MZDLJvspWFq89yub9XdOKFWS7mH5ZMVdNGIDL\neeIWf/5QHf914B22NOwAYFL+OL4+4kZ87jPfY+9MdAAyL9XO3FQ/81LtzE3BDQW3s1XTGGLp+go+\n3OYnFk+Q5rBy9YQBlH6lmMLc9O7lDrYeZtH+tznYehiLYeFi73gm5Y9jbN4Y0u2uz9nC6ekAZF6q\nnbmpfual2pmbghsKbudKWyjK8i3VfLCpkpZgFIAJw/Mo/Uox44fnYjEMkskk2xp28l8HF1MTqgXA\nYlgYmT2cifljmZA/lnxXbo+3qQOQeal25qb6mZdqZ24Kbii4nWuxeIJNe+t5b2Ml+ytbAfDlpjPj\nsmKuHD8Ap8NKMpmkOuRna/1OtjXs5Eigovv1I7KGcfXAyVzsnXDGabV0ADIv1c7cVD/zUu3MTcEN\nBbfedMQf4N0NFazbVUssniTdaeO6i4so/UrxZ27ou71hF5vqtrKneT8A6TYXV/gu5aqiyRRl+E65\nfh2AzEu1MzfVz7xUO3NTcEPB7XxoDUb44OMqPvi4ikB7JxbD4JLR+Vw7qYhxQ3O7R6MC1Lc38lHN\nOlbXrCcQDQIwImso0wZfy8T8sVgMS/eyOgCZl2pnbqqfeal25qbghoLb+dQZi7N2Zx3vbqigoq4r\nlOVmOrl6wgCunjCA/E/MyhBPxNnWuItVVWvY1bQXAK8rj2mDrmHKgMtwWB06AJmYamduqp95qXbm\npuCGgltfSCaTHPYHWLGlmrU7a+mIxjGAkqE5XDupiEtGebHbTvSs+UO1vF+xkrX+TcQSMdy2dK4Z\nOIVbL55FNNA7szJI79KXh7mpfual2pmbghsKbn0tEo2zYU8dK7ZUs+/YYIYMl50rx/u4dlLRSVNr\ntUUDrKhczYqqjwh1tuOw2rm6aArTB3+VLOfZvZnl/NKXh7mpfual2pmbghsKbv1JTWOIlVtq+HB7\nDYH2TgBGFmdxzcQBXD6mgDRH1419o/Eoq2s2UF6xnMZwM3aLrSvADbmObGdWX+6C9JC+PMxN9TMv\n1c7cFNxQcOuPYvEEm/c1sHxLNTsPNZEEnHYrl13k5aoJAxg9OBuLYZCdm8Zfty1jyeH3aY60YLPY\nmOL7CpO84xmVPRz7GW4nIn1HXx7mpvqZl2pnbgpuKLj1dw0tYT7c7ufDbTU0tHYAkJ+VxpXjfcy5\nbiTWRIJYIsZa/0aWHH6fxo5mAOwWO6NzRjA27yLG5l5EQXp+X+6GfIq+PMxN9TMv1c7cFNxQcDOL\nRDLJvooWVm2rYcPueiKdcQBGF2dx5YQBXHZRAU6Hwf6WQ+xo2s3Oxj3dszMADPYM5KvFV3Np4STs\nFtvpNiPnib48zE31My/VztwU3FBwM6OOaIyNe+pZv6eebfsbSAJ2m4VLRuUzdZyPsUNzsdssNHU0\ns7NxD9sadrGjcTdJknjsGVw9cDJXD5yi6+H6kL48zE31My/VztwU3FBwMzOv18Pu/fWs3uHno+1+\n/E3tALicViYMz+PS0V4mDM/D5bTRGG5mZdVqPqxeS3ssjMWwcIl3ApcWTmJMzijSbM4+3psLi748\nzE31My/VztwU3FBwM7NPHoCSySQHa9pYv6uOTXvru6+Hs1ktjB2aw+VjCrh0tBerLcF6/8csq/yQ\n6pC/axmLjTE5I5mQP5bx+SXqiTsP9OVhbqqfeal25qbghoKbmZ3uAJRMJqmoC7Jpbz0b99ZTVR8C\nwGGzcOloL1PG+Rg7NJuqUDVbG7omuq8K1nS/fkjmIC7xTuBi7wS86XnnbX8uJPryMDfVz7xUO3NT\ncEPBzcx6egCqbWpnzc5aVu/wU9ccBiAz3c4VJYVcNqaAkQOzaIo0s61hJ1sbdrK/5SCJZAKAgRkD\nukJcwQR86QUYhmZqOBf05WFuqp95qXbmpuCGgpuZfdED0PHTqWu217J2Vy3BcNdNfjPdDi4dlc+l\nF3kZMziHjniYrQ072Vy/jd1N+4gnu0aw5qflMi6/hPF5Y3SfuLOkLw9zU/3MS7UzNwU3FNzM7GwO\nQLF4gl1Hmtm4p56P99V3z9SQ7rQxcWQek0bkM354LhZbjO0Nu9lcv53dTXvpiEcAcFjsXJQ7inF5\nYxibO5o8V+45268Lgb48zE31My/VztwU3FBwM7NzdQBKJJLsq2xh4956Nu2tp6mtK5xZDIPRg7KY\nOCKfSSPzyM92cLD1CNsbu24vUtte372OgvR8SnIvoiR3FKOyR2iU6hnoy8PcVD/zUu3MrV8Ht8ce\ne4xly5aRl5fHX//6VwBaW1t54IEHqKqqori4mBdffBGPp2snnnzySVasWIHL5eLZZ5+lpKSkR9vR\nG9i8euMAdHxgw5b9DWze38ihmrbu3xXkuLh4ZD6TRuYzqjiLpkgTu5r2sqtpD3ubDxCJRwGwGlZK\nckczecBXmJBXolOqp6AvD3NT/cxLtTO3fh3cNmzYgNvt5pFHHukObs899xzZ2dl897vfZeHChbS1\ntfHwww+zfPlyXn75ZRYuXMiWLVt46qmn+NOf/tSj7egNbF7n4wDUGoqy9UADW/Y3suNQU/eMDS6n\njQnDc7tPqbrSLBxqPcKupn1sb9zVPUrVZUvj0oKJXOH7CiOyhmpwwzH68jA31c+8VDtzO9vg1qvz\nBl122WVUVVWd9Fx5eTn/+Z//CUBZWRnf/va3efjhhykvL2fu3LkATJo0iUAgQENDA/n5mp9Szk6W\n28E1E4u4ZmIRnbEEe442s2V/I5v3N7BuVx3rdtVhAMOLMpkwIo+JI67ka8Nn4Q/Vss6/ifW1H/Nh\n9To+rF5HXloOE73jmJg/jhFZQ7FarH29eyIicgE57xM+NjU1dYcxr9dLU1MTAHV1dfh8vu7lCgsL\nqa2tVXCTc8puszB+eB7jh+dx54xRVNWH2HKggW0HGtlf1caB6jZeX3mITLeDCcNzmTjiMqZfWkpl\n+Cjr/JvYXL+NDypW8UHFKtJtLsbllTDRO5aS3NG4bGl9vXsiIpLi+s1M3ac6Y6tTUtKbDMOguCCD\n4oIMbpo6lFBHJzsONbHtQCPbDjby4TY/H27zYzEMRhZnMXHEFH5w0SzabX62N+5ia8NO1tduYn3t\nJiyGhSGeQYzJHclFOaMYljUYm6XffLxERCRFnPdvlry8vO5ToPX19eTmdt2CobCwEL/f372c3++n\noKCgR+s82/PF0rf6S/28wNBBudx07UgSiSQHqlrYsKuOjbtq2VvRzN6KFgCyMhxMHDmWshHXkOuL\ncCS0j621u9nfdJhDbUd453A5TpuTsd5RXD5wIpcPnERWWmbf7lwv6S+1ky9H9TMv1e7C1evB7dM9\nadOmTeO1117j3nvvZdGiRZSWlgJQWlrKyy+/zOzZs9m8eTOZmZk9Pk2qizTNqz9fZJudZmP6JUVM\nv6SItvYoOw41sf1gI7uONLNycxUrN3ddv5mX6aRkyHXMG3wjjpwWjrYfZk/zfj6u2c7HNdv5jw1/\nZGT2MC4umMDF3vEpM49qf66dnJnqZ16qnbn161GlDz30EGvXrqWlpYX8/Hx+8IMfMH36dO677z5q\namooKipiwYIFZGZ29UY88cQTrFy5EpfLxTPPPMO4ceN6tB29gc3LjAegZDKJv6md3Uea2XWkmd1H\nW7pncAAo9roZOzSX4oEWgo4KdjTv4GDrke7fD/IMZLCnmMHH/izK8JnytKoZaycnqH7mpdqZW78O\nbueL3sDmlQoHoEQySUVtkJ1Hmth5qIm9la10xrrmSTWAQQUZDB5kx5ZbS13iEBWhCmKJWPfrbYaV\nogwfxRlFDPQUdf2Z4cNlc/XRHvVMKtTuQqb6mZdqZ279+nYgIhcCi2EwxOdhiM/DjZOH0BmLs7+y\nlV1Hm9lb0crB6jaO1iUAB3AR3pyJDB6cxJMfJuFsoS7ipypYzdFAFdScWG9eWi7FnqKTeucyHO6+\n2k0REekH1OMmfepC+J9jZyzBYX8beyta2FfZyr7KFsKRePfv87PSGD0ok4IBCZyZIYI0UB30Uxms\nJtgZOmlduWk5DPYUMyJrCBfljqLI7euz0dcXQu1SmepnXqqduelUKQpuZnYhHoASia4pufYcbWZP\nRQt7K1oIdZw4depOszFiYBbDizIZ4LNgSW+jOlzN0UAlR9sqTwpzHnsGo3NGcFHuSMbkjCLPlXve\n9uNCrF0qUf3MS7UzNwU3FNzMTAegrmvkqutD7KtqZX9lKweqWqlrCXf/3mIYDCrMYOTALEYOzMTr\nhbpYJXua97OnaT+t0RNzsQ7JHMQVhZfylcJJeBwZvdpu1c7cVD/zUu3MTcENBTcz0wHo1FpDUQ4c\nC3L7qlo44g8Qi5/4qOZ4nIwYmMWIAR5yvDGCNj87m3axu2kfSZJYDAtjckZxue8SxuZdRIb93F8b\np9qZm+pnXqqduSm4oeBmZjoA9UxnLM5hf4D9n+iVa2s/cQsSm9VgSKGHogE2kpnVVCf2UhOu7v69\n25aONz0fryufgvQ8CtO9jMgedlb3lFPtzE31My/Vztw0qlTkAmC3WRlVnM2o4myY3HUvufrWDg5W\ntXKgqo391a0c9gc4UJ0E0oCJOD2jyCluxOppI5YMcDRQyeG2oyet1+cupCRnFGNyRzEqZwROq6NP\n9k9ERHpGwU3EhAzDoCDbRUG2iynjfABEO+NU1AU57A9wqKaNw/4ANbtcnOhST+DydFI4IElGTgcd\njlrq2iv5ILSKDypXYTWsDPYMpNBdgC+9gIJ0L750L/muPKwWa1/tqoiIfIKCm0iKcNitXde9DTxx\n+jMciXG0NsBh/7FHTRuH94bp6pXLBmMU7rwAmb5W4ul1HG6r4NCneuUshoUit48hmYMYklnM0MzB\n+NJ7No+wiIicW7rGTfqUrtU4/9o7OjnsD3DEH+DQsTDX0NrR9UsjgeFsx5ERJisvijOjg4SjjdZE\nI/HkiVuWOCx2huUOxpdWSHFG12wPA9yF2K32Ptor+aL02TMv1c7cdI2biHwh6Wl2xg7NZezQE/d8\nC4Y7Oexvo6I2SEVdkIr6IP597cQTx/5fZySwuIJkettx5QSJO5vZ23CQPRzoXofFsOBLL2BgRhHF\nngEMdA9goGcAmY6zO0iJiMgJCm4iQobLzvhheYwfltf9XGcsQU1jqCvIHXscrQ3QcuRYz5sRx0gP\n4vAEychpx0gPUBtqpDrkZ33tiXV7HBkMdA/A5y6gML0An9tLYXoBmQ5Pn836ICJiVgpuInJKdpuF\nwYUeBhee6DFLJpM0ByIcrQvSFIqy70gzVfVBavYc751LYji7QpzDEyQtq51Ioo3d0X3sbt530vrT\nrGn43AX43AUMcBfiSy/A5y4kNy0bi2E5z3srImIOCm4i0mOGYZCbmUZuZtpJ19nE4gnqmsNUNYSo\nbghR1RCipiGEv/JYoLPEMFwhLGlBbO520jwdJNOCHIl/9hYldoudfFcueWm55LlyyU/LIc+Vh9eV\nhzc9H7tFhy0RuXDpCCgiZ81mtVCU76Yo/+QZGo4HuuqGEDWNIWoa27seNSGinYnuwRCGK4jFFcLp\nacdID1EXb6ImVPuZ7VgMC/muXAakF+JzF+JzF1CQnk++Kw+3LV2nXkUk5Sm4iUivOV2gSySTNLV1\n4G9qx9/YTs2xP/3V7TQHIkASrJ0YzjCGM4w1LYwrswNrejuNiVbq2hvY0rDjpHW6bGnku/LId+VR\n4MqnyF1IUcYACtO9ug+diKQMBTcROe8shkF+lov8LNdJAyIAOqIx6prD1DaHqW1q73o0h/Efbact\n3AkkwRbF4gp29dSlteNwR+hMa6ey009FoOqk9dkMK4XuAorcA/C6cvE4PGQ6MvA4PHgcGWQ6PKTZ\nnOdx70VEvjwFNxHpV9Icts8MijguHOkKdXUtYeqauwJdQ0uY+qoOmgIdJJNJsEewpIUw0gNYXEES\n7gBV8TqqgjWn3WaG3U2eKxfvsR674712A9wFpNvTe3N3RUS+EAU3ETENl9PGEJ+HIb7PhrpYPEFj\nWwcNLR3Ut4RpaO2goTVMfVMHDa3tBOOtGI4ODHsE7FEMewTDHsXijNCe1kEwWsWRtorPrDfT4aHI\n7esa+eouIDcthyxnJpkOD257ukbAish5peAmIinBZrVQmJNOYc6pe8gi0TgNbR00fCLUNbR00NDc\n1VsXao92BTtnO0ZaO0ZaCIsrSIsrSNspbmcCYMFChj2DnLQsctOyyTn+cGaTk5aFx55But1FmjVN\nAydE5JxQcBORC4LTYWVgvpuBnxoocVy0M05zIEJjWwdNbRGaAp/4szpIc6yBqLWtq6fOEcGwR0g4\nOmixR2iNVHIk8NneuuMMDFw2F267iwy7m4J0L4Xp3q772KUXkO/K0wAKEekRBTcREcBht1KYm05h\n7umvaWvviNEc6KDxE8Guua2DxpYOmsOttEXbiBghDEcHFmcYbJ0Y1k6wdRK0xgjZgtTbmjj0qXvX\nGVhwWzNwWdPJcKTjcbrJdLpx29PJdmYd69HLIceZhcvmUu+dyAVMwU1EpIfS02ykp2Uw0Jtx2mUi\n0TgtwQjNgQgtoQgtgSgtwUjXozVKcyBMW6yVTltb18jYtK572LXZIwRsIeqjcQievg1W7LitHtw2\nN1nOTLLTMslPzyLXlYXbnk6aLY00qxOXLa37Z/XmiaQOBTcRkXPI6Thzzx103fakNRSlNdgV7NpC\nUdraO2kJhWlpD9IaCRKMhmhPBIlb27uuv3OESTg7aLUHaLM3URMB2nrQJsNFujUDj91DpsNDTloW\ng7wFOBMOMp0ZZNgz8DgyNNhCxAQU3ERE+kCaw0aaw3bawRSfFInGCbR3Bbu29iiBUJTW9g4aw620\nhAO0dbYR6gzREe8gkuggYXSCNYZhiYGtk7AjQoe9ieZYPYSBVuCzE1MAYEnasWHHbjhxWJw4rU7S\nbS7c9q7TuJlON1muDLLTMnA70km3uXDZ0ki3p+Ow2HUaV6SX9bvgtmLFCp5++mmSySS33nor9957\nb183SUSkTzkdVpwOF/nZrh4tH+mMEwp3EmjvJBCO0t4RI9QRo7U9REukjdZoGxHCtIYDdCTCRJPt\nxIwISWuEhDVGzBojYg2CtQUjkYROugLfmSQNrEkHNpzdwS/Nmtb1sDmPBbyuR4YzjQxnOllpbjzO\ndNLtLtJtLuwKfyKfq18Ft0QiwT//8z/zm9/8hoKCAm677TZKS0sZMWJEXzdNRMQ0nHYrTruV3My0\n0y7j9Xqorw+c9FykM34s5HUSCnc9WjvCtIVDBKLtBKIh2jvbaY+F6YiHiSYjdCYjxIgSN6JgjR4L\nfhE6bEGMZAJidD0iPWx80sBI2LFgw4odKzZshh27YcduceKwOHBauwKhy55Gms3R9bA7cNkcpDmc\npNsduB1ppDucuB0uHFYHdotNp4ElJfSr4LZ161aGDBnCwIEDAbjpppsoLy9XcBMROQ+OB74cz5eb\nAqwzliAcjRGOxOiIxGkLd9DWEaK1I0Qo2kF7Z5hwLEJHrIOOWIRIIkL02CNGVwBMGFHilhhxS4yY\npR2scQxLomsDSSB+7AE96wX8pIQVI2nDSFqxJG3HwqENq2HDihWbYcNqORYULXZsFit2ix2HxY7d\nasdhPfazzY7Tasdhs+G0OnDYbKTZ7Dhsdpw2B2k2G06749hzNmwWq3oR5ZzpV8GttraWAQMGdP+9\nsLCQbdu29WGLRESkp+w2C3abg8x0x7FnPID3S60rFk8Q6YwTicYJRzsJRjoIRsIEox2EomHao2Ha\nYxEisQiReJRovJPOeCfRRCediRixZCexZCdxOkkkY8SNGAk6SRpxEkacuCUMlgSGNX7qBiSOPc6B\nZBJIWjCSBiQtgAUjeeyBBQPrJ362YDEMDKxYsGAxjj2HFYthwWpYsNvtJOJJrMd/d+x5i2HFYhif\n+PnY7ywWDIxjfzewWixYsGAc+7n7d5YTy1gMA8Mwjm0fDMOCBQPD4NjzBgYWMLruU2gYHFsncHw5\nusJq1+uPP9/1Wuhaz0lx9hM9op/sGz2ReY1PPGecctmTnCIsf5H4fNqw3cP12i12PPbPzvLi9X72\nuS+iXwW3ZDLZ100QEZF+wGa1YLNacKfZgTS6QuC5k0wmiSeSdERjdHRGCUejhGNdf3bEonR0RumI\ndRKNR4nEuoJh9JPBMB6jMxkjluh6xJMx4sk48WScRDJOnK4/EyRIEidJ4tjPCZJGVypMGDEwEiSN\nJBgJMBKnygSfleDkpJA89pB+J7LzChLB3JOe++vPbj6rdfar4Obz+aiuru7+e21tLQUFBWd83dmm\nV+lbqp95qXbmpvqJmE+/ulJzwoQJHD16lKqqKqLRKG+99RalpaV93SwRERGRfqFf9bhZrVZ+/OMf\nc/fdd5NMJrnttts0MEFERETkGCOpC8tERERETKFfnSoVERERkdNTcBMRERExCQU3EREREZMwdXBb\nsWIFN9xwA7NmzWLhwoV93Rz5HH6/n29/+9vMnj2bOXPm8Lvf/Q6A1tZW7r77bmbNmsU999xDIBA4\nw5qkryQSCcrKyvje974HQGVlJbfffjuzZs3iwQcfJBaL9XEL5XQCgQDz58/nxhtv5KabbmLLli36\n7JnIb37zG772ta8xZ84cHnroIaLRqD5//dhjjz3GlVdeyZw5c7qf+7zP25NPPsnMmTO5+eab2bVr\n1xnXb9rgdnxe01/96le8+eabvPXWWxw4cKCvmyWnYbVaefTRR3n77bd55ZVXePnllzlw4AALFy5k\n6tSpLFmyhMmTJ/PLX/6yr5sqp/G73/3upFHezz//PHfddRdLlizB4/Hw6quv9mHr5PM89dRTXHfd\ndbzzzju88cYbDB8+XJ89k6itreX3v/89r732Gn/961+Jx+O89dZb+vz1Y7fccgu/+tWvTnrudJ+3\n5cuXc/ToUZYuXcoTTzzB448/fsb1mza4fXJeU7vd3j2vqfRPXq+XkpISANxuNyNGjKC2tpby8nLK\nysoAKCsr47333uvLZspp+P1+li9fzrx587qfW7NmDbNmzQK6avfuu+/2VfPkcwSDQTZs2MCtt94K\ngM1mw+Px6LNnIolEgnA4TCwWo6Ojg4KCAtauXavPXz912WWXkZmZedJzn/68Hc8r5eXlzJ07F4BJ\nkyYRCARoaGj43PWbNridal7Turq6PmyR9FRlZSW7d+9m0qRJNDY2kp+fD3SFu+bm5j5unZzK008/\nzSOPPNI9d19zczNZWVlYLF2HEJ/Pp89fP1VZWUlOTg6PPvooZWVl/PjHPyYcDuuzZxKFhYXcdddd\nfPWrX+Xaa6/F4/EwduxYMjMz9fkzkaamppM+b01NTQDU1dXh8/m6lyssLKS2tvZz12Xa4Kbbz5lT\nKBRi/vz5PPbYY7jd7tNP4iv9xrJly8jPz6ekpKT7c5dMJj/zGVQt+6dYLMbOnTu58847WbRoES6X\ni4ULF6peJtHW1kZ5eTkffPABK1euJBwOs2LFis8sp3qa06myzJlq2a9mTvgivuy8ptJ3YrEY8+fP\n5+abb2b69OkA5OXl0dDQQH5+PvX19eTm5p5hLXK+bdq0iffff5/ly5cTiUQIhUI8/fTTBAIBEokE\nFosFv9+vz18/5fP58Pl8TJgwAYCZM2fyH//xH/rsmcRHH33EoEGDyM7OBmD69Ol8/PHHtLW16fNn\nIqf7vBUWFuL3+7uX60ktTdvjpnlNzeexxx5j5MiRfOc73+l+btq0abz22msALFq0SDXshx588EGW\nLVtGeXk5L7zwApMnT+b5559n8uTJLF68GFDt+rP8/HwGDBjAoUOHgK5rE0eOHKnPnkkUFRWxZcsW\nIpEIyWSSNWvWMGrUKH3++rlP96Sd7vNWWlrK66+/DsDmzZvJzMzsPqV6Oqae8mrFihU89dRT3fOa\n3nvvvX3dJDmNjRs38q1vfYvRo0djGAaGYfDAAw8wceJE7r//fmpqaigqKmLBggWfuahT+o9169bx\n0ksv8Ytf/IKKigoefPBB2traKCkp4bnnnsNut/d1E+UUdu/ezQ9/+ENisRiDBg3imWeeIR6P67Nn\nEj//+c956623sNlsjB07lieffBK/36/PXz/10EMPsXbtWlpaWsjPz+cHP/gB06dP57777jvl5+2J\nJ55g5cqVuFwunnnmGcaNG/e56zd1cBMRERG5kJj2VKmIiIjIhUbBTURERMQkFNxERERETELBTURE\nRMQkFNxERERETELBTURERMQkFNxERM7CunXruidwFxHpbQpuIiJnSfNEisj5Ytq5SkVEzmTr1q08\n//zzhEIhAObPn8/IkSO59dZbKSsr48MPPwTgJz/5CZdddhkAr7/+Or/61a+wWCwMHjyYf/qnf+qe\nV/CXv/wlb775JhaLhfT0dP74xz8CXfPw/uQnP2Hz5s1YLBZeeOEFhg8f3gd7LCIpLykikoLa2tqS\nc+fOTdbX1yf///bu3yW1OA7j+GNKiUQFzQVCkNAQDQ1FSPgHZOCgbS3tIW5uLeESbQ6NEUFDEP0D\ntqWIS8PBqQZdpMCl5RA8d0q493Iv3OEWR9+v5Xz5cOCcz/bw+Z4ftj0YDJzNZh0EgVdXV313d2fb\nbrVazmazDsPQ3W7XOzs7fn19tW2fn5/7+PjYtn17e+tisej393fb9nA4tG03m02vra05CALbdr1e\nd/+MxtkAAAGGSURBVKVS+dJeAUwOJm4AxlKn01Gv19PR0dHoh8/xeFwfHx+anp7W3t6eJGlzc1PJ\nZFLPz89qtVra3d3V4uKiJKlUKimfz0uSGo2GDg4OlEqlJEnz8/Oja6XTaWUyGUnS+vq6Go3GV7UJ\nYMIQ3ACMrUwmo8vLy59q/X7/t/NsKxaLjY6/1j/XfzIzMzNaf4ZDAPgfeDkBwFja2NjQy8uLms3m\nqPb09CTbCsNQ9/f3kqR2u60wDJVOp7W1taWHhwe9vb1Jkm5ubrS9vS1JyuVyur6+Hj0vNxwOv7gj\nAGDiBmBMzc3NqV6vq1ar6fT0VGEYanl5WdVqVQsLCwqCQBcXF5Kks7MzJRIJraysqFwu6/DwUFNT\nU1paWtLJyYkkaX9/X4PBQMViUfF4XLOzs7q6uvrOFgFMoJj/Nv8HgDHT7/dVKBT0+Pj43bcCAP+M\nrVIAE4fvrgGIKiZuAAAAEcHEDQAAICIIbgAAABFBcAMAAIgIghsAAEBEENwAAAAiguAGAAAQET8A\nDw2uroq0y2QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55f68f9c3b50>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnIAAAD+CAYAAAC+w+gfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xt8XHWd//HXmfs9k8vM5J60Kb3QK7S1hVZKubQIKJZW\nBURUFHUVl/2tCru4iiu6+2N3xZ+g68LKZb2grmARKUixQLkUCoXe703b3DO5T+Z+O+f3xyTThqZN\n2iakmXyej8c8ppM5c853+k1m3ud7O4qmaRpCCCGEEGLc0Y11AYQQQgghxJmRICeEEEIIMU5JkBNC\nCCGEGKckyAkhhBBCjFMS5IQQQgghxikJckIIIYQQ49SoBrm7776biy++mI9+9KMn3eYHP/gBK1as\n4LrrrmPv3r3Zn69du5aVK1eycuVKnn766dEsphBCCCHEuDSqQe7666/nkUceOenzGzdupL6+nvXr\n1/P973+fe+65B4BAIMDPfvYznnzySf7whz/w05/+lGAwOJpFFUIIIYQYd0Y1yC1YsACXy3XS5zds\n2MDHP/5xAObOnUswGKSjo4PXX3+dJUuW4HQ6cblcLFmyhNdee200iyqEEEIIMe6M6Ri5trY2iouL\ns4+Li4vx+/34/X5KSkqyP/f5fPj9/rEoohBCCCHEOWtMg9z7rw6maRqKopzwcwBFUT6oYgkhhBBC\njAtjGuR8Ph+tra3Zx62trXi9XoqLi2lubj7h50ORy8YKIYQQYiIxjPYBThWuLr/8cn7zm99w9dVX\ns23bNlwuF0VFRSxdupQf//jHBINBVFVl06ZNfPOb3xzyWIqi0N4ukyLGK4/HKfU3jkn9jV9Sd+Ob\n1N/45fE4z3ofoxrkvvGNb7B582Z6enq49NJL+frXv04ymURRFD71qU+xbNkyNm7cyJVXXonVauVf\n//VfAcjLy+OrX/0qq1evRlEUbr/99lNOmhBCjCxV0+gNJ3DZTehkWIMQQpyzFC3H+iPlrGT8Opuz\nymRKZUdtB7FEGpvZgNVswGbJ3DusRqzmoc9Z0qpKIJTAZNRjNevR6waOPAjHkjT4QzS0hahvC9Lc\nEcFmMeB1W/G4rfjyrXjyrbjsJmLxFJF4ikis7xZP4bQaqfA5KHRZThjzqWoaTe1h9tV3s7++h2g8\nhdthJt9pxu0w4XaYcdlNROIpesMJekJxAuEEvaEE0UQKnU7BoNOh1yno9Qp6nYLdYiTPYSLPntmH\ny27CaTOh1ynodAo6BXQ6BUVR6AjEqGvt5WhrkLrWIPVtIeKJNFazgepiJ9UlTiYVu6guceJ2mAlH\nk4T6buFYinAsSb7bRjKWxGLSYzEbsJj06BSFYCRJMJIgGO27jyRR1cx4WEUBXd+9Xq/gsJpw2oy4\nbH33dhMOqxGD/vRHgaRVlXhCJZ5Mk0ylSaU1UmmVtJq5T6VUkmmNZEollVaz9zaLgaI8K0V5Fpw2\n45DjczVNozeSpCMQpTMQoycYp8BlocLnwOO2njQIh2NJOnpimE16Cl0WjIaxG+kiLTrjm9Tf+HXO\nt8iJ3JdKqxxtDdLRE6Wk0E5pkQ2jQT/k6xLJNN3BOF29Mbr67pMqtHdHCEeThGNJwtEUsUSKmrI8\nFp3vY+6UIszGgfsOhOK8vLWJV7Y10xtOnPR4DqsRb741c3Nn7tNpjdauSPbW1h0lrR47rzEb9dgs\nBmxmA7FEis7e+IB96nUKaVVj92n+n9nMBsq9Diq9DvKdZg4397K/oYdQNHmaexodigIlhXZ8+VZa\nOiPsretmb133mJbJZjbgtBlx9gU8u8VIKp0JabFEmkQyTSyZJp7IPM6EN/Wsj2sy6ijKs5LvNKOQ\nCdyq2nfT+sJYIHbSY5lNeso9dio8Dhw2Ex09UfzdUdq6I4Rjqex2CuB2mvHkWShyWyl0WXDYjDgs\nRuxWA3Zr5t+xRJr2nigdgRjtPVHaA1F6gglsFgOuvuDrsplw2k1YjPpMYO0PrmkVVdWYVOJiaoV7\n2OFY0zQ6AzEONQU41BTA3xWhyG2ltMieuRXacTtMMiFNiDEiLXITXFt3hIONAVJpFUVR0CkKOl1m\nvKFRr8Pa17plNeuxmQ2YjHoa2kLsr+9mf0MPh5oCJJLHvsR0ioKvwEqF10G5x4FBr6M3nCAQjvfd\nJ+gJJYYMLQa9DofVgF6no7M3BmSC1QVTi1g0w4fTZmLDu428vddPWtWwmQ18eG4JpYV2IvEU0fix\nlrBgJJn50usZGNSOZzUbKC6w4XFbSKU1IrEk0XiaSDxzr9crVHgd2Vul14mvwEoiqWb33dYdpa0n\nSiiSzLYI9gdBq9lATyhOfV+Lnr8rwvElKXCZmV6Zz7RKN9Mr83E7TPSEEnQH4/SE4vSEEvSGE9gt\nBlx2E3kOE267GZfDhM1sIK1qpNMaqqaRTquk0hqhaJJAOE4glPl/D4QShGJJ0qqGpmqkVS0bTPLs\nJqqKnVQXu6jwOjCbjgXmSCyVba070tJLOJbKBAurEYfVgMNixGYxYrWZaO8M94WrFLFEmrSq4bQe\nC2BOmxGH1YRBr6BpoKGhaZmAlEqphKJJgpEkvX0td73hxHGteUlCkSTqIB9Zep2CxaTHZNRnWgRN\nesxGPRaTAZNRh8mgx6BX0Ot1GPQKBn2m9dJo0GHU6zAadBj0OgwGHaFoks6+oNQZiNEeiBGNp044\npk5RsJr1FLmtFLksFOZZKMqzkOcw0xGI0tCWqevWzsiA3zu9TsHTdzLhybMSS6bo6InREYjS1Rvn\ndD+QzSY9iUT6tF5ntxiYN6WIC6d6mDmpgLJSN+3tQeLJNO19v8f+rghHWno51BSgJ3TykyTI/P1M\nLnFywVQPF5znId9pPmGbVFrlQEMPO2o76Q7GyXeaKXCaKXBZyHeZyXeYicZTdARidARidAZidPTG\niCVSVHqdTC51MbnUhdtx4r4nOmmRG79GokVOglwO0jSNcCyF0aDDZNANOFNOJNPsb+hhZ20nOw93\n4u+OntWxSovsTKtwU1xgo7UrQkN7iMa2ELFEetDtrWYDeXYT+U4zhS4LBa7MB3mB00x1RT6JaAKH\n1YjpuJa3pvYQm/f6eWu3n45AbMD+SgptXDG/nItmFWMxnbqBWVU1unpj+PtCl16nUFxgo7jANqwu\ntJEUT6Rp7AjR1RunqtiJJ+/E7tbx5oP4MlE1jUhfV65Rr8PcF9jOpOv1dMSTaRQyXdGZbunh11Uy\npdLSGSYUTeJ1WylwWdDpBn99Kq3S2RujqzeebZnOdl9Hk5gMeorcFjzuTNevx23FajaQVlVC0Uy3\ne28kQTCcIJ5MZ8JpX3jV63Woqsaeo11sPdhBdzDTwmwy6phUmoe/MzxoYMuzm5hSnseUssyttMhO\nRyBGc0c4c+vM3Ld0RrKvmVTi4sKpRcyoKqChLciO2k721HUTP8nnwukocJmZXOKiMM+SDeT979No\n0GGzGPpOMox9LZkGdDql78Qsc5IXjaeIJ9KYjHrslkyLp73vxOv9QypGQiqtZodXjMbfuQS58UuC\n3CAm2i9zWlVp6YxQ7w9S7w9l7yN9LQiKQl8LRWbMUmcgRqKvG8hs0nN+VT7nVxdgsxj6uouOtZAk\nUyrReIrYcR+AsUQab76VaRVupla6cdlMJ5SpvyumsT2Mpmm4HCby7Jnbqbpdh/ow0jSNwy29bN7j\npzecYOnsEs6fVCCD8c8R8mUyfqiaxtGWIO8daOfdA+20d0cocFmOG3pgw+O2UulzUDTMk4yu3hhb\nD3bw3oF29tf3nNBy6iuwMXtyAXNqCikttNMTSgwYWtETimM1GyjK62/dzIRVo0HH0dYgh5t7OdLc\ny+HmAL2R0RuGYDLq+sZtHhtHqlMU3E4zpX3DR8qKHJQW2ShwWbKt/qG+VuRgNEl3MNOi2N+q2B2M\no2ngy7cyf5qX+dM8VBc7T/r/GkukiMYzYzsTqcz4zUQy8+/+noZILEkklvlcrizNY5LXQYXPcdLP\nw0gsRW1zgGg8RZ49M+42z2Ea8gRYjC4JcoPI1S+S/i6Hls7+s+AILZ1h/F0RUuljVagA3gIbJQU2\nUqpKLJEmFj/W1ZXnMDF7UiGzJxdw3mmMk/kgSBAY36T+xq+CQgddnaER218ommT7oQ4ONvZQ7nEw\nu6YQX75tRPataRqdvTFC0SSpVN/4PzUznCCZUvvG1x43GSeaIq1q2eEh/cNFLCY98WSacCwTisLR\nTEtvNJFG0zRUNXMsVcsMQ+jqjZNKn964S0Uh2/tgMRnY39CdHYpS6LIwf5qHMo+djp5MV35bX2/B\nmY6XddlNzJpUwKxJBVSXuKj3BznYEOBgYw8NbaFBu9/NJj1uh5myIjuVPgdVPieVPueAcY/ReIrO\n3kww7QrGcdlMmck8OdCTMNYkyA3iXPwiCYVCvPjiX1i1as0pt0ulVWqbAuyt66a9J8Zffvfv1Fx0\nC8GEbtAuCbNJT2mhjTJP/x9fZlzacGZonoskCIxvUn/jl9Td0NKqSkdPbEB3cncwjs1iHDARx2kz\n4rabKcyzkO80DzhZjifT7DrcxbsH2th+qINofODnul6nUOS24nFbsFuMmfGbxsw4zv6xnnaLAavF\ngM2c6Q62mPQEYmk2bW9i15GuQSd9GQ06Jpe4OK8ijzy7OTPrPZSgp28MbVdvbMDkGyA7caY7eOJz\n/axmPeWezLjhcq8jO4O/wGUelS7qXCRBbhDn4odRS0szd931f/jlL38/4OeqqtLVG2fXkS52Hu5k\nb133CWPLHFYjboc5M5bMaaak0E5JkY3SQntmJl0OnQ3Jl8n4JvU3fkndffCSKZW9dV10B+OZyS9D\njJ88lf76UzWNxrYQOw930tgepsLrYGq5m6pi5ymXt9E0jZ5Qgjp/kPrWYObeHyQUTVHgyoTS/gk9\n+U4z3cH4sck8XRHenyJ0ikJhnhmPO7Mck9VkwGLODPGxmvTYrUamVbgpcFlO+73mGglygzgXP4zu\nuedu3njjVSorq9Dr9ZhMZgwmK/sO1lK29Bs0vfM/pGIB9EqahUuv5ROrV1PqsfPVL36SRx/9NZFI\nhG9+82+ZPXseu3Ztx+Px8X//748wmU4cnzaeyZfJ+Cb1N35J3Y1vY1l/8WSa5o4wTe3h7JI4mZn8\nsVMuCQVQ6XUwZ0oR86YUUV3iHHR8n6pqJI9b5zHZP2YwlSYcTfVNBkr2TQ5KHRtn6crMIs/vax1U\nVY1gJLNqQiCcWQkgnVYxGDItnSZDpvXToNMRjiX7Jg5lZtD3hhNEYqlsOTK3zNqUDquRQlfmeIV5\nmQl8Noshs0pDKEF336oDPaE4sXh6wJqVyZTK7//lmrOug/HZB3eG/velQ7yzr21E97lwupdPXjbl\nlNv8zd98naNHD/Poo79h01ub+fbd36Diw/+HsqUfZ2Z1PquX/iMLZ1aSZ9Nz2223MPmrN+JyWVGU\nY2dQjY0N/PM//yt33fVtvvvdf+SVV15ixYqrRvS9CCGEEKfDbNQzqcTFpJITr74UT6QJRZOZyRuJ\nvrHa8cwaojsPd7Kvvpv6thDPbjqKy2akzOMglkhnZhb3bRtPnt1MZ52iYLMYCMeSJ7Qcni5FAZNB\nn1m2yJCZMe3vjtDQNvT4UkUBq8mAoW/JI4vZgNM2Mj1qEyrIjSVNg1e2NfE/z+7B4CynoryMm66Y\nysxJBTzyyEPc+YuNALS1tdHYWM/558+C44amlpSUUlOTCYzTpk2ntbV5LN6GEEIIMSxmk37AmpTH\nu3JhBdF4ij1Hu9l+qIMdtR3srevGoNdhNeuxmgy47WYsJn12nGD/uo/9rWgOa//SMZnlZmwWA5F4\nKjNbOBDNrkkYjCQpLrThtpvIc2SudJNnN2My6kgk1ezs4ERKJZ1WsVuMOO2ZK8z0jxW0WQyDTg7s\nX+6rMxCjqzczSzkaS+HquyJPft/xnDbTGXWbD8eECnKfvGzKkK1no+FoS5C27ii//Mt+UmmNcp+b\nf771Qxj0OrZufZf33tvCww8/jslk4utf/zKJxInN0cd3o+p0+kG3EUIIIcYLq9nA/Gke5k/z9C1m\nro3pperOhKIo2XULq4rPfrzbmZhQQe6D1tUb48lXanlj2xFisQgXzfQxrcDKumd2ZpN9OBzC6XRi\nMpmoqzvK7t27Bt1Xjg1lFEIIIbJ0ioLOMD4m7yXTSQ72HKa25wjVeZXMKpwxphMPJciNgngyzQub\n63nurToSKZXJlT488y/k9Sfv5W2TmYKCguy2ixZdzNNPP8XNN3+SysoqZs2afdyejv1i5NLsVCGE\nELlN1VR2dOwh35xHlavilNum1TQv1L1Ec9jP9PwpzCycTr7FfVbH1zSNcDJCV7ybaDJGqaMYp8lx\nxvvrjvWwq3Mfuzv3sr/rEAn12Fp/FY5SPjLpCuYUzRz0uzqtpmkKtVBgzcdhtJ9xGU5GZq2OsPcO\ntPPEXw/Q1Rsnz27i+mWTWTK7RK4+MAwyc258k/obv6Tuxrdzqf7Sapot/m38pW4DbZEOdIqOVTVX\ns7ziw4OGnGgqyqO7nmBP1/4BPy9zlDCzcDozC6dT7ijFYhj8GruaptEZ6+Zw4CiHA3W0RzroinfT\nHeshqQ5c/y7f7KbKVU6ls5xKVznVrgqsBusp38u29l280vg6hwN12Z/7bF5mFk6jxj2J9/zbea9t\nBxoaZY4SPlJ9BXM9M2kJ+9nfdZD93Yc42HOYeDrBQt+FfG7mDQOOIcuPDGIsf5lf3trEr17Yj0Gv\nsGJhJddcVDVuF+cdC+fSh5E4fVJ/45fU3fj2QdVfR7SLnR170Ct6vLYivLYi3OY8dIqOtJrmbf9W\nXji6gfZoJzpFx0LfBeztOkBvIsh871xumr5mQCDriHbxXzseoyXs5/zCaXy85moOdh9md+c+DvTU\nkjouiOWZnHhtHjzWzHH1Oj2HA3Uc7jlKINE7oJwOo518i5sCs5t8ixuz3kxTqJm63kaCyWMzTHWK\njpq86mxgLLH7UBSFcDLCG82b2di4iZ54AIDp+ecx23M+swqnU2QtHHC81rCf549u4F3/djQ0jDoj\nyeNa7Hw2D1Pzp7Cs/GJK7L4Br5UgN4ix+jB6cUsDv/3rQVw2I9+44QIqvGfehDtRyZfJ+Cb1Nz69\n07qVbV3buX7Sxyi0Fgz9AnHOGc2/vd5EkPf8O9ji38aR3roTnjfqDHisRcTScbpi3egVPReVLmRF\n5XIKrfkE4r38YtevORw4Sondx5dm34LX5uFw4CgP7fgfQskwl5Yv4fop16LXHZvhGk8nONB9iH1d\nB2kNt9EW7aA71oP2vguNuUxOavKqmeyupiavmhK7D5N+8DVWNU2jJx6gPthIXW8j+7oPUt/bmN1n\ngSWfCmcZezr3k1STmPQmLipZwLLyJfhsniH/r/zhNv5S9xJHe+updlUyLX8K0/KnnLKbWILcIMbi\ni+T5zXX84eVa8hwm7rzxAkoKR74PfCKYiEEgraYHfHiNZxOx/sa7YCLE9968j1g6jtucx+3zvnhC\ni4E4943U356maQQSvTSFWmkOtbCvr2tQQ0NBYVr+FC70zcGgGGiPdtAW6cjep7Q0F5d8iBVVl54Q\nXFJqij8eWsfGxjew6C18uGwxLze8horGJ867jkvKLxpW+ZLpJB2xLtoiHSTSCapdlRRZC85qDHkw\nEWJP5352d+5jT9cBoqkohZZ8lpUv4aKShdiMJ+96HQkS5AbxQX+R/HnTUda+eph8p5k7b7wAX8HI\nXBh6IppIQUDTNJ478iIv1L3MouL5XDt5BXnmExfUHE/OxfpLq2m2d+wmpaZwmhw4jQ6cJicOoy1n\nAvRg0moaDQ2D7tRDO367/4+83vQWc3wz2OHfi91o42tzvzDk4HQB29p28o5/K/N985hbNPOkv0+a\nplEbOMqezv1cXPohikah1fNM//b6y/Ze2w4ag820hFuJpKIDtpnkqmS+bx4XeueSZx48dGiahqqp\nQ/5Nvd36Hk/se4qkmsRqsPCFWTczo2DqaZd7tKTVNB3RTjy2InTKB7MMigS5QXxQXySapvGn14/w\nzBtHKXSZ+dZNF+J1j25yz3XnYhA4laSaQq/oTvsPXtM0/nz4BV6oewmdokPVVEw6I1dULuPyymUn\nHdR7RmVMJ2kMtVAfbKS+t5FQMsSF3rlc4J2DSW8csePA6NdfIp2kNnCE/V2HaA63MrtoBotLFmI8\nSVip7TnK7/b/keZw66DPF1kLWT3lWuZ4Zo5amSFT30d66wgmwswpOn/I1oOdHXt4vWkz87yzme+d\nc9JuovcLxHvZ3bmf3Z172dd1EJvRxrcW3I7LNPgXRXOolX95+8d4bR5+fM13WbdzI0/sexKT3shX\n5nyOqfkf/Jqb76dpGqFkGIvBctJ67hdPJ0imkzhMZ94jklJTpNT0Kf8GE+kETx78M280b87+LN/s\nZln5xSwp/RA2Y+ZkPpqKsrn1PV5veouWsB8Am8HK52bexMzCaWdcxsGc7t9ed6yHza3v8lbLFtqj\nnQAoKHhshZTaSyh1FFNqL6bSWU6hNX9Ey9oYbOa1pjdZXrGUYmn9lSA3mA8qCDz92mGeeeMoHreF\nb914AUV5EuLO1ngJcol0kueP/pW/1m/EZXIy3zeXBb55VDjKhvyS1jSNP9U+z4v1r+C1FnH7vNvY\n13WAPx95gWAihMvk5NpJK1hcsuCMW4yS6SR/PvJCNvComnrCNnaDjUUl81latjg79uP42V+1PUdo\nDrdi0VsyLVn9N6MDDY3uWIDueDddsR664wF6YgFU0idcAsesN1Fs9/Z9Ofiy9woKvYkQoWSIYCJE\nbyJENBVFr+gx6gwYjru1RdrZ33WIw711AwY/Q+YLdEXVpVxUshBjXzANJcI8Xfscb7a8A8BFJQup\nclUQSoQI9h0vmAhxJFBHSkuz0Hchn5j6MezG02tNj6cTpNVU9ov7/VJqivfadvByw+vUBxsBuLzi\nElZNueakvydb23by6O7fZOvMarCyuGQ+S0sXU2z3ZrdTNZXOaDfN4Rbqg03s7txHQ7Ap+7zT5CCY\nCHGeezJfn3fboL9LP932C/Z2HeBv5nye5TM+RHt7kPfadvD47t+iKAq3zvw0c0cw5EZTUY4E6qkN\nHMWgGFhesQSL4eQXTQ/Ee/nlnt+zr/tg9v/CabL3tao6SGtqti6DyRCJdGaR9Gn5U7iy6lKm5583\n5N9jJBnlSG9mwHxt4ChHextQNZULvXNZXrHkhJbJplALj+5+gtawnzJHCR+vuZqdHXt5q3ULiXQC\nk87IopIFfTM3t5JQk+gVPfM8syh3lLLu6Iuk1TTXTLqSldWXDfskMJqKsqH+NQ71HMZrKxoQthwm\n+5CfndFUjJZwK02hFra372Zf18G+QfkG5nlms7hkAZPzqkf85E4MTYLcID6IILD+nQZ+t+EgHreF\nu266kALXyT+MzsSVV17Ciy++OqL7HA/OJsh1x3rY0bEHk95ElbOcYrt3VJrGD3Qf4ol9T9Ee7cRt\nziOejhNNxQDw2opY4LuABd65+I770u2naRpra9exof5VvLYi7rjgy7jNeQDEUnH+Wr+RDfUbSahJ\nCi35mX355lHqKB52+ZJqiod3/g97Ovdj1Bkod5RR6Sqnqm+6vUExsKnlbTY1v00oGQZgav4UHEYb\nte+b/aWgnDCweDB2gw23JQ+LyUQqOTA0RlNR2qOdw9rPqSgolDtKmFowhWn55+GzFfFq45u82vQm\nSTWJ25zHlZWXYtQZ+FPt84RTEcocJdwwbRWT86oH3WdzqJVf7f1f6oON5Jmc3Dh9NbOLzj9lOZLp\nJLs797HFv41dnXtJqikKLQXH/o/7WjDead3Ga02bCCSCKCjMKToff6Sd1kgbF5Us5MZp158Qrt5r\n28Fju5/AqDNw84xP0hRs5o2WtwkmMrPsznNPpshaSHOolZZw64B1rPSKninuSczqm33ntXn4712/\nYnv7Li6vvITrp1w74Fi7O/fzn9sfYXr+edw+74t4va7s397ergM8vON/SGlpLvTOwW60YzVYMje9\nBbvRhs/uxWstOunJRlpN44+00xRqyZwYBI7SHGod8HvgNudxw7RVg/6f7+rYy6/2/i+hZJhJripM\nemM2tIWS4ex+9Ip+wElGPB2nNnAUgApnGVdWLuMC7xx0ig5N0+iK9VAbOMLhQB21PUdoCfuz+1JQ\nKHUUk1LT+COZa3JPzqtmecVS5hbN5I3mzTx16FlSaopl5UtYVXN19uQhkoyyqeVtNjZuoivWDUCh\nJZ+lpYtZXLog2ypa39vIf+/6FV2xbmYXzeCWGTeccgxWIp1gY+Mm1te9fEKXZz+XyYnHXoABI1aD\nBUtfXekUHf5wG02hVrrjPQNeM8lVxUUlC7jQN+eUy2+I0SdBbhCjHeRe39HCo8/tJc9h4h9vnj8q\n3akrVixj/fqNI77fc93xQU7VVHZ37mNb2y5cZmdm3R9nOQUWd/YsO5QIs7U9M5vqUM+RAfsy6YxU\nODMhptxRSp7ZlT2Tdxjtp93aFUlGWHvoOTa1vI2CwvKKpVw7eSU6Rceezv1s8W9lZ8ee7LpFHmsh\nswpnMLNwOlPyJ2NQ9Pzx0LO81PAaPpuXOy740qBj4nriAZ478le2+LcS72thKLUXs8A3j/m+eacc\nX5NSU/xi16/Y2bGX8wun8aVZt2S/aAbbdnv7Ll5reouDPYeBTCtOTd4kavpmf5U7Sklp6b4v0GD2\nixTITO23uHGb3dluqJMF8UQ6SWvYT3O4tS+E+FEUJVsf/TebwUpaU0mpSVJqmqSaJKmmyDO7mJpf\nM+hCmsFEiL/Wb+TVxk3ZYGPWm7h28kqWlV08ZD2n1TQv1m/kuSMvktbSLCqez7Lyi1EY2JLTmwiy\ntW0n29p3EUtngrvP5qXA4qY+2Eg4GTlh3xa9mYtLP8Sy8ospshYSSoT52fZHqA82Ms8zi8/NvCnb\nXfiufxuP7/kdJp2Rr837QjZ8ptQUOzr28FrTWxzoPgRkwkumlTPTIlPqKGaKe9IJrVvRVIx/2/IA\nbZEOvjDrZi70zsm+53955//hD7fxjx/6O8ocJSfU3eFAHQ/teDwb9gdjUPT4jmttRYOmcAstYT/+\ncBsp7dhRj/AhAAAgAElEQVTFzo06A1WuCmryJjE5r4qjvQ2sr3uZtJZmvncua6Z+DJfJSVJN8afa\n53i54XUMOgOrplzDsrKLB7SsqZpKOBlBr+iwGqwntLrV9zayvv4VtrXtREOjyFJAhaucI4G67FIS\nmTIZqXZVUOOexOS8aibnVWI1WNE0jX3dB3m54XV2d+4DwGqwEE3FsBttfGbGJ08a+NNqmr1dB9Dr\n9EzLnzLoyWQoEeax3U+wr/sgHmshN01fQ6ElPxvCdIqOlJpiU/PbPH90A72JIFaDlRWVl7KkbBE9\n8QBNoZZsoG8O+wkmQyTTyUFKlAl6/b8npfZiJudVDXqiKcaGBLlBjGaQe+9AOz9buxOb2cBdn76Q\ncs/wlhj5+c8fpLi4hFWr1gDw6KMPoygK27ZtJRQKkkqluO22r7B06TJgYge5Q41NbGp+hzeaN59w\nFgmZ9YEqneUA7Os+mO2CmuKexHzvPADqgg3U9zYOONt+P7vRdkKQcBqd2I02VNS+sTKZW0JN8k7r\nVnoTQcocJXx6+ppBB4PHUjF2dOxhW/su9nUdyAYxk85Isd1HfbCRYruPv533pZMOGu6XSCfY2bGX\nLf5t7Oncl/1SvNA7h+tqPnLCOkZpNc0ju3/D9vZdTM8/j6/M+dxJQ9z7dUS70DTtrGd/jWXXeDAR\n4pXGN4ilYlxZdWm2pXO4Mq1zv6f+uO7JweSb3dlQXe4oQVGUvpaebur6xiH6I+1Mza9hcckCrIOE\nq4d2PM7BnsNMzz+P22bfwq6OPTy+53eY9WZun/cFJuVVDXrsjmgXSTV5ypawwd7Xv7/7UxTgzgVf\np9ju49XGN/n9gbUsKf0QN03PfCYNVnfJdJJAIkg0FSOWihJNxYimYgSTIVrC/mwoP369LMj8vpdk\ng4OP6rwqKpxlJ4xxaw618sS+JznSW4/NYOUjk65gc8u7NIaa8dm83DrzJsqdpcN6n4Npi7Tz1/pX\n2dz6bmayi9GRPUmZ7K6mwlE25P+jP9zGK42beKt1C9WuSj57/qdO+3drMKqm8ufDL7C+7uUTnjPr\nTSjoiKVjmPQmLqv4MJdXXHLKljuPx0mzv5tYXx3FUjFSWgqv1XNWYwbF6JMgN4hTfZH88dCzbG3b\neUb7TabS9IYzH1h5dhOGvgv7XuCdfUK3xfsdPLifn/zkR/z0pw8DcPPNn+T++x/E4XBis9kIBHr4\n8pc/z+9+txaYmEGutucob7S9yTtN21E1FbPexMLiC7moZAHxVCKz7k+wkfreBjr7ui4qnGWZL1Xv\n3EHX6UmkEzSGmmkOtWbH0fQmQoT6xmQFE8GTdle8n0Fn4OrqK7iictmwvkRTaopDPUfY3bmP3Z37\n8EfaKbH7+NsLvnTSwecnE0lG2NbXelYfbESv6FlWfjFXVV+O3WgjraZ5bM9v2dq2g6n5U/ibOZ8f\nk7Eu42WM48mk1TRvNL9Ne7TjhOeMOiPnF05jcl7VWXfZJ9NJHtn9G3Z27MFn89IWae8LcV9kUl7l\nWe17MO/6t/Ho7ifw2bx8fd4X+b/v/ISkmuSexXdlTyjOtO5UTaUj2kVzuBUFKLWXUGjNH/b/kaqp\nvNr0Js/UPp898bm45EOsmfoxzMOc5DGUUDJMNBk7qxOVtJpGp+hG/FKJezr3s6tzbzYk9wexRDrB\nzMLprKhePqzPi/H+tzeRjUSQk8sODEMqpWZDnMtuzIa44TrvvGn09PTQ2dlBd3c3LpeLwsIifvKT\n/2D79m3odAodHe10d3eRn39uLMi5vX0X/kg7y8uXDrtl5/0iySgmvfGUSyD4w238qfZ5tnfsBjKX\nZflw2WIW+i4Y0FU0reDYDLpQIkw8nRhyNpVJb+rrMqk+6TYpNUUoGSaYCBPqG3+jU3TZAff994XW\ngtMKYAadgekF5zG94DxWn/dRumM92I22Yc9APJ7NaOPi0g+xuGQB77Xt4Jna53mp4TXebNnCVdWX\n0RBsYmvbDqa4J/GVOZ+TActnSK/TD3s9q7Nh1Bu5bdZn+PW+P/B263tYDVa+Pu+Lo7bkx3zfPI72\nNvBSw2v86zv/j3AywkcnXzVkq/Bw6BRddoX/M339peVLmFs0kxfqXmZa/hQu8M4e+oWnwWG0n/X1\nLUdrqZrzC6dx/gjPYBUTz4RqkTsTgVCcf/rFZiLxFF/9+GzmTxt6defBPPLIQ+Tl5dHZ2UlRkQeb\nzcZbb23innt+gE6n4xOf+BgPPvgwxcXFY9oiF08nePLAn9jUN+Ov1F7MZ8+/4ZRdHOFkhH1dB2iL\ndNAW7aA9klkgMpyKYNFbmFU0nXme2ZxfOC17lh1MhHjuyF95vfktVE2lJq+az85fTYHmHfGz3lyT\nVFNsbHyDvxx9iWhfi+LkvGq+NvcLI7p0yemSVoHTo2oq77RupcpVPurLMKTVND/Z+jC1gSPkm918\nd/G3BgR+qbvxTepv/JIWuQ/AkxtrCcdS3HD5eWcc4gAuu+xK/u3ffkAgEOCnP32YDRteJD+/AJ1O\nx3vvbaG1tSW77Vhl64ZgM4/t/g3+SDsVjlIqnOVsanmbf9/yINdOXsnllZcM6DLpjvXwUsNrvN68\nOTv1HzKDsYusBVTnVdIa9rPFv40t/m0YdQZmFEzDZ/PwWtObxNJxvNYiPj7lauYUzcTrccmH0TAY\ndQauqFzGRSULWV/3Mj3xADdOu35MQ5w4fTpFx6KS+R/IsfQ6PV+YdTNPHXyGD5ctllZbIXKIBLlT\nqG0O8MbOViq8Dq6YX35W+5o0aTKRSASPx0dBQSErVlzFXXf9PZ/97I1Mnz6DqqpJ2W1Hq0VqZ8ce\n2iMdlDiKKbWX4DI5soO1X2l8g6cPrSOlpbms4sN8rOYjmTWGvLP49d4/8HTtc+zu3MdnZnyKhJrg\nxbpXeMe/FVVTcZvzWFl1GRXOMrzWIgos7mxXhKZpNIaa2da+i23tu9jR14VqN9r4RM11fLh0cU6v\nsD+a7EYbq6ZcM9bFEONEntnJrbM+PdbFEEKMMOlaPQlV0/jhL7dwpCXIP3z6QqZWnPyit+e6RDrB\n7w88zVstWwb83G60UWovRkPjUM8RHEY7n5nxSWYVzRiwXSgR5on9T7G9fRdGnTE7S63Y5uWKqktZ\n6Js35KWA+rWG22gKNXN+4bQT1i+S7oHxTepv/JK6G9+k/sYv6VodRW/sbOFIS5BF5/vOuRCnaRoN\noSa2t+1iT9d+Ci0FXFG1jGrXiTPeWsNtPLLr1zSHW6l0lrG84sO0RdppDrXSHG7lUM8RNDSm55/H\nLed/atC1zRwmO7fN+gxvtb7LUwefodxRypVVlzK7aMZpz+ArtnsHrFAvhBBCiDMnQW4QkViKp16p\nxWTU8YlLa8a6OEBmYPThQB3b+7oo+1cP1yk66oNNbG3fyVR3DVdWXcqMgqkoisI7rVt5Yv9TJNIJ\nLim7mOvPu/aEtZzi6QSBeC9F1oJThjJFUbioZAGLii/8wC4mLIQQQohTkyA3iGfeOEJvJMn1l0we\n8ctvnQ5N06gPNrLFv413/duzl0+y6C0s9F3APM8sZhRO42ignhfrX2Fv1wEO9NRS7iil2O5li38b\nFr2ZW2fexHzfvEGPYdabTmvpAAlxQgghxLlj1IPcq6++yr/8y7+gaRqrV6/mS1/60oDnm5ubufvu\nu+nq6sLtdvPv//7v+HyZqfgzZsxg+vTpaJpGaWkp//mf/znaxaWpI8yGdxvxuC2s/NDorOs0FH+4\njXf829ji30p7tBMAm8HKxSULmeedw7T8mgFj0qYVTGFawRQagk28WPcK77XtoDHUTJmjhC/Mujl7\nUXQhhBBC5JZRDXKqqnLvvffy+OOP4/V6WbNmDZdffjk1Nce6K++77z5WrVrFddddx+bNm/nRj37E\nv/3bvwFgtVpZu3btaBZxAE3T+O1fD5BWNW64/DyMhg9uNmX/tUVfbnid/X3XVDTpjCzwzWOBbx4z\nCqYOOaGgwlnGrbM+zUcjV3Go5zDzffNkmQEhhBAih41qkNuxYwdVVVWUlZUBcM0117Bhw4YBQa62\ntpa7774bgEWLFvHVr341+9wHPaF268EO9hztZtakAuZNObOVyk9XLBXjrZZ3eaXx9Wzr21R3DUtK\nP8Rsz8wzukyNx1aIx1Y49IZCCCGEGNdGNcj5/X5KSkqyj30+Hzt3DrzW6fTp01m/fj2f+cxnWL9+\nPZFIhEAgQF5eHslkkjVr1mAwGPjiF7/IFVdcMWplVTWNP756GJ2icOMV543KWm6aptEd78nOGG0K\ntbCrYx+xdAyDzsDFJQu5tGIpZY6SoXcmhBBCiAlvVIPccFrU7rzzTu69917Wrl3LggUL8Pl86PWZ\nLs2XX34Zj8dDQ0MDn/3sZ5k2bRoVFacet3ama7K8tauF5o4wly2oYM704jPax2BUTeWthq28cGgj\ndT2NRJIDL9Lutri4bsaVXFnzYVyWs19PZrwbiTV1xNiR+hu/pO7GN6m/iWtUg1xxcTHNzc3Zx36/\nH6934BpiXq+XBx98EIBIJML69etxOBwAeDyZQfoVFRUsWrSIvXv3DhnkzmRRRE3T+O0L+wBYPrdk\nRBZW1DSN3Z37+PPhF2gMNaOg4LN5mJZ/HmX24r6rKxRnl/2IB6E9OLEXdJRFLcc3qb/xS+pufJP6\nG7/O+QWBZ8+eTX19PU1NTXg8HtatW8f9998/YJvu7m7cbjeKovDQQw+xevVqAHp7e7FYLJhMJrq6\nunjvvff44he/OCrl3F/fw+HmXi44r4gyj+OU26bVNId6jrCtfRdNoWY81iJKHcWZm70Yl8nJoZ4j\nPHP4LxwOHEVBYaHvQq6ZdKWMWxNCCCHEiBrVIKfX6/nOd77DrbfeiqZprFmzhpqaGh544AFmz57N\n8uXLefvtt7n//vtRFIWFCxfy3e9+F8hMgvjud7+LXq9HVVW+/OUvD5gkMZKee6sOgKsXVw36fDKd\nZF/3Qba172Jn+x7CqUj2udrA0QHbWg1WoqlM9+ncoplcO3klpY6R66oVQgghhOg34a+1Wtca5J8f\nf4fplW7uvOnCE55vCrXw4Lb/JpgIAZBncjHXM4t5nllMzquiM9ZFc9hPc6iF5lArLWE/RdZCrp50\nJZPyTrxkljg56R4Y36T+xi+pu/FN6m/8Oue7VseDbGvcRSe2xvnDbTy49b8JJkMsL1/KfN9cqlwV\nA65uUGz3UWz3caF3zgdWZiGEEEIImOBBzt8VYcv+Nip9DmZWFwx4riPaxQPbMiHuU1M/ziXlF49R\nKYUQQgghBjehL5z5/OZ6NA2uuah6wLpxPfEAD2x9mJ54gI/XXC0hTgghhBDnpAkb5LqDcTbtasGX\nb2X+1GPXIg0mQjyw9b/pjHVxdfUVXFl16dgVUgghhBDiFCZskHvxnQZSaY2PLK5Cp8u0xoWTER7c\n9t/4I21cXnEJV0+6coxLKYQQQghxchNyjFw4luTlbU24HSYumplZGqQj2slDO/6H5nArS8sWs2rK\nNaNymS4hhBBCiJEyIYPctoMdxBNprr2oCqNBx/6uQzyy69eEUxGWlS9hzXkflRAnhBBCiHPehAxy\nu490ATC3ppBXGt7gqUN/RkHhpumrWVK6aIxLJ4QQQggxPBMuyKmaxu6jXeQ59bzS+TxvtmzBaXRw\n2+xbqHFXj3XxhBBCCCGGbcIFuca2EMF4hKJ5O3mzpY1KZxlfmv1Z8i3usS6aEEIIIcRpmXBBbveR\nLoyV+wjr25jvncvNMz6JSW8c62IJIYQQQpy2CRfk3mnajcHTRImthM+efwN6nX6siySEEEIIcUYm\n1DpygWiYVvtboCl8duanJMQJIYQQYlybUEHu1zv/hGKKUalcQIWzdKyLI4QQQghxViZMkNvXdZA9\noW2oEQcfmXT5WBdHCCGEEOKsTYggF0vFeWLfk6ApqPVzmFFRMNZFEkIIIYQ4axMiyP2p9nk6Y90k\nWyYxtbAao0HGxgkhhBBi/Mv5IHewu5ZXmzbh0heQaqphZrW0xgkhhBAiN+R0kNM0jf898CcUFHyh\nxaDpmTVJgpwQQgghckNOB7mmUAvN4VbmeGZy9LCBPLuJMo99rIslhBBCCDEicjrIbfFvA2CSZTq9\n4QTnVxegKMoYl0oIIYQQYmTkbJDTNI1327Zj0ZtJdBYBMHNS/hiXSgghhBBi5ORskDvSW0dXrJs5\nnpnsq+sF4HyZ6CCEEEKIHJKzQW6LfzsA8wrncKChh3KPA7fDPMalEkIIIYQYOTkZ5NJqmvfatmM3\n2lDCHpIpVbpVhRBCCJFzcjLIHew5TDAR4gLvHPbVBQCYKcuOCCGEECLH5GSQ65+tusA7j3113Rj0\nOqaWu8e4VEIIIYQQIyvnglxSTbGtfSducx417moC4QT5ThMmo1yWSwghhBC5JeeC3J7O/URTMeZ7\n56JTdMQSKSwmw1gXSwghhBBixOVckHu3r1t1vm8uqqYRi6exmqQ1TgghhBC5J6eCXCwZY0fHHjzW\nQiqd5cQTaTTAYpYWOSGEEELknlEPcq+++ipXXXUVK1eu5OGHHz7h+ebmZj73uc/xsY99jFtuuQW/\n3599bu3ataxcuZKVK1fy9NNPD3msLc07SKpJFvjmoSgKsUQaAIu0yAkhhBAiB41qkFNVlXvvvZdH\nHnmEZ599lnXr1lFbWztgm/vuu49Vq1bxzDPP8LWvfY0f/ehHAAQCAX72s5/x5JNP8oc//IGf/vSn\nBIPBUx7vjfotACzwzQMglkgBYJUWOSGEEELkoFENcjt27KCqqoqysjKMRiPXXHMNGzZsGLBNbW0t\nixcvBmDRokXZ519//XWWLFmC0+nE5XKxZMkSXnvttVMeb1vrHsocJRTbfQBE45kWOatMdhBCCCFE\nDhpWkFu3bh2pVOq0d+73+ykpKck+9vl8tLW1Ddhm+vTprF+/HoD169cTiUQIBAKDvvb4btfBpNV0\ntjUOjrXISdeqEEIIIXLRsILcs88+y2WXXcZPfvKTIcPU8TRNG3KbO++8k7fffpvrr7+eLVu24PP5\n0Ov1g75WUZRT7utj01fw4bLF2cf9LXIy2UEIIYQQuWhYCefnP/85TU1N/O53v2P16tVceOGF3HTT\nTdku0ZMpLi6mubk5+9jv9+P1egds4/V6efDBBwGIRCKsX78eh8NBcXExmzdvzm7X2to65PFunrtq\nwGNTXXfmGIV2PB7n0G9UjDmpp/FN6m/8krob36T+Jq5hN1WVlZXxjW98g+XLl/P3f//3vPbaa5SX\nl3PPPfewYMGCQV8ze/Zs6uvraWpqwuPxsG7dOu6///4B23R3d+N2u1EUhYceeojVq1cDsHTpUn78\n4x8TDAZRVZVNmzbxzW9+c8hytrcfmxDR1hEGIJlIDfi5ODd5PE6pp3FM6m/8krob36T+xq+RCODD\nCnKJRILnnnuO3/72t6TTaf7u7/6Oq6++mh07dnDnnXfy0ksvDfo6vV7Pd77zHW699VY0TWPNmjXU\n1NTwwAMPMHv2bJYvX87bb7/N/fffj6IoLFy4kO9+97sA5OXl8dWvfpXVq1ejKAq33347LpfrtN5c\nNN43a1XGyAkhhBAiBynaMAayLV26lEWLFnHzzTdzwQUXDHju29/+Nj/84Q9HrYCn6/izkidfqeW5\nt+q4++b5TCnPG8NSieGQs8rxTepv/JK6G9+k/savD6xF7o9//OMJY9v6nUsh7v2i/bNWzdIiJ4QQ\nQojcM6xZq08//TQ9PT3Zx93d3fziF78YtUKNlFi2a1VmrQohhBAi9wx7HTm32519nJ+fz7PPPjtq\nhRop2Ut0SYucEEIIIXLQsILcYMPo0un0iBdmpPVPdpAFgYUQQgiRi4YV5Kqrq3nsscfQNA1VVXn0\n0UeprKwc7bKdtWgijcmoQ68b1SuRCSGEEEKMiWElnG9/+9u8/PLLzJkzh3nz5rFx48bsMiHnslgi\njUXGxwkhhBAiRw0r5fh8Pn75y18SiUQAsNlso1qokRKLp2QNOSGEEELkrGE3VwWDQY4cOUI8Hs/+\nbOHChaNSqJESTaRwO81jXQwhhBBCiFExrCD33HPPcd9999Hb24vX66W+vp7p06ezdu3a0S7fGVNV\njURSlRY5IYQQQuSsYY2R+6//+i/++Mc/UlVVxQsvvMAvfvELZs+ePdplOyux/sWAZYycEEIIIXLU\nsIKcwWCgsLAwu+TIkiVL2Llz56gW7Gz1ryFnlTXkhBBCCJGjhtVcZTKZ0DSNqqoqfvWrX1FWVpad\n+HCuOraGnLTICSGEECI3DSvl3HHHHYRCIb75zW/yve99j2AwyD333DPaZTsrUbmqgxBCCCFy3JBB\nLp1OU19fz0UXXYTT6eTxxx//AIp19mSMnBBCCCFy3ZBj5PR6Pb///e8/iLKMqFi8b4yczFoVQggh\nRI4a1mSHRYsW8Ze//GW0yzKi+sfIWc3SIieEEEKI3DSslLN27Voee+wxLBYLVqsVTdNQFIU333xz\ntMt3xvpnrVqkRU4IIYQQOWpYQe6pp54a7XKMuGj/GDlpkRNCCCFEjhpWyikrKxvtcoy4Y2PkJMgJ\nIYQQIjcNK+UsXrwYRVFO+Pm53bXaP2tVulaFEEIIkZtOu2s1Ho/z5z//GYPh3G7pimav7HBul1MI\nIYQQ4kwNa9ZqWVlZ9jZ58mTuuOMONm7cONplOyvHruwgLXJCCCGEyE3DCnLv19DQQGdn50iXZUT1\nz1o1S5ATQgghRI467TFyqqqSSqX49re/PaoFO1uxeAqLSY9ukLF9QgghhBC54LTHyBkMBoqKitDr\nz+2WrmgiJePjhBBCCJHThtW1Gg6Hyc/Pp6ysDJ/PRzwe5+DBg6NdtrMSS6RlfJwQQgghctqwgtw/\n/MM/YDQas48NBgN33XXXqBVqJETjaSyyhpwQQgghctiwglw6nR4Q5EwmE+l0etQKdbZSaZVUWpUW\nOSGEEELktGEFOYPBQENDQ/ZxfX39OT1GLiZryAkhhBBiAhhW0rn99tu58cYbWbZsGQAbN27kBz/4\nwagW7Gz0ryFnlRY5IYQQQuSwYQW55cuX8+tf/5o33ngDgC996UtUVVWNasHORn+LnIyRE0IIIUQu\nG1bS6erqorS0lE9/+tMAJJNJurq6KCgoGNXCnansVR3M0iInhBBCiNw1rDFyX/7ylwdMbkgmk3zl\nK18Z1gFeffVVrrrqKlauXMnDDz98wvMtLS3ccsstrFq1iuuuuy576a+mpibmzp3LqlWrWLVqFd/7\n3veGdTyAWKKva1XGyAkhhBAihw0r6SQSCaxWa/axzWYjHo8P+TpVVbn33nt5/PHH8Xq9rFmzhssv\nv5yamprsNj//+c+5+uqrueGGG6itreW2227jpZdeAqCyspK1a9ee7ns6rmtVWuSEEEIIkbuGfa3V\nrq6u7L87OztRVXXI1+zYsYOqqirKysowGo1cc801bNiwYcA2iqIQCoUA6O3txefzDbdIJ3VssoO0\nyAkhhBAidw0r6XzmM5/hxhtv5LrrrkPTNJ555hluu+22IV/n9/spKSnJPvb5fOzcuXPANrfffju3\n3norv/rVr4jFYjz22GPZ5xobG7n++uux2+3ccccdLFiwYFhvKhrva5GTMXJCCCGEyGHDCnJr1qyh\nsrKSV155BYAf/vCHwwpVmqYNuc26detYvXo1n/vc59i2bRvf+ta3WLduHR6Ph1deeYW8vDx2797N\n1772NdatW4fdbh9yn/1j5GTWqhBCCCFy2bCSTjAY5LXXXuPgwYPEYjF27doFwC9/+ctTvq64uJjm\n5ubsY7/fj9frHbDNk08+ySOPPALAvHnziMfj2RmxJpMJgJkzZ1JRUcHRo0eZOXPmKY/p8ThR+hYr\nLvW58Hicw3mL4hwh9TW+Sf2NX1J345vU38Q1rCB39913U1NTw9GjR7njjjt46qmnhgxUALNnz6a+\nvp6mpiY8Hg/r1q3j/vvvH7BNaWkpmzZtYtWqVdTW1pJIJCgoKKCrqwu3241Op6OhoYH6+noqKiqG\nPGZ7e5CunggAsUic9vbgcN6iOAd4PE6pr3FM6m/8krob36T+xq+RCODDCnJ1dXU8+OCDbNiwgWuv\nvZYVK1Zwyy23DPk6vV7Pd77zHW699VY0TWPNmjXU1NTwwAMPMHv2bJYvX85dd93FP/3TP/H444+j\n0+m47777ANiyZQsPPPAABoMBnU7H97//fVwu17DelMxaFUIIIcREMKwg19/FaTQa6enpIS8vb8As\n1lO55JJLuOSSSwb87G//9m+z/66pqeG3v/3tCa9bsWIFK1asGNYx3i/aP0ZO1pETQgghRA4bVtKp\nrq6mp6eHj370o3zqU5/C6XQOq2t1rMQSaRQFTIZhr64ihBBCCDHuDCvI/cd//AcAn//855k9ezbB\nYPCEVrZzSSyewmoyoCjKWBdFCCGEEGLUnHbf43DXchtL0Xgaq6whJ4QQQogcl5N9j7FEStaQE0II\nIUTOy7kgp2kasURaruoghBBCiJyXc0EumVJJq5pcZ1UIIYQQOS/ngpysISeEEEKIiSLngpysISeE\nEEKIiSLnglwsnmmRk65VIYQQQuS63Aty/S1y0rUqhBBCiByXc0Eu2t8iJ12rQgghhMhxuRfksmPk\npEVOCCGEELkt54KczFoVQgghxESRe0EunmmRk8kOQgghhMh1ORfkojLZQQghhBATRM4FuZhMdhBC\nCCHEBJFzQU4WBBZCCCHERJFzQU4mOwghhBBiosi9ICeTHYQQQggxQeRckIsm0hj0CkZDzr01IYQQ\nQogBci7txBJpLNIaJ4QQQogJIOeCXDSekvFxQgghhJgQci7IxRIpWXpECCGEEBNCTgU5TdOIxdPS\nIieEEEKICSGnglwskUZDFgMWQgghxMSQU0EuEksCsoacEEIIISaGnApy0Xj/dValRU4IIYQQuS+n\nglwk1rcYsFla5IQQQgiR+3IqyEVj0iInhBBCiIkjp4JcJHt5LmmRE0IIIUTuy6kgF433TXaQWatC\nCH7eGU0AAA3bSURBVCGEmAByK8hlu1alRU4IIYQQuW/Ug9yrr77KVVddxcqVK3n44YdPeL6lpYVb\nbrmFVatWcd1117Fx48bscw899BArVqzgIx/5CK+//vqQx8p2rUqLnBBCCCEmgFFNPKqqcu+99/L4\n44/j9XpZs2YNl19+OTU1Ndltfv7zn3P11Vdzww03UFtby2233cZLL73EoUOHeP7553nuuedobW3l\n85//POvXr0dRlJMeLztrVSY7CCGEEGICGNUWuR07dlBVVUVZWRlGo5FrrrmGDRs2/P/27j6m6vL/\n4/jrnIMWGnjHzZFCp1CKDrXNRrpGKk5SNCUorbVUWtYfRYbdTE1dpvJzOVauZeoopzlbc95Mj2WF\nAi4nzMybLewPsiEoBxXtICIn5Pr+4TzfHylYfTscPuc8H/94zsfr+LnOrr3da9f1OdfVpo3NZtPV\nq1clSR6PR7GxsZKkAwcOaMqUKQoLC9MDDzyggQMH6uTJkx3e77/7yLG0CgAAgp9fp67cbrf69+/v\nex8bG6tTp061afPqq68qJydHW7Zs0fXr1/X555/7Pjtq1Kg2n3W73R3ez3eyA/vIAQCAEODXIGeM\nuWsbl8ulrKwszZkzR8ePH9dbb70ll8t1x892tKwq/XdpNT6ut+7r0f2fdRoBFR0dEegu4H/A+FkX\nY2dtjF/o8muQczqdOnfunO+92+1WTExMmzbbt29XYWGhJGnUqFFqbm5WfX29nE6nzp8/72tXW1t7\n22f/7NbS6tWGJjU1Nv9bXwOdJDo6QhcuNAS6G/iHGD/rYuysjfGzrn8jgPv1Gbnk5GRVVVWppqZG\nXq9XLpdLaWlpbdrExcXp8OHDkqTKykp5vV717dtXEyZM0L59++T1enX27FlVVVVpxIgRHd7vWnOL\nuofZ5bAH1a4qAAAAd+TXGTmHw6ElS5YoJydHxhhlZ2crISFBa9euVXJyssaPH6933nlH7777rjZt\n2iS73a7Vq1dLkhITEzV58mRlZGQoLCxMy5Ytu+vSatP1P9gMGAAAhAyb+SsPslnE7Pe+UZjDrv97\neUygu4J/gOUBa2P8rIuxszbGz7q6/NJqZ7t2vYU95AAAQMgIqiB33XuDPeQAAEDICKogJ3E8FwAA\nCB1BF+TYDBgAAISK4AtyPCMHAABCRNAFuXCekQMAACEi6IIc+8gBAIBQEXxBjhk5AAAQIoIuyLGP\nHAAACBVBF+T6RN4T6C4AAAB0iqCaviqYn6pIth8BAAAhIqhm5B6M7yO7zRbobgAAAHSKoApyAAAA\noYQgBwAAYFEEOQAAAIsiyAEAAFgUQQ4AAMCiCHIAAAAWRZADAACwKIIcAACARRHkAAAALIogBwAA\nYFEEOQAAAIsiyAEAAFgUQQ4AAMCiCHIAAAAWRZADAACwKIIcAACARRHkAAAALIogBwAAYFEEOQAA\nAIsK8/cNSktLtWrVKhljlJWVpXnz5rX5+/z8fJWVlclms+natWu6fPmyysvLJUlJSUkaOnSojDGK\ni4vTJ5984u/uAgAAWIZfg1xra6vef/99bdq0STExMcrOzlZaWpoSEhJ8bRYuXOh7/cUXX6iiosL3\nPjw8XDt37vRnFwEAACzLr0urJ0+e1MCBA3X//ferW7duysjIUFFRUbvt9+7dq6lTp/reG2P82T0A\nAABL82uQc7vd6t+/v+99bGys6urq7tj23Llzqqmp0aOPPuq79scffyg7O1uzZs3S999/78+uAgAA\nWI5fl1b/zoyay+VSenq6bDab79rBgwcVHR2ts2fPavbs2RoyZIji4+P90VUAAADL8WuQczqdOnfu\nnO+92+1WTEzMHdvu27dPy5Yta3MtOjpakhQfH6+UlBRVVFTcNchFR0f8j71GIDF+1sb4WRdjZ22M\nX+jy69JqcnKyqqqqVFNTI6/XK5fLpbS0tNva/frrr/J4PBo1apTvmsfjkdfrlSTV19fr2LFjbX4k\nAQAAEOr8OiPncDi0ZMkS5eTkyBij7OxsJSQkaO3atUpOTtb48eMl3ZyNy8jIaPPZyspKLV26VA6H\nQ62trXr55ZcJcgAAAP+PzfDTUAAAAEviZAcAAACLIsgBAABYFEEOAADAooImyJWWluqJJ55Qenq6\nNmzYEOjuoAO1tbV64YUXNGXKFE2bNk2bN2+WJP3+++/KyclRenq6XnzxRTU0NAS4p+hIa2urMjMz\n9corr0iSqqur9cwzzyg9PV15eXlqaWkJcA/RnoaGBuXm5mry5MnKyMjQiRMnqD+L2LRpk6ZOnapp\n06ZpwYIF8nq91F4XtmjRIo0dO1bTpk3zXeuo1lasWKFJkyZp+vTpbY4s7UhQBLlbZ7oWFhZq7969\ncrlcqqysDHS30A6Hw6GFCxdq3759+vLLL7V161ZVVlZqw4YNGjNmjPbv36+UlBStX78+0F1FBzZv\n3tzml+Rr1qzR3LlztX//fkVERGj79u0B7B06snLlSj3++OP6+uuvtXv3bg0ePJj6swC3260tW7Zo\nx44d2rNnj27cuCGXy0XtdWFPPfWUCgsL21xrr9ZKSkpUVVWlb7/9VsuXL79tb932BEWQ+7tnuiKw\noqOjlZSUJEnq2bOnEhIS5Ha7VVRUpMzMTElSZmYmx7J1YbW1tSopKdHTTz/tu3bkyBGlp6dLujl+\n3333XaC6hw5cvXpVR48eVVZWliQpLCxMERER1J9FtLa2qqmpSS0tLbp+/bpiYmJUVlZG7XVRo0eP\nVmRkZJtrf661W3mlqKhIM2bMkCSNHDlSDQ0Nunjx4l3vERRB7u+c6Yqupbq6WqdPn9bIkSN16dIl\nRUVFSboZ9i5fvhzg3qE9q1at0ttvv+07Uu/y5cvq1auX7Pab/6U4nU5qsIuqrq5Wnz59tHDhQmVm\nZmrJkiVqamqi/iwgNjZWc+fO1bhx45SamqqIiAgNGzZMkZGR1J6F1NfXt6m1+vp6SVJdXZ2cTqev\nXWxsrNxu913/vaAIcmyFZ02NjY3Kzc3VokWL1LNnzzbn7KLrKi4uVlRUlJKSkny1Z4y5rQ4Zz66p\npaVFP//8s5577jnt3LlT4eHh2rBhA+NlAR6PR0VFRTp48KAOHTqkpqYmlZaW3taOsbSmO2WZvzKW\nfj3ZobP8nTNd0TW0tLQoNzdX06dP18SJEyVJ/fr108WLFxUVFaULFy6ob9++Ae4l7uTYsWM6cOCA\nSkpK1NzcrMbGRq1atUoNDQ1qbW2V3W5XbW0tNdhFOZ1OOZ1OJScnS5ImTZqkjRs3Un8WcPjwYcXH\nx6t3796SpIkTJ+qnn36Sx+Oh9iykvVqLjY1VbW2tr91fHcugmJH7q2e6outYtGiREhMTNXv2bN+1\nCRMmaMeOHZKknTt3MoZdVF5enoqLi1VUVKSCggKlpKRozZo1SklJ0TfffCOJ8evKoqKi1L9/f505\nc0bSzWcbExMTqT8LiIuL04kTJ9Tc3CxjjI4cOaIHH3yQ2uvi/jzT1l6tpaWladeuXZKk48ePKzIy\n0rcE25GgOaKrtLRUK1eu9J3pOm/evEB3Ce348ccf9fzzz+uhhx6SzWaTzWbTG2+8oREjRmj+/Pk6\nf/684uLi9NFHH932kCi6lvLycn322Wf69NNPdfbsWeXl5cnj8SgpKUkffPCBunXrFugu4g5Onz6t\nxYsXq6WlRfHx8crPz9eNGzeoPwv4+OOP5XK5FBYWpmHDhmnFihWqra2l9rqoBQsWqKysTFeuXFFU\nVJRee+01TZw4Ua+//voda2358uU6dOiQwsPDlZ+fr+HDh9/1HkET5AAAAEJNUCytAgAAhCKCHAAA\ngEUR5AAAACyKIAcAAGBRBDkAAACLIsgBAABYFEEOAP4l5eXlvsPoAaAzEOQA4F/EOZcAOlNQnLUK\nAHdz8uRJrVmzRo2NjZKk3NxcJSYmKisrS5mZmfrhhx8kSUuXLtXo0aMlSbt27VJhYaHsdrsGDBig\n9957z3cu4vr167V3717Z7Xb16NFD27Ztk3TzHOGlS5fq+PHjstvtKigo0ODBgwPwjQGEBAMAQc7j\n8ZgZM2aYCxcuGGOMqaurM6mpqaaiosIMGTLE7N692xhjTHl5uUlNTTVer9f88ssv5rHHHjMXL140\nxhjz4Ycfmvnz5xtjjNmxY4eZOXOmaWxsNMYYc+XKFWOMMWVlZWb48OGmoqLCGGPMunXrzJtvvtmp\n3xVAaGFGDkDQO3bsmKqrq/XSSy/5DrB2OBxqaWlR9+7d9eSTT0qSHnnkEd177706c+aMysvLNW7c\nOPXr10+SNGvWLE2fPl2SVFxcrGeffVY9evSQJPXq1ct3r0GDBmno0KGSpJEjR6q4uLizviaAEESQ\nAxAShg4dqi1btrS5VlNTc1s7Y4xsNpvvzz9fv/W6Pffcc4/v9a2wCAD+wo8dAAS9hx9+WL/99pvK\nysp8106dOiVjjLxer/bs2SNJOnr0qLxerwYNGqQxY8aopKREly5dkiR99dVXGjt2rCRpwoQJ2rZt\nm+95uytXrnTyNwKAm5iRAxD0IiMjtW7dOq1evVr5+fnyer0aMGCAFi9erN69e6uiokIbN26UJBUU\nFCgsLEyJiYnKy8vTnDlzZLfbFR8fr+XLl0uSZsyYobq6Os2cOVMOh0P33Xeftm7dGsivCCBE2UxH\nawQAEMRqamqUlZWlI0eOBLorAPCPsLQKIKSx7xsAK2NGDgAAwKKYkQMAALAoghwAAIBFEeQAAAAs\niiAHAABgUQQ5AAAAiyLIAQAAWNR/AOnHOb4V3YD+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x55f69bb1d690>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 43s, sys: 22 s, total: 4min 5s\n",
      "Wall time: 2min 46s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# Specify the model hyperparameters (NOTE: All the defaults can be omitted):\n",
    "model_params = {\n",
    "        'hidden_sizes' : [512, 512], # List of hidden layer dimensions, empty for linear model.\n",
    "        'l2_lambda' : 1e-3                     # Strength of L2 regularization.\n",
    "}\n",
    "\n",
    "# Specify the training hyperparameters:\n",
    "training_params = {\n",
    "        'num_epochs' : 200,     # Max epochs/iterations to train for.\n",
    "        'report_every' : 1,     # Report training results every nr of epochs.\n",
    "        'eval_every' : 1,         # Evaluate on validation data every nr of epochs.\n",
    "        'stop_early' : True,    # Use early stopping or not.\n",
    "}\n",
    "\n",
    "# Build, train, evaluate and plot the results!\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True    # Modify as desired.\n",
    ")\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4F5tXNkiHSOQ"
   },
   "source": [
    "You should get around **97.4**%. Shouldn't deeper do better?! Why is it that the 2-hidden layer model\n",
    "\n",
    "* ***took much longer to train*** (2min 46s versus 1min 3s on our system), and\n",
    "* ***got roughly the same accuracy*** as the 1 hidden layer model (sometimes worse)?\n",
    "\n",
    "This illustrates the **fundamental difficulty of training deep networks** that have plagued deep learning research for decades (and to some extent, still do): \n",
    "\n",
    "> ***Although deeper networks can give you more powerful models, training those models to find the right parameters is not always easy & takes a lot of computing power (time)!***\n",
    "\n",
    "For a long time people just believed it wouldn't work because a) fewer people tried to make it work, and those who did b) didn't have enough data or computing power to really explore the vast space of hyperparameters. These days, we have more data and more compute power, however one can never have 'enough' resources :) This is where the art of doing proper hyper-parameter selection comes in. In fact, we should be able to squeeze out another percentage point or so by choosing better:\n",
    "\n",
    "* Optimizer + its hyperparams (learning rate, momentum rates, etc)\n",
    "* batch-size,\n",
    "* choice of regularization.\n",
    "\n",
    "For a new problem, we will typically spend most of our time exploring these choices, usually just by launching many different training runs (hopefully in parallel and using GPUs!) and keeping the best ones.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tzyUXqInbqaB"
   },
   "source": [
    "# EXTRA: How to Choose Architectures?\n",
    "\n",
    "How does one design new architectures? Should you use:\n",
    "\n",
    "* a tapered architecture (large-medium-small),\n",
    "* a \"regular\" (like the Levi's :) architecture (large-med-med-...-small),\n",
    "* a \"bottlenecked\" architecutre (large-small-large)?\n",
    "* an \"over-complete\" architecture (small-large-small). \n",
    "\n",
    "Unfortunately it mostly comes down to just developing your own intuition over time, and trying different approaches in hyperparameter search. \n",
    "\n",
    "However, a good pattern to follow in general is something like the following:\n",
    "\n",
    "1. Start with a basic architecture and overfit a portion of your training data.\n",
    "2. As you train on more data, add capacity and prevent overfitting by\n",
    "    * Adding more units to the hidden layer (often times wide still beats deep)\n",
    "    * Add one or two more hidden layers, trying some architectural choices mentioned above.\n",
    "    * Add dropout.\n",
    "    \n",
    "The goal is to match your architecture to your data, meaning you have just enough capacity to fit the data, but not too much to easily overfit. The easiest way to do this is to gradually build up the capacity of your model in this way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vkrPMrQoSfK0"
   },
   "source": [
    "# EXTRA: Hyperparameter Selection (USE A GPU INSTANCE FOR THIS)\n",
    "\n",
    "## Random grid-search\n",
    "\n",
    "Most of the deep learning practitioner's time will be spent training and evaluating new model architectures with different hyperparameter combinations.    General rules of thumb exist, but often times change for each new architecture or dataset. Several approaches exist for automatic hyperparameter selection (grid search, random search, Bayesian methods). In practice, most people just use a **randomized grid-search** approach to try out different hyperparams, where one defines sensible values for each hyperparameter, and then randomly samples from the (cross-product space) of all of these possible combinations. This space grows exponentially, making exhaustive search infeasible for all but the simplest models. In practice, randomized search have also been found to find the best values, quicker (see [Bergstra and Bengio 2009](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf) if you're interested in the details).\n",
    "\n",
    "Below, we will illustrate the basic idea with a skeleton randomized grid search implementation. In practice, one would launch these different runs in parallel on a computing cluster, making it easier to explore multiple options at the same time. Here, we would just try out a few different options to get a sense for how that would work. \n",
    "\n",
    "**NOTE**: We will keep the models small here (so we won't get SOTA results), in order to keep the training times reasonable, but to illustrate how dependent results are on these choices.\n",
    "\n",
    "**Architecture Selection**: Let's consider 1 and 2-hidden layer models. For each layer, we'll try out [128, 256] neurons per layer. We'll stick to ReLUs for this.\n",
    "\n",
    "**Hyperparameters**: Let's pick SGD with Momentum as our optimizer (a good, stable workhorse), with a fixed computational budget of 20 training epochs. We'll need to pick ranges for the following:\n",
    "* *learning rate*: we'll use a log-scale from 1e-5 to 1.\n",
    "* *momentum coefficient*: we'll use a log-scale from 5e-1 (0.5) to 1.\n",
    "* *L2 regularization*: we'll use a log-scale from 1e-5 to 1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 71,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 522,
     "status": "ok",
     "timestamp": 1503658401647,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "kyf_7dWqd_p1",
    "outputId": "cb4464de-cff7-4eee-a798-e11eaef911ec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'hidden_sizes': [256, 128], 'l2_lambda': 0.0054235896697678509}\n",
      "{'eval_every': 1, 'report_every': 1, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x55725cae8a90>, 'num_epochs': 20, 'stop_early': True}\n"
     ]
    }
   ],
   "source": [
    "def sample_log_scale(v_min=1e-6, v_max=1.):\n",
    "    '''Sample uniformly on a log-scale from 10**v_min to 10**v_max.'''\n",
    "    return np.exp(np.random.uniform(np.log(v_min), np.log(v_max)))\n",
    "\n",
    "\n",
    "def sample_model_architecture_and_hyperparams(max_num_layers=2,\n",
    "                                              lr_min=1e-6,\n",
    "                                              lr_max=1.,\n",
    "                                              mom_min=0.5,\n",
    "                                              mom_max=1.,\n",
    "                                              l2_min=1e-4,\n",
    "                                              l2_max=1.):\n",
    "        '''Generate a random model architecture & hyperparameters.'''\n",
    "        \n",
    "        # Sample the architecture.\n",
    "        num_layers = np.random.choice(range(1, max_num_layers+1))\n",
    "        hidden_sizes = []\n",
    "        layer_ranges=[128, 256]\n",
    "        for l in range(num_layers):\n",
    "            hidden_sizes.append(np.random.choice(layer_ranges))\n",
    "        \n",
    "        # Sample the training parameters.\n",
    "        l2_lambda = sample_log_scale(l2_min, l2_max)\n",
    "        lr = sample_log_scale(lr_min, lr_max)\n",
    "        mom_coeff = sample_log_scale(mom_min, mom_max)\n",
    "        \n",
    "        # Build base model definitions:\n",
    "        model_params = {\n",
    "                'hidden_sizes' : hidden_sizes,\n",
    "                'l2_lambda' : l2_lambda}\n",
    "\n",
    "        # Specify the training hyperparameters:\n",
    "        training_params = {\n",
    "                'num_epochs' : 20,\n",
    "                'optimizer_fn' : tf.train.MomentumOptimizer(\n",
    "                        learning_rate=lr, \n",
    "                        momentum=mom_coeff),\n",
    "                'report_every' : 1,\n",
    "                'eval_every' : 1,\n",
    "                'stop_early' : True}\n",
    "        \n",
    "        return model_params, training_params\n",
    "\n",
    "# TEST THIS: Run this cell a few times and look at the different outputs. \n",
    "# Each of these will be a different model trained with different hyperparameters.\n",
    "m, t = sample_model_architecture_and_hyperparams()\n",
    "print m\n",
    "print t\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VPXaIIZteNU-"
   },
   "source": [
    "We will play around with the strength of the L2 regularization. Let's use SGD+Momentum, and run for a fixed budget of 20 epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 1108,
     "output_extras": [
      {
       "item_id": 10
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 189573,
     "status": "ok",
     "timestamp": 1503227140737,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "Kyb7EOJWiehX",
    "outputId": "f221c2a9-4443-4a6d-977c-5b65e5087b74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RUN: 0 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.017639122106750327}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca7c8c90>}\n",
      "Validation loss stopped improving, stopping training early after 3 epochs!\n",
      "Optimization Finished!\n",
      "RUN: 1 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.00071915651503389648}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca247f10>}\n",
      "Optimization Finished!\n",
      "RUN: 2 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.034917746813186976}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca7c8a10>}\n",
      "Validation loss stopped improving, stopping training early after 5 epochs!\n",
      "Optimization Finished!\n",
      "RUN: 3 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [128, 128], 'l2_lambda': 0.0011477153620692811}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecba07d10>}\n",
      "Optimization Finished!\n",
      "RUN: 4 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [128, 128], 'l2_lambda': 0.0066110195159220803}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556e59bd68d0>}\n",
      "Optimization Finished!\n",
      "RUN: 5 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.00064575036694946794}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecabea5d0>}\n",
      "Optimization Finished!\n",
      "RUN: 6 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [128, 128], 'l2_lambda': 0.013143246333050255}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eb9135650>}\n",
      "Optimization Finished!\n",
      "RUN: 7 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [128, 128], 'l2_lambda': 0.00017646189551092479}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ec9811090>}\n",
      "Optimization Finished!\n",
      "RUN: 8 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.00012706891665017044}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecb78fc10>}\n",
      "Validation loss stopped improving, stopping training early after 9 epochs!\n",
      "Optimization Finished!\n",
      "RUN: 9 out of 10:\n",
      "Sampled Architecture: \n",
      "{'hidden_sizes': [256], 'l2_lambda': 0.0029062557341502037}\n",
      "Hyper-parameters:\n",
      "{'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca24d890>}\n",
      "Optimization Finished!\n"
     ]
    }
   ],
   "source": [
    "results = []\n",
    "\n",
    "# Perform a random search over hyper-parameter space this many times.\n",
    "NUM_EXPERIMENTS = 10\n",
    "\n",
    "for i in range(NUM_EXPERIMENTS): \n",
    "    \n",
    "    # Sample the model and hyperparams we are using.\n",
    "    model_params, training_params = sample_model_architecture_and_hyperparams()\n",
    "    \n",
    "    print \"RUN: %d out of %d:\" % (i, NUM_EXPERIMENTS)\n",
    "    print \"Sampled Architecture: \\n\", model_params\n",
    "    print \"Hyper-parameters:\\n\", training_params\n",
    "    \n",
    "    # Build, train, evaluate\n",
    "    model, performance = build_train_eval_and_plot(\n",
    "            model_params, training_params, verbose=False)\n",
    "    \n",
    "    # Save results\n",
    "    results.append((performance['test_acc'], model_params, training_params))\n",
    "    \n",
    "# Display (best?) results/variance/etc:\n",
    "results.sort(key=lambda x : x[0], reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 207,
     "output_extras": [
      {
       "item_id": 1
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 65,
     "status": "ok",
     "timestamp": 1503228968350,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "A6E0f3M02xB9",
    "outputId": "87d3d8a3-36dc-4f5a-d720-4ef6762f986a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.95090008, {'hidden_sizes': [128, 128], 'l2_lambda': 0.0066110195159220803}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556e59bd68d0>})\n",
      "(0.94180018, {'hidden_sizes': [256], 'l2_lambda': 0.00064575036694946794}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecabea5d0>})\n",
      "(0.92350012, {'hidden_sizes': [128, 128], 'l2_lambda': 0.013143246333050255}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eb9135650>})\n",
      "(0.8925001, {'hidden_sizes': [256], 'l2_lambda': 0.034917746813186976}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca7c8a10>})\n",
      "(0.86520004, {'hidden_sizes': [128, 128], 'l2_lambda': 0.0011477153620692811}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecba07d10>})\n",
      "(0.71679997, {'hidden_sizes': [256], 'l2_lambda': 0.0029062557341502037}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca24d890>})\n",
      "(0.69389999, {'hidden_sizes': [256], 'l2_lambda': 0.017639122106750327}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca7c8c90>})\n",
      "(0.28809997, {'hidden_sizes': [256], 'l2_lambda': 0.00071915651503389648}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556eca247f10>})\n",
      "(0.20929998, {'hidden_sizes': [256], 'l2_lambda': 0.00012706891665017044}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ecb78fc10>})\n",
      "(0.1135, {'hidden_sizes': [128, 128], 'l2_lambda': 0.00017646189551092479}, {'stop_early': True, 'report_every': 1, 'eval_every': 1, 'num_epochs': 20, 'optimizer_fn': <google3.third_party.tensorflow.python.training.momentum.MomentumOptimizer object at 0x556ec9811090>})\n"
     ]
    }
   ],
   "source": [
    "for r in results:\n",
    "    print r    # Tuples of (test_accuracy, model_hyperparameters, training_hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cgu2OxMS41sF"
   },
   "source": [
    "**Notice the huuuuge variance in the test accuracies!** This is just a toy run, but hopefully it illustrates how important choosing the right architectures and training hyperparameters is to getting good results in deep learning.\n",
    "\n",
    "Below, we've included some hyperparameter settings which achieve near state-of-the-art results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bsCJW0_gRFX8"
   },
   "source": [
    "# EXTRA: Known Good Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "x0pO_4fQm8xi"
   },
   "source": [
    "## 784-500-300-10 w/ L2 + SGD + Momentum\n",
    "\n",
    "**Best so far: 98.02% test accuracy when we ran this (well, in one of our training runs :).**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "autoexec": {
      "startup": false,
      "wait_interval": 0
     },
     "height": 3602,
     "output_extras": [
      {
       "item_id": 49
      },
      {
       "item_id": 88
      },
      {
       "item_id": 89
      },
      {
       "item_id": 90
      },
      {
       "item_id": 91
      }
     ]
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 129328,
     "status": "ok",
     "timestamp": 1503057210512,
     "user": {
      "displayName": "Stephan Gouws",
      "photoUrl": "//lh4.googleusercontent.com/-6znVyM1oxdg/AAAAAAAAAAI/AAAAAAAAABI/vEPo2Ce7Rpc/s50-c-k-no/photo.jpg",
      "userId": "102606466886131565871"
     },
     "user_tz": -60
    },
    "id": "J1spLw65l7mY",
    "outputId": "ac425ebc-96c1-4025-e493-ad3818bfc5b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0001 Training cost= 327.262532238\n",
      "Epoch: 0001 Validation acc= 0.435399979\n",
      "Epoch: 0002 Training cost= 251.312214827\n",
      "Epoch: 0002 Validation acc= 0.542799890\n",
      "Epoch: 0003 Training cost= 224.936292614\n",
      "Epoch: 0003 Validation acc= 0.605999947\n",
      "Epoch: 0004 Training cost= 201.409107805\n",
      "Epoch: 0004 Validation acc= 0.626999915\n",
      "Epoch: 0005 Training cost= 180.312732350\n",
      "Epoch: 0005 Validation acc= 0.656599879\n",
      "Epoch: 0006 Training cost= 161.427582259\n",
      "Epoch: 0006 Validation acc= 0.692799866\n",
      "Epoch: 0007 Training cost= 144.570377253\n",
      "Epoch: 0007 Validation acc= 0.730599880\n",
      "Epoch: 0008 Training cost= 129.454500316\n",
      "Epoch: 0008 Validation acc= 0.729999840\n",
      "Epoch: 0009 Training cost= 115.930559138\n",
      "Epoch: 0009 Validation acc= 0.764599860\n",
      "Epoch: 0010 Training cost= 103.779861076\n",
      "Epoch: 0010 Validation acc= 0.817999840\n",
      "Epoch: 0011 Training cost= 92.894099121\n",
      "Epoch: 0011 Validation acc= 0.848599792\n",
      "Epoch: 0012 Training cost= 83.177957764\n",
      "Epoch: 0012 Validation acc= 0.874799788\n",
      "Epoch: 0013 Training cost= 74.441788441\n",
      "Epoch: 0013 Validation acc= 0.866999805\n",
      "Epoch: 0014 Training cost= 66.607071221\n",
      "Epoch: 0014 Validation acc= 0.908799767\n",
      "Epoch: 0015 Training cost= 59.628462719\n",
      "Epoch: 0015 Validation acc= 0.920999765\n",
      "Epoch: 0016 Training cost= 53.376457901\n",
      "Epoch: 0016 Validation acc= 0.927999735\n",
      "Epoch: 0017 Training cost= 47.795069421\n",
      "Epoch: 0017 Validation acc= 0.921199799\n",
      "Epoch: 0018 Training cost= 42.801629826\n",
      "Epoch: 0018 Validation acc= 0.938799739\n",
      "Epoch: 0019 Training cost= 38.321246601\n",
      "Epoch: 0019 Validation acc= 0.939399719\n",
      "Epoch: 0020 Training cost= 34.317467527\n",
      "Epoch: 0020 Validation acc= 0.943799734\n",
      "Epoch: 0021 Training cost= 30.729907542\n",
      "Epoch: 0021 Validation acc= 0.950399756\n",
      "Epoch: 0022 Training cost= 27.518849869\n",
      "Epoch: 0022 Validation acc= 0.948799729\n",
      "Epoch: 0023 Training cost= 24.648165835\n",
      "Epoch: 0023 Validation acc= 0.949599802\n",
      "Epoch: 0024 Training cost= 22.077698746\n",
      "Epoch: 0024 Validation acc= 0.954999745\n",
      "Epoch: 0025 Training cost= 19.774227052\n",
      "Epoch: 0025 Validation acc= 0.954399645\n",
      "Epoch: 0026 Training cost= 17.714090944\n",
      "Epoch: 0026 Validation acc= 0.957399666\n",
      "Epoch: 0027 Training cost= 15.867918540\n",
      "Epoch: 0027 Validation acc= 0.960799754\n",
      "Epoch: 0028 Training cost= 14.219327608\n",
      "Epoch: 0028 Validation acc= 0.961799681\n",
      "Epoch: 0029 Training cost= 12.738986726\n",
      "Epoch: 0029 Validation acc= 0.963399768\n",
      "Epoch: 0030 Training cost= 11.412021542\n",
      "Epoch: 0030 Validation acc= 0.968799710\n",
      "Epoch: 0031 Training cost= 10.229510890\n",
      "Epoch: 0031 Validation acc= 0.965799749\n",
      "Epoch: 0032 Training cost= 9.169845919\n",
      "Epoch: 0032 Validation acc= 0.967199683\n",
      "Epoch: 0033 Training cost= 8.216691914\n",
      "Epoch: 0033 Validation acc= 0.969199717\n",
      "Epoch: 0034 Training cost= 7.367491474\n",
      "Epoch: 0034 Validation acc= 0.970399737\n",
      "Epoch: 0035 Training cost= 6.608459154\n",
      "Epoch: 0035 Validation acc= 0.972999692\n",
      "Epoch: 0036 Training cost= 5.925226277\n",
      "Epoch: 0036 Validation acc= 0.969599724\n",
      "Epoch: 0037 Training cost= 5.316913984\n",
      "Epoch: 0037 Validation acc= 0.973999679\n",
      "Epoch: 0038 Training cost= 4.772619098\n",
      "Epoch: 0038 Validation acc= 0.973199725\n",
      "Epoch: 0039 Training cost= 4.282892211\n",
      "Epoch: 0039 Validation acc= 0.973599672\n",
      "Epoch: 0040 Training cost= 3.846757963\n",
      "Epoch: 0040 Validation acc= 0.973599792\n",
      "Epoch: 0041 Training cost= 3.455926691\n",
      "Epoch: 0041 Validation acc= 0.976799786\n",
      "Epoch: 0042 Training cost= 3.105613444\n",
      "Epoch: 0042 Validation acc= 0.975199699\n",
      "Epoch: 0043 Training cost= 2.793994174\n",
      "Epoch: 0043 Validation acc= 0.976199746\n",
      "Epoch: 0044 Training cost= 2.512941882\n",
      "Epoch: 0044 Validation acc= 0.975399733\n",
      "Epoch: 0045 Training cost= 2.262076859\n",
      "Epoch: 0045 Validation acc= 0.976999760\n",
      "Epoch: 0046 Training cost= 2.037694932\n",
      "Epoch: 0046 Validation acc= 0.975999653\n",
      "Epoch: 0047 Training cost= 1.837552240\n",
      "Epoch: 0047 Validation acc= 0.976399660\n",
      "Epoch: 0048 Training cost= 1.657969395\n",
      "Epoch: 0048 Validation acc= 0.975599706\n",
      "Epoch: 0049 Training cost= 1.496847012\n",
      "Epoch: 0049 Validation acc= 0.976999760\n",
      "Epoch: 0050 Training cost= 1.352709585\n",
      "Epoch: 0050 Validation acc= 0.978399754\n",
      "Epoch: 0051 Training cost= 1.223720786\n",
      "Epoch: 0051 Validation acc= 0.978799701\n",
      "Epoch: 0052 Training cost= 1.108979440\n",
      "Epoch: 0052 Validation acc= 0.978399634\n",
      "Epoch: 0053 Training cost= 1.005705943\n",
      "Epoch: 0053 Validation acc= 0.977999747\n",
      "Epoch: 0054 Training cost= 0.913236865\n",
      "Epoch: 0054 Validation acc= 0.978799701\n",
      "Epoch: 0055 Training cost= 0.830339956\n",
      "Epoch: 0055 Validation acc= 0.979199767\n",
      "Epoch: 0056 Training cost= 0.755614541\n",
      "Epoch: 0056 Validation acc= 0.978599668\n",
      "Epoch: 0057 Training cost= 0.689879866\n",
      "Epoch: 0057 Validation acc= 0.979399741\n",
      "Epoch: 0058 Training cost= 0.631272063\n",
      "Epoch: 0058 Validation acc= 0.978599787\n",
      "Epoch: 0059 Training cost= 0.578013867\n",
      "Epoch: 0059 Validation acc= 0.978799701\n",
      "Epoch: 0060 Training cost= 0.530160095\n",
      "Epoch: 0060 Validation acc= 0.979799747\n",
      "Epoch: 0061 Training cost= 0.487254938\n",
      "Epoch: 0061 Validation acc= 0.979799747\n",
      "Epoch: 0062 Training cost= 0.449552791\n",
      "Epoch: 0062 Validation acc= 0.980399728\n",
      "Epoch: 0063 Training cost= 0.415439274\n",
      "Epoch: 0063 Validation acc= 0.978799701\n",
      "Epoch: 0064 Training cost= 0.384874637\n",
      "Epoch: 0064 Validation acc= 0.979999721\n",
      "Epoch: 0065 Training cost= 0.356752845\n",
      "Epoch: 0065 Validation acc= 0.978199661\n",
      "Epoch: 0066 Training cost= 0.332616655\n",
      "Epoch: 0066 Validation acc= 0.979799747\n",
      "Epoch: 0067 Training cost= 0.309994660\n",
      "Epoch: 0067 Validation acc= 0.979199767\n",
      "Epoch: 0068 Training cost= 0.290989620\n",
      "Epoch: 0068 Validation acc= 0.979599655\n",
      "Epoch: 0069 Training cost= 0.272564379\n",
      "Epoch: 0069 Validation acc= 0.980599701\n",
      "Epoch: 0070 Training cost= 0.256773828\n",
      "Epoch: 0070 Validation acc= 0.980799794\n",
      "Epoch: 0071 Training cost= 0.242967232\n",
      "Epoch: 0071 Validation acc= 0.979999661\n",
      "Epoch: 0072 Training cost= 0.230392076\n",
      "Epoch: 0072 Validation acc= 0.979199708\n",
      "Epoch: 0073 Training cost= 0.218723012\n",
      "Epoch: 0073 Validation acc= 0.980599761\n",
      "Epoch: 0074 Training cost= 0.208804401\n",
      "Epoch: 0074 Validation acc= 0.980199695\n",
      "Epoch: 0075 Training cost= 0.198739098\n",
      "Epoch: 0075 Validation acc= 0.981199682\n",
      "Epoch: 0076 Training cost= 0.190688753\n",
      "Epoch: 0076 Validation acc= 0.979799688\n",
      "Epoch: 0077 Training cost= 0.183180254\n",
      "Epoch: 0077 Validation acc= 0.980599761\n",
      "Epoch: 0078 Training cost= 0.176371708\n",
      "Epoch: 0078 Validation acc= 0.980599761\n",
      "Epoch: 0079 Training cost= 0.170845800\n",
      "Epoch: 0079 Validation acc= 0.979599714\n",
      "Epoch: 0080 Training cost= 0.165155359\n",
      "Epoch: 0080 Validation acc= 0.980199695\n",
      "Epoch: 0081 Training cost= 0.160126905\n",
      "Epoch: 0081 Validation acc= 0.979999721\n",
      "Epoch: 0082 Training cost= 0.155851784\n",
      "Epoch: 0082 Validation acc= 0.982599735\n",
      "Epoch: 0083 Training cost= 0.152026351\n",
      "Epoch: 0083 Validation acc= 0.980199695\n",
      "Epoch: 0084 Training cost= 0.147996796\n",
      "Epoch: 0084 Validation acc= 0.981599748\n",
      "Epoch: 0085 Training cost= 0.146238381\n",
      "Epoch: 0085 Validation acc= 0.980599761\n",
      "Epoch: 0086 Training cost= 0.142040186\n",
      "Epoch: 0086 Validation acc= 0.981599689\n",
      "Epoch: 0087 Training cost= 0.138474341\n",
      "Epoch: 0087 Validation acc= 0.981799722\n",
      "Epoch: 0088 Training cost= 0.137078403\n",
      "Epoch: 0088 Validation acc= 0.980999768\n",
      "Validation loss stopped improving, stopping training early after 88 epochs!\n",
      "Optimization Finished!\n",
      "Accuracy on test set: 0.9798\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmsAAAD+CAYAAABleEI9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4lHW+9/H3PS110icFCBAIXUAERIqAAYKiCAGx7hHR\nS9zd4+JajgXP6lnXVY/u+uh5PMdFn3U9uqy7ioCropQgRHpvEjoICaSR3jMz9/NHIDZQUIZJZj6v\n68rFzD13+X5Ngh/u8vsZpmmaiIiIiEirZPF3ASIiIiJydgprIiIiIq2YwpqIiIhIK6awJiIiItKK\nKayJiIiItGIKayIiIiKtmM2XO29sbOS2226jqakJj8fD+PHjuffee3nsscfYsGEDTqcTwzB49tln\n6dmzJwBPP/00OTk5hIWF8dxzz9GrVy9fligiIiLSqvk0rDkcDt566y3CwsLweDzccsstXHnllQA8\n8sgjZGZmfmP9lStXcvToUZYsWcL27dt58skneffdd31ZooiIiEir5vPLoGFhYUDzWTa3241hGACc\naSze7OxsJk+eDED//v2pqqqipKTE1yWKiIiItFo+D2ter5fJkyczfPhwhg8fTr9+/QB46aWXmDRp\nEs899xxNTU0AFBUVkZyc3LJtUlIShYWFvi5RREREpNXyeVizWCwsXLiQnJwcduzYwYEDB3jwwQf5\n5JNPmDdvHuXl5bz++uvAmc+2nT4TJyIiIhKMLtrToJGRkQwePJjPP/+chIQEAOx2O1OmTGHHjh1A\n85m0goKClm0KCgpITEz83v1qalMREREJZD59wKC0tBS73Y7T6aS+vp61a9cyc+ZMiouLcblcmKbJ\nsmXL6N69OwBjxoxh7ty5TJgwgW3bthEVFdUS7M7GMAyKi6t82Uar5HI51XcQUd/BRX0HF/UdXFwu\n53lv49OwVlxczKOPPorX68Xr9TJhwgRGjRrF9OnTKSsrwzRNevXqxW9/+1sARo0axcqVKxk3bhxh\nYWE8++yzvixPREREpNUzzAC4jhisyVx9Bw/1HVzUd3BR38Hlx5xZ0wwGIiIiIq2YwpqIiIhIK6aw\nJiIiItKKKayJiIiItGIKayIiIiKtmMKaD1RXV7Ngwbzz3u7hh39NTU21DyoSERGRtkphzQeqqipZ\nsOC97yz3er3fu93zz79ERESkr8oSERGRNsing+IGqz/96RWOH8/nzjtvw2q14nCE4HQ6OXr0S/72\nt/d57LGHKC4uorGxgWnTbmHixMkATJt2PX/+89vU1tby0EOz6Nv3Unbt2o7LlcRzz/0Rh8Ph585E\nRETkYmvzYe3/vLOFW8ekYznLhO/vLj/Axj1FF/SYg3smcmNG+lk//8UvfsWRI4d44425bN26mYcf\nvp+3336X5ORkAGbPfhKn00lDQwN33307o0ZlEBUVBXzVQ17eMX7722d55JHHeeKJx1ixYjmZmVdf\n0D5ERESk9WvzYW35pmOMG9iepNhwf5dyVr1792kJagDvvvs3Pv98JQBFRUXk5R2ld+9LgK8mk0hJ\naUfXrs2BsEePnhQUHL+oNYuIiEjr0ObDGkBRWd1Zw9qNGenfexbsYggNDW15vXXrZrZs2cRrr72J\nw+HgV7+6h8bGxu9s8/VLnhaL9YzriIiISOALiAcMisrq/F3CN4SHh1NbWwvAt6derampxul04nA4\n+PLLI3zxxa4z7iMApmwVERGRCyAgzqwVltX6u4RviIqKpm/f/kyffjMORwhxcXEtnw0ZMoyFC9/n\nZz+7kY4dO3HJJX2/tuVX96wZZ7kHT0RERIKLYbbxUzgTH/yAfl3j+fW0/v4u5aJyuZwUF1f5u4yL\nTn0HF/UdXNR3cAnmvs9Xm78M6gx3UNjKLoOKiIiIXChtPqy1S4igpLwOzw8MOCsiIiLSFrX5sJaS\nEIHHa1Ja2eDvUkREREQuuIAIa9D6nggVERERuRACKKy1ridCRURERC6EgAlreshAREREApFPw1pj\nYyPTpk1j8uTJTJw4kVdeeQWAvLw8brzxRsaPH88DDzyA2+1uWf/+++8nMzOTm266iePHf3iKpZT4\ntn8ZdNy4kf4uQURERFopn4Y1h8PBW2+9xcKFC1m4cCE5OTls376dP/zhD8yYMYPFixfjdDqZN28e\nAPPmzSM6OpolS5Ywffp0XnjhhR88RlSEg7AQG0XlbTesaQBcERERORufXwYNCwsDms+aud1uDMNg\n/fr1jB8/HoCsrCyWLVsGQHZ2NllZWQCMHz+etWvX/uD+DcMgKTaMorI6vK1kfN9XX/2/LFgwr+X9\nG2+8xl/+8jr33fdL7rrrX5g+/RZWrVrpxwpFRESkrfD5dFNer5cpU6Zw9OhRbrvtNlJTU4mKisJi\nac6JycnJFBYWAlBUVERycjIAVquVqKgoysvLiYmJ+d5jJMaGcaSgivKqBuKiQr/x2fwDH7G1aOcF\n7WlAYl+mpF931s/Hjs3k5Zf/SFbWDQAsX76MF1/8v9x0022Eh4dTUVHOPffMYMSIURe0LhEREQk8\nPg9rFouFhQsXUl1dzb/+679y8ODB76xz+jLgt2e+Mk3znC4RJsaGA80PGXw7rPlDt249KC8v5+TJ\nEsrKyoiKiiI+PoGXX/4D27dvw2IxKCkppqyslNjYuB/eoYiIiAStizaRe2RkJIMHD2b79u1UVlbi\n9XqxWCwUFBSQmJgIQFJSEgUFBSQlJeHxeKiuriY6OvoH953eMRbWHKHO7f3OnFv3uG4BbvFFS9/r\n2muvYePGVZSUlDB58vWsXfsZDQ21fPTRP7FYLGRkZBAZacflcmIYP26usB+zTSBQ38FFfQcX9R1c\ngrXv8+XTsFZaWordbsfpdFJfX8/atWuZOXMmQ4YM4dNPP2XChAksWLCAMWPGAJCRkcGCBQvo378/\nn376KVdcccU5HSfM1nz27eDRMoq7xvusn/NxxRWjeP75p6moqOCVV14jO3spYWFOTp6sYcuWTRw/\nfpyTJ2uw26vwes3znsw2mCfAVd/BQ30HF/UdXIK57/Pl07BWXFzMo48+itfrxev1MmHCBEaNGkWX\nLl144IEHePnll+nVqxc33NB8b9e0adP4t3/7NzIzM4mJieHFF188p+OcvgzamobvSEvrQm1tLS5X\nEnFx8WRmXs0jjzzA9Om30LNnLzp1SmtZV0+DioiIyNkY5rdvFGuDiooq+eX/ycEVHcZTd13u73Iu\nimD+F4n6Dh7qO7io7+ASzH2frzY/gwGcGr4jJoyi8trvPKQgIiIi0pYFRFiD5uE7Gpu8VNQ0+rsU\nERERkQsmgMJa67tvTUREROSnCqCw1jxTQmFZrZ8rEREREblwAiasJZ0KazqzJiIiIoEkYMKaLoOK\niIhIIAqYsBYd6cBhsyisiYiISEAJmLBmMQxcsRq+Q0RERAJLwIQ1gMSYMOoaPFTVNfm7FBEREZEL\nIqDCWpLuWxMREZEAE1BhLTHu1PAdpRq+Q0RERAJDQIW1pBgN3yEiIiKBJaDCWsvwHeUKayIiIhIY\nAiqsxUaFYLNaKNIsBiIiIhIgAiqsWQwDV0yoLoOKiIhIwGjzYe3fl71Ao+eroTqSYsOpqXdTreE7\nREREJAC0+bC27+QhthXvbHmfqDlCRUREJIC0+bAGsPbEppbXX4U13bcmIiIibV+bD2u9XN3YV3aA\nkrqTgM6siYiISGBp82EtI20YAOtOnV07PXxHocKaiIiIBACfhrWCggJuv/12JkyYwMSJE3n77bcB\neOWVVxg5ciRZWVlkZWWRk5PTss2cOXPIzMzkmmuuYdWqVT94jCGpAwi1hrD2xCa8ppf4qBCsFoOi\ncl0GFRERkbbP5sudW61WHnvsMXr16kVNTQ1Tpkxh2LDmM2EzZsxgxowZ31j/4MGDfPLJJyxatIiC\nggJmzJjBkiVLMAzjrMcItYUwMOlSVh9fz57S/fSO70FCtIbvEBERkcDg0zNrLpeLXr16ARAREUHX\nrl0pKioCwDTN76yfnZ3NhAkTsNlsdOjQgU6dOrFjx44fPM7QlMEArD2xEWi+FFpV20RtvftCtSIi\nIiLiFxftnrW8vDz27NlDv379AJg7dy6TJk3i8ccfp6qqCoDCwkJSUlJatklKSqKwsPAH9905KpXk\niCR2FH9BdVPNVw8Z6FKoiIiItHEXJazV1NQwa9YsZs+eTUREBLfeeivLli3jgw8+ICEhgeeeew44\n89m277sE+vV1hqUMxm162Fiwlc7JTgC27S+5sI2IiIiIXGQ+vWcNwO12M2vWLCZNmsTYsWMBiIuL\na/n8xhtv5Oc//zkAycnJnDhxouWzgoICEhMTf/AYLpeTa5xX8sHBRWws2sxTo8fyTvZ+Vu0s4M5J\nfbFa2/xDr2fkcjn9XYJfqO/gor6Di/oOLsHa9/nyeVibPXs26enpTJ8+vWVZcXExLpcLgKVLl9K9\ne3cAMjIyeOihh7jjjjsoLCzk6NGjLZdNv09xcRVg0DehN9uKd7E7/wBDeyeTvSWPZeuOcFl3l096\n8yeXy3mq7+CivoOL+g4u6ju4BHPf58unYW3z5s18+OGHdO/encmTJ2MYBvfffz8fffQRubm5WCwW\n2rdvz1NPPQVAeno611xzDddeey02m40nn3zynC6DnjY0ZTDbinex9sRGRg0YR/aWPFZszQ/IsCYi\nIiLBwadhbeDAgeTm5n5n+ciRI8+6zT333MM999zzo47XK6470Q4nGwu3kpV+Hekdotl1uJSi8joS\nY8J+1D5FRERE/CmgbuayWqwMSRlEnbue7cW7uOrS9gDkbDvu58pEREREfpyACmsAQ1MGAbDmxEYG\n9XQREWrj8x3HcXu8fq5MRERE5PwFXFhLDHeRHpPGvrIDVDSVM6JfClW1TWzZV+zv0kRERETOW8CF\nNYBhKZcD8Hn+OkaduhS6Ymu+P0sSERER+VECMqxdltQfpz2S1cc3EBtto1enWPYcLefEyRp/lyYi\nIiJyXgIyrNktNka0H0Kdu46NBVsYPaD57NpKPWggIiIibUxAhjWAEe2vwGJYWJG3mkvT44mKcLB6\n5wkamzz+Lk1ERETknAVsWIsJieayxH6cqCnkUOVhruyXQk29m417ivxdmoiIiMg5C9iwBjC6w3AA\nVuStZlT/dhjAim160EBERETajoAOa52jOtLJmcrOkt0QUsclXeI5mF/JsaJqf5cmIiIick4COqwZ\nhsHo1OGYmOTkr2H0gHYALNt0zM+ViYiIiJybgA5rAAMS++G0R7Lm+EZ6do4iKTaMtV8UUFHd4O/S\nRERERH5QwIe15mE8rqDOXcemoq1kXt4Rt8cke0uev0sTERER+UEBH9YArvzaMB7D+iQRGWbnsy35\nNDRqGA8RERFp3YIirEWHRHFZYj8Kago5XH2YMQM7UFPv5vMdGiRXREREWregCGsAozuMAJqH8bjq\nsvbYbRaWbDyGx+v1c2UiIiIiZxc0YS0tuiOdolLZVZJLo1HF8L4plFTUs3lvsb9LExERETmroAlr\n0DxIronJ8mOrGD84FQNYvOEopmn6uzQRERGRMwqqsDYwsT+xITGsOb6esEgPA7q7OHyiin3Hyv1d\nmoiIiMgZBVVYs1qsZHYaTZPXzfKjn3P15R0B+HT9UT9XJiIiInJmPg1rBQUF3H777UyYMIGJEyfy\n1ltvAVBRUcGdd97J+PHjueuuu6iqqmrZ5umnnyYzM5NJkyaRm5t7wWsamjKYKIeTnPw1pCTZSW8f\nzfaDJzleUnPBjyUiIiLyU/k0rFmtVh577DEWLVrE3//+d+bOncvBgwd57bXXGDp0KIsXL2bIkCHM\nmTMHgJUrV3L06FGWLFnCU089xZNPPnnBa7Jb7YzpOJIGTyMr8lYz/tTZtcUbdHZNREREWh+fhjWX\ny0WvXr0AiIiIoGvXrhQWFpKdnU1WVhYAWVlZZGdnA5Cdnc3kyZMB6N+/P1VVVZSUlFzwuka0u4II\nWzgrjq2iV5pTU1CJiIhIq3XR7lnLy8tjz5499O/fn5MnT5KQkAA0B7rS0lIAioqKSE5ObtkmKSmJ\nwsLCC15LqC2Eq1JHUOuuY/WJdS1TUC3brCmoREREpHW5KGGtpqaGWbNmMXv2bCIiIjAM44zrnWkI\njbOt+1ON6jCMUGsI2cdyGNwrnqgIB9mb86iua/LJ8URERER+DJuvD+B2u5k1axaTJk1i7NixAMTH\nx1NSUkJCQgLFxcXExcUBzWfSCgoKWrYtKCggMTHxB4/hcjl/RGVOru4+moW5i9lb/wXTxnTnz//c\nxee7Crh9Qu8fsb+L78f13fap7+CivoOL+g4uwdr3+fJ5WJs9ezbp6elMnz69ZVlGRgbz589n5syZ\nLFiwgDFjxgAwZswY5s6dy4QJE9i2bRtRUVEtl0u/T3Fx1Q+ucyZXxA/hY8tyFnyxmNmDHiQ6wsE/\ncw4xvE8SUeGOH7XPi8Xlcv7ovtsy9R1c1HdwUd/BJZj7Pl8+vQy6efNmPvzwQ9atW8fkyZPJysoi\nJyeHu+++mzVr1jB+/HjWrl3LzJkzARg1ahQdOnRg3LhxPPHEEz55GvTrnI5Ihre7nLKGcrad3MG1\nQzvR0ORhscZdExERkVbCp2fWBg4ceNax0t58880zLn/iiSd8WNF3je04is/z17H0y894dND9fLL+\nKNlb8si8vCPREa377JqIiIgEvqCaweBMYkNjuCJlIEV1Jews/YLrhnaiscnLJ+u+9HdpIiIiIgpr\nAOM6XoXFsLDo8FKG9k0iLiqEz7bmU65x10RERMTPzimsLVq0iOrqagBefvll7rrrLnbt2uXTwi4m\nV3g8Q1MGUVhbzObiLVw3rDNNbi+LdHZNRERE/Oycwtqrr75KZGQkO3bsYNWqVUyePJmnn37a17Vd\nVBPSxmG32Fh0eBmX904gITqUFVuPU1als2siIiLiP+cU1my25ucQVq9ezbRp05g4cSINDYEVYmJC\norkq9UrKGypYfWItE4d1xu3x8vHaI/4uTURERILYOYU1wzBYtGgRixYtYujQoQA0NQXeSP/jOo4m\n3BbG4i8/o3+PKFwxoeRsP05pZb2/SxMREZEgdU5h7d///d/56KOPuOGGG0hNTeXIkSMMGTLE17Vd\ndOH2MDI7XUWdu47leTlcPzwNt8fkozVH/F2aiIiIBKlzCmuXXXYZ//M//9MyC0Hnzp35zW9+49PC\n/GVUh+HEhESzIm8VPbqGkhQXTs72E5w4WePv0kRERCQInVNYe+6556iqqsLtdnPrrbdy6aWX8sEH\nH/i6Nr9wWO1cm5ZJk9fN4i+zuXF0V7ymyT+WH/B3aSIiIhKEzimsrVmzBqfTyapVq0hKSmLx4sW8\n8cYbvq7Nb4YkX0ZyeCJrT2wkuZ2Xnh1j2HHwJF8cLvV3aSIiIhJkzmtQ3I0bNzJu3DiSkpIwDMNX\nNfmd1WLl+q5XY2Ly0eHF3DymGwbw9+X78Xi9/i5PREREgsg5hbX4+HiefPJJPvnkE4YPH47b7cbj\n8fi6Nr/ql9CHtKhObCvehSe0lBH9UsgvruHz7Sf8XZqIiIgEkXMKa3/84x9JS0vjxRdfJDo6moKC\nAmbMmOHr2vzKMAwmdb0GgAUHFpF1ZRohDisLPj9Ebb3bz9WJiIhIsDinsBYXF8fPfvYzIiIiOHDg\nAMnJyUyZMsXXtfldt9gu9E3ozcGKwxyoyeXaKzpRVdukgXJFRETkojmnsLZz507GjRvHvffeyy9/\n+UsyMzP54osvfF1bq3BDt4nYLDbmH/iYkQMSiY8KYemmYxSV1/m7NBEREQkC5xTWfv/73/PMM8+w\nePFilixZwjPPPMPvfvc7X9fWKiSExZPZcTQVjZUsy1/O1NFdcXtM5n2moTxERETE984prNXV1bVM\nMwVwxRVXUFcXPGeWxnW6ivjQOD47toqOHaFruyg27S1m37Fyf5cmIiIiAe6cwlpYWBjr1q1reb9h\nwwbCwsJ8VlRr47Damdb9eryml/f2fcBNGekAvJO9H69p+rk6ERERCWS2c1lp9uzZ3HfffTgcDqB5\nEvf/+q//8mlhrU3fhN5cEt+LXSdzGdH+CFf0TmLd7kJWbM0n47IO/i5PREREAtQ5hbV+/fqxZMkS\nDh8+jGmadOnSBbvd7uvaWp1p3a9nz/r9vL//I349ahY7Dp5k3oqDXJqeQFxUqL/LExERkQD0vZdB\n6+rqWr7cbjepqal07NgRt9t9TveszZ49m2HDhjFx4sSWZa+88gojR44kKyuLrKwscnJyWj6bM2cO\nmZmZXHPNNaxateontOUbX3/YYHXx59yYkU59o4e5S/f5uzQREREJUN97Zm3AgAEYhoF56r6s01NM\nmaaJYRjk5uZ+786nTJnCv/zLv/Dwww9/Y/mMGTO+M6juwYMH+eSTT1i0aFHLoLtLlixpddNajet0\nFesLtrD82Oc8NnggPVJj2Lq/hM17ixjYI9Hf5YmIiEiA+d6wtmfPnp+080GDBpGfn/+d5eYZbsrP\nzs5mwoQJ2Gw2OnToQKdOndixYwf9+/f/STVcaA6rnRu6TWTOzv/lvf3/5F/G38p//GUTf126j16d\n4ggPPacryyIiIiLn5Lwmcr9Q5s6dy6RJk3j88cepqqoCoLCwkJSUlJZ1kpKSKCws9Ed5P6j5YYOe\n7Cs7wJHGXCYO60RFdSPzVh70d2kiIiISYC56WLv11ltZtmwZH3zwAQkJCTz33HPAmc+2tbZLoKcZ\nhsHNPaYQag3l/f0fMuTSaNonRLBia77GXhMREZEL6qJfs4uLi2t5feONN/Lzn/8cgOTkZE6cONHy\nWUFBAYmJ53YPmMvlvLBFnssxcTKjaRqvbnybBUc+4L6bb+OR/17F3GX7ePmB0dhtVt/X4Ie+WwP1\nHVzUd3BR38ElWPs+Xz4Pa98+Y1ZcXIzL5QJg6dKldO/eHYCMjAweeugh7rjjDgoLCzl69Cj9+vU7\np2MUF1dd2KLPUZ/IS+gT35MdhblcErOV0QPa89mWfN768AuuH5Hm02O7XE6/9e1P6ju4qO/gor6D\nSzD3fb58GtYefPBB1q9fT3l5OaNHj+ZXv/oV69evJzc3F4vFQvv27XnqqacASE9P55prruHaa6/F\nZrPx5JNPttrLoKcZhsGtPafy9Po/8v6BD3nw8vvYuq+Yj9YeYWAPF+1dkf4uUURERNo4wzzTzWJt\njL+T+foTm3kr9x90j03nyojJvDJ/F6mJkfz77YOw23xzW2Aw/4tEfQcP9R1c1HdwCea+z5dfngYN\nNJcnX0bfhF7sKztAdfhBRvZP4VhRNQtyDvm7NBEREWnjFNYuAMMwuKXHVMJtYSw4+DHjhseTGBvG\n4g1HyT1S6u/yREREpA1TWLtAokOimNZ9Eo2eRt47MJ+7r+uNYRj8v49zqa5r8nd5IiIi0kYprF1A\ng5MG0D+hD/vLD3HIvZVJIzpTVtXAW4v3nnEcOREREZEforB2ARmGwS09pxLtcPLPQ5/Sqzd06xDN\npj1FrNlV4O/yREREpA1SWLvAnI5IZvS5FdM0eXP3O9x2dWfCQqz8dek+isrr/F2eiIiItDEKaz7Q\nLbYr16ZlUtZQzsf5/+S2sd1paPTw+odf4PF6/V2eiIiItCEKaz4yvvNV9Iztxq6TudRG7ePyXokc\nzK9k4eeH/V2aiIiItCEKaz5iMSxM73MzUQ4nHxz6hFHDwkiMCePjtV+yZV+xv8sTERGRNkJhzYei\nHE5m9LkF0zSZu+8f3Hl9Og67hf/30W5OnKzxd3kiIiLSBiis+Vj32HQmpI2lrKGc5SUfM/3qHtQ3\nenhl/k7qGtz+Lk9ERERaOYW1i+DqzmPoEZvOzpLdVEXuIXNwKidO1vLGx7kaf01ERES+l8LaRWAx\nLNzR5xZiQqL558FP6dG3np4dY9i8r5hF6770d3kiIiLSiimsXSRRDic/73cHdouNt3L/waSx8cQ6\nQ5ifc4hdh0/6uzwRERFppRTWLqJUZ3um976ZRk8jb++fyx0T07BaDOZ88AXFGjBXREREzkBh7SK7\nNLEv13e5mrKGcj4tms8t47pSU+/mv+btoLZeE76LiIjINyms+UFmp6sYnHQZRyqPcsS+ijED25Nf\nUsMr83fS5NYMByIiIvIVhTU/MAyD23pOJS2qE5sKtxGXnsdl3V3sOVrOXz7RE6IiIiLyFYU1P7Fb\n7czsdzuxITF8dHgxl1/hoWv7KNZ9Ucj8nEP+Lk9ERERaCYU1P4pyOPlF/xmEWB3M3fsPrhkbTlJs\n85RUn23N93d5IiIi0gr4NKzNnj2bYcOGMXHixJZlFRUV3HnnnYwfP5677rqLqqqqls+efvppMjMz\nmTRpErm5ub4srdVoH5nCzL7TAXh739+Ydm08znA7f12yl237S/xcnYiIiPibT8PalClT+POf//yN\nZa+99hpDhw5l8eLFDBkyhDlz5gCwcuVKjh49ypIlS3jqqad48sknfVlaq9Izrht3XnIbTZ4m3jk0\nl9smpmC3WvjTP3dx6Hilv8sTERERP/JpWBs0aBBRUVHfWJadnU1WVhYAWVlZZGdntyyfPHkyAP37\n96eqqoqSkuA5s9TfdQm39ZpGrbuOhfnvcOu1HWhye3npve3kFVX7uzwRERHxk4t+z1ppaSkJCQkA\nuFwuSktLASgqKiI5ObllvaSkJAoLCy92eX41NGUQU7tNpKKxiuyyedyUmUp1XRMv/H0r+SU1/i5P\nRERE/KDVPGBwpuEqDMPwQyX+lZF6Jdd0HkNJfSkbGz/k5nGdqKpt4oV3tnLipAKbiIhIsLFd7APG\nx8dTUlJCQkICxcXFxMXFAc1n0goKClrWKygoIDEx8Zz26XI5fVKrv9yRMBXT5uHTAyv4Im4xd06a\nyBsf7OeP/9jGs78cQTtXJBB4fZ8r9R1c1HdwUd/BJVj7Pl8+D2vfPmOWkZHB/PnzmTlzJgsWLGDM\nmDEAjBkzhrlz5zJhwgS2bdtGVFRUy+XSH1JcXPXDK7Ux16ZeTVl1FesLNtMQ+T5TrprA/M/yePS/\nV/HIrQPo0z0pIPv+IS6XU30HEfUdXNR3cAnmvs+XTy+DPvjgg9x8880cPnyY0aNH8/777zNz5kzW\nrFnD+PH5GbrTAAAbe0lEQVTjWbt2LTNnzgRg1KhRdOjQgXHjxvHEE08E1dOgZ2IxLPys1zSGtxvC\nserjbONDrh+VQllVA8+/s5UCXRIVEREJCoYZAHMbBXIyN02T9/d/yGd5q0gMS6CP9xoWfV5MYmwY\n90/rT1JcuL9LvKiC+V9i6jt4qO/gor6DS6s7syY/nWEYTO02kcxOV1FUV8J24yOuHhFPUVkdz/x1\nM0cKNA6biIhIIFNYawMMw2BS12uY2GU8pfVlbONDbr2+PdW1Tfzn37byxZFSf5coIiIiPqKw1oZc\n3XkMU9Kvo6Kxkuyyf3DDhHg8Hi8vvbudDbnBNSadiIhIsFBYa2PGdBzJzT2mUNVYw+LSd5k0IQyH\n3cKcD75g2aZj/i5PRERELjCFtTboyvZX8PCIX2AYBp8ULiAjs5HICDt/W7af+TkHzzjAsIiIiLRN\nCmtt1MB2fbn/sp8T5YhkeeFSBowqwBUTwkdrvmTOP7+gocnj7xJFRETkAlBYa8M6Ojvwb4N+RbuI\nZDaWbKTD5Xvo0iGcDblFPPfXLZRW1vu7RBEREfmJFNbauNjQGB4Y+Et6xnZjT/lerN3Xc3k/J18W\nVvHUmxvZn1fu7xJFRETkJ1BYCwBhtlB+2f9OhqVcTl51PoecHzNmtIPqOjfP/20rOduP+7tEERER\n+ZEU1gKE1WLl1p5TublHFg3uBtbWfsjwcZWEOCy8+cke5i7Zh9vj9XeZIiIicp4U1gKIYRhc2X4o\nDwz8JXGhMWwqX03a8FxSEm1kb8nj+b9tpaSizt9lioiIyHlQWAtAnaJSeWTwfVwS34tDVYcwu+fQ\n5xI4kF/Bf7yxkc17i/xdooiIiJwjhbUAFWEP555+05nU5RoqG6s4ErGEy0dX4vZ4+O8Fu3h78V4a\nNbyHiIhIq6ewFsAshoXMzlcxa8BMnPYIdtauodOIXSSnmHy2NZ+n39rE8ZIaf5cpIiIi30NhLQh0\nj+3K7CEPMMDVl7zaY9R3/ozel9WQV1zNU29u5LMteXg164GIiEirpLAWJCLtEdx1yc+Y3vtmDCwc\ntn1Oj5EHsTqaeHvJPv7wzlYKy2r9XaaIiIh8i8JaEDEMg8uTL+PxIffTLaYLR+sPEHHpGrr2qmXP\n0XKe/PMGPl1/FK9XZ9lERERaC4W1IBQXGsusATOZkn4dDZ56jjtz6H7lfuzhDbz72QGe+etm8our\n/V2miIiIoLAWtCyGhTEdR/LY5c1n2Y41HMTaO4e0/sUcOlHOf/xlIws/P6QnRkVERPxMYS3IJUck\nct+Ae7i9102EWB0UhGym/bCtRMRX8c/VR3j89fVs2lOEqQcQRERE/MLmrwNnZGQQGRmJxWLBZrMx\nb948KioquP/++8nPz6dDhw689NJLOJ1Of5UYNAzDYEjKQC5J6MUHBxex+vgGSCsmLa0HR7e3438W\n7qJXp1huGduNDq5If5crIiISVPx2Zs0wDN5++20WLlzIvHnzAHjttdcYOnQoixcvZsiQIcyZM8df\n5QWlCHs4t/a8gQcu+yXtIpIpYC8RA1bR/pJ8co8V8x9vbGTu0n3U1Df5u1QREZGg4bewZpomXu83\nJxbPzs4mKysLgKysLJYtW+aP0oJe15jOPDr4Pm7tMZUwWyil4TuJu3wt0R1PkL35KI/+aS2frP+S\nBt3PJiIi4nN+uwxqGAZ33XUXhmFw8803M23aNE6ePElCQgIALpeLsrIyf5UX9KwWK8PbD2Fg0qVk\nH8th2dGVNCZtw5UcS/Whrrz3WRNLNh5j4rDOjOzfDptVtz+KiIj4gt/C2t///ndcLhelpaXceeed\npKWlYRiGv8qRswi1hXBt2jhGtBvCx4eXsub4BkjbRGJaAuWHOvHXJQ18uv4ok0akMbRPMhaLvoci\nIiIXkmG2gsf8XnnlFcLDw3nvvfd4++23SUhIoLi4mNtvv51PPvnE3+XJ1+RVnGDeFx+z9tgWTEwi\njXjKD3Sk6WQiHRKdTBvTnZED2utMm4iIyAXil7BWV1eH1+slIiKC2tpa7rzzTu69917Wrl1LdHQ0\nM2fO5LXXXqOyspKHHnroB/dXXFx1EapuXVwup1/7Lqgp5NMjy9lUuA0Tk1BvDFWHO+E+mUR8VBiZ\nl3dkZL92hDisF/S4/u7bX9R3cFHfwUV9BxeX6/xHufDLZdCSkhLuvfdeDMPA4/EwceJERowYwSWX\nXMKvf/1r3n//fdq1a8fLL7/sj/LkHCRHJHFHn1u4Jm0si48sZ2PhVuxdtxPZJZLq/FTeWV7Dh6uP\nMHZgBzIGdiAyzO7vkkVERNqkVnEZ9KcK1mTemvouqTvJsqM5rDuxiSZvE1bseIo7UJ+fisN0MuyS\nZDIua0/7nzhOW2vr+2JR38FFfQcX9R1c2syZNQk8CWHx3Nwji+u6ZLI6fz0r81ZT4TpMqOsI1soU\nVh5oz2db8+iRGkvGwA4M6Jag+9pERETOgcKaXFCR9gjGd85gTMeRbC7czvJjn5PHcUKijmNzOzl4\nvB17PyokJtTJ6Evbc2X/dsQ6Q/xdtoiISKulsCY+YbPYGJIykMuTL+NgxRFWH1/P1qId2DvuxZG6\nn/qKRP65owMfrI6nT1o8wy9JYUC3BBz2C/tAgoiISFunsCY+ZRgG6TFppMekMa3b9Wwo2Mrq4+s5\nbhQQElOAxR3GnuIkvljWjtDFsQzplcTwvil0aRelcfdERERQWJOLKNwezujU4YzqMIwjlcdYc3wD\nW4t34k05gj3lCEZDJKuKUlj5bgqu8HgG90xkcM9EUhMjFdxERCRoKazJRWcYBmnRHUmL7siNPSaz\n++QeNhZsZWdJLvYO+7F32E9lTTSLjySxaGsiSeHNoW1wr0QSEn7a06QiIiJtjcKa+JXdYqO/6xL6\nuy6hzl3P9uJdbCzYyj4OYkRUYE/dR3ldJJ98mchH25JIiWhHvy7x9O8aT3qHaKwWPVEqIiKBTWFN\nWo0wWyhXpAziipRB1DTVsrNkN9uLv2B36V6MsEPY2x+itDGU7MIElux1EdqYSL+0ZC5NT+CStHjC\nQ/XjLCIigUf/d5NWKcIe3hLcGjyN7D65l23FO8kt3UeNIw9bYh6mabClKpZNGxIwl7roHNuePp3j\n6JMWR1pKlMZxExGRgKCwJq1eiNXBgMS+DEjsS1x8OJsO5bL75B52n9zLl0Ye1qhSSN1HfpOdY4Vx\nfLw/Dnudix5JqfTpHEfPjrG0c0Vg0UMKIiLSBimsSZtitVjpEt2JLtGduK7LeKoaq8kt3cee0v3s\nLT1Aub0Qa1whkMveJge7D8fh3RFLSFMC3eJT6ZEaR7fUaDolOXXmTURE2gSFNWnTnI5ILk++jMuT\nL8M0TUrqStlXfoD9ZYfYU3qAKnsBxBdgAns9VnKPR+PdF4ulLo5Ozg6kp7jokhJNl3ZRmklBRERa\nJYU1CRiGYeAKj8cVHs/wdkMwTZPiuhIOVXzJoYoj7Cs9TLG1GGt0KXCQPDZytDaMZTuiMddGE0E8\nXaJTSU9JoGOSk45JkTjDHf5uS0REgpzCmgQswzBIDHeRGO7iipRBANQ21XK48iiHyo9wqOIYX1Ye\noyG0+exbI7AH2F0ajpnnxFvrJII4OkS2o0tCEp2So+jgiiAhJkz3v4mIyEWjsCZBJdweTp/4nvSJ\n7wmAaZqU1pfxZVUeRyvzOFD2JceNEzSENt/71ggcAg7W2/DmRmJuicTS6CTekUCHqGQ6xyfSITGS\nlLgIYqNCFOJEROSCU1iToGYYBvFhccSHxXFZYj+gOcCVN1SQX32C/OoTHCnP52jVccptpeAsB6Ds\n1NeOSitmYSTe+nAsTZFE22JJDE8gNTqJ1IRYkmLDSYwNIyLU7r8mRUSkTVNYE/kWwzCIDY0hNjSG\nSxJ6tSxv8ropri3hRE0hx6sL+LL8BCdqCqmwlmGJrACg6tTXQTeYxxyYB8Lw1odj80QQZY8hPjSO\nFKeLDtHxJMaEEx8TRpwzRE+miojIWSmsiZwju8VGu8hk2kUmMzCpf8tyj9dDWUM5RbUlFNYWc6y8\nkOPVRZQaJ6m1VbYEucpTX4c9YJ4E80QoZmMoNIbhMCNx2qKICY0mPiyGpMg4kqNiiI8OJdYZSny8\n6Z+mRUTE7xTWRH4iq8VKQlg8CWHx9I7vAalffebxeihvqORk/UmKa0vJqyjiRHUJZfXlVBuV1Dsq\nwCjHA5Sf+jpiAlVgVhiYh0KgKQSaQrETRpglgkhbJNEhUcSGReGKiCEhMpq4yHCiIuxERziw26z+\n+Q8hIiI+obAm4kNWi5X4sFjiw2LpHgu0/+bnHq+HisZKSuvLKakto6DqJMU1ZZTVV1DZWEmtUU2D\noxKMCjxA9amvAoAmWhKe6bFiNjnA7cDwhOAgjBBLKKHWcCLs4TjtEUSFRhITGklsuJO4iEiiwkKJ\nDLMTHmrTZVgRkVasVYa1nJwcnnnmGUzTZOrUqcycOdPfJYn4hNViJS40lrjQWNJj0s64jtf0EhJl\ncOREARUNlRTXVFBcVcbJugoqGqqoaaqhzqylwVGHO6QSDJMmmrNcNVByekf1p76an5HA9FjBY8N0\n2zG8DmymA5vhwG6EEGIJIdQWSpg1jHBHGBG2ECIcYUSGhOMMDcMZGk5MWDjhIQ5CHVZCHFY9CSsi\n4iOtLqx5vV5+97vf8eabb5KYmMgNN9zAmDFj6Nq1q79LE/ELi2EhJtRJ+0iD9pEpEH/2dU3TpM5d\nR1VTDdWNNZTWVlJaU0V5fTUV9dVUNdZQ566j3lNPg1FPk60et60B01KNB/AADTSHvBbuU1/1Zzie\nxwJeG6bHimHasJg2LKYdK3Zshq05/Fns2C12HBYHIdbmL4fNQajNTqjNQag9hDC7g1Bb859hjhDC\n7Q7CHA5CwkNoaPJgt1kUBkUkaLW6sLZjxw46depE+/bN14uuvfZasrOzFdZEzoFhGITbwwm3h5MU\n7qJrzLlt5zW91LvrqXXXU9tYR3ldDeX1NVTV11DVUEttUwN1TXXUuRto8DTQ4G2g0duAmyY8RhMe\nexNeox7TaMJjNIe+xrMd7Oup8AeYpgFeS/OXacEwrWBasZhWDCwYWLGc+rIaViyGFeup11bLqT8N\nGzaLFZvlq+W2U8vs1ub3dstX69isX723W23N708vs1qxn9quebmt+bXFit1mxWaxYLUqWIrIhdXq\nwlphYSEpKSkt75OSkti5c6cfKxIJfBbD0hLyCIOO0T9uP6Zp0uR10+hppN5dT01jA9UN9VQ31lPb\nUE9dU0Nz4HM3Ue9upMHdSIOngSavmyZPE27Tjdvrxm024THdeA0PbtONiQevxYuJB9NowGt4wPCC\ncR5PyXpP/en5cb2dK9M04PQXzX8ap1+ffn/qtYHRHEK//h4Di2EBk5Zlxtc+MwzLt5YaGMap7Vq2\n/2p/hvHN9QwMMMDyte2+/Xnza765X4NvHY+vHYdTdfHVfk6tj9H8+Vfrfe2d8bX1gbBQBw317lPL\nObXtV/v6attmxteOf/r96YpPL2w+ZsurU3Xzje1Ov7B8e7uWN9887pnCuPG1ZcbXdnKm/XzrsEQW\nhFJT3fDd7b+1ydeXn+Vlc+9n2Paba3279jMvP9s/Oozv2ddZNjij6PIwqirPcMr++4o66yHO/x9I\nljNsEmoNI8oR9Z3ldquFhJiw8z7GhdLqwpppaogCkbbKMAwcVjsOq51IRwQJ4T9tfy6Xk+LiqrN+\n7jW9uL0emjxNNHiaqG9qosF9+stN0+nXHjdNHjduj5smr6f5tddNo8eNx+vB7fXgMT24vV48phuP\n14vH9OA1vXjNb73Gi2l6MWl+78Vsec2pd+a3/uTUO7MlYHoxDbPls+ZlJubp8Hnq/UU/QWee5fXF\ncvZvtchFU79jBGZ95HeW33N9H4b0TvJDRa0wrCUnJ3P8+PGW94WFhSQmJn7vNi6X09dltUrqO7io\nbxGRi+AmfxfwXa3uef2+ffty9OhR8vPzaWxs5OOPP2bMmDH+LktERETEL1rdmTWr1cpvfvMb7rzz\nTkzT5IYbbtDDBSIiIhK0DFM3iYmIiIi0Wq3uMqiIiIiIfEVhTURERKQVU1gTERERacVa3QMG5yNY\n5hCdPXs2K1asID4+ng8//BCAiooK7r//fvLz8+nQoQMvvfQSTmdgDXFQUFDAww8/TElJCVarlWnT\npnH77bcHfO+NjY3cdtttNDU14fF4GD9+PPfeey95eXk88MADVFRU0KdPH55//nlstjb9K/wdXq+X\nqVOnkpSUxJ/+9Keg6BkgIyODyMhILBYLNpuNefPmBfzPOUBVVRWPP/44+/fvx2Kx8Mwzz9C5c+eA\n7vvw4cPcf//9GIaBaZocO3aM++67j0mTJgV03wBvvvkm8+bNwzAMunfvzrPPPktRUVHA/47/7//+\nL/PmzQP48f8fM9soj8djjh071szLyzMbGxvN66+/3jxw4IC/y/KJjRs3mrt37zavu+66lmXPP/+8\n+dprr5mmaZpz5swxX3jhBX+V5zNFRUXm7t27TdM0zerqajMzM9M8cOBAUPReW1trmqZput1uc9q0\naea2bdvM++67z1y0aJFpmqb5xBNPmO+8844/S/SJv/zlL+aDDz5o3nPPPaZpmkHRs2maZkZGhlle\nXv6NZcHwc/7II4+Y8+bNM03TNJuamszKysqg6Ps0j8djDh8+3Dx+/HjA911QUGBmZGSYDQ0Npmk2\n/27Pnz8/4H/H9+3bZ1533XVmQ0OD6Xa7zRkzZphHjhw57+93m70M+vU5RO12e8scooFo0KBBREV9\nc/qL7OxssrKyAMjKymLZsmX+KM2nXC4XvXr1AiAiIoKuXbtSWFgYFL2HhTVPa9LY2Ijb7cYwDNav\nX8/48eOB5r6XLl3qzxIvuIKCAlauXMm0adNalq1bty6gez7NNE28Xu83lgX6z3l1dTWbNm1i6tSp\nANhsNpxOZ8D3/XVr1qyhY8eOpKSkBEXfXq+Xuro63G439fX1JCYmBvzfawcPHuTSSy/F4XBgtVoZ\nNGgQS5cuZfny5ef1/W6zYe1Mc4gWFRX5saKLq7S0lISEBKA51JSVlfm5It/Ky8tjz5499O/fn5Mn\nTwZ8716vl8mTJzN8+HCGDx9OamoqUVFRWCzNv7LJyckB9/P+zDPP8PDDD7fMsVhWVkZ0dHRA93ya\nYRjcddddTJ06lffeew8g4H/O8/LyiI2N5bHHHiMrK4vf/OY31NXVBXzfX7do0SKuu+46IPC/30lJ\nScyYMYPRo0czcuRInE4nvXv3Dvi/17p168bGjRupqKigrq6OnJwcCgoKzvv73WbDmqnh4YJGTU0N\ns2bNYvbs2URERHxjwuRAZbFYWLhwITk5OezYsYODBw9+Z51A+u+wYsUKEhIS6NWrV8vvtmma3/k9\nD6Sev+7vf/878+fP5/XXX2fu3Lls2rQpYHs9ze12s3v3bm699VYWLFhAWFgYr732WsD3fVpTUxPL\nly/n6quvBgL3Z/u0yspKsrOz+eyzz/j8889bgsu3Bdp/h65du3L33XczY8YMZs6cSc+ePbFaree9\nnzYb1n7MHKKBJD4+npKSEgCKi4uJi4vzc0W+4Xa7mTVrFpMmTWLs2LFA8PQOEBkZyeDBg9m+fTuV\nlZUtl8oKCgoC6ud9y5YtLF++nDFjxvDggw+yfv16nnnmGaqqqgK2569zuVwAxMXFMXbsWHbs2BHw\nP+fJyckkJyfTt29fADIzM9m9e3fA931aTk4Offr0aekv0Ptes2YNqampxMTEYLVaGTt2LFu3bg3o\nv9dOmzp1KvPnz+ftt98mOjqazp07n/f3u82GtWCbQ/TbZxgyMjKYP38+AAsWLAjY3mfPnk16ejrT\np09vWRbovZeWllJVVQVAfX09a9euJT09nSFDhvDpp58Cgdf3Aw88wIoVK8jOzubFF19kyJAh/OEP\nfwjonk+rq6ujpqYGgNraWlatWkX37t0D/uc8ISGBlJQUDh8+DDTfn5ienh7wfZ/28ccft1wChcD/\ne61du3Zs376dhoYGTNNk3bp1dOvWLSh+x0tLSwE4fvw4S5cu5brrrjvv73ebnm4qJyeH3//+9y1z\niAbq0B2nzzSUl5eTkJDAr371K8aOHct9993HiRMnaNeuHS+//PJ3HkJo6zZv3szPfvYzunfvjmEY\nGIbB/fffT79+/fj1r38dsL3v3buXRx99FK/Xi9frZcKECfziF7/g2LFjPPDAA1RWVtKrVy9eeOEF\n7Ha7v8u94DZs2MAbb7zBn/70p6Do+dixY9x7770YhoHH42HixInMnDmT8vLygP45B9izZw+PP/44\nbreb1NRUnn32WTweT8D3XV9fz+jRo1m2bBmRkZEAQfH9fuWVV/j444+x2Wz07t2bp59+moKCgoD/\nHb/tttuoqKjAZrPx2GOPMWTIkPP+frfpsCYiIiIS6NrsZVARERGRYKCwJiIiItKKKayJiIiItGIK\nayIiIiKtmMKaiIiISCumsCYiIiLSiimsiYj8BBs2bGiZjFxExBcU1kREfqJAm89QRFoXm78LEBHx\nlR07dvCHP/yhZTqnWbNmkZ6eztSpU8nKymL16tUAPPHEEwwaNAiAhQsX8uc//xmLxULHjh357W9/\n2zJv35w5c/joo4+wWCyEh4fzzjvvAM1z2D7xxBNs27YNi8XCiy++SJcuXfzQsYgEJFNEJABVVlaa\nkydPNouLi03TNM2ioiJz5MiRZm5urtmjRw/zgw8+ME3TNDds2GCOHDnSbGxsNPfu3WuOGDHCLCkp\nMU3TNF966SXz17/+tWmapjl//nzzpptuMmtqakzTNM3y8nLTNE1z/fr1Zp8+fczc3FzTNE3z1Vdf\nNR966KGL2quIBDadWRORgLRlyxby8vK4++67MU/Nqme1WnG73TgcDq6//noABg8eTGhoKIcPH2bD\nhg2MHj2a+Ph4AG6++WYmTZoEwIoVK7jlllsIDw8HIDo6uuVYaWlp9OzZE4D+/fuzYsWKi9WmiAQB\nhTURCVg9e/bk7bff/say/Pz876xnmiaGYbT8+e3lp1+fTUhISMvr04FQRORC0QMGIhKQBgwYwJEj\nR1i/fn3Lsp07d2KaJo2NjXz44YcAbNq0icbGRtLS0hg6dCgrV67k5MmTALz77rsMGzYMgIyMDN55\n552W+9/Ky8svckciEqx0Zk1EAlJUVBSvvvoq//mf/8mzzz5LY2MjHTt25PHHHycmJobc3Fxef/11\nAF588UVsNhvp6ek88MAD3HHHHVgsFlJTU3nqqacAmDx5MkVFRdx0001YrVYiIyOZO3euP1sUkSBh\nmN93bl9EJMDk5+czdepU1q1b5+9SRETOiS6DikjQ0bhoItKW6MyaiIiISCumM2siIiIirZjCmoiI\niEgrprAmIiIi0ooprImIiIi0YgprIiIiIq2YwpqIiIhIK/b/AVFgBeHPDd4FAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x556eb9081210>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmgAAAD+CAYAAACOT/k+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XmcFPW97/9XVfUy+74yrA47jssRg6jBsESIYJCAGuMx\nOXp/muUYTEyOGnM8ufea6I3J4XE05/6inhhJjMcsRkziGIOCiiuKoigIyjrAMAuz9/RML1V1/+iZ\nZkYQBkLTMz3v5+MB3V1d3f35dvdMvef7rfqW4bqui4iIiIgMGmayCxARERGR/hTQRERERAYZBTQR\nERGRQUYBTURERGSQUUATERERGWQU0EREREQGmYQGtNtvv53zzz+fSy+99BPX+eEPf8jFF1/M4sWL\n+eCDDxJZjoiIiMiQkNCA9oUvfIGHHnroE+9/8cUXqampYfXq1fzv//2/+cEPfpDIckRERESGhIQG\ntOnTp5OTk/OJ969Zs4bLLrsMgDPPPJOOjg4OHjyYyJJEREREBr2k7oPW0NBAWVlZ/HZpaSn19fVJ\nrEhEREQk+ZIa0I50linDMJJQiYiIiMjg4Unmi5eWllJXVxe/XVdXR0lJyTEf57qugpyIiJw0tuMS\njtiEIza24+KxTDyWgddjYpkmpnlom2PbDpGoQ6T3MurEltkO0aiD7bhEog5R28E0DXweE6/Hwusx\n8XpMfF4LwwDbdonasfVs2yViO7iui2WaWKaBaRp4rNh1y+q53lNX73LDMHB6ag/1/gvHLh3HxXVd\nXAAXevtEHLf/6/ZedxwXs+d1TcPod713k2sQv9L3Ina9zw3XJf66Ts8Lu66L23PbdV0cl3iNQKzd\nVv82W6ZJNP5e2/H3O2o7WKaJzxt7b31eE1/Pe2w7bvx9CPd5TyLR2GdrO268vbYde398XhO/14r9\n88Uuc7P8nDGhGMtMTt5IeEA72rnY586dy6OPPsoll1zCO++8Q05ODkVFRcd8TsMwaGzsOJllDgnF\nxdlq9zCidg8vhYVZ1De0xzYajovjxi5dJ7Yh6924ua6LA7hO74bVjQeDaE9I6H0O23EPPd9R1ned\njxXTu/E1wGuZsbDiMXuuG1hWbKMZ7tlghqMOkYhDxLaJRPts/HuCim3HXsA0Y4+1ejb6lmWQlual\nuyvSs9EmHiicni18bCMfW9Z7f9R24wEp2icoffyPd8PoaUrfZX2a6bgu4eihUHU0vYEpajscZbN2\nylmmccza5cTd+qWzmTQ6/+9+nuLi7ON+TEID2ne+8x3Wr19Pa2srn/nMZ/jmN79JJBLBMAyuvPJK\nLrroIl588UU++9nPkp6ezt13353IckRkGHPdQ6ElEnX69TiEIw6hiB3fUDtuLND0hqRYuOl5fN/w\nYTuEw86hnovIob/YI1EX2+npXekTjnp7DD4eurSJHTjTMOK9Ud6e4Jju98TDVyzM9fbaHHrcx99j\n0yDWo9WndyvWY2Yc6lnq+cwjPb1Nh4LqoR42T89tyzLwxHuBYr0/ruv26/XpDZZenwc7asd6ikwz\nvr5hEP/+9X5fbTv2Xer7Pez9DtqOg8eM9cr19iL5e3qVLCv2jhg9/xnEesIMg3id8V4r08Awjfgf\nBL3f/95a+up9T90jLexZ3huMzXjYj/W9ZWWl0RUMxW4bseWmaUCfn89D7XZwXBePGfsDwWOZeHt6\n2DyWGf8ZjL2/dvy9NQ0Dn9fC7+n/vvR+RrH33OjXMxqJ2oR7fi/0Xnosk9NG5J7IV/SkMNyjdXEN\nYsPxL+zh2rOgdqeu3g1W31+M6Rk+9h1oJ9gdobM7Smd3hM6uKN3haGyj1RueejYktt3z+GgsaPX9\nBRsPVc6p7fXwWL3BwfjYkFRsA2H2bJjiGyjA5/fg9AyJWR8bXjLNnnXp/1hPvHfL6BMaDm2Eep8r\n9prGJ6/fZwin79t0qNett7eqN7Q4eK2eoSWPidcbCy0+rxULLT1Dcd4+tfQOxR3q1YsFjPyCTFqa\nO2Nt7AkQZk/Xl0Fv59ehIbbe92OoGw4/30cynNt9vJK6D5qIDH2uG9vfIxCM0NEVoSMYpiMYif3r\nCtPZFY2HrWBP4Ap2Rwn17OtzslimEf9r2e+xyEjz4On5C9my+gQV08DXs8+K32vFehx8seseq2f9\nvvvgGEb/fYBMI9570rsPTO8+K73/TiRADIcNV2/ojLEAyM9OI9odSV5RknS2Y9Nth8jwpGv/8j4U\n0ESEQFeEuuYgdU1B6pqDtHeGiTq9+yn16UWJOh/bETnWUzXQmOX3xoJTQY6fNJ+nZ3ipZxjCY+L1\nWuTnpmO6DplpXjLSPGSme8lM85Du88R7hfr2LFnmoWEmOXliw64upnHi72vEidId7aY7GsJx7SOu\n0+3roL69lYgTJeJEiDpRIk4U13XJ9mWS5c0ix5dNhjf9sFpsxyYY7SIY7aI72k26J40cXw5pHv9R\n64o6UQKRTsJ2BK/pwWt68VpevKan32u4rovt2kQdG8e1iTg2ITvWnm67m65oiO5oNyE7hO06OD3/\nYvsJxq57DA8+y4fP8uHv+eezfLSZeQQ7o/jMQ8s9ZmyTHLJDdEaCBCKd8cvuaDeGYWIZFh7TwjJM\nLNODZZiE7QjdvXVFu+m2Y5eW6SHLm0GmN5NMbwZZ3kyyvJlYpkXYDhOyw4R7/oWcMBEnesT3y8DA\nb/lI86SRZvlJ77n0e/w4rkPEiRJ1ooTtCFEnEvv8cDExYz3EhoGBiWmY7I3A9rp9NHYdpCF4kMau\nJpq7W3BcB7/lozCtgKL0QgrT8ylKKyTPn0O3HYq/F53xyyAe00Oa5Y/V5fGTZsUuXdelO9pNl93/\n/Yg4kZ5hWwcXN/5Z5aXl8tWqr8Tf/8FicFUjIn+3SNQm2B0lGIrGL7t6rneFepaHonR1RznY1k1d\nc5BA18B6MCzTiB/llJHmJT87tr+L3+chK91LdkbvPx/Z6bHLzHQPGWmxkHWsENUV7SYr10uowz3p\nvyxtxyYQCRKIBHo2OH7SPX78lh/LtI762IEcOe66LiE73G8jEuh3GbsedsJEHRvbsWMbf9fGcWws\nj4lrG1imhcewsEyrz8bYii83ey8Nk7ATIRQNE3bC/Ta4UdfGdqLxYBG7bWMaJl7Tg6dPIPGaXsCN\nP7bvpUvsc/CbhwJGLGx44wEuvsFzHRxcQtFQfIMY/YRQdiJMwyTTm0GmJ4NuO0Qw2kXYDh9xXb/l\nI9eXQ44/m1xfDrbr0BHuoCMSoCPcSVe06xNfx+p5b23Xxjns6InEig1hx157uMj2ZjE2ZxQZnnSa\nu1tp6m6mtrPu2A88AaZhYmJgGD3BETPWQ26Y8SNNBxMFNJEhKNgdpfZgJ/saA+xv7GT/wQAHmoN0\ndkWJ2gPfqJiGQXFeGpUjcigtyKCsMIPyggzysv39doT2WCZRN0JLuAWv6cFv+eMb7CP1sDiug+3Y\nfTbQDhE3TCT+B7pLc3cr+wMHqA3UcaCzjv2BOlpCrfHnSPekk+PL6ulBycLv8ROywz09Mof+Oo7Y\nkViIiIeHniBheumKdtMeCRAIB+iMBPmkXfF9ppc0TxqWYWG7seDUW7/txNrgtbyHBRWPadEV7SYQ\n7qQzGiT6CT0Qn8TA6AlisQ1Gb3A70UMGTMPEZ3rxmJ54uPOafjJ6Qp7jOrGeDidCKBwi0tPbEXsP\nfPgtL37LR7YvC7/lwzIswnaEUE8ADEQ6CXW3EHEiPfuLGfGNnmmYGIaJ3/KR5c2kKL3wUO/GUUJw\nZrqfaNjtqduL14qFRheXQLizJ1T1/ouFrDTLT2l6EeneDDI86WR40knz+OmKdtMWbqc91EFbuJ3G\n1qb4e2lgkOXNJN+fy6jsCnJ8WXhNb79eu4gdez9s1z5iOLZMq1+bentu/JYfj2FhGLENfuw9iX2m\ntmsfMfxafmgNBD4WsEM4rhsLoj09Xpk9PWAZnrTYEax9vp+262C7dvz727c2v8ff80dJ/z8WOiOd\nRJ1oz8+wt8/Piw+v5T3iZ9T7x0esl663BzH282caFj6rJ/T3BH6P6cHEiPcixnoUY0G+ICebDCeb\nkowiitILSfekHfZanZEgB7ubaOpqpi3UTponrV8vYGbP5267zqGa7FC8B9E0zMM+ozTLP+h6yI5F\nBwkMIcNhH5UjSeV2d4WitHSECHRF6OyKEOjZIT7QFcE1Ddrauw8dHdgzpNgeDNPSEer3PAZQmJtG\ndoaXDL+H9J4eqwy/hzSfhekPEbUCdBvtBN02AnYrbdFWMrx+RmZXMDJrBCOzyinPLI3/kg6EO9nR\ntovtrbF/+wK1R+xR8PT8UnZdJx5oTiRg5PqyGZFVTn5WDgc7Wgn0bJQDkc7Dns80zJ5hljR8lpeI\nEyVkh3qGayL91s/wpJPtyybbl0m2N4tsXxZgfGxIKHY96trxDXHf3iqAsBM5bFgo6tqkWWmHNhy+\nDDI9mfFhpSxfz2WfYaZY8PHgMa1+4bbv97xvwO3taesdauvdODuuE9/A+iwffjM2RHYi+/Ac79yS\nJ3MuykT+fNuOTUckgGVYZHoz/q7h2pMtlX+vHc1wbvfxGlpxchALBAI8++wzLFmy7Lged8st3+IH\nP/ghmZlZCapMBgvXdTnY1s32fW18tL+ND2vrqY/UAAZOVxZudwa4Rx9q83lNMtO8TBmXQ1ERZOfa\npGdFMbzdBOwGgpHY/jhd0S6aeq4HI11EQ4f37HhNL9FglB1tu+PLTMOkNKMY13WpCzbEl1uGxdic\nUYzIKsd27P69AU6YiB3BMsx4oLHiQ3TmJ27Ic3w5jMgqoyKzjPKsMrK8mcDhv8Ad16EzEqQ7Gur3\nl/AnPa/rukScKGEnnPC/mh3XSchG3zRMTMvEy5F7NE624w1bQ2VHbsu0yPMnb5oEkb+HAtpJ0tHR\nzqpVfzgsoDmOg2l+8i/we+75j0SXJqeA47oEuiK0B8J0BMO0B/sezRimNRBmZ10bHXYTVl4jZl4j\n5uhWfP22cwbZZi4F/iJK0ksoycujrbOdkNtFl91FMBokGAnSEQmwOxJkN0B7z7+P6e1hyvCkk+/P\noyAtj+KMIorTiyhOL6Q4o5BcXw5hJ0JtoI79gVr2BQ6wP1DL/sABXGBy/gTG541jfN44xuSMxvcJ\nwx+JZhom2b7enq9jMwyjp2cp8fUOph4ZEUktCmgnyf33/ye1tfu57rqrsSwLn89PdnY2NTV7+O//\n/iPf+953aWxsIBwOcfnlV3HppZcBcPnln+ehhx4hGAzy3e8up6rqLN5//12Ki0v5P//n3/H5fElu\nmUCsV6a5PURtUyf7GztpaO2itSNEW2eI1kCY9s5wbMdeK4rhCWN4w+AJY3hDGN4whq8bz4Qm0rzd\nQGx/mHE5Y6kqmoLX8lLXWc+BzgbqOuvZ07WdPV3bobl/DQZGz1BaFqOyKsjz55KXlkueP5d8fy65\n/pz4vhl+yz+gXg6/5WNc7mjG5Y6OL+sdxlT4EBFJnpQMaL9fu503tzYce8XjcO7kEq6YM/4T7//6\n17/J7t07+eUvH2Xjxre45ZZv88gjv6esrAyA22//AdnZ2YRCIa6//stcdNEccnJy6HvikX379vK/\n/tfd3Hrr9/m3f/seL7ywlosvXnBS2yFH5jhuvKerN3S1dYZpbOmitqmT2uY2wp5WjIwOzIx2jLQg\nhj+KkRHFKLfxmVEwj37kVYYng6mFZ1FVOIUphZPI9GYcto7rugQinRzorMefaWJ3mfH9m9I9aack\nNCmYiYgkX0oGtMFg6tRp8XAG8Pvf/zcvvfQiAA0NDezbV8PUqafTd97u8vIRVFbGQuCkSZOpq6s9\npTUPB67r0tIRoqY+QE19B3vqO6ipD9Dc0R2bad5wMNI6MdI6MdMDmBkdmEUdGCODfHxmJa/pje2o\n7snomX8njXTLT1bPcFy2Lyu+U3qOL4vi9KJjTudgGEb8scN1Z1oREUnRgHbFnPFH7e06FdLSDh06\nvHHjW7z99gYefHAlPp+Pb37zq4TDh8/f03c40zStI64jAxcK2+xrDPBBXQ0ftG6jPlxDVySCHTXA\nsXAdE1wTX7GXvDERHF8HYbMDjP5HDGZ6MqjIHs/IrPLY0Y7ZIyjJKMY7xA7ZFhGRoUNbmJMkIyOD\nYDAIHDpJb6/OzgDZ2dn4fD727NnN5s3vH/E5huiMJwkVCHcSbG0jEjLJ8mYc1gMV7I5wsK27z78u\nGlo72RvcTcC7P7Yzvr8rNpLsj/37eB+WA3QTm3frtMzRlGWUUp5ZQllmKeWZpeT5c4fMUWsiIpIa\nFNBOkpycXKqqzuQrX/kiPp+fgoKC+H0zZpzPk0/+kX/8xysYPXoMp59e1eeRhzb8CgExYTvMpoNb\neLPubbY0fxjfad3AwG+mYTpp2CEvoW4Dm9i+X4Zlxy5NG/IiGAUOHsByfYzwTWBqwWTOG1VFUVZO\nnwkpI/HTy2R5M8nxZeszEBGRQUET1Q4hqbxPkuM6fNiygzfrNvJO43t027GJWPOtEnzRQpo62wk5\nwdgRkd4whqfPqYlcA4/hjc2E7vGR6U1nUmElpxdOoTJ37DH3+xqsUvnzPhq1e3hRu4eX4dzu46Ue\nNEkKx3XYF6hle+sudvTMVB+IdALgczLxNI8mUFtKbXds7qt0v8WEEbmML8plfEUuo8sysDwOXsvX\nc4oV9XyJiEjqUECTk85xHba37qIzEuwZRozEhxW77RB72veyq21PvJcMwIikE20ZiX1wBF2BfLLS\nffzD6Dwmj8lnxhkjyLAMTFMhTEREhgcFNDmpIk6Uhzf/N+82HvlAiF5WJJtoSzFORz5ORz5pRjan\nj85n8ox8Jo/Jp6I4E7OnV2y4domLiMjwpYAmJ03YjvBf7/+aLU3bqMwdx9klVXhMD00tYXbVBtm5\nL0BXt4sTzMLjpjG+Ipep0wqYOraAsWXZ6iETERHpoYAmJ0V3NMQDm1byYesOphRMZF7RZbyzrYU3\nPqinNRAFfORkljF7UglnTihkwsg8/N6hufO+iIhIoimgyVG1dLfyZt1GQk6Yc0rOZERW2WHrBCNB\n7n37IfZ17iU7Mppt68bzdnATAJlpHmadWc6nppQyaXQe1lFOHC8iIiIxCmhJ8tnPzuLZZ9clu4wj\nijhRNjVu5rUDb7K1+SPcntNRPbN7DaOzK5hRNp1peaeze3+IjTv38577V5z0VqJN5TTsnEx+lpez\nqvI5Z1IJp48rwGMplImIiBwPBbQkGYzTQtQG6ni5dj0b6jbSGY2dFWFczmhmlp9LmieNl/a+wfb2\n7dR0/InfO3/GaSvG8AcxMwLkhSuZe9rnmDq3iNL89EHZPhERkaFCAe0k+fnPf0ZZWTlLliwD4Je/\nfBDDMHjnnY0EAh1Eo1Guv/5rXHjhRUmu9HARO0L1rmd5ruZFXFyyfVnMG30RM8unk+ctZPUbe3l1\ncx0NLZXgHYmnsJb0sjqi+Q0AzKo4n8snfh7TUE+ZiIjIyZCSAe2J7U+xseG9k/qcZ5dU8YXxiz7x\n/nnzLubee/89HtDWrn2OFSt+xpVXXk1GRgZtba189avXDrqAtquthkc++D31wQYK0wr4woRFVBVO\nAQxeevcAT778Ou2dYfw+i3MmFnPG+ELOOG0OuVl+9nbU0tzdwhlFU9VjJiIichKlZEBLhgkTJtHa\n2kpT00FaWlrIycmhsLCIe+/9Ke+++w6maXDwYCMtLc3k5xcc+wkT7OO9ZheNPJ/FlZfgM728u72J\nP7ywnQNNQXxek89fMJYFM0aT5uv/dRmVPYJR2SOS1AIREZHUlfCAtm7dOu666y5c12Xp0qXccMMN\n/e6vra3l9ttvp7m5mby8PH7yk59QWlr6d73mF8YvOmpvV6LMnj2X559/jqamJubOvZjVq/9KW1sb\nDz/8KKZpcvnlnycUCp/yuj5uV9seHvngD9QHGyhKK+Afp1zOhPxKdh1o5/dr32fb3lYMA2adOYLL\nPj2OvCx/sksWEREZVhIa0BzH4c4772TlypWUlJSwbNky5s6dS2VlZXydH//4xyxZsoTFixezfv16\n/v3f/5177rknkWUlzJw5n+Wee35IW1sb//mfD7JmzbPk5xdgmiZvv72BuroD8XWTdY762kAdK97+\nOY7rcNHIC1hc+Tk6Omwe/PNmXt9SD8CZlYUsmz2eiqLMpNQoIiIy3CU0oG3atIkxY8ZQUVEBwMKF\nC1mzZk2/gLZjxw5uv/12AGbMmME3vvGNRJaUUOPGnUYwGKS4uJSCgkIuvngBt956M1/5ylVMnjyF\nMWPGxddN1j5ba/auw3Ed/mnqVUzLO50/rdvDcxv2EbUdxpRlc8Xs8UwZk5+U2kRERCQmoQGtvr6e\n8vLy+O3S0lLee6//zvuTJ09m9erVXHPNNaxevZpgMEhbWxu5ubmJLC1hfvWr38av5+bmcf/9vzzi\neqtXv3iqSoprD3ewoW4jxelFtNQUcutvX6OzO0phjp8vXFTJjKml8fNfioiISPIkNKANZBjvlltu\n4c4772TVqlVMnz6d0tJSLEunAEqEl/a/TtS16aip4Le7tpPut1j2mUrmnTMSn067JCIiMmgkNKCV\nlZVRW1sbv11fX09JSUm/dUpKSvjZz34GQDAYZPXq1WRlZR3zuYuLs09usUPEibY7Ykd4+ZXXMR0f\nLXuKmX/eGK753BRyh8gBAPq8hxe1e3hRu4eX4dru45XQgFZVVUVNTQ379++nuLiY6upqVqxY0W+d\nlpYW8vLyMAyDBx54gKVLlw7ouRsbOxJR8qBWXJx9wu1+7cAG2kMdROrHMX1COVdcdBrhrjCNXck/\nqvRY/p52D2Vq9/Cidg8vavfwciKhNKFTv1uWxR133MF1113HokWLWLhwIZWVldx33308//zzALzx\nxhssWLCABQsW0NzczNe+9rVEljQsua7LX3e8gOsa5AQn8E+fm6yJZUVERAaxhM+DNmvWLGbNmtVv\n2fLly+PX58+fz/z58xNdxrC28cA2msINOC1l/POiT5GR5k12SSIiInIUOnliinMcl/9+528AfGbk\nhYwrz0lyRSIiInIsCmgp7nevvEvQv5+0aCFXfGp6sssRERGRAVBAS2Fb97Twwr5XMAxYMmUupqmP\nW0REZCjQFnuI29VWwx8/+gt72vf2W94eDHN/9TtYRfvI8mQzs+LsJFUoIiIixyvhBwlI4nSEAzzw\n3ko6wgHW7n2JcTmjuWjkBZxdUsVjz31EMGMnXstm7ugLsUxNRCsiIjJUKKANUa7r8tjWP9IRDvDp\nipm0dLewuWkbu7Y8xu+3/Zm2tjLSRtRhml4uqJiR7HJFRETkOCigDVGvHdjAuwc3MyHvNK6YuBjT\nMGkMNvH83pd5sWY93ood2MDM8vPI9GYku1wRERE5DgpoQ9DBriYe/+hPpHvS+PLUKzGN2K6ExRmF\nWHWn07UxnTOmd5NT0sGCMXOSXK2IiIgcLwW0IcZ2bH615beE7DBfmfpFCtLy4/ftawzwzPoaCrOy\n+PqFc/H7tN+ZiIjIUKSjOIeY1XteYGfbHs4pOZNzSw8dmem4Lr9+Zhu243LN/IkKZyIiIkOYAtoQ\nsqN5D0/vfpY8fy5fnLSk3/k0171by/b9bUyfXMIZlUVJrFJERET+XhriHCLCdpifvfkwjutwzZQr\nyOiz439bIMTjz+8g3W9x1dwJSaxSRERETgb1oA0RT+54mtqOeuaM+jSTC/qHsMfWfEQwFGXpRZXk\nZ/uTVKGIiIicLApoQ8Ce9r2s2/caFTllfP60Bf3u27SjiTc+aOC0ETl85uyKJFUoIiIiJ5OGOAc5\nx3X43YdP4uLy/51zFV7DG7/vYGsXv3hqC5Zp8JUFkzH77JMmIiIiQ5d60Aa51w68yZ72vUwvPYtp\nJRPjy0Nhm/v++B6BrghXf3Yio0qykliliIiInEwKaINYZyTIn3b8Fb/lY8n4hfHlruvyUPUW9jUG\n+MzZFRraFBERSTEKaIPYn3c+Q2ckyCXjPkuePze+/KlXd7NhWyMTR+XxpXk6alNERCTVKKANUjXt\n+3hl/3rKMkuZPfLC+PKNHzay6qVdFOb4+cZlp+Ox9BGKiIikGm3dB6G+BwZcOXExlhk7K8CeunYe\nfGoLPo/JN5eeQU6mL8mVioiISCIooA1Crx/YwO72Gs4pOZOJ+eMBCHRF+NEv3yAUtrlu4RRGl2Yn\nuUoRERFJFAW0Qab3wACf5eMLExbFl6/861YONHWycOYYPjWlNIkVioiISKIpoA0yT+38G4FIJ5eM\nnRc/MKCmvoO3P2xkytgClsw6LckVioiISKIpoA0i7eEOXq5dT2lGMbNHHTow4Jn1NQBcMW+iJqMV\nEREZBhTQBpGNDe/huA6frpiJx4yd5KGxtYs3PmhgZHEm50wuSXKFIiIiciokPKCtW7eOBQsWMH/+\nfB588MHD7j9w4ABf/vKXWbJkCYsXL+bFF19MdEmD1lv172BgcHZJVXzZ396owXFdPnfeGAz1nomI\niAwLCT0Xp+M43HnnnaxcuZKSkhKWLVvG3LlzqaysjK/z85//nEsuuYQvfvGL7Nixg+uvv561a9cm\nsqxBqaW7lR1tu5mQd1p837P2YJiXNx2gMCeNT01R75mIiMhwkdAetE2bNjFmzBgqKirwer0sXLiQ\nNWvW9FvHMAwCgQAA7e3tlJYOzyMU327YBMA5pWfGl63ZsI9w1GH+p0ZhmRqNFhERGS4S2oNWX19P\neXl5/HZpaSnvvfdev3VuvPFGrrvuOh555BG6u7t5+OGHE1nSoPVW/buYhslZxbHhze5wlLVv7yMr\n3cunzxyR5OpERETkVEpot4zrusdcp7q6mqVLl/Liiy/ywAMP8C//8i+JLGlQagw2sadjL5Pyx5Pt\nywJg3Tu1dHZHmXfOSPxeK8kVioiIyKmU0B60srIyamtr47fr6+spKem/L9Xjjz/OQw89BMBZZ51F\nKBSiubmZgoKCoz53cXHqzKT/0paXAZg9/jyKi7OJRB2ee2sffp/F5RdP7ndKp1Rq9/FQu4cXtXt4\nUbuHl+Ha7uOV0IBWVVVFTU0N+/fvp7i4mOrqalasWNFvnREjRvDqq6+yZMkSduzYQTgcPmY4A2hs\n7EhU2afHo0LIAAAgAElEQVTcup1v4DEsxvkraWzs4OVNBzjY1s1np48iFAzRGAwBsS91KrV7oNTu\n4UXtHl7U7uFlOLf7eCU0oFmWxR133MF1112H67osW7aMyspK7rvvPqqqqpg9eza33nor//qv/8rK\nlSsxTZMf//jHiSxp0DnQWU9tZx1nFE0jw5uO47r8df0eLNNg/qdGJbs8ERERSYKEBjSAWbNmMWvW\nrH7Lli9fHr9eWVnJY489lugyBq236t8FDh29+e72gxxoCnL+6WUU5KQlszQRERFJEs3dkESu6/JW\nwzv4TC9VRVMB+OvrsdM6fW7G6GSWJiIiIkmkgJZE+wK1NAQPcnrRFPyWj/0HO9m+v42q0wqpKM5K\ndnkiIiKSJApoSXRoePMsANZvqQfg/NPLklaTiIiIJJ8CWpK4rsuG+ndIs/xMK5iE67qs31KH32tx\n1viiZJcnIiIiSaSAliS72mtoCbVyZvHpeC0vuw500NjazdkTi/D7NDGtiIjIcKaAliRv1b8DHDp6\n8/UtdQDMmDI8z0UqIiIihyigJYHjOrzdsIlMTwaT8yfgOC5vftBAVrqXaeOOPUmviIiIpLYBBbTq\n6mqi0Wiiaxk2NjVupj3cwVklVVimxdaaFto6w0yfXILHUmYWEREZ7gaUBp566inmzJnDvffeS319\nfaJrSmm2Y/OXnX/DNEzmjvo0AK/3HL05Y0rJ0R4qIiIiw8SAAtrPf/5zHnvsMaLRKEuXLmX58uW8\n/vrria4tJb1R9zZ1wQbOK5tOaWYJkajDW9sayc/2M2FUXrLLExERkUFgwONpFRUVfOc73+G+++5j\n06ZNfP3rX+fSSy9lw4YNiawvpUTsCNW7nsVjerhk3DwA3tvZRFcoyoyppZiGkeQKRUREZDAY0Lk4\nw+EwTz/9NI899hi2bfOtb32LSy65hE2bNnHLLbewdu3aRNeZEl6qfZ2WUCtzR80iPy3WW3ZoeFNH\nb4qIiEjMgALanDlzmDFjBrfddhtnn312fPn06dOZOXNmwopLJd3Rbv62ey1pVhoXj50NQFcoyrvb\nD1JemMHoUp3aSURERGIGFNCeeOIJSkqOvAP7j370o5NaUKpas/clApFOFo2bT5Y3E4CNHzUSiTrM\nmFqKoeFNERER6TGgfdCefPJJWltb47dbWlr4xS9+kbCiUk1HOMCamhfJ9mYxe9SF8eXx4c2pGt4U\nERGRQwY8D1pe3qEjDPPz83nqqacSVlSq+duetYTsMAvGziXN4wegPRhmy64WxpVnU5qfkeQKRURE\nZDAZUEBzXfewZbZtn/RiUlFzdwsv7XuNwrR8LqiYEV++YWsDjusyY2pZEqsTERGRwWhAAW3s2LE8\n/PDDuK6L4zj88pe/ZPTo0YmuLSVU73qWqGuzcNzFeM1Du/y9vqUeAzh3sianFRERkf4GdJDA97//\nff7lX/6FFStWYBgGZ599Nvfcc0+iaxsy1h94i6d3PYtlWnhNL17Tg9f04rE8fND0IeWZpZxbdujo\n1z11HWzf18aUMfnkZ/uTWLmIiIgMRgMKaKWlpfz6178mGAwCkJGhfaZ6he0Iq7ZX0xkNkunJIOK0\nE3Gi2G5sCNjAYMn4hZhGrLOyrTPMz57YBMD8T6kXUkRERA43oIAG0NHRwa5duwiFQvFl5557bkKK\nGkpeqV1PRyTAgjFzuLRyQXy54zpEnNgJ5v2WD4BI1OE/n9hEc3uIL8w6jTMqC5NSs4iIiAxuAwpo\nTz/9ND/+8Y9pb2+npKSEmpoaJk+ezKpVqxJd36AWcaI8V/MiPtPL7J4Tn/cyDTMezCB2oMWvntnK\njv3tnDe1lIUzx5zqckVERGSIGNBBAvfffz9PPPEEY8aM4W9/+xu/+MUvqKqqSnRtg976AxtoDbXx\n6YqZZPkyj7ruM2/U8Or7dYwrz+GfPjdZE9OKiIjIJxpQQPN4PBQWFsan1rjgggt47733ElrYYGc7\nNqv3PI/H9DB39KyjrvvORwd5/Pkd5Gf7+ebSKnxe6xRVKSIiIkPRgIY4fT4frusyZswYHnnkESoq\nKuIHDAxXb9ZvpKm7hYtGnk+uP+cT19vXGOCBv2zG6zH55tIq8rJ01KaIiIgc3YAC2k033UQgEOC7\n3/0u//N//k86Ojr4wQ9+MKAXWLduHXfddReu67J06VJuuOGGfvfffffdrF+/HsMwCAaDtLS08MYb\nbxx/S04hx3X42561WIbFvNEXfeJ67cEw9z2+iVDY5uuXnc7Ysk8OciIiIiK9jhnQbNumpqaGmTNn\nkp2dzcqVKwf85I7jcOedd7Jy5UpKSkpYtmwZc+fOpbKyMr7O9773vfj13/zmN3zwwQfH14Ik2Niw\niYbgQc4v/xQFaflHXCcUsbnv8U0cbOvm8xeM1YS0IiIiMmDH3AfNsix+97vfndCTb9q0iTFjxlBR\nUYHX62XhwoWsWbPmE9d/6qmnWLRo0Qm91qniuA7P7F6LgcHFY2YfcR3bcXjgT5vZWdvOzGllLL5w\n3CmuUkRERIayAR0kMGPGDJ555pnjfvL6+nrKy8vjt0tLS2loaDjiurW1tezfv5/zzjvvuF/nVHrv\n4AfUdtYxvfRsijMOn8fMdV0eXf0h72w/yNSx+Vx7iY7YFBERkeMzoH3QVq1axcMPP0xaWhrp6em4\nrothGLz22mtHfdyRTrL+Saqrq5k/f/6Aw0xxcfaAn/tkcV2X5zY+j4HBVWcvojj38Bp+99w2Xnin\nltNG5PKD62eSkeY9qTUko92Dgdo9vKjdw4vaPbwM13YfrwEFtD/+8Y8n9ORlZWXU1tbGb9fX11NS\ncuR9sZ5++ukBH3gA0NjYcUI1/T02N21jZ0sNZxdX4Q9nHVbDy5sO8Ju/bqUwJ41/XnI6nR3ddHZ0\nn7TXLy7OTkq7k03tHl7U7uFF7R5ehnO7j9eAAlpFRcVxPzFAVVUVNTU17N+/n+LiYqqrq1mxYsVh\n6+3cuZP29nbOOuusE3qdU8F1XZ7Z/RwAC8bOPez+93c28atntpKZ5uHbV5ypk6CLiIjICRtQQDvv\nvPOOOPR4rCFOy7K44447uO6663Bdl2XLllFZWcl9991HVVUVs2fHdrJ/+umnWbhw4QmUf+psbf6I\nnW17qCqaysjsEf3u27K7mf/75PsYhsHyZWcwoujoZxUQEREROZrjHuIMhUL85S9/weMZ2HnWZ82a\nxaxZ/WfaX758eb/bN95444CeK1lc1+WpXasBWDjus/HlO2rbeHLdTjbvbsEAvrHkdCaMzEtSlSIi\nIpIqTmiI86abbuKKK67gn//5nxNS1GCzuWkru9trOKv4dEZlV7CnroMnX9rJuzuaAJg2Np8lsyo5\nbYQmohUREZG/38C6wT5m7969NDU1nexaBiXXdane9SwAMwov5P+ueo+3tjUCMHFkLktmncak0Uee\nrFZERETkRBz3PmiO4xCNRvn+97+f0MIGi/ebPqCmYx9nFlXx6ycP0Nwe4rQROSz59GlMHZuvOc5E\nRETkpDvufdA8Hg9FRUVYlpWwogYL13Wp3rkaAwN/82Sa2ztY8KnRXD67UsFMREREEmZAZxLo7Owk\nPz+fiooKSktLCYVCfPTRR4muLek2HdzM3kAtU/Om8fIbAQpy/Cy+cJzCmYiIiCTUgALabbfdhtd7\naEZ8j8fDrbfemrCiBgPHdaje9SwGBu27xmA7Ll+cMwG/L/V7DkVERCS5BhTQbNvuF9B8Ph+2bSes\nqMHg3cbN7A8coDJjCh9+ZDN1bD7nTCpOdlkiIiIyDAwooHk8Hvbu3Ru/XVNTk9L7oDmuw9M9vWf7\n3x+BZRpc/dmJGtoUERGRU2JABwnceOONXHXVVVx00UUAvPjii/zwhz9MaGHJtLHhPWo76yg3JrLz\noIfPzRhFeaHODiAiIiKnxoAC2uzZs/nNb37DK6+8AsANN9zAmDFjElpYstiO3dN7ZlKzqZy8LB+L\nzh+b7LJERERkGBlQQGtubmbEiBFcffXVAEQiEZqbmykoKEhoccnw/L6XqQs2kNs9nrqudK78/ATS\n/Sc0n6+IiIjICRnQPmhf/epX+x0UEIlE+NrXvpawopKlubuF6p2rSTPTqdsymkmj8vjUlJJklyUi\nIiLDzIACWjgcJj09PX47IyODUCiUsKKS5fcf/omwE4HaqZi2n6sv1oEBIiIicuoNKKBBbJizV1NT\nE47jJKSgZHm3cTPvHdzCqIwxtNQUMWNqCSOLs5JdloiIiAxDA9q56pprruGqq65i8eLFuK7Ln//8\nZ66//vpE13bKdEdD/OHDP2EZFqMjM/mQVs6aoDnPREREJDkGFNCWLVvG6NGjeeGFFwD40Y9+xPTp\n0xNZ1ylVvWs1LaFWPjd2Lu++7GAaBtPG5ie7LBERERmmBhTQOjo6eOmll/joo4/o7u7m/fffB+DX\nv/51Qos7FfZ21PLCvlcoSi/k/JILWVW7ngkjc8lI8x77wSIiIiIJMKB90G6//XYsy2L37t1ceeWV\nWJbFGWeckejaEs5xHX677Qkc1+GLE5fw4Z4OXKCqsjDZpYmIiMgwNqCAtmfPHr71rW+RlpbGokWL\neOCBB9iwYUOia0u4V2rXs7u9hnNKzmRK4UQ27WwC4IzKoiRXJiIiIsPZgAKaz+cDwOv10traitfr\n7XdU51DU1NXCn3b8lXRPGksnfB7HcXl/ZzP52X5GFuu0TiIiIpI8A9oHbezYsbS2tnLppZdy5ZVX\nkp2dzbRp0xJdW8IEI0H+/3cfoivazZcmLyXXn82O/W0EuiLMOrNcc5+JiIhIUg0ooP30pz8F4Npr\nr6WqqoqOjg5mzZqV0MISJeJEefC9X1MXbGD2qAu5YMQMAN7rGd6sOk3DmyIiIpJcx32SyaE8vYbr\nujz6wR/4qHUnZxWfzhfGL4rft2lHE5ZpMFXTa4iIiEiSDfhMAqngqZ1/4836jYzLGc1Xpl6FacSa\n39YZZnddBxNH5enE6CIiIpJ0CQ9o69atY8GCBcyfP58HH3zwiOs8/fTTLFy4kEsvvZTvfve7Canj\nlf3reWbPWorSC/nqGf+Ezzo0z9n78eFNTa8hIiIiyZfQ7iLHcbjzzjtZuXIlJSUlLFu2jLlz51JZ\nWRlfZ8+ePfziF7/gd7/7HVlZWQk5OnRz0zZ+++EqMr0Z/POZ15Ht63+Ozfj+Z5r/TERERAaBhPag\nbdq0iTFjxlBRUYHX62XhwoWsWbOm3zq///3v+dKXvkRWViw0FRQUnNQa6jobeOj9R7AMk6+dcS0l\nGf3PsWk7Du/vbKYwJ40RhRkn9bVFRERETkRCA1p9fT3l5eXx26WlpTQ0NPRbZ/fu3ezatYurrrqK\nL37xi7z00ksntYZXa98gZIf54qQvcFrumMPu37G/nWAoyhmVhZpeQ0RERAaFhA5xuq57zHVs26am\npoZHH32U2tparr76aqqrq+M9an+vzU1b8Zpezik584j3v6f9z0RERGSQSWhAKysro7a2Nn67vr6e\nkpKSfuuUlpZy9tlnY5omI0eOZNy4cezevZvTTz/9qM9dXJx9zNdvCBykLtjAP4yoYkTZkYdOt+xp\nwWOZfPqcUaQNgSM4B9LuVKR2Dy9q9/Cidg8vw7XdxyuhiaSqqoqamhr2799PcXEx1dXVrFixot86\n8+bNo7q6mssuu4zm5mb27NnDqFGjjvncjY0dx1xn3b63AJiYPf6I67d0hNhV2860cQV0tHdx7GdM\nruLi7AG1O9Wo3cOL2j28qN3Dy3Bu9/FKaECzLIs77riD6667Dtd1WbZsGZWVldx3331UVVUxe/Zs\nPv3pT/PKK6+wcOFCLMvilltuITc396S8/uamrQBMLZh8xPt7hzfP0PCmiIiIDCIJH9ObNWvWYaeF\nWr58eb/bt912G7fddttJfd2wHeHDlu2UZ5ZSmH7kswO8t0PTa4iIiMjgk7JnEviodQcRJ8q0wiP3\nnkVth827mynJS6c0P/0UVyciIiLyyVI2oL1/MDa8efonBLQN2xroDtuaXkNEREQGnZQMaK7rsrnp\nA9KsNE7LHXvY/VHb4cl1u7BMg3nnHvuABBEREZFTKSUDWn2wkabuFqYUTMAyrcPuf2nTARpau7jo\nrBGU5Gl4U0RERAaXlAxovUdvHmn/s1DE5s8v78LnNbn0/LGnuDIRERGRY0vJgPZ+7/QaRwhoz23Y\nS1tnmIvPHUVulv9UlyYiIiJyTCkX0Lqi3exo3cXo7Apy/f0nhuvsjvDX12vITPOw4FOHn5dTRERE\nZDBIuYC2rWU7tmsfcXjzr6/XEAxFWThzLBlpg/+0TiIiIjI8pVxA23zwyPuftXSEeG7DXvKz/cz5\nh4pklCYiIiIyICkV0GLTa2wly5vJmJz+02f85dXdhKMOiy8ch897+JGdIiIiIoNFSgW0fYEDtIXb\nmVIwCdM41LT65iDr3qmltCCDC6rKklihiIiIyLGlVEDrnV7j9MJJ/ZavemknjuuydNZpWGZKNVlE\nRERSUEqllc1NWzEwmNInoNXUd/DGBw2MKcvmnEnFSaxOREREZGBSJqAFIp3satvDuNwxZHoz4svf\n3NoAwKKZY3XOTRERERkSUiagbW36EBf3sKM3t+1txTBg6tj8JFUmIiIicnxSJqBta9kOwNTCifFl\n4YjNrtp2Rpdmk+7XvGciIiIyNKRMQNvfWYdlWFRklseX7ahtx3ZcJo3KS2JlIiIiIscnJQKa4zrU\nddZTmlGMZR6a4+zDva0ACmgiIiIypKREQGvpbiVkhxmR1X+Os201LRjABAU0ERERGUJSIqDVdtYB\nUJ5ZGl8WiTrsqG2nojiLrHRvskoTEREROW4pEdAOdNYDUJ55qAdtd107kaij4U0REREZclIioNUG\nYgFtRJ+Atq2mZ/+z0QpoIiIiMrSkREA70FmH1/RSmH5orrPeAwS0/5mIiIgMNUM+oDmuQ12wgfLM\nkvgJ0m3H4aP9bZQXZpCb6UtyhSIiIiLHZ8gHtMauJqJOtN/+Z3vqAoTCtvY/ExERkSEp4QFt3bp1\nLFiwgPnz5/Pggw8edv+qVauYOXMmS5YsYcmSJTz++OPH9fwHAocfwdk7vDlRAU1ERESGoISe/8hx\nHO68805WrlxJSUkJy5YtY+7cuVRWVvZbb+HChfzrv/7rCb1G7xQbfedA21bTAsCk0Tr/poiIiAw9\nCe1B27RpE2PGjKGiogKv18vChQtZs2bNYeu5rnvCr9E7xUbvEZyO4/LhvjZK8tLJz/af8POKiIiI\nJEtCA1p9fT3l5YfOjVlaWkpDQ8Nh661evZrFixdz0003UVdXd1yvUdtZT5qVRp4/F4B9jQG6QlEN\nb4qIiMiQldCANpCesTlz5rB27Vr+9Kc/MXPmTG699dYBP3/UidIQbKQ8sxTDMADNfyYiIiJDX0L3\nQSsrK6O2tjZ+u76+npKSkn7r5Obmxq9fccUV/PSnPx3QcxcXZ1PTuh/HdTitaBTFxdkA7G4IAHDe\nmRUUF2b+vU0YdHrbOdyo3cOL2j28qN3Dy3Bt9/FKaECrqqqipqaG/fv3U1xcTHV1NStWrOi3TmNj\nI8XFxQCsWbOG8ePHD+i5Gxs72Fy/E4B8q4DGxg5c1+W97QcpyPFj2jaNjR0nt0FJVlycnXJtGgi1\ne3hRu4cXtXt4Gc7tPl4JDWiWZXHHHXdw3XXX4bouy5Yto7Kykvvuu4+qqipmz57NI488wtq1a/F4\nPOTm5nL33XcP+Pk/PsVG7cFOAl0RZp52aMhTREREZKhJaEADmDVrFrNmzeq3bPny5fHrN998Mzff\nfPMJPXdt7xGcPVNsaP4zERERSQVD+kwCBzrryPJmku3NAmDb3t4DBDT/mYiIiAxdQzaghe0wB7ua\n40dwuq7Ltr2t5Gb6KM1PT3Z5IiIiIidsyAa0us4GXNz4OTgbWrpoC4SZOCpP+5+JiIjIkDZkA1r8\nDAJZsQMEDg1vav8zERERGdqGbEDrPQdnbw/a5l3NAEzSAQIiIiIyxKVAQCslHLHZtKOJkrx0RhSl\n3uS0IiIiMrwM2YB2IFBPri+HTG8G7+9qJhSxOWdysfY/ExERkSFvSAa0YKSLllBrfP6zDdtiJ2Cf\nPqnkaA8TERERGRKGZEDb13YAiA1vRqIO73x0kKLcNMaW6fxeIiIiMvQNyYBW0xY7AXt5ZhmbdzfT\nHbaZPqlEw5siIiKSEoZkQNvXE9BGZJXy1tbY8OY5k4uTWZKIiIjISTMkA9re9lhAK04rZuNHBynI\n8XNaeU6SqxIRERE5OYZkQKtpO0BhWj479gYJhqKcM1HDmyIiIpI6hmRAa+tupzyz7NDRmxreFBER\nkRQyJAMaQGlGCRs/bCQ3y0dlRW6yyxERERE5aYZsQKM7m87uKNMnlmBqeFNERERSyJANaHW1FqDh\nTREREUk9QzKgGRhs/TBCToaXCSN1cnQRERFJLUMyoM0deTGBTod/mFSCaWp4U0RERFLLkAxodt04\nAM6dpOFNERERST1DMqC99t4BstK9TByt4U0RERFJPUMyoLV0hPiHicVY5pAsX0REROSohmzC0dGb\nIiIikqqGZEArzE1j8uj8ZJchIiIikhBDMqDde/Nn8FhDsnQRERGRY0p4ylm3bh0LFixg/vz5PPjg\ng5+43jPPPMPkyZPZvHnzMZ8zN8t/MksUERERGVQSGtAcx+HOO+/koYce4qmnnqK6upodO3Yctl5n\nZye/+c1vOOussxJZjoiIiMiQkNCAtmnTJsaMGUNFRQVer5eFCxeyZs2aw9a79957uf766/F6vYks\nR0RERGRISGhAq6+vp7y8PH67tLSUhoaGfut88MEH1NXVcdFFFyWyFBEREZEhw5PIJ3dd95j333XX\nXfz4xz8e8GNEREREUl1CA1pZWRm1tbXx2/X19ZSUlMRvd3Z2sn37dq655hpc1+XgwYN84xvf4Oc/\n/znTpk076nMXF2cnrO7BTO0eXtTu4UXtHl7UbjmahAa0qqoqampq2L9/P8XFxVRXV7NixYr4/VlZ\nWbz22mvx29dccw3f+973mDp1aiLLEhERERnUEhrQLMvijjvu4LrrrsN1XZYtW0ZlZSX33XcfVVVV\nzJ49u9/6hmFoiFNERESGPcNVIhIREREZVDQdv4iIiMggo4AmIiIiMsgooImIiIgMMgk9SCAR1q1b\nx1133YXruixdupQbbrgh2SUlxO23384LL7xAYWEhf/nLXwBoa2vj29/+Nvv372fkyJH8x3/8B9nZ\nqXW4cl1dHbfccgsHDx7Esiwuv/xyvvzlL6d828PhMFdffTWRSATbtpk/fz433ngj+/bt4+abb6at\nrY1p06Zxzz334PEMuR/bo3Ich6VLl1JaWsr9998/LNoMMGfOHLKysjBNE4/Hw+OPP57y33OAjo4O\nvv/97/PRRx9hmiZ33XUXY8eOTel279q1i29/+9vxA+H27t3LTTfdxOLFi1O63StXruTxxx/HMAwm\nTpzI3XffTUNDQ8r/fP/qV7/i8ccfB/j7tmHuEGLbtjtv3jx33759bjgcdj//+c+727dvT3ZZCfHm\nm2+6W7ZscRctWhRfds8997gPPvig67qu+8ADD7g/+clPklVewjQ0NLhbtmxxXdd1A4GAe/HFF7vb\nt28fFm0PBoOu67puNBp1L7/8cvedd95xb7rpJvfpp592Xdd1/+3f/s197LHHklliQjz88MPud77z\nHferX/2q67rusGiz67runDlz3NbW1n7LhsP3/NZbb3Uff/xx13VdNxKJuO3t7cOi3b1s23YvuOAC\nt7a2NqXbXVdX586ZM8cNhUKu68Z+rp944omU//n+8MMP3UWLFrmhUMiNRqPutdde6+7evfuEPush\nNcQ50HN7poLp06eTk5PTb9maNWtYsmQJAEuWLOG5555LRmkJVVxczJQpUwDIzMyksrKS+vr6YdH2\n9PR0INabFo1GMQyD9evXM3/+fCDW7meffTaZJZ50dXV1vPjii1x++eXxZa+//npKt7mX67o4jtNv\nWap/zwOBABs2bGDp0qUAeDwesrOzU77dfb366quMHj2a8vLylG+34zh0dXURjUbp7u6mpKQk5X+n\n7dixg7POOgufz4dlWUyfPp1nn32WtWvXHvdnPaQC2kDO7ZnKmpubKSoqAmJBpqWlJckVJda+ffvY\nunUrZ555Jk1NTSnfdsdxuOyyy7jgggu44IILGDVqFDk5OZhm7Me0rKws5b7vd911F7fccguGYQDQ\n0tJCbm5uSre5l2EY/I//8T9YunQpf/jDHwBS/nu+b98+8vPz+d73vseSJUu444476OrqSvl29/X0\n00+zaNEiILU/79LSUq699lo+85nPMGvWLLKzs5k6dWrK/06bMGECb775Jm1tbXR1dbFu3Trq6upO\n6LMeUgHN1ZRtw0ZnZyfLly/n9ttvJzMzM74BT2WmafLkk0+ybt06Nm3axI4dOw5bJ5XehxdeeIGi\noiKmTJkS/9l2Xfewn/NUanNfv/3tb3niiSf4r//6Lx599FE2bNiQsm3tFY1G2bJlC1/60pdYtWoV\n6enpPPjggynf7l6RSIS1a9eyYMECIHW/2wDt7e2sWbOG559/npdeeikeVj4u1d6DyspKrr/+eq69\n9lpuuOEGJk+ejGVZJ/RcQyqgHevcnqmusLCQgwcPAtDY2EhBQUGSK0qMaDTK8uXLWbx4MfPmzQOG\nT9shdgq0c889l3fffZf29vb4MFhdXV1Kfd/ffvtt1q5dy9y5c/nOd77D+vXrueuuu+jo6EjZNvdV\nXFwMQEFBAfPmzWPTpk0p/z0vKyujrKyMqqoqAC6++GK2bNmS8u3utW7dOqZNmxZvXyq3+9VXX2XU\nqFHk5eVhWRbz5s1j48aNKf07rdfSpUt54okneOSRR8jNzWXs2LEn9FkPqYDW99ye4XCY6upq5s6d\nm+yyEubjPQlz5szhiSeeAGDVqlUp2/bbb7+d8ePH85WvfCW+LNXb3tzcTEdHBwDd3d289tprjB8/\nnmKmAFkAAASLSURBVBkzZvDMM88Aqdfum2++mRdeeIE1a9awYsUKZsyYwU9/+tOUbnOvrq4uOjs7\nAQgGg7z88stMnDgx5b/nRUVFlJeXs2vXLiC2v+H48eNTvt29qqur48ObkNq/10aMGMG7775LKBTC\ndV1ef/11JkyYMCx+vpubmwGora3l2WefZdGiRSf0WQ+5Uz2tW7eOH/3oR/Fze6bqNBu9PQqtra0U\nFRXxzW9+k3nz5nHTTTdx4MABRowYwb333nvYgQRD3VtvvcU//uM/MnHiRAzDwDAMvv3tb3PGGWfw\nrW99K2Xbvm3bNm677TYcx8FxHC655BK+/vWvs3fvXm6++Wba29uZMmUKP/nJT/B6vcku96R74403\n+OUvf8n9998/LNq8d+9ebrzxRgzDwLZtLr30Um644QZaW1tT+nsOsHXrVv5fe/fzCu0exnH8Y2ZC\nmlDWMzWaMrGQItHTNPkDGFkgGxtbSTZSU6QmpclCSbKSpixE2FmMhWKSxGKyYkHJj0IpTaPrWZzz\nTJ1zOqdznp7D7fZ+bebu29TcV9+mPl33j2tiYkKFQkGBQEDJZFJvb2+ur/v19VWxWEy7u7vy+/2S\n5Pr9np+f187Ojnw+n+rr6zU9Pa2bmxvX/78HBgb09PQkn8+n8fFxtba2/tRef7qABgAA4Haf6hIn\nAADAV0BAAwAAcBgCGgAAgMMQ0AAAAByGgAYAAOAwBDQAAACHIaABwH+UzWaLA78B4P9AQAOAn+C2\nGYIAnMX30ScAAL/S6empZmdni6OUhoeHFQ6H1dPTo+7ubu3v70uSEomEmpubJUkbGxtaXl6Wx+NR\nMBjU5ORkcVbe4uKitre35fF4VFFRoXQ6Lem3mbGJREInJyfyeDxKpVKqra39gIoBuJIBgEs8Pz9b\nPB63u7s7MzO7vb21aDRquVzO6urqbHNz08zMstmsRaNRy+fzdn5+bt++fbP7+3szM5ubm7ORkREz\nM1tfX7fe3l57eXkxM7PHx0czMzs8PLSGhgbL5XJmZrawsGBjY2PvWisAd6ODBsA1jo+PdXV1paGh\nIdnvU+y8Xq8KhYJKS0vV2dkpSWppaVF5ebkuLi6UzWYVi8VUU1MjSerr61NXV5ckKZPJqL+/XxUV\nFZKkqqqq4m+FQiFFIhFJUmNjozKZzHuVCeALIKABcJVIJKKVlZU/rF1fX//le2amkpKS4uef138c\n/52ysrLi8Y8QCAC/Cg8JAHCNpqYmXV5e6vDwsLh2dnYmM1M+n9fW1pYk6ejoSPl8XqFQSG1tbdrb\n29PDw4MkaW1tTe3t7ZKkjo4OpdPp4v1sj4+P71wRgK+KDhoA16isrNTCwoJmZmaUTCaVz+cVDAY1\nMTGh6upq5XI5LS0tSZJSqZR8Pp/C4bBGR0c1ODgoj8ejQCCgqakpSVI8Htft7a16e3vl9Xrl9/u1\nurr6kSUC+CJK7J96+ADgAtfX1+rp6dHBwcFHnwoA/Ctc4gTwJfDeMgCfCR00AAAAh6GDBgAA4DAE\nNAAAAIchoAEAADgMAQ0AAMBhCGgAAAAOQ0ADAABwmO/L2XRoDivmFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x556eae026510>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 47s, sys: 32.3 s, total: 3min 19s\n",
      "Wall time: 2min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# Specify the model hyperparameters (NOTE: All the defaults can be omitted):\n",
    "model_params = {\n",
    "        'hidden_sizes' : [500, 300], # List of hidden layer dimensions, empty for linear model.\n",
    "        'l2_lambda' : 1e-3                     # Strength of L2 regularization.\n",
    "}\n",
    "\n",
    "# Specify the training hyperparameters:\n",
    "training_params = {'num_epochs' : 100,     # Max epochs/iterations to train for.\n",
    "                        'optimizer_fn' : tf.train.MomentumOptimizer(learning_rate=2e-3, momentum=0.98),\n",
    "                        'report_every' : 1, # Report training results every nr of epochs.\n",
    "                        'eval_every' : 1,     # Evaluate on validation data every nr of epochs.\n",
    "                        'stop_early' : True,    # Use early stopping or not.\n",
    "}\n",
    "\n",
    "# Build, train, evaluate and plot the results!\n",
    "trained_model, training_results = build_train_eval_and_plot(\n",
    "        model_params, \n",
    "        training_params, \n",
    "        verbose=True    # Modify as desired.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tB5r6-oLTzgE"
   },
   "source": [
    "# NB: Before you go (5min)\n",
    "\n",
    "Pair up with someone else and go through the questions in \"Learning Objectives\" at the top. Take turns explaining each of these to each other, and be sure to ask the tutors if you're both unsure!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3ryLRtd2XfeX"
   },
   "source": [
    "# Additional Resources\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PTnXEgI1oMY9"
   },
   "source": [
    "* TensorFlow Debugging (useful tips and code patterns): https://github.com/wookayin/tensorflow-talk-debugging\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0ZrPt_Ao3z1U"
   },
   "source": [
    "# Feedback\n",
    "\n",
    "Please send any bugs and comments to dli-practicals@googlegroups.com."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "default_view": {},
   "last_runtime": {
    "build_target": "",
    "kind": "local"
   },
   "name": "Practical 2: Deep MNIST and Best Practices (Solution)",
   "provenance": [],
   "version": "0.3.2",
   "views": {}
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}