MLPs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Networks: Multilayer perceptrons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of today's session is to get you familiar with neural networks, backpropragation and training. To do that, we will not use any external libraries and we will code from scratch a vanilla multilayer perceptron. \n",
    "\n",
    "After this session, we will use standard deep learning libraries, as coding a neural network from scratch is not that easy, as you will see."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activation functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Activation functions introduce non-linearities in our network. They are a fundamental ingredient in a neural networks, as they allow them to learn highly non-linear mappings between the input and the target domains."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now we will look only at the classical sigmoid activation function, defined as \n",
    "\n",
    "\\begin{align}\n",
    "S(z) = \\frac{1}{1 + e^{-z}}.\n",
    "\\end{align}\n",
    "\n",
    "To train our networks, we will need to compute its derivative:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{dS}{dz}(z) = \\frac{d}{dz} \\left( \\frac{1}{1 + e^{-z}} \\right) = -1 (1 + e^{-z})^{-2} (-e^{-z}) = \\frac{e^{-z}}{(1 + e^{-z})^2} = S(z)(1-S(z))\n",
    "\\end{align}\n",
    "\n",
    "Implement the sigmoid function and its gradient below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Sigmoid:\n",
    "    @staticmethod\n",
    "    def forward(z):\n",
    "        return 1 / (1 + np.exp(-z))\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(z):\n",
    "        return Sigmoid.forward(z) * (1 - Sigmoid.forward(z))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you are done, you can test your implementation by running the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAhNklEQVR4nO3de5xU9X3/8ddnZ3dZkIUFF1hgQVBuonhd0cR4SYwXSAxqm6q1rbFJjX3Emrb212jTNPk1ze+XyyNNml8uhhhrkiYaTbxQxYBJagwxGLnDguBy3WUvLPcF9jYzn98fM+g4zrLDMrNnLu/n4zHMnHO+M+ezZ2bfnP3OOd9j7o6IiOS/kqALEBGRzFCgi4gUCAW6iEiBUKCLiBQIBbqISIFQoIuIFAgFuhQcM7vDzJbm2nrN7CUz+9hg1iTFRYEuecvM3mNmr5jZITPbb2a/M7NL3P3H7n7dYNcT1HpFjisNugCRgTCzEcBzwF8DTwDlwBVAd5B1iQRJe+iSr2YAuPtj7h5x9053X+ru68zsI2a27HhDM7vOzDbH9+S/bWa/Od71EW/7OzP7mpkdNLNtZvbu+PxGM9tjZncmvNZIM/uhmbWb2U4z+2czK0l4rcT1Xmtmr8fX+03ABm3rSFFSoEu+2gJEzOwHZjbPzEalamRm1cDPgAeB04HNwLuTml0KrIsv/wnwOHAJMA34M+CbZjY83vb/ASOBM4GrgL8A7upjvT8H/hmoBrYClw/0hxVJhwJd8pK7HwbeAzjwPaDdzBaZ2bikpvOBend/yt3DwDeA1qQ22939P909AvwUmAT8q7t3u/tSoAeYZmYh4FbgQXfvcPcdwFeBP09R4nxgo7v/zN17ga+nWK9IRinQJW+5+yZ3/4i71wLnAhOIBWeiCUBjwnMcaEpq05bwuDPeLnnecGJ72uXAzoRlO4GJKcpLtd7GFO1EMkaBLgXB3V8HHiUW7IlagNrjE2ZmidMnaS/QC5yRMG8ysDtF2xZie/qJ652Uop1IxijQJS+Z2Swzu9/MauPTk4DbgeVJTZ8H5pjZTWZWCnwCqBnIOuNdMk8AXzCzSjM7A/h74L9SNH8eOMfMbomv976BrlckXQp0yVcdxL7MfNXMjhIL8g3A/YmN3H0v8GHgy8A+YDawgoEf3vg3wFFgG7CM2JeojyQ3SljvF+PrnQ78boDrFEmL6QIXUkzihxg2AXe4+/8EXY9IJmkPXQqemV1vZlVmNgT4J2LHgyd3zYjkPQW6FIN3ETsOfC9wI3CTu3cGW5JI5qnLRUSkQGgPXUSkQAQ2OFd1dbVPmTIlqNWLiOSllStX7nX3MamWBRboU6ZMYcWKFUGtXkQkL5nZzr6WqctFRKRAKNBFRAqEAl1EpEAo0EVECoQCXUSkQPQb6Gb2SPwyXBv6WG5m9g0zazCzdWZ2UebLFBGR/qSzh/4ocMMJls8jNpLcdOBu4DunXpaIiJysfo9Dd/eXzWzKCZosAH4YvyLL8vggSOPdvSVTRYpIYXJ3eiNOVzhCd2+Urt4I3eEokagTjh6/99h9xFPPjzqRaJRIFKLu4OA47hBNeOzuOLz5OLYsPj+xHbHXOT4qyvG2b9Wc9DPgKZe9Y1CVhIV1U0Zz5YyU5wadkkycWDSRt19aqyk+7x2BbmZ3E9uLZ/LkyRlYtYgExd053Bmm/Ug3e+O3A0d7ONwVpqMrTEdXLx1dYY50v/W4qzdCV2+U7vBb99EiGk7KLHZ/z1Vn5WygW4p5Kd8id18ILASoq6srordRJP+4Oy2Huti+9yiN+4/RdKCTxgPHaNx/jJZDXew70kNPJJryueWhEiorSqmsKGV4RSmVQ8qYNHoYw8pDVJSGGFJWQkVZiCGlb90POX5fWkJZqIRQiVFaYvH7+HTIUs8vMUrMMCN+M0oMjIR5xx/DW20xrCQ2z8zetgxizyuJzzd7K+qSQy9h0dvaDbZMBHoTb79WYi3QnIHXFZFBEok6r7ceZvWug7zeepjNrR1sbu3gcFf4zTahEmP8yAomjRrGu8+qZkzlEKqHl1M9fEjsVlnO6NPKGVFRRkVZKMCfpnhlItAXAfea2ePELgl2SP3nIrktEnXWNh3kN5vbWbnzAKt3HeBoTwSAyopSZo6r5MbzJzCrppKzxg5n0qhhjB9ZQWlIRzrnsn4D3cweA64Gqs2sCfgsUAbg7g8Bi4H5QANwDLgrW8WKyMB19Ub41aY9/HJTG7/Z0s7+oz2UGMyqGcEtF9VSN2UUF00eRe2ooYF2G8jApXOUy+39LHdiV1IXkRzj7ry6fT9Pr9rN4vUtdHSHGX1aOVfPGMPVs8Zy5fRqqoaVB12mZEhgw+eKSPZ09UZ4ds1uHlm2g81tHZxWHuKGc8dzy0UTuezM0wmVaA+8ECnQRQpIV2+E/1q+k++8tJV9R3uYVVPJl//4PD543niGlevXvdDpHRYpANGo8+TKRr724hu0Hu7iiunV/PVVZ/Gus05Xf3gRUaCL5Ln65kN8+ukNrGk8yIWTq/jarRfwrrNOD7osCYACXSRP9YSjfPXFzTz82+1UDS3ja7eez00XTNQeeRFToIvkoW3tR7jv8dVs2H2YW+sm8eD8WTpaRRToIvnmFxta+Psn1lJeWsJ3//xirj+nJuiSJEco0EXyhLvzzV838NUXt3Dh5Cq+c8fF1IysCLosySEKdJE8EI5E+cefr+OpVbu55aKJ/J+b52i8FHkHBbpIjusOR7jvsdUsqW/j/mtncO/7pumLT0lJgS6Sw7p6I9z9o5W8vKWdz944m7sunxp0SZLDFOgiOSociXLfY6t5eUs7X/qjOdx6iS4KIyemsTBFcpC7809Pr2fpxjY+e+NshbmkRYEukoO+9uIWnljRxH3XTFc3i6RNgS6SYxavb+Ebv27gwxfX8nfvnx50OZJHFOgiOWRTy2Huf2ItF06u4t9uPldHs8hJUaCL5IjDXb3c/aMVjBhaynf/7GKGlOo4czk5OspFJEd85pkNNB/s4omPX8bYEToDVE6e9tBFcsDTq5t4dk0zn7xmOhefMTrociRPKdBFAta4/xifeaaeuVNG84n3Tgu6HMljCnSRALk7Dz61HoB/v/V8XetTTokCXSRAP1+1m2UNe/nUvFnUjhoWdDmS5xToIgHZe6Sbf3t+I3VnjOKOuToTVE6dAl0kIF94fhNHu8P831vmUKKuFskABbpIAFbu3M/Tq3fz8SvPYvq4yqDLkQKhQBcZZNGo86/PbWLciCH89dVnBV2OFBAFusgge3btbtY2HuQfr5/FaUN0bp9kjgJdZBAd6wnzpRc2c17tSG6+cGLQ5UiBUaCLDKIfvLKT1sNdfOaDs/VFqGScAl1kkHR09fLdl7dy9cwxXDJFp/dL5inQRQbJI8t2cPBYL/dfOzPoUqRApRXoZnaDmW02swYzeyDF8pFm9t9mttbM6s3srsyXKpK/Dh3r5eFl27hu9jjm1I4MuhwpUP0GupmFgG8B84DZwO1mNjup2SeAje5+PnA18FUzK89wrSJ56+Fl2+joCvN3184IuhQpYOnsoc8FGtx9m7v3AI8DC5LaOFBpscurDAf2A+GMViqSpzq6enn0lR3MO7eGs8ePCLocKWDpBPpEoDFhuik+L9E3gbOBZmA98El3jya/kJndbWYrzGxFe3v7AEsWyS8/eXUXHV1hnUQkWZdOoKc6tsqTpq8H1gATgAuAb5rZO3ZF3H2hu9e5e92YMWNOslSR/NMdjvD9Zdu5fNrpnFdbFXQ5UuDSCfQmYFLCdC2xPfFEdwFPeUwDsB2YlZkSRfLXM6t3s6ejm3uu0t65ZF86gf4aMN3Mpsa/6LwNWJTUZhdwDYCZjQNmAtsyWahIvolGne++vI1zJozgPdOqgy5HikC/A0m4e9jM7gWWACHgEXevN7N74ssfAj4PPGpm64l10XzK3fdmsW6RnPfSlj1saz/KN26/kNjxAiLZldbIQO6+GFicNO+hhMfNwHWZLU0kvz36yk7GjRjCvHNrgi5FioTOFBXJgq3tR3h5Szt3XHoGZSH9msng0CdNJAt+9PudlIWM2+ZO6r+xSIYo0EUy7Eh3mJ+tbOIDc8YztrIi6HKkiCjQRTLsqVVNHOkOc+e7pwRdihQZBbpIBrk7P/z9Ts6rHckFk6qCLkeKjAJdJINW7TpAw54j3HHpZB2qKINOgS6SQT99rZFh5SE+cN6EoEuRIqRAF8mQI91hnlvXwo3nTWC4Lv4sAVCgi2TIc2ubOdYT4U8u0aGKEgwFukiGPP5aI9PHDueiyVVBlyJFSoEukgFb2jpY03iQWy+ZpC9DJTAKdJEMeOK1RspCxs0XJl/7RWTwKNBFTlE4EuXZtc1cPXMspw8fEnQ5UsQU6CKn6JWt+2jv6OYW7Z1LwBToIqfomTW7qawo5b2zxgZdihQ5BbrIKTjWE2bJhlY+MGc8FWWhoMuRIqdAFzkFL25s42hPhJvU3SI5QIEucgqeWb2bCSMrmDtldNCliCjQRQZq75FuXn5jLx+6YCIlJTr2XIKnQBcZoOfXtRCJuo49l5yhQBcZoEVrm5lVU8nMmsqgSxEBFOgiA9J8sJOVOw/wwfPGB12KyJsU6CID8MKGVgDmz1GgS+5QoIsMwPPrmjl7/AjOHDM86FJE3qRAFzlJzQc7WbXroLpbJOco0EVOkrpbJFcp0EVO0vPrmpk9fgRTq08LuhSRt1Ggi5yE490tH1B3i+QgBbrISVi8vgVQd4vkJgW6yElYvL5F3S2Ss9IKdDO7wcw2m1mDmT3QR5urzWyNmdWb2W8yW6ZI8NTdIrmutL8GZhYCvgVcCzQBr5nZInffmNCmCvg2cIO77zIzjfQvBWdJvY5ukdyWzh76XKDB3be5ew/wOLAgqc2fAk+5+y4Ad9+T2TJFgre0vo0Z44aru0VyVjqBPhFoTJhuis9LNAMYZWYvmdlKM/uLVC9kZneb2QozW9He3j6wikUCcOBoD3/YsZ/rZtcEXYpIn9IJ9FQDPXvSdClwMfAB4HrgM2Y24x1Pcl/o7nXuXjdmzJiTLlYkKL9+fQ+RqHPdOeOCLkWkT/32oRPbI5+UMF0LNKdos9fdjwJHzexl4HxgS0aqFAnY0o2t1IyoYM7EkUGXItKndPbQXwOmm9lUMysHbgMWJbV5FrjCzErNbBhwKbAps6WKBKOzJ8JvtrRz3TnjMNOViSR39buH7u5hM7sXWAKEgEfcvd7M7okvf8jdN5nZL4B1QBR42N03ZLNwkcGyrGEvXb1R9Z9LzkunywV3XwwsTpr3UNL0V4CvZK40kdywtL6VyopSLj1TF4KW3KYzRUVOIByJ8stNbVwzayxlIf26SG7TJ1TkBFbuPMCBY71cd466WyT3KdBFTmBJfRvlpSVcOUOH2UruU6CL9MHdWbqxlfdMq2b4kLS+bhIJlAJdpA+bWjpoOtDJdbN1MpHkBwW6SB+WbmzFDK45W4Eu+UGBLtKHpfVtXDx5FGMqhwRdikhaFOgiKTTuP8bGlsMau0XyigJdJIUXN7YBcK3ODpU8okAXSWHpxlaNfS55R4EukuTA0R7+sF1jn0v+UaCLJPnV63uIOuo/l7yjQBdJsrReY59LflKgiyTo7Inw8hsa+1zykwJdJMFv32jX2OeStxToIgmWbmzT2OeStxToInHhSJRfbWrjfRr7XPKUPrUicSviY59fr7HPJU8p0EXilmrsc8lzCnQRNPa5FAYFugga+1wKgwJdBI19LoVBgS6Cxj6XwqBAl6Knsc+lUCjQpehp7HMpFAp0KXoa+1wKhQJdiprGPpdCokCXoqaxz6WQKNClqGnscykkCnQpWhr7XAqNAl2K1rKGvRr7XApKWoFuZjeY2WYzazCzB07Q7hIzi5jZH2euRJHsWFrfqrHPpaD0G+hmFgK+BcwDZgO3m9nsPtp9CViS6SJFMi0cifKr1/do7HMpKOl8kucCDe6+zd17gMeBBSna/Q3wc2BPBusTyYo/7NjP/qM96m6RgpJOoE8EGhOmm+Lz3mRmE4GbgYdO9EJmdreZrTCzFe3t7Sdbq0jGvLC+lYqyEt47S2OfS+FIJ9BTff3vSdNfBz7l7pETvZC7L3T3OnevGzNGv0gSjEjU+UV9K++dOZZh5Rr7XApHOp/mJmBSwnQt0JzUpg54PH7oVzUw38zC7v5MJooUyaSVOw/Q3tHNvDnjgy5FJKPSCfTXgOlmNhXYDdwG/GliA3efevyxmT0KPKcwl1y1eH0LQ0pLeN+ssUGXIpJR/Qa6u4fN7F5iR6+EgEfcvd7M7okvP2G/uUguiUadFza0cNWMMbrUnBSctD7R7r4YWJw0L2WQu/tHTr0skexY3XiAtsPdzFd3ixQgHYArRWXx+lbKQyW872x1t0jhUaBL0XB3XljfwhXTqxlRURZ0OSIZp0CXorG26RDNh7p0dIsULAW6FI3F61soCxnXnq2xz6UwKdClKESjzvPrWrh8WjUjh6m7RQqTAl2KwqpdB9h9sJMFF0wIuhSRrFGgS1F4dk0zFWUlXKvBuKSAKdCl4PVGojy/voX3nz1OJxNJQVOgS8Fb1rCX/Ud7WHDBxP4bi+QxBboUvEVrmhk5tIyrZmiETylsCnQpaJ09EZbUtzJ/Tg3lpfq4S2HTJ1wK2i83tXGsJ8KHzld3ixQ+BboUtGfXNFMzooK5U3UhaCl8CnQpWHuPdPPS5j186IIJhEpSXXhLpLAo0KVgPbN6N+Go8+GLa4MuRWRQKNClILk7P1vZxPmTqpg+rjLockQGhQJdCtKG3Yd5vbVDe+dSVBToUpCeXNlIeWkJN56vsVukeCjQpeB0hyM8u6aZ68+pYeRQjawoxUOBLgXnlxv3cKizV90tUnQU6FJwHvvDLiaMrODyadVBlyIyqBToUlC27z3Ksoa93D53so49l6KjQJeC8uPlOyktMW6dOynoUkQGnQJdCkZXb4QnVzZx/bk1jK2sCLockUGnQJeC8dy6Fg519vJnl54RdCkigVCgS8H40fKdTBs7nMvO1EBcUpwU6FIQ1jcdYm3jQe64dDJm+jJUipMCXQrC95dtY/iQUv5Ix55LEVOgS95rPtjJf69r4dZLJjGiQmeGSvFSoEvee/SVHQDcdfmUQOsQCVpagW5mN5jZZjNrMLMHUiy/w8zWxW+vmNn5mS9V5J06unp57NVdzDu3htpRw4IuRyRQ/Qa6mYWAbwHzgNnA7WY2O6nZduAqdz8P+DywMNOFiqTy09ca6egOc/eVZwZdikjg0tlDnws0uPs2d+8BHgcWJDZw91fc/UB8cjmgb6Yk67p6I3zvt9uYO3U059VWBV2OSODSCfSJQGPCdFN8Xl8+CryQaoGZ3W1mK8xsRXt7e/pViqTw5IpG2g53c9/7pgddikhOSCfQUx3U6ykbmr2XWKB/KtVyd1/o7nXuXjdmzJj0qxRJ0h2O8O2XtnLxGaO4fNrpQZcjkhPSCfQmIHGko1qgObmRmZ0HPAwscPd9mSlPJLWfrWyi5VAXn7xmuk4kEolLJ9BfA6ab2VQzKwduAxYlNjCzycBTwJ+7+5bMlynylp5wlG//z1YunFzFFdM15rnIcaX9NXD3sJndCywBQsAj7l5vZvfElz8E/AtwOvDt+N5S2N3rsle2FLOfvLqT3Qc7+cLN52rvXCRBv4EO4O6LgcVJ8x5KePwx4GOZLU3knQ539fKNXzfw7rNO56oZ+h5GJJHOFJW88t3fbGX/0R4enHe29s5FkijQJW+0HOrk4d9uZ8EFE5hTOzLockRyjgJd8sZXfrEZd/iH62YGXYpITlKgS15Yvm0fT63ezV9dOZVJozVmi0gqCnTJeT3hKJ95ZgO1o4Zy73t1VqhIX9I6ykUkSI/8bjtv7DnC9++sY2h5KOhyRHKW9tAlp+3Ye5T/+OUbXDd7HNecPS7ockRymgJdclYk6tz/5FrKQsb/XnBO0OWI5Dx1uUjOWvjyNlbuPMDXb72A8SOHBl2OSM7THrrkpI3Nh/n3Fzczf04NCy6YEHQ5InlBgS45p6Orl0/8ZBVVw8r5t5vm6IxQkTSpy0Vyirvzv55cx679x3jsry5j9GnlQZckkje0hy455Xu/3cYv6lt5cN4s5k4dHXQ5InlFgS45Y2l9K1984XXmz6nho++ZGnQ5InlHgS45YfWuA9z3+Grm1Fbx1Q9foH5zkQFQoEvgtrUf4aM/WMHYygqdDSpyChToEqit7Ue4beFyDHj0rkuoHj4k6JJE8pYCXQKztf0Ity9cTtSdx+6+jDPHDA+6JJG8psMWJRCrdx3gYz9YgRn85K8uY8a4yqBLEsl72kOXQbe0vpXbv7ec04aU8tOPv0thLpIh2kOXQROJOt/8dQNf/9UWzq+t4uE769RnLpJBCnQZFPuOdPO3P13Db9/Yyy0XTuQLN8/R0SwiGaZAl6xyd55b18LnFtXT0R3mi7fM4dZLJuk4c5EsUKBL1jQf7OSzi+p5cWMb59eO5Mt/fD4za9RfLpItCnTJuMNdvXznpa18f9l2DPin+bP4y8unUhrSd/Ai2aRAl4w5eKyHH/5+J//5u+0cONbLzRdO5B+un8nEKl2cQmQwKNDllG1tP8JPXt3FY3/YxbGeCFfPHMP9185kTu3IoEsTKSoKdBmQQ529LKlv5ckVjby24wChEuPG88bz8avO4uzxI4IuT6QoKdAlbU0HjvHS5naW1Lfy+637CEedM6tP41M3zOKPLprI2BEVQZcoUtQU6JJSNOrs2n+MVbsO8Put+1i+fR+N+zsBmFp9Gh+9YirXn1PDhZOqdAiiSI5QoBc5d2fvkR527DvK9r1H2dzawYbdh9jYfJiO7jAAVcPKuHTqaD56+VQun1bNtLHDFeIiOSitQDezG4D/AELAw+7+xaTlFl8+HzgGfMTdV2W4VjlJ4UiUw11h9h3ppu1wN22Hu2jr6GJP/HHTgU527D36ZnADVJSVMHv8CG66cCLnThzBnIlVzKqppKREAS6S6/oNdDMLAd8CrgWagNfMbJG7b0xoNg+YHr9dCnwnfi/E9oIjUSccv0UiTjgaTT0dOd42Sm/E6eqN0NkboSt+6+yJ0BWOxu8jdPVEONYT4VBn75u3jq4whzp7OZIQ1IkqK0oZN6KCCVVDuWhyFVOqT2Nq/DaxaqiOFxfJU+nsoc8FGtx9G4CZPQ4sABIDfQHwQ3d3YLmZVZnZeHdvyXTBL23ew+efi63a4/84sdA8Ps8dHI/d+1vPdfc3l8faxtuQ2C5xXqw9x1/z+PSbzz/xa+IQiYd5NpSXljC0LMTQshAjh5YxcmgZtaOGMiL++Pht9Gnl1IyoYNyICsaOGMKwcvW0iRSidH6zJwKNCdNNvHPvO1WbicDbAt3M7gbuBpg8efLJ1gpAZUUZs2pGQLwHwGKvG79/5zwM4o8w4812b5sXb/j258faHH9OvP6E10nxmseXJ6w3VAKlJSWUlhihkFFaYrHpkBEq6Xs6FDLKSkqoKCuhoizE0PJQ7L4sREVZCUNKQ4TUDSIiCdIJ9FSpkbzLmU4b3H0hsBCgrq5uQLutF58xiovPGDWQp4qIFLR0OkubgEkJ07VA8wDaiIhIFqUT6K8B081sqpmVA7cBi5LaLAL+wmIuAw5lo/9cRET61m+Xi7uHzexeYAmxwxYfcfd6M7snvvwhYDGxQxYbiB22eFf2ShYRkVTSOtzB3RcTC+3EeQ8lPHbgE5ktTUREToYOOBYRKRAKdBGRAqFAFxEpEAp0EZECYe7ZOS293xWbtQM7B/j0amBvBsvJlFytC3K3NtV1clTXySnEus5w9zGpFgQW6KfCzFa4e13QdSTL1bogd2tTXSdHdZ2cYqtLXS4iIgVCgS4iUiDyNdAXBl1AH3K1Lsjd2lTXyVFdJ6eo6srLPnQREXmnfN1DFxGRJAp0EZECkbOBbmYfNrN6M4uaWV3SsgfNrMHMNpvZ9X08f7SZvWhmb8TvM35VDDP7qZmtid92mNmaPtrtMLP18XYrMl1HivV9zsx2J9Q2v492N8S3YYOZPTAIdX3FzF43s3Vm9rSZVfXRblC2V38/f3w46G/El68zs4uyVUvCOieZ2f+Y2ab45/+TKdpcbWaHEt7ff8l2XQnrPuF7E9A2m5mwLdaY2WEz+9ukNoOyzczsETPbY2YbEuallUUZ+X1095y8AWcDM4GXgLqE+bOBtcAQYCqwFQileP6XgQfijx8AvpTler8K/Esfy3YA1YO47T4H/EM/bULxbXcmUB7fprOzXNd1QGn88Zf6ek8GY3ul8/MTGxL6BWJX5LoMeHUQ3rvxwEXxx5XAlhR1XQ08N1ifp5N5b4LYZine11ZiJ98M+jYDrgQuAjYkzOs3izL1+5ize+juvsndN6dYtAB43N273X07sTHY5/bR7gfxxz8AbspKocT2SoA/AR7L1jqy4M2Lf7t7D3D84t9Z4+5L3T0cn1xO7MpWQUnn53/z4ufuvhyoMrPx2SzK3VvcfVX8cQewidj1efPFoG+zJNcAW919oGehnxJ3fxnYnzQ7nSzKyO9jzgb6CfR1Qepk4zx+1aT4/dgs1nQF0Obub/Sx3IGlZrbSYhfKHgz3xv/kfaSPP/HS3Y7Z8pfE9uRSGYztlc7PH+g2MrMpwIXAqykWv8vM1prZC2Z2zmDVRP/vTdCfq9voe8cqqG2WThZlZLuldYGLbDGzXwI1KRZ92t2f7etpKeZl7djLNGu8nRPvnV/u7s1mNhZ40cxej/9PnpW6gO8Anye2XT5PrDvoL5NfIsVzT3k7prO9zOzTQBj4cR8vk/HtlarUFPMGdPHzbDCz4cDPgb9198NJi1cR61I4Ev9+5Blg+mDURf/vTZDbrBz4EPBgisVBbrN0ZGS7BRro7v7+ATwt3QtSt5nZeHdvif/JtycbNZpZKXALcPEJXqM5fr/HzJ4m9ufVKQVUutvOzL4HPJdiUVYu7J3G9roT+CBwjcc7D1O8Rsa3Vwo5e/FzMysjFuY/dvenkpcnBry7Lzazb5tZtbtnfRCqNN6bIC8YPw9Y5e5tyQuC3Gakl0UZ2W752OWyCLjNzIaY2VRi/8v+oY92d8Yf3wn0tcd/qt4PvO7uTakWmtlpZlZ5/DGxLwY3pGqbKUl9ljf3sb50Lv6d6bpuAD4FfMjdj/XRZrC2V05e/Dz+fcz3gU3u/u99tKmJt8PM5hL7Pd6Xzbri60rnvQnygvF9/qUc1DaLSyeLMvP7mO1vfQd6IxZETUA30AYsSVj2aWLfCG8G5iXMf5j4ETHA6cCvgDfi96OzVOejwD1J8yYAi+OPzyT2jfVaoJ5Y10O2t92PgPXAuviHYnxyXfHp+cSOotg6SHU1EOsnXBO/PRTk9kr18wP3HH8/if0Z/K348vUkHG2VxZreQ+xP7XUJ22l+Ul33xrfNWmJfLr8723Wd6L0JepvF1zuMWECPTJg36NuM2H8oLUBvPL8+2lcWZeP3Uaf+i4gUiHzschERkRQU6CIiBUKBLiJSIBToIiIFQoEuIlIgFOgiIgVCgS4iUiD+P5M1KGshnA7fAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAt2klEQVR4nO3deXwb9Z3/8ddHsuU7tmM7TmI7dwjkJjEJ91WuQHfTQg8oLW23Lctu2W53t7+Fbbtst8c+ut2etHRZSu+WAi3QUghXSylQIOS0cwfntOIkduzYjo/40uf3h0ZBCDmWE0kjyZ/n4+GHpTk/Gtlvjb4z8x1RVYwxxmQuj9sFGGOMSSwLemOMyXAW9MYYk+Es6I0xJsNZ0BtjTIazoDfGmAxnQW9SnohcJCI7ErTsz4rI/YlY9knWmbDXczrrFZGfiMiXk1mTSQ4LehNXIrJXRHpF5JiItIvIKyJym4ic8t+aqr6kqnPiUNulIuKPWPZ/qerHT3fZUdY1T0SeFZGjznZYJyLXOuuMy+sZLbfWa9xnQW8S4a9UtQiYCnwVuAP44aksSESy4llYEv0eeA6oBCYAnwI6Xa3IjFkW9CZhVLVDVR8H3g98WETmA4hIjoh8XUT2i8hhEblXRPKccZeKiF9E7hCRQ8CPw/fEReROEflN+HpE5Dsicrfz+KMiss35RrFbRP7WGV4APAVMFpEu52eyiHxBRH7hTPO0iNwesew6EbneeXymiDwnIm0iskNE3hftdYtIOTAd+IGq9js/f1HVl8NfY9j0S0Rkg1Pzr0XkoVATStj2+FcRaRaRgyLyLhG5VkR2OrV8NmxZOSLybRFpcn6+LSI5w6z3bBFZ76z3ISB3dO+wSRcW9CbhVPV1wA9c5Az6b+AMYDEwC6gC7gqbZSIwnuA3glsjFvcr4FoRGQcgIl7gfcADzvhm4J3AOOCjwLdEZImqdgMrgCZVLXR+miKW/QBwU+iJiMx1anjS+aB4zplmgjPd90VkXpSX3Ao0AL9wQrlyuG0jIj7gMeAnzmv+FfDuiMkmEgzh0Hb6AfBBYCnBbXqXiMxwpv0ccC7BbbsIWAZ8fpj1/hb4ubPeXwM3DFenSW8W9CZZmoDxIiLAJ4B/UtU2VT0G/BdwY9i0AeA/VLVPVXvDF6Kq+4D1wLucQZcDPar6mjP+SVXdpUF/Bp7lzQ+YkTwGLBaRqc7zm4FHVbWP4IfHXlX9saoOqup64BHgPZEL0WAHUpcBe4FvAAdF5EURmR1lnecCWcDdqjqgqo8Cr0dMMwB8RVUHgAeBcuA7qnpMVbcAW4CFYTV/UVWbVbUF+E/gQ8OsNxv4trPe3wBrRtxCJi1Z0JtkqQLagAogH1jnHKRsB552hoe0qOrxkywrfM/7A7y5N4+IrBCR15wmjXbgWoLBOCLnQ+dJ3vzQuRH4pfN4KrA8VLOz7JsJ7m1HW5ZfVW9X1ZnOvN3Az6JMOhk4oG/tXbAxYppWVR1yHoc++A6Hje8FCsOWty9s3D5nWCzr3RdlOpMBLOhNwonIOQSD/mXgCMFgmqeqJc5PsaoWhs0yUpeqvwYuFZFqgs0cDzjrySG4l/11oFJVS4BVgMS4XAg2ndwkIucBecCfnOGNwJ/Dai5xmn/+bqQFqmojcA8wP8rog0CV800npCaGOofTRPCDJWSKMyyW9U45jfWaFGZBbxJGRMaJyDsJNjf8QlU3qWqAYBvzt0RkgjNdlYhcHetynSaJF4AfA3tUdZszygfkAC3AoIisAK4Km/UwUCYixSdZ/CqCQflF4CGnXoAngDNE5EMiku38nCMiZ0V53aUi8p8iMktEPM7B2b8BXouyvleBIeB2EckSkZUE29VP1a+Az4tIhbPeu4BfDLPeQeBTznqvP831mhRmQW8S4fcicozgXvDngG8SPDAacgfBg5WviUgn8AdgtOd3PwBcQVizjdP08ingYeAowWadx8PGbycYhLud5pe3NWk47fGPDrPsqwg25zQBhwgeVM6JUls/MM15XZ3AZqAP+EiU9fUD1wMfA9oJHmR9wpn+VHwZWAvUA5sIHs9420VQYev9CMFt9X6Cr9tkILEbjxiTWkRkNXCvqv7Y7VpMZrA9emNcJiKXiMhEpwnlwwTPoHna7bpM5kjXqw6NySRzCDY3FQK7gPeo6kF3SzKZxJpujDEmw1nTjTHGZLiUbLopLy/XadOmuV2GMcakjXXr1h1R1Ypo41Iy6KdNm8batWvdLsMYY9KGiAx7ZbM13RhjTIazoDfGmAxnQW+MMRnOgt4YYzKcBb0xxmS4mIJeRK5xbp3WICJ3Rhl/s4jUOz+viMiisHF7RWSTiGwUETuVxhhjkmzE0yudW7XdA1xJ8HZwa0TkcVXdGjbZHuASVT3qdA17H7A8bPxlqnokjnUbY4yJUSzn0S8DGlR1N4CIPAisBE4Evaq+Ejb9a0B1PIs0xm37W3t4dushAFYsmERVSZ7LFRkTu1iCvoq33trMz1v31iN9DHgq7LkCz4qIAv+nqvdFm0lEbsW5EfSUKXajG5MaVJX7X9rDV5/ezlAg2C/U157ewV1/NZcPnjt1hLmNSQ2xBL1EGRa1JzQRuYxg0F8YNvgCVW1y7ib0nIhsV9UX37bA4AfAfQC1tbXW05pJCT/+y16+smob1y6YyOevm8tQQLnrd5v5/G83k+0V3n+O7ZSY1BfLwVg/b72HZTVR7kEpIguB+4GVqtoaGq6qTc7vZuAx7HZlJk1saergK6u2cfW8Sr530xIml+RRMz6fH9xSy0Wzy/n3321hx6FjbpdpzIhiCfo1wGwRmS4iPoK3Uns8fAIRmULwNmQfUtWdYcMLRKQo9Jjgrdg2x6t4YxJlKKB89tFNlOZn87UbFuHxvPnFNsvr4dvvX0y+z8u//3Yz1tW3SXUjBr2qDgK3A88A24CHVXWLiNwmIrc5k90FlAHfjziNshJ4WUTqgNeBJ1XV7pxjUt4T9U3U+Tv4/HVzKc7Pftv4ssIc/vXqM3l9bxt/3NbsQoXGxC4lbzxSW1ur1nulcUsgoFz97RcRgaf/8eK37M2HGxgKcPk3XqCsIIfH/v58RKJPZ0wyiMg6Va2NNs6ujDUmwnPbDvNGcxefvGzWsCEPkO318HeXzGJjYzuv7moddjpj3GZBb0yEX67ez6TiXK5bMGnEaW9YWkVpfja/WD1sV+DGuM6C3pgwB9p7eemNFt67tJos78j/HjlZXm5YUs2zWw7TcqwvCRUaM3oW9MaE+c1aP6rw3tqakSd23LhsCoMB5ZH1/gRWZsyps6A3xqGq/HpdIxfOKqdmfH7M882aUMjZU0p4fOPbLi8xJiVY0BvjqPd34D/ay8rFk0c973ULJrH1YCd7jnQnoDJjTo8FvTGOpzYfIssjXDm3ctTzXuscuF216WC8yzLmtFnQG0Ow2ebpzQc5b2YZJfm+Uc8/uSSPJVNKeKLegt6kHgt6Y4Adh4+xt7WHa+ZPPOVlXLtgEtsOdrKv1ZpvTGqxoDcGeHbLYUTgqrmnHvRXnBVs8nlhR0u8yjImLizojQH+vLOFBVXFVBTlnPIyppUXML28gD/tsL5vTGqxoDdjXkfvABsb27nkjIrTXtalcyp4dVcrvf1DcajMmPiwoDdj3isNRxgKKBfHIegvmzOBvsEAr+22vm9M6rCgN2Pen3e2UJSTxeKaktNe1rLp48nL9lrzjUkpFvRmTFNVXtzZwvmzysiOoW+bkeRme1k2fTyvWG+WJoVY0JsxbVdLF00dx+PSbBNy3swyGpq7aO48HrdlGnM6LOjNmBba875oVvyC/vyZZQC8au30JkVY0JsxbfXuNiYV51IzPi9uy5w3uZii3Cy7GYlJGRb0ZsxSVVbvaWP59PFxvQ2g1yMsn15me/QmZVjQmzFr95FujnT1sXxGWdyXfd7MMva19nCgvTfuyzZmtCzozZi1encbEDwlMt5OtNNb841JARb0Zsx6fU8r5YU5zCgviPuy51QWUZyXzdq9bXFftjGjZUFvxqQT7fMz4ts+H+LxCEumlLB239G4L9uY0bKgN2NSY1svBzuOszwBzTYhtdPG09DcRXtPf8LWYUwsLOjNmLR6T7DtfPn0+B+IDVk6tRSAdbZXb1xmQW/GpDp/O0U5WcyeUJiwdSyqLiHLI9Z8Y1xnQW/GpHp/B/OrivF44t8+H5Ln8zKvqph1ey3ojbss6M2Y0zc4xLaDnSysKU74umqnllLnb6d/MJDwdRkzHAt6M+ZsP3iMgSFlUXVJwtdVO7WUvsEAm5s6Er4uY4ZjQW/GnHp/OwALqxO/R790mnNA1ppvjIss6M2YU+fvoKzAR1VJ/DoyG86EomCHaev3W9Ab98QU9CJyjYjsEJEGEbkzyvibRaTe+XlFRBbFOq8xyVbvb2dhdXFCLpSKZnFNKXWN7UlZlzHRjBj0IuIF7gFWAHOBm0RkbsRke4BLVHUh8CXgvlHMa0zSdPcN0tDcxcIktM+HLKoupqnjuN2IxLgmlj36ZUCDqu5W1X7gQWBl+ASq+oqqhr6bvgZUxzqvMcm0+UAHAYVFSTjjJuTsKSVAsMnIGDfEEvRVQGPYc78zbDgfA54a7bwicquIrBWRtS0tLTGUZczo1Tthu6CqJGnrnDe5GK9H2Nho7fTGHbEEfbSGTI06ochlBIP+jtHOq6r3qWqtqtZWVMTvtm7GhKvztzO5OJeKopykrTM328uZE4uoa7Q9euOOWILeD9SEPa8GmiInEpGFwP3ASlVtHc28xiTLpgMdSW2fD1lUU0Kdv51AIOp+jjEJFUvQrwFmi8h0EfEBNwKPh08gIlOAR4EPqerO0cxrTLK09/Szr7UnKVfERlpcU8Kx44PsPtKd9HUbkzXSBKo6KCK3A88AXuBHqrpFRG5zxt8L3AWUAd93TlkbdJphos6boNdizEmF2ueTcUVspMU1wXXWNbYzK4EdqRkTzYhBD6Cqq4BVEcPuDXv8ceDjsc5rjBtCV8TOr0r+Hv3MikIKfF7q/O3csLR65BmMiSO7MtaMGXX+DmaUF1Ccl530dXs9wsLqErtwyrjCgt6MGaErYt2yqKaErQc7OT4w5FoNZmyyoDdjwuHO4xzu7HPljJuQxTXFDAwpWw92ulaDGZss6M2YEGoySeYVsZEWOQdkN9kVsibJLOjNmFDv78DrEeZOci/oJ47LpbzQx6YDFvQmuSzozZhQ529n9oRC8nxe12oQERZUFdsevUk6C3qT8VSVTQc6XDl/PtKCqmLeaD5Gb78dkDXJY0FvMl5jWy/tPQOuXBEbaUF1CQGFrQdtr94kjwW9yXh1zoVSqbJHD29epWtMMljQm4xX72/Hl+VhzsQit0uhclwOFUU5dkDWJJUFvcl4df4O5k4aR7bX/T93EWGhHZA1Seb+X74xCTQUUDYf6GCRi1fERppfVcyuli66+wbdLsWMERb0JqPtaumip3/I1StiIy2sLnYOyNoVsiY5LOhNRkuFK2IjhQ7IWvONSRYLepPR6v0dFOZkMaM8dfqAnzAul8pxdkDWJI8Fvclo9f525leNw+OJdvti9yyoKragN0ljQW8yVv9ggG0Hj6XE+fORFlSVsKuliy47IGuSwILeZKzthzrpHwqwIIXOuAlZWF2MKmyxvXqTBBb0JmPVuXiP2JGEbmdozTcmGSzoTcba5G+nND+b6tI8t0t5m4qiHCYV51rQm6SwoDcZq97fwcLqEkRS60BsyHw7IGuSxILeZKSe/kF2Hj6WUlfERlpYVczulm6OHR9wuxST4SzoTUba0tRJQEmpK2IjzXc+hLY02RWyJrEs6E1GCl0Rmwp90A/HrpA1yWJBbzJSvb+DScW5TCjKdbuUYZUX5lBVkke9tdObBLOgNxmp3t/OwhRunw+ZXzWOzRb0JsEs6E3G6egZYG9rT0q3z4csrC5hz5FuOu2ArEkgC3qTceoPtAOpeaFUpNCFU7ZXbxLJgt5knND9WEMHO1OZHZA1yWBBbzJOXWM708ryKc7PdruUEY0v8FFVkmcXTpmEiinoReQaEdkhIg0icmeU8WeKyKsi0icin4kYt1dENonIRhFZG6/CjRnOpgMdadE+H7Kw2q6QNYk1YtCLiBe4B1gBzAVuEpG5EZO1AZ8Cvj7MYi5T1cWqWns6xRozkuZjxznYcTwtzrgJWVBdzL7WHjp67ICsSYxY9uiXAQ2qultV+4EHgZXhE6hqs6quAewv1biqvtHpsbKmxN1CRiHUTr+5yfbqTWLEEvRVQGPYc78zLFYKPCsi60Tk1uEmEpFbRWStiKxtaWkZxeKNeVO9vx2PwLzJ49wuJWahoK/zt7tbiMlYsQR9tK7/dBTruEBVlxBs+vmkiFwcbSJVvU9Va1W1tqKiYhSLN+ZNdf4OzqgsIt+X5XYpMSvJ9zG1LP/EtxFj4i2WoPcDNWHPq4GmWFegqk3O72bgMYJNQcbEnaqmzRWxkRZVl1Bve/QmQWIJ+jXAbBGZLiI+4Ebg8VgWLiIFIlIUegxcBWw+1WKNORn/0V6O9gyk1Rk3IQuri2nqOE7zseNul2Iy0Ijfb1V1UERuB54BvMCPVHWLiNzmjL9XRCYCa4FxQEBEPk3wDJ1y4DHnxg9ZwAOq+nRCXokZ80Jt3OlwRWykxc7B4/rGDq6Ym7odsZn0FFNDpqquAlZFDLs37PEhgk06kTqBRadToDGxqvd34MvyMGdikduljNq8ycV4PUKdv50r5la6XY7JMHZlrMkYGxvbmTd5HL6s9PuzzvN5OaOy6MQNzY2Jp/T7jzAmisGhAJv8HWnZbBOyqLqYen87qqM5qc2YkVnQm4zQ0NJF78DQibbudLSopoT2ngH2t/W4XYrJMBb0JiOEbh2YTlfERgqdFrrReS3GxIsFvckIGxs7GJebxbSyfLdLOWVnVBaRm+050c2yMfFiQW8ywsbGdhbVlOCcypuWsr0e5k0uPvHtxJh4saA3aa+nf5Cdh4+ldft8yKLqEjY3dTA4FHC7FJNBLOhN2tvS1MlQQNP6jJuQRTXFHB8IsPNwl9ulmAxiQW/SXqipY2FN+vVxEyn0YWX93ph4sqA3aW9jYztVJXlMKEr/rgOmluVTnJdtXRabuLKgN2mvzt/OogzYmwcQERZWF1NnXRabOLKgN2mttauPxrbejGifD1lUXcKOw8fo7R9yuxSTISzoTVoLnXOezhdKRVpUU8JQQNl60PbqTXxY0Ju0trExeOvA0O34MkGoGWrD/nZ3CzEZw4LepLWNje3MmlBIQU763DpwJBOKcqkuzbOgN3FjQW/SViCgbNh/lCVTSt0uJe6WTi1l7b4268nSxIUFvUlbu1q66Dw+yJKpmRf0S6aUcrizj6YOu7WgOX0W9CZtrd9/FAju/Waa0Gtat++oy5WYTGBBb9LWun1HKcnPZkZ5gdulxN2ZE4vIy/ay3oLexIEFvUlb6/YdZemU0rTusXI4WV4Pi2qKT3xrMeZ0WNCbtHS0u59dLd0Z2T4fsnRqKVubOu3CKXPaLOhNWtrQmLnt8yFLppQyGFDr4MycNgt6k5bW7TuK1yMZ1fVBpLOd00bXWfONOU0W9CYtrdt3lHmTx5Hn87pdSsKML/Axo6LADsia02ZBb9LOwFCAusaOjLxQKtKSKaWs399uF06Z02JBb9LO9oPH6B0Yyuj2+ZClU0tp6+5nb2uP26WYNGZBb9LOun1tQGYfiA0Jvca1e9tcrsSkMwt6k3Ze39vG5OJcJpfkuV1Kws2qKKQ4L5s1FvTmNFjQm7Siqry+p43lM8rcLiUpPB7hnGnjWb3Hgt6cOgt6k1Z2tXRzpKuf5dPHu11K0pw7Yzz7Wns4ZB2cmVNkQW/SyuvOnu2yMRT0y6cHv72s3tPqciUmXcUU9CJyjYjsEJEGEbkzyvgzReRVEekTkc+MZl5jRmP1nlYqinKYnoEdmQ1n7uRxFOVk8dpua74xp2bEoBcRL3APsAKYC9wkInMjJmsDPgV8/RTmNSYmqsrq3W0snz4+IzsyG47XI9ROK7U9enPKYtmjXwY0qOpuVe0HHgRWhk+gqs2qugYYGO28xsSqsa2XQ53Hx1T7fMjyGWXsbumm+Zi105vRiyXoq4DGsOd+Z1gsYp5XRG4VkbUisralpSXGxZux5DVnj3asnHETLvTh9rqdfWNOQSxBH+07cqzXY8c8r6rep6q1qlpbUVER4+LNWPL6njZK87OZVVHodilJN7+qmHyf14LenJJYgt4P1IQ9rwaaYlz+6cxrzFus3tPKsunj8XjGTvt8SLbXw9Kppay2A7LmFMQS9GuA2SIyXUR8wI3A4zEu/3TmNeaE/a09NLb1cv7McrdLcc25M8rYcfgYbd39bpdi0syIQa+qg8DtwDPANuBhVd0iIreJyG0AIjJRRPzAPwOfFxG/iIwbbt5EvRiTuV5uOALABbPGdtADvLrLzr4xo5MVy0SqugpYFTHs3rDHhwg2y8Q0rzGj9ZeGI0wcl8vMirFz/nykRdXFFOVm8XJDC9ctnOR2OSaN2JWxJuUFAspfdh3hglnlY+r8+UhZXg/nzSjjxZ1HrH96MyoW9CblbT3YSXvPABfNHrvNNiEXzS7nQHuv9U9vRsWC3qS8l94Its+fP2vsnT8f6cLZwVOPX37DrjUxsbOgNynvLw1HmFNZxISiXLdLcd20snyqSvJOfPgZEwsLepPSjg8M8fretjF9tk04EeGi2eW8uquVwaGA2+WYNGFBb1La2r1H6R8McOFsa7YJuXB2Ocf6Bqnzd7hdikkTFvQmpb2woxmf13OiT3YDF8wsRwRetuYbEyMLepPSnt/RzPIZ4ynIiemSjzGhtMDHwqpiXtjZ7HYpJk1Y0JuUta+1m90t3Vx+5gS3S0k5l59ZycbGdo509bldikkDFvQmZT2/PbjHakH/du84awKq8MIOO83SjMyC3qSs57c3M6OigKllY7fbg+HMmzyOynE5PL/9sNulmDRgQW9SUnffIKt3t3H5HNubj0ZEuPzMSl7ceYT+QTvN0pycBb1JSX9pOEL/UMCabU7iHWdOoKtvkDV7rY96c3IW9CYl/WlHM4U5WdROG3v3h43VBbPKycny8Idt1nxjTs6C3qScQED5w7ZmLj6jHF+W/YkOJ8/n5fyZZfxxW7P1ZmlOyv6LTMpZt/8oLcf6uGa+9bk+knecVcn+th52HD7mdikmhVnQm5Tz1KZD+LI81j4fg6vnTcQjsGrTIbdLMSnMgt6kFFXl6c0HuXh2OYV2NeyIKopyWDZ9PE9tOuh2KSaFWdCblFLn76Cp47g124zCdQsm8UZzF29Y840ZhgW9SSlPbT5Ilke48qxKt0tJG1fPn4gIPGl79WYYFvQmZQSbbQ5x3swyivOz3S4nbUwoyuWcaeN5ytrpzTAs6E3KqPN3sK+1h3cutGab0bpuwSR2HD5GQ3OX26WYFGRBb1LGbzccwJflYcUCC/rRusZpvnmivsntUkwKsqA3KWFwKMAT9U1ccdYExuVas81oVY7L5bwZZTy24YBdPGXexoLepISXG45wpKuflYur3C4lbd2wpJp9rT2s23fU7VJMirGgNynhtxsOUJyXzaVzKtwuJW1dM38iedleHll/wO1STIqxoDeu6+4b5Nmth7l2wSRysrxul5O2CnKyWDF/Ik/UN3F8YMjtckwKsaA3rvt9XRM9/UO8Z2m126WkveuXVHPs+KD1aGnewoLeuO5Xr+9nTmURS6aUuF1K2jtvZhmTinN5aE2j26WYFGJBb1y1+UAHdf4OblpWg4i4XU7a83qE959Tw0tvHGFfa7fb5ZgUEVPQi8g1IrJDRBpE5M4o40VE7nbG14vIkrBxe0Vkk4hsFJG18SzepL8H1+wnJ8vDu8+2Zpt4ufGcKXg9wgOr97tdikkRIwa9iHiBe4AVwFzgJhGZGzHZCmC283Mr8L8R4y9T1cWqWnv6JZtM0dM/yO82NHHdgknW5UEcTSzO5cqzKnl4bSN9g3ZQ1sS2R78MaFDV3araDzwIrIyYZiXwMw16DSgREbu80ZzUo+sPcKxvkA8sn+J2KRnng+dO5WjPgPV/Y4DYgr4KCD+y43eGxTqNAs+KyDoRuXW4lYjIrSKyVkTWtrS0xFCWSWeBgPKjl/ewsLqYpVNL3S4n45w/s4zp5QX89NW9bpdiUkAsQR/tCFnkNdYnm+YCVV1CsHnnkyJycbSVqOp9qlqrqrUVFXbRTKZ7YWczu49087ELp9tB2ATweISPnD+NDfvbWbu3ze1yjMtiCXo/UBP2vBqI7Dlp2GlUNfS7GXiMYFOQGeN++PIeJo7L5VrrwCxh3ltbTUl+Nv/34m63SzEuiyXo1wCzRWS6iPiAG4HHI6Z5HLjFOfvmXKBDVQ+KSIGIFAGISAFwFbA5jvWbNLS1qZO/NLTy4fOnke21M3wTJd+XxS3nTuUP2w6zq8W6Lx7LRvwvU9VB4HbgGWAb8LCqbhGR20TkNmeyVcBuoAH4AfD3zvBK4GURqQNeB55U1afj/BpMmrnnTw0U5mTxgWV2EDbRbjl/Gj6vh/tfsr36sSymuy+r6iqCYR4+7N6wxwp8Msp8u4FFp1mjySA7Dx9j1eaD/P2lM+2UyiQoL8zhPUur+fVaP7dfPpuqkjy3SzIusO/NJqm++3wD+dlePn7hDLdLGTP+/rJZAHzv+TdcrsS4xYLeJE1D8zGeqG/ilvOnUVrgc7ucMaOqJI8bl9Xw67V+9rf2uF2OcYEFvUmarz+zk/xsL5+4yPbmk+2Tl83C6xG+80fbqx+LLOhNUqzZ28bTWw5x2yUzGW9780lXOS6XD507lcc2+Nl+qNPtckySWdCbhAsElC8/uY3KcTl83PbmXfPJy2ZRlJvNF3+/1e4rO8ZY0JuE+319E3WN7Xzmqjnk+ewOUm4pLfDxz1eewSu7Wnl2q92YZCyxoDcJ1Xl8gK88uY15k8dx/RLrithtNy+fwuwJhXzlyW12u8ExxILeJNTXn9nBka4+/uvdC/B6rE8bt2V5PfzHX81jf1sP33u+we1yTJJY0JuE2bD/KD9/bR+3nDeNRTUlbpdjHBfOLueGJdXc++ddbGnqcLsckwQW9CYhjg8Mcccj9VQW5fIvV53hdjkmwr+/8yxK8rO545F6BocCbpdjEsyC3iTEV5/azs7DXXztPQspyrWuDlJNSb6PL66cz+YDndxtTTgZz4LexN0LO5r5ySt7+egF07j4DLu3QKq6dsEkrl9Sxfeef4NXd7W6XY5JIAt6E1dN7b38y8N1zKks4o5rznS7HDOCL62cz9SyAj790AbauvvdLsckiAW9iZvjA0P87c/X0TcY4J6bzyY3286ZT3UFOVl896azOdo9wCd/uZ4Ba6/PSBb0Ji5Ulc8+tolNBzr41vsXM2tCkdslmRjNryrmqzcs4NXdrXzh8S121WwGiqk/emNG8q3ndvLo+gN8+orZXDm30u1yzChdv6SanYe7uPfPu5heXmBdVWQYC3pz2n76yl7ufr6B99fW8I/vmO12OeYU/b+r57CvtZsvP7mNotws3n+O3QEsU1jTjTktD6zezxd+v4Ur51bylXfPR8Sufk1XXo/w7RsXc8kZFdz56CZ+t/GA2yWZOLGgN6fsRy/v4bOPbeLSMyr47k1nk2U3+k57OVle7v3gUs6ZNp5/emgjD63Z73ZJJg7sP9OMWiCgfPO5nXzxia1cM28i//ehWjvDJoPk+bz8+CPncOHsCu54ZBP3/KnBDtCmOQt6Myo9/YN88oH13P3HN3jv0mq+94Gz8WXZn1GmKcjJ4v5bannX4sn8zzM7uPORTdbbZRqzg7EmZg3NXfzDrzaw41Ann7v2LD5+0XRrk89gviwP33zfYqaMz+fu5xvYerCT79+8hJrx+W6XZkbJdsXMiFSVX67exzu/+xKHOnr54UfO4RMXz7CQHwM8HuGfr5rD/bfUsre1m2vvfonfrPNbU06asaA3J7WrpYsP/nA1n3tsM+dMG8/Tn76Yy+ZMcLssk2RXzK3kiX+4kDMnFvGZX9fx8Z+uxX+0x+2yTIwkFT+Za2trde3atW6XMaZ1Hh/g3hd28YOXdpOb7eVfr57Dzcun4rGbh4xpQwHlJ6/s5X+e2Y4q3HrxDG67ZCYFOdYK7DYRWaeqtVHHWdCbcN19g/zklb3c9+JuOnoHuH5JFf+24iwqinLcLs2kkAPtvXzt6e38bmMTFUU5/O3FM/jA8ink+yzw3WJBb0bU2NbDL17bx4NrGunoHeDyMyfwT1ecwYLqYrdLMyls3b6jfOPZHbyyq5XS/Gw+dN40blpWw6TiPLdLG3Ms6E1UPf2D/GFbM7/bcIDndzTjEeHqeZV84qIZnD2l1O3yTBpZt+8o9/ypgee3N+MRuPzMCby3toZLzqiwayySxILenHCkq4+X3zjCH7c384eth+kdGGLiuFzes7Sam8+dYnti5rTsb+3hwTX7eXitnyNdfRT4vFx+ViUr5k/kgpnlFOfb3cYSxYJ+DDvUcZyNje1saDzKy28cYUtTJwDjC3ysmD+Rv140mXOmjbeDrCauBoYCvLqrlac2H+LZLYdo7e7HI8Eukc+bWca508tYWF1MWaEd+4kXC/ox4PjAELtaumho7mJXcxc7Dh+jrrGDQ53HAcj2CmfXlHLxGeVcfEYF8ycXW7ibpBgcCrChsZ2/NBzhlYZWNjQeZWAomDtVJXksrC5m7qRxzJxQyMyKQqaW5Vtzzyk47aAXkWuA7wBe4H5V/WrEeHHGXwv0AB9R1fWxzBuNBf2bVJXu/iGOdvdztKeflmN9NHUcp6m9l4PtvTR1HOfA0V6aOnoJvZUegSnj81lYXcLimhIWTylh7qRx9s9jUkJP/yD1/g42+Tuo87dT7+9gf9ub5+SLQHVpHtUl+UwqzmVSSS4Ti/OYXJxL5bhcSvKzGV/gIy/baxfthTlZ0I94LpSIeIF7gCsBP7BGRB5X1a1hk60AZjs/y4H/BZbHOG/aUVUCGvx6OhhQBgYDDAwFGHAeDwYC9A+qMz74eDAQnOb4QICe/iF6+gfp7huit3+Q7v6htwzr6hvgaPcAR3uC4R7a+wmX5REmFucyuTiP2mmlTC+vZtaEQmZNKGRaWYGFuklZ+b4szp1Rxrkzyk4M6+kfZHdLN7uPdLOruYs9R7ppau9l9Z42DnUeZyjw9v8BX5aH0vxsSvN9lOb7KMjJoiDHS74vi0Lnd0HY75wsLz6vh+wsDz6vB1+W4PN68WV5yPYKvhPDPXg8glcEr0fwiOCRYDfO6frBEstJr8uABlXdDSAiDwIrgfCwXgn8TINfD14TkRIRmQRMi2HeuHnnd1+it38IVVAgoEpANfhcOfE44AQ1BH+HDw9OGzaciOcJaOkq8HnJc/4Y87K9FOVmMbUsn7OnlFCS72N8QXbwd76PskIfk0vyKC/MwWtNLyZD5PuymF9VzPyqt5/OOxRQ55tsLy3H+mjv6aete8D53c/RnuDjA+29zs6SsxOVgE7YRMAjwQ8Bj4fgb5HgB0PYh4JHBBEQOPHhIOL88NZxAuA8LyvI4eHbzot73bEEfRXQGPbcT3CvfaRpqmKcFwARuRW4FWDKlFO7s82sikIGAorAiQ0uzgb3OBvU47xB8NY35M03xhnukRNvhEcIm+bN5WR5BZ/XQ5ZXyPZ63vI4+8RvT9h0HnKyPBT4ssjP8ZLv85Kb5bW2cmNOwut8e51YnDuq+YYCSu/AED19g3T1DdI/FKDf+fbdNxhgYEhPPO8fdH6cxwFVhgLKkLNzNxQIPg/tPA4FnB1GZ5oTvzXYjfeJnUZwfgefhHYc3xz+5nMUxuUl5oKzWJYaLYUi92uHmyaWeYMDVe8D7oNgG30Mdb3Nt288+1RmM8ZkIK9HKMzJojAni7HeO1MsQe8HasKeVwNNMU7ji2FeY4wxCRRL75VrgNkiMl1EfMCNwOMR0zwO3CJB5wIdqnowxnmNMcYk0Ih79Ko6KCK3A88QPEXyR6q6RURuc8bfC6wieGplA8HTKz96snkT8kqMMcZEZRdMGWNMBjjZefR24xFjjMlwFvTGGJPhLOiNMSbDWdAbY0yGS8mDsSLSAuw7xdnLgSNxLCderK7RsbpGx+oanUysa6qqVkQbkZJBfzpEZO1wR57dZHWNjtU1OlbX6Iy1uqzpxhhjMpwFvTHGZLhMDPr73C5gGFbX6Fhdo2N1jc6Yqivj2uiNMca8VSbu0RtjjAljQW+MMRkuLYNeRN4rIltEJCAitRHj/k1EGkRkh4hcPcz840XkORF5w/ldmoAaHxKRjc7PXhHZOMx0e0VkkzNdwntyE5EviMiBsNquHWa6a5xt2CAidyahrv8Rke0iUi8ij4lIyTDTJWV7jfT6nS6573bG14vIkkTVErbOGhH5k4hsc/7+/zHKNJeKSEfY+3tXouty1nvS98Wl7TUnbDtsFJFOEfl0xDRJ2V4i8iMRaRaRzWHDYsqhuPwvqmra/QBnAXOAF4DasOFzgTogB5gO7AK8Ueb/GnCn8/hO4L8TXO83gLuGGbcXKE/itvsC8JkRpvE6224GwZvH1AFzE1zXVUCW8/i/h3tPkrG9Ynn9BLvlforgXdTOBVYn4b2bBCxxHhcBO6PUdSnwRLL+nmJ9X9zYXlHe00MELypK+vYCLgaWAJvDho2YQ/H6X0zLPXpV3aaqO6KMWgk8qKp9qrqHYP/4y4aZ7qfO458C70pIoQT3ZID3Ab9K1DoS4MQN4VW1Hwjd1D1hVPVZVR10nr5G8G5kbonl9a8EfqZBrwElIjIpkUWp6kFVXe88PgZsI3hf5nSQ9O0V4R3ALlU91SvuT4uqvgi0RQyOJYfi8r+YlkF/EsPdpDxSpQbvgIXzO5G3lLwIOKyqbwwzXoFnRWSdBG+Qngy3O1+ffzTM18VYt2Oi/A3Bvb9okrG9Ynn9rm4jEZkGnA2sjjL6PBGpE5GnRGRekkoa6X1x+2/qRobf2XJje0FsORSX7ZaYW47HgYj8AZgYZdTnVPV3w80WZVjCzh+NscabOPne/AWq2iQiE4DnRGS78+mfkLqA/wW+RHC7fIlgs9LfRC4iyrynvR1j2V4i8jlgEPjlMIuJ+/aKVmqUYZGvP6l/a29ZsUgh8AjwaVXtjBi9nmDzRJdz/OW3wOwklDXS++Lm9vIBfw38W5TRbm2vWMVlu6Vs0KvqFacwWyw3Mgc4LCKTVPWg8/WxORE1ikgWcD2w9CTLaHJ+N4vIYwS/qp1WcMW67UTkB8ATUUbFuh3jWpeIfBh4J/AOdRoooywj7tsrilhef0K20UhEJJtgyP9SVR+NHB8e/Kq6SkS+LyLlqprQDrxieF9c2V6OFcB6VT0cOcKt7eWIJYfist0yrenmceBGEckRkekEP5lfH2a6DzuPPwwM9w3hdF0BbFdVf7SRIlIgIkWhxwQPSG6ONm28RLSLvnuY9SX9pu4icg1wB/DXqtozzDTJ2l6xvP7HgVucs0nOBTpCX8MTxTne80Ngm6p+c5hpJjrTISLLCP6Ptya4rljel6RvrzDDfqt2Y3uFiSWH4vO/mOijzYn4IRhQfqAPOAw8EzbucwSPUu8AVoQNvx/nDB2gDPgj8Ibze3yC6vwJcFvEsMnAKufxDIJH0euALQSbMBK97X4ObALqnT+YSZF1Oc+vJXhWx64k1dVAsC1yo/Nzr5vbK9rrB24LvZ8Ev1Lf44zfRNjZXwms6UKCX9vrw7bTtRF13e5smzqCB7XPT0JdUd8Xt7eXs958gsFdHDYs6duL4AfNQWDAya6PDZdDifhftC4QjDEmw2Va040xxpgIFvTGGJPhLOiNMSbDWdAbY0yGs6A3xpgMZ0FvjDEZzoLeGGMy3P8Hvk1P09MV79QAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-10,10,1000)\n",
    "S = Sigmoid.forward(z)\n",
    "dS_dz = Sigmoid.gradient(z)\n",
    "\n",
    "plt.plot(z, S)\n",
    "plt.title('Sigmoid')\n",
    "plt.show()\n",
    "\n",
    "plt.plot(z, dS_dz)\n",
    "plt.title('Derivative Sigmoid')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another very popular activation function, the Rectified Linear Unit (ReLU), is defined as \n",
    "\n",
    "\\begin{align}\n",
    "ReLU(z) = \\begin{cases}\n",
    "               0               & z<0\\\\\n",
    "               z               & z\\geq 0\\\\ \\end{cases}\n",
    "\\end{align}\n",
    "\n",
    "and its (sub-)derivative reads:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{d ReLU}{dz}(z) = \\begin{cases}\n",
    "               0               & z<0\\\\\n",
    "               1               & z\\geq 0\\\\ \\end{cases}\n",
    "\\end{align}\n",
    "\n",
    "Implement this function and its derivative below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReLU:\n",
    "    @staticmethod\n",
    "    def forward(z):\n",
    "        return z * (z > 0)\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(z):\n",
    "        return 1. * (z > 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you are done, you can test your implementation by running the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdr0lEQVR4nO3deXhU5fnG8e9jWGRfJCB7QBbFBQgRAXexVXFBbVUUWq3+tCCouBZr1bbWWltrXYpY21oXIouCShX3tW5UEkJYwr5vSVjCDtne3x8z9ooxkUnmzJw5k/tzXbkymXMy55kzmXvevGfmOeacQ0REgucwvwsQEZHaUYCLiASUAlxEJKAU4CIiAaUAFxEJKAW4iEhAKcBFRAJKAS5Jx8zWmNl+M9tjZlvM7DkzaxrB731sZv9Xze2dXem6a8zsMy/rFqkpBbgkqwudc02BfkB/4G5/yxHxngJckppzbgvwDqEgx8wGmdkXZlZkZvPN7AwfyxOJigJckpqZdQLOA1aYWUfgTeB3QGvgDmCGmaX6WKJIrSnAJVm9Zma7gfVAAXA/MAqY7Zyb7Zwrd869B8wFhvlYp0itKcAlWV3snGsGnAEcDbQBugKXhadPisysCDgFaH+I2yoF6le6rj5Q4mnFIjVUz+8CRGLJOfeJmT0HPALMAV50zl1fw5tZB6RVuq4bsDbqAkWioBG41AWPAT8APgMuNLNzzCzFzA43szPC8+TfqBe+/puv+sA0YLyZHW0hGcC1wNS43xORChTgkvScc4XAC8B4YDjwS6CQ0Pz4nXz7eTAJ2F/h61/A38Pf/w3sDN/WPc65t+NzD0SqZjqhg4hIMGkELiISUApwEZGAUoCLiASUAlxEJKDi+j7wNm3auLS0tHhuUkQk8LKysrY6577T8iGuAZ6WlsbcuXPjuUkRkcAzsyo/NKYpFBGRgFKAi4gElAJcRCSgFOAiIgGlABcRCahDBriZPWtmBWa2sMJ1rc3sPTNbHv7eKrZliohIZZGMwJ8Dzq103QTgA+dcT+CD8M8iIhJHhwxw59ynwPZKVw8Hng9ffh642NuyRESSw4GSMn49axHb9xZ7ftu1nQNv55zbDBD+3ra6Fc3sBjOba2ZzCwsLa7k5EZFguve1hTz/5RoWbtzp+W3H/CCmc+4Z51yGcy4jNVUn/xaRumPa1+t4OWsDN53Zg9N6eZ9/tQ3wfDNrDxD+XuBdSSIiwbdw407ufX0Rp/Zswy1n94rJNmob4LOAq8OXrwZe96YcEZHg27m/hBszszmiSQMeu6IfKYdZTLYTydsIpwBfAr3NbIOZXQf8AfiBmS0ndLLYP8SkOhGRgCkvd9w+fT6bivbz16vSOaJpw5ht65DdCJ1zV1azaKjHtYiIBN7fPl3F+3n53H9hHwZ0je1HZPRJTBERj3y5cht/emcJ55/QnmuGpMV8ewpwEREP5O86wE1T5tGtTRMe/tEJmMVm3ruiuJ7QQUQkGZWUlTPupWz2HizlpetPomnD+ESrAlxEJEp/fHsJX6/ZweMj+tGrXbO4bVdTKCIiUXh74Wb+/p/V/HRwV4b36xjXbSvARURqafXWvdz5ci59O7fknvOPifv2FeAiIrWwv7iMMZOzqJdiPDUynYb1UuJeg+bARURqyDnHr15byNL83Tz3s4F0bNnIlzo0AhcRqaGpX69nRvYGbj6rJ6fHoElVpBTgIiI1sHDjTu6fFWpSdfPQnr7WogAXEYnQzn0ljJ6cRZsmDXh8RP+YNamKlObARUQiUF7uuG16Dvm7DjD954Np3aSB3yVpBC4iEolJn6zkgyUF/Or8PvTvkhjncVeAi4gcwucrtvLnd5dyYd8O/HRwV7/L+R8FuIjI99iy8wA3T5lH99Sm/OHS4+PSpCpSmgMXEanGN02q9peUMW1UOk3i1KQqUolVjYhIAvnDW0uYu3YHT17Znx5t49ekKlKaQhERqcLsBZv552eruWZIGhf27eB3OVVSgIuIVLKqcA93vZJL/y4t+eWw+DepipQCXESkgn3FpYyZnE2Deocx8ap0GtRL3JjUHLiISJhzjl+9upBlBbt54dqBdPCpSVWkEvelRUQkzl767zpmztvI+KG9OLWnf02qIqUAFxEBcjcU8ZtZizm9Vyo3ndXD73IiogAXkTqvaF8xYyZnk9qsIY9d0Y/DfG5SFSnNgYtInVZe7rh1Wg4Fuw/w8ughtEqAJlWR0ghcROq0iR+t4KOlhdx3QR/6dW7pdzk1ogAXkTrrs+VbefT9ZQzv14FRgxKnSVWkFOAiUidt3rmfm6fOo0dqUx5KsCZVkVKAi0idU1xaztjMbA6WlDFp1AAaNwjm4cBgVi0iEoWH3soje10RE69Kp0fbpn6XU2sagYtInfJG7ib+9fkafnZyGuef0N7vcqISVYCb2a1mtsjMFprZFDM73KvCRES8tqJgD794JZf0Li25+7zEbVIVqVoHuJl1BG4GMpxzxwEpwAivChMR8dK+4lJuzMyiYf0UJo5M7CZVkYr2HtQDGplZPaAxsCn6kkREvOWc45czF7C8YA9PjOhP+xaJ3aQqUrUOcOfcRuARYB2wGdjpnHu38npmdoOZzTWzuYWFhbWvVESklibPWcdrOZu47exenNKzjd/leCaaKZRWwHCgG9ABaGJmoyqv55x7xjmX4ZzLSE1N/O5eIpJc5q8v4oF/L+bM3qmMPTMYTaoiFc0UytnAaudcoXOuBJgJDPGmLBGR6O3YW8yNmaEmVX8JUJOqSEUT4OuAQWbW2EIfYRoK5HlTlohIdMrLHeOn5VC4+yCTRqXTsnFwmlRFKpo58DnAK0A2sCB8W894VJeISFSe/HAFnywr5L4L+3BCp5Z+lxMTUX0S0zl3P3C/R7WIiHji02WFPPbBMi7p35GRJ3Xxu5yYCf4bIUVEKthUtJ9bps6jV9tmPHjJcYFsUhUpBbiIJI3i0nJuzMympMwxaVR6YJtURSq5752I1Cm/n51HzvoinhqZTvfU4DapipRG4CKSFGbN38RzX6zhulO6Mez4YDepipQCXEQCb0XBbibMyCWjaysmnHe03+XEjQJcRAJt78FSRk/OpnGDFP56VTr1U+pOrGkOXEQCyznH3TMXsKpwD5OvO4kjW9StjtZ156VKRJLOi1+tZdb8Tdz+w94M6ZE8TaoipQAXkUDKXreDB95YzNCj2zLm9KP8LscXCnARCZzte4sZl5lNu+aH8+jlydekKlKaAxeRQCkrd9wydR5b9xQzY8wQWjSu73dJvlGAi0igPPHBcv6zfCu/v+R4ju/Uwu9yfKUpFBEJjI+XFvDEh8v5UXonrhzY2e9yfKcAF5FA2Fi0n/HTcujdrhm/uzi5m1RFSgEuIgnvYGkZN2ZmU1bmmDRqAI0apPhdUkLQHLiIJLwH38xj/voinh6VTrc2TfwuJ2FoBC4iCe31nI288OVarj+1G+ceVzeaVEVKAS4iCWt5/m4mzFjAiWmtuOvcutOkKlIKcBFJSHsOljJ6chZNGtarc02qIqU9IiIJxznHhBm5rN66lyev7E+75nWrSVWkFOAiknCe+2INb+Ru5o5zejP4qCP8LidhKcBFJKFkrd3Bg2/mcfYxbRl9Wt1sUhUpBbiIJIxtew4y7qVs2rc8nD9fVnebVEVK7wMXkYQQalKVw7a9xcys402qIqURuIgkhMffX8ZnK7bywPBjOa5j3W5SFSkFuIj47qOlBTzx4QouG9CJK07s4nc5gaEAFxFfbdixj1un5XBM++Y8cPFxfpcTKApwEfHNt5pUjUzn8PpqUlUTOogpIr554I3F5G7Yyd9+MoA0NamqMY3ARcQXr83byOSv1vHz07pzzrFH+l1OIEUV4GbW0sxeMbMlZpZnZoO9KkxEktfSLbu5e+YCBnZrzZ3n9Pa7nMCKdgrlceBt59yPzawB0NiDmkQkie0+UMKYb5pUXdmfempSVWu1DnAzaw6cBlwD4JwrBoq9KUtEkpFzjl/MyGXt9n1k/t9JtFWTqqhE89LXHSgE/mVm88zsH2b2naMQZnaDmc01s7mFhYVRbE5Egu7Zz9cwe8EW7jynN4O6q0lVtKIJ8HpAOjDJOdcf2AtMqLySc+4Z51yGcy4jNTU1is2JSJDNXbOdh2bn8YM+7fj5ad39LicpRBPgG4ANzrk54Z9fIRToIiLfsnXPQca+lE3HVo145LK+OqO8R2od4M65LcB6M/vmEPJQYLEnVYlI0gg1qZpH0b4SnhqZTotGalLllWjfhXITkBl+B8oq4GfRlyQiyeQv7y3j8xXb+OOPT+DYDmpS5aWoAtw5lwNkeFOKiCSbD5fk89ePVnBFRmcuz+jsdzlJR2/AFJGYWL99H7dOm0+f9s35zfBj/S4nKSnARcRzB0pCTarKnePpUQPUpCpG1MxKRDz32zcWs2DjTv7+0wy6HKEPaMeKRuAi4qkZWRt4ac46Rp9+FD/o087vcpKaAlxEPLNkyy7ueW0Bg7q35o4f9vK7nKSnABcRT+w6UMKYydk0P7w+T6hJVVxoDlxEouac466Xc1m3fR9Trh9E22ZqUhUPeokUkaj987PVvL1oC784tzcDu7X2u5w6QwEuIlH5es12HnprCecc247rT1WTqnhSgItIrRXuPsjYzGw6t2rEn9SkKu40By4itVJaVs7NU+ax60AJz187kOaHq0lVvCnARaRWHn1vGV+u2sYjl/XlmPbN/S6nTtIUiojU2PuL83nq45VcObAzPx7Qye9y6iwFuIjUyLpt+7hteg7HdWzO/ReqSZWfFOAiErEDJWWMycwCYNJINanym+bARSRiv561iEWbdvHPqzPo3FpNqvymEbiIROTlueuZ+vV6bjzjKIYeoyZViUABLiKHtHjTLn712kIGdz+C236gJlWJQgEuIt9r14ESbszMokUjNalKNJoDF5FqOee4Y/p81u/Yz9QbBpHarKHfJUkFeikVkWr9/T+reHdxPnefdzQnpqlJVaJRgItIleas2sbDby/lvOOO5LpTuvldjlRBAS4i31Gw+wDjpsyjS+vG/PHHJ6hJVYLSHLiIfEtpWTk3vTSP3QdKePG6gTRTk6qEpQAXkW955N1lzFm9nUcv78vRR6pJVSLTFIqI/M97i/N5+pOVXHVSFy5NV5OqRKcAFxEA1m7by23Tczi+Ywvuu6CP3+VIBBTgIsKBkjJGT87mMDOeGpmuJlUBoTlwEeG+1xeSt3kXz16jJlVBohG4SB03/ev1TJ+7gXFn9uCso9WkKkgU4CJ12KJNO7n39YWc3OMIblWTqsCJOsDNLMXM5pnZG14UJCLxsXN/CWMmZ9OqcQMeH9GflMP0YZ2g8WIEfguQ58HtiEicOOe44+X5bCraz8SR/WnTVE2qgiiqADezTsD5wD+8KUdE4uFvn67ivcX53D3sGAZ0VZOqoIp2BP4YcBdQXt0KZnaDmc01s7mFhYVRbk5EovXVqm388e0lnH98e649Oc3vciQKtQ5wM7sAKHDOZX3fes65Z5xzGc65jNTU1NpuTkQ8ULDrAONemkdamyb84UfHq0lVwEUzAj8ZuMjM1gBTgbPMbLInVYmI50rLyhk3ZR57D5by9KgBalKVBGod4M65u51znZxzacAI4EPn3CjPKhMRT/3pnaX8d/V2Hrr0eHq1a+Z3OeIBvQ9cpA54e+EW/vbpKkYN6sLF/Tv6XY54xJOP0jvnPgY+9uK2RMRbq7fu5c6X59O3UwvuVZOqpKIRuEgS219cxpjJWaSkGBNHptOwnppUJRM1sxJJUs457n19IUvzd/PsNSfSqZWaVCUbjcBFktS0r9fzStYGbjqzB2f2but3ORIDCnCRJLRw407um7WIU3u24Zaz1aQqWSnARZLMzn0ljMnM4ogmDXjsin5qUpXENAcukkTKyx23v5zD5qIDTPv5YI5Qk6qkphG4SBJ5+tOVvJ9XwD3nH8OArq38LkdiTAEukiS+WLmVR95ZygUntOeaIWl+lyNxoAAXSQL5uw5w85R5dGvThId/dIKaVNURmgMXCbiSsnLGvZTNvuIyplw/iCYN9bSuK/RIiwTcw28t4es1O3h8RD96qklVnaIpFJEAe2vBZv7x2Wp+Orgrw/upSVVdowAXCahVhXu485Vc+nZuyT3nH+N3OeIDBbhIAO0vLuPGzGzqpxhPqUlVnaU5cJGAcc5xz2sLWJq/m+d+NpCOLRv5XZL4RCNwkYCZ8t/1zMzeyM1n9eT0XjrPbF2mABcJkAUbdvLrcJOqm4f29Lsc8ZkCXCQgivYVMyYzizZNG/D4iP5qUiWaAxcJgvJyx23T55O/6wDTfz6Y1k0a+F2SJACNwEUCYNInK/lwSQH3XtCH/l3UpEpCFOAiCe7zFVv587tLuahvB34yqKvf5UgCUYCLJLAtO0NNqrqnNuWhS49Xkyr5Fs2BiySokrJyxr6Uzf6SMqaNSleTKvkO/UWIJKiHZi8ha+0OnryyPz3aqkmVfJemUEQS0Ju5m3n289VcMySNC/t28LscSVAKcJEEs7JwD3e9Mp/+XVryy2FqUiXVU4CLJJB9xaWMmZxFw/opTLwqnQb19BSV6mkOXCRBOOe459WFLC/YwwvXDqSDmlTJIejlXSRBZM5Zx6vzNjJ+aC9O7akmVXJoCnCRBJC7oYjf/nsxp/dK5aazevhdjgRErQPczDqb2Udmlmdmi8zsFi8LE6krduwtZszkbFKbNeSxK/pxmJpUSYSimQMvBW53zmWbWTMgy8zec84t9qg2kaRXXu64dXoOhbsP8vLowbRSkyqpgVqPwJ1zm51z2eHLu4E8QGdVFamBiR+t4OOlhdx7YR/6dm7pdzkSMJ7MgZtZGtAfmFPFshvMbK6ZzS0sLPRicyJJ4T/LC3n0/WVc3K8Do07q4nc5EkBRB7iZNQVmAOOdc7sqL3fOPeOcy3DOZaSm6si6CMCmov3cMjWHnm2b8ns1qZJaiirAzaw+ofDOdM7N9KYkkeRWXBpqUnWwpIxJowbQuIE+jiG1U+u/HAsNGf4J5DnnHvWuJJHk9vvZecxbV8TEq9I5KrWp3+VIgEUzAj8Z+AlwlpnlhL+GeVSXSFL69/xNPPfFGn52chrnn9De73Ik4Go9AnfOfQZo4k4kQisK9jBhRi7pXVpy93lqUiXR0ycxReJg78EKTapGqkmVeENHT0RizDnHL19dwIrCPbx47Um0b6EmVeINDQNEYmzyV2t5PWcTt53di1N6tvG7HEkiCnCRGMpZX8Rv31jMmb1TGXummlSJtxTgIjGyY28xYzOzadvscP6iJlUSA5oDF4mBsnLHLdNCTapeGTOYlo3VpEq8pxG4SAw8+eFyPl1WyP0X9eGETi39LkeSlAJcxGOfLCvk8Q+Wc2n/jlw1UE2qJHYU4CIe2li0n/FT59GrbTMevERNqiS2FOAiHikuLWdsZjYlZY5Jo9Jp1CDF75IkyekgpohHHnxzMTnri3hqZDrd1aRK4kAjcBEPzJq/iee/XMt1p3Rj2PFqUiXxoQAXidLy/N1MmJFLRtdWTDjvaL/LkTpEAS4Shb0HSxmTmU3jBin89ap06qfoKSXxozlwkVpyzjFh5gJWFe5h8nUncWSLw/0uSeoYDRdEaumFL9fy7/mbuP2HvRnSQ02qJP4U4CK1kL1uB797czFDj27LmNOP8rscqaMU4CI1tG3PQcZmZtOu+eE8ermaVIl/NAcuUgNl5Y7x03LYtreYmWOG0KJxfb9LkjpMI3CRGnj8g+X8Z/lWfnPRsRzXsYXf5UgdpwAXidDHSwt48sPl/Ci9EyNO7Ox3OSIKcJFIbNixj/HTcujdrhm/u/g4NamShKAAFzmEg6VljM3MpqzMMWnUADWpkoShg5gih/C7N/KYv2EnT49Kp1ubJn6XI/I/GoGLfI/Xczby4ldruf7Ubpx7nJpUSWJRgItUY1n+bibMWMCJaa2461w1qZLEowAXqcKeg6WMnpxFk4b11KRKEpb+KkUqcc7xixm5rNm6lyev7E+75mpSJYlJAS5SyXNfrOHN3M3ccU5vBh91hN/liFRLAS5SQdba7Tz4Zh5nH9OO0aepSZUkNgW4SNjWPQcZmzmPDi0b8efL+6pJlSS8qALczM41s6VmtsLMJnhVlEi8lZU7bpk6j+37inlqZDotGqlJlSS+Wge4maUAE4HzgD7AlWbWx6vCROLpsfeX8fmKbTwwXE2qJDii+STmQGCFc24VgJlNBYYDi70orKInP1jOrPmbvL5ZEQDKnGNV4V4uG9CJK07s4nc5IhGLJsA7Ausr/LwBOKnySmZ2A3ADQJcutXtypDZrSM92TWv1uyKRGHp0W27/YW+/yxCpkWgCvKojPO47Vzj3DPAMQEZGxneWR2LEwC6MGKiRkYhIRdEcxNwAVGyK3AnQPIeISJxEE+BfAz3NrJuZNQBGALO8KUtERA6l1lMozrlSMxsHvAOkAM865xZ5VpmIiHyvqPqBO+dmA7M9qkVERGpAn8QUEQkoBbiISEApwEVEAkoBLiISUOZcrT5bU7uNmRUCa2v5622ArR6W4xXVVTOqq2ZUV80kal0QXW1dnXOpla+Ma4BHw8zmOucy/K6jMtVVM6qrZlRXzSRqXRCb2jSFIiISUApwEZGAClKAP+N3AdVQXTWjumpGddVMotYFMagtMHPgIiLybUEagYuISAUKcBGRgEqoADezy8xskZmVm1lGpWV3h0+evNTMzqnm91ub2Xtmtjz8vVUMapxmZjnhrzVmllPNemvMbEF4vble11HF9n5tZhsr1DasmvXieiJqM/uTmS0xs1wze9XMWlazXlz216Huv4U8EV6ea2bpsaqlwjY7m9lHZpYX/vu/pYp1zjCznRUe3/tiXVd4u9/7uPi0v3pX2A85ZrbLzMZXWicu+8vMnjWzAjNbWOG6iHLIk+eicy5hvoBjgN7Ax0BGhev7APOBhkA3YCWQUsXv/xGYEL48AXg4xvX+GbivmmVrgDZx3He/Bu44xDop4X3XHWgQ3qd9YlzXD4F64csPV/eYxGN/RXL/gWHAW4TOODUImBOHx649kB6+3AxYVkVdZwBvxOvvKdLHxY/9VcVjuoXQB13ivr+A04B0YGGF6w6ZQ149FxNqBO6cy3POLa1i0XBgqnPuoHNuNbCC0EmVq1rv+fDl54GLY1IooZEHcDkwJVbbiIH/nYjaOVcMfHMi6phxzr3rnCsN//gVoTM3+SWS+z8ceMGFfAW0NLP2sSzKObfZOZcdvrwbyCN0ztkgiPv+qmQosNI5V9tPeEfFOfcpsL3S1ZHkkCfPxYQK8O9R1QmUq/oDb+ec2wyhJwXQNoY1nQrkO+eWV7PcAe+aWVb4xM7xMC78b+yz1fzbFul+jJVrCY3WqhKP/RXJ/fd1H5lZGtAfmFPF4sFmNt/M3jKzY+NU0qEeF7//pkZQ/SDKj/0FkeWQJ/stqhM61IaZvQ8cWcWie5xzr1f3a1VcF7P3P0ZY45V8/+j7ZOfcJjNrC7xnZkvCr9YxqQuYBDxAaL88QGh659rKN1HF70a9HyPZX2Z2D1AKZFZzM57vr6pKreK6yvc/rn9r39qwWVNgBjDeOber0uJsQtMEe8LHN14DesahrEM9Ln7urwbARcDdVSz2a39FypP9FvcAd86dXYtfi/QEyvlm1t45tzn8b1xBLGo0s3rApcCA77mNTeHvBWb2KqF/maIKpEj3nZn9HXijikUxORF1BPvrauACYKgLTwBWcRue768qRHL/fTlZt5nVJxTemc65mZWXVwx059xsM3vKzNo452LauCmCx8XPk5ufB2Q75/IrL/Brf4VFkkOe7LegTKHMAkaYWUMz60bolfS/1ax3dfjy1UB1I/ponQ0scc5tqGqhmTUxs2bfXCZ0IG9hVet6pdK84yXVbC/uJ6I2s3OBXwAXOef2VbNOvPZXJPd/FvDT8LsrBgE7v/l3OFbCx1P+CeQ55x6tZp0jw+thZgMJPXe3xbiuSB6XuO+vCqr9L9iP/VVBJDnkzXMx1kdpa3hE9xJCr0wHgXzgnQrL7iF01HYpcF6F6/9B+B0rwBHAB8Dy8PfWMarzOWB0pes6ALPDl7sTOqo8H1hEaCoh1vvuRWABkBv+Q2hfua7wz8MIvcthZZzqWkFori8n/PW0n/urqvsPjP7m8ST0r+3E8PIFVHg3VAxrOoXQv8+5FfbTsEp1jQvvm/mEDgYPiUNdVT4ufu+v8HYbEwrkFhWui/v+IvQCshkoCWfXddXlUCyei/oovYhIQAVlCkVERCpRgIuIBJQCXEQkoBTgIiIBpQAXEQkoBbiISEApwEVEAur/AfSjrXX6Zy+HAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWQ0lEQVR4nO3dfbRldX3f8fenM0CioIgMCsPDkHYwjl1qYURto6EVlSE2xKzUglYN0dBZlTa2q2tB6ira2mVqjWnqCjhrgmgeVJJW1AmOIumK2iyLMlAeRXQYeRgHYRCf8AEc+faPs++wOe5775k75947e/t+rXXXnLP37+zzPfvc+dzf+f32PjtVhSSp//7OchcgSZoOA12SBsJAl6SBMNAlaSAMdEkaCANdkgbCQNcBJcmLkty+SNv+D0kuXYxtz/GcSfL+JN9K8sWlfG797DHQtWBJ7kzywyTfS/LtJJ9PsjHJgn+vqur/VNUzplDbaUl2jm37HVX1xv3d9j76JeClwLFVder4yiS/meQnSR5K8t0kNyZ5xSQb7nqNrW3+bcfyO5OcvpAXoX4w0LW//mlVHQacAPxX4ALgfQvZUJKV0yzsAHECcGdVfX+ONv+3qg4FDgcuAS5PcvgS1KaBMdA1FVX1naraAvxz4PVJ/j5AkkOS/H6Su5Pcl2RTkp9v1p2WZGeSC5J8A3h/u9eZ5MIk/6v9PEn+R5L3NLfPTXJb8wlhR5J/2Sx/IvBJ4Jim5/tQkmOSvC3JnzdtPpXk/LFt35jk15vbv5jk6iQPJrk9yatme+3Ntrc0bbcn+e1m+RuAS4EXNjX8p3n24aPAnwFPBNbOt/+kcQa6pqqqvgjsBF7ULHoncBLwXODvAauBi1oPeTpwBKOe7Hljm/swcGaSJwEkWQG8CvhQs/5+4BXAk4Bzgf+e5OSmN7wB2FVVhzY/u8a2/SHgnJk7SdY1NXyi+YNwddPmqKbdJUmeNcvL/nDzmo8BfgN4R5KXVNX7gI00PfCqeussj5+pYUXzOn4M3NUsnm//SXsZ6FoMu4AjkgT4beDfVtWDVfU94B3A2a22jwJvraqHq+qH7Y1U1V3A9cCvNYv+CfCDqrqmWf+JqrqjRj4LfJrH/pDM56PAc5Oc0Nx/DXBFVT3M6I/EnVX1/qraU1XXAx9hFNaPk+Q4RuPkF1TVj6rqBka98tdOWAfAC5J8G/gR8PvAv6iq+yfcf9JeBroWw2rgQWAV8ATgumbS9NvAp5rlM3ZX1Y/m2Fa7J/1qHuudk2RDkmuaoY5vA2cCR05SYBOOn+CxcDwb+GBz+wTg+TM1N9t+DaNPE+OOAWbCdsZdjPbBpK6pqsOBpwBbeOyP0iT7r8se4KCO5Qcx6v1roIY4CaVllOR5jMLsb4EHgB8Cz6qqr8/ykPm+7vN/Au9OcizwSuCFzfMcwqjX/Drg41X14yQfAzLhdmE0VPLWJJ8Dfh74m2b5PcBnq+qlE2xj5tPIYa1QPx6Y7fXOqqoeSvKvgDuSXAbcyPz7r8vdwPFJUs3XqSZ5AqPho7vmfKR6zR66piLJk5rD7S4H/ryqbm4m+f6Y0dj2UU271UlePul2q2o38Bng/cDXquq2ZtXBwCHAbmBPkg3Ay1oPvQ94apInz7H5rYx64/8Z+IumXoArgZOSvDbJQc3P85I8s6O+e4DPA7+X5OeSPBt4A4/19vdJVX2T0ZDNRZPuv+Z59/4AX2A0fHNhs+yJjI5A2oaBPmgGuvbXXyX5HqNe7VuAP2A0sTfjAmA7cE2S7wJ/DezrceYfAk6nNdzS9Ib/DfCXwLcYDcdsaa3/MqMe+I5muOKY8Y024+VXzLLtlzEahtkFfIPR5OQhs9R3DrCmaftRRnMCV+/ja2z7Q0aTwc9m/v23mlEvvv1zHPArwGmMJmt3MBoaetVMj13DFN9fSRoGe+iSNBAGuiQNhIEuSQNhoEvSQCzbcehHHnlkrVmzZrmeXpJ66brrrnugqjpPLlu2QF+zZg3btm1brqeXpF5KMuu5BA65SNJAGOiSNBAGuiQNhIEuSQNhoEvSQMwb6EkuS3J/kltmWZ8k72kuvXVTkpOnX6YkaT6T9NA/AJwxx/oNjK5/uJbRJcTeu/9lSZL21bzHoVfV55KsmaPJWcCfNl/LeU2Sw5McXVX3TqtIaSl97P99nR27H1ruMjRg69ccwYtPmu/CU/tuGicWrWb0XdgzdjbLfirQk5xHcyHg448/fgpPLU3fv/vLG3i0IJm/rbQQG3/57x6wgd71a9/5JetVtRnYDLB+/Xq/iF0HpEcL3nz6Wt58+knLXYq0T6ZxlMtORldImXEsoyu3SL3jBV/UZ9MI9C3A65qjXV4AfMfxc0laevMOuST5MKNrEx6ZZCfwVuAggKraxOhCu2cyuu7hD3j89SSlXkrnSKJ0YJvkKJdz5llfwJumVpG0jBxxUZ95pqgkDYSBLrXMdNA9ZFF9ZKBL0kAY6FIHO+jqIwNdavE4dPWZgS5JA2GgSy1OiqrPDHRJGggDXeoQu+jqIQNdanFOVH1moEvSQBjoUkt1f5W/1AsGuiQNhIEudXBOVH1koEstToqqzwx0SRoIA13q4BWL1EcGuiQNhIEuSQNhoEstM5OiHuWiPjLQJWkgDHSpgx109ZGBLrV46r/6zECXpIEw0KUWJ0XVZwa6JA2EgS518ExR9ZGBLrU4Jao+M9AlaSAMdKmlmllRJ0XVRwa6JA3ERIGe5IwktyfZnuTCjvVPTvJXSW5McmuSc6dfqiRpLvMGepIVwMXABmAdcE6SdWPN3gR8qaqeA5wGvDvJwVOuVVp0ToqqzybpoZ8KbK+qHVX1CHA5cNZYmwIOSxLgUOBBYM9UK5UkzWmSQF8N3NO6v7NZ1vZHwDOBXcDNwO9U1aPjG0pyXpJtSbbt3r17gSVLi+exM0WdFVX/TBLoXb/Z459MXw7cABwDPBf4oyRP+qkHVW2uqvVVtX7VqlX7WKokaS6TBPpO4LjW/WMZ9cTbzgWuqJHtwNeAX5xOidLSs3+uPpok0K8F1iY5sZnoPBvYMtbmbuAlAEmeBjwD2DHNQqUl4ayoemzlfA2qak+S84GrgBXAZVV1a5KNzfpNwNuBDyS5mVHn5oKqemAR65YkjZk30AGqaiuwdWzZptbtXcDLpluatPRmLnDhnKj6yDNFJWkgDHRJGggDXWrZexz68pYhLYiBLkkDYaBLLTNHLXqmqPrIQJekgTDQJWkgDHSpxSsWqc8MdEkaCANdatk7KbqsVUgLY6BL0kAY6JI0EAa61FKPHYi+rHVIC2GgS9JAGOhSy96vz13mOqSFMNAlaSAMdEkaCANdapv5+lzHXNRDBrokDYSBLrU8dqaoXXT1j4EuSQNhoEvSQBjoUks5KaoeM9AlaSAMdKml9k6LSv1joEsdHHFRHxnokjQQBrrU4qSo+sxAl6SBMNClFqdE1WcTBXqSM5LcnmR7kgtnaXNakhuS3Jrks9MtU1panvqvPlo5X4MkK4CLgZcCO4Frk2ypqi+12hwOXAKcUVV3JzlqkeqVJM1ikh76qcD2qtpRVY8AlwNnjbV5NXBFVd0NUFX3T7dMaWnU3lnR5a1DWohJAn01cE/r/s5mWdtJwFOSfCbJdUle17WhJOcl2ZZk2+7duxdWsSSp0ySB3tVXGZ87WgmcAvwK8HLgPyY56aceVLW5qtZX1fpVq1btc7GSpNnNO4bOqEd+XOv+scCujjYPVNX3ge8n+RzwHOArU6lSWiKOuKjPJumhXwusTXJikoOBs4EtY20+DrwoycokTwCeD9w23VIlSXOZt4deVXuSnA9cBawALquqW5NsbNZvqqrbknwKuAl4FLi0qm5ZzMKlxRRPFVUPTTLkQlVtBbaOLds0dv9dwLumV5okaV94pqgkDYSBLrU4Kao+M9AlaSAMdKll5opFzomqjwx0SRoIA12SBsJAl1q8YpH6zECXpIEw0KUWr1ikPjPQpQ5esUh9ZKBL0kAY6FLLzBWLnBRVHxnokjQQBrrU4qSo+sxAl6SBMNAlaSAMdKnlsTNFnRVV/xjokjQQBrr0OE6Lqr8MdKmDAy7qIwNdkgbCQJda/Ppc9ZmBLkkDYaBLLU6Jqs8MdKmDX5+rPjLQJWkgDHSpxUlR9ZmBLkkDYaBLLeW0qHrMQJc6OOKiPjLQJWkgJgr0JGckuT3J9iQXztHueUl+kuQ3pleitHTKERf12LyBnmQFcDGwAVgHnJNk3Szt3glcNe0ipaXmUS7qo0l66KcC26tqR1U9AlwOnNXR7l8DHwHun2J90pKyh64+myTQVwP3tO7vbJbtlWQ18Epg01wbSnJekm1Jtu3evXtfa5WWkF109c8kgd71mz3ej/lD4IKq+slcG6qqzVW1vqrWr1q1asISJUmTWDlBm53Aca37xwK7xtqsBy5vrsN4JHBmkj1V9bFpFCktFY9DV59NEujXAmuTnAh8HTgbeHW7QVWdOHM7yQeAKw1z9ZmTouqjeQO9qvYkOZ/R0SsrgMuq6tYkG5v1c46bS33ipKj6bJIeOlW1Fdg6tqwzyKvqN/e/LGl52UFXH3mmqCQNhIEuSQNhoEsd4qyoeshAl6SBMNCllr1XLFreMqQFMdAlaSAMdKnFM0XVZwa61ME5UfWRgS5JA2GgSy17J0XtoauHDHRJGggDXWpxSlR9ZqBLHeKR6OohA12SBsJAl1rKL0RXjxnoUhdHXNRDBrrUYv9cfWagSx3soKuPDHRJGggDXWpxTlR9ZqBLHbxikfrIQJcexy66+stAlzrYP1cfGeiSNBAGutTipKj6zECXOjgnqj4y0KUWO+jqMwNd6uDX56qPDHRJGggDXWpxUlR9NlGgJzkjye1Jtie5sGP9a5Lc1Px8Pslzpl+qtHScFFUfzRvoSVYAFwMbgHXAOUnWjTX7GvDLVfVs4O3A5mkXKi0FL3ChPpukh34qsL2qdlTVI8DlwFntBlX1+ar6VnP3GuDY6ZYpLS076OqjSQJ9NXBP6/7OZtls3gB8smtFkvOSbEuybffu3ZNXKUma1ySB3tVZ6fxcmuQfMwr0C7rWV9XmqlpfVetXrVo1eZXSEnHARX22coI2O4HjWvePBXaNN0rybOBSYENVfXM65UnLxDEX9dAkPfRrgbVJTkxyMHA2sKXdIMnxwBXAa6vqK9MvU1oazomqz+btoVfVniTnA1cBK4DLqurWJBub9ZuAi4CnApc0FwbYU1XrF69sSdK4SYZcqKqtwNaxZZtat98IvHG6pUnLx1P/1UeeKSq1lNOi6jEDXergmaLqIwNdarODrh4z0CVpIAx0qYMjLuojA11qccRFfWagSx3irKh6yECXWjxTVH1moEvSQBjoUgdHXNRHBrrU4pmi6jMDXepgB119ZKBLLU6Kqs8MdEkaCANd6uCkqPrIQJdaHHFRnxnokjQQBrrUyTEX9Y+BLrWUh7moxwx0qYOTouojA11qsX+uPjPQJWkgDHSpgyMu6iMDXWpzzEU9ZqBLHbxikfrIQJda/Ppc9ZmBLkkDYaBLHRxwUR8Z6FKLJ4qqzwx0qYNzouojA11qsYeuPpso0JOckeT2JNuTXNixPkne06y/KcnJ0y9VkjSXeQM9yQrgYmADsA44J8m6sWYbgLXNz3nAe6dcp7Sk4rSoemjlBG1OBbZX1Q6AJJcDZwFfarU5C/jTGn336DVJDk9ydFXdO+2CP/uV3fyXK780f0NpAb7/8J7lLkFasEkCfTVwT+v+TuD5E7RZDTwu0JOcx6gHz/HHH7+vtQJw6CErWfu0Qxf0WGkSL/65g/wdUy9NEuhdnz3Hp44maUNVbQY2A6xfv35B00+nnPAUTjnhlIU8VJIGbZJJ0Z3Aca37xwK7FtBGkrSIJgn0a4G1SU5McjBwNrBlrM0W4HXN0S4vAL6zGOPnkqTZzTvkUlV7kpwPXAWsAC6rqluTbGzWbwK2AmcC24EfAOcuXsmSpC6TjKFTVVsZhXZ72abW7QLeNN3SJEn7wjNFJWkgDHRJGggDXZIGwkCXpIFILdPXyyXZDdy1wIcfCTwwxXKm5UCtCw7c2qxr31jXvhliXSdU1aquFcsW6PsjybaqWr/cdYw7UOuCA7c269o31rVvftbqcshFkgbCQJekgehroG9e7gJmcaDWBQdubda1b6xr3/xM1dXLMXRJ0k/raw9dkjTGQJekgThgAz3JP0tya5JHk6wfW/e7zQWpb0/y8lkef0SSq5N8tfn3KYtQ418kuaH5uTPJDbO0uzPJzU27bdOuo+P53pbk663azpyl3ZwX/16Eut6V5MvNhcQ/muTwWdotyf46EC9+nuS4JH+T5Lbm9/93OtqcluQ7rff3osWuq/Xcc743y7TPntHaFzck+W6SN4+1WZJ9luSyJPcnuaW1bKIsmsr/x6o6IH+AZwLPAD4DrG8tXwfcCBwCnAjcAazoePx/Ay5sbl8IvHOR6303cNEs6+4EjlzCffc24N/P02ZFs+9+ATi42afrFrmulwErm9vvnO09WYr9NcnrZ/SV0J9kdEWuFwBfWIL37mjg5Ob2YcBXOuo6DbhyqX6f9uW9WY591vG+foPRyTdLvs+AFwMnA7e0ls2bRdP6/3jA9tCr6raqur1j1VnA5VX1cFV9jdF3sJ86S7s/aW7/CfBri1Ioo14J8Crgw4v1HItg78W/q+oRYObi34umqj5dVTNXYb6G0ZWtlsskr3/vxc+r6hrg8CRHL2ZRVXVvVV3f3P4ecBuj6/P2xZLvszEvAe6oqoWehb5fqupzwINjiyfJoqn8fzxgA30Os12QetzTqrlqUvPvUYtY04uA+6rqq7OsL+DTSa5rLpS9FM5vPvJeNstHvEn342L5LUY9uS5Lsb8mef3Luo+SrAH+AfCFjtUvTHJjkk8medZS1cT8781y/16dzewdq+XaZ5Nk0VT220QXuFgsSf4aeHrHqrdU1cdne1jHskU79nLCGs9h7t75P6qqXUmOAq5O8uXmL/mi1AW8F3g7o/3ydkbDQb81vomOx+73fpxkfyV5C7AH+OAsm5n6/uoqtWPZgi5+vhiSHAp8BHhzVX13bPX1jIYUHmrmRz4GrF2Kupj/vVnOfXYw8KvA73asXs59Nomp7LdlDfSqOn0BD5v0gtT3JTm6qu5tPvLdvxg1JlkJ/Dpwyhzb2NX8e3+SjzL6eLVfATXpvkvyx8CVHasW5cLeE+yv1wOvAF5SzeBhxzamvr86HLAXP09yEKMw/2BVXTG+vh3wVbU1ySVJjqyqRf8Sqgnem+W8YPwG4Pqqum98xXLuMybLoqnstz4OuWwBzk5ySJITGf2V/eIs7V7f3H49MFuPf3+dDny5qnZ2rUzyxCSHzdxmNDF4S1fbaRkbs3zlLM83ycW/p13XGcAFwK9W1Q9mabNU++uAvPh5Mx/zPuC2qvqDWdo8vWlHklMZ/T/+5mLW1TzXJO/Ncl4wftZPysu1zxqTZNF0/j8u9qzvQn8YBdFO4GHgPuCq1rq3MJoRvh3Y0Fp+Kc0RMcBTgf8NfLX594hFqvMDwMaxZccAW5vbv8BoxvpG4FZGQw+Lve/+DLgZuKn5pTh6vK7m/pmMjqK4Y4nq2s5onPCG5mfTcu6vrtcPbJx5Pxl9DL64WX8zraOtFrGmX2L0Ufum1n46c6yu85t9cyOjyeV/uNh1zfXeLPc+a573CYwC+smtZUu+zxj9QbkX+HGTX2+YLYsW4/+jp/5L0kD0cchFktTBQJekgTDQJWkgDHRJGggDXZIGwkCXpIEw0CVpIP4/bbZYS5jqxAMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "z = np.linspace(-10,10,1000)\n",
    "T = ReLU.forward(z)\n",
    "dT_dz = ReLU.gradient(z)\n",
    "\n",
    "plt.plot(z, T)\n",
    "plt.title('ReLU')\n",
    "plt.show()\n",
    "\n",
    "plt.plot(z, dT_dz)\n",
    "plt.title('Derivative of ReLU')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now define our first neural network through the Network class.\n",
    "\n",
    "We implemented for you the constructor of the class, which takes as input a specification of the dimension of each layer $l$ and a list of per-layer activation functions (so that each layer can have a different activation function). \n",
    "\n",
    "The initializations for our network are done in the following way:\n",
    "\n",
    "\\begin{align}\n",
    "W_{(l)}^{(i,j)} &= \\frac{\\mathcal N(0,1)}{\\sqrt{I}} \\\\ \n",
    "W_{(l)}^{(i,0)} &= 0 \\, \\, \\, i=1,\\dots,I \\\\\n",
    "i&=1,\\dots,I \\, \\, \\,  j=1,\\dots,J \n",
    "\\end{align}\n",
    "\n",
    "where $J,I$ are the number of neurons respectively at the input/output of the $l$-th layer. $W_{(l)}^{(i,j)}$ denotes the **weight** from node $j$ to $i$ in layer $l$ and $W_{(l)}^{(i,0)}$ is the **bias** term for layer $l$, node $i$. Let's change this notation a bit, and call the weights in layer $l$ (without the bias) as $\\tilde{\\mathbf{W}}_{(l)}$ and the bias as $\\mathbf{b}_{(l)}$.\n",
    "\n",
    "You will have to implement the forward pass of the network, which generates its output $\\mathbf{y}$.\n",
    "Specifically, the input is fed to the network and recursively multiplied with the weights of every layer $l$.\n",
    "The mathematics of the forward pass are the same for every layer $l$ \n",
    "Using the notation we introduced, this can be written as \n",
    "\n",
    "\\begin{align}\n",
    "\\mathbf{a}_{(l)} &= \\tilde{\\mathbf{W}}_{(l)}^T \\mathbf{z}_{(l-1)} + \\mathbf{b}_{(l)}\\;. \\\\\n",
    "\\end{align}\n",
    "\n",
    "Followed by the nonlinear activation \n",
    "\\begin{align}\n",
    "\\mathbf{z}_{(l)} &= f_{(l)}(\\mathbf{a}_{(l)})\n",
    "\\end{align}\n",
    "where $f_{(l)}$ denotes the activation function for layer $l$, which is applied independently to every element of the vector $\\mathbf{a}_{(l)}$. $\\mathbf{z}_{(l)}$ is then the output of layer $l$, and we denote by $\\mathbf{z}_{(0)}$ to input ${\\bf x}$ (i.e., we can think of the input to the network as the output of layer $0$).\n",
    "\n",
    "Implement the forward pass in the method $\\texttt{feed\\_forward}$ below, which takes as input ${\\bf x}$ returns ${\\bf z}$ and ${\\bf a}$, two dictionaries containing the output dimension and activation function for every layer. Keeping track of computed quantities will be useful for training the model. \n",
    "\n",
    "Method $\\texttt{predict}$, which we have implemented for you, is just a wrapper around $\\texttt{feed\\_forward}$ yielding the output of our network. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network:\n",
    "    def __init__(self, dimensions, activations):\n",
    "        \"\"\"\n",
    "        :param dimensions: list of dimensions of the neural net. (input, hidden layer, ... ,hidden layer, output)\n",
    "        :param activations: list of activation functions. Must contain N-1 activation function, where N = len(dimensions).\n",
    "        \n",
    "        Example of one hidden layer with\n",
    "        - 2 inputs\n",
    "        - 10 hidden nodes\n",
    "        - 5 outputs\n",
    "        layers -->    [0,        1,          2]\n",
    "        ----------------------------------------\n",
    "        dimensions =  (2,     10,          5)\n",
    "        activations = (      Sigmoid,      Sigmoid)\n",
    "        \"\"\"\n",
    "        \n",
    "        self.n_layers = len(dimensions)\n",
    "        self.loss = None\n",
    "        self.learning_rate = None\n",
    "        \n",
    "        # Weights and biases are initiated by index.\n",
    "        self.w = {}\n",
    "        self.b = {}\n",
    "\n",
    "        # Activations are also initiated by index. \n",
    "        self.activations = {}\n",
    "        for l in range(1, self.n_layers):\n",
    "            self.w[l] = np.random.randn(dimensions[l - 1], dimensions[l]) / np.sqrt(dimensions[l - 1])\n",
    "            self.b[l] = np.zeros(dimensions[l])\n",
    "            self.activations[l] = activations[l-1]\n",
    "\n",
    "\n",
    "    def feed_forward(self, x):\n",
    "        \"\"\"\n",
    "        Execute a forward feed through the network.\n",
    "        :param x: (array) Batch of input data vectors.\n",
    "        :return: (tpl) Node outputs and activations per layer. The numbering of the output is equivalent to the layer numbers.\n",
    "        \"\"\"\n",
    "\n",
    "        # w(x) + b\n",
    "        a = {}\n",
    "\n",
    "        # activations: z = f(a)\n",
    "        z = {0: x}  # First layer has no activations as input, so we consider input itself as the first activation.\n",
    "        \n",
    "        for l in range(1, self.n_layers):\n",
    "            # current layer = l\n",
    "            a[l] = np.dot(z[l - 1], self.w[l]) + self.b[l]\n",
    "            z[l] = self.activations[l].forward(a[l])\n",
    "\n",
    "        return z, a\n",
    "    \n",
    "    def predict(self, x):\n",
    "        \"\"\"\n",
    "        :param x: (array) Containing parameters\n",
    "        :return: (array) A 2D array of shape (n_cases, n_classes).\n",
    "        \"\"\"\n",
    "        _, a = self.feed_forward(x)\n",
    "        return a[self.n_layers - 1]\n",
    "    \n",
    "\n",
    "    def back_prop(self, z, a, y_true):\n",
    "        \"\"\"\n",
    "        The input dicts keys represent the layers of the net.\n",
    "        a = { 0: x,\n",
    "              1: f(w1(x) + b1)\n",
    "              2: f(w2(a2) + b2)\n",
    "              }\n",
    "        :param a: (dict) w^T@x + b\n",
    "        :param z: (dict) f(a)\n",
    "        :param y_true: (array) One hot encoded truth vector.\n",
    "        :return:\n",
    "        \"\"\"\n",
    "\n",
    "        # Determine partial derivative and delta for the output layer.\n",
    "        y_pred = z[self.n_layers - 1]\n",
    "        delta = self.loss_function.gradient(y_true, y_pred) * self.activations[self.n_layers - 1].gradient(y_pred)\n",
    "        \n",
    "        dw = np.dot(z[self.n_layers - 2].T, delta)\n",
    "\n",
    "        update_params = {\n",
    "            self.n_layers - 1: (dw, delta)\n",
    "        }\n",
    "\n",
    "        # Determine partial derivative and delta for the rest of the layers.\n",
    "        # Each iteration requires the delta from the previous layer, propagating backwards.\n",
    "        for l in reversed(range(1, self.n_layers - 1)):\n",
    "            delta = np.dot(delta, self.w[l+1].T) * self.activations[l].gradient(a[l])\n",
    "            dw = np.dot(z[l - 1].T, delta)\n",
    "            update_params[l] = (dw, delta)\n",
    "        \n",
    "        # finally update weights and biases\n",
    "        for k, v in update_params.items():\n",
    "            self.update_w_b(k, v[0], v[1])\n",
    "\n",
    "    def update_w_b(self, index, dw, delta):\n",
    "        \"\"\"\n",
    "        Update weights and biases.\n",
    "        :param index: (int) Number of the layer\n",
    "        :param dw: (array) Partial derivatives\n",
    "        :param delta: (array) Delta error.\n",
    "        \"\"\"\n",
    "\n",
    "        self.w[index] -= self.learning_rate * dw\n",
    "        self.b[index] -= self.learning_rate * np.mean(delta, 0)\n",
    "\n",
    "    def fit(self, x, y_true, loss, epochs, batch_size, learning_rate=1e-3):\n",
    "        \"\"\"\n",
    "        :param x: (array) Containing parameters\n",
    "        :param y_true: (array) Containing one hot encoded labels.\n",
    "        :param loss: Loss class (MSE, CrossEntropy etc.)\n",
    "        :param epochs: (int) Number of epochs.\n",
    "        :param batch_size: (int)\n",
    "        :param learning_rate: (flt)\n",
    "        \"\"\"\n",
    "        if not x.shape[0] == y_true.shape[0]:\n",
    "            raise ValueError(\"Length of x and y arrays don't match\")\n",
    "        # Initiate the loss object with the final activation function\n",
    "        self.loss_function = loss\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "        for i in range(epochs):\n",
    "            # Shuffle the data\n",
    "            indices = np.arange(x.shape[0])\n",
    "            np.random.shuffle(indices)\n",
    "            x_ = x[indices]\n",
    "            y_ = y_true[indices]\n",
    "\n",
    "            for j in range(x.shape[0] // batch_size):\n",
    "                k = j * batch_size\n",
    "                l = (j + 1) * batch_size\n",
    "                z, a = self.feed_forward(x_[k:l])\n",
    "                self.back_prop(z, a, y_[k:l])\n",
    "\n",
    "            if (i + 1) % 10 == 0:\n",
    "                z, _ = self.feed_forward(x)\n",
    "                print(\"Loss at epoch {}: {}\".format(i + 1, self.loss_function.loss(y_true, z[self.n_layers - 1])))\n",
    "                \n",
    "                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have now created your first neural network, which can produce outputs based on inputs. \n",
    "To make sure everything is right, you can run the cell below. It will define a network with 1 input layer, 1 hidden layer and 1 output layer, with 1 node each. This is defined by the `dimensions` parameter, as `(1, 1, 1)`. The activation functions of the hidden layer and output layer are both sigmoid, defined by the `activations` parameter as `(Sigmoid, Sigmoid)`. \n",
    "\n",
    "We then generate a random input `x` and pass it through the `feed_forward()` function of the `Network` object. This produces two outputs `z` and `a` which will both be dictionaries, containing the `z` and `a` values the network produces. You will be seeing the following dictionary outputs:\n",
    "\n",
    "a:\n",
    "{\n",
    "\t1: \"a values of the hidden layer\", \n",
    "\t2: \"a values of the output layer\"\n",
    "}\n",
    "\n",
    "z:\n",
    "{\n",
    "\t0: \"inputs x\",\n",
    "\t1: \"activations in the hidden layer\",\n",
    "\t2: \"activations in the output layer\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{1: array([[0.11413095]]), 2: array([[-0.12187239]])}\n",
      "{0: array([[0.0825075]]), 1: array([[0.52850181]]), 2: array([[0.46956956]])}\n"
     ]
    }
   ],
   "source": [
    "# initialize  network structure\n",
    "nn = Network(dimensions=(1, 1, 1), activations=(Sigmoid, Sigmoid))\n",
    "x = np.random.rand(1,1)\n",
    "z,a = nn.feed_forward(x)\n",
    "print(a)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loss functions map the predictions of our network into a real number, measuring the error in our predictions.\n",
    "\n",
    "In this first session we will take the Mean Squared Error (MSE) as our loss function.\n",
    "\n",
    "Let $\\hat{\\mathbf{y}}_i \\in \\mathbb R ^ C$ be the prediction generated by our network for $i$-th data sample, and $\\hat{\\mathbf{Y}}  \\in \\mathbb R ^{N \\times C} $ be the matrix of N predictions generated for $N$ data points.\n",
    "\n",
    "Similarly let $\\mathbf{Y} \\in \\mathbb R ^{N \\times K}$ be the matrix of true labels $\\mathbf{y}_i \\in \\mathbb R ^ C$.\n",
    "\n",
    "We then define the MSE of the predictor as\n",
    "\n",
    "\\begin{align}\n",
    "\\mathcal L _{\\text{MSE}} =\\frac {1}{N} \\sum _{i=1}^{N} \\frac {1}{C} \\sum _{c=1}^{C}\\left(\\hat{\\mathbf{y}}_{i}^{(c)}-{\\mathbf{y}_{i}^{(c)}}\\right)^{2}.\n",
    "\\end{align}\n",
    "\n",
    "We can compute its derivative w.r.t. the predictions $\\hat{\\mathbf{y}}_{i} ^{(c)}$ as\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\mathcal L _{\\text{MSE}} }{\\partial \\hat{\\mathbf{y}}_{i} ^{(c)}} =\\frac {2}{NC} \\left(\\hat{\\mathbf{y}}_{i}^{(c)}-{\\mathbf{y}_{i} ^{(c)}}\\right).\n",
    "\\end{align}\n",
    "\n",
    "Re-writing this in matrix form gives\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\mathcal L _{\\text{MSE}} }{\\partial \\hat{\\mathbf{Y}}} =\\frac {2}{NC} (\\hat{\\mathbf{Y}}-\\mathbf{Y}).\n",
    "\\end{align}\n",
    "\n",
    "Implement this loss function and its derivatives in matrix form below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MSE:\n",
    "    @staticmethod\n",
    "    def loss(y_true, y_pred):\n",
    "        \"\"\"\n",
    "        :param y_true: (array) One hot encoded truth vector.\n",
    "        :param y_pred: (array) Prediction vector\n",
    "        :return: (flt)\n",
    "        \"\"\"\n",
    "        return np.mean(np.mean((y_pred - y_true)**2, axis=1),axis=0)\n",
    "\n",
    "    @staticmethod\n",
    "    def gradient(y_true, y_pred):\n",
    "        return (2.0/(y_pred.shape[0]*y_pred.shape[1])) * (y_pred - y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training your network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss function $\\mathcal L$ gives us information about the prediction error of the model. Our goal is to minimize the loss function by optimizing the weights and biases. \n",
    "\n",
    "We know from calculus that if we take the partial derivative of the loss function with respect to a certain weight $W_{(l)}^{(i,j)}$ we get the rate of increase of $\\mathcal L$ in that direction. \n",
    "\n",
    "The opposite direction of $\\frac{\\partial \\mathcal L }{\\partial W_{(l)}^{(i,j)}}$ will then point in the direction that minimizes the loss function.\n",
    "We can therefore, choosing a step size $\\eta$, reduce the loss function by updating our weight as follows\n",
    "\\begin{align}\n",
    "W_{(l)}^{(i,j)} =W_{(l)}^{(i,j)} - \\eta \\frac{\\partial \\mathcal L }{\\partial W_{(l)}^{(i,j)}}\n",
    "\\end{align}\n",
    "This weight optimization is called gradient descent and is done w.r.t. every weight and bias of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To determine the partial derivative of $\\mathcal L$ with respect to a certain weight, we can apply the chain rule in differentiation\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}\n",
    "\\end{align}\n",
    "\n",
    "In practice this allows us to break down the problem within each layer. We call the whole process backpropagation.\n",
    "\n",
    "\n",
    "To understand backpropagation, we will take a step back and consider a very simple neural network with one hidden layer (with a single node) taking scalar input $x \\in \\mathbb R$ and outputting a scalar $y \\in \\mathbb R$\n",
    "\n",
    "\\begin{align}\n",
    "a_{(2)} &= \\tilde{W}_{(1)} x + b_{(1)} \\\\\n",
    "z_{(2)} &= S(a_{(2)}) \\\\\n",
    "a_{(3)} &= z_{(2)} \\tilde{W}_{(2)} + b^{(2)} \\\\\n",
    "y &= S(a_{(3)}) \\\\\n",
    "\\mathcal L &= (\\hat{y}-y)^2\n",
    "\\end{align}\n",
    "\n",
    "Using the chain rule, compute derivatives of $\\mathcal L$ with respect to all weights and biases:\n",
    "\\begin{align}\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\hat{y}} &= 2 (\\hat{y}-y) \\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial a_{(3)}} &= \\frac{\\partial \\mathcal L}{\\partial \\hat{y}}  \\frac{\\partial \\hat{y}}{\\partial a_{(3)}} = 2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)}))  \\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\tilde{W}_{(2)}} &= \\frac{\\partial \\mathcal L}{\\partial a_{(3)}} \\frac{\\partial a_{(3)}}{\\partial \\tilde{W}_{(2)}} = 2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\cdot z_{(2)} \\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial b_{(2)}} &= \\frac{\\partial \\mathcal L}{\\partial a_{(3)}} \\frac{\\partial a_{(3)}}{\\partial b_{(2)}} = 2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial z_{(2)}} &= \\frac{\\partial \\mathcal L}{\\partial a_{(3)}} \\frac{\\partial a_{(3)}}{\\partial z_{(2)}} =  2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\cdot \\tilde{W}_{(2)}\\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial a_{(2)}} &= \\frac{\\partial \\mathcal L}{\\partial z_{(2)}} \\frac{\\partial z_{(2)}}{\\partial a_{(2)}} = 2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\cdot \\tilde{W}_{(2)} \\cdot S(a_{(2)})(1-S(a_{(2)}))\\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial \\tilde{W_{(1)}}} &=\\frac{\\partial \\mathcal L}{\\partial a_{(2)}}  \\frac{\\partial a_{(2)}}{\\partial \\tilde{W}_{(1)}} = 2 (\\hat{y}- y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\cdot \\tilde{W}_{(2)} \\cdot S(a_{(2)})(1-S(a_{(2)})) \\cdot x\\\\\n",
    "\\frac{\\partial \\mathcal L}{\\partial b_{(1)}} &=\\frac{\\partial \\mathcal L}{\\partial a_{(2)}} \\frac{\\partial a_{(2)}}{\\partial b_{(1)}} = 2 (\\hat{y}-y) \\cdot S(a_{(3)})(1-S(a_{(3)})) \\cdot \\tilde{W}_{(2)} \\cdot S(a_{(2)})(1-S(a_{(2)}))\n",
    "\\end{align}\n",
    "\n",
    "As things get quite tricky with indices, we have implemented backpropagation for you in method $\\texttt{back\\_prop}$. Look at it and convince yourself that it generalizes the derivations above to the multi-dimensional case. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fitting your model to MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have implemented your first neural network, and have understood backpropagation, we will fit our model to the MNIST dataset.\n",
    "\n",
    "The MNIST database contains a total of 70,000 handwritten digits with 10 different classes, from 0 to 9. 60,000 examples are taken as training dataset and the remaining 10,000 as testing set. The digits have been size-normalized and centered in a fixed-size image. \n",
    "\n",
    "First download MNIST from: https://www.python-course.eu/data/mnist/mnist_train.csv\n",
    "and https://www.python-course.eu/data/mnist/mnist_test.csv\n",
    "and put these 2 csv files in the same folder as your jupyter notebook file.\n",
    "\n",
    "To make sure everything went well, we will first load the dataset and visualize some samples from it. It might take a while to load it into memory!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/guillard/miniconda3/envs/py37torch2/lib/python3.7/site-packages/ipykernel_launcher.py:18: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n",
      "/Users/guillard/miniconda3/envs/py37torch2/lib/python3.7/site-packages/ipykernel_launcher.py:19: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.\n",
      "Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations\n"
     ]
    }
   ],
   "source": [
    "# load the dataset\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "image_size = 28 # width and length\n",
    "no_of_different_labels = 10 #  i.e. 0, 1, 2, 3, ..., 9\n",
    "image_pixels = image_size * image_size\n",
    "train_data = np.loadtxt(\"mnist_train.csv\", \n",
    "                        delimiter=\",\")\n",
    "test_data = np.loadtxt(\"mnist_test.csv\", \n",
    "                       delimiter=\",\") \n",
    "\n",
    "x_train = np.asfarray(train_data[:, 1:])\n",
    "x_test = np.asfarray(test_data[:, 1:])\n",
    "y_train = np.asfarray(train_data[:, :1])\n",
    "y_test = np.asfarray(test_data[:, :1])\n",
    "# transform labels into one hot representation\n",
    "lr = np.arange(no_of_different_labels)\n",
    "y_train = (lr==y_train).astype(np.float)\n",
    "y_test = (lr==y_test).astype(np.float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOSklEQVR4nO3df4xU9bnH8c8jgqgQg7JQYsnd3kZNjcnd4kiuQQiXegnyDxDsTUlsaCTdxh9JMcRcw02sPxJDzKUVo2myvSD0ptdaBQQTc4sSEkOi1VFRQfydtWxZYYlKhSgt8Nw/9nCz4sx3lpkzc4Z93q9kMzPnOWfP47gfzsx8z5mvubsAjHznFN0AgNYg7EAQhB0IgrADQRB2IIhzW7mziRMnemdnZyt3CYTS29urQ4cOWaVaQ2E3s3mS1kgaJem/3H1Vav3Ozk6Vy+VGdgkgoVQqVa3V/TLezEZJelTSDZKulLTEzK6s9/cBaK5G3rNPl/SBu3/k7n+T9HtJC/JpC0DeGgn7pZL2DXncly37GjPrNrOymZUHBgYa2B2ARjQS9kofAnzj3Ft373H3kruXOjo6GtgdgEY0EvY+SVOHPP62pP2NtQOgWRoJ+yuSLjOz75jZGEk/krQ1n7YA5K3uoTd3P25mt0v6owaH3ta5+57cOgOQq4bG2d39WUnP5tQLgCbidFkgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCaGgWV7S/kydPJuvHjh1r6v43bNhQtXb06NHktm+//Xay/tBDDyXrK1eurFp75JFHktuef/75yfrq1auT9VtuuSVZL0JDYTezXklfSDoh6bi7l/JoCkD+8jiy/4u7H8rh9wBoIt6zA0E0GnaXtM3MXjWz7kormFm3mZXNrDwwMNDg7gDUq9Gwz3D3aZJukHSbmc06fQV373H3kruXOjo6GtwdgHo1FHZ335/dHpS0WdL0PJoCkL+6w25mF5rZ+FP3Jc2VtDuvxgDkq5FP4ydL2mxmp37P/7j7/+bS1Qhz+PDhZP3EiRPJ+htvvJGsb9u2rWrt888/T27b09OTrBeps7MzWV+xYkWyvnbt2qq1iy66KLntzJkzk/U5c+Yk6+2o7rC7+0eS/inHXgA0EUNvQBCEHQiCsANBEHYgCMIOBMElrjno6+tL1ru6upL1zz77LMduzh7nnJM+1qSGzqTal6EuW7asam3SpEnJbceNG5esn41ng3JkB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGfPwSWXXJKsT548OVlv53H2uXPnJuu1/ts3bdpUtXbeeeclt509e3ayjjPDkR0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgmCcPQe1rqtev359sv7UU08l69dee22yvnjx4mQ95brrrkvWt2zZkqyPGTMmWf/kk0+q1tasWZPcFvniyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQZi7t2xnpVLJy+Vyy/Z3tjh27FiyXmsse+XKlVVrDz74YHLbHTt2JOuzZs1K1tFeSqWSyuWyVarVPLKb2TozO2hmu4csu9jMnjOz97PbCXk2DCB/w3kZv17SvNOW3SVpu7tfJml79hhAG6sZdnd/QdKnpy1eIGlDdn+DpIX5tgUgb/V+QDfZ3fslKbutOnGWmXWbWdnMygMDA3XuDkCjmv5pvLv3uHvJ3Utn42R4wEhRb9gPmNkUScpuD+bXEoBmqDfsWyUtze4vlZS+DhJA4Wpez25mj0uaLWmimfVJ+oWkVZL+YGbLJP1Z0g+b2eRIV+v702uZMKH+kc+HH344WZ85c2ayblZxSBdtqGbY3X1JldIPcu4FQBNxuiwQBGEHgiDsQBCEHQiCsANB8FXSI8Dy5cur1l5++eXktps3b07W9+zZk6xfddVVyTraB0d2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcfYRIPVV0z09Pcltt2/fnqwvWLAgWV+4cGGyPmPGjKq1RYsWJbfl8tl8cWQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYsjm4Wte7z5t3+pyeX3f48OG6971u3bpkffHixcn6uHHj6t73SNXQlM0ARgbCDgRB2IEgCDsQBGEHgiDsQBCEHQiC69mDmz59erJe63vj77jjjmT9ySefrFq7+eabk9t++OGHyfqdd96ZrI8fPz5Zj6bmkd3M1pnZQTPbPWTZPWb2FzPblf3Mb26bABo1nJfx6yVVOo3qV+7elf08m29bAPJWM+zu/oKkT1vQC4AmauQDutvN7M3sZf6EaiuZWbeZlc2sPDAw0MDuADSi3rD/WtJ3JXVJ6pe0utqK7t7j7iV3L3V0dNS5OwCNqivs7n7A3U+4+0lJv5GU/kgXQOHqCruZTRnycJGk3dXWBdAeal7PbmaPS5otaaKkA5J+kT3ukuSSeiX9zN37a+2M69lHnq+++ipZf+mll6rWrr/++uS2tf42b7zxxmT9iSeeSNZHotT17DVPqnH3JRUWr224KwAtxemyQBCEHQiCsANBEHYgCMIOBMElrmjI2LFjk/XZs2dXrY0aNSq57fHjx5P1p59+Oll/9913q9auuOKK5LYjEUd2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcXYk7d+/P1nftGlTsv7iiy9WrdUaR6/lmmuuSdYvv/zyhn7/SMORHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJx9hKs15dajjz6arD/22GPJel9f3xn3NFy1rnfv7OxM1s0qfqNyWBzZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtnPAkeOHEnWn3nmmaq1++67L7nte++9V1dPeZgzZ06yvmrVqmT96quvzrOdEa/mkd3MpprZDjPba2Z7zOzn2fKLzew5M3s/u53Q/HYB1Gs4L+OPS1rh7t+T9M+SbjOzKyXdJWm7u18maXv2GECbqhl2d+9399ey+19I2ivpUkkLJG3IVtsgaWGTegSQgzP6gM7MOiV9X9KfJE12935p8B8ESZOqbNNtZmUzK9c6TxtA8ww77GY2TtJGScvd/a/D3c7de9y95O6ljo6OenoEkINhhd3MRmsw6L9z91NfJ3rAzKZk9SmSDjanRQB5qDn0ZoPXCa6VtNfdfzmktFXSUkmrststTelwBDh69Giyvm/fvmT9pptuStZff/31M+4pL3Pnzk3W77333qq1Wl8FzSWq+RrOOPsMST+W9JaZ7cqWrdRgyP9gZssk/VnSD5vSIYBc1Ay7u++UVO2f2B/k2w6AZuF0WSAIwg4EQdiBIAg7EARhB4LgEtdh+vLLL6vWli9fntx2586dyfo777xTT0u5mD9/frJ+9913J+tdXV3J+ujRo8+0JTQJR3YgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCCLMOHtvb2+y/sADDyTrzz//fNXaxx9/XE9Lubnggguq1u6///7ktrfeemuyPmbMmLp6QvvhyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQYQZZ9+4cWOyvnbt2qbte9q0acn6kiVLkvVzz03/b+ru7q5aGzt2bHJbxMGRHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCMHdPr2A2VdJvJX1L0klJPe6+xszukfRTSQPZqivd/dnU7yqVSl4ulxtuGkBlpVJJ5XK54qzLwzmp5rikFe7+mpmNl/SqmT2X1X7l7v+ZV6MAmmc487P3S+rP7n9hZnslXdrsxgDk64zes5tZp6TvS/pTtuh2M3vTzNaZ2YQq23SbWdnMygMDA5VWAdACww67mY2TtFHScnf/q6RfS/qupC4NHvlXV9rO3XvcveTupY6OjsY7BlCXYYXdzEZrMOi/c/dNkuTuB9z9hLuflPQbSdOb1yaARtUMu5mZpLWS9rr7L4csnzJktUWSduffHoC8DOfT+BmSfizpLTPblS1bKWmJmXVJckm9kn7WhP4A5GQ4n8bvlFRp3C45pg6gvXAGHRAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IIiaXyWd687MBiR9PGTRREmHWtbAmWnX3tq1L4ne6pVnb//g7hW//62lYf/Gzs3K7l4qrIGEdu2tXfuS6K1ereqNl/FAEIQdCKLosPcUvP+Udu2tXfuS6K1eLemt0PfsAFqn6CM7gBYh7EAQhYTdzOaZ2btm9oGZ3VVED9WYWa+ZvWVmu8ys0Pmlszn0DprZ7iHLLjaz58zs/ey24hx7BfV2j5n9JXvudpnZ/IJ6m2pmO8xsr5ntMbOfZ8sLfe4SfbXkeWv5e3YzGyXpPUn/KqlP0iuSlrj72y1tpAoz65VUcvfCT8Aws1mSjkj6rbtflS17UNKn7r4q+4dygrv/e5v0do+kI0VP453NVjRl6DTjkhZK+okKfO4Sff2bWvC8FXFkny7pA3f/yN3/Jun3khYU0Efbc/cXJH162uIFkjZk9zdo8I+l5ar01hbcvd/dX8vufyHp1DTjhT53ib5aooiwXypp35DHfWqv+d5d0jYze9XMuotupoLJ7t4vDf7xSJpUcD+nqzmNdyudNs142zx39Ux/3qgiwl5pKql2Gv+b4e7TJN0g6bbs5SqGZ1jTeLdKhWnG20K90583qoiw90maOuTxtyXtL6CPitx9f3Z7UNJmtd9U1AdOzaCb3R4suJ//107TeFeaZlxt8NwVOf15EWF/RdJlZvYdMxsj6UeSthbQxzeY2YXZBycyswslzVX7TUW9VdLS7P5SSVsK7OVr2mUa72rTjKvg567w6c/dveU/kuZr8BP5DyX9RxE9VOnrHyW9kf3sKbo3SY9r8GXd3zX4imiZpEskbZf0fnZ7cRv19t+S3pL0pgaDNaWg3q7T4FvDNyXtyn7mF/3cJfpqyfPG6bJAEJxBBwRB2IEgCDsQBGEHgiDsQBCEHQiCsANB/B/B/E1sUrHmQgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit is:  5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOb0lEQVR4nO3db6yU5ZnH8d8lLf4BJCAHgvbE4yImahOhmZBNNA2bug3oCyTqBqKENUQaAkpN/ReMqTGayLotSlyJsBBwbWkaipEXZq2SRuwLG0egwpHs6uIRzpFwDhFSq9Hy59oX57E54pl7hpln5hm4vp9kMjPPNfd5roz+eGbmfmZuc3cBOPedV3QDAFqDsANBEHYgCMIOBEHYgSC+08qdTZgwwbu6ulq5SyCUnp4eHTlyxIarNRR2M5sl6VlJIyT9p7s/lXp8V1eXyuVyI7sEkFAqlSrW6n4Zb2YjJP2HpNmSrpE038yuqffvAWiuRt6zz5D0obvvd/e/SfqNpDn5tAUgb42E/TJJB4fc7822fYOZLTazspmVBwYGGtgdgEY0EvbhPgT41rm37r7W3UvuXuro6GhgdwAa0UjYeyV1Drn/PUmfNNYOgGZpJOzvSJpqZleY2UhJ8yRty6ctAHmre+rN3U+Y2TJJr2lw6m2Du3fn1hmAXDU0z+7ur0p6NadeADQRp8sCQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EERLl2zGuefgwYPJ+rPPPluxtmrVquTY++67L1lfvnx5st7Z2ZmsR8ORHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ4dSX19fcn69OnTk/Vjx45VrJlZcuwzzzyTrG/atClZHxgYSNajaSjsZtYj6TNJJyWdcPdSHk0ByF8eR/Z/cvcjOfwdAE3Ee3YgiEbD7pJ+b2bvmtni4R5gZovNrGxmZd5DAcVpNOzXu/sPJM2WtNTMfnj6A9x9rbuX3L3U0dHR4O4A1KuhsLv7J9l1v6SXJc3IoykA+as77GY2yszGfH1b0o8l7c2rMQD5auTT+EmSXs7mSr8j6dfu/t+5dIWW+fjjj5P1mTNnJutHjx5N1lNz6WPHjk2OPf/885P1/v7+ZH3//v0Va5dffnly7IgRI5L1s1HdYXf3/ZKuy7EXAE3E1BsQBGEHgiDsQBCEHQiCsANB8BXXc8Dx48cr1qpNrc2aNStZr/ZT0Y2YNm1asv7kk08m6zfccEOyPnXq1Iq1tWvXJscuWrQoWT8bcWQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSCYZz8HPPDAAxVrzz33XAs7OTNvvvlmsv75558n63Pnzk3Wt27dWrG2a9eu5NhzEUd2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCefazQLXvlL/00ksVa+7e0L6rzWXfeuutyfqdd95ZsdbZ2Zkce/XVVyfrDz30ULK+ZcuWirVGn5ezEUd2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQjCWjnfWCqVvFwut2x/Z4u+vr5k/brr0ovlHjt2rO5933HHHcn6unXrkvX3338/Wd+5c2fF2rx585JjL7roomS9mtSyy6NGjUqO7e7uTtarnSNQlFKppHK5POw62VWP7Ga2wcz6zWzvkG3jzex1M/sgux6XZ8MA8lfLy/iNkk5fNuRhSdvdfaqk7dl9AG2satjdfYekT0/bPEfSpuz2Jkm35NsWgLzV+wHdJHc/JEnZ9cRKDzSzxWZWNrPywMBAnbsD0Kimfxrv7mvdveTupY6OjmbvDkAF9Yb9sJlNlqTsuj+/lgA0Q71h3yZpYXZ7oaRX8mkHQLNU/T67mW2WNFPSBDPrlfRzSU9J+q2ZLZJ0QNLtzWzybHfkyJFkfeXKlcn60aNHk/VJkyZVrF1xxRXJsUuWLEnWR44cmaxXW2O9Wr0oX3zxRbL+9NNPJ+urV6/Os52WqBp2d59fofSjnHsB0EScLgsEQdiBIAg7EARhB4Ig7EAQ/JR0Dk6cOJGs33///cl66qegJWns2LHJ+muvvVaxduWVVybHHj9+PFmP6qOPPiq6hdxxZAeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIJhnz8GBAweS9Wrz6NW8/fbbyfpVV11V99++8MIL6x6LswtHdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0Ignn2HCxdujRZr7Ys9ty5c5P1RubRIzt16lTF2nnnpY9zrVzKvFU4sgNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEMyz12jXrl0Vazt27EiONbNk/fbbWfG6GVJz6dX+m5RKpbzbKVzVI7uZbTCzfjPbO2TbY2bWZ2a7s8tNzW0TQKNqeRm/UdKsYbavcvdp2eXVfNsCkLeqYXf3HZI+bUEvAJqokQ/olpnZe9nL/HGVHmRmi82sbGblgYGBBnYHoBH1hn2NpCmSpkk6JOkXlR7o7mvdveTupY6Ojjp3B6BRdYXd3Q+7+0l3PyVpnaQZ+bYFIG91hd3MJg+5O1fS3kqPBdAeqs6zm9lmSTMlTTCzXkk/lzTTzKZJckk9kn7SvBbbw5dfflmx9tVXXyXHXnrppcn6zTffXFdP57pq696vXr267r992223JesrVqyo+2+3q6phd/f5w2xe34ReADQRp8sCQRB2IAjCDgRB2IEgCDsQBF9xbYELLrggWR89enSLOmkv1abW1qxZk6w/+OCDyXpXV1fF2iOPPJIcO3LkyGT9bMSRHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ69BRYsWFB0C4Xp6+urWFu5cmVy7PPPP5+s33XXXcn6unXrkvVoOLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBDMs9fI3euqSdLGjRuT9UcffbSeltrC5s2bk/V77rmnYu3o0aPJsffee2+yvmrVqmQd38SRHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCYJ69RmZWV02Sent7k/XHH388WV+0aFGyPmbMmIq17u7u5NgXXnghWX/rrbeS9Z6enmR9ypQpFWvz5s1Ljq02z44zU/XIbmadZvYHM9tnZt1mtjzbPt7MXjezD7Lrcc1vF0C9ankZf0LSz9z9akn/KGmpmV0j6WFJ2919qqTt2X0Abapq2N39kLvvzG5/JmmfpMskzZG0KXvYJkm3NKlHADk4ow/ozKxL0nRJf5I0yd0PSYP/IEiaWGHMYjMrm1l5YGCgwXYB1KvmsJvZaEm/k/RTd/9LrePcfa27l9y91NHRUU+PAHJQU9jN7LsaDPqv3H1rtvmwmU3O6pMl9TenRQB5qDr1ZoPzSusl7XP3Xw4pbZO0UNJT2fUrTenwHHDy5MlkvdrU2/r165P18ePHV6zt2bMnObZRs2fPTtZnzZpVsbZs2bK820FCLfPs10taIGmPme3Otq3QYMh/a2aLJB2QdHtTOgSQi6phd/c/Sqp01siP8m0HQLNwuiwQBGEHgiDsQBCEHQiCsANB8BXXGl177bUVazfeeGNy7BtvvNHQvqt9RTa1LHI1EycOe5bz3y1ZsiRZP5t/BjsajuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATz7DW6+OKLK9a2bNmSHPviiy8m6838yeQnnngiWb/77ruT9UsuuSTPdlAgjuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EIS5e8t2ViqVvFwut2x/QDSlUknlcnnYX4PmyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQVQNu5l1mtkfzGyfmXWb2fJs+2Nm1mdmu7PLTc1vF0C9avnxihOSfubuO81sjKR3zez1rLbK3f+9ee0ByEst67MfknQou/2Zme2TdFmzGwOQrzN6z25mXZKmS/pTtmmZmb1nZhvMbFyFMYvNrGxm5YGBgca6BVC3msNuZqMl/U7ST939L5LWSJoiaZoGj/y/GG6cu69195K7lzo6OhrvGEBdagq7mX1Xg0H/lbtvlSR3P+zuJ939lKR1kmY0r00Ajarl03iTtF7SPnf/5ZDtk4c8bK6kvfm3ByAvtXwaf72kBZL2mNnubNsKSfPNbJokl9Qj6SdN6A9ATmr5NP6Pkob7fuyr+bcDoFk4gw4IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxBES5dsNrMBSR8P2TRB0pGWNXBm2rW3du1Lord65dnb5e4+7O+/tTTs39q5WdndS4U1kNCuvbVrXxK91atVvfEyHgiCsANBFB32tQXvP6Vde2vXviR6q1dLeiv0PTuA1in6yA6gRQg7EEQhYTezWWb2P2b2oZk9XEQPlZhZj5ntyZahLhfcywYz6zezvUO2jTez183sg+x62DX2CuqtLZbxTiwzXuhzV/Ty5y1/z25mIyT9r6R/ltQr6R1J8939/ZY2UoGZ9UgquXvhJ2CY2Q8l/VXSi+7+/Wzbv0n61N2fyv6hHOfuD7VJb49J+mvRy3hnqxVNHrrMuKRbJP2rCnzuEn39i1rwvBVxZJ8h6UN33+/uf5P0G0lzCuij7bn7DkmfnrZ5jqRN2e1NGvyfpeUq9NYW3P2Qu+/Mbn8m6etlxgt97hJ9tUQRYb9M0sEh93vVXuu9u6Tfm9m7Zra46GaGMcndD0mD//NImlhwP6eruox3K522zHjbPHf1LH/eqCLCPtxSUu00/3e9u/9A0mxJS7OXq6hNTct4t8owy4y3hXqXP29UEWHvldQ55P73JH1SQB/DcvdPsut+SS+r/ZaiPvz1CrrZdX/B/fxdOy3jPdwy42qD567I5c+LCPs7kqaa2RVmNlLSPEnbCujjW8xsVPbBicxslKQfq/2Wot4maWF2e6GkVwrs5RvaZRnvSsuMq+DnrvDlz9295RdJN2nwE/n/k/RIET1U6OsfJP05u3QX3ZukzRp8WXdcg6+IFkm6RNJ2SR9k1+PbqLf/krRH0nsaDNbkgnq7QYNvDd+TtDu73FT0c5foqyXPG6fLAkFwBh0QBGEHgiDsQBCEHQiCsANBEHYgCMIOBPH/oSRW25O4eG4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit is:  0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAM+0lEQVR4nO3db6hc9Z3H8c9ntUFM+iCaqxts2MQYNFLctAxxwbW4RIP6wFilSyOULMqmgkIKFVb0QcUnyrJtaWSp3K6h6dK1FloxSNiNxKoUJHgjd01sXONqbPPHZEKUGgWj9373wT1ZrvHOmcnMmTlz7/f9gmFmzvece76MfnLOnN/M/BwRAjD3/UXdDQAYDMIOJEHYgSQIO5AEYQeSOHeQO1u0aFEsXbp0kLsEUjlw4ICOHz/umWo9hd32jZJ+IukcSf8WEY+Wrb906VKNjY31sksAJRqNRsta16fxts+R9K+SbpJ0paT1tq/s9u8B6K9e3rOvlvRWRLwdEack/UrSumraAlC1XsJ+iaQ/TXt+sFj2ObY32h6zPdZsNnvYHYBe9BL2mS4CfOGztxExGhGNiGiMjIz0sDsAvegl7AclLZn2/CuSDvfWDoB+6SXsr0haYXuZ7XmSvi1pWzVtAaha10NvEfGZ7Xsl/Zemht62RMTrlXUGoFI9jbNHxHZJ2yvqBUAf8XFZIAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuhpFldgmO3bt69l7frrry/ddnx8vLQ+MjLSTUu16instg9I+lDShKTPIqJRRVMAqlfFkf3vIuJ4BX8HQB/xnh1Iotewh6Qdtnfb3jjTCrY32h6zPdZsNnvcHYBu9Rr2ayLi65JuknSP7W+cuUJEjEZEIyIas/GiBjBX9BT2iDhc3B+T9LSk1VU0BaB6XYfd9nzbXz79WNJaSXuragxAtXq5Gn+xpKdtn/47/xER/1lJV32wf//+0vr7779fWl+9mpOW2WbXrl0ta2vWrBlgJ8Oh67BHxNuS/rrCXgD0EUNvQBKEHUiCsANJEHYgCcIOJJHmK647d+4srb/xxhuldYbehk9ElNbLhlvffPPNqtsZehzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJNOPsmzdvLq2vXbt2QJ2gKidPniytP/LIIy1rmzZtKt12Lv6qEkd2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUgizTj7xMRE3S2gYnfffXfX265cubLCTmYHjuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kMScGWc/fPhwaf3QoUMD6gSDcuLEia63veGGGyrsZHZoe2S3vcX2Mdt7py27wPZztvcX9wv72yaAXnVyGv9zSTeesex+STsjYoWkncVzAEOsbdgj4iVJZ54vrZO0tXi8VdKt1bYFoGrdXqC7OCKOSFJxf1GrFW1vtD1me6zZbHa5OwC96vvV+IgYjYhGRDTm4o/4AbNFt2E/anuxJBX3x6prCUA/dBv2bZI2FI83SHqmmnYA9EvbcXbbT0q6TtIi2wcl/UDSo5J+bfsuSX+U9K1+NtmJHTt2lNY//vjjAXWCqnz00Uel9T179nT9ty+88MKut52t2oY9Ita3KK2puBcAfcTHZYEkCDuQBGEHkiDsQBKEHUhiznzFde/eve1XKrFq1apqGkFlHnzwwdJ6u681X3XVVS1r8+bN66qn2YwjO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kMWfG2Xt19dVX193CrPTJJ5+U1nfv3t2yNjo6WrrtU0891VVPp23evLll7bzzzuvpb89GHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2QsffPBBbftu973sycnJ0vqLL77YsvbOO++Ubnvq1KnS+mOPPVZan5iYKK3Pnz+/ZW3t2rWl27YbC//0009L6ytXriytZ8ORHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSmDPj7Oeff35p3XZp/ZZbbimtX3755WfdU6defvnl0npElNbPPbf1f8YFCxaUbtvue/z33Xdfaf3aa68trZf9Hn/ZGLwkLVmypLTebkrnkZGR0no2bY/strfYPmZ777RlD9k+ZHu8uN3c3zYB9KqT0/ifS7pxhuU/johVxW17tW0BqFrbsEfES5JODKAXAH3UywW6e22/VpzmL2y1ku2NtsdsjzWbzR52B6AX3Yb9p5KWS1ol6YikH7ZaMSJGI6IREQ0umAD16SrsEXE0IiYiYlLSzyStrrYtAFXrKuy2F097+k1Jvc2XDKDv2o6z235S0nWSFtk+KOkHkq6zvUpSSDog6bv9a7EzDz/8cGl9+fLlpfUXXnihwm7OzooVK0rrd9xxR2n9sssua1lbtmxZVz0Nwvbt5YM47733Xmn9iiuuqLKdOa9t2CNi/QyLn+hDLwD6iI/LAkkQdiAJwg4kQdiBJAg7kMSc+YprOxs2bOipjuo9++yzPW1/5513VtRJDhzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJNOPsmHtuu+22uluYVTiyA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBJ8nx1DKyJK6++++25p/dJLL62ynVmv7ZHd9hLbv7O9z/brtjcVyy+w/Zzt/cX9wv63C6BbnZzGfybp+xGxUtLfSLrH9pWS7pe0MyJWSNpZPAcwpNqGPSKORMSrxeMPJe2TdImkdZK2FqttlXRrn3oEUIGzukBne6mkr0naJeniiDgiTf2DIOmiFttstD1me6zZbPbYLoBudRx22wsk/UbS9yLiz51uFxGjEdGIiMbIyEg3PQKoQEdht/0lTQX9lxHx22LxUduLi/piScf60yKAKnRyNd6SnpC0LyJ+NK20TdLpeY43SHqm+vaQme3S2+TkZOkNn9fJOPs1kr4jaY/t8WLZA5IelfRr23dJ+qOkb/WlQwCVaBv2iPi9JLcor6m2HQD9wsdlgSQIO5AEYQeSIOxAEoQdSIKvuGLWev7550vra9YwWDQdR3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJxdgytdj8ljbPDkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHbW5/fbbS+uPP/74gDrJgSM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTRdpzd9hJJv5D0l5ImJY1GxE9sPyTpHyU1i1UfiIjt/WoUc0+733VnjvVqdfKhms8kfT8iXrX9ZUm7bT9X1H4cEf/Sv/YAVKWT+dmPSDpSPP7Q9j5Jl/S7MQDVOqv37LaXSvqapF3Fonttv2Z7i+2FLbbZaHvM9liz2ZxpFQAD0HHYbS+Q9BtJ34uIP0v6qaTlklZp6sj/w5m2i4jRiGhERGNkZKT3jgF0paOw2/6SpoL+y4j4rSRFxNGImIiISUk/k7S6f20C6FXbsNu2pCck7YuIH01bvnjaat+UtLf69gBUpZOr8ddI+o6kPbbHi2UPSFpve5WkkHRA0nf70B+AinRyNf73kjxDiTF1YBbhE3RAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkHBGD25ndlPTutEWLJB0fWANnZ1h7G9a+JHrrVpW9/VVEzPj7bwMN+xd2bo9FRKO2BkoMa2/D2pdEb90aVG+cxgNJEHYgibrDPlrz/ssMa2/D2pdEb90aSG+1vmcHMDh1H9kBDAhhB5KoJey2b7T9P7bfsn1/HT20YvuA7T22x22P1dzLFtvHbO+dtuwC28/Z3l/czzjHXk29PWT7UPHajdu+uabeltj+ne19tl+3valYXutrV9LXQF63gb9nt32OpDcl3SDpoKRXJK2PiD8MtJEWbB+Q1IiI2j+AYfsbkk5K+kVEfLVY9s+STkTEo8U/lAsj4p+GpLeHJJ2sexrvYraixdOnGZd0q6R/UI2vXUlff68BvG51HNlXS3orIt6OiFOSfiVpXQ19DL2IeEnSiTMWr5O0tXi8VVP/swxci96GQkQciYhXi8cfSjo9zXitr11JXwNRR9gvkfSnac8Parjmew9JO2zvtr2x7mZmcHFEHJGm/ueRdFHN/Zyp7TTeg3TGNOND89p1M/15r+oI+0xTSQ3T+N81EfF1STdJuqc4XUVnOprGe1BmmGZ8KHQ7/Xmv6gj7QUlLpj3/iqTDNfQxo4g4XNwfk/S0hm8q6qOnZ9At7o/V3M//G6ZpvGeaZlxD8NrVOf15HWF/RdIK28tsz5P0bUnbaujjC2zPLy6cyPZ8SWs1fFNRb5O0oXi8QdIzNfbyOcMyjXeracZV82tX+/TnETHwm6SbNXVF/n8lPVhHDy36ulTSfxe31+vuTdKTmjqt+1RTZ0R3SbpQ0k5J+4v7C4aot3+XtEfSa5oK1uKaevtbTb01fE3SeHG7ue7XrqSvgbxufFwWSIJP0AFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEv8H3Hn9kA5jwPcAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit is:  4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAMe0lEQVR4nO3dYYhc9bnH8d8v3kZhUySa1YY0dNuqeKXYNAxB8FKUcqtGJFawNC9KFGkqKLRYQbEvEvBNuF5bCl4KiYampbUUWzWC3BsJBS1CySq5Gg1qKmmTGLITgtQg0uo+fbHHso07ZyZzzpkzyfP9wDIz5zlz/g+H/e2ZmXNm/44IATj7LWq7AQCjQdiBJAg7kARhB5Ig7EAS/zbKwZYtWxZTU1OjHBJI5eDBgzp+/LgXqlUKu+3rJf1E0jmSHo2ILWXrT01NaXp6usqQAEp0Op2etaFfxts+R9L/SLpB0hWS1tu+YtjtAWhWlffsayQdiIi3I+Jvkn4taV09bQGoW5Wwr5B0aN7jw8Wyf2F7o+1p29PdbrfCcACqqBL2hT4E+MS1txGxNSI6EdGZnJysMByAKqqE/bCklfMef1bSO9XaAdCUKmHfI+lS25+3vVjStyTtrKctAHUb+tRbRHxo+25J/6e5U2/bI+K12joDUKtK59kj4llJz9bUC4AGcbkskARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSVSaxRVo0rZt20rrd955Z2l9dna2Z+2NN94ofe5ll11WWj8TVQq77YOS3pP0kaQPI6JTR1MA6lfHkf3aiDhew3YANIj37EASVcMeknbZfsn2xoVWsL3R9rTt6W63W3E4AMOqGvarI2K1pBsk3WX7q6euEBFbI6ITEZ3JycmKwwEYVqWwR8Q7xe2MpCclramjKQD1Gzrstidsf/rj+5K+LmlfXY0BqFeVT+MvlvSk7Y+386uI+N9aukIKu3fvLq3fc889pfVFi4Z/YVr83qYydNgj4m1JX66xFwAN4tQbkARhB5Ig7EAShB1IgrADSfAVV7TmzTffLK1/8MEHI+okB47sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE59nRqNdff71nbfPmzZW2vXr16tL6rl27etYmJiYqjX0m4sgOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0lwnh2VHDhwoLS+du3anrUTJ05UGnvLli2l9fPPP7/S9s82HNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnOs6OSRx99tLR+6NChobd9yy23lNavvfbaobedUd8ju+3ttmds75u37ALbz9l+q7hd2mybAKoa5GX8zyRdf8qy+yXtjohLJe0uHgMYY33DHhHPSzr1usZ1knYU93dIurnetgDUbdgP6C6OiKOSVNxe1GtF2xttT9ue7na7Qw4HoKrGP42PiK0R0YmIzuTkZNPDAehh2LAfs71ckorbmfpaAtCEYcO+U9KG4v4GSU/X0w6ApvQ9z277cUnXSFpm+7CkTZK2SPqN7Tsk/UXSrU02ifa8//77pfWHHnqotL5oUe/jyYUXXlj63AcffLC0jtPTN+wRsb5H6Ws19wKgQVwuCyRB2IEkCDuQBGEHkiDsQBJ8xTW5d999t7S+bt26xsbuN2Xz5Zdf3tjYGXFkB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkOM+e3AsvvFBaf/HFFytt/9Zbe3/7+bbbbqu0bZwejuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATn2c9ye/bsKa1v2LChtN7PTTfdVFrftm1bz9p5551XaWycHo7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE59nPAmX/+/2qq65qdOxLLrmktD4xMdHo+Bhc3yO77e22Z2zvm7dss+0jtvcWP2ubbRNAVYO8jP+ZpOsXWP7jiFhV/Dxbb1sA6tY37BHxvKQTI+gFQIOqfEB3t+1Xipf5S3utZHuj7Wnb091ut8JwAKoYNuw/lfRFSaskHZX0cK8VI2JrRHQiojM5OTnkcACqGirsEXEsIj6KiFlJ2yStqbctAHUbKuy2l897+A1J+3qtC2A89D3PbvtxSddIWmb7sKRNkq6xvUpSSDoo6bvNtYh+Hn6457soLVrU7HVT9913X6PbR336hj0i1i+w+LEGegHQIC6XBZIg7EAShB1IgrADSRB2IAm+4noGOHLkSGn9iSeeaGzs22+/vbTOVZFnDo7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE59nPAJ1Op7R+/Pjxobd93XXXldYfeeSRobeN8cKRHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4Dz7GWBmZqa0XuXfRff7V9CLFy8eetsYLxzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJzrOPgXvvvbe0Pjs729jYV155ZWPbxnjpe2S3vdL2723vt/2a7e8Vyy+w/Zztt4rbpc23C2BYg7yM/1DSDyLi3yVdJeku21dIul/S7oi4VNLu4jGAMdU37BFxNCJeLu6/J2m/pBWS1knaUay2Q9LNDfUIoAan9QGd7SlJX5H0R0kXR8RRae4PgqSLejxno+1p29PdbrdiuwCGNXDYbS+R9FtJ34+Ivw76vIjYGhGdiOgwCSDQnoHCbvtTmgv6LyPid8XiY7aXF/Xlksq/mgWgVX1Pvdm2pMck7Y+IH80r7ZS0QdKW4vbpRjo8C1SdcrnfV1jPPffcnrVNmzaVPndiYqK0jrPHIOfZr5b0bUmv2t5bLHtAcyH/je07JP1F0q2NdAigFn3DHhF/kOQe5a/V2w6ApnC5LJAEYQeSIOxAEoQdSIKwA0nwFdcROHnyZGm933n4fqampnrW+v2raOTBkR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS4PvsI7BixYrS+o033lhaf+aZZ+psB0lxZAeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAaZn32lpJ9L+oykWUlbI+IntjdL+o6kbrHqAxHxbFONnsmWLFlSWn/qqadG0whSG+Simg8l/SAiXrb9aUkv2X6uqP04Iv67ufYA1GWQ+dmPSjpa3H/P9n5J5ZeEARg7p/We3faUpK9I+mOx6G7br9jebntpj+dstD1te7rb7S60CoARGDjstpdI+q2k70fEXyX9VNIXJa3S3JH/4YWeFxFbI6ITEZ3JycnqHQMYykBht/0pzQX9lxHxO0mKiGMR8VFEzEraJmlNc20CqKpv2G1b0mOS9kfEj+YtXz5vtW9I2ld/ewDqMsin8VdL+rakV23vLZY9IGm97VWSQtJBSd9toD8ANRnk0/g/SPICJc6pA2cQrqADkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4k4YgY3WB2V9Kf5y1aJun4yBo4PePa27j2JdHbsOrs7XMRseD/fxtp2D8xuD0dEZ3WGigxrr2Na18SvQ1rVL3xMh5IgrADSbQd9q0tj19mXHsb174kehvWSHpr9T07gNFp+8gOYEQIO5BEK2G3fb3tN2wfsH1/Gz30Yvug7Vdt77U93XIv223P2N43b9kFtp+z/VZxu+Acey31ttn2kWLf7bW9tqXeVtr+ve39tl+z/b1ieav7rqSvkey3kb9nt32OpDcl/aekw5L2SFofEa+PtJEebB+U1ImI1i/AsP1VSScl/TwivlQs+y9JJyJiS/GHcmlE3DcmvW2WdLLtabyL2YqWz59mXNLNkm5Ti/uupK9vagT7rY0j+xpJByLi7Yj4m6RfS1rXQh9jLyKel3TilMXrJO0o7u/Q3C/LyPXobSxExNGIeLm4/56kj6cZb3XflfQ1Em2EfYWkQ/MeH9Z4zfceknbZfsn2xrabWcDFEXFUmvvlkXRRy/2cqu803qN0yjTjY7Pvhpn+vKo2wr7QVFLjdP7v6ohYLekGSXcVL1cxmIGm8R6VBaYZHwvDTn9eVRthPyxp5bzHn5X0Tgt9LCgi3iluZyQ9qfGbivrYxzPoFrczLffzT+M0jfdC04xrDPZdm9OftxH2PZIutf1524slfUvSzhb6+ATbE8UHJ7I9IenrGr+pqHdK2lDc3yDp6RZ7+RfjMo13r2nG1fK+a33684gY+Y+ktZr7RP5Pkn7YRg89+vqCpP8vfl5ruzdJj2vuZd3fNfeK6A5JF0raLemt4vaCMertF5JelfSK5oK1vKXe/kNzbw1fkbS3+Fnb9r4r6Wsk+43LZYEkuIIOSIKwA0kQdiAJwg4kQdiBJAg7kARhB5L4Byx6uPlgroKuAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit is:  1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAANzklEQVR4nO3da4hcdZrH8d/PrImYGdBs2hicuO0OQTcom5EiiMrgMuzE6AujMFFBjXiJiMIoA94WnEB8IcvOeEEZyazRzOJ6wRk1L8SdIIJEZEgp0cQNu3FD7yRj7LT4wguE2STPvujjbhu7/tWp2ynzfD/QVNV56tR5qOTXp/r8T52/I0IAjn3H1d0AgMEg7EAShB1IgrADSRB2IIm/GOTG5s+fH6Ojo4PcJJDK2NiYPvnkE09X6yrsti+W9IikWZL+OSIeLD1/dHRUzWazm00CKGg0Gi1rHX+Mtz1L0uOSVkhaIulq20s6fT0A/dXN3+zLJH0YEbsj4s+SnpN0WW/aAtBr3YT9NEl7pjzeWy37GttrbDdtNycmJrrYHIBudBP26Q4CfOPc24hYHxGNiGiMjIx0sTkA3egm7HslLZry+HuSPuquHQD90k3Yt0pabPsM27MlXSVpU2/aAtBrHQ+9RcRB27dL+jdNDr1tiIgPetYZgJ7qapw9Il6V9GqPegHQR5wuCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiiqymbbY9J+lzSIUkHI6LRi6YA9F5XYa/8XUR80oPXAdBHfIwHkug27CHp97bfsb1muifYXmO7abs5MTHR5eYAdKrbsF8QEedKWiHpNts/PPIJEbE+IhoR0RgZGelycwA61VXYI+Kj6na/pJckLetFUwB6r+Ow255r+7tf3Zf0Y0k7etUYgN7q5mj8Akkv2f7qdf41Il7rSVcAeq7jsEfEbkl/28NeAPQRQ29AEoQdSIKwA0kQdiAJwg4k0YsvwmCIjY2NFetPP/10sf7aa+XR1K1btx5lR//vmWeeKdYXLVpUrG/evLlYv/7661vWRkdHi+sei9izA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLMfA956662WtVWrVhXXHR8fL9Yjoli/4oorivU9e/a0rF1zzTXFddtp11vpMmiPP/54V9v+NmLPDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMM4+BA4fPlyst/tO+qWXXtqy9sUXXxTXXblyZbH+wAMPFOuLFy8u1g8dOtSydsMNNxTXfe6554r1ds4///yu1j/WsGcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZx8Cb7zxRrG+fPnyjl/7yiuvLNY3bNhQrM+ZM6fjbUvSli1bWta6HUdvd+33yy+/vKvXP9a03bPb3mB7v+0dU5bNs73Z9q7q9uT+tgmgWzP5GP+0pIuPWHaPpNcjYrGk16vHAIZY27BHxJuSPj1i8WWSNlb3N0pa2du2APRapwfoFkTEPkmqbk9p9UTba2w3bTdL1wQD0F99PxofEesjohERjZGRkX5vDkALnYZ93PZCSapu9/euJQD90GnYN0laXd1fLemV3rQDoF/ajrPbflbSRZLm294r6eeSHpT0gu0bJf1R0k/62eS33aOPPlqs33nnncW67WL9/vvvb1m7++67i+t2O47ezh133NG3137++eeL9RNPPLFv2/42ahv2iLi6RelHPe4FQB9xuiyQBGEHkiDsQBKEHUiCsANJ8BXXHnjiiSeK9XZDa+2Gv6666qpi/d57721ZO/7444vrtnPw4MFi/b333ivWd+3a1bLWbsrldkOWjUajWMfXsWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ5+hAwcOtKytW7euuG67r6i2G0dvd7nnbnz66ZGXF/y6dpeibncZ7JJbbrmlWL/55ps7fm18E3t2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfYZOnToUMva+Ph4V6/90EMPFetffvllsf7iiy+2rLW73PLbb79drH/22WfFertzCEr1m266qbju7Nmzi3UcHfbsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+wzNGvWrJa1U089tbjuxx9/XKzPmzevWG83lt2N008/vVg/6aSTivU9e/YU6wsWLGhZO/fcc4vrorfa7tltb7C93/aOKcvW2v6T7W3VzyX9bRNAt2byMf5pSRdPs/yhiFha/bza27YA9FrbsEfEm5LK1y4CMPS6OUB3u+33q4/5J7d6ku01tpu2mxMTE11sDkA3Og37ryR9X9JSSfsk/aLVEyNifUQ0IqIxMjLS4eYAdKujsEfEeEQciojDkn4taVlv2wLQax2F3fbCKQ8vl7Sj1XMBDIe24+y2n5V0kaT5tvdK+rmki2wvlRSSxiSVLwB+DDjhhBNa1rZs2VJc97zzzivW2x3LWLJkSbF+7bXXtqxdd911xXXnzp3b8WtL7cfZb7311mIdg9M27BFx9TSLn+xDLwD6iNNlgSQIO5AEYQeSIOxAEoQdSIKvuPbA6Ohosd7uK6512rVrV7H+8ssvF+vHHVfeX5x11llH2xL6hD07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHtyBw4cKNbbjaO3u8z1ihUrjron9Ad7diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnH25M4555y6W8CAsGcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ09u+/btdbeAAWm7Z7e9yPYbtnfa/sD2T6vl82xvtr2ruj25/+0C6NRMPsYflPSziPgbSedJus32Ekn3SHo9IhZLer16DGBItQ17ROyLiHer+59L2inpNEmXSdpYPW2jpJV96hFADxzVATrbo5J+IOkPkhZExD5p8heCpFNarLPGdtN2c2Jiost2AXRqxmG3/R1Jv5V0R0R8NtP1ImJ9RDQiojEyMtJJjwB6YEZht328JoP+TET8rlo8bnthVV8oaX9/WgTQC22H3jx5reAnJe2MiF9OKW2StFrSg9XtK33pEH21e/fuulvAgMxknP0CSddK2m57W7XsPk2G/AXbN0r6o6Sf9KVDAD3RNuwRsUVSq5kAftTbdgD0C6fLAkkQdiAJwg4kQdiBJAg7kARfcU1u2bJlxfrhw4eL9XZTOmN48C8FJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzp7cwoULi/Wzzz67WN+5c2exPj4+3rJ2xhlnFNdFb7FnB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGdH0cMPP1ysL1++vFi/6667WtYee+yx4roLFiwo1nF02LMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBIzmZ99kaTfSDpV0mFJ6yPiEdtrJd0saaJ66n0R8Wq/GkU9LrzwwmJ91apVxfoLL7zQsjZ//vziuo888kixPnv27GIdXzeTk2oOSvpZRLxr+7uS3rG9uao9FBH/1L/2APTKTOZn3ydpX3X/c9s7JZ3W78YA9NZR/c1ue1TSDyT9oVp0u+33bW+wfXKLddbYbtpuTkxMTPcUAAMw47Db/o6k30q6IyI+k/QrSd+XtFSTe/5fTLdeRKyPiEZENEZGRrrvGEBHZhR228drMujPRMTvJCkixiPiUEQclvRrSeUZAgHUqm3YbVvSk5J2RsQvpyyfelnSyyXt6H17AHplJkfjL5B0raTttrdVy+6TdLXtpZJC0pikW/rQH2o2Z86cYv2pp54q1s8888yWtXXr1hXXXbt2bbHOV2CPzkyOxm+R5GlKjKkD3yKcQQckQdiBJAg7kARhB5Ig7EAShB1IwhExsI01Go1oNpsD2x6QTaPRULPZnG6onD07kAVhB5Ig7EAShB1IgrADSRB2IAnCDiQx0HF22xOS/nvKovmSPhlYA0dnWHsb1r4keutUL3v7q4iY9vpvAw37NzZuNyOiUVsDBcPa27D2JdFbpwbVGx/jgSQIO5BE3WFfX/P2S4a1t2HtS6K3Tg2kt1r/ZgcwOHXv2QEMCGEHkqgl7LYvtv0ftj+0fU8dPbRie8z2dtvbbNf65ftqDr39tndMWTbP9mbbu6rbaefYq6m3tbb/VL1322xfUlNvi2y/YXun7Q9s/7RaXut7V+hrIO/bwP9mtz1L0n9K+ntJeyVtlXR1RPz7QBtpwfaYpEZE1H4Chu0fSvpC0m8i4uxq2T9K+jQiHqx+UZ4cEXcPSW9rJX1R9zTe1WxFC6dOMy5ppaTrVeN7V+hrlQbwvtWxZ18m6cOI2B0Rf5b0nKTLauhj6EXEm5I+PWLxZZI2Vvc3avI/y8C16G0oRMS+iHi3uv+5pK+mGa/1vSv0NRB1hP00SXumPN6r4ZrvPST93vY7ttfU3cw0FkTEPmnyP4+kU2ru50htp/EepCOmGR+a966T6c+7VUfYp7s+1jCN/10QEedKWiHpturjKmZmRtN4D8o004wPhU6nP+9WHWHfK2nRlMffk/RRDX1MKyI+qm73S3pJwzcV9fhXM+hWt/tr7uf/DNM03tNNM64heO/qnP68jrBvlbTY9hm2Z0u6StKmGvr4BttzqwMnsj1X0o81fFNRb5K0urq/WtIrNfbyNcMyjXeracZV83tX+/TnETHwH0mXaPKI/H9J+oc6emjR119Leq/6+aDu3iQ9q8mPdf+jyU9EN0r6S0mvS9pV3c4bot7+RdJ2Se9rMlgLa+rtQk3+afi+pG3VzyV1v3eFvgbyvnG6LJAEZ9ABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBL/C0VtGPnOQShRAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The digit is:  9\n"
     ]
    }
   ],
   "source": [
    "# visualize the image and label from the training set\n",
    "for idx in range(5):\n",
    "    example_img = np.resize(x_train[idx],(28,28))\n",
    "    plt.imshow(example_img,cmap=\"Greys\")\n",
    "    plt.show()\n",
    "    print ('The digit is: ',y_train[idx].argmax())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are now ready to fit your model to MNIST. Start by considering a simple one hidden layer MLP with the following specifications:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "dimensions =  (28*28,     50,        10)\n",
    "activations = (      ReLU,  Sigmoid)\n",
    "epochs = 100\n",
    "batch_size = 128\n",
    "learning_rate = 1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the Network class implemented above, initialize the network and fit it to the dataset using the MSE loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss at epoch 10: 0.01413166036948466\n",
      "Loss at epoch 20: 0.011675064830289648\n",
      "Loss at epoch 30: 0.010268625246906846\n",
      "Loss at epoch 40: 0.009542317245590013\n",
      "Loss at epoch 50: 0.00915979217050333\n",
      "Loss at epoch 60: 0.008417085255329375\n",
      "Loss at epoch 70: 0.008069097288688055\n",
      "Loss at epoch 80: 0.007718254587191162\n",
      "Loss at epoch 90: 0.007452544924186071\n",
      "Loss at epoch 100: 0.007174542517659386\n"
     ]
    }
   ],
   "source": [
    "# initialize  network structure\n",
    "nn = Network(dimensions, activations)\n",
    "# fit network to training data\n",
    "nn.fit(x_train, y_train, loss=MSE, epochs=epochs, batch_size=batch_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to previous exercise sessions, let's define a metric to measure the accuracy of our predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(prediction, ground_truth):\n",
    "    match = prediction.argmax(axis=1) == ground_truth.argmax(axis=1)\n",
    "    return float(np.sum(match)) / ground_truth.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then look at how our network is performing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loss on TRAIN split: 249.73988169103737\n",
      "Accuracy on TRAIN split: 0.9585333333333333\n",
      "------------------------\n",
      "Loss on TEST split: 252.42316767701908\n",
      "Accuracy on TEST split: 0.9442\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAANQklEQVR4nO3df4hd9ZnH8c/H2UYw0RjN6A7paNIgsrqyprnEhSxFKRbjP0bQpUFiFmRTQaGFIivZP6qgRJZtS/5YAnETmq5ZpWCDgnGthIpUoTjRaKIhG1fGNs2QTAiSKYLZJM/+Mcdlksw99+aec3/o837BcO89zzn3PB785Jy53zP364gQgK+/S/rdAIDeIOxAEoQdSIKwA0kQdiCJv+jlzhYuXBiLFy/u5S6BVMbHx3X8+HHPVqsUdtt3SdokaUjSv0fEM2XrL168WGNjY1V2CaBEo9FoWuv4Mt72kKR/k7RK0k2S1ti+qdP3A9BdVX5nXyHp44j4JCJOSXpB0j31tAWgblXCvkjSH2e8PlwsO4ft9bbHbI9NTk5W2B2AKqqEfbYPAS649zYitkREIyIaw8PDFXYHoIoqYT8saXTG629KOlKtHQDdUiXs70i6wfYS23MkfV/Sy/W0BaBuHQ+9RcRp249Kek3TQ2/bIuLD2joDUKtK4+wRsUvSrpp6AdBF3C4LJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSKLSlM22xyVNSToj6XRENOpoCkD9KoW9cEdEHK/hfQB0EZfxQBJVwx6SfmN7j+31s61ge73tMdtjk5OTFXcHoFNVw74yIr4taZWkR2x/5/wVImJLRDQiojE8PFxxdwA6VSnsEXGkeDwmaaekFXU0BaB+HYfd9lzbl3/5XNL3JO2vqzEA9aryafy1knba/vJ9/jMi/quWrgDUruOwR8Qnkv6mxl4AdBFDb0AShB1IgrADSRB2IAnCDiRRxx/CpDc1NVVaX7NmTWl97dq1pfV77723tD5nzpzSehWt/tsmJiZK688991zT2lNPPVW67aFDh0rrS5cuLa3jXJzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlr8N5775XWd+3aVVp/9dVXS+sLFiword9///1Nazt37izdNiJK61988UVp/fPPPy+tnzlzpmmt+PPoppYvX15af+utt0rrN998c2k9G87sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+xtOnv2bNPaG2+80bd9S9LWrVub1kZGRkq3nT9/fmn9kkvKzwcPPPBAaX3evHlNa5s3by7ddv/+8mkIDh48WFpnnP1cnNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2ds0Pj7etPbkk09Weu9XXnmltH7LLbeU1j/77LOmtSVLlpRue9lll5XWu+ntt98urbcaZ8fFaXlmt73N9jHb+2csu8r267YPFY/l364AoO/auYz/haS7zlv2uKTdEXGDpN3FawADrGXYI+JNSSfOW3yPpO3F8+2SVtfbFoC6dfoB3bURMSFJxeM1zVa0vd72mO2xycnJDncHoKqufxofEVsiohERjeHh4W7vDkATnYb9qO0RSSoej9XXEoBu6DTsL0taVzxfJ+mletoB0C0tx9ltPy/pdkkLbR+W9BNJz0j6le2HJP1BUvMvLv+a2LhxY8fb3nHHHaX1O++8s7Q+NDRUWl+0aNFF94R8WoY9ItY0KX235l4AdBG3ywJJEHYgCcIOJEHYgSQIO5AEf+Lapk8//bRpbXR0tHTbF198sbTeamjtq6xsyuYjR46UbttqOulVq1Z11FNWnNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2dv07LPPNq1NTU2VbnvllVfW3M1Xx0cffdS01mqq66uvvrq03mo6aZyLowUkQdiBJAg7kARhB5Ig7EAShB1IgrADSTDO3qbrr7++3y18JT399NMdb7thw4bS+qWXXtrxe2fEmR1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkmCcHZWcOHGitP7aa691/N6NRqPjbXGhlmd229tsH7O9f8ayJ2z/yfbe4ufu7rYJoKp2LuN/IemuWZb/PCJuLX521dsWgLq1DHtEvCmp/FoNwMCr8gHdo7Y/KC7zFzRbyfZ622O2xyYnJyvsDkAVnYZ9s6Slkm6VNCHpp81WjIgtEdGIiMbw8HCHuwNQVUdhj4ijEXEmIs5KelbSinrbAlC3jsJue2TGy3sl7W+2LoDB0HKc3fbzkm6XtND2YUk/kXS77VslhaRxST/oXosYZC+88EJp/eTJkx2/97JlyzreFhdqGfaIWDPL4q1d6AVAF3G7LJAEYQeSIOxAEoQdSIKwA0nwJ66oZPPmzaX1iGhae/jhh0u3vfzyyzvqCbPjzA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOjlL79u0rrR88eLC0fsUVVzStbdy4saOe0BnO7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPsyZ06daq0vnr16tL6mTNnSuuPPfZY09r8+fNLt0W9OLMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMsyf3/vvvl9bHx8dL60NDQ6X1Bx988GJbQpe0PLPbHrX9W9sHbH9o+4fF8qtsv277UPG4oPvtAuhUO5fxpyX9OCL+StLfSnrE9k2SHpe0OyJukLS7eA1gQLUMe0RMRMS7xfMpSQckLZJ0j6TtxWrbJa3uUo8AanBRH9DZXixpmaTfS7o2Iiak6X8QJF3TZJv1tsdsj01OTlZsF0Cn2g677XmSXpT0o4g42e52EbElIhoR0RgeHu6kRwA1aCvstr+h6aDviIhfF4uP2h4p6iOSjnWnRQB1aDn0ZtuStko6EBE/m1F6WdI6Sc8Ujy91pUN01Y4dOyptf99995XWR0dHK70/6tPOOPtKSWsl7bO9t1i2QdMh/5XthyT9QdL9XekQQC1ahj0ififJTcrfrbcdAN3C7bJAEoQdSIKwA0kQdiAJwg4kwZ+4Jrdp06bS+vRtFs2tXbu2znbQRZzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtmTazWO3qp+22231dkOuogzO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTj719yRI0cqbb9y5crS+vz58yu9P3qHMzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJNHO/Oyjkn4p6S8lnZW0JSI22X5C0j9KmixW3RARu7rVKDpz+vTpStvPnTu3tD40NFTp/dE77dxUc1rSjyPiXduXS9pj+/Wi9vOI+NfutQegLu3Mzz4haaJ4PmX7gKRF3W4MQL0u6nd224slLZP0+2LRo7Y/sL3N9oIm26y3PWZ7bHJycrZVAPRA22G3PU/Si5J+FBEnJW2WtFTSrZo+8/90tu0iYktENCKiMTw8XL1jAB1pK+y2v6HpoO+IiF9LUkQcjYgzEXFW0rOSVnSvTQBVtQy7p79edKukAxHxsxnLR2asdq+k/fW3B6Au7Xwav1LSWkn7bO8tlm2QtMb2rZJC0rikH3ShP1R03XXXldZvvPHG0vrhw4dL63v27CmtL1++vLSO3mnn0/jfSZrty8MZUwe+QriDDkiCsANJEHYgCcIOJEHYgSQIO5AEXyWd3IEDB/rdAnqEMzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJOGI6N3O7ElJn85YtFDS8Z41cHEGtbdB7Uuit07V2dv1ETHr97/1NOwX7Nwei4hG3xooMai9DWpfEr11qle9cRkPJEHYgST6HfYtfd5/mUHtbVD7kuitUz3pra+/swPonX6f2QH0CGEHkuhL2G3fZfug7Y9tP96PHpqxPW57n+29tsf63Ms228ds75+x7Crbr9s+VDzOOsden3p7wvafimO31/bdfept1PZvbR+w/aHtHxbL+3rsSvrqyXHr+e/stock/bekOyUdlvSOpDUR8VFPG2nC9rikRkT0/QYM29+R9GdJv4yIvy6W/YukExHxTPEP5YKI+KcB6e0JSX/u9zTexWxFIzOnGZe0WtI/qI/HrqSvv1cPjls/zuwrJH0cEZ9ExClJL0i6pw99DLyIeFPSifMW3yNpe/F8u6b/Z+m5Jr0NhIiYiIh3i+dTkr6cZryvx66kr57oR9gXSfrjjNeHNVjzvYek39jeY3t9v5uZxbURMSFN/88j6Zo+93O+ltN499J504wPzLHrZPrzqvoR9tmmkhqk8b+VEfFtSaskPVJcrqI9bU3j3SuzTDM+EDqd/ryqfoT9sKTRGa+/KelIH/qYVUQcKR6PSdqpwZuK+uiXM+gWj8f63M//G6RpvGebZlwDcOz6Of15P8L+jqQbbC+xPUfS9yW93Ic+LmB7bvHBiWzPlfQ9Dd5U1C9LWlc8XyfppT72co5Bmca72TTj6vOx6/v05xHR8x9Jd2v6E/n/kfTP/eihSV/fkvR+8fNhv3uT9LymL+v+V9NXRA9JulrSbkmHiserBqi3/5C0T9IHmg7WSJ96+ztN/2r4gaS9xc/d/T52JX315LhxuyyQBHfQAUkQdiAJwg4kQdiBJAg7kARhB5Ig7EAS/wfQKfKUIfkCXwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  7\n",
      "Ground truth annotation:  7\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAANSklEQVR4nO3db6xU9Z3H8c8HLERtJVCuiFaXboPJkjVL8UpM3KCGbFUSg31QLTENG03oA0xa08TV+gD0AZLNtlXD2kiVgBvWppEaMVG3hDQqiUEvhioucWXJXUoF7kWi2MRYL3z3wT1sLnjnzGXOmT/L9/1KJjNzvnPO+WZyP3Nmzm/m/hwRAnDum9TtBgB0BmEHkiDsQBKEHUiCsANJnNfJnc2cOTPmzJnTyV0CqQwODuro0aMer1Yp7LZvlvSYpMmSnoqItWWPnzNnjgYGBqrsEkCJ/v7+hrWW38bbnizpXyXdImmepGW257W6PQDtVeUz+0JJ+yJif0T8RdKvJS2tpy0AdasS9ssk/XHM/YPFstPYXmF7wPbA8PBwhd0BqKJK2Mc7CfCl795GxPqI6I+I/r6+vgq7A1BFlbAflHT5mPvfkPRhtXYAtEuVsL8laa7tb9qeIun7krbW0xaAurU89BYRI7bvkfQfGh162xAR79XWGYBaVRpnj4iXJL1UUy8A2oivywJJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kERH/5U08hkZGWlYW7x4cem6U6dOLa2vW7eutH7llVeW1rPhyA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOjra67777GtZef/31Stteu7Z00mBt2LCh0vbPNRzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJxtlR6vjx46X1lStXltY3b97c8r4ffPDB0vqqVata3nZGlcJue1DSp5JOSBqJiP46mgJQvzqO7DdGxNEatgOgjfjMDiRRNewh6Xe2d9leMd4DbK+wPWB7YHh4uOLuALSqativi4gFkm6RtNL2ojMfEBHrI6I/Ivr7+voq7g5AqyqFPSI+LK6HJD0vaWEdTQGoX8tht32h7a+dui3pO5L21NUYgHpVORs/S9Lztk9t598j4pVaukLHRERpffXq1aX1KuPoZb91l6SHHnqotD5pEueXz0bLYY+I/ZL+rsZeALQRL41AEoQdSIKwA0kQdiAJwg4kwU9ck3v55ZdL648++mil7c+dO7dh7YEHHihdl6G1evFsAkkQdiAJwg4kQdiBJAg7kARhB5Ig7EASjLOf4w4fPlxaf/jhhyttv9lPZN94442GtWnTplXaN84OR3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9nPcq6++Wlp/8803K23/3nvvLa1Pnz690vZRH47sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+zngJMnTzasbdmypdK2L7rootJ6s//9XkzpjR7Q9Mhue4PtIdt7xiybYXub7Q+Ka745AfS4ibyN3yjp5jOW3S9pe0TMlbS9uA+ghzUNe0S8JunYGYuXStpU3N4k6bZ62wJQt1ZP0M2KiEOSVFxf3OiBtlfYHrA9MDw83OLuAFTV9rPxEbE+Ivojor+vr6/duwPQQKthP2J7tiQV10P1tQSgHVoN+1ZJy4vbyyW9UE87ANql6Ti77Wcl3SBppu2DklZJWivpN7bvlnRA0vfa2STKff755w1rzz33XKVtL1y4sLTOR7P/P5qGPSKWNSgtrrkXAG3E12WBJAg7kARhB5Ig7EAShB1Igp+4ngMee+yxltdt9hPUNWvWtLxt9BaO7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPs54Cnnnqq5XXPO6/8T2DevHml9VdeeaW0Pn/+/Ia1Sy65pHRd1IsjO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTh7cl988UVpfdq0aaX1EydOlNYnTWp8PNm5c2fpuldffXVpHWeHIzuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4O0rNmDGjtH777beX1p944omGtVtvvbV03cHBwdL6lClTSus4XdMju+0Ntods7xmzbLXtP9neXVyWtLdNAFVN5G38Rkk3j7P8FxExv7i8VG9bAOrWNOwR8ZqkYx3oBUAbVTlBd4/td4q3+dMbPcj2CtsDtgeGh4cr7A5AFa2G/ZeSviVpvqRDkn7W6IERsT4i+iOiv6+vr8XdAaiqpbBHxJGIOBERJyX9StLCetsCULeWwm579pi735W0p9FjAfSGpuPstp+VdIOkmbYPSlol6Qbb8yWFpEFJP2xfizhw4EBp/dix1s+fLllSPmpaNk4uSVdccUVp/cknn2xYO3z4cOm677//fmn9qquuKq3jdE3DHhHLxln8dBt6AdBGfF0WSIKwA0kQdiAJwg4kQdiBJPiJaw/47LPPSuvXXnttaf3jjz9ued+PPPJIab3Z0Nonn3xSWj958uRZ93TKrl27SusMvZ0djuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATj7D1gZGSktP7RRx+V1sumRW42zt1s281+Pnv99deX1sv2f80115Sue8cdd5TWcXY4sgNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyz94DJkyeX1rdt21Zav+uuuxrW9u/fX7ruM888U1rfsWNHaX3fvn2l9TJr1qwprZ9//vktbxtfxpEdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgnL0HXHDBBaX1RYsWldZvvPHGhrVm4+wbN24srVd16aWXNqwtWLCgrfvG6Zoe2W1fbvv3tvfafs/2j4rlM2xvs/1BcT29/e0CaNVE3saPSPpJRPyNpGslrbQ9T9L9krZHxFxJ24v7AHpU07BHxKGIeLu4/amkvZIuk7RU0qbiYZsk3damHgHU4KxO0NmeI+nbknZKmhURh6TRFwRJFzdYZ4XtAdsDw8PDFdsF0KoJh932VyVtkfTjiDg+0fUiYn1E9EdEf19fXys9AqjBhMJu+ysaDfrmiPhtsfiI7dlFfbakofa0CKAOTYfebFvS05L2RsTPx5S2SlouaW1x/UJbOkRT69ata1h78cUXS9cdGmrva/TAwEDD2vTpDOB00kTG2a+T9ANJ79reXSz7qUZD/hvbd0s6IOl7bekQQC2ahj0idkhyg/LietsB0C58XRZIgrADSRB2IAnCDiRB2IEk+InrOWDq1KkNa5s2bWpYk6Q777yztD5lypTS+uOPP15anzVrVmkdncORHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYJz9HHfTTTeV1o8ePdqhTtBtHNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgiaZht3257d/b3mv7Pds/Kpavtv0n27uLy5L2twugVRP55xUjkn4SEW/b/pqkXba3FbVfRMS/tK89AHWZyPzshyQdKm5/anuvpMva3RiAep3VZ3bbcyR9W9LOYtE9tt+xvcH29AbrrLA9YHtgeHi4WrcAWjbhsNv+qqQtkn4cEccl/VLStyTN1+iR/2fjrRcR6yOiPyL6+/r6qncMoCUTCrvtr2g06Jsj4reSFBFHIuJERJyU9CtJC9vXJoCqJnI23pKelrQ3In4+ZvnsMQ/7rqQ99bcHoC4TORt/naQfSHrX9u5i2U8lLbM9X1JIGpT0wzb0B6AmEzkbv0OSxym9VH87ANqFb9ABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeScER0bmf2sKT/GbNopqSjHWvg7PRqb73al0Rvraqzt7+KiHH//1tHw/6lndsDEdHftQZK9GpvvdqXRG+t6lRvvI0HkiDsQBLdDvv6Lu+/TK/21qt9SfTWqo701tXP7AA6p9tHdgAdQtiBJLoSdts3237f9j7b93ejh0ZsD9p+t5iGeqDLvWywPWR7z5hlM2xvs/1BcT3uHHtd6q0npvEumWa8q89dt6c/7/hndtuTJf2XpH+QdFDSW5KWRcR/drSRBmwPSuqPiK5/AcP2Ikl/lvRMRPxtseyfJR2LiLXFC+X0iPinHulttaQ/d3sa72K2otljpxmXdJukf1QXn7uSvm5XB563bhzZF0raFxH7I+Ivkn4taWkX+uh5EfGapGNnLF4qaVNxe5NG/1g6rkFvPSEiDkXE28XtTyWdmma8q89dSV8d0Y2wXybpj2PuH1Rvzfcekn5ne5ftFd1uZhyzIuKQNPrHI+niLvdzpqbTeHfSGdOM98xz18r051V1I+zjTSXVS+N/10XEAkm3SFpZvF3FxExoGu9OGWea8Z7Q6vTnVXUj7AclXT7m/jckfdiFPsYVER8W10OSnlfvTUV95NQMusX1UJf7+T+9NI33eNOMqweeu25Of96NsL8laa7tb9qeIun7krZ2oY8vsX1hceJEti+U9B313lTUWyUtL24vl/RCF3s5Ta9M491omnF1+bnr+vTnEdHxi6QlGj0j/9+SHuxGDw36+mtJfygu73W7N0nPavRt3RcafUd0t6SvS9ou6YPiekYP9fZvkt6V9I5GgzW7S739vUY/Gr4jaXdxWdLt566kr448b3xdFkiCb9ABSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBL/C9Rl+Vg3kdI6AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  6\n",
      "Ground truth annotation:  6\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPI0lEQVR4nO3df6xU9ZnH8c+j4i+oinsvSASWUogs/gIy4iYalZhVICZAYosmEpoY6R8YWlOj2NVoYjRqtjQSfxC6ghRZSJNa5Q91MdiEVJPqiCwXJLu6eLcgV7mIWksIFfvsH3c0V5zznXHOzJyR5/1KbmbueeY752G4n3vmznfOfM3dBeD4d0LRDQBoD8IOBEHYgSAIOxAEYQeCOKmdO+vq6vJx48a1c5dAKL29vTpw4IBVq+UKu5nNlPSopBMl/bu7P5S6/bhx41Qul/PsEkBCqVTKrDX8NN7MTpT0uKRZkiZLutHMJjd6fwBaK8/f7NMlvevuu939b5I2SJrTnLYANFuesJ8rac+g7/dWtn2NmS0ys7KZlfv7+3PsDkAeecJe7UWAb7z31t1XunvJ3Uvd3d05dgcgjzxh3ytpzKDvR0val68dAK2SJ+xvSJpoZt83s5Ml3SBpY3PaAtBsDU+9uftRM7tV0n9qYOptlbvvbFpnAJoq1zy7u78g6YUm9QKghXi7LBAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBtHXJZrTG0aNHM2svvvhicuzcuXOT9eXLlyfrixYtStaHDBmSrKN9OLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBDMsx8HnnzyyczabbfdlhxrZsn6kiVLkvXt27cn63feeWdmbezYscmxJ53Ej2cz5Xo0zaxX0meSvpB01N1LzWgKQPM141fnDHc/0IT7AdBC/M0OBJE37C5pk5m9aWZV3yRtZovMrGxm5f7+/py7A9CovGG/zN2nSZolabGZXXHsDdx9pbuX3L3U3d2dc3cAGpUr7O6+r3K5X9LvJU1vRlMAmq/hsJvZUDP73pfXJV0jaUezGgPQXObujQ00G6+Bo7k08Kr+f7j7A6kxpVLJy+VyQ/uL7NChQ8n6hAkTMmt5Xyep9fNRa54+5YorvvFX39e88sorDd93VKVSSeVyuep/SsNTb+6+W9LFDXcFoK2YegOCIOxAEIQdCIKwA0EQdiAIziH8Dpg8eXKynppeu+iii5Jjx48fn6xfcsklyfqGDRuS9Z6enszali1bkmNr9XbTTTcl63fccUdmbdiwYcmxxyOO7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQRMOnuDaCU1yre+utt5L1adOmJesnnJD9O/v2229Pjn344YeT9bxWrFiRWbv77ruTYw8ePJis1zq99sILL8ysbdq0KTl2xIgRyXqnSp3iypEdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Jgnr0NDh8+nKxPnDgxWe/r60vWU/PN69atS4694IILkvXzzz8/Wc/jwIH0eqAjR45M1vN8jHUt7733XrI+ZsyYlu07D+bZARB2IArCDgRB2IEgCDsQBGEHgiDsQBB8bnwbbNu2LVn/4IMPct3/LbfcklmbOXNmcmyRn5/e1dWVrK9atSpZv//++5P13t7eb9vSV84777xkfevWrcn6pEmTGt53q9Q8spvZKjPbb2Y7Bm0728xeNrN3KpfDW9smgLzqeRr/tKRjDw9LJW1294mSNle+B9DBaobd3bdIOvbzgeZIWlO5vkbS3Oa2BaDZGn2BbqS790lS5TLzA7vMbJGZlc2snFqTDEBrtfzVeHdf6e4ldy91d3e3encAMjQa9g/NbJQkVS73N68lAK3QaNg3SlpYub5Q0vPNaQdAq9ScZzez9ZKuktRlZnsl3SvpIUm/NbObJf1Z0g9b2eR33fLly1t6/wsXLsysnXnmmS3ddyul/l2SdO211ybrd911V2Zt7dq1ybFHjhxJ1mfMmJGsv/7668l6EefD1wy7u9+YUbq6yb0AaCHeLgsEQdiBIAg7EARhB4Ig7EAQnOLaBJ988kmy/tprryXrtT7O+6yzzkrWJ0yYkKwfr84555xkffXq1Zm10aNHJ8c+8MADyfr+/en3kd1zzz3Jeur03dQS3HlwZAeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIJhnr9PRo0cza8uWLUuOff/995P1WksPL168OFmv9ZHMUe3evTuz9tFHHyXH1vo/qVV/5plnkvXrr78+s3bdddclxzaKIzsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBME8e53Wr1+fWXvwwQdbuu958+a19P4j2rlzZ6H7f/vttzNrzLMDyIWwA0EQdiAIwg4EQdiBIAg7EARhB4Jgnr1Otc5JzyO1tLAkTZ06tWX7Pp6NHz8+s7Zp06bk2NNPPz3XvhcsWJCsL1myJNf9N6Lmkd3MVpnZfjPbMWjbfWb2vpltq3zNbm2bAPKq52n805JmVtn+K3efUvl6obltAWi2mmF39y2SDrahFwAtlOcFulvNbHvlaf7wrBuZ2SIzK5tZub+/P8fuAOTRaNiflPQDSVMk9Un6ZdYN3X2lu5fcvdTd3d3g7gDk1VDY3f1Dd//C3f8u6deSpje3LQDN1lDYzWzUoG/nSdqRdVsAnaHmPLuZrZd0laQuM9sr6V5JV5nZFEkuqVfST1rXYmf49NNPM2u11levZe7cubnGo7rPP/88s9bT05McW+v/9JRTTknW58+fn6yfeuqpyXor1Ay7u99YZfNTLegFQAvxdlkgCMIOBEHYgSAIOxAEYQeC4BTXOj3yyCOZtVrL944YMSJZnzRpUkM9IW3p0qWZtUcffTQ5ttb/6WOPPZasz5xZ7dyxYnFkB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgmGdvg2nTpiXrJ598cps6+W5JnaIqpefRJWn16tUN7/vZZ59N1q+55pqG77soHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjm2et05ZVXZta2bNmSHPvSSy8l64cPH07WhwwZkqx/V+3ZsydZv/zyy5P1PMtob9y4MVmfPfv4W5iYIzsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBME8e51Snxt/6aWX5rrvWks2P/fcc8n6aaedllmrNUd/5MiRXPWPP/44Wb/33nsza2vXrk2OrbVscq3H/YknnsisTZ06NTn2eFTzyG5mY8zsD2a2y8x2mtlPK9vPNrOXzeydyuXw1rcLoFH1PI0/Kunn7v5Pkv5Z0mIzmyxpqaTN7j5R0ubK9wA6VM2wu3ufu2+tXP9M0i5J50qaI2lN5WZrJM1tUY8AmuBbvUBnZuMkTZX0J0kj3b1PGviFIKnqgmZmtsjMymZW7u/vz9kugEbVHXYzGybpd5J+5u5/qXecu69095K7l7q7uxvpEUAT1BV2MxuigaCvc/cvP3bzQzMbVamPkrS/NS0CaIaaU282sHbtU5J2ufuyQaWNkhZKeqhy+XxLOuwQF198cWZt1qxZybG1TnGtdYrs8OHpiY7U/seOHZscu2PHjmT91VdfTdZrLW2cZ+z8+fOT9RUrViTrZ5xxxrfu6XhWzzz7ZZIWSOoxs22Vbb/QQMh/a2Y3S/qzpB+2pEMATVEz7O7+R0lZv4Kvbm47AFqFt8sCQRB2IAjCDgRB2IEgCDsQBKe41il1qujTTz+dHNvT05Osz5kzJ1k/dOhQsl5rHr+VRoyo+i7pr8ybNy+ztnjx4uTYyZMnN9QTquPIDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM/eBF1dXcn6jBkzkvV9+/Yl648//niyvmHDhsxarXPGb7jhhmT96qvTJzZOmjQpWR86dGiyjvbhyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQVitZXGbqVQqeblcbtv+gGhKpZLK5XLVN1dwZAeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIGqG3czGmNkfzGyXme00s59Wtt9nZu+b2bbK1+zWtwugUfV8eMVRST93961m9j1Jb5rZy5Xar9z931rXHoBmqWd99j5JfZXrn5nZLknntroxAM31rf5mN7NxkqZK+lNl061mtt3MVpnZ8Iwxi8ysbGbl/v7+fN0CaFjdYTezYZJ+J+ln7v4XSU9K+oGkKRo48v+y2jh3X+nuJXcvdXd35+8YQEPqCruZDdFA0Ne5+7OS5O4fuvsX7v53Sb+WNL11bQLIq55X403SU5J2ufuyQdtHDbrZPEk7mt8egGap59X4yyQtkNRjZtsq234h6UYzmyLJJfVK+kkL+gPQJPW8Gv9HSdXOj32h+e0AaBXeQQcEQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiirUs2m1m/pP8btKlL0oG2NfDtdGpvndqXRG+NamZv/+juVT//ra1h/8bOzcruXiqsgYRO7a1T+5LorVHt6o2n8UAQhB0Iouiwryx4/ymd2lun9iXRW6Pa0luhf7MDaJ+ij+wA2oSwA0EUEnYzm2lm/21m75rZ0iJ6yGJmvWbWU1mGulxwL6vMbL+Z7Ri07Wwze9nM3qlcVl1jr6DeOmIZ78Qy44U+dkUvf972v9nN7ERJ/yPpXyTtlfSGpBvd/e22NpLBzHolldy98DdgmNkVkv4q6TfufkFl2yOSDrr7Q5VflMPd/c4O6e0+SX8tehnvympFowYvMy5prqQfq8DHLtHXj9SGx62II/t0Se+6+253/5ukDZLmFNBHx3P3LZIOHrN5jqQ1letrNPDD0nYZvXUEd+9z962V659J+nKZ8UIfu0RfbVFE2M+VtGfQ93vVWeu9u6RNZvammS0qupkqRrp7nzTwwyNpRMH9HKvmMt7tdMwy4x3z2DWy/HleRYS92lJSnTT/d5m7T5M0S9LiytNV1KeuZbzbpcoy4x2h0eXP8yoi7HsljRn0/WhJ+wrooyp331e53C/p9+q8pag//HIF3crl/oL7+UonLeNdbZlxdcBjV+Ty50WE/Q1JE83s+2Z2sqQbJG0soI9vMLOhlRdOZGZDJV2jzluKeqOkhZXrCyU9X2AvX9Mpy3hnLTOugh+7wpc/d/e2f0marYFX5P9X0r8W0UNGX+Ml/Vfla2fRvUlar4GndZ9r4BnRzZL+QdJmSe9ULs/uoN7WSuqRtF0DwRpVUG+Xa+BPw+2StlW+Zhf92CX6asvjxttlgSB4Bx0QBGEHgiDsQBCEHQiCsANBEHYgCMIOBPH/J3F/FLPr20oAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  0\n",
      "Ground truth annotation:  0\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOO0lEQVR4nO3db6hc9Z3H8c9nY2skrRI31xBsMLYGWV3U1CEIGnGtW0yexD5waRBRCMa/2GIfGCpSUdY/y9aisCrpmjRqTVO0EpGQrcRC0AfFUVyTbHSNMVvTRHNDIKagcU2/++COyzXe+c11zpk/yff9gsvMnO+cOV8m95Mzd37nnJ8jQgCOfX8z6AYA9AdhB5Ig7EAShB1IgrADSRzXz43NmDEj5syZ089NAqns3LlT+/bt80S1SmG3fbmkhyRNkfTvEXF/6flz5sxRs9msskkABY1Go22t64/xtqdI+jdJCyWdJWmJ7bO6fT0AvVXlb/b5krZHxI6I+FTSbyQtrqctAHWrEvZTJb0/7vGu1rIvsL3MdtN2c3R0tMLmAFRRJewTfQnwpWNvI2JFRDQiojEyMlJhcwCqqBL2XZJmj3v8LUm7q7UDoFeqhP1VSXNtn27765J+KOn5etoCULeuh94i4jPbt0j6D40Nva2MiK21dQagVpXG2SNivaT1NfUCoIc4XBZIgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJPo6ZTOGz5IlS4r1tWvXFusLFiwo1jds2NC2dsIJJxTXRb3YswNJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoyzHwMOHz7ctrZ06dLiup3G0W0X6y+//HKxvnLlyra1VatWFdd99913i/VObrvttra1O++8s9JrH40qhd32TkkHJR2W9FlENOpoCkD96tiz/0NE7KvhdQD0EH+zA0lUDXtI+r3t12wvm+gJtpfZbtpujo6OVtwcgG5VDfuFEfFdSQsl3Wz74iOfEBErIqIREY2RkZGKmwPQrUphj4jdrdu9kp6TNL+OpgDUr+uw255m+5uf35f0fUlb6moMQL0cEd2taH9bY3tzaexb/acj4p9L6zQajWg2m11tL7Pt27cX62+99Vbb2uLFi4vrdvr37zTOPsxOO+20trUdO3b0sZP+aTQaajabE/6jdT30FhE7JJ3bdVcA+oqhNyAJwg4kQdiBJAg7kARhB5LgFNch8N577xXry5cvL9afe+65Yn2QZs+e3bZ27rnlwZwXXnih7nZSY88OJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzt4HH3zwQbE+f375mh/79++vs51a3X333cX6jTfe2LbW6X1hnL1e7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2fvg0KFDxXovx9HnzJlTrHcay+50KekzzjijWD/uuPa/Yp3G2as688wze/r6Rxv27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOPsfTBz5sxivdNY90MPPVSs33PPPW1rp59+enHdGTNmFOtVla6Jf/vtt1d67U7HEKxdu7bS6x9rOu7Zba+0vdf2lnHLTrb9ou13WrfTe9smgKom8zH+V5IuP2LZckkbI2KupI2txwCGWMewR8QmSUcez7lY0urW/dWSrqi3LQB16/YLupkRsUeSWrentHui7WW2m7abo6OjXW4OQFU9/zY+IlZERCMiGiMjI73eHIA2ug37h7ZnSVLrdm99LQHohW7D/ryka1r3r5G0rp52APRKx3F222skXSJphu1dkn4m6X5Jv7W9VNKfJF3ZyyaPdlOnTi3WFy5cWKk+zErj7OvXr6/02nPnzi3WTzzxxEqvf6zpGPaIWNKm9L2aewHQQxwuCyRB2IEkCDuQBGEHkiDsQBKc4oqiw4cPF+sPP/xwsX7vvfd2ve1zzjmnWF+zZk3Xr50Re3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJxdhQtXbq0WH/yySd7tu1O4+jTp3NR46+CPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJME4e3JLlrS7ePCYqtMel85J37RpU3FdLgVdL/bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE4+zHgC1btrStLViwoLjugQMHinXbxfq8efOK9ZdeeqltjXH0/uq4Z7e90vZe21vGLbvL9p9tv9H6WdTbNgFUNZmP8b+SdPkEy38REee1ftbX2xaAunUMe0RskrS/D70A6KEqX9DdYvvN1sf8thcDs73MdtN2c3R0tMLmAFTRbdgflfQdSedJ2iPp5+2eGBErIqIREY2RkZEuNwegqq7CHhEfRsThiPirpF9Kml9vWwDq1lXYbc8a9/AHktqP/QAYCh3H2W2vkXSJpBm2d0n6maRLbJ8nKSTtlHR971o89n3yySfF+vXXl9/edevWta0dPHiwq54+12kcfePGjcU6Y+nDo2PYI2Kiqxs83oNeAPQQh8sCSRB2IAnCDiRB2IEkCDuQBKe49sHu3buL9csuu6xYf/vtt+ts5ws6XSp60aLyCY3Tpk2rs52v5NChQ8X6K6+80rZWOi1Ykm699dauehpm7NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2Wvw8ccfF+t33HFHsd7LcfROrrzyyoFtu5P333+/WH/ggQeK9UcffbRt7eKLLy6uyzg7gKMWYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTj7JJXG0m+66abiuk888UTd7dTmo48+KtarXgq6dC7/9u3bi+t2OgZg3759xfrxxx/ftnbfffcV1z0WsWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ2/pNG3yDTfc0Lb21FNP1d1O31x00UXF+vz58yu9/rPPPtu21mmMv5PSOLokPfLII21rF1xwQaVtH4067tltz7b9B9vbbG+1/aPW8pNtv2j7ndbt9N63C6Bbk/kY/5mkn0TE30m6QNLNts+StFzSxoiYK2lj6zGAIdUx7BGxJyJeb90/KGmbpFMlLZa0uvW01ZKu6FGPAGrwlb6gsz1H0jxJf5Q0MyL2SGP/IUg6pc06y2w3bTdHR0crtgugW5MOu+1vSHpW0o8jYtLfrETEiohoRERjZGSkmx4B1GBSYbf9NY0F/dcR8bvW4g9tz2rVZ0na25sWAdSh49CbbUt6XNK2iHhwXOl5SddIur91u64nHdbk008/Ldavu+66Yv3pp5+us50v6PSJ59prry3WH3vssba1gwcPFtfdvHlzsb5169ZivZc6DQs++OCDxfr5559fZztHvcmMs18o6WpJm22/0Vr2U42F/Le2l0r6k6ThvQA5gM5hj4iXJblN+Xv1tgOgVzhcFkiCsANJEHYgCcIOJEHYgSTSnOL6zDPPFOtVxtGnTJlSrM+bN69Y37BhQ7E+fXr5hMKrr766bW3BggXFdTud2tvJ1KlTi/XSqaTLli0rrnvppZcW61Uvc50Ne3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSCLNOPvYafntnXTSScX6gQMH2tZWrVpVXPeqq64q1qs6++yz29b2799fXHfHjh3Femmq6k7bxnBhzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTgi+raxRqMRzWazb9sDsmk0Gmo2mxMeVMKeHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeS6Bh227Nt/8H2Nttbbf+otfwu23+2/UbrZ1Hv2wXQrclcvOIzST+JiNdtf1PSa7ZfbNV+ERH/2rv2ANRlMvOz75G0p3X/oO1tkk7tdWMA6vWV/ma3PUfSPEl/bC26xfabtlfannCOItvLbDdtN0dHR6t1C6Brkw677W9IelbSjyPiI0mPSvqOpPM0tuf/+UTrRcSKiGhERGNkZKR6xwC6Mqmw2/6axoL+64j4nSRFxIcRcTgi/irpl5Lm965NAFVN5tt4S3pc0raIeHDc8lnjnvYDSVvqbw9AXSbzbfyFkq6WtNn2G61lP5W0xPZ5kkLSTknX96A/ADWZzLfxL0ua6PzY9fW3A6BXOIIOSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQRF+nbLY9Kul/xi2aIWlf3xr4aoa1t2HtS6K3btXZ22kRMeH13/oa9i9t3G5GRGNgDRQMa2/D2pdEb93qV298jAeSIOxAEoMO+4oBb79kWHsb1r4keutWX3ob6N/sAPpn0Ht2AH1C2IEkBhJ225fbftv2dtvLB9FDO7Z32t7cmoa6OeBeVtrea3vLuGUn237R9jut2wnn2BtQb0MxjXdhmvGBvneDnv6873+z254i6b8l/aOkXZJelbQkIv6rr420YXunpEZEDPwADNsXS/qLpCci4u9by/5F0v6IuL/1H+X0iLh9SHq7S9JfBj2Nd2u2olnjpxmXdIWkazXA967Q1z+pD+/bIPbs8yVtj4gdEfGppN9IWjyAPoZeRGyStP+IxYslrW7dX62xX5a+a9PbUIiIPRHxeuv+QUmfTzM+0Peu0FdfDCLsp0p6f9zjXRqu+d5D0u9tv2Z72aCbmcDMiNgjjf3ySDplwP0cqeM03v10xDTjQ/PedTP9eVWDCPtEU0kN0/jfhRHxXUkLJd3c+riKyZnUNN79MsE040Oh2+nPqxpE2HdJmj3u8bck7R5AHxOKiN2t272SntPwTUX94ecz6LZu9w64n/83TNN4TzTNuIbgvRvk9OeDCPurkubaPt321yX9UNLzA+jjS2xPa31xItvTJH1fwzcV9fOSrmndv0bSugH28gXDMo13u2nGNeD3buDTn0dE338kLdLYN/LvSrpjED206evbkv6z9bN10L1JWqOxj3X/q7FPREsl/a2kjZLead2ePES9PSlps6Q3NRasWQPq7SKN/Wn4pqQ3Wj+LBv3eFfrqy/vG4bJAEhxBByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/B+ArlCbqBZS3wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  2\n",
      "Ground truth annotation:  2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOYklEQVR4nO3df4xU9bnH8c8jgj8AFburbqjXbRvEEpUfmeA1GOSKNmpMkD9qyh8NNzahMZC0kRiESwL+CNlcLzRXc9MIVwRvWghJa8FIvDWEBBsVHQwqFhU0W6AQWCQKTTQoPPePPd4suOc7y8yZH+zzfiWbmTnPfDlPhv3smZnz42vuLgCD3wXNbgBAYxB2IAjCDgRB2IEgCDsQxIWNXFlbW5t3dnY2cpVAKN3d3Tp69Kj1V6sp7GZ2t6T/lDRE0n+7e1fq+Z2dnSqXy7WsEkBCqVTKrVX9Nt7Mhkj6L0n3SBonaZaZjav23wNQX7V8Zp8saa+7f+ruJyWtlzSjmLYAFK2WsI+WtL/P4wPZsjOY2RwzK5tZuaenp4bVAahFLWHv70uA7xx76+4r3b3k7qX29vYaVgegFrWE/YCka/s8/r6kg7W1A6Beagn725LGmNkPzGyYpJ9J2lRMWwCKVvWuN3f/xszmSfpf9e56W+3uHxTWGYBC1bSf3d03S9pcUC8A6ojDZYEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IoqFTNqPxjh8/nqzv2LEjWX/55ZeT9TVr1iTrl112WW5t1apVybHTp09P1nFu2LIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBDsZx8Eurq6cmvLly9Pjv3ss8+S9aFDhybrixYtStbb2tpya5V6u/XWW5P1Sy+9NFnHmWoKu5l1Szoh6ZSkb9y9VERTAIpXxJb9X9z9aAH/DoA64jM7EEStYXdJfzazHWY2p78nmNkcMyubWbmnp6fG1QGoVq1hn+LukyTdI2mumU09+wnuvtLdS+5eam9vr3F1AKpVU9jd/WB2e0TSi5ImF9EUgOJVHXYzG25mI7+9L+knknYV1RiAYtXybfzVkl40s2//nd+7+yuFdIUzrFixIllfvHhxbu306dPJsQ8//HCyPm/evGS9s7MzWU+ZO3du1WNx7qoOu7t/Kml8gb0AqCN2vQFBEHYgCMIOBEHYgSAIOxAEp7i2gO3btyfrjzzySLI+duzY3Nq6deuSY2+++eZkPdu1Whdff/11sr5///5kvbu7O1m//fbbc2tDhgxJjh2M2LIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBDsZ28BCxcuTNbdPVl/+umnc2vjx9f3xMSvvvoqWV+2bFlubf369cmxlS5j9vzzzyfr9TxG4HzElh0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgmA/ews4erS2eTGvuOKKYhrpx7Fjx5L1O++8M1nfuXNnbu3BBx9Mjl2wYEGyPmbMmGQdZ2LLDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBsJ+9Bdx1113J+q5d6Wnvn3zyydxaV1dXcuypU6eS9YceeihZ//zzz5P1w4cP59ba2tqSYzkfvVgVt+xmttrMjpjZrj7LrjSzV81sT3Y7qr5tAqjVQN7Gr5F091nLHpW0xd3HSNqSPQbQwiqG3d23STr7mMkZktZm99dKur/YtgAUrdov6K5290OSlN1elfdEM5tjZmUzK1e6phiA+qn7t/HuvtLdS+5eam9vr/fqAOSoNuyHzaxDkrLbI8W1BKAeqg37Jkmzs/uzJW0sph0A9WKVrkluZuskTZPUJumwpCWS/iRpg6R/krRP0k/dPX3is6RSqeTlcrm2jgehL774IlmfNm1asv7uu+/m1irNQz58+PBk/cYbb0zWX3nllWR9xIgRyTqKVSqVVC6X+z1AoeJBNe4+K6c0vaauADQUh8sCQRB2IAjCDgRB2IEgCDsQBKe4toDLL788Wd+2bVuyPmXKlNxapdNjjx8/nqzfd999yTq71s4fbNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAj2s58HRo4cmazPnDkzt1ZpP3slixcvTtbffPPNZH39+vW5tYsvvriqnlAdtuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EAT72QeBtWvX5tauueaa5NhNmzYl6wsXLqxp/Pjx43NrL730UnLs9ddfn6zj3LBlB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEg2M8+CPT09OTWlixZkhxbKpWS9Ur7whcsWJCsP/PMM7m1W265JTn2jTfeSNZvuOGGZB1nqrhlN7PVZnbEzHb1WbbUzP5uZjuzn3vr2yaAWg3kbfwaSXf3s/w37j4h+9lcbFsAilYx7O6+TdKxBvQCoI5q+YJunpm9l73NH5X3JDObY2ZlMyunPlsCqK9qw/5bST+SNEHSIUnL857o7ivdveTupfb29ipXB6BWVYXd3Q+7+yl3Py1plaTJxbYFoGhVhd3MOvo8nCmptusVA6i7ivvZzWydpGmS2szsgKQlkqaZ2QRJLqlb0i/r1yLeeuutZP3LL7+s27orXdt9+fLcT3CSpNRHt8ceeyw5dvLk9BvG3bt3J+ujR49O1qOpGHZ3n9XP4ufq0AuAOuJwWSAIwg4EQdiBIAg7EARhB4LgFNfzwE033ZSsX3TRRQ3q5LsuvDD9K5Sa8vnEiRPJsU899VSyPm/evGR9w4YNubWhQ4cmxw5GbNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAj2s58HLrnkkmT9ggvOz7/ZlU5xrXQZ640bNybr27dvz63ddtttybGD0fn5WwLgnBF2IAjCDgRB2IEgCDsQBGEHgiDsQBDsZx8E7rjjjtza66+/nhx78uTJZH3YsGFV9TQQlS5T3dHRkax/+OGHyfrevXtza+xnBzBoEXYgCMIOBEHYgSAIOxAEYQeCIOxAEOxnHwSeeOKJ3NqkSZOSYx9//PGa6vU8l37cuHHJ+tatW+u27sGo4v+UmV1rZlvNbLeZfWBmv8qWX2lmr5rZnux2VP3bBVCtgfxZ/kbSfHf/saR/ljTXzMZJelTSFncfI2lL9hhAi6oYdnc/5O7vZPdPSNotabSkGZLWZk9bK+n+OvUIoADn9IHLzDolTZS0XdLV7n5I6v2DIOmqnDFzzKxsZuWenp4a2wVQrQGH3cxGSPqDpF+7+/GBjnP3le5ecvdSe3t7NT0CKMCAwm5mQ9Ub9N+5+x+zxYfNrCOrd0g6Up8WARSh4q43MzNJz0na7e4r+pQ2SZotqSu7TV/XF3UzYcKE3NoDDzyQHLts2bJk/ZNPPknWly5dmqyPHTs2t/bxxx8nx7722mvJOs7NQPazT5H0c0nvm9nObNki9YZ8g5n9QtI+ST+tS4cAClEx7O7+F0mWU55ebDsA6oXDZYEgCDsQBGEHgiDsQBCEHQjC3L1hKyuVSl4ulxu2PkgfffRRsp66DLUkHTx4MFmvdIrrddddl1vbt29fcuypU6eS9alTpybrmzdvzq0NHz48OfZ8VSqVVC6X+917xpYdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4LgUtKDXOp8cknas2dPsv7ss88m6/Pnz0/Wu7u7k/WUiRMnJusvvPBCsj5Y96VXiy07EARhB4Ig7EAQhB0IgrADQRB2IAjCDgTB+ezAIML57AAIOxAFYQeCIOxAEIQdCIKwA0EQdiCIimE3s2vNbKuZ7TazD8zsV9nypWb2dzPbmf3cW/92AVRrIBev+EbSfHd/x8xGStphZq9mtd+4+3/Urz0ARRnI/OyHJB3K7p8ws92SRte7MQDFOqfP7GbWKWmipO3Zonlm9p6ZrTazUTlj5phZ2czKPT09tXULoGoDDruZjZD0B0m/dvfjkn4r6UeSJqh3y7+8v3HuvtLdS+5eam9vr71jAFUZUNjNbKh6g/47d/+jJLn7YXc/5e6nJa2SNLl+bQKo1UC+jTdJz0na7e4r+izv6PO0mZJ2Fd8egKIM5Nv4KZJ+Lul9M9uZLVskaZaZTZDkkrol/bIO/QEoyEC+jf+LpP7Oj82f/BpAy+EIOiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBANnbLZzHok/a3PojZJRxvWwLlp1d5atS+J3qpVZG/XuXu/139raNi/s3KzsruXmtZAQqv21qp9SfRWrUb1xtt4IAjCDgTR7LCvbPL6U1q1t1btS6K3ajWkt6Z+ZgfQOM3esgNoEMIOBNGUsJvZ3Wb2kZntNbNHm9FDHjPrNrP3s2moy03uZbWZHTGzXX2WXWlmr5rZnuy23zn2mtRbS0zjnZhmvKmvXbOnP2/4Z3YzGyLpY0l3STog6W1Js9z9rw1tJIeZdUsquXvTD8Aws6mS/iHpBXe/MVv275KOuXtX9odylLsvaJHelkr6R7On8c5mK+roO824pPsl/aua+Nol+npADXjdmrFlnyxpr7t/6u4nJa2XNKMJfbQ8d98m6dhZi2dIWpvdX6veX5aGy+mtJbj7IXd/J7t/QtK304w39bVL9NUQzQj7aEn7+zw+oNaa790l/dnMdpjZnGY304+r3f2Q1PvLI+mqJvdztorTeDfSWdOMt8xrV83057VqRtj7m0qqlfb/TXH3SZLukTQ3e7uKgRnQNN6N0s804y2h2unPa9WMsB+QdG2fx9+XdLAJffTL3Q9mt0ckvajWm4r68Lcz6Ga3R5rcz/9rpWm8+5tmXC3w2jVz+vNmhP1tSWPM7AdmNkzSzyRtakIf32Fmw7MvTmRmwyX9RK03FfUmSbOz+7MlbWxiL2dolWm886YZV5Nfu6ZPf+7uDf+RdK96v5H/RNK/NaOHnL5+KOnd7OeDZvcmaZ1639Z9rd53RL+Q9D1JWyTtyW6vbKHe/kfS+5LeU2+wOprU223q/Wj4nqSd2c+9zX7tEn015HXjcFkgCI6gA4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEg/g+UqlJFdXnxUAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  8\n",
      "Ground truth annotation:  8\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAANqElEQVR4nO3dYahc9ZnH8d/P2IQYKxpz40YbN90myMrCRhliMEtVxGKCoA24NGDJgvT2RSItFFTcFwb0RYjb1oJaSI00la610IoKwRq1IDUgGSVrksZVN2SbmGiuGGwqiV3jsy/usVzjnTPXOefMTPJ8P3CZmfPMOf8nw/3lzMx/5v4dEQJw+jtj0A0A6A/CDiRB2IEkCDuQBGEHkjizn4PNmTMnFixY0M8hgVT27dun9957z5PVKoXd9vWSfiJpmqSHI2J92f0XLFigdrtdZUgAJVqtVsdaz0/jbU+T9KCk5ZIulbTK9qW9Hg9As6q8Zl8i6a2I2BsRf5X0K0k31tMWgLpVCftFkvZPuH2g2PYZtkdtt223x8bGKgwHoIoqYZ/sTYDPffY2IjZGRCsiWiMjIxWGA1BFlbAfkDR/wu2vSDpYrR0ATakS9u2SFtn+qu3pkr4l6al62gJQt56n3iLiY9trJf1O41Nvj0TE7to6A1CrSvPsEbFF0paaegHQID4uCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKVVnEFmrR8+fLS+vbt20vre/fu7Vg755xzeurpVFYp7Lb3SToq6YSkjyOiVUdTAOpXx5n9moh4r4bjAGgQr9mBJKqGPSQ9a/sV26OT3cH2qO227fbY2FjF4QD0qmrYl0XE5ZKWS1pj++sn3yEiNkZEKyJaIyMjFYcD0KtKYY+Ig8XlYUlPSFpSR1MA6tdz2G3Psv3lT69L+oakXXU1BqBeVd6Nv0DSE7Y/Pc5/RsQztXSFFO67777S+tatW0vrEVFa37BhQ8favffeW7rv6ajnsEfEXkn/XGMvABrE1BuQBGEHkiDsQBKEHUiCsANJ8BVXNOqNN97oWFu3bl3pvt2m1rqZMWNGpf1PN5zZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJ5tlPA8eOHetYmzlzZqNjHz9+vLR+5ZVX9rxvN2eddVZpfc2aNZWOf7rhzA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDPfgo4ePBgaX3p0qUdawsXLizd94UXXuipp09t27attH7kyJFKxy/zwAMPlNZnz57d2NinIs7sQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AE8+xD4MMPPyytr1ixorT+9ttvd6y98847pfs+80z5KtvXXHNNaf3mm28urVexZMmS0vott9zS2Nino65ndtuP2D5se9eEbbNtb7X9ZnF5XrNtAqhqKk/jfy7p+pO23Snp+YhYJOn54jaAIdY17BHxoqT3T9p8o6TNxfXNkm6qty0Adev1DboLIuKQJBWXczvd0fao7bbt9tjYWI/DAaiq8XfjI2JjRLQiojUyMtL0cAA66DXs79qeJ0nF5eH6WgLQhF7D/pSk1cX11ZKerKcdAE3pOs9u+zFJV0uaY/uApLslrZf0a9u3SvqTpOYmWxPYvn17aX3nzp09H3v69Oml9UsuuaS0vmnTptJ6t++r2y6tl1m/fn1pfdq0aT0fO6OuYY+IVR1K19bcC4AG8XFZIAnCDiRB2IEkCDuQBGEHkuArrn1w+HD5Z45uuOGGxsY+//zzS+vz588vrd9zzz11tvMZ115bPqGzbNmyxsbOiDM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBPHsfbNmypbR+7NixSsefNWtWx9pzzz1Xuu/u3btL690+I1DF2rVrS+svv/xyaf3MM8t/fS+//PKe963y1dxhxZkdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Jgnr0G3ZZcvuOOOxodf926dR1rixYtKt339ttvr7mbqduwYUNpfceOHaX1Kp9P2LZtW2n9iiuu6PnYw4ozO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTx7DQ4ePFhaHxsba3T8drvdsTY6Olq678MPP1xp7Ijoed9uc91NevTRR0vrKefZbT9i+7DtXRO2rbP9tu0dxc+KZtsEUNVUnsb/XNL1k2z/cUQsLn7K/xQLgIHrGvaIeFHS+33oBUCDqrxBt9b2a8XT/PM63cn2qO227XbTr10BdNZr2H8q6WuSFks6JOmHne4YERsjohURrZGRkR6HA1BVT2GPiHcj4kREfCLpZ5KW1NsWgLr1FHbb8ybc/KakXZ3uC2A4dJ1nt/2YpKslzbF9QNLdkq62vVhSSNon6bvNtTj8zj777NL6zJkzS+vHjx+vNP7jjz/e875N/330Qf799RUrOs8IL1y4sI+dDIeuYY+IVZNs3tRALwAaxMdlgSQIO5AEYQeSIOxAEoQdSIKvuNZg3rx5pfWnn366tH7dddfV2U5fzZ07t7T+0EMPdaxdeOGFpfsuXry4l5b+Zvr06R1rZ5yR7zyX718MJEXYgSQIO5AEYQeSIOxAEoQdSIKwA0kwz94HV111VWm92zz8Bx98UFo/cuRIx9ptt91Wum9VDz74YGl95cqVjY6PqePMDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJuMqSu19Uq9WKsuWF0ZujR492rJ177rmNjn3ixIlGj48vptVqqd1uT/r3uzmzA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EASfJ/9NPDSSy8NugWcArqe2W3Pt/1723ts77b9vWL7bNtbbb9ZXJ7XfLsAejWVp/EfS/pBRPyjpKWS1ti+VNKdkp6PiEWSni9uAxhSXcMeEYci4tXi+lFJeyRdJOlGSZuLu22WdFNDPQKowRd6g872AkmXSXpZ0gURcUga/w9B0qSLftketd223R4bG6vYLoBeTTnsts+W9BtJ34+IP091v4jYGBGtiGiNjIz00iOAGkwp7La/pPGg/zIifltsftf2vKI+T9LhZloEUIeuU2+2LWmTpD0R8aMJpackrZa0vrh8spEO0dX+/fsbO/bChQsbOzb6ayrz7MskfVvSTts7im13aTzkv7Z9q6Q/Sbq5kQ4B1KJr2CPiD5Im/TK8pGvrbQdAU/i4LJAEYQeSIOxAEoQdSIKwA0nwFddTwEcffVRav//++xsb+7LLLmvs2OgvzuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kATz7KeAGTNmlNYvvvjijrXXX3+90thz507618ZwCuLMDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMM9+Grj77rs71p599tnSfVeuXFlaX716dU89YfhwZgeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJKayPvt8Sb+Q9HeSPpG0MSJ+YnudpO9IGivueldEbGmqUXS2dOnSjrUTJ070sRMMs6l8qOZjST+IiFdtf1nSK7a3FrUfR8R/NNcegLpMZX32Q5IOFdeP2t4j6aKmGwNQry/0mt32AkmXSXq52LTW9mu2H7F9Xod9Rm23bbfHxsYmuwuAPphy2G2fLek3kr4fEX+W9FNJX5O0WONn/h9Otl9EbIyIVkS0RkZGqncMoCdTCrvtL2k86L+MiN9KUkS8GxEnIuITST+TtKS5NgFU1TXsti1pk6Q9EfGjCdvnTbjbNyXtqr89AHWZyrvxyyR9W9JO2zuKbXdJWmV7saSQtE/SdxvoD0BNpvJu/B8keZISc+rAKYRP0AFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRPRvMHtM0v9O2DRH0nt9a+CLGdbehrUvid56VWdvfx8Rk/79t76G/XOD2+2IaA2sgRLD2tuw9iXRW6/61RtP44EkCDuQxKDDvnHA45cZ1t6GtS+J3nrVl94G+podQP8M+swOoE8IO5DEQMJu+3rb/237Ldt3DqKHTmzvs73T9g7b7QH38ojtw7Z3Tdg22/ZW228Wl5OusTeg3tbZfrt47HbYXjGg3ubb/r3tPbZ32/5esX2gj11JX3153Pr+mt32NElvSLpO0gFJ2yWtiog/9rWRDmzvk9SKiIF/AMP21yX9RdIvIuKfim0bJL0fEeuL/yjPi4g7hqS3dZL+MuhlvIvViuZNXGZc0k2S/k0DfOxK+vpX9eFxG8SZfYmktyJib0T8VdKvJN04gD6GXkS8KOn9kzbfKGlzcX2zxn9Z+q5Db0MhIg5FxKvF9aOSPl1mfKCPXUlffTGIsF8kaf+E2wc0XOu9h6Rnbb9ie3TQzUzigog4JI3/8kiaO+B+TtZ1Ge9+OmmZ8aF57HpZ/ryqQYR9sqWkhmn+b1lEXC5puaQ1xdNVTM2UlvHul0mWGR8KvS5/XtUgwn5A0vwJt78i6eAA+phURBwsLg9LekLDtxT1u5+uoFtcHh5wP38zTMt4T7bMuIbgsRvk8ueDCPt2SYtsf9X2dEnfkvTUAPr4HNuzijdOZHuWpG9o+JaifkrS6uL6aklPDrCXzxiWZbw7LTOuAT92A1/+PCL6/iNphcbfkf8fSf8+iB469PUPkv6r+Nk96N4kPabxp3X/p/FnRLdKOl/S85LeLC5nD1Fvj0raKek1jQdr3oB6+xeNvzR8TdKO4mfFoB+7kr768rjxcVkgCT5BByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ/D+Byw9jNWfOOAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  4\n",
      "Ground truth annotation:  4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOrUlEQVR4nO3da4xUdZrH8d+jgBdALktjUJBmiCbrJTKTCq6yjm4mjugbJGbWIaaDAdMTxWQwxEjwgiYmXuLMaHAZg0oGFZ2QzBgw3lBCQvAFoSSIuMSFNcD02Gma8EJQkhF49kUfZ1vs+p+m6tQFnu8nqVTVeerU/6HSP05VnXPqb+4uAGe+s5rdAIDGIOxAEIQdCIKwA0EQdiCIIY0cbNy4cd7e3t7IIYFQ9u7dq4MHD9pAtZrCbmYzJT0v6WxJL7v7U6nHt7e3q1wu1zIkgIRSqVSxVvXbeDM7W9J/SbpF0uWS5pjZ5dU+H4D6quUz+3RJe9z9S3f/h6Q/S5pVTFsAilZL2C+W9Ld+97uyZT9gZp1mVjazcm9vbw3DAahFLWEf6EuAHx176+4r3L3k7qW2trYahgNQi1rC3iVpUr/7EyV9VVs7AOqllrBvlXSpmU0xs2GSfi1pXTFtASha1bve3P2Ymd0n6QP17Xpb6e6fF9YZgELVtJ/d3d+V9G5BvQCoIw6XBYIg7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiCIhk7ZjPp49tlnK9YeeOCB5LovvPBCsr5gwYKqeirCpk2bkvUbbrih6ud+9dVXk/WOjo6qn7tVsWUHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSDYz34a2LZtW7L+4IMPVqyZWXLd559/PlkfOXJksp6np6enYu3hhx9Ornv8+PFkPe/fds4551SsXXPNNcl1z0Q1hd3M9ko6LOm4pGPuXiqiKQDFK2LL/h/ufrCA5wFQR3xmB4KoNewuab2ZfWJmnQM9wMw6zaxsZuXe3t4ahwNQrVrDPsPdfybpFkkLzOznJz/A3Ve4e8ndS21tbTUOB6BaNYXd3b/Krg9IekvS9CKaAlC8qsNuZsPNbOT3tyX9UtLOohoDUKxavo2/UNJb2b7OIZLecPf3C+kqmGeeeSZZf/zxx5N1d6967D179iTrd911V9XPXW/nnXdesr5169aKtcsuu6zodlpe1WF39y8lXV1gLwDqiF1vQBCEHQiCsANBEHYgCMIOBMEprg1w8GD6PKG800yPHj1a9djz589P1vN2261cuTJZv/POO5P1KVOmJOspd9xxR7I+derUZP3cc8+teuwzEVt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiC/ewFyNuPPmPGjGS9u7u7pvFvv/32irUXX3yxpud+7rnnkvW8fdlDhvAn1irYsgNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEOwELcDy5cuT9d27d9f0/O3t7cn6Qw89VLG2efPm5Lpvv/12st7V1ZWsX3fddcn61VdX/gHiVE2SRo8enazj1LBlB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEg2M8+SKnzup944om6jn3o0KFkffbs2RVr+/btK7qdH1izZk3V606cODFZX7x4cbJ+7733Vj12RLlbdjNbaWYHzGxnv2VjzexDM9udXY+pb5sAajWYt/F/kjTzpGWLJW1w90slbcjuA2hhuWF3902STn4fOUvSquz2Kkm3FdsWgKJV+wXdhe7eLUnZ9fhKDzSzTjMrm1m5t7e3yuEA1Kru38a7+wp3L7l7qa2trd7DAaig2rD3mNkEScquDxTXEoB6qDbs6yTNzW7PlbS2mHYA1EvufnYze1PSjZLGmVmXpKWSnpK0xszmS9ov6Vf1bLIVPPnkkxVrx44dq+vYF1xwQbJ+zz33VKx1dHQU3c4PvPHGG8n6smXLKtb279+fXHfhwoXJ+gcffJCsr13LNqi/3LC7+5wKpV8U3AuAOuJwWSAIwg4EQdiBIAg7EARhB4LgFNdM3tTGeaeZplxyySXJ+oIFC5L1+fPnJ+tjx4495Z6KsmjRomQ9NZ30/fffn1w3b9fZe++9l6yvWLGiYq2zszO57pmILTsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBGHu3rDBSqWSl8vlho13Kq699tpkfcuWLRVrkyZNSq67adOmZH3y5MnJ+pkq79TgvOMLXnvttWR9+PDhFWt5U1GPGjUqWW9VpVJJ5XLZBqqxZQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIDifPZP3s8Sp85+ffvrp5LpR96PnGTIk/ef3yCOPJOt5+9m/+eabirXUFNyStHTp0mT9dMSWHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeC4Hx2tKy8893nzZuXrL/++usVa2edld7Obdy4MVm//vrrk/Vmqel8djNbaWYHzGxnv2WPmdnfzWx7drm1yIYBFG8wb+P/JGnmAMv/4O7Tssu7xbYFoGi5YXf3TZKqn/sIQEuo5Qu6+8xsR/Y2f0ylB5lZp5mVzazc29tbw3AAalFt2P8oaaqkaZK6Jf2u0gPdfYW7l9y91NbWVuVwAGpVVdjdvcfdj7v7CUkvSZpebFsAilZV2M1sQr+7syXtrPRYAK0h93x2M3tT0o2SxplZl6Slkm40s2mSXNJeSb+pX4uIKu9890cffTRZT+1nP3HiRHLdL774Illv1f3sKblhd/c5Ayx+pQ69AKgjDpcFgiDsQBCEHQiCsANBEHYgiDA/JZ13qO7555+frKem/0VzXHTRRXV77pdeeilZv/vuu+s2dr2wZQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIM6Y/ewfffRRsj5z5kC/mfn/rrrqqmR9y5YtFWvDhg1LrovTz9atW5vdQuHYsgNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEGfMfvabbropWTcbcBbbf5owYUKynrc+0OrYsgNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEGfMfvbx48cn63m/G//+++8n60ePHq1YGzp0aHJd1MfHH3/c7BZOK7lbdjObZGYbzWyXmX1uZr/Nlo81sw/NbHd2Pab+7QKo1mDexh+TtMjd/1XSv0laYGaXS1osaYO7XyppQ3YfQIvKDbu7d7v7tuz2YUm7JF0saZakVdnDVkm6rU49AijAKX1BZ2btkn4qaYukC929W+r7D0HSgB+azazTzMpmVs773AygfgYddjMbIekvkha6+9eDXc/dV7h7yd1LbW1t1fQIoACDCruZDVVf0Fe7+1+zxT1mNiGrT5B0oD4tAihC7q436zu38xVJu9z99/1K6yTNlfRUdr22Lh0OUk9PT7I+evToZP3rr9NvVm6++eaKtfXr1yfXHTFiRLJ+Jp8++91331Wsffvtt8l1ly1blqy/8847VfU0GPPmzavbczfLYPazz5DUIekzM9ueLVuivpCvMbP5kvZL+lVdOgRQiNywu/tmSZU2Pb8oth0A9cLhskAQhB0IgrADQRB2IAjCDgRxxpzimufTTz9N1q+88spkPTVl86hRo5LrLlmyJFnPW7+zszNZHz58eLJei8OHDyfrL7/8crK+evXqirUdO3ZU1VMR8vaj5/27Tkds2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCHP3hg1WKpW8XC43bLxTsW/fvmT9iiuuqFjLOy8b9ZF3fMHy5csr1jo6OopupyWUSiWVy+UBz1Jlyw4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQYQ5nz3P5MmTk/UjR440qBOgPtiyA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQuWE3s0lmttHMdpnZ52b222z5Y2b2dzPbnl1urX+7AKo1mINqjkla5O7bzGykpE/M7MOs9gd3f7Z+7QEoymDmZ++W1J3dPmxmuyRdXO/GABTrlD6zm1m7pJ9K+n4upPvMbIeZrTSzMRXW6TSzspmVe3t7a+sWQNUGHXYzGyHpL5IWuvvXkv4oaaqkaerb8v9uoPXcfYW7l9y91NbWVnvHAKoyqLCb2VD1BX21u/9Vkty9x92Pu/sJSS9Jml6/NgHUajDfxpukVyTtcvff91s+od/DZkvaWXx7AIoymG/jZ0jqkPSZmW3Pli2RNMfMpklySXsl/aYO/QEoyGC+jd8saaDfoX63+HYA1AtH0AFBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Iwd2/cYGa9kvb1WzRO0sGGNXBqWrW3Vu1LordqFdnbZHcf8PffGhr2Hw1uVnb3UtMaSGjV3lq1L4neqtWo3ngbDwRB2IEgmh32FU0eP6VVe2vVviR6q1ZDemvqZ3YAjdPsLTuABiHsQBBNCbuZzTSzL8xsj5ktbkYPlZjZXjP7LJuGutzkXlaa2QEz29lv2Vgz+9DMdmfXA86x16TeWmIa78Q040197Zo9/XnDP7Ob2dmS/kfSTZK6JG2VNMfd/7uhjVRgZnslldy96QdgmNnPJR2R9Kq7X5kte0bSIXd/KvuPcoy7P9givT0m6Uizp/HOZiua0H+acUm3SbpLTXztEn39pxrwujVjyz5d0h53/9Ld/yHpz5JmNaGPlufumyQdOmnxLEmrstur1PfH0nAVemsJ7t7t7tuy24clfT/NeFNfu0RfDdGMsF8s6W/97nepteZ7d0nrzewTM+tsdjMDuNDdu6W+Px5J45vcz8lyp/FupJOmGW+Z166a6c9r1YywDzSVVCvt/5vh7j+TdIukBdnbVQzOoKbxbpQBphlvCdVOf16rZoS9S9KkfvcnSvqqCX0MyN2/yq4PSHpLrTcVdc/3M+hm1wea3M8/tdI03gNNM64WeO2aOf15M8K+VdKlZjbFzIZJ+rWkdU3o40fMbHj2xYnMbLikX6r1pqJeJ2ludnuupLVN7OUHWmUa70rTjKvJr13Tpz9394ZfJN2qvm/k/1fSQ83ooUJfP5H0aXb5vNm9SXpTfW/rvlPfO6L5kv5F0gZJu7PrsS3U22uSPpO0Q33BmtCk3v5dfR8Nd0janl1ubfZrl+irIa8bh8sCQXAEHRAEYQeCIOxAEIQdCIKwA0EQdiAIwg4E8X8tH2VMpxTKBQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  5\n",
      "Ground truth annotation:  5\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAANQ0lEQVR4nO3db4hd9Z3H8c/HGEVilcSMbjTBqeKDVSGpDGEhS3QpWxIfOKnQJT6oKYaNiIqVPlhxwaoPVBbbUGQpTtfQZOlaCk0wD9RNCAX1Sc0kZDUxrv+IbZo4M6JS6x+aSb77YI7LGOeeO95z7j138n2/YLj3nu+553znJJ85d87v3vk5IgTgzHdW0w0A6A3CDiRB2IEkCDuQBGEHkji7lztbvHhxDA4O9nKXQCpHjhzR+++/75lqlcJue42kn0maJ+k/IuKxsvUHBwc1OjpaZZcASgwNDbWsdfwy3vY8Sf8uaa2kqyXdYvvqTrcHoLuq/M6+UtJbEfFORPxV0q8lDdfTFoC6VQn7ZZL+OO3x0WLZl9jeZHvU9ujExESF3QGookrYZ7oI8JX33kbESEQMRcTQwMBAhd0BqKJK2I9KWjbt8VJJx6q1A6BbqoR9r6SrbH/T9jmS1kvaWU9bAOrW8dBbREzavkvSf2tq6G1LRByqrTMAtao0zh4Rz0p6tqZeAHQRb5cFkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJFFpymbbRyR9LOmkpMmIGKqjKQD1qxT2wj9ExPs1bAdAF/EyHkiiathD0i7b+2xvmmkF25tsj9oenZiYqLg7AJ2qGvZVEXGdpLWS7rS9+vQVImIkIoYiYmhgYKDi7gB0qlLYI+JYcTsuaYeklXU0BaB+HYfd9gLb3/jivqTvSDpYV2MA6lXlavwlknbY/mI7/xURz9fSFb7kvffeK60vWbKkZa349+maa665prR+++23t6ytWLGi9LmrVq0qrXf7ezvTdBz2iHhH0vIaewHQRQy9AUkQdiAJwg4kQdiBJAg7kEQdH4RBRZ9++mlpffPmzaX1s85q7mf2a6+9Vlq/5557Ot72+Ph4af2iiy7qeNsZcWYHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQYZ+8Dzz9f/sngxx9/vONtn312+T/xwoULO962JH344Yel9cnJyY63/eKLL5bW161b1/G2M+LMDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMM7eA6dOnSqt7969u2v7fuKJJ0rrmzbNOGvXrK1Zs6a0XuV727VrV2n9pptuKq03+Tn/fsTRAJIg7EAShB1IgrADSRB2IAnCDiRB2IEkGGevQbtx9G3btpXWR0ZGKu3/gQceaFlbv359pW2fOHGitP7ZZ59V2n6ZJ598srTe7ntbvXp1ne3MeW3P7La32B63fXDaskW2d9t+s7it9hcQAHTdbF7G/1LS6W+Tuk/Snoi4StKe4jGAPtY27BHxgqQPTls8LGlrcX+rpHX1tgWgbp1eoLskIo5LUnF7casVbW+yPWp7dGJiosPdAaiq61fjI2IkIoYiYmhgYKDbuwPQQqdhH7O9RJKK2/LpNgE0rtOw75S0obi/QdIz9bQDoFvajrPbflrSDZIW2z4q6ceSHpP0G9sbJf1B0ve62WS/aze/+saNGytt//LLLy+t33vvvS1rF1xwQaV9P/LII6X1l156qdL2q1i+fHlj+56L2oY9Im5pUfp2zb0A6CLeLgskQdiBJAg7kARhB5Ig7EASfMR1Djj//PNL6+eee27H237jjTdK6+0+ZtpN7f5U9HnnndejTs4MnNmBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnG2eeAQ4cOldaHh4db1pYuXVr63Oeee660PjY2VlrvpiuuuKK0fs455/SokzMDZ3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9jPA7t27m26hK3bs2FFaf/jhh0vrCxYsqLOdOY8zO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwTg7KrnyyitL62+//XbH23733XdL65OTkx1vO6O2Z3bbW2yP2z44bdmDtv9k+0DxdWN32wRQ1Wxexv9S0poZlm+OiBXF17P1tgWgbm3DHhEvSPqgB70A6KIqF+jusv1K8TJ/YauVbG+yPWp7dGJiosLuAFTRadh/LulKSSskHZf0k1YrRsRIRAxFxNDAwECHuwNQVUdhj4ixiDgZEack/ULSynrbAlC3jsJue8m0h9+VdLDVugD6Q9txdttPS7pB0mLbRyX9WNINtldICklHJN3evRb7X7t5wvfu3Vtav/nmm0vrH330UWl97dq1pfUyg4ODpfXbbruttN7u78pff/31X7cldEnbsEfELTMsfqoLvQDoIt4uCyRB2IEkCDuQBGEHkiDsQBJ8xLUG8+bNK61fd911pfX9+/eX1j///PPS+qWXXlpa76ZPPvmksX0fO3astH7hhRf2qJO5gTM7kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBOHsfWLRoUdMtzEnbtm0rrT/66KM96mRu4MwOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzo45a9myZaX1EydOtKzNnz+/7nb6Hmd2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcXbMWXfffXdp/dZbb21ZY5x9BraX2f6d7cO2D9m+p1i+yPZu228Wtwu73y6ATs3mZfykpB9FxN9K+jtJd9q+WtJ9kvZExFWS9hSPAfSptmGPiOMRsb+4/7Gkw5IukzQsaWux2lZJ67rUI4AafK0LdLYHJX1L0u8lXRIRx6WpHwiSLm7xnE22R22PTkxMVGwXQKdmHXbb50v6raQfRsSfZ/u8iBiJiKGIGBoYGOikRwA1mFXYbc/XVNB/FRHbi8VjtpcU9SWSxrvTIoA6zOZqvCU9JelwRPx0WmmnpA3F/Q2Snqm/PQB1mc04+ypJ35f0qu0DxbL7JT0m6Te2N0r6g6TvdaVDALVoG/aIeEmSW5S/XW87ALqFt8sCSRB2IAnCDiRB2IEkCDuQBB9xRSXXXnttaX3lypUtay+//HLd7aAEZ3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJxdlTy+uuvl9b37dvXo07QDmd2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcXZUMjk5WVo/efJkjzpBO5zZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJtuPstpdJ2ibpbySdkjQSET+z/aCkf5Y0Uax6f0Q8261G0Z+WL19eWt++fXvL2h133FH63LGxsdL6Qw89VFqfP39+aT2b2bypZlLSjyJiv+1vSNpne3dR2xwRj3evPQB1mc387MclHS/uf2z7sKTLut0YgHp9rd/ZbQ9K+pak3xeL7rL9iu0tthe2eM4m26O2RycmJmZaBUAPzDrsts+X9FtJP4yIP0v6uaQrJa3Q1Jn/JzM9LyJGImIoIoYGBgaqdwygI7MKu+35mgr6ryJiuyRFxFhEnIyIU5J+Ian1DH4AGtc27LYt6SlJhyPip9OWL5m22nclHay/PQB1mc3V+FWSvi/pVdsHimX3S7rF9gpJIemIpNu70B/63NS5oLXh4eGOaqjfbK7GvyRppn9RxtSBOYR30AFJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JwRPRuZ/aEpHenLVos6f2eNfD19Gtv/dqXRG+dqrO3yyNixr//1tOwf2Xn9mhEDDXWQIl+7a1f+5LorVO96o2X8UAShB1IoumwjzS8/zL92lu/9iXRW6d60lujv7MD6J2mz+wAeoSwA0k0Enbba2z/r+23bN/XRA+t2D5i+1XbB2yPNtzLFtvjtg9OW7bI9m7bbxa3M86x11BvD9r+U3HsDti+saHeltn+ne3Dtg/ZvqdY3uixK+mrJ8et57+z254n6Q1J/yjpqKS9km6JiNd62kgLto9IGoqIxt+AYXu1pL9I2hYR1xbL/k3SBxHxWPGDcmFE/Euf9PagpL80PY13MVvRkunTjEtaJ+kHavDYlfT1T+rBcWvizL5S0lsR8U5E/FXSryUxNcgMIuIFSR+ctnhY0tbi/lZN/WfpuRa99YWIOB4R+4v7H0v6YprxRo9dSV890UTYL5P0x2mPj6q/5nsPSbts77O9qelmZnBJRByXpv7zSLq44X5O13Ya7146bZrxvjl2nUx/XlUTYZ9pKql+Gv9bFRHXSVor6c7i5SpmZ1bTePfKDNOM94VOpz+vqomwH5W0bNrjpZKONdDHjCLiWHE7LmmH+m8q6rEvZtAtbscb7uf/9dM03jNNM64+OHZNTn/eRNj3SrrK9jdtnyNpvaSdDfTxFbYXFBdOZHuBpO+o/6ai3ilpQ3F/g6RnGuzlS/plGu9W04yr4WPX+PTnEdHzL0k3auqK/NuS/rWJHlr0dYWk/ym+DjXdm6SnNfWy7oSmXhFtlHSRpD2S3ixuF/VRb/8p6VVJr2gqWEsa6u3vNfWr4SuSDhRfNzZ97Er66slx4+2yQBK8gw5IgrADSRB2IAnCDiRB2IEkCDuQBGEHkvg/ObPpX/CFxeQAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  9\n",
      "Ground truth annotation:  9\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAM10lEQVR4nO3dYahc9ZnH8d9P24re9EXcXOPFhk2sCqsGk3CJC1maLHWLCjFWyZIoJQVpiqikUHBFX1R8pXXb0hdLIV1D06WbUEwkeSFrQghKIhSvEjVusNoQ01tDMiGBWkGj5tkX97hcr3fO3Mw5M2eS5/uBYWbOc86ch8P95cyc/0z+jggBuPBd1HQDAPqDsANJEHYgCcIOJEHYgSS+0s+dzZkzJ+bPn9/PXQKpHDlyRCdPnvR0tUpht32rpF9KuljSf0bEk2Xrz58/X2NjY1V2CaDE6Oho21rXb+NtXyzpPyTdJul6SWttX9/t6wHorSqf2ZdKejciDkfEGUlbJa2qpy0AdasS9qsk/XnS8/Fi2RfYXm97zPZYq9WqsDsAVVQJ+3QXAb703duI2BgRoxExOjw8XGF3AKqoEvZxSfMmPf+GpPertQOgV6qE/RVJ19peYPtrktZI2llPWwDq1vXQW0R8avtBSS9oYuhtU0S8VVtnAGpVaZw9Ip6X9HxNvQDoIb4uCyRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKVZnEFOnn66afb1h5++OHSbZ999tnS+t13391VT1lVCrvtI5I+kPSZpE8jYrSOpgDUr44z+z9HxMkaXgdAD/GZHUiiathD0i7br9peP90KttfbHrM91mq1Ku4OQLeqhn1ZRCyRdJukB2x/a+oKEbExIkYjYnR4eLji7gB0q1LYI+L94v6EpOckLa2jKQD16zrstodsf/3zx5K+I+lgXY0BqFeVq/FzJT1n+/PX+e+I+J9aurrAvP7666X1o0ePltZXrlxZZzt9Vfx9nHNNks6ePVt3O6l1HfaIOCzpphp7AdBDDL0BSRB2IAnCDiRB2IEkCDuQBD9x7YP33nuvtP7QQw+V1s/nobdrrrmm6223bt1aWl+9enXXr50RZ3YgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIJx9gGwcOHCplvomUWLFnW97dVXX11fI+DMDmRB2IEkCDuQBGEHkiDsQBKEHUiCsANJMM7eB7t37y6t33zzzX3q5PyyePHiplu4oHBmB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGevwccff1xa37JlS2l9w4YNdbZzwdi2bVtp/Z577ulTJxeGjmd225tsn7B9cNKyy23vtv1OcT+7t20CqGomb+N/I+nWKcsekbQnIq6VtKd4DmCAdQx7RLwk6dSUxaskbS4eb5Z0Z71tAahbtxfo5kbEMUkq7q9ot6Lt9bbHbI+1Wq0udwegqp5fjY+IjRExGhGjw8PDvd4dgDa6Dftx2yOSVNyfqK8lAL3Qbdh3SlpXPF4naUc97QDolY7j7La3SFohaY7tcUk/kfSkpN/bvk/SUUmpJ8o+c+ZMaf3UqanXN/MYGRnpqob6dQx7RKxtU/p2zb0A6CG+LgskQdiBJAg7kARhB5Ig7EAS/MS1Bvv27au0/fLly2vqZPBccsklbWtz584t3fbDDz8srZ89e7a0ftFFnMsm42gASRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKMsw+A06dPN91CI1avLv9l9GOPPVZa/+ijj0rrl1122Tn3dCHjzA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOPgD43TX6gb8yIAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCcfYBYLvpFnrm5Zdfblt78cUXK732+Ph4af26666r9PoXmo5ndtubbJ+wfXDSssdt/8X2geJ2e2/bBFDVTN7G/0bSrdMs/0VELCpuz9fbFoC6dQx7RLwk6VQfegHQQ1Uu0D1o+43ibf7sdivZXm97zPZYq9WqsDsAVXQb9l9J+qakRZKOSfpZuxUjYmNEjEbE6PDwcJe7A1BVV2GPiOMR8VlEnJX0a0lL620LQN26CrvtkUlPvyvpYLt1AQyGjuPstrdIWiFpju1xST+RtML2Ikkh6YikH/auxQvf/v37S+s33XRTz/Z9+PDh0vrevXtL69u3by+tv/32221rn3zySem2nZS9tsQ4+1Qdwx4Ra6dZ/EwPegHQQ3xdFkiCsANJEHYgCcIOJEHYgST4iWsNVqxYUVq/5ZZbSutPPfVUpXovDQ0NldaXLVtWWr///vvb1l544YXSbXfu3Flax7nhzA4kQdiBJAg7kARhB5Ig7EAShB1IgrADSTDOXoNLL720tL5jx45K9V27dpXW58yZ07a2ZMmS0m1vuOGG0vqCBQtK67NmzSqtl1m+fHlpnXH2enFmB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGfvg07j8GvWrKlUP1+dPn266RZS4cwOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kwzo7GXHnllU23kErHM7vtebb32j5k+y3bG4rll9vebfud4n5279sF0K2ZvI3/VNKPI+IfJP2jpAdsXy/pEUl7IuJaSXuK5wAGVMewR8SxiHitePyBpEOSrpK0StLmYrXNku7sUY8AanBOF+hsz5e0WNIfJM2NiGPSxD8Ikq5os81622O2x1qtVsV2AXRrxmG3PUvSNkk/ioi/znS7iNgYEaMRMTo8PNxNjwBqMKOw2/6qJoL+u4jYXiw+bnukqI9IOtGbFgHUoePQm21LekbSoYj4+aTSTknrJD1Z3Jf/f8jAFJ2G3mbPZoCnTjMZZ18m6XuS3rR9oFj2qCZC/nvb90k6Kml1TzoEUIuOYY+IfZLcpvztetsB0Ct8XRZIgrADSRB2IAnCDiRB2IEk+IkrGjM0NFRav+OOO0rr+/fvL62vXLnynHu6kHFmB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkGGfHwFqwYEFp/Yknniit33vvvW1rCxcu7Kqn8xlndiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgnF2DKzly5eX1u+6667S+o033lhnO+c9zuxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kMRM5mefJ+m3kq6UdFbSxoj4pe3HJf1AUqtY9dGIeL5XjSKfTuPsner4opl8qeZTST+OiNdsf13Sq7Z3F7VfRMS/9649AHWZyfzsxyQdKx5/YPuQpKt63RiAep3TZ3bb8yUtlvSHYtGDtt+wvcn27DbbrLc9Znus1WpNtwqAPphx2G3PkrRN0o8i4q+SfiXpm5IWaeLM/7PptouIjRExGhGjw8PD1TsG0JUZhd32VzUR9N9FxHZJiojjEfFZRJyV9GtJS3vXJoCqOobdtiU9I+lQRPx80vKRSat9V9LB+tsDUJeZXI1fJul7kt60faBY9qiktbYXSQpJRyT9sAf9AajJTK7G75PkaUqMqQPnEb5BByRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSMIR0b+d2S1J701aNEfSyb41cG4GtbdB7Uuit27V2dvfR8S0//9bX8P+pZ3bYxEx2lgDJQa1t0HtS6K3bvWrN97GA0kQdiCJpsO+seH9lxnU3ga1L4neutWX3hr9zA6gf5o+swPoE8IOJNFI2G3favtt2+/afqSJHtqxfcT2m7YP2B5ruJdNtk/YPjhp2eW2d9t+p7ifdo69hnp73PZfimN3wPbtDfU2z/Ze24dsv2V7Q7G80WNX0ldfjlvfP7PbvljSHyX9i6RxSa9IWhsR/9vXRtqwfUTSaEQ0/gUM29+S9DdJv42IG4tlP5V0KiKeLP6hnB0R/zYgvT0u6W9NT+NdzFY0MnmacUl3Svq+Gjx2JX39q/pw3Jo4sy+V9G5EHI6IM5K2SlrVQB8DLyJeknRqyuJVkjYXjzdr4o+l79r0NhAi4lhEvFY8/kDS59OMN3rsSvrqiybCfpWkP096Pq7Bmu89JO2y/art9U03M425EXFMmvjjkXRFw/1M1XEa736aMs34wBy7bqY/r6qJsE83ldQgjf8ti4glkm6T9EDxdhUzM6NpvPtlmmnGB0K3059X1UTYxyXNm/T8G5Leb6CPaUXE+8X9CUnPafCmoj7++Qy6xf2Jhvv5f4M0jfd004xrAI5dk9OfNxH2VyRda3uB7a9JWiNpZwN9fIntoeLCiWwPSfqOBm8q6p2S1hWP10na0WAvXzAo03i3m2ZcDR+7xqc/j4i+3yTdrokr8n+S9FgTPbTp62pJrxe3t5ruTdIWTbyt+0QT74juk/R3kvZIeqe4v3yAevsvSW9KekMTwRppqLd/0sRHwzckHShutzd97Er66stx4+uyQBJ8gw5IgrADSRB2IAnCDiRB2IEkCDuQBGEHkvg/GjLH0QSMC4EAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  4\n",
      "Ground truth annotation:  4\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAN5UlEQVR4nO3db4xU9b3H8c8XWwJCE8Fdybo10tvwoMYIbQa8RkSu9Rb/PEAwbcoDgomGJv5Ji425WBOrz/TmtsQHphGEwDW9Nk2KkQdoa7DR8IQ4GK7CRbtIaFlcl8F/UFfT6n77YA83W9j5zXLOmTkj3/crmczM+c7Z82V2P5yZ+c05P3N3ATj/Tam6AQCdQdiBIAg7EARhB4Ig7EAQX+nkxnp6enzu3Lmd3CQQypEjR3TixAmbqFYo7GZ2k6QnJF0g6Wl3fyz1+Llz56perxfZJICEWq3WtJb7ZbyZXSDpSUk3S7pC0iozuyLvzwPQXkXesy+SdMjdD7v73yT9RtLyctoCULYiYe+XdHTc/cFs2T8xs7VmVjezeqPRKLA5AEUUCftEHwKc9d1bd9/o7jV3r/X29hbYHIAiioR9UNJl4+5/XdK7xdoB0C5Fwv6apHlm9g0zmyrph5J2lNMWgLLlHnpz98/N7F5Jv9fY0NsWdz9QWmcASlVonN3dd0raWVIvANqIr8sCQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxBER08ljXx27kwfWHjXXXc1rQ0PDyfXHR0dTdanTCm2P1i3bl3T2kMPPZRcd8aMGcn61KlTc/UUFXt2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQjC3M+axKVtarWaM4vr2T7++ONkff78+cn64OBg7m23+v2bTTj7b0csWbIkWX/88ceT9YULF5bZzpdCrVZTvV6f8JfGnh0IgrADQRB2IAjCDgRB2IEgCDsQBGEHguB49i7wzDPPJOtFxtG/zF555ZVkfdmyZcn6Cy+80LR29dVX5+rpy6xQ2M3siKRTkr6Q9Lm718poCkD5ytiz/5u7nyjh5wBoI96zA0EUDbtL+oOZ7TWztRM9wMzWmlndzOqNRqPg5gDkVTTs17r7dyTdLOkeMzvryAV33+juNXev9fb2FtwcgLwKhd3d382uj0t6TtKiMpoCUL7cYTezGWb2tdO3JX1P0v6yGgNQriKfxs+R9Fx2vPNXJP2Pu79YSlfBHD58uOoWvpROnjyZrN9///1Na7t27UquO23atFw9dbPcYXf3w5LSZ1UA0DUYegOCIOxAEIQdCIKwA0EQdiAIDnHtAitXrkzWL7300rZt+8Ybb0zW582bV+jnb9++vWntvvvuS6576tSpQtves2dP09qHH36YXLevr6/QtrsRe3YgCMIOBEHYgSAIOxAEYQeCIOxAEIQdCIJx9i6wePHiQvVutnr16qa1VlMyX3nllcn6yMhIrp6k1lNVn4/YswNBEHYgCMIOBEHYgSAIOxAEYQeCIOxAEIyzozKXX355sv7www8n6w8++GDubWenQA+FPTsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSBaht3MtpjZcTPbP27ZbDN7ycwGsutZ7W0TQFGT2bNvlXTTGcvWS9rl7vMk7cruA+hiLcPu7q9K+uCMxcslbctub5N0W7ltAShb3vfsc9x9SJKy60uaPdDM1ppZ3czqjUYj5+YAFNX2D+jcfaO719y91tvb2+7NAWgib9iHzaxPkrLr4+W1BKAd8oZ9h6Q12e01kp4vpx0A7dLyeHYze1bSUkk9ZjYo6eeSHpP0WzO7U9JfJH2/nU0ipg0bNlTdwnmlZdjdfVWT0ndL7gVAG/ENOiAIwg4EQdiBIAg7EARhB4LgVNKozAMPPJCsDw8PJ+utTge9YsWKprU5c+Yk1z0fsWcHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAYZ0dbDQwMNK099dRTyXVHR0eT9VZTPm/atKlpbcqUePu5eP9iICjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCcXYU8tZbbyXr119/fdPayMhIct1WY+HXXHNNsn7RRRcl69GwZweCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIBhnz3z66afJ+jvvvNOhTs5dT09P09q0adOS67Yaix4aGkrW77777mT9/fffT9ZTFi9enKynjlfH2Vru2c1si5kdN7P945Y9YmbHzGxfdrmlvW0CKGoyL+O3SrppguUb3H1BdtlZblsAytYy7O7+qqQPOtALgDYq8gHdvWb2RvYyf1azB5nZWjOrm1m90WgU2ByAIvKG/VeSvilpgaQhSb9o9kB33+juNXev9fb25twcgKJyhd3dh939C3cflbRJ0qJy2wJQtlxhN7O+cXdXSNrf7LEAukPLcXYze1bSUkk9ZjYo6eeSlprZAkku6YikH7Wvxc5Yv359sv7kk0/m/tnunqy3mme8lYsvvrhpbfr06cl1+/v7k/VDhw4l60XG0Wu1WrL+xBNPJOszZ87Mve2IWobd3VdNsHhzG3oB0EZ8XRYIgrADQRB2IAjCDgRB2IEgwhzi+t577yXrTz/9dIc6KV+R4a+jR48m60WHBVNefPHFZJ1TQZeLPTsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBBFmnH10dDRZ/+yzzyrbdqupidupm3tDufhNAkEQdiAIwg4EQdiBIAg7EARhB4Ig7EAQYcbZZ8+enazffvvtyfr27dtzb7vVWHU7jxlvpcreli1blqw/+uijyfrSpUuT9VbTVUfDnh0IgrADQRB2IAjCDgRB2IEgCDsQBGEHgggzzt5qzHXbtm3J+oEDB5rW3n777Vw9dYOrrroqWf/oo4+S9VbnnU/Zu3dvsn7rrbcm63fccUeyvnkzkw2P13LPbmaXmdkfzeygmR0wsx9ny2eb2UtmNpBdz2p/uwDymszL+M8l/dTdvyXpXyXdY2ZXSFovaZe7z5O0K7sPoEu1DLu7D7n769ntU5IOSuqXtFzS6de+2yTd1qYeAZTgnD6gM7O5kr4taY+kOe4+JI39hyDpkibrrDWzupnVG41GwXYB5DXpsJvZTEm/k/QTdz852fXcfaO719y91tvbm6dHACWYVNjN7KsaC/qv3f304V/DZtaX1fskHW9PiwDK0HLozcaOcdws6aC7/3JcaYekNZIey66fb0uHHTJ9+vRkfevWrU1rN9xwQ3LdkZGRPC2VolarJesvv/xyst5qOuglS5Yk60WG5loZGBho288+H01mnP1aSaslvWlm+7JlP9NYyH9rZndK+ouk77elQwClaBl2d98tqdkZDL5bbjsA2oWvywJBEHYgCMIOBEHYgSAIOxBEmENci1q4cGHT2u7du5PrfvLJJ8n6unXrkvVWY+UrV65sWrvuuuuS606dOjVZv/DCC5P1Vv/2RYsWNa0NDw8n121l/XqOvToX7NmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjG2Uswf/78Quvv2bOnpE46r7+/P1k/duxYhzpBK+zZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQRB2IIiWYTezy8zsj2Z20MwOmNmPs+WPmNkxM9uXXW5pf7sA8prMySs+l/RTd3/dzL4maa+ZvZTVNrj7f7WvPQBlmcz87EOShrLbp8zsoKT06UkAdJ1zes9uZnMlfVvS6fMo3Wtmb5jZFjOb1WSdtWZWN7N6o9Eo1i2A3CYddjObKel3kn7i7icl/UrSNyUt0Nie/xcTrefuG9295u613t7e4h0DyGVSYTezr2os6L929+2S5O7D7v6Fu49K2iSp+Qx+ACo3mU/jTdJmSQfd/ZfjlveNe9gKSfvLbw9AWSbzafy1klZLetPM9mXLfiZplZktkOSSjkj6URv6A1CSyXwav1uSTVDaWX47ANqFb9ABQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAIOxAEYQeCMHfv3MbMGpL+PG5Rj6QTHWvg3HRrb93al0RveZXZ2+XuPuH53zoa9rM2blZ391plDSR0a2/d2pdEb3l1qjdexgNBEHYgiKrDvrHi7ad0a2/d2pdEb3l1pLdK37MD6Jyq9+wAOoSwA0FUEnYzu8nM3jazQ2a2vooemjGzI2b2ZjYNdb3iXraY2XEz2z9u2Wwze8nMBrLrCefYq6i3rpjGOzHNeKXPXdXTn3f8PbuZXSDpT5L+XdKgpNckrXL3/+toI02Y2RFJNXev/AsYZrZE0l8l/be7X5kt+09JH7j7Y9l/lLPc/T+6pLdHJP216mm8s9mK+sZPMy7pNkl3qMLnLtHXD9SB562KPfsiSYfc/bC7/03SbyQtr6CPrufur0r64IzFyyVty25v09gfS8c16a0ruPuQu7+e3T4l6fQ045U+d4m+OqKKsPdLOjru/qC6a753l/QHM9trZmurbmYCc9x9SBr745F0ScX9nKnlNN6ddMY0413z3OWZ/ryoKsI+0VRS3TT+d627f0fSzZLuyV6uYnImNY13p0wwzXhXyDv9eVFVhH1Q0mXj7n9d0rsV9DEhd383uz4u6Tl131TUw6dn0M2uj1fcz//rpmm8J5pmXF3w3FU5/XkVYX9N0jwz+4aZTZX0Q0k7KujjLGY2I/vgRGY2Q9L31H1TUe+QtCa7vUbS8xX28k+6ZRrvZtOMq+LnrvLpz9294xdJt2jsE/l3JD1URQ9N+voXSf+bXQ5U3ZukZzX2su7vGntFdKekiyXtkjSQXc/uot6ekfSmpDc0Fqy+inpbrLG3hm9I2pddbqn6uUv01ZHnja/LAkHwDTogCMIOBEHYgSAIOxAEYQeCIOxAEIQdCOIfUhgjdXlMMbsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network prediction:  2\n",
      "Ground truth annotation:  2\n"
     ]
    }
   ],
   "source": [
    "# predict on training split\n",
    "prediction = nn.predict(x_train)\n",
    "print(\"Loss on TRAIN split: {}\".format(MSE.loss(prediction, y_train)))\n",
    "print(\"Accuracy on TRAIN split: {}\".format(accuracy(prediction, y_train)))\n",
    "\n",
    "print(\"------------------------\")\n",
    "\n",
    "# predict on testing split\n",
    "prediction = nn.predict(x_test)\n",
    "print(\"Loss on TEST split: {}\".format(MSE.loss(prediction, y_test)))\n",
    "print(\"Accuracy on TEST split: {}\".format(accuracy(prediction, y_test)))\n",
    "\n",
    "# visualize the image and label from the training set\n",
    "\n",
    "\n",
    "for _ in range(10):\n",
    "    idx = np.random.randint(0, x_test.shape[0])\n",
    "    example_img = np.resize(x_test[idx],(28,28))\n",
    "    plt.imshow(example_img,cmap=\"Greys\")\n",
    "    plt.show()\n",
    "    print ('Network prediction: ',prediction[idx].argmax())\n",
    "    print ('Ground truth annotation: ',y_test[idx].argmax())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play around with the architecture, activation functions and hyperparameters to increase the accuracy on the TEST split. You should be able to reach 96% accuracy! (Note that, in principle, one should not tune their model to the test data, but create a validation set.)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.10 ('py37torch2')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "635196d2e7d96e8482f8171ea2c3eea7d8992e2ed178bb98cf4adc6e0453abe3"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}