16. Using a Neural Network to Recognise Hand-Written Digits.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "import numpy as np\n",
    "np.set_printoptions(precision=3)\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using a Neural Network to recognise hand-written digits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In previous examples, we worked with a data set of 10 points. Each data point was a geometrical point in a plane, containing 2 numbers to encode point's $x$ and $y$ coordinates. Each of the data points was labeled with two possible labels, `1` and `-1`. Our goal was to predict the label of an unknown point given our 10 data points. We achieved this by defining a cost function, and using calculus (gradient descent) to minimise it. In each epoch of learning we found the derivatives of weights and biases for all 10 data points in our dataset, and then we averaged over them.<br>\n",
    "For a simple task, such is binary classification of 2 dimensional points, 10 data points were sufficient. However, for more complex tasks such is recognising written digits, situation is the following:\n",
    "- Each data point (an image of a handwritten digit) contains 784 numbers (brightness values of each pixel)\n",
    "- Each data point is labeled with 10 possible labels (0, 1, 2, 3, 4, 5, 6, 7, 8 or 9)\n",
    "- Algorithm needs to learn as many as possible multiple ways of writting a digit by hand.\n",
    "\n",
    "For this reason, we need a large data set! We will be working with a MNIST dataset, containing 70'000 grayscale images of hand written digits, 28x28 pixels each. If we used the same strategy as before, we would need to compute gradients of 70'000 data points in one batch! That would be too ineficient. For that reason, we will be using the technique called __stochastic gradient descent__. The idea is to split the data set into smaller managable chunks called *mini batches*, and within each epoch train only a single mini batch. In this way, we will still be able to minimize our cost function by making small steps at a time. The difference is that the steps taken will be less precise than if we worked with the whole dataset at a time, but over a large number of epochs, the steps will still converge towards the function's minimum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with datasets in machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our previous example, we worked with a dataset of 10 data points. We used those points and their labels to tweak weights and biases of our neural network so that we can predict label of new unseen points.<br>\n",
    "The part of the dataset that is used to train the neural network (tweak its weights and biases) is called a __training set__.\n",
    "We used the visualisations to indicate how well our network is trained. To create visualisations, we defined a grid of $(x,y)$ points and passed it to our trained algorithm as inputs. The algorithm then predicted a matching label for each point, and visualised them on the screen, along with our original inputs. The grid that we used to visualise the results of our network is an example of a __testing set__ in machine learning. Testing set contains all the data that wasn't part of the original dataset, so it was never used to tweak weights and biases of the neural network. Its role is to test the prediction quality of our neural network.\n",
    "The last type of dataset used in machine learning is called a __validation set__. Its role is two-fold. Its first role is to tweak meta-parameters of our neural network or test different models in order to improve prediction. The second, and more important role is that it is used to prevent __overfitting__. Overfitting means that our network has learned our training set *to well* that it became biased towards it. It will acurately predict all the training data and the similar data, but it will fail to generalise to data beyond it. <br> If we have a large amount of data, we should first split it into these categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our dataset, the data is already split into the training, validation and testing set. We first load the data from the archive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with gzip.open('data/mnist.pkl.gz', 'rb') as f:\n",
    "    train_set, valid_set, test_set = pickle.load(f, encoding='latin1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the amount of data points in the _training set_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50000"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_set[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the amount of points in the _validation set_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(valid_set[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the amount of points in the _testing set_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(valid_set[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these datasets can be split into data points and their labels. Here we will split the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_points, train_labels = train_set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wish to see the any particular data point we can do it with indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.012,\n",
       "       0.07 , 0.07 , 0.07 , 0.492, 0.531, 0.684, 0.102, 0.648, 0.996,\n",
       "       0.965, 0.496, 0.   , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   , 0.   , 0.   , 0.   , 0.117, 0.141, 0.367, 0.602,\n",
       "       0.664, 0.988, 0.988, 0.988, 0.988, 0.988, 0.879, 0.672, 0.988,\n",
       "       0.945, 0.762, 0.25 , 0.   , 0.   , 0.   , 0.   , 0.   , 0.   ,\n",
       "       0.   , 0.   ], dtype=float32)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_points[0][:200] #:200 means first 200 values (out of 784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Its corresponding label is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also visualise it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAH0CAYAAADVH+85AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHQBJREFUeJzt3X2wJXV5J/DvI5PACvKixlgmm0XcAEoiLhBAKJWXwpdNxWgEy8qLlKUpE8kiBi1TicZRs4mp2qwiuJpEIxWo2omFCSlWomwBKgaySYYoawXfQaQi4jgLOCLEYX77x+mJk5t75+X0mXvu/Z3Pp+pU39Pdz/k909N1v7fP6e5TrbUAAH161LwbAAD2H0EPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB3bMO8G9oequiPJoUnunHMrADCtI5M80Fp78pgX6TLoMwn5xw4PAFhYc33rvqp+tKr+pKr+qaoerqo7q+pdVXXEyJe+cxb9AcCc3Tn2BeZ2RF9VT0lyc5InJPnLJJ9LcnKS1yZ5flWd3lr71rz6A4AezPOI/n9kEvIXttZe1Fr7jdbaWUnemeSYJP91jr0BQBeqtbb6g1YdleTLmbwl8ZTW2o5dlj0mydeTVJIntNa+M8Xrb05ywmy6BYC5ubW1duKYF5jXEf1Zw/S6XUM+SVpr307y10keneTU1W4MAHoyr8/ojxmmX1hh+ReTPDfJ0UmuX+lFhiP35Rw7fWsA0I95HdEfNkzvX2H5zvmHr0IvANCttXodfQ3T3Z5AsNLnFj6jB4CJeR3R7zxiP2yF5YcuWQ8AmMK8gv7zw/ToFZb/+DBd6TN8AGAvzCvobxymz62qf9XDcHnd6Um+m+RvVrsxAOjJXIK+tfblJNdlcsP+C5YsfmuSg5P86TTX0AMA3zfPk/Fek8ktcN9dVWcnuT3JKUnOzOQt+9+aY28A0IW53QJ3OKo/KcnlmQT8xUmekuTdSZ7pPvcAMN5cL69rrX0tySvm2QMA9GyuX1MLAOxfgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOrZh3g0A0zvggANG1R9xxBEz6mT1bdy4ceraQw45ZNTYT3va06auPffcc0eNfeWVV05d+6xnPWvU2Nu3b5+69o/+6I9GjX3BBReMql9kczuir6o7q6qt8LhnXn0BQE/mfUR/f5J3LTN/22o3AgA9mnfQ39da2zjnHgCgW07GA4COzfuI/sCq+sUkP5bkO0luS/LJ1toj820LAPow76B/YpIrlsy7o6pe0Vr7xJ6Kq2rzCouOHd0ZAHRgnm/dfzDJ2ZmE/cFJfjLJHyY5MslfVdXx82sNAPowtyP61tpbl8z6bJJfqaptSS5OsjHJi/fwGicuN3840j9hBm0CwLq2Fk/Ge98wffZcuwCADqzFoL93mB481y4AoANrMeifOUy/MtcuAKADcwn6qjquqh67zPz/kOSy4en0N3QGAJLM72S885L8RlXdmOSOJN9O8pQkP53koCTXJvlvc+oNALoxr6C/MckxSf5TJm/VH5zkviSfyuS6+itaa21OvQFAN+YS9MPNcPZ4QxzYW0cdddSo+oMOOmjq2uc973mjxj7nnHOmrj388MNHjX3qqaeOql9UDzzwwNS1H/rQh0aNffLJJ09d+/DDD48a+2tf+9rUtddff/2osZneWjwZDwCYEUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQsWqtzbuHmauqzUlOmHcf7JtnPetZU9ded911o8Y+8MADR9Wzvoz9vXfxxRdPXbtt27ZRY48x5vvkk+See+6ZuvYzn/nMqLEX2K2ttRPHvIAjegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI75mlrWjMc//vFT137+858fNfYRRxwxqn4R3XHHHaPqv/3tb4+qP+6446aufeSRR0aNfdBBB42qh33ga2oBgJUJegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI5tmHcDsNOWLVumrn3DG94wauyXvvSlU9fecssto8Z+y1veMqp+jLvvvnvq2uOPP37U2Nu2bRtVf9JJJ01d+7a3vW3U2LCeOKIHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDoWLXW5t3DzFXV5iQnzLsP1o/DDz986tr7779/1Ngf+chHpq59/vOfP2rs1772tVPXXnrppaPGBvbKra21E8e8wEyO6Kvq3Kq6tKpuqqoHqqpV1ZV7qDmtqq6tqq1V9WBV3VZVF1XVAbPoCQBINszodd6U5Pgk25LcneTY3a1cVT+b5MNJHkryZ0m2JvmZJO9McnqS82bUFwAstFl9Rv+6JEcnOTTJr+5uxao6NMkfJ3kkyRmttVe21t6Q5BlJbklyblW9bEZ9AcBCm0nQt9ZubK19se3dB/7nJvmhJJtaa3+/y2s8lMk7A8ke/lgAAPbOPM66P2uYfnSZZZ9M8mCS06rqwNVrCQD6NI+gP2aYfmHpgtba9iR3ZHLuwFGr2RQA9GhWJ+Pti8OG6UrXJO2cv8frnYbL6Jaz25MBAWBRrMUb5tQw7e8CfwBYZfM4ot95xH7YCssPXbLeila6iYAb5gDAxDyO6D8/TI9euqCqNiR5cpLtSb6ymk0BQI/mEfQ3DNPl7t357CSPTnJza+3h1WsJAPo0j6C/KsmWJC+rqpN2zqyqg5L8zvD0vXPoCwC6M5PP6KvqRUleNDx94jB9ZlVdPvy8pbX2+iRprT1QVb+cSeB/vKo2ZXIL3BdmcundVZncFhcAGGlWJ+M9I8n5S+Ydle9fC//VJK/fuaC1dnVVPSfJbyV5SZKDknwpya8nefde3mEPANiDmQR9a21jko37WPPXSf7zLMYHAJY3j8vrYM2577775jb21q1b5zb2a17zmqlr3/Oe94wae8eOHaPqgb2zFm+YAwDMiKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI5Va23ePcxcVW1OcsK8+4C9ccghh0xd+3d/93ejxj7mmGOmrv35n//5UWNv2rRpVD0siFtbayeOeQFH9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMd9HD+vYU5/61FH1//AP/zB17UMPPTRq7M2bN4+qv+mmm6aufetb3zpq7B5/b7Jm+T56AGBlgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOrZh3g0A07v99ttH1V9wwQVT11522WWjxj7zzDPnVn/IIYeMGvuSSy6ZuvZrX/vaqLFhXzmiB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COVWtt3j3MXFVtTnLCvPuAnp1yyimj6j/wgQ+Mqn/a0542qn6Ma665ZuraCy+8cNTYX/3qV0fVs+7c2lo7ccwLzOSIvqrOrapLq+qmqnqgqlpVXbnCukcOy1d6bJpFTwBAsmFGr/OmJMcn2Zbk7iTH7kXNZ5Jcvcz8z86oJwBYeLMK+tdlEvBfSvKcJDfuRc2nW2sbZzQ+ALCMmQR9a+1fgr2qZvGSAMAMzOqIfhpPqqpXJ3lckm8luaW1dtsc+wGA7swz6M8ZHv+iqj6e5PzW2l178wLD2fXL2ZtzBACge/O4jv7BJG9PcmKSI4bHzs/1z0hyfVUdPIe+AKA7q35E31q7N8lvL5n9yap6bpJPJTklyauSXLIXr7XstYWuoweAiTVzZ7zW2vYk7x+ePnuevQBAL9ZM0A++OUy9dQ8AM7DWgv7UYfqVuXYBAJ1Y9aCvqlOq6geXmX9WJjfeSZJlb58LAOybmZyMV1UvSvKi4ekTh+kzq+ry4ectrbXXDz//fpLjhkvp7h7mPT3JWcPPb26t3TyLvgBg0c3qrPtnJDl/ybyjhkeSfDXJzqC/IsmLk/xUkhck+YEk30jyoSSXtdZumlFPALDwZnUL3I1JNu7luh9IMu77KQGAveL76IG5eOxjHzuq/uUvf/nUtX/wB38wauwx3+lx++23jxr7uOOOG1XPurM2vo8eAFibBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdMzX1AILZ/v27aPqH/Wo6Y+RduzYMWrsl770pVPX/vmf//mosZkLX1MLAKxM0ANAxwQ9AHRM0ANAxwQ9AHRM0ANAxwQ9AHRM0ANAxwQ9AHRM0ANAxwQ9AHRM0ANAxwQ9AHRM0ANAxwQ9AHRsw7wbANanU089dVT9K17xirmNP+b75Me65557RtVfffXVM+qEReGIHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGO+phbWseOPP35U/caNG6euPfvss0eNfcghh4yqn6cdO3ZMXbtly5a5jc1ickQPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB0T9ADQMUEPAB3zffQw0o/8yI+Mqv+1X/u1qWtf/epXjxr78MMPH1W/Xt11112j6jdu3Dh17eWXXz5qbNhXo4/oq+pxVfWqqvqLqvpSVX23qu6vqk9V1Suratkxquq0qrq2qrZW1YNVdVtVXVRVB4ztCQCYmMUR/XlJ3pvk60luTHJXkh9O8nNJ3p/kBVV1Xmut7Syoqp9N8uEkDyX5syRbk/xMkncmOX14TQBgpFkE/ReSvDDJR1prO3bOrKrfTPK3SV6SSeh/eJh/aJI/TvJIkjNaa38/zH9zkhuSnFtVL2utbZpBbwCw0Ea/dd9au6G1ds2uIT/MvyfJ+4anZ+yy6NwkP5Rk086QH9Z/KMmbhqe/OrYvAGD/n3X/vWG6fZd5Zw3Tjy6z/ieTPJjktKo6cH82BgCLYL+ddV9VG5K8fHi6a6gfM0y/sLSmtba9qu5IclySo5LcvocxNq+w6Nh96xYA+rQ/j+jfkeQnklzbWvvYLvMPG6b3r1C3c/5iXvcDADO0X47oq+rCJBcn+VySX9rX8mHadrtWktbaiSuMvznJCfs4LgB0Z+ZH9FV1QZJLkvxjkjNba1uXrLLziP2wLO/QJesBAFOaadBX1UVJLkvy2UxC/p5lVvv8MD16mfoNSZ6cycl7X5llbwCwiGYW9FX1xkxuePPpTEL+3hVWvWGYPn+ZZc9O8ugkN7fWHp5VbwCwqGYS9MPNbt6RZHOSs1trW3az+lVJtiR5WVWdtMtrHJTkd4an751FXwCw6EafjFdV5yd5WyZ3urspyYVVtXS1O1trlydJa+2BqvrlTAL/41W1KZNb4L4wk0vvrsrktrgAwEizOOv+ycP0gCQXrbDOJ5JcvvNJa+3qqnpOkt/K5Ba5ByX5UpJfT/LuXe+LDwBMr3rMVJfXLZ4nPelJo+pPO+20qWsvu+yyUWM/4QlPGFW/Xt1xxx2j6n/3d3936toPfvCDo8besWPHnleC2bh1pUvJ99b+vgUuADBHgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjgh4AOiboAaBjG+bdAP14/OMfP6r+mmuumbr26KOPHjX2EUccMap+vfryl788de3v/d7vjRp706ZNo+offPDBUfWwKBzRA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdEzQA0DHBD0AdMzX1HbmnHPOGVX/9re/ferapz71qaPGfsxjHjOqfr363ve+N3XtFVdcMWrsiy66aOrabdu2jRobWB2O6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY4IeADom6AGgY76PvjO/8Au/MKr+5JNPnlEnq+sb3/jGqPqPfvSjU9du37591NhvfOMbp67dunXrqLGB/jmiB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6JigB4COCXoA6Fi11ubdw8xV1eYkJ8y7DwAY6dbW2oljXmD0EX1VPa6qXlVVf1FVX6qq71bV/VX1qap6ZVU9asn6R1ZV281j09ieAICJDTN4jfOSvDfJ15PcmOSuJD+c5OeSvD/JC6rqvPZv3zr4TJKrl3m9z86gJwAgswn6LyR5YZKPtNZ27JxZVb+Z5G+TvCST0P/wkrpPt9Y2zmB8AGAFo9+6b63d0Fq7ZteQH+bfk+R9w9Mzxo4DAOy7WRzR7873hun2ZZY9qapeneRxSb6V5JbW2m37uR8AWCj7LeirakOSlw9PP7rMKucMj11rPp7k/NbaXfurLwBYJPvziP4dSX4iybWttY/tMv/BJG/P5ES8rwzznp5kY5Izk1xfVc9orX1nTwMMl9Et59hpmwaAnuyX6+ir6sIklyT5XJLTW2tb96JmQ5JPJTklyUWttUv2omZ3Qf/ove8YANak0dfRz/yIvqouyCTk/zHJ2XsT8knSWtteVe/PJOifPbzGnmqW/ce7YQ4ATMz0FrhVdVGSyzK5Fv7M4cz7ffHNYXrwLPsCgEU1s6CvqjcmeWeST2cS8vdO8TKnDtOv7HYtAGCvzCToq+rNmZx8tzmTt+u37GbdU6rqB5eZf1aS1w1Pr5xFXwCw6EZ/Rl9V5yd5W5JHktyU5MKqWrrana21y4effz/JccOldHcP856e5Kzh5ze31m4e2xcAMJuT8Z48TA9IctEK63wiyeXDz1ckeXGSn0rygiQ/kOQbST6U5LLW2k0z6AkAiK+pBYC1bP5fUwsArF2CHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA6JugBoGOCHgA61mvQHznvBgBgBo4c+wIbZtDEWvTAML1zheXHDtPP7f9WumGbTcd2m47ttu9ss+ms5e12ZL6fZ1Or1tr4VtaZqtqcJK21E+fdy3phm03HdpuO7bbvbLPpLMJ26/WtewAggh4AuiboAaBjgh4AOiboAaBjC3nWPQAsCkf0ANAxQQ8AHRP0ANAxQQ8AHRP0ANAxQQ8AHRP0ANCxhQr6qvrRqvqTqvqnqnq4qu6sqndV1RHz7m2tGrZRW+Fxz7z7m5eqOreqLq2qm6rqgWF7XLmHmtOq6tqq2lpVD1bVbVV1UVUdsFp9z9u+bLeqOnI3+16rqk2r3f88VNXjqupVVfUXVfWlqvpuVd1fVZ+qqldW1bK/xxd9f9vX7dbz/tbr99H/G1X1lCQ3J3lCkr/M5LuHT07y2iTPr6rTW2vfmmOLa9n9Sd61zPxtq93IGvKmJMdnsg3uzve/03pZVfWzST6c5KEkf5Zka5KfSfLOJKcnOW9/NruG7NN2G3wmydXLzP/sDPtay85L8t4kX09yY5K7kvxwkp9L8v4kL6iq89oudz+zvyWZYrsN+tvfWmsL8UjysSQtyX9ZMv+/D/PfN+8e1+IjyZ1J7px3H2vtkeTMJD+epJKcMexDV66w7qFJ7k3ycJKTdpl/UCZ/fLYkL5v3v2kNbrcjh+WXz7vvOW+zszIJ6Uctmf/ETMKrJXnJLvPtb9Ntt273t4V4676qjkry3ExC6z1LFr8lyXeS/FJVHbzKrbFOtdZubK19sQ2/Ifbg3CQ/lGRTa+3vd3mNhzI5wk2SX90Pba45+7jdSNJau6G1dk1rbceS+fcked/w9IxdFtnfMtV269aivHV/1jC9bpn/9G9X1V9n8ofAqUmuX+3m1oEDq+oXk/xYJn8U3Zbkk621R+bb1rqxc//76DLLPpnkwSSnVdWBrbWHV6+tdeNJVfXqJI9L8q0kt7TWbptzT2vF94bp9l3m2d/2bLnttlN3+9uiBP0xw/QLKyz/YiZBf3QE/XKemOSKJfPuqKpXtNY+MY+G1pkV97/W2vaquiPJcUmOSnL7aja2TpwzPP5FVX08yfmttbvm0tEaUFUbkrx8eLprqNvfdmM3222n7va3hXjrPslhw/T+FZbvnH/4KvSy3nwwydmZhP3BSX4yyR9m8nnWX1XV8fNrbd2w/03nwSRvT3JikiOGx3MyObHqjCTXL/jHbe9I8hNJrm2tfWyX+fa33Vtpu3W7vy1K0O9JDVOfGy7RWnvr8FnXN1prD7bWPtta+5VMTmL8d0k2zrfDLtj/ltFau7e19tuttVtba/cNj09m8u7b/0nyH5O8ar5dzkdVXZjk4kyuHvqlfS0fpgu3v+1uu/W8vy1K0O/8C/awFZYfumQ99mznySzPnmsX64P9b4Zaa9szuTwqWcD9r6ouSHJJkn9McmZrbeuSVexvy9iL7basHva3RQn6zw/To1dY/uPDdKXP8Pm37h2m6/KtrFW24v43fF745ExOCvrKaja1zn1zmC7U/ldVFyW5LJNrus8cziBfyv62xF5ut91Z1/vbogT9jcP0ucvcDekxmdxA4rtJ/ma1G1vHnjlMF+aXxQg3DNPnL7Ps2UkeneTmBT4DehqnDtOF2f+q6o2Z3PDm05mE1b0rrGp/28U+bLfdWdf720IEfWvty0muy+QEsguWLH5rJn+l/Wlr7Tur3NqaVlXHVdVjl5n/HzL56zhJdnvbV5IkVyXZkuRlVXXSzplVdVCS3xmevnceja1lVXVKVf3gMvPPSvK64elC7H9V9eZMTiLbnOTs1tqW3axufxvsy3breX+rRblvxTK3wL09ySmZ3KnrC0lOa26B+69U1cYkv5HJOyJ3JPl2kqck+elM7rJ1bZIXt9b+eV49zktVvSjJi4anT0zyvEz+2r9pmLeltfb6JetflcktSTdlckvSF2ZyKdRVSV66CDeR2ZftNlzSdFySj2dyu9wkeXq+f534m1trO4OrW1V1fpLLkzyS5NIs/9n6na21y3epWfj9bV+3W9f727xvzbeajyT/PpPLxb6e5J+TfDWTkzMeO+/e1uIjk0tL/mcmZ6jel8lNJr6Z5H9nch1qzbvHOW6bjZmctbzS485lak7P5I+j/5fJR0X/N5MjhQPm/e9Zi9stySuT/K9M7mi5LZNbut6Vyb3bnzXvf8sa2mYtycftb+O2W8/728Ic0QPAIlqIz+gBYFEJegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI4JegDomKAHgI79f5cA/kaMAf/SAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 253
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(train_points[0].reshape((28, 28)), cmap=cm.Greys_r)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exact same procedure can be applied to the validation and testing sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting our dataset into column vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting label digits to neuron representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen the provided labels in our dataset are digits from 0-9. Our goal is to convert these to adequate activations of the output layer. Since we have 10 different digits, we will use 10 neurons in the output layer, each of them representing the probability that of a certain digit. For example, a label 5 would mean that the sixth neuron (we are counting from 0) should output 1 and all the others output 0. A label 1 would mean that the second neuron should output 1 and all the others 0, etc. <br>\n",
    "This can be easily achieved. We just need to create a vector with 10 zeros, and fill the corresponding location with the value 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lab = np.zeros((10,1))\n",
    "lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to convert the label `5` to the neuron, we simply turn array's fifth element into one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lab[5]=1\n",
    "lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can turn this into a function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_label(digit):\n",
    "    neuron = np.zeros((10, 1))\n",
    "    neuron[digit] = 1.0\n",
    "    return neuron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [1.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.],\n",
       "       [0.]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "convert_label(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use this function to convert the labels of the entire training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_labels = np.array([convert_label(digit) for digit in train_labels])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting image data points to column vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To convert a single point, we simply need to reshape it into a column vector of the size `(784,1)`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.012],\n",
       "       [0.07 ],\n",
       "       [0.07 ],\n",
       "       [0.07 ],\n",
       "       [0.492],\n",
       "       [0.531],\n",
       "       [0.684],\n",
       "       [0.102],\n",
       "       [0.648],\n",
       "       [0.996],\n",
       "       [0.965],\n",
       "       [0.496],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.117],\n",
       "       [0.141],\n",
       "       [0.367],\n",
       "       [0.602],\n",
       "       [0.664],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.879],\n",
       "       [0.672],\n",
       "       [0.988],\n",
       "       [0.945],\n",
       "       [0.762],\n",
       "       [0.25 ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.191],\n",
       "       [0.93 ],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.98 ],\n",
       "       [0.363],\n",
       "       [0.32 ],\n",
       "       [0.32 ],\n",
       "       [0.219],\n",
       "       [0.152],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.07 ],\n",
       "       [0.855],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.773],\n",
       "       [0.711],\n",
       "       [0.965],\n",
       "       [0.941],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.312],\n",
       "       [0.609],\n",
       "       [0.418],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.801],\n",
       "       [0.043],\n",
       "       [0.   ],\n",
       "       [0.168],\n",
       "       [0.602],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.055],\n",
       "       [0.004],\n",
       "       [0.602],\n",
       "       [0.988],\n",
       "       [0.352],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.543],\n",
       "       [0.988],\n",
       "       [0.742],\n",
       "       [0.008],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.043],\n",
       "       [0.742],\n",
       "       [0.988],\n",
       "       [0.273],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.137],\n",
       "       [0.941],\n",
       "       [0.879],\n",
       "       [0.625],\n",
       "       [0.422],\n",
       "       [0.004],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.316],\n",
       "       [0.938],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.465],\n",
       "       [0.098],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.176],\n",
       "       [0.727],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.586],\n",
       "       [0.105],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.062],\n",
       "       [0.363],\n",
       "       [0.984],\n",
       "       [0.988],\n",
       "       [0.73 ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.973],\n",
       "       [0.988],\n",
       "       [0.973],\n",
       "       [0.25 ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.18 ],\n",
       "       [0.508],\n",
       "       [0.715],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.809],\n",
       "       [0.008],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.152],\n",
       "       [0.578],\n",
       "       [0.895],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.977],\n",
       "       [0.711],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.094],\n",
       "       [0.445],\n",
       "       [0.863],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.785],\n",
       "       [0.305],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.09 ],\n",
       "       [0.258],\n",
       "       [0.832],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.773],\n",
       "       [0.316],\n",
       "       [0.008],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.07 ],\n",
       "       [0.668],\n",
       "       [0.855],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.762],\n",
       "       [0.312],\n",
       "       [0.035],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.215],\n",
       "       [0.672],\n",
       "       [0.883],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.953],\n",
       "       [0.52 ],\n",
       "       [0.043],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.531],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.988],\n",
       "       [0.828],\n",
       "       [0.527],\n",
       "       [0.516],\n",
       "       [0.062],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ],\n",
       "       [0.   ]], dtype=float32)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_points[0].reshape(784,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can use this to convert the points of the entire training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_points = [point.reshape(784,1) for point in train_points]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Combining data points with their labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's introduce an example of combining two lists `L1` and `L2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "L1 = np.array([1,2,3,4,5,6,7,8,9])\n",
    "L2 = np.array(['A','B','C','D','E','F','G','A','H'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use `zip` and `list` functions to create a list of list containing an element from the first list with the element of the second list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(1, 'A'),\n",
       " (2, 'B'),\n",
       " (3, 'C'),\n",
       " (4, 'D'),\n",
       " (5, 'E'),\n",
       " (6, 'F'),\n",
       " (7, 'G'),\n",
       " (8, 'A'),\n",
       " (9, 'H')]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(zip(L1,L2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "by using `np.array()` we can turn this into a numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['1', 'A'],\n",
       "       ['2', 'B'],\n",
       "       ['3', 'C'],\n",
       "       ['4', 'D'],\n",
       "       ['5', 'E'],\n",
       "       ['6', 'F'],\n",
       "       ['7', 'G'],\n",
       "       ['8', 'A'],\n",
       "       ['9', 'H']], dtype='<U21')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L1zipL2 = np.array(list(zip(L1,L2)))\n",
    "L1zipL2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this in mind, we ca finally put together the training points and their labels as `train_set` by using the function `zip` and converting it into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set = np.array(list(zip(train_points, train_labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each element of this data set now has data points associated with the index 0 and the corresponding label associated with the index 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting the validation and testing data sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the same procedure to convert the validation and testing sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_points = [point.reshape(784,1) for point in valid_set[0]]\n",
    "valid_labels = np.array([convert_label(digit) for digit in valid_set[1]])\n",
    "valid_set = np.array(list(zip(valid_points, valid_labels)))\n",
    "\n",
    "test_points = [point.reshape(784,1) for point in test_set[0]]\n",
    "test_labels = np.array([convert_label(digit) for digit in test_set[1]])\n",
    "test_set = np.array(list(zip(test_points, test_labels)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating mini-batches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The idea of mini-batches is to take the data set containing for example 50'000 points, and split it into smaller, managable chunks of data that can be trained more efficiently using our algorithm. Let's see how this can be accomplished on our toy example with `L1zipL2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['1', 'A'],\n",
       "       ['2', 'B'],\n",
       "       ['3', 'C'],\n",
       "       ['4', 'D'],\n",
       "       ['5', 'E'],\n",
       "       ['6', 'F'],\n",
       "       ['7', 'G'],\n",
       "       ['8', 'A'],\n",
       "       ['9', 'H']], dtype='<U21')"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L1zipL2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define the size of chunks that we would like to subdivide our data into with `mb_size`. Then we compute the length of our data with `data_len`: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chunk size: 3, number of data points 9\n"
     ]
    }
   ],
   "source": [
    "mb_size = 3\n",
    "data_len = len(L1zipL2)\n",
    "print (f'chunk size: {mb_size}, number of data points {data_len}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main idea is that in every epoch, we shuffle our data set by using `np.random.shuffle`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['6', 'F'],\n",
       "       ['5', 'E'],\n",
       "       ['9', 'H'],\n",
       "       ['1', 'A'],\n",
       "       ['3', 'C'],\n",
       "       ['7', 'G'],\n",
       "       ['4', 'D'],\n",
       "       ['8', 'A'],\n",
       "       ['2', 'B']], dtype='<U21')"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.shuffle(L1zipL2)\n",
    "L1zipL2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use the function range to define the indexes of cuts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 3, 6]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(range(0, data_len, mb_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can take parts of our data and extract them according to the indexes until we reach the end of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch 1:\n",
      "[['6' 'F']\n",
      " ['5' 'E']\n",
      " ['9' 'H']]\n",
      "\n",
      "batch 2:\n",
      "[['1' 'A']\n",
      " ['3' 'C']\n",
      " ['7' 'G']]\n",
      "\n",
      "batch 3:\n",
      "[['4' 'D']\n",
      " ['8' 'A']\n",
      " ['2' 'B']]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "mini_batches = [L1zipL2[a:a+mb_size] for a in range(0, data_len, mb_size)]\n",
    "\n",
    "for i, batch in enumerate(mini_batches):\n",
    "    print (f'batch {i+1}:\\n{batch}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The flexible neural network algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "code_folding": [
     0,
     3,
     36
    ]
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1.0/(1.0+np.exp(-z))\n",
    "    \n",
    "def backprop(x, y, weights, biases, num_layers):\n",
    "    # storing derivatives for the current data point\n",
    "    d_biases =  [np.zeros(b.shape) for b in biases]\n",
    "    d_weights = [np.zeros(w.shape) for w in weights]\n",
    "    \n",
    "    a = x # current activation\n",
    "    activations = [x] # here we store all the activations, layer by layer\n",
    "    weighted_sums = [] # here we store all the weighted sums, layer by layer\n",
    "    \n",
    "    # calculate all the weighted sums and activations\n",
    "    for b, w in zip(biases, weights):\n",
    "        w_sum = np.dot(w, a)+b #current weighted sum\n",
    "        weighted_sums.append(w_sum) \n",
    "        a = sigmoid(w_sum) # current activation is updated and then the loop goes to the next layer\n",
    "        activations.append(a)\n",
    "    \n",
    "    # computing the cost for a single example\n",
    "    cost = 0.5*sum((y - a)**2)[0]\n",
    "        \n",
    "    #backward pass\n",
    "    # output layer\n",
    "    d_w_sum = (activations[-1] - y) * (activations[-1] * (1 - activations[-1]))\n",
    "    d_biases[-1] = d_w_sum\n",
    "    d_weights[-1] = np.dot(d_w_sum, activations[-2].T)\n",
    "        \n",
    "    # other layers\n",
    "    for i in range(2, num_layers):\n",
    "        act = activations[-i]\n",
    "        d_w_sum = np.dot(weights[-i+1].T, d_w_sum) * (act * (1 - act))\n",
    "        d_biases[-i] = d_w_sum\n",
    "        d_weights[-i] = np.dot(d_w_sum, activations[-i-1].T)\n",
    "    return (d_biases, d_weights, cost)\n",
    "\n",
    "def prediction(a_set, weights, biases):\n",
    "    data_len = len(a_set)\n",
    "    number_correct = 0\n",
    "    for point, label in a_set:\n",
    "        a = point\n",
    "        for b, w in zip(biases, weights):\n",
    "            w_sum = np.dot(w, a)+b #current weighted sum\n",
    "            a = sigmoid(w_sum) # current activation is updated and then the loop goes to the next layer\n",
    "            a = np.round(a)\n",
    "            if np.array_equal(a,label):\n",
    "                number_correct += 1\n",
    "    #print (f'correct: {number_correct} of {data_len}')\n",
    "    return number_correct/data_len\n",
    "    \n",
    "def train_neural_network(sizes, train_set, step_size, mini_batch_length, no_epochs, iter_info):\n",
    "    \n",
    "    num_layers = len(sizes) # number of layers\n",
    "    data_len = len(train_set) # number of data points\n",
    "    \n",
    "    # initialising random weights and biases \n",
    "    biases  = [np.random.randn(y, 1) for y in sizes[1:]]\n",
    "    weights = [np.random.randn(x, y) for x, y in zip(sizes[1:], sizes[:-1])]\n",
    "    \n",
    "    for epoch in range(no_epochs):\n",
    "        total_cost = 0 # total cost for all training examples\n",
    "        np.random.shuffle(train_set) # shuffle \n",
    "        mini_batches = [train_set[a:a+mini_batch_length] for a in range(0, data_len, mini_batch_length)]\n",
    "        \n",
    "        for mini_batch in mini_batches:\n",
    "            # in avg_d_biases/weights we store the averaged partial derivatives\n",
    "            avg_d_biases =  [np.zeros(b.shape) for b in biases]\n",
    "            avg_d_weights = [np.zeros(w.shape) for w in weights]\n",
    "            \n",
    "            for x, y in mini_batch:\n",
    "                d_biases, d_weights, cost = backprop(x, y, weights, biases, num_layers)\n",
    "                # averaging the derivatives of weights and biases\n",
    "                avg_d_biases  = [nb+dnb for nb, dnb in zip(avg_d_biases, d_biases)]\n",
    "                avg_d_weights = [nw+dnw for nw, dnw in zip(avg_d_weights, d_weights)]\n",
    "                total_cost += cost\n",
    "\n",
    "            # updating the weights and biases\n",
    "            weights = [w - (step_size/len(mini_batch) * dw) \n",
    "                   for w, dw in zip(weights, avg_d_weights)]\n",
    "            biases =  [b - (step_size/len(mini_batch) * db)\n",
    "                  for b, db in zip(biases, avg_d_biases)]\n",
    "        \n",
    "        if (epoch%iter_info==0):\n",
    "            print (f'epoch: {epoch+1} | total cost: {round(total_cost,2)}')\n",
    "            train_prediction = round(100*prediction(train_set, weights, biases),4)\n",
    "            valid_prediction = round(100*prediction(valid_set, weights, biases),4)\n",
    "            print (f\"Training set prediction:   {train_prediction}%\\nValidation set prediction: {valid_prediction}%\\n\")\n",
    "            if train_prediction-valid_prediction > 3.0:\n",
    "                break\n",
    "    return (weights, biases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 | total cost: 9543.52\n",
      "Training set prediction:   77.23%\n",
      "Validation set prediction: 77.86%\n",
      "\n",
      "epoch: 6 | total cost: 1799.96\n",
      "Training set prediction:   93.158%\n",
      "Validation set prediction: 91.63%\n",
      "\n",
      "epoch: 11 | total cost: 1190.02\n",
      "Training set prediction:   95.014%\n",
      "Validation set prediction: 92.46%\n",
      "\n",
      "epoch: 16 | total cost: 846.46\n",
      "Training set prediction:   95.804%\n",
      "Validation set prediction: 92.53%\n",
      "\n",
      "epoch: 21 | total cost: 642.72\n",
      "Training set prediction:   96.712%\n",
      "Validation set prediction: 92.77%\n",
      "\n",
      "epoch: 26 | total cost: 508.69\n",
      "Training set prediction:   97.242%\n",
      "Validation set prediction: 92.65%\n",
      "\n",
      "epoch: 31 | total cost: 424.06\n",
      "Training set prediction:   97.556%\n",
      "Validation set prediction: 92.8%\n",
      "\n",
      "epoch: 36 | total cost: 388.5\n",
      "Training set prediction:   97.756%\n",
      "Validation set prediction: 92.54%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "weights, biases = train_neural_network([784, 100, 10], \n",
    "                                       train_set, \n",
    "                                       step_size = 3.0, \n",
    "                                       mini_batch_length = 10, \n",
    "                                       no_epochs = 50, \n",
    "                                       iter_info = 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction quality on the testing set: 92.24\n"
     ]
    }
   ],
   "source": [
    "print(f'Prediction quality on the testing set: {round(100*prediction(test_set, weights, biases),4)}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}