Handwritten_digits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ứng dụng Nerual Network trong nhận dạng chữ số"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Module này mình hoạ feedforward neural network với sô layer bất kỳ. Những khái niệm về linear projection sẽ được đưa ra thông qua matrix multiplication, non-linear transformaiton."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import những thư viện cần thiết"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python imports\n",
    "import numpy as np # Matrix and vector computation package\n",
    "import matplotlib.pyplot as plt  # Plotting library\n",
    "# Allow matplotlib to plot inside this notebook\n",
    "%matplotlib inline\n",
    "# Set the seed of the numpy random number generator so that the tutorial is reproducable\n",
    "np.random.seed(seed=1)\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import datasets, metrics, cross_validation # data and evaluation utils, \n",
    "from matplotlib.colors import colorConverter, ListedColormap # some plotting functions\n",
    "import itertools\n",
    "import collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handwritten digits dataset\n",
    "\n",
    "Tập dataset sử dụng được cung cấp bởi scikit-learn. Tập này bao gồm 1797 8x8 images số được viết bằng tay từ 0 - 9. Tập dataset này nhỏ hơn rất nhiều so với tập MNIST."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load the data from scikit-learn\n",
    "digits = datasets.load_digits()\n",
    "\n",
    "# Load the targets\n",
    "# Note that the targets are stored as digits, these need to be\n",
    "# converted to one-hot-encoding for the output sofmax layer.\n",
    "T = np.zeros((digits.target.shape[0], 10))\n",
    "T[np.arange(len(T)), digits.target] += 1\n",
    "\n",
    "# Devide the data into a train and test set.\n",
    "X_train, X_test, T_train, T_test = cross_validation.train_test_split(digits.data, T, test_size=0.4)\n",
    "\n",
    "# Devide the test set into a validation set and final test set.\n",
    "X_validation, X_test, T_validation, T_test = cross_validation.train_test_split(X_test, T_test, test_size=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABl4AAADjCAYAAADkM2mGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAewgAAHsIBbtB1PgAADuhJREFUeJzt27FSm9faBeCPM/QSuYHI3ABynD7yDNQohWmhssuICjor\nnemgdBW5jYvIdVyI3kzEBSQoNxDJV8C5gv8ffNYbJFnP07/by2Lr21tao637+/sGAAAAAACA3H+W\nHQAAAAAAAOBroXgBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongB\nAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAA\nAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAo\nongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongB\nAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAA\nAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAo\nongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongB\nAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAA\nAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAo\nongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongBAAAAAAAoongB\nAAAAAAAoongBAAAAAAAoongBAAAAAAAosr3sAI/kftkB3r9/H82fnZ3FGQ4ODqL5N2/exBl2dnbi\nNQpsPdK/s/R9l+r1evEai8Uimh8Oh3GGfr8fr1HgMfbd2u+5yWQSr5H+vbvdbpyh4v9RYGOedRcX\nF9H8+fl5nOHJkyfR/M3NTZzBGbte0vOxaZrm5OQkmh+Px3GGFbERZ2x6L+t0OnGG0WgUr/GV8Kx7\noFX4PDGdTuMMK2IjnnWXl5fRfMX5mp6Pt7e3cYZWqxXNz2azOEO73d6YZ91gMIjmK+5U6b0u/T80\nTdO02+14jQIbs+/S7y4qnncr8t3FKvhX9p1fvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAA\nAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAA\nABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABRRvAAAAAAAABTZXnaATXF2dhbN393dxRnm83k0\n/80338QZfv3112j+xYsXcQYert1ux2tcX19H85PJJM7Q7/fjNXiY6XQazT9//jzO0Gq1ovnZbBZn\n4OHOz8/jNdKz5e3bt3GGV69eRfM3Nzdxhv39/XgNHs9oNIrX6Ha7eRDWRno+pXeypmmad+/eRfPf\nfvttnME5/bjG43E0X7HvXr9+Ha/B5qj4DHt5ebnU+aZpmsViEc1XvA6bJP0cWyG9G1Z8d1Kxxqao\nuI98+PAhDxLa2tqK5vf29uIMq/D++7f4xQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAA\nAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEAR\nxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAECR7WUH\nWAc3NzfxGnd3d9H8n3/+GWfY3d2N5g8ODuIM6Wv54sWLOMMmmU6n0fxkMqkJEuh2u8uOwBcYj8fR\n/N7eXpyh3+9H8z///HOcgYd7+fJlvMbZ2Vk0/+zZszjDkydPovn9/f04A49rsVhE86PRKM4wGAyi\n+dlsFmdIdTqdZUdYG+12O5r/+++/4wytViua7/V6cYb0vZe+jptmOBwuO0J8t2O9pGdbhXTfV5yv\nq/BZfJOk3ztU3GfSu2HF+Zbuu4pzfl2k95EKP/zwQ7xGunc9q/5/fvECAAAAAABQRPECAAAAAABQ\nRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPEC\nAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAAAABQRPECAAAA\nAABQZHvZAdbBfD6P1/juu++i+d3d3ThD6tmzZ8uOsFEuLy/jNYbDYTT/+fPnOEOq1+stOwJfYDAY\nRPOdTmfpGQ4PD+MMPFzF+fbXX39F83d3d3GG/f39aL7irrGzsxOvwcONRqNofjabxRlOTk6i+fR5\n2TRN0263o/n0rrJJ0jPy9vY2zpDeDbvdbpwh3XN8mcViEc3v7e3FGSr2DY9jMpmsxBqpis/iqfF4\nHM2nd4RNk75eT58+jTOkd8OK87Hi8/imWIXXKn1ONE3T9Pv9aD69J3zt/OIFAAAAAACgiOIFAAAA\nAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACg\niOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIFAAAAAACgiOIF\nAAAAAACgiOIFAAAAAACgyPayA6yD+Xwer3FwcFCQZLkqXoednZ2CJJthMBjEa5ycnETzq/D3WiwW\ny46wMSpe68vLy2h+PB7HGVKj0WjZEfhCu7u70fw///wTZ9jf31/qfNM0zcePH6P5VXjmP5aKZ83p\n6Wk0f3x8HGdIXV1dxWv88ssvBUl4iHTfTiaTOMN0Oo3m0/dNhYo79iZJ74edTifOkN4v+/1+nKHi\n/7EJKl6n9DlT8axLVdwzer1eHoQHW4XvHa6vr6P5u7u7OINn3cO12+14jb29vWi+4vPbTz/9FM2n\nz+ymaZrZbBbNr/K+9YsXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACA\nIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAIooX\nAAAAAACAIooXAAAAAACAIooXAAAAAACAIooXAAAAAACAItvLDrAOdnZ24jVubm4KkmTm83k0/+nT\npzjD0dFRvAabZTqdxmt0u92CJF+/4XAYr3F1dZUHCf3222/RfLvdLkrCuqg45z9+/BjNv3r1Ks5w\ncXERzb958ybOsC4q3uetViuaf/fuXZyh4oxM9fv9ZUfggXq93rIjlJjNZsuOsFE6nU40f319HWdY\nLBbR/OnpaZzhjz/+iOY35fNIul+apmnG43E0v7W1FWdIP098Lc/bdVFxH3r+/Hk0//r16zhDer5V\n3MnS91/FM2CTpHv3a/m+bDAYRPPpvv03+cULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAA\nAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABA\nEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAEcULAAAAAABAke1l\nB1gHu7u78RqfPn2K5t+/fx9nqFgjdXZ2tuwIwP/h5OQkXmMymUTzt7e3cYYff/wxmj88PIwzpK9l\nv9+PM2yS8/PzaH5/fz/OMJ/Po/nff/89znB0dBSvsSl6vV68xmKxiOan02mcIf1/HB8fxxna7Xa8\nBg8zHo+j+Yq/1XA4jNdIOSMfV3qnOT09jTN0Op1ofjabxRnS91+3240zbIrBYBDNt1qtOEPFPYHH\nkz4jmibfN+m+bZr8WfX06dM4w2g0iuZX4Z6wSSrOlnTvpnumafIzdpX5xQsAAAAAAEARxQsAAAAA\nAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEAR\nxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsA\nAAAAAECR7WUHWAe7u7vxGhcXF9H82dlZnOH777+P5m9ubuIMPK52ux3NHx4exhk+fPgQzU8mkzjD\nyclJvMYm6Ha78RrT6XSp803TNMPhMJpP92zTNE2n04nm+/1+nGGT7OzsRPMvX74sSvK/Ozo6itd4\n+/ZtQRIeS3pGN03TfP78OZp3Pq6X9E50dXVVEyRwfHwcr9Hr9fIgPFj6nJjNZnGG0WgUzVfsGXez\nx5M+69L90jQ1ZzSPp+LvlT4n0s8jTdM0rVYrmq/4/mYwGMRr8HDp613x/clisYjmK76zq/gualX5\nxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsA\nAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAA\nAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAECRrfv7+2VnAAAAAAAA+Cr4xQsAAAAAAEARxQsA\nAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAA\nAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEAR\nxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsA\nAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAAAEARxQsAAAAA\nAEARxQsAAAAAAECR/wLZLyYExbz0CgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11689a610>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 113,
       "width": 815
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot an example of each image.\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "fig = plt.figure(figsize=(10, 1), dpi=100)\n",
    "for i in range(10):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    ax.matshow(digits.images[i], cmap='binary') \n",
    "    ax.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generalization of the layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the non-linear functions used\n",
    "def logistic(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "def logistic_deriv(y):\n",
    "    return np.multiply(y, (1 - y))\n",
    "\n",
    "def softmax(z):\n",
    "    return np.exp(z) / np.sum(np.exp(z), axis=1, keepdims=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define layers used in this model\n",
    "class Layer(object):\n",
    "    \"\"\"Base class for the different layers.\n",
    "    Defines base methods and documentation of methods.\"\"\"\n",
    "    \n",
    "    def get_params_iter(self):\n",
    "        return []\n",
    "    \n",
    "    def get_params_grad(self, X, output_grad):\n",
    "        return []\n",
    "    \n",
    "    def get_output(self, X):\n",
    "        pass\n",
    "    \n",
    "    def get_input_grad(self, Y, output_grad=None, T=None):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LinearLayer(Layer):\n",
    "    \"\"\"The linear layer performs a linear transformation to its input.\"\"\"\n",
    "    \n",
    "    def __init__(self, n_in, n_out):\n",
    "        \"\"\"Initialize hidden layer parameters.\n",
    "        n_in is the number of input variables.\n",
    "        n_out is the number of output variables.\"\"\"\n",
    "        self.W = np.random.randn(n_in, n_out) * 0.1\n",
    "        self.b = np.zeros(n_out)\n",
    "        \n",
    "    def get_params_iter(self):\n",
    "        \"\"\"Return an iterator over the parameters.\"\"\"\n",
    "        return itertools.chain(np.nditer(self.W, op_flags=['readwrite']),\n",
    "                                np.nditer(self.b, op_flags=['readwrite']))\n",
    "    \n",
    "    def get_output(self, X):\n",
    "        \"\"\"Perform the forward step linear transformation.\"\"\"\n",
    "        return X.dot(self.W) + self.b\n",
    "    \n",
    "    def get_params_grad(self, X, output_grad):\n",
    "        \"\"\"Return a list of gradients over the parameters.\"\"\"\n",
    "        JW = X.T.dot(output_grad)\n",
    "        Jb = np.sum(output_grad, axis=0)\n",
    "        return [g for g in itertools.chain(np.nditer(JW), np.nditer(Jb))]\n",
    "    \n",
    "    def get_input_grad(self, Y, output_grad):\n",
    "        \"\"\"Return the gradient at the inputs of this layer.\"\"\"\n",
    "        return output_grad.dot(self.W.T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LogisticLayer(Layer):\n",
    "    \"\"\"The logistic layer applies the logistic function to its inputs.\"\"\"\n",
    "    \n",
    "    def get_output(self, X):\n",
    "        \"\"\"Perform the forward step transformation.\"\"\"\n",
    "        return logistic(X)\n",
    "    \n",
    "    def get_input_grad(self, Y, output_grad):\n",
    "        \"\"\"Return the gradient at the inputs of this layer.\"\"\"\n",
    "        return np.multiply(logistic_deriv(Y), output_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class SoftmaxOutputLayer(Layer):\n",
    "    \"\"\"The softmax output layer computes the classification propabilities at the output.\"\"\"\n",
    "    \n",
    "    def get_output(self, X):\n",
    "        \"\"\"Perform the forward step transformation.\"\"\"\n",
    "        return softmax(X)\n",
    "    \n",
    "    def get_input_grad(self, Y, T):\n",
    "        \"\"\"Return the gradient at the inputs of this layer.\"\"\"\n",
    "        return (Y - T) / Y.shape[0]\n",
    "    \n",
    "    def get_cost(self, Y, T):\n",
    "        \"\"\"Return the cost at the output of this output layer.\"\"\"\n",
    "        return - np.multiply(T, np.log(Y)).sum() / Y.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Mô hình mạng"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define a sample model to be trained on the data\n",
    "hidden_neurons_1 = 20 # Number of neurons in the first hidden-layer\n",
    "hidden_neurons_2 = 20 # Number of neurons in the second hidden-layer\n",
    "\n",
    "# Create the model\n",
    "layers = [] # Define a list of layers\n",
    "# Add first hidden layer\n",
    "layers.append(LinearLayer(X_train.shape[1], hidden_neurons_1))\n",
    "layers.append(LogisticLayer())\n",
    "# Add second hidden layer\n",
    "layers.append(LinearLayer(hidden_neurons_1, hidden_neurons_2))\n",
    "layers.append(LogisticLayer())\n",
    "# Add output layer\n",
    "layers.append(LinearLayer(hidden_neurons_2, T_train.shape[1]))\n",
    "layers.append(SoftmaxOutputLayer())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the forward propagation step as a method\n",
    "def forward_step(input_samples, layers):\n",
    "    activations = [input_samples] # List of layer acctivations\n",
    "    # Compute the forward activations for each layer starting from the first\n",
    "    X = input_samples\n",
    "    for layer in layers:\n",
    "        Y = layer.get_output(X) # Get the output of the current layer\n",
    "        activations.append(Y)\n",
    "        X = activations[-1]\n",
    "    return activations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define the backward propagation step as a method\n",
    "def backward_step(activations, targets, layers):\n",
    "    param_grads = collections.deque() # List of parameter gradients for each layer\n",
    "    output_grad = None # The error gradient at the output of the current layer\n",
    "    # Propagate the error backwards through all the layers.\n",
    "    # Use reversed to iterate backwards over the list of layers.\n",
    "    for layer in reversed(layers):\n",
    "        Y = activations.pop() # Get the activations of the last layer on the stack\n",
    "        # Compute the error at the output layer\n",
    "        # The output layer error is calculated different then hidden layer error\n",
    "        if output_grad is None:\n",
    "            input_grad = layer.get_input_grad(Y, targets)\n",
    "        else:\n",
    "            input_grad = layer.get_input_grad(Y, output_grad)\n",
    "            \n",
    "        # Get the input of this layer (activations of the previous layer)\n",
    "        X = activations[-1]\n",
    "        # Compute the layer parameter gradients used to update the parameters\n",
    "        grads = layer.get_params_grad(X, output_grad)\n",
    "        \n",
    "        param_grads.appendleft(grads)\n",
    "        # Compute gradient at output of previous layer (input of current layer)\n",
    "        output_grad = input_grad\n",
    "    return list(param_grads)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tính sai số cho việc đạo hàm - Gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No gradient errors found\n"
     ]
    }
   ],
   "source": [
    "# Perform gradient checking\n",
    "nb_samples_gradientcheck = 10 # Test the gradients on a subset of the data\n",
    "X_temp = X_train[0:nb_samples_gradientcheck,:]\n",
    "T_temp = T_train[0:nb_samples_gradientcheck,:]\n",
    "# Get the parameter gradients with backpropagation\n",
    "activations = forward_step(X_temp, layers)\n",
    "param_grads = backward_step(activations, T_temp, layers)\n",
    "\n",
    "# Set the small change to compute the numerical gradient\n",
    "eps = 0.0001\n",
    "# Compute the numerical gradients of the parameters in all layers.\n",
    "for idx in range(len(layers)):\n",
    "    layer = layers[idx]\n",
    "    layer_backprop_grads = param_grads[idx]\n",
    "    # Compute the numerical gradient for each parameter in the layer\n",
    "    for p_idx, param in enumerate(layer.get_params_iter()):\n",
    "        grad_backprop = layer_backprop_grads[p_idx]\n",
    "        # + eps\n",
    "        param += eps\n",
    "        plus_cost = layers[-1].get_cost(forward_step(X_temp, layers)[-1], T_temp)\n",
    "        # - eps\n",
    "        param -= 2 * eps\n",
    "        min_cost = layers[-1].get_cost(forward_step(X_temp, layers)[-1], T_temp)\n",
    "        # reset param value\n",
    "        param += eps\n",
    "        # calculate numerical gradient\n",
    "        grad_num = (plus_cost - min_cost)/(2*eps)\n",
    "        # Raise error if the numerical grade is not close to the backprop gradient\n",
    "        if not np.isclose(grad_num, grad_backprop):\n",
    "            raise ValueError('Numerical gradient of {:.6f} is not close to the backpropagation gradient of {:.6f}!'.format(float(grad_num), float(grad_backprop)))\n",
    "print('No gradient errors found')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lan truyền ngược - Backpropagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create the minibatches\n",
    "batch_size = 25  # Approximately 25 samples per batch\n",
    "nb_of_batches = X_train.shape[0] / batch_size  # Number of batches\n",
    "\n",
    "# Create batches (X,Y) from the training set\n",
    "XT_batches = zip(\n",
    "    np.array_split(X_train, nb_of_batches, axis=0),  # X samples\n",
    "    np.array_split(T_train, nb_of_batches, axis=0))  # Y targets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Define a method to update the parameters\n",
    "def update_params(layers, param_grads, learning_rate):\n",
    "    for layer, layer_backprop_grads in zip(layers, param_grads):\n",
    "        for param, grad in itertools.izip(layer.get_params_iter(), layer_backprop_grads):\n",
    "            # The parameter returned by the iterator point to the memory space of\n",
    "            #  the original layer and can thus be modified inplace.\n",
    "            param -= learning_rate * grad  # Update each parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Perform backpropagation\n",
    "# initalize some lists to store the cost for future analysis        \n",
    "minibatch_costs = []\n",
    "training_costs = []\n",
    "validation_costs = []\n",
    "\n",
    "max_nb_of_iterations = 300  # Train for a maximum of 300 iterations\n",
    "learning_rate = 0.1  # Gradient descent learning rate\n",
    "\n",
    "# Train for the maximum number of iterations\n",
    "for iteration in range(max_nb_of_iterations):\n",
    "    for X, T in XT_batches:  # For each minibatch sub-iteration\n",
    "        activations = forward_step(X, layers)  # Get the activations\n",
    "#         print \"activations = \", len(activations)\n",
    "        minibatch_cost = layers[-1].get_cost(activations[-1], T)  # Get cost\n",
    "        minibatch_costs.append(minibatch_cost)\n",
    "        param_grads = backward_step(activations, T, layers)  # Get the gradients\n",
    "        update_params(layers, param_grads, learning_rate)  # Update the parameters\n",
    "    # Get full training cost for future analysis (plots)\n",
    "    activations = forward_step(X_train, layers)\n",
    "    train_cost = layers[-1].get_cost(activations[-1], T_train)\n",
    "    training_costs.append(train_cost)\n",
    "    # Get full validation cost\n",
    "    activations = forward_step(X_validation, layers)\n",
    "    validation_cost = layers[-1].get_cost(activations[-1], T_validation)\n",
    "    validation_costs.append(validation_cost)\n",
    "#     if len(validation_costs) > 3:\n",
    "#         # Stop training if the cost on the validation set doesn't decrease\n",
    "#         #  for 3 iterations\n",
    "#         if validation_costs[-1] >= validation_costs[-2] >= validation_costs[-3]:\n",
    "#             break\n",
    "    \n",
    "nb_of_iterations = iteration + 1  # The number of iterations that have been executed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyQAAAIqCAYAAADVd47AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAWJQAAFiUBSVIk8AAAIABJREFUeJzs3Xl8FdX9//HXJySEPQTCvgUxbK7gDsoiuG9YqbhUwaVW\n26pU1H61UKLiT6u2RbSo1QrWupRaRbEi2gq44AYKCLihrAqCxrBvIef3x8xNbm7mhpuF3CXv5+Nx\nHzd35szMZ5Y7mc+dc+aYcw4REREREZF4SIt3ACIiIiIiUncpIRERERERkbhRQiIiIiIiInGjhERE\nREREROJGCYmIiIiIiMSNEhIREREREYkbJSQiIiIiIhI3SkhERERERCRulJCIiIiIiEjcKCERERER\nEZG4UUIiIiIiIiJxo4RERERERETiRgmJiIiIiIjEjRISEZEKmNnJZvY/Mys0s2Izc2Y2Kt5xSfyY\nWb5/HEyNdyz7YmaD/FhXxjuWZGFmU/1tlh/vWOJF20BqW3q8AxCJJ/+CYmTE4CJgM/AjsAT4AHjG\nObeidqOTeDOzE4CZeD/e7AU2Ag7YEc+4aoqfWOUC051zC+MbjUjiC7tAn+icK4xnLFVhZs2B0QDO\nufz4RiNSSgmJiGcPUOD/bUAzoAXQDTgHmGBmzwG/dM59H58QJQ6uw0tGpgGjnHMpkYiEGQUMBFYC\nSkhEPOuAz4Ggc/14/30qkHQJCdCc0nXIr6BcRdtApMapypaIZ55zrq3/auOcawhkA6cB/8T7Vfyn\nwEIz6xjPQKVWHeS/P5mCyYiIBHDO3eKc6+mcezDescSLtoHUNiUkIlE45wqdc6865y4AzgB2Ah2A\n5+IbmdSihv771rhGISIiksKUkIjEwDn3KnCj//EYMzsrqJyZ1TezX5vZW2ZWYGa7zGyVmT1uZr0q\nWoaZtTSz28xsgd+AeruZfWFmz5rZsIiyo/wGh3P8zxeb2Vwz+8EfHlm+iZndamYfmtkmM9tpZl+a\n2SQz6xQlnqb+cqaZ2RI/ph1mttzM/mpmeRWsS1MzG+evyxYz221m35rZfDO718wOjjJdpeOMhZll\nmtkNZva+P98dZva5mf3JzNoGlHdm5vDaVwDMDg0LbfNKLDvDzK7yG8ZvDDsmXvOHN65uvGHTHWZm\nfzezlf5ytpjZ12b2qpmNNrNGfrlR/voN9CedErZ+VWoAbWaDzex5M1vv7+/1ZvaCmZ0YULa/v5zd\nZtaignl2MLO9ftnDAsbnmtkD/rbZ7q/vAjP7bdB29acJrWOumfUysyfMbI2Z7TGz6VVY7zQz+42Z\nLTKzbf538CUzO7qCafqa2d1m9raZrfb31Q9mNsfMrjSzevtYZmMzu9HM5pl3ntnp7+eXzDsXZFQi\n/qFmttXfJneHDS9puF/ZdbSIhvRmdpqZzTSzDeY9GGJ0RPk2ZvZHM/vM34+bzOwDMxtjZplRllHS\n6NrMGph37vzM/65sMLNnzKx7rNsh2rwjh4UVWxHxnZkaMJ9WZnaXmX3ib+Nt5p1L74x23Jv33XX+\nNuxgZpP9fbvLzBaGlevoHwOvmneO3G5mm83sY39bNA+Y9xxgRdhnF/Eqt74WpVG7VeEcZeX/b51l\nZrPN+9+y1czeM7MLg6aVOsA5p5dedfaFVw/YAXNiKFsfWO+XfyZgfDu8evjOf+3Faxwf+rwD+EmU\neZ+AV1c3VHYX8ANeA3vnfVXLlB8VihuYFLa8Av99WFjZXnhtBELz3oP3i3/ocwHQPyCmX4eVKfLj\n2RU2bCswNGC6LGBpxHYIxRUadnfAdFWKM4b91gr4KGw+OyP2SwFwbMQ06/3X3rAyoWHPV2LZHYCP\nI7ZF5HYcVN14/elOB3ZHTLcp7LMDevplR/jrEiq/KWz91gMfVnIbTwhbRjHeAyGKw4bdFVHe8C6M\nHHBVBfO9wS+zNGDcT/C+U6FlbItY/8VAm4DpQuMv8adx/vbdgde4P5b1zfenewJ4Pux4LQybfxEw\nIsr030fE/WPEfvoPkB5l2t5h2y603B/899Cw3LDyg/xhKwPmda5/nDjg/2pqHcOXCYyJOC6KgNFh\nZY/24w/NM7QvQp8XAq0DljE1dGwB71J63gw/5rcBA6pwzgjNOz9s2P2Unv8d3gMuwr8z90fM4/iI\n9doVsV6rgR4By17pj7+K0odobMM7Fy4MK/dcxLx/oOw5djnQMWLez4fN00XEvx64saJtUAPnqFGU\n/t8aR+k5MfyYcuHHh1515xX3APTSK54vKpGQ+OWf9suvjRiegfc0Lgf8FzgOyPDHtQP+HPaPpVvE\ntN3C/ol+DAwG6vnjGgInAf+OmCZ0Yt+C94/+90Bzf1wz/H/geMlB6OJlGnBo2LwPAJ4K+8fUPGIZ\nF+BdaB4F1PeHGdAT+Ic/3QagccR0vw8bdwb+hZW/jfKA3wI/j5imynHGsM9mhv2T/GnYfI/Eu2gN\nzTcnYNqVBCQNMS43k9J/2huBS0PbCqgH9PWPi2NqIl7ga3/cDKB72PBmeAnvXwm7UPXHzfGnGVWN\n79AFlF5IPBCKC2hJabLsgJ9FTHeXP3x2BfP+0C/zu4jhR+ElH3v8Y7RD2HY9Lmy6WQHzDMWzxV//\ng8OO7W4xrnO+P49CvAvs3wANw77Pr/njtwfNE+88cgHQNmxYY+BneI2JHXBTwHQt8C5knb+/z6H0\nu5kB9AceJ+xClCgJiX88FuGdP66pyXUMW+YOf9q/4CeHQINQfHjt9L6lNIE8Kmw/Dsf7Djjg9YD4\npobFtw0vwQydcw8HFlD6Xcmu5DEdmnd+BcdPbgXTd6E0yZwMHIhXIyUNOBiY5Y9biv/9DjjnbPG3\nSb+wcQeG/X0HcC3eOTUt7BgYSOn/ov8ExJYbWodqbIOqnqNGRRxTYyn9v9UG+FfYcdOiquckvZLz\nFfcA9NIrni8qn5DcEvYPKSNs+JX+sDfDh0dM+7Bf5sGI4dP84Z8DTWOMY1RYHP+vgnKhX66frqBM\n6J/LjbEs25/GgNf96UZGjHvFH/7bSsxvf8V5Qth2OiVgfBtKL3puDxgfujgYVIVj65eU/np46P6M\nF2gdNl25uwIVLG8O1UhI/OPgS38e5e4a+mVCSfwK/Asnf/ghlP5C2iFgugPD1qlrxLi3/eG/iLLM\nFpRe6B4ZMS40z6/wL7CrsN75YfP5XcD4BsBn/vjHKjnv0DGwImDcPZQmuOW2WZT5DSIiIcG7kC3G\nS+h+FmW6Kq9j2DL39Z0O/Ur+I2HJWdj4k8Pmc2LEuKlh4y4OmDaH0jtRYyu5D0Lzzg8YF0tCEvrB\n5q4o4+sDi/wywyPGrQzbJjF/lwOO/w3+Ps6NGJcbWoeqbAOqcU6l7P+toGOqoR+3Ay6tyrrrlbwv\ntSERqZyCsL/D6wCP9N/vd87tiTLtU/77SaEBZtYEr9oEwO+dc1sqGc9e4E8VjA/F9ccKyjwdGde+\nOO+/x3/8j/0jRm/239vFOj/2U5x4v7ICzHfOzYoc6Zz7Di9RBDi/EvONxaX++xTn3OIYp6lqvFvx\nLj6gctu9ug7HSxzASyqD3Oa/5+JVzwHAOfcJXj8/aXhVyCKF6pK/58L6ADKzbnjHXCHwt6AFOucK\n8BJYiH68POiq/+S07cDEgOXvpPRYPs/MLNYZOufewlu3XDNrHzE6dEzd55z7pgrxYmbj8O5c7ca7\nGP7HPiap7jreW8G8Q8f7Y8659QHLeA2vOhZE/36uovTcED7t98AjEcvZ78xrp/VTvO9j4LnZObeb\n0oejRDs+/+5/3yvNP/7n4f1g0K8q86hATZxTdxJ8TO3Au3sE3p0kqUPUD4lI5ZT7p2tm6ZReaD1i\nZn+JMm2ooWp44+wj8b6HDni1CvEsd1H6RTGvEXjoEcWvRDTIDFc/IK7QPDri/Zo6FK+aRlPKPwwj\n8qLpFbwLzOvMrCXexcLb0ZKtmoizAn3999kVlHkD785XdzNr7JzbVon5B/IbFR/hf3ylEpNWKV7n\n3HYzm4tX3W+WmT0AvAx84pzbW8nwKyMU70bn3NKgAs65z83sG7z2NH2B98JGPw38P+Aiyl+8XRhW\nJlzoAqsJsLaCa/0m/nu04+XdKMMrY34Fx8tc/7050BWvilUJM/spcDHeNmmFd8chUnu8Oz2YWS7e\nr89QuWMqbJH2J7yqV9uAc5xz/4thuiqvI17Vm0VRgqlP6UXnvo734yg91srF4P9AEi2+W4GDzay+\nnwjsb0fgnasc8EkFx2foCX5VPj79hwpcjfed6IhX7S9S5Pm5umrinLqsgmMqlGhnVyNGSUJKSEQq\nJ/wkGbpb0oLSi+WWMcyjYdjfoQuMTc65TVWIZ2MF48J/KW8dw7wahX8ws4F4F7VNwgZvwvt1C7z1\naEbEP0Hn3N/NrD9eo8yf+a9iM1uM177hIefcupqKcx9a+e8V/Zq81n83vGoe1U5I8I6J0Pl1dSWm\nq068V+Ltr1549cvvALaa2ZvAM8CzzrmiSsRSU/GCF3OHsPIhzwB3AkeYWZ5z7ksAMzscbz324vUD\nFC50vKRT+v2pSLTjpaLvTqwqWu/wca3wL9b9HzCmUXpnFLxGyd/jrW+ofBplv1vh61qZYyqkM14y\nAl6bkViSEajCOob5wTlXTLAWlP64EcvxHnnsVCa+enjn7irdcaik0PFp7Mfj08xuxKvCF8p49uJV\n8wolXVl4SW7g0+aqoSbOqRXVBAj9f4n5SXGSGlRlS6RyDvXf14ZVzQr/HvVxztm+XjUYT0W/fofH\nlR1DXLmhwv4v/P/AS0b+CwzAq2/f3PkdSOI9AQkC7ho5536B9+vn7XjtFHbhVe8ZB3xpZuHVFKoc\nZyUE/fqcyCodr3Pua7zj81y8Buyf4u2/04Engff9KoL7Q5W2r3NuJaW/BF8UNip0d+R/zrkNEZOF\njpdFsXzXnHOjoix+f945qsjP8fbRduA6oJNzroFzrlXYd+tbv2xNnivWU3pH4y6/6tv+Fus2Trbv\nZ0VCx+emGI/PQVHmE3XbmdlBwB/wjo8H8TpwzXTOtQg7hkJVwmryGAqXSvtMEoASEpEY+VUMQn0q\nvBU2KvS4RfB+hayM0C92WWaWVY3wKpo3VD6u4/CqABTgVe14y68zHq7CX/+cc0udc+Odc4PxqnSc\nBXyC94vdE1baV0J14tyX0K+MFc03VF3M4f1KXRMK8J4iA94Td2JVrXidc0XOuenOuV8453rj/Vp7\nE96vjn2B8ZWIpTLx7qsaXSjmoF99Q1WyLgSvXhHeE6jCx4ULHS9V7pemBlVUHSZ8XPh6/9R/v8M5\n94Bzbm3YOMzrgyQnYH7h35PKHFMhu4AzgXfw7la9YWaxzKcq6xiLAkrbPcVyvEebfyzxhe4e1IbQ\nfmq2H87pIefhXb/Ncs5d65xbFlA1M5a7M1URr3OqpDglJCKx+zmlVYpCDdTx75TM9z+eVsl5zse7\ncLUqTFshvyFw6J9jZecd+ofyhXNue5QyQysRy27n3MuUXoy1w3tcZXXj3JeP/PeBFTS6DSWZX9RE\n+xEoOSYW+B9Pr8SkNRqvc269c+4+ShuQDowoErogrOqvqKF4G1v0TvK6410Ah5cPNw3vO9DDzPri\n1YfvjJdEPR9QPnRHpYWZHVPFuGvKkX4j5iChbV1IWGd0lH63Po4yXX8Cfn327yaFGn5X5pgKn8dW\nf9oP8LbxG347sYpUZR1jiWU33kMNwGv7FE3oeA86dsJjqGjckhpsPxJqrxLtOxN+Tj+1hpYZqcJj\nyLxOQY+NMm1xWLmqfO/jck6V1KeERCQGZnYKpU+Ledc595+IIlP991EW0KN0xLxK2qH4Fwgv+B9v\nM7OmNRBuUFw3mlmHaIXME96zb6g9S56Zlbs4MrOTiXIR4d9Jiib8qUbhPTBXNc59CVVbOAivz4bI\n+bXBaxQK3oVxTfq7/z7KzA6tsGSpKsVrXm/wFV1chLZ7ZK/XoSeiVWabhluI1wEbeI2Hg+T77yvx\nLoTLcM5txKsWCN5dklDVrZeDHoTgnPuM0obx91gFvZKbWSOL0tN3DWkMXB+w3ExKqzQ+F9HoOvTd\nOiRgunSiP60MvKp3AGMq+p5UxDm3GTgF78LyALykpKIns1VlHWMVOt5HBcXgn2eO8z9G+37mBvXu\nbV5P6Ff5H/9VhdiiqfA74x+z//Y/3l7ROd3M0qtYjTLqMeT7Hd4DSIJsDvu7Kt/7eJ5TJZW5BHj2\nsF56xetFBf2Q4DUKPAWv4W2oB9zVQPuAshmU9hb8A97dlGZh49vjPdr2Lco/1/1ASnu5DXWMGOro\nqiFe54KvREwzKlrcEeWa4/W34PAej3k+YX0v4D0Z5xq8J+GMipgu1Iv1NKBdWDyX4zVK/D4oBry+\nWCZR2u4kNPwgvKevOLw68unVjTPGfRzeiddwSjvxOoLSvgDWAy0Dpl1J1fshyaS0l/aNeB23NfLH\n1cN7wtqjVNwxYkzx4rXPWQqMBroDFnZcnkdpT8j3RCzrTn/4W0BWFb9DIyjtW+CBUFyU7xixXF8R\nYfO4xC+zhtJ+CM6toPxRlPYw/iZer9ih70w9f3vc5m+n3Ihp99mPRAzrnO/PI9TB2/WUdhp4AN4T\n8xxeIhjZaeD/88dtwrugC+3fnv6+34n3GOdyx52/Tdf6474GzqZsx4gDgWeJrWPEFmHH0zKgVQ2u\nY+AyA7ZjeMeIi/D7jPH34XnE3jHiVrwnloU6YT2U0s4xv6NmO0Z8xx/3ZyI6NQwrk0tpL+2f4N0p\nCXXaGOpc9ia8c17kPl4ZtO8jyoT3z3ILpeeVVng/nIWqSkVbh2/8cTdUcRtU6ZxKDP+3wo67qVX9\nfuqVnK+4B6CXXvF8hZ10d/sn0PX+P7DtYSd8h3eb+58E9OYdNq/WlHbY5vCSmB8ovbAPvcYHTDuY\n0p59Hd5Fyfd4FwKOiE6sYjmxh5U9EO+CIzTvIn/eOyLiGhkx3XUR4wvxOlJzeBfa1wbFgPerefg2\nKIhY1jZgSE3FGcP6t6I0MQhdQG0O+1wAHBdl2pVUMSHxp++Ed0ESuU67woYNqm68eBfg4dtoJ6Vt\nm0LDPiQsSfan6xkWyx68C5WVeI9prsx6TqD8Pg9fdmAHcWHTN6Hsd+5HvEa6FU1zGqWJVvh3Zg9l\nt0WXiOlCw3Orcd7I9+fxBF61stA5JPw7XARcEDBtC7y7Si5suk1h04yq6LjD+1V8TcT0keudG1Z+\nEFGSA/9YW0ppQtCyhtYx6jIDyh5NaeLh8I718O/8IqB1wHRTQ8cW3h2z0DGwKWzabcCAKuzf0Lzz\nA8ZdFjb/HXg/oKzE6xsmvNxRlF74h++n8O++AwZW5ZyDdxcmNI9iStvkOOCxfazDbWHTbvWXuRIY\nHeM2qNI5FSUkelXwUpUtEU8GXiPANni/Qu7E+wXyJbzb3wc450a4KH1+ADjvaUAD8X6pewXvV/Gm\neCfXz/Cq8JwP3B0w7WygB96TU5bg/aNvgPcL2jN4v4RWiXNuOdAHr+fw2XgXFFl4FzCL8Z7IdAbe\nU7XCp5sE/ATvF8HteI9Z/QyvYXQ/oj+68Uq/zGy8O0qhxxx/hvdEmINdwCNHqxpnDOu/Ea/ax414\n9bv34D2m+Uu8thUHOedqok+KoGWvwbsTch1esroF7+J7HV4HYFcSUY2pivF+ivdL5cN4FwqFeI9k\n3uQv91qgv/Oq64Qv6zO8jtle9cu2xWswva92BZHrORYYAryId9HVBC8hegkY6py7ZR/Tb8V7JHTI\n8865XfuYZibe3aAJeNWPduHdaduE1ync3cARzrlVlVmXSnJ47aJuwNsH9fGO25eBfs65ZwPiLsCr\n3/8QpY9H3QFMx7s4nVrhAr0OJQ8CxuIdHzvwqlWt9udxYdh8Kw7eO9aGAJ/j3VV4PaBKZKXXsTKc\ncx8AvfHuOHyBdy4u8tftJrw7iJFPWgu3Cy8Buh0vOaiPd+59FujrnHuzOvEFxDsF7w74B36cnfC+\nMzkR5T7ES/h/i3c8bsU7PrfjrdskvP09l6oZAfwf3j7Zg3fn5R28H2yu3Me0t/txLfan6+K/YqrC\nFc9zqqSu0G19EREREQDMLB/vh4UnXPRHJ8eNmU3FqwZ7m3MuP77RiEh16Q6JiIiIiIjEjRISERER\nERGJGyUkIiIiIiISN0pIREREREQkbtSoXURERERE4qZO3CExs5ZmdqWZvWBmy81sh5ltMrO3zewK\nM4t5O5jZSjNzUV7r9+d6iIiIiIikmvR4B1BLfor3zPd1lPaN0Aavj4XHgNPM7Kcu9ttFm/CetR1p\naw3EKiIiIiJSZ9SJKltmdiJex1H/cc4Vhw1vi9e5USdguHPu3zHMayWAcy53vwQrIiIiIlKH1Ikq\nW865N5xzM8KTEX/4eryejcHr6VVERERERGpRXamyVZE9/ntRJabJNLOfAZ2BbcBi4E3n3N6aDk5E\nREREJJXV6YTEzNKBS/2Pr1Zi0rbAkxHDVpjZZc65uTEue0GUUQfjtUVZWYl4REREREQqKxfY7Jzr\nGs8g6nRCAtyNlwC84pybFeM0U4C3gKXAFuAA4NfAVcBMMzvOObeoGjHVq1+/fosuXbq0qMY8JE6K\ni71agWlpdaI2ZErRvktu2n/JTfsveWnfJbdVq1axe/fueIdRNxq1BzGz64D7gc+A/s65gmrO7z5g\nDDDdOXduNeazIC8vr+8XX3xRnXAkTubMmQPAoEGD4hqHVJ72XXLT/ktu2n/JS/suuXXv3p0vv/zy\nI+fcEfGMo06ms2b2a7xkZBkwuLrJiC/UOH5ADcxLRERERKROqHMJiZmNBh4AluAlIzXVmeFG/71x\nDc1PRERERCTl1amExMx+C/wZWIiXjGyowdkf679/XYPzFBERERFJaXUmITGzcXiN2BcAQ5xz31dQ\nNsPMeppZt4jhvcys3B0QM8sFHvQ//qPGghYRERERSXF14ilbZjYSuB3Yi/eErOvMLLLYSufcVP/v\nDsCnwCq8x6GFjADGmNmb/rgtQDfgDKAB8Apw335ZCRERERGRFFQnEhIg9GzlesDoKGXmAlP3MZ/Z\nQA+gD9Afr71IIfA2Xr8kT7q6+tgyEREREZEqqBMJiXMuH8ivRPmVQLlbKH6nhzF1fCgiIiIiIvtW\nJxISERGRZFRcXExBQQFbtmxh165dpOpN+EaNGgHw6aefxjkSqSztu8RhZmRmZtK0aVNatGiRVJ1V\nKiERERFJQMXFxaxZs4bt27fHO5T9LnRRK8lH+y5xOOfYuXMnO3fuZNu2bXTq1ClpkhIlJCIiIgmo\noKCA7du3k56eTtu2bWncuHHSXFxU1pYtWwBo2rRpnCORytK+SxzFxcVs27aN9evXs337dgoKCsjJ\nyYl3WDFJzTObiIhIkgtd6LVt25amTZumbDIiIjUjLS2Npk2b0rZtW6D0HJIMdHYTERFJQLt27QKg\nceNy3V+JiEQVOmeEziHJQAmJiIhIAgo1YNedERGpjFBfe8n0EAyd5UREREREUkRA598JTwmJiIiI\niIjEjRISERERERGJGyUkIiIiIvvRnj17GD9+PHl5eWRmZmJmTJ8+vUrzmjp1KmbG1KlTywzPzc0l\nNze3+sHuJ9HirqyVK1diZowaNapG4pLEoIREREREZB/y8/MxM+bMmVPpaf/4xz9y++230759e268\n8UbGjx9Pz549az7ISjAzBg0aFNcYJDZ1YV+pY0QRERGR/ejll1+mSZMmvP7669SvXz/e4cTFueee\ny7HHHku7du2qNZ8OHTrw6aefkpWVVUORSSJQQiIiIiKyH3377be0bNmyziYjAFlZWTWSRGRkZMT9\n7pLUPFXZEhERkYT3wQcfMGLECDp06EBmZibt2rXj5JNPZtq0aeXKTps2jQEDBpCVlUXDhg055JBD\nuOuuuwI7ilu8eDEXXnghubm5ZGZm0qpVK/r27cvo0aPZs2cP4LXPuO222wAYPHgwZlbyqsioUaMw\nM1asWMGqVatKpgm19ZgzZw5mRn5+fuD0+6NdSKgtB8DcuXPLrEsojvB2Gl988QUjRoygdevWpKWl\nlVRZW7BgAddffz39+vWjc+fONGjQgLy8PMaMGcOPP/4YdbnR2r5s27aNm266ic6dO5OZmcmBBx7I\nH/7wh3J9aURrQxLa1itXruSRRx7hkEMOoUGDBrRp04arrrqKTZs2BW6PWbNm0b9/fxo3bkyLFi0Y\nNmwYn332WZn5xeLrr7/mqquu4sADD6Rhw4a0aNGCQw45hKuvvpoffvihXPlnnnmGwYMH07x5cxo0\naECvXr2YMGFCmWM0ln2VKnSHRERERBLao48+yjXXXEO9evU4++yzycvLY8OGDcyfP5/Jkydz/vnn\nl5S99dZbueuuu8jJyeGiiy6iSZMmzJw5k1tvvZVZs2bx2muvldypWLx4Mccccwxmxtlnn03Xrl3Z\nvHkzy5cvZ/LkyUyYMIGMjAxGjx7N9OnTmTt3LiNHjow5SRg2bBi5ublMnDgRgNGjRwPQvHnzmt1A\nlXD44Yczfvx4brvtNrp06VLmwj6yncJXX33FMcccQ/fu3bn44ovZsWMHzZo1A7x98sILL9C/f38G\nDRpEeno6CxYs4E9/+hMzZ87k/fffp2nTpjHFtGfPHk455RS+/fZbTjvtNNLT05k+fTr/93//x86d\nOxk/fnzM63fzzTcza9YszjrrLE4++WRmz57No48+yvLly3njjTfKlH322We56KKLaNCgAeeffz7t\n2rVj3rx5HHfccRx22GExL3PdunUcddRRbN68mdNPP53zzjuPnTt3smLFCp588kl+/etf07Jly5Ly\nl19+OVOmTKFjx46cd955NG/enPfee49x48bxv//9j9dff5309PRK7auk55zTK4FewIK8vDwnyWn2\n7Nlu9uzZ8Q5DqkD7Lrml4v5btmyZW7ZsWeA4SK7XvmzevNlt3rw5cNzSpUtdenq6y87OdkuWLCk3\nfs2aNSV/z5s3zwGuU6dObt26dSXD9+zZ484880wHuDvvvLNk+A033OAAN3369HLzLSgocHv37i35\nPH78eAdU6Tjr0qWL69KlS7nhs2fPdoAbP358zNNNmTLFAW7KlCkxLSMawA0cODBw3IoVKxzgAHfL\nLbcEllmK+WhTAAAgAElEQVS5cqUrKioqt+8ee+wxB7i777475rgBd9ppp7nt27eXDP/uu+9cVlaW\ny8rKcrt37y4X28iRI8vMZ+TIkSX7ftWqVSXD9+zZ40444QQHuPfff79k+ObNm13z5s1d/fr13cKF\nC8vM67e//W3J+q9YsSJw/cNNmjTJAW7ixInlxm3durXMeoW2w7nnnltmuHOlx1jkfCraV9FUdP4I\nl5eX54AFLs7Xv6qyJSIiIgnroYceoqioiHHjxnHQQQeVG9+xY8eSvx9//HEAxo4dS9u2bUuGp6en\n88c//pG0tDQee+yxcvNo2LBhuWHZ2dmkpdXty6Q2bdpEvTvRpUsX6tWrV2745ZdfTrNmzZg1a1al\nljVp0qQy+6F169acc845bNq0ic8//zzm+fz+97+nc+fOJZ/T09O57LLLAK/aX8iLL75IYWEhF198\ncbm7IWPHjq3SXayg46hx48Zlht9///2kp6fz+OOPlys/btw4WrZsyVNPPVXpZSc7VdkSERGRhPXe\ne+8BcNppp+2z7EcffQTAiSeeWG5c9+7d6dixIytWrGDTpk1kZWUxYsQI7r//foYNG8bw4cMZOnQo\n/fv3p1u3bjW7EknqsMMOIzMzM3Dcnj17eOSRR3jqqaf47LPP2Lx5M8XFxSXjv/nmm5iXk5WVxYEH\nHlhueKdOnQAC26REc+SRR8Y0n48//hiA448/vlz5Jk2acPjhh8f8iOezzz6bW2+9lV/96lfMmjWL\nU045hf79+9O7d+8y7Yy2b9/OokWLyMnJKanGFykzM5NPP/00puWmEiUkIiIikrAKCwsB73Gv+xJq\nuBzt0bLt2rVj9erVFBYWkpWVxdFHH81bb73FnXfeyXPPPceTTz4JQI8ePRg/fjwXXnhhDa1Fcgq/\nyxRpxIgRvPDCC+Tm5nLGGWfQqVOnkuRl4sSJgQ8QiCba3Yj0dO8yde/evdWaV9B8QsdKmzZtAucT\nbXiQLl268MEHH5Cfn8+rr77K888/D3iJ0I033sh1110HeAmRc46NGzeWPCRBPEpIREREkoxz+y6T\nKkIXmN98880+H/caeqzs+vXrA+9yrFu3rkw5gOOOO46XX36ZXbt2sWDBAl599VUeeOABLrroIlq1\nasXQoUNralXKCVUJKyoqChxfWFgY1wbw0Z4iNn/+fF544QWGDh3KtGnTSE9PL2nAXlxczD333FOb\nYVZJqHH+d999Fzg+2vBoevXqxT//+U+KiopYtGgR//3vf3nggQe4/vrrady4MVdccUXJcdenT5+S\nu3niqduVI0VERCShHXvssQDMnDlzn2X79OkDEFjVZvny5axdu5auXbsGXuRnZmbSr18/br/9diZN\nmgR47QxCQu0lKvNr/b5kZ2cDsGbNmsB4oz2qtiakpaVVeV2WL18OeFWVQncfQj744AN27NhR7fj2\nt9Cx8vbbb5cbt3XrVhYuXFil+aanp3PEEUfw29/+lmeeeQaA6dOnA15VsIMOOoilS5dSUFAQ8zyr\ns6+ShRISERERSVjXXHMN6enp3HHHHSxbtqzc+LVr15b8ffnllwMwYcIENm7cWDJ879693HjjjRQX\nF3PFFVeUDJ83b17gxXPo1/FGjRqVDAs9tnX16tXVXKNSPXv2pFmzZrz44ots2LChZPiOHTtKqvns\nLy1btgxMhGIR3o9KuA0bNvCrX/2qmpHVjnPOOYesrCyeeuopFi1aVGbchAkTSqoKxmLBggWByWPQ\ncXTDDTewe/duLr/88sBl/Pjjj+XunlRnXyULVdkSERGRhNW7d28mT57M1VdfTZ8+fTjnnHPIy8vj\nhx9+4MMPP6RZs2bMnj0bgH79+nHzzTdzzz33cPDBBzN8+HAaN27MzJkzWbJkCccffzw33XRTybzv\nuece3njjDU444QS6du1KkyZNWLp0KTNnziQ7O5urrrqqpOzgwYNJS0vjlltuYcmSJSV3N8aOHVvl\ndcvIyOD666/njjvuoE+fPpx77rkUFRXx+uuv0759e9q3b1/lee/LkCFDePbZZznrrLPo27cvGRkZ\nDBgwgAEDBuxz2qOOOor+/fvz/PPPs3btWo477jh+/PFHZs6cSY8ePfZr3DWlWbNm/OUvf+GSSy6h\nX79+ZfohWbRoEQMHDmTu3LkxPWntySef5JFHHuH444+nW7duZGdn89VXXzFjxgwyMzNL+p8BL2le\nsGABkydPplu3bpxyyil07tyZgoICVqxYwZtvvslll13Gww8/XDJNdfZVslBCksg2bfJeYY+vExER\nqWt+/vOfc/DBB3PfffcxZ84cpk+fTk5ODoceeihXXnllmbJ/+MMf6NOnDw8++CB///vf2bNnD926\ndWPChAmMGTOmpFNEgF/+8pdkZ2fz/vvv8/bbb1NUVETHjh355S9/yZgxY+jSpUtJ2V69evHEE09w\n3333MXnyZHbu3AlULyEBuO2222jUqBGPPvoof/3rX2nbti0XXHAB+fn59O7du1rzrsj999+PmfG/\n//2PV155heLiYsaPHx/TRW69evV46aWXGDt2LP/5z394+OGH6dChA1deeSVjx47dr3HXpIsvvpgW\nLVpwxx138M9//pPMzEwGDBjAu+++y4033giUtjWpyIUXXsiuXbuYN28eCxYsYMeOHXTo0IELLriA\nMWPGcPDBB5cp/5e//IXTTjuNhx9+mP/+978UFhbSokULOnfuzE033cTPfvazMuWrs6+Shbm61DIu\nCZjZgry8vL5fTJ4Ml18O7dvD229DunLHZBC6fZ1yPajWAdp3yS0V91/o0Z+9evWKcyT735YtWwBi\n7tlbEkcq7ru9e/dywAEHsHv37pIHISSbWM8f3bt358svv/zIOXdEbcQVjdqQJKAGGzbASSfBmjXw\n/vtw333xDklEREQkpRQWFrJ9+/Yyw5xzTJgwgdWrV3PuuefGKbK6Rz+7J6CMyIZR48fDGWfAIYfE\nJyARERGRFPPee+8xYsQITj75ZHJzc9m6dSvvvfceCxcupFOnTuTn58c7xDpDCUky2L0bRo707pZk\nZMQ7GhEREZGk16NHD84880zeeecdXnnllZI2RNdddx233norrVu3jneIdYYSkmTx8cdw552gbF1E\nRESk2rp27cpTTz0V7zAEtSFJSMXRdsudd8Inn9RuMCIiIiIi+5ESkgS0q1Wr4BFFRV5SIiIiIiKS\nIpSQJKCi5s0oOObU4JH/+hd8/XXtBiQiIiIisp8oIUlQLf79GLszGpUfUVwMf/pT7QckIiIiIrIf\nKCFJVB06kPbzK4PHPf44bNxYu/GIiIiIiOwHSkgSWPpNv6E4rV75ETt2wF/+UvsBiYiIiIjUMCUk\niSw3l73DRwSP+8c/wLnajUdEREREpIYpIUlwGbfeHDziq6/g889rNxgRERERkRqmhCTRHXYYRYcc\nHjxuxozajUVEREREpIYpIUkC6cPOCh6hhERERKTOMzMGDRpUZlh+fj5mxpw5c2Kez6BBgzCzmg0u\nQFC8UrcpIUkGZ0VJSN55B374oXZjERERqYOqcoFfV+Xm5pKbmxvvMBLC1KlTMTOmTp0a71ASmhKS\nZHDEERS3blt+eHExzJxZ+/GIiIhIQvv1r3/Np59+ytFHHx3vUMr59NNP+fvf/x7vMCSBKCFJBmlp\npJ11RvA4VdsSERGRCDk5OfTs2ZNGjQI6WY6znj170rlz53iHIQlECUmyiFZt69VXYe/e2o1FRESk\nln3wwQeMGDGCDh06kJmZSbt27Tj55JOZNm1aubLTpk1jwIABZGVl0bBhQw455BDuuusudu3aVa7s\n4sWLufDCC8nNzSUzM5NWrVrRt29fRo8ezZ49ewCvCtJtt90GwODBgzGzkldFnn32WcyM3/zmN4Hj\nd+3aRXZ2Nu3ataOoqAiATZs2ce+993LiiSfSsWNH6tevT6tWrTj77LN59913Y95eFVUxe/bZZzni\niCNo2LAhrVu35pJLLuHbb78NnM/u3bt58MEHOf300+nSpQuZmZm0aNGCoUOHMjOilsacOXMwM1at\nWsWqVavKbKdRo0aVlIvWhmTTpk3ccsst9OjRgwYNGpCdnc0pp5zCf//733JlQ8vKz89n4cKFnHHG\nGTRv3pxGjRoxcOBA5s2bF/O2AnjppZcYMmQI7dq1IzMzk/bt2zNw4EAmT55crmxBQQG33HILvXr1\nomHDhmRlZTFkyBBee+21MuUGDRrEZZddBsBll11WZnusXLmyUvGluvR4ByAxGjoUl5mJRZ5MN2/2\nHv/bu3d84hIRkdpVC42O94tq9J316KOPcs0111CvXj3OPvts8vLy2LBhA/Pnz2fy5Mmcf/75JWVv\nvfVW7rrrLnJycrjoooto0qQJM2fO5NZbb2XWrFm89tpr1K9fH/CSkWOOOQYz4+yzz6Zr165s3ryZ\n5cuXM3nyZCZMmEBGRgajR49m+vTpzJ07l5EjR8bcPmLYsGFkZWXx9NNPc++995KeXvay68UXX6Sw\nsJAxY8aUjPv000/53e9+x4ABAzjjjDPIzs5m9erVvPTSS8ycOZMZM2Zw6qmnVnlb/vnPf+aGG26g\nefPmXHrppTRv3pxZs2bRr18/srKyypUvKCjg+uuvp1+/fpx00km0atWKdevWMWPGDE4//XQeeOAB\nRo4cCXiJ2/jx45k4cSIAo0ePLpnP4YdHeWKor7CwkP79+7Ns2TKOOuooRo8ezffff8+0adM4+eST\neeihh/jFL35Rbrr58+dzzz33cNxxx3HllVeyevVq/v3vfzNkyBAWLlxIjx499rlN/vrXv/KLX/yC\ntm3bctZZZ5GTk8OGDRtYvHgxU6ZM4Ze//GVJ2VWrVjFo0CBWrlzJCSecwKmnnsq2bdt4+eWXOfXU\nU3nkkUf4+c9/DsCoUaNo3rw5L774Iuecc06ZbdC8efN9xlWnOOf0SqAXsCAvL88FOuEE57xTetnX\nk08Gl5daN3v2bDd79ux4hyFVoH2X3FJx/y1btswtW7as/Iig/wPJ8KrA5s2b3ebNmwPHLV261KWn\np7vs7Gy3ZMmScuPXrFlT8ve8efMc4Dp16uTWrVtXMnzPnj3uzDPPdIC78847S4bfcMMNDnDTp08v\nN9+CggK3d+/eks/jx493QKWPs6uuusoBbsaMGeXGnX766Q5wixcvLhlWWFjoNm7cGLie7dq1cz17\n9iw3DnADBw4sMywo3hUrVriMjAyXnZ3tVqxYUTJ879697ic/+YkDHBH7aufOnWW2cXicBx10kGve\nvLn77rvvyozr0qWL69KlS7lpKoo3tJ2uuuoqV1xcXDL8iy++cM2aNXP169cvE/Ps2bNL4p0yZUqZ\neT388MMOcNdcc03UGML17dvX1a9fv9x6OOfK7YuBAwc6M3PPPPNMmeE//vijO+yww1yDBg3c+vXr\nS4ZPmTIlMMb9Ler5I0JeXp4DFrg4X/+qylYy6ds3ePhHH9VuHCIiIrXkoYceoqioiHHjxnHQQQeV\nG9+xY8eSvx9//HEAxo4dS9u2pQ+DSU9P549//CNpaWk89thj5ebRsGHDcsOys7NJS6v+ZVLo7sET\nTzxRZvj69euZNWsWffr04ZBDDikZnpWVRU5OTrn5dOzYkeHDh/PZZ5+xevXqKsXy1FNPsWfPHq69\n9toyd3nS0tK49957A9c3MzOzzDYOj/Pyyy+nsLCQj6p5HbJ7927+8Y9/0KRJE+66664yVeHy8vK4\n7rrr2L17d2BD+P79+5epDgZw+eWXk56ezgcffBBzDOnp6WRkZJQbHr4vFi1axNy5cznvvPO44IIL\nypRr3rw5t912Gzt37uTf//53zMsVj6psJZMjjggeroRERERS1HvvvQfAaaedts+yoQvjE088sdy4\n7t2707FjR1asWMGmTZvIyspixIgR3H///QwbNozhw4czdOhQ+vfvT7du3Wos/n79+tG9e3dmzJjB\njz/+SHZ2NuAlB3v37i13MQ3wzjvvcP/99/Puu++yYcMGdu/eXWb8N998U6VG4aHtM3DgwHLjDjjg\nADp16sSqVavKjVu6dCn33nsvb775JuvWrWPnzp1lxkdrfxKrzz//nO3bt9O/f39atGhRbvyJJ57I\nhAkT+Pjjj8uNO/LII8sNy8jIoE2bNvz4448xLf/iiy9mzJgx9O7dmwsuuICBAwfSv39/WrVqVaZc\nqA3Ppk2byM/PLzefjRs3Al61O6kcJSTJJNodko8/9h4BXAO/5IiIiCSSwsJCADp06LDPsps2bQKg\nXbt2gePbtWvH6tWrKSwsJCsri6OPPpq33nqLO++8k+eee44nn3wSgB49ejB+/HguvPDCGlmHkSNH\n8rvf/Y5nn32Wa665BvDumGRkZHDRRReVKfvCCy8wfPhwGjRowEknnUS3bt1o3LgxaWlpzJkzh7lz\n5wY2zo9FaPu0adMmcHzbtm3LJSTvvfceJ554IkVFRQwZMoSzzz6bZs2akZaWxsKFC3nxxRfLJUxV\njaui/Qalx0K4aG0x0tPT2RvjQ39uuOEGcnJymDx5MpMmTWLixImYGQMHDuTee+8tSXp+8Pt+e/31\n13n99dejzm/r1q0xLVdKKSFJJj164Bo2xHbsKDt882b4+ms48MD4xCUiIrKfhC44v/nmG3r27Flh\n2VCj7PXr1wfe5Vi3bl2ZcgDHHXccL7/8Mrt27WLBggW8+uqrPPDAA1x00UW0atWKoUOHVnsdLrnk\nEsaNG8cTTzzBNddcw8cff8wnn3zCOeecU6561rhx46hfvz7z58+nV69eZcb94he/YO7cuVWOI7Te\n3333XWD1t/Xr15cbNmHCBHbs2MHs2bPLPRnrrrvu4sUXX6xyPJFxBS0fgvdbTbv00ku59NJLKSws\nZN68ebzwwgs8/vjjnHLKKXz22We0atWqZPn3338/11133X6LpS7ST+rJJD0dO+yw4HGqtiUiUjfE\nv3l61V5VdOyxxwKUe8RskD59+gAEPup2+fLlrF27lq5duwb+qp6ZmUm/fv24/fbbmTRpEkCZi+16\n9eoBxPyre7hOnTpx4okn8v777/P555+XtCcJtS+JjLN3797lkpHi4mLefvvtSi87XF+/pkVQUvP1\n11+zZs2awHhatGgR+JjeaMlRvXr1KrWdevToQaNGjVi0aFHgXZDZs2eXiX9/at68OaeffjqPPvoo\no0aNoqCggDfffBMoPRbfeuutmOdXneOmLlFCkmzUsF1EROqQa665hvT0dO644w6WLVtWbvzatWtL\n/r788ssB71f9UH1+8C4Gb7zxRoqLi7niiitKhs+bN48dkbUO8O4gAGU6FWzZsiVAlRuUh9qK/O1v\nf+OZZ54hJyeHM888s1y53NxcvvzyyzLtMpxz5OfnB65/ZVx88cVkZGTwwAMPlOkHo7i4mJtuuoni\n4uLAeAoKCli8eHGZ4X/729+YNWtW4HJatmzJxo0bA7dtkPr163PxxRezZcsWxo0bV2bcV199xaRJ\nk8jIyOCSSy6JaX6VNXv2bFxA0rxhwwag9Dg48sgjOeGEE3j++edLHqAQ6ZNPPimZDqp/3NQVqrKV\nbJSQiIhIHdK7d28mT57M1VdfTZ8+fTjnnHPIy8vjhx9+4MMPP6RZs2Ylv6D369ePm2++mXvuuYeD\nDz6Y4cOH07hxY2bOnMmSJUs4/vjjuemmm0rmfc899/DGG29wwgkn0LVrV5o0acLSpUuZOXMm2dnZ\nXHXVVSVlBw8eTFpaGrfccgtLliwpaZw+duzYmNbj3HPPpVmzZkycOLHkSVdBT3X6zW9+U7Ku5513\nHhkZGbzzzjssW7aMs846ixkzZlR5W+bm5nL33XczZswY+vTpw4gRI8jKymLWrFkUFhZy6KGHlks8\nRo8ezaxZszj++OM5//zzycrKYv78+bz99tsMHz6c5557rtxyhgwZwocffsipp57KgAEDyMzM5LDD\nDuOsaJ08A3fffTdvvfUWDz74IB9++CGDBw8u6Ydky5YtPPjgg3Tt2rXK616Rc889lyZNmnDssceS\nm5uLc4633nqLDz/8kCOOOKJMtb2nn36aE088kSuuuIJJkyZxzDHH0Lx5c9auXcvixYtZsmQJ7777\nLq1btwa8KoGNGjVi4sSJ/PDDDyVPf7v22mv3axW0pBPv5w7rVfZFRf2QOOfcRx8F3gwvbtnSubDn\ndkt8pGJfCHWF9l1yS8X9F2s/Aqmgon5IQubNm+d+8pOfuFatWrmMjAzXrl07d8opp7h//etf5co+\n88wzrn///q5JkyYuMzPT9e7d202YMMHt2LGjTLlZs2a5UaNGuV69erlmzZq5Ro0aue7du7trr73W\nrVy5stx8n3zyyZK+Jgjos2NfrrjiipLp5s+fH7XclClT3GGHHeYaNWrkWrZs6YYNG+YWL14ctS8U\nYuyHJOTpp592ffr0cZmZmS4nJ8ddfPHF7ptvvnEDBw4MXKcZM2a4Y445xjVp0sRlZWW5k046yc2d\nO7ekj42HHnqoTPmtW7e6q6++2nXo0MHVq1fPAW7kyJEVxuuc15fHzTff7A488EBXv359l5WV5YYO\nHepmzZpVrmyoH5Lx48cHbsN99YUS7qGHHnLDhg1zXbt2dQ0bNnTZ2dnu8MMPd3/4wx8Cj8vNmze7\nO++80/Xt29c1btzYNWjQwOXm5rrTTz/dPfLII27r1q1lys+cOdMde+yxrnHjxiX7P7xPlf0h2foh\nMeeqXq9Tap6ZLcjLy+v7xRdfBBfYvRvXpAm2Z0/5catWQRUeAyg1J1RvOaiurSQ27bvklor7L/To\n0Mi2BKloy5YtADRt2jTOkUhlad8lpljPH927d+fLL7/8yDkXpW+J2qE2JMmmfn0srAOlMhYtqt1Y\nRERERESqSQlJMjr00ODhy5fXbhwiIiIiItWkhCQZ5eUFD1dCIiIiIiJJRglJMorWAaISEhERERFJ\nMkpIkpESEhERERFJEUpIklG3boGD3cqVsHt37cYiIiIiIlINSkiSUVYWtGpVbrAVF0NYz6siIiIi\nUrckY5ceSkiSlaptiYikNDMDoLi4OM6RiEgyCSUkoXNIMlBCkqz0pC0RkZSWmZkJwLZt2+IciYgk\nk9A5I3QOSQZKSJKV7pCIiKS0UM/X69evZ8uWLRQXFydlVQwR2f+ccxQXF7NlyxbWr18PlJ5DkkF6\nvAOQKlJCIiKS0lq0aMG2bdvYvn07a9eujXc4+9XevXsBqFevXpwjkcrSvktMjRo1okWLFvEOI2ZK\nSJJVlITELV9O8tQYFBGRaNLS0ujUqRMFBQVs2bKFXbt2pewdku3btwPJ9YuueLTvEoeZkZmZSdOm\nTWnRogVpaclTEUoJSbKKdodkxQooKoJ07VoRkWSXlpZGTk4OOTk58Q5lv5ozZw4ARx99dHwDkUrT\nvpOakDypk5SVnQ0Bt+KsqAhWr45DQCIiIiIilaeEJJmpHYmIiIiIJDklJMlMCYmIiIiIJDklJMks\nWkLy1Ve1G4eIiIiISBUpIUlmubnBw9esqdUwRERERESqqk4kJGbW0syuNLMXzGy5me0ws01m9raZ\nXWFmldoOZtbRzB43s2/NbJeZrTSziWaWvb/WIVDnzsHD1ahdRERERJJEXXk27E+Bh4B1wGxgNdAG\n+AnwGHCamf3UxfCAdzPrBswDWgMvAp8BRwPXA6eaWX/n3A/7ZS0ideoUONitWaO+SEREREQkKdSV\nhOQL4GzgP8654tBAM7sV+AA4Dy85+XcM85qMl4xc55x7IGxefwJ+A9wJXF1zoVegY8fg4evWwZ49\nkJFRK2GIiIiIiFRVnaiy5Zx7wzk3IzwZ8YevBx72Pw7a13z8uyMnAyuBv0SMHg9sAy4xs8bVjTkm\njRpBQGdZ5hx8802thCAiIiIiUh11IiHZhz3+e1EMZQf7768FJDdbgHeARsCxNRfePkSptqWG7SIi\nIiKSDOpKla1AZpYOXOp/fDWGSXr4719EGf8l3h2U7sD/9rHsBVFG9SwuLmbOnDkxhAMHN2xI+Xsk\nsGzWLDbs3RvTPKTmbNmyBSDm/SeJQ/suuWn/JTftv+SlfZfciouL912oFtT1OyR3AwcDrzjnZsVQ\nPst/3xRlfGh48+oGFqudbdoEDm+wYUNthSAiIiIiUmV19g6JmV0HjMF7StYltb1859wRQcPNbEFa\nWlrfQYMGxTajDz6AF14oN/iAjAwOiHUeUmNCvxDFvP8kYWjfJTftv+Sm/Ze8tO+SW1paYtybSIwo\napmZ/Rq4H1gGDHbOFcQ4aegOSFaU8aHhhdUIr3LUhkREREREklidS0jMbDTwALAELxlZX4nJP/ff\nu0cZn+e/R2tjUvPUOaKIiIiIJLE6lZCY2W+BPwML8ZKRyja0mO2/nxzZu7uZNQX6A9uB96oba8yi\n3CEpXq07JCIiIiKS+OpMQmJm4/AasS8Ahjjnvq+gbIaZ9fT7HSnhnPsKeA3IBX4VMdltQGPgSefc\ntpqMvULt20NA/b+0HwtgW+2FISIiIiJSFXWiUbuZjQRuB/YCbwHXmVlksZXOuan+3x2AT4FVeMlH\nuF8C84BJZjbEL3cMXh8lXwC/q/k1qEB6upeUrF1bftyaNdCzZ62GIyIiIiJSGXUiIQG6+u/1gNFR\nyswFpu5rRs65r8zsSLwE51TgdGAdXiP525xzP1Y72srq1EkJiYiIiIgkpTqRkDjn8oH8SpRfCZS7\nhRI2fg1wWXXjqjGdO8O775YfridtiYiIiEiCqzNtSFJatEf/6klbIiIiIpLglJCkAvVFIiIiIiJJ\nSglJKlBfJCIiIiKSpJSQpIIod0jct9/WciAiIiIiIpWjhCQVtG8fONh9o4RERERERBKbEpJU0Lo1\n1KtXbnDapkLYvj0OAYmIiIiIxEYJSSqoVw/atg0ep2pbIiIiIpLAlJCkiijVtpSQiIiIiEgiU0KS\nKpSQiIiIiEgSUkKSKqIlJN98U7txiIiIiIhUghKSVNGhQ/Bw3SERERERkQSmhCRVqMqWiIiIiCQh\nJeNA0/4AACAASURBVCSpQgmJiIiIiCQhJSSpIlrniGvVhkREREREEpcSklQRpQ2J+/ZbcK6WgxER\nERERiY0SklSRnQ2ZmeUGp+3cAZs2xSEgEREREZF9U0KSKszUjkREREREko4SklSivkhEREREJMko\nIUkl6otERERERJKMEpJUoipbIiIiIpJklJCkEiUkIiIiIpJklJCkErUhEREREZEko4QklVTUF4mI\niIiISAJSQpJKovXW/o0SEhERERFJTEpIUkm7doGDbf06KC6u5WBERERERPZNCUkqadrUe0WwoiLY\nuDEOAYmIiIiIVEwJSapRXyQiIiIikkSUkKQaPfpXRERERJKIEpJUo4RERERERJKIEpJUo75IRERE\nRCSJKCFJNWpDIiIiIiJJRAlJqlGVLRERERFJIkpIUo0SEhERERFJIkpIUk3U3trVhkREREREEo8S\nklQTrbf2DRtgz55aDkZEREREpGJKSFJNZibk5ASPW7++dmMREREREdkHJSSpSO1IRERERCRJKCFJ\nReqLRERERESShBKSVKS+SEREREQkSSghSUWqsiUiIiIiSUIJSSpSQiIiIiIiSUIJSSpSGxIRERER\nSRJKSFKR2pCIiIiISJJQQpKKovXWroRERERERBKMEpJU1Lo1pJXftVZYCNu3xyEgEREREZFgSkhS\nUb160LZt8DjdJRERERGRBKKEJFWpHYmIiIiIJAElJKmqXbvg4evW1W4cIiIiIiIVUEKSqtQXiYiI\niIgkASUkqUoJiYiIiIgkASUkqUpVtkREREQkCSghSVW6QyIiIiIiSUAJSapSQiIiIiIiSUAJSapS\nlS0RERERSQJKSFJVq1a4evXKD9+8GbZtq/14REREREQCKCFJVWlpWLTe2nWXREREREQShBKSVKZ2\nJCIiIiKS4JSQpDIlJCIiIiKS4JSQpDI1bBcRERGRBKeEJJXpDomIiIiIJDglJKlMCYmIiIiIJDgl\nJKlMVbZEREREJMEpIUllukMiIiIiIglOCUkqi3aHRAmJiIiIiCQIJSSpLFpv7Vu2wNattR+PiIiI\niEgEJSSpLC0NUzsSEREREUlgSkhSnRISEREREUlgSkhSnRq2i4iIiEgCU0KS6pSQiIiIiEgCU0KS\n6lRlS0REREQSmBKSVKc7JCIiIiKSwJSQpDolJCIiIiKSwOpMQmJmw83sATN7y8w2m5kzs39UYT4r\n/WmDXuv3R+zVoipbIiIiIpLA0uMdQC0aCxwGbAXWAj2rMa9NwMSA4YnX26DukIiIiIhIAqtLCclv\n8BKR5cBAYHY15lXonMuviaD2u5wcXHo6VlRUdniot/YmTeITl4iIiIgIdajKlnNutnPuS+eci3cs\ntSotDdq2DR6nalsiIiIiEmd16Q5JTco0s58BnYFtwGLgTefc3viGFczat4e1a8uP+PZbyMur/YBE\nRERERHxKSKqmLfBkxLAVZnaZc25uLDMwswVRRvUsLi5mzpw51YmvjIMzMsgJGL7sf/9jQx27YbS/\nbdmyBaBG95/UDu275Kb9l9y0/5KX9l1yKy4ujncIQB2qslWDpgBD8JKSxsAhwCNALjDTzA6LX2jB\ndrVsGTi8/vff13IkIiIiIiJl6Q5JJTnnbosYtAS42sy2AmOAfODcGOZzRNBwM1uQlpbWd9CgQdWM\nNMzbb8NLL5UbfGCjRhxYk8uRkl+IanT/Sa3Qvktu2n/JTfsveWnfJbe0tMS4N5EYUaSGh/33AXGN\nIoj6IhERERGRBKWEpOZs9N8bxzWKIOqLREREREQSlBKSmnOs//51XKMIEu0OiRISEREREYkzJSQB\nzCzDzHqaWbeI4b3MrNwdEDPLBR70P/5j/0dYSdHukKjKloiIiIjEWZ1p1G5mw4Bh/sdQT4HHmdlU\n/+/vnXM3+n93AD4FVuE9PStkBDDGzN70x20BugFnAA2AV4D79tMqVF1FvbVv2QJNm8YnLhERERGp\n8+pMQgIcDoyMGHaA/wIvwbiRis0GegB9gP547UUKgbfx+iV5MiF7gk9Lw7Vpi30T0DniunVKSERE\nREQkbupMQuKcy8d7JG8sZVcCFjB8LhBTx4eJJq1De4iWkHTvXvsBiYiIiIigNiR1h560JSIiIiIJ\nSAlJXaG+SEREREQkASkhqSt0h0REREREEpASkrpCfZGIiIiISAJSQlJXqC8SEREREUlASkjqClXZ\nEhEREZEEpISkrlCVLRERERFJQEpI6oqcHFy9gG5ntm71emsXEREREYkDJSR1RVoae9vo0b8iIiIi\nkliUkNQhaR1UbUtEREREEosSkjokrYOetCUiIiIiiUUJSV2ihu0iIiIikmCUkNQl6otERERERBKM\nEpK6RH2RiIiIiEiCUUJSl6jKloiIiIgkGCUkdYmqbImIiIhIglFCUpeoypaIiIiIJBglJHVJy5YU\nq7d2EREREUkgSkjqkrQ0/n979x1vV1Xn///1ublJIAVCAqGGBEILApEiVSDICKjMfNERe8MCjvoV\nROc7MzqjML9B0UFFRRwLqIODjoMOqCggJZQEQhENLRBKQpBQQkJIb3f9/ljnets5t556z+v5eOzH\nPmetvfdZyWaH8z57r7U27+Bs7ZIkSaofBpImE6UmR/SxLUmSJNWAgaTJtO7mSFuSJEmqHwaSJlPy\nDomPbEmSJKkGDCTNxrlIJEmSVEcMJM3GuUgkSZJURwwkzca5SCRJklRHDCTNxke2JEmSVEcMJM2m\n1B2SP/8ZUqpuWyRJktT0hhRIIuKuiLgoIqaWq0GqsEmT2NI6qmf52rWwcmX12yNJkqSmNtQ7JIcD\nnwKOKENbVA0tLWycvFvxumeeqW5bJEmS1PTK9cjWVmU6jqphVwOJJEmS6kO5AsnkUhURsXdE/O1A\nH+uKiFER0Tr0pqm71j0MJJIkSaoP5QokU3qpezXwP8ATETGirwNFxJSIuAFYC6yLiOsi4q/K1E4B\nI6eVCCRLllS3IZIkSWp65Qokr+6lbuvCOjq97s2vgRPJbRsBnARcFxH/OKQWqsNu3iGRJElSfShH\nIAng0Igo1Y9kx06vd+r1QBEzgYMKb5cAlwH3kdt5gXdKysRAIkmSpDpRjkCSyHc+/qZE/d6dXu/Z\nx7E6P/p1fErpI+QRvK4rlH1yUC1UVwYSSZIk1YlyPbIVwOcjotjxjun0+ug+jjOusN6UUloEkFJq\nA35Q+Iwjh9hOAUwp0eXHQCJJkqQqK1cgeQCYAXyvcyiJiKMK5YkcKP66j+O0P961plv51eRHuc4u\nS2ub3eTJbGkpMoDZK6/kRZIkSaqScgWSbwGrgDOAeyPiYxHxfuBn5DAyH1gEvDoiju/lOO2d45d2\nLkwptaWUHkwp/bRM7W1uLS2sn7hr8bo//7m6bZEkSVJTG2oguRPYCLwAfIAcPmaSA8rldPQJ+Tnw\nE/Jdku9GxMTuB4qIscCphWM8NMR2qQ+bd7YfiSRJkmpvSIEkpXQMsA1wY0rpavJwvX8iB48obHYt\n8FVySFlJ7uQ+JyJObD9ORIwGvgtMKhTdOJR2qW8tuxtIJEmSVHtDngk9pbQJ2FR4fStwSETsBkwF\nnk8pPV7Y9MWI+ARwBbAPcENEvAgsBqYD2xW2W0meSFEVtNV0A4kkSZJqb8iBpJiU0jNAj2+2KaX/\nKsxX8k3yUMGTgR0K1UF+XOvslNLLlWiXOozc05G2JEmSVHvl6tTebymly4D9ga8BDwMbyP1Q5gGn\npZSuqHabmpJzkUiSJKkOVOQOSV9SSouBzxQW1YKBRJIkSXWg6ndIVCdKBZIlS6rbDkmSJDU1A0mz\n2mkntsSInuUrVsCa7vNSSpIkSZVhIGlWI0awdtudi9f52JYkSZKqxEDSxDZMLjHS1tNPV7chkiRJ\naloGkiaWpkwtXrFoUVXbIUmSpOZlIGlirdNLBJLFi6vbEEmSJDUtA0kTG7u/gUSSJEm1ZSBpYqP2\nNpBIkiSptgwkzWyqgUSSJEm1ZSBpZqUCyZ//DJs3V7ctkiRJakoGkmY2bhyrR0/sWb5lSw4lkiRJ\nUoUZSJrcK9v52JYkSZJqx0DS5DbuZCCRJElS7RhIml2pfiROjihJkqQqMJA0udH7Tite4R0SSZIk\nVYGBpMltc6CPbEmSJKl2DCRNbswMA4kkSZJqx0DS5GJaiUDy9NPQ1lbVtkiSJKn5GEia3cSJrBsx\ntmf5hg3wwgvVb48kSZKaioGk2UWwfLyPbUmSJKk2DCRizfYO/StJkqTaMJCIzbuUCCRPPVXdhkiS\nJKnpGEhEy157Fq944onqNkSSJElNx0Aitj5gevEKA4kkSZIqzEAitjvMQCJJkqTaMJCI8TNLPLK1\nZEke/leSJEmqEAOJiG3Gs2zE5J4VKTn0ryRJkirKQCIAnh/nY1uSJEmqvqYJJBHx1oj4VkTcHhGv\nRESKiJ8M8li7RcTlEfFsRGyIiEURcXFEbFfudlfLykkGEkmSJFVfa60bUEX/DMwEVgPPAPsN5iAR\nMR2YC0wGrgEWAIcDZwOnRMQxKaWXytLiKtqw23R4skiFgUSSJEkV1DR3SIBPAfsA2wB/N4TjXEoO\nI59MKZ2WUvrHlNLrgK8D+wIXDLmlNTBib+cikSRJUvU1TSBJKd2SUlqYUkqDPUbh7shJwCLg292q\nvwCsAd4bEWMH3dAaGTfTR7YkSZJUfU0TSMrkhML6hpRSW+eKlNIqYA4wBjiy2g0bqkmHlwgkTz6Z\nR9uSJEmSKqCZ+pCUw76F9WMl6heS76DsA9zU24Ei4r4SVfu1tbUxe/bsQTVwsDZthEmMZRxrulas\nX8/cX/yCjdtvX9X2NKpVq1YBVP38aeg8d43N89fYPH+Ny3PX2Nra2vreqAq8QzIw2xbWK0vUt5dP\nqEJbymrkKHh6xB5F67Z+9tkqt0aSJEnNwjskNZJSOrRYeUTc19LScsisWbOq3CK4bdu9YPmDPcoP\n3mYbqEF7GlH7L0S1OH8aGs9dY/P8NTbPX+Py3DW2lpb6uDdRH61oHO13QLYtUd9e/nIV2lJ2q3e0\nY7skSZKqy0AyMI8W1vuUqN+7sC7Vx6SubZlaYujfhQur2xBJkiQ1DQPJwNxSWJ8UEV3+7iJiPHAM\nsBa4q9oNK4cRM0rkrEcfLV4uSZIkDZGBpIiIGBkR+xXmHfmLlNITwA3ANODj3XY7HxgLXJFS6jZU\nVWMY/5oSk9c/+ijUySgMkiRJGl6aplN7RJwGnFZ4u1NhfVRE/KjwellK6TOF17sCjwCLyeGjs48B\nc4FvRsSJhe2OIM9R8hjwuUq0vxp2PmxXVjGO8azuWrFuHSxZAlOn1qZhkiRJGraaJpAArwbe361s\nz8ICOXx8hj6klJ6IiMOAfwVOAd4ILAW+AZyfUlpRthZX2ZTdg/nsx2u4t2flggUGEkmSJJVd0zyy\nlVI6L6UUvSzTOm27qHtZt2MtSSmdkVLaOaU0KqU0NaV0TiOHEYDRo+HprUs8trVgQXUbI0mSpKbQ\nNIFE/bNshxKB5JFHqtsQSZIkNQUDibpYt7t3SCRJklQ9BhJ1tZ+BRJIkSdVjIFEX4169F1uK/Wfx\n/POwoqG7yEiSJKkOGUjUxbR9R/MkJWZsd4JESZIklZmBRF3stRcswMe2JEmSVB0GEnUxZQo81mIg\nkSRJUnUYSNTFiBGwvNTQvwYSSZIklZmBRD1s2KNEIHn44eo2RJIkScOegUQ9jDxoRvGKxx+HtWur\n2xhJkiQNawYS9bDLARN5hl17VqQEDz1U/QZJkiRp2DKQqIe99oL5HFS8cv786jZGkiRJw5qBRD1M\nn24gkSRJUnUYSNTDtGnwYBhIJEmSVHkGEvUwahQs26WXQJJSdRskSZKkYctAoqJiv33ZyMieFcuX\nw7PPVr9BkiRJGpYMJCpq2t4jeZj9i1f62JYkSZLKxECiohxpS5IkSdVgIFFRBhJJkiRVg4FERRlI\nJEmSVA0GEhU1fXovQ/8uWAAbNlS3QZIkSRqWDCQqaqutYMweO/ICO/Ss3LwZHnig+o2SJEnSsGMg\nUUn7zQju5+DilffdV93GSJIkaVgykKikGTPgPg4tXmkgkSRJUhkYSFTSfvsZSCRJklRZBhKV1Osd\nkgcesGO7JEmShsxAopL22w8WM5VlTOpZuWmTHdslSZI0ZAYSlTRxIkyeHD62JUmSpIoxkKhX9iOR\nJElSJRlI1CsDiSRJkirJQKJe2bFdkiRJlWQgUa/aO7a/xMSelXZslyRJ0hAZSNSrGTMAgns5rPgG\n8+ZVszmSJEkaZgwk6tWUKTBmDNzN4cU3mDOnug2SJEnSsGIgUa9aWvJdkjkcU3yDuXOr2yBJkiQN\nKwYS9emAA+AujqSN6Fm5eDH8+c/Vb5QkSZKGBQOJ+nTggbCSCTzEq4pv4F0SSZIkDZKBRH068MC8\nnsvRxTewH4kkSZIGyUCiPh1wQF7bj0SSJEnlZiBRn3beGSZO7OUOyf33w9q11W2UJEmShgUDifoU\nkR/beoLpPM/knhts3gz33FP9hkmSJKnhGUjUL7kfSZS+S3LHHdVsjiRJkoYJA4n6pc9+JDffXL3G\nSJIkadgwkKhf2kfaupXji28wZw6sW1e9BkmSJGlYMJCoX9rvkPyBQ1jBhJ4bbNjg8L+SJEkaMAOJ\n+mWbbWD33aGNEdzCCcU3uumm6jZKkiRJDc9Aon5rf2zrJk4svoGBRJIkSQNkIFG/HXxwXt/IXxXf\n4N57YcWK6jVIkiRJDc9Aon475JC8fox9eIZde26QEsyeXdU2SZIkqbEZSNRv7YEEovRjWzfeWK3m\nSJIkaRgwkKjfdt8dJk7Mr0s+tnXddflOiSRJktQPBhL1W0THXZKSd0iefBIee6x6jZIkSVJDM5Bo\nQA49NK+Xsgv38+riG117bfUaJEmSpIZmINGAdPQjgWt5U/GNDCSSJEnqJwOJBqRzIPktbyy+0e23\nwyuvVKdBkiRJamgGEg3InnvmWdsB5nEEy5jUc6NNmxxtS5IkSf1iINGAtLR0TJDYxgiu45TiG/rY\nliRJkvrBQKIB63c/ki1bqtMgSZIkNSwDiQbssMM6Xl/PyWwp9p/R88/DTTdVr1GSJElqSAYSDdhR\nR3W8XsFE7mg5rviGP/lJdRokSZKkhmUg0YBNmwaTJ3e8/8+29xTf8Je/hDVrqtImSZIkNSYDiQYs\noutdkl/wt2xuHd1zwzVr4H//t3oNkyRJUsMxkGhQOgeSlUzgvl3/pviGPrYlSZKkXhhINCidAwnA\n5RtLPLb1+9/DkiWVb5AkSZIakoFEg3LYYdDa2vH+h0tPoW1ikUkS29rg0kur1zBJkiQ1FAOJBmXM\nGJg5s+P9Jkax+Kh3Ft/4u9+FtWur0zBJkiQ1FAOJBu3II7u+v3qXjxXfcMUKuOKKyjdIkiRJDcdA\nokHr3o/kmsdmwCmnFN/44ovz41uSJElSJwYSDdrRR3d9P28ebPz4OcU3XrAArr++8o2SJElSQzGQ\naNCmTYMpUzrer18P87Y5CWbMKL7DV75SlXZJkiSpcRhINGgRcNxxXctuvS3g7LOL7zB7Ntx9d8Xb\nJUmSpMZhINGQHH981/e33gq8732www7Fd/AuiSRJkjppqkASEbtFxOUR8WxEbIiIRRFxcURsN4Bj\nLIqIVGJ5rpLtr0fdA8ncubCpdWv45CeL7/DLX8Jjj1W+YZIkSWoIrX1vMjxExHRgLjAZuAZYABwO\nnA2cEhHHpJRe6ufhVgIXFylfXY62NpK994addoLnClFs7Vq47z448mMfgwsvhDVruu6QElx0EXzv\ne9VvrCRJkupOM90huZQcRj6ZUjotpfSPKaXXAV8H9gUuGMCxXk4pnVdkuagSDa9nESUe25o4Ec48\ns/hOP/5xR4KRJElSU2uKQFK4O3ISsAj4drfqLwBrgPdGxNgqN21YKBpIAD71KWgtchNu40b4xjcq\n3i5JkiTVv6YIJMAJhfUNKaUus/OllFYBc4AxwJHddyxhdES8JyI+GxFnR8QJETGijO1tKN0DyR13\nwKZN5DGB3/3u4jt95zvwyisVb5skSZLqW7P0Idm3sC7Vm3oh+Q7KPsBN/TjeTsAV3cqeiogzUkq3\nFtuhu4i4r0TVfm1tbcyePbs/h6kLKcF22x3NihWjAFi1Ci655I8cfPDLjJk1i8N//OOeO61cyRN/\n//cseec7q9zaylq1ahVAQ50/ZZ67xub5a2yev8bluWtsbW1tfW9UBc1yh2Tbwnplifr28gn9ONYP\ngRPJoWQscCDwXWAa8LuImDn4ZjamCDj88OVdyu68cxIAa6dNY1n3Kd0Ldr/ySka+/HLF2ydJkqT6\n1Sx3SMompXR+t6IHgY9GxGrg08B5wJv7cZxDi5VHxH0tLS2HzJo1a4gtra5ly+D66zvez58/hVmz\nCtO4f+Ur8NrX9thn5OrVHPOb38APflClVlZe+y9EjXb+5LlrdJ6/xub5a1yeu8bW0lIf9ybqoxWV\n134HZNsS9e3lQ/m5/j8K6+N63WqYOumkrv3XH30UFi4svDnmGDj22OI7XnYZzJtX8fZJkiSpPjVL\nIHm0sN6nRP3ehfVQZux7sbBuypG6ttmmZ+f2a6/t9OZrX8vPdhVz5pmwfn3F2iZJkqT61SyB5JbC\n+qSI6PJnjojxwDHAWuCuIXxG+whdTw7hGA3t1FO7vv/Nbzq9OewwOOus4jvOnw/nnluxdkmSJKl+\nNUUgSSk9AdxA7nj+8W7V55PvalyRUloDEBEjI2K/wvwlfxERM4rNVRIR04BLCm9/UtbGN5A3vanr\n+9tu6zay7wUXwKRJxXf+znfg5z+vWNskSZJUn5oikBR8DHgB+GZEXB0RX4qIm4FPkR/V+lynbXcF\nHqHnEMBvB56LiGsj4tKI+HJEXFXYdi/gt0DTzdbebu+9YZ9OD8Vt2gS//nWnDSZOzB3cS/nwh+Hx\nxyvWPkmSJNWfpgkkhbskhwE/Ao4gj4g1HfgGcGRK6aV+HOYW4DeF/d4FnAscD9wBvB84NaW0seyN\nbyBv7ja+2JVXdtvgjDPg9NOL77xqFbz97bBhQ0XaJkmSpPrTNIEEIKW0JKV0Rkpp55TSqJTS1JTS\nOSmlFd22W5RSipTStG7lt6aU3plS2i+lNCGlNDKltENK6fUppf9MKaWq/oHq0Lve1fX99dfDiy92\nKoiA738f9tyz+AH+8Af4zGcq1j5JkiTVl6YKJKq8Aw+EV72q4/2WLfA//9Nto223zf1FRo0qfpBL\nLoFis7tLkiRp2DGQqKwiet4l6fHYFsChh8JFvXS3+chHcq94SZIkDWsGEpXdO9/Z9f2cObBoUZEN\nP/EJeMtbih9k06bcIeWRR8rdPEmSJNURA4nKbo894Oiju5ZdcUWRDSPyTO177FH8QMuXw6xZ8NBD\n5W6iJEmS6oSBRBXx7nd3fX/ZZbk/SQ8TJsCvfgXjxxc/0Asv5FAyf365myhJkqQ6YCBRRbzrXbDV\nVh3vFy+G3/++xMYHHJB7vo8YUbx+2TJ43evgj38sezslSZJUWwYSVcSECXlKkc6+971edjj5ZPj2\nt0vXv/RSDiX33VeW9kmSJKk+GEhUMWee2fX9r34FS5f2ssNZZ+Uhf0tZsSI/vvW735WjeZIkSaoD\nBhJVzFFH9ZyT5Ec/6mOnj38cvvOd0vWrV8Nf/3Uft1skSZLUKAwkqpiInndJvv99aGvrY8ePfjRv\nGFG8fsuWfDflc5+DlMrSVkmSJNWGgUQV9Z73dO3c/tRTcNNN/djxwx/OQ3OVCiUAX/xiHs5rzZoh\nt1OSJEm1YSBRRU2cCKef3rWs309bnXEG/OxnMGpU6W1++lM44ggnUJQkSWpQBhJVXPfHtq6+Gp5/\nvp87v+1tcOONsN12pbd56CE4+GD4zGfyaFySJElqGAYSVdwxx8CMGR3vN2/uR+f2zo49Fu68s/SM\n7gAbNsBXvwrTp8Pll9u3RJIkqUEYSFRxxTq3f+c7sHHjAA6y7745lLzmNb1vt3IlfOhDcOqpzu4u\nSZLUAAwkqor3vhdGj+54v3gx/PjHAzzIjjvC7Nn5YH357W9h5kw47jj4+c9h06YBfpgkSZKqwUCi\nqpg0CT74wa5l//ZvA7xLAjBmTE4yP/hB1+G7Srn99jxl/NSp8OlP5/ebNw/wQyVJklQpBhJVzWc/\n23XArKefzt09BiwiP5b14IPwjnf0b5+lS+FrX8t3TCZNypMrfv3r8Mc/9mNiFEmSJFWKgURVs9tu\nPfuSXHABrF07yANOn56H/b333vx4Vn+98gr85jdw7rl5dK4JE+Doo+EDH8h3Ub7yFfj973N/FEmS\nJFVUa60boObyT/+UJ2HfsCG/f+aZ/OjWF784hIMeeijcfXc+0Je/PPDnwFatyh3m77yza3kE7L8/\nHHlkx7L//tBijpckSSoXv1mpqnbZBT7+8a5lF11UhnkNR42Cf/1XWLQIzj8fdt55iAckDx380EN5\nxviPfAQOPDDPh/L618O//Atcey0sWzb0z5EkSWpiBhJV3Re+kINJu02b4GMfK9PUITvvDJ//fB7G\n6+c/h+OPL8NBO3nllTxR47/9Wx5aeIcdYO+94X3vg0svZezjj8OWLeX9TEmSpGHMR7ZUddtsAxdf\nnCdhbzd7Nnzve3DWWWX6kJEj4fTT8/L00/CrX+Xl9tth/foyfUjB44/n5YoreA2weezYPBtk+3LE\nETBuXHk/U5IkaZgwkKgm3vpWOPlkuP76jrJzz4UTToB99inzh+2+O3ziE3nZsAHmzYObb4Zbbsl9\nT8ocUFrXrIEbbsgLwIgRudN955Cy225l/UxJkqRGZSBRTUTk2doPOghWr85la9fmOQ/vuCPf4KiI\n0aPz0L/HHQfnnZcfr3rySXj4YXjhBXjxxTzD+1135ce+ymHLFvjDH/LyrW/lsilT8qzzhx3WsZ4w\noTyfJ0nScJdSHrZ/y5b6W9dDG3pr24YNeQCgDRsY+9RTtT6TgIFENbTHHvCNb+QpRdrdfTf8aRYE\nWgAAHCdJREFU8z/nwbKqYsSI3Adk77171i1dmoNJ+3LPPbBuXXk+d8mSvPzylx1le+2Vw8nMmXk0\nr1e9CqZNc1QvSapHxb58DnYZ7P518AX5wBdfJNra8o9q1fxSX5aOp6qXbxgGEtXUGWfAr38NV1/d\nUfaVr+QbBqefXrt2AbmD/JvfnBfIve8ffDAPD9weUhYuLN/ntfdF+elPO8q23jo/w7bHHjmctK/b\nX48fX77Pl9R8Uspf8DZv7li3L53f91ZXhm2nPvFE/lJ7003V/UI/lP0FwKRaN0DDgoFENRWRO7Pf\nfTc8+2xH+RlnwIwZcMABtWtbDyNH5okUDz44DwsGedjfefNg7lyYM4ctd93FiPZJVsph3Tr405/y\nUsy4cTB5Muy4Y17aX++wQ/61asIE2HbbvLS/3mYb77qoebW1df1yXIUv23W9bVtbrc8IAHvUugGS\naspAoprbYQe46qo8Qu+mTblszRp405vy9/xdd61t+3q1/fa5oW96EwB33Hgj4x5/nEPXr4c5c/Ky\ndGnlPn/16rw8+eTA9hs/vmtYGT8+h5vBLGPGQGtrTpdqTO2/km/a1PFFtf11sbJSr2tY/5pXXiE2\nb84/HPS2bZ18AZckdTCQqC4cdVTu7/3Rj3aUPf00vOENcNttjdPfO7W2smq//WDWLDjnnPxFb9Gi\nHEwKd1F44IHaP/u6alVeliwpz/Ei8uSUo0f3vnTeprW16zJixNDKigWi/pa1tbHzgw8Smzblu1Eb\nN+YvsZs2dbzesKH4snFjPp8RXZeWlp5l/dmmexu7l3X/db/Yl+6BlG/aNCwePxlb6wZIkgbNQKK6\nceaZcP/98N3vdpQ98ACcdhpcdx1stVXt2jZoEbmvxx57wHvek8tWr84jbt1zT8cy0Dsc9Salji/o\nDWrfWjdAkjQwEfkHqZaW/q0Hsm29r4eyb6cfCNe84Q35h9MaM5CobkTAJZfkJ5x+9auO8ltvzcMB\n/+xn+TpqeOPGdQw93O6ll+Dee3MCe+ihPAzxww93jIksSao/nb8cDmYZyv7d963RF+D5Dz0ELS0c\ndPDB1f1S3n6HWUPSVrF5FgbGQKK60tqaB5n6q7/Kg1m1u+qq/ATUN785TP/9mTQpzxR58skdZSnB\nM8/kXy6eeqrn+plnhsWjNpJqrP3LZqlHIytZV/jsxc88Q2ppYdr06dX7Ql+OQCCWb7NNfjFrVk3b\nocZmIFHdGTMmDwX82tfCggUd5Zdckh/b+spXhmko6S4iT6A4ZQoce2zP+rY2WL48T+j4/PNd18uW\nwcqVHcvLL3e89q6Lmt3IkT37IVXzS3i16vqz7YgRdfEP6lOzZwMwzS+1UlMykKguTZqU+40cfXTX\n4YAvuij/v/PLX66L/4fWVktLHuVr++3zRIr9tXkzvPJK16CyZk3HiF0DWVatgvXrvVMzHHT+str5\nC3v766GUVeE4d//hD6TWVo445pjej+Ov2pJUdwwkqltTp+ZQcvzxsGJFR/m//3v+TvGlLxlKBqW1\nFSZOzEu5bNnS+yhUxcoHMldCf7bprthIZqVGN4tg6Usv0dbayq7TpuUvriNH5k5/ndelRg2LyMfu\n79I+y3D3pXsbu5el1PNLdrGlt/qRI3P46PyFfRg8frJ2+fL8Yg9ntJCkRmMgUV078MA8ce+JJ3YN\nJe13SL74RUNJXRgxIj9rN2ZMrVsyaI8WHhnZ1UdGJEmqqsb+SUxN4eCD4cYbe85FcuGF8OlPO8+Z\nJElSIzOQqCEcckjxUPL1r8MHPtAxw7skSZIai4FEDePQQ+H3v+8ZSq64Ak49NffPliRJUmMxkKih\nHHYY3Hwz7Lhj1/IbbsgjcjX6hOeSJEnNxkCihnPwwTBnDuy5Z9fyRx6Bww+H22+vTbskSZI0cAYS\nNaTp02HuXDjiiK7lL72UR+T64Q9r0y5JkiQNjIFEDWvHHeGWW+Ad7+havmkTfPCD8Pa35wnLJUmS\nVL8MJGpoW28NV14J553Xs+7nP4dXvQquuabqzZIkSVI/GUjU8CLgC1+An/4Uttqqa90LL8Bpp8H7\n3gftEzlLkiSpfhhINGy84x1wzz15zpLurrgC9tkHvv99J1KUJEmqJwYSDSsHHAB33QXnnw+trV3r\nXnoJzjwzj9J1zTWQUm3aKEmSpA4GEg07I0fC5z8P8+blgNLd/Pn5Ma7DD4frrjOYSJIk1ZKBRMPW\nIYfAvffCF7+YO793d++98IY3wJFHwuWXw+rV1W+jJElSszOQaFgbPRr+6Z9gwQJ45ztzB/ju7r4b\nPvQhmDwZ3vhG+OY34bHHvHMiSZJUDQYSNYXdd8/DA8+fD295S/Ft1q2D3/0Ozj4b9t03T7748Y/D\nr3/t3RNJkqRKMZCoqRxwAPziF3DfffCmN/W+7VNPwaWXwt/8DUycmGeA//d/hwce8O6JJElSuRhI\n1JQOOQR+8xtYuBA++9l8B6U3mzbBzTfD//t/cNBBMHVq7puyYkV12itJkjRcGUjU1PbaCy64ABYt\ngocegq9+FV7/ehg1qvf9liyBz30OpkyBT30KFi+uSnMlSZKGHQOJRO7svv/+cO65cMMNeVb3a6+F\nT3wih5ZS1qyBiy/O/U3e/W54/PFx1Wu0JEnSMNDa9yZS8xk7No+49cY35vePPw7XX5/nLbnpptwB\nvrMtW3Kn+SuvPIxDD13OF7+Y77QUG9VLkiRJHbxDIvXDXnt1jLj17LNw4YWw007Ft73vvomcfHKe\nEf4b34Dnn69uWyVJkhqJgUQaoAkT4B/+Ifc7uewymDGj+HZ/+hOccw7suiuccgr85CfwyitVbaok\nSVLdM5BIgzR6NHzwg/Dgg/nOybHHFt9uy5b8uNd73wuTJsHxx8OXvgT33w9tbdVtsyRJUr0xkEhD\n1NICp54Kt90G3/72fRx33Isl+45s3py3++xn89DDO+0Ep52W5zeZOxfWr69u2yVJkmrNTu1SGe2/\n/yrOP/8h9tprFj/9aX5Ma/780tu/+CJcc01eIA83fNhhcPTR8JrXwKtfnUfwGjGiOu2XJEmqNgOJ\nVAG77QZ///d5eeAB+K//yqFjwYLe99u4Md8pmTu3o2zMGDjwQJg5MweUmTNhv/3y7PGSJEmNzkAi\nVdiBB+ZRuS68MHeEv/56+N3v8vDBq1f3vf/atTBvXl46mzQJ9tmn57LXXjnESJIkNQIDiVRF06bB\nWWflZfPmfPdkzpx8R2TOHHj66f4f66WX4M4789LdlCn5s6ZOzevOr6dMyR3yJUmS6oGBRKqR1tY8\nV8nBB+cZ4QGeeSaHk3nz8rDBf/xjDh4DtWRJXm6/vWddBOy8c0dImTo1D028885dl622GtIfT5Ik\nqV8MJFId2W03eNvb8gKQUp6I8U9/6ggoDz8MCxfChg2D+4z2Yz77bNe+Kt1NmAC77NIzqEyeDNtv\nDzvskNfbb+8jYpIkafAMJFIdi8h3L3bdFd74xo7ytrZ8B+Sxx/Ly6KMdrxctyqFjqF5+OS8PP9z3\ntmPGdISTzkFl4sQcbNqX7bbr+n7cOEoOkSxJkpqDgURqQC0tHY9bvf71XevWr4fFi/OyaFFe2l8v\nXpzvjJQjsHS2dm3u/zKQPjCQhzPedtueQaV9GT8+h5bx44sv7XVjxxpsJElqVAYSaZjZaivYd9+8\nFLNhQ+6r0h5WliyBpUtzUFm6NC/PP59nmK+0LVtg+fK8DEVEz+BSLMiMHZvv5owd2/P1ggXjGT16\nC9OmddRtvXUOf5IkqXIMJFKTGT06T7Y4fXrpbbZsgWXLegaV557L5S++mNftrzdtql77i0kJVq3K\ny+AdWrR0zJjSIabz61J1W2/d99Lqv8SSpCbm/wYl9TBiBOy4Y15e/eret20PA50DSvu6vR9K92XF\nirxeu7Y6f56hWLs2L8uWVe4zWlu7BpRRo/IycmTHuvPrYmWVru9e5p0jSVK5NFUgiYjdgH8FTgEm\nAUuBq4HzU0orqn0caTiIgG22ycueew5s340bYeXKnkGlfVm9uuPOR7GlvX7dusr82apl8+Zy3OGp\nrpaW4oGltTUH2tbWrq/7ux7MPiNGwOLFUxgxIvHAA31v29LSc12srBzbFtunpcU+T5LUWdMEkoiY\nDswFJgPXAAuAw4GzgVMi4piUUp8zPpTrOJLyl9gddsjLUGzeXDy8dC9buxbWrMlL99fPPbeK9etb\ngLF/KV+/vix/zGGprS33Rxrs8NPl18sziHUoonLhKKJr8On+uq/6Wuy3aNFUWloSd95Z+c+LKL30\nVT/QpdzHq+UxpUpqmkACXEoOEZ9MKX2rvTAivgZ8CrgA+GgVjyOpTFpbO0bmGqzZs+8DYNasWX8p\n27Il330pFWL683rdur6XtrYh/gWo4aSUg7Ta7VHrBqgfigeV44AciCsVmrp/duf3vdWV+/1w/Kzn\nn6+PWZCbIpAU7mqcBCwCvt2t+gvAmcB7I+LTKaU1lT6OpMYwYkQerWvcuMp9Rkp5UIB16zoCzMaN\nuWzTpuKv+6rvz+uhHqPcQ0dLqn8pFbv2c4eyWg9uosEaWesGAE0SSIATCusbUkpdfotMKa2KiDnk\noHEkcFMVjiNJQP6Fqr0T+7bb1ro1/bdlS/HAsnlzriu27q2uP9v0Vrdo0RK2bAl22mm3Xvdva8vv\ni617qyvHPu2vDXOS1FWzBJL2GRkeK1G/kBwk9qH3IFGu4xAR95Womrl48WL22Wef3nZXnWorPHvT\n4hBEDcdz19iqff7a+yUMTfwlnHQOKR2vo2h5X/XdX0MUKevtff+27ygb6PY990kpkRJEt84KA21T\nX9v3t129bdtXXbG/+8Eeq1TbqnUsNYNHAKbVuBFNE0jaf3dcWaK+vbyvJ9DLdZzetGzcuHHLwoUL\n/zSEY6h29iusF9S0FRoMz11j8/w1Ns9f4/LcNbaZQAUfTO6fZgkkdSelVHQWtvY7J6XqVd88f43L\nc9fYPH+NzfPXuDx3ja2XJ3aqqlmeTWi/c1HqCe328perdBxJkiRJNE8gebSwLtUxY+/CulTfkHIf\nR5IkSRLNE0huKaxPioguf+aIGA8cA6wF7qrScSRJkiTRJIEkpfQEcAN5FIGPd6s+HxgLXNE+d0hE\njIyI/Qrzjgz6OJIkSZJ610yd2j8GzAW+GREnksc5O4I8t8hjwOc6bbtroX4xPYdCG8hxJEmSJPUi\nUhMNQB0RU4B/BU4BJgFLgf8Fzk8prei03TTgKWBxSmnaYI8jSZIkqXdNFUgkSZIk1Zem6EMiSZIk\nqT4ZSCRJkiTVjIFEkiRJUs0YSCRJkiTVjIFEkiRJUs0YSCRJkiTVjIGkTkTEbhFxeUQ8GxEbImJR\nRFwcEdvVum3KCucklVieK7HP0RHx24hYHhHrImJ+RJwTESOq3f7hLiLeGhHfiojbI+KVwnn5SR/7\nDPj8RMT7I+LuiFgdESsjYnZEnFr+P1FzGcj5i4hpvVyLKSJ+1svneP7KLCImRcSHI+J/I+LxwrW0\nMiLuiIgPRUTR7xpef/VhoOfP66++RMSXI+KmiFhSOHfLI+L+iPhCREwqsU/dXXvOQ1IHImI6efb3\nycA1wALgcPLs748Cx6SUXqpdCwU5kAATgIuLVK9OKV3Ubfv/A/wCWA/8N7Ac+GtgX+CqlNLpFW1w\nk4mIPwIzgdXAM8B+wH+llN5TYvsBn5+IuAj4dOH4VwGjgHcAE4H/m1K6pMx/rKYxkPMXHZPX/gm4\nusjhHkwpXVVkP89fBUTER4HvkCcJvgV4GtgReAuwLfk6Oz11+sLh9Vc/Bnr+vP7qS0RsBP4APAy8\nAIwFjgQOA54FjkwpLem0fX1eeykllxovwPVAKpzUzuVfK5T/R63b6JIAFgGL+rntNoV/GDYAh3Uq\n34ocPhPwjlr/mYbTQg7wewMBzCr8Hf+kXOcHOLpQ/jiwXafyacBL5H/cp9X676FRlwGev2mF+h8N\n4Piev8qdu9eRv9C0dCvfifzlNgF/26nc66+OlkGcP6+/OlqArUqUX1D4O7+0U1ndXns+slVjhbsj\nJ5G/7H67W/UXgDXAeyNibJWbpqF5K7AD8LOU0r3thSml9cA/F97+XS0aNlyllG5JKS1MhX8p+zCY\n8/PRwvqClNKKTvssIl+7o4EzBtn8pjfA8zcYnr8KSSndnFL6dUqprVv5c8B/FN7O6lTl9VdHBnH+\nBsPzVyGF66aYnxfWe3cqq9trz0BSeycU1jcU+cdgFTAHGEO+/abaGx0R74mIz0bE2RFxQolnLl9X\nWF9XpO42YC1wdESMrlhL1ZvBnJ/e9vldt21UHbtExFmF6/GsiDiol209f7WxqbDe3KnM669xFDt/\n7bz+6ttfF9bzO5XV7bXXOtQDaMj2LawfK1G/kHwHZR/gpqq0SL3ZCbiiW9lTEXFGSunWTmUlz2tK\naXNEPAW8CtgTeKQiLVVvBnR+CncodyX3FVpa5HgLC+t9KtFYlfT6wvIXETEbeH9K6elOZZ6/GoiI\nVuB9hbedv8x4/TWAXs5fO6+/OhIRnwHGkfv9HAa8lhxGLuy0Wd1ee94hqb1tC+uVJerbyydUoS3q\n3Q+BE8mhZCxwIPBd8nOUv4uImZ229bzWt4GeH89nfVkL/H/AocB2heV4cofcWcBN3R5z9fzVxoXA\nAcBvU0rXdyr3+msMpc6f1199+gz5Uf9zyGHkOuCklNKLnbap22vPQCL1U0rp/MKzts+nlNamlB5M\nKX2UPPjA1sB5tW2h1BxSSi+klD6fUvpDSunlwnIb+W7yPGAv4MO1bWVzi4hPkkflWQC8t8bN0QD1\ndv68/upTSmmnlFKQfzR9C/kux/0RcUhtW9Y/BpLaa0+X25aoby9/uQpt0eC0d/o7rlOZ57W+DfT8\neD4bQEppM/CDwluvxxqJiE8A3yAPQ3pCSml5t028/upYP85fUV5/9aHwo+n/kgPiJOA/O1XX7bVn\nIKm9RwvrUs/ftY+OUKqPiWqv/XZo51vUJc9r4bncPcidBJ+sbNNUwoDOT0ppDfBnYFxE7FzkeF6n\n9aPH9ej5q56IOAf4FvAg+ctssUljvf7qVD/PX2+8/upESmkxOVS+KiK2LxTX7bVnIKm9Wwrrk4rM\nhjoeOIb8vOZd1W6Y+q19BLTO4eLmwvqUItsfRx45bW5KaUMlG6aSBnN+etvnDd22Ue0Uux7B81dx\nEfEPwNeBP5K/zL5QYlOvvzo0gPPXG6+/+rJLYb2lsK7fa2+oE5m4lGVSGydGrPMFmAGMLVI+jTzK\nRAI+26l8G/IvRU6MWJvzNYu+J0Yc0PnBib3q6fwdQrdJ3ArlJxbOQwKO9vxV9Zz9S+Hv915gYh/b\nev3V2TLA8+f1VycL+U7HtkXKW+iYGHFOp/K6vfaicFDVUGFyxLnAZOAa8jCwR5DnKHmMfGG/VLsW\nKiLOI3fwuw1YDKwCpgNvIl/IvwXenFLa2Gmf04CryBfrz4DlwN+Qh927Cnhb8gIsm8Lf92mFtzsB\nJ5N/pbu9ULYspfSZbtsP6PxExFeBc4FnCtuMAt5Ofk73/6aULqnIH64JDOT8FYYW3Zv87+YzhfqD\n6BgL/19SSv9W5DM8fxUQEe8HfkT+FfZbFB+RZ1FK6Ued9vH6qxMDPX9ef/Wj8Ijdl4A7gKfIAWFH\n8qhnewLPASemlB7utE99Xnu1Tncuf0maU8jDyi4FNpK/9F5MpzTqUtPzczzwU/KIIy+TJ4t6Efg9\neZz2KLHfMeSwsgJYBzwAfAoYUes/03BbyKOcpV6WReU4P8AHgHuANeRgeitwaq3//I2+DOT8AR8C\nfgMsAlaTf+17Gvhv4Ng+PsfzV/1zl4DZRfbz+quDZaDnz+uvfhbysMyXkB+zW0bu/7Gy8Hd8HiXu\ndtXjtecdEkmSJEk1Y6d2SZIkSTVjIJEkSZJUMwYSSZIkSTVjIJEkSZJUMwYSSZIkSTVjIJEkSZJU\nMwYSSZIkSTVjIJEkSZJUMwYSSZIkSTVjIJEkSZJUMwYSSZIkSTVjIJEk9SkipkVEiogf1bot5RQR\nsyMi1bodktTMDCSSpEGJiB8VQsq0WrellEZooyQ1u9ZaN0CS1BD+DMwAVta6IWX2PmBMrRshSc3M\nQCJJ6lNKaROwoNbtKLeU0tO1boMkNTsf2ZIk9al7H5JCv4v3F6qfKtSliFjUbb+JEfGliHgkItZF\nxMqIuCkiTiryGR8oHOMDEXFKoX/Hys59PCLitIj4SUQ8FhFrCst9EfHJiGjpdrw+21iqD0lEtETE\nRyPinohYXficeyLi77p/TvtnFY61fUR8LyKWRsSGiHgoIs7o51+zJDUl75BIkgbjfOA0YCbwDeDl\nQnn7moiYCswGpgG3A9cBY4FTgesi4qyU0veLHPutwCnA74D/AKZ2qrsQaAPmkR8j2xZ4XaENrwHe\nO5A29uIK4F3AEuAHQALeDFwKvBZ4d5F9JgBzgI3AVcBo4HTg8ohoSyn9uB+fK0lNJ1JycBFJUu8K\nncKfAn6cUvpAoexH5DsQe6SUFhXZZzZwHPCulNLPOpVPIAeVfYFpKaXnC+UfAH5I/vL/xpTSdUWO\nOT2l9ES3spbCfu8DjkwpzetU1582Hp9Sik5l7wSuBO4HjksprS6UjwVuBQ4F3p1SurLTPu3/M70M\nOCultKVQvj8wH3gspbR/98+XJPnIliSpAiJiJnA88IvOYQQgpfQy8AVgK+Bvi+x+TbEwUtj3iSJl\nbeQ7IAAnD6XdBR8srP+xPYwUPmcN8A+Ftx8ust9a4Nz2MFLY52HyXZMZETGuDG2TpGHHR7YkSZVw\nVGG9bUScV6R+h8J6RpG6u0sdNCImAX8PvBHYk/wIWGe7DqyZRR1CfixsdpG6W4EtwMFF6hamlF4p\nUr6ksN4OWF2kXpKamoFEklQJkwrr1xeWUordNXiu2IaFR73uAfYgh5b/BJYDm8n9N84m99sYqm2B\n5Smljd0rUkqbI2IZMLnIfqX6pmwurEeUoW2SNOwYSCRJldA+X8nZKaVvDnDfUp0bP0wOI+enlM7r\nXBERR5EDSTmsBCZGxMjCcMedP6cV2B4odidEkjQI9iGRJA1We1+JYr/831VYH1vGz9ursP5Fkbrj\nS+zTWxtLuZ/8/8fjitQdVzjWHwZwPElSLwwkkqTBeqmw3r17RUrpXvJQv2+JiA92rweIiAMjotij\nT6UsKqxndTvOwcA/DbSNvbi8sP5SRPxlFvfC6wsLby8bwPEkSb3wkS1J0mDdRO5g/v2I+AWwCng5\npXRJof5dwM3AZRHxSfLcIS8DuwEHAQeQO7+/0M/P+8/C510cEScAC4G9yfOa/BJ4+yDa2ENK6cqI\n+D/A24CHIuJq8mNkp5EfGfvvlNJ/9bPNkqQ+GEgkSYOSUro+Ij4NfAQ4BxgFLAYuKdQ/ExGHAv+X\nPLzvu8mPOz0HPAx8C3hgAJ/3bEQcS75L8VryEL8LgI8BN1IkkPTVxl68kzyi1geBswpljwBfBb7T\n3zZLkvrmxIiSJEmSasY+JJIkSZJqxkAiSZIkqWYMJJIkSZJqxkAiSZIkqWYMJJIkSZJqxkAiSZIk\nqWYMJJIkSZJqxkAiSZIkqWYMJJIkSZJqxkAiSZIkqWYMJJIkSZJqxkAiSZIkqWYMJJIkSZJqxkAi\nSZIkqWYMJJIkSZJqxkAiSZIkqWYMJJIkSZJq5v8HkrA/0H3NseYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116811290>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 277,
       "width": 402
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the minibatch, full training set, and validation costs\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "minibatch_x_inds = np.linspace(0, nb_of_iterations, num=nb_of_iterations*nb_of_batches)\n",
    "iteration_x_inds = np.linspace(1, nb_of_iterations, num=nb_of_iterations)\n",
    "# Plot the cost over the iterations\n",
    "# plt.plot(minibatch_x_inds, minibatch_costs, 'g-', linewidth=0.5, label='cost minibatches')\n",
    "plt.plot(iteration_x_inds, training_costs, 'b-', linewidth=2, label='cost full training set')\n",
    "plt.plot(iteration_x_inds, validation_costs, 'r-', linewidth=3, label='cost validation set')\n",
    "# Add labels to the plot\n",
    "plt.xlabel('iteration')\n",
    "plt.ylabel('$\\\\xi$', fontsize=15)\n",
    "plt.title('Decrease of cost over backprop iteration')\n",
    "plt.legend()\n",
    "x1,x2,y1,y2 = plt.axis()\n",
    "plt.axis((0,nb_of_iterations,0,2.5))\n",
    "plt.grid()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Test kết quả với mô hình học được"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 4 4 4 0 4 5 7 7 0 5 3 7 3 6 4 5 1 1 0 7 3 3 1 0 4 2 3 0 6 2 3 7 9 3 7 9\n",
      " 1 5 4 3 0 1 5 8 8 1 3 4 8 7 5 8 6 9 1 7 8 0 3 3 4 3 2 0 0 2 5 5 0 2 0 8 9\n",
      " 4 5 5 0 3 0 8 7 9 6 1 9 1 9 7 0 5 5 5 7 8 2 9 0 2 1 8 7 7 6 3 4 3 6 7 9 8\n",
      " 3 0 5 7 9 5 9 9 0 7 7 0 6 1 9 3 2 5 9 3 0 3 4 0 0 9 7 7 1 2 9 9 7 4 1 5 3\n",
      " 6 8 5 4 8 6 6 0 8 4 8 7 0 9 3 9 5 4 9 1 5 2 5 1 6 6 7 6 9 6 1 4 2 0 5 1 7\n",
      " 9 5 1 6 9 4 8 9 4 6 2 3 1 9 3 4 5 2 0 0 7 1 6 4 6 0 4 9 0 1 6 3 4 5 4 0 1\n",
      " 9 3 4 9 3 4 4 4 2 8 2 1 8 8 0 3 3 3 4 5 1 5 7 7 3 4 3 1 9 7 1 8 4 8 4 9 2\n",
      " 3 8 4 2 4 5 8 4 8 8 0 6 2 7 1 2 7 5 2 2 4 7 0 9 9 7 2 1 2 4 1 8 0 0 5 0 1\n",
      " 9 8 1 1 5 8 8 7 2 7 9 0 5 3 3 0 6 7 4 3 3 7 7 5 2 7 5 9 1 1 9 6 2 1 4 9 1\n",
      " 8 8 2 5 6 0 7 3 3 4 6 0 7 7 6 2 8 0 5 8 6 8 2 7 7 4 1]\n",
      "[3 4 4 4 0 4 5 7 7 0 5 3 7 3 6 4 8 1 1 0 7 3 3 1 0 4 2 2 0 6 2 3 7 9 3 7 9\n",
      " 1 5 4 3 0 1 5 8 8 1 3 4 8 7 5 5 6 9 1 7 8 0 3 3 4 3 2 0 0 2 5 5 0 2 0 8 9\n",
      " 4 5 5 0 3 0 8 7 9 6 1 9 1 9 7 0 5 5 5 7 8 2 9 0 2 1 8 7 7 6 3 4 3 6 7 9 8\n",
      " 3 0 5 7 9 5 9 3 0 7 7 0 6 1 9 3 2 5 5 3 0 3 4 0 0 9 7 7 1 2 9 9 7 4 1 9 3\n",
      " 6 8 5 4 8 6 6 0 8 4 1 7 0 9 3 9 5 4 9 1 5 2 5 1 6 6 7 6 9 6 1 4 2 0 5 1 7\n",
      " 9 9 1 6 9 4 8 9 4 6 2 3 1 9 3 1 5 2 0 0 7 1 6 4 6 0 4 9 0 1 6 3 4 5 4 0 1\n",
      " 9 3 4 9 3 4 4 4 2 8 2 1 8 8 0 3 3 3 4 5 1 5 7 7 3 4 3 1 9 7 1 8 4 8 4 9 2\n",
      " 3 8 4 2 4 5 8 4 9 8 0 6 2 7 1 2 1 5 2 2 4 7 0 9 9 7 2 1 2 4 1 8 4 0 5 0 1\n",
      " 9 8 1 1 5 5 1 7 2 7 9 0 5 3 3 0 6 7 4 3 3 7 7 8 2 4 5 9 1 1 9 6 2 1 4 9 1\n",
      " 8 8 2 5 6 0 7 3 3 4 6 0 7 7 6 2 8 0 5 8 6 8 2 7 7 4 1]\n",
      "The accuracy on the test set is 0.96\n"
     ]
    }
   ],
   "source": [
    "# Get results of test data\n",
    "y_true = np.argmax(T_test, axis=1)  # Get the target outputs\n",
    "activations = forward_step(X_test, layers)  # Get activation of test samples\n",
    "y_pred = np.argmax(activations[-1], axis=1)  # Get the predictions made by the network\n",
    "print y_true\n",
    "print y_pred\n",
    "test_accuracy = metrics.accuracy_score(y_true, y_pred)  # Test set accuracy\n",
    "print('The accuracy on the test set is {:.2f}'.format(test_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3 4 4 4 0]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0oAAADjCAYAAAC7IeRWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAewgAAHsIBbtB1PgAAB9hJREFUeJzt2TFSk3sUxuFwh958rkCzAkdYAEXS25iWVNJilZSWSWlJ\nJbbS0MsSyAwbEFyBiSvILSiu81bMPZ/5gDxPf/4cnCOT32Rvs9n0AAAA+M8/XS8AAADw2AglAACA\nIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAA\nIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIA\nAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQA\nAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAgl\nAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhC\nCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAsN/1Aluy\n6XqBxWJRmp/NZuUdXr9+XZpfLpflHZqmKb/Rgr0t/ZzO765qvV6X35hMJqX5y8vL8g6PxDbuzs31\n3Nwf/K17IHfXqp25u4uLi9L8dDot7zAajUrz8/m8vMNz/mznGyUAAIAglAAAAIJQAgAACEIJAAAg\nCCUAAIAglAAAAIJQAgAACEIJAAAgCCUAAIAglAAAAIJQAgAACEIJAAAgCCUAAIAglAAAAMJ+1ws8\nBbPZrPzGt2/fSvNnZ2flHU5OTkrzy+WyvMNwOCy/wfacn5+X33jz5k19EXaGm6ML7o7/Yzqdlubv\n7u7KO6xWq9L8y5cvyztUP+O+f/++vMPf4hslAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACC\nUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIOxtNpuu\nd9iG0i95e3tbXqBpmtL8wcFBeYeqNv4dHom9Lf2czv9zrdfr0vzR0VF5h9PT0853qHr16lUbz2zj\n7txcz839wd+6B3J393bp7pbLZXmBw8PD0vyPHz/KOwwGg9L8aDQq71D9jDqfz8s79P7S3flGCQAA\nIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIA\nAAhCCQAAIAglAACAIJQAAADCftcLPAWDwaD8xu3tbWn+7u6uvMNwOCzNr1ar8g5N05Tf4OHOz89L\n8z9//izvMJlMSvOnp6flHfr9fmn+06dP5R12hZu75+a2y93dc3cP18Znmrdv35bm2/h8WXVwcND1\nCo+ab5QAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhC\nCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAsN/1ArtiMBiU5n/9+lXeYTgcdjrf6/V6V1dX\npfmmaco7PBWXl5flNz5+/FiaPz4+Lu9Q9fnz5/IbX758aWGT58/N3XNz2+Xu7rm77VqtVuU3RqNR\nC5t0q41/h+f82cw3SgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAA\nAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQ9rtegIdpmqb8xtXVVWn+5OSkvMNisSjN\nz+fz8g5PRb/fL7/x4sWL0vzXr1/LO9zc3JTfqHr37l3XKzwJbq49bu7h3F173N3DtfG5arlctrBJ\nzWq1Ks1fX1+XdxiPx+U3HivfKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoA\nAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABhv+sFdsVsNivND4fD\n8g6r1ao0//379/IO4/G4/MauODo6Kr+xXq9L8zc3N+Udqr/H8fFxeYd+v19+Yxe4uXtubrvc3T13\nt12DwaD8xvX1dWn+4uKivEMbb1RNp9OuV/hrfKMEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIA\nAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAGG/6wV2RdM0\npfkPHz60tMn/Nx6Py2+cnZ21sAnb0u/3y2/8/v27ND+ZTMo78HS4Obrg7nbPYDAov7FYLErz0+m0\nvMPh4WFpfrlclnd4znyjBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAASh\nBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAIS9zWbT9Q4AAACPim+UAAAA\nglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAA\ngCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkA\nACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgPAvsYocdeThKSwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x117baaad0>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 113,
       "width": 421
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "real_num = np.argmax(T_test[0:5], axis=1)\n",
    "print real_num\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "fig = plt.figure(figsize=(10, 1), dpi=100)\n",
    "for i in range(len(real_num)):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    ax.matshow(digits.images[real_num[i]], cmap='binary') \n",
    "    ax.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0oAAADjCAYAAAC7IeRWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAewgAAHsIBbtB1PgAAB9hJREFUeJzt2TFSk3sUxuFwh958rkCzAkdYAEXS25iWVNJilZSWSWlJ\nJbbS0MsSyAwbEFyBiSvILSiu81bMPZ/5gDxPf/4cnCOT32Rvs9n0AAAA+M8/XS8AAADw2AglAACA\nIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAA\nIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIA\nAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQA\nAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAgl\nAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhC\nCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAsN/1Aluy\n6XqBxWJRmp/NZuUdXr9+XZpfLpflHZqmKb/Rgr0t/ZzO765qvV6X35hMJqX5y8vL8g6PxDbuzs31\n3Nwf/K17IHfXqp25u4uLi9L8dDot7zAajUrz8/m8vMNz/mznGyUAAIAglAAAAIJQAgAACEIJAAAg\nCCUAAIAglAAAAIJQAgAACEIJAAAgCCUAAIAglAAAAIJQAgAACEIJAAAgCCUAAIAglAAAAMJ+1ws8\nBbPZrPzGt2/fSvNnZ2flHU5OTkrzy+WyvMNwOCy/wfacn5+X33jz5k19EXaGm6ML7o7/Yzqdlubv\n7u7KO6xWq9L8y5cvyztUP+O+f/++vMPf4hslAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACC\nUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIOxtNpuu\nd9iG0i95e3tbXqBpmtL8wcFBeYeqNv4dHom9Lf2czv9zrdfr0vzR0VF5h9PT0853qHr16lUbz2zj\n7txcz839wd+6B3J393bp7pbLZXmBw8PD0vyPHz/KOwwGg9L8aDQq71D9jDqfz8s79P7S3flGCQAA\nIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIA\nAAhCCQAAIAglAACAIJQAAADCftcLPAWDwaD8xu3tbWn+7u6uvMNwOCzNr1ar8g5N05Tf4OHOz89L\n8z9//izvMJlMSvOnp6flHfr9fmn+06dP5R12hZu75+a2y93dc3cP18Znmrdv35bm2/h8WXVwcND1\nCo+ab5QAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAIJQAAACCUAIAAAhC\nCQAAIAglAACAIJQAAACCUAIAAAhCCQAAIAglAACAsN/1ArtiMBiU5n/9+lXeYTgcdjrf6/V6V1dX\npfmmaco7PBWXl5flNz5+/FiaPz4+Lu9Q9fnz5/IbX758aWGT58/N3XNz2+Xu7rm77VqtVuU3RqNR\nC5t0q41/h+f82cw3SgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAA\nAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQ9rtegIdpmqb8xtXVVWn+5OSkvMNisSjN\nz+fz8g5PRb/fL7/x4sWL0vzXr1/LO9zc3JTfqHr37l3XKzwJbq49bu7h3F173N3DtfG5arlctrBJ\nzWq1Ks1fX1+XdxiPx+U3HivfKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoA\nAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABhv+sFdsVsNivND4fD\n8g6r1ao0//379/IO4/G4/MauODo6Kr+xXq9L8zc3N+Udqr/H8fFxeYd+v19+Yxe4uXtubrvc3T13\nt12DwaD8xvX1dWn+4uKivEMbb1RNp9OuV/hrfKMEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIA\nAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAEEoAQAABKEEAAAQhBIAAEAQSgAAAGG/6wV2RdM0\npfkPHz60tMn/Nx6Py2+cnZ21sAnb0u/3y2/8/v27ND+ZTMo78HS4Obrg7nbPYDAov7FYLErz0+m0\nvMPh4WFpfrlclnd4znyjBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAASh\nBAAAEIQSAABAEEoAAABBKAEAAAShBAAAEIQSAABAEEoAAABBKAEAAIS9zWbT9Q4AAACPim+UAAAA\nglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAA\ngCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgCCUAAAAglACAAAIQgkA\nACAIJQAAgCCUAAAAglACAAAIQgkAACAIJQAAgPAvsYocdeThKSwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112e1d7d0>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 113,
       "width": 421
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "activations = forward_step(X_test[0:5], layers)  # Get activation of test samples\n",
    "pred_num = np.argmax(activations[-1], axis=1)  # Get the predictions made by the network\n",
    "print \n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "fig = plt.figure(figsize=(10, 1), dpi=100)\n",
    "for i in range(len(pred_num)):\n",
    "    ax = fig.add_subplot(1,10,i+1)\n",
    "    ax.matshow(digits.images[pred_num[i]], cmap='binary') \n",
    "    ax.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}