6.diy_neural_network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DIY neural network\n",
    "\n",
    "Neural networks are the primary method that we will use for most of the rest of these guides. It is important to understand how neural networks actually work, rather than seeing them like a black box. \n",
    "\n",
    "In this guide, we will derive and implement a simple 2-layer neural network with a sigmoid activation function. We will do so by starting with the linear regression example from the previous guide, and building on top of it. \n",
    "\n",
    "If you want more details about their history, design, and use cases, take a look at the [ml4a chapter on neural networks](http://ml4a.github.io/ml4a/neural_networks/), Michael Nielsen's [Neural Networks and Deep Learning](http://neuralnetworksanddeeplearning.com/), Goodfellow, Bengio, and Courville's [Deep Learning](http://www.deeplearningbook.org/) book, Yoav Goldberg's \"[A Primer on Neural Network Models for Natural Language Processing](http://arxiv.org/abs/1510.00726)\", or [Francis Tseng's AI notes](http://frnsys.com/ai_notes/machine_learning/neural_nets.html) on neural networks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is a neural network?\n",
    "\n",
    "A neural network is pretty simple - it's just a _network_ of _neurons_! That is, it is some neurons - often called \"units\", the terminology I'll try to stick to - that are connected in some way. Each unit is a little function: it takes some input from the (incoming) units that are connected to it and passes some output to the (outgoing) units it's connected to.\n",
    "\n",
    "Generally these units are organized into layers, and each layer connects to the next. The most basic architecture (i.e. how the units are connected) is a _dense_ layer, in which every input is connected to every unit (these are also called _affine_ or _fully-connected_ layers). The input to a unit is a tensor consisting of the outputs of each incoming unit or it may be the input you give to the network (if they are in the first layer).\n",
    "\n",
    "It's a vector if each incoming unit outputs a scalar, it's a matrix if each incoming unit outputs a vector, and so on.\n",
    "\n",
    "![A feedforward neural network](../assets/feedforward.jpg)\n",
    "[source](http://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Architecture/feedforward.html)\n",
    "\n",
    "To keep things simple, we'll just consider the case where the input is a vector (i.e. each unit outputs a scalar).\n",
    "\n",
    "Each unit has a vector of weights - these are the parameters that the network learns.\n",
    "\n",
    "The inner operations of the basic unit is straightforward. We collapse the weight vector $w$ and input vector $v$ into a scalar by taking their dot product. Often a _bias_ term $b$ is added to this dot product; this bias is also learned. then we pass this dot product through an _activation function_ $f$, which also returns a scalar. Activation functions are typically nonlinear so that neural networks can learn nonlinear functions. I'll mention a few common activation functions in a bit, but for now let's see what a basic unit is doing:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review linear regression\n",
    "\n",
    "Let's quickly recall our linear regression example from the last section.\n",
    "\n",
    "Suppose we have a set of points $(X, Y)$, and we'd like to find a line, $y=mx+b$ which best fits that dataset. We would like to find the parameters, $m$ and $b$ which minimize the sum squared error, $J = \\frac{1}{2} \\sum_i{(y_i - (mx_i + b))^2}$. Although there exists a quicker analytical way of solving a linear regression, we instead introduced gradient descent as a way of solving a linear regression.  \n",
    "\n",
    "With gradient descent, we calculate the gradient of the loss function $J$ with respect to the parameters $m$ and $b$. \n",
    "\n",
    "$$\n",
    "\\nabla J = \\left[ \\frac{\\partial J}{\\partial m}, \\frac{\\partial J}{\\partial b} \\right] = \\left[ -\\sum_i{x_i \\cdot (y_i - (mx_i + b))}, -\\sum_i{(y_i - (mx_i + b))} \\right]\n",
    "$$\n",
    "\n",
    "And then we picked a random initial $m$ and $b$, and then defined an update rule where we calculate the gradient and then adjust the parameters $m$ and $b$ with the following formula:\n",
    "\n",
    "$$ m := m - \\alpha \\cdot \\frac{\\partial J}{\\partial m} $$\n",
    "$$ b := b - \\alpha \\cdot \\frac{\\partial J}{\\partial b} $$\n",
    "\n",
    "We do this some number of times, and $m$ and $b$ converge to the optimal solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural networks\n",
    "\n",
    "We are now going to innovate our linear regression in a few ways and turn it into a multiple-layer neural network!\n",
    "\n",
    "First, let's start by adding another input variable, so that now our input is two dimensions. We'll go back to the Iris dataset, and pose the following problem: given the sepal length and sepal width (first two features), try to predict the petal length (third feature).\n",
    "\n",
    "Let's load the data first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "data, labels = iris.data[:,0:2], iris.data[:,2]\n",
    "\n",
    "num_samples = len(labels)  # size of our dataset\n",
    "\n",
    "# shuffle the dataset\n",
    "shuffle_order = np.random.permutation(num_samples)\n",
    "data = data[shuffle_order, :]\n",
    "labels = labels[shuffle_order]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like the 1-dimensional problem previously, we can still do linear regression, except now we have two variables and therefore two weights as well. Let's denote the input variables as $x_1$ and $x_2$ and instead of using $m$ as the coefficient variable, let's use $w_1$ and $w_2$. So for linear regression, we would have the following function:\n",
    "\n",
    "$$\n",
    "f(X) = w_1 x_1 + w_2 x_2 + b \n",
    "$$\n",
    "\n",
    "Rather than having two parameters as in the 1-d example, now there are three parameters: $w1$, $w2$, and $b$.\n",
    "\n",
    "We can visualize this graphically using a neuron diagram where $y=f(x)$:\n",
    "\n",
    "![2 input neuron](https://ml4a.github.io/images/figures/neuron2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, suppose set $w = [0.2, 0.6]$ and $b = -0.3$. Let's calculate the resulting $y$. We can program this as a function called `weighted_sum`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "for x=[4.70, 3.20], predicted = 2.56, actual = 1.60\n"
     ]
    }
   ],
   "source": [
    "def weighted_sum(x, w, b):\n",
    "    return b + np.dot(w, x)\n",
    "\n",
    "# set our paramters\n",
    "w = [0.2, 0.6]\n",
    "b = -0.3\n",
    "\n",
    "# for example, let's use the first data point\n",
    "X, y = data, labels\n",
    "\n",
    "pred_y = [weighted_sum(x, w, b) for x in X]\n",
    "\n",
    "# let's print out the first prediction\n",
    "print(\"for x=[%0.2f, %0.2f], predicted = %0.2f, actual = %0.2f\" % (X[0][0], X[0][1], pred_y[0], y[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that we can evaluate the quality of our predictions using the sum-squared error function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "313.0096\n"
     ]
    }
   ],
   "source": [
    "# sum squared error\n",
    "def cost(y_pred, y_actual):\n",
    "    return 0.5 * np.sum((y_actual-y_pred)**2)\n",
    "\n",
    "error = cost(pred_y, y)\n",
    "print(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a dataset, we can use this setup to do linear regression, just as we did in the case of the one-dimensional dataset in th previous section. In this case, we are no longer able to visualize the loss surface, because our function has 3 parameters ($w_1$, $w_2$, and $b$) and therefore requires four dimensions to plot the loss surface with repsect to the parameters). But the regression would work just the same way; we can optimize our three parameters using calculus and finding the partial derivative of the cost function with respect to all the parameters.\n",
    "\n",
    "Since we now have two $w$ and two $x$, we will use the subscript to differentiate them. So $x_1$ is the first element in the input variable $X$. We will use the superscript to denote the $i-th$ point in the dataset. So $(X^1, y^1)$ is the first point in the dataset and $(X^2, y^2)$ is the second, and for example, $x_2^1$ is the second element in the first data point (be careful not to confuse the meaning of the subscript and the superscript).\n",
    "\n",
    "We find that our partial derivatives are:\n",
    "\n",
    "$$ \\frac{\\partial J}{\\partial w_1} = - \\sum{x_1^i \\cdot (y^i - (w_1 x_1^i + w_2 x_2^i+ b))} $$\n",
    "$$ \\frac{\\partial J}{\\partial w_2} = - \\sum{x_2^i \\cdot (y^i - (w_1 x_1^i + w_2 x_2^i+ b))} $$\n",
    "$$ \\frac{\\partial J}{\\partial b} = - \\sum{y^i - (w_1 x_1^i + w_2 x_2^i+ b)} $$\n",
    "\n",
    "Again, we use the following update rule, where we calculate the gradient and then adjust the parameters $w_1$, $w_2$, and $b$:\n",
    "\n",
    "$$ w_1 := w_1 - \\alpha \\cdot \\frac{\\partial J}{\\partial w_i} $$\n",
    "$$ w_2 := w_2 - \\alpha \\cdot \\frac{\\partial J}{\\partial w_2} $$\n",
    "$$ b := b - \\alpha \\cdot \\frac{\\partial J}{\\partial b} $$\n",
    "\n",
    "If we implement this in code, as we did in the previous guide for the 1-D example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial parameters: w1=0.080, w2=0.220, b=0.259\n",
      "initial cost = 5.384\n",
      "final parameters: w1=1.902, w2=-0.923, b=-0.221\n",
      "final cost = 0.665\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAADgCAYAAAAKV5uJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8XXWd//H3597cJG2Wbkm6t6ELFAqUpeyrsigOgjAKCIooIzKK47iMD3TGnwzjzDg6zqiDyoAygCIKKAiiqIhAWVpooQVKF7rShSZp2jRLs977+f1xTpLbNEmTNjfnJnk9H4/7OOd+z/a599tze985yzV3FwAAAAAA2SIWdQEAAAAAAKQjqAIAAAAAsgpBFQAAAACQVQiqAAAAAICsQlAFAAAAAGQVgioAAAAAIKsQVAEA6AczO9fMtg7Qum43s68NxLoOoYaVZnZulDUAANAVQRUAMGSZ2dVmttTM6s3sHTP7vZmdeYjr3GRm5w9Ujb1x9xvd/V/C7Q5YAO6Jmd1tZt/oUsN8d386k9sFAKC/CKoAgCHJzL4g6buS/k3SREkzJP1Q0qVR1hUVM8uJugYAAAYKQRUAMOSY2RhJt0r6jLv/2t0b3L3V3R9z938I58kzs++a2fbw8V0zywunlZjZb82sxsx2mdkiM4uZ2U8VBN7HwqO0X+6lhq+a2c7wCOw1YdtJZlZhZvG0+S43sxU9rONuM/uGmRVI+r2kKeF2681sSljTzWa23syqzewBMxsfLltuZm5m15vZ25KeCtsfNLMdZrbHzJ41s/lh+w2SrpH05XD9j4XtHUeQD/CenWtmW83si2ZWGR7B/vhBdyIAAL0gqAIAhqLTJOVLeriXef5R0qmSjpO0QNLJkv4pnPZFSVsllSo4GvtVSe7uH5X0tqT3u3uhu3+rh3VPklQiaaqkj0m6w8yOcPeXJVVLujBt3o9Kure3F+PuDZIukrQ93G6hu2+X9FlJH5B0jqQpknZL+kGXxc+RdKSk94TPfy9prqQySa9Iui/cxh3h+LfC9b+/m1J6e8/aX/eY8HVfL+kHZjaut9cGAMDBIKgCAIaiCZJ2untbL/NcI+lWd6909ypJ/6wgNEpSq6TJkmaGR2IXubv3s4avuXuzuz8j6XFJV4Tt90j6iCSFRz/fI+nn/Vx3uxsl/aO7b3X3Zkm3SPpgl9N8bwmPKDdKkrvf5e51afMvCI9A90Vv75kUvG+3hu/Z7yTVSzriIF8bAAA9IqgCAIaiakklB7guc4qkzWnPN4dtkvRtSesk/dHMNpjZzf3c/u7wKGh36/6ZpPeHp/NeIWmRu7/Tz/W3mynp4fAU5RpJqyQlFRwFbrelfcTM4mb2zfBU4VpJm8JJJX3cXm/vmSRVd/njwF5JhX1cNwAAfUZQBQAMRS9KalZwWmxPtisIeu1mhG0Kjzh+0d1nSbpE0hfM7Lxwvr4cWR0XBtHu1r0trO9yBUcjf9qH9fW03S2SLnL3sWmP/HAb3S13tYKbSZ2v4BTd8rDdetlGuh7fMwAABhNBFQAw5Lj7Hkn/T8E1kh8ws9FmljCzi8ys/brS+yX9k5mVmllJOP/PJMnMLjazOWZmkvYoOEqZCperkDSrD2X8s5nlmtlZki6W9GDatHslfVnSMZJ+3ceXVSFpQpfTdG+X9K9mNjOsu9TMerurcZGCAF8tabSCOyJ33UZvr63H9wwAgMFEUAUADEnu/h1JX1Bws58qBUcfb5L0SDjLNyQtlfSapNcV3Fio/TdE50p6UsE1li9K+qG7/yWc9u8KwlqNmX2ph83vUHBjo+0KblB0o7uvTpv+sMLTdt19bx9fz2oFQXFDuO0pkr4n6VEFpyjXSVos6ZReVnOvgtN1t0l6M5w/3U8kHRWu/5GuC6v39wwAgEFj/b93BAAAOBAzWy/pU+7+ZNS1AAAw1HBEFQCAAWZmf63getCnoq4FAIChqLe7JQIAgH4ys6clHSXpo+6eOsDsAACgG5z6CwAAAADIKpz6CwAAAADIKgRVAAAAAEBWyaprVEtKSry8vDzqMgAAAAAAA2zZsmU73b20L/NmVVAtLy/X0qVLoy4DAAAAADDAzGxzX+fl1F8AAAAAQFYhqAIAAAAAsgpBFQAAAACQVQiqAAAAAICsQlDto4/8eIkWfuNJVdQ2RV0KAAAAAAxrBNU+2lnfrJ31zaqqa466FAAAAAAY1giqfVRWnC9JqqzjiCoAAAAAZFJGf0fVzDZJqpOUlNTm7gszub1MKi3MkySOqAIAAABAhmU0qIbe5e47B2E7GVVWTFAFAAAAgMHAqb99NLEoCKrbajj1FwAAAAAyKdNB1SX90cyWmdkN3c1gZjeY2VIzW1pVVZXhcg7e7LJCSdL6yvqIKwEAAACA4S3TQfVMdz9B0kWSPmNmZ3edwd3vcPeF7r6wtLQ0w+UcvLllRZKkNRV1cveIqwEAAACA4SujQdXdt4XDSkkPSzo5k9vLpInFeSorytOexlatreCoKgAAAABkSsaCqpkVmFlR+7ikCyW9kantZZqZ6cy5JZKkx1Zsj7gaAAAAABi+MnlEdaKk58xshaSXJD3u7k9kcHsZd/XJMyRJ9y3ZrIbmtoirAQAAAIDhKWNB1d03uPuC8DHf3f81U9saLCfOHKfjZ4zV7r2t+sbjb0ZdDgAAAAAMS/w8TT+Ymf798mOUG4/p/pe26LevcQowAAAAAAw0gmo/zZtUrK++b54k6YsPrNCrb++OuCIAAAAAGF4IqgfhY6eX68MnT1dzW0qfvHeZtu7eG3VJAAAAADBsEFQPgpnp1kuP1hlzJmhnfbOuv3upaptaoy4LAAAAAIYFgupBSsRj+uE1J2pOWaHWVNTphnuXqrktGXVZAAAAADDkEVQPwZhRCf3fdSeprChPizfs0hd+uULJlEddFgAAAAAMaQTVQzR9/Gjd/fGTVZSXo8dff0e3PrZS7oRVAAAAADhYBNUBcNSUYt1x7ULlxmO658XN+uHT66MuCQAAAACGLILqADlt9gT995XHyUz69h/W6MGlW6IuCQAAAACGJILqAPqrYyfrlvfPlyTd/OvX9dTqiogrAgAAAIChh6A6wD52erk+867ZSqZcf/uzV7RkQ3XUJQEAAADAkEJQzYAvXXiErjppuprbUrr+nqVasaUm6pIAAAAAYMggqGaAmelfLztG718wRfXNbfrY/72kNTvqoi4LAAAAAIYEgmqGxGOm/7pigc4/skw1e1t1zY+XaOPOhqjLAgAAAICsR1DNoEQ8ptuuPkFnzJmgnfXNuubOxdpW0xh1WQAAAACQ1QiqGZafiOuOjy7UiTPHafueJl1z52JV1jVFXRYAAAAAZC2C6iAoyMvRXdedpPlTirWpeq8++uOXVF3fHHVZAAAAAJCVCKqDZMyohO79xMmaU1aoNRV1uvrOJYRVAAAAAOgGQXUQTSjM088/eQphFQAAAAB6QVAdZGVF+YRVAAAAAOgFQTUChFUAAAAA6BlBNSKEVQAAAADoHkE1Qt2F1ao6wioAAACAkY2gGrGuYfWK/31R22oaoy4LAAAAACKT8aBqZnEze9XMfpvpbQ1VZUX5+sUNp+qoycXauLNBV9z+ojbtbIi6LAAAAACIxGAcUf2cpFWDsJ0hraQwT/ffcKqOnzFW22oa9aH/fVFrdtRFXRYAAAAADLqMBlUzmybpryT9OJPbGS7GjEroZ9efotNnT1BVXbOuvONFvba1JuqyAAAAAGBQZfqI6nclfVlSqqcZzOwGM1tqZkurqqoyXE72K8jL0V3XnaTz5pWpZm+rrr5ziV7auCvqsgAAAABg0GQsqJrZxZIq3X1Zb/O5+x3uvtDdF5aWlmaqnCElPxHX7R89URcfO1n1zW269q4lemp1RdRlAQAAAMCgyOQR1TMkXWJmmyT9QtK7zexnGdzesJKIx/S9q47XlQunq6k1pU/eu0wPvLwl6rIAAAAAIOMyFlTd/SvuPs3dyyVdJekpd/9IprY3HMVjpm/+9TG66V1zlEy5vvyr13TbU2/J3aMuDQAAAAAyht9RzXJmpi+95wjdeul8mUn/+ce1+vqjK5VMEVYBAAAADE+DElTd/Wl3v3gwtjVcXXtauX5w9QnKjcd074ubddPPX1FTazLqsgAAAABgwHFEdQh53zGTde/1J6soL0e/f2OHPnbXS9qztzXqsgAAAABgQBFUh5hTZ03QAzeeponFeVqycZcu+9Hz2lzdEHVZAAAAADBgCKpD0JGTi/XrT5+heZOKtKGqQZf98AUt3cRvrQIAAAAYHgiqQ9TUsaP04I2n6dwjSrWroUVX37lEv1m+LeqyAAAAAOCQEVSHsKL8hH587UJde9pMtSRT+twvluu7T67l52sAAAAADGkE1SEuJx7TrZcera+//yjFTPruk2/p879cruY27ggMAAAAYGgiqA4THz/jMN157UKNzo3rkeXbdfWdS1RZ2xR1WQAAAADQbwTVYeS8IyfqoRtP15Qx+Vq2ebfef9tzWr6lJuqyAAAAAKBfCKrDzFFTivXoZ8/UyYeNV0Vts664/UU9sHRL1GUBAAAAQJ8RVIehksI83fc3p3TcZOnLD72mWx5dqdZkKurSAAAAAOCACKrDVCK8ydI3Lz9GufGY7n5hkz7y4yWqrm+OujQAAAAA6BVBdZi76uQZuv+GU1VWlKclG3fpktue1wquWwUAAACQxQiqI8CJM8fpsc+eqeNnjNW2mkZ98PYXdM8Lm/i9VQAAAABZiaA6QkwsztcvbzhN151ertak6+uPrtRn739V9c1tUZcGAAAAAPsgqI4guTkx3XLJfN129fEqyI3rt6+9o0v+5zmt3lEbdWkAAAAA0IGgOgJdfOwUPfbZMzVvUpE27GzQB37wvB5atjXqsgAAAABAEkF1xJpVWqiHP32GPnTiNDW1pvSlB1foHx5coQZOBQYAAAAQMYLqCDYqN65vf2iBvvXBY5WXE9ODy7bq4v95Tq9v3RN1aQAAAABGMIIqdMXC6R2nAm/c2aDLf/S87nh2vVIp7goMAAAAYPARVCFJOnxikR75zBkddwX+t9+t1rV3vaTK2qaoSwMAAAAwwhBU0SE/Edctl8zXXdct1PiCXD23bqfe+71FevLNiqhLAwAAADCCEFSxn3fPm6gnPneWzppbol0NLfqbe5fq5l+9xm+uAgAAABgUfQqqZvbTvrRh+Cgrztc9Hz9Z//i+I5Ubj+kXL2/Re7/7rF5cXx11aQAAAACGub4eUZ2f/sTM4pJOHPhykE1iMdMnz56lxz57po6eWqytuxv14TsX65ZHV6qxJRl1eQAAAACGqV6Dqpl9xczqJB1rZrXho05SpaTfHGDZfDN7ycxWmNlKM/vnAawbg+iISUV6+NNn6O/Pn6ucmOnuFzbpr76/SK+8vTvq0gAAAAAMQ+Z+4J8gMbN/d/ev9GvFZiapwN3rzSwh6TlJn3P3xT0ts3DhQl+6dGl/NoNB9vrWPfrig8u1tqJeMZM+dc5sfe68ucpPxKMuDQAAAEAWM7Nl7r6wL/P29dTf35pZQbjyj5jZf5nZzN4W8EB9+DQRPvhhziHumGlj9OhNZ+pT58ySS/rR0+v1vu8t0pINXLsKAAAAYGD0Naj+SNJeM1sg6YuS1ku690ALmVnczJYrOFX4T+6+pJt5bjCzpWa2tKqqqh+lIyr5ibi+ctGReujG0zWnrFAbdjboyjsW66sPv67aptaoywMAAAAwxPU1qLZ5cI7wpZJuc/cfSCo60ELunnT34yRNk3SymR3dzTx3uPtCd19YWlran9oRsRNnjtPjf3emPnfeXCXipp8veVvnf+cZPfHGjqhLAwAAADCE9TWo1pnZVyR9VNLjZhZTcCpvn7h7jaS/SHpv/0tENsvLievzFxyux//uLJ0wY6wq65p148+W6cafLlNFbVPU5QEAAAAYgvoaVK+U1CzpE+6+Q8ER0m/3toCZlZrZ2HB8lKQLJK0+hFqRxQ6fWKSHbjxdt146XwW5cT2xcofO/84z+r/nN6otmYq6PAAAAABDSJ/u+itJZjZR0knh05fcvfIA8x8r6R5JcQWB+AF3v7W3Zbjr7/CwvaZRX3vkDf15dfBP5MjJxfqXS+drYfn4iCsDAAAAEJX+3PW3rz9Pc4WCI6hPSzJJZ0n6B3d/6BDq3A9Bdfhwdz25qlK3PLpS22oaJUkfPHGabr5onkoK8yKuDgAAAMBgy0RQXSHpgvajqGZWKulJd19wSJV2QVAdfhpbkvrh0+v0v89sUEsypeL8HP3De47Q1afMVDxmUZcHAAAAYJBk4ndUY11O9a3ux7IYwUblxvXFC4/QHz5/ts4+vFS1TW362m9W6pLbntNifnsVAAAAQDf6GjafMLM/mNl1ZnadpMcl/S5zZWG4OaykQPd8/CTd/pETNWVMvlZur9VVdyzWjT9dps3VDVGXBwAAACCL9Hrqr5nNkTTR3Z83s8slnRlOqpF0n7uvH8hiOPV3ZGhsSerORRv0o6fXq7E1qdx4TNedUa6b3j1Hxfl9/tUjAAAAAEPIgF2jama/lfQVd3+9S/sxkv7N3d9/SJV2QVAdWXbsadK3/7BGv3plqyRpQkGuPn/B4brqpOnKiXNmOQAAADCcDOQ1qhO7hlRJCtvKD6I2oMOkMfn6zhUL9NhNZ+rk8vGqbmjRPz3yht73/UX686oK9fWnkwAAAAAMLwcKqmN7mTZqIAvByHXMtDH65adO1Y+uOUHTx4/S2op6XX/PUl3xvy/q5U27oi4PAAAAwCA7UFBdamaf7NpoZn8jaVlmSsJIZGa66JjJevIL5+hrFx+l8QW5ennTbn3o9hd1/d0va/WO2qhLBAAAADBIDnSN6kRJD0tqUWcwXSgpV9Jl7r5jIIvhGlW0q2tq1Y8XbdSPF21QQ0tSZtJlx03V5y84XNPHj466PAAAAAD9NGA3U0pb4bskHR0+XenuTx1CfT0iqKKrnfXNuu2pdbpvyWa1Jl2JuOmqk2bob8+drSljOfscAAAAGCoGPKgOFoIqerJl117995/W6uHl2+Qu5cZjuvKk6QRWAAAAYIggqGLYequiTt9/ap1++9r2jsB6xUnT9Olz5xBYAQAAgCxGUMWw1zWwJuKmK0+aTmAFAAAAshRBFSPGWxV1+p+n1umxtMB62fFTdcPZszWnrDDq8gAAAACECKoYcdZV1un7fw6OsKZcMpMuPGqi/vbcOTpuem8/BwwAAABgMBBUMWJt2tmgOxZt0ENLt6olmZIknTZrgm48d7bOnlsiM4u4QgAAAGBkIqhixKusbdJdz2/SfYs3q665TZJ01ORifeqcWXrfMZOViMcirhAAAAAYWQiqQKi2qVX3LX5bP3luo3bWN0uSJhXn69rTZ+rDJ83QuILciCsEAAAARgaCKtBFU2tSv3plq+56bqPWVzVIkvITMV12/DR94oxyzZ1YFHGFAAAAwPBGUAV6kEq5Fq3bqbue26hn1lZ1tJ81t0SfOOMwnXN4qWIxrmMFAAAABhpBFeiDdZX1uvuFjfrVsm1qbE1KkmaVFOjqU2bogydO09jRnBYMAAAADBSCKtAPNXtb9IuXt+jeFzZp+54mSVJuTkwXHztZ15wyUyfMGMvdggEAAIBDRFAFDkJbMqU/r67UfUve1rNppwUfOblY15wyQx84fqoK83IirBAAAAAYugiqwCHaXN2g+1/aogeWbtGuhhZJUkFuXJceP1VXLJyuBdPGcJQVAAAA6IesCKpmNl3SvZImSnJJd7j793pbhqCKbNPcltQTb+zQfUve1ksbd3W0zy0r1IcWTtNlx09TaVFehBUCAAAAQ0O2BNXJkia7+ytmViRpmaQPuPubPS1DUEU2W1tRp1++vEWPvLpN1eFR1njM9K4jyvShhdP07nllSsRjEVcJAAAAZKesCKr7bcjsN5Juc/c/9TQPQRVDQUtbSn9ZU6kHl27VX9ZUKpkK9qEJBbn6wPFT9dcnTNORk4s4NRgAAABIk3VB1czKJT0r6Wh3r+0y7QZJN0jSjBkzTty8eXPG6wEGSlVdsx55dZseXLZFayvqO9rnlhXq0uOm6JIFUzVjwugIKwQAAACyQ1YFVTMrlPSMpH9191/3Ni9HVDFUubte27pHDy7bosdfe0e797Z2TDt+xlhdumCK/urYKVzPCgAAgBEra4KqmSUk/VbSH9z9vw40P0EVw0FrMqXn3tqp3yzfpj++WaG9LUlJUsykM+aU6NLjpurC+RNVnJ+IuFIAAABg8GRFULXgAr17JO1y97/vyzIEVQw3e1va9OSqSj26fJueXlOltvB61kTcdOacEl109GRdcNREjSvIjbhSAAAAILOyJaieKWmRpNclpcLmr7r773pahqCK4Wx3Q4t+/8YOPbpim17auEthZlU8Zjpt1gRddMwkXXjUJE4PBgAAwLCUFUH1YBBUMVLsrG/WH1dW6PdvvKMX1ld33Dk4ZtJJ5eN10dGTdMH8SZo6dlTElQIAAAADg6AKDCG7G1r0p1UVeuKNHVr0VpVak5375JGTi3X+kWU678iJOnbqGMVi/OQNAAAAhiaCKjBE1Ta16qlVlXrijR169q2qjhsxSVJpUZ7OmxeE1jPnlGhUbjzCSgEAAID+IagCw0BTa1KLN1Trz6sq9edVFdq+p6ljWl5OTGfMKdG755XpnMNLNX08v9UKAACA7EZQBYYZd9eqd+r051UVenJVhVZs3bPP9MNKCnT23BKdc0SpTp01QaNzcyKqFAAAAOgeQRUY5iprm/TU6ko9s7ZKz63bqbqmto5pufGYFpaP09mHl+rsuaU6cnKRgl+LAgAAAKJDUAVGkLZkSsu31OjZtVV65q2dem1rjdJ369KiPJ05p0SnzZqg02ZP4DRhAAAARIKgCoxguxta9Ny6nXpmbZWeXVulyrrmfaZPGzdKp82aoNPnTNBps0o0aUx+RJUCAABgJCGoApAUXNu6tqJeL6zfqRfXV2vxhmrVpp0mLAXXt542e4JOmzVBp8war7IigisAAAAGHkEVQLeSKdeqd2r14vpqvbihWi9t3KX65n2D68wJo7Vw5nidVD5OC8vHa3ZpAde4AgAA4JARVAH0SVsypde37dGLG6r14vpqvbJ5txrSfrtVksaNTmhheWdwPXrKGOXmxCKqGAAAAEMVQRXAQWlLprR6R51e3rRLSzft1subdu13jWteTkwLpo/V8dPH6rjpY3XcjLGaVJzPUVcAAAD0iqAKYEC4u7bubtTLm3bp5U27tXTTLr1VWb/ffGVFeTpu+tiOAHvMtDEqyk9EUDEAAACyFUEVQMbsbmjRq1t2a/nbNXp1S41WbKnZ7wZNZtKc0kItCI+6HjN1jI6YVKT8RDyiqgEAABA1giqAQePu2rizQcvD0Lp8S43efKdWrcl9P1viMdPcskLNnzJG86cU6+ipY3TUlGIV5uVEVDkAAAAGE0EVQKSaWpNa9U6tlm+p0Wtb92jl9j1aV1mvVDcfN4eVFGj+lGLNnzJGR08NhuMLcge/aAAAAGQUQRVA1mlsSWrVjlqt3F6rldv26I3te7R2R71akqn95i0tytO8SUU6fGKRjphUpHmTijS3rEijcjl1GAAAYKjqT1DlnDsAg2JUblwnzBinE2aM62hraUvprcq6tPBaq9Xv1KqqrllVdc1a9NbOjnnNpPIJBTp8YqGOmFSseZOCEFs+oUDxGHccBgAAGE44ogogq6RSrm01jVq9o05rdtRqTUW91uyo1YaqBrV1c+5wbk5Ms0oKNLusULNLCzW7tCAcFnIEFgAAIItwRBXAkBWLmaaPH63p40frgqMmdrQ3tyW1oapBa3bUafWOOq2tqNOaHXUdoXb1jrr91jV17CjNLivUnNJCzS4LAuycskJNKMjld18BAACyGEEVwJCQlxPXkZOLdeTk4n3a65pataGqQeur6rW+ql7rKuu1vqpBm3Y2aFtNo7bVNOrZtVX7LFOcn6PDSgo0c0KByieMDoYlwZAQCwAAED2CKoAhrSg/oQXTx2rB9LH7tLcmU3p7116tD4NrZ4itV21Tm1Zs3aMVW/fst77CvBzNnDBa5WnhtTwMtKVFeYRYAACAQUBQBTAsJeKxjmtV07m7quqb9Xb1Xm2q3qvN1Q0dw407G1TX1Bbc3Gl77X7rzE/ENG3caE0bNyp8jN5nyNFYAACAgUFQBTCimJnKivJVVpSvheXj95nm7qrZ26pN1Q3aXL13v+GuhhatqwyOzHantyA7ZUy+SgrzFOMOxQAAAAdEUAWAkJlpXEGuxhXk6vi0n9FpV9fUqm01jdq6q1Fbd+/V1t2NwaMmGK/Z29prkM2JmSYW52vymHxNGtM+HKXJ4fjkMaNUWpTHz+0AAIARL2NB1czuknSxpEp3PzpT2wGAwVKUn9C8SQnNm1Tc7fSeguyW3Xu1Y0+TqhtaOm7w1JN4zFRWlNcZZIuDIFtalKeyojyVFeeptChfxfk5nGYMAACGrUweUb1b0m2S7s3gNgAgaxwoyDa1JlVR26R39jRpx55g+M6exn2e76xvDtub9Gov28rLiXWE12CYnxZkO5+PL8hVTjyWmRcMAACQIRkLqu7+rJmVZ2r9ADDU5Cfimjkh+FmcnjS3JVVZ29wRYtsDbFV9s6pqm1VZ16SqumY1tCQ7jtj2JmbS+II8lRTmakJhriYUBOG1pDBX4/cZz9WEwjyO1AIAgKwQ+TWqZnaDpBskacaMGRFXAwDRysuJa/r40Zo+fnSv8zU0t6myrllVdUF4raxtVlV9syrTwmxVXbOqG1q0s75ZO+ub+7T9RNw0bnQQWicUBOF2fEFuOJ6ncaMTGjs6V2NHJzR2VDDMT8QH4qUDAAB0iDyouvsdku6QpIULF3rE5QDAkFCQl6PD8nJ0WEnPR2el4Pdkd9Y3q7q+RdUNLdrV0DleXd+sXQ3t4y3a1dCi+jAAV9b1LdhKwd2Ox43O1ZhRCY0dndC49iA7Oldjw7b28XEFwXDM6ITycgi4AACge5EHVQBA5iTiMU0eM0qTx4zq0/xNrckgvNa3qLqhOW08CLY1ja2q2duimr2t2r23VXsaW9TUmuq4rrY/8hMxFecnVJSfo+JRiV7Gc1Scn1BdELTcAAAOY0lEQVTxqJywPRgflYhzmjIAAMMUQRUA0CE/EdeUsaM0ZWzfgq27a29LUrvD8LqnsbVjvD3Qpofb9PGm1pSaWvt39DZdTsz2C7YFeTkqzMtRQV5chXkJFebFVZCX3p7T2Zab07FMghtOAQCQVTL58zT3SzpXUomZbZX0dXf/Saa2BwAYfGbWEQSn7f/Tsz1qD7h1TW2qbWpVXVOrahuD8dqmNtU2tobt7eNt4Tyd402tKe0Oj+weqtycWEfALchND7XBY3ReXKNz4xqdGxzJHZUbPB+VCNv2ed4+PYffxAUA4CBl8q6/H87UugEAQ1t6wJ00Jv+g1tHSlgrCa1qwbWhuU31zMhy2qSF8tLc1tATt9U1tnfO0JNXSltKuthbtahjY15mbE+sSXuManchRfm5co7u0j0rElZeIKz8RV34ipvycuPLCYXtbXk44LRFOS8SVlxNTbjzGadAAgGGFU38BAENSbk4suDtxYd4hrcfd1dyW6gi2wTAItnVpYbexJam9rUk1tiTTxtu0tyWpvR1tbWpsSQXtrUEAbmlLaU/joR/17Y2ZwkDbGV7z24NvTqxj2BGC2wNuTky58XgwDB958c7x3PTx9OfdtOflEJYBAAOHoAoAGNHMLAxwcZUcYuhN1x6AgyDbpqbW5L6htpv2prakmltTam5LhtfwJtXUmlRzW/t4qtt52lKuxtakGluTkjIbinuTiFsP4Ta+XwhOxE058WB6TsyUyIkpEQvaEvFweiymRI4pEeucPxE3JeKxYDzWPm7hMsF4bjjMicU6xzvag+3lxmOKcWo2AGQtgioAABmQHoDHF+RmdFttyVRnmA2HzWGo7RjfJ/AG87W0pdSaDIbNbSm1hOMdj7TnzR3jyW7na016+EiqoSWZ0dc7UGKmfQJuTiwIsfFYEGzjMQufp7XvM72b9vT54z20tz+PH8z6Y4rFpLgFz2Mx6xw365iW3m4mxcPnsbRhLGyPhfO1twNANiCoAgAwxOWERwoL8qL7bz2VcrWmegi53YTg1mRKrSlXa1tKban2oJtSW9LVEg7bUqnO8WRKLeGwLdU+z77Lta8zaA/bUim1toXraguWbd9GyqXmsD506inAdm3vHGq/wNy1/cDBOfjjTsyC5UxSzCxsU0cI3+e5BevqWKbL8/2Wty7b6MM86dvoOtx3/q41Wsdr2GeeWDd1KFindTPeXqfa29VZQ/u8UrBeU/fraF/O0mpSx/TO2tLX1/Fa05azsG5gsBBUAQDAIYvFTHmxuPJy4lGX0ifurmTK00JvEGaTYZBtnxYMU/s+T/bQnnIlU6lulg/bU65ksof29ufJHtq7rD8V1p/04I8EyVTQ1t6ecgXTUx68VnclU+qcHs6fdFcqpXC6S5JSLqWSLsmj7SRkpR6Dr7oG7GB6+7TOML1v8O0asGNhGO4atmNd1td9wE5fd5dp4Uqth9ehtDqUtkxHWw/rVbfr6Az17etQl1p6Wu++2+59vermNU4fP1p/d97cvnRl1iOoAgCAEcfMwmtXg98PRqCnANtd+/4BWd0E5r61u6tjeud48AeF9OfB9M7x9nk6xw88T8c2Ui6X9tt+x/yp/ZcP5k/bRjfzpG/D1TlPt9vYZ97O1+vqbEuFIx3T02r28G8J7etKX869c/7OYfo29m1vf31KW08qbXq79mU7G/mDRjY5dtoYgioAAACGl1jMFJPxBRHd8rSQ3RF81RmauwZsDw/Mdxd8PUzf+wfs7tfXvm11CePpfxToDO5dnquz5vbp6evpWKbLPO01dq6j9/Wq2zp63vaB1iulr6f39bZve9zozN4TYTDxOQQAAADggMxM8fZzToEMi0VdAAAAAAAA6QiqAAAAAICsQlAFAAAAAGQVgioAAAAAIKsQVAEAAAAAWcXSb30cNTOrkrQ56jp6USJpZ9RFYD/0S/ahT7IT/ZJ96JPsRL9kH/okO9Ev2Sfb+2Smu5f2ZcasCqrZzsyWuvvCqOvAvuiX7EOfZCf6JfvQJ9mJfsk+9El2ol+yz3DqE079BQAAAABkFYIqAAAAACCrEFT7546oC0C36JfsQ59kJ/ol+9An2Yl+yT70SXaiX7LPsOkTrlEFAAAAAGQVjqgCAAAAALIKQbWPzOy9ZrbGzNaZ2c1R1zNSmNl0M/uLmb1pZivN7HNh+y1mts3MloeP96Ut85Wwn9aY2Xuiq354M7NNZvZ6+P4vDdvGm9mfzOytcDgubDcz+37YL6+Z2QnRVj/8mNkRafvDcjOrNbO/Z18ZfGZ2l5lVmtkbaW393jfM7GPh/G+Z2ceieC3DRQ998m0zWx2+7w+b2diwvdzMGtP2mdvTljkx/NxbF/abRfF6hose+qXfn1l8Rxs4PfTJL9P6Y5OZLQ/b2VcGQS/fhYf//yvuzuMAD0lxSeslzZKUK2mFpKOirmskPCRNlnRCOF4kaa2koyTdIulL3cx/VNg/eZIOC/stHvXrGI4PSZsklXRp+5akm8PxmyX9Rzj+Pkm/l2SSTpW0JOr6h/Mj/MzaIWkm+0ok7//Zkk6Q9EZaW7/2DUnjJW0Ih+PC8XFRv7ah+uihTy6UlBOO/0dan5Snz9dlPS+F/WRhv10U9Wsbyo8e+qVfn1l8R8t8n3SZ/h1J/y8cZ18ZnD7p6bvwsP9/hSOqfXOypHXuvsHdWyT9QtKlEdc0Irj7O+7+SjheJ2mVpKm9LHKppF+4e7O7b5S0TkH/YXBcKumecPweSR9Ia7/XA4sljTWzyVEUOEKcJ2m9u2/uZR72lQxx92cl7erS3N994z2S/uTuu9x9t6Q/SXpv5qsfnrrrE3f/o7u3hU8XS5rW2zrCfil298UefOu7V539iIPQw77Sk54+s/iONoB665PwqOgVku7vbR3sKwOrl+/Cw/7/FYJq30yVtCXt+Vb1HpaQAWZWLul4SUvCppvCUxruaj/dQfTVYHJJfzSzZWZ2Q9g20d3fCcd3SJoYjtMvg+sq7ftFgn0lev3dN+ifwfUJBUcg2h1mZq+a2TNmdlbYNlVBP7SjTzKnP59Z7CuD5yxJFe7+Vlob+8og6vJdeNj/v0JQxZBgZoWSfiXp7929VtKPJM2WdJykdxScioLBdaa7nyDpIkmfMbOz0yeGf0XltuKDzMxyJV0i6cGwiX0ly7BvZBcz+0dJbZLuC5vekTTD3Y+X9AVJPzez4qjqG4H4zMpeH9a+fwRlXxlE3XwX7jBc/18hqPbNNknT055PC9swCMwsoWDHvM/dfy1J7l7h7kl3T0m6U52nLNJXg8Tdt4XDSkkPK+iDivZTesNhZTg7/TJ4LpL0irtXSOwrWaS/+wb9MwjM7DpJF0u6Jvyip/DU0upwfJmC6x8PV/D+p58eTJ9kwEF8ZrGvDAIzy5F0uaRftrexrwye7r4LawT8v0JQ7ZuXJc01s8PCoxVXSXo04ppGhPB6iJ9IWuXu/5XWnn5942WS2u9O96ikq8wsz8wOkzRXwQX9GEBmVmBmRe3jCm5K8oaC97/9LnIfk/SbcPxRSdeGd6I7VdKetNNVMLD2+Ys3+0rW6O++8QdJF5rZuPDUxwvDNgwQM3uvpC9LusTd96a1l5pZPByfpWDf2BD2S62ZnRr+33StOvsRA+QgPrP4jjY4zpe02t07TullXxkcPX0X1gj4fyUn6gKGAndvM7ObFHRmXNJd7r4y4rJGijMkfVTS6xbeDl3SVyV92MyOU3CawyZJn5Ikd19pZg9IelPBqVyfcffkoFc9/E2U9HDw2akcST939yfM7GVJD5jZ9ZI2K7jpgiT9TsFd6NZJ2ivp44Nf8vAX/tHgAoX7Q+hb7CuDy8zul3SupBIz2yrp65K+qX7sG+6+y8z+RcGXcEm61d37etMZdNFDn3xFwR1k/xR+li129xsV3PX0VjNrlZSSdGPae/9pSXdLGqXgmtb061rRTz30y7n9/cziO9rA6a5P3P0n2v/eBxL7ymDp6bvwsP9/xcIzXQAAAAAAyAqc+gsAAAAAyCoEVQAAAABAViGoAgAAAACyCkEVAAAAAJBVCKoAAAAAgKxCUAUAoBdm9kI4LDezqwd43V/tblsAAIx0/DwNAAB9YGbnSvqSu1/cj2Vy3L2tl+n17l44EPUBADCccEQVAIBemFl9OPpNSWeZ2XIz+7yZxc3s22b2spm9ZmafCuc/18wWmdmjkt4M2x4xs2VmttLMbgjbvilpVLi++9K3ZYFvm9kbZva6mV2Ztu6nzewhM1ttZveZmbWvz8zeDGv5z8F8jwAAGGg5URcAAMAQcbPSjqiGgXOPu59kZnmSnjezP4bzniDpaHffGD7/hLvvMrNRkl42s1+5+81mdpO7H9fNti6XdJykBZJKwmWeDacdL2m+pO2Snpd0hpmtknSZpHnu7mY2dsBfPQAAg4gjqgAAHJwLJV1rZsslLZE0QdLccNpLaSFVkv7OzFZIWixpetp8PTlT0v3unnT3CknPSDopbd1b3T0labmkckl7JDVJ+omZXS5p7yG/OgAAIkRQBQDg4Jikz7r7ceHjMHdvP6La0DFTcG3r+ZJOc/cFkl6VlH8I221OG09Kar8O9mRJD0m6WNITh7B+AAAiR1AFAKBv6iQVpT3/g6S/NbOEJJnZ4WZW0M1yYyTtdve9ZjZP0qlp01rbl+9ikaQrw+tgSyWdLemlngozs0JJY9z9d5I+r+CUYQAAhiyuUQUAoG9ek5QMT+G9W9L3FJx2+0p4Q6MqSR/oZrknJN0YXke6RsHpv+3ukPSamb3i7tektT8s6TRJKyS5pC+7+44w6HanSNJvzCxfwZHeLxzcSwQAIDvw8zQAAAAAgKzCqb8AAAAAgKxCUAUAAAAAZBWCKgAAAAAgqxBUAQAAAABZhaAKAAAAAMgqBFUAAAAAQFYhqAIAAAAAsgpBFQAAAACQVf4/QXTwgF1vKFgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b20d0b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "# grab our data\n",
    "X, y = data, labels\n",
    "\n",
    "# always a good idea to normalize\n",
    "X = X / np.amax(X, axis=0)\n",
    "y = y / np.amax(y, axis=0)\n",
    "\n",
    "# choose a random initial m, b\n",
    "w, b = [random.random(), random.random()], random.random()\n",
    "\n",
    "# our function w1 * x1 + w2 * x2 + b\n",
    "def F(X, w, b):\n",
    "    return np.sum(w*X, axis=1) + b\n",
    "\n",
    "# what is our error?\n",
    "y_pred = F(X, w, b)\n",
    "init_cost = cost(y_pred, y)\n",
    "\n",
    "print(\"initial parameters: w1=%0.3f, w2=%0.3f, b=%0.3f\"%(w[0], w[1], b))\n",
    "print(\"initial cost = %0.3f\" % init_cost)\n",
    "\n",
    "# implement partial derivatives of our parameters\n",
    "def dJdw1(X, y, w, b):\n",
    "    return -np.dot(X[:,0], y - F(X, w, b))\n",
    "\n",
    "def dJdw2(X, y, w, b):\n",
    "    return -np.dot(X[:,1], y - F(X, w, b))\n",
    "\n",
    "def dJdb(X, y, w, b):\n",
    "    return -np.sum(y - F(X, w, b))\n",
    "\n",
    "# choose the alpha parameter and number of iterations\n",
    "alpha = 0.001\n",
    "n_iters = 2000\n",
    "\n",
    "# run through gradient descent\n",
    "errors = []\n",
    "for i in range(n_iters):\n",
    "    w[0] = w[0] - alpha * dJdw1(X, y, w, b)\n",
    "    w[1] = w[1] - alpha * dJdw2(X, y, w, b)\n",
    "    b = b - alpha * dJdb(X, y, w, b)\n",
    "    y_pred = F(X, w, b)\n",
    "    j = cost(y_pred, y)\n",
    "    errors.append(j)\n",
    "    \n",
    "# plot the error\n",
    "plt.figure(figsize=(16, 3))\n",
    "plt.plot(range(n_iters), errors, linewidth=2)\n",
    "plt.title(\"Cost by iteration\")\n",
    "plt.ylabel(\"Cost\")\n",
    "plt.xlabel(\"iterations\")\n",
    "\n",
    "# what is our final error rate\n",
    "y_pred = F(X, w, b)\n",
    "final_cost = cost(y_pred, y)\n",
    "\n",
    "print(\"final parameters: w1=%0.3f, w2=%0.3f, b=%0.3f\"%(w[0], w[1], b))\n",
    "print(\"final cost = %0.3f\" % final_cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice thing to do is to visualize our function. We can plot the prediction surface, $f(X)$ as a function of  $X=\\left[x_1,x_2\\right]$ within some range. We do that in the next cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x10b2af550>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzsvWmQJGd5NXpq33vfl5neu2fT7DOS4BLEBUwYh9EF/LFGYC4mzA8jsHEYFCauP2MbIyTEFgb7OiwDFy98MrJDNp8MYSss7AAto83SjDS9VPXe1Vt17UsulXl/VL3Zb2VlVmVWZc1U1+SJ6OiZ7uysrCVPPnne85zHIooiTJgwYcLE7Yf1dh+ACRMmTJgowCRkEyZMmGgSmIRswoQJE00Ck5BNmDBhoklgErIJEyZMNAlMQjZhwoSJJoFJyCZMmDDRJDAJ2YQJEyaaBCYhmzBhwkSTwK5ze7Otz4QJEyb0w6JlI7NCNmHChIkmgUnIJkyYMNEkMAnZhAkTJpoEJiGbMGHCRJPAJGQTJkyYaBKYhGzChAkTTQKTkE2YMGGiSWASsgkTJkw0CUxCNmHChIkmgUnIJkyYMNEkMAnZhAkTJpoEJiGbMGHCRJNAb7iQCRMVIYoi8vk8AMBms8Fi0ZSpYsKECZiEbMIgCIKAfD4PnufBMIz0c4vFApvNJn1ZrVZYrVZYLBaTrE2YkMEkZBN1QRAE8DwvVcUWi0UiXFEspLUSopb/XTabRWdnJ+x2u0nUJkzAJGQTNUAURYiiCI7jIAgCAEhESkiY/Iz+ToNlWaysrMDn84Fl2ZK/sVqtsNlsJlGbuONgErIJzRBFUaqI5USsF+RvbDab4mPk8/kSoibbyuUPk6hNtBJMQjZRFXIiJiRYDxHKq2n650r7pYlaFMWSbeLxOHp7e6XKmq6qTZg4SjAJ2YQqRFEEz/OS/msEEROoEXKl7dWIemlpCe3t7WVETQhaaUHRhIlmhEnIJspArGs8z+P69esYHR1FR0eHoY+hl5Ar7Yc4OWjQOjfLsiZRmzgSMAnZhARSEdOygJzotKAZiK1SRQ0APM+D47iS3yUSCXR0dMDpdJpEbeK2wCRkE4rShNVqlf5NFvCMhFEVci2PS38nEEUR6+vrcLvdZc+XXJho14fZ9GKiETAJ+Q4G3cwBKFeVVqu1IcR5uwhZDeR4CNkSyL3U5M6B3tZsejFhFExCvgOh1MyhRiC1VMjyxbVK2zU7KlXUgHLTC8/zYFkWnZ2dJlGb0AWTkO8Q0ItcCwsLGBkZgcfjqUoQjayQmxFaj6sSUedyOWxtbcHr9Zb9jdn0YqISTEJucSg1c2SzWeTzeU0kUKuGXK1KbjbJAjCmYifPWS59kP2bTS8mKsEk5BZFpWYOm82mmWStVqsuQs5ms1hfX4fD4YDf74fP54PD4aj1aRxJqF2Maml6YVkWTqcTHo9HqqhN50frwiTkFgPtISYnt5wI9FSnWrdNpVIIhULIZDIYGhoCz/PY2dlBOp0Gz/NwOBzw+XwlX80Grdq30fupRNSbm5tob29HZ2en6aW+A2AScotAiYiJdU0OPVVvtW2TySSCwSBYlsXExAS6u7vBcZxipZdOp5FOpyWiTqfTePnllyWC9vv98Hq9R76iNorY6bwPu/3wVDWbXloXJiEfcRAP8cbGBgYGBioSMYEeQlarkOPxOILBIARBwMTEBLq6uirux+l0wul0orOzU/rZtWvXcOrUKYmct7e3SypqInmQL5qUGoHbVSFXgiAIZe9nLU0vVqsVPM/D7XbD4XCYRN2kMAn5iELezLGysoKhoSFNf6u3QqZP7mg0imAwCKvVisnJSbS3t+s/eApKRE2qv1QqhXQ6jXA4jHQ6jXw+D6fTecdIH2Rf1S6wBNUsevPz8zh+/HiJ+4MQs91uNyvqJoBJyEcMWpo5qkFvhSwIAiKRCILBIJxOJ2ZmZtDW1qb72LXCYrHA6XSiq6urpPIWRbFE+iBEnU6n8corr5RJH42uqNVgdIVc777I34uiKBEv+T9gNr00E0xCPiLQ08xRDVoJWRRFJJNJhMNhdHV14eTJk/D7/TU9phGwWCxwuVxwuVwlRP3888/jxIkTEjlvbm4ik8kgn8/D5XKVVNOViPqoSBZG7auWphciicmrapOojYFJyE2MSpM51LbXclJUI2RRFLGzs4Pl5WU4HA709fXh5MmTtT2JWwA1oiYVNZE+KhG1kdKH0ZKFUfvK5/OawqKqEbUgCLh58yb6+vokycpsejEGJiE3IeSOCaB6RUy8xVpOODVCFgQB29vbWFlZQWdnJ86fP49kMoloNKrr+Jul6YMm6u7ubunnoiiCYZiSijqdTiOVSuH69etl0ofexLujUiHrBU3UoijC4XCUyB9m00v9MAm5iUA3c2xsbIDjOIyNjWmuerVWQPKFOkEQsLW1hdXVVfT09ODixYtwuVwAgHQ63ZC0t9sJi8UCt9sNt9tdQtTXrl3D9PQ0stks0uk0NjY2pOfvdrvLpA+11/p2Lepp2Vejqm0tTS/y41DSqO/0BUWTkJsASh5iu90OhmE0fzhr8Rbn83lsbGxgY2MDfX19uHz5MpxOZ8m2zVLt3goQovZ6vWUVdS6Xkyrqg4MDZDIZVaJutkU9gtslf1QjavJ6joyMALizvdQmId9GVGrmsNlsJVVFNehphxZFEZFIBFtbWxgcHMSVK1dUmzH0tk4fZVRqefZ4PPB4POjp6SnZXomoGYaBw+EAwzCS9OHxeGoK+zdSsjAS+Xy+bvmDvNbkc26z2e74pheTkG8DlCZzyD/cegmZSBaVwHEc1tbWsLGxAY/Hg7vvvruqNexOqpD1Qo2o19fXkc/n4ff7kUqlSipqj8dTVlFXIjYjq1ojiUvreoUW5PN56XNYa9NLqxC1Sci3EJUmc8ihp+IFKleyLMtidXUVu7u7GB0dxcmTJxGJRDT5dO+kChkwjrRcLhd6enrKKmqiT6fTaezv7yOTyUAURVWiNpKQjYSR2jZp+KmEas4PNaL+5je/iQceeODItOObhHwLIAgCOI7D+vo6hoeHNa0wa6l4aSgROMMwWFlZwf7+Po4fP4577rkHVqsV0WhUd2OICe2oJH14vV54vV709vaWbK9G1LlcDsvLy1IbOUl9u90w8q6J53l4PJ6a/rYaUT/++OP4whe+UN8B3kKYhNxA0M0cZGbb6Oiopr+tRbKg846Xl5cRi8UwNjaG6enpkpNY7wKgKVnog97XS42oBUHAtWvXEAgEkE6nsbe3h0wmAwBlFXU1om7m95CWLIwCLbU14x2GGkxCNhhqzRx6q5paJItMJoNwOIxkMonx8XGcOHFC8cNoZNqbEo7SCdAIGHU7T3TQ3t7eMqKmK+rd3V1ks1kA6kRt9OKgke8xz/OG6dFKOEqfR5OQDYLSZI56zO96KuRUKoXd3V0IgoDZ2VmcOnWq4uPqkSHMRT39aLTua7VaFbsLKxG12+0GwzDY29uDz+eD2+2umaCN/jxotdDpRaOJvhEwCblOVJrMUQ+0aMiJRAKhUAgsy6KjowMdHR3o6+vTtG+tJ1WjF/WaddGqHtyu51SJqOPxOBYXF5FKpbCzsyMRtdfrLauoqx270dV2IyQLoBARW28a4a2GScg1QstkjnpQiTRjsRiCwSBEUcTk5CQ6OzuxtrbWEBlCb4VM4jmJ7Yv4cH0+n2K1YhJyZRixH6vVCpfLBa/Xi/HxcenngiAgk8kgnU4jmUxie3sbuVwOQGWiNpqQG1XJJhIJdHR0GL7fRsIkZJ2gifill17ChQsXdOXV1vNhPjg4QCgUgtVqxdTUVMnVX48roxEacjQaxdLSEux2OyYnJ2Gz2aSTnWRF0J1tfr8fgiA0beNDPWjGi4zS62y1WuH3+8sS/JSIOpvNSouPLpcLPM8jk8loqqi1HFsjCDkWi5kVcqtCqZlDT2szANjtdt0dTqSrLhQKwel0YnZ2FoFAoGw7m81W5sNUg5EVMk3Ec3NzCAQCktfa7XaXpa+RzrZUKgWWZfHyyy8DKCxG0RW12+1uOlLTCqM0ViO1Wj0XvmpEfXBwAEEQEAwGkc1mYbVayypqPe9foy5gpmTRgqjWzKHnw0QW6rSY1IlT47nnnoPX662aRdyoClntuSkRsZZ90Z1tkUgEZ8+ehdVqRTabRSqVKqnIaE2UkLXT6TwSRN2KucqEqAkxz83NAShowKSijsfj2NraQi6X00zUjXo/4/G4KVm0CrRM5iAEq3VBwmazlQV+y0FnEfM8j8uXL5eM3FFDI3VhGrUQcaXjANQXo/L5vOQYiEQiWFtbA8uysNvtJSTdbC4Qo2xvzUbIBPK7PJvNhkAgUPZZqEbU5P0TBKEhVbJJyC0APZM57HY7eJ7XRchqVawgCAiHw1hdXZWyiF966SW43e669y1HLR98PUSsdf/VLgw2mw1tbW1l46I4jpNkD+IYeP755+F0Oktkj1qyjI2AUeTSTFnI8n1peV21EHU0GkUul8O1a9dKLszky+Vy1fxaxmIxzY1YzQKTkHFoXUsmk1IOsBbHBCFkrVAiTUEQsLm5ibW1tbIsYtIcouVEapQ9LRqNIp1OY2Vlpe6KWI5aK3WHwyHZ/IDCavqlS5dKpoPQWcakWeJWtR8bFZlpZF6E0RVyPRc6mqg5jkMikcCFCxdK7oii0Sg2NjbAMEzNRG26LI4YaA9xPp/HSy+9hHvvvVe3JqwV9PZasoj1SCJ6O/uqga6I3W43zp8/b9i+CYy8Ra00HUStWYLom0ZLH0btx8gsZCObL4zeF/l8q90RqRG1zWYrkT7kawyJRMJc1DsKINa1fD4vfehrGStTS4XMsiyWl5exubmJoaEhXL16VZVw9RC+3jAiNShJE7/4xS/q3q8aGq3/VsqJkOubmUwGL7zwQglJ+/3+qklkao9bL5pVsqg3C1m+r2rkrkbUxHpHsqjX19fBMAwA4Dvf+Q7i8Thu3ryJmZkZDA4O6npP1tfX8dGPfhQ7OzuwWCz4zd/8TXzmM5/BwcEBPvCBD2BlZQVjY2N47LHH0NnZqf+Jq+COImSjmzn0ECbHcYhGo0ilUhgbG9OURdzozAkaRi7WacXtbMumrV39/f0ACiOczp07h0wmg1QqhUgkgtXVVXAcB7vdXtboctSmVxvVDScIQk0XKSXoWYORw263KxI1wzD42Mc+hocffhgvvfQSnnjiCTidTjzxxBO69v3II4/gwoULSCaTuHjxIt7xjnfge9/7Ht72trfhgQcewIMPPogHH3wQX/nKV2o6fsXHNWxPTYxKkzmUttV6EmipkFmWxcrKCvb29hAIBDA2NoaxsTFN+9dbIddCyLeDiAmaMSdD7STnOE7Sp7e3t5FKpUqmVxOSNnKEUzNXyM0ofxC4XC689a1vxYMPPoivfvWrZe+lFgwODmJwcBAAEAgEcOLECWxubuKJJ57A008/DQD49V//dbz1rW81CVkr9ATCA7XZ2NQIM5fLYWVlBZFIRMoi3tnZkVpT692/0rZ6CJnneVy7dk0zETdj99mthMPhQGdnZ8ntqXx6NZkMkslkwPM82tvbNcdjKqFVF/VoNDIAKJ1Ol1kpa8HKygpefvllXL16FTs7OxJRDwwMYGdnp+7902hJQhZFEalUSpppplWW0GtjI4NIaciziGdnZ6XH1uJDpqG32UPLtqQiZlkWZ8+e1bToQSpZPXY2rds1W4WsBxaL8vTq//7v/8bx48elqnp3dxeZTAYWi6XEKeD3+yu6BYxe1GtGQm5UsBD5XNX7nFOpFN73vvfhG9/4RlmlbWR2DUFLETLdzBGLxRCNRjE7O6v57/Uu0tntdikwPJPJIBQKIZlMYmJiQjGLuJGDS/W2OL/++uuaq4dGhdQfdUKuBI/Hg46ODsWFxFQqhXg8js3NTcktoNSReDu8w1rQrOQuR72EyXEc3ve+9+EjH/kI3vve9wIA+vv7EQ6HMTg4iHA4rCldUQ9agpCVmjkcDodu10EtrolMJoNXX30V2WwWExMTFbOI67HJVYPeFme9nX2NCoBpRUJWu5tQy4jgeV6SPfb29rCysiLlklgsFmxtbUmEXWs12azkXs/4pkqo9+5CFEX8xm/8Bk6cOIHPfvaz0s/f/e534/vf/z4eeOABfP/738d9991nxOFKONKELIoiWJZVDIR3OByaw3YI9BByIpFAMBhEOp3GXXfdha6urqofgEYSshzVFutupYNDDa2qSevV2+12O9rb28vko3A4jGg0inw+j3A4jHQ6jXw+LyXm0R2J1ci2WTXkRkkWiUSirgXqn//85/jBD36AM2fO4Ny5cwCAP/3TP8UDDzyA97///Xj00Udx/PhxPPbYY0YdMoAjTsj0gEP5CaC32tX6NySLGABGRkawt7dXoh9WQi1z8vQSslbXxK3Kvrgd+73dMGoBlDQ+0O2/ZCGROD4ikUjZnD26I7ERGcbN7rIACudpPV16b37zm1U/m0899VTN+62GI03IgDqxGF0hHxwcIBgMwm63S1nEuVwO29vbmvdfS4XMsqymbfW2ODdLhWwSsjqUSJReSOzp6SnZlnQkyhPzvF6vpFvbbLa6E/OMJPd6fMiVcBSDhYAWIGQ11EIidru9hMRFUcT+/j5CoRDcbncZ0el1TZA8ZK3QQuC1tjg3skJOJpM4ODhAIBBQnRRC9tuKuB2NIXTeA73QRIJ8bty4gWQyid3dXWkhkW459vv9mmJhyXE1Y7VN4yhmIQMtTMi1nBB2ux3ZbBaiKGJ3dxfLy8vw+Xw4ffq0oiOh0RJEJZeFWouz1pO4ERVyKpXC4uIieJ5HR0dHyaQQEkBPvkiKnVkhq8OISpQE+TgcDmmSC3C4kJhKpbC3tyfFvTqdzjLHRyMT88zxTaU48oRsZJVls9kQj8fx7LPPoq2tDXfddVfFLGK9djC9x6pE4JU0YkLgWj7gegm50vNMp9MIBoPI5XKYmppCR0cHOI6Tni8J+CEB9OFwGLlcDizLIpfLobu7W3eV1uwwipCNIis5uSstJJKhCESfVhu9RTJgjKiSzQGnpTjyhFwNWj44JIs4FArBYrHg0qVLmnOIGwm6AteyWEcI3GhCJrY3ObLZrOQ0mZycRHd3t+K2dMAPfTs9Pz8vNe7QVRrdjqzVRdCKMFIaAKpfJCwWC5xOJ7q6uqqO3nrxxRcB1D96y+jnSBCLxaSMkqOEI0/Ild58h8Mh3YYpgc4i7u3txenTp7G6utoUZAwUCDmbzWpucW7UQp28Qs7lcgiFQojH45icnERvb2/Z+6Dltp3onvJ2ZJJrTAJ+iIuAjlqs1uXWCmiWAbD06K3u7m7s7e3h8uXL0kJiPaO3GiVZJRIJzMzMNGTfjcSRJ+RKIK4JOSHn83msr69jY2MDAwMDuHLlChwOBxiG0W2VAxqT8xCNRrGwsIBMJoNLly5p8lTq6eyrpUJmWRahUAgHBweq3Yhkey2vh9JioVqusVqXm91uRy6Xw+bmZtUUtqOGZswPoS8SWkZvHRwcqI7eMiJnQg3xeNzQWMxbhSP/ya1WIdOuCZ7nsba2hq2tLQwNDZVFYNbiXSbEplXrI+SmVvnQ0sT09DSCwaBmg3ujBp2Kooj19XWk02mMj4+X5HPUAz3uDbUuN47j8MILL0AUxbLmCbI9SWFrNnKrhmapkGnUk19MRm+l02ns7OwglUohk8nglVdeMXz0lml7a0IQguU4Dqurq9je3sbIyAjuuecexTe8VqucnpViogvLTzRCxA6HQ5ImSIi+VhihC9PgeR4rKysIh8MYGBjAXXfdZThB1HvL6nA4YLfbMTIyUrLPXC4nyR4k3Icerkm+jMr1bQSasUKuJ8dCPnqLTOk5ceKEpE8bNXrLrJCbEBaLBWtra8hkMhgdHcW9995b8Q2t1SqnhzTJ9sRNoETEBHovEEZJFvl8XrqTGB0dxejoKAKBgK4TUQuZNIpsaM2TDvchnlx5+LzD4YDP55PmuzXa6qUVRlXIRuq0RocUORwOSaKSLyRqHb2ltJB4FMc3AS1AyEonNckiDofD6OnpwT333NOwWz+9zSFk+0pETGCETa7StvLjprX1kZER3H333bDZbFheXtZ9UteqITcSalOQWZZFMpnE3t5eidWLPvFrcRDUC6McCEZW2rcqC1nP6K1cLifp2aIoIhgMIpPJ1Jxl8fGPfxw//vGP0dfXh+vXrwNAw0c3ERx5QqZBsojj8TjGxsbg9/vB83xNweB6poboqZB5nsdrr70Gj8dj+ISOWp0TtNtkYGCgTFvXIm/UgmZpnSZWL5fLhRMnTgAofAbIiU97p0lUJq15Nso7bVQecjPnWOhdgFUavQUcztcLBoN48sknsbW1hcuXL6Orqwsf/vCH8YlPfELzY3zsYx/Dpz71KXz0ox+Vfvbggw82dHQTwZEnZIvFgnQ6jVAohFQqVbL6T9/maEUtU0O0VMikIiYxnbTmaRT0EnI+n8fm5iZWVlbQ19cnuU2Utm0G4mwklNweSq3Iah1utHfaqMYJo4j0KA841Qoyeuv8+fP4zne+g7e85S14+eWXcXBwgFQqpWtfb3nLW7CyslLys0aPbiI48oQsiiIWFhYwMjKCnp6ekoqinsQ3PVNDKlXIcmkiHA7D5XLpOiat0Kohi6KIWCyGcDiMoaEhXL58ueLiVqtXyHqg1uFGe6fpxgmlRUStVa9RUoORJGp0FnIjtHpyZ2GxWNDd3a05jbESGj26ieDIE7LFYsGFCxcUT+xaEt9q1YTlUNOId3d3dUdq6smnqLRvktERCoXgcrnQ19enaaKK1WrV/TpqQTMRcj3kJ/dO7+7uSo0TZBExGo1iY2ND8k5rmWBtZIXcjHGZjWqbTqVSDR3W24jRTQRHnpArodYKWa9rgiaraot1tQQS6cmnUNo3Sa0jnuZz584hm81qvsqbeci1oZJ3muRFVPJO3wkaMtmX46nvAs7SDlnu//hQTfuNx+M1TZquhEaPbiJoCUJWO7EbFVJPw2azIZfLaXJNkO1rieDUchLYbLaySjYSiWBpaQler7ckLIlhmNueh9yK0HKBUZtgLfdOx+NxvPrqqyXVdC3eaaMJudaq1vGv/2/J/4+T5zFful2tREzQiGChRo9uImgJQlZDLYRM8i+0IpvNYn19HclkUpNrgrT6akUtC3XAYaXudDpx6tSpsirNnBhSituRYUxDyTv94osv4syZM1Kwj9w7TWvTlbrbbmWFbHvs4cPnJL9wuIoVMPl5sSKul4DlqHdayIc+9CE8/fTT2N/fx8jICL74xS82fHQTQUsQstoJUMuJodc1IYoiOjo6pLlbRu2f3l6PtzibzeLFF1+E1Wo1bISTOTFEO4z0/IqiKLkH5Lfg9CLi+vo6MpmMqnfa6AGn3f/4TQgAbJ7SxWkrFcpFyNiiQMKCvwP587+MpaWlQrKcIUd2iHrbpv/+7/9e8eeNHN1E0BKEbCTkmrAccmkCAJaXlzXvv1GDTpPJJEKhEHK5HM6dO1f1lq2RFTLP89jb25OM/ZXIoFkI+XZXyHr3pRaTqeSdJmsQLpdLkj+qeaf5b32u7GdWlxMTCtsSIi4hYX8b4HJDcBR+lz//y+WPYY5vKsMdQch6Gz0Yhin7uZpGnMlkGjpJupqVLZ1OY2lpCSzLYmhoSHPLaCMqZEEQsL6+jvX1dXR2dmJvb0+KzvT5fAgEAlLl5nA4zApZA/TsS807vbm5iVQqBYvFgp2dHaTTack7PflP35G2s8sqXrunQKZWV4FoSUVMCNjaU3wMR7H6LVbDlUiYRiPHNxlhdbsdaAlCrvSh1dvoIdedqy3W1bIIaMTYJ9KVlMlkMDU1he7ubiSTScRiMc37NapCJklry8vL6O/vx9WrV0scAiRDgrQnk2YKURThdDolPfQoJrLJ0YyBQK6HPgcXAIfXBXolweY6rJIJGctJ2Nlf0LMl6YF8L5Iw3zUAABDtTmD2TbqOq1E+5GQyiYkJpVq++dEShFwJxIusl5C1uiYaTcjy7XO5HILBIJLJJCYnJ0uaYRoVv6m2rSiK2NvbQzAYRGdnp9RgQkYB0c9BniEhiiI2NjaQSqWQzWalappkEtALVrdirFMzShZ6sPWJXyv7mdN3WPE6vIV/ExKWE7CjI1AiPZQQsL9NqnqF4kKcaC9WxTpJmEYjxzeZksVtRKUTQC9hkqxWjuM0uSb0thXXSsgMwyAUCiEWi2FiYgInT54se96NnBgi3/bg4ACLi4vw+Xw4d+4cPB6PtidUhMVikVLWjh07Jv2chJvLW5Npj67f74fH42m6ShQwXkOmsfh/HUoALn+pg8HuPrxoESKWk7BvuCAxkOq3TPvtLVS7pPqlSZh3t4G3F/YXTFuRSRYWEd1FW14t70sjJQuTkJsUWrv1SEVMKjStrgm90EvIoihia2sLy8vLGB8fx9zcnOoHvlELdfS2iUQCi4uLsFqtinY6+ri1xG/Kj0Ep3Fzu0d3Z2ZFGBfn9frAsi1gsBr/fX3PF1SwV8o23v036tw/AjeK/nRQBEzImJEwTsKuj8H6QylfSfYskbO8t6r4KEgTv64DgcEGwFf6GELDr+GnYANgAXLt2DZcuXZLeO7X3RR7ApOadbsRF9ahmIQMtQsj1VMhyacLj8Ug5BI2AVkImofobGxvo6urC+fPnq1qX9JC9nhPBarWCZVm88sor4Hke09PThhjvtV4U1PKNSdBPJBLBzs4OgsFgSccbWUS8lbGZWgn52al7pH+7ekslGaePqnaL5Csn4fZjhdfhUII4JGB7e5skOUheYIqABW+bJD0IjgLpEhJOeXsK+x88DrXalTy/SrnT5C6nkne6UQu6ZoXcxFAjZDWNWBRF3VkTelDtZCVjpsLhMEZHRzE1NQWO4zT5SBvhF87lclhcXEQikcC5c+cMXb2u12VBgn6cTqeUyUFXbfLYTJoM/H5/2e2y0RXyf3ZdKvu9s6uUfAkZExKmCdjb7ZdkB+BQ9yUk7BkskKAkPdAk3NYB0emCCEAsVrqEhHP+wt/xdjd4a5HgB48Xvtf2tEugNsJJyTv9/PPPlw2vrfcCmkwmj2Q4PXAHELJcsqi2WFdrl1W95nu1cPidnR1FG57acRhOCxChAAAgAElEQVRVdbAsi+XlZUQiERw7dgwsy+om49uxwFWpmiZkEA6HkUqlpDFBZIJ1Pp+v65hpAv5P2e8IEdPVsNPnQGDAD4e3QMIOT/G7lyy4ueDqCJTJDmW2syIRi84i8RYJmG0r/J5ID7y9sB1vdSBja8Ng3629rae906IoIp1O4+LFiyWTq+UXUHqBV6scJQjCkR10ezSPWoZqkgXDMJpdE7VAbbq1FgiCgI2NDayvr2NwcLAsHF6Pc8II8uN5Xpo/ePz4cczMzIDjOITDYc37IGlY1S4Ot9KHbLfbS+a5AYdjglKpFGKxGNLpNK5duyalsdFBP2qLT0pVMAEh4e7ThccksgMhYKBAwp7uNqr6lXl+XU7Y2jtgIZGtRdIlJMx3FEhXLj3wdheyrnbw1sL+ODhLCPh2149kQU9tcjWdO72zs4NQKCR5p+UDUelCSBTFI+1tbwlCBtRP7lwupytrgqCRU0OAAhFvbW1hdXVV8u4qXdX1LgLWCtLUsbGxgeHh4ZKxV63aOk2PCQoEAmAYBmfOnJHS2FKpVNlIJ7/fj+iHCpMkkvOZkv31XDoke4e3WBFTJOztDlB2s1ICdg0cVvOS9ECRsBhoh+iQ6b7FBbicux0cIWKKgNOCF1N9Lty4cQNjY2N1v15GvlfVHBZqudMMw0jvTSQSkRqPvF4vYrEYtre3DR3X9pOf/ASf+cxnkM/n8YlPfAIPPPCAYftWQssQshykIgaA9vZ2Xa4JPZGXgL58CuLRfeaZZ9Db21s1HL7RhKzU1CG/MDQy7a0Zqxl5GtvL97wFNgAcgCi1XWDWC1eAEG6BgLvGD2Udu7u4EEeRsJPy+xIJQiLgogQhSQ8UAXOeduRtxWaMovRASDhhKRwnKzoAAZgaKO24a8bJI7W0TVssFrjdbrjdbvT09JQcVyaTQTQaxVNPPYVwOIzz58+jq6sLn//85/HOd76zpmPM5/P4rd/6Lfzbv/0bRkZGcPnyZbz73e/GyZMna9qfFrQMIZNqSy5N2Gw2LCws6NoXkSC0ErKWClkURenWSxRFnDt3ruw2TQl6JknrAX1h6OrqqnhhaGTaW7NAfkf08j1vUd22e7JAgER6ILYzQsC+wQJZ0OE7hITtRNeWQncK25AFOK69QMr5IunSJJxxtIGHA5xYIH+2+H2qv/IEGqOItFFZyPWC2B/f/OY3Y2xsDJFIBE8++SQODg7q+tw+//zzmJqakrr+PvjBD+KJJ54wCVkLSKOCXCNmWVb3tAtCyFpHLVWqkEk4/NLSEtrb23HhwgXcuHFD84dRj4asFeS14nkely9fljKS1dAo4rzdkoUcnv/5OQSL/24bOvRXOzwF4qO137aRAumqab+OYpaEJDtQBCz42ktkB+CQgFPeAmFz1qIEgcJjc6IDac4LTih8bu4a1f66GUV+t2ridD2gLW908FIt2NzcxOjoqPT/kZERPPfcc3XtsxpahpBzuZwhWRO1/I1ahUyHw589e1YiPj0SRy2ShZr+LW/quHHjRk0LkUbhdhNy8H3vUv0dIeH2Y0XilckPAOAeILazws8sxWqXkLDY3VewnSlov4y7Q6p+ifTAWV3Iil5JeuDyDgUC1v96GVkhN0PQfSU0Ipz+VqJlCHl4eFiRuGqZmFxLPoVSIJHT6cTp06fLpAk9i4C1psPR1QedCDczMyN9YG/3JJDbIVmEf/N/AAAykbT0M2+3ryT3QdJ8iyTsLUoQkvZLEbClrf2w+gUgOpwSCfPeNqny5e3uEgLOWAPgxcLpR6QHjnMgzTnB5AvvnW3vZxgbG4Pf74cgVI4xrQSj7IdHYcCpkYQ8PDyM9fV16f9kwbuRaBlCNhK1VMg8zyMej2NxcRE2m82wMU56SZNIHGS0FAkimp6eLvMR325CBhq/qEcIWA6ahO0eF9ydhffKJpMgVLVf4v11OJH3dUAohu3QBJxzBiTpgUNRCy6ScIwpSmpF8mXyNrxpkgHAA+BxcHCAtaQbDMOUuAloX24gENAcvGTU9Opm1JBpGNmld/nyZSwuLmJ5eRnDw8P44Q9/iL/7u78zZN9qaBlCNrLa0kvILMtic3MT+/v7mJmZqTpgUS8h6yEtq9UKhmGkpo7JyUnFICKybaMI+Xb6kOO//xvI7CcUf1em/VIEbO9oh4WeekHl/Ir+tkPpgbQdF0k47esHAPCkAqZIOM4VPgtEeiAEHMkUtnnnSVKpl37e7HY73G43jh8/Lv1MEISSluSVlZUSby4d8GOk9YvAaEJ2U6+1UTCSkO12O/7sz/4M73znO5HP5/Hxj38cp06dMmTfqo/Z0L03CeiqUQu0EnIqlcLS0hLS6TTa29tx1113adp/o6xsPM8jm83ilVdewfj4OGZmZipeqBpFyKTLz2q1SnkSSlq1kYQc//3fKPm/t6dAhER2AIq2s652WJ2k+i1KD6T12FFsOVZIOwOArL/Y+UY8v0UCjlsKi0dcsfrluKIEIdiQZJ1geULGFlUClkNJ9yWvpzzGlGVZJJNJKSEvm81KYfV+vx88z0tZEvXASA25kZIFvRBXL971rnfhXe9SX2swGi1DyFoChvQQcqV25Uwmg6WlJWSzWUxNTUmTGLTCaEKmu/3Igp3RU0O0IJ/PY2VlBdvb2xgZGQGAknAZUskRkq6HjGkCnpH9zu52wtXVBhsJW5cRsLW9o2zKBdF+uY5CBKXc88vbXMg6ApL0QKxnnGhHmvWAyxeIisnbSgg4mbHi1y4ldT8/OuC/EiwWC1wuF1wuV4k3lw744Xker732Wkk1Td6D2xWXaWYhK6NlCLkSSJ6FVhubWoWczWYRDAaRSqWkKR0WiwWJRKKhIfVqUGrquHnzpua/N4qQBUHA5uYm1tbWpC6/fD5fQip0l1UymZRGCXEch4WFBU1tyszDv13yf6L7ApT0QJGwraP9MGhdRsCHLceHt815mxM5V2nXG03AGc4DVihWwRQBJ3N2cHzheTJc4XstJExDFMW6qlE64GdzcxMXLlxQfA/oGFP6S4ksjVzUa5SGnEgkTEJudtS6SEdAh8NPTk7i1KlTJVVFLU4IrYFBSqAndXR0dJQ0dehpJNFLyPLVenIcS0tL6OnpwZUrV6TbYvnrodRllU6nEQqF0NfXh2QyKbUpi6IotTP3/6+vKR6Lb6iwD1c/1XIsSz0Tu4q2s6LWexg56QbjapOkB3nbcVwo3F0Q6YGQ8EGm8PdMkXw53gKGsyCVEfGxt6QAAOFwuPjZqe+22SirGn0Xotbpls/nSzKN5TGm5Ivned2DCNTQKMlC60zJZkXLEHI1yUJPcwghZDrxrFI4fC2En06nq29YBJ0mF41GsbCwAK/XqzipQ28Ykd5Ae/L8Y7GYdBwXLlyoeYHGYrGUhP6Ij/5PAAB7EC/ZztnVfig5kAkXhIC7SctxkYhxSMK5dpkEQdqNHQXHCS09EO/vQa7wmhLpgRBwJF4gSI4vkBwhYRrNEnRPoIXYbTabYm6EPHw+Ho/DbrdLwwDIHU0t0kMjJYujGk4PtBAhV4LD4dBFmKIoIh6P49q1axgbG6u6OFbvnLxqsFqtiMfjCIVCVSd1NGqMEyHkdDqNhYUFCIKAEydO1JWaJ8kZRRKmQRMwQE26kGu/zoLfl0gPvKzlOOkuVIJy7ZfYzojnl0gQ+8kiQRMJgi0QcCIp4NO/Ul2GuJVE2sj9KMWYLi4uoqOjA06nE8lkEtvb20ilUsjn81KMKfmqlml8FGxvtwMtQ8hGzNUj0ZMkapJOPKuERhJyJpNBJpPBwsIC5ubmqt6O6SVkPQtrr7/+OtLpNGZmZqq2pVYjJdvjX4MnFofcROTok005dhy6JITuQrVL7GY0CTPOgHLiWd4Lvig5EOmByduQZhxS5ctwFomAI9HCa8ewhe+f/pXSRLdqqFf7pffTTMRO9uV0OhWr6UqZxmQRkV4fMPK4aNQag9ssaBlCBtRtVA6HA9lsVvXv8vk81tbWsLW1hZGREdxzzz147rnnNH9g9J44WgiZburweDw4c+ZM1cwJrfsm0ELePM9jZWUFyWQSIyMjOH36dE1EYXu8XAu2dxROamnIplz7lfl9s4Gi35cQLkXAGdEHVihUtzx3SMBJxnm4AEcRcCpjAcMWPiscJwIQwbCibgKWo9mI1Oh2Z6Wqlo4x7StmeAClQwG2trakoQBer1dqeCHDAYySeY46WoqQ1aBWIZMM4PX1dQwPD0tTOhqNSqTJcRxCoRAikYg0Xfq11167ZdOkCWgr3cjICDo7O9HT06PrxLH/49cVf25xOmFpK95WFitg0emWSJj3tB1KD3aKdJ1t4MXDsB0ABRLOA1Gm0J5OdF9CwpFkUQdmUfwuguNEJJIcGKbwHrz/4uuw2WzIZDIQhFN1EViztSnfTmeE2lCATCaDV155BfF4HJubm2AYRtdQgGpophRBvbgjCFmuIdPh8AMDA2VTOhoNJULO5/OSXHL8+HFMT0+XBMTrqXrrqZDpmNDe3l4pH1lLlKH44+9I/7YBEABYA8WuRZkEITrd4H0d0uIbcKj/pjxF3ZdUwEXyjbKFffGU9AAUSDiSKtrRKO03kRLAcYXXgpAvw+SRSjB48JNF4uY4pFK92NvbQzqdlgbc+nw+yavr9/s1N1UYVaUZJX0YLVnUS+6kYcXhcGByclL6udJQAFEUJW2abjBSI9xcLmeYC+R2oaUIWU2yIC4L2rfb19dXYtMyAlqrI5qQ6UpUrUrXI0PYbDbNjhI5eRMHh9/vL3NOqDkyaBIu239PH0RfmyQ95IuaLyHhNImalNnP4nxBymDZwseTEPB+uhjMni/1/EZihccrSBBiUYIA9vay4JjChZgQcAGH/yZh9OR9m5mZKWlR3tvbw/LyMnie1zTNutmI1EhCNkr+UDom+VAAsl02m0UymUQsFsPGxgYYhimbXu3z+WC1WhGLxarGFjQ7WoqQ1WCz2ZBOpzWFsRPoHVxKZBEtBE/S4ba2tipO6qC3b1SFTCqThYUFWCwWVQcHvQAo/qwYsJKMHW4QOGw5JgQMFEiYc7eVSA+AQuKZ4AD4gu6b5lyU5axoNSt2vbHFaw3DipIGHIkUPN2kCuYYHsl4Ft/+nB9QHWZfCvpiSrcoDw4OSr9XmmZtt9tLKmmtHXZ6jqceGE3IRsgfWj3IavP25NOr0+k0/vVf/xUvvvgikskkfvrTn+Ls2bMYGBio+Rj/4R/+AX/4h3+IN954A88//zwuXTqcnfjlL38Zjz76KGw2G771rW/VPJFECS1FyErVCmmgyOVyuPfeezX7ZfUOLiWkWY2QyTGl02nE43FNF4dGNXvk83mEw2Hs7u5iZmZG1b8p/uzvcAYAXrxemsZLkTDbXlhwI8Qr1345sTTtjBUcSOXch6E7FAGnclawxeqXJuBEigdXdD8Q8o3spcHmChsVCBgokLCyLbBWqE2zJhe0ZDKJ9fV1HBwcwGazYW9vrybJg6AZK2TAuNS4eiRCeno1wV133YXHH38cjz32GJ566il87Wtfw5/8yZ/g8uXLNT3G6dOn8Y//+I/45Cc/WfLz119/HT/84Q9x48YNbG1t4e1vfzsWFhYM0+lbipAJRFGUwuH9fj/Onj2LV155RVfzAtGdtRKyFmsd3dTh9Xpx4sQJTfvWU/VqqaZ5nsfy8jK2trbQ3t6Os2fPlp1o+Rd+DGu6PDGN6zhcRRco8s252sFbiwtuRIIoknCMJVGTRZ23SMIH6cLvGYl8C9/3owIKrocC+XKsgMhewf3AFCUImoRffPG6rpmJRkJ+qx0MBtHW1ga3241kMom9vT2EQqEyr24gEKjoLmjGCtkoNMKD7HK50N3djYsXL+Khhx6qe39q5+YTTzyBD37wg3C5XBgfH8fU1BSef/553HPPPXU/JtCChHxwcIClpSW43W6cOXNG09w6Jejt7qtEhMlkEgsLCyVNHb/4xS8M2bcc1ZwTZLL06OgoTp48iYODA+nEz7/w49LtfQU9jmi/gt0F1t0Gzlb0/1oPqz4p8UywS9IDUCDhNOsskR4AYD9mBVvUeon0wLB5pCj3A9F/GYZHMpoGx3L4/pf6i4+oLZdEK4zssLPZbIqpbLlcDslkUlXyCAQC0lj7ZpyDZxRqGXCqBbeiKWRzcxN333239P+RkRFsbm4atv+WIuStrS2Ew2GcPHlStZNNK/SMWQKUK+RMJoPFxUWwLIvp6emaPyx6FuqU5A1RFLG9vS0tZhK9Onfjv9CVjSP/wmLZfnIBMmyTLLgVSDhhJ1GTJGKymPOQLbzeh57fwvcDme2MkPBepPADji11QRzsJsFkye8Kz7lAwsYS8K0GLXnQXl2O46TozNXVVSmIPp/PSxcItbAfLbhTKmRAPyG//e1vx/b2dtnPv/SlL+G+++4z8tA0o6UIeWhoCP39/Yq/q2WRrtbuO7qpY2pqqiTIpZbj0RNGJJc3IpEIFhcX0dbWhosXL0Jceh78fFhK4+U8BUeDXPvNOCkJQjxsO05zbkl6kCSIvBXJnA0sXyo9MCyQTAuS9MAwAjg2j0ScAceWSg9MlkU6ngabY/D4d6Y0PVcjcbsyKBwOR5kems/npUG4u7u7UtiP3AKmpaFCEARDqlEjmy4amfSmZyHv3//933U/RqPHOrUUIVfSWvUu0tUSGJTL5TA/P1/S1KF2whAC10LItXiLiUxyyp3BuQAAMQJxKVKyLdF+DzveDgk4nfeCk3J9Dwk4xdgPiZci4ERKpCSI4oLbfq7w/1zhdSQkHAlHwTEFImZzhQvN//rWGKzW2m83m6VLywjbm81mg81mQ39/vyS50e3JiURCaqigLWC05EHQjNLHUZYs3v3ud+PDH/4wPvvZz2JrawuLi4u4cuWKYftvKUKuhEYScj6fRzQaRTwex/T0dElThxq0ujLobbWAZVlMtdsg7oYw3WEHcsWfu4ujhIrSAyHhBAofYFYoTjouLrhFs4U2bSI9sLLEM6L7ssWuN7n0wOR4HOwUEtvYXFGCYDikonHwRf3iy5+zYnJyEj6fD4IgSM+RfLdYLNJXpdezmTqzGlVpq7Un0xYwInmQ5otAIIBMJqM5B7wSjLbPNTsh/9M//RPuv/9+7O3t4Vd+5Vdw7tw5/PSnP8WpU6fw/ve/HydPnoTdbse3v/1tQ6v9liLkSicCCanXCrvdXjH/Aiht6vD7/Th27Jg0KaMa9GjUWmxvieB/AwDkr0DaVzh55dovUCDhNOcqkR6AAgknszZwxcMj+u9+lAcgSNIDUCDgZCwrVb9E/43uHBQeI5uTCPjJvz2PlRUekUgOExMTiq3YgiBAFMWS70DhJCbVpxaS1gsjXQ230h2hZAGjp4WQinptba0myYPep5EVshEXCTmMJOT3vOc9eM973qP4uy984Qv4whe+YMjjyNFShFwJ9YbU05B3/F29ehUHBweIx+OK26vtvx4r2/b6ivRvL1t43IyzGKxuKXp+cZj7kGI91KDN4kJcuvB7hiM5v4X97R0IAASw3KH2C0CynhHdl2N5JKMpsDnmUILI5vDk/3cWwKFXVxRFbG1t4dq1axgZGcHly5dVyYb8nD75q5E0qa6tVuttX8AysnW6VmKnp4WkUin09vaio6NDkjzoDAmHw1Hil5ZLHgRHZXzTUc5CBu4gQtabiaxEyKIoYn9/H0tLS4qTOhoVwUk0ZJqEacRdBfLLil4p8Yz2/KYYJ9h8qfTAcFakMoftxocShIBknAPLHUoPbI5DKp6RpAei+/7DtyYAVI4DjUQiCAaD6OzsxKVLl2pqVVcj6WQyicXFRYlE8vm89JoS+xndfVcNRkkfzeQfJvkT1SSPZDKJSCRSJnkQojaakM3xTcpoKUJuxNQQgmg0isXFRXg8HsVJHXorcK2EHNrYh1dMYaD78IOWsfhLW45R9PtyLrBE86UIOJGxgqM63grfBSRTPNhc4RgIAe9vF5pBDhfdWPz9V0cRDqcQDK6iq6tLqr4qkQYJsrfZbDhz5oyhoS8syyIUCiGZTGJmZkbK5iWVsyAIFSUPOrSJoBkqWxq3SvqoJHmQuXvBYBAsW7gYr6ysSLJHpaCfSmjU+CaWZWueXNMsaClCBqoHDGkFIVi6qaOSv9moCjm0sV/2s4yl8JiEhKO5wv8l90ORhCMpIkEU/o483YL2eyg9sLk8kknm0HJW/P7oH3QD6JYelxAaz/Po7e1FZ2en1NgQCoWQyWRKch/a2trgdDql/OR6vNdKIJr95uYmxsbGMDs7W0IISkRL/o4QtCiKiouH5Pf1wshFvdvVOk1LHgR7e3s4ODiAx+NRlTwCgQA8Hk/Vx2uEZNEsLpt60XKErAaHwyGZ7rWA4zjEYjHcvHlTE7HUUyFfX03DaytfQCQtx0T7BQoknMzZqcSzIhnHyqUHANjfLeyXZfMSAX/nc20APBBFN5555hnce++9JY9LiJhUl2QRzel0oru7G93dh6RNLlqJREKaKOJyuSTyBoBAIFB3RUQySXp7e3HlyhVd+1PSlukKOp1OY3NzE/39/VIqYK2Lh0dxpp7W/bjdbvT395d4/VmWlRpbaMlDPimEJuBGVchAczluakHLEXKlClkLYeZyOYRCISQSCdjtds3hJHoq5OuracDSC/BAdLUw7DSTLw7WlGm/8pbjvVjx/9yh9AAAkX2SeFZsuGDz+ManPQBom1+pbKAUxkRXkoSQqj1vlmURDofR39+PY8eOAYCkS9KTInw+H9ra2qSKWoueTJLonE4nzp07Z9gtqdVqlaahRKNRzMzMoK2tTXXxkP67SiRtVGULGBfk00gfstJFmp5iTc/d83q98Pv9YBhG8iIbRaAMwxzp0U0ELUfIaqhme+M4DsvLy9jf38fExAROnDiBZ555RvP+qxH+s0t5+B05xd9Fc2TaxaHtLJWzSdIDsZ0VJl3wUvVL9N8//r+tAMhJ58QvfvGLsqq3GsitPCFiLSdxPB6XFtXOnz9fYmWSz10jlWgymZS6z3ieh9frlUiaSB6F58pJ3Y5GSx/EJbO6uorR0VFMTU2VEEMlhwft6iD7ohcPm+3W2aiJIXoW4tSmWGcyGYmc5+fnwbIsnE5nWWNLLSQdj8erzps8CrhjCFmNMOWTOu6+++6ap/TK8exSacWc4kjAOomctCKRsx9OuaAIOJUWDjXfIgH/Px+Sn+zqx6nnljefz5fJE5WQy+WwtLQElmUxNzenKTeE1pqHhoakY8xkMkgkEjg4OMDq6ipYloUoimBZFoODgzh9+rShCzXxeBwLCwtoa2vT5PpQc3iQ7zRh8zyPXC6HfD4Pnucb4pfWCyMli3oqUOLc8Pl8WFtbk9L5GIaR7qb29/eldQnaL+33+6teDI76tGmCliNkNTJRGuNEmjqGhoYMmadn7btaRsIEB+lC9ShJEKTzTab9MoyA33l3tuzDX0iH01b1kkaSSs+HkIjL5cJzzz0Hv99fIico/S25xY9EIpicnFTM6NAD+iQdHBzE/v4+FhcX0dXVhY6ODqRSKdy8eRO5XA4ul0uqosnikZ5KimEYLC0tgWEYnDhxoq7wKfniIV1xHzt2DG63W1Hy0Np5aDSMkj4aofu6XC4pOpN+LCJ5hMPhkuGo8sYWgkQiceSnhQAtSMhqIJ1x5ORZWVkpmRmnBC0BQE/doH93+G9CvOTfqaylRHoARLxl5GWMzI1gYWEBLpcL09PTRXtYfVoY8eQqnUDyBbuzZ89CEARFzZcm6WQyic3NzaqNHbWAWOTsdruipRAoEGoikUAymcT29jYymYy0wk+O0efzKXb+ra+vIxwOY2JiAr29vYYu/CQSCczPz6tW3HQFTb/u1ToPm036MIqQtTyvapJHNBrF+vo6GIaBy+XCK6+8gpWVFUl2q+c4f+/3fg//8i//AqfTicnJSXz3u9+VKu9GTgohaDlCrnSycRyHZ599Fh0dHbh48WLV9k21/ItSEj4EaTlmZS3Hn/g/y5Pa9vb28Prr+8hms5idndV0ddcqQ6gNL1VbsFM6AYjmu7W1hVAoBKvVCofDgVgshnw+L5FgPTMJyYTteDyOmZmZirecLpcLvb29JdM6yAo/CYKnbXhtbW3I5/OSe+Ly5cuGVngcx2FpaQmZTAZzc3Ml2cc09HYe0mim6EwjtehaJUFyN0VcHkTaikQi+PnPf44bN27g0qVL8Hg8+Od//uea7uDe8Y534Mtf/jLsdjs+//nP48tf/jK+8pWvNHxSCEHLEbISSFMHz/O4cuWK5iYFQsj/+7XD29s2d4FtE7miB5hMu+CBGcd/4pcuXazosWRZFsFgEAcHB3C5XCWzuiqBkKyWD4A8+6KWBbtMJoOlpSXYbDZcvXoVHo8HgiBImi+xofE8L3V1EZKupjUKgoDNzU1sbGzg+PHjmJmZqalqVbPh7e/vS1M67HY79vb2kM1mq0oyWiCKIjY3N7G+vo6xsTHMzc3pPvZKJE0ytNvb2+vuPDQSRro1jPIgWywWuFwuvO1tb8Pq6ire9KY34f7770cqlYLX661pn7/0S78k/fvuu+/Gj370IwCNnxRC0HKETJ8c8qaOV199VfMC0T+96ARwFStvlP58P+nAB+5mALBlf/PCCzbVD1w+n8fKygq2t7cxPj6O8fFxXL9+XfPzIrY6rcMh6ZwHPQt2dBec3N1AFlto/ZVemItEIlheXgbHcfB6vSUkTe5GyGit7u5uXL582dAGATKaKhqN4sSJE1KuAdEklSQZQtBaqv14PI75+Xmpbd7o5oa1tTXs7OxgZmYGXV1dVTsPgeq6tFHSh1GSRSPD6ScmJgCg7uEUBH/913+ND3zgAwAaPymEoOUIGTis7hiGKSEVQmrVTqQCGQNz3v9Gf39/SVtpJZCKmpZCSEW1urqK4eFh3HPPPZIHVm9nn55BpzzPS4uYWohYEASsra0hHA5jfHy8rAtODfKFOeAwuzeRSCAajWJ1dRW5XE6ayhxhCKIAACAASURBVH3s2DH09PQYdmKSiSgrKyuqNjatNjxS7dM2PJZlsbS0hFwuh1OnTtU8FkwNZIhAf38/rly5UrZoWKmppRJJG4lbPXFaL/S4LLRMCvnSl74Eu92Oj3zkI4YeZzW0HCGnUim8+uqrmJqaQnd3d8kHk3iRqxHyey4Wqt/Fxdq778h0aVINXrlypaQC09tqrSWknpycbW1tuHHjBhwOh9QC29bWpuhMEEURu7u7WF5elgih3hOGDrLp7u5GKBQCz/OYnJyE1WqVuvrIogw5vkAgALfbrYtMyKJaIBDQFV6kxYa3srKCTCYDnufR09ODkZER2Gw2w7rocrkcFhYWIIoizp49q1lKq6ZLk69oNAqr1QqO4+p2eBhZITci6U1PsFC1SSHf+9738OMf/xhPPfWU9D43elIIQcsRciAQwNWrVxVPGCMjOCttH4vFsLCwAI/Hg/PnzyueaHpP6EoELl+wGxoawvDwMFiWRSKRQCKRwM7OjuRMIARosViwtrYGv9+PCxcuGNrpRGutx44dK9GJSdqYKIpgGEZqvd7c3EQul4PT6SxpFlG6kJCqNZvN1m1jI6CrfY/Hg1gshsHBQQwODiKdTpccYz02PGK53NrawuTkZMlCZa2gSZphGCwsLEAQBJw5c0a6iJBKmnikyd9p0aWbcfIIDaMaQ37yk5/goYcews9+9rMSHbrRk0IIWo6QAe1e5GrQS8iCIGBxcRF2ux0nTpxQXXmvBWqEXGnBzul0oqenp2S1mWVZ7O/vY3l5GSzLwm63I51OY2VlRSKYWrulCA4ODiQ/cSWt1WKxwO12w+12l5ASTdLyC4nf70c6nZY6Kvv6+gy9PWcYBouLi+A4rkSe8Pv9JRkOtdrwYrEY5ufn0dPTY7jzQxRFKXxJjejltjs9iXhGvM6NHN9kRBbypz71KTAMg3e84x0ACgt7f/EXf9HwSSEELUfIRkdwVpsaAhROzmAwiL29PQwPD2N6elrzY2iFknNC74Idz/NYW1tDJBLB9PS0RNTEPpZIJLC7u4tsNiuNpyfVtBaSzmQyWFhYgMViwV133VVz5CZpFpBfSDY3N7G0tCRlIKytrSEWi5VcSGqt4vT6lfXY8EiVH41Gkc/ncfr0acN16GQyiZs3b0oLjmpkoUS0SouH9EWeTsmrt6mlkRWyEYS8tLSk+rtGTgohaDlCBtQDhoyukEnn2s7ODiYmJuD1ehty9Qfqc07Q8sHo6GhZY4eSfYzjOEnuIORCxzLSFSDP8wiFQojFYpiamtK8CKoVhOitVisuX74sET3HcdKFZHl5Gel0usSH3NbWBp/PV5VASEVfb9Wq9jouLy9jZWUFPp8PPM/j+vXrJe6Oemx4PM8jGAwikUhU9ENXQqXFw0wmIzlLjOg8bNT4plwuZ2jm9u1CSxKyGuoNqScgPtq1tTWMjIxIzonNzU1d+9fSCUhgs9nAcZwu5wRwaDOrJh/I4XA4VEmaVIDpdFrKbejt7TVMyyXI5/NYXl6WKno50TscjrJwdRIHmkwmsbq6ilQqJdn1yIWEZCPkcjksLi4in8/XVdGrgSw4tre3401vepP02htlwyPukNHR0Zq93Gqg7xhmZ2el6rNS5yFQnaQbmYV81KM3gTuQkBmmvGuu0vY0IRNHQjAYRE9PT1nbNTnJtYLowpUImXz4fT4fFhYWsLW1VeKcUBtUmUqlJD3bKLKhSfrg4ABLS0vo7OxEd3c30uk0lpeXpUpaLnfozRQmNja9rdp2ux2dnZ0lt6/5fF4i6Y2NDaRSKTAMA0EQ0N/fj9HRUUMXNElSXSqVUrxIabXhkchKedNNNpvF/Pw87HY7Ll68aHjsZCKRwM2bNyV3EP3a19J5SDe1mFnIldGShFxJsqi1Qo5Go1hYWIDP58OFCxcUG0xqDalXq4RoLa+7uxv33nsvcrlcmSuBto55PB6sr68jnU4bHlsJQOokA4DTp09LK9G0lqokJdAkTeQOJZJNJBJYWFiA3++veQafHDabDR0dHejo6JA8v8PDw+ju7kYqlVLN7wgEArqqOfpCcvz4cc1ebqC6DY803aTTaQiCgN7eXvT390skaFSAEJE/Kk3HUTp2QDkRj148zOVyiMfj6OvrA8uyqouHetGohcLbgdZ4FhpRi42NYRi8/PLLEEURp06dqvghrWWMk9LxVNKJ5a4EYh2LxWJYX19HPB6Hw+GA3+9HJBIBx3GSv7cekC64g4MDRfmAhpqUQOSOlZUVSe+lbWPhcBi5XA6zs7OGOlQAIJvNSguOtOeXvmCRKjWRSGB7exuLi4vS3Ql9MVG6SKRSKczPz8Pr9Rp2IZHb8KLRKIaHhzE4OChNj6ZDdupJwyPDe0dGRjA9PV03wdNES1+oJiYm0N7eXpYrXY8uHY/HWyLpDWhRQjbC9kbsT6lUCrOzs5oWqmohfJrAa1mwAwpWquXlZQwODuLcuXOwWq2SLYs+ad1ud5ncUQ2iKGJrawtra2sYHR3FlStXajpZ7Xa7IkmT44tGo3A4HHA6ndjY2NC1KFcJgiBgZWUFe3t7UrOQGugqlZj+q+V3+Hw+HBwcIJFIYHZ21vCQdDrEiHZn0CE7QKkNLxwOI5vNarLhsSyL+fl5CIJg6EQWglwuhzfeeEPKbVG6UFWSPLSM02qVLGSgRQlZDVoW9UgluLe3h4mJCcTjcd2t01pBKupaiZiErfv9/jItkVTSdBNGLpdDIpGQqmmGYeDxeEr0XpqkSShTR0eHYVWf/PiXlpbQ09MjNTCoLcrR7cx+v18TSRMCHRgYqDkyVC2/I51OY2NjA6urq3A4HLBarSVebjq/oxbQVaWWECOtNjwiHZFRSjs7O5iampI+J0aB9kSTbA411JqIR4i6VaaFAC1KyGof3EqESbqn1tbWcOzYMWlySCgU0vy4tbRDcxxX4vnUQsTZbBZLS0vgeV6zs8FiscDj8cDj8ZTEF9Ikvba2Jo3VYRgGdru9aixmLaDlA/mCY6VFuUQigbW1tTLnhJykiU3OZrM1pOrLZrNYXFyE0+nEvffeC6fTqZjfQcbS01ZBtUVYGplMBjdv3oTb7a7rQqiWhre3tyd9rm02mzS6jL7g1bPwlk6n8cYbb6Ctra1mG6EWkib/fvLJJxsS9HM70JKErAaiZ9EQRRE7OzsIhULo7e3F3XffXfMCgZ4KWRRFuN1uLC4uYn9/H+3t7VUbMGgdt9rttxbISZrsf39/H319fRBFEcFgECzLqs6+04NqNjY10Ity9L7kJE0WczmOw/j4OIaGhgyNqSSJffv7+5iZmSm5aND5HQMDAwBKL3hyvVcpv4OWV2ZnZw2/EBIr2+7uLk6fPi1VlcSGRxaK5TY8cpzVzgtBELC6uord3V3Mzc0ZXrXKSXp3dxe/+7u/C6vVim9+85uGPtbtgkVnPF9zjTFQAfHGKoEeAEoaAvx+P6amphRvMZ955hlcvXpV84ldbcCoXJ7geV46GRKJBNLpdFkokMvlQjgclho7hoeHDbX40COIRkZGMDw8XDa9glR/5ItEbNLHWcktsrOzg+XlZcX9G3H8RJ7o6uqC2+2WRgBZLJYy50Qtj00WvQYHBzE6Olrz8cvzOxKJBHK5HCwWCxiGQVdXF8bHxxX13npAWrb7+vpw/PjxqsdPL3AS2UPNhgccWuV6enowNjbW0LxmURTx+OOP4+GHH8YXv/hFvOc97zkKljdNB9iSFXK1N4eMlrdYLFWdE2pTQ/RCTSd2Op1li11E+4vH41hbW0MymYTT6URfXx8cDgdyuZzuVDQ1kCCk9vZ21dtjteovk8kgmUyW5SDTJJ3L5TA/Pw+fz9cQzywZ/+R0OnHhwoWyiypd/REPMoAyuUPttpocv8ViMUT+kOd3sCyLhYUFMAyD0dFRqQ2fzu+oJ2OE53ksLS0hnU7jzJkzmoPbad2eQG7DW1lZAcuykuQ2NjZm+IgsOXZ2dvDZz34WPp8P//Ef/1H3XMdmQ0sSshpyuRxyuRxu3LhRdsuphnoJuZYFO6fTCZfLhVgsBo/HgzNnzkixlYlEAltbW5L/mEgdWl0TBEQHzefzNWX80pYsOUmTUKDr168jn8+jvb0dHo8HqVSq7rFPBPl8HqFQCNFotKLOrdSEoYWkvV4vNjc3sb29jenp6brlITlo9woJAlJyQJBKWt6+riW/gzSY6PVEq0GefR2NRnHz5k0MDAzA7/cjlUqVxaoSUtdrw5NDEAT86Ec/wiOPPII//uM/xn333XcUqmLdaElClr9RJE9gf38fbrcbZ8+e1Vzp6F2oI6AXHfQs2JERT6lUCjMzMyVEQq+ik1tfuT7p8XhKKlT5hYTWcY3QoWkQTToSiSAej2N2dhZ9fX2S3EFkBXLrSx+nVt2edEuGQiGpi6+WKNNKJB0KhXBwcCD5qWl3ghFdZmSadiAQqNjOrpaNUS2/w2azYXFxETabrSF3JTzPY3FxEdlsVnEoLS3LyG14NElrlWW2t7fxO7/zO2hra8PTTz9t+MWxmdCSGjIZfkimYJDZbcPDw3jttdcwOTmpuQvp9ddfx+DgoOYkqWeeeQZXrlxRHCZaCfl8XhrhMz4+XlOsJL2IRL7oBTmywk50aKN1PjqkZ2xsTJW8iG2MHCPRJ30+X0n1Jycq0nzh8XgwNTVlONHQ0Ztzc3NwuVwl+r68m48cq1aSpqv6ubk5w5oZiFWQNLSkUim43W50dnaW5XfUCzJ04fjx4xgcHNT1GaVteIlEouRCR95z2ncuCAIee+wxfP3rX8eXvvQl/Oqv/upRroo1HXhLEjIh4lAohIGBARw/flw6uW/cuIHh4WHNK9gLCwvo7OzUFCIuiiJeeOEFtLW1oaOjA+3t7VVvz+kFr8HBQRw7dszwBa+dnR0Eg0HYbDYpNY6Qn94KVQnExgYAMzMzNeVm0A0YcpL2+/1IJpNSF5/Rq/fEM7uxsYHJycmKnlxBEGoiabIoODw8jJGREcOJJZVK4Y033kBHR4c0W44mP7ksozdljjSQiKKI2dlZwxLbaN85Oc7/+q//wgsvvICdnR309vbiL//yLxsyneMW484m5Ndffx3Hjx8vq6L0ECwAhEIhuN1uKV9ACbROzDAMotGo9AHL5/OqJ2ssFsPi4iICgQAmJiYMr/hImhnP85iZmZF0YrUKVW/lR9vA9NjYtILcNayvr8Pr9Uqvca0VqhLI4FLibqhlXzRJE3Ihx0lanu12O+bm5gz3RBMJ6uDgoOpQBDplTuliopTfQTeoVLtYGQFBEPD9738ff/M3f4Nz586B53m89tpr+MY3vlEyZPQI4s4lZKBwRVd6bqFQCB6PRxrIWQ1ra2sAgGPHjpX9TsuCHW0fisfjSKVS0nh3q9UqTb1ohF+WtAtrWYmmj5MQiyAIJdokue2ldVxS8RktfySTSczPz8Pv92NyclK601A7Tr0kzXGcpIPOzc0ZHhhPgnq2t7cRCATAcVxN3t5KODg4wMLCgnRnVUvVrfZ6kvyMSCQCn8+H2dlZwzs15dja2sJnPvMZ9Pf345FHHjEkcL6JYBKy0nNbW1uDxWLB6Oiopv1sbW2BYRiMj49LP6t1wY40XkQiEQwNDcFisUiVisViKZEQavGh0tWMEUQpvz0nlTTP8/B4PJiYmEBnZ6ehZExHV2oNGSKkEo/HpQpVFEXF23Pa3TA+Po7+/n7D5QPiyZVX3XJvbyKRkMivknYuB8dxWFhYAMuymJubMzzHmVTd4XAYbW1tYFm2JL9D7kGuF4Ig4G//9m/x7W9/Gw8++CB++Zd/+ShrxWq4swmZVCRykBVforNVw+7uLuLxuDSWST7eRssHhwTab2xsqDZ20Asz8XgcmUwGdru96tRoApJr0Sj5g3Z/DA0NIZ/PSxeTWqZ0yEE3p9SyYCSHktbL8zw4joPP55NSx4zM5iXTO5LJJObm5jQtHNM5yHLtXJ4wR683NOpiQrc9T05OSq8PbWkkx0p85/Xkd2xubuLTn/40hoeH8dWvfrVlQoIUYBKyEiHv7e1J3lUtODg4wM7ODubm5moKANrf30cwGER3dzfGxsZ03aLSY5Ti8Tiy2WxJ2y1ZpV9aWgLLspiZmTF0YgdQejEZGxvDwMBAxYsJ6Takxz1Va2ogkzXa2towMTFh+K0xIcp4PI7R0VHpYpJMJgGgTO6o5WJCJJxjx45Jdz+1QmmBk+M4cBwHt9uN8fFxdHZ2Gvo61dL2THdwkvdfa36HIAj4wQ9+gD//8z/HQw89hHe+852tWBXTMAlZiZBjsRi2trZw8uRJTfuJx+MIBoM4efKkNPVAyweHdAM6nU5MTU0ZtphDvMexWAw7OztgGAZ+vx89PT1Sk4hR1TGxsXV3d+te8KL9ssTiJK/47Xa71JU2Oztr+MWErijViJL2H9NuBLl2rkbS2WwWN2/ehNPpxPT0tOF3JiR/YmtrS9KJyetKZARa7qiFpI1se6atl+Q46bzmZ599FhMTE3j44YcxPj6Ohx56qGWS2qrgziZknucVGzpSqRSCwSDOnj1b8e+JRkwWf0g1FQgEJOLz+/1lJzjd+jo9Pd0QixYhmaGhIYyMjIBlWamKJreS9djaSBefKIqYnp7W3G5bDXRL+O7uLtLpNLxeL3p6eqTjNKolPJ1O4+bNmzV5lpVIms7EIBX/xsYGtre3q8ZL1gp6lJKSr5u4ZegLn1zrrUTSxBcdi8UMn4coP06GYbC7u4vf//3fx0svvQSbzYaJiQn82q/9Gj75yU825HGbDCYhKxEywzC4fv06Ll68qPh3lRbsSMIYIT761jwQCCCVSiESiUjOCaNvweLxOBYXF+Hz+TA5OalKMkTvI8ep1dZGuzMa0S5MnsP8/Dw6OzsxPj4uyQd00I5cltFzd0HbwIz0LNPpcvv7+1KgPn1nUm+YPv1Y9CRpPURJa7203EF3RgYCAaTTaczPz2NoaAijo6MNlwvW1tZw//33Y3JyEg8//DACgQB2dnYQiUQ0360ecdzZhKyW+JbP53Ht2jVFT2MtC3Ysy2J1dRVbW1twOBywWCwSoZATtV4TfS6XQzAYBMMwNevEtBOB1k9JJcXzPLa2tjAyMtIQGxvLslhcXATDMJidnVW1mdEt4eSLTDuhczuULkaki2x4eLghJEPulnK5nNTJR1enRixwNqKBhCZpcnfC8zw6OjrQ2dlZV6RqNQiCgO9+97v4q7/6KzzyyCN429ve1upasRpMQlabV/fMM8+URGTWMzpJqbGDzsCVty8TUtEiIdDt1BMTE4YnaeXzeWkxikwGli/G1RsDSS8K1nrnUK0l3O12IxwOSzquUV1k9OMTO2E1d4NS5xmZHUjLHXKSpkcpzc7OGt5AApS2PQ8MDCCbzZZcUOSuiXpJenV1FZ/61KcwNzeHr3zlKw2TRI4I7mxCJvqvEkhmca1ETCel0R1wapBXKHIJob29vSRrgF61b0Q7NXDo900mk5idnZUcG2QYqdpiXHt7u2adl0R71tMFpwainwaDQUSjUWklv9bQIjUQLdrr9WJqaqqmRTM1FwrxRzMMg+3t7YaMUgK0tz3Lc6+TySRYlq0aWCWHIAh49NFH8d3vfhdf//rX8da3vvVOrYppmIRciZCvXr2qm4g5jsPKyoohEztonyzp4CNyRyqVgt/vl26LjQSd26BmY5ODLBqSL7n9rr29veQ46ZCe2dlZwxYFaUQiESwuLmJgYEC6YBnVEg4c6umRSKQh+Rk8z0sXXTK8k77w6UlDU4MRbc9q1jZC0qSaJu//8vIy7r//fpw6dQoPPvig4R2QADA/P48PfOAD0v9DoRD+6I/+CL/9278t/ezpp5/GfffdJzV0vfe978Uf/MEfGH4sOmASspyQyYLdCy+8IGmS7e3tVXU++rbbCJ+pEhiGwcLCAv7/9s49qokzf+PPQAAFBArWS0ARCKDVChJAPa3VWq3VurRaW9HuaretVbcou7pVW39aqlXb1baupRat1W7tBa2nihcOuoq4vWgCLGhFETCgXCIilYBAgIT394e+szNJIJOQIMJ8zuEcSSbJDCbfvO/38jz19fXw9vaGVqvt0OrUFNS0lBbUOrJ6NEwh0Dwv3Q0EBQUJHk+39HULCgpACBEkZGTJSDiFBvuOuoO0d06055dr1WSqVdCSfm4uXLfn4OBgm/Ysm0ohvfvuu6ipqYFarcbSpUsxf/78dvVfbIVer4evry8UCgX8/f3Z2zMyMrBlyxYcPXrU7ucgkJ7rGAIYayJzC3bh4eG4c+cONBoN62xM3/g0SNMcHh3s6Nu3b7vatdZiLk/MXZ2q1WojYXpPT0+zW0gaxFpbWzFixAibrFgNXa2rq6tRUFAANzc39O3bF+Xl5SgpKWHb7zw9PTuk3UD7cdVqtWB9DoDvfEEVw7i7k/LycrbA6ebmxuoLP/roo3ZZ3XGtlAydsKn+MreFjhukaTulRCLhfaFwg7Qlbs/WYujFqFKp0NraiqioKIwfPx55eXlYsmQJDhw4YPdUxalTpxAUFMQLxg8y3XaFTDWRheaJDafi6uvr0dLSAhcXF/j7+6Nv3742rULTPHFxcTFv2y3kcU1NTWzBkFvg4nYhSCQS6PV6diVmSRCzhPZWrIYpBKrd4O7uzuvlNpdCoEHMnM6ytRBCUFpaiuvXr8Pb2xuEEDaF1NGRcArXSqmjYkbc9ypNIUkkEvTu3RsajQaenp4IDQ21+eLBEL1ej507d+Lbb7/F1q1b8cQTT9j19Uzx6quvIiIiAnFxcbzbMzIy8MILL8DPzw9SqRRbtmzB8OHDO/38OPTslAUNrF5eXmwQFvJtTQc76uvrMWTIELS2trLPRQcu6CraWunHuro6FBQUoHfv3ggKCupwntiw75gG6ZaWFnh7e8Pf3x8eHh423XrTbXdlZaVFwd6UYBHAH7ihgY/bKjd06FC75KLr6uqQn58PT09PBAYG8oJYW33n3HMVkkLgWil1VKPDFK2trVCpVKisrIS3tzeam5uNHDrMaaFYSlFREZYsWQK5XI7333/fLv835mhuboZUKkVeXh769+/Pu6+2thYODg5wd3dHamoq4uPjUVhY2OnnyKFnB2SlUonly5dDo9Fg6NChkMvliIqKQlhYmMm8oxDHDu6Kj3ZLEELYDyjNR7f1pudO8YWEhLCdDbaEO7JNW5u402aGgc+aDyjNsfbv31+Qg7E5uEMXXCEgnU6HAQMGwM/Pz+YuzHT4gr4/hKjKAcJGwmngowapDg4OCA0NtUufb3tjz9x0V11dHRoaGuDs7Gw0dGPJ31Wv1+Pzzz9HcnIytm3bhscff9zm1ySUlJQUfPbZZzhx4oTZY4cMGYKsrKz7aYraswMypaWlBXl5eTh37hwyMzORm5sLBwcHjBo1ChEREYiIiMDPP/+M/v37IyIiwuIiDjeY0FQH/YBytSVo/tNeU3wtLS1QqVSora018uKj0PYr7oqPrqLoubb3AaXOIAzDICQkxC69slRoqE+fPvDx8WGHWWwRTCi0H5cOwXT0/8IwhdDQ0IDW1lbodDr4+vrC19fXZiPhFGvHntvqmOGmZto614KCAixduhTR0dFYv369zWU/LSU2NhZTpkzBn//8Z6P7bty4wfaLK5VKzJo1C9euXbuf7XdiQDYFzQ9mZ2cjOTkZBw4cgJ+fH3x8fBAREQG5XI7o6OgOSRu2tLRAo9FAo9Hg1q1brMdZ//79WZ8zW1W9CSEoLy9HaWmp1T5n3CEWrVbLU+uiEpU0F22vkWraF11fX9+m0JCpc7XEeZu7Yg0JCbF5SyHwPysld3d3+Pj4sF/WHR0J53L79m2bjj0bTkZyz/XatWvw9fXF8ePHceDAAXz66ae8oSpbM2TIEDYVKJFIkJWVxbufEIL4+HgcPXoU169fx6lTpzB+/HgAQFJSEgBg0aJFSExMxOeff87m1j/++GO7nrcAxIDcHk1NTVi4cCHefvtthISEQK1WQ6lUsitpWgiTy+WIjIzEqFGjTIoJtQXNE/fq1QtBQUEghPCCCRWBocHEmnw0HbywRRsbxbCl6datW2hoaICrqysGDBhg0aSh0NejOshC+6K5j+U6bxtO8NEfR0dHdodiry8UroZGWwambbUKcr/82ktrcN2ehw0bZtcVKv27fvLJJzh+/Diqq6sxatQoREdHY/Xq1XbZHQHmUwupqan49NNPkZqaCoVCgfj4eCgUCruci40RA3JH0Ov1uHLlChQKBRQKBXJyctDS0oKRI0eyQfqRRx4xWulSIff6+vp288SG1k51dXWCc7xteeXZkoaGBly5cgVOTk6QyWRGE3zcXl46aWhpLvnOnTvIz883smnqCHSQgQbo27dvo76+nrXt8vLyYoO0reBaKVmS8jI3Es5Va+uI27M16HQ6JCYm4scff8Rnn32G6OhoVFRUIDs7267uz+YC8sKFCzFhwgTMmTMHABAaGoqMjAy79LzbGDEg25qGhgbk5ORAqVRCqVTi0qVL6NOnD+RyOcLDw5GTk4PBgwfjhRdesCpPzK3qc11DuC1iN2/etLizwdJzoMalISEhbfqatba28opbdXV1RpoNbX2h6HQ6qFQqaDQa3ti2LTFMgQDgCSsZDodYI0xPxYZoF4gtVqyG48s1NTVsd4dUKoW3t7dNdyimuHz5MpYsWYLx48fj3Xfftdtq2BRUfJ9hGCxcuBBvvPEG7/7p06dj1apVbDHxqaeewocffojIyMhOO0cr6dmDIfbA1dUVjz32GB577DEAdz88t27dQmJiIt555x34+/sjIyMD6enpiIqKglwuh1wuZ1vvzOHo6AgvLy+ejU1zczM0Gg3UajXy8vLg6OgIT09PdkVtq3w0IQRVVVVQqVSQSqVGQwuGODg4sJ0lFO4qWqVSsUVDbo5Xo9GguLgYgwYNQnBwsM1XWly9aH9/f4SGhrKv4e7uzhsOoV8oZWVlFvUd29NKiepx9O7dG4QQP+sZ+gAAGA9JREFUVp7S3d0dtbW1qKqqwtWrV1mbJ8PUTEfQ6XT45z//icOHD2P79u2IioqyyTVZws8//wxfX1/cvHkTkydPxtChQ+9Lf/P9QgzIHYBhGPTt2xe9e/dGdnY2pFIp2xOqUChw8uRJfPDBB6ivr8cjjzyCyMhIREZGYuTIkYILSs3NzSgtLYWLiwsee+wxODs7Q6vVsgVDlUrF6jVw89GWrPaoNq6LiwsiIiKsbs+SSCRGk2a0EHfr1i1cvnwZhBB4eNw1zqyurrap7CNNszg7O0Mul7f7vKa+ULg7lJKSEpNWVAzDsH+ryMhIuzgxc8eeua/h5ubGbs25Ka8bN26gsLDQ7Eh4e1y6dAlLlizBxIkT8fPPP9ul4CkE+oXZr18/zJgxA0qlkheQfX19UVpayv5eVlbGPqY7IKYsOoHm5mZcuHCBzUf/9ttvcHZ2xqhRo9ggLZPJeEFUSBsbhat1zBUq4haLTA0wUBds6jFoD4NJbgqEivQYypNyB26sWe1xtSHaS7NYA21p02g0qKysRENDA9zc3ODj49Ph9jtDOjr23N7QDTdIG77Ptm7dimPHjmH79u122fqXlpZi3rx5qKysBMMweOONNxAfH887JiMjAzExMfD394ejoyOmT5+O9PR0rF27Fs888wx73LFjx5CYmMgW9ZYuXQqlUmnzc7YDYg65q0I7LjIzM6FQKKBUKnH16lUMHDgQERER7CotISHBaiEjUz3Hzs7O7IRhc3Nzuy7YtoCKrZsrdpkauKGrPW7+3NTjb9++jYKCAvTr188mQyqmoMMXVEbUsMBpqlXQ0lV/W27PHaUtO6p9+/ahT58+SEtLQ0xMDNasWWO3VbFarYZarWbf23K5HIcOHeI5hWRkZGDdunWorq4GcPf9O3fuXKxevZrXzkYIQVxcHNLS0uDq6oo9e/Y8CPljQAzIDxaEEBw5cgQrVqyAt7c3XFxc2MIaXUWHh4cLVvsyBfU1KykpASEEEomEJ/5jq2KRLQZIuDleuuqnRUNPT0/07t0bZWVldpX45E7ztTd8YUm3hKnrtNTtuaNotVqsWbOG7Wa4ceMGfHx8cPz48U4ZnHjuuecQFxeHyZMns7d1QXU2WyMG5AeNkydPYtCgQWxXgE6nw+XLl9ne6JycHBBCEBYWxgZpoSIyOp2O9WmjKRDDFrHa2lro9Xqr29m4+hb26PfV6XTQaDQoLy/HrVu34OTkZOTEYspy3ho6aqVk6G1H/7bcL0BCCAoLC23i9iyU3377DUuXLsXUqVPxzjvvsCt52mtub0pKSvDEE0/g4sWLvO6aLigGZGvEgNzdoFv77OxstvWOGobS3uioqChemoMrUi5Ey5nmIbktYtzCFl2dGj4H7cW1lb6FKah7BzV5dXJyYrtQ6Eq6qamJFU+nQdqSwps9rZRorr+mpgbl5eVoaGjgeQVa288thObmZmzZsgUnT55EUlISwsPDbf4a5rhz5w7Gjx+P1atXY+bMmbz7uqAYkK0RA3JPgMp40oJhZmYm1Go1AgICMGjQIPzyyy9Yv349xo0bZ3U3A82Z0sDX0NDAjta6urri5s2bAO426dtjeswSJ2maPuCu+ulUJFeX2TBHy50YtNZdQwiGY8+EEJP93LYQgaKcP38e8fHxbA+vPUSOzNHS0oLp06djypQpWLZsmdnju4AYkK0RA3JP5c6dO1i8eDEyMzMxevRoFBQUQKvVYsSIEazq3fDhwzv0wWxsbIRKpcKtW7fY57GVGD0XqiwnlUqtdsOmOwvuqp+q9NG2u+vXr8Pd3R0ymcwuQxeWjD1zPfjogBDXQEGolGZTUxM2b96M06dPY8eOHRg5cqStL0sQhBDMnz8f3t7e2Lp1q8ljuqAYkK0RA3JPRafT4fDhw5gxYwb7hm5qakJubi6bj7548SJcXV0RERHB5qOF5jFNmZea0mQmhLD90UKssrhQSyt7uTDr9XrU1taipKQEGo0GTk5ORmpyHSmgcrHF2LMpRTnaNcMVVqLPnZubi/j4eDz//PNYsWKFXfqlKWlpaYiPj4der8frr7+OVatW8e5PT0/HU089BWdnZ0gkEvj7+2PLli24fv06gC4rBmRrelZA/uGHH5CQkIDLly9DqVTyWmE2bdqEL7/8Eo6Ojti2bRumTJli9Pji4mLExsaiuroacrkce/fuvS9bu86CEILbt28jMzOTDdIlJSXw8/NjA7RcLoe3tzcvqF+9ehVarRahoaFmNTRMdUqYssriBiiuCatMJsPDDz9sl+vXaDTIz8/ntcuZSs0YBj1LvhiEuj1bi6Gwkkqlwo4dO+Dm5oaysjIkJSXZfcqNOq//+9//hp+fH6KiovD999/zWtq2b9+OCxcuICkpCcnJyTh48CD27dtn1/PqgvSsgHz58mU4ODhg4cKF2LJlCxuQL126hDlz5kCpVKKiogKTJk1CQUGBUQ7xpZdewsyZMxEbG4tFixYhLCwMixcvvh+Xct+gXRLcfHRdXR1CQ0Ph5OSEa9euYdeuXR2SfDS0ympsbGR7eJ2cnKBWq+Ht7Y3AwECbWzUBllspGQY9WjTkBmnD1act3J6tISsrC6tWrYJMJkO/fv2Qk5ODqKgofPDBB3Z7zbNnzyIhIQHHjx8HcHfxAwBvv/02e8yUKVOQkJCAsWPHsqYDVVVV3SkdIYSepWUxbNgwk7enpKQgNjYWLi4uCAgIgEwmg1KpxNixY9ljCCFIT0/Hd999BwCYP38+EhISelxAdnBwQEBAAAICAhAbGwvgbpvUK6+8goceeghBQUF4+eWX4ejoyAr8R0VFITg4WHDwdHJygo+PD9sSR/O7RUVFqK2tZfuvGxsbO2yVZQjXSomrcdEeLi4uePjhh9mVOlf8hzu6TicNe/XqhbKyMvTq1ctuo9WGaLVabNq0CWfPnsUXX3zRqe1i5eXlGDRoEPu7n5+fkRwm9xgqllVdXd2dCnY2o9sE5LYoLy/HmDFj2N/9/PxQXl7OO6a6uhpeXl5sMcfUMT0VNzc37Nmzhy0I0a6A7OxsnDt3Du+//z4KCwvx8MMP81rvhAruULGcQYMGISwsDAzD8Cb31Go1a6JqOLkndIXFFaY3p3FhDir+Q/Whgf+1s5WUlEClUsHZ2ZlVgqPpmY4YpLZHVlYW/va3v2H27NnIyMiwu7GpiH15oP73Jk2ahBs3bhjdvmHDBjz33HM2f73Zs2fjypUrAO4Wsry8vJCbm2t0nDmXgweZwMBA3u9UI+PJJ5/Ek08+CeBukK6oqGAF/nfs2IGqqioEBwezincRERG89q3GxkZcuXIFEonEKEgyDAN3d3e4u7tDKpUC4FtlUeEfQ6us9vLRwcHBdluR0Wvx8PDAuHHj4OjoyDvfa9eu8fLn7fVzC0Wr1WLjxo1QKBT45ptv2twh2hshYj/0GD8/P3a4xx4mAd2BByognzx50uLHCHnD+Pj4oKamBjqdDhKJhD2GW3hYvnx5u/2vp0+f7rFbMIZh4OvrixkzZmDGjBkA7gbQ/Px8KBQKHDp0CGvXroVer8eIESOg1WrR2NiIbdu2Cc6vmpIm5Qr/VFRUsJoSVEuioqICXl5eiI6Otks+ur2x5/bOt7a2Fjdv3uT1c9MgLaTwRw1858yZg9OnT9/XVXFUVBQKCwtRXFwMX19fJCcns6k/SkxMDP71r39h7NixOHDgACZOnNjT8seC6TZFPcqECRN4Rb28vDzMnTuXLeo99dRTKCwsNPqAvvjii3jhhRfYot7IkSPxl7/8BcDdldbgwYORnp6O4OBgo9fshk3sduHXX3/FggULMHjwYHh4eCA/Px8eHh68VIevr6/VW3ua6rh69SpqamrY4NZRqyxTtOf2bAlNTU28VsGmpqY2NTAaGxvx/vvv47///S927NiBoUOHdvg6hPDWW2/hyJEjcHZ2RlBQEPbs2cP7oklNTcVf//pXFBcXw9vbGwMHDsTNmzeRlJSEmJgYaLVa/OlPf0JOTg68vb2RnJxstPPqAfSsLouDBw9iyZIlqKqqgpeXF8LDw9nK74YNG7B7925IJBJs3boVU6dOBQBMmzYNu3btglQqhUqlQmxsLH7//XeMGjUK33zzDfuB/s9//oNly5a1mYow53IgcpecnBy4urqyWh1U4F+pVLKqd2VlZfD39+e13nl6egpaUdEpOK66XEesskxhrduzUEzpiyQlJeHGjRsoLi5GTEwMNm7caPPXbY8TJ05g4sSJkEgkWLlyJQDgww8/NDpOXJi0S88KyNYiJC+9ePFiyGQyLF++3ORzlJeX81wOPv30U6P+z4SEBHzxxRdstX7jxo2YNm2a0XOZa7Lv7rS2tuLq1atsgM7KykJDQwNP4P/RRx/lbe0ttVIyZ5VFUweGQdrWbs9CaGhowLp165CXl4cnn3wSpaWlyMnJwd69e9kvts7k4MGDOHDgAL799luj+8SA3C5iQLYFOp0Ovr6+yM7Ohp+fn9njExIS4O7ujr///e+CbucipMm+J9Lc3Izz58+z/dEXL16Ei4sLwsPD4ejoiKtXryIxMbFD5p/U2YSuTLVaLdtv7ObmhqqqKjQ1Ndnd7ZnLr7/+irfeegvz58/HkiVL7JIHt5Q//OEPmD17Nv74xz8a3SfuFNulZ/Uh24uTJ09i6NChbQbj+vp6Vky9vr4eJ06cwNq1a616LaVSCZlMxubXYmNjkZKS0uMDsrOzM6KiohAVFYW4uDgQQnDx4kUsWLAAer0eAwcOxPPPPw+pVMr2RkdGRqJv376CA7SzszP69u3Lru6oSFFpaSkuXboEJycnNvhba5UllPr6eqxbtw4XL17E/v37TdYtbI2QneKGDRsgkUjw8ssvm3yOnu6HZwvEgGyG5ORk1nKcUlFRgddffx2pqamorKxkOwuoywHXcoZLYmIivv76a0RGRuKjjz4yshoS0mQvAjbIfvDBB5gwYQKAuwH0+vXrUCgUOHv2LLZt28ZaU3EF/oW2mrW0tKCoqAiEEIwdOxYuLi48qyyuMao5qyyhEELwyy+/YOXKlXj11VexdevWTlsVm+tg+uqrr3D06FGcOnWqzesz54cnYh4xZWFD2ltljBkzhl2xrVmzBmq1Grt37+Ydd+DAAaSlpWHXrl0AgL1790KhUCAxMZE9xlzFm9Kde6OFotPpkJeXx46B5+TkgGEYI4F/btCzdOy5PassS1rZ6uvrkZCQgPz8fOzcuRNBQUEdvn5bkZaWhmXLluHMmTNtaosY7hQnT55s5IfXwxFzyF2VkpISTJ8+HRcvXuTdLkQXQKx4Ww9ti8vKyoJSqURmZiauXLkCb29vyOVyBAQEICUlBf/3f/+H8PBwq8eeTelfGDqb0N5hQgh++uknrFq1CgsWLMDixYs7xTkEEF5olkqlqKqqgoODA3x8fBATE4OkpCTeTlGlUhntFFevXt0p1/GAIAbkroRarWYt3D/55BMoFAokJyfzjtHpdAgJCcGpU6fg6+uLqKgofPfdd21qE4gV745DhenXr1+PgwcPIiwsDBUVFQgMDGS1o0eNGoU+ffp0KBVh2MqmUChw5swZtLS0oKamBt988w1CQkJsfHXtIxaaOxWxqNeVWLFiBXJzc8EwDIYMGYIdO3YA4OejJRIJEhMTMWXKFOj1erz66qvtCsXs3r0bs2fPNnkfwzB4+umnxYq3GWgOeMCAAVCpVHB1dUVraysKCgpw7tw5HDlyBO+99x6am5uNBP6FrqC5+hcDBw4EIQQ1NTU4ePAgAgMDIZVKMXfuXLzyyiuIi4uz8xVbhlho7lzEFXIXRGjFOysrCz/++KPJlVt7vdHmep2bmpowb948ZGdnw8fHB/v27cOQIUNsf6EPEFqtlifwn5eXBzc3N57AvxAvwbq6OqxZswYlJSXYuXMn7+9KCOnUkeKEhAR89dVX8PDwaLPQLKSuISIIcYX8oGLPirder8ebb77J24LGxMTwVjxffvklHnroIRQVFSE5ORkrV67siYLiPHr16oUxY8awyoGEEPz++++swP/+/ftx7do1DBo0iDdlSPtyCSHIyMjAO++8gzfffBNJSUlGwdsewbi9L/fFixdjzZo1bKF5+fLlRoVmkc5FDMgPGGlpafjHP/6BM2fOtGnb3l5vtJAtaEpKChISEgAAs2bNYnt/RUGY/8EwDHx8fPDMM8+wnQStra0oKSnBuXPncPr0aWzevBl1dXUICQnBzZs30bt3bxw5cgSDBw/utPMUKsi1YMECTJ8+3eh2IeJcIrZDDMgPGHFxcWhqasLkyZMBAGPGjDGqeLfXGy0KitsPBwcHBAYGIjAwEHPnzgVwt5/5woULOHLkCNauXdtpHRRC4BaaDx48iBEjRhgdI0TNTcR2iAH5AaOoqMjk7VKpFKmpqQDuahifP3/e7udSWlqKefPmobKyEgzD4I033kB8fDzvmIyMDDz33HMICAgAAMycOdPqScYHEScnJ1YTujOwRMO7pqYGTU1NYBgGkyZNskmhWaRjiAG5h2FLQXGJRIKPPvoIERERqKurg1wux+TJk40q8OPGjcPRo0ftc0EiPCzR8C4qKjK56+F+uQN3VRFN9SeL2J6us38S6RS4W9Dm5mYkJycjJiaGdwwVFAfQrqD4wIEDERERAQDo06cPhg0bJlpfdREIIdi/f7/R2L9I10YMyD0M7hZ02LBheOmllzB8+HCsXbsWhw8fBgC89tprqK6uhkwmw8cffyzItbikpAQ5OTkYPXq00X1nz55FWFgYpk6diry8PJtfk4gxP/30E/r379+mMBHtU5fL5di5c2cnn51IW4h9yCId5s6dOxg/fjxWr16NmTNn8u6rra2Fg4MD3N3dkZqaivj4eBQWFpp8HnP6G4QQxMfHIzU1Fa6urvjqq6/YFXpPorM0vEVsirAWJUKIJT8iIjyam5vJ008/TT766CNBx/v7+5OqqiqL7yOEkGPHjpFnnnmGtLa2krNnz5Lo6Girzrm709LSQvr160dKS0sFHf/uu++SzZs32/msejyCYqyYshCxGkIIXnvtNQwbNgzLli0zecyNGzdA7u3ClEolWltbrXYcTklJwbx588AwDMaMGYOamhqo1Wqrz7+7IkTDu66ujv33iRMnTLa8iXQ+YkAWsZpffvkFe/fuRXp6OsLDwxEeHo7U1FQkJSUhKSkJwN2i4IgRIxAWFoalS5ciOTm5zQETc3lNUz3U3bmI+MMPP2D48OFwcHAwSt9s2rQJMpkMoaGhrDoghWp4FxcXY/To0ZDJZIiJiWF70SsrK/H4448jLCwM0dHRePbZZ0WZzK6C0KU0EVMWInamrKyMEEJIZWUlGTlyJDlz5gzv/meffZb89NNP7O8TJ04kmZmZRs+Tn59PwsLC2J8+ffqQTz75hHfM6dOniYeHB3vMe++9Z4cr6hiXLl0i+fn5ZPz48bzrzMvLIyNHjiRarZaoVCoSGBhIdDqd0eNffPFF8v333xNCCFm4cCHZvn17p527iBFiykLkwcKU/obh/ULGeENDQ5Gbm4vc3FxkZ2fD1dWVnVzkMm7cOPa4rjisMmzYMJNGpikpKYiNjYWLiwsCAgIgk8mM/laEEKSnp2PWrFkAgPnz5+PQoUOdct4i1iMGZJEugZC8ZkxMDL7++msQQnDu3Dl4enqyo79tcerUKQQFBcHf399u597ZCEndVFdXw8vLixXC7+7pne6COKkn0iVoS3+D5qIXLVqEadOmITU1FTKZDK6urtizZ4/Z5zXliUih/dFSqRRbtmy5LyPBQlrYRHoOlvYhi4g8MDAM4wygAsBwQkilwX0eAFoJIXcYhpkG4J+EEPvbO1sBwzAZAP5OCMm69/vbAEAI2XTv9+MAEgghZzmPYQBUARhACNExDDP23jFTOvv8RYQjpixEujNTAfzXMBgDACGklhBy596/UwE4MQzzoMjZHQYQyzCMC8MwAQCCAfCSyOTuSus0gFn3bpoPIKVTz1LEYsSALNKdmQPge1N3MAwz4N4qEgzDROPuZ6G6E8/NLAzDzGAYpgzAWADH7q2EQQjJA7AfwCUAaQDeJITo7z0mlWEY6b2nWAlgGcMwRQB8AHzZ2dcgYhliykKkW8IwjBuA6wACCSGae7ctAgBCSBLDMHEAFgPQAWgEsIwQ8uv9Ol8REUAMyCIiIiJdBjFlISIiItJFEAOyiIiISBdBDMgiIiIiXQQxIIuIiIh0EcSALCIiItJFEAOyiIiISBfh/wGwSNQRE0E1SgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b20a898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.gca(projection='3d')\n",
    "x1, x2 = np.meshgrid(np.arange(-10, 10, 1), np.arange(-10, 10, 1))\n",
    "y = b + w[0]*x1 + w[1]*x2\n",
    "ax.plot_surface(x1, x2, y, rstride=1, cstride=1, cmap=plt.cm.coolwarm, linewidth=0, antialiased=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We found a respectable cost applying linear regression to our dataset. But we can still do better. The problem with linear regression is the linearity. Sometimes, data is not so nicely shaped that a plane can fit it well. Real-world datasets are much more often irregularly shaped, with data being distributed along apparently curved surfaces. \n",
    "\n",
    "In order to give our function defined before, $F(X) = w_1*x_1 + w_2*x_2 + b$ more \"flexibility,\" we can complicate it by introducing a non-linearity. One good non-linear transformation is a sigmoid function, which is given below.\n",
    "\n",
    "$$\n",
    "\\sigma(z) = \\frac{1}{1 + e^{-z}}\n",
    "$$\n",
    "\n",
    "Let's implement the sigmoid function, and take a look at its graph.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x10b5259e8>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYXHWd7/H3t7p6ydJZSDr7RiAJSdgJq2xKAgEdwqhoGFEQRtQRL47cuQ8O8zCA17njODrjjDiIiggom8MSJJAAE0CRLYFsnQU6G+lOekln6yS9Vn3vH3Uai6Y6XUmq+lRVf17PU+lT5/yqzrdPn3z69O+cOj9zd0REpLBEwi5AREQyT+EuIlKAFO4iIgVI4S4iUoAU7iIiBUjhLiJSgBTu0mvM7AtmtjjX1mtmL5nZX3ezzMzsV2a2y8zezF6VKdf9rJld05vrlMJhus5dMsnMzgX+BZgJxIC1wLfc/a1QCzsIM3sJeNDdf5Fi2XnAQ8A0d9+fxRpuB45196uztQ7pW6JhFyCFw8wGAb8Hvg48CpQA5wGtYdZ1hCYCm7MZ7CLZoG4ZyaSpAO7+kLvH3L3Z3Re7+0oAM7vWzP7Y2djMLjaz9Wa2x8x+amYvd3aPBG1fNbN/M7PdZrbRzM4J5m81s/rkLgszG2xm95tZg5ltMbN/MLNIN+udY2brgvX+BLBU34yZXQ/8AjjbzPaZ2R1d3yto52Z2bDB9n5ndZWbPmFmTmb1hZscktZ1pZs+b2U4zqzOzvzezucDfA58P1rMiaPtBd5GZRYLvaUvwvd9vZoODZZOCGq4xs/fNbIeZ3XrYP0UpCAp3yaR3gZiZ/drMLjWzod01NLPhwO+A7wDDgPXAOV2anQmsDJb/FngYOB04Frga+ImZDQza/icwGJgMXAB8CfhyN+t9HPgHYDiwAfhYqhrd/ZfA14DX3H2gu/9jTxsgMB+4AxgKVAHfC9ZdDrwAPAeMCb6PF939OeCfgEeC9ZyU4j2vDR4fD77HgcBPurQ5F5gGXATcZmbT06xXCpDCXTLG3feSCBgHfg40mNkCMxuZovllQKW7P+7uHcB/ALVd2mxy91+5ewx4BBgP3Onure6+GGgDjjWzIhKB+h13b3L3zcAPgS8eZL2/c/d24N9TrPdIPeHubwbf12+Ak4P5nwJq3f2H7t4S1PpGmu/5BeBH7r7R3feR+KU438ySu1bvCP5aWgGsAFL9kpA+QuEuGeXua939WncfBxxP4gj131M0HQNsTXqdA9Vd2tQlTTcH7brOG0jiCLwY2JK0bAswNs31bk3R7kgk/7I4ENQIiV9OGw7zPcfw0e8vCiT/4uxuvdIHKdwla9x9HXAfiZDvajswrvOJmVny80O0A2gncfKz0wSgppv1ju+y3vEp2nVnP9A/6fWjDuG1W0l0qaTS02Vr2/jo99fBh38BinxA4S4ZY2bHmdnNZjYueD4euAp4PUXzZ4ATzOyKoGvhG8ChBOUHgm6bR4HvmVm5mU0Evg082M16Z5rZp4P1/q9DXO+K4PUnm1kZcPshvPb3wGgz+5aZlQa1nhksqwMmdZ4ETuEh4G/N7OjgPENnH33HIaxf+hCFu2RSE4mToG+Y2X4Sob4auLlrQ3ffAVxJ4pr4RmAGsJTDv2zymySOqjcCfyRxAvbeg6z3n4P1TgFeTXcl7v4ucCeJE6PvBetK97VNwBzgL0h0obxH4gQpwGPB10YzezvFy+8FHgBeATYBLSS+Z5GU9CEmyQnBEWs18AV3XxJ2PSL5TkfuEhozu8TMhphZKYnrvI3UXTgicogU7hKms0lcPbKDRFfFFe7eHG5JIoVB3TIiIgVIR+4iIgUotBuHDR8+3CdNmhTW6kVE8tKyZct2uHtFT+1CC/dJkyaxdOnSsFYvIpKXzGxLz63ULSMiUpAU7iIiBUjhLiJSgBTuIiIFSOEuIlKAegx3M7s3GNZrdTfLzcz+w8yqzGylmZ2a+TJFRORQpHPkfh8w9yDLLyVxZ70pwA3Afx15WSIiciR6vM7d3V8xs0kHaTIPuD8Y0eb14EZQo919e4ZqFJECF487rR1x2jritHbEaO2I0x6L0xF32pKm22NxYnGnI+7Eu351JxZ34g5xT8zvnHZ3PFiPA3EnMc/B6fzKh5536rxFS2ebzukPlieNs/Lh+Um63ObloukjOWn8kAxsue5l4kNMY/nwMGXVwbyPhLuZ3UDi6J4JEyZkYNUiEjZ3Z09zOw1NrTTub2PX/jZ2Hmhj94F29ra0s7e5nb0tHexvTTz2tcZobuvgQFuM5vYYre1x2mLxsL+NrDP78/SIQWV5Ee5pc/d7gHsAZs2apTuWieQBd6e+qZVNO/azpXE/mxsPsG13c/BooaGptdtwLimKMKhfMYPKogwojTKwNMrYIcX0K4nSv7iIfiVFlBUXURqNUFocoTSamC6JRiiNRiguihCNWOJrkRGNJL4WRYwiC74mP8yImBGJkPgaTBuGWee84HkkcY9pMwu+/rkd/DmMPzQvaN853Sk5uC35SYgyEe41fHgMynGkHrtSRHJceyzOu3VNrKzew6qaPbxb28T6uiaaWv48ml80YoweUsaYwf044+ijGDmojIryUoYPLKFiYClDB5QwtH8JQ/oXU1ZcFOJ307dlItwXADea2cMkhljbo/52kfzQEYvzztbdvL6hkdc3NbJsyy5a2hNH4YPKohw3ehDzTh7D1JHlTB4+kInD+jN6cBnRIl1Fnet6DHczewi4EBhuZtXAPwLFAO5+N7AQuAyoAg4AX85WsSJy5Pa3dvDiunpeXFvHknX17G3pwAymjxrE/NMncMqEIZw0bggTh/XPmS4GOXTpXC1zVQ/LncTI9SKSo+Jx508bGnn87WqeXV1Lc3uMowaUcPHMUVx03AjOPmYYQ/qXhF2mZFBot/wVkezb39rB75ZV86tXN7G58QDlZVGuOGUMV5w8llmTjqIooiPzQqVwFylATS3t/PwPm7jv1U3sbeng5PFD+PGcqVwyc5ROcvYRCneRAtLSHuPB17dw15Iqdh1o55KZI7nh/GM4beLQsEuTXqZwFykQf9qwg1ufWM2mHfs5b8pw/u6SaZw4LrsflJHcpXAXyXN7DrTzvYVreHRpNROH9ef+687g/Kk9DrEpBU7hLpLHlm3ZxY2/fZv6pla+dsEx3HTRFPqVqE9dFO4iecnduffVzfy/hWsZPaSMJ/7mHHXByIco3EXyTEt7jJsfW8EzK7dz8YyR/ODKkxjcrzjssiTHKNxF8sjelna+8uulvLFpJ7dcehxfPX+yPkUqKSncRfJE3d4Wrrn3TTY07OPH809m3sljwy5JcpjCXSQP1O1t4cq7X6NxXyv3Xns6503R1TBycAp3kRy3a38bX/zlGzTua+XBvz6TUyboA0nSM4W7SA7b19rBtb96k82NB7jvy6cr2CVtuimzSI5qj8W54f6lrN62l5/+1amcc8zwsEuSPKJwF8lR33tmLX/a0Mj3P3Mis2eMDLscyTMKd5Ec9NjSrdz3p81cf+7RfPa0cWGXI3lI4S6SY1Zs3c2tT67mnGOG8Z1Ljwu7HMlTCneRHLLnQDtff3AZFQNL+c+rTtFYpXLYdLWMSA65/elK6ppaefzr5zBsYGnY5Uge02GBSI5YuGo7T7xTwzc/cSwnjddNwOTIKNxFckB9Uwu3PrGKE8cN5hsfPzbscqQAKNxFQubu/P3jqzjQFuNHnzuJYvWzSwZoLxIJ2bOra3lhbT1/d8k0jh1RHnY5UiAU7iIhOtDWwf/9/Rqmjx7EtedMCrscKSAKd5EQ/XTJBrbtaeG782bqskfJKO1NIiHZtGM/97yykU+fMpZZk44KuxwpMAp3kRC4O3c8XUlJNMIt+hSqZIHCXSQEf6zawUvrG7jpoimMGFQWdjlSgBTuIr3M3fnBovWMHdKPL50zMexypEAp3EV62aLKOlZW7+Gm2VMojRaFXY4UKIW7SC+KxZ0fPb+eyRUD+PQpGuBasietcDezuWa23syqzOyWFMsnmNkSM3vHzFaa2WWZL1Uk/y1YUcO7dfv49pypuvRRsqrHvcvMioC7gEuBGcBVZjajS7N/AB5191OA+cBPM12oSL5rj8X5t+ffY/roQVx2/Oiwy5ECl86hwxlAlbtvdPc24GFgXpc2DgwKpgcD2zJXokhheGr5Nt7feYCb50wlErGwy5ECl064jwW2Jj2vDuYlux242syqgYXAN1O9kZndYGZLzWxpQ0PDYZQrkp/iceeeVzZw3KhyLpo+IuxypA/IVKffVcB97j4OuAx4wMw+8t7ufo+7z3L3WRUVFRlatUjuW7K+nnfr9vHVCyZjpqN2yb50wr0GGJ/0fFwwL9n1wKMA7v4aUAYMz0SBIoXgZy9vZOyQfnzqxDFhlyJ9RDrh/hYwxcyONrMSEidMF3Rp8z5wEYCZTScR7up3EQGWbdnFm5t3cv25R+te7dJretzT3L0DuBFYBKwlcVVMpZndaWaXB81uBr5iZiuAh4Br3d2zVbRIPvnZyxsY3K+Yz58+vufGIhmS1gDZ7r6QxInS5Hm3JU2vAT6W2dJE8t+Ghn08v7aOGz9+LANKNR699B79jSiSRQ+8toXiSIQvnT0p7FKkj1G4i2TJ/tYO/ntZNZedMIqK8tKwy5E+RuEukiVPLd9GU2sHXzxbd36U3qdwF8kCd+f+1zYzffQgTp0wNOxypA9SuItkwdvv72JdbRNfPGuiPrQkoVC4i2TBA69tobw0yryT9aElCYfCXSTDduxrZeGqWj5z2jhd/iihUbiLZNh/L6umLRbn6rMmhF2K9GEKd5EMcnceW1bNaROHcuyI8rDLkT5M4S6SQcu37qaqfh9XnjYu7FKkj1O4i2TQY8uqKSuO8MkTNdKShEvhLpIhLe0xnl6xjUuPH015WXHY5Ugfp3AXyZBFlbU0tXSoS0ZygsJdJEMeW1rNuKH9OGvysLBLEVG4i2RCze5mXt2wg8+cOk6DX0tOULiLZMCT79TgDp9Vl4zkCIW7yBFyd558p4bTJw1l/FH9wy5HBFC4ixyxdbVNvFe/j8tPHht2KSIfULiLHKGnlm8jGjE+eYKubZfcoXAXOQLxuPP0im2cN2U4Rw0oCbsckQ8o3EWOwLL3d1Gzu5l56pKRHKNwFzkCC5Zvo6w4wpwZI8MuReRDFO4ih6k9FueZVduZPX2k7tsuOUfhLnKY/li1g53729QlIzlJ4S5ymJ5esY1BZVHOnzo87FJEPkLhLnIY2jriPL+mjotnjqI0WhR2OSIfoXAXOQyvVu2gqaWDy04YFXYpIikp3EUOwzOrtlNeFuXcYyvCLkUkJYW7yCFq64izuLKWOTNGUhLVfyHJTdozRQ7Rqxt2sLelQ7cbkJyWVrib2VwzW29mVWZ2SzdtPmdma8ys0sx+m9kyRXLHs6u2U14a5dwpukpGclePn7wwsyLgLmAOUA28ZWYL3H1NUpspwHeAj7n7LjMbka2CRcLUHouzeE0ds2eM1FUyktPSOXI/A6hy943u3gY8DMzr0uYrwF3uvgvA3eszW6ZIbvjThkZ2H2jnMnXJSI5LJ9zHAluTnlcH85JNBaaa2atm9rqZzU31RmZ2g5ktNbOlDQ0Nh1exSIieXbWdgaVRzlOXjOS4TJ1QjQJTgAuBq4Cfm9mQro3c/R53n+XusyoqdAmZ5JdY3Hl+TR0fP24EZcXqkpHclk641wDjk56PC+YlqwYWuHu7u28C3iUR9iIFY+nmnTTub2PuTH1wSXJfOuH+FjDFzI42sxJgPrCgS5snSRy1Y2bDSXTTbMxgnSKhe66ylpJohAun6a9OyX09hru7dwA3AouAtcCj7l5pZnea2eVBs0VAo5mtAZYAf+fujdkqWqS3uTuLK+s4f8pw3d5X8kJae6m7LwQWdpl3W9K0A98OHiIFZ3XNXmp2N3PTbPU2Sn7QJ1RF0rCospaiiDF7ukZckvygcBdJw3OVtZwx6SgNgi15Q+Eu0oOq+n1U1e9j7vG6Skbyh8JdpAeLKmsBuHimumQkfyjcRXqweE0dJ40bzOjB/cIuRSRtCneRg6jd08KKrbu5WB9ckjyjcBc5iOfX1gFwibpkJM8o3EUOYnFlLZOHD+CYioFhlyJySBTuIt3Y09zOaxsamTNzJGYWdjkih0ThLtKNl9bX0xF3Lp6h/nbJPwp3kW4sXlPH8IGlnDL+I3evFsl5CneRFFo7Yry8voE5M0YSiahLRvKPwl0khdc2NLKvtUMfXJK8pXAXSWHxmjoGlBRxzjHDwi5F5LAo3EW6iAfD6V0wrYLSqIbTk/ykcBfpYnn1bhqaWrlEn0qVPKZwF+licWUd0Yhx4bQRYZcictgU7iJdLF5Ty9nHDGNwv+KwSxE5bAp3kSRV9fvY2LCfi2foKhnJbwp3kSTPr0ncKGy2wl3ynMJdJMniNbWcqHu3SwFQuIsE6ve28M77u9UlIwVB4S4S6Lx3uwbmkEKgcBcJLK6sY9Kw/kwZoXu3S/5TuIsAe1va+dOGHVw8c5Tu3S4FQeEuAixZV097zPWpVCkYCncR4LnVtYwo173bpXAo3KXPa2mP8dL6Bi6eqXu3S+FQuEuf98q7DTS3x9QlIwVF4S593qLKOgaVRTlrsu7dLoVD4S59Wnsszgtr65g9fSTFRfrvIIUjrb3ZzOaa2XozqzKzWw7S7jNm5mY2K3MlimTPm5t2sqe5XR9ckoLTY7ibWRFwF3ApMAO4ysxmpGhXDtwEvJHpIkWy5bnVtZQVR7hgakXYpYhkVDpH7mcAVe6+0d3bgIeBeSnafRf4PtCSwfpEsiYedxZV1nLB1Ar6lWg4PSks6YT7WGBr0vPqYN4HzOxUYLy7P3OwNzKzG8xsqZktbWhoOORiRTJp2fu7qG9q5bITRoddikjGHfEZJDOLAD8Cbu6prbvf4+6z3H1WRYX+DJZwPbNyOyXRCJ84TsPpSeFJJ9xrgPFJz8cF8zqVA8cDL5nZZuAsYIFOqkoui8ed51bXcv6UCsrLNJyeFJ50wv0tYIqZHW1mJcB8YEHnQnff4+7D3X2Su08CXgcud/elWalYJAPe2bqL2r0tfPJEXSUjhanHcHf3DuBGYBGwFnjU3SvN7E4zuzzbBYpkwzMraykpinDRdA3MIYUpmk4jd18ILOwy77Zu2l545GWJZE887jy7ejvnTx3OIHXJSIHSR/Kkz1levZvte1q49HhdJSOFS+Eufc7CldspLjJma6xUKWAKd+lT3J1nV9dy3pQKBvdTl4wULoW79Clvv7+Lmt3NfFIfXJICp3CXPuWp5dsojUa45HhdAimFTeEufUZ7LM4zK7cze8ZIBpamdaGYSN5SuEuf8WrVDhr3tzHvpDFhlyKSdQp36TMWrNjGoLIoF0zTfY2k8CncpU9oaY+xaHUtl50wmtKobu8rhU/hLn3Ci2vr2d8W43J1yUgfoXCXPuGp5TWMKC/lTA2CLX2Ewl0K3u4Dbby0voFPnTiGooiFXY5Ir1C4S8FbsGIbbbE4nzltbM+NRQqEwl0K3mNLq5kxehAzxwwOuxSRXqNwl4K2rnYvq2r2cOWscWGXItKrFO5S0B5bWk1xkTHvZHXJSN+icJeC1R6L8+Q7NcyePpKjBpSEXY5Ir1K4S8H6n3X1NO5v47OnqUtG+h6FuxSsx5ZWU1FeygVTdbsB6XsU7lKQ6ptaWLK+nk+fMpZokXZz6Xu010tBeuTNrcTizudPHx92KSKhULhLwemIxfntm+9z3pThTK4YGHY5IqFQuEvBeXFdPdv3tHD1WRPDLkUkNAp3KTgPvr6FMYPLuOi4EWGXIhIahbsUlI0N+/jDezv4qzMn6ESq9Gna+6WgPPj6+xQXGZ/TiVTp4xTuUjAOtHXw2LKtzD1+NCPKy8IuRyRUCncpGL9bVk1TSwdfOlsnUkUU7lIQOmJxfv6HjZw6YQizJg4NuxyR0CncpSAsXF3L1p3NfPWCYzDTaEsiaYW7mc01s/VmVmVmt6RY/m0zW2NmK83sRTPT38XSa9ydn728gckVA5gzfWTY5YjkhB7D3cyKgLuAS4EZwFVmNqNLs3eAWe5+IvA74F8yXahId16taqRy216+ev5kIhojVQRI78j9DKDK3Te6exvwMDAvuYG7L3H3A8HT1wHdY1V6zd0vb2BEeSlXnKIBOUQ6pRPuY4GtSc+rg3nduR54NtUCM7vBzJaa2dKGhob0qxTpxsrq3fyxagfXnXs0pdGisMsRyRkZPaFqZlcDs4AfpFru7ve4+yx3n1VRoXtsy5H74eJ3GdK/mL86c0LYpYjklHTCvQZI/rjfuGDeh5jZbOBW4HJ3b81MeSLde2vzTl5+t4GvXXAMg8qKwy5HJKekE+5vAVPM7GgzKwHmAwuSG5jZKcDPSAR7febLFPkwd+cHz62noryUa86eFHY5Ijmnx3B39w7gRmARsBZ41N0rzexOM7s8aPYDYCDwmJktN7MF3bydSEa88t4O3ty8k29+4lj6laivXaSraDqN3H0hsLDLvNuSpmdnuC6Rbrk7/7poPWOH9GP+6eprF0lFn1CVvPPMqu2sqtnDt2ZPoSSqXVgkFf3PkLxyoK2Df3pmLdNHD+IvdV27SLcU7pJX7lpSxbY9Ldw5b6YG4xA5CP3vkLyxacd+fv7KJj59ylhOn3RU2OWI5DSFu+QFd+eOpyspiUa45dLjwi5HJOcp3CUvLKqs5aX1DXxr9hRGDNIoSyI9UbhLztuxr5Vbn1jNjNGDuOacSWGXI5IX0rrOXSQs7s6tT6yiqaWD337lZIp1ElUkLfqfIjnt8bdrWFRZx/++ZCrTRpWHXY5I3lC4S86q2d3M7QsqOWPSUVx/7uSwyxHJKwp3yUmtHTG+8Zu3ibvzr1eeRJFGWBI5JOpzl5zj7tz2ZCXLt+7m7qtPZcKw/mGXJJJ3dOQuOec3b7zPI0u38o2PH8Pc40eHXY5IXlK4S055a/NO7ni6kgunVfDtOdPCLkckbyncJWes2baX6+97i/FD+/Pj+aeon13kCCjcJSds2rGfL937BgNKo9x//RkM7qdh80SOhMJdQrdtdzNX/+IN3OGB689k3FCdQBU5Ugp3CdWGhn1cefdr7G1u59fXncGxIwaGXZJIQdClkBKad97fxXX3vUXEjN9+5SyOHzs47JJECobCXULxP+vq+MZv3qGivJT7rzuDScMHhF2SSEFRuEuvisWdf3/hXX6ypIqZYwZx77WnM6Jct/AVyTSFu/Sa+qYWbnpoOa9tbOTzs8Zzx7yZlBUXhV2WSEFSuEvWuTtPvFPDd3+/hub2GP965Ul89rRxYZclUtAU7pJV7zce4NYnV/GH93Zw6oQhfP8zJzJlpG7dK5JtCnfJisZ9rdy1ZAMPvr6FkmiE786byRfOnEhEnzoV6RUKd8mohqZWHnhtM7/84yaa22Ncedp4/nbOVEYN1klTkd6kcJeMWLNtL796dRNPLd9GWyzOpceP4uaLp+lDSSIhUbjLYWtoamXBim08/nY1ldv2UlYc4XOnj+PLHzuaYyoU6iJhUrhL2tydqvp9vLiunhfW1PH2+7uIO5wwdjD/+BczuOLksQwdUBJ2mSKCwl0Ooj0WZ31tE8u37uaNTTt5fWMjDU2tAMwcM4gbPzGFvzhxtK5+EclBCnchHnfqmlrY2LCfd+uaeLduH+tq97Jm215aO+IAjCgv5ezJwzhr8jAumFbB2CH9Qq5aRA4mrXA3s7nAj4Ei4Bfu/s9dlpcC9wOnAY3A5919c2ZLlcPR2hFj94F2duxrpaEp8ahvaqVmdzPbdzdTs7uZLY0HPghxgMH9ipk2qpyrz5rIieMGc9K4IUwc1h8zXcYoki96DHczKwLuAuYA1cBbZrbA3dckNbse2OXux5rZfOD7wOezUXA+i8edmDuxeOLR0fk1Fqc97sRiTns8TnssTnuH0xaL0doRpy14tHTEaWmP0doeo7k9xoG2GM1tMfa3dbC/NUZTSwf7WtvZ09zB3uZ29jS3s6+1I2UtQ/sXM2ZIPyYOG8AFUyuYOGwAk4YNYOrIgVSUlyrIRfJcOkfuZwBV7r4RwMweBuYByeE+D7g9mP4d8BMzM3f3DNYKwKNvbeWeP2z84Hl3q/BunnROuvuH2nS+jeO4Jz1PaueeWB7/YHnndKJNPJ54bdwT82PueBDm8YxviYTSaIQBpVEGlkYZUBqlvDTK2CH9mD66nMH9ihk2oIShA0o4qn8JIwaVUjGwjIryUvqV6J4uIoUsnXAfC2xNel4NnNldG3fvMLM9wDBgR3IjM7sBuAFgwoQJh1Xw0AElTOt6Aq+bg8zk2clHovbBvNRtLPjHsA/adL7cMCKRYMogktQuYkbEEtNFkT/PKzIjEjEiBtFIYrrIjGhRhGjEKIoYxUVGUSRCcZFRUhShuChCtMgojRZREo1QGjzKiosoLY7QvyRKv+IijTMqIin16glVd78HuAdg1qxZh3UsO2fGSObMGJnRukRECk06w+zVAOOTno8L5qVsY2ZRYDCJE6siIhKCdML9LWCKmR1tZiXAfGBBlzYLgGuC6c8C/5ON/nYREUlPj90yQR/6jcAiEpdC3uvulWZ2J7DU3RcAvwQeMLMqYCeJXwAiIhKStPrc3X0hsLDLvNuSpluAKzNbmoiIHK50umVERCTPKNxFRAqQwl1EpAAp3EVECpCFdcWimTUAWw7z5cPp8unXHJKrteVqXZC7tamuQ5erteVqXXDotU1094qeGoUW7kfCzJa6+6yw60glV2vL1bogd2tTXYcuV2vL1boge7WpW0ZEpAAp3EVEClC+hvs9YRdwELlaW67WBblbm+o6dLlaW67WBVmqLS/73EVE5ODy9chdREQOQuEuIlKAcjbczexKM6s0s7iZzeqy7DtmVmVm683skm5ef7SZvRG0eyS4XXE26nzEzJYHj81mtrybdpvNbFXQbmk2aumyvtvNrCaptsu6aTc32I5VZnZLtusK1vkDM1tnZivN7AkzG9JNu17ZZj1tAzMrDX7OVcE+NSlbtSStc7yZLTGzNcH/g5tStLnQzPYk/YxvS/VeWarvoD8bS/iPYJutNLNTe6GmaUnbYrmZ7TWzb3Vp02vbzMy+zQfuAAAEpUlEQVTuNbN6M1udNO8oM3vezN4Lvg7t5rXXBG3eM7NrUrXpkQfjfObaA5gOTANeAmYlzZ8BrABKgaOBDUBRitc/CswPpu8Gvt4LNf8QuK2bZZuB4b24/W4H/ncPbYqC7TcZKAm264xeqO1iIBpMfx/4fljbLJ1tAPwNcHcwPR94pBe20Wjg1GC6HHg3RV0XAr/vrX3qUH42wGXAsyRGqTwLeKOX6ysCakl84CeUbQacD5wKrE6a9y/ALcH0Lan2feAoYGPwdWgwPfRQ15+zR+7uvtbd16dYNA942N1b3X0TUEViEO8PWGIw1E+QGKwb4NfAFdmsN1jn54CHsrmeDPtg8HN3bwM6Bz/PKndf7O4dwdPXSYzuFZZ0tsE8EvsQJPapiyx5UN4scPft7v52MN0ErCUxVnG+mAfc7wmvA0PMbHQvrv8iYIO7H+6n4I+Yu79CYnyLZMn7Une5dAnwvLvvdPddwPPA3ENdf86G+0GkGrC7604/DNidFCCp2mTaeUCdu7/XzXIHFpvZsmCg8N5wY/An8b3d/PmXzrbMtutIHOGl0hvbLJ1t8KEB4IHOAeB7RdANdArwRorFZ5vZCjN71sxm9lZN9PyzCXvfmk/3B1phbTOAke6+PZiuBVINCJ2RbderA2R3ZWYvAKNSLLrV3Z/q7Xq6k2adV3Hwo/Zz3b3GzEYAz5vZuuA3e1bqAv4L+C6J/4TfJdFldN2RrC9TtXVuMzO7FegAftPN22R8m+UbMxsI/DfwLXff22Xx2yS6HfYF51SeBKb0Umk5+7MJzq9dDnwnxeIwt9mHuLubWdauRQ813N199mG8LJ0BuxtJ/BkYDY60UrVJW091WmJQ8E8Dpx3kPWqCr/Vm9gSJ7oAj+s+Q7vYzs58Dv0+xKJ1teVjS2GbXAp8CLvKgozHFe2R8m6VwKAPAV1svDgBvZsUkgv037v541+XJYe/uC83sp2Y23N2zfoOsNH42Wdu30nAp8La713VdEOY2C9SZ2Wh33x50U9WnaFND4txAp3Ekzj0eknzsllkAzA+uYDiaxG/dN5MbBGGxhMRg3ZAYvDubfwnMBta5e3WqhWY2wMzKO6dJnFBcnaptpnTp3/zLbtaXzuDn2ahtLvB/gMvd/UA3bXprm+XkAPBBn/4vgbXu/qNu2ozq7Ps3szNI/H/ujV866fxsFgBfCq6aOQvYk9QdkW3d/hUd1jZLkrwvdZdLi4CLzWxo0J16cTDv0PTGWePDeZAIpGqgFagDFiUtu5XEFQ7rgUuT5i8ExgTTk0mEfhXwGFCaxVrvA77WZd4YYGFSLSuCRyWJrolsb78HgFXAymCHGt21ruD5ZSSuxNjQG3UF66wi0ae4PHjc3bW23txmqbYBcCeJXz4AZcE+VBXsU5N7YRudS6JLbWXSdroM+FrnvgbcGGybFSROTJ/TSz+/lD+bLrUZcFewTVeRdMVblmsbQCKsByfNC2WbkfgFsx1oD7LsehLnal4E3gNeAI4K2s4CfpH02uuC/a0K+PLhrF+3HxARKUD52C0jIiI9ULiLiBQghbuISAFSuIuIFCCFu4hIAVK4i4gUIIW7iEgB+v+0edQcU9EjCQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b2eadd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def sigmoid(z):\n",
    "    return 1.0 / (1.0 + np.exp(-z))\n",
    "\n",
    "x = np.arange(-10.0, 10.0, 0.2)\n",
    "sig = sigmoid(x)\n",
    "plt.plot(x, sig)\n",
    "plt.title('Sigmoid function')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basically, a sigmoid just squashes any input it gets to between 0 and 1.\n",
    "\n",
    "So we now modify our basic function in the following way: instead of just outputting the weighted sum by itself, $b + \\sum w_i x_i$, we now pass that through the sigmoid function. \n",
    "\n",
    "In graphical terms, this looks like:\n",
    "\n",
    "![2 input neuron sigmoid](https://ml4a.github.io/images/figures/neuron2sig.png)\n",
    "\n",
    "So instead of $y = w_1 x_1 + w_2 x_2 + b$, our function now looks like:\n",
    "\n",
    "$$\n",
    "z = w_1 x_1 + w_2 x_2 + b \\\\\n",
    "y = \\sigma(z)\n",
    "$$\n",
    "\n",
    "Thus, the full equation is:\n",
    "\n",
    "$$\n",
    "y = \\frac{1}{1 + e^{-(w_1 x_1 + w_2 x_2 + b)}} \n",
    "$$\n",
    "\n",
    "But it's easier to imagine it as the two-step process we saw before.\n",
    "\n",
    "Suppose we use the same parameters as before:\n",
    "\n",
    "$$\n",
    "w = \\left[0.2, 0.6\\right] \\\\\n",
    "b = -0.3\n",
    "$$\n",
    "\n",
    "We can compute the predictions like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "829.4187739184515\n"
     ]
    }
   ],
   "source": [
    "def weighted_sum(x, w, b):\n",
    "    return b + np.dot(w, x)\n",
    "\n",
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "# reset our parameters\n",
    "w = [0.2, 0.6]\n",
    "b = -0.3\n",
    "\n",
    "X, y = data, labels\n",
    "\n",
    "# get weighted sum like before\n",
    "Z = [weighted_sum(x, w, b) for x in X]\n",
    "\n",
    "# now transform the weighted sums with a sigmoid\n",
    "y_pred = [sigmoid(z) for z in Z]\n",
    "\n",
    "# evaluate error\n",
    "error = cost(y_pred, y)\n",
    "print(error)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That migh look much worse, but we haven't optimized yet!\n",
    "\n",
    "Given the above weights, we can take a look at the graph of our neuron's activity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x10b20a208>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAADuCAYAAAAOR30qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYW3W9/9/ZZrLN3nams7Szd19npguoFFEriGVTKXq94H1A4EqpgijKFXsRBGWHCuUCwr2iFkUR1P4KIlQBodOWVuk2SzL7vmbfTnJ+fwzn9CRz9pyTZNLzeh4e2uTk5EwneeeT9/f9/Xx0JElCQ0NDQyP96NN9ARoaGhoaM2iCrKGhoZEhaIKsoaGhkSFogqyhoaGRIWiCrKGhoZEhaIKsoaGhkSFogqyhoaGRIWiCrKGhoZEhaIKsoaGhkSEYJR6vbevT0NDQkI5OzEFahayhoaGRIWiCrKGhoZEhaIKsoaGhkSFogqyhoaGRIWiCrKGhoZEhaIKsoaGhkSFogqyhoaGRIWiCrKGhoZEhaIKsoaGhkSFogqyhoaGRIWiCrKGhoZEhaIKsoaGhkSFIbS6kocELSZKIRqMAAIPBAJ1OVE8VDQ0NaIKsoRCxWAzRaBQEQSAUCtG363Q6GAwG+j+9Xg+9Xg+dTqeJtYZGApogayRFLBYDQRB0VazT6WjBJcmZbq2UUCc+LhAIoKioCEajURNqDQ1ogqwhA5IkQZIkIpEIYrEYANBCSokwdRvz/0zC4TC6u7ths9kQDofjHqPX62EwGDSh1jjr0ARZQzQkSdIVcaIQS4V6jMFgYH2OaDQaJ9TUsYn2hybUGtmEJsgagiQKMSWCyQhhYjXNvJ3tvEyhJkky7hiXy4X58+fTlTWzqtbQmEtogqzBCUmSIAiC9n+VEGIKLkHmO55LqDs7O1FQUDBLqCmBZltQ1NDIRDRB1pgFFV0jCALHjx9HVVUVCgsLFX0OqYLMdx4qycGE6XOHw2FNqDXmBJoga9BQFTHTFkgUOjFkgrDxVdQAQBAEIpFI3H1utxuFhYXIycnRhFojLWiCrMFqTej1evrP1AKekihVIct5Xub/KUiSRF9fH8xm86yfl/pgYqY+tE0vGmqgCfJZDHMzB8BeVer1elWEM12CzAV1PZTYUiRmqalvDsxjtU0vGkqhCfJZCNtmDi4BkVMhJy6u8R2X6fBV1AD7pheCIBAOh1FUVKQJtYYkNEE+S2AucrW3t6OyshIWi0VQIJKtkE9+5lOc91kBfBjhF/tVb70p+7nlIFYw+YQ6GAxicHAQVqt11mO0TS8afGiCnOWwbeYIBAKIRqOiRECuh0ySJE5t/TTvMVEBMSaCBI5u/gT7+aNnPiQMAND6ttRLnH1OhVIfAGZZH9T5tU0vGnxogpyl8G3mMBgMokVWr9dLEmS2LdRsiBFjLphiTPHBho/Hnz8w+/wtH77L+5xKwWXZyNn0Eg6HkZOTA4vFQlfUWvIje9EEOctgZoipN3eiEEhZUJO7+MZXHSstxnHnZhFiikOrzgUAxAjuc+j+9Tfe84tBrIdOPyePUA8MDKCgoABFRUValvosQBPkLIFNiKnoWiJSql6pFTLA7xunS4wBfiGmOLKM3SIBgE2d7wk+HpAuyFww+30YjWfeqtqml+xFE+Q5DpUh7u/vR1lZGa8QU0gRWakVsvPLFyO/PJ/z/kggwntfbl4O631sQh2cPtN3WQkxJiP8x/yjehPv/ed0vz9zHoUEGZhJxCT+PuVsetHr9SAIAmazGSaTSRPqDEUT5DlK4maO7u5ulJeXi3qs1Ao58c0tFyExloq5MBfATNVtsnEfxxTz8PRsYRcSYkCcoL9bsSHu78OMP5870Cr4eDaovLMYhCJ6bW1tWLx4cVz6gxJmo9GoVdQZgCbIcwwxmzmEUKtCdn75Ys77khFjPhtDqgWSUxj/kueyQcKTM48TV1kL/1smijWFkFBTC7LJQD2eJElaeKm/A9qml0xCE+Q5gpTNHEKo7SEriZJinAifJ51TbEQ0EANfJ4+IJypKjPmghJpN+D8+cojVspBL4rnkbHqhLLHEqloTamXQBDmD4ZvMwXW8mDeFGoKsRnUsJKh8JCPGgLAnDQAGix6wcIsl4Ra+fr4K/O3SFgDAJMt9Hx85JHjuRKLRqKhmUUJCHYvFcPr0aSxYsAAFBQX0sdqml+TRBDkDSUxMAMIVMZUtFvOGU3NRLxG1xJivOlZbjMUuEBos3L+LaCAq6jxcUGKdCJ9QJ1ttM4WaJEmYTKY4+0Pb9JI8miBnEMzNHP39/YhEIqiurhZd9YqtgKQs1IkRb77qmElRdUnc38PeIOexYV+Y8z65SQ2AX6yD06GUpDViBAmdSQ+DifuYaCAq+DxssAk1dT2W155UTAQTX2tiNr0wb6M2KCV61Gf7gqImyBkAW4bYaDQiFAqJfnGq5QuLrZCL68tm3Rb2+DmPV0OMhRCqnE02k+i0BjA7saFUWoOMxKA3cv/epVTWzGsKbL0RbBvMPzF5WPT5KKTYH0JCPTk5Cb/fj8rKSgBnd5ZaE+Q0wreZw2AwxFUVQqi1HZrvWIIg4L3rBlYxTgfJiLWcBUJmYoPPBlE6scEl1szzi/lwoPh7cTPr7XxCHY1Gk7Y/KIGlXucGg+Gs3/SiCXIaYJvMkfjilirIlGUh9thkKuRIJIKenh6MjIxgBcfjUl0dpzI6l4iQJy0msRGeTD7rrTfqaFHWmRIW5CQINAWXUAOA9fU9sqbJsBGNRumdiHI3vWSLUGuCnEL4JnMkIqXiBdSzLJjHhsNh9PT0YHR0FFVVVVjx2nOiry+dZHpaI0aQMObzvxWTTWwkCjQTOWLt/8wNeIfldrn2R04Ot+8PCCc/uIT60Ucfxe233w6TicewzyA0QU4BsVgMkUgEfX19qKioELXCLKXiBdSzLHQ6HQiCQHt7O8bGxrB48WJs3rwZer0e06Kv7gypro4zOa0BCNsYlFjyJTYiHvkfOAC3WCtZVfMJNUEQsFgskp8LEBbq3/3ud7jjjjtknTsdaIKsIszNHNTMtqqqKlGPlWNZKC3IoVAI3d3dmJycRGlpKerr60X5hnx2hRwowU1MaQD8Ag8kt0AoV+jFpDUA8WIsdA6heJ1c2IRajkgD/ELNtCyUgmm1zSXrQhNkheHazCF1AUSOZaGUhxwMBtHV1YWpqSlUVFQgGAyioqIi7pjpO68VfW0UQtUx1+KgXIFPhxgDyac1AOUSG0qlNcRek1QooR5iuU+O/cGGJshnIWyTOZIJv6tZIXNNAQkGg3A6nXC5XKiursbSpUsRDocxNjYm+jrkiqccMRaqjuWSTFpDDunqryFWrNUQYjFQYi1XmAmCUGzhMVVogpwkfJM5kkFtD5mZnAgEAnA6nXC73aipqcGyZcviRhElnldOdWwrn4dwez8AoKixMu6+iMIWB5CetAaQHf01gDNiPbOJJfnEhhySrZBdLhe9tXuuoAmyTMRM5kgGqcNF5VgWfr8fTqcTXq8XtbW1WL58+azrl7J1mq2aLVxWCwCIuNyzhDgZ5C4OJkM6xRhIXX8NQF5iQ0mhVsKucLvdKCwsVOBqUocmyBJhCvEHH3yA9evXS+pXq2T3LiZSBDkYDMLtduPDDz9EbW0tVqxYwflBIsUKocRXCnzVsdKLg0B60hpiHp+qxIaYBUC5PTaUiNYp5RsDwPT0tFYhZytsmzmkbG0GAKPRmPQOJy4MBoNgfwqv1wun0wm/3w+TyYQNGzYIXn9ihZxoV+Qtb6D/HHV7WM8RcbmFLl8SyUbn2NIaSpyX7345/TWoiSipSmzoTB/tEmWJ7CaT1pg5N39VraQQU2iWRRYitJlDyrgeaqFOjZA6X4Xs8XjgcDgQiURQW1uLwsJCtLa2irruxGOYAsyES4zlovTioNA51bJAkvGkzYW5SU9DAZRJbBgsBk5vOpmudTqTDuY/PSH78Xy4XC7NssgWxEzmoARWbIbSYDDMavitFGzWgtvthsPhAEEQqKurQ3FxMYAz0bx0o/Rinn1hMed9atgfgLrNjpKdhgKwWyFUWoMi2dSG3Ggd1Sr00CHpvZ3FoAlyFiBlMofRaARBEJIEWUpygroeMRYH89wulwsOhwMkSaK2thZFRUVxx8pdeIzu/Ynkx8ixK8RWskqlNdJVHfMhJMZi4PKlc4pnXq+UFZLoKkc8jFaZSV4Hm1jHCJIWYzWn0UxPT4veiJUpaIKMM9E1j8eD3NyZwZliEhOUIItFbgc3MYKs1+sRCARw5MgR6HQ61NXVZbR/JlU8mWmNTEHN+JwQai4SmvJmJDpGkKyJDbFJDTYSZwiKbeMpBy1lMcdgZoij0Sg++OADnHPOOZI9YbHIPV6oAp+amkJbWxuCwSDWr1+P/Px80c+hBEr7x1xpDT4xlpvWyMS+zJme2KAaIXF501wLgGwDXdXYNk3hdrszuihh46wUZCq6Fo1G6c0ccsbKpKJC5jt+cnISDocDJpMJtbW1GBwcTLkY8yGlmqUWC5UW91Sg1CQUQLnEBpXQSESJxAbAv1DIjNZFA1HeydpqVsiah5zhKL2ZQ+0KmW2hjiRJWohzc3OxbNky2O12BAIBVf04Of4xF+aqCpgVO5u61bHUSShqIsUGMRfmzrpfKLERHA8pMqKKCZ8YA5C0BiMVl8s1a/0k0zkrBJlvMgfbsWIFWmqFTOWQxcIUcJIkMT4+DqfTCYvFghUrVsBmO/PukrKBQ0mkVLTWVasAALFptjnK/KTaO5Y7BUXN6piPZD1pADDa+StVygpJTGlwsanzPcFj1K6QNUHOIKQ0hAfkxdikCqzY4aLU8QRBYHR0FE6nE3a7HatWrYLVamU9VqogS/nwkQoloJQIiyGVWeZE4WQmNlK9exBIrvucGJT0pamUBhvhSQJLDu2LKxZ4n1fFBkA+n0/0dWQKWSnIJEnC6/UiFArBZrOJtiWkxtioQaRikZJDJkkSgUAAJ0+eRHFxMVavXs0qxBRSmxFRO/DE/Lts7GUbjcmNceVamNxTrPfJqY75yKS0RiZuLgGSF2MpVP3tJYyNjaG7uxuRSAQmkwk2mw02mw12ux02my3u/aXWoh6Vs1djV6yaZJUgMzdzTE9PY2pqCkuWLBH9eDkWhN8vXhDEVNQkSWJkZARdXV0AgOrqalFZSilNgIAzFkcyL9jEita4cq3sc/EhVzxTmdZIhkxObADCqQ2K9a2zP7jD4TB8Ph98Ph+Gh4fh8/lAEARycnJgs9kQDodhNptVsS6UbPaVKrJCkNk2c5hMJsmbMOSkJpRKWZAkiaGhIXR3d6OoqAjr1q3D0NCQ6Bep1BeeEp4zlwDrOKpjPuTYFWzima60hlrVsW0e/1duJXts+MZm/3smI8YAkJOTg5ycnDgvlyRJWqj7+vrgcrlw7NgxRKNR5Obm0pW0zWaD1WqVJdRUemquMacFmfrFsjWEN5lMkvxaILWLdBSxWAxDQ0Po6elBcXExmpqa6M0pcnb2iUWsIHOlK6LLm2H0ShdeJeHqq5HJJC4WJtOAX+k2o7b58ZaYmOo5OB3iFGMudDodcnNzkZubi6mpKRQWFqKkpAQkSSIUCsHn88Hr9WJychJ+vx+xWAwWi4UWaUqo+b7dud1u5OXlSbquTGBOCzJzwGHip6FUcZXzmGQq5FgshsHBQfT09GDevHlobm6eNXlXqi8sBTkVcnQ591h4MSiVrlArraFGfI6vv0YyKJFn5kOsryxVjBNhWhU6nQ5msxlmsxklJWey3dR6CmV9jI+P01YhU6jtdjssFgt0Oh2mp6fnXAYZmOOCDHALS6ZWyARBoK+vD729vViwYAFaWlo4R6AbDAaEw+o0WxcS5OnpaXR2dmI1hwgrWR2LsRfSmdaQA5XaUEPkk0UpMV7++htJX4uYRXSdTger1Qqr1Yr58+fTt8disTihHh0dRSAQwMsvv4wTJ07A5/Phj3/8I1auXInFixeLXi/Zv38/du7ciWg0imuvvRa333573P29vb24+uqrMT09jWg0ivvuuw8XXXSR9B+ehTkvyFzIqQCNRqOsWJoYotEohoaGMD09jaKiImzYsEGwDWc6LAuXy4XOzk7odDo0NjYC3dI6ccnxj7nIlLQGn3DayueBy+VVYzQVoG6eWSxKiDGQXA5Zr9fT1TGTVatW4cUXX8Sf//xnHDx4ED//+c/x7W9/G+eee66o6/nGN76Bv/zlL6isrERLSwu2bduG5cuX08fcfffd+NKXvoQbb7wRJ0+exEUXXYTu7m5ZP0MiWSvIcgx9o9GIQCAg+ngxghmNRtHX14eBgQGUlZXBarWivr5e9PnVyhYnCrLb7UZnZydIkkR9fT0KCgoQ/tdfJT03H2IFlLlQqKS4A8lH3dhSG3LPqVZ1nCqrQinUyCGbzWYUFxejpaUFd911l6THtra2or6+HrW1M7/r7du345VXXokTZJ1OB7d75vfucrlQXl6u2LXPeUFWciVVqmXBN/eOsiYGBwexcOFCbNy4EUajESMjI5LOL6c7nJgXOCXIHo8HnZ2diEajqK+vF+W7qbGYp1RkTkm7IhP7a6Qzz0zhu+de2deQiFo5ZLnTQgYGBuJippWVlTh48GDcMbt27cJnPvMZPP744/D5fHjjDWW+LQBZIMhCSMnaylkITIQgCPT29mJoaAgVFRW0EMtBTu8LsV8BCYKgrYn6+vq0bDGlkhpS/3WUtivMVRWInOxgTW3wiXGm5ZntpfyNpZRIbVQ+/3tFG8qTJKnK5o3p6WmUlpYqfl4A+PWvf41rrrkGt956K9577z189atfxfHjxxX5Oea8IPNVyCaTiQ6hiyEZQY5EIujt7cXw8DAqKyuxadOmpL+KKdGMKBGfzweHw4HJyUlUVVWhrq4uqWtkItZiSDatIQcu8bSuWoXY9GTGROiSSWyoHaGrfP73ik+aUWtyjdvtnlkDkUhFRQX6+vrov/f396OioiLumGeffRb79+8HAGzevBnBYBDj4+NYsGBBcheNLBBkPiiBVVOQSZJER0cHRkdHUVVVpYgQU0j1kPmO9/v9cDgc8Pv9qK+vh9Vq5d3nr6R/HCmvhWF6UpIQq7m5REpiQw7pqI6TOa8UG0StqelKI7exUEtLCzo6OtDV1YWKigrs3bsXv/rVr+KOWbRoEf7617/immuuwalTpxAMBuPSH8kw5wVZqEKWkpqQIsjhcBjd3d3w+XzIycnB5s2bRb1QdTqdpCkgciwLJoFAAA6HA16vF3V1dZg3bx69KCFnp55U/9jTsAFm3zirGMvxouXaFVwizHc+uXaFXNRKbCgRoat8/vcAlO3OpuZcR7m9kI1GI3bv3o2tW7ciGo3iP/7jP7BixQrceeedaG5uxrZt2/Dggw/iuuuuw8MPPwydTofnn39esbWsOS/IfMhZpBMSqVAohO7ubkxMTGDRokUoKipCWVmZ6KqBsiGkzskTA/P6g8EgHA4H3G436urqsGLFirgXDfXBoBSJFa2nYYNi55YLtVCodFpDKRJTG2okNgQfK3GRUOxrVwxqVtvJtN686KKLZuWKmWmN5cuX4913303q+rjIakGmPGSx8H3KhUIhdHV1YXJyEtXV1WhoaIBer8f4+Lgk0aQ2kwhlkAHpWWqDwYBgMIiBgQG4XC7U1tZi+fLlrD+XWv2TlRBiOQKqW1QHHD+mWoMjJnIX86yVZeDq16dWv2clq2MAolM8YtDGN81mzgsyn4hK3ejBRjAYRFdXF6amplBTU4MlS5bEPaec7dNij5fyNSgUCmFqagpjY2NobGzEsmXLeB+v1+uTTpQw4RJis29csecQQkkxViLqlrhQKPec6dpgAsSLMaCsZaFmL2SPx5NR48zEMucFmQ+5gkySJC3ELpcLNTU1WLp0KavAKdFgKBnC4TC6urowMTEBi8WC8vJyUXEfvgw114Iem+c7vGgjAMAekubtKuEfM33pVPrRXFCJjVSixAYTKdNRlBRkNStkkiRVE3s1mfOCLLSoJ6VfMcXx48fh9XpRW1srWGkq2YJTCkwhrq6uRmNjI5xOp+iFkmQtC0qIAelinCxqpzX4SLQWmIuFmbRAKISYpkeG2x6etfNTSQ9ZrfFNai4Wqs2cF2Q+pCzq+Xw+OJ1O+P1+LF68GCtXrhQ9ZSSVFXIkEkF3dzdGR0exePFi2sumzi1WZOUIMlOE1YJPQLmEWOmdg3ziqXZkLpFkemyE2/vjbhM7pori9OnTCIVCMBqNdI/iSCSiWKJALcuC6oWs9UNOE1zTMsTE3rxeL5xOJwKBAOrq6kAQBAoKCkT/MlNRIVOzAXt6ejAyMoJFixaxxuykxOSkCnJXYTPyIK2SU8I/ZnrTSvrRUq0F2p/O0MQGk8JltYi43HECLBXrfz2BNR/9ORKJxPUopjZCMJvJ2+12wR7FiahlWXi93jnZCxnIEkHmgq9C9ng8cDqdCIVCqKurQ3FxMXQ6HYaHhyWnJqR2iJM6+87hcGBkZARVVVW8eWclBDnRP+4qnKlK8/TKfa0WU9EmLhLKEeNk7Qopi4Rq2BVCiY3wKSfnmCo+pC4SmkwmFBYW0rlevV6PhQsXIhQKwev1wufzYWJigrYHqU1Hdrsddrsdubm5rAWOWpaFy+Wakwt6QJYIMleFzCbIVDMdgiBoIRZ6DB9U1EzK8WJEk+qJ4fV6UVpaKmoHoJSp1kIVMiXEqSbd+WXdojrON0Um5JmZDY/kzAxMFqqqZTaTnzdvHn1/LBaD3++Hz+eDy+XC4OAggsEgDAZD3KBTu90OgiBExT+lIrexUCaQFYLMBVNcXS4XHA4HYrEY6urqOEPjUrPLcpra8wl4NBpFb28vBgcHUVFRgaKiIpSXl4vu4JZshSxViJVa0BtetFHWuZRKV1D+dLrHUjFRo7+GmOrY+l9PcN4nVNXq9Xq6MmamfQiCoG2PsbExdHV1we/3IycnBz6fL06sk100nKvTQoAsEWQuv1en0yEajeLIkSMAgLq6OsFflNqeMNf5mX2Ty8vL6YqYmkogBim+sFobQyjEWgypWCjkQ4lGR0raFWLGU6WzHajcjSFGoxEFBQVxlWtHRwfy8/NhNBrpgad+vx8kSdKjmSiRpkYziUHutulMICsEmY2pqSk4HA6EQiGsXr1a9FcYqZ6wHIuDKbCxWAz9/f3o6+uL65vMdbyUc/PBJshtvS5Rj02G8eJGlHkPShJipf1jOSKspl2hZHIjWbuCrzoGlM8hWywW5Ofnz5qhR9keHo8Hw8PDCAQC9IQQpu3B1jhME+QMYnJyEg6HA0ajEY2NjTh+/Lgkg99oNCIUCok+Xk6FHI1GEYvFMDAwgN7eXpSVlXH2TVYryiblWCUX9ID0VcWUP80l8Km0K/jGU6mFEjv+lBZktnPpdDp6NBOzpWU0GqVtj4mJCfT09CASicBkMtEiPTY2hvHxcZSVid/skklkjSBPTEzA4XAgJycHS5cupWMvlACKjdfI8YSl9svweDx47733sGDBAsHZelJ8YbVzyIlI8XzptAZHdE5N/1it6JwQidaC2PFU6bIrhKpjQNmNIVJzyAaDAfn5+bMKrHA4TKc9nn32WbzzzjsgCAL79u3DqlWrcOedd4p+HqEBpwDwm9/8Brt27YJOp8OaNWtmtedMhqwQ5NHRUQwMDGD58uWw2+1x91FZ5HQLciwWw9DQEJxOJwBg48aNovo0S7Uh5FoWXq9X1OPEwBS9bE9rCOWZU9HsiEJuhA6YWUAU88rJxOZCOTk5KC4uRnFxMR577DF8//vfx8UXX4yGhgacPn1a9PWKGXDa0dGBe++9F++++y6KioowOjqa9PUzyQpBLisri/OgmKhd8fL1hABm/LChoSF0d3ejpKQEa9euRVtbm+im+cn6wkLHBgIBdHZ2IqdgkajHiYFLhOVYH3IqWm+psukEWc3yPxpPlYkkk95IhWWRLC6XC8XFxaitraWHlYpBzIDTp59+Gt/4xjfolJYSU0KYZIUg86Fmk3o+SJLE8PAwurq6UFxcjKamJuTm5oIgCMXHMsk5NhwOIxAI4NixY6irq8MURxJPqoi2YQXy4ZP0GDkkil2yvTWUEE+xi4XpsCuERDh6+bdEnUdpEVVje7PcXshiBpy2t7cDAM4991xEo1Hs2rULn/3sZ5O7YAZZIchCLTjVrJATIUkSIyMj6OrqQmFhIdavXw+z2UzfL3cRUKljI5EIurq6MD4+DoPBgE2bNkGn02FKYsIiUfTasELS4/nOJQW10xp88I2mSnV1LLfpkRwyvUeEmikLgiDQ0dGBAwcOoL+/H5/4xCfw4YcfKvZ8WSHIfCQbSxMLSZIYGxuDw+FAQUEB1q1bFyfEFFJfzAaDQXTqg69CZk7DXrRoETZt2oT3338/6TdXohDnm9SvjoHUJTWE7Ip0DGxlI9VNjzIZj8cja6eemAGnlZWV2LhxI0wmE2pqatDY2IiOjg60tLQkfd3AWSDIUi0LqQJFNf55//33kZ+fj7Vr18JisUi9TE6kJCfYtpDHYjH09fWhr6+Pcxq2nPyx1IpYKf84nWkNIHMSGxRKND0Sa1coiZotMmOxmKzFQjEDTi+99FL8+te/xte+9jWMj4+jvb1dkk8tRFYIspBlISVXLBaSJOmoXTQaxZo1a1TZPy8lOcH8dyBJEgMDA+jp6UFZWRk2bdok+UXKJqInXNUAgEKz8v+mfDAXCpXORYtBSmqDT+Dl+sdx509IbihxzlSjZi9kuWIvZsDp1q1b8frrr2P58uUwGAy4//77OQMFsq5BsTOlmWRacLKR2JibycTEBDo7O2GxWLBq1SpJqQmpSLVQqMVEp9OJkpIStLS0KHJtlBADyoqxUEWrVGxObjWbCaOpgNTG57hQsqpVuzm9XCtOaMCpTqfDQw89hIceekj+RfKQNYLMhZxFOsqLTXzBTE5OorOzE7m5uVi5ciVstpmh7VL7X1ATn5WcPE2SJMbHx+H3+zE5OTlrMVEuTCEWQkn/uA0rkF+YGj+aDbnNjpRG7aZHUuwKJadEEwShSi9kv99Pvy/nIlkjyHwVslRBpkScEuSpqSl0dnbCZDKxbj6ROzVErCALechTU1Po6OiA2WyGxWKjHFClAAAgAElEQVSJy00KwVX1HB+Zj3yr8JgfpaG8aTniroR/rOZioVhrIXGhUK4FIobp6WnY7XZR4jgXMsjT09NztvUmkEWCzIWcQaeUIFObJgwGQ9x27ETkdogT0wuWz0N2u93o6OiAXq+nPyj+8Y9/iL4OnU6H9r54P/b4yHwAQL5VuUGsYjzfZGJzyaKkCMutZNVIa4jxj0dGRuh1ELPZTLfOtNvtszqszYWJ03O5sRBwlgiy1Ao5Fovh+PHjMJlMaGxsFGxOJKdCFntNbJaF1+ulm+w3NDTMqgj4/G8mzAqdEmIhlPSPB0ILASyU9Bgld/t1FTYjrzg1uwe5EGp4pCbRy7+FJR/9mZq07vV64fV6MTIyMqvDmpICqtb4prncnB7IIkHmEiChrc1M3G43Ojs74fF4UFNTg0WLxG0nllohSxFwpiBTFbvf70d9fT3r6i5lcYhtaC9WiIWQajGccFWnPKlBIWahUG3/ON2TURLR6XSwWCywWCyYP//Ma4LZYW1ychIulwutra30PD1mNS3FX1azQtYEeY5DjXWKRqOor6+nBziKRc3NJ3q9HgRB4NSpU5ienkZdXR3mz5/P+wEk9qtlL84Vfc1KIWaRUC3/OFGIUxmf85Y2IM/dyirEfNWx2hE6IZgd1iwWC3Jzc1FfX0/P06MmgAQCAbptJlOouWw5NftYaJZFBiAn5kJ99Y9EIqivr6f3v09PT0sW2HA4LOl4MYJMbXP2+Xyora3F0qVLBX9OJdpqquEfS0lrKE2qOs4J+ceZUhVHlzcDSz8u/XEfiSjXPD1mNU2NaSIIArm5uXFCbbVaac9aaTRBngMkVo0+nw8OhwPBYBD19fVJDzpVukJO3OZss9mwcKE4r1XKzj6pyLEYuIRYzrlkNTpSeCu3FK832YZHSqLEoqFQMoitXzFJknS/YqqxvN/vRygUgs1mQygUooVaiby8y+WKaxA018gaQRbTYCgUCsHhcNAebHFxMevjUjU1JBGubc7M/fVCSNnZpyZqpDXEkmxiI1nxVKrhkVy7gmp6BCib3pBjM+h0OuTm5iI3NzduzaOtrQ15eXnQ6XRx0z9ycnLiLA+r1SrJm9Yq5DmATqfDyZMnEQ6HUVdXh5KSkrR2iGObqzc4OJjUNmcKsZbF7w4pt7OQWYUmu0iYjH8sRYjV8I+5hDgd1TGvEMuwKwBlm9PHYjHk5eXFRUm5qmkAsFqtcUKdk5PD+h52u92aIGcqgUAATqcTLpcLNTU1qK6uFuU1q90hjurgRrXqVHKbc7KWhZyKlkuEU1UdpzPDDHwUn0tDf41E1I7Qic3Oi4EtZcFVTcdiMfj9fni9XkxNTaGvrw/hcJiepcdcQHS73VrKIhNgCm0wGKSFuK6uDgaDAXa7XfTCXyoqZLfbjYMHDyI/P19wm7OUrdZqWRZcnu/xPhvybdJ6HCjlHws1OlLbP07XeComqVwoVHqnnthvgXq9nhZdJsxqur29HTfccAPcbjfuuOMObNiwAeeeey4+9rGPiXoOMbP0AOB3v/sdvvCFL+DQoUNoblb+9581ggwAoVAITqcT09PTqKmpwbJly6DT6eD1etO6SMeE2oYdi8XQ1NQEq9Uq+BjKhhAryGot6lEc7zvTK0CqGCuBmokNOfE5ucj1j72lDUCp9HMmi5IDTpUQd+YsvUWLFuHQoUPYsmUL/uu//gunTp1Cb2+v6GsRmqUHzMRjH330UWzcqN72+qwR5FgshmPHjmHRokWz4mHpTk0A8duca2pqMDU1JUqMmecXU1GIEWS5/jFTiNWAr6JNZ1qDQo4QK+0fJ930SKZ/DCjvISsl7kyi0SjWrl2LdevWiX6MmFl6APCDH/wA3/3ud3H//fcres1MlP8XSRPUOKKFCxfOsiaktuCU6gnzWSFerxfHjh1De3s76uvrsW7dOuTl5aV1jBMXXJ7vyS69ZDFW0j9WajehXNqwgtenToV/PLxoY8qmpHCR6fP05LYHZZulNzAwEHfMBx98gL6+Pnzuc59L6hqFyJoKmQ+pFa8SLxS+bc5yFgGVGnQ6NTUFzu+7DE52nfmszrcr97ktpaJNJjqnhH/MFOFUjaZiwibA6cwzKynIak4MUVroY7EYbrnlFjz//POKnpeNs0KQ5bTglAvTx+ba5ixVkKUs1HEd6/V60dHR8dEbgV2QmSIsBrX843RVxHKaHcmFz+sdL24EZHQ+FfSPk7ArAGUtCzUIBoOyxqcJzdLzeDw4fvw4tmzZAgAYHh7Gtm3b8Oqrryq+sJdVgszVE1lOC06pkCSJ9vZ2jI+Po6amhnebs9qTp5k/azAYRGdnJ3w+HxobG/Fm52wxPu2IwG5X7qUgp6LtnbQCEOepUyjpH8ttdsRnV0itZoXmBaYbpRb11PKPp6enBTszsiE0S6+goADj42c+7LZs2YIHHnhAS1nIRU4LTrFRM2qbs9/vR25uLjZt2iT4GDUFmaqQqT4Y4+PjqKurw4oVK+I+IE47xH1AKWlXsEF50+lIawDqNTuSQibE58SglGWRab2QxczSSxVZJchcFamcuXqUiHNt1Ejc5lxUVITS0lJRn/xSPS6pmz0mJycxMjKCxYsXz/qAECvEatA/ZkRh1UwVKnaRUC3/OJ3NjijYhFhuxZ2qfspK+LOZ2AtZaJYekwMHDsh6DjFklSBzISebS/nOiYLMtc3Z5XKp5lOL8ZBJksTg4CAcDgdycnLoPhhM7tmr3GKH3IpW7eicEErG5+QyZKoGCtmvQy1Oxkpg7+2F3W5HXl6eYjvu5KDm+Ka5vG0aOEsEWc6neqLvLLTNOZm4mRB856aGm3Z2dqKoqAjLli3D6OiopBe82v4xtViYb59118ztMsRdqoDOzAhU9vcjtZql5wUi9YmNqqoquj9Ed3c33RYzLy+Pc2STWqg14HSuNxYCskyQlXwxUVNAmIJXUFDAuc1Zjk8t5VrYLJfp6Wm0t7fDbDZj7dq1sFgs8Hg8iu3US8Y/lprYYH1+BQQ0Hc2OEkl3rw0ArI18qCbzHo+HHtlEtRmgKmmbzaZ4Nas1p+cmqwSZDyn9IADQNoTD4YDZbMaaNWt4d9bJ2Uwi9nqoZkQUPp8P7e3tiMViWLZsWdwbjcveSJVdwSXEai8OUvROWrGy1JcRm0mkooZ/TPnUNQm3czWZJwiC7g8xMDAAn88HkiRhsVgQCoUwMTGBvLy8pJpgqSXIbrcbZWVlip83lWSVIItpqSnmheR2uzE8PAyj0YhVq1bNamrCd36xUAIupWFQMBiEw+GA1+tFQ0PDrMb61LFSKmQl7ArmQqHU8ymZrqAaHUkVYyX9Y6H4XKo2mDAXDK06NwBx+Vyj0YjCwsK4SjMWi8Hj8czqtsbsXZyXlwer1SrqW6pmWXCTVYLMhxhBZk5zLi0tpceii0HqoFNKkMUuroyPj2NsbAy1tbVYvnw570y9REGWUx3zVbRcSQ0lvWg+EgVPzEKh2v5xJqQ2APbkRrKZX71eD4vFArPZjPr6evp2puUxPj5Oz9VjtsS02+2zxFfNlIUmyBkE36czX/SNbZvz0NAQAoGA6OeWM2VESMBjsRh6e3vR29sLs9mM5uZmxTPOYulwzFR2Npv01Xk5doUYAU0UYqWzzHM5PsdEiU0YbDYDW+9iaq4e5Us7HA5Eo1FYLBa6kg4Gg7I2cAihCfIcgs1S4NvmLKdDnM8n/uuo0WjkTU4MDQ2hq6sLZWVlWL16Nfr6+hRvv8lX0VICnGmc7NIj357e6BygvBDL8Y/FzgxUS5DZ4JqrFwgE4PV64XK5MDY2hrGxMQwODs6yPJK5VpfLRQ8qnqucNYLM7GfB3MXGtc05FVND2I4fHx9HR0cHCgsL6Widz+cTfe7E7eNS7IqerhlRMFulV8Fy7AqxFa1ajY4AefE5gLuCT4V/LGXBMDTdA0N5edLPmcxCnE6ng9VqhdVqxYIFC0AQBMrKymCxWOgFxN7eXrqgsdlscXE8sbbeXB/fBGSZIAst6lEV8dDQEOsutsTjU9lD2eVyob29Hbm5ubMSHVLEXmz0b6DXBbNFmvCmyq6gUCI6ByTvHycuEqZjcCsgL7mhVN8IJZvTU1unmQ3mKWKxGG15jI2NoaurCwRB0Os5lFCbzeZZr/VwOMw7eWcukFWCDLA3GIrFYpiensbk5CRqampYd7ElkqoK2e/3o729HQRBYMmSJazeWrJz8gZ6XbNukyrGqeJMo6P0dp6T0+xILZLJMSslyEp2euNb1NPr9ayZ6WAwSFfTQ0NDCAaDMBgMdG/x8fFx2T+n0Pimhx56CM888wyMRiPmz5+Pn//851i8eLGs5xIi6wSZCXObs91uR3l5Oaqrq0U9Vu0KmSRJ9PX1IRqNoqGhIW5hJBG5c/L+86fSu4alyq4YHA4jv27mudLV6EjJOYHJILcDnZAFUlNmwb9GlauQlRJkqc2FdDodLBYLLBYL5s8/820lEonA6/XixIkT2LNnD3p7e9Hc3IylS5fiuuuuo9tl8iFmfNO6detw+PBhWK1WPPnkk/jOd76DF198UdLPLJasE2Rqw0XiNmev14uRkRHR51GrQiYIAt3d3ejv70dJSQlWrlwpaDMoPSdPTnUsx64QIhWNjsTaC0r02FDKP1ZywVApq0HpAadKnMtkMqGoqAgf+9jH0NzcjM997nN499130dbWBptN3O9TzPim888/n/7zpk2b8MILLyR97VxknSCPj4+jra1t1jZnqR3fpG7DFhJwqjtcf38/qqqq0NDQgHA4LOp55GwJT1V1zAezok1MbXAJfKqyzIA8EVbTP1YjQqeU1aD07jql+yFTnd5MJhNWrlwp+nFs45sOHjzIefyzzz6LCy+8MKlr5SPrBJkgCNZtzmr2mgC4RZMkSQwPD8PpdKK0tBQbN26E0WjE6Ogo/H6/atfDhdLesZzonNLVtlR7QY1mR3KZaXrEPh5EiYpbSQ85me3SapOKDPILL7yAw4cP429/+5tqz5F1grxw4UJW6yCVY5woJiYm0NHRgfz8fDQ3NyM3N5e+T83ucC+8I75CEIMYAaUicxRqVttyUTM+xwebeCYzL1AMNWVntkor1cM4k8c3ud1uWZtNhMY3Ubzxxhu455578Le//S3ufaw0WSfIXEjd2gxIb0hE4Xa70d7eDpPJhNWrV7M2JVJTkLngq47ZBHS4bxoAULc0PvZFpTbMFhMmxpS7PqXtiv4REqkarC42z5zupkdyUUqQ1RpuKrcXstD4JgA4evQorr/+euzfvx8LFixQ6pJZyTpB5qoG5PZEFtuQCJj5WvfPf/4T4XAYjY2NvNML5AgySZKCP4dY73hsaDru76YcdrE2W3NYY3NySYVdQS0WKr1ZRW41O1dFmImSXrQa8/TUHN902223wev14otf/CIAYNGiRXj11VeV/hFmrkeVs2YJYgU5HA7D4XAgEAigoaEBpaXsU52ZyJk8LeZNMTkyzXq7KdcED/tdnGIsl3TYFVISG6myK4TmBfIJfCZ0jGOiZFoj0xoLCY1veuONN5K6NilknSALVZBiqkwKoYVAgiDQ09OD4eFh1NTUwOv1in5RyN1IwifI22/tFX0+MZit3B9EqVwc5KLD4cuItEYizPRGOoa3Mv1jpSwCpSwLNZvTU9G1uUzWCTIflKiJ/YTmEuRYLIb+/n56wOnmzZuh1+sxOjpKj8YRey1Srp0vi8wnxqZcbvFUujrmI1m7Qmx0Tg34qtn+MSP6M+itRBCE4qOYsnXidKaROa+iFEBlkeUKMnOu3vz58+kIG4UUkZVjWaRyEVBudaykXdHT5YbZaoKE/TyC8NkVSsfnUg3lVVfNm0YsFsPU1BT0ej0ikQh0Oh39nxzrQckKWQ3LIhsaCwFZKMhipoaIhXn85OQkOjo6kJeXh6amJtYqWMr5pVYwfAI+F6pjPibG/LDb82dF5+Sitl0htumR3AVCqf4x26IhNeJr1apVMBgMIEmS/obFrKD1en3cn7lQskmRWhUy3yL6XCHrBJkPqVlko9EIr9eLI0eOwGAwYOXKlbxbMtMxeVoNMVaiOqYicxTM6FxiasNsMUkW41TaFUD6ZwWywSbETfPbcfToAN3fOxFKlKPRKEiSjBNq6ja9Xk+LL1OElbBB1BzfNNd7IQNZKMhCFbLY7dOBQACDg4MIhUJYs2aNqE9ftSdPMz1kkiRxyfWnZx1nzVO3eXtiXI4Jn8hzRedSZX8IoXR8Ti2EInTBYBAtLS2cVSib0FKvq1gshlgsBpIkaXGmuidSt8m1PCjUrJA1Qc5Q2FpwAuIq5HA4DKfTiampKcyfPx/RaFT0VyE1K2TKQ6YqGq4FPr/HR19LyB9kv06TEQGwj6cymozwe9i3dKtRcSsNn3gqHZ9LJUIROoqGhgbJ52YTaWBGoP1+P9ra2lBYWBhXSQOQ5UuLXfSWSjAYhMUibpBrJpOVgswFX4UcjUbR3d1NR9iWLFmC6elpDA0NKXJ+NqTsBDQYDIhEIvQHyhXf6OQ9dq7DVx0raVeoNStQCf9Y6szAS9eLnwEpBNUMa2hoCEuWLKGrT6qCpooCqSKtxqIeVXwpnSxJB2edICcOIo3FYhgYGEBvby8qKiroCBt1vNQWnMEge1XKdbxQ4J568dtsNrS3t2NwcBA/epp7WV9IjA0m7l+5kee+VPvRaiIlPpdqu0KJNqDJ4na7cfr0aZSUlGDDhg1xr0/qz8zXGVOkEz1pYOY1bDAYoNPpVIu9AZogZyx8lgVVwZIkidHRUTgcDsybNw8bNmyYNbsrVWOcuGaGMb28kpISnHPOObjsxnbe8/E+n0wxnitwiaca8TmlyYQIXTQahcPhgNvtxvLly2G3i7sYLpGmzkl50MFgEC6XCwsWLEA4HOZcPJSKWguF6SA7fgqRUII5NTWF9vZ22Gy2uJ7JXMeLRc5mD7bzJ/rE1FfAz13z4axjcyzi/Dg+MRYinWmNRPiqWaVic8nCZy30j5BYXjPz58TkRrIZaaqzYF5eHiwWi6SKcXx8HJ2dnaisrERDQ0PS1SZTaKkWtN3d3aitrUVBQQH9GqfeL8n40i6XS1ant0wkKwWZ68UUDocxNjaGSCSCFStWCFYAUjvEyamomQLOJcQAWMUYAMKBEH2tUbA/tyWf/2uwXKtCbRKjc8CZ+JyUOYFyBT7ZBUIulBreyuTCFdNwuwvh8XgwNDSEQCAAk8mEvLw8WqRtNhvrYNC2tjbEYjGsXbtW8SGhwWAQp06dQm5uLpqbm1m/DfJZHlQMj0+ks2WXHpClgpxIIBBAZ2cnfD4fcnNzsW7dOlGPk/o1Sm5FLUeIE8/DhU6vQ9DL3QhfbzQgHOTeiGDwG2AvymO9T2x1nNjwSLDR0QT3OeXE59INM7mhliedm5uL+fPnx2WPw+EwPB4PPcHZ7/fTg0HtdjtCoRBGRkZQX1+veFtJkiTR39+PgYEBNDY2xk2WTkSOL009TqfTZc2mECBLBZkSs0gkAqfTicnJSdTV1aGkpAStra2qPa+c7dCRSCQu88msYITEWMgz1ukFZvUZBTznj87vnfLMvo8nOgcAfk9WvrREwxafU6vi5iInJwclJSVxA3QJgsDY2BicTieAmd9xV1cXxsfH6WrabrcntfDm8/lw6tQp5Ofn82ai+RAj0tSf9+3bh4GBAdnXm0lk5bsmGo2iq6sLg4ODqK6uRmNjIy10ajXIBqRVyCRJwmw2o6OjA+Pj4ygoKEB+fj6sVit0Oh0u/LdjnI/VGw0pE2M5ZFJaI5V2BdV9bpDzkfJQqmMcFWUbHR3FypUr6aoyGo3C6/XC7XZjYGAAXq8XsVgMdrs9zvIQWjiLxWLo6enB6Ogoli5dqnjVmijSo6OjuPXWW6HX6/Hoo48q+lzpIisFeXJyEsDMhNhkIzZSssLUAgYfTHuivLwcCxYsoN8MDocD3/2p8HTpGBFFjGAZU5U7I1pqi7Ea0bm5CNfMwFQjJn88PT2NtrY2LFiwAC0tLXGvZ4PBgIKCgjgBjcVi8Pl8cLvdGBkZQWdnJ6LRKKxWa5xIU73CqajcvHnzZp1faUiSxO9+9zvcf//9+O///m9cdtllWRF5A7JUkBcsWKDYNkqpU0O44PKJc3JyUFxcjOLiYt6qWAyRUJhXjCnB5iMZMU6GVFfHcqBSHFznzaQ8MwVBEPT6yapVq1jHibGh1+uRl5eHvLwz6wckScLv98PtdmNiYgLd3d0Ih8O05VZdXY358+erKo4jIyO45ZZbYLPZ8NZbb2HevHmqPVc6yEpB5oPagiy2ck5WkPkW7JhsvepI/HXKqOyFKmMi8pGdEmG3VYQidEJiPJeqYz7xnBjzc84KVKO/hpItQZlQOfvFixdjyZIlSQulTqeDzWaDzWbDwoULMTU1hdOnT6OsrAx2ux1erxcnT55EKBRCbm4uXUXLieElEovF8NJLL+HBBx/Ej370I1xyySVZUxUzyUpB5vtFUf0sxAqy3P4UzEUHtgU7ikQhpogxnpMptDod+5tXSIx1Ir5CRsMEZ3QOAHShMADAbJ9dZSWzsUTNPhhs0TlAXnwuEykLv4XWVj1tI+Tn58NgMKCjowMGgwFNTU1Jf7tLhCAIdHR0IBAIYO3atbN6SJAkiVAoRCc8mDE8pkizxfDYGB4exre+9S3k5+fjwIEDcYuU2UZWCjIfVL8JsQ1OpEbZdDpd3O4kvoA7lxjHnS9BaEmS3WPWgS/6xi/GYj6cmNeRGKMTjM6xNDriitJxwTYrMNXxOTUWCOXATHBc+sVNIAgCHo8HbrcbJ0+ehNfrhdlsRlFREUZHR+mYmxJblsfGxtDZ2YnFixdj6dKlrIKq0+lgNpthNptFx/CoDxSbzUa/X2KxGH7zm9/g4Ycfxj333IPPf/7zWVkVMzkrBVmtzR4kScJoNKKzsxOFhYUoKChgDcJ/5spDs25LFE2hijeRGEcVb+DYlk3fL1GME5G7QEhF6dToPDeX4LMrBofDyK+b+Tn5utAZjUaYTCaMjo6iuLgYzc3NAECLX39/P7xeLwDAbrfHValiRZraQEKSJNavXy+rYxtXDI+6zp6eHni9Xrz99ts4fPgwRkZGMH/+fOzfvx8VFRWSn28ukpWCLGRZSOnIJqaDG9MnXrZsGaampjAxMYGuri5Eo1H6TXDtbdzNFEhGO02dXg8yRkoW5UR0Oj1rGgMQF52buRb515BNC4SphorQ8Qnx7V8k6Ijn5OQkli1bFrcIV1hYGLeDjYq3eTweDA4OxsXbmCLNjLcxtz3X1dUpvoHEaDSiqKgorpvcqVOnMD4+jqamJhAEgSuuuAKPPPIINm3apOhzZyJZKcgAd4MhJStktgU7i8US56lR8aErrp3dTH7WNSdUyWSMfUFH0C/m8JmZcEXnAOXic3zMtQVCte0KORG6yclJtLe3Y+HChWhpaRH8Oi8UbxseHkZHRwfdXdBisWBiYgI2m41z27OSDA4OYufOnSgtLcW+ffuyouG8VLJWkLmQI8iJLTvFLtgBwGdF+cTSMptsQk2JpxgxFoLg+UZAiXW6NpYIkSmN8oUQ6kAnpj9zT08P1qxZk1Rjdma8jbIFqKp7aGgI+fn5CAQCOHz4MGw2G2sGOVlisRh++ctf4mc/+xnuu+8+XHjhhVnvFXORtYLM14IzEBDfyNtoNMLnO1O9JI634VqwY/OJuUi0K+RAiTSJ+KpXanxOqCqeyTrreaNzarYBVWtoq9p5ZqU70V3z8X6Ulq5VXLiY257POecc+neZmEHu6upCJBKZtVFEqrc8MDCAm2++GRUVFfj73/+eNU2C5JK1gsyFXMtCbJ4YkCbGiTDFmUKuSANnFvuUiM6JuRbB6Jxeh0gozBqdEyIZq0Kp6phveCsQH6MzW0yq5ZlLSkoUFWOhbc+JGWRgRqQDgQDcbjempqbQ09ODcDgMs9lMR/AokU681lgshl/84hd48skn8dOf/hRbt249a6tiJmedIEudPG0wGBAIBBAKheipB2oIMR/JinQqonOAtGZHbB3oxHSeo0iMzcmpjqkondz4HMA9vDUZxNgVx44dA0EQsNlstPDl5+fL8nnlbnvW6XSwWq2wWq0oKysDALoRvdvthsvlQl9fH71RJC8vD++//z5qa2tx//33o6amBm+//XbWdGpTgqwVZC7RFDv3jqqIc3NzYTKZ8MEHHwAA8vLy6EZAdrsdOp1ONSHmvT4RIp2q6ByQus5zFIkd6IRsEK74XDKkK7Fx+xcJAC0gSRI+nw8ejwfj4+NwOp20SDM3inCJdDQahdPpxPT0tKQJIXwwF7ZLS0sBnNkoMjo6itdeew0ffPABDAYDSJLE3r17cf311yf9vNlC1goyF0IVcuKCndFoxPLlywHMvIA9Hg9cLhe6u7tx693i5+elglRH54DM6DyXjmkoyaDU9mudTge73Q673R5nI1Be7/j4eJzXy7QRfD4f2traUF5ejubmZlXtAp1Oh9HRUezYsQN1dXX48MMPkZeXh5GREUxM8Hz9OAvJWkGWUyELLdgZDAY625mOqlgMSkXngOTic5nQeQ5QLz6nVmIj2YnaXF4vJdJjY2M4ceIECIJAYWEhotEoJiYmkJ+fr/gWa2DGK37uuefwzDPP4MEHH8QFF1xAvzdLS0vpKlpjhqwVZC70ej29MEeRqgU7tZHiK6sdnePrPCc2OidEMmIsRKYmNmbsCmlQIu33+zE1NYX6+nqUlZUhEAjA4/FgcnIS3d3ds1ITyYp0T08PbrrpJixduhTvvvuuIpZItpO1gswlqMzbs0WIKSjLIt3RuZlr4K6MxUTngOSyzEJkY3XMBde2Z6qSZi7IUamJyclJOjVhsVhogRYj0rFYDM8++yyee+45PPzww9iyZYuWoBBJ1gqyENFoNGuEOBG1onMz5xGuopPtPBcOhHiHtgLJDW4VQq3qmAsqSpcYoQPYUhz8PzcTqdueuVITbL+uN/cAAB/dSURBVNE2SqSpapoS+a6uLuzYsQMrVqzAu+++C5tN/PWKpa2tDVdeeSX9d6fTibvuugvf/OY36dsOHDiASy65BDU1MyO+L7/8ctx5552KX4vS6CSONFJv/pHCxGKxWV4xtWB3+PBhmM1mehsps8MUk7koxlJIJjrHhVA1rXR8jvUaeKwQa55NsDpOZrcfJchjQyzd6QSEXujc990AesSXEMxpzw0NDYpue2ZG26j/fvjDH2J6ehpDQ0O4+eabcfXVV6O8vFyx5+QiGo2ioqICBw8exOLFi+nbDxw4gAceeAB/+tOfVL8GkYh6A2VthZz4omUu2K1duxZerxcul4vuMGUwGJCfn0+LtNlsxmt7mzE+Po6v3NSdnh9CZcRW0lKSGlzROb3BkHYxBoCQPzirFWjc8ycxvJUvzyyEGBvE4XDA7/fDaDTG+bxMkZYy7VkuidE2p9OJWCyGlpYWnHfeeThx4gR27NiBl156SXWr4q9//Svq6urixHguk7WCTMHmE+v1+llNViKRCP1pPzw8DJ/PR/dNfv7hxZg3bx7tnWVz5azENm7285IgY7PFmimgmR6fS5cNAgCP32oDsBpA/Gt1dHQUgUAARqMRFosFLpcLBQUFaG5uFhxKmizRaBT/8z//g1/+8pd45JFH8IlPfELV52Nj7969uOqqq1jve++997BmzRqUl5fjgQcewIoVK1J8ddLJWsuC2ilUWFhIe8RiPq1DoRAcDgd8Ph+qq6sRi8Xoc0UiEdhsNlrMmf1ks02khcRYStUsN7XBnAEo5vnSNbwVEF4kTLbp0YwgsxOLxeB0OjEyMoLi4mKEw+FZEzry8/OTHqPEpLOzEzt27EBTUxPuvvtu0bP6lCQcDqO8vBwnTpyYFZ9zu93Q6/Ww2+3Yt28fdu7ciY6OjpRfIwNR//BZK8itra249dZb4XK5sHTpUjQ1NaGlpYWzO1Y0GkVvby9GRkZQU1ODBQsWsNoeVKtCl8sFj8cDkiTp3XuUH009bi6KdLJVsZieGcmckyJxYGsm55mT9Y75xJi57bm6ujpuLSQcDtOVtMfjgd/vR05OTlxiwmw2SxLpaDSKJ598Env37sVjjz2Gj33sY6IfqzSvvPIKfvazn+H1118XPLa6uhqHDx9O51DUs1uQKSKRCE6cOIH3338fhw4dwrFjx6DX67Fu3TqsX78e69evxzvvvIPS0lKsX78eVVVVkkaYU7v3KJH2+XwwGo20H03FhPr6+jA0NIQfPpL5LpGSVgUgLzZHIafhUeKwVrWtinRUx8xtz8uWLROd8WWKtNvtRiAQoPtMCIl0e3s7br75ZmzYsAE/+tGPkmr7qQTbt2/H1q1b8bWvfW3WfcPDwygtLYVOp0Nrayu+8IUvoKenJ53xO02Q2SBJEl6vF0eOHMHevXvx0ksvobKyEiUlJVi/fj2ampqwYcMG+pcph0gkApfLBZfLhfHxcXrGWWlpKYqKimb1F5gLlbTSIq3T6wQraCW6z3GJMZ11TqMYA/IEeWpqit72XFVVlbTIhEKhOJEOBoP01Oienh5UVFTgtddew0svvYTHH38c55xzTlLPx0d1dTVtBRqNRhw+fDjufpIksXPnTvzpT39Cb28v/vrXv+K8884DAOzZswcAcMMNN2D37t148sknaW/9oYceUvW6RaAJMh+hUAjXX389vve976GxsRFDQ0NobW2lK+nR0VHU19ejqakJzc3NWLduHd1MSAwejwft7e0wm82oq6sDSZJ0Fe12u+kmMFQVPRf9aDkiLXbaiRJTtJUYUaU3GmDN47YMkumFIVWMmdOely1bpmqFSon0ww8/jNdeew0TExNYt24dNmzYgDvuuANms1mV5xWyFvbt24fHH38c+/btw8GDB7Fz504cPHhQlWtRGE2QkyEajaKtrQ0HDx7EwYMHcfToUUQiEaxevZoW6eXLl8/Kd4bDYXpRsLGxEfn5+aznZ47OofxonU4X101uLvrRfEKZTKMjpu2RSjFO5jmSrb6LSwvx0M05MJlMcdOeFy5cqPpXb4IgsHv3bvz+97/Hz372M2zYsAGDg4M4cuSIqtOfhQT5+uuvx5YtW+hkxZIlS3DgwAG6b0cGowmy0vj9fhw9ehStra1obW3FyZMnkZeXh6amJqxduxZHjx7FokWLcMUVV7AuCgrB7CbncrnozCmz3efo6ChGRkZQX1+PL3+jS6WfVFl0en3SXefocyXYHGyimQ1iDAA/vNaH6elp+Hw+GAwGlJeXo7i4GPn5+apG2k6dOoUdO3bgvPPOww9/+EPVqmE2ampqUFRUBJ1Oh+uvvx5f//rX4+6/+OKLcfvtt9OLiRdccAF+8pOf0JO2M5ize2OIGlitVpx77rk499xzAcz4WePj49i9eze+//3vY/HixThw4ADefPNNtLS0oKmpCU1NTXT0TghmNzmKcDgMl8uFoaEhnDhxgh5U6fF48OsnajPej6aqWb5mRqLPxeI5s3Wao25LTGKIfd5MEGNgpvf2xMQE3auY6tbmcDgQjUbp5vTUf8n2/iAIAo8++iheffVVPPHEE2hpaUnqfHJ45513UFFRgdHRUXz605/G0qVL05JvTheaICeBTqfDvHnzYLFYcOTIEZSXl9OZ0IMHD+KNN97AfffdB5/Ph+XLl6O5uRnNzc1YvXq16Nlj4XAYfX19yM3NxbnnnoucnBwEg0F6wdDpdCIajcJut6OgoAC//Z9G5OXl0UmRdAq0kLUgRaTlROgioTDr+bmEGki+A51SfPuqUUxN5cZNe2a21OSaFs1MS9jtdtEiffLkSezYsQOf/OQn8c4770iejacU1KDVBQsW4LLLLkNra2ucIFdUVKCvr4/+e39/P/2YbECzLFJAOBzGv/71L9qP/vDDD5GTk4N169bRIl1fXx8Xt4tEInA6nXC73WhsbOQdc0O9OSmrw+v1QqfT0W/MgoKCuO21aot0JsXmmIhdKEyMzTFJRXW84/I+WdueY7EYvF5vXPYYwCyRTnydPfLII/jzn/+MJ554QpWv/n19ffj3f/93jIyMQKfT4etf/zp27twZd8yBAwewbds2LF68GAaDARdffDHefPNN3HnnnfjsZz9LH/fnP/8Zu3fvphf1br75ZrS2tip+zSqgeciZCpW4OHToEA4ePIjW1lY4HA4sXLgQ69evp33kXbt2oby8XNYCCkEQ9Hncbjd8Ph9ycnLoHYbhcBj9/f2oqqpCRUUFtm4/LHzSJFBapAHxQq1EfA44I8Zcgq2UVfF/95Yp1no0Go3GiTT1Yf3iiy8iLy8P+/fvx7Zt2/CDH/xAtap4aGgIQ0ND9Gu7qakJf/jDH+hJPMCMIN911130BBGCIPDlL38Zd9xxR1ycjSRJ3HTTTdi/fz+sViuee+65ueAfA5ogzy1IksQf//hHfOc730FxcTFyc3MxPj6OxsZGuopeu3at6G5fbFBzzbq7u+nxVJQPSS0cMheLMr2SZhPaRGtDaTEWez1sU7XFCPIvf6r+1+9gMIgf/OAHdJpheHgYJSUleO2111KyceKSSy7BTTfdhE9/+tP0bRnYnU1ptEW9uQTVi/aVV17BkiVLAMxUCadOncL777+P3/72t/j+978PkiSxZs0aWqSXLFkiasWdIAh0d3fD7XZj9erVKCgooHvdulyuuMUi6ivuS08vmfUVV0mRTqZvM5fQJk7UJj9a8+OqptWK0CVO1RaaqA2AN++sFB9++CFuvvlmXHjhhXj77bfphll+vz8lYtzd3Y2jR49i48aNs+6bi82AlEarkOcQVC+NI0eO0NG7trY2FBUV0dnolpaWOJuD2aR80aJFghYI5UNSVofH46Fbk1KVdGKTmlRW0krF5yiEJmqnIkJH8b8/KZv1AagU4XAYDzzwAN544w3s2bMHa9euVfw5hPB6vTjvvPNwxx134PLLL4+7LwObASmNZlmcDZAkidHRUXrB8NChQxgaGkJNTQ2qqqrw7rvv4kc/+hE+/vGPy56PRhBE3C5Dv99Pb621Wq0YHR0FMBPSp3aPqSHSSgqzUGpDzDRtMdchVowf+76F/gDU6/WcG4Tk8M9//hM7d+6kM7xqDDMVIhKJ4OKLL8bWrVtxyy23CB6fAc2AlEYT5LMVr9eLG2+8EYcOHcLGjRvR3t6OYDCIlStX0l3vVqxYkdQbMxAIwOl0Ynx8nD4P04/Oy8tTzI9WerJJsl3olJqoTfHyk41xf6cWZKkPQb/fHzdAQWwrzVAohPvvvx9vvfUWnnrqKaxevVrU9SgNSZK4+uqrUVxcjEceeYT1mAxsBqQ0miCfrRAEgVdffRWXXXYZ/YIOhUI4duwY3avj+PHjsFqtWL9+Pe1HJ7Zv5GJ6ehrt7e0oLi5GTU0NDAYDPWqeqqLdbjdIkqTz0WyjssSItBLpDCWnaSeeMxFmxlmuILPBbEpPfUuhUjOUnZSbm0v/vo8dO4adO3fi0ksvxXe+8x1FRzglsn//fuzcuRPRaBTXXnstbr/99rj733zzTVxwwQXIycmB0WjE4sWL8cADD6C3txdAxjYDUpqzS5B/+9vfYteuXTh16hRaW1vjojD33nsvnn32WRgMBjz22GPYunXrrMd3dXVh+/btmJiYQFNTE37xi1+k5atdqiBJElNTUzh06BAt0t3d3aisrKQFuqmpCcXFxXGi7nA4EAwGsWTJEsEBlrFYLK7S4xqVxeVHqxGVo0h1S1CAPS4nRoy5oBoAUR+CTqcTTz31FGw2G/r7+7Fnzx7Vd7lFo1E0NjbiL3/5CyorK9HS0oJf//rXcZG2J554Av/617+wZ88e7N27Fy+//DJefPFFVa8rAzm7BPnUqVPQ6/W4/vrr8cADD9CCfPLkSVx11VVobW3F4OAgPvWpT6G9vX2WP/ilL30Jl19+ObZv344bbrgBa9aswY033piOHyVtxGIx9PT0xPnRHo8HS5YsgclkQk9PD5555pmkWj4yKz2Xy4VAIACz2UxvAR8aGkJxcTFqa2vp39HWq44o+WOyosQ07TPHifsg+fPzq0QdJ5bDhw/j9ttvR319PRYsWICjR4+ipaUF9913n6LPw+S9997Drl278NprrwGYKX4A4Hvf+x59zNatW7Fr1y5s3rwZBEGgrKwMY2Nj2WRHiOHsir0tW7aM9fZXXnkF27dvR25uLmpqalBfX4/W1lZs3ryZPoYkSbz55pv41a9+BQC4+uqrsWvXrrNOkPV6PWpqalBTU4Pt27cDmIlJXXPNNSgqKkJdXR2+8pWvwGAw0A3+W1pa0NDQIHojg8lkQklJCUpKSgCcSY50dnbC7XbT+etAIEBX0fteWDvr/EqKNFNoE2NzwIxIK5VnplBSjIPBIO6991689957ePrpp1MaFxsYGEBVVRX998rKylntMJnHUM2yJiYmsmnBTjGyRpC5GBgYwKZNm+i/V1ZWYmBgIO6YiYkJFBYW0otQbMecrdhsNjz33HP0ghBJkvB4PDhy5Ajef/993H333ejo6MD8+fPjondiG/xT+eeqqiqsWbMGOp0ublTW0NAQ2tvb40Zl5efnY/+v1s86vxyRFiO0JBmj88xMpLYEpVBSjA8fPoxvfetbuPLKK3HgwAHVB5tqqMuc+u196lOfwvDw8Kzb77nnHlxyySWKP9+VV16JtrY2ADMLWYWFhTh27Nis44SmHMxlamtr4/5O9cg4//zzcf755wOYEenBwUG6wf9TTz2FsbExNDQ00B3v1q9fHxffCgQCaGtrg9FoRFNTU5xfr9PpYLfbYbfbUV5eDiB+VFZ3dzfrqKxEkSZJEp/98gesP5cSeeZYNPrR9erjJmrzLeY99L0cDA0Nsea5pRAMBvHjH/8YBw8exAsvvMD5DVFtxDT7oY6prKwEQRBwuVz0NySNeOaUIL/xxhuSHyPmBVNSUoLp6WkQBAGj0Ugfw1x4uPXWW3kb/Lz11ltn7VcwnU6HiooKXHbZZbjssssAzAjo6dOncfDgQfzhD3/AnXfeiWg0ipUrVyIYDCIQCOCxxx7DggULRD0HW2tSyo92uVwYHBxEMBiE2WxGQUEBcnJyMDg4iMf/uxB1dXVxloeidofIlqB6owF/eHop7Z+Pjo7G5bmpDxYx/SSoAb5XXXUV3nrrrbRWxS0tLejo6EBXVxcqKiqwd+9e2vqj2LZtG/73f/8XmzdvxksvvYRPfvKTZ5t/LJqsWdSj2LJlS9yi3okTJ/DlL3+ZXtS74IIL0NHRMcuT/OIXv4grrriCXtRbvXo1/vM//xPATKW1aNEivPnmm2hoaJj1nFkYYleFf/zjH7juuuuwaNEi5Ofn4/Tp08jPz4+zOioqKmTvVKOsDofDgenpaVrcuEZlJfLZr7BX01xIidH9vxfYd8aFQqG4qGAoFILVao3rc0xF1gKBAO6++2588MEHeOqpp7B06VJJ1yuX2267DX/84x+Rk5ODuro6PPfcc3EfjPv27cM3v/lNdHV1obi4GAsXLsTo6Cj27NmDbdu2IRgM4qtf/SqOHj2K4uJi7N27d9Y3r7OAsytl8fLLL2PHjh0YGxtDYWEh1q5dS6/83nPPPfj5z38Oo9GIRx55BBdeeCEA4KKLLsIzzzyD8vJyOJ1ObN++HZOTk1i3bh1eeOEF+g3997//HbfccgunFSE05UBjhqNHj8JqtdK9OqgG/62trXTXu/7+fixevDgueldQUCCqoqKGfy5cuJCeHi51VFYiXCKthBizwewvQon0nj17MDw8jK6uLmzbtg0//vGPRU+ZVoLXX38dn/zkJ2E0GvHd734XAPCTn/xk1nFaYcLL2SXIchHjS994442or6/HrbfeynqOgYGBuCkHjz/++Kz8565du/D0009j/vz5AIAf//jHuOiii2adSyhkn+3EYjE4HA5aoA8fPgy/3x/X4H/VqlVxX+0jkQg6OjoQCoWwdOlSweGfQqOyKOsgUaQpwf/v3eJHGkkRYzb8fj/uuusunDhxAueffz76+vpw9OhR/OIXv6A/2FLJyy+/jJdeegm//OUvZ92nCTIvmiArAUEQqKiowJEjR1BZWSl4/K5du2C32/Htb39b1O1MxITsz0bC4TD++c9/0vno48ePIzc3F2vXzsThHA4Hdu/endTwz3A4HLfJIhgMwmKx0DsMx8bGEAqFOKc9X/hvsxd7kxXjf/zjH7jttttw9dVXY8eOHYr1SE6Gz3/+87jyyivxb//2b7Pu074p8nJ25ZDV4o033sDSpUs5xdjn89Gjc3w+H15//XXceeedsp6rtbUV9fX1tL+2fft2vPLKK2e9IOfk5KClpQUtLS246aabQJIkjh8/juuuuw7RaBQLFy7EpZdeivLycjob3dzcjHnz5okW6JycHMybN4+u7kiSRDAYRF9fH06ePAmTyUSLP9OPpvzuZMWXic/nw1133YXjx4/jN7/5Deu6hdKI+aZ4zz33wGg04itf+QrrOc72eXhKoAmyAHv37qVHjlMMDg7i2muvxb59+zAyMkInC6gpB8yRM0x2796N//u//0NzczMefPBBFBUVxd0vJmSvAVpk77vvPmzZsgXAjID29vbi4MGDeO+99/DYY49hampqVoN/sVGzSCSCzs5OkCSJzZs3Izc3N25UVn9/v+CorP/f3t3GxFllARz/X9JudSr4QqLtGKUMszSEpoMlUkxsmhjpe8bYoHb7gZo21RKIJLgR10ql8UMTa62bNAaJrcQaS9GIlHbCmlAgpAu0dIvGaVAJ4rYpIsGwpWYLhTn7oTDLMAMMFMYBzi8hgXnuTO5DwuU+95x77mSJCOfOnSMvL4+dO3fy/vvvh2xWPFEGU3FxMadPn6aqqmrM+5voPDw1MV2ymEbjzTJSU1O9M7b8/Hw6Ojo4duyYT7svvviCyspKPvroIwCOHz9OY2MjR44c8baZKOI9bC7nRgdrYGAAt9vt3QZ+6dIljDF+Bf5HDnoj60fHxcVNmJY33lFZk0ll+/333ykoKKClpYWioiLi4uLu+P6nS2VlJbm5udTW1npjIKONflJMS0vzOw9vntM15HDV3t7Oli1b+O6773xeD6YugEa8p244La6pqYnz589z4cIFvv/+ex544AGSk5OJjY2lvLycN998k6SkpClXSBtd9GdkKtvoo7JEhLq6Ol5//XV2795NZmbmjBSoDyTYQLPVaqWrq4uIiAiio6NxOp0UFhb6PCm2tbX5PSnu3bs3JPcxS+iAHE46Ojq8R7gfPnyYxsZGSkpKfNoMDAwQHx9PVVUVDz/8MI8//jifffbZmLUJNOJ950SEjo4O3n77bcrKynA4HFy7dg2bzeatHf3YY48RGRl5R0sRo1PZGhsbqa2t5datW/T09PDpp58SHz/1ym9ToYHmkNKgXjh57bXXaG5uxhjDsmXL+PDDDwHf9egFCxZw5MgR1q9fz+DgIDt37hy3UMyxY8d44YUXAl4zxrBu3TqNeE9geA14yZIltLW1YbFY8Hg8/PDDDzQ0NFBRUcH+/fvp7+/3K/Af7Ax6+LxEi8XC0qVLERF6enooKyvDZrNhtVrZvn07L774ItnZ2TN8x5OjgebQ0hlyGAo24t3U1MSXX34ZcOY2Xm70RLnOfX19ZGRkcPHiRaKjozl58iTLli2b/hudRW7evOlT4N/tdrN48WKfAv8xMTETLjf09vaSn59Pe3s7RUVFPr9XEQnpluKCggKKi4uJiooaM9AcTFxDBUVnyLPVTEa8BwcHycrK8nkEdTqdPjOeo0ePcv/999Pa2kpJSQl5eXnzsaC4j7vuuovU1FRv5UAR4bfffvMW+C8tLeXnn3/mkUce8dllOJyXKyLU1NTwxhtvkJWVRWFhod/gPROD8Xj/3DMzM8nPz/cGml999VW/QLMKLR2QZ5nKykreeecdamtrsVgsAduMlxsdzCNoeXk5BQUFAKSnp3tzf7UgzP8ZY4iOjmbDhg3eTAKPx0N7ezsNDQ1UV1dz8OBBent7iY+P59dff+Xuu++moqKCRx99NGT9DLYg1+7du9myZYvf68EU51LTRwfkWSY7O5u+vj7S0tIASE1N9Yt4j5cbrQXFZ05ERAQ2mw2bzcb27duB2/nM3377LRUVFezbty9kGRTBGBloLisrY8WKFX5tgqnmpqaPDsizTGtra8DXrVYrLpcLuF3D+Jtvvpnxvly5coWMjAw6OzsxxvDSSy+Rk5Pj06ampoZnnnmG2NhYALZu3TrlnYyz0cKFC701oUNhMjW8e3p66OvrwxjD008/PS2BZnVndECeZ6azoPiCBQs4dOgQq1atore3l+TkZNLS0vwi8GvWrOH06dMzc0PKx2RqeLe2tgZ86hn5zx1uV0UMlJ+spl/4PD+pkBj5CNrf309JSQlOp9OnzXBBcWDcguJLly5l1apVAERGRpKQkKBHX4UJEaG0tNRv278KbzogzzMjH0ETEhJ4/vnnSUxMZN++fZw6dQqAXbt20d3djd1u57333gvq1OL29nYuXbrE6tWr/a7V19fjcDjYuHEjbrd72u9J+aurq+Ohhx4aszDRcJ56cnIyRUVFIe6dGovmIas7duPGDdauXcvevXvZunWrz7Xr168TERHBPffcg8vlIicnhx9//DHg50xUf0NEyMnJweVyYbFYKC4u9s7Q55NQ1fBW0yq4FCURmcyXUj76+/tl3bp1cujQoaDax8TESFdX16SviYicOXNGNmzYIB6PR+rr6yUlJWVKfZ7rbt26JQ8++KBcuXIlqPZvvfWWHDx4cIZ7Ne8FNcbqkoWaMhFh165dJCQkkJubG7DNL7/8ggw9hZ0/fx6PxzPlE4fLy8vJyMjAGENqaio9PT10dHRMuf9zVTA1vHt7e73ff/311wFT3lTo6YCspuzcuXMcP36cs2fPkpSURFJSEi6Xi8LCQgoLC4HbQcEVK1bgcDh45ZVXKCkpGXODyUTrmoFyqOdyEPHzzz8nMTGRiIgIv+WbAwcOYLfbWb58ubc64LDhGt4//fQTq1evxm6343Q6vbnonZ2dPPnkkzgcDlJSUti8ebOWyQwXwU6lRZcs1Ay7evWqiIh0dnbKypUrpba21uf65s2bpa6uzvvzU089JRcuXPD7nJaWFnE4HN6vyMhIOXz4sE+b6upqiYqK8rbZv3//DNzRnbl8+bK0tLTI2rVrfe7T7XbLypUr5ebNm9LW1iY2m00GBgb83v/cc8/JiRMnRETk5Zdflg8++CBkfVd+dMlCzS6B6m+Mvh7MNt7ly5fT3NxMc3MzFy9exGKxeHcujrRmzRpvu3DcrJKQkBDwINPy8nK2bdvGokWLiI2NxW63+/2uRISzZ8+Snp4OwI4dO/jqq69C0m81dTogq7AQzLqm0+nkk08+QURoaGjg3nvv9W79HUtVVRVxcXHExMTMWN9DLZilm+7ubu677z5vIfy5vrwzV+hOPRUWxqq/MbwWvWfPHjZt2oTL5cJut2OxWPj4448n/NxAZyIOG86PtlqtvPvuu3/IluBgUtjU/DHZPGSlZg1jzJ+Aa0CiiHSOuhYFeETkhjFmE/B3EZn5452nwBhTA/xVRJqGfv4bgIgcGPr5H0CBiNSPeI8BuoAlIjJgjHliqM36UPdfBU+XLNRcthH41+jBGEBErovIjaHvXcBCY8xsKWd3CthmjFlkjIkF/gz4LCLL7ZlWNZA+9NIOoDykvVSTpgOymsv+ApwIdMEYs2RoFokxJoXbfwvdIezbhIwxzxpjrgJPAGeGZsKIiBsoBS4DlUCWiAwOvcdljLEOfUQekGuMaQWigaOhvgc1ObpkoeYkY8xi4N+ATUT+M/TaHgARKTTGZAOZwADwXyBXRP75R/VXKdABWSmlwoYuWSilVJjQAVkppcKEDshKKRUmdEBWSqkwoQOyUkqFCR2QlVIqTPwP8bPFTJbR7ssAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b4f05c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.gca(projection='3d')\n",
    "x1, x2 = np.meshgrid(np.arange(-10, 10, 1), np.arange(-10, 10, 1))\n",
    "y = sigmoid(b + w[0]*x1 + w[1]*x2)\n",
    "ax.plot_surface(x1, x2, y, rstride=1, cstride=1, cmap=plt.cm.coolwarm, linewidth=0, antialiased=False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function we posed above can already be considered a neural network. But let's complicate things a bit further, by adding a hidden layer. Neurons can be arranged in layers. So instead of having just two input neurons and an output neuron, let's place a layer of three neurons in the middle. \n",
    "\n",
    "Now it looks like this:\n",
    "\n",
    "![2,3,1 net](https://ml4a.github.io/images/figures/neural_net_231.png)\n",
    "\n",
    "It's important to realize that everything works as it did before. We start at the input layer on the left, and compute each middle neuron individually. Once they are calculated, we can perform the same operation for the output neuron and get the final result. We still have a function that takes in two inputs and outputs one final value. \n",
    "\n",
    "Note: it is typical to skip the non-linearity for the output layer. So we have sigmoids on the 3 middle layer neurons, but not the final output neuron.\n",
    "\n",
    "Compared to our earlier linear regression, we have complicated things in two ways:\n",
    "1) The addition of a non-linear sigmoid to each neuron's activation function (except for in the output neuron).\n",
    "2) The addition of hidden layer in the middle.\n",
    "\n",
    "The end result of this is our neural net is much more \"flexible.\" It can bend and curve in various ways, depending on how the parameters are set. But it comes at a heavy cost: we can no longer do linear regression on it, and doing gradient descent on it will be more difficult, as we shall see.\n",
    "\n",
    "Let's implement this in code. We now have two layers of weights. Let's call them $W1$ and $W2$. $W1$ has $2 \\times 3$ connections, and $W2$ has $3 \\times 1$ connections. We can create random initial weights with numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W1= [[ 0.55604015 -1.30425457 -1.51214943]\n",
      " [-0.06828511 -0.63519787 -0.59453199]]\n",
      "W2= [[-0.09451191]\n",
      " [-2.21387541]\n",
      " [-0.4512179 ]]\n"
     ]
    }
   ],
   "source": [
    "W1 = np.random.randn(2, 3)\n",
    "W2 = np.random.randn(3, 1)\n",
    "\n",
    "print(\"W1=\", W1)\n",
    "print(\"W2=\", W2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's just keep all the biases, $b$ fixed to 0 for now. We'll bring them back layer.\n",
    "\n",
    "To do the full operation from the input layer to the output layer is called a \"forward pass.\" The next cell will implement a forward pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted 2.56 for example 0, actual 1.60, total cost 201434.60\n"
     ]
    }
   ],
   "source": [
    "X, y = data, labels\n",
    "\n",
    "# first layer weighted sum z\n",
    "z = np.dot(X, W1)\n",
    "\n",
    "# project z through non-linear sigmoid\n",
    "z = sigmoid(z)\n",
    "\n",
    "# do another dot product at end (sigmoid is omitted)\n",
    "y_pred = np.dot(z, W2)\n",
    "\n",
    "# what is our cost\n",
    "error = cost(y_pred, y)\n",
    "\n",
    "print('predicted %0.2f for example 0, actual %0.2f, total cost %0.2f'%(pred_y[0], y[0], error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's turn this basic operation into We can turn this into a class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "class Neural_Network(object):\n",
    "    def __init__(self, n0, n1, n2):        \n",
    "        self.n0 = n0\n",
    "        self.n1 = n1\n",
    "        self.n2 = n2\n",
    "        \n",
    "        # initialize weights\n",
    "        self.W1 = np.random.randn(self.n0, self.n1)\n",
    "        self.W2 = np.random.randn(self.n1 ,self.n2)\n",
    "        \n",
    "    def predict(self, x):\n",
    "        z = np.dot(x, self.W1)\n",
    "        z = sigmoid(z)\n",
    "        y = np.dot(z, self.W2)\n",
    "        return y\n",
    "  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate a neural network with 2 input neurons, 3 hidden neurons, and 1 output neuron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = Neural_Network(2, 3, 1)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to do a forward pass, we can simply run the networks `predict` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted 2.56 for example 0, actual 1.60, total cost 211597.43\n"
     ]
    }
   ],
   "source": [
    "X, y = data, labels\n",
    "y_pred = net.predict(X)\n",
    "error = cost(y_pred, y)\n",
    "\n",
    "print('predicted %0.2f for example 0, actual %0.2f, total cost %0.2f'%(pred_y[0], y[0], error))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient descent\n",
    "\n",
    "To review, we have a 2x3x1 neural network with 9 weights and 4 biases for 13 total parameters. How can we optimize the parameters to minimize our cost function.\n",
    "\n",
    "As with our linear regression example from before, we know we can solve it with gradient descent. Actually, now we *have* to solve it with gradient descent! Linear regression could actually be solved analytically, but for neural networks, no analytical solution can be used to find the parameters, since it is non-linear.\n",
    "\n",
    "So how will gradient descent work here? Recall that the goal of gradient descent is to find the gradient of the loss function $J$ with respect to the parameters $w$ (note, we are including the biases in $w$).\n",
    "\n",
    "$$\\nabla J = \\left[ \\frac{\\partial J}{\\partial w_1}, \\frac{\\partial J}{\\partial w_2}, ..., \\frac{\\partial J}{\\partial w_n} \\right]$$\n",
    "\n",
    "So that we may be able to apply an iterative update rule to each of the parameters:\n",
    "\n",
    "$$ w_i := w_i - \\alpha \\cdot \\frac{\\partial J}{\\partial w_i} $$\n",
    "\n",
    "Let's start with the simplest way of calculating the gradient, the numerical (experimental way). Recall that the definition of any derivative is:\n",
    "\n",
    "$$f^\\prime(x) = \\lim_{\\Delta x\\to 0} \\frac{f(x + \\Delta x) - f(x)}{\\Delta x} $$\n",
    "\n",
    "Thus, the partial derivative of the loss with respect to any weight can be calculated as:\n",
    "\n",
    "$$ \\frac{\\partial J}{\\partial w_i} = \\lim_{\\Delta w_i \\to 0} \\frac{J(w_i + \\Delta w_i) - J(w_i)}{\\Delta w_i} $$\n",
    "\n",
    "This provides us a very simple, straightforward way to get the gradient. We just pick some small $\\Delta w_i$ and then for each weight, compute the value of the loss at the current $w_i$ and at $w_i+\\Delta w_i$, and take the difference.\n",
    "\n",
    "Note that while this method is very simple, it is also very slow and computationally inefficient. That's because it requires doing a forward pass of the network for every single parameter, for *each* update. At 13 weights, that's no big deal, but if our network has 1,000,000 weights, that's at least 1,000,000 forward passes, each of which have at least 1,000,000 multiplications. We shall see later that there exists a fast and efficient way of computing the gradient, called _Backpropagation_. But for now, we will use the slow way because it is much easier to understand and implement. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next cell, we implement the numerical method for calculating the gradient, given a network and dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "def get_gradient(net, X, y):\n",
    "    w_delta = 1e-8\n",
    "\n",
    "    # get the current value of the loss, wherever the parameters are\n",
    "    y_pred_current = net.predict(X)\n",
    "    error_current = cost(y_pred_current, y)\n",
    "\n",
    "    # grab the current weights and copy them (so we can restore them after modification)\n",
    "    dw1, dw2 = np.zeros((net.n0, net.n1)), np.zeros((net.n1, net.n2))\n",
    "    W1, W2 = np.copy(net.W1), np.copy(net.W2) \n",
    "    \n",
    "    # for the first layer, iterate through each weight, \n",
    "    # perturb it slightly, and calculate the numerical \n",
    "    # slope between that loss and the original loss\n",
    "    for i, j in itertools.product(range(net.n0), range(net.n1)):\n",
    "        net.W1 = np.copy(W1)\n",
    "        net.W1[i][j] += w_delta\n",
    "        y_pred = net.predict(X)\n",
    "        error = cost(y_pred, y)\n",
    "        dw1[i][j] = (error - error_current) / w_delta\n",
    "\n",
    "    # do the same thing for the second layer\n",
    "    for i, j in itertools.product(range(net.n1), range(net.n2)):\n",
    "        net.W2 = np.copy(W2)\n",
    "        net.W2[i][j] += w_delta\n",
    "        y_pred = net.predict(X)\n",
    "        error = cost(y_pred, y)\n",
    "        dw2[i][j] = (error - error_current) / w_delta\n",
    "\n",
    "    # restore the original weights\n",
    "    net.W1, net.W2 = np.copy(W1), np.copy(W2)\n",
    "\n",
    "    return dw1, dw2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above function `get_gradient` calculates the gradient of a 2-layer network `net`, given a dataset `X`, `y`. In the next cell, we now go through the learning procedure. We load our dataset, instantiate a neural network, and train it on the data using the gradient method made above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial cost = 85.579\n",
      "final cost = 0.562\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAGbRJREFUeJzt3Xlwm4d55/HvA4CHRFKkKFKyLouSb9mSbZlR7FixJ3aaOI5zbdqOO23ujtPZdCfZ7k4mmcxk2tnuTrs7m7a728brNkmTxrmao/GmTRq7ceQ4kiVTsixbh3XL1mGJFHVRFi/g2T/wgoRoEgRlAO/7Ar/PDPSCL16874MX4E8vn/eAuTsiIhIfibALEBGRmVFwi4jEjIJbRCRmFNwiIjGj4BYRiRkFt4hIzCi4RURiRsEtIhIzCm4RkZhJlWOmHR0d3tXVVY5Zi4hUpS1btvS5e2cx05YluLu6uujp6SnHrEVEqpKZHS52WrVKRERipqjgNrM2M/u+me02s11mdke5CxMRkckV2yr5K+Bn7v6bZlYPzC5jTSIiUsC0wW1mrcBdwEcB3H0YGC5vWSIiMpViWiXLgV7ga2b2nJn9nZk1TZzIzB4ysx4z6+nt7S15oSIiklVMcKeANcCX3f1W4ALwuYkTufsj7t7t7t2dnUUd0SIiIpehmOA+Ahxx903Bz98nG+QiIhKCaYPb3V8FXjGz64JR9wI7S13IvpPn+a2HN/D5H24v9axFRKpKsUeV/Afg0eCIkgPAx0pdyMBQmmcPnWZoNFPqWYuIVJWigtvdtwHd5SzExpZVzqWIiMRfZM6ctCC5HSW3iEgh0QnuYJtbW9wiIoVFJ7hzW9wKbhGRgiIX3Bklt4hIQdEJ7rHdkyIiUkh0glutEhGRokQvuHVUiYhIQdEJbh1VIiJSlOgE99gWt4iIFBKd4A6Grk1uEZGCohPc2uIWESlKZIJ7bJtbyS0iUlBkgltb3CIixYlOcAdD9bhFRAqLTnAHm9yKbRGRwqIT3MFQG9wiIoVFJ7h15qSISFGiE9w6c1JEpCjRCW5dZEpEpCiRCW4RESlOZIJ7fItbm9wiIoVEKLh1OKCISDGiE9zBUBvcIiKFRSe4dTigiEhRohPcOhxQRKQo0QluXWRKRKQo0QnuYKgtbhGRwqIT3LmjSpTcIiIFpYqZyMwOAeeBNDDq7t2lLkStEhGR4hQV3IG3uXtfuQrR9bhFRIoTvVZJyHWIiERdscHtwM/NbIuZPTTZBGb2kJn1mFlPb2/vjAvRzkkRkeIUG9zr3H0N8C7gU2Z218QJ3P0Rd+929+7Ozs4ZF6JrlYiIFKeo4Hb3o8HwJPAjYG2pCxk7AafUMxYRqTLTBreZNZlZS+4+8A7gxZJXMtYrKfmcRUSqSjFHlSwAfhTsPEwB33L3n5W6kFyrJKNWiYhIQdMGt7sfAG4udyF1iezG/2hGwS0iUkhkDgdMJbOb3ApuEZHCohPciWxwpzNORuEtIjKlyAS3mVEXbHWPZDIhVyMiEl2RCW6AumTQ505ri1tEZCqRCu5cu2QkrS1uEZGpRCq461NJAC6OpEOuREQkuiIV3Cs6mgDYdfxcyJWIiERXpIJ7zbK5AGw5fDrkSkREoitSwX1bENw9hxTcIiJTiWRwb3vlDIPqc4uITCpSwd3eVM/1V7QwNJph68va6hYRmUykghvgzqs7ANiw71TIlYiIRFPkgvstV80DYMP+sn29pYhIrEUuuNcubyeZMJ4/cpbzgyNhlyMiEjmRC+6WxjpWL2klnXGePdQfdjkiIpETueAGuPOqbJ/71+pzi4i8TiSDe7zPreAWEZkoksG9Ztlc6lMJdh0/x6mBobDLERGJlEgGd2NdkrVd7QA8vU9Hl4iI5ItkcAPcdW22z71+T2/IlYiIREtkg/vua+cD8NSePn2VmYhInsgG97ULmrliTiN9A0PselWXeRURyYlscJvZWLvkqT3qc4uI5EQ2uAHuurYTgPV7ToZciYhIdEQ6uNdd3UHCsl+sMDA0GnY5IiKREOngbptdz81L2xhJOxt1Mo6ICBDx4Aa4O2iXPKXDAkVEgBgE93ifW8EtIgIzCG4zS5rZc2b2k3IWNNHNS9ponVXHy/2vcajvQiUXLSISSTPZ4v40sKtchUwlmTDWXaOzKEVEcooKbjNbArwb+LvyljO5u69Rn1tEJKfYLe6/BD4LZKaawMweMrMeM+vp7S1twOb63Bv2n2JoVN/+LiK1bdrgNrMHgJPuvqXQdO7+iLt3u3t3Z2dnyQoEuKK1kesWtHBxJM2Ww/r2dxGpbcVscd8JvNfMDgHfAe4xs2+WtapJ6PR3EZGsaYPb3T/v7kvcvQt4EPiFu/9e2Sub4K1Bn/tXe9XnFpHaFvnjuHPWLm+nIZVgx7Fz9OlbcUSkhs0ouN39l+7+QLmKKaSxLsna5cG34uxVu0REaldstrgB7sodFqh2iYjUsHgF97W5Pncf7vpWHBGpTbEK7msXNDO/pYHe80PsfvV82OWIiIQiVsFtZjq6RERqXqyCG3Q8t4hI7IJ73dUdmMHmQ/1cHNbp7yJSe2IX3POaG7hpUSvDoxk2H+oPuxwRkYqLXXADvPWaXLtEfW4RqT0xDW7toBSR2hXL4L5t2Vxm1yfZc2KAV88Ohl2OiEhFxTK461MJ7lgxD9BZlCJSe2IZ3DDe5/6VrlsiIjUmtsGd+x7KjftP6fR3EakpsQ3uqzqb6WxpoG9giH0nB8IuR0SkYmIb3GbGW67K9rk37D8VcjUiIpUT2+AG8oJbfW4RqR0xD+5sn/uZA/2kM+pzi0htiHVwL22fzZK5szh7cYRdx8+FXY6ISEXEOriBseO5N6rPLSI1IvbB/Zar1ecWkdoS++C+Y0W2z735YD8j6UzI1YiIlF/sg/uK1kZWdDZxYTjN9iNnwy5HRKTsYh/cMH5Y4Ea1S0SkBlRJcAenvx/QDkoRqX5VEdy3B0eW9Bw6zeCIvs5MRKpbVQR3e1M911/RwtBohudePhN2OSIiZVUVwQ1wR9DnfkbtEhGpclUT3Ll2ifrcIlLtpg1uM2s0s81m9ryZ7TCzP6lEYTP15uXtmMG2l8+ozy0iVa2YLe4h4B53vxm4BbjPzG4vb1kz1za7nhuumMNwOsPWw6fDLkdEpGymDW7Pyn1TQV1wi+Sl+NTnFpFaUFSP28ySZrYNOAk87u6bylvW5VGfW0RqQVHB7e5pd78FWAKsNbObJk5jZg+ZWY+Z9fT2hvPN62tzfe5XznBxWH1uEalOMzqqxN3PAE8C903y2CPu3u3u3Z2dnaWqb0ZaZ9Vx46I5jKSdLepzi0iVKuaokk4zawvuzwJ+A9hd7sIu19j1uQ/ouiUiUp2K2eJeCDxpZtuBZ8n2uH9S3rIu3/gOyv6QKxERKY/UdBO4+3bg1grUUhLdXe0kDJ5/5QwXhkZpapj2JYqIxErVnDmZM6exjlWLWxnNqM8tItWp6oIbdFigiFS36gxunYgjIlWsKoP7TV3tJBPG9iNnGRgaDbscEZGSqsrgbm5IsWpxK+mM03NIR5eISHWpyuAG9blFpHpVbXDreG4RqVZVG9zdy+aSShgvHj3L+cGRsMsRESmZqg3upoYUq5dk+9zPqs8tIlWkaoMbxvvcapeISDWp6uDO9bk37tcOShGpHlUd3Lctm0td0thx7CxnL6rPLSLVoaqDe3Z9ipuXtJFxePag2iUiUh2qOrhB30MpItWn6oNbJ+KISLWp+uBec+Vc6pMJdh4/x9nX1OcWkfir+uCeVZ/klqVtuMOmg9rqFpH4q/rghvzLvGoHpYjEX20E94p2QH1uEakONRHca66cS30qwa7j5zh9YTjsckRE3pCaCO7GuiS3Lm0DYJOO5xaRmKuJ4AYdzy0i1aNmgnv8glMKbhGJt5oJ7luWttGQSrD71fOcGhgKuxwRkctWM8HdWJdkzZVzAdisPreIxFjNBDfkXeZV7RIRibGaCm71uUWkGtRUcN+8tJXGugR7TgzQpz63iMTUtMFtZkvN7Ekz22lmO8zs05UorBwaUkm6l2XPotyk099FJKaK2eIeBf6Tu68Ebgc+ZWYry1tW+eROf9+wvy/kSkRELs+0we3ux919a3D/PLALWFzuwsrlzqs7APjlS724e8jViIjM3Ix63GbWBdwKbCpHMZVw85I2OprrOXrmIntODIRdjojIjBUd3GbWDPwA+Iy7n5vk8YfMrMfMenp7e0tZY0klEsbbrpsPwBO7ToRcjYjIzBUV3GZWRza0H3X3H042jbs/4u7d7t7d2dlZyhpL7t4bssH9i90nQ65ERGTmijmqxICvALvc/UvlL6n81l3TSX0ywdaXT9Ovy7yKSMwUs8V9J/Ah4B4z2xbc7i9zXWXV3JDizSvacYcntdUtIjFTzFElT7u7uftqd78luP1LJYorp3uvV7tEROKpps6czHfvDQsAWL+nl8GRdMjViIgUr2aDe2n7bG5cNIeBoVF+tVcn44hIfNRscAO8e/VCAP55+7GQKxERKV5tB/eqbHA/vvOE2iUiEhs1HdzL5jWxanErF4bT/PKl6J40JCKSr6aDG+CBXLvkheMhVyIiUpyaD+77g3bJEztPMDA0GnI1IiLTq/ngXto+m7XL27k4ktZOShGJhZoPboDfum0JAN/rORJyJSIi01Nwk22XNNUn2XL4NPt7dalXEYk2BTfQ1JDigdWLAPhezyshVyMiUpiCO/Dbb1oKwD/2HNEx3SISaQruwJor21i1uJX+C8P8eNvRsMsREZmSgjtgZnx8XRcAX3n6oL6PUkQiS8Gd592rFjG/pYE9Jwb49b5TYZcjIjIpBXee+lSCD9+xDIC/fnJfyNWIiExOwT3Bh27voqUxxcYDp9i4X1vdIhI9Cu4JWmfX8fvrVgDwF0/sUa9bRCJHwT2Jj63ronVWHZsP9utLFkQkchTck5jTWMcf3H0VAP/lJzsZSWdCrkhEZJyCewofX9dF17zZ7D05wNc3HAq7HBGRMQruKTSkknzxPSsB+Ksn9nLi3GDIFYmIZCm4C7jn+gW8/Yb5nB8a5bPf364dlSISCQruafy3D6yibXYd6/f08s1NL4ddjoiIgns68+c08qfvvwmA//rPO9l57FzIFYlIrVNwF+GB1Yv4zduWMDiS4ZPf7OH0heGwSxKRGqbgLtKfvv8mVi1u5ZX+i/z7R7cyNKpLv4pIOBTcRWqsS/J/P3QbHc0NbDxwik9/exujOr5bREIwbXCb2VfN7KSZvViJgqJsUdssvvHxtbQ0pvjZjlf57A+2K7xFpOKK2eL+e+C+MtcRGysXzeFrH30Ts+qS/HDrUT71ra36xhwRqahpg9vdnwL6K1BLbHR3tfMPn1jLnMYU/7rjBB/92mb6tcNSRCpEPe7L1N3Vznc/eQedLQ08c6Cf9/zvp3nx6NmwyxKRGlCy4Dazh8ysx8x6ent7SzXbSLth4Rwe+8M7uXlpG0fPXOSDX97ANzYeIpPRGZYiUj4lC253f8Tdu929u7Ozs1SzjbyFrbP43idv58E3LWVoNMMXf7yDD391M8fOXAy7NBGpUmqVlEBDKsmffXA1f/O7a5g7u46n9/XxG19az8Pr9+t4bxEpuWIOB/w2sBG4zsyOmNknyl9WPN2/aiE//493884bF3BhOM2f/XQ39/3lr3h85wldoEpESsbKESjd3d3e09NT8vnGyfo9vfzJ/9vBgd4LAKxe0spn3n4Nb7tuPmYWcnUiEjVmtsXdu4uaVsFdPsOjGb75zGH+5pf76RsYAuDGRXP42J3LeWD1QhrrkiFXKCJRoeCOmIvDaR7ddJiH1++nbyB7vPe8pnp+Z+2VfPC2JSzvaAq5QhEJm4I7ogZH0jz2/DG+vuEQO/IuD3vL0jY+cOti7l+1kM6WhhArFJGwKLgjzt3pOXya7z77Cj994TgXhrNHnphlQ/ze6+dzz/ULuGFhi/rhIjVCwR0jF4fTPL7rBP/03FGe3tvHcN5FqzqaG3jz8nbevKKdtcvbuXZ+C4mEglykGim4Y+rC0Ci/3tfHL3af5Be7T3Ly/NAlj7c0prhx0RxuXNTKTYuzwxUdTaSSOhxfJO4U3FXA3TnQd4HNB/vZfLCfTQdOcezs679pvi5pXNk+m+UdzazobGJ5RxNd85pY1NbIFa2NNKR05IpIHCi4q9TJc4O8eOwsO46eyw6PnePI6cKn1nc017OwdRYLWxtZ2NpIe1MD7c31zGuqp71pfNg2u56k2jAioZlJcKfKXYyUzvw5jdwzp5F7rl8wNu7icJqDfReC2wAHei9wuP81jp+5yInzQ/QNDNM3MMwL01y50Aya61M0N6ZobhgftuR+bqijuSFJY32SxlSShroEjakkjXVJGusSY8OG1PiwIZUglUyQShp1iewwlTDtcBV5gxTcMTerPsnKRXNYuWjO6x5LZ5yT5wc5fnaQ42cGefXcIP0Xhui/MDx2OxUMz7w2wvmhUc4PjZa95mQiG+B1yVyYJ6hL2oSAT5BMGImEkTBImJE0wyz7/MSE+9lbMF1ikuly0yQunQ6y/2kZFgyDn80mH5/3M2YYjC0j/7lTzTcx9pjlzTN/edmRiSmWTf70efPM/V84tuzgsez9vMfHxufXOP7c8fuXPp9Lnj9huknmxRTjL3mdM6klb5pLhvmvbcplBlNOeA2F1tOly7Ipp8srf2x826y6su93UnBXsWTCgjbJLLiy8LTpjDMwNJq9DY4yMDTC+cH8n0c5PzjK4GiaoZEMgyPp4JZhaDQ7HAyGQ8Fjw2lnNJNhNO2MpDOMZpx03m1oVF/7JtXniT+6m6vnN5d1GQpuAbIh3zqrjtZZdWVdjrszkgv0jDOadkbTGUYywTAv7NMZJ+NOxskOM07aHQ9+Tmey9183Xd5jl0znedMF/3l4UFO2NnA8GDL2/Fzd+ePzpyOY58TnOtkffMLzM8H9sflOsezx50wzX8guP1fr2D/jz534+nKTjO/imjjeXzdN/vj8+Y3dn2yZk9TiY8/3vPvj43nd+EuXkf/8KWuZuIxpX9ul64kJz5/qtU22/lIV2Fek4JaKMjPqU0a9rigsctn02yMiEjMKbhGRmFFwi4jEjIJbRCRmFNwiIjGj4BYRiRkFt4hIzCi4RURipixXBzSzXuDwZT69A+grYTmlorpmRnXNjOqamWqsa5m7dxYzYVmC+40ws55iL21YSaprZlTXzKiuman1utQqERGJGQW3iEjMRDG4Hwm7gCmorplRXTOjumampuuKXI9bREQKi+IWt4iIFBCZ4Daz+8zsJTPbZ2afq/Cyl5rZk2a208x2mNmng/F/bGZHzWxbcLs/7zmfD2p9yczeWcbaDpnZC8Hye4Jx7Wb2uJntDYZzg/FmZv8rqGu7ma0pU03X5a2TbWZ2zsw+E9b6MrOvmtlJM3sxb9yM15GZfSSYfq+ZfaRMdf0PM9sdLPtHZtYWjO8ys4t56+7hvOfcFnwG9gW1v6Er9U9R14zfu1L/zk5R13fzajpkZtuC8RVZXwWyIdzPV/ZbNsK9AUlgP7ACqAeeB1ZWcPkLgTXB/RZgD7AS+GPgP08y/cqgxgZgeVB7sky1HQI6Joz778DngvufA/48uH8/8FOy34J3O7CpQu/dq8CysNYXcBewBnjxctcR0A4cCIZzg/tzy1DXO4BUcP/P8+rqyp9uwnw2B7VaUPu7ylDXjN67cvzOTlbXhMf/J/DFSq6vAtkQ6ucrKlvca4F97n7A3YeB7wDvq9TC3f24u28N7p8HdgGLCzzlfcB33H3I3Q8C+8i+hkp5H/D14P7Xgffnjf+GZz0DtJnZwjLXci+w390LnXBV1vXl7k8B/ZMscybr6J3A4+7e7+6ngceB+0pdl7v/3N1z38j8DLCk0DyC2ua4+zOeTYBv5L2WktVVwFTvXcl/ZwvVFWw1/zbw7ULzKPX6KpANoX6+ohLci4FX8n4+QuHgLBsz6wJuBTYFo/4w+JPnq7k/h6hsvQ783My2mNlDwbgF7n48uP8qsCCEunIe5NJfprDXV85M11EYNX6c7NZZznIze87M1pvZW4Nxi4NaKlHXTN67Sq+vtwIn3H1v3riKrq8J2RDq5ysqwR0JZtYM/AD4jLufA74MXAXcAhwn+6dapa1z9zXAu4BPmdld+Q8GWxWhHBpkZvXAe4F/DEZFYX29TpjraCpm9gVgFHg0GHUcuNLdbwX+CPiWmc2pYEmRfO/y/A6XbiBUdH1Nkg1jwvh8RSW4jwJL835eEoyrGDOrI/vGPOruPwRw9xPunnb3DPC3jP95X7F63f1oMDwJ/Cio4USuBRIMT1a6rsC7gK3ufiKoMfT1lWem66hiNZrZR4EHgN8NfukJWhGngvtbyPaPrw1qyG+nlKWuy3jvKrm+UsC/A76bV2/F1tdk2UDIn6+oBPezwDVmtjzYinsQeKxSCw/6Z18Bdrn7l/LG5/eHPwDk9nY/BjxoZg1mthy4huwOkVLX1WRmLbn7ZHdsvRgsP7dX+iPAj/Pq+nCwZ/t24Gzen3PlcMlWUNjra4KZrqN/Bd5hZnODNsE7gnElZWb3AZ8F3uvur+WN7zSzZHB/Bdl1dCCo7ZyZ3R58Tj+c91pKWddM37tK/s6+Hdjt7mMtkEqtr6mygbA/X5e7V7PUN7J7Y/eQ/Z/zCxVe9jqyf+psB7YFt/uBfwBeCMY/BizMe84Xglpf4g3u5S9Q1wqye+ufB3bk1gswD/g3YC/wBNAejDfgr4O6XgC6y7jOmoBTQGveuFDWF9n/PI4DI2R7h5+4nHVEtue8L7h9rEx17SPb68x9zh4Opv1g8B5vA7YC78mbTzfZIN0P/B+CE+dKXNeM37tS/85OVlcw/u+BP5gwbUXWF1NnQ6ifL505KSISM1FplYiISJEU3CIiMaPgFhGJGQW3iEjMKLhFRGJGwS0iEjMKbhGRmFFwi4jEzP8HOFy4dr83uAYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b8d6c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load the data and labels\n",
    "X, y = data, labels.reshape((len(labels),1))\n",
    "\n",
    "# it's always a good idea to normalize the data between 0 and 1\n",
    "X = X/np.amax(X, axis=0)\n",
    "y = y/np.amax(y, axis=0)\n",
    "\n",
    "# create a 2x3x1 neural net\n",
    "net = Neural_Network(2, 3, 1)    \n",
    "\n",
    "# what is the current cost?\n",
    "y_orig = net.predict(X)\n",
    "init_cost = cost(y_orig, y)\n",
    "print(\"initial cost = %0.3f\" % init_cost)\n",
    "\n",
    "# Set the learning rate, and how many epochs (updates) to try\n",
    "n_epochs = 2000\n",
    "learning_rate = 0.01\n",
    "\n",
    "# for each epoch, calculate the gradient, then subtract it from the parameters, and save the cost\n",
    "errors = []\n",
    "for i in range(n_epochs):\n",
    "    dw1, dw2 = get_gradient(net, X, y)\n",
    "    net.W1 = net.W1 - learning_rate * dw1\n",
    "    net.W2 = net.W2 - learning_rate * dw2\n",
    "    y_pred = net.predict(X)\n",
    "    error = cost(y_pred, y)\n",
    "    errors.append(error)\n",
    "        \n",
    "# plot it\n",
    "plt.plot(range(0, len(errors)), errors, linewidth=2)\n",
    "\n",
    "# what is the final cost?\n",
    "y_pred = net.predict(X)\n",
    "final_cost = cost(y_pred, y)\n",
    "print(\"final cost = %0.3f\" % final_cost)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we have implemented a neural network with manual numerical gradient computation for gradient descent and used it to optimize a small network.\n",
    "\n",
    "In the next guide, we are going to use [Keras](http://www.keras.io), a high-level machine learning library which we will use throughout the class from this point forward."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}