chap10-introduction-to-ANN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 10 - Introduction to Artificial Neural Networks with Keras"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:04.389615Z",
     "start_time": "2020-01-03T11:05:02.020111Z"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import sklearn\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "%matplotlib inline\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:04.407854Z",
     "start_time": "2020-01-03T11:05:04.391500Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version:     sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)\n",
      "TensorFlow version: 2.0.0\n",
      "Keras version:      2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "print('Python version:    ', sys.version_info)\n",
    "print('TensorFlow version:', tf.__version__)\n",
    "print('Keras version:     ', keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Main content:</font>**\n",
    "1. introduce ANNs, starting with a quick tour of the very first ANN architectures, leading up to MLPs (Multi-Layer Perceptrons).\n",
    "\n",
    "2. implement NNs using the popular Keras API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ANNs are at the very core of Deep Learning. They are versatile, powerful, and scalable, making them ideal to tackle large and highly complex Machine Learning tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 Artificial Neurons"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should drop the biological analogy (类比) altogether (e.g., saying **units** rather than neurons), lest (以免) we strict our creativity to biologically plausible (看似可信的) systems.\n",
    "\n",
    "We are now witnessing yet another wave of interest in ANNs. A few good reasons to believe that this time is different:\n",
    "\n",
    "+ **A huge quantity of data available** to train NNs, and ANNs often **outperform other ML techs** on very large and complex problems.\n",
    "\n",
    "\n",
    "+ The **tremendous increase in computing power**, due to Moore's law (the number of components in integrated has doubled about every 2 years over the last 50 years), but also thanks to **gaming industry**, which has stimulated the production of powerful **GPU** cards by the millions. Moreover, **cloud platforms** have made this power accessible to everyone.\n",
    "\n",
    "\n",
    "+ The training algorithms have been improved. \n",
    "\n",
    "\n",
    "+ Some theoretical limitations of ANNs have turned out to be benign in practice.\n",
    "\n",
    "\n",
    "+ ANNs seem to have entered a virtuous circle (良性循环) of funding and process.\n",
    "\n",
    "### 1.1 The Perceptron (感知机）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **<font color='red'>perceptron</font>** is one of the simplest ANN architectures, based on a slightly different artificial neuron called a **<font color='red'>threshold logic unit (TLU)</font>** or sometimes a **<font color='red'>linear threshold unit (LTU)</font>**.\n",
    "\n",
    "![threshold linear unit](./figs/chap10-figs/threshold-linear-unit.png)\n",
    "\n",
    "\n",
    "The inputs and output are numbers (instead of binary on/off values), and each input connection is associated with a weight. The **TLU** computes a weighted sum of its inputs ($z = w_1x_1 + w_2x_2 + ⋯ + w_nx_n = x^Tw$), then applies a step function to that sum and outputs the result: $h_w(x) = step(z)$, where $z = x^Tw$.\n",
    "\n",
    "The **most common step function used in Perceptrons** is the **<font color='red'>Heaviside step function</font>**. Sometimes the **<font color='red'>sign function</font>** is used instead.\n",
    "\n",
    "$$ heaviside(z)= \\begin{cases}\n",
    "0, & \\text {if z < 0} \\\\ \n",
    "1, & \\text{if z $\\geq$ 0} \\end{cases}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "$$ sgn(z)= \\begin{cases}\n",
    "-1, & \\text {if z < 0} \\\\ \n",
    "0, & \\text {if z = 0} \\\\\n",
    "1, & \\text{if z $\\gt$ 0} \\end{cases}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "**A single TLU can be used for simple linear binary classification.**\n",
    "\n",
    "\n",
    "A Perceptron is simply composed of a single layer of TLUs, with each TLU connected to all the inputs. When all the neurons in a layer are connected to every neuron in the previous layer (i.e., its input neurons), the layer is called a **<font color='red'>fully connected layer</font>**, or a **<font color='red'>dense layer</font>**. \n",
    "\n",
    "Computing the outputs of a fully connected layer:\n",
    "$$h_{w,b}(X) = \\phi(XW+b)$$\n",
    "\n",
    "> $\\phi$ is called the **activation function**: when the artificial units are TLUs, it's a step function.\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='blue'>How is a Perceptron trained?</font>**\n",
    "\n",
    "**Hebb's rule or Hebb's learning** -- “Cells that fire together, wire together”; that is, the connection weight between two neurons tends to increase when they fire simultaneously. Perceptrons are trained using a variant of this rule that takes into account the error made by the NN when it makes a prediction. **The Perceptron learning rule reinforces加强 connections that help reduce the error**. More specifically, the Perceptron is fed one training instance at a time, and for each instance it makes its predictions. For every output neuron that **produced a wrong prediction**, it **reinforces the connection weights from the inputs** that would have contributed to the correct prediction.\n",
    "\n",
    "**Perceptron learning rule (weight update):**\n",
    "\n",
    "$$w_{i, j}^{(next  step)}=w_{i, j} + \\eta(y_j - \\hat {y_{j}})x_i$$\n",
    "\n",
    "+ $w_{i, j}$ is the connection weight between the $i^{th}$ input neuron and the $j^{th}$ output neuron.\n",
    "\n",
    "+ $x_i$ is the $i^{th}$ input value of the current training instance.\n",
    "\n",
    "+ $\\hat {y_{j}}$ is the output of the $j^{th}$ output neuron for the current training instance.\n",
    "\n",
    "+ $y_j$ is the target output of the $j^{th}$ output neuron for the current training instance.\n",
    "\n",
    "+ $\\eta$ is the learning rate.\n",
    "\n",
    "<br>\n",
    "\n",
    "**The decision boundary of each output neuron is linear, so Perceptrons are incapable of learning complex patterns (just like Logistic Regression classifiers)**. However, if the training instances are linearly separable, this algorithm would converge to a solution. This is called the **<font color='red'>Perceptron convergence theorem</font>**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:04.636874Z",
     "start_time": "2020-01-03T11:05:04.410107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A `Perceptron` class with a single-TLU network in Sklearn\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.linear_model import Perceptron\n",
    "\n",
    "iris = load_iris()\n",
    "x = iris.data[:, (2, 3)]  # only two features\n",
    "y = (iris.target == 0).astype(np.int)\n",
    "\n",
    "perceptron_clf = Perceptron(max_iter=1000, tol=1e-3, random_state=42)\n",
    "perceptron_clf.fit(x, y)\n",
    "\n",
    "perceptron_clf.predict([[2, 0.5]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>The Perceptron learning algorithm strongly resembles Stochastic Gradient Descent</font>**. In fact, Scikit-Learn’s `Perceptron` class is equivalent to using an `SGDClassifier` with the following hyperparameters: `loss=\"perceptron\"`, `learning_rate=\"constant\"`, `eta0=1` (the learning rate), and `penalty=None` (no regularization).\n",
    "\n",
    "Note that contrary to Logistic Regression classifiers, **Perceptrons do not output a class probability**; rather, they **make predictions based on a hard threshold**. **<font color='blue'>This is one reason to prefer Logistic Regression over Perceptrons.</font>**\n",
    "\n",
    "\n",
    "**<font color='blue'>A serious weakness of Perceptrons</font>:** they are incapable of solving some trivial problems (e.g., the Exclusive OR (XOR)) classification problem. This is true of any other linear clf model (such as LR clf).\n",
    "\n",
    "It turns out that some of the limitations of Perceptrons can be eliminated (消除) by stacking multiple Perceptrons. The resulting ANN is called a **<font color='red'>Multilayer Perceptron (MLP)</font>**. **An MLP can solve the XOR problem.**\n",
    "\n",
    "> **XOR**: with inputs (0, 0) or (1, 1), the network outputs 0, and with inputs (0, 1) or (1, 0) it outputs 1.\n",
    "\n",
    "![XOR](./figs/chap10-figs/XOR-classification-problem-and-an-MLP-that-solves-it.png)\n",
    "\n",
    "> All connections have a weight equal to 1, except the four connections where the weight is shown."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 The Multiplayer Perceptron and Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An **<font color='red'>MLP</font>** is composed of one input layer, one or more layers of TLUs, called **<font color='red'>hidden layers</font>**, and one final layer of TLUs called the **<font color='red'>output layer</font>**. The layers close to the input layer are usually called the **<font color='red'>lower layers</font>**, and the ones close to the outputs are usually called the **<font color='red'>upper layers</font>**. Every layer except the output layer includes a bias neuron and is fully connected to the next layer.\n",
    "\n",
    "![MLP](./figs/chap10-figs/mlp.png)\n",
    "\n",
    "The signal flows only in one direction (from the inputs to the outputs), so this architecture is an example of a **<font color='red'>feedforward neural network (FNN)</font>**.\n",
    "\n",
    "\n",
    "When an ANN contains a deep stack of hidden layers, it is called a **<font color='red'>deep neural network (DNN)</font>**. The field of Deep Learning studies DNNs, and more generally models containing deep stacks of computations. Even so, many people talk about Deep Learning whenever neural networks are involved (even shallow ones).\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='red'>Train MLPs?</font>**\n",
    "\n",
    "Use **<font color='red'>backpropagation training algorithm</font>**. In short, it is Gradient Descent using an efficient technique for computing the gradients automatically: in just two **passes through the network (one forward, one backward)**, the backpropagation algorithm is able to **compute the gradient of the network’s error with regard to every single model parameter**. In other words, it can **find out how each connection weight and each bias term should be tweaked in order to reduce the error**. Once it has these gradients, it just performs a regular Gradient Descent step, and the whole process is repeated until the network converges to the solution.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "    <center><b>Autodiff</b></center><br> Automatically computing gradients is called <b>automatic differentiation</b>, or <b>autodiff</b>. There are various autodiff techniques, with different pros and cons(有不同的利弊). The one used by backpropagation is called <b>reverse-mode autodiff</b>. It is fast and precise, and is well suited when the function to differentiate has many variables (e.g., connection weights) and few outputs (e.g., one loss). If you want to learn more about autodiff, check out Appendix D.\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='red'>Backpropagation training algorithm:</font>**\n",
    "+ It handles one mini-batch at a time (for example, containing 32 instances each), and it goes through the full training set multiple times. Each pass is called an **<font color='red'>epoch</font>**.\n",
    "\n",
    "\n",
    "+ Each mini-batch is passed to the network’s input layer, which sends it to the first hidden layer. The algorithm then computes the output of all the neurons in this layer (for every instance in the mini-batch). The result is passed on to the next layer, its output is computed and passed to the next layer, and so on until we get the output of the last layer, the output layer. This is the **<font color='blue'>forward pass</font>**: it is exactly like making predictions, except all intermediate results are preserved since they are needed for the backward pass.\n",
    "\n",
    "\n",
    "+ Next, the algorithm **measures the network’s output error** (i.e., it **uses a loss function** that compares the desired output and the actual output of the network, and returns some measure of the error).\n",
    "\n",
    "\n",
    "+ Then it **computes how much each output connection contributed to the error**. This is done analytically by applying the **chain rule** (perhaps the most fundamental rule in calculus), which makes this step fast and precise.\n",
    "\n",
    "\n",
    "+ The algorithm then **measures how much of these error contributions came from each connection in the layer below**, again using the chain rule, working backward until the algorithm reaches the input layer. As explained earlier, this reverse pass efficiently measures the error gradient across all the connection weights in the network by propagating the error gradient backward through the network (**hence the name of the algorithm**).\n",
    "\n",
    "\n",
    "+ Finally, the algorithm **performs a Gradient Descent step to tweak all the connection weights in the network**, using the error gradients it just computed.\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='red'>BP summary:</font>** for each training instance, the backpropagation algorithm first makes a prediction (forward pass) and measures the error, then goes through each layer in reverse to measure the error contribution from each connection (reverse pass), and finally tweaks the connection weights to reduce the error (Gradient Descent step).\n",
    "\n",
    "<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <center><b>随机初始化权重的重要性</b></center><br> <b>It is important to initialize all the hidden layers’ connection weights randomly, or else training will fail.</b> For example, if you <b>initialize all weights and biases to zero</b>, then all neurons in a given layer will be perfectly identical, and thus backpropagation will affect them in exactly the same way, so they will remain identical. In other words, despite having hundreds of neurons per layer, your model will act as if it had only one neuron per layer: it won’t be too smart. If instead you randomly initialize the weights, you break the symmetry and allow backpropagation to train a diverse team of neurons.\n",
    "</div>\n",
    "\n",
    "<br>\n",
    "\n",
    "In order for this BP algorithm to work properly, its author made a key change: they **<font color='blue'>replaced the step function with the logistic (sigmoid) function</font>** (不止logistic 函数，其他激活函数也行). This was essential because the **step function contains only flat segments, so there is no gradient to work with** (Gradient Descent cannot move on a flat surface), while the logistic function has a well-defined nonzero derivative everywhere, allowing Gradient Descent to make some progress at every step.\n",
    "\n",
    "+ **<font color='red'>logistic (sigmoid) function</font>**\n",
    "\n",
    "$$\\sigma(z) = \\frac{1}{1+e^{-z}}$$\n",
    "\n",
    "\n",
    "+ **<font color='red'>the hyperbolic tangent function</font>**\n",
    "\n",
    "$$tanh(z) = 2\\sigma(2z) - 1$$\n",
    "\n",
    "S-shaped, continuous, and differentiable, but its **output value ranges from -1 to 1** (instead fo 0 to 1 in the case of the logistic/sigmoid function). This range **tends to make each layer's output more or less centered around 0 at the beginning of training, which often helps speed up convergence.**\n",
    "\n",
    "\n",
    "+ **<font color='red'>the rectified linear unit function</font>**\n",
    "\n",
    "$$ReLU(z) = max(0, z)$$\n",
    "\n",
    "The ReLU function is continuous but unfortunately **not differentiable at z = 0 (the slope changes abruptly, which can make Gradient Descent bounce around)**, and its derivative is 0 for z < 0. In practice, however, it **works very well and has the advantage of being fast to compute, so it has become the default**. Most importantly, the fact that it **does not have a maximum output value** helps reduce some issues during Gradient Descent.\n",
    "\n",
    "**<font color='red'>Why need nonlinear activation function?</font>**\n",
    "\n",
    "If you chain several linear transformations, all you get is a linear transformation. So, if you don’t have some nonlinearity between layers, then even a deep stack of layers is equivalent to a single layer, and you can’t solve very complex problems with that. Conversely, a large enough DNN with nonlinear activations can theoretically approximate any continuous function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:04.642961Z",
     "start_time": "2020-01-03T11:05:04.639209Z"
    },
    "code_folding": [
     0,
     4,
     9
    ]
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "\n",
    "def relu(z):\n",
    "    return np.maximum(0, z)\n",
    "\n",
    "\n",
    "# scipy.misc.derivative\n",
    "def derivative(f, z, eps=1e-6):\n",
    "    return (f(z+eps) - f(z-eps)) / (2*eps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.051144Z",
     "start_time": "2020-01-03T11:05:04.644659Z"
    },
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3cAAAF5CAYAAAA8ptNPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3xW5f3/8dcnOyEQCEsE2SAuREVEAQ0qilpxoLUuinVbtdo6W+uqP6vYr9UWa6VqRUErKFYFxVGNosgQ2SBbNmEkJCRk39fvj3MnhJCEjHslvJ+PR7zHuc51PvcJ5tyfcy1zziEiIiIiIiKNW1S4AxAREREREZGGU3InIiIiIiLSBCi5ExERERERaQKU3ImIiIiIiDQBSu5ERERERESaACV3IiIiIiIiTYCSOzlkmNlPZnZPCI7zqJktCcFxoszsJTPbZWbOzNKCfcyDxPOamU0NZwwiItL4mNloM8sN0bGcmV0WimOJhINpnTuJRGZ2IjAXmOWcG1THfR8FLnPOHVvp/bZAnnNub4Bi7AqsA052zn1f4f1kIN45tysQx6nh+D8DpgBpwFog0zlXFMxj+o+bBnwJtHXO7azwfgre35TdwY5BRERCx8xeA37pf1kCZAFLgXeAcc654gbWnwg0d85tb0g9lep8DWjjnPtZpfcPA7Kcc4WBOpZIJFHLnUSqG4B/AMea2VGBqNA5tyNQid1BjpMb7MTOryew1Tk30zm3LRSJXU2cc9lK7EREmqzPgQ5AV+Ac4EPgMWCGmTWrb6VmFuucyw9kYlcT//VSiZ00WUruJOL47+BdBYzDuyt4fRVlDjezif4uiXvNbIGZDTWz0cAjwDH+rhfO/95+3TLN7E0ze7dSnVFmttHMfut/PdzMZphZlpllmtknlRLNdf7Huf7jpPv3269bpr/eP/rrLjSzxWZ2UYXtXf37jzSzz/yfZ5mZDavhHL0G/BXo7N/3J//76WY2tnLZit0l/WX+YWZPmtlOM9tuZn8xs6gKZeL829f7Y15rZnf6Wyu/9Bfb4T/2a9UcJ97MnjOzDDMrMLNZZja4wvY0//5nmdls/+f+3t9qW1Ymxcze8MdY4I/jrurOi4iIBE2hPzHa7Jxb4Jx7Fq/nyInAfVB+7XjazDb5/6bPNbNzyyqo8Hf/fDObY2ZFwLkVu2WaWW9/meMqHtzMbvJfs2LNLNrMXjGzdWaWb2arzOy+suuYvwfPL4ELKnwXSPNvK++WaWYzzez/Kh2nhb/OS2v5mWLN7G9mtsV/vdxoZk8F9MyL1IGSO4lElwHrnXOLgTeAUWYWW7bRvDuEX+HdPbwYOA543L/5beD/gBV4dxg7+N+rbALeH/2UCu+d4S//lv91M+A5YADeBSwb+NDM4vzbB/gfh/v3u7Saz/Mb4F7gfn+s7wFTzKxfpXL/D/gbcDxel9T/mNfFs7o6Hwc2+Y99cjXlqnM1Xtea04DbgbuAKypsHw+MAn4LHIWXYO8GNgIj/WWO8R/7N9UcY4y/zl8BJwCLgelm1qFSuT8DD+B9QdgFTDQz8297Au+c/Qw40l/X5jp+VhERCQLn3BJgOvuuC//Gu5ZeBRyLdy350MyOr7Tr08BDQB9gdqU6V+JdA6+utM/VwCR/F9AovGvBz/GuUX8Afg9c5y/7F2AS+1obOwAzq/gIE4BfVLy56f8sBcC0Wn6mO4FLgF8AvfCueyuqOJZISMSEOwCRKlyPl9SBl8TtBS7Ca8UD7w/sYcCpFcZ8rSnb2X/3r8Q5t62GY3yKl6xdBrzif+9q4Avn3FYA51zllr3rgBy8pO4bYId/066DHOse4C/OuTf9rx82s9P9719TodxfnXMf+o/1e7zkqp//WPtxzmWb2R6g9CDHrs4y59zD/ucrzexG4CzgLTPrhXeROs85N91fZm3ZjmaW6X+6veKYu4r8CfitwA3OuWn+924BzgR+jXdRL/NH59yX/jKP+z9vR7zEtQvwg3Nujr/s+np8VhERCZ5lwNlm1gO4EujqnNvg3zbWzM4GbgZuq7DPo865T8te7LufV24C8Dsze9A558ysMzAEeBDAn+A9XKH8T/5eH1cCrzjncs0sH39rYw2xv413E3co8D//e1cDk51zhbX8TF2AlcAM501ksYGqE0mRkFDLnUQUM+sJDAbeBPD/oZzI/l0zTwAWVZdY1IZzrgTvj/rV/uPG492tm1Ahlh7mdd9cY2Y5QAbe/zOd6/B5WgCHA99W2vQNcHSl9xZVeL7F/9iutseqo0WVXm+pcKwTAB/7ul/WRw8glgqf2zlXCnxH3T73i8AVZrbQ33X0jAbEJCIigWeAw+t9YcAyM8st+wEuwLsmVPQ9NfsP3rVziP/1lcA651x50mRmt/i78u/wH+du6nB9BvCPj5/Ovu8Ch+MlemXfBWrzmV7DuxG70sxeMLMLKrUEioSUWu4k0twARAMbKtzJMwAzO8I5tzGAx5oAfGdmHYFTgDi82SfLTMVrPboZr/tHCd4dyjgCo/JUteWzjfnvVELdb8D48J+vCmKrKFd5ZjNXj2PVV7Wfu8K2KADn3Mdm1gU4D69lcZqZTXbOXYeIiESCo/F6d0Th/Q0/mQOvMfmVXufVVKFzbruZfYaXdH3tf5xYtt3MrsBrcbsHr5UsB69XyCX1iH8C8C8zuw2v18pGYIZ/20E/k3PuB/PGo5+Ld50aDyw0s2HOOV894hFpEN1ZkIhhZjF4A6AfxLsLVvZzPF7rTtkX+vlAXzNrU01VRXgJYo38Xf1W490RvBp43zlXNqC7Nd5YgCedc58755YDzdn/hkjZ7JTVHss5l4PXGlV5OYfBeIlioO3AG1tQUeWxDgezAO9vw9Bqth/0c+N1ky2iwuc2s2jgVOr4uZ1zO51zbzjnRuO14P7S39IqIiJhZGbH4o07fwfv2mzAYc651ZV+6jNWegJwuZmdhDf2ekKFbYOB2c65sc65H5xzqzmwdbBW3wWAD/yPP8P7LvCm27dOWK0+k3Nuj3PuHefcrXitemfizWgtEnJquZNIcgHQBvhX5aUEzOw/wC1m9ie8LpsPAO+b2QN4rWrHAnv8Y7d+Arr4+99v8L9f3bTHE/FaC7uy/4QoWcBO4EYz24g3BuwZvNa7Mtvx7tyda95slQXOuewqjvEM8LiZrQLm4Y2zG4LX3SPQvgCeM7MReAO6bwaOwDsnteKcW2lmk4CXzew3wA9AJ7wxB2/gjXtzeBPSfAjklyXFFerIM7MXgafNbCfezKJ3A+3xlrioFf8YvB/w1lOKwfsdrdU01iIiIRdv3hpxUUBbvFaq3+Nd1/7i/7s/EXjNzH6H97c7Ff9arM65KVVXW63/Ai/hjYuf659opcxKYLSZnYd3k/YXeJOeZFUo8xNwnpkdiTdZV3ZV6/E55wrMmz37IbyboddW2LbyYJ/JvBm2t+LdGC3GmxcgB6/nj0jIqeVOIsn1wJfVrBE3GS8BG+acy8P7I74Jb52dJXhr7ZTdaXsX+AhvcPQOvJa56kzAm4UxG2+SFQD8XSmuAPr6638B+CNQWKFMCd4sWTfgtc69X80x/oaX4I3x13UJMNI5t7CGuOrr1Qo/3wJ78GbnrKtReEn034Af8cYUpAD471Y+gje7ZwYwtuoquB9vXOO/8S56fYHhZRPW1FKh/zgL8T5Pc+DCun0UEREJgLPxkpgNeNfXEcCjwOn+6zJ4PWz+jXe9+xFveMPp1GMyLP+6tO/hJVwTKm1+CW82zDfxZtbsijdTdkX/Apbjje/bwYE9aCqa4D/OfOdc5d4lB/tMe/BmxJ6Dl/z1w5uQLOjr6opUxfa1PIuIiIiIiEhjpZY7ERERERGRJkDJnYiISJCZ2e3+adsLzey1Gsr90szmmVmOmW0yszH+yaZEREQOSsmdiIhI8G0BnsAbD1uTJOAuvMmlTsGbtOKe4IYmIiJNhe4GioiIBFnZTIFm1h9v9tnqyr1Y4eVm/0x91S1LIiIish+13ImIiESu0/GWAhERETmoRtVy16ZNG9e1a9dwh3GAvLw8mjVrFu4wGg2dr7rR+aobna+6ieTzNW/evJ3OubbhjiNczOxXQH+85VaqK3MTcBNAYmLiSUcccUSIoqs9n89HVJTuJdeWzlfd6HzVjc5X3UTq+Vq5cmW118dGldx17dqV77//PtxhHCA9PZ20tLRwh9Fo6HzVTWM7X2sfXEvh5kKOuO8Iko9NDvnxG9v5CrdIPl9mVue1sZoKM7sY+DNwtnNuZ3XlnHPjgHEA/fv3d7pGNn46X3Wj81U3Ol91E6nnq6brY+SloiLSaDnnyJiYQcYbGbgiraEpUh9mNhxvAeYLnXOLwx2PiIg0Ho2q5U5EIl/f6X3J+jyL5H6hb7UTiVT+5QxigGgg2swSgBLnXEmlcmcCE4FLnHNzQh+piIg0Zmq5E5GAMTOaHd2MTnd2wqIs3OGIRJKHgHzgAeAa//OHzKyzmeWaWWd/uT8CKcBH/vdzzezj8IQsIiKNjVruREREgsw59yjwaDWbkyuU07IHIiJSb2q5E5GAKN1byg+DfmDdI+twTuPtREREREJNyZ2IBMTur3aTMzOHzI8yMVOXTBEREZFQU3InIgGROT0TgNThqWGOREREROTQpORORAIi8xMvuWt1bqswRyIiIiJyaFJyJyINlv9TPvkr8olOiabFwBbhDkdERETkkKTkTkQaLOuTLABandWKqBj9WREREREJB30LE5EG03g7ERERkfBTciciDeIr9pH1P6/lLvVcJXciIiIi4aLkTkQaJOe7HEr3lJJ0VBIJnRPCHY6IiIjIIUvJnYg0iLpkioiIiEQGJXci0iBK7kREREQig5I7Eam3oowicufnEpUQRcqQlHCHIyIiInJIiwl3ACLSeJXmltL2522xaCM6MTrc4YiIiIgc0pTciUi9JfZI5Ji3jwl3GCIiIiKCumWKiIiIiIg0CUruRKRe8tfms3PqTkpyS8IdioiIiIig5E5E6injzQyWXLiENfesCXcoIiIiIkKAkzszu93MvjezQjN77SBl7zazbWaWY2avmll8IGMRkeCK7xRP81Oa0/r81uEORUREREQIfMvdFuAJ4NWaCpnZucADwFlAF6A78FiAYxGRIOowugMnzTqJNiPahDsUERERESHAs2U656YAmFl/oFMNRX8JvOKcW+ov/ydgIl7CJyIiDeCc9+Pz7Xus+FPxvezsWLZvP7Ccc/vqqvhY1XvB2CYiIiJ1F66lEI4B3q/weiHQ3sxaO+d2hSkmkYgUv2MHXH45lJaGO5RymRmdiUvIo1mLXZiFO5r9HbNzJ7QJXWtiQWksu4ubkV2cRHZxEntKEtlbGs/e0njyS+PIL43b7/l+r31xFPliKPbFUOyiKfFFU+yiq3gdTUkV75e6KHzO8OF/dIbDcHXqlDEoaOdGREREQitcyV0ykF3hddnz5sB+yZ2Z3QTcBNC+fXvS09NDEV+d5ObmRmRckUrnq27iV68mb+5c1v3qV+EOBfBaVrL+OgBfdgItb5lHTIe8cIe0n4KCAjISEuq9f05+PNv3tGBXbjN25TVjV24ymXnNyl9n5jUjJz+B3MIE9hTEU1wamcuFmvnTPHOYeY9R5sAof+6V8RFl7F8Ghyt7rPA83vZ91jwK8OFw/v86K3sGzYilmcVjOAooZhe5/prAmc9fg9dE19laEe1PRjPIZi9FYI6WJLF7RwhPmIiISBMQrm8luUCLCq/Lnu+pXNA5Nw4YB9C/f3+XlpYW9ODqKj09nUiMK1LpfNXNgh9+oFn37hz78MPhDgWAvGV5zH10LrHtYjn+hd9iUZHVdJeenk7/Gv59OQcbN8LSpbBiBfz00/4/2dnV7lql2FhISYGWLb3H5s2hWTNITPR+kpJqfh4X59URGwsxMfueV35d1bboaO8nKgrMvMey52YGGEWlRczcOJOde3eSlZ9FVkEWuwt2k5Wfxe7C3Zweezq3jrgVgGe+fYY/fvlHCksLD/icbZLasPnefdlWl+e6sCF7Q5Xn5O5BD/Dns/8MwKdrPuXcCefut90wEmMTSYhJYMbN8+mc0hmAez8dwzcbvyEuOo6rjr2KW06u2+9CRETkUBeu5G4pcDwwyf/6eCBDXTJFquAckdT3MfOTTABSz0mNuMSusr174YcfYO5cWLLES+iWLYM9B9xG2qdZM+jUCdq33/fTrt3+z1NTvUQuJQUSEsLz65m7eS4LMxayOWczGXkZbM/bzo69O9iRt4Nj2x3LpMu9P695RXkMHT+02nq69+le/jw2OpbC0kJio2JJSUghOS65/Kd14v6zov765F+zt3jvfmWaxTYjMTaRHq16lJcb3Hkw6+9aT0JMQvlPbFSsP/nc3zPnPLPf61u4pV7nRkRE5FAV0OTOzGL8dUYD0WaWAJQ45yqvcvw68JqZTcSbYfMh4LVAxiLSVJhzXnNMhMic7k/uhqeGOZID7dgB6eltmTQJvvsOFi+ueqhi27ZwzDFw1FHQvTt07brvp3Xr8OXSRaVFrNq1inW717Euax0/7f6JTXs2sWXPFjbnbOajqz+iT5s+ALww9wXGLxxfZT3xMftWlmmZ0JK0rmmkJqaSmpBKy4SW5T+tElsRtXnfv62bT7qZm0+6mYSYhCqTr4ruG3RfrT5TUmxSecuciIiIBFegW+4eAh6p8Poa4DEzexVYBhztnNvgnJtuZmOAL4FE4N1K+4lIGZ8vYpK70r2l7P5qNxi0OqdVuMOhtBRmzYL//hc++wwWLgRvviZPVBQcfzwMGOA9HnOM99O2bdhCZk/hHlbsWsGPO39kxc4VdGvVjV+d4I2nXL5jOf1e6lftvhuzN5Ynd2ld0zAzOjbvSIfkDrRt1pa2SW1p26wt7Zu1L9/HzPjyl19WW2f6rvTy54mxiQ38dCIiIhJOgV4K4VHg0Wo2J1cq+yzwbCCPL9IURVLL3e6vd+MKHcknJRPXNi4sMfh88MUXMGkSvP8+bN++b1tCAhx9dBYjR7ZiyBA48USvm2W4PTfrOaaunMqPO39k857N+207t8e55cld15Zd6ZXai26tutGtZTe6tuxKpxad6Ni8Ix1bdKRLSpfy/Ub3G83ofqND+TFEREQkwkXmNG8isk8EtdyFs0vmunXw2mvez4YK83h07QqXXAI/+xmcdhrMmrUwpBP2+JyPn3b/xIJtC8p/FmYs5ItRX9Aj1Rt7tmT7Ev637n8AxEfH06t1L/q06cORrY/k5MP3zRqSkpDCyjtWhix2ERERaVqU3IlEOIOImVAl1Mmdc/DVVzBmDHz88b73u3aFa6+FkSOhb9/wnJ6N2Ru57v3rmLtlLjmFOQdsX7BtQXlyd0v/W7j0qEvp06YPXVK6EB0VHepwRURE5BCg5E4k0kVIy13+T/nkr8gnukU0LU5pcfAdGqC01BtH9/TT3kyX4HW5vOwyuO46SEsLzSlxzrF853K+3fAt3236joSYBP5xwT8Ab2mAr9Z/RYmvhMOSD6PfYf3o176f93hYP3qm9iyvp//h/YMfrIiIiBzylNyJRLhIGXOX9UkWAK3ObkVUbPDi+fxzuOeesslRoE0buOMOuO0273mwbczeyPTV0/nipy/4ct2XZORllG9rldCKseePJcqiSIxNZPrV0zmq7VEc3vzw4AcmIiIichBK7kQiXYS03JV3yTw3OF0yly6F++6Djz7yXh9xBNx/v9dSl5QUlEMC3uyVxb5iUhO9z/Xu8ne5+5O7y7d3SO7A6V1O59ROp3LqEafut+9Z3c8KXmAiIiIidaTkTiTCRULLnfM5sr/LBgKf3BUUwGOPeePqfD5o3hx+/3v4zW8gMUgz86/OXM3UlVOZtmoaX/30FQ8MfoDHhz4OwLDuwxh51EjO7HYmZ3Y7kyNbH3nQNd9EREREIoGSO5FI51zYJ1SxKGPguoHkzM4hoUtCwOqdM8drmVu2zMtfb7sNHnkE2rUL2CHK/bjzR95e8jaTl01m6Y6l5e8bxuacfcsTHNPuGN75+TuBD0BEREQkyJTciUS6COmWGZ0YTau0wCxcXlzsJXFPP+19vCOP9JY4GDgwINWXc86Vt7o9OeNJ3lj0BgAtE1oyvOdwLuh1AcN7DqdNUggG84mIiIgEmZI7kQgXCd0yfSU+omICE8P27XD55fD1197Huvder1tmoLpg7i7azfOznmf8wvE8fMbDXNznYgCu7XstMVExXH705ZzV/SziosOzCLuIiIhIsCi5E4l0YW65K8ooYnbv2bS+oDVHTTyqQePP5s6FSy+FTZugQweYPBkGDQpAjKVFTF05lfELxzNt5TRKXSkAby99uzy5G9ZjGMN6DGv4wUREREQiVPj7eolIjcLdcpf9bTalOaWUZJU0KLEbPx6GDPESu0GDYN68wCR2z816jo7PdmTkpJF8sOIDnHNc0OsCJl02iX9f9O+GH0AkAMzsdjP73swKzey1g5S928y2mVmOmb1qZvEhClNERBo5tdyJRLowT6jS9tK2DFw/kJKcknrX8cwz3jIHALfeCs89B3H17BXpcz6KSotIiPEmdikoKWDn3p0c2+5YRh8/mm653bj0nEvrHatIkGwBngDOBarthGxm5wIPAGf693kPeMz/noiISI2U3IlEuHC33AEkdK7fDJnOeROn/OlP3uuxY+HXv65fDHlFebw6/1X+PufvjDp+FA+d/hAAN554I0M6D+G0I07DzEhPT6/fAUSCyDk3BcDM+gOdaij6S+AV59xSf/k/ARNRcidSbs8eeP55WLsWtm07ki1b4CzLYPMLmw++cwVdH+lK6jBveZ+M/2Sweexm2l3Rjk53eP+L7pm/h1V3rKpTnVXtn9wvmd5jewNQureUhecsrFOdVe0fnRjN8Z8dX15m2ZXLKNhYcPDKsuGHlB+q3f/oN48uv+ave3QdWZ9n1SnWqvav6jzXRVh/T/7zVRsB/T351Wd/JXcikS6MY+5cqcOi69dq6Bz87nfw17964f/73zBqVN3r2bl3J2PnjGXsnLHsyt8FwEerPipP7lontWZQ5wD07xSJDMcA71d4vRBob2atnXO7Khc2s5uAmwDat28fkTc3cnNzIzKuSKXzVbNvvmnN88/3ZufOst7KHfj4Y7izQzaXbM2pU12LvloEsf4XXwPfQk6HHFYft9p7b4H3Xl1UtX/Onhy2pG/x3suvR51V7Z/A/v9OvgU21rI+cqrdf1b6LOjsf29G3WOtav+qznNdhPv3lEPt/l0F+vcE9dtfyZ1IhAtny936P68nY0IG3R7vRruf137xOee8Nev++U+IjYW33oKRI+t27A3ZG3jm22d4Zf4r5JfkAzCw00DuPe1eRhw5om6ViTQeyUB2hddlz5sDByR3zrlxwDiA/v37u7S0tGDHV2fp6elEYlyRSuereh99BA8/7F1jBgyAm64sZuvHP/L57E68ubULS9u141//glaptasvqXcSce28MQKFvQrJvyqf+MPjSezu9ZwuOaGE3P65dYqxqv1jWsSQ3DcZ8G6aZs/IrqmKA1S1v0UZKaellJfZ894eSvNLD1rXgvkL6HdCv2r3b35ic6KTogHY22EvRTuK6hRrVftXdZ7rIpy/p7LzVRuB/D2VqXb/ITXEUevaRSQ8fL6wjbnL+iSL/BX5WGzdjv/gg15il5AA770Hw4fX/diLMxYzdu5YAC7odQH3DbqPIZ2HNGhSF5FGIBdoUeF12fM9YYhFJGIsXw5XXukldg88AE88Ablz9zL/7l0MO7GIu+NP4n/fxXPN0/DFFxBfx2mI4jvGE99x/51iUmJoObhlvWOuan+LtgbVWd3+zU9qXrsKSqj1/klHJpF0ZFKdY6xp/6rOc12E/PdUzfk6mAb/nqpRm/01W6ZIpAtTy11JdgnZ32VjMUarM2u/ePn//Z+3OHlMDLzzTu0Tu43ZG5mwaEL56/N6ncf9g+5n0S2LmHrVVE7vcroSOzkULAWOr/D6eCCjqi6ZIoeKoiKv90dOjrdO6pNPQnQ0xLaKhSHQ5uyWTJkCnTrBzJnw+9+HO2KR8FFyJxLhwtUtM+t/WVAKLU5tQUxK7Rr5330X7rnHez5+PFxwwcH32bpnK3d+fCc9/96T0f8dzepMrw98lEXx1NlPcVz74+r7EUQihpnFmFkCEA1Em1mCmVX1P9brwPVmdrSZtQQeAl4LYagiEeef//Ra7nr39sZvl93nSzoyCR6HHk/34LDDYMoUb9vf/gYrV4Y3ZpFwUXInEunCNKFK5vRMAFKH127wwty5cM013vOnn4arrqq5fF5RHo98+Qg9/taDv8/5O0WlRVx29GXERKm3uDRJD+ENsX8AuMb//CEz62xmuWbWGcA5Nx0YA3wJbADWA4+EJ2SR8MvKgsce854/8ww0a1Z92ZNPhuuug5ISuP/+0MQnEmn0LUokwoWj5c45V6fkLiMDLr4YCgrg+uvh3ntrLj9+wXge/N+DbM3dCsBFR17E40Mfp2/7vg2OXSQSOeceBR6tZnNypbLPAs8GOSSRRuHJJyEzE9LS4MIL999Wml8K26BoRxFxbb0JN/70J/jPf+C//4Wvv4bTTw99zCLhpJY7kUgXhglV9v64l8KNhcS2jSW5X3KNZUtK4IorYMsWGDwY/vGPg4f74coP2Zq7lf6H92fGdTP47y/+q8RORET2k5XlXVPAG89d+dqS810OXAnLrlhW/t7hh++7wfjUUyEKVCSCKLkTiXAGIW+5K2+1OzcVi6o5U/vDH+Crr+Cww2DSJIiLO7DMhuwN/Ljzx/LXY4aN4Y1L3mD2DbMZ3HlwQGMXEZGm4dVXYe9eGDYMTjzxwO3O57wnlS6Rd9zhzdb88ccaeyeHHiV3IpEuDGPuatsl89NPYcwYb9ayyZOhQ4f9txeXFvOXmX/h6BeOZtR7oyj1eWu7dG/VnWv6XkOU6U+QiIgcqLQUxnqr4XDnndUU8nkPlW9Ctm69bwx4WR0ihwp9sxKJcKEec1eaX0r2194Cq62GVb8EwvbtMGqU9/yxx7wumRXN3DiTk8adxL2f3UtecR5dW3YlrzgvWGGLiEgTMnUq/PQT9OgB559fdZnqWu7Aa70Db3bNnJyghCgSkZTciUS6ELfc7f5qN74CH8knJRPXrqvF//4AACAASURBVIo+lnhL7910kzeRSlqat6Bsmcz8TG768CYGvTqIxdsX061lNz666iMmXT6JFvEtqqxPRESkohde8B5vv72GS2A1LXcAfft616fcXHjjjaCEKBKRlNyJRDhzLqQTqmR9kgV44+2qM3EivP8+tGgBr7/udcsErxvmgH8N4F8//IvYqFgeGvIQS29bynm9zgtF6CIi0gRs3Qr/+x/Exu7rIVKVmlruwLsJCd41S+RQoaUQRCJdiLtlpp6XSklOCW0ualPl9i1b9nV3ee45OOKIfdtio2P5zSm/YfKyyYy7cBx92vQJQcQiItKUvP2212nlwgshtaah394w7mon/hoxwlsX77vvYO1a6N498LGKRBq13IlEulAnd+ek0ueVPrQYUHUXyjvvhN27vTEQo0fD+z++z4RFE8q3/3rAr0kfna7ETkRE6uXNN73Hq6+uudzBWu6aNfPWYAV4663AxCYS6ZTciUQ4C8NsmdWZOhXefde7YD713G6ufe8aLn77Ym6ddiubcjYBEGVRmgVTRETqZdUqmDsXmjeHn/3sIIXLxtxFVz90oSxBnDjRu1cq0tSpW6ZIpAthy92mv28itm0srX/Wmpjk/f885OV5A9sBRt+9mvOnDmVTziYSYxJ5YugTHN788JDEKCIiTVdZq92ll0JiYs1ly1rualqP9eyzoU0bWL4cFiyAE04IVKQikUm310UinPl8IZlQxVfsY90f1rH8yuWU7Co5YPtTT8H69dC+xxZesKPYlLOJgZ0GsujWRfxm4G/UWiciIg327rve45VX1qKwv+Wupm+zsbFw+eXe8ylTGhKZSOOgb2MikS5ELXeuyNHl4S4cdt1hJHRJ2G/bTz/BM894zzPSRhIV4+OPp/+Rr0d/Tc/UnkGPTUREmr5162DxYq9L5tChBy9fm5Y72Dfu7oMPGhqhSORTt0yRCBeqRcyjm0XT+Z7OVW675x4oLITzR2ax6vhdvDziS07vcnrQYxIRkUPHhx96j+edB3FVL7O6n/iO8TAEmp/cvMZyZ5zhJYyLFnk3K7t2bXCoIhFLLXcikS6ME6rsLd7L71+ZzrvvQlISvPRcK5b/erkSOxERCbiylrURI2pXvuXpLeFxOOK3R9RYLj4ehg/3npclkCJNlZI7kQgXipa7ou1FrHtkHTlzcsrfW7FzBaf8ayB/fjQFgHvvhU6dIDoqOqixiIjIoWf3bvjqK4iO9lruAq0sYVTXTGnqlNyJRLoQTKiS+Wkm6x9fz7qH1wEwaekk+v+rP0u+7gGbTiW1TTG/+11QQxARkUPY9OlQUgJDhhxk4fIKSvaUwDYoziw+aNnzz/cSx/R0yM5uWKwikUzJnUhjEOSWu8zpmQCkDEvhjo/u4Ip3riA3P5/mM8YC8PijsTSveUiDiIhIvU2b5j1eeGHt99n+9na4Etbct+agZVNTYdAgL4H89NN6BinSCCi5E4lwwV7E3PkcWZ9kAXDH3jsYO3cssVGxXGPT2bOlI927w403Bu3wIiJyiPP54LPPvOdlY+NqI7pZNLSH2NaxtSpfVnfZsUSaIiV3IpEuyGPucufnUryzmLjOcaxquYrOKZ1JH/UN3008G4BHHqndrGUiIiL1sXgxZGRAx45w1FG136/9le3hP9Dj6R61Kn/OOd7jp596l1aRpkhLIYhEuGC33O38eCcArYe3ZtrV02gR34Kpk9qyZg306gVXXRW0Q4uIiJR3kzznnOAOMT/hBGjdGtavh1WroHfv4B1LJFzUcicS6ZwLytWuqLSIW6bewty35gKQOjyVHqk9aBXfliee8Mo8/DDE6BaQiIgEUcXkLpiiomDYsP2PKdLUKLkTiXDBWAphW+42ho4fyoRvJ9D6x9YQA63ObAXA5Mmwdq13R/PKKwN6WBERkf3k58OMGd49zLPPrtu+W8ZtgYth7R/W1nqfil0zRZoi3ZMXiXQB7pa5YNsCLnzrQjblbOLibRcT7YsmZVAKMSkxOAfPPOOVu+ceb9poERGRYJkxAwoL4aSToE2buu1bmlcK2f7HWiprufvySyguhtjazcUi0mio5U4kwgWy5e6DFR8w+NXBbMrZxGlHnMYjPAJ4XTIBvvgC5s+H9u3h2msDckgREZFqff6591iWdNWJz3uw6NoPXejUyZu0JTcXZs+uxzFFIpySO5FIF6CWu/eWv8fF/7mYvOI8rul7DZ9f+zn5/8sH9iV3Za12d9wBCQkNPqSIiEiNvvzSezzzzLrv63zelJcWVbdx6WXHSk+v+zFFIp2SO5EIZwGaUOXs7mfTt31fnhj6BK9f/Dq+VT4KNxYS2y6W5H7JLFoEn3wCSUlwyy0BCFxERKQG2dnwww/exF2nnVaPCvwtd3X9NpuW5j0quZOmSGPuRCJdA7plZuZn0iy2GfEx8TSPb87sG2YTHxMPQP6afKJTokk9JxWLMv7v/7x9rr/emypaREQkmL75xuucMnAgNGtW9/3r23J3+une48yZ3ni/+Pi6H1skUqnlTiTS1TO5W7lrJQNfHsiNH96I86/WWpbYAbS5sA2Ddg6i53M92bQJ3nzTO8zddwcschERkWqVtZyVtaTVWT1b7tq1g2OO8WbqnDu3nscWiVBK7kQiXH0WMf96/dcMfHkgqzJXsXj7YrILs6ssFxUTRWzrWJ5/HkpK4PLLoVu3QEQtIiJSs7LkbujQ+u1f35a7isdU10xpapTciUS6OrbcTVo6iWFvDCOrIIsRR45gxnUzaJnQcr8yRTuLyqeOzsuDf/3Le/+eewIWtYiISLXKxtvFxsKpp9azknq23MG+1sKyCV1EmgoldyIRzny+Wk+o8tfv/soV71xBUWkRdwy4gyk/n0JyXPIB5Tb8vw18k/oNW1/ZyttvexfZU0+F/v0DHb2IiMiBZszwxtsNGFC/8XbQsJa7yuPuRJoKJXcika6WLXf/nv9vfvvpbwEYc/YYnh/+PNFRVa9CXpRRhCt2JPZO5J//9N7TDJkiIhIqM2Z4j2VJVn24Ui+5q8+32bZt4eijoaAA5s2rfwwikUbJnUiEq+0i5lccewVDOg9h4qUTuXfQvVgNrX1Hv3k0p20/jdVxLZg7F1q18sbbiYiIhMLMmd7joEENqKQei5hXVHbsslhEmgIldyKRroYJVbLysygoKQAgKTaJ9NHpXHXcVbWqNq5NHONe8er95S8hMTEw4YqIiNSksHDfLJX1Hm8HxKTEQDuIblF1L5WDKUvuvv22/jGIRBoldyIRrrqWu/W71zPo1UFcM+UaSn3e5ChRdvD/pQs2FuCcIzvbW/4A4OabAxqyiIhItX74wUvwjj4aUlPrX0/n+zvD29Dp9k712r9icudfMUik0dMi5iKRzrkDJlRZsG0B5008j22524iOiiarIIs2SW0OWpWv2MfcY+YSkxLDwrtOJi8vhrQ06NMnSLGLiIhUUtZS1qAumQHQo4e35t327bB6NfTqFd54RAJBLXcika5Sy92naz5lyL+HsC13G2ld05hx3YxaJXYAObNyKN1TSnRyNP8Y793b0UQqIsFnZqlm9p6Z5ZnZejOrsv+0mcWb2T/NLMPMMs3sQzPrGOp4RYKpLLk77bTwxmG2LwZ1zZSmQsmdSISruIj5+AXjueDNC8gtyuXKY69k+tXTD1jDriaZ0zMBKOibyuLF3mxhl1wSlLBFZH8vAEVAe+Bq4EUzO6aKcr8BTgX6AocDWcDfQxWkSLA5F6DJVIC1f1gLF8OWl7bUuw5NqiJNTUCTuzrcmXzUzIrNLLfCT/dAxiLSZPhb7v77438Z/f5oSnwl3HfafUy4dALxMfF1qqosuft4lzfI4frrIS4u4BGLSAVm1gwYCfzROZfrnPsG+AC4tori3YBPnHMZzrkC4G2gqiRQpFFas8brBtm2LfTs2bC6SveUQjb4Cn0HL1wNTaoiTU2gx9xVvDPZD5hmZgudc0urKPu2c+6aAB9fpMkpa7k7r+c5DO06lEuPupTbB9xe53qKMorI/SEXS4jinzNTAC+5E5Gg6w2UOOdWVnhvIXBGFWVfAZ43s8OB3XitfB8HP0SR0Kg43q6GFXtqpduT3dh85mY6nNOh3nWceCLEx8OyZZCZ2bAJXkQiQcCSuwp3Jo91zuUC35hZ2Z3JBwJ1HJFDSV5RHnlRJbQ2Iz4mns+u/azahckPJvMzr9Vub68Udi+O5rTTGn7XVERqJRnIqfReNtC8irKrgI3AZqAUWAxUezfHzG4CbgJo37496enpAQg3sHJzcyMyrkjV1M/X5Mm9gcNp334N6ekbG1xfbkwuM+bMaFAdvXqdwJIlKbz00iJOPTWzwTFFsqb+7yvQGuP5CmTLXV3uTAJcaGaZwFZgrHPuxQDGItLobc/bzoVvXUjKCav5yBwxUO/EDvZ1yZxR6N2WHDUqEFGKSC3kAi0qvdcC2FNF2ReAeKA1kAfch9dyd0pVFTvnxgHjAPr37+/S0tICE3EApaenE4lxRaqmfr5u99+qGDWqB6ed1qPB9QXifJ1/PixZArm5fWnCpx5o+v++Aq0xnq9AJnd1uTM5Ce9ilIF3wXrXzHY7596qXFB3JZsena+D25y/mfsW3ceWgi10bhbHF4tmEpfYrP4V+oCp3tOJK1OJjfVx+OEzSU8vCUi8kUT/vupG5yskVgIxZtbLObfK/97xQFVDFvoBf3DOZQKY2d+Bx82sjXNuZ2jCFQmOrCxYutTrBnnSSQ2vb+NzG+FN2PXYLlqf17re9WjcnTQlgUzuan1n0jm3rMLLmWb2PHAZcEByp7uSTY/OV81mb5rNXW/dxc6CnZzU4SQmvFZEn8eH0pDbiXvm7WFe9jwKW8azYXcSI0cYF144OHBBRxD9+6obna/gc87lmdkUvCTtBrwE7iKgqong5wKjzCwd2AvcBmxRYidNwXffeY/9+3sJXkPlLcmDuVC4ubBB9ZQthzBnDhQXQ2xsw2MTCZdAzpZZfmeywnvV3ZmszAENHFYr0vh9uOJDho4fys69Oxneczjpo9NpXxCz3zp39VHWJXOupQKmLpkioXcbkAhsx7uReatzbqmZDTGz3Arl7gEK8Mbe7QDOB7RgiTQJAV+83D9JpkU17CtkmzZw5JGQnw/z5wcgLpEwClhy55zLA8ruTDYzs0F4dybfqFzWzC4ys1bmGQDcCbwfqFhEGqMv133JxW9fTH5JPr/q9ys++MUHJMcle7NlNnBKsbLk7pOsVFq3huHDAxGxiNSWcy7TOXexc66Zc66zc+5N//sznHPJFcrtcs5d7Zxr55xr6Zwb7JybE77IRQIn0IuXO5/zngTg26wWM5emItCLmNf2zuQvgNV4XTZfB552zo0PcCwijcrgzoM5u/vZPHLGI7w84mVio/39Qvzr3NVXSXYJ2d9l44sy5tOKK6/U2nYiIhJaxcVet0cIXHIXqJY70GLm0nQEdJ07/wDwi6t4fwbehCtlr68M5HFFGqvCkkIKSwtpEd+C2OhYpl01jZioSv9bNjC5i0qM4pj3j+PeX+STlxvDtVUtmywiIhJECxd63R579/YWMA8EVxq4lruKk6o41/A1+ETCJdAtdyJSS7sLdjN84nAufftSikqLAA5M7Ni3iHl9RcVFMddaMz63E0ceCSefXO+qRERE6mXWLO9x4MAAVlrWchfd8Eysd29o2RK2boVNmxpcnUjYKLkTCYMN2RsY9Oog0n9KZ+mOpWzI3lB94Qa23AG8+ab3eO21uhspIiKhN3u293hKlSs21k/ZmLtAdMuMioIBA7znZbGKNEZK7kRCbMG2BQx8eSDLdizj6LZHM+v6WfRM7Vlt+YZMqLJ3xV4WXbyE3e9mAHDFFfWqRkREpEGCkdyVtdwF6ttsWWxK7qQxC+iYOxGp2SerP+GyyZeRW5RLWtc0pvx8Cq0SWx18x3q23O36aBeZ7++kL9FsOaE9PavPIUVERIIiMxNWrYKEBOjbN3D1BrLlDpTcSdOgljuREJm5cSYXvHkBuUW5XHXcVUy/enqtEruGjLlre3lbZpzcm4/owM9/Xq8qREREGqRslswTTwzwAuEBbrkr65Y5bx6UlASmTpFQU8udSIic0vEULuh9Ace0PYYnznyCKKvl1agBY+5c6wTGLD+cXODdy+tVhYiISIMEpUsmgW+5a9sWuneHtWthyRLo1y8g1YqElJI7kSAqKCkgryiP1kmtiY6K5t2fv1vljJg1aUjL3SefQG6ud7e0R496VSEiItIgwUruAt1yB16Ma9d6MSu5k8ZI3TJFgiQjN4Oh44cy4j8jKCgpAKpe6uCg6rngzoa/bGDjA6s5gr3qkikiImHh3L5umYFO7o6aeBS8B6nnpAaszrIYy2IWaWyU3IkEweKMxQx4eQCzNs1iU84mtuzZUv/K6tktc8vL2zhuxSZaUsTl6pIpIiJhsHYt7NoF7dpBly6BrTumRQy0hKj4wH2d1aQq0tgpuRMJsGkrp3Haq6exIXsDAzsNZM4Nc+jeqnu966tPt8yC9QUUrNhLHtEknNiC7vU/vIiISL2VJUkDBjSOdVb79fMmfVm2DHJywh2NSN0puRMJEOccz816jhH/GUFuUS6/OPYXfDHqC9ont29oxXVO7jI/yQRgHq247Ar9by4iIuERtPF2wJr718B9sGf+noDVmZAAxx/vXXq//z5g1YqEjL71iQTIhys/5O5P7sbnfDx6xqO8eembJMYmNrheq0dyt2Oal9zNJVVdMkVEJGyCmdztmbsH5kJJVmDXLVDXTGnMlNyJBMjPev+MUceP4q2Rb/FI2iNYoPqf+Hx16sviK/aR+XkWAAXHpdKtW2DCEBERqYvCQpg/33t+8smBr7/7093haUg+Pjmg9Sq5k8ZMSyGINMCCbQtondiaI1KOIMqiGH/x+IAfo64tdzmzcrC9pawniTOvTAh4PCIiIrWxcCEUFUGfPtCyZeDrb3FyC8iD2NaBXBl9/+SunhNWi4SNWu5E6umtxW9x2iuncemkS8kvzg/egeqY3O38yOuSOYdULrkkWEGJiIjULJhdMoOpVy9o1Qq2bYONG8MdjUjdKLkTqaNSXyn3fXYfV025ivySfI5rd1zgumBWoa6zZW5610vutnRMpU+fYEUlIiJSs2And5tf3AwvQ8GGgoDWa+bN7gnqmimNj5I7kTrYtXcX5795Ps/MfIaYqBjGnjeWV0a8QkJMELs/1qHlrmh7EazKpZAo+vw8JXgxiYiIHESwk7ttr22DiVC0tSjgdWvcnTRWGnMnUkuzN83m5+/8nA3ZG2ib1JbJl0/mjK5nBP24VocJVcqWQFhICiMujw5mWCIiItXatQtWr/aWFjjuuCAdxOd/DEJThZI7aayU3InU0syNM9mQvYEBHQcw+fLJdE7pHLqD17Llbu1/vORueXIq9zSyMQ6BtH37dnbv3k1paWm4Q4l4KSkpLF++POTHjY6OpmXLlrRp04aoOi71ISKRb+5c7/HEE71FwYPB+RwAFhX4oRFls3vOmwclJRCjb8zSSOifqkgNnHPl4+nuGngXLeJbcO3x1xIXHRe6IOow5m5NVhyJxNF6eGpdl8ZrMpKSkigsLKRr167ExsYGdTxkU7Bnzx6aN28e0mM65yguLiYjI4NNmzbRuXMIb5SISEiEZDKVILbctW0L3bvD2rWwZAn06xf4Y4gEwyH69U/k4BZlLOLUV05lXdY6AMyM60+8PrSJHXVbCuHZ/J5czqmcOTopyFFFrtjYWDp27EhcXJwSuwhlZsTFxdGxY0fy8vLCHY6IBEEokrtgttyBumZK46TkTqQS5xwvzn2RU14+hdmbZ/NI+iPhDaiWLXfr18OCBdCsmXHmWYd2UqNufo2Dfk8iTZNzMGeO97yxttyBkjtpnNQtU6SCHXk7uOHDG/hgxQcA/Krfr/j7+X8Pa0xWyxVU/zcmkxY055zzYknQ2uUiIhIma9Z4E6q0awddugTvOGq5EzmQkjsRv8/WfMao/45iW+42UuJTeOlnL3HFsVeEO6xaLYVQkl1C1xcXMRlj5/DBgGbKFBGR8KjYJTOoveOD3HLXr583Gczy5ZCTAy1aBOc4IoGkPjEiwLbcbVz41oVsy93G4M6DWXjLwshI7KjdIuY7VhaxwLVkMS0571IldiIiEj4hmUwFcKXBbblLSPASPOf2zf4pEumU3IkAhyUfxtNnP83jaY/z5S+/pEvLIPYjqatatNx9uSqJ39GPaWl9adUqRHFJwO3YsYPbbruNrl27Eh8fT/v27TnrrLP47LPPAOjatSt/+ctfwhyliEjNQpbc+btlBrOzirpmSmOjbplySCouLebP3/yZXqm9uPK4KwH4zcDfhDmqqtWm5W7aNO/xwhGH9kQqjd3IkSPZu3cvr7zyCj179mT79u189dVX7Nq1K9yhiYjUSmGhN7kX7FsrLmj83TKD1XIHXnI3dqySO2k8lNzJIWdRxiJG/3c087fNp3Viay488kKS45LDHVb1DjKhyt6Nhaz5sIAoWnDBBUruGqvdu3czY8YMPvvsM8466ywAunTpwsn+b0dpaWmsX7+ee++9l3vvvRfwZnYFmDlzJg8++CBz586lVatWjBgxgqeffpoW/gEiaWlp9OnTh/j4eF5//XUAbrjhBp5++ulQf0wRaeIWLICiIjjqKEhJCe6xylvugtgPrWLLXS3nNxMJK3XLlENGcWkxf/rqT/Qf15/52+bTrWU3Jl8+ObITOzhot8zvx2znqT3zebTFCnr3DmFcElDJyckkJyfzwQcfUFBQcMD2KVOm0KlTJx5++GG2bt3K1q1bAVi8eDHnnHMOI0aMYOHChUyZMoUFCxbwq1/9ar/9J06ciM/n47vvvuOll15i3LhxPPfccyH5bCJy6Chr4Ro4MPjHOur1o+BpiGsfvPVne/aE1FTIyIANG4J2GJGAUcudHBK+3fAtt0y7hSXblwDw65N/zVNnPxX5iR0HX8R8+9RM2gBJp7YMXVCNUThut/pb1mojJiaG1157jRtvvJFx48ZxwgknMGjQIC6//HJOOeUUUlNTiY6Opnnz5hx22GHl+z3zzDNcccUV/O53vyt/78UXX+SEE05g+/bttGvXDoAOHTrwt7/9DTOjT58+rFy5kmeffZYbb7wxcJ9XRA55s2Z5j8EebwfQ8vSW4IPoxOANujPzPsvHH3ufLZhLO4gEglrupMnzOR+3TruVJduX0L1Vd74Y9QVjzx/bKBI7oMZFzEvzS2mxPhuAfjekhjKqxse50P/U0ciRI9myZQsffvgh5513HjNnzmTgwIE8+eST1e4zb948JkyYUN7yl5yczKBBgwBYs2ZNebmBAwdiFRLcU089lc2bN5OTk1PnOEVEqhOqyVRCSZOqSGOiljtpkpxz5JfkkxSbRJRF8Y8L/sGnaz7lwcEPkhibGO7w6qSmlruVk3YT53ysjkpm1IjgdUuR0ElISGDYsGEMGzaMhx9+mBtuuIFHH32Ue+65p8ryPp+PG264gbvvvvuAbR07dgx2uCIi5XbsgLVrISkJjj02+Mdb/+R6WAYlJ5UQ0zx4X2nLupgquZPGQMmdNDkLti3gzo/vpFurboy/eDwAgzsPZnDnwWGOrJ5qGMG9bHwWrYHdvVKJU27XJB199NGUlJRQUFBAXFwcpaWl+20/8cQTWbp0KT179qyxntmzZ+OcK2+9mzVrFocffnj5pCsiIg1Vlvz07w8xIfiGuem5TbADfM/6oHnwjjNggPc4b543WYyutxLJ1C1Tmoyte7Zy/fvXc+JLJzJjwwymr57Orr2NfAp557yWu+rGi83JBKDjxeqS2djt2rWLM888kwkTJrBo0SLWrVvH5MmTGTNmDGeddRYtWrSga9euzJgxg82bN7Nz504A7r//fubMmcMtt9zC/PnzWb16NVOnTuXmm2/er/4tW7Zw1113sWLFCt555x2eeeaZKlv7RETqK5STqQB0/n1nuB6ik4O40B3QqhUceaS3zMOiRUE9lEiDqeVOGr29xXt59rtneeqbp8grziMmKobbT76dh894mFaJjXxFb+dwZvuNlSqT9WMBrfP2kkc0ab9W60tjl5yczMCBA3n++edZvXo1hYWFdOzYkauuuoqHHnoIgMcff5ybb76ZHj16UFhYiHOOvn378vXXX/PQQw9xxhlnUFpaSvfu3bnkkkv2q//qq6+mtLSUU045BTPj+uuv5+6772bv3r3h+Lgi0gSFcjIVgCPuOoI16WuITgpucgfeZ1qxwvuM/fsH/XAi9abkThq1vKI8jvnHMazPXg/ARUdexJhhY+jduomsCeDzVdtqN3tsJgnA2patuOAINcI3dvHx8Tz55JM1Tp4ycOBAFi5ceMD7/fv3Z/r06TXWHxMTw9ixYxk7dmyDY5W6M7NU4BXgHGAn8KBz7s1qyp4IPAecCOQBTzrnng9VrCL14fPBnDne86Y0mUqZU06B11/3Widvvz3c0YhUT8mdNDplCzebGc3imjGs+zC+3/o9z57zLEO7DQ1zdAHm8+GioqgqvdsxLZMjgPjB6pIp0gi8ABQB7YF+wDQzW+icW1qxkJm1AaYDdwPvAHFApxDHKlJnK1ZATg506gShmstp5/s7YRH4BvmIig3uTU5NqiKNhW73S6PhnGPqyqmcNO4kPljxQfn7fx3+V76/8fuml9hBtZOplBb5aLM+C4ATblZyJxLJzKwZMBL4o3Mu1zn3DfABcG0VxX8LfOKcm+icK3TO7XHOLQ9lvCL1EeoumQDLrl4GD4Ov0Bf0Yx13HCQkwKpVsKuRD+eXpk0tdxLxnHN8suYTHk1/lNmbvVtmL37/Ihf1uQig8axXVx8+H66K5G7RWzkkulI2Rydx5fkJYQhMGpP09PRwh3Co6w2UOOdWVnhvIXBGFWUHAovNbCbQE5gN/No5t6Gqis3sJuAmgPbt20fk7zo3Nzci44pUjfV8vfdeb+Bw2rZdQ3r6xtAcXX7NuAAAIABJREFUtMR7+OabbyAEl8JevfqxeHFLXn55Eaeckhn8AwZBY/33FS6N8XwpuZOIVeIrYdLSSYz5dgwLM7xxRu2atePBwQ9y80k3H2TvJqKaBcyXT8jicGB379TqlsATkciRDFReLT6bqidv74Q31m4YsBgYA7wFDKqqYufcOGAcQP/+/V1aWlpgIg6g9PR0IjGuSNVYz1fZ5LtXXdWDIUN6hOSYX7mvcDiGpA0hOiH4k6qccw4sXgz5+X1phL8ioPH++wqXxni+lNxJxHr5h5e5ddqtAHRI7sBdA+/itpNva9otdZVV03L375IubKMVD9+gxXZEGoFcoPKUti2APVWUzQfec87NBTCzx4CdZpbinMsObpgi9ZOX5y0REB0NJ50UwgP7e2NaVDXLBQVYWZfTsi6oIpFIyZ1EjA3ZG1i2YxnDew4H4Jq+1zB+4XiuP+F6ru17LfEx8WGOMAyqaLnLzYWvvo2iNKolZ/4yTHGJSF2sBGLMrJdzbpX/veOBpVWUXQS4Cq9dFWVEIsq8ed7l6oQTICkpdMd1Pv8Ea9GhSe7KJlWZM6fajjUiYafkTsLK53x8vvZzXpj7AlNXTqVlQks23b2JxNhEkuOS+e7678IdYni5A7/XffklFBd7F5nWrcMQk4jUiXMuz8ymAI+b2Q14s2VeBJxWRfF/A/+fvfsOj6L6Gjj+nfTeCITQQiihd5AOAZWAIB1FbCiCiIqKgggqWPCnKCqi8oqiAiKKHRBBBQIB6b3X0AkphJJKkp33j5tsEpJAEjY7m+R8nmef2Z29M3N2yLJz5rZfNE37BJX8vQasl1o7YcuMGEwFMNfcWWt4wGrVIDAQLlxQA6vUq2ed4wpRFJLcCUPEJMawYM8C/m/b/3H0krqR7WjnSFjtMK6kXsHV0dXgCG1E5lQIOcW8fJjppGDXtBb5d9kRQtigMcDXQDQQBzyl6/p+TdM6A3/puu4BoOv6ak3TJgF/Am7AemCYQTELUShZ0wNk1WxZg57j5qdWwHywlqZp6jP+9pv6zJLcCVskyZ2wuvPJ57n7w7tJN6lhrqp7VefJVk/yRMsnCPAIMDg6G3PDJOYmk06FI7HUIg3nbtIeRIjSQtf1S0D/fNZHoAZcybluNjDbSqEJcduykjur1txZudYuS9u22cndI49Y99hCFIZcHYoSpes6W89t5aONH5nXBboE0rhSY3rX7c1v9//GiedOMLnLZEns8nNDzd2xYxrDM9oww6MhbQZbsWODsAmhoaE888wzRocBFC6Wxo0bM3XqVOsEJIQwxNmzcO4ceHtDSIj1jpvV3w7rVNqZyaAqwtZJzZ2wOF3X2Xp+Kz/t/4mfD/7MycsnAehbry+1/WqjaRpbntiCo72jsYGWBjfU3K1YAZdxwvPeSjjIt7fMiYmJYcqUKSxfvpwLFy7g4+ND48aNmThxInfffTe//vorjo628b2xpViEEMbJWWtn1QFGDKq5a91afc49eyA5GVylF4mwMXJ5KCzmUvIl3ol4h58P/MypK6fM6wM9AhnScEiuZE4Su0LS9VxTIaxYoZY9exoUjyhRgwYNIikpiblz51KnTh2io6NZu3YtcXFxAPj5+RkcYTZbikUIYRxDmmRiXM2dhwc0bqySux07oGO+M1AKYRxplimKLS4pjohTEebXLg4ufLb1M05dOUUVzyqMvWMs64av4+y4s8zsNZMa3jUMjLaUyjHW8rWodB5esYnnOcLdd8vo6GXN5cuXiYiI4N133+XOO+8kKCiINm3a8NJLLzF06FAgb1PIixcv0rdvX1xdXQkKCuKbb77J0xRS0zRmz55Nv379cHNzIyQkhDVr1nD27FnCwsKoXLkyzZs3Z8eOHbni+fXXX2nSpAnOzs5Ur16dadOm5RrA4MZYoqOj6devnzmWr7/+uoTOlBDClmQ1T7TmYCoAekbm/0cGXMlmfVZpmilskdTciUJLTU9l87nNrI5czT8n/mHT2U14OHkQOz4WR3tH3BzdmNlzJo0qNqJ99fbYaXLv4LblmMR802fxBOopNHZPJDDQyrcqywDtjYLP2Rd9vmBUq1EAzNk+hyeXPVlgWX1KdoLTak4rdlzYcctyheHh4YGHhwdLliyhU6dOuLi43HKbRx99lAsXLrB69WpcXV158cUXOXXqVJ5yb7/9Nh988AEffPABU6ZMYejQoTRv3pwxY8bwv//9j1dffZXhw4ezZ88eALZv386QIUN49dVXefDBB9m6dStPPvkkXl5ePPvss/nGMnz4cE6dOsW///6Lm5sbL7zwAidPnizSORBClC7p6WqOO4A77rDuse0c7QieFkzkqUjrHhhVSzlnTnatpRC2RJI7cUs7LuzglVWvEHEqguT0ZPN6RztH2lRpQ0xSDFU8qwCYL5CFheSouTv3+yVqAnpraQ5XFjk4OPDtt98ycuRI5syZQ4sWLejYsSNDhgyhbT7tnQ4fPszKlSvZuHEj7TJvI3/77bfUrFkzT9lHHnmEBx54AIBJkyaxaNEiwsLC6NevH9euXWPChAl069aN2NhY/P39+fDDD+natStvvPEGACEhIRw9epT33nsv3+TuyJEj/PXXX6xfv56OmW2U5s2bR61atSx1eoQQNmjfPkhKgjp1wN/fuse2c7YjaFIQkeHWT+6yau4kuRO2SJI7YRaVEMXGMxvZeHYjQd5BPH3H04Bqbvn38b8BaFSxEd2Du5sfXs5eRoZc9mXW3Om6juehSwCEPCTJXXEUtiZtVKtRhb5JsX3U9tsJKY9BgwbRu3dvIiIi2LhxIytWrGDGjBlMmzaNSZMm5Sp76NAh7OzsaN26tXld9erVqVKlSp79Nm3a1Pw8IECNStukSZM866Kjo/H39+fgwYP07t071z46derEG2+8wdWrV/Hyyv29P3jwIHZ2dtyR49Z9UFBQvrEIIcoOwyYvN1j9+uDlBadPqwnNAwONjkiIbJLclWM7L+xk7am1bDm3hY1nN5pHtQRoV62dOblr4N+AxYMX0yWoi0xXYG26DprG0dVJVEhP5bLmyD0Pe9x6O1Fqubi4cPfdd3P33Xfz+uuv88QTTzB16lReeumlYu8z56iWWZP95rfOZDJxKzebLNhaEwkLIWyDUYOpAJhSTcQti4MjQKh1j21nB23awKpVKsEdMMC6xxfiZiS5K+NMuomTl0+yO2o3u6J2Mbz5cIJ9gwH4Ztc3zNoyy1zWw8mDtlXb0q5aOzrX6Gxer2kaQxoNsXrsAvNUCLu+uEQlIKqaL07OcgFdnjRs2JD09HRSUlJyra9fvz4mk4nt27ebm22ePXuW8+fP3/YxGzRowIYNG3KtW79+PdWqVcPT0zNP+axYtmzZQocOHQA4ffq0RWIRQtiurP8m2re3/rHTr6Szf/B+8AFesf7x27dXyd1//0lyJ2yLJHdlTHJaMu+uf5dDcYc4HHuYw3GHSUnPviisW6GuObnrUbsHqemptAxsSfvq7WlUsRH2dvZGhS7ykzmJedJa1STTp4c0ySyr4uLiGDJkCI8//jhNmzbF09OTbdu2MX36dO688848TSHr1atHWFgYo0ePZvbs2bi4uDB+/Hjc3NxuuwbtxRdfpE2bNkydOpVhw4axdetWZsyYwTvvvJNv+Xr16tGzZ0+efPJJ5syZg6urK+PGjcNVJoASosyKjoajR8HNDZo1s/7xNScN/4H+xCbFWv/gZE+BcMN9MCEMJ8ldKWHSTZy/dp7I+EgiL0dyIv4EkZcjiYyPxNfVlz+G/gGAk70T0/+bniuhC/QIpFnlZjQPaE6jio3M6/uE9KFPSB+rfxZRBCYTKbo7gdFXAGj7tCR3ZZWHhwft2rVj5syZHDt2jNTUVKpWrcqwYcN49dVX890mawCW0NBQKlWqxJtvvsmJEycKNdLmzbRs2ZKffvqJKVOm8M477xAQEMDEiRNzTX1QUCzdu3fH39+fKVOmEB0dfVtxCCFs13//qWXbtuBowNS1jj6ONP6lMeHh4dY/OKrmTtPUaKEpKXCb/+0KYTGS3NmA1PRUzl87z7lr5zh39Zx5Obz5cJoEqEEPJq+azLsb3s13ey9nL3RdR9M07O3seffOd/Fz9aOefz3qVaiHt4u3NT+OsCSTia2JfXDGxGkXD0JbOBkdkSghzs7OvPPOOwXWjgF5LmIqV67M0qVLza9jY2MZNWoUderUMa/LOTcdgL+/f5519evXz7Nu4MCBDBw4sNCxBAQEsGTJklzrnnjiiQK3F0KUblk1VuV1Em9vbzWZ+d69sG0bdOpkdERCKJLclYDU9FRikmKITYolJjGGmKQYYhLV6wpuFXi+3fMAJF5PpObMmgU2KWga0NSc3AX7BlPJvRLBPsEE+wYT7BNMLd9a5tc5PdfuuZL9gMJ6dJ1zSe2pBaQ2lVo7kdvq1au5du0aTZo0ITo6msmTJ+Pv70/Pnj2NDk0IUcYZndyZ0kyknksFY1plAuqz792rzoUkd8JWWDS50zTND5gL9EB93V7Rdf37fMppwLtA1m3dr4CJ+o23jq1I13WuZ1wnMS2RhOsJJF5PxM3RjSCfIADikuL4cf+PXEm5wuWUy1xJvcKVVPX8TPQZfmz4I40qqSaPTy9/mrk75+Z7nKYBTc3JnZujG1dSrmCv2RPoGUgVzypU9ayqHl5VaRnY0rzdyJYjZQ65cigtPp2A6/6ATq1hFYwOR9iYtLQ0Xn31VU6cOIGbmxvt2rVj3bp1uLu7Gx2aEKIMS0lRzRE1zZjBVABSz6ayudZmqAwMNiaGjh3h//5P+t0J22LpmrvPgOtAANAc+FPTtN26ru+/odwooD/QDNCBf4BI4P9utvOktCTWn15PSnoKqemppKSnqOcZqbQKbEWzyqpH7+6o3Xy/93vzezmXyWnJ/D70d1wcVOPoAT8OYE3kGhKuJ5ChZ+Q63rAmw1g4cCEAMUkxPL386QJju5h4kUao5C7QI5BAj0D83fyp6F5RLd0qUtGtIjV9apq30TSNc+PO4efqd8uBTGSI8fLpwMwU3NHZZefD6CdlTkGRW1hYGGFhYUaHIYQoZ7Ztg+vXoUkT1TzREFkzt9gZdHyyay3/+888c5EQhrNYcqdpmjswCGis63oCsF7TtCXAw8DEG4o/CszQdf1s5rYzgJHcIrk7c/4M4yaNy/e9x1o8RrVG1QCIPBXJmog1XHe8zu6au81lWp5oiX2GPVf7XMXFRyV33oe9CTkSAoCDnQMuDi7mRxOHJsStiAPANdWVKUzBw8mD9LbpeHt64+3ije9hX6L2RtHMMXuoqFeqvcLz15/PG2QKEIV5nwB22HGZy3i29sTJX/WnSjqSRPKJZNzquuFaW402lxqVSsKuhJudnjzy294pwAnPFmoo84zkDC6vvVykfea3vb2rPT5dfcxl4lfFY0q7yXxZeyAuJfscFLS9b3df7JzU/9pXt14lLS6tSLHmt31+57korP3vtLuGP/tJI6ZTNVxc5FdDCCGE8YxukgmgmzIbexn401izpprA/MIFOHxYTW4uhNEsWXMXAqTrun4kx7rdQNd8yjbKfC9nuUb5lMul8qXKTF84Pf83F8Je9gLggw/Tmc71ytc5vuS4OVkL6ByA3RU73Ga4mTd7Zu0zJKwu+GI8a58AoZmzZHaM7YhjBTU01O4Xd+P6jytaRw1UbknUt1Gc/t/pW32cXJqubIpf5jD3WdsHTwsmaJJqFnp1w1U1n0sR5Le9/0B/Gv/SGIDrF6+zt9fem+0ij/y2dw5ypv3J7HYZ++/bT/ql9JvuJ+d5LWj7jrEdsaugkrPIyZHE/xNfpFjz2z6/81wU1v53+nOvL4vx49P7irRLIYQQosRkJXeG9jOzgZo7TVPn4Kef1DmR5E7YAksmdx7A1RvWXQHyzniryl65oZyHpmnajf3uNE0bhWrGSbBzMDQtfEBOvk40SGyQvaI5kATbtm2DrC4plYE2hd8nwIZNG7K3rwjpLdPZE7kHwjPXZRR9n/ltH5kSSWR4pFp3vuj7zG/7WO/Y7FHuLhV9n/ltn+qbmnvkvCZAUsH7SM9Ix8E++0+voO1vPM+3++9Em/zPc1FY899p1aq1rNigJqf28dlEeHhKwTsRZl5eXly7ds3oMEqNjIwMQ89XSkqKYUOZCyGKTtezp0Eo7zV3oM5BVnI3YoSxsQgBlk3uEoAbOwV5AfldNdxY1gtIyG9AFV3X5wBzAFq3bq2HbgktfoTh+ay7jd1lbR8eHk5oaGiudbe7z3zXPXub+8xv+4JHOi+c/LYPv/kmec5XYba/SfFCyW/7ktqnhf6d/vsPriZBbceTPPhgu9vYafmyc+dOPD3zu6ck8nPt2jVDz5eLiwstWrQw7PhCiKI5fBji4qBKFQgKMjCQrJo7G0juQAZVEbbDkpXZRwAHTdPq5ljXDMivjdr+zPduVU6IcmvFCrW8y2O9sYEIIYQQmdZn/iR17GjsACJ6RmZ9gIHNMgGaNQM3NzhyBKKjjY1FCLDgV0LX9UTgV+BNTdPcNU3rCPQDFuRTfD4wTtO0qpqmVQFeBL61VCxClAVZyd2dnnI7UAghhG2whcFUwHaaZTo6QlvVg8LcXFUII1n6fscYwBWIBhYBT+m6vl/TtM6apuUcteQLYCmwF9gH/Jm5TggBxMSooaadnUx08thmdDiiDBk6dCiDBxs0KZQQotSzleTO3Czz5jNJWYU0zRS2xKLJna7rl3Rd76/ruruu6zWyJjDXdT1C13WPHOV0Xdcn6Lrul/mYYOQE5kLYmn/+UZ3Wuza7gqt9qtHhiBKmadpNH8OHDzc6RCGEIDoajh5VzRCbNbt1+ZJkKzV3IMmdsC2WnsRcCGEBf/2llj3viIMIG/jlEiXqwoUL5ufLli1j5MiRuda5uroaEZYQQuSS1eywbVvVHNFQNjAVQpb27VX/w+3bISUFXFyMjkiUZzbwlRBC5GQywcqV6nnP1nHodvI1LesqV65sfvj4+ORZ5+3tDcC4ceOoW7curq6uBAcHM3nyZK5fv27ez8SJE2ndujXz588nODgYLy8vBg8eTHx83jki33//fQIDA6lRowYjR44kNVVqiIUQN5dzMBWj2VLNnbc3NGkC16/Dli1GRyPKO7lqFMLG7Nyp+tzVqAH1qyUYOxxZGaFpxjwszdvbm/nz53Pw4EE++eQTvvnmG95///1cZQ4fPszSpUtZunQpy5cvZ+PGjUydOjVXmX/++YeTJ0+yZs0avvzyS3744Qc+//xzywcshChTsqakvNlsRtbi6OuI/0B/NYexDejaVS3XrjU2DiEkuRPCxmSNktmzJ2i6SWruhNmUKVNo3749NWvW5N5772XChAksWrQoT7lvv/2Wxo0b06lTJx5//HFWrVqV631/f39mzZpF/fr1CQsLo3///nnKCCFETpcvq5uPjo6qGaLR3Oq50fiXxjDK6EiUrIQ3KwEWwijS504IG7N8uVr26oUaVUVq7m5bWRmuadGiRcyaNYsTJ06QkJBAeno6Tk5OucrUqlULd3d38+sqVaoQfcPkS40bN8Yux02DKlWqcPjw4ZINXghRqq1fr7oNdOigBlQRuXXpopb//QepqeDsbGw8ovySKgEhbMilS7Bpk7ozeuedqF9SSe4EEB4ezsMPP0zfvn1ZtmwZO3fu5PXXX8/V5w7A8YZRDjRNw2QyFbmMEELkZEtNMgEykjNIPpkMebsUG8LfX/W7S0mRfnfCWFJzJ4QN+ftvlc+FhoKnJ2CSZplC2bBhA7Vr12bixInmdSdPnjQuICFEubJmjVraSnJ39b+r7L5rN7QABhgdjRIaCnv3qnPVubPR0YjySq4ahbAhWVMg9OqVuUJq7kSmkJAQIiMjWbx4McePH+eTTz7hl19+MTosIUQ5YGv97QA0Zw3nGs7ga3Qk2aTfnbAFktwJYSNMpuzBVHImd1JzJwAGDx7Ms88+y5gxY2jevDnr169nypQpRoclhCgHIiJU3+W2bW2nv51PJx/an2oPrxkdSbasfncbN6rmmUIYQZplCmEjduyA6Gg1BULDhpkry8pIIKLQBg8ejJ7Pv7umacyYMYMZM2bkWj927Fjz83fffTfPdqNHj2b06NHm1z/88EOeMvltJ4QQWWytv52tyup3t3ev6neXlewJYU1SJSCEjcjZJNPcElNq7oQQQhgsK7nr1s3QMEqFrHMkTTOFUeSqUQgbkTUFwj335Fgpfe6EEEIYKKu/nZMTtGtndDTZLq28xPoK6+FtoyPJTfrdCaNJcieEDYiNhc2b1Y9n9+453pCaOyGEEAZat872+tsBmFJMpF9Kh2SjI8kt53x30u9OGEGuGoWwAX//rX48O3cGD48cb0jNnRBCCAPZan873ZTZN9nGrmQrVICmTdVE5ps3Gx2NKI9s7CshRPmU1d8uV5NMUBmfJHdClAmapvlpmvabpmmJmqad0jRt2C3KO2madlDTtLPWilGIG9lqcocpc2mDP5HSNFMYSZI7IQyW7xQIOd7UJbkToqz4DLgOBAAPArM1TWt0k/LjgRhrBCZEfuLjYdcu2+tvB7ZbcweS3Alj2eBXQojyZds21eeuZk2oX/+GN00mkD53QpR6mqa5A4OA13RdT9B1fT2wBHi4gPLBwEPA/6wXpRC52Wp/O8Cma+5yzneXbGN9AkXZJ/PcCWGwrFEyc02BkEVq7oQoK0KAdF3Xj+RYtxvoWkD5WcAkbjFchKZpo4BRAAEBAYTbYFVBQkKCTcZlq2zpfH39dV2gKrVrRxIefsrocHLbpxZppjSbOV851a3biqNHPZk1azd33BFvdDhmtvT3VRqUxvMlyZ0QBss5v10eUnMnRFnhAVy9Yd0VwPPGgpqmDQDsdV3/TdO00JvtVNf1OcAcgNatW+uhNtcxCsLDw7HFuGyVLZ2vESPU8skng2nXLtjYYG5w8dxFDnIQR0dHmzlfOQ0aBO++C1FRzWyqv6It/X2VBqXxfMlVoxAGiomBrVvzmQIhi65LzZ0QZUMC4HXDOi/gWs4Vmc03pwNjrRSXEPk6dgxOnABfX2jTxuho8rLlPncAYWFquXKlsXGI8sdGvxJClA8rV6r+DF27grt7PgVkKoRyZfjw4WiahqZpODg4UKNGDZ566ini4wvfpCc8PBxN04iNjS3wGH369CnyduK2HQEcNE2rm2NdM2D/DeXqAjWBCE3TooBfgUBN06I0TatphTiFALKTkrvuAnt7Y2PJlw33uQPo0EFNbXTgAJw5Y3Q0ojyR5E4IAxU4BUIWmcS83Lnrrru4cOECJ0+e5KuvvmLp0qWMGTPG6LDEbdJ1PRGVqL2paZq7pmkdgX7AghuK7gOqA80zH08AFzOfyyWisJqs5K5nT2PjKIieYds1dzlb5Pz9t7GxiPLFRr8SQpR9GRk3mQIhi9TclTvOzs5UrlyZatWq0aNHD+6//37+znFlcOXKFUaNGkWlSpXw9PSka9eubNu2zcCIRRGMAVyBaGAR8JSu6/s1TeusaVoCgK7r6bquR2U9gEuAKfN1hnGhi/Lk+nVYs0Y979HD2FgKYuvNMkGaZgpjyIAqQhhk61a4dAmCgyEkpIBCUnNnMeFaeJHKe7T0oPX21nm2D9VDzeu2tdpGwo6EfLfPWa64Tpw4wYoVK3B0dARA13V69+6Nt7c3y5Ytw8/Pj3nz5tG9e3cOHz5MYGDgbR9TlBxd1y8B/fNZH4EacCW/bcKBaiUbmRC5/fcfJCRAo0ZQzUb/+uzd7XEOcibVK9XoUAqUldz9+6+6oWuTzVtFmSNXjUIYZOlStezd+yaVc7putXiEbVixYgUeHh64urpSu3ZtDhw4wMsvvwzAmjVr2LVrFz///DN33HEHderU4a233qJWrVosWHBj6z4hhCierJqmrOTEFgU8EED7k+3hSaMjKVjt2uoRH69u6AphDVJzJ4RBspK7e++9SSGZCsFibrcmLb/tc9bsWUqXLl2YM2cOycnJfPnllxw/fpyxY9XAidu3bycpKYmKFSvm2iYlJYXjx49bPBYhRPlUGpK70iIsDD7/XJ3Tdu2MjkaUB3LVKIQBTp6EvXvB01ONlFkgmcS83HFzc6NOnTo0adKETz75hKSkJN566y0ATCYTAQEB7Nq1K9fj0KFD5jK34uXlxZUrV/Ksv3z5MnZ2dnh65pl2TQhRjly8CDt3gosLdO5sdDSln/S7E9YmNXdCGCCr1i4sDJydb1JQau7KvSlTptCrVy9GjRpFy5YtuXjxInZ2dtSqVatY+6tXrx4LFy4kOTkZV1dX8/odO3YQFBSE803/IIUQZd0//6hl166Q478Im3P+i/OcmHwCwoBQo6MpWLdu4OAAmzer5pm+vkZHJMo6uWoUwgBZyV3fvrcoKDV35V5oaCgNGzbk7bff5q677qJjx47069ePv/76i8jISDZu3MiUKVOIiIjItd2+ffvy1PCZTCYefPBBHBwceOSRR9i+fTvHjx/nm2++4eOPP2b8+PEGfUohhK3IGsXZ1ptkZiRnkB6XDrY7ngqgWuh07Kju1f77r9HRiPJAkjshrOzqVQgPVxVyBc5vl0XXZSoEwYsvvsjcuXM5ffo0y5cvp3v37owcOZJ69epx3333cfjwYapUqZJrm27dutGiRYtcj6SkJHx8fIiIiCAjI4O+ffvSqVMnZs6cyYcffsjo0aMN+oRCCFtgMmXPyWbryV2VJ6vQIaYDPG50JLcmTTOFNUmzTCGsbOVKSEuDTp2gQoVbFJapEMqVb7/9Nt/1w4YNY9iwYebXM2fOZObMmfmWDQ0NRb/FKKshISH8+uuvAFy7dk362QkhANXXLiZGTX/QoIHR0dycvas99q724GZ0JLcWFgaTJqlaUblnK0qaXDUKYWWFbpIJMom5EEIIq1myRC379JGfHktq3hyqVIFz51QCLURJkuROCCtKT4fly9Xzm06BkEVq7oQQQlhJkW4+GuziDxfZHbZzGmPUAAAgAElEQVQb/jI6kluzs8v+zc86x0KUFLlqFMKKNm6EuDioUwfq1SvEBiZTicckhBBCnDmjapXc3dUIj7Yu5XgK8X/HwzmjIymcrOQuq3ZUiJIiyZ0QVpTzrmihmrzoukyFIIQQosRl/T716KHmuLN1uimzb3Ep+Yns3h3c3GDHDjh71uhoRFlWSr4SQpQNWT+ehWqSCTIVQjHdakARYRvk30kI25FVo1QammQCkNWwpZT8RLq6qsQZpGmmKFmS3AlhJUeOwKFD4OOj5rwpFJnEvMh0XSc5OdnoMEQhJCcn4+joaHQYQpR7167BmjWqRUnv3kZHUzilreYOshNnaZopSlIp+koIUbpljjzPvfdCoa9npeauyBITEzl37hxJSUlSM2SjdF0nKSmJc+fOUalSJaPDEaLc+/NPuH5d3XisWNHoaApHz8j8/70U/UT26aPu165aBZcvGx2NKKtknjshrCQruRs0qAgbyVQIRZaRkUFAQADnz58nLS3N6HBsXkpKCi4GdLBxdHQkICAALy8vqx9bCJHbL7+oZZF+n4yW1SyzFFVTVKwIXbpAeDgsWwYPPWR0RKIskuROCCs4fRq2blWdqbPa3BeKrstUCMXg5eUlSUMhhYeH06JFC6PDEEIYJCkpe4qegQONjaUoSmOzTFAJdHi4SqgluRMloZR9JYQonX77TS3vuUd1qi40qbkTQghRglauVAle69ZQo4bR0RRBKRtQJcuAAWq5YgUkJBgbiyibJLkTwgqK3eRFJjEXQghRgkplk0xKb81d1arQrh2kpMBfpWACdlH6lLKvhBClz8WLsH49ODmpmrsikZo7IYQQJSQ1NXtY/tKW3JXWmjvIPtc//2xsHKJskuROiBL2++9qLvK774YidwOT5E4IIUQJWbECrl6FZs2gbl2joyma0lpzBzB4sFouWwaJicbGIsqeUviVEKJ0KdYomVl0XaZCEEIIUSIWLVLLBx4wNo5iKcU1dzVrQvv2qq+jzHknLE2SOyFKUHw8rF4N9vZqfrsik0nMhRBClIDExOwmmfffb2wsxVHrf7XoENMBSsmk6zfKSqh/+MHYOETZI1eNQpSgpUshPR26dgV//2LsQCYxF0IIUQKWLFE1R+3bq5qk0sbe3R4nfyew/jSdFjFkiLp3+9df6kawEJYiyZ0QJei2mmSC1NwJIYQoEVk1RqWySWYZULkydOsGaWnZ1wpCWIJcNQpRQq5eVZ3VAfr3L+ZOTCZ0i0UkhBBCQGysqjGys1M1SKXRmY/OsDtsN2w2OpLiy0qsv/vO2DhE2SLJnRAl5Pff1TDTXbpAlSrF3ImuS82dEEIIi1q0SNUYhYWpGqTSKHFfIvF/x0Os0ZEU3+DB4OIC4eFw8qTR0YiyQq4ahSgh33+vlrfV5EX63AkhhLCwefPU8tFHjY3jdlQfV50my5ugty697Vu8vWHAAPV8wQJjYxFlh4PRAQhRFsXEwL//goND9nw2xSJ97oQQQljQvn2wfbtKLPr1Mzqa/GWYMjgUe4i90Xs5EX/C/LiQcIH7G93P1NCpuDdy56DvQbp/2R2HtxxwsnfCz9WPyh6VCfQIJNAjkBc7vEhIhRCjP85NPfqoqkmdNw9efVWmthW3T5I7IUrATz9BRgbcc08xR8nMIjV3QgghLCir1u7++1WTQFuQnJaMq6Or+XWrOa3YfXF3vmVPXTllfp5uSjcv003pJKUlcfbqWfP7Y9uONT+ftGoS+6L30SWoC12CutCicgsc7R0t/VGK7K67VNeN48dhwwbo1MnoiERpJ8mdECXAIk0yQdXcSXInhBDCAtLSspv/Gd0k82jcUX45+Asrj69k45mNnB13Fn83dTe0YcWGXE65TIvAFtT1q0st31rU8q1FVc+qBHgEAHDxh4tU2luJ1cGr6fRYJ1IzUolLiuNCwgUuXLvA2atnqVuhrvl460+vJ+J0BEuPqMn9PJ086VW3F/3q9aNXnV74uvpa/ySg5sF9+GF47z34+mtJ7sTtk+ROCAs7fVrdfXNxsUCTFxlQRQghhIUsWQIXL0LDhmp+O2uLjI/kx/0/snj/YnZG7TSvt9Ps2H5+O2F1wgCY13/eLWvVYn+PJebHGLTXNBztHXG0d8TDyYMgn6B8yy8YsIB1p9apx+l1HIk7wuL9i1m8fzEjWozgq75fWe6DFtHjj6vk7ocf4KOPVJNZIYpLkjshLCxr7qC+fcHT8zZ3Js0yhRBCWMicOWo5apT1G4VEJURRZ1YdTLoJAC9nL/rX78+9IffSPbg7fq5+5rKFai5pylwW8nME+QTxsM/DPNzsYUAlmksOL2HJkSUMqD/AXO7PI3/y04GfGNFiBJ1qdEKzwokKCVFz3q1ZAwsXwpgxJX5IUYZJcieEhS1apJYWmRhWBlQRQghhASdOwN9/q1YlDz9c8sc7feU03+/9npc7voymaVT2qEzfen1xc3Tj/kb3E1Y7DGcH52LvXzdljpJZzJ/IYN9gnmv3HM+1ey7X+i93fMkfh/9g3u55NA1oyjNtnmFYk2G4O7kXO9bCePJJldx98QU89ZT0yBDFJ8mdEBZ08CDs2qWaVPTqZYEdSs2dEEIIC/gqs9XhkCHg53fzsrdj67mtfLjpQ37a/xMZegadanSiUw3VkezX+361XE1YEWvuCuv9u9+nUcVGfLXzK/Zc3MOoZaOY8O8ERrcazQvtX6CSeyXLHjBT//5qALY9e2DzZmjXrkQOI8oBi1UJaJrmp2nab5qmJWqadkrTtGE3KTtV07Q0TdMScjxqWSoWIYySVWs3aBA4F/+GZDYZUEUIIcRtSknJTu5GjSqZY4SfDCf021Du+OoOftin+icMazKMim4VzWUs2cTxdmvuClK3Ql2m3TmN08+f5rsB39GuWjsup1zm3Q3vMnvrbMseLAdnZ3jsMfX8s89K7DCiHLDkV+Iz4DoQADwIzNY0rdFNyv+o67pHjscJC8YihNWZTPDdd+q5RZpkAui61NwJIYS4LT/8oOZfbdECOna07L51Xafvor50m9eNtafW4u3szfgO44l8LpKFAxdSz7+eZQ+YpYRq7rI4OzjzYNMH2ThiIxtHbGRIwyE82/ZZ8/urTqziaNxRix5zzBjVE+PHHyEqyqK7FuWIRZI7TdPcgUHAa7quJ+i6vh5YAlihVbcQtmHtWoiMhOrVVcdoi5A+d0KUCYVt3aJp2nhN0/ZpmnZN07RITdPGWztWUbboOnzyiXo+dqxlGoPoum4eGEXTNJoGNMXXxZe3ur3FqedPMf3u6VT3rn77B7pZDCVUc5efdtXasXjIYvOgL6npqTz6+6PU/6w+j//xOKevnLbIcWrWVIOxpaWpvndCFIelvhIhQLqu60dyrNsN3Kzm7l5N0y5pmrZf07SnLBSHEIaZO1cthw9X89ZYhPS5E6KsKGzrFg14BPAFegLPaJo21GpRijJnwwbYuRMqVoShFvhL2n5+O3ctuIsvtmVnHxM6TiDyuUhe7fIq3i5WGse/hGvubiYxLZGedXpip9nxza5vqDurLuNWjiM2Kfa29/1c5vgus2fD9eu3vTtRDllqQBUP4OoN664ABQ0EvxiYA1wE2gK/aJp2Wdf1RTcW1DRtFDAKICAggPDwcAuFbDkJCQk2GZetKovnKyHBgZ9+ag/Y06DBJsLDUyyy30bR0aRcv17mzldJKot/XyVJzlfJy9G6pbGu6wnAek3Tslq3TMxZVtf16TleHtY07Q+gI/CDteIVZctHH6nlk0+qkTKLKzI+ksmrJ7Non7pUi02KZXTr0WiahpezlwUiLRpr1tzdyM/Vj6/6fsXEThN5fc3rLNq3iI82fcRXO77ipQ4vMaHjBFwcineyu3aFJk1g717Vj9/oyeZF6VOo5E7TtHCgawFvbwCeBW78ZnsB1/LbQNf1Azle/qdp2kxgMJAnudN1fQ4qEaR169Z6aGhoYUK2qvDwcGwxLltVFs9X1h227t3hgQcsOMSVnx8XXV1pXcbOV0kqi39fJUnOl1UU1LqloN9VADQ1+kRnQBpoiWI5fBh++w2cnIo/d1psUizT1k3js62fkWZKw9nembFtx/JKp1esMgdcgQysuctSx68O3w/6nvEdxjNp9SRWHFvBwr0LmdR5UrH3qWkwbpwaXOX999W0FdI7QxRFoZI7XddDb/Z+5l1JB03T6uq6ntW7tBmwv5Bx6Bj69RTi9nz9tVqOGGHhHcuAKkKUBUVt3ZJlKqpe4puCCkjrlrLHkufr/ffroeuB9OhxnsOHj3D4cNG2P3rtKC/sfoHEjEQ0NHoE9ODxmo8T4BjA7s27LRJjscWpRXJqsk38fb1c9WXC3MPI0DNYv249AFfSrnAi4QQtfFsUaV9Vq2pUrNiW/ftdePfdvXToEGexOOX7WDSl8XxZpFmmruuJmqb9CrypadoTQHOgH9Ahv/KapvUD1gGXgTbAWKD4tzmEMNCePbBtm5rbbsAAC+9cBlQRoixIoAitWwA0TXsG1feus67rqQWVk9YtZY+lzte5c/DPP6om6MMPq1C3bpUi76NjRkc+PvMx1byqMf2u6TSr3Oy247KUXd67uMxlXN1cbebvK5TQXK+f/vNpPt/zOf3r9+f9u9+njl+dQu/rlVdUDd7y5U2YZMErZPk+Fk1pPF+WvGocA7gC0ajmlU/pur4fQNO0zpqmJeQoOxQ4hvphmw+8p+v6PAvGIoTVZNXaDRsGrq4W3rkMqCJEWXCEzNYtOdYV2LpF07THUX3x7tR1/awV4hNl0IcfqlEXBw+GunVvXR7gaNxR7v/5fqIS1Dj8jvaOrH9sPSsfWmlTiR1Arem1aLqiKdQ2OpKC1fCugbujO78f+p2GnzVk/N/juZJypVDbjhwJvr5qQJy1a0s4UFGmWCy503X9kq7r/XVdd9d1vYau69/neC9C13WPHK8f0HW9Qub8dvV1Xf/EUnEIYU2pqdlz21m8SSZIzZ0QZYCu64lAVusWd03TOqJatyy4saymaQ8C7wB3y/yvoriiolRfcFA1QLdyNfUqE/6ZQKPPG7F4/2Kmhk81v+fr6lsyQd4mr9Ze+IX5gZUG5yyOlzu9zJFnjzC8+XDSTGl8sPED6s6qyxfbviDDlHHTbT08skfOnDq15GMVZYdcNQpxG5Ysgbg4aNoUWrYsgQOYTOglsFshhNXl27oln5YtbwMVgK2apiVkPv7PgHhFKfbee5CcDP37q4nLC2LSTczbNY96n9bj/f/eJ82UxmPNH2Nq6FSrxVrWVfGswjf9vmHbyG10qtGJmKQYRv85mu0Xtt9y2+efBx8fCA+HNWtKPlZRNlhqKgQhyqX/y7zkGjHCMhPD5qHrUnMnRBmg6/oloH8+6yNQA65kvQ62Zlyi7Dl/PrvW7mY1PrujdjNq2Si2nNsCqIm6P+n5CW2qtin5IC3g3OfnSD2bqho4lwKtqrRi3fB1/HzgZzaf28wdVe8wvxeXFEcFtwp5tvH2hhdfhNdegylTIDS0hK41RJkiV41CFNO+fbB6Nbi7wyOPlNBBpM+dEEKIInjrLdVlYNAgaHaTxCc1I5Ut57ZQ2aMy8/rPY8PjG0pNYgcQNS+K0/87bR41szTQNI0hjYbwQY8PzOsiTkVQ/aPqTFkzhaS0pDzbjB0Lfn4QEQErV1ozWlFaSXInRDHNmqWWjz6qmk2UCOlzJ4QQopAOHYIvvwR7e5Xk5ZSansqvB381v76j6h0sHryYI88c4ZFmj2Cnla7fmipPVSH47WCoaHQkt2dV5CqS05N5c92b1P+0Pj/u+xFdz+6Q4eWV3W9ywgTIuHlXPSEkuROiOC5dggWZQyE880wJHkhq7oQQQhTSxInq4v+JJ6BBA7VO13WWHl5K49mNGbR4EKtOrDKXH9JoCJ7Ot5pu0TYFDg8kaHJQqU/upoZOZd3wdbSo3IIzV88w9JehdP22Kzsv7DSXeeYZCAqCvXth/nwDgxWlgiR3QhTD3Lmqs3qPHtk/oCXCZJIG9kIIIW5p3Tr44w/VVSCrr92BmAP0XNiTvj/05dilYzTwb4CTvZOhcYq8Ogd1ZuvIrczpMwd/N38iTkfQak4rFu1dBICLC0ybpsq+9hokJhoYrLB5ktwJUUTp6fDpp+r52LElfDAZUEUIIcQtpKdntyKZMAGcveN57q/naDq7KX8f/xsfFx9m9pzJ7tG76RzU2dhgLeTSP5eI+TUGEm5dtjSwt7NnZKuRHH32KC+0e4GK7hXpUbuH+f0HHoDWrdXk9FmJnhD5katGIYpo6VI4fRpq14ZevUr4YNIsUwghxC3Mnq2a7AUHw/jxMH3DdD7Z8gk6Ok+1foqjzx5lbNuxONo7Gh2qxRwfd5z9g/aryUXKEB8XHz4M+5DjY4+bR9BMTU+l34/3MmzCRgA++ACOHDEySmHLJLkToog++UQtn33WCpVqMqCKEEKIm7h4UTXVA/j4Y3B1VZNnD2wwkJ1P7uTz3p/j7+ZvbJAlQDdlDjpSRu9/ejiZZ0hh7s65LDuyjHEHOlA99G/S0lTLIV0mwhX5kKtGIYpgzx41maiHBwwfboUDSs2dEEKIm3h89DWuXAGXBmu4q2cyoGp/frnvF5oGNDU4upKjZ2RmNuXgSnZEixFMv2s6nk6enGn9ELjEs3IlfD0/79QJQpSDr4QQlvNB5tQ0w4eryUVLnAyoIoQQIh/RidH0fu1Llv/uCY4J0HsMO6K2Gx2W9Zgyl+XgJ9LZwZnxHcdz5NkjPNapD/R4EYCRY5KZG7HU4OiErZHkTohCOnUKvv9ezR/04otWOqiuS82dEEIIs8TriUxbN43a77Zh+ce9AWj36B8ce+1fOtXoZHB01mNulmlvbBzWVNmjMl/3+5rNnz2Fd4Mt6EkV+PLtsls7K4pHkjshCumDD9T8QQ88ADVrWumg0udOCCFEDr0W9uLV1a+S8Ov7kFCFFncksuGLB6nqVdXo0KyrHNXc3eiOam3YsawNLm7pbP47iIUL1frZW2cTcSrC2OCE4eSqUYhCiI6Gr75Sz19+2YoHlj53QghRrum6TnJasvn1M3c8Q81Tr8OB+/DwgJ8XuZfLe4Dmmrty+NkBatXS+PQTBwDGjIH1e87y/Mrn6fJtF+5ddC/7ovcZHKEwSjn9SghRNDNmQEoK3HsvNG5sxQNLzZ0QQpRLuq6z4tgK2n7Vlgn/TDCvb2o/hNifpwJqztVatQwK0GjluOYuy+OPw4ABcPUqvPhkIBPaTcbd0Z1lR5bRdHZThv8+nMj4SKPDFFYmV41C3EJ0dPak5a+/buWDy4AqQghRrui6zqoTq+j0TSd6LezF1vNbWXJkCanpqSQmwuDBGgkJGkOHwiOPGB2tccp7zR2oy4Mvv4Rq1WDLJnuuLXud42OP80ybZ3Cwc2De7nmEfBrCE0ueIN2UbnS4wkrK8VdCiMJ5/31ISoI+faB1aysfXAZUEUKIckHXddZEruGF3S9w14K7+O/Mf1RwrcD0u6ZzYMwBnOydGTkS9u+HBg3URX25/nmQmjsAKlSAn34CR0eYORPWLAtg1j2zOPTMIR5u+jAm3UR0YjQOdg5GhyqsRJI7IW7i4kX47DP1fOpUAwKQZplCCFEuHI47TPf53dl9ZTe+Lr5M6z6NyOciGd9xPO5O7vzvf7BoEbi7wy+/qPlWyzOpucvWrh18+KF6/thjsHUr1PKtxfwB8zn09CE+6PGBuey+K/t4/I/HORx72KBoRUmTr4QQN/HWW5CcDH37QqtWBgQgA6oIIUSZlG5KZ+WxlebX9f3r83DTh3ms5mNEPhfJpM6T8HT2BODnn2HyZFVT9913quau3MvIXMpPJABPPw0jRqjxAfr2hdOn1fq6FeoSUiHEXG7+qfl8s+sbGnzWgIE/DmTT2U0GRSxKiiR3QhTg2DH44gtVcTZtmkFBSM2dEEKUKddSrzFr8yzqfVqPngt78t+Z/8zvzR8wn0eCHsHbxdu8LjwcHnpIPX/vPejf38oB26gak2sQPC0YXI2OxDZoGnz+OXTrBlFR0KsXxMXlLfdc3ecY1XIUjvaO/HboN9rPbU+Xb7rw55E/MemmvBuIUkeuGoUowKuvQno6PPqolUfIzMlkQjfo0EIIISwnMj6ScSvHUe2jaoxdMZYT8Seo41eHhOsJBW6zY4eqhUlNhdGj4aWXrBiwjav+fHWCJgWBi9GR2A4nJ1XL26gRHDgA99wD167lLlPVtSpf3PsFp54/xSudXsHb2ZuI0xH0WdSHWZtnGRO4sChJ7oTIx+bN8OOP4OwMb7xhYCC6LjV3QghRyj3+x+PUmVWHjzZ9xNXUq3Su0Zlf7vuFQ08fokftHvlus3Mn3H23uji//341arO00he34ucHf/8NNWvCli3Qu3feBA+gskdl3rnzHU6/cJoP7v6AYJ9g7mt0n/n9NZFrOBp31HqBC4uRq0YhbmAywXPPqecvvADVqxsbjPS5E0KI0iUuKY7E64nm1xVcK2Cv2fNIs0fYPmo76x5bx8AGA7G3s893+23boHt3uHRJjdQ8fz7Y51+03Ir5PYaYX2Oy+94JsypV4N9/oWpViIiAnj3hypX8y3o5e/Fihxc5NvYYgZ6BAGSYMnh8yeOEfBrCPQvvYcnhJTKVQikiyZ0QN/juO1VzFxgIkyYZHIz0uRNCiFJB13XWnVrHQ78+RNUPq7JgzwLzexM6TuD0C6eZ138eLQNb3nQ/27f70q0bXL4M/fqpkTGdnEo6+tLn4IMH2T9oP1w3OhLbVLs2rF2rblD/9x907QoXLhRc3k7LvtZIuJ5A95rdcXFw4a9jf9Hvh35U/6g6E/+dKLV5pYBcNQqRw9WrMHGiev7uu+DpaWw8UnMnhBC27fil47wR/gYhn4bQ9duuLNy7kOsZ13MNNV/RvSKVPSrfcl8LFsDEiU1ISIAHHoDFiyWxK4h/P3/8B/iD1GgWqHZtWLcO6taF3buhQweIjHS75XbeLt7M7TeXsy+cZfpd0wmpEEJUQhTvbXiPkE9Dcg0CJGyPzGgoRA6vvqrubLVtmz06maFMJulkIYQQNur5Fc8zc/NM8+sqnlUY0WIEI1qMIMgnqND7yciAV16B998HsGPcOPVcGm4UrOH3DQEIDw83NhAbV7MmbNig+t5t3QpjxrTCzU3147yVCm4VGN9xPC91eIkNZzYwd+dcNp7ZSNuqbc1lpm9QyV+vOr1wdnAuuQ8iCk2SOyEybdumOqzb22dPgWA4GVBFCCFsQmxSLH8c+oOWgS1pEdgCgBaVW+Du6M7ABgN5uOnDdA/uXmA/uoKcPQuPPAJr1oCDAzz99BFmzAi59YZCFFLFimpKjSefhO++s2foUNX95L33wNHx1ttrmkanGp3oVKMTGaYM8994XFIck1dPJt2UjrezN4MaDGJo46GE1gzF0b4QOxYlQpI7IYC0NPWfnq6rQVSaNTM6okzSLFMIIQxz4doFfjv0G78c/IW1J9eSoWfwVOun+Lz35wDc3/h+BjccjLuTe7H2/8svMHIkxMdDpUrwww+gaecBSe5uJTkyGc1OQ+YLKhw3NzUwj5/fET7/PISPPlI1efPmQa1ahd9PzpsX9nb2TOs+jUX7FrErahdf7/qar3d9jY+LD31C+vBG6BvU8i3CzoVFSJWAEKi7Vzt2QFAQTJ1qdDQ5yIAqQghhdV9s+4I2X7ahyodVeHr506yOXI2maYTVDqNrUFdzORcHl2IldleuwIgRMHiwSuzuuQf27lUTUItb03WdzbU2s6nmJqNDKVU0DQYMOM/atWpEzfXr1Ty+H32kmgYXlY+LDxM6TmDnkzs5MOYAr3V5jYYVG3I55TIL9yzEzTG7f1/EqQhOXzltwU8jCiI1d6Lc270b3nxTPf/6a3Av3g3YkiE1d0IIUaKiEqJYHbma0JqhVPGsAsCxS8fYdn4bLg4u9Kjdg0ENBnFvyL34uvre1rFMJlVTMnEiREeDiwt88AGMGSPdq4skq7ZOy3yIIunQAXbtguefh++/h3HjVK3xV19BkybF22eDig14s9ubvNntTY7EHWHT2U3mQYR0Xeeh3x7i9JXTNPBvQFjtMMLqqBslro6uFvxkAiS5E+VcSgo8+qhqljlmjJpXyKbIgCpCCGFRl5IvsfbkWlZHrmb1ydUciDkAwKe9PuXpO54GYETLEXQL7kZozdBctQ+3Y8sWePZZtQTo2FH1727UyCK7L1d0U2Z2Jw1biq1iRVi4UI3KOnq0+rts3hwee0y1YKpWrfj7DqkQQkiF7KbFiWmJtApsRXxyPAdjD3Iw9iAfb/4YFwcX2ldrz5SuU+has+tN9iiKQpI7Ua5NnKhq7mrXVk0zbY6uS82dEEIUk67raDn+D+36bVciTkWg5+io5eboRpegLuZaO4D6/vWp71/fIjFs2QJvvw1Ll6rXgYFqJMxhw+TeXbGZ1EKz03L9W4qi69MHDhyAyZNh9myYO1clfc8+q66R/Pxu/xgeTh78ev+vpGWksensJlYcW8HK4yvZfmE7a06u4fWur5vLLtq7iN0Xd9M1qCsdqnfA28X79gMoZyS5E+XWn3/CzJlqdLJFi8DDw+iI8iF97oQQotAup1xmx4UdbDyzkY1nN7IzaifHxx7HxcEFUH3kHO0daV+tPXcG30n34O60qdoGJ3vLTian62oC6f/9D/7+W61zdYWxY9VFtOFzqJZyUnNnWV5eMGuW+vt89VU1v+L778Nnn8Hw4fDccxBigTF+HO0d6RzUmc5BnZl25zRik2KJOBWRa2qFhXsX8ufRP3lvw3toaNT3r0/rKq1pU6UNnWp0Mo9UKwomyZ0ol06eVENPA0ybBm3aGBpOwaTPnRBC3NSh2EO8suoVdkXt4uTlk3ne3xW1i3bV2gEwt+9cKrhWKLF+PpcuqREJ58yBgwfVOg8PePpp1a+pUqUSOWz5IzV3JaJuXfjxR5gwQSV5K1bA55+rR+/eqvlmWFjhpmIvUDoAABmNSURBVE8oDH83fwY0GJBr3QvtXqBJpSasO72Oree2mptxLtizgPsb3c8Pg38AID45nh/2/UDjSo1pXKnxbfeHLUskuRPlTnIyDByofoTvuQdeesnoiG5Cau6EEOWYruucuXqGQ7GHOBR7iIMxBzkUd4iWlVsyI2wGAE72Tvx+6HdA1cw1qdSEtlXb0r56e9pXa09Nn5rm/VXzuo2ORAVISoLly+Gnn+CPPyA1Va2vXBlGjVK1IRUqWPyw5ZrU3JWsVq3gr79g/374+GNYsEC1dvrzT9VXb+hQePhhaN3a8k2L76x1J3fWuhOA1PRU9lzcw7bz29h6fivdamYPJ7v53GbGLB9jfl3Vs6o50WtcqTGDGw7Gw8kWm2SVPEnuRLmi62o+u5071bwu331n47mTDKgihCjjTLqJqIQoTsSfUJOCZ04t8NxfzzF351wS0xLzbHMt9Zr5ebBPMAsHLqR55eaEVAjBwa7kL23On4d//1X96JYvVwkeqP+uw8LU70yfPpar4RA3yFFzJ0pOo0bw5ZfwzjuqL978+apGetYs9aheXf2d9+mjBqRzcbHs8Z0dnGlTtQ1tqrbhKZ7K9Z6viy/Dmw9nX/Q+9kfv59y1c5y7do6Vx1cCMKB+do3g+L/HE5ccR12/utStUJe6fnWp41en2PNT2jpJ7kS58vbb6g6Umxv89hv42notvgyoIoQo5TL07Am0YpNimbtjLmevniXyciQn4k8QeTmSlPQUADaN2ETbaqr/jYOdA4lpiVRyr0R9//o08G9gXjaqlD3EpKZpDGsyrEQ/w4ULsHGjmhfsn39g377c77dtC0OGqHnrgoJKNBSB1NxZW8WKanCVl19WcwIvWKD65Z05owZhmT1bJXbt20PXrurRtq3qZ1pS2lZra/6/IsOUQeTlSPZF72Nf9D7OXj2bayCWXw/9yon4E3n2UcG1Ai+0e4HJXSYDcDHhIutOrSPIJ4ga3jWo5F4621FLcifKjYUL4fXX1Z3VRYugaVOjIyoEaZYphLBBGaYM7DQ780iUayLXsD9mP9GJ0Vy4doGz185y7qq6k97EvQl3dlPNrJLTkpm4amKe/fm7+VPLtxbppnTzupc7vczkLpPxc7XAcH2FZDLB6dNqQvF9+9Royps2walTucu5uUFoKPToAQMGQI0aVguxfJs+XXWSb9YJKKDmbs0a2LpVdRwTFqVpqslmq1bw4Ycq0Vu6FJYtU8/XrFEPAHt7aNgQWrSAli3VsnlzNXiLpdnb2VPHrw51/OrQv37/PO9/de9XHI47zNG4oxy5dISjcUeJvBxJXHJcrtF0t5zbwn0/32d+7WTvhJ+jH0HHgqjsUVn12XVTbay3nttKhp5BZY/KBLgH2NR8fZLciXJh+XI14hPAjBnQt6+h4RSeDKgihChBaRlpxKfEE58cT3xKPBVcK1C3Ql0ATsSf4POtn3M55TLxKfFcSr5ETGIMMUkxxCbFcuzZYwT7BgMwe9tsfjrwU77HiHOMMz8P9AxkXLtxVPOqRrBvMLV8axHsE4ync97hI0vqrrmuQ1QUnDgBkZHZj4MHVUKXkJB3G09PVRPRvj3ceadaOll2gE1RGG3awH33oX+5GNDy1tytWQP33aeqlUSJsrNTfe5at4Y33oCYGIiIUKPErl2rbpBkPebPz96uenU1cEudOrmX1auXTOIH0C24G92Cu+VaZ9JNRCdG42iX3Xbax8WH/vX7c/rKaU5fOU1sUixRGVFEnYsC4HvH781lx/8znrWn1ppfuzq44ufqRwW3CgyoP4CpoVMBiEuK4+udX5vf83P1w8fFB08nTzydPfF18cXezt6in1eSO1HmRUSopjLp6TB+PDz/vNERFZKuq4ckd0KUCZqm+QFzgR5ALPCKruvf51NOA94Fnshc9RUwUdf1mw4LGJsUy8ebPibhegIJ1xO4lnqNhLQEktOSWTwk+2K318JebD+/nYTrCSSnJ+fax5jWY/is92cAxCTGMGPjjJseLyu5C6sdhr+bP5XcKxHgHkA1r2pU9apKNa9q7NuS3YbRwc7BPBCKpeg6pKTA5cvZj4sXVQJ38WLe5+fOqfIFCQiAxo2hSRP1aN1a9T2yt+z1lyiObt1U4jb4CWAuWvp1guZ/pzJyFxc1I/fixaqcsKqKFdVgdQMHqtdJSbBnjxrjYMcOtdy7VzXlPHMGVq/Ouw8PDzV5etWq6lGlCvj7qwGJbnz4+t7ed9JOs6OyR+Vc67KmaciSeD2R31f9Ts1GNYlOjMbN0c38XsOKDUlKSyIqIYqLiRdJTk829/trX629udzJyyeZ8G/Btcibn9jM/7d390FyVWUex79P9/S8pWcgTCZREZIASTCJGwhRgy7rgG65u2oJq6sWLm/GBfFl0fjuxoJFpYrs6q7FskiqgihYsFmLCCzU7lrIFC9raRI0kGiAsGFIwiqZBDLTM5mZnpln/zg97z3TfSc90y/5fapudd/b595+cubk3H763D73rae+FYDrH72e+5+9n4aaBhprGmmoHnlc0rSET675JBAmmpqKkjupaI89FmbEPHYM1q0LNyovm1xp6D9v2QQsIjncCvQBC4BzgIfMbKe77x5X7mrgYmAV4MDPgH3A96c6eNtrbXz+vz6f9bW+gb7he7m91vMah7oPARC3OCfXnszcurnMrZ07ZjbJM+aewc3vvpm5tXOHy8yfM5/m+mbm1c8jEU/gHi5l/NjydXx4yTr6+yGdDl+mpfug4//gwP4kzzwzant64vN0OvTT3d1hGXo+2baOjpFE7uhR6OuL9odoaoLFi8cuS5eGZK65OdqxZJZdeCG2+Xbm/eXjVKWOsejOO8NvLQYGwmU6SuxKQn09rF0bliHpdLjE+fnnYe/ekce9e+HAgZCj79kTllzMwkhfMhmWhobsz5PJkPfX1Excsm2vrg5JY1UVxONziB89g4Wx8znj5DA6OfTaP7T8a6YMxGJOd38XR44d4XD3YRprRoYgT6k7hfVr13OkJ7x2+NhhOno76OjtoLO3k5NqRn4b+MKrL7DzDzuz/nvPf+P5w8ld38DUHZ6SO6lYjzwSLr/s7g5T9t5+e5nlSe76vZ1IhTCzOcAHgZXungKeMLMHgMuA8T9CuwL4jrsfyOz7HeBvyJHcLfz92fzdt3+AMdTRjXR4d3/7GSBkip/wW1gHPFh/Gv9TPR9344zedj50rI3tVU00Vofu5/UDc/hiz7tw4FXgCPCCp3FeBl7OGsMvaOJHLALgVLrZwO84SJIrRpW5lR1TzoNRm1mG5rs6SB23sXzC/p/mXAYzR/pabA9nxrqIxyFeBYmqMFNlVSI8JhIj2xKJ8IHs3F+cS6wq7L/nqj10be6i7o5l0BymT39p40sc+vdDU1X5BMvuWEbyzWP3P+1LpzH/w+ES0/YH22m7sW2qQ0An7GjYMbyabf+m9zWx6PpFAHQ/383vLv1dpDiz7V93Vh3L7xmp5x1rd8DAJAfIYrL9J9Tzromzn04l2/7LruhjZc1NI0OwPT3hk/pUQ7JSdIlEuAzzrLMmvuYevqw5eDAsBw6EiYwOHx67tLeHx6Evdo4enemoz89dBCMWSxKPJ6mqOp1YLHx8MwOzxZh9J/N89HaoNmj555Htg/yQBX4HMIgPLTaAM8gLsThLhwYobOppeJXcSUXasiUkdH19cMUVYQrfsrukRpOpiFSSpUC/uz83attO4J1Zyq7IvDa63Ios5TCzqwkjfSxlKUvS+X+4jXWkac8kgDHSLKGTvQNz6Mzcp22AQZbSOcURJjpIgnkcIkGahXRxNp0kOcYKdpEgTYI0y+gmSne8iP1s5h+pp5t6umngOow4u1jJXI5wMq+xe/C7dA4uh36gd/JjpTMLAIlqhrKXLm6lk+UM/NFbgZAo9fC3dHJJlqNMLtv+fR+5Bj7y08z7/xmdfCXncTpH1Xu2/eds3wI3bARgkDPoZHOkOLPtP7j9N3DvSDNL8TM8wsfESffPUs+RZPs7bf8iMK6t9/TA+98f7dgnoJZiBzAJI3yhMxdYmUf5fuJ00kCKJCmSkz5PkaSHWnqpGV7Gr4/enibBAHEGiNNPVdbH8dsGiTM4GD62pdO5Y59cPLNk98rws6k/Gyq5k4riHi69/PrXw/PrrgszOpVljqTkTqSSJIGOcduOAhNnEgllj44rlzQzG/+7O3ffBGwCWHHmud5082piNvTtLsPPbejbYjLdisG/nFbL7fPD9sEjTaT3r2Z9c4KvLwrbvKeOnj2rw7fKo4/D2GOO9rZ5Cb65OMwaN9A9QNfuBbDrKXZdNfJxrWN7RxhCzFOsLkZy5cgMeEP7N5y3e3i2xLN/28VAV4RhJsDOS4d/2Kj969/0NCTDR6PT23p43SvRrvfMtn/t6f8GC8IlsU2H+lj94tQJ+FM7nmL1eauH17Ptn5j3Nlh8MwB13QOs3h1tNCzb/rG6NbBy3XCZcyP/nbLvn62eo8j6d3pxI1z+V2NH6mprw53k3/e+SMc/0bS2ttLS0lLsMI5bFSPJ4EzKp77cw1XBo5ehS9aHpk8YvRRi+/IpviNRcicVI5WCq66Cn/wkrN90U7gvS1ldijmakjuRSpICxs8F1whZh8bGl20EUrkmVKmbG+fNH5rmdHPN1bBs3PSPjXFOmj/96evi9XEa39II4/KOxjXHNyVetv3nLD++mxFn2792YS21C6d/V+Zs+1c3V1PdnGOazS5CvWWRbf/hep6myfYv6b/T0drwCbq2Fu/txWpqhtdFZptZ+B1eVYlkVSUShsjx2bsXLr4Ydu8OP7C9664yut3BZAYHyzgzFZFxngOqzGyJuz+f2bYKGD+ZCpltq4Bf5SgncuJ59NEwK+bDD0NPDy9u3criSy7RbJkiGUrupKy5h/unXHdd+FHt2WfDT38Ky5YVO7IC0IQqIhXD3bvM7D7gRjP7BGG2zA8Ab89S/EfAejN7mHBh3BeAW2YtWJFSNfo+dpkEri2ZZPHQZXNbtkx4XeREo0+OUrYOHgy/nb7yypDYXXIJ/PKXFZLYgS7LFKk8nwLqCL+Lvwe41t13m9kFZjb61tm3Aw8CzwC7gIcy20RObNu2TZ24Dd0Hb9u22Y1LpIRo5E7KzuAg3HknrF8fkrqTToLvfQ8uv7zCrmJUcidSUdz9COH+deO3P06YRGVo3YEvZxYRGfLlPP5LXHihRu3khKbkTsrKo4/CF74Av/51WH/ve8P96049tbhxzQgldyIiIiISgT45SlnYtStcgnnRRSGxO/VUuPtuePDBCk3sIPzmrqKGIkVERERkJmnkTkqWOzz+OGzcCA89FLYlk/C1r8HnPgf19cWNb8Zp5E5EREREIlByJyXn2DG47z645ZYwQQpAXR2sWwcbNsCCBcWNb9YouRMRERGRCJTcSUlwD5Nb3XEH3HtvmCgF4JRT4LOfhU9/GpqbixvjrFNyJyIiIiIRKLmTohkYgCefhK1bw73pXnxx5LW3vAU+/nG47DKYM6doIRaXkjsRERERiUDJncwad9i7Fx544PXcdhv8/OfQ3j7y+uteB5deClddBStXFi/OkqEJVUREREQkAiV3MmMOH4bt28OybVtYXn4ZYOQu42edFW4+fvHFsHatBqrG0MidiIiIiESg5E6OW3s7PPvs2OXpp2Hfvollm5pg5cpX+OhH53PRRbBkiQanJqXkTkREREQiKEhyZ2afAa4E3gzc4+5X5ij/eeArQD3wE+Bad+8tRCxSOAMD8OqrcOhQWA4cgP37xy5tbXDkSPb96+pg9WpYsyb8hm7NmpDMPfbYb2lpmT+7/5hypORORERERCIo1Mjdy8C3gPcAdVMVNLP3AF8FLsrstxX4+8w2OU7ukE5DTw/09obHnh7o7AxLR8fI8/HbhhK59vbweORIyC9ySSZh2bKxy/Ll8KY3QZXGhqdPyZ2IiIiIRFCQj97ufh+Ama0B3pij+BXAZnffndnnm8CPySO5a2+HTZtCAuM+9N7FX9+3bxGPPJK7/OAg9PeHEbGpHvMp09c3Nnkbet7bO/LehTB3LsybF25D8IY3wGmnjV1OPz1MhKJLK2eAJlQRERERkQiKMa6yArh/1PpOYIGZNbn74al2bGuDa66Z0dimaVGxAxgjQR811ket9Q4vSeumIZai0VI0xFI0WBcNsS4aLBUeYylOtg6a40dojh1mXvxVmmKvkrB+6AUOZJZfHX98a3t7oabm+A9U6dLpkFmLiIiIiOShGMldEjg6an3oeQMwIbkzs6uBqwHqa5fxrvNfwAzMHBsu45nHkXUDGLc+PAgybv14y/en+6iuTowaZPFMjJnyOBjEzInHnXh8kHjMicecWCxsi8WceCyzPR6ex4afZyszSE31ANWJQaoTA5llkEQivJ5bLFPlDRNe6cwsWeZDKYju7m7q6+tn6OiVZSCZJJVK0draWuxQyobqKxrVl4iISOXImdyZWSvwzkleftLd/zjie6aAxlHrQ887sxV2903AJoA1a9b4Az8/M+LbzbzW1lZaWlqKHUbZaG1t5XzVV97UvqJRfUWj+hIREakcOZM7d28p8HvuBlYBWzLrq4A/5LokU0RERERERCZXkKn4zKzKzGqBOBA3s1ozmyxx/BGwzsyWm9nJwAbgzkLEISIiIiIicqIq1DzrG4BjhBkv/zrzfAOAmZ1uZikzOx3A3f8T2Ag8CrwEtAHXFygOERERERGRE1KhboVwA3DDJK+9RJhEZfS27wLfLcR7i4iIiIiISOFG7kRERERERKSIlNyJiIiIiIhUACV3IiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIiIiIhUACV3IiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIiIiIhUACV3IiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIiIiIhUACV3IiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIjMIDM7xcy2mlmXmbWZ2aVTlP2Sme0ys04z22dmX5rNWEVEpLxVFTsAERGRCncr0AcsAM4BHjKzne6+O0tZAy4HngbOBP7bzPa7+72zFq2IiJQtjdyJiIjMEDObA3wQ+Ia7p9z9CeAB4LJs5d19o7s/5e797v4scD/wjtmLWEREypmSOxERkZmzFOh39+dGbdsJrMi1o5kZcAGQbYRPRERkgrK6LHPHjh3tZtZW7DiymAe0FzuIMqL6ikb1FY3qK5pSrq+FxQ6gAJJAx7htR4GGPPa9gfAl7A8mK2BmVwNXZ1ZTZvbsNGKcaaXcxkqR6isa1Vc0qq9oSrW+Jj0/llVy5+7NxY4hGzPb7u5rih1HuVB9RaP6ikb1FY3q6/iYWSvwzklefhL4LNA4bnsj0JnjuJ8h/PbuAnfvnaycu28CNuUbbzGojUWj+opG9RWN6iuacqyvskruRERESom7t0z1euY3d1VmtsTdn89sXsUUl1qa2ceBrwJ/4u4HChWriIhUPv3mTkREZIa4exdwH3Cjmc0xs3cAHwDuylbezD4G3AT8qbv/7+xFKiIilUDJXWGU9CUxJUj1FY3qKxrVVzSqr5n3KaAOeAW4B7h26DYIZnaBmaVGlf0W0ARsM7NUZvn+rEdcWGpj0ai+olF9RaP6iqbs6svcvdgxiIiIiIiIyHHSyJ2IiIiIiEgFUHInIiIiIiJSAZTczQAzW2JmPWZ2d7FjKVVmVmNmm82szcw6zew3ZvbnxY6rlJjZKWa21cy6MvV0abFjKlVqT9On/kpmk9pbburP8qNzZP7UpqavHPssJXcz41ZgW7GDKHFVwH7C/aFOAjYAW8xsURFjKjW3An3AAuBjwG1mtqK4IZUstafpU38ls0ntLTf1Z/nROTJ/alPTV3Z9lpK7AjOzjwKvAY8UO5ZS5u5d7n6Du7/o7oPu/h/APuC8YsdWCjL3xvog8A13T7n7E8ADwGXFjaw0qT1Nj/ormU1qb/lRf5abzpHRqE1NT7n2WUruCsjMGoEbgfXFjqXcmNkCYClT3Nj3BLMU6Hf350Zt2wnoW8k8qD3lpv5KZpPa2/SpP8tK58jjoDaVWzn3WUruCuubwGZ3P1DsQMqJmSWAHwM/dPc9xY6nRCSBjnHbjgINRYilrKg95U39lcwmtbdpUH82KZ0jp0ltKm9l22cpucuTmbWamU+yPGFm5wDvBv6p2LGWglz1NapcDLiLcN38Z4oWcOlJAY3jtjUCnUWIpWyoPeVH/ZUUks6P0ej8WBA6R06D2lR+yr3Pqip2AOXC3Vumet3MPgcsAl4yMwjfKsXNbLm7r57xAEtMrvoCsFBRmwk/hv4Ld0/PdFxl5DmgysyWuPvzmW2r0CUUk1J7iqQF9VdSIDo/RqPzY0HoHBmR2lQkLZRxn2XuXuwYKoKZ1TP2W6QvEhrGte5+qChBlTgz+z5wDvBud08VO55SY2b3Ag58glBPDwNvd3edvLJQe8qf+iuZTWpv0ak/y03nyGjUpvJX7n2WRu4KxN27ge6hdTNLAT3l0AiKwcwWAtcAvcDvM9+MAFzj7j8uWmCl5VPAHcArwGFCp6KTVhZqT9Gov5LZpPYWjfqzvOkcmSe1qWjKvc/SyJ2IiIiIiEgF0IQqIiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIiIiIhUACV3IiIiIiIiFUDJnYiIiIiISAVQciciIiIiIlIBlNyJiIiIiIhUACV3IiIiIiIiFeD/AbcozCDM0604AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# activation function and its derivate\n",
    "z = np.linspace(-5, 5, 200)\n",
    "\n",
    "plt.figure(figsize=(15, 6))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(z, np.sign(z), \"r-\", linewidth=1, label=\"Step\")\n",
    "plt.plot(z, sigmoid(z), \"g--\", linewidth=2, label=\"Sigmoid\")\n",
    "plt.plot(z, np.tanh(z), \"b-\", linewidth=2, label=\"Tanh\")\n",
    "plt.plot(z, relu(z), \"m-.\", linewidth=2, label=\"ReLU\")\n",
    "plt.grid(True)\n",
    "plt.legend(loc=\"center right\", fontsize=14)\n",
    "plt.title(\"Activation functions\", fontsize=14)\n",
    "plt.axis([-5, 5, -1.2, 1.2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(z, derivative(np.sign, z), \"r-\", linewidth=1, label=\"Step\")\n",
    "plt.plot(0, 0, \"ro\", markersize=5)\n",
    "plt.plot(0, 0, \"rx\", markersize=10)\n",
    "plt.plot(z, derivative(sigmoid, z), \"g--\", linewidth=2, label=\"Sigmoid\")\n",
    "plt.plot(z, derivative(np.tanh, z), \"b-\", linewidth=2, label=\"Tanh\")\n",
    "plt.plot(z, derivative(relu, z), \"m-.\", linewidth=2, label=\"ReLU\")\n",
    "plt.grid(True)\n",
    "#plt.legend(loc=\"center right\", fontsize=14)\n",
    "plt.title(\"Derivatives\", fontsize=14)\n",
    "plt.axis([-5, 5, -0.2, 1.2])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Regression MLPs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Typical regression MLP architecture:</font>**\n",
    "\n",
    "| **Hyperparameter** | **Typical value** |\n",
    "| :--- | :--- |\n",
    "|**input units**| one per input feature|\n",
    "|**hidden layers**| depends on the problem, but typically 1 to 5|\n",
    "|**units per hidden layer**| depends on the problem, but typically 10 to 100|\n",
    "|**output units**| 1 per prediction dimension|\n",
    "|**hidden activation**| ReLU (or SELU)|\n",
    "|**ouput activation**|None, or ReLU/softplus (if positive outputs) or logistic/tanh (if bounded outputs)|\n",
    "|**loss function**|MSE or MAE/Huber (if outliers)|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>How many outputs?</font>**\n",
    "+ **predict a single value**, just need a single output unit: its output is the predicted value.\n",
    "\n",
    "\n",
    "+ **predict multiple values at once (multivariate regression)**, need one unit per output dimension\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='blue'>Which activation function should be used in regression mlps?</font>**\n",
    "\n",
    "+ don't use any activation function for the output units, the outputs are free to **output any range of values**\n",
    "\n",
    "\n",
    "+ use the ReLU or softplus activation function in the output layer to guarantee that the **outputs will be always be positive**\n",
    "\n",
    "\n",
    "$$softplus(z) = log(1+e^z)$$\n",
    "\n",
    "\n",
    "+ use the logistic function or the hyperbolic tangent, and then scale the labels to the appropriate range to guarantee that the **predictions will fall within a given range of values**\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='blue'>Loss function used in regression mlps</font>**\n",
    "\n",
    "typically **the mean squared error (MSE)**, but if you have a lot of outliers in the training set, you may prefer to use **the mean absolute error (MAE)** instead. Alternatively, you can use the **Huber loss**, which is a combination of both.\n",
    "\n",
    "\n",
    "$$ huber(y, \\hat y)= \\begin{cases}\n",
    "\\frac{1}{2}(y-\\hat y)^2, & \\text {if $|y-\\hat y| < \\delta$, typically 1} \\\\ \n",
    "\\delta |y-\\hat y| - \\frac{1}{2}\\delta^2, & \\text{otherwise} \\end{cases}\n",
    "$$\n",
    "\n",
    "> **<font color='red'>Huber loss的优点: </font>** The linear part makes it less sensitive to outliers than the mean squared error, and the quadratic part allows it to converge faster and be more precise than the mean absolute error."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Classification MLPs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Typical classification MLP architecture:</font>**\n",
    "\n",
    "| **Hyperparameter** | **Binary classification** | **Multilabel binary classfication** | **Multiclass classification**|\n",
    "| :-- | : -- | :-- | : -- |\n",
    "|**input units**| one per input feature | one per input feature | one per input feature |\n",
    "|**hidden layers**| depends on the problem (typically 1 to 5)| depends on the problem (typically 1 to 5| depends on the problem (typically 1 to 5|\n",
    "|**units per hidden layer**| depends on the problem (typically 10 to 100)| depends on the problem (typically 10 to 100)| depends on the problem (typically 10 to 100)|\n",
    "|**output units**|1|1 per label| 1 per class|\n",
    "|**output layer activation**|sigmoid|sigmoid| softmax|\n",
    "|**loss function**|cross entropy| cross entropy| cross entropy|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **binary classification**, just **need a single output neuron using the logistic (sigmoid) activation function**: the output will be a number between 0 and 1, which you can interpret as **the estimated probability of the positive class**. The estimated probability of the negative class is equal to one minus that number.\n",
    "\n",
    "\n",
    "+ **multilabel binary classification tasks**, need multiple output units, both using the sigmoid activation function. Note that the output probabilities do not necessarily add up to 1, which lets the model output any combination of labels.\n",
    "\n",
    "\n",
    "+ **multiclass classification**, need to have one output neuron per class, and you should use the softmax activation function for the whole output layer. The softmax function will ensure that all the estimated probabilities are between 0 and 1 and that they add up to 1 (which is required if the classes are exclusive). \n",
    "\n",
    "<br>\n",
    "\n",
    "Regarding the **loss function**, since we are predicting probability distributions, the **cross-entropy loss** (also called the **log loss**, see Chapter 4) is generally a good choice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Implement MLPs with keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow itself now comes bundled with its own Keras implementation, `tf.keras`. It only supports TensorFlow as the backend, but it has the advantage of offering some very useful extra features.\n",
    "\n",
    "<br>\n",
    "\n",
    "![Keras vs tf.keras](./figs/chap10-figs/keras-vs-tf.keras.png)\n",
    "\n",
    "<center>Two implementations of the Keras API: multibackend Keras (left) and tf.keras (right)</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Both [Keras](https://keras.io/) and [Pytorch](https://pytorch.org/) APIs are inspired by [Scikit-Learn](https://pytorch.org/) and [Chainer](https://chainer.org/). Once you know Keras, it is not difficult to switch to PyTorch, if you ever want to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Classifier using the sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**dataset:** Fashion MNIST\n",
    "\n",
    "> a drop-in replacement of MNIST; has the exact same format as MNIST; more diverse, and the problem turns out to be significantly more challenging than MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.408129Z",
     "start_time": "2020-01-03T11:05:05.052563Z"
    }
   },
   "outputs": [],
   "source": [
    "# Load the dataset\n",
    "fashion_mnist = keras.datasets.fashion_mnist\n",
    "(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Keras vs Scikit-Learn when loading MNIST or Fashion MNIST:</font>**\n",
    "+ every image is represented as a 28 × 28 array rather than a 1D array of size 784;\n",
    "\n",
    "\n",
    "+ the pixel intensities are represented as integers (from 0 to 255) rather than floats (from 0.0 to 255.0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.413376Z",
     "start_time": "2020-01-03T11:05:05.409818Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   0,\n",
       "        193, 228, 218, 213, 198, 180, 212, 210, 211, 213, 223, 220, 243,\n",
       "        202,   0],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   1,   3,   0,  12,\n",
       "        219, 220, 212, 218, 192, 169, 227, 208, 218, 224, 212, 226, 197,\n",
       "        209,  52],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   0,   6,   0,  99,\n",
       "        244, 222, 220, 218, 203, 198, 221, 215, 213, 222, 220, 245, 119,\n",
       "        167,  56],\n",
       "       [  0,   0,   0,   0,   0,   0,   0,   0,   0,   4,   0,   0,  55,\n",
       "        236, 228, 230, 228, 240, 232, 213, 218, 223, 234, 217, 217, 209,\n",
       "         92,   0],\n",
       "       [  0,   0,   1,   4,   6,   7,   2,   0,   0,   0,   0,   0, 237,\n",
       "        226, 217, 223, 222, 219, 222, 221, 216, 223, 229, 215, 218, 255,\n",
       "         77,   0]], dtype=uint8)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0][10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Fashion MNIST 是彩色图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.420688Z",
     "start_time": "2020-01-03T11:05:05.417140Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((60000, 28, 28), (10000, 28, 28))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape, x_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.426779Z",
     "start_time": "2020-01-03T11:05:05.423384Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('uint8')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.431902Z",
     "start_time": "2020-01-03T11:05:05.428514Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 0, 0, ..., 3, 0, 5], dtype=uint8)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.685901Z",
     "start_time": "2020-01-03T11:05:05.433821Z"
    }
   },
   "outputs": [],
   "source": [
    "# train the NN using Gradient Descent, so we must scale the input features\n",
    "x_train = x_train / 255.0  # for simplicity\n",
    "x_val, x_train = x_train[:5000], x_train[5000:]\n",
    "y_val, y_train = y_train[:5000], y_train[5000:]\n",
    "\n",
    "x_test = x_test / 255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.689836Z",
     "start_time": "2020-01-03T11:05:05.687444Z"
    }
   },
   "outputs": [],
   "source": [
    "# class names\n",
    "class_names = [\n",
    "    \"T-shirt/top\", \"Trouser\", \"Pullover\", \"Dress\", \"Coat\", \"Sandal\", \"Shirt\",\n",
    "    \"Sneaker\", \"Bag\", \"Ankle boot\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.694671Z",
     "start_time": "2020-01-03T11:05:05.691412Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Coat'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class_names[y_train[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:05.765070Z",
     "start_time": "2020-01-03T11:05:05.696442Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAKN0lEQVR4nO3d20rWWx/F8WllWeYuUwtCSsIMCkqKiCDIrqOjovPooDvoIjrpCjrrHhZC1EHuyHaWFaXltkxts87eo/UfI3xe1zMe1vdzOpg+mxz9wR9zzqbfv38XAHl21PsNAPhnlBMIRTmBUJQTCEU5gVC7TM6fcvF/4yYDTU1N/9I7ifOPH5wnJxCKcgKhKCcQinICoSgnEIpyAqEoJxDKzTmxDcbGxiqzBw8eyLWjo6My//nzp8wPHTok85MnT1ZmV65ckWsvXLgg8//wHHNLeHICoSgnEIpyAqEoJxCKcgKhKCcQinICoZhzbsHExITMr1+/LvNHjx5VZj9+/JBrd+3S/2Q7duj/b13+/fv3La8dHByU+e3bt2V+48YNmf/X8OQEQlFOIBTlBEJRTiAU5QRCUU4gVJM5rrBhj8b89etXZeZGAk5fX5/M5+fnZd7R0VGZueMjm5ubZe5GMTt37pS523KmLCwsyPzIkSMyf/v27ZZfu1Z1PraTozGBRkI5gVCUEwhFOYFQlBMIRTmBUJQTCNWwW8bUHLOU2maZi4uLMndzzpaWFpnv27evMhsaGpJr3XY1N49z713NOd+8eSPXdnZ2yrytrU3mjx8/rsyGh4flWmc7f1+2S947AlBKoZxALMoJhKKcQCjKCYSinEAoygmEit3PuZ1zqYsXL8p8ZmZG5u69uVnj0tJSZaau4CullOXlZZm/ePFC5m4Ge+LEicrMzSndfkx17GYppWxsbFRm7t97bm5O5o7bx+r2wdaI/ZxAI6GcQCjKCYSinEAoygmEopxAKMoJhIrdz1nrOaF37typzJ4/fy7X9vf3y9ydDetmiWre52aFp06dkrmaoZbi91yq9/b69Wu51hkYGJC5Os/35cuXcu3Nmzdlfu/ePZlv8xxzS3hyAqEoJxCKcgKhKCcQinICoSgnECp2y1itLl++XJmtr6/LtW6Ms7a2JvM9e/bIfO/evZXZysqKXLt//36Zt7a2ytxtKVOvf+zYMbn28OHDMnff29evX7f0vkrx3/lff/0l8zpjyxjQSCgnEIpyAqEoJxCKcgKhKCcQinICoWK3jDnuKMMvX75UZmrOWEop7e3tMldX+JWij3h0uZvXuRltrcd2njt3rjJzM1Z3daLb9tXd3V2Z7dqlf1Xn5+dl7q4vdNsE64EnJxCKcgKhKCcQinICoSgnEIpyAqEoJxCqYeec7po+tf/Pzes2Nzdl7mZublapZrTu2E33s3t7e2XuZrBqT+WnT5/k2t27d8u8q6tL5up7cfNdd72gm4My5wTwxygnEIpyAqEoJxCKcgKhKCcQinICoRp2zun2Birfvn2TuZr1leLnpG4WqWaZ7mxXtxd1dXVV5u6zqxmum2O6a/Tce1teXq7M3Hm8bn/v+Pi4zIeHh2VeDzw5gVCUEwhFOYFQlBMIRTmBUJQTCEU5gVANO+d0c6sdO6r/31lYWJBr3717J/PTp0/L3M371CzT7bd059K2tbXJ3O0XVe/NzRLdfNftufz48WNldvDgQbnWfefufs5r167JvB54cgKhKCcQinICoSgnEIpyAqEoJxCqYUcps7OzMlcjB/dn99+/f8vcjQzcljN19KZ7b24U4o6QVCOmUkppbm6WueLemxulqO/NjYjctYxTU1MyT8STEwhFOYFQlBMIRTmBUJQTCEU5gVCUEwjVsHPOyclJmatZZVNTU02v7WaRbmuVmiW6WWCt3JYzNYN1Vx+6z+3WqyNH3WzZHds5NjYm80Q8OYFQlBMIRTmBUJQTCEU5gVCUEwhFOYFQDTvnfPr0qczVLFLN8v6Eu0bP7ZmsZQbrZoVuL2otM143I3V5S0uLzNWxoO5nO3NzczJ/9uyZzAcHB2t6/a3gyQmEopxAKMoJhKKcQCjKCYSinEAoygmEatg554cPH2R+4MCBysztmezs7JS5m7m5vYVqnudmgW5G686tddSc1O3XdK/tZqzq7Fn3ud2ZuY67UpI5J4D/oZxAKMoJhKKcQCjKCYSinEAoygmEatg5p9szqeZibh7nzkh1s0h3rq2a97n9mG6e5+7XdLNG9fPdXtJaPrd7bXfnqZstOx0dHTWt3w48OYFQlBMIRTmBUJQTCEU5gVCUEwjVsKMU92d59af1xcVFubanp0fmbqSwuroq871791Zma2trcq373K2trTJ3R0TW8tpqy1cppSwsLMj8+PHjldnU1JRc60ZrXV1dMndHY46MjMh8O/DkBEJRTiAU5QRCUU4gFOUEQlFOIBTlBELFzjndNXtue9L+/fsrs8+fP8u1Bw8elLnjZm7btbYUf+yn25Kmtpy5ozHdVjuXnz9/vjJ79eqVXOu2fLnZ9PT0tMzrgScnEIpyAqEoJxCKcgKhKCcQinICoSgnECp2zumOQnS5OmbR7Xns7e2V+fv372Wurh8spZSlpSWZK25PZa3r1ffmZrDuyNDZ2VmZqxlse3u7XDszMyNzd22ju1KyHnhyAqEoJxCKcgKhKCcQinICoSgnEIpyAqFi55zubFl19mspeu+hm3kNDAzIfHl5WeZuHqhy994ct2fSUd+bO5fWzTnb2tpkrv5N3Wu7ubebk6r9v/XCkxMIRTmBUJQTCEU5gVCUEwhFOYFQsaMUd1WdGxmo7UduFOKOl1THR5ZSyubmpsxrobZ0leKPDHXfmzqS1I2I3HGmtVyd6I7ldNzozX1v9cCTEwhFOYFQlBMIRTmBUJQTCEU5gVCUEwgVO+d0M7Pdu3fLXB0B6bYHdXd3y3xiYkLmtcxg3RV97nM77mhMNcOtdcZay/x3aGhI5g8fPpR5T0+PzN1nqweenEAoygmEopxAKMoJhKKcQCjKCYSinECo2DnnysqKzN0xjGqed/To0S2vLaWUz58/y9wdran2i7q9pG6G+uXLF5nPz8/LXB0h6eaYtcyeS9HX8F27dk2udXNOtwfX/T7VA09OIBTlBEJRTiAU5QRCUU4gFOUEQlFOIFTsnNNd6dbR0SFzde7tyMiIXHvo0CGZu6vs3DV+6+vrlZmbxzlufWdnp8zVflK3H9Pl7ho/NQe9evWqXOu4c2/d71s98OQEQlFOIBTlBEJRTiAU5QRCUU4gFOUEQsXOOd28zt31qOZ1Z8+elWtHR0dl/uTJE5m7M1bX1tYqM7fn0c1Ya51F1nI/58bGxpZ/din6fs6+vj651p1L62bPzDkB/DHKCYSinEAoygmEopxAKMoJhIodpbg/+bsjJJXp6WmZ379/X+b9/f0yX1hYkLn6s737XO7IUDeKccd2qpGDGnWU4rejufHYpUuXZK64MY4aX5VSyuTk5JZfe7vw5ARCUU4gFOUEQlFOIBTlBEJRTiAU5QRCxc45z5w5I/Ph4WGZj4+PV2Zuu5mbx929e1fm+PfdunVL5m67m9tGWA88OYFQlBMIRTmBUJQTCEU5gVCUEwhFOYFQTeoISQD1w5MTCEU5gVCUEwhFOYFQlBMIRTmBUH8DscHqopQEqFAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot an image\n",
    "plt.imshow(x_train[0], cmap='binary')\n",
    "plt.axis('off')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.780769Z",
     "start_time": "2020-01-03T11:05:05.767002Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqoAAAEjCAYAAAD60iPnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOydeZhdRbX2f9Vzkk7SIekkhISQEBAwzLMGAQFlHhxRwMtVUJBZHgW9qKAIfFxFrnBFkNEhjIIGUC4oMoPMhACBQAYSMnaSTnoeTtf3R+23Tp2hO0mPp2O9z9NPd589nF1rr6pa612rVhlrLRERERERERERERGFhqKBfoCIiIiIiIiIiIiIfIiGakREREREREREREEiGqoREREREREREREFiWioRkREREREREREFCSioRoREREREREREVGQiIZqREREREREREREQSIaqgMEY8wzxphTOzk21RhT38+PtNnAGLPQGHPoQD9Hd2CMscaYaZt6bAP3PNUY80zPn67/EeWRiSiPiIiIfzf0q6FqjPmqMeZlY0y9MWaZMeZvxpgZPbznE8aY03rrGTfwXfXBT4cxpin4/6Te+h5r7XxrbeUGniWvoWuMOcAY85QxpiSZuLbprefqDowxM4wxzxlj1hlj1hhjnjXG7D2Qz9QfSPRyrTGmfKCfpa9gjDnIGLNkI8+N8sg8N8qjb75zUM8xvY1/Z3kkhEWTMabOGFObzENnGGMiQZdgsOhHv70wY8x3gGuBK4BxwNbAr4Hj+usZegprbaV+gA+BY4LP/tgfz2CMKdpARzsK+Gt/PMuGYIwZATwEXAdsAWwFXAa0DORzbSyMMSXdvG4b4ADAAsf24iMNSkR5ZCLKo2+wOcwxvYkoD8DN0cOBycBVwEXALflONMYU9+eDDTQGlX5Ya/v8BxgJ1ANf7OR4OU5gS5Ofa4Hy5NgonLGzClib/D0xOfYzIAU0J/e/vj/ak3z3QuDQDZwzFJgJrAZqgReBMcmxZ3BG23NAHfAIsEVybJp7Nf4+zwA/BZ4HmoC7s9p9bXDubGCX5L4WaEjO+Xxy/Azg/eSZ/gxsmXxekpx/DrAAqMF17KIeyGgvoLaTY6cm7fp58l4XAEdk6cwtwDLgI+ByoDg5ti3weNKGGuCPQFW+dwPsmNz7K8n/E4A/Jfq0ADg3uO5S4D7gD8B64LRutvtHwLPANcBDWcduB/4XeDh57/8Ctg2OW2Ba8vcMYDFwUJ5j5YnsPgRWAL8BhnQh62eB64F1wFzgkOD4BGAWsCbRjdM31DeBYYkudiT6VQ9MiPKI8thUefTGD5vhHBPl0WMZLCRrjgb2SXRyetLXbsAROw3AoV31G2BMIovapC88TTI/4gzgj5I++27YfwrxZ7DpR38J5XCgHSjp5PhPgBeAsUA1zsj6aXJsNPB5nNE3HLgX+HNw7RN006DoYZtyOkGec87CGYNDgGKc4VaZHHsGmAdsl7TtaeDy5Fg+Q3UhzugqxRmVzwCnZn3fJODD5G8ZntsExz8DrAR2Aypw3tPjWef/PVHEybhJ6dTuyCe55wicMXkHcAQwKjh2KtAGnJ7I5sykQ5jk+APAjbgJbyzOyP9WIJ/Dks5UDTxFprG+EDfo7IEbcI5OPi8CXsEZCmXAVGA+8Nnk+KXJMx2fnJt3Yt+Idr8PfBvYM7nfuODY7YlM9klk/kfgruC4Tdp3OM4I2Sf7WPL3L3HGwxa4fvEgcGUnz3Mqrv9dkOjPl3EGiRyjpxJdqEh0YxXw6Y3omwcBS6I8ojx6Io/e+GEznGOiPHosg4XkmaNxc8KZSV9bB3wSN95XdNVvgCtxhmtp8nMAYICPJX1xQnLeNgTOZSH+DDb96C+hnAQs7+L4B8CRwf+fBRZ2cu5uwNq+FMpGtilvJ8g655s4g3LnPMeeAS4O/j+XhF0hv6H6ozzXn5r12beAG5O/8xmqdwBXBP+PwHk/E4PzD816pv/roZx2TAaEJUnHmIULM5wKvB+cNzT5/vHJ8RYCQxH4CvDPTr7jeOC1rHdzWfKdBwWf70tiyAeffR+4Lfn7UuCpHrZ3Bs74EHM+F7ggOH47cHPw/5HA3OB/mzzTImB61r1lpBgcAxAybfsDCzp5plMJnIDksxeBU3DOTQoYHhy7Erg9+bvTvslGGCJRHlEePelPm9DvNrs5JsqjxzJYSH5D9QXgv5K+9rvg8y77Dc54+wuJMxicMw1HAB0KlA50uzdH/eivHNXVwJgucv4m4AZeYVHyGcaYocaYG40xi4wx63HefVWh5ZMYY4qzFltNwHWEvwP3GGM+MsZclSWD5cHfjUBXC6gWb8RjHEnX+akZcrbWrsdR91t18j3+PXQX1tp3rLWnWmsn4sItE3BhBAjab61tTP6sxLG5pcCyJAm+FseujgUwxowzxtyVyHQ9LlQ/JuurzwCes9Y+EXw2GZigeyb3/QHOMBY2Rs5d4T+AR621Ncn/M5PPQmzovZ8P3GOtndPJd1TjDPtXgnY8knzeGT6yySiSQO92ArDGWluXdUw60Wnf3EhEeWQiyqNvsNnPMZuIKI/OsRUudA+Z4/2G+s1/46Ihjxpj5htjLgaw1r6P65OXAiuTuWkg+sCmYFDpR38Zqs/jGLLjOzm+FGdECFsnnwFciKPW97XWjgA+lXxukt/h4DpgsNambLDYylq71Frbaq291Fq7I45JOQHnyXTrK7r63xhTlnzH3zs5H7LkbIwZjgvzfxScMyn4O3wPPYa1di7OeJ++gVMX4/RljLW2KvkZYa39eHL8Clz7dk504mTS+iCcAWxtjPll1n0XBPesstYOt9YeGT5m91oHxpghwJeAA40xy40xy3Hh1F2NMbtuwq2+CBxvjDmvk+M1uPy/jwftGGm7rhSxlTEmlJHe7VJgi0QXwmPSia76ZpeyivLIRJRHn2Kzn2M2EVEeeWBcxZmtcBFJyGxLl/3GWltnrb3QWjsVtwjyO8aYQ5JjM621M3AytcD/66cmdReDSj/6xVC11q7D5QX+rzHm+MQiLzXGHGGMuRq4E7jEGFNtjBmTnPuH5PLhOOWpNcZsAfw46/YrcLmGBQdjzKeNMdOTVfrrcSG/jl66fXa7DwResdY2gDOccV5TeM6dwDeMMbsYVxbnSuBpa21YQuZ7xpgqY8zWuND/3d19QGPMDsaYC40xE5P/J+FC+C90dZ21dhnwKPALY8yIpNLBtsaYA5NThuMStdcZY7YCvpvnNnW4PJxPGWOuSj57EagzxlxkjBmSsODTTe+VyzoeFybdCRcO2Q2X+vA08LVNuM9S4BDgPGPMmdkHrbUdwG+BXxpjxDJvZYz5bBf3HAucm/S7LybP9Vdr7WJc/tGVxpgKY8wuwDdI97+u+uYKYLQxZmQn3xnlkYkojz7Cv+sc0xmiPDKRzCNHA3cBf7DWvpl9zob6jTHmaGPMtMShW4fryx3GmI8lc305bhGRFhEWLAadfvRmHsGGfnBs4su4PJDluJWtn8AlMf8Kt8J7WfJ3RXLNBFzOQz3wHi4P05IkAeNySN7DhbB/1Y9tWciGc1RPTp6tPmnvtaRXrmfkmAKnAU8kf+fLUT01694zcIuxanGrh68Fzs8656zke2uBzwWffYALfcwCtko+D1f9L8QZuVfTs1X/WwH34NiXhuT3jbjc2FOBZ7LOt6QXg4zErchcghsUXgNOTI59HLcoqh54HefhLQnu498NLin+DdKJ4BNwnXB5ojMvBOdeihvEutveR4Bf5Pn8S8n3leAY5cuDYwdlPXsogym4kMtpeY5V4Jjl+Tgn6B2CCgZZ338qmau63wM+ExyfiFu5uSbRjTOCY532zeT4raSrWkyI8ojy2Fh59MUPm9EcE+XR47YvxBlUdYleP4+b/zQHZ/S1DfUbXPRjYSLLJcAPk893ISFBkj7yUH/o+r+TfmiFdcQghzHmPdzq9ve6eX0JjvGdYq1d2JvPFhERERERERHRHcQdGjYDGGMqgFu6a6RGRERERERERBQiIqMaAURGNSIiIiIiIqLwEA3ViIiIiIiIiIiIgkQM/UdERERERERERBQkoqEaERERERERERFRkOhsV4KNxYDnDVhryaxTvUno9oWdYJPkMWeO21imoaGBd955B4AbbrgBgJkzZwKw7bbbdnmPZ55xdYsvv/xyAH76059SXOw2iJgyZQoAo0aN2thHGlB5FCCiPDIR5ZGJKI9MFIQ8wnS27LnhyCOPpLLS7XvQ3t4OwGc/+1m+9a1vZZzX0eHKYBYV9YjLGVB5dCWHf/7znwCcddZZlJeXA9Dc3Oyve/DBBwHYbrvtMq7r6Ojw9+rGvFsQ+hHiH//4B4Cff3fccUemTZuWcU5tbS21tbUA3HfffQAcdNBBABx++OEMGzasu1/f2/KAbupIvnepPtHR0cFxxx0HwJo1bkOvRx55hFWrVgHw2GOPbdJ9N4C8F/Q0R7XXBlYZbX/605/417/+BUAqlQJg/Pjx7LjjjgAcfPDBAOy777698bUD0nH+8AdXN7e+vh6A6upqPvaxjwHw/e9/H4AnnngCgIkTJ/KJT3wCgCFDhvhj77//PgAtLS2AG2wBrr32WmbPng3AihUrAJg8eTLHHnvsxjxawQ0kA4woj0xEeWQiyiMTBWGY5Zscv/tdtyfIjTfe6A0RTbplZWXcfvvtAH6s7SUUnH786U9/AuALX/gCALvuuitr164F8AZXeXk5b7/9NgCzZs0C0vNLxsNsujEyoPJoaGgA4OKLL2bu3LlAeg7eZpttADffSj9kiH3wwQfeqREWLlzo/5bj87e//W0TH3/gDdUQNTVuV+evfOUrADz77LOA6x9y3PSuOzo6PCGmz37zm98A8OUvfznn3qlUyp+/ARSWoaqO8I1vfAOAl19+GXBebkmJI3rlzRYVFXlvT59tv/32AFx44YWcdtpp3X2Mfu84Dz30EI8//jgAJ598MgBLly6lqqoKwBus8mivueYa38HUgd58803GjHFb23/ve98D4Ktf/SoAL730kpfV0KFDAbjrrrs4/PDDgfwDToCCG1gHGFEemYjyyESURyYKRh7nned2l33xxReB9CS8xRZbsHix295dY+7w4cNpamoCnKECcO655wKOMesBu9rv8sgXYbzhhhu49957AXjvPVfBcPhwtyPuMccc441z2QL33nsvr732GpBmnSdNcjtrn3DCCZxzzjkZ9+/o6NhY2Qyofui5a2tr/fwpyGCtqKjwhqf0o6SkxBNDgmyU+vp6f62M/3yGWifoV0M1n2Px3HPPAc6OeP311wEYMWIEAGPHjgVg5cqV/nyx74BnmcePHw/g+9WoUaP48Y/dZlXdsM3yyiTmqEZERERERERERBQk+oVRzeeRjhs3Dkh7uiNHuu2grbWUlpYCaW+uuLjYpwEICldMnDjRW/J5H7Dr8ES/e3jXX389H330EQA77bQTAFtvvbU/XlFRAaQ9+46ODp8Dsn79egD22WcfqqurAccQAMyfPx+AtrY2L+8lS5b4Y2JXzz///K4er2AYkQJBlEcmojwyEeWRiYKQxw033MDVV18NwPTp04H0fLFmzRrPGDU2NgIu5L3lllsCsHz58oxjYpm6iX6XR8hu/va3vwVc2oMYUc2pitItXrzYzwmaQ2bNmsVWW20FpFlFzb8fffQRZ511FgBXXnnlpj7/gOiH1nFcdtllgGMClWOaHdIXUwrpkH5zc7O3UyQP6UlJSYm/RqzrzTffvMG1JQkGLPR/2223AWmZdHR0eLtL9oNskeXLlzN58mQgrQdz5szxTKr6U1tbG+BsLdkqWiejyAZ0zyaLjGpERERERERERERBoqer/jeIfPkrtbW1nlGV1S7Gb4cddvD5q7K4x40b5y35Dz/8EMjML3r11VcB2GOPPTK+F3q8arPX8cYbb/g81Lq6OsB5bFooVVZWBqS9sxEjRnjvT8nI7e3trFu3DnD5rZCWI6Q9GyWDV1RU+NykiM0LYU6adMJaS2trK5DOG9L/bW1tPs9IfWjs2LG+f2Xnbq1atcpHAHbbbbe+bEpERK/gySef9OOhxsLRo0cDbnyV/qsaSllZmdd/jcMaV1955RX23HPP/nv4HiKc7+655x7A5RBq7tDiW/0/efJkz7xqztx+++39eCG5aF7acsstefLJJ/u6Gb2KGTNmAOk1Hk888UQOQyr2NIRyT5ubm70+iYFVfuaYMWP8+hqxrJdeeim///3v+6QtvYUf/vCHQNruSqVSXnfEeGqtS3V1tZeTFiBOnjzZR3OlI9Ipa62P+Gr+eemll9h77727/byFZcVFREREREREREREJOgzRjUfo7n//vsDsGjRopxyB2L/hg4d6o998MEHgGNRxUKqjIQs9ZUrV3LYYYdlfNeqVav839nW/kCjoqLCr6JTm5YtW+ZzOcSyytOprKz0n0kuY8eOzWmPPOWWlhbPrOmcpUuX+mt7UN+s4NBVW8JjYlUkg7Kyss2i/ZDZ9v/8z/8EYMGCBf4zsQKSwfLly713rGurq6s9Q6BcpL322guAo48+mj/+8Y8A3HrrrX3Wjk1F9rvvSQRlc+oT/YVFixYB8Oc//5mzzz4bKJwxdv369Z4N0lgoRrWysjJjrgEXlZP+67euf/HFFwcVowqwevVqIB2Vq6io8P1DbHLIfml1tyofFBUVeYZR86d+FxUV+QiL8n43oU73gEDPrtzaiy++2Ocun3HGGUA6kiSmFTLzVTVmSi+Un7lw4UIfaZLu/PznP++bhvQSWltbfSkyjXmpVMrnLWf341Qq5eUim2z8+PE+j1u6JbS1tfnIhO7/wAMPeEa1O+Nsnxmq4cNcdNFFQLrjbL311p5CF6UuBVi8eLFXIg0yVVVV/nhYvwxg6tSpfiGWksG/+c1vctNNNwGFM3gqPGCt9ca2aPSpU6f6tqrtwtKlS72MFI556623fHkRpVCokzQ2NvoBWIo0adIkb7Coxuquu+7ayy3sf4Q6ptQGlS37xS9+AbjUiW9+85v9/3D9hLa2Np8Er9rCb7/9th8o9Fv9YOedd/YDtxyghoYGPygrtUaTeWNjo69BWUjIHuxCQ/Wpp54C0qXbtttuO99m9T2Vhttpp51y7lVfX+/TjzTeaGL61Kc+1cst6T/ImS0vL+fRRx8F4KSTTgLS9TU31L7f/e53AL580fnnn8/TTz8NpIugDzQUtoe0o6Z3PHHiRN9f1CfKysr84g+NmcKzzz7LmWee2efP3JvQAjC979LSUj9/yuBSKL+lpSWn5OOiRYv8cclD/cda6++lueTAAw/s2wb1EHrP4dwqAzUM4UNm+SXZKCUlJf5z6ZPOr6+v59RTTwXSqQWakwsVzz33nNd36UNjY6MfG6U3ckgqKiq8rSKHr6KiIiOdDMiwa7JTRh555BGuuOKKbj9zDP1HREREREREREQUJPqFUX3++ecBxxzqmLwVhd1k2RcXF/tjCrl88MEH3vPRDlUqJ9LU1OQpayX3vvnmm33Uqu5DBfzHjx/vPXqxgOvWrfMlqsJFUeDKdsnjVYJydXU1y5YtA/C7eIkhXbNmjWeWFZ6bOnWql5d2z9gcGNUQ2kFFIQrJ7N133+XXv/41kJbfdtttx5FHHgmk01HkKQ42hOXlVBKloqIiJ1IRhnjECshzLikp8d6xdFO7mo0ePdr3uUJCV+F6PbtY0CFDhniWTTu+qeTb1KlTueuuuwD41a9+BbgdVqQrYgvEluy///5eRoMNYYhO6Udi0k8//XTA6ZDGU+kEpOWtMke67rHHHuP444/v4yffOIj9U6QA0gyY2tTU1JRT6nDJkiW+nwh6x/Pmzeuz5+0riO3WOwvTYSQPMYRFRUVeHooetLW1eRZScpE8jDG+z73wwgtA4TOq+SBGVOyzxorKysqMRVTgZKbxVHaI7JX6+vpB1/4HH3wwo2+D6+uyEcLoNjg9kg4pWgtpnRA7K/sL0mysrgvT0bqDyKhGREREREREREQUJPq8PFUqlfL5DcqXGzFihLfMZdnrd3l5uWd6wgVXWtyhJG95NvPnz/dsmLz8mpoan2sXFtMfSCiR+KWXXvL5b9rW7jOf+Yz3RpQIv/vuuwOOYQ49O3AejjYEkEzl3Q4dOtQztffffz8AX//6133y9D777NNXTRwwrF692stUhYy1BWJ5ebnP/xVLVlNT4xlY5TKLYf7c5z7nZT/YINbPGNNpjtnQoUP9MeWhtre3e883eyvJnuQV9SWymdQwF10bXeiccEGdFoCIBfnLX/7iy9ppTJk0aZLvj5KL2IPByqZCeqyAdBF0MUMaa59//nkvN42nbW1tfsHMxz/+cSCdyzx+/PickmYDBbGfLS0tOfqheaOkpMSPp6HOZDNmg/l9q1xj9kJDSOdd6lgoA31mrfXjhdqv/lNWVub1QnPOYEG4sFp6LEZV7728vNxH5cSyQrov6LfOV17mYILGwBClpaU8++yzQJoZ1RzQ3Nzs51BFK4YPH+5tNsngjTfeAFyEWHOtWP0RI0Z4OzBkXjcWfW6oLlq0yDdOg0VbW5t/4QpBSIna29v9Z1qN2Nra6kM3Cllpsh01apS/VgZuuJNEoRiqRx99tP8tRfnrX/8KuPD9pz/9aSBtWGgnh5133tm3Xcb92rVr/T008GgSGjdunE8H0CR0ySWXFPzKzGxszEpsvffKykovP32migpXXXWV3zlDC9DGjBnjw+Q6T6sZL730Uv7yl7/0alv6Evl2lquoqPAGVj75aZAN6+bpPA0+OmewINQXTTCaZNetW+fTYLSI6phjjgFc6Fp9SItJysrKcgwUGfCDEfkqnygdS+1U+8aOHes/ky40Nzf7sUROgEJ7SqEpBOiZUqmU1wGNnZpThg0b5kOVGjNTqZSfiLUQRueIHBhMkBMhWGt9jU+1LxwXpP8yaEtLS31/ku5ovglrskregwXhTpdKZ9Jnshfa29u97oT9JXsHK+lFmPZSaBWGOkNDQ0PO7p8NDQ3eRlDbZLeNHj3az5dKq2pra8swQiFtfy1fvtzLUwZrY2Mjb731FgAHHHDAJj9zDP1HREREREREREQUJPqcUdXCHkizhQ0NDZ5dlacry76pqcl7urLsGxsbPQMrJlVeS319vfd+Fd5OpVLeeg93qyoUyHPRwqazzz7be7PyeN955x3A1bPU+fpswoQJnj7/xz/+AaTZwnnz5nlv8fLLL8/4vsECa62XR1jvDzKZADHSd999Ny+99BLgFsJAOrQ5bNgwrxfy8g488EDPEEl3pI9iWAcLwlp/Ibsq71asqfbubm5u9l6xZNDe3u77le6nY4MFoV4o8qAE/m222cbr07vvvgukF3bW1dX5vqMFih0dHT6ao5BwIS4o21hks+rz5s3zKUbSC7EnRUVFOXWHm5qaPLsalrjS+YUC9e/ly5f7msIaY8UYr1u3zrdB40JFRQVz5swB4IQTTgBcCZ/wnoMJGjPDufXzn/88gC/dpndcXl7u9UNj37vvvuvHAb33/fbbD3CRJ73zsJTTYEA4Vkof9JnC3O3t7V5+YuNLSkq8LaLzpS/5WNdCZ1QXLFiQMyY0Njb6d61a2mKIV61a5W032RItLS0+lC85qa9UVlb69BqNIe3t7X5Hs8ioRkREREREREREbDboc0b1rbfe8h6Y8lw++ugjdt55ZyDtfcjDaW1t9Va4mI729nZ/XFa+vLmQIVJSf3Fxsc/BOuWUU/qwdZuOMB9QbS8pKfHMnhgcMVwvvPACX/3qV4E0Iz1//nzvESspXCzB/PnzvYzCUleDIX8mZE2znzP0AKVHWhj22GOPeXbk0ksvBdJsyciRI33uoTB//nyvW2JSdf6aNWsKbiFeVwjlIh2or69nu+22A9IMgI6tWrXKRyzk9ZaUlPj7ZEczBgtCOWhxpXQozIl/5JFHAHj44YcB13bpgtrc3t7u76c+VygLhrqDbNbzz3/+s89Rkw6o74XjU7ioSuOv9Ei5quEGIwONcJHITjvtBKTLkGlOCQvci0GqrKz0x8UwaxOZhQsXesZIY0ShQ0yXxv+5c+dyzz33APhIo3JWwz3uFYHQ3ANp5lW7On3pS1/y7ONgy2MPGU+VWRO0uHLFihUZdoQgG0PjiGQbLqYKGdtCxpIlS/yzakHtueeeyx133JHxmcbF8vJyrwc6Bum2a5yQPhx//PE+QqONjkpLS/2C5+4gMqoREREREREREREFiT53AZYsWZI331CMqLxVWexhwf8wzy57Fa7OaW5u9veQBzB06FDmzp3bZ23qLSgfdeTIkV5GyvPQZgevv/461113HZDOIZozZ473muUZiiVIpVLeIxY7EB4vFHS1qr+trc0zWWI2xDiXlZX5slv/93//B7gV3GKdtR+9csyqqqo84yP2uaamxjPMuq/k/7vf/c5vK9nbjGp395Rvb2/P8dbVX8J+ceONNwKu/IdWror5CvOJ1HbdQ98BuWVYwi1a+xMbklVYuq6z88JqGWrPiSeeCKRZo6efftrrmlavFhcX++LXykULS9UUIjqTV0dHR07fnzlzpmeNJKOu5AjpfqJtUsXErlq1yue0DTSUYwzpyIByccO96kMWEdIygHRVGTGyr732mt8sQlGKQkZzc7OfN8X6lZaW+rlR8tCxlpYW37/Dyjuaq/VZvn3aszdJGEzQeKdt3bWFsMbLEOHYq9x9MdKDjVUG15eVg69IyS9/+Uu/GYqYT42HoT5Ib1auXOnvoTlaUez999/f66Dm6LAaUXfQ54bqO++8k3cQze4I2aGoEB0dHX5SldLoupKSkpyFWWVlZX6CKWTopVdVVfm/Ra2LOleKBKTDNkcccYQfiLXPtmQ8evRorziFHIoIB9Ns/Whra/N6oI6kEMKcOXM4++yzM+4xe/Zs/v73vwPpxQJK2DbGeENVv/faay9v6MiA04Bz2GGHFVzIP3yP+QzUmTNnAi6kC3Dcccd5R03tk7FSVFTkdU2hzKampoydZ8LrPvzwQ1+apFCQSqW8zuTTcZUXUwrA4sWLfZhX5VI0VhQXF/vJWzqRSqX8ooEwDFrI6MzADI1U1Rj+4IMP/I5s0if1t3CnIiGs5XzYYYcBaSf79ddfLxhDVUZm+Lf0OBwTNfdINmnpmu8AACAASURBVOGCOtXVVC3loqIiX7pqMODDDz/MmStlUEDaENthhx0Ap/PS+9D5k97LQdH7rq6uzinVtGrVKt+vChnhWKGxQXLQGBc6MeECKn0eEmQwuBadyiELyaywxq5S5DQvqPxYR0eHH1/U7rKyMn9cC+Zl2O62225+TPjud7/rz5fD1x0UFs0WERERERERERERkaDPKbc333wzY1GDoHBbWHQcnDcnzycfE5vt/VVUVHi2JPQOxExql6bsBTUDhXzMx5gxY3J2QlFooaWlxYchJau33norp1yMZFZaWprXu93UcHNfI1zoFe5DD85rVfK6mHGVV7n++ut9Mr9KBi1dupRXXnkFcN4cpMtoVFdXe13Qwory8nLPMkyZMgVIy2/8+PE5e8X3FsJ3kF2oP9T1cGGLfmfvkCTcdttt/PjHPwYc0w6uvEz2Jhhik9vb23PC5pBZ7BvS7+Ktt94aEEY1fLbshYDhIgeNKSpV9vDDD/sQsNoybtw47/3feeedQDpNaOnSpb6/iElobW31TJLkrsUoKntU6ND4UFZW5v+Wnuy2225evorchH0wu9h7SUmJjzho9xmdP3PmTI477rg+b8/GQMx4uPONWCS94zA6Jz0KF2+KPdX4sXLlykEV3l2+fLl/f2rnxIkTfcRJx8Sohex5WM5K8pDu3HXXXYBjHrU5inTgww8/HBSMagiVdRRDqAjljBkz/Dk6FjLI2alRd999N9OnTwcKe6EypMtbjh071o/3YcRFrLmitJJJUVGR140w6qnP1Gc01yxYsCCnBFV5eblnnzUfb4rOREY1IiIiIiIiIiKiINHnjOqyZcu8hxvmdciil2cnz62iosJ7grLQgRwvX8fCHERdF3pAyussFEY1H4YNG+Zlo/aJ8bHW+tyOkIWWZ5OdQ9fS0tKtvXT7GmKolCMl1nLFihWe+VJO0P777+8ZsquuugpIe7rf//73PTug4v4rVqzwOWW77LILkJZVWVmZz53RZyFjoP3BdU5HR4dn33bddddea3+IVCrVZfmtrthvRQhuueUWAGbNmuVLqyxcuBDILC2VzVavWrUqJ78zZJSkd/r/tdde49hjj93UJvYYocefLau6ujoeeOABILeg+4gRIzxLLs/9/fff90y7vH8tGBg9erRfWCS5lZeXe8ZfrIJKH+V7d4UEjX3huKDthRWJGDlypJdDNoMeMqrSnbCwu/qGcvtWrVpVMKXv9L5HjBjh2fHsRYJh1E26XlZW5nNaNQ+FeZ2DiVFdsWJFDls2fPhwP56KKc6O2mRD95D+a3HqtGnTPKMqhLnBgwX33XcfkNYL9f05c+b4fqKoZgjlqoalvwYL9KxhZDtkNbVJjqIQmhNTqZSfO9WvGhoafMRR/UNjyYsvvsjXvva1jO/u6Ojw44TywJXvvjHoc0M13Okk3NVELzoMv4DrOBpcwgUe2QNOeF24Xy1kDtJKoi9kpFIpPxlkLxbr6OjwA6rkGIbNZQCGO8VkL4YYaCxYsMCv+FPHUDrDbrvt5ieHf/7zn4ALXavqgWq7XX/99f5e6hgaNEInRJ1R3zN69Gg/6Uj/tthii5wqC/q/rq6uz/Z139QJXe979uzZPvQsHVebDzzwwJw9mrfYYoucRWJCaWmp7x/qc+ECq+xnC3eW60tkG0dhSEoGiKo9PPnkkzkGWVjLUxOM7jlp0iQf9tJ4cPDBBwMuNUnnh3Wbs1dOS0+efPJJb/gNBMJd20JjI6zJHOKYY47x+q+qFy+//HJGFQ1I9418NWRXrlzp9U2r4aVr9fX1flc47V40UFC/HTduHIsWLQLSehQ6bDK+NL80Njb695zdb8aNG+fHp8GAhoYGFi9eDKSdifXr1+fsLBTOL9l9r7i4OGOsBOewgtvRTTKVHirlotARjm0ylqZNmwak9bmystLrgsaYysrKLutKKx1GxEahOrMhGZadaghpJzS7LnJxcXHGLnWQaXdlp429+uqr/trsmsXQvd3eYug/IiIiIiIiIiKiINHnjGpY6kJeanV1dQ7lLi+3qanJe3aimcPdEHRMVv/atWu9VyQ2raioyHuOCpUOJAuyIZSWlmYwyiFSqZT3RuQFNzY25oT8N7R7SndrePYEerdDhw714WMt3tD7r62t9SEHHRsyZIhnYHUP1btbu3atb6O8tZqaGu8Fy2uUrm255ZY5O5fV1NT43aeyd2lav359nzGq8rJXrFjBlVdeCWR6nwATJkzw7dJzDBs2jL322guAQw89FEjL4+mnn85YKAWOUVRoV/eSvMeNG+eZWulEXV2d/zvbmw53++lL5NuPHhyLqjC9dKGqqirHU1fb6+rqfFv1TltaWvwuKmKIVe5szz339G2UzNrb2/136bnE0D/xxBN9NpaEDGl2aDZfmkY+aPHh6aefDriIgpjou+++G3BlzMQsv/nmm0A6IjNq1CjfdzSGbrPNNj56IfZMLNL777/v2eqBZlTFGi5fvtzLQQtEwoWaYfQJnA5l7wj47LPPAk53JKPBgJD5k84sXbrUtyt7EVVnixbFkmms1XtfsmSJl5VSADRuFyrCSC648nUKW0teYeREuh5G7nSe9EmoqqriwQcfBNKMaiGyqZCeT+rq6rwsFNmEdHvD3bbAjUHZkV7ovFTX7Nmz/bilSExtba0fXxXN2RRERjUiIiIiIiIiIqIg0WeMqrzb4uLiHG883Dc7e9eU8P9wH+rspH95BOXl5X7XDO1VPHLkSO8BiJkpFIR7aat9bW1t3gvLToQPPRgda29vzylTFBaCz84nGTJkyICUpwpzwPR82e1rb2/35T2UuzJv3jzvkem8T37yk4Bj+JRLIxa5qKgoZ39m/W5qasooTQOOcclmGuUNjhgxImNHr77A1Vdf7fvEueeeC6SZ1WXLlvkkdrGbY8eO9e0T+yxmsLS01C8uE3PQ2trq37cYALEezc3N3hMWqxZuspH9fvpr0wjlid58881AOhJSUlLiWSD1+aamppwSQ/q/vr7e653k0dDQ4OUhtkAlrJ544gk+8YlPAJmbHYhJ0nfr/j3ZXWVj0dUuctZa/y61iOX555/3mz2oTNvJJ58MwOWXX861114LwP/8z/8ATte1k5s2FLnhhhsA1/fELp133nmAezdiULUYUlGscePGdbogp78hue23334ZC0EgczOH7DzocGxU3xA7/Pjjj/vc5cGAVatWef0PFyp3FWlT+xVdam1tzZl/dK+1a9f6vzU29FfUpbsIFwMC/OEPf/D6q8hUGJXNXiA1ZswYf56ic+oj06ZN82NXoSwq7Axhbqjemca+e++9N2cHUY2f4bqhsHRoWM4MyIh0KiKhwv81NTVep7qjL5FRjYiIiIiIiIiIKEj0GV0itquhoSHH4h47dqwvuaRVheE2ddnsX7jCTFa5PJolS5Z4j15e9KJFi7x3EO7/XGjQytS2tracldvyzioqKryHEzIX2TKSPIqKijI2BgB8fmN/Q6v4R4wY4ctGaVWt3n9VVRUTJkwA8FuXHnDAAT73MGSFBckh1Am977CkmaB7KAfniCOOyNjHOER5eXmXjFZPoFzLZcuWefb/3XffBdK5QsOHD/fvWzpRUlLioxHy9uXZFxcXe9moz4U6IzmKhQ7zb8O+IYZX8hOD2B/bR65evZqf/OQnQLp/q0RMe3u7b3uY157t4YcIS0lB5qp4RXrEmo8ZM8a/F0VmUqmUz4sXayCWatmyZV6Penv7xLBPP/nkk0D6fav02EcffeQZVclq3LhxnHDCCUC6MLue90c/+hHXXXcdAPvssw/g2HWVdhNTr1XdDQ0NXp/ExFZWVvr3oeiE+uqjjz5aMFuoKlcW4MILLwTI2dwiHEPD6jLSFeXPaWMHbQE5WFBbW+ujKOrrxcXFvp9kbwDS0tKSM5e0tbXlRLTCld06pu8p5Dk2hNr+/vvv+3ev/qWoVJiPrd/heKP+ov8XLlzoo1zaTETRjEJDuG5B71BRtQcffNDbYuFGH5C5haquq66u9nOXbA+dM2rUKB8Zk56F9+hOubc+M1T1MEOGDMkJv0ydOjVn55fsDgGZlL3uoYZLOMOHD/cDq441NDR4QyTcu7fQECYVZw8WoRzC0Cy4TqKOkl3nsb293ctG4YqBMlS1c9Rll13my9pocNNuR0OHDs3Y7QKcESajS51Hxkp5ebmXjYyO8vJyPzFnHxsyZIjXC02yoWErSP+am5v9gNbbu608/vjjgAu3SB4y2FVSpqamJqfMWnl5uZ901K6wJqxkFLYpu46kjKoddtjBOwa6btSoUf58/ZYxVFpamtdZ6A1Ir3/84x/79gvq7w0NDTmh5YaGBt/W7IVTqVQqp9ZyKpXycsgOy3V0dPi2atIaP358xuIbSDsNjY2NXH311QBcccUV3Wx5fmgB0He+8x2vj+oTmiB33nln9thjj4xjkyZN8pPGD37wAyBd1m3YsGH+2WfPnu2/S2OEZCsjtrq62r9vOTfz5s3zqSb77rtvxvUtLS1st912vdL+3oSMp67G1XDBUBjmBPxiyzBNbTAgrEutvlFeXp5TSkgId8ILndjOnPWSkpKcGt7Sk0KHQvTLly/3YXDZBy+88ALgiC+dpwVXzc3NfnyRTNVXlyxZ4tOJlB5TqIaqUoVKS0v9WC59ePXVV/1x6UqYypC9ANEY441ctVvnrFq1yhMpssPKysq8fuUjkzaEGPqPiIiIiIiIiIgoSPQZoxqGzuSxiUFsbm72Xl+4S4KQzX6Ul5dn7MoEaUappKQkJ1wOafan0BKbQw9fXsfQoUNzktaFkpKSLmWl9uULhYqZGygoJHjrrbf6NAftIyw2qrKy0i9W0O9wMYSY+TC8IM9MHt3ixYu93NTmkE2Qxy85VlVV5Xh1kn9TUxPHH398bzQ/B9/+9rcBt8uLFguJ6dSilqampoxi6uB0XSkT2aVgSktLvWx0r/Lycs8UjB49GkgzgsXFxV6P1Oa6urqcHax0r/nz5/uQUW8zqpdffjng3rHem963PP01a9bkbEZQWlqaE5IPIyw6Lyznls1Qqi35FmuNHDnSM8tinaVrpaWl/l31NsIdfyQHMTli+F5//XW/+YPQ1taWUbAfMsvUSR5iiCorK708pFcvvvgi4OQhhlT33Gqrrfz5GrMUwXnrrbf6LFWmJ8i3UEpQu0Ld0Xn5zi/0RTIhamtrve6oT4VRq+wNHsrKynLkEe7cl71oV98B6TF2sOzOpOeePn267ycaN3Ssvr7eM6piXffaay8eeeQRIJ2CozmitrbW99GLL764H1rRfag9ZWVlOePEnDlzmDlzJpBmyDXmrF692pdo0z1Gjhzp06e0658it3vttZePGJ5xxhmA60MaQ7uz+LLwRpiIiIiIiIiIiIgI+qE8VVlZmffmQpZHCxjEYoQFrrNZRWNMhrcHuWVpIL1l3EMPPeTzC7MXyxQSxI6VlZXlsBLy4ltbW7081OZ8DIY8w7a2ti4XFg0UxK7qd5jHI49cJS1qa2u9NyfvLmRIxQJdcMEFQKbXL+ZQTFhVVZVfVCbGdv369f4e4d7FOqevtt3Ve5sxYwYzZswA0u9IuarLli3zucWKQLS0tHi56d1KJ8Li8GIJR40a5bfLFPt3++23A3DNNdd4llXXlZaW5mxlLG955cqVOeVdegsqCbNo0SKfs6z3ESbvS7fDbfj0TGFJLshcCCKdCVlnyS8sw5Wdg9vS0uLvq1zH8P0oX1PvsLdw3HHH+d9a3PGPf/wDSOeBlZSUeDZTbTbG5OSxCxUVFT73LIywSPbKOdVix/fff5/LLrss45yamhrPOGk7VvWvNWvWeHlowVUhQO9UuqDxNHzfmo/CKJTOD2U1EKX9uotbbrklY5EuwFlnneXnY/WDcN2I2pedv5rvs3Xr1vlSaGLNBsM25eDKUkHmFuSCIkhLlizxDKFsh5qaGg466CAgrSuh7ohlfP755wE4+uij+6gFPYPGi6FDh/poXWhLqGRdb0Jjaltbm5+zumOT9ZmhKiMsDKso6dgY4xfHTJ06FUiHuJqbm/0kJGOipqbGh4I1oYb7lWsyOeWUUwBnqGaHBAsRGkiGDh3qO4CUKVxkphcb7i6VvROTfhcXF/uBQ/IrdMjB0O/ehgzagUY46KtPSNenTJnif6u2XQgNLNKBsC7txiz4OOqoowBnxMoIlTHW0dGRsfMRZKbRyPjvbXzpS18CnOMq40u6q7qyQ4cO9f1bg97o0aN9PU8Z8GrLyJEjvVEV7rQiJ0ir1XX+q6++6hcnabHR+PHj/fiiULcmpnCXtb6EdiDT7xDSBRmgtbW13hDJB4X8ZXhuCKpIoTEmXAmthXjSuQkTJhTkYqpshGNn6PBs6HxIT6zZBEohYquttsqpA51KpXx7sqsghP1cKC4uzkl3CFOjPvWpT/XNw/cxXn/9dcD1g9AIhbSuT58+3RulSgGYO3eun5tk0Or62tpab9f87//+L1C4hqpsCmutf5+yqyA9p0g3ssmLzpDtDHZ0dPi+ElYEyU432qRn3+QrIiIiIiIiIiIiIvoBfV5HdeTIkX5hlfbIHj9+vA/DZoepwzBuuPglm0kK2UhZ/Icccoi/NixvU+goKiry7c/eEShfOKa9vT1nF4mwXIzKMQ0GBuDfCT1ZdNLTMmsK2fbVQrHuQHp6zDHH5BxTGLy3cOaZZ/bq/QYS2XUeexsqbTWY0VmUoaioKKdmqjGmy3qrgwmpVCpn0deyZcty0n00FoXsacg6C9ls2pQpU3LOC9PTChGaWzVHlpSUeBZU6SqK0Cxfvjxn1zulAkCaUdW9Jk6c6HVN7OyiRYsKprZwPhQVFXn7IoyW6X3mq02drz9kL0DUdWE6pqI4ZWVleY9v9DNv8hURERERERERERER/YA+Y1TDosOyxnfffXfA7ZetnVCU+6EEd2OMZ1tD9jS7PJXyjBobG33OlorJV1dXew+5kBnVcMFZ9u5T8j5SqVQOE9fW1paTO6Rck8bGRi+3cEMBIZ/XHBEREbE5QTmEGufCDUGy55CSkpKcve27w/oUAvKN61OmTMmYXyHNFobsaxi90/yTvfClqKgo5zsKvWzXaaedBqSL9Dc3N3v2U+WmxIrW19f7MnGyK6qqqny+qhY5aoOAEIpwnH/++TzwwAN90ZQeQe8t3MAgtC3C/tDZtRuDUI/U71pbW/0aA+XWbwoioxoREREREREREVGQ6DNGVR5pWDZl3rx5ANx2221+9a1W+4r5bG5u9hUDZMVPnTrVW+mhlwPOYv/kJz+Z8d2tra3egwz3fy40TJ8+HXCrjbO3zZSXGjLSYmDb2tp8jkz2XuyrV6/2+Ugbu8o3IiIiYrBDc05paSlf+MIXALj//vuBdP5gcXFx3mL2ymNUmbSw2kI+hqlQEeYSih1eu3atZ8tUZUTRtsrKypyKACFrms2WNjU1+Tlb+Y2Fns+r6h/KNd1zzz158sknAXJW/7e3t3PfffcB6VX/7e3tnH/++QD+mErT1dfXc/jhhwNwySWXAOlSgIUGVecIq16oIgj0HjMesrQqNzh58mSvU2GlgY2F6aGSdXqxaParrrrK17E8+OCDAVfLsS9x2WWXeWEp3aCTkhG9Hf/utjAVblApHpVwaGxs9AaqBp5UKuXTHGSwa2HKmDFj/GDbDRSMPAoEUR6ZiPLIRJRHJgZUHvnSmjQPPfPMM4Crh/vyyy8D6dKI++23nzdatYhPREB7e3tPDNV+l0eY2iBccsklvtZtuBsdOKNCBqqMtvb29rwpE+AWD916660Z98+3gKsTFER/WbRoUc7OfbfccgvgHJTshVDnnHOOTx9Qze8vf/nL/rhqfcvo2wSDry/y77o9hhRIWmDeL4+h/4iIiIiIiIiIiIJETxnViIiIiIiIiIiIiD5BZFQjIiIiIiIiIiIKEtFQjYiIiIiIiIiIKEhEQzUiIiIiIiIiIqIgEQ3ViIiIiIiIiIiIgkQ0VCMiIiIiIiIiIgoS0VCNiIiIiIiIiIgoSERDNSIiIiIiIiIioiDRb4aqMeYZY8ypnRybaoyp769niYgYrDDGnGqMeaaL438zxvxHfz5TRGEh6khEROcwxmxjjLHGmJLk/yeMMacN9HNFdI4uDVVjTH3w02GMaQr+P6m3HsJaO99aW7mBZ8lr6BpjDjDGPGWMKUmUb5veeq7uoL9ktrnCGLMwkNlaY8zDxphJA/1c/Q1jzAxjzHPGmHXGmDXGmGeNMXtv6Dpr7RHW2ju6uG+XRkwhIdCFOmNMbSKPM4wxMRJE1JF8MMZ81RjzcjJ+LEuM8hk9vOegMGT+HftL1nyxwhhzuzGmS1siIhODYc7tUoGttZX6AT4Ejgk++2N/PKAxpmgDHe0o4K/98Swbg02Vmby6gUQhPEMWjknktyWwArhugJ+nX2GMGQE8hGv3FsBWwGVASw/vW2jveWNwjLV2ODAZuAq4CLgl34nGmI3eZHuwI+pILowx3wGuBa4AxgFbA78GjhvI5+pn/Dv2F80XewB7AZcM8PNsEAUo+4Kec3vV0zLGDDXGzDTGrE48uheNMWOCU6YkXl6dMeYRY8wWyXXTjDE2uM8zxpifGmOeBxqAO4H9gd8kVv+1wT2PxBmqTyX/v5Wc8/nkXmcYY95PnunPxpgtk8/FwJ5jjFlgjKkxxlzV196nMeZyY8zdxpg7jTF1wMnGmApjzK8SBuAjY8w1xpiy5PzTjDFPBNdnMMfGmKONMe8kMl1ijLkgOPdYY8wbybt4xhgzPTi2xBjzXWPMmzgZFxystc3AfcBOAMaYo4wxrxlj1htjFhtjLg3PN8Z8zRizKHnXP0w8xUMH4NF7iu0BrLV3WmtT1toma+2j1trZOsEY8/PE+11gjDki+NyzPwkz9qwx5pfGmNXA3cBvgP2TPlLbz+3qNqy166y1s4AvA/9hjJmesCc3GGP+aoxpAA42xpQnsvkwYVh+Y4wZAmCMGWOMeSjpD2uMMU+rvxtjLkr6Xp0x5l1jzCED2NyNQdSRAMaYkcBPgLOstfdbaxustW3W2gettd9N9OJaY8zS5OdaY0x5cu2oRC9WJfJ6yBgzMTn2M+AA4PpEHtcPXCs3Hv+O/cVa+xHwN2B69thvjLnUGPOHDd3DOGLskmQeWWmM+V2iW0qZOTvr/DeMMZ9L/t7BGPNYIqt3jTFfCs7LkX0vNbtXUahzbm8bZf8JDAUmAqOBbwPNwfGvAv+B83aHAd/p4l6nAF8HRgAnAc8DZyTM5PkAxtHTVcng/Knkuo8n5/zJGPMZ3OD1BRzjsBTIZjWPw3lieybnfa0b7d5UnADMBEbiJoYf4TzBXYDdgU8C39/Ie90GfCPxoncBngQwLgT4W+A03Lu4FfiLSQzgBCcCRwBVPWxPn8AYMxQ30L6QfNSAez9VOCb9TGPM8cm5O+HYk5NwXuFI3DsfjHgPSBlj7jDGHGGMGZV1fF/gXWAMcDVwizHGdHKvfYH5uD53MnAG8HzSRwryvXcFa+2LwBKc8QBuTPkZMBx4BscibQ/sBkzD6cCPknMvTK6txsnjB4A1xnwMOBvYO+lHnwUW9kNzeoKoI5nYH6gAHujk+H8B++H0YldgH9LMWxFuHJ2MY2GbgOsBrLX/BTwNnJ3I42wGEf6d+ktiDxwJvNaD25ya/BwMTAUqSXQBR5h9Jfi+nXA687AxZhjwGG5eH4ubW3+dnCNky77gUKhzbm8bqm24gXFa4uW/bK0NF0ndYq2dZ61tBO7FdY7OcKu19p3EK27v5JwjcR5UZzgJuNla+3riKVwMHChvOcFV1tq11tpFwK8IFLEP8Uzi6XdYa5uS57zUWrvKWrsSZ1yfspH3agN2MsYMt9ausda+mnz+TeDX1tqXkndxa/J5mMP2P9baJckzFBL+nDA564DDgP8GsNY+Ya19M5HbbNzAcWByzReAB621z1hrW3GDrc1z74KHtXY9MAP3/L8FVhljZhljxiWnLLLW/tZamwLuwA0S4/LfjaXW2uuste0F+J67i6W4cDfAX6y1z1prO3Bh728CFyR9oQ4XBj4xObcNJ6vJybjytLXWAimgHNePSq21C621H/RrizYRUUdyMBqo6WKuOAn4ibV2pbV2FS5N4hQAa+1qa+2frLWNic78jPS4sjlgc+8vmi+ewRE1V/TgXicB11i3bqYeRxidaFxKzAPAbsaYycG591trW4CjgYXW2tuSfvQa8Cfgi8G9vewTe6SQUNBzbrcNVWNMsclcODQBuB34O3BPEha4ymTmPC0P/m7EeSudYfFGPIbC/p1hArBI/ySD+1oyrf7wexYl1/Q1stuW8ZzJ3xvrmZwAHAt8aFxIb9/k88nARUnYpjZRwi3pvO2FhOMTJqcC57k/aYwZb4zZ1xjzzyREtw7H/Ci1ZAJBexJnaHV/P3hvIXHSTrXWTgSm49qnlJflwXmNyZ+d9aVCfcc9wVbAmuTvsH3VuIjOK4HOP5J8Dm7wfR941Bgz3xhzMYC19n3gfOBSYKUx5q5kPCtoRB3JwGpgjOk8xzbfGDsBfMrajUkIcz0ujazKFF4eYXexufeX4621Vdbaydbab/fQ2cqnJyXAuMSQf5i0If8V0hHaycC+WfPtScD44F6F3M8Kes7ttqGasHSVwc9Sa22rtfZSa+2OOG//BNzL6tZXdPV/EsKegTOM850PzpOcHFwzHBgFfBScE65u2zq5pq+R/awZz5k8h56xATeYCKHiY639l7X2WFy44SHgruTQYuCypAPrZ6i19p4unqOgkOjY/TgPfgYurDILmGStHYnLpVM4cxku5QQA4/KsRvfvE/cNrLVzcU7g9A2cmvfyDfw/qJCktGxFOnQWtqcGF7b9eKDzI21SUcRaW2etvdBaOxXn3H3HJLl11tqZ1toZuH5ogf/XT03qFUQd4XkcQ3h8J8fzjbEa6y8EPgbsa60dQTqNTGPLYJQH8G/dX7qcN7tA3Avd5QAAIABJREFUPj1pxy0wgiT8b4xRqsk/k88XA09mzbeV1tozg3sVvB4V6pzb24upPm1c0nYRsB4XOujopduvwOWMCAcCr1hrG8AJGGfNh+fcCXzDGLOLcYnzVwJPW2uXBOd8zxhTZYzZGjgXlzPa37gT+JFxyevVwA8BJX6/AexijNk5UYQf6yJjzBDjyrGMsNa2AXWk5f1b4CxjzN7GodIYc0ySSzMokDz3cTjn4h1cbs8aa22zMWYfXM6PcB9wjDHmE4kTcynpDjWoYFxS/oUmvaBjEs57f6HrKzcKK4CJWbnKBQ9jzAhjzNE4R+wP1to3s89Jwpm/BX5pjBmbXLeVMeazyd9HG7dw0+BCXCmgwxjzsWTsKsfl1DfRe+NWnyDqSCastetwocf/NcYcn7Ckpcbl716NG2MvMcZUG7fA90ekx9jhuHdea9wC3x9n3T577il4xP7C67iQfakxZi9cmHpjcCdwgTFminFlrq4A7g5SSv6KM2R/knyudj8EbG+MOSX5ztJk7t2x95rU9yjUObe3c1QnAPfjjNS3cGznzF6697U4T6bWGHMN+ctS/RiYmZzzOWvtIziFegBn/W9NLsP7IE6pX0vOu72XnndTcBnOIJ0DzAb+hTOqsda+jessT+AWRzyVde1/AApZfQO3GAJr7QvAmcANuHSH93RsEOBB4zaAWI/LF/sPa+1buMV5PzGuWsKPAM8OJ8fPwQ3My4B6YCU9LNczQKjDLXD5l3ErRF/A6caFvXDvx3F9c7kxpqYX7tfXeDB534txC2KuwS3a7AwX4cKVLyR94u84tgxgu+T/ehwD92tr7T9x+XZX4Rim5bjoxMYuZhwoRB3JgrX2F7gFupcAq3A6czbwZ+By4GXc+Pom8GryGbi5ZQju/b+AC3+H+B/gC8ZVBPhVHzejp4j9xeGHwLa4ue8yNt4OuRX4PW6eXYAzxM/RwSQf9X7g0PCeSVrAZ3BpAUtxcvl/OFkNBhT0nGtcbvTggzHmPeBoa+173by+BMf4TrHWLuzNZ4sYeCTecC2wnbV2wUA/T0RERERExOaKvpxzB+WOFcaYClwFgW4ZqRGbJ5LUhqFJesPPcczJwoF9qoiIiIiIiM0P/TXnDkpD1VrbbK0ttOTtiIHHcbiwy1Jc2OpEO1hDBhEREREREYWNfplzB23oPyIiIiIiIiIiYvPGoGRUIyIiIiIiIiIiNn9EQzUiIiIiIiIiIqIg0dkuHhuLHucN/PWvrsLUkUce2eV569atA+Dvf3f1/T//+c/nPkySxmA63dI6B71d86vH8njmGVeXec6cOQCUl5dTXOw2SNl+++0BaGxsZO3atQDMmDEDwP8/fvx4qqq6vT13v8vDWpvzvlpbW1m0yG0O0tHhytStWeM2Vlm/fj1tbW0Z53d0dFBS4lRZ9xo2zJWLnTJlCqWlpYCTTTba2115PF2fhYLTjwFGwcnjl7/8JQB1dXUAXHPNNey3334AfO5znwPggw8+oKzMlQRVPxkzxm2u8u1vf5uxY8d29+sLRh6djX1r1qzhH//4BwATJ7ra3I2NjX6M2HPPPXPuswnjZzYKQh6pVMqPmdlYvXo1f/yj20xoxx1dicu5c+fy0Uduf5WrrrqqO1/ZGQpCHo2NjcyfPx/AtzOVSgFQXFzM0KGuLv6//vUvAI466ij++U9Xx36HHXYAoKjIcVr77bcfFRUV3X3+gpBHPtx5550AvPHGG1RWug3c9Hv16tXe/vjZz34GwPDhw3vja/ui5uhmOcf0NEd1ky7+4AO3HfAvfvELXnnlFQAWLHBVDDRxFBcXs+uuuwJpI+Wdd96hpsaV89PzbrfddoAbbK688koARo4c6a9Tx9oACq7jfPOb3wTwk8uOO+7o5TZ9utt0Zvjw4d6w+trXvgY44w6goqKCT3ziE939+n6TR76J9ZFHXPnCDz/8kA8//BDAG6z19fWAe7eahGSAtrW1+fvoM73/4cOHs8ceewBpnZk6dSrbbLNN3ufJeqYB1Y+GhgYAHn74YT/RPPvsswDsvvvugNOPhQsXAngDfu+992bpUrfpjmRaXe12Rdxjjz0YN85t+X7UUUcBbGxfgQLqLy+//DIABxxwAABf/aqrQ11eXs4NN9wAwNNPP+3P0Zhy2GGHAXDzzTcDcOaZZ3LFFd3eGnxA5KFxcWPe27e//W1mz54NwBZbuO3eR48eTXOz22pcE/SGvm8wjKddyUVG2Mknn+zHiIMOOgiAZcuW+X713e9+N+N3xsMMMiLkpz/9KQArV65k9Wq3s6UclGXLlgFu/Hj99dcB/O8//vGPXHfddRnny3A966yzePTRRwH44Q9/CKT74EagIMaPJUuW+D4hg/3yy11J3ba2NnbeeWcAfve73wGuzZpvm5rc7qzSnWnTprHTTjsBaXJkExAN1VwMnKH6/PPPA/D1r38dgIULF3qvbMSIEUCa0dpiiy0YPdrtwqXBtKqqyhtimrA16I4cOZKDDz4YcAoFTmE2cjAviI4T4lvf+haAl8+wYcP8ICrvdp999vGDym677QbgjdOioiI+9rGP0U30uTzyDfaaLGWQL1682A8kQ4YMAdIDa1VVlTc6XnrpJSA92ECaed1yyy399bqv2OcjjzzS/z1lypROn4sB0g+19ec//zkAo0aNYvJkt6tfbW0tkGaCW1tbee211wDHNkPmxCHjVcZpeH8NuhdccEFetjkPCqa/vP322wAccsghQHocOemkk7w+rFy5EnBsq2Ry2223ZVx/yy238MUvfrG7j1Ew8pg7dy4Af/vb34C0YdbW1uajVhpDOzo6vAFy+OGHA3gZHHLIId7h7wYKRh6/+c1vALjnHlefXMZpR0cHL774IpA2LKy13pGT0fHOO+8AcMIJJ/CDH/wAwLPym4ABkYf0/7TTTgPcXKmIg973448/DsDWW2/tx1EZs1dffTX33XcfkJ5zpFeHHnooDzzwgL8vwB/+oA2+NogBkcebb7pNuRSNbWlp8fqvufKtt94CHEEkYmPUqFGAc+xEmGicEdu6dOlSb2torjrjjDP83xtAvxqqoU0UMurZEHu89957A46RF5EouU2aNIlf/crtfSE59RLyyiTmqEZERERERERERBQkepqjmoNsZqq+vt7nxYj5WbVqlf9bTMBXvvIVwHkvulYh/cMOO8x7PmJZJ0yY4BpQUuK95v/8T7dT3D333LMp4cyCgHJT5dkrz+7111/3bFjYJnl0+qyxsRHIn4dZSMjWj8WLF/u0DrHqu+++u/dgv/SlLwH4cyoqKjj33HMBPMtYXFzsWfiWFrd7mxiU0tJS7w2+8cYbgJOxWCMxqnqeHubo9QoefvhhIJ2qMGzYMN9+Pa/Y07a2Nk488UQg7e3Pnz+f5cuXA/j8s6233hqAFStWeN2SjGfNmuVTTgYLxA5kR4SuueYadtllFyCdk9nW1ubD+4pUiEkQ0zSYICZc4em3337b67MY9DDKsM8++wAwb948wKVHiC3ReKp7VVdX+7FVqUbnnXee71+FjPfffx+Aiy66yPcPsaAhGypZKV+5vr7e9zVhq622AlyqzXHHHQekZfTpT3+6r5rQK5BOq2+sX7/ez8FquyIzY8aM8Uyq5p45c+b4sVj6IfZ5xYoVPpqjPljIqKur8xEFjZ3FxcWe8VR61V577QW4nG1FIzS3rl692uexS0aaI0LmVJGqm266ifPOO6/vGrWJyBc1z2cfKcr08Y9/HIDPfvazgItESk6KWP7+97/3kV5FuYVNSBXaaAwuay4iIiIiIiIiIuLfBn3OqC5YsIDnnnsOgKeeegpw+VDHHnsskF7QIYajubnZsx4nn3wy4Fi3bE9GFv4tt9zi8xHlEdTU1HgWrRsJ8AMCyUgei1b9t7W1eVYsbLs+k8erBTXFxcWeDSgk6D1ke1orVqzwHpk83hEjRnhm9JprrgFcTgw4D1aMqtpsrfX3Fbt+9tlnA7Dtttv6e4mBra+v94xjvuccaF0Ro6rc67Fjx3p9FxMihqi0tNQzxepD1dXVnkFVHpmuq6qq8jqjds6ePXtD1Q8KFtmsztixY3nvPbezshZclZaW+jwqyUhtF1M9mKDok8bJ6dOn+/6l1cjS77/97W++X02dOhVw7JhYJvWrL3zhC4Bja8XKKtp1+umnc//99/dto3oByrtcvXq1j0KJXdQYMGHCBM8OhizjtGnTgHQ/UT+oqqry9xCDVOiMqphi9X1rrWf+ysvLgTQrX1VV5edWtbO1tdWfp/x+6VpLS4tn6zWOrF+/3kdzCg0LFy70LLJ+p1Ip/+waD6QfdXV1nlHU2FJSUuIZeo3DYW6nxk6xrjU1Nf5+kuNAQmNdGDXMZxcp31+RKC3EzYcbb7yRbbfdFoBLLrkESC9I64todmRUIyIiIiIiIiIiChK9Tp9ks1EjR47kk5/8JJDOo9x11129Rb9ixQogvZps4cKF3uNVXtSIESP8fZVPo2PHHHMMjz32GJBe+b5mzRrPqA4WaNVhdi5Yc3OzZ0nkza1bty7HC5Y8C5FNhXTOXDZjt3TpUq8DqmtojPF/f+pTnwLSKxAvv/xyLr30UsDlogHMnDnTMwTXX389kM6pamho8MeE8ePH+xxfVaQQo1JdXT2gLPyqVat86SwxI0VFRb6GrhgfPeOwYcNyViqHOhAyQ+DYErEDwqhRo3x+lRi2QofYDumVWMCQPRLrnI/9yB5/Bgvmz5/v26V3VV5e7sdWyUMs6s9+9jOfu6ljra2t7L///hn3DRkgRWs0hi5atMivnFbpnkKEWOTy8nJf4UDjgCJVjY2NPodZOj9u3DjPCKp/haWG9Hf2OFKouPXWW4E049nW1ubL+6nsn/rPBx984OcQ/V66dKk/T+UjP/OZz/hjmo8k05kzZ3LGGWf0baO6iYaGhpx8VGNMTgWHkG2UrPRZeXm5H1/0mVjDVCrl+5U+a25u9my9ohiFgHClf/YzX3fddb7/HHrooRnXhTmnYeRNFYeuvfZaIM2o9gX63FB9++23/cKpJUuWAM7Q0uSq8JIS+MvKyjyFrnDT8uXLOf744wG49957AZcOAK5EkcJ6ovZvvPFGfvGLX+R9nkKFJlNNDjKk2tra/MCgxRBr1qzxJZoU/pbhqgG6kBCmcwhhXT+979DoloGlDqJyKWvWrPGGqjB79mwf6lf7f/SjHwEuVUCTkMoVrVixwm8woTDeHXfcAbjFW+qMWpDVn1AfgbTxsHjxYv8s0nX9bm1tzSgZImgg1qAbDqxKtdA9Jk2a5A22wWKoypiSnsjwTKVSOZNKmBqiAVrnyxkYLFi8eHFO+lNRUZGXh9qZXc4vxPjx471uZRtfJSUl/tpwLBkMhuqqVasA12+zHRmNq2vWrPFOsIz70aNHe7lJjupTjY2N/h6acwodmktUZmn69OnMmjULgAcffBBIl6m67bbbfPrdQw89BLhxUmPmgQceCKTTKo4++mhPpqhEYiEvtFu/fn1O2llJSYl3PqTjoXGq8V9zcLj4Sv0l1C/NWxpzR4wY4ReqFpKhGo6H2YSR7CqAU045JeNYW1ubb1toT5155plA2k7T5isXXHBBl2SPZJedktAVYug/IiIiIiIiIiKiINFnKyfklVx00UU+rKpQy5VXXukZQIUpVX6qpqbGs20qmROGYRSyUjmnO+64w4fNjz76aCBdhmgwQV6ZEpnFMNfU1Hi2SyU0fv3rX3uZaNGAFpQVIioqKrwnOnPmTCBdjPrmm2/2C3/CrVHFIKqosNp+//33e89fi6pOOeUUv12mmJ//+q//ApwXLYZB3vOrr77KMcccA6QXXYVM4kAwqcLcuXO956/w0YgRIzwzpvctNm3YsGE+BKfE/cbGRn9cuhOWU8kOd0+ZMsUzAJJzoSO7PFtYzFrvOyzRtLlg3rx5Xj/D0kHZrES42E6pHmIQww1Rsrcettb6fhKyrQqJFzJUqLy4uDiHtZGejBo1yrPJKma/7bbb+rEne1FQuFhPfarQkR1xgnTpLoX0Nd4NHTrURyLFjE6YMMFHdlTaTIzsUUcd5efzwYCmpqacMcJam8OMKuJnjPH9RKxguMBWY0pYlkohc81jZWVlBRmpCaMtGh+kD08//bSfk7XbpRAuCAvTqJQiINlpgeYFF1zg5RWmCvQkpS4yqhEREREREREREQWJPmNU5anffPPNOfuOh15O9nafVVVVPhdIjNL222/vPZl3330XSBc8X7RokS9MffrppwPOOxhM5Xba2tq8R6+cFnm3ixYt8qyz8oVuuummHJZV3kp2LmihQDnGaqcWwO2///5+i0ItIho2bJj3WMUOaKHQQw89xMUXXwykk/nHjh3LSSedBOQujjHG5JTuWbx4sWcQ//u//xuA3/72t4AraXTOOef0Spu7A5VWgjQ7vN9++/lnF7Mlz7ajo8P/rX5WWlrqGXd5r/Kgx48f75k45TzvtNNOeXMZCxnKKRM7Jr1PpVJ5Pfbsotebkh9VSFi+fLlnk8XatLe3+zEiu5RdcXGxf/eK1pSUlHi5aZzU+a2trb6PikGsrKz08i5kqNTYkCFDfPRO/SDMQVX0SuskSkpKPIuWnftcW1vrdUvHBiM03koekoExxs+t0pN169b5MULsvXJ2X3rpJc+oDobSj01NTb7N0vXGxkZfuktsYbhRht6zIguhDaFxWLpWXl7u+5K+p76+3suvEJC9nXy4kCyMPmsO3FSo7GNYHk72S7hxRL7v31j0mRUXLlaQwSmKfNasWb5GqpLcNShWVFT41WSaPOvq6rzyaEAR7XzTTTfxve99D0gnd8+aNcvvvjMYVv/X1tbmrKrT4FFTU+OT3fXy161b51MDwgkG0iHRQkJTU5M3ovR8559/PuDqxcox0US6cuVKb4TqOuEnP/mJH1wuuOAC/7lSQfTeFdoqKirysg1rAWaHMm+66SbAdeqBNFTXrl3r2yfjIAw5yijV73DfZvW5IUOG+IlIC6Z0r7KyMj94aJA68cQT/WA7WKCFcdnhfWut/yzfzjnZButgSw9Ys2aNH+g12dbV1eUYkjIwwoVkOicMW+pemlgbGhr83+ECrUKaeDuD5oaRI0f60K70WmNLQ0ODHwc0xgwbNszLQeOoZLRu3TqvMzLWCh35DEh9poWrOlZbW5uzV3u+8UbykQMA6f6Vb7/4QkFra6vXf+lEKpXy71lt0P+pVMrrRz6SS/dQX6qoqPAOo3Tm/7N35mFyVeW6/63qudPpTkIIGQgkhABBQIYAIkERMAwCgsogciEqR5TD8Rz1KB5FAedzrvM9V+AiKIiCHFQGERRkDAmDzCCDmRMyD93pKemh9v1j7XfVql3VnU7Sw67Oep+nn66qvWvXXmt/61trvd9UXV1dkFllKFEsr+mf//xnIOciV11d7YLvtO6SK82kSZMKTPlNTU1ujtUYk9vM6aefzre//W0gF6xcjEDbno1OMP0HBAQEBAQEBASkEgNuF99zzz2dad5Pf/GHP/wByK24f/7znwN2h6c0BzIJjxkzxuXTVBohmYnb2tocuyh2ZJ999nEBWaXAqG7YsMGxaNpl6H13dzfjx4/PO3/58uXuuHbD2hlqx5MmRFHkTPLafcqE8vLLL7sduRj3qVOnutdik4844ggAbrvtNu666y4Al7uvpqbGVSVLVl2CQlNDMfPweeedB+RMZEMFv+qYWNGysjInx/rMD6ZS+7SzfeGFFwpS8GhXXVVVVdD2FStWlFw+UbW1N0bUD4YoVu8a6LFCWVqxYcMGN3bE4GzdutU9U8m/z1botWShoqLCMUgae2KAuru7Xd+KQYyiyLEraYY/9qUHk3lzR4wY4XSQdGhVVZVjz6RH1Qdbtmxx/ZxmE3dv6OzsdAFT6gexX11dXQVuYy0tLc7yJdZZ+kQuU5BuJlXw8wKLDc1kMnmuMZB7tn5gomTAzzuqvvLdQHSePmttbU0Foyo3smuvvRaA22+/HSiu82pqalwaUUHyIL0BuTEzffp0N1a0BtE4efvtt12Vq1mzZgE2qF5WnpNPPhnID+Dc1tgKjGpAQEBAQEBAQEAqMWCM6gUXXOBeKzm9qlzsttturoqUHHGvueYawFYK0kpeFQ/a2tpcXdlkcviPf/zjfP/73wdyK/QXXniB+++/H4DHHntsIJrXr1iwYEEBK6ZCCMUSsL/vfe9zPjLa1WgHl8bky7W1tS7YQ7sv+cE89NBDbncv/yd/py62XIzpe9/7XheU9+yzzwJWZpSqSinKVA2tvr6+wD+mqqrK/aYS/3/xi18E4O677+6HFm8/9PxGjhzpdq1iTTs6OtzuWLthVdfp7Ox0MiNZePPNNx0DLad/+XTW19e7nb/vRy7GVfeR1trdgsZ/MUY16ZPl+6qqzWLQfJ+7UkBLS4sb42IQ29vb3bjyk5qDHUtiRDQG29ranL7x2RKwY1Xsup/+yk8dl1b4frnqD8m9/o8fP97JjHx8oZB51bVmzpzJyy+/DOSnQhuIeuYDhU2bNjmdmqxy2NzcXJCqqbOz0/WHxpnaWyq+7LrPzs5OJ+O+H6pkQP/1vLPZbEEVRT9Bvq6h8dPV1ZXHDOp7xfzjBxPXXnutCzqWTtc8OHr0aFe5TkHKkEvX5qfqAjtfii2VlaWurq6AXX7++ecBa/V53/veB+TWfBdffLGzCio+6Wtf+1re7/SG0hltAQEBAQEBAQEBuxT6nVEVg6Pa7J/5zGecb4TqsB911FEuE4B8YOQDlc1mXVlQ7YpVHADgsMMOA3LR47/61a8cyyrfo/POO8+ViCsFbNq0ye3Y1C4xaMVKsM2cOdOxHTpfiXvTmp5K96WkyNp9rVu3zrFa2rWtWrXK7ehVOvW5554D4Morr3QyI79lyNW3lp+NsgWUl5c7ORI7u3LlShcFnPT5VBqtwYZk99lnn3XWA+1Ys9msS/Svnbzed3R0FPgKNTY2us/UTn0viiLn3/3qq6+639Z3dR9pZ1R7Sr5eLDl3dXV1ASMohqjUfFSjKCpg3BsaGtx4EXPm+9eprX76pmSZWd8vT+drbNTW1pZEsnvp0K6uLhfB/PTTTwP5vrtizyTjfmED9Yv6eOLEiQW+e77+KAV0dXW5cS023vffFooluNc83Ref8DRBzKfPAEue6+rq3FyTPE++vJCfgUey5TO1kD++1Ddbt24dMkZV4/Syyy5z9ywWVGPAz4rhzyvJDCBCeXm5kx+1q7Gx0fWVrLsaTzNmzHDX2G+//dz5ki9lBPCL8iRTaCXR7wtVPVzlFKupqeGqq64C4OyzzwbgxBNPdHSxFpm33norYNMLyUSlxWt5ebkTAn1PCmjSpEk8+eSTQM6c/MMf/tDR2ErDIAfeNGL16tXOBUIPW4NKTuw+/FrCEkY9YJl20ga5f0jgpQjXrFnjFqoaNFVVVW5wXXzxxUDu+V1zzTWceeaZQM695JlnnnH1hlVBRab/sWPHOtOd8v91dHS4gaT+k6uFqpwNNrSIGD9+vFs8yVSy2267uf5IBoYZY9ziRObN9vZ2p4iTdavHjh2bFyii89UPug/JY1rh16GHfPO+ZKvYZJEMokhD0ENf4AdzSDfo2c6aNcvlMJQ+1bEtW7a4SdWveS4ZkN5Qf6xatcoFQsybN89dy1/MpQ1ql9qZzWbdplTjxc+zq4Wq9E59fb0bL+oHpa879thj8yr6gCVX0rxQTZpS29ra3AJVz1H/u7q6CgLJMpmM6y/pHZlwSyWXrG/ST6Ye89cTgt9nyRSRfq7qYpXckq4CfmW0wYZcHisqKlyOernySKa7urrca99dQTpB40P/q6ur3fyhfujs7HTtVr9qc1dTU8O6deuA3PhraGjI2/hCbq4+66yzQjBVQEBAQEBAQEBAaaLfGVWZE+Wk66eGEcva2dnpgmNkChazsXLlyoJa6wsWLHBsmFbeMok//vjj7jyltZo8eXLRIKS0YtOmTa5KSrLOsBgSH7vttpszf2tXI/YtrSlDVEf4lltuAXL3u3TpUvfs9f+YY45x3xMbKteGuro6t0u78cYbgXxTlczaH/zgBwEbWKR61WJcysvLndlCrKKu+fzzz7sCBIMpQzKfTJw4kb/+9a9Ajl2fOHFiQf16mXiy2WzBbrSsrMztbpOVrDo6OgqKBqxbt85ZMdLMnPnojQlNBjcUC6ZS25PBRGmF2A5jjJNj6dODDz7Ypf5Lmu/89FRy7+jo6HA6Vuf5DLPcq/72t78BhUUS0gaNHZ8lk0VA/aH/mUymoP/84BexYj5Llmx/qYwRoa2tzcm5Ashk2cpkMk4W/IIJss5IFsSGpdVil0RyHoUc8zd16lQnMzrPr/Sn88T8+eyh/ot1zWazeakywcqaGOnBrpApPd7Z2cn06dMBXKpOoa2tjeOOOw7IzasjR45kxYoVefcqOW9ubnYWB42diooK1zbNNTq/oqLCret8FwvpH60NlaI0MKoBAQEBAQEBAQEli35f5qu0qf4DjqESLr30UvdaZU8XLFgAWAZAbNszzzwD2BW9dgf6THXQX375Zbc7/MQnPgGUjh+NUFtb6/xJ5Avk1+BOlhobM2aMc3JPOv+LJUgbZs6cCcAjjzwC5HZaNTU1BaxWR0dHQS13vZ8wYYI7z2cVtdP/9a9/DeRkYcyYMY6tFrvY0dHh+lTX1/fb2tpcSjOl0RgMqA/q6+sdW6hne+CBBxaMId+fLJlCxk9anUy7s3nzZudfp3Pa29sLfKPTjiRL4LOnYkd6YwL7ck6aIN/h8vJyJxfyO9xvv/2KMkiC5ELfK5Zqyi+zqkAkn+VIczom6YFiuiHpd5fNZh2LLLZw48aN7vxksvdXXnnFfSamqRQCy3ysXLkyz38X8v3TBd/XV+eJGROT2NDQUDAf9SVh+2DDl3E9e/lZTpgwwa0jxLT7wVJ6re91dHS48ZW0VHV3dzt2UrqzqqrK9Z/mrcEqv+tbIxVQnFwT7Le7yPz9AAAgAElEQVTffm7+VSwP4FJWac6QbPjzifp11KhRBcV1NIeMHz/etVtrsxUrVjgdIkulrKvXXXddXlnnYhhwPrq7uzvPAResqUqBVXfccQeQq0zl185VQExFRYVT1MmobGUXgPwF6vbUkR1q+M78ouJV0QEK2zJt2jQnJMma9WmE77CvfKdXXHEFYKPS1Xbf1CAFony7MmncfvvtbnMjU8X69eu56KKLgJzi0WZn/PjxeYElYJWvzBD+AhGsvCrKdzAXqr6S06JU/VJbW+sUnxSl7+Ihudcmp7m52SkF9YdvutJrObxPnjzZuZIk636nFcnFQrEFmv+Z+k//fb2gRYn6No1QYGkmk3FyrIVqfX19QUU6PWN/0SEdWl1d7eQjOQH5n8k9prGx0V0/jX3lB72AHQ8aw8mglo6ODrdgmT9/PmB1i/STJlhN7uvWrStwISmVXKLC6tWrC/Kn+hkP/IpNkB8MlNST2WzW6Y1ibmlpQ2dnp5N76cmamhr3DGWi911DRBT4zz0ZxKr3TU1Nbm5SQK4f6Km1zGAtVH2ojSImRAZWVVWxcOFCAP7xj38AcNxxxzkXBgWq65533313N7foma9evdqNFfWhxtBLL71U4Cry0ksvufvS3KzrX3fddXzuc5/rtS3p3SYHBAQEBAQEBATs0uh3RjXJ/vkmIz/gRQyBzC8y5fk7Xj+oSq+10xWj9OEPf7jgN/1a7qXAqJaXlztGS2yJWLV3vetdBTnGKioqXK5N7eLEcBx99NGpc33wAxJ0n2JUr732Wne/2rW/9dZbbsf/wx/+EMjtCh955BEeeughID9n5Fe+8hUAl4P361//OmB3w9rp+cF82unqv/oT8qt1DBb8nHfJVFTd3d2ujyQffqUYtctPDyKoT/X9rq6uAvbNr26VZA7Sip4sCb6+8RmRJOPq6wUxLmJG0gg/tY5YMbm0+MeTNckzmYx73spHvXXrVtd+yUcyAA1yptK1a9e664tt8nNbDzU0JvxcskkWTWOqs7PTne8HTKn9YoDECI0bN86lulKan1Ko0uVDgVGQ03dyk/ArKwl++j7JgG9x0vXSzKj6Lk9+kBjYeSD5mXSn5mHI6ZK2tjbXD8kco/5Ykg7fvHmzm5uS5w80xJRCzmLmu8SAtbzNnj0bwFUwbGlpcayp+kQBiW1tbW5+0LP3rRBqo9YilZWVjonXOFyyZIlzI9GYVJ///ve/D4xqQEBAQEBAQEBAaaLfGdXk7sx/r916ZWWlW40rTZXYjEwm43YjWo3Pnz/f+TXIWViM7D777ON2uL6fSSkwqUJ1dbVL1Ktdhl+DPJlyau3ata5v3vGOdwC5NE5+ZY004zvf+Q5gGdUki7Fx40bHmMnhWjvgNWvW8KUvfQnItXX16tV5TuSQcySfOnVqQeDAypUrC4K15O83YsQId2wwofsoLy/n4IMPBnLslZ/aRGNIDJcfHCKmrby83B3X+T6Lqv4Wi7xq1Sq32x3qGtV9hc989IRt+a0KpRAcI9k1xhRUffGRTGSezWbdM9U1fD9lyYXYI79/xo8fD1gdLd2qcZkmRjVZNcefX6RTZJGpqqpy41tsUmNjYwGDqPEzduxYxywpfiDNgWXF8OKLLzpdqefuB7D6afsgv2BIsjBEdXW1060HHXQQkE6rpdrk+2/7AceSFT1nPf/m5mYn9z7jrvOSxUQ2bdrkxoZY+9bW1gK2cbCgokeQs1JLptWuZcuWuQI50n2dnZ3uGWut9eCDDwJ2rGt+FGs6YcIEJyNai6i/Ro4c6ZhX6ahDDjnEyVzSb1cpTXtDaY24gICAgICAgICAXQaDk4U2hp/uQSttsTvyo8hms27FrWiy1atXuzQKyajFI444wrEr2jH4KWpKAX6tbjEA2tVA4Y61urraRbUecsghQK6OfRp3t1Dou6z/mzdvdqm51Pb29nb3nO+66y4AlwR/5cqVjs1RicfTTz/dJSdXqhBFMFZUVLjflly1trbmlXaD/Kj4wUrOXAzFEvL7ZQCTaU+6u7vdGNJ9V1ZWOj+gJMPW1tbm2AGVrF24cKGTH1kq0g6fYYTiTGkyNZcPf5wky7GmEWrvyJEjne+Z71Or42KS/Eh49YMY+oqKijzZ8s/35UW65dZbby1I8ZYmaCzoOe+xxx6uzdKTvv+qZFxt8RnSZDxAc3Ozu65YJd/nsxSwfv1650+afM7l5eWO9RPj2NLS4vzedUx9UF1d7Xx10wwxwVEUOb2vtEg+q655Qrqzrq7OzT3SC52dnY6VlFxo/dHY2OjGnPpY1k2gIB5goCE/U/+35beqOaG6utqNf7GilZWVro3qGxWHqKysLCit3dzc7GQjmTmhpqbGzUmSMz9uQvO8xlNf5GlQ01MJl1xyCZdddhmQ60RNHOeee65Lo6DqU9OnT3c0sdIcqHPuvPNOTjnlFCC3UC01jBo1yg0ODRzVey+GKIrcIJIZRoMwrQtVCawG+ptvvgnY+5Xw+7XqJehXXXUVAGeeeSYAjz76qGvjl7/8ZQDOOeccl6f1u9/9LgBXXnklYM1TupaU14YNG9zmRoNRSrqzs3NI5EiD1zdD6xk3Nja6fksqvsrKSje5JhWSD3+DoGehcVlbW1tQtznt0KKrmBk2uXjd1kJVY02mzDRCsusHb/j165NtVb/4rhwKcCgmY8nKZ5DLoZnJZNx101iVKZlqq6qqym3C/H4Da8b084WCXaxrzGjhrnPa2tp45zvfCeSCLJM5jdOOMWPGFFRiK7Yx0WKtra3NVYLU81b/lJeXu41SmqF2VVVVuTZrM+67BkrvS/92dHS4ceXn3vU3OkBeSkX9lhZeVVVVBS4CgwU/GFbtUDpHLUTLy8tde3VOdXW1e9Zqo+Shq6vLtbEYkaFx5LtPJNPeGWPyZMg/1pdUd8H0HxAQEBAQEBAQkEoMGKNajMXQinvMmDFudS9a/oILLgDg+OOPdztYP2WIzDRKAC8moK2tzZmO/d8upYT/++yzj3NSltO/djrFkMlk3O7ET1uTZmh3KrPC+9//fgBmzJjhdncyp1RXVztHcLVLLgD/8z//43aNYk8vvPBCx7SrYpmCtVatWuWYJJkyM5mM28XNmDEDyDl2r1u3rmiQykBDu/HXXnvN3ZuCqjZt2uT6RuyAnntNTY3b3fumG8l/Ui7KysrcONT1X3zxReeEL+tE2pFMq5Q02foolq7Ot/L0Zr1IG6IoKmDMfYbPT9EH+eyR/vvuMH6Nc11fkIuNn94rjYyqrEoyT65evdpZRTSu/fSGflo7sGyh9JKOqa86OjrcNXT9vgTypQHJCneQe86+ZUavNee0trYWsM7SSZWVlc4almb4BS+kK33XQLVHcqFnWldXV1AUxNcf/nXB6loxzL4Lm+aoZNXFgYb/e8lCFbr3rVu3umfuFy5IssaSH8jNI74u6WltVV5e7vSwfvOtt95y/S8Lp9jZvhTQCIxqQEBAQEBAQEBAKjFgNFyxhP/asR1zzDF8/vOfB+D8888Hcv4jkJ9mBGxJL+1gtEvQbn/mzJkFu/y0s4tJjBs3Lq98KOR2Nc3NzW4HInR0dBTsStJU0rAYbrrpJgC++tWvAjmn7Nra2oIAsiiKHEuS9Bc999xzCxIY33nnnQVBQ76vrxgnyd8ee+zhHN61a9Q9bNmyxfm7DibkU9fW1uZ8CGVZ8NMJqQ1+2hPthP2Ubcm0MkJ5ebljEOWDuGXLFsc6JOtCpxXJ2uxCseT+fglf6RafUS0Fnzs94+7u7oKiDOvXry9gRH0/ZL3WOKivry/wSfX7IxkkkclkUu27rDLa8mOfPXs2r732GpCTD40Hvx2+LCTLfEufrlu3zpXynjNnTt7vpR3SB2VlZQVjwk/Bl2RZt27dWuCz7gcfKbA1zfCDqSTPWjP4wWLJgMuGhoaCYO5MJuNiGXRdWa8OOOAANxdL15aXlzumdrCLQ/iMqm9B6Ql6rn4AbrEUo31JyeZbsXW++qasrMwFLqtP1Pd9YZ0HfEXnN1o3mMlkXJS2JmhNnuPHj3c13GUOPe6449xC4uabbwbg8ssvB/IrKsjEG0VRSZj8hREjRjjT7ttvvw3kotEXLVrkFixCR0dHnukXehfGocYPfvAD7rnnHiBnftdCo729Pc/JG2DFihVu4+LnhQO499573SJU6OzsdBOToBrG/rnKTfvmm286OfrBD34AkJeb9fTTT9/Blu449Dz3228//vznPwO5DAZdXV1ucSIlKlnfsGGDUzaSgb333rtg8EuJ+GZOKaY99tjDbRJKpY65zLB9ze6RNOf5KIWFqhYdURTlRdCCXVgmgzZ8E50mDUUlt7a2ukm4WMUubRIVzDpq1KgCM2KaoHzACqCEXPRzcsHQ1NRUkEPWr9YmHaQ+27Bhgxtrn/70pwe0Hf0NP0AomStVrnR+hhU/OC8Z/Cr9U1dXl5eRJu3o6OgoCDCsq6tj6dKlQH4eUbBjqVilS+mbJCm2ePHiojozafoeLPjEjp+1IHlPSTnv6urqMeern9veJyCLkQRCMqC1s7MzL1MN5NZ8fakIGEz/AQEBAQEBAQEBqcSAM6o+u6lV+dq1a11FqmSaoDVr1riVvXYxTz/9NMcffzyAM8vefffdgA2eUQDNb3/7W6A0AqiS8FNbQI7t+sc//lHAqFZUVLjcY341jLTihBNO4IknngAKd3mVlZUucErPG3KMoczT73nPewC4//77HaN09tlnA9YUpzy7F110EZALSvPNpOqzsWPHOjOhmNef/OQngHUjGAqI1evu7nayoD7YvHmzY7f8lClgd7ZiT9XWkSNHun5OptspLy93Y03j8ZhjjnEMgxjbtEOuIzJn9xQ85h+DHIvmM6ylEEzlm8uSFgWftdFz9ytN+RUBda1kP/jprZJs/Lhx45x+8QMs0oJk4E9lZWUBk+Wn60qm8vI/EztWzEVEKJZyMY3wn5WevUzeYlv32msvlxZQZtoxY8bksbH+91etWlUS86ueZ1VVlbNO+s9MKTDVziTDCIVjyYeY1dbW1gJZ6O7udvN4saDygcT06dPda92D7r+Yu5iPntwV/Op2OwOt9TRXy/3zi1/84ja/GxjVgICAgICAgICAVKLfGdXe0kKJRW1vb+ess84CciyoWJ6pU6c6f80lS5YAMHfuXE477TQg5/wrH5u999570P1ABgLJBNWCzzIKxpiC9DK9pbMaahx22GHu/rTL1zNesGCB852Tv+i//uu/FiSmVj3gCRMmuN2dmNXa2lonP9oV6vpbtmxxO0ox09dccw0/+tGPgBwzn2SoBxti0MeMGeOsB2IC/Gernb/6ZfLkyY6NFSMyYsQIJ0fJilaVlZVubIrJHj16tPssGaiTViR3/T4boD4q5red9L2rrq5OZbWlJOS3v3Xr1oJn1Nzc7AIMk0yJn6xfbLnfL8nCAN3d3QUMUlVVlfORHex0O31BsZRjGk/F0m9Jb+hYZWWla1cylU8x/7m+BJakAfI1bmpqcnEA6hfpu2w261hFtaujo6OgEpOfIF8MrFIqiiFLE9TOlpYWZ6HyoeBeWWSK+bCLkfTnW1/H6n1yrdPQ0ODkSXPbYOHoo48GijO5zz//PACHH364kw0xy8cee2yqrQSlMeICAgICAgICAgJ2OfQ7o5rcXfg+qmINf/jDHzrGS75Py5YtA2wkmHxDxICNGjXK+VeIZVWkZ3V1tYuUL2WoPQ888ACQ882U76CPFStWuB2b2q6Sd2nFhRdeCOTSU2m3OmXKFB555JG8cz/wgQ+4dul5i4n106qICYBcf4kl1O5w1KhRLrH91KlTAdvH2hE/+uijeb89VP5nZ5xxhnsttvAb3/gGYFms5557Dsj1m1jW2tpad7++zBRL7A2WLRFLol11TU0N1157bf83agAhhk/sqfRDeXm5Yx+LQf6dGj+dnZ2OGUozxIA1NzfnyT3YIiiKeBdbqOdfX19fkNrPL7sr5l390d7e7uRIaGxsdJadhx9+GICPf/zj/di6/oGfsFzyIbbcZ1Q15/gWPsmPGDNdK5lhoZRwxBFHAHZO1fPV8xa7boxxOlZtz2azbgw99NBD7hqQn8ZI83kaoft/4okniqZulLVK//sTr7zyiutf+WXOnj27339ne3H44Ye718qq46cFTTMGNeGozLPPP/+8UyB6gJo0W1pa3GSiyXnNmjVuASezlRTnyy+/7AJnfJRSZSqAU089FcgtyNRXxUxP+++/v3OFOOyww4CcUkordL+/+93vAPinf/onIFdpzEdtba1zCvedw/sLfvUlbYo0kaUhzZfu4Zvf/CZgJ00FDGrj4ldmS5pjKysrnXKWCU/m4traWpfCRJsjBW2VEi6++GIgp2jV3hNOOIGf/vSnQC7wcty4cS4d2Uc+8hEArrvuOsCOpWOOOWbwbnwH8e1vfxuwelL5IIUxY8Ywb948AK6//nogF5y3detWt5jXAreiosKZtrVh0zg755xznMwIF154IY899hhAQWBnmuBvMN/3vvcBuXlCLj3l5eVu8SCyxM81q4WcFrOqXOejVOYUBRy+/vrrTh608FRu2PPOO8+l8lJ1v9mzZ7v5+b777gNyVexOO+20IUnft71QlagDDjigaG7onoKc/M/955xMt+QjKQ+nnnqq2/wedNBB23nnAcUQTP8BAQEBAQEBAQGphBns9AkBAQEBAQEBAQEBfUFgVAMCAgICAgICAlKJsFANCAgICAgICAhIJcJCNSAgICAgICAgIJUIC9WAgICAgICAgIBUIixUAwICAgICAgICUomwUA0ICAgICAgICEglwkI1ICAgICAgICAglQgL1YCAgICAgICAgFSiJBeqxpglxpiThvo+AgJKAWG8BATsujDGTDHGRMaY8vj9o8aYS4b6vgYbxpi5xpg5PRzbxxjTMsi3FNBH7PRC1RgzyxgzzxjTZIzZaIx50hhzZH/c3HBCvFhoN8Y0G2Ma4z77tDGmJDcLAwFjzAXGmL8ZY1qMMauMMfcbY2bt5DVTpZTDeMlH/Kz1l43HiN5/bKjvL00IOmTbGO46xJOBFmPMGmPML40xdUN9XwOFwdIPURQtiqKo137saaFrjDnOGPO4MaY83hBM6a/7GgwkZGqTMeY+Y8zkob4vHzul4Iwx9cAfgf8DjAEmAdcAW3f+1gYe2mEOIs6IomgksDfwPeAK4MZiJxpjygbzxoYaxpjPAz8GvgPsAewF/Az44FDeV38ijJdCRFFUpz9gGXaM6LNfD8Y9bC+G+B6CDukBu4IOiXFGPF4OB2YCVw7x/WwTOyqL26sfBgLGmMw2NoMfAP40GPcygJBMTQDWYOeo9CCKoh3+ww6Sxh6OzQHmAt8HNgGLgVO94w1YBbsKeBv4FlAWH5sGPAxsANYDvwZGed9dApwUv54RX/uj8fuJwO+AdfHnn/W+dzVwJ3ArsBm4ZGfav5195e7Z++woIAscBPwSuBYr8K3ASUBV3H/LsMJzHVATf3csdtHTCGwEngAy8bEr4j5tBt4EThysdu5g3zQALcA5PRyvwk5AK+O/HwNV8bHRcT+si+Xsj8Ce8bFvA93Alvj6/z3E7QzjZfvHyLeA3wK3xfI8B6gGfur1xQ+Byvj8S4BHve+XAxEwJX5/OvB6fK0VwOe8c88EXorH1FzgIO/YCuCLwCvA1iGSn2L9E3RItEvpkDwZAP53fL/Jz68Gbo1fT4nHQHn8/lGNZSxZdSWwFFgL3AI0xMfuBy5P/P5LwIfi1wcAD8ay8yZwrndegSwOhPwXOacW+A1WFzYCzwBj42NzscTAvFiuHwDGxMf2BSLvOnOBbwLzgXasDvLl4MfeuS8Dh8TXjeL2tgAfjo9/GlgQ39NdwIT4c+mmf8Hq3vXYzWdmiGXqNOCt+PUHgBew+n85cHXiuxfFsrMB+FpfntEO3eNONrA+vsGbgVOB0d6xOUAn8E9AGfAZrIIw8fE/ANcDI4BxsUBd6gnN+7HKZXfg8YRgLMEq4cOxCvh0b9A9B3wdqAT2ARYBJ3uDtxM4Kz63ZqiEwft8Wdw3vwSagGPje6sGfgTcg2XfRgL3At+Nv/dd7KRTEf8dBxhg/1igJsbnTQGmDabg70DfnAJ0ESvSIse/ATwVy8nuWIXwzfjYbsCHsQpqJPA/wF3edx9lEDckYbz07xjBLlQ7gDN0D1jGbF7c1nHA08BV8fnbWqiuA94dvx4DHB6/PhK7kDsy7v9PAAvJLYBXxH2150D3w/b0T/x50CG7jg5xMgBMBl7DLqjyZIO+L1Q/gV1E7QPUAb8HfhUfuwh40rvmgdjFXxVWDy0HPh6PscOwC60D43MLZHGg5D9xzj9jF4M18TieCdTFx+YC/wCmx8/6CeBb8bFiC9Ul2I19RdzGucCcxO9NBpbFr/N0TfzZbOwG4FDsePwZ8HDi/Iewm6W942cxZ0f6p59kqhY7P90Svz8eODh+hodgdeRZnjy0ALOw88f3sfNFuhaq8c3OiIVyBVZR3IM1u8wBFnjn1cYPZXx8fCuewgc+CjzSw2+cBbyQ6Nhr4t883vv8aAmN99l/AL/wBu/jgykExYQh8flTwFfjPrzF+9xgd2bTvM+OARbHr78B3A3sm7jevvHAOAmoGIq27kDffAxY3cvxhcBp3vuTgSU9nHsosMl7/ygpmWTi+wnjpee+KRgj2IXqw4nPlgKzvfcfUN+x7YXqyvickYlr3kC82E3I3bHx6xXARUMsO0GH9Nw3u4QOiWWgBbtgXIpd+NQkZYO+L1T/ClzmfW9/7GKjHLtobwX2jo99G7gpfn0e8ETi3q4nt2HMk8V+bPu2Fqqfwi4oDy5ybC7wZe/9Z4E/xq+LLVS/XuT7cxKfXQpcH78utlC9GfiO974ey8zu6Z1/UuKe/jyEMtWJ1ZEF/Ref+2PgR/HrrwO3ecdqsaRCvy9Ud9oJP4qi16MomhNF0Z5Y89PEuDEAq73z2uKXddidQwWwKg4KaMQK+TgAY8wexpjbjTFvG2M2Y02PYxM//WlgXhRFj3qf7Q1M1DXj634FO9ELy3e2zf2MSVjTCeTf2+7YB/+c15YH4s/BmnwWAH8xxiwyxnwZIIqiBcC/YRXV2rgfJw58M3YKG4Cxvfj+TcQqZWFp/BnGmFpjzPXGmKWxrDwOjEqrf14YLzuE5D0Uk4dJfbzW2VgT/7I4SObo+PO9gSsSfTEhcd009EUxBB2yC+kQLKM1KoqivaMouiyKovaduFaxfikH9oiiqBm4Dzg/PvZRrFsR2PFydGK8fAy7sRYGdLwYY8oSwVYTsQvkh4A7Yn34vYRMrPZet2H1a0/oy/2fRu/+qXn9G0XRZqx7SU96xcnlIOOsKIpGYVnfy4HHjDHjjTFHG2MeMcasM8Y0YecRzS0T8e49nrM2DMTN9Wu0aBRFb2AF5aBtnLocyxCNjQfcqCiK6qMoekd8/DvYncbBURTVAxdi2QEfnwb2Msb8KHHdxd41R0VRNDKKotP829yx1vU/jI32noTdqUH+va3H+sa8w2tLQxRHJkZR1BxF0ReiKNoHO/F+3hhzYnzsN1EUzcIqkwj4z0Fq0o5iPlYezurh+EpsW4S94s8AvoBlAY6OZeU98eeSl9Q87yTCeOkzkvdQTB7ejl+3Yhdngj9xEkXR01EUnYld5P8RuD0+tBy4JtEXtVEU3dHLfQw5gg5x2CV1iIde5b4XFOuXLqyJF6xv+EeNMcdgFzGPxJ8vBx5LjJe6KIo+411rQPstiqLuyAu2iqJoZRRFHVEUXR1F0QysSfps7AJ6h36it/fGmMr4Nx7q4XxI9K8xZiTWzP+2d44fYe/L5aAj7tPfY1nfWVh/33uAyVEUNWBdhTQuVmGZYQCMMTVYN5p+x85G/R9gjPmCMWbP+P1k7K7rqd6+F0XRKuAvwA+MMfVxVN00Y8x741NGYqnoJmPMJGwQQxLNWL+k9xhjvhd/9gzQbIy5whhTE++4DjIpS/8Tt/l07CR5axRFryTPiaIoizVH/sgYI+ZskjHm5Pj16caYfY0xBusL1A1kjTH7G2NOMMZUYR2/27HBFqlFFEVNWDPC/zXGnBUzHBXGmFONMf+FVZZXGmN2N8aMjc+9Nf76SGwbG40xY4CrEpdfg/W/GnKE8dJvuA34ujFmrDFmd6wTv+ThJeAQY8zBseJ08hC38QJjTH0URZ3YPtHYuAH4Z2PMkcaizhhzhjFmxOA1q+8IOiQfu4oO6QUvAufHbZ4JfKSP37sN+JwxZqqxaa6+A/w2iqKu+PifsAutb8SfSw7+COxnjPlf8W9WxGNnRv81afsRy+1Bxkbpb8aasvtLdpNy8F7guSiKWsEu8rCMon/ObcAnjTGHxOPpu1iXiRXeOV8yxowyxuyFNf3/tp/ud7sR674PYhfTr2PHxsYoirYYY44CLvBOvxM4wxjz7njRfjWFBEm/YGcZ1Wasn9vTxphW7IT7KnaHui1chHXA/TuWCr8Ta2oD6093OFZ53od18C5AFEWN2CCSU40x34wF5XSsj9FiLKPwc2xEaBpwrzGmGbsb/So2WvnjvZx/BdY095SxJqmHsDt/sA7hD2EXKPOBn0VR9AjW0f172LavxjJH/9H/TelfRFH0A+Dz2AjUddg+uhzrGP8t4G/Y6MpXgOfjz8CazWuw7X0Ka9r08RPgI8bmh/vpADdjWwjjpX9wDXZB+ipWJp7GTgBEUfR37GT7KDYS+fHEdy8Glsbj6ZNY9pkoip7CBiRdi+3ft3QsZQg6pAfsIjqkJ3wNm/1jE3Z8/KaP37sJ+BV2nCzGbkz+RQejKNqK1Scn+deM3QJmY90CVmLl5D+xsjOUmIi9383YQLOH6HtfbAs/xrLLjcaYH1I8LdVVwG/icz4URdED2EX+H7AM5F4UMrz3YjcaL8Tn/bKf7nd7cK+xBQ82Y32RL46i6DXgMuAbsc75OuAsTPHxf8Fulldh9chaBiDdoiKKA7SkPtgAACAASURBVAICAgICAgIC+gBjzFvYDCpv7eD3y7GM79Qoipb0570NBWJGvhGYHkXR4v68dqhoEhAQEBAQEBDQRxhjqoEbd3SROlwQu0fVxi5S38daK5b09++EhWpAQEBAQEBAQB8RRdGWKIrSHmA4GPgguSIa04HzowEw0wfTf0BAQEBAQEBAQCoRGNWAgICAgICAgIBUoqfkyH1FqdOx/Z1KYaf7o73d5m6+8847AXj44YeZOnUqAGvXrgVg3bp1TJhgA773398G8H7wgx8EYOLEncoVnLr+WL9+PQCPPGLT9y1atIjKykoAli61eZQnTZrE+9//fgDe8Q6bWrSioiJ3E7HVwGbh2S6krj+GGKE/8pH6/rj11ls55ZRTABg71ubpbm1t5Q9/+AMA732vzXA2efLk4hfYPqS2Pzo7OwG48cYbnY5obm4GYNasWdTX1/d8E0F/9BcGvT+6u7vJZCwfV+z5NTY2AvDFL9qMfjNnzuSCC2wGJsnHxIkT+elPbbKHBQsWAPCjH9l01GVlO1UTYiBSOQ1LGQmMakBAQEBAQEBAQCqxsz6qw3L1vhPY4f7Qjv+II44A4KSTTgKgq6uLF154AYANG2x1slGjRnH66acDOcbx7bdtoYubbrqJESN2OEf5kPRHNmvzMWvnu2zZMk4++WQA3njjDQAaGmxqz4qKCtfmMWPGANDW1saWLVvyrnn++bbq32233Za7me1nRlIjHylBavrj6quvBuA73/kOANOmTQMsQ6Ln3NLSAsB5553HDTfcAOTk4oEHbKrM1atXU1vrF/TZLqSmP5KYPXs2AIsXL6ary+ZulyUik8k4BlGM0Lx58/rjZ1PXH089ZWtpqH1z585l3bp1AJSXW4PihRdeyIUX2pS5ra2tQE63QE5vCKWmP+bOncvdd98NwO9/b1MsT58+HYAjjzzS6dbq6mrAWu4ef9ymH5Zu/shHbP2AU0891X13BzBo/eE/Mz0vMaSvvPIKGzfaisMjR47MO3bjjTfS3d0NWEsdwPz583nppZcA+H//7/8BcPTRtvLysmXLGDVqFACHHXYYwPbMv4FRLUTRPgkL1f7FTvfH5ZdfDuAmz/POO8+Z6TSAVq9ezcUXXwzAfffdB+TcAm6++ead+flU9MfEiRP55Cc/CeBcHK644goA6upypZmlgNrb293C9je/sbmdNQEvX76cPfe0Vd6SC+I+IBX9kSKkpj/e/e53A/D6668DuY2MMYa2tjYgt9hYtWqVW4Dsvrstc791q81J/eyzz7LPPjtcdCg1/SEsX25Lb2uhWlVV5SZSX+732GMPIDdBn3nmmQB86lOf2pmfT0V/LFq0iL/+9a8A3H///QDO/WH16tXOfKv+OP/8852OePPNNwE46qijAOsSUWoL1V/96lcA/PKXvwRg48aNrg1VVTYfv3RhV1eX28AI7e3t7jwt5kUEGGN417veBcDPfvaz7b3/IemP559/HoDXXnsNgNGjR7s2q1+kP8aOHcv8+fOBnG7JZrN84hOfAHLzz6uvvgrY/hGBJJ1y/PHHO3naBsJCtRDB9B8QEBAQEBAQEFA62NlgqoB+RkdHBwD77bcfYE132sE//fTTABxwwAFuR6yd4D/+8Y/BvtUBw8EHH8yDDz4I5Jgh7XyjKHK7YblLtLe3u36TCU/BIfPmzePcc88FcgxKFEU7EhhRMshmswWs8Wc/+1kAFxRQ6lD7xHDIbJnNZp18KDCxrq7OMfOSo7fesnm6169fvzOMauogk6aCRKIoctYZ9VVXV5djnZuamoCdDsJMFe6880723ntvAI477jggZ2F5z3vew2OPPQbkWNMpU6Y4JlqMu5jVcePGOXaxFFI5Pvfcc/znf9r0njJrjx492unKpPtTJpNxc4lQU1NToD9qamrc95599lkgx77LHJ5W3HTTTQAceuihgNULYj8VdLts2TLAutDJjUhBd/X19axatQrI6Q2ho6PD9Y1Y57vuustZRgP6B4FRDQgICAgICAgISCUCo5oyaOcvVujvf/87CxcuBHI75Ewmw9/+9jcA53/mp2MqVRxyyCGA9SnU7lQsshiztra2ort9+fGq38QenXfeebz44otALvBmuDOqPvPz3HPPAbkgigMOOIDLLrsMyPk872SKlSGBAuoUKCTGqKOjw7VLMpPJZNi0aRNAQeDU8uXLHbM2HPDOd74TwDFAp556qvPNS6ZlAnjiiScG+Q4HDitXrgSsPItRlqVFMjFq1CgefvhhAOfX3tnZ6Vgx6dg1a9YAlmErJcb9Jz/5iXutcd3a2up0pnxONW6AAn/NbDbrWFbpSR0rLy93ac5eeeUVABYuXOhYyLThjTfecPertjc1NbnXet6+RUZjRzqlrKzMyYf6SnKl7+g8sNYMBeyJoQ/YOQRGNSAgICAgICAgIJUIjGrKIN8o+UW9+eabjh048MADAesDIzZAuzgxq6UIpZCSn+3uu+/uGGLfr07/xQ4okru8vLzAB1FMwLhx4xyDImxH1H9JwmeL5aep/rzhhhtc6jP5QZci5FspGRAjIqYEcixaFEUFx5MprIYrTj75ZJdaR36o1dXVbrwMJ0gWRo0a5XwOlWJIY37jxo0uKl5+rB0dHS5dl+RDDPzChQsdo1oKVphFixblZUMBywLqM59JBdtezSFiEH1onPhJ8zWudK3XXnsttYzqvHnz3L1v3rwZsDKhNifTGlZVVTkZ0Pey2axra7Kvqqur3XXlH15WVsbf//53IFdQY7Bx4403usw5xSBGWP/9dveHnKu/Fi1aBPQ+13zoQx/i0ksvBXJWjiRSs1DtLcdlseAQ4Xe/+x0f/vCHC65VCkqlGJSjTou3KIpc2+UCUFZW5kzdWqB+4xvfGOxb7Tdo4pAZpqKiokBp+ulS1B9afFRWVjrlqe/p/LKyMjdpyVwsU9BwQ7FgD/WR+mfTpk3MmjULgI9+9KNAvrmwVJA0t2m8+5Vo/HRkydRkOj85UQ03NDY2urGgtjc1NeXlCYWdqr6UGshcn8lknKlWOvO0004DYMmSJW7Dr4VFdXV1Qa5NBZvttttu7vql0EcbNmxw7i1aqJaVlRVs0PxgKp8EgFzgFBTq07q6OkecaEylOZD3b3/7G4cffjiQk4V58+a5fMrFXObUD2pfNpt1hIn0jR+MpRy9Wqw3NDSwZMkSYOgWqpdcconT+cVSzs2ZMwfApdGqr693C27Bdw1Te5NBeT7UN01NTe613I7e//73O9c7Qak07733Xhfw3BOGN7UUEBAQEBAQEBBQskgNo1psl5o0O/ifidZ+/fXX+d73vgfg0mb0tuP1U3Gk0QT8b//2b0But1FfX1+QPqS7u9vtkJVseMqUKYN3k/2MxYsXA7kdWTabdUygdrf+Dq7Yc9NnxY7JvKsqNarqNdwgufflX6lqtCMeM2aMYxEkY3fccYdjRfyCCmCfRbHrDjWSY6IY26VzfKtEEqWQcmhn8Pvf/94FdMg0XlFRwcsvvwzsUBGM1EJMX01NjXstllUYP368C9A85phj3OeSG/WRgmGmT5/uWHfppDSjqanJWaY05ru7u12BB7XFD6DUs9dnHR0d7rWOKYjIGONYZ81BaWZUN27c6Nw6xo0bB8DPf/5zF1goFlTtbG9vLyiA0NHR4fpSMiA9mclkHPvuB3HKejdU+NKXvuTG/Yc+9CEATjjhBMDOtxofPmsq1jyp57u6ulz7pS/0Pf8zn4lWH8rd6L777nPyMnfuXADe9773AXb+2ZYeLn3tFBAQEBAQEBAQMCyRGka1GIqxJBdddBGQK4+41157uVRNX/jCFwD4r//6rx5T7qSdOZgxYwaQ8z3dunWrY8B07/5uRjsX+R2WIpRse/To0YDdmSUdu31fxCTDl8lknKwkd8N+8JXKzQ5XRtUfL3LmV6k/MQDt7e2uL7Xb3bRpk2Nh/vKXvwDWpwjSO16SQSFqu+/fLB2xZs0a53eXlKvkdYYbFi9e7Fij1atXA1a3vP322wAudZv8+EoZ8rErKytzzJf8NXVsjz32cLpVPovyWYWcfIhxPu6445yPeykEH8ovFXJM1/r1691Y0Nzh69BiFrtk6jrp1Q0bNrgAHDGVSguWJuj5TZ06tSBFWW1trWM8NedI/0VRVLDu8PtD11KfjR492lns1PfV1dVOjt544w3ApgUcDKiYRVlZGWeffTYA3/zmNwG45ZZbgPziHnq+vn9qcu3U3d1dEAsAhUyqWNeqqqqCOIEpU6Y4Nl99qWJFp5xyCr/97W97bVdJLFQht5iRspWJZu3atU6ByKQzYsQIt+C74IILgNykNXHiRE499dRBuPudg+8Qn3zoxhgnHKVgjuoN2Wy2oA57U1NTQX3y3szOxcwGfiUrDTK5GAxX+H2kACkpA42byspKt5CT4hg5cqSbdFQJ7MYbbwRwNa7TCk2g/gSs563qZMuWLXMLVR2TfAzXhar05ZgxYwoilcvKypwuUS7M4bBQXbp0KWA3ZQqCUnYIjY3Nmze7hakW611dXXlZQoA8eVmxYgWQ7oWqb2pObuT9wFQtoJLjAIq7TyUJAD8Hr66vvKNpgjboCxYsYObMmQD86U9/AmxgndooHaj51p9P/UqGyah/31Vg3333BXJVqyZPnuxcThYsWAAM3kJV+bIPPfRQF3Gv569A7c7OTkd++cG2yYWkv4HRNXzTv7+Qh9xCtaGhwc07cidYvHixWwwrwOyBBx4AbNVEnd8T0kmXBAQEBAQEBAQE7PJINaPqmx/ElmrVrxRDLS0tbrejFf6MGTNc/jyZALSa37JlC1OnTgUGb5ezI9AOJpPJuH7wd7rJ3UypQkwf5FdBSbIC2xv04gcGCAqQGG4oFhCjlCly9NfOubOzs+D85uZmZ/oSO/Dtb38bsGnPzjvvPCAXmJUGJB3/xfhs3LiR8ePHA3D00UcDlklRGpakWctPxTOcIEYpk8m4QBr1WW1trbMwiVUcDhAr1tzczBFHHAEUmqWjKHJjQqySMcbpCbnIKAClqanJMUVphq9Hk7qyWOCUzxYKvq4V85o0eXd1deXlJ4Z0pniTJXX27NmurZL/rVu3OjcwMejSH9lstiCPqu9O5F8DrNuU0mMqJdXRRx/tPpNeHSxI73/2s591ayax7WLD161b54Kv5bYQRZFrm9ZTeq6+/Pvm/mKuiGDnmuSxvffe21k0pZskU6+//vo2ZSgwqgEBAQEBAQEBAalEKhnVYoEx2h2I8RGmTJniAke0c+zu7nY7JTGv2ilv2rQp1bWb5dsi36qamhq3w9Eupaury7EBOk/9IxapVCBfOtj+9Ee+H2qSIfCvpR2y/1vDCcl+i6LIpR+R3PssiMaJdtg1NTWuj8Qwym+4ra3N+XilCdr9ix0Tm7Zs2TLXFvmif+1rX8urY+6j1H28e4KYxOrq6oI65ZWVlQUpjEoZStyvZ/zud7/byYMgmchms07+pUP9FGzqD/mxPvbYY8533g8WSRvE5vn+lMLo0aMdYzVixIi8Y8UYxGw2WzBOxH4dcsghLnhZvyNdkUb4xV38INpbb70VyFUgk5V1y5YtefMs5BeZSaYqW79+vWPv9X8ooWcxbdo0ZxVTFTr5gY4bN64gmLS1tTWvqp+Pjo6OgjiZYr796hOfUVV/VVRUuLgT+Y+//vrrgJVdrWN6QmBUAwICAgICAgICUolUMqpJhuiZZ55xUcgqhSc26IADDnAreTGrLS0tLppVq3ffxySZwihNUBJ2+Y6MGDGigC3MZDKuj7TDuf7664HSY1R78q0S61Es4X/ynGJRqoKffDiNaVT6A0kW+ZVXXnE762Tpv66uLjd2dGzEiBHuM7FMYioPO+wwzjnnnMFoxnZBjKDkX6yhMcb5X/qR7GKUkyUTtxVtWqoQo97R0eF0nxjBuro699pPZ1SqEBujOITa2tqCrA7SAZ2dnQU6xRjj2CT1h+aNKIqcj5+OpZFR1fOuqqpy7RKrPHHiROcfqHRMaoufnqo3HSvdsueee/LQQw8BuTGXxswZ/rNN6se2tjbXD8nywl1dXXn+/GBlR22U/vDjRqQ7/RRXSQxWsRR/PCsLgWTZj2vRs/PT+kkniD3fXt3gZ0JIxhC0tra6/kzGBVRVVbk0eT1hpxaqA1X7WB0lh+eFCxfy1a9+FYAHH3wQyFViWr58uRMafdbZ2ekCZ0Q9J2sdpxX//d//DeRodP9+/ddSLlJGv/jFLwC46aabBuU++wvNzc1OfnzhTqal8h39kwEBvpkqmXalrKysaOBAKSGKooLa9r3h/vvvd2NIcqSNjzHGmWB0zdbW1oI+lVIZ6gorPUFyr0WKxndXV5czdRXTS8nPkjXvhwtkCm5tbXXmTclEbW2t04/akJQykpuQ+vp6t3jQAs6vmpMMBvL1hzZ4WtguW7bMyZMfmJk2aHz7wbfaiI4dO7agepQ/Hxabx5OprfR/w4YNBYFWaURva5KKigonF0o5pkWZHzjlX8t3HYH8KorFNi5DVcVPz3LFihUubZhcEvx2SZb1v1j1Pj9Puf8ayHMTSI6jrVu3FrS/rKzMLYSlu0UcrV+/3umonhBM/wEBAQEBAQEBAanETjGq/blr0E729ttvd0yqdmzTpk1zOxixptohbt261SUr145h2rRpbNiwAcgPSvLPSRvE/CZNtn6qJp9B1C5GuzmxZEuXLmXvvfcevBvfSWzZsqUoS5hkPXxZ84OowO7WkmypvztMmqZWrlyZV52jFNAbk5rcCf/iF79wye7FHIht8gMDxCJks1n3mRg2yZMSVqcN2pVLTtSWKIoKWFJjjBv3GleCgi6HG/x0L0n57+7udp8NB0ZVJn8FPdXV1bn5QYGzPuPuVxDS9zWHCNKru+22mwtASrObhBhynxH0A8qKFY2BntNTJc/Xed3d3QVFA4wxrm9KId2bzyLrfsXANzQ05BXG0PmC9I1fECIZeDaU0HpnxYoV7r6l87QmymQyjt30gynVtmIJ/4tZI5Pzjs7p6OgoSIE4YsQIJ4/6TGPzxRdf3GbqyMCoBgQEBAQEBAQEpBL9Ekzl+9D15hfmH1MQzZ133gnA888/D9jV/P777w/kUjW98MILbgej1bgSVXd2dualoAC7E9DuWsmu5YuxePHighQTaYB8b+UPVSwRt88EJPtUabvuu+8+LrvssgG/3/7Cpk2b8tKKgW1Tsn3+bi+ZTDiTybjPxEjLWR4K2chXX321pBhVf9wka3D7kC9SNpt1LJC/ywXLuCR97crKylxQUTKtiMZP2iCWoFi6FCX3FxoaGvKCE30k3w8XSLfV1NS4YAq/RrsY8+HQfrFIfvotzR1qn18C0vcvBCv/PZWDnDFjhgtKSTJIaYJ8/Kqrq91YECvc2tpaEHiq9vm+hj6D5utiyOmRhoaGgu9GUeRkqxQY1a6uroKSyrIsjBs3zrWvWAxEspRoTymdhgp+7I7GveAHYOv+ff2Z1KW+VbO3wjvFLL7JIMYtW7a46ymdnO715Zdf3qZ1vl962a+v3Rdce+21LrpdCyy/UopM//41Zd6RoOj8VatWuUGqjlq/fr3rIJl0RDN3dnY6ClzVq9KAJ598Esjd76xZswB4/PHHXdtVSWvp0qVuAXHkkUcCNuAM4I033hi8m+4HbN68uWBRunXr1rxKKJBfScU3+et7Wlj5A0P/k+betWvXDlh7BhrJcXb33Xdz/vnnAzkztr8IT0Yqd3V15eUHBKuQtMDXZ1LWadrM+UguVH0FmqzJ7mfOSJrBh8NCrRikM6qqqlxfabFeWVnpJurhkPVAuk8YO3ZsXuCYj7KysoJFmO9KJRcZ6Q9jDEuXLgVyC2K51aQJMl2Xl5e75yyyprGxsaB6o9DV1VU0e4of8Q45eVK9eMhfzKbVpa4YtmzZ4sZ9cuPv908xk37SpD1UQVM9YcaMGQDMnz/fbUaT8F1/imVsSC5Gu7u7C8gk/7XmDD8vsaDrl5WVOT0kWRIhee+997LXXnv12q70bhEDAgICAgICAgJ2afQbb53cfWj3+fbbb7NixQrAsoNgzf6HHnookGNuVP/VT5+jFfrWrVvd6l07IbGoEyZMYNq0aQC89NJLgN1dqtKEzhdjVFNTk8q0GjIvLVu2DIATTjgBsIypWFLVMM9ms7zzne8EciZuVXmQ60CpYPPmzXk1t8GyytrBJRlBn1H0d/TJvKtipn0zhCDGYSiRTOlRbCdfzKz05z//GYDLL78csBYFVY7yc0Fq16q+9c0ySRNmTU1NQa1l9beYmrRBTKBkwH/GYpKE+vr6gjRFyUCD4QblzfR1p8ybmUzGjas0p1zqK0488UQgp+MzmUzBuJJ8+8f8OSvZD5KXd7zjHU4npznwTu2rqqpyaYnkAtPZ2enamgwIi6Ko6HyYDFiVPtl3333d2NM1J06c6Po+6YaXRrS1tTm9KCbYr9SW1M1+9S7BZ1bTFEw1Z84cAH7wgx+49YIsyH6u7KQe9OfVZEqyYindijGrvqUu2V/t7e0FuZulj1pbW3nPe97Ta7sCoxoQEBAQEBAQEJBK7BSjqpX01VdfnVc3HHI7Dr8SiD4bNWqUW8End7Ll5eXumFb0Ptskpla7ukMPPdTtIOU/c9BBB7mVv3Z6er9+/fpUphmRX6F2Haow1dXV5Riw1157DbC7WjFBxx57LAA33HADkPPFLUWo7cX8UH2oP3y/omTFId8hXteSX/O2UmEMFIqlgumtfUJrayvHH388gKuzrff777+/25n6LJl8y9QPGpfV1dV5VXr0vWTtZ+2IxSylDclgsd7Q0NDg2lGqRR+2F8uXLwes7Ch4QXpyt912c8FGaS3osD2Q9cxH0vfQr1SVZNOSDDzkdMSMGTO46KKL+v+m+xka576/vvplyZIlPfaHz7b6SLKKvsVCVjv5+ldUVKRyTu0JZWVlTgZ0337QdTJwyA8+Surm7u7uVFloZ8+eDcC///u/F6Qp0/u2tjYXeyB52LJli5ObZDCVf54vK8niML6fcrEUZmJ2xfRK99TW1vKpT32q13YFRjUgICAgICAgICCV6JeE/xdffLFLL/Xmm28Cud2W7/um3cvmzZvdjlUrbn1v0qRJLsm4VuhdXV0uib18rw466CDA+uyJLVFUvO+DKF87MUrl5eWpjHQ944wzALjrrrsAXKTpAQccwKOPPgrk2lJfX+8im+X/q51OGtvWG5YuXeoYDbWhsbHR7fiSvi7d3d1FGQA/VRXkmHp/V6hrzZs3rz+b0Gf0JUJ0w4YNPPfccwA88MADANx2221uF6qd56JFiwCb6iO5y6+qqnLtVj/IsuD7qPpjI8my6vstLS0ulZzuIQ2QnBdLfZcsaThy5MgCJlXnq93DDb7uTNa9h1xBh+HKMCfT4wi+7pAlorOzs4BFK7V+8Utua86V39+TTz5ZoHt8vdhb2q1kcZCqqirH1GruqampKRo9nlbU1tY6vah+Uf+1tbUVTWRfzK8Z8tM6pQlPPPGEe/7F2OCknPuZdorJfrE2Jq2CfmYJP6OGzkkWadE6b+LEidv0bd6phaocrBsaGjj33HOLntPU1OQoX53f0tLiBlPS7NLZ2ekChHzH3WS1CHVKU1OTc4TWsYqKCqegZCJXJ/lVNNKEd73rXQAcddRRANx0000AfO5zn3NmTpl32tranKP81772NSCnlL7whS8M3k33A9ra2lzNXz93p9qaTJ3iB0cla3zruH9+e3t7gfla+duGCn/4wx+49tprgZz5Q2PEVwgavFOmTHGBHH//+98BXI685ubmglypxZSnzvGVla+ktYjXOPOVlfovTQvV5ObGR3KhWldXV2DCTGvarf6Cn89Si3oFoNbV1bmxNlzTcyVdWfzgEb3WPNDd3V20Ah4UDz5MI/wNvV4r/Q8ULkyKbfCEYnms9b0lS5Y40ujhhx8GrI7envSUQ43W1lbnQqh0ThoPxaoc+umZkibwTCaTyoDEhoYGt4ZQgJXmjGK5tDs7O3uV894Cpoql/ivmuiddo3WgdPBf//rXbbYn/SMwICAgICAgICBgl8ROMapiKZcuXepM1VolKz3Q6NGjHYPlr9hlwlcQlhjQTCbjPvOTvSdX9H6y2WQxgLa2th4dnLu7u50z+DHHHLODLe9/LFmyBMhV6Prwhz8M2GTW+uxDH/oQYE052h197GMfA+Cee+4B4P777+fUU08dtPveWfzlL3/h+uuvB+Ccc84B4JJLLuHuu+8GcImAi1VNEbLZbIEZwq//LSZJpr5kEvDBgkz6V111lXPeV/vktuIzOGLF1q1b51xp9JnSmY0ZMyavzj1YZjWZbsrfCWucaFftp2RJOuBnMpnUJbWGHDuQDPqAQka1tra24LzkOcMNal93d7djMiQ7lZWVTk8P136QlSXJJHZ1deUVutCxJNNYzAWgJ9Y1TfCDxQTpwmIwxhQUQDDGFIwXXXPVqlVFU7qlrUJTbyjmpiB96QcE+f2h9iULy/jnpQ2qVPiVr3wFgCuvvBKwazM9w95SSvkpIU855RT3XYBnnnnGme41X0lGysvLndz4FSWV5k1B77/73e/63JbAqAYEBAQEBAQEBKQSO7UN0q5i+vTpbgcmNlQs1vr1613den/1rpQp+q/dfk1NTUE5Mz+dhFb9/s5X7IA+23333d01fBZB56SxzvuBBx4I5Mo/ijGbPn06H/3oR4EcI+inJBLbqp1gKdRaTuLSSy/Ne7906dKC8m9+sJRkwN8Z67XkRDKhlDwwdEyqMH/+fMAypGIE5f+pYKeKigq3axfj6e9Qk+WC16xZU+C/nclk3G64WKJqHfNTyCUZFPV3WuVJqcaSvohQmPKrvLy8gPUotaDD7YWfli/JIGazWSdjpRY01FckffD8oJFk8ZDu7u6iQZv+99IO3wdbMRvCqlWrXNxHsdRDgl+mOlkSVcfWrl3rrD9CR0dHKuM+ekJnZ6dbD+i5nqXUFwAAIABJREFU+2k1k/qjoqLCnad5Rd9PW3oqv9CFdJ7WD1pbXHPNNa6IkB+0nJx3/LiG6667Dsj5mfrMvfpLOmfEiBEFBQUqKipcufdbbrkl7559K0dP6De+3q8q5P8P6BuSCywtYJYsWeJoei1SdtttN7ch0EJfgTfbqvCQNhQLVvDz1vnVMaBnM4tfxQxyg0fvt/Wbg4HTTjsNgNtvv93lxE1mN6isrHSvNaFWVlYWROon/wN5Dv/FzJv6nzRrZjIZd30pG/Xz5MmTeeGFF4D84IyhhhaqmjD8BUXS9Onn0lV/yPViuELkQENDg8uZqv/SGZDLzjLcIHO3nrPv2pJcfHZ0dBQEyaRp8dEX+AutYjXek8FRxYKf/HOkV/SZrtnY2Ogy7gjd3d0uIPaQQw7Z6bYMNPyNvHSF5MWfe6RDOzo63HnSj/730xRI1luQnFwB7rnnHl555RUgFxD3+uuvuwVq0jUsiqK8DBlg258kjHx3Irl7qoLmKaec0mMVwL64jQTTf0BAQEBAQEBAQCpROh7QuwjEFCkXbHt7Oy+++CIAZ599NgAPPvigS88jM48clUshlYqPYve75557uuAyPy0V5KeS6a3Sk44Vq9Q1VOY83cvcuXNdsNjNN98M5NwClixZ0qf7U3v94Kj+hO+CIuf3NEGmzCR7uueeexaYPquqqgry+vW0ux8u8NMyqc3F0peJKSllJIOcWlpaCtxhfNN4MqWZ/77UGdXOzk6XulB47rnnnKuRXF789qnffNO/+jSZtu7xxx/nxhtvBHJm4Gw2666fRiTdwsrKylw/iAVUW3x23a9gJgZVekNyMnLkyFQFJPY1sOvggw/O+592lNaqJiAgICAgICAgYJdBYFRThpkzZwI4Z+fOzk4OPfRQIOdbtv/++7sgIe2MTz755MG+1QGDX3+8GFPqpzITkpVo5Bi+evVq588rpi0NARIf/OAH8/77kL+X/AxXrVrFwoULgeI+SPIf86uviQ1Qf/i7/uSu2w9WTDrIjx07lkmTJu1YIwcQsiiI8VEwR0dHR0Egjf9ZMt3OcMeIESOcXIg1qq2tLajsVcpIMqpdXV1OfpMFLPxKXUJLS4vro6Sve6n0j++LLqucMH/+fFdQRLEjGi/ZbLagP3wLjRhEjR8/kEr6tK2trcCKkWasW7fOWRL0nP1KVdKVOmfkyJHOwqnz1N61a9cWzC8B/Y/AqAYEBAQEBAQEBKQSgVFNGRRVqB1+dXW1i7IUg5jJZNx5fuGDUkWy7On06dOdj6pK3Wkn21MalGT0vHa3J554YsFON01RmsWg9GlpTKOWFugZqtCIrA4vvvhigf9pfX29K6wgtkilE4cr/IwO0hUaGxs3bnQWmVmzZg3NDfYjipU9VdYUtVN9oOM+qqur8zJsQC7Nol8+M80Q+9fY2FigIxXtPVDYuHGjK+2cTF2VBiTjIEaNGuV8M1UWWn3W2trqMgDoe4sWLWLfffcFcnpHFokJEyYM+3LMaYDZSTPo0NtQdw79bdfZ6f744x//CMADDzwAWLOUlK0WayNGjHDmHU0+J5xwAgAXXnjhzvx86vrj1VdfBeCpp54C7MJEZn0/RYheH3744QDMnj278Ga2v7JM6vpjiJGa/kjqrSEy0aamP5LQQu0//uM/nLlXOZdPOukk5yqkCbifgstS0x9z5861F0iMeb8uvfRpZWVlgYuM/hcLxtwODFp/PPvsswDcd999zn3s9NNPt1+KooL0fsUCUfsCf9EneVq9erWrhriNa6VGPnqD0rgptdmKFSsKAtT6CQOhtIblHBNM/wEBAQEBAQEBAanEzjKqAQEBAQEBAQEBAQOCwKgGBAQEBAQEBASkEmGhGhAQEBAQEBAQkEqEhWpAQEBAQEBAQEAqERaqAQEBAQEBAQEBqURYqAYEBAQEBAQEBKQSYaEaEBAQEBAQEBCQSoSFakBAQEBAQEBAQCpRsgtVY0xkjNm3D+dNic8t3RqjfUAp9kdv99zX9hT53hxjzNydv7uA4YxSHC8Bg4dSlY+gU3uGMWauMWZOD8f2Mca0DPItBfQR/b5QNcbMMsbMM8Y0GWM2GmOeNMYc2d+/UyrYFfrDGPOoMWaTMaZqqO9loGCMOd4Ys6IfrtPi/WWNMe3e+4/1x72WMnaF8bKjMMYsieWl2RjTGPfTp40xJUs4bC92FfkIOtWdMyj6MoqiRVEU9VpHuKeFrjHmOGPM48aY8ngzMKW/7qs/Ucr6o19v0BhTD/wR+D/AGGAScA2wtT9/p1SwK/RHPCiPw9YYPnNIb6YEEEVRnf6AZcAZ3me/Tp6fBqZmsO5hVxgv/YAzoigaCewNfA+4Arix2InGmLLBvLGBxq4iH0Gn5rC9+nIgYIzJbGMx9wHgT4NxL/2A0tQfURT12x8wE2js4dg04GFgA7Ae+DUwyju+BPh34GWgCfgtUO0d/yKwClgJfAI7iPeNj30AeAHYDCwHrva+NyU+t7w/2xr6w13v68CTwA+BPyaO/RL4v8B9QDPwNDDNO+7f86z4Xo8vcqwK+D5WUa0BrgNqerifOfH9/Hfcb28AJ3rHJwL3ABuBBcA/eceqgB/Hfboyfl0FjADagSzQEv9N7Ie+WwKclPjsW/Gzvi3uszlANfDT+Hm/Hfd1ZXz+JcCj3vfL476bEr8/HXg9vtYK4HPeuWcCLwGNwFzgIO/YiljGXgG2hvEy+Pqjj/JyVCyXB2HH27XYSbMVOKm3sQOMxS78GuPx8ASQiY9dEctaM/Am3hgawvbvEvJB0Kl9lv8i59QCv4nloBF4BhgbH5uL3djMi/vuAWBMfGxfIPKuMxf4JjA/vs/fAt3Alvhef+yd+zJwSHzdCDv2WoAPx8c/HffLBuAuYEL8uXT1vwCLsXL7PeIxGPSHd5/93BH18cO4GTgVGO0d2xd4f9zw3YHHEw97SSxUE7G75deBT8fHTok76aBYwH9D/qA7HjgYyxAfEp971kApktAfeW1cAFwGHAF0Ant4x34Zt/8o7KD8NXC7dzyK++EUrEI9Knksfv0jrCIcA4wE7gW+28P9zAG6gM8BFcB5WOUqhfQ48DPs4u9QYB1wQnzsG8BTwLj4mcwDvun16Yp+lo8lFF+odgBnxM+vBvhOfC+7x/f2NHBVfP62FqrrgHfHr8cAh8evj4zl4kigDDs5LyS3AF4BPAfsSQ8TWBgvg65PCuQl/nwZ8BnseGsCjo3bUk0vYwf4LnbiqYj/jgMMsD92PE70+mDaYLVzV5cPgk7dLvlPnPPP2MVgDVavzQTq4mNzgX8A07EL2ieAb3nyE3nXmRv/3oy4zeXxZ3MSvzcZWBa/ztO98WezgbVxv1TH/fRw4vyHgNFYlnNB8jeC/ujnhWp8UzPiBq+IhfsevIHmnXcW8EKiEy/03v8XcF38+ibge96x/fAGXZFr/xj4kddJQzbRDOf+wO7YO8ntWN8gn7H7JfBz7/1pwBve+wj4D2ApHpvnHds3FvxW8lmDY4DFPdzTHOzO3XifPQP8L6xS6QZGese+C/wyfr0QOM07djKwJH59PIO3UH048dlSYLb3/gPAgvj1thaqK+NzRiaueQPxYtf7bCFwbPx6BXBRGC8lsVB9Cvhq3G+3eJ/3Onawi4i7k/0Qj7u1WEalYqjbvSvJB0Gnbrf8J875FHZBeXCRY3OBL3vvP0vMWFN8ofr1It+fk/jsUuD6+HWxherNwHe89/Vxf+3pnX9S4p7+PEBjp2j/UQL6o9+daKMoej2KojlRFO2J3aFOBH5sjNnDGHO7MeZtY8xm4FYsdexjtfe6DZBz80TsCl1Y6n/JGHO0MeYRY8w6Y0wTlmpPXntIMMz742LgL1EUrY/f/yb+zEdPbRD+DbgjiqJXe/iN3bG73+diB/BGrMlm917u6+0oHjExlmL7bCKwMYqi5sSxSfHrieT3pb432FieeF/svibRN5yNNfEviwM0jo4/3xu4Qn0a9+uExHWT9zHgGObjZaAwCWt6g/x2bmvs/G8sg/MXY8wiY8yXAaIoWoAdl1cDa+N+H4pxUIBdQD6CTu0jjDFliWCridjF1kPAHbEsfC/hY7+tvvPRF/13Gr37p+a1P4qizcAmetazQzHnpF5/DGi0VxRFb2AF5yCs+TLC7nTqgQuxK/a+YBV25ybslTj+G+zOenIURQ1YOrqv1x40DKf+MMbUAOcC7zXGrDbGrMaaht5pjHnndlzqHOAsY8y/9nB8PdZH6B1RFI2K/xqi3iM0Jxlj/PbuRc5HaowxZmTi2Nvx65XYBVzye2Cf1WAh+VvF7kv33IpVJsL4vAtF0dNRFJ2JNb39Ebg9PrQcuMbr01FRFNVGUXRHL/cxqBhO42WgEEe8T8KyPZD/zHodO1EUNUdR9IUoivbBbmY+b4w5MT72myiKZmHlLgL+c5Ca1GcMN/kIOnX7EEVRd+QFW0VRtDKKoo4oiq6OomgGlp0+G9jR7ADJ+8t7b4ypjH/joR7Oh0T7434aTa5/oFD2VjJIKBX90d9R/wcYY75gjNkzfj8Z+CiWWh6JdTBuMsZMwjqv9xV3AHOMMQcaY2qBqxLHR2J3dVuMMUcBF+xsW/oDw7w/zsKaMA7E+t8cijXLPQFctB3XWQmcCPyrMeYzyYNRFGWxZuofGWPGARhjJhljTu7lmuOAzxpjKowx58T39acoipZjfaS+a4ypNsYcAnwSy76ADWC60hizuzFmLDaoQcfWALsZYxq2o239hduArxtjxhpjdge+5t3XS8AhxpiD44nOyYIxpsYYc4Expj6Kok6sY3s2PnwD8M/GmCONRZ0x5gxjzIjBa1Y+hvl46VcYY+qNMadjNx63RlH0SvKcbY0dY8zpxph9/397Zx4dVX3G/e/MJCEhkAQIAoksyuKCCyKuVUu1alFqPS0eq63iWrWolaMCr1qXt76tWtG6VK2ctlorimgVEVsFqQiCiuICqIRNCMEgJBAmySSTzNz3j8v3ub+5cwkJzHJDn885nCGTO5O7/Nbvs+1agNTB7s/xQCBwSCAQOD1gp0ZqghP0klX+B9qHjqn7yK52e0TAjtLfCduNIlVtdwuAg42fvw/gE8uyGgB74Qzbf9g85gUAVwYCgaN29ac/AFhoWZaZlmtSIBAoCQQCA2Cb/mek6Hx3S2cbP1KtqIYBnADgw0Ag0AB7AFkB4GbY0XYjYV/QHAD/au+XWpb1b9h+QfNhS83zXYf8GsD/DQQCYdgd4SX4g/35fowH8HfLsjZallXNf7AjQ38R6EBKI8uyNsIeWKcEAoGrPA6ZDPs6PwjYZr15sB22d8eHsB3mtwH4fwDGWZZVs+t3F8H2K9sM4FXYfprcEd8L4GPYUZzLASzb9R7VmxcArAvYZpBMmmfugb0gXbHr3D6EPeDBsqwvYatJ78KOrnzP9dnxADbsum9XwlaaYFnWB7Ad6J+EbYqq4O+yyP7cX1LF7F3nWQnbr+whAJe3cXxbfWforp/rYUc3P2FZ1n9hByTdB7v/VMNepPyf1F9Kh9nf24eOqftOGexnvxPAyl3XNT0F3wvYbeSiXef6ELzTUt0FYPquY35qWdZ/YPtyvgpbuR+AZIV3NoDPYGeWeBW2lSBddMrxI5DodqIoiqIoiqK0RSAQqAAw1rKsir38fA5sxfcgy7K+SeW57W/4viKBoiiKoiiKXwgEAvkA/rq3i1SlY6iiqiiKoiiKkkFUUW0/ulBVFEVRFEVRfIma/hVFURRFURRf0u4owt3Q2eXYVOfC0/uRiN6PRPbqfrz44ovIz88HAOTl5QEA4vHkbB/BYFBeaSnp0qVLwu+amprwox/9aG9OA/DJ/fARWb0ftbV2ju6tW7di8eLFAID6+noAwA033NDmZ++8804AwJgxYwAAkUgEADBixAj07NmzI6dhou0jEb0fiWT1frCNNzY2YtEiO21oWZmdaOC4445r13fU1NiJDpYvt7M5DR48GDk59jKqX79+HTkdID25mve6jfDaVq9eDQB49dVXAQBXXHEFDjkkMSHEzJkz8fHHHwMArrnmGgDAwQcfjBTgeU9UUVUURVEURVF8yb76qOoOLxG9H4no/UikQ/dj48aNAIC7774bpaV2xUZTNSX8f2BX4RjLsuT/VFRzc3MB2IrbTTfdBADo1atXR89f20ciWbkf9957LwAgFosBAMrLyxEKhQAA06ZNAwAcfbRdyGjMmDGikBYUFAAAJk6ciJ///OcAgDPOOAMA8Omnn8r3H3rooQBsdbWDaPtIRO9HIhm/H/X19Vi/fj0ASB/p0aMHWlpaADj9hcrqySefjD//+c8AgHDYrgo7bNgwlJfbFU+pHq5atQoA0LdvX2zebBeSampqAmD3x96926pGK/hGUb355puxYoVdcZfzzrZt2+SVimrXrnYRRMuyUFVlF9c64QS7KjcV1gULFmDYsGEAHMufOV/tAc97sq+mf0XJONxcBQLJbfqjjz4CANTV1QGwTeXdutmVAfv3tyvVHXDAAW1+t9f3ZgMOGL1795Zzp+mfi5Tc3FwZILkoBSADMRej/Hn79u3YunVrwu8U/8LnzEl21apVMiGMGjUKAHDggQeitbUVAHDjjTcCsN1FAGDx4sU4/PDDAQBPPfUUAHtyveoqOwc8+wsXp7FYDNXVdjl0vvbtm1CVV1E6DZs3b0ZhoV1sr3t3u8prLBaTse/yy+1c9/fddx8Ae8O2du1aALaLAI+nm813330HADKnhMNhlJSUAAB27NgBAKisrGzvQjXrcO6YPn06iovtAmFcXLLf5+bm4uKL7WJsCxYsAACsX79exJPKysqE77z++uvx9ttvA+jQArVN1PSvKIqiKIqi+BJVVJVORSwWE3WJzJ8/H//6l10xcefOnQAcM+fAgQMlwIQ73qKiIhx22GEAgEsvtUtoU0X1i5oKOOb6goIC+T/VZPMe0Jnf7QIAOEoqj8nJyRG1+X+ZtlR5APjqq68AAK+//joAYPLkyZk5MRfutr5w4UIJ2vjyyy8BAIcccoi08QMPPBCAEyS1Zs0aCSIZOXIkAODXv/61mC75/dFoFIDdv9h3GDDSu3dvOc6t8CqKH+FY39jYKOop23gwGJTAIfaXp59+GgCwdu1a+SwZOnQoioqKADjtn8pqPB6XMZbKbWtrq3wH1Va/Mm+eXek2HA5LwK57Htm2bRuOOOIIAI55PxaLiQWPqiw/v3379pSfpyqqiqIoiqIoii9RRVXpVJhKzsyZMwHYaTTouzlgwAAAjnN8dXW1+Bpxp7hz50688cYbAIC33noLgJOeZOLEiem+hHZDJaywsFD8q/geryUajcrOl/empaVFFNjm5uaE7wyFQqIAdGb2pIjuCa/AM7J+/Xrx9aRSSV+2tvybU4lbuaSP3OLFizF8+HAAwD/+8Q8AwKBBg8QPlcefeuqpAOzzZxtngFUkEpHvY6AV/140GpU2RgVq48aNOOigg9JynZ2RSZMmiSWGSpMqzf6ioaEBgB38w7GC417Xrl2lz1NZ5XMbNGhQ0jOMRqPi1+8ed8xj6dtZUFAg/cvviuqHH34IwL437pSHHAcGDx6MCRMmAHBiJAoLC6XN01pHRbWqqgrffPMNAPt+pgJVVBVFURRFURRf4ktFlSt1c5fa0Z3qY489BsCJ9LvssssA2LueVEWiKdllyZIlAOzoREYsUgH75JNPANgRiYzQ5C67R48eEkVPvv76awB28nS/RGyamQsY1e1WSs30cu6UVPws4Oz28/LyxKeoM5MqX2Lze2bNmgUAmDp1qowRvG8skrBs2bKU/N094R7vGJ3fvXt3SSn1wAMPAADmzp2LY445BoATjUyl9IQTTsArr7wCALjuuuuSvpvtg+ppXl6eKClkzZo1oqju74qhl1L/7rvvAgAeeughALZfI9MVkf1tTumoxcJ9PCPnn376adx///1pOMP2EQqFpA9z7DQtTsRUWLnu4OdCoZAc716bBINB+R3H1Vgs5qtYh7bgPBkMBmVO4fVwzunatauso/heJBIRv91vv/0WgLPWamlpkbk5VYqqLxeqHAy9BkXeKP7Oq0GsW7cOTz75JABnADnvvPMA2AO4mmn2Dzh5hsNhcV6neYc58EpKSmSRQXN/XV2dLGz79OkDAPIzHeL9ANtpQUGBLD7Z3s3BlwMMrz0UCok5xnwPsBexXJQoNpdccgkAJx1Yz549JSiPrxdddFF2Tm4Xzz//PADg+9//fpLZvqamRoKjmGaKqaWKi4sxZcoUAJANWG1trbR3t2tBcXGxLF5Jc3Oz3Bu61uyvuOeTWbNm4dFHHwXg9McnnnhCfu+eS/yU3q69eC1K+X+OLWxPAwYM8Lw+93uDBw8GAMyZMwc//elPATj5NjOBOe55VfFz5/c07wHHTmJZlhzHeYb3paysTAQQ3oOCggJpF35nzZo1AOxz57zgzq2fk5Mj128u1Hk8xSGa/uPxOBYuXAggdePm/rUNVBRFURRFUfYbfKOoUinNycmRShLvvfceAGD8+PFynHu348X48eNFIWAQBNWHeDyuSmonxlQsmGLqvffekyoaNL8wgGr06NE48cQTAUBSWBUUFIjiylcqRay84Qe468/JyUlyXDdVL/YdU90xTVMAElKJdJbd/t7iVgTaUrjuvfdeGW8Y+DBw4EAx8dNczuCqTGCmYKOSxZRRBxxwADZt2gTAUfF4/gCSAqFisZgk7jYDpvh/jpNsT2+88Ya4EVA1Kikpkb+xPymqVN3cZmDASU82Z84cqcrzxz/+Mek491zS2dRUINnqYl7TaaedBsBJ6l5aWiptkrXdy8vLJZiPqunYsWMBAI8//jg2bNiQ8Lt0wnZtBk5xTuAc0a9fP/m9e5wMBAJJ44d5HOcHvsZiMWzZsgWAY/ru2bOnBGn53XrL5xoKhRJcxgAkFJLh+MMxoWvXrqKkuufMeDwuwVSpQhVVRVEURVEUxZdkXVH1SmD+29/+FoDj6DtjxgzxcznllFMAOL5YJkwxtGnTJklu/Yc//CFNZ549zIAwqm7cwTQ2NoofDX+3du1a8ec88sgjATgO0Eyv0pmJRCKS1Jm7QpZ369GjByoqKgA4yf2fe+458U1lwJJfAqhMqJQGg0F53lTMzATs7EN83qZPFj9HP6pQKNQpVZ+OYAZB7I533nkHgO2DyD7Bzz3zzDO45ZZbAGRWSSXmeTOIiqpeeXm5BP5R5Rg7dizWrVsHAKJeMZF5bW2tfJ/b9xRw2hNV12HDholiS7V1zJgxWLx4MQDbR7Yz4fa/NC0ybSmprPd+/vnn45xzztmnv+l32C6oloVCIUlbxMT2bB/RaFQKTnBMWbdunbSP2bNnA3DK9VZVVeHvf/97Ji4DgONTznnADHJbuXIlANuKRh9atn9aqnb3zNwBVgxaXLVqlSjMDNCl9ZZ/C/Bvmiqm26usrJQ4Ht6L5557DoBdIMetMgeDQVlncK3FubS+vh6rV69O6Xn6ZqHKBvXdd9/JIMuBeN26dXjwwQcBAC+88AIAp6HcdtttEtVqSvB0gCec9E06W6Qm71Vra6sMLm+++SYASGTloEGDRLKn+S8SicjClAuyqqoqAHZHczuW+xnTdYMTaSwWwy9/+UsAzjXQLFFRUSETL9vVhAkTMGzYMABORRIvc0+24TOur6+XAZTXx8EkFArJBMPnnZ+fL8e5K1P9L+BeoJqLk08//RQA8JOf/AQAMHz4cGlH/N2VV14pm2WSLRMeFwqc/KqqqmRi/O9//wvAHgu58eKGnnkfi4uLk6pPAc718Hs///xzAJB+BDgRu0cddZT0J7+bMt24Fx7mz4xMtiwLr732GgBIJDNN2XwFnEVNfn5+wsLX/f2dZYFKOP6bpt+rrroKgL3RN4+JxWIy9zLwtLi4WISQESNGAHDu7fbt2+W9TMCxkPN9YWEhNm/eDMARuUKhkLiGeW1W3CKQCZ83F6BlZWWyQGXVpsMOO0zWLmwzfluocszjWsGyLJx55pkAnIp3JBgMynE08zc1Ncl48r3vfQ+AI36tWrUq5fON/1cmiqIoiqIoyv8kWZdZ3CreAQccgPvuuy/hvaqqKtkBcEfDKivhcFhqy1I9Ov300zFkyJCE7+DnvHZQfsZUg/hqmvAo3dMM09zcnOTcfOKJJ4qaSEVk/vz58vvOoKR6QfVjx44dUqWHuzuSl5cn10z3iEMPPRTPPPMMAGDp0qUAgNtvvz0DZ9wxmEJo8+bN8n+3WtO9e3dRjBnwMnLkSLlmthXucOPxuKcJeH8mEAjgs88+AwCcdNJJAOwgO8BWCGheP/744wE4+TJNqCA2NTWJukj3knRARZSKDAOcvv76a3FhobI1ZcoUUf7Y92m6PfLIIxPUd34nP0tzHdVTM5DrjjvuAGD3EZp7maaqs1aqWr58OV566SUAzjUMGjRIxtGjjjoKAPDFF18AQEJOWabfMels6qkX7vH/xRdflLRFnGdp5jXnFq88o2xPtNhlOnDTHTwaDAalzbKPtLS0yHzZ3vWAOz0Tr71Xr14SREVLR11dnfQ5d05iv0ALimllYVvmfEKamppEoaYyHAwGZb1FdfrYY48FADz77LPyHu8JXUf2ls65QlEURVEURVH2e7KuqHrhdkYvLy9PqgTCY84888yENAoA8Lvf/S7pO7lzCofDos4OHDgwDWfvjZeDPd8zd51mmgz38fSBmTZtGv7yl78ASFaBvIJlQqGQqEAMtvDyv/EL7U2azd19SUmJ7PjnzZsHwKl1vnXrVlFOqLJ/9dVXkpaDPnqmT41f/PCoGJsJpwnvTzgcxumnnw4A+Pe//w3AVjPc6ba4sw8EAp7KkN9IZUDK8uXLMWbMGABOhSkqiu+//7742jF9mQnv2z333APA9gVnoMg111yzz+e2OzhG8dVMT+X2+RsyZIhYBhgwwuuLRCJJ1pRQKCSqOt9j/zHbPP0Ur7jiCmlHHEf4yr+lzPzSAAAZWElEQVTjV9x9+c033xQViemVtm/fLkGl7nGxsrJS0uB5tUX6RNJa8/DDD0ugzq233prKS0kLZmAug2cuvfRSmVeomvG+RCIRURD5GolEpA9RcWO/oVKdKajocr7fsmWLjPV8j223vViWJe2H94H3JRqNyu+o0m7YsAFDhw4F4B0b4weoMrN/FBcXi+J85513AnDmn/z8fLln9M0tLCwUK99bb70FwAm0LCoqkj7G4ERVVBVFURRFUZT9Et8pqmYUpTvtDpCsci1atEgUA6qFr7zyCm666SYAtpoCONHdc+fOxYUXXgjAUUkygRkl6k6f01aEXDwel6wGVMJyc3Olzjfv0bPPPgvAVge40+Vuv1+/frLDoQLCn2traxPSafiF9iRtN6P+mZ6Kuzz63PXu3VvUKL6aZVLLysoAZFZdby/u+tJAss9pbW2tpC5ihPjrr78uFgh3+hU/1aFmP/Aq3eguFQskljt0F0Dwylwxd+5cAMCFF14oRR/Y5+gDumnTJlFcyKpVq3DXXXcBcHyYWQpwxowZHU5XtDfw7/I5sq17Jdy/6qqrJJUflQ/64ALONbP9mwUh2D6GDx++23OJx+PyvTyevmfuWAC/4Z4vJk+e7Hkck8HTasUxcfz48ZgxYwYAJ/l9TU2NFKOh7zMVtBEjRnimTswmbVmozP5C3+fDDjtMxkhGzHO+KC4uTsp40KNHDxmr2D5o5eTcnCk4/pulX3luHCuampqS0lHxWsyxxYTvcQ6mcmseyzlo9erVMp9wLvYbTM9HhbS0tFRSe3E85NgTj8flHtL3NBAIiI8+yzfzu0KhkMxFXLv84Ac/2Kfz9d1C1atDmQMrWbFiBQDbtECzFCtf7Ny5U9IPMZ8Xb3pOTg6uvfba9Jx8G5iTstuMsHLlSqxduxaA00hoSsrJyZHOwcZ/6KGHJqSfAewKO4CdA5KdlN8fDofl+2jOY8f85JNPJC1Fqtlb8217j+cg2tLSIhMpTXgcWB977DFZ3F122WUAgD59+iQ5x3Ng9RMc5FpbWxMqggDOBNPa2irtiQEvQOJiHHAWuOFw2DeDp5d7C+GEx8WVG/P6gcTNHoMx6Q5z6qmnynPmAMrPjRs3Tu4NN7AfffQRrrjiCgBONSIOwkuXLpVzS6fZ+5FHHgEA/OY3vwHgmObNdEmkrq5OJk5eC10ADjzwQFl0uY8xYfCMV5DUDTfcgMceewyAc/94P/26UHXPF3ty4+Gky2A7uhD97Gc/k8BL5tudP3++BPCeffbZAJxAkmg06rtUcHsaT7nR5SZo0KBBsmjhuGou8mjep1m/a9euYibmq7mRziRuV4P+/fsnbNpIR9MRusUlvtIFAHDml+bmZt+7V7k3EOPGjZPNF+HitKGhQcZhXhfnI8BxufzPf/4DwHaJYrq3UaNGpeR81fSvKIqiKIqi+JKsbP3aGyxj4j6e6S9aWlpEDeMu8IEHHhBFiek1SDAYlOTYmcBd0ACApFLiLv6UU04RlYsmOO5cSktLxSz1+9//HoCtorLwARU2miNbW1tFOaFKe8EFF+DFF18E4JjEWcN82rRpaVNUO/qM21JgvQKczJRlVLmYBJ2VUvLz88Ukc/nllwOwFQQqINwZUiFx/41swnba0tIi18ddrpnEnTtaWg2am5tFCWE74j1tbW0V1SPbmOZ9d7AYd/D19fVyDaa5zl0AgYrn1VdfLamZmOC7qalJAh2oJlJRXbJkiRzHIgCTJ09OStVCRbN79+4ZUUuo3nGMYKEOr6CE22+/Xa7fi7aS9fN+0G1q+fLl8rcJ+4r5XexTfsIcP3bXh5cuXSqmSo6PZp1zqopUis877zwJErntttsA2G4CHEc5D9HCddJJJ+3WCpAt4vG4KILsQ7Q49ezZU+4VzbNfffWV9A+q8Xzu0WhU3qMF54MPPpB5i9UhaeljGiQ/0dLSstdjPNcaXmMoxwy/q6mAs5bgK+BYB7gW4dhgpuRj2zaLQ1Bhp9vM9OnTE743FaiiqiiKoiiKoviSlCiqbSmk8Xhc/Di4Ct+bYA634kKfqebmZtnFcZf73HPPyQ6vpqYGgONLEolE0p7g3rIsTyUVsBWfM844A4CTKmf69OmSDoS+tSYs0UYFsaGhQfxx6czPwJHly5eLckifE9Mn7YknngDglIscOnSolD4zfRyzQVvtwtwBL1q0CICjEA0ePBjvvvsuACftBp93IBAQpZ1tIhqNippC5b2iogIAJOjGD9AHsn///uJTxPbM+9G/f39pY9zZlpSUiLpKpY27/MLCQt8oqsSrP9ICcOutt4pvNp874PiwTp8+HYATLNm3b18ZD6gC1NTUyNjDz3399dcAbP9Vlh+mWlBRUSEKFO8plajCwsKkghqphhYRwFGm2Pej0WiSYheNRqV9sK9TMdu0aZP8jipZXl5eUllV9v3q6uokRRVwnhGDqPhMqqur9zn1TKrwSv3HYg5U2fPz88UXmffWCz6DeDwufnZUmhYsWJCUKJ5t6NhjjxX1O524fXDbUpODwWBSWVAGfw0fPlyCv3iPunXrluDTDiCh/9D6xOIq69evx4IFCwA48wqVx4MPPlisEZksI2r6oHqVu+X10KJm3h+v8chdupq0ZcnobHAOZBvhvAk4CjLHz5aWlqTAXhacMUlVmsGUL1TduUFzcnISZOL2wM/yu3JycsR0RzMdf540aZJEcnJh9uijjyZEoAFOtFomBpG26j1/9tlnYi5i4FR1dbU8ZNYY93rAzAX5/PPPS7Q6TXC8zo0bN8rC1oQBAcwBycE3Ho/Loi3bC1USi8VkUnW3nRUrVki+Sy4YvvjiC+k0W7ZsAeAsQBsbG5OCXgYNGiQ1rLnwc9c39hPz5s2TBcWNN94IwMlm8atf/UoW5Zw8v/32W9x9990AIIGDbPczZ870zWLcDDB09xdG3Z955pnSPmfOnAnAngy5aGWwJNt/bW2tDJxcQPXt21dM2wwq/NOf/gTADiJgIBEnLTN4010hqm/fvml3DVmyZIk8U5pZeQ+8AqFCoZC0Z563WZmMYyHvSzgcln7lzgjwzTffyDWbYyWDVHg8z6O2tjarC1WvBcny5cslmIMbem5eevXqhb/+9a8AgJdffhmAHWDK4Dm6DFEAWLBggVTo4gLXC47H4XA4bVk12uPa4MWmTZswdepUAJD82wyg6tOnj7i3mHllOY5ys8dnHIlEsG7dOgDOIi03N1cWMgxmZV9atWqVuFrwWWSCtp5BMBiU/uzOLGS2J3PB6s6va+aX3VPgZ2fDvQmLRCJJFbZyc3NljOHx3BCbmYTcQWh7i5r+FUVRFEVRFF+SEkXV3HlwJ2Oms6CT7fXXXw/ADiZiTWW3GgQkqgGAvcunozfVoOeffz7pb5smTSpQ/C7+nM7UPNyZLlu2TMwpVPP42rNnz6Ta7EOGDBGTNZVfqoWmKYJ57mbPni33jelqxo4dC8A2BbuDJsLhsCgFDBIwK3VlAnc6EK/8mCQUCiXtwGbPng3AVkx57VSmzXrFvA+mGcJtvu3fv7+kN6NiwPvuR7Zt2ybnx+pTNAUPHz48yRy1bds2MbdRSWR7mj17tqQ8yoR1oS28TGwcI/jM+vfvL+Zmqqz9+vWTa2Y6JbZvLxoaGiQoiXXeaVFYuXKlqIv8zi5duojixH5JRTMTlJeXJwWx8Py9lInPP/8c559/fsJ7HDu9jveqSscxo0uXLgn9ibAKFxU54nVsJvFSzl5++WVMmDABgHdlJLpGTZs2DQBw9913S/WpJ598EoAzfsyaNSspcMzL1Y2uR926dZPvSjXm32Te3GXLlgFw3DqKi4vFDM9KU/369ZNnTmWZCnxFRUVSG2lsbJT2Q+scP9+nTx9RDul209raKv2Fbm2cZ5csWZJ2V7v2wmuIxWJJKSJJIBBo83zd6mlubq6ox51JUfWy2DJo2+3aZF6XO/8s4IzjvA/pGBP80YIURVEURVEUxcU+KarchTc3N8sqnDs77qwKCwvF+Zqr8E8//VQUVbc/BOCoAVQzjjvuOPziF78A4PiWecEay4CzqnerTelMTUWFqqCgAO+//z4A537w75599tmijrH6TGVlpfjcMnE/U+XQwRlI3LFQCWXwFRN8L126VNQ27nTy8vLkGVB15nnV1NRkpB6zW4Foy2fFsizxG1y4cGHC52tqauT+mQFCbH9MT0XVetSoUVIYgCpJXV2dKHFU7tjmmpubO+xTnW6mTp2Km2++GYCT2oxt4ayzzko6/uKLLxblkGnMeE2jRo3yXeUcAJgyZQoASL+hKrV69Wp5zjzvnJwcaf98trQ28P6YlJWVYc6cOQAcCwT7XlFRkfQTBoKEw2HpH7SEUIHKhDpUV1cn7ZEKjldFKgb8FBQUSHtm3/dSVHlNXgVUzCp5XrXQ+f0MQHL7rPkBntOQIUPa7MN8lqxDDkD6F33b2Y569eqVZKHyUnH5fOjbmU6uvvpq/O1vfwPgjF+cR6PRqIz1TM04ePBgefa0tPA8+/TpI8+bilpra6vMm/xejq+RSEQCpdj+cnNzxfebSd9Nv2W/FEBg+4jH4+3ymfRSDfk53k8g+1aFvcFLUWWMCtuDmcrPXUXQDJJn2+B31tXVpdxvXRVVRVEURVEUxZfs01aHuwvTp4G7J/pYbdiwIcnH57rrrsP48eN3+73c9TH9xQUXXNCmkkroF2P6CLl9KtIZocpdvJlAn9fC14MOOkjU05NPPhmAHVnM3Z7pVwrYEer0EeF9vuSSS9pUDBhlyR1PTk6O7Pr4Oe5+tm7d6pkSK9Xw+rgz5c+1tbWiflJBr66uFqWAKtoHH3wAwI5IZeolphjaunWrXDNVbd4r+nIBTpssKytLSvxOP8j6+nrfKao1NTUSTctMEbw+Mxk72blzp6jkvEdsC6kqaZdKPvroI/G1Y1tkf6mtrZW0c0yXYlmWKHuMtmbGj6FDh4rv4cSJEwHYWUDom0dfQipFZuoe9okBAwZIyjaqaWyvmSg/u3LlyoQoa8Ap5mDCcWHgwIFyXu5+ZqpZ7owAJmYEr5keyw3HJUZ+Z7qWOzEVIap4PO+LL744SRVuK03O2LFjRWV9+OGHATg+wQCS2ofXd1CJTWdZXV7Ta6+9Jr6jjNTnOFBWVib9hRaCiooKmXPc6l9LS4s8UzNTD61W7HO0atDnHfAuQ8pxmj6rQHI552xhZilg/+c1m/6rploKJFpR3Gm+AH+W3+4oO3fuTIrToNUuLy8vqVBELBZLGmN4LysrKxOefyrYp4XqK6+8AsB2sKdpiPnZOHBaliUXx8Zx9NFHe6ZAAezORbM3U1Ex9yeQHHzl5dheUFAgHZMNyqxNm0nM3JaZoiONJBNm4HXr1kkaIZqcmA7qu+++k0GXi46SkhLJ0/j2228DcCpHlZaWShAVJ4XS0lIxedL9gz9v375d3C6YiiocDsvgwgmK7WPz5s2+q7oTDodlgclgMUJznft4bpZ+/OMfA3DSuWViU9JeuDG59tprZTIzc33ylc+GxzQ0NEh74LPi73bs2CGBlzfccAMAO1fqjBkzADjBhBxk6+vrZfFqprNyp1X54osvEj6fTrxSUHkFavDczI0uxxkzByIxU1JxTObfMhfkXgtZwuo1nPS9FtCpxrKspOfhFQRy7rnnyntLliwB4OSq9lpc3nPPPQDsyXfSpEkAEheoxCsPp1eVL8Aez9IF3d9aWlpk081z4pjY3Nwsz4ab72g0KgtNbjB4/g0NDXJvObdaliVthf2E4+URRxyBkSNHAnDui9e95d8rKysTd55sjz1mSiq3C4+5iXNfj1fKKtMVgH3J3ac6E01NTTLOUgThz+baifeiS5cuCW4A5u8496YSNf0riqIoiqIovmSfFFUqd6WlpQkJkgFHLRk4cKCnpMydLusoM+ijpKQE48aNAwA89NBD8hmu2r2Cr9xUVVUlBckwiCidwVSKNx9//LEoD0yazeCvTZs2iZmVakZubq6YkLjLZ1trbGwUdYmmvq1bt4qSQbcPKm5ffvmlHMf3gsGgfIdbQamoqPCszJNNunfvLgn+zaAvwLkmk5KSElGsGdTI+59Oxaej8FpGjx4tYwRdFah+RyIROWe2hdLSUjmelhm6Aqxfvx633347ACdAZubMmaKWUoWnchuPx0XpoapSX18vihXVKbahTFhGmD5pT1AhqqmpSUrE7065BSSqf7xWd2GNWCzWpqLK8TqT7CnBPQMof/jDHwJIVLmo+jC91lNPPYUHH3wQACRF3YQJE9rV59tKIs/7nM6qZaNHjwZgj2nuAhB0nyotLRVlyzRvs62w/XM+NV0VqIKGQqGktE0cb+rr62VupzUuEolIO+Lf5vPKz89v05UkE/CczGJEbkW8rXSJeyrgwM9Sde6MiuqOHTvkGfJ63e5x5nuWZSW4FpqwLZrfta+ooqooiqIoiqL4kn1SVBngwcS/JlQiKisrRRFhqqWqqirZEXL1fcsttwCw/XC8Ap52lxbGa8X+9ttvyw6Qzv7cUdKXVkk/DHDKz8+X5/34448DcHZmjY2NsiOlymmWwWRQEJWz1atXi78rfWFaWlrks1RS2F66dOmStKNnoA6QrCh5+ahlG1MlYbvm7t3LwlBYWCi/5z3iNWcr+MULqjnnnnuu+CQz3RTHhbq6Okl9wzFlx44dcj1sR3zeU6dOTUhtBSQWwTBruAO2ukL/Vvqh5uTkiF/d8ccfD8DxgW1ubs5qcIiZrJxjWnV1tdwvpkli/zFTUXn5VfI49p/dBRLuzicznfB5r1mzRu65aRUB7OdIZY8p7eLxuFjQ7rzzTgCO4v7qq69K4N0555wDwEkF2BHc8xGV1HQG2zE4cOLEiXjnnXcAAI888ggAJ0iQ98AkJycnSfUy/W7bEwxkBpTRH5Y+sKa/ojtl0ZYtWyQVZbZwW3QLCgrkPInZrt1lVdtKSRcMBuV493d2JrwUVTMeiPfAtMTwOHfKLrMNpkpRTVuCM3bYww8/XHIX0nSRbrLdMRQbRnKbVX3Y+LmAqqmpkUmEgVY1NTWyYGEQkJmlgIszc/Dl79112xksBTgdqUePHjKR8XM8LlOVujpCUVFRQkQukJg30Y05cZCOuM5kCp7L4MGDJaKZz2XEiBEA7OfByGYz8JILTtPEyPfZBnh8ly5dkiJT2Q67desmbYCZAbp16yY5Bfm3uWHyU0YILvQbGhqSFuAkGAwmLTLN99gnuOhtbW31XIzuboFqLpxTDd0/Xn/9dVmo8jo5LuTn58tChIu28vJyiU5noCaD7pYuXYr7778fAFJaQYpizNy5c3HRRRel7Ht3B116+EqampokkIxCwYYNG6R/uU3dzc3NEgDNTXpRUZHM39zAsP8Eg0EZc/gda9askf/zc3SdKSgokOw22cK9EPdaeJrBUe5sBma1Ki+3GPfCrjPgvsb169cnZUAgra2tCTmYAbs98D33PfEK8N1X1PSvKIqiKIqi+BJ/lIxQ9kuoLKxZs0ZMVTTZclfe3Nws6id3ZoFAQFRWwh19cXGx/J87ZbOajjt9Wa9evURt4w6wqKhIlDvuHvnzP//5T5xwwgkAklOnZQvTVEUlqS0KCwuTUobwlffCD5g5fd2O+2ZKKvdzMIOd3Mpx165d5Xju/nNzc8VkTIWI7SsejyfUSQdsUyHbJ3NRsm2mq457ezHVS15LXl6eKKK8b7y3ZropvmdWXzODqHj83p5PqmE+0Lvuuittf2Nvcbe7O+64I0tnkkh+fr6kZ+NrujnttNMy8nf2Fo6dpopsBlMCTjtuaWnxVFSJu73H4/E21dbOwo4dO5JyxJqmffd1BwIBGbPdLhLM9W1+x76iiqqiKIqiKIriS1RRVdLOkCFDJFG66WsK2OmImO6EO7FwOJzk/M/dW35+vqiCVEH79euXkCAeSKxbTeXJrFFNX1QqSqZq5yfVEbCVPirEVALc9ZVNvOpsc2frt2sDEgMcqWRSOa6vrxcfRLNwiDs1iumry2vkvSkoKEhKK8Xn3trampSyZ968eUm+ebzf6Uw/lAp4nVRWTTWebSgUCiWpQFRMotGoqNUknX6oipJu2P7NNu+eX4jXeGqqgl6VqTqjj6qbWbNmyThLf253XIT5nmVZSanIODbu7t7uC6qoKoqiKIqiKL5EFVUl7Zh1gbn7ZBQ1X9PxN4lbPYpEIqKumjtEnl8m6rl3FCqMvH9UEL1Sy0SjUXmfu3y++qXu9u5wK+Nm1oZMsTfpirJJKBSSdFrMsMIUQmbNe7PcKd+nekollp9TlP0FdwEEsySoe/yPx+Oilpq+l5xDaK0xS6m6S9B2BtzK8U033SSFd5gOsL1xERw7eC8XLVqUylMFoAtVJQNkw2zo9TdpkujevbsvF6NtwQU9Tf9cyLnNtICd7onmGw7OdJNI18ZAyR7XXHMNXnrpJQDOQtMMtOKmhovT2trahPRBgBPkeMwxx0guVkXZH3AvUE33J7oA8Zh4PC4LLlNccYsd5qLUnYu4M+A2z5911lk466yzAECqIM6fPx+A7SrHYFQGWwaDQbl33ABz3GDV0VSipn9FURRFURTFlwS8nIcVRVEURVEUJduooqooiqIoiqL4El2oKoqiKIqiKL5EF6qKoiiKoiiKL9GFqqIoiqIoiuJLdKGqKIqiKIqi+BJdqCqKoiiKoii+5P8Dx95AmO8R5MEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x345.6 with 40 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_rows = 4\n",
    "n_cols = 10\n",
    "plt.figure(figsize=(n_cols * 1.2, n_rows * 1.2))\n",
    "for row in range(n_rows):\n",
    "    for col in range(n_cols):\n",
    "        index = n_cols * row + col\n",
    "        plt.subplot(n_rows, n_cols, index + 1)\n",
    "        plt.imshow(x_train[index], cmap=\"binary\", interpolation=\"nearest\")\n",
    "        plt.axis('off')\n",
    "        plt.title(class_names[y_train[index]], fontsize=12)\n",
    "plt.subplots_adjust(wspace=0.2, hspace=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.842734Z",
     "start_time": "2020-01-03T11:05:06.782269Z"
    }
   },
   "outputs": [],
   "source": [
    "# build the model using the sequential API\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.Flatten(input_shape=(28, 28)))\n",
    "model.add(keras.layers.Dense(300, activation='relu'))\n",
    "model.add(keras.layers.Dense(100, activation='relu'))\n",
    "model.add(keras.layers.Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='red'>Flatten layer</font>** converts each input image into a 1D array: if it receives input data `X`, it computes `X.reshape(-1, 1)`. This layer does not have any parameters; it is just there to do some simple preprocessing.\n",
    "\n",
    "\n",
    "+ You should specify the **`input_shape`** in thee first layer in the model, which doesn’t include the batch size, only the shape of the instances. You could add a **`tf.keras.layers.InputLayer`** as the first layer, setting `input_shape=(28,28)`.\n",
    "\n",
    "\n",
    "+ `activation='relu'` is same as `tf.activation=keras.activations.relu`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.881771Z",
     "start_time": "2020-01-03T11:05:06.844373Z"
    }
   },
   "outputs": [],
   "source": [
    "# Instead of adding the layers one by one above, \n",
    "# you can pass a list of layers when creating the Sequential model\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.InputLayer(input_shape=(28, 28)),\n",
    "    keras.layers.Flatten(),\n",
    "    keras.layers.Dense(300, activation='relu'),\n",
    "    keras.layers.Dense(100, activation='relu'),\n",
    "    keras.layers.Dense(10, activation='softmax'),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.889037Z",
     "start_time": "2020-01-03T11:05:06.884395Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "flatten (Flatten)            (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 300)               235500    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 100)               30100     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1010      \n",
      "=================================================================\n",
      "Total params: 266,610\n",
      "Trainable params: 266,610\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **model’s `summary()`** method **displays all the model’s layers, including each layer’s name** (which is automatically generated unless you set it when creating the layer), its **output shape** (`None` means the batch size can be anything), and **its number of parameters**. The summary ends with the total number of parameters, including trainable and non-trainable parameters. Here we only have trainable parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**`Dense` layers often have a lot of parameters**. The first hidden layer has 784 x 300 connection weights, plus 300 bias terms, which adds up to 235,500 parameters. This gives the model quite a lot of flexibility to fit the training data, but it also means that the model runs the risk of overfitting, especially when you do not have a lot of training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.895070Z",
     "start_time": "2020-01-03T11:05:06.891186Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tensorflow.python.keras.layers.core.Flatten at 0x15215d400>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x15215d9e8>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x15215dda0>,\n",
       " <tensorflow.python.keras.layers.core.Dense at 0x1521740f0>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.901285Z",
     "start_time": "2020-01-03T11:05:06.896999Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Dense at 0x15215d9e8>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fetch a layer by its index, or you can fetch it by name\n",
    "hidden1 = model.layers[1]\n",
    "hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.907979Z",
     "start_time": "2020-01-03T11:05:06.903675Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dense'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.915446Z",
     "start_time": "2020-01-03T11:05:06.910645Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.non_trainable_variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.928361Z",
     "start_time": "2020-01-03T11:05:06.924511Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.non_trainable_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.936598Z",
     "start_time": "2020-01-03T11:05:06.932857Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.946376Z",
     "start_time": "2020-01-03T11:05:06.938801Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'dense/kernel:0' shape=(784, 300) dtype=float32, numpy=\n",
       " array([[ 0.02448617, -0.00877795, -0.02189048, ..., -0.02766046,\n",
       "          0.03859074, -0.06889391],\n",
       "        [ 0.00476504, -0.03105379, -0.0586676 , ...,  0.00602964,\n",
       "         -0.02763776, -0.04165364],\n",
       "        [-0.06189284, -0.06901957,  0.07102345, ..., -0.04238207,\n",
       "          0.07121518, -0.07331658],\n",
       "        ...,\n",
       "        [-0.03048757,  0.02155137, -0.05400612, ..., -0.00113463,\n",
       "          0.00228987,  0.05581069],\n",
       "        [ 0.07061854, -0.06960931,  0.07038955, ..., -0.00384101,\n",
       "          0.00034875,  0.02878492],\n",
       "        [-0.06022581,  0.01577859, -0.02585464, ..., -0.00527829,\n",
       "          0.00272203, -0.06793761]], dtype=float32)>,\n",
       " <tf.Variable 'dense/bias:0' shape=(300,) dtype=float32, numpy=\n",
       " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.trainable_variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.960979Z",
     "start_time": "2020-01-03T11:05:06.948521Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'dense/kernel:0' shape=(784, 300) dtype=float32, numpy=\n",
       " array([[ 0.02448617, -0.00877795, -0.02189048, ..., -0.02766046,\n",
       "          0.03859074, -0.06889391],\n",
       "        [ 0.00476504, -0.03105379, -0.0586676 , ...,  0.00602964,\n",
       "         -0.02763776, -0.04165364],\n",
       "        [-0.06189284, -0.06901957,  0.07102345, ..., -0.04238207,\n",
       "          0.07121518, -0.07331658],\n",
       "        ...,\n",
       "        [-0.03048757,  0.02155137, -0.05400612, ..., -0.00113463,\n",
       "          0.00228987,  0.05581069],\n",
       "        [ 0.07061854, -0.06960931,  0.07038955, ..., -0.00384101,\n",
       "          0.00034875,  0.02878492],\n",
       "        [-0.06022581,  0.01577859, -0.02585464, ..., -0.00527829,\n",
       "          0.00272203, -0.06793761]], dtype=float32)>,\n",
       " <tf.Variable 'dense/bias:0' shape=(300,) dtype=float32, numpy=\n",
       " array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden1.trainable_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.967629Z",
     "start_time": "2020-01-03T11:05:06.963045Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_layer('dense') is hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.974112Z",
     "start_time": "2020-01-03T11:05:06.970280Z"
    }
   },
   "outputs": [],
   "source": [
    "# use `get_weights()` and `set_weights()` to access all the parameters of a layer\n",
    "weights, biases = hidden1.get_weights()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.981142Z",
     "start_time": "2020-01-03T11:05:06.976648Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.02448617, -0.00877795, -0.02189048, ..., -0.02766046,\n",
       "         0.03859074, -0.06889391],\n",
       "       [ 0.00476504, -0.03105379, -0.0586676 , ...,  0.00602964,\n",
       "        -0.02763776, -0.04165364],\n",
       "       [-0.06189284, -0.06901957,  0.07102345, ..., -0.04238207,\n",
       "         0.07121518, -0.07331658],\n",
       "       ...,\n",
       "       [-0.03048757,  0.02155137, -0.05400612, ..., -0.00113463,\n",
       "         0.00228987,  0.05581069],\n",
       "       [ 0.07061854, -0.06960931,  0.07038955, ..., -0.00384101,\n",
       "         0.00034875,  0.02878492],\n",
       "       [-0.06022581,  0.01577859, -0.02585464, ..., -0.00527829,\n",
       "         0.00272203, -0.06793761]], dtype=float32)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights  # shape: (784, 300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:06.990093Z",
     "start_time": "2020-01-03T11:05:06.983064Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "biases  # shape: (300,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Dense` layer's weights (<font color='blue'>kernel is another name for the matrix of connection weights</font>) were initialized randomly (to break symmetry), and the biases were initialized to zeros."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red' size=4>Note:</font>**\n",
    "\n",
    "The shape of the weight matrix depends on the number of inputs. This is why it is recommended to specify the `input_shape` when creating the first layer in a Sequential model. However, if you **do not specify the input shape, it’s OK**: Keras will **simply wait until it knows the input shape before it actually builds the model**. This will happen either when you **feed it actual data (e.g., during training)**, or when you **call its `build()` method**. **Until the model is really built, the layers will not have any weights, and you will not be able to do certain things (such as print the model summary or save the model)**. So, if you know the input shape when creating the model, it is best to specify it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:07.069236Z",
     "start_time": "2020-01-03T11:05:06.992645Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAH3CAYAAAB0Cd46AAAAAXNSR0IArs4c6QAAAERlWElmTU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAABb6ADAAQAAAABAAAB9wAAAAAomfLfAABAAElEQVR4AeydB7glRfH2ewkCApIlyyJRMksOwpJkyQssS84ggiRByRnclT+KZAUEyXlJApIUydEASJCMZEGSiErqr36lPd+cOXPmzJx0Z86pep57Z6a7p8M799b0VL9VPcx7f4UzMQQMAUPAEKgSAtdMIr3drEo9tr4aAoaAIWAIuOcmMhAMAUPAEDAEqoeAKe/qPTPrsSFgCBgCzpS3/REYAoaAIVBBBLB518irr77q7r///po0uzAEDAFDwBAYGgSmmmoqt84669Q1Xqe8Udxjx46tK2gJhoAhYAgYAr1HYN5553XPPvtsXcN1yjuUEAphOLWjIVBKBB577DG3+OKLu7/85S9u/vnnL2Ufy9KpKaec0p1xxhlu++23L0uXrB85EBg/frw799xzU0uazTsVFks0BAwBQ6DcCJjyLvfzsd4ZAoaAIZCKgCnvVFgs0RAwBAyBciNgyrvcz8d6ZwgYAoZAKgKmvFNhsURDwBAwBMqNgCnvcj8f650hYAgYAqkINKQKppa2REOgzxCAErvGGmu4KaaYwt144419NrrWh/Ob3/zGXX/99VEFyyyzjNtmm22i63By3HHHuTFjxrgFF1zQ/e1vf3M//OEP3R/+8Af31a9+1W222WZugw02cNAUi8hdd93lzjzzTPfyyy+7eeaZx+25556O9uPy0UcfuVNOOcXdd999btiwYW7TTTd1G2+8sZtmmmnixXKdd6I9aJjTTTed23LLLWvavO2229wNN9wQpS299NJu2223ja7bOpE/3hq54oorIHjXpNmFIVBGBB599FH9WxWed8vd++yzz/yMM87oZ5ttNv/555+3XE8nbvziiy86UU1qHV/+8pf9eeedl5qXlvijH/1IsT3xxBP9G2+84d97772aYv/5z3+8cMb9z372M03/5z//6RdYYAF/2WWX+ddee80ffPDBer8oK//JJ5/U3Jt1IZxmvW/OOef0E088sZ5zvOWWW6LbwGnFFVf0a6+9thfnFf/II4/41Vdf3c8yyyz+rbfeisrlOelUe4xfJgH+sMMOq2mW9Jdeeslfe+21OhZ5odXkN7sYN26cFyedtGLj6rS0Ke80nCytjAh0QnkzLhTTBx98MKRD5MWx2mqreV4m3ZBWlfef//znuu7QR5TlgQceGOVdd911fqKJJvKPP/54lIbSYSJ47733RmlZJzyHmWee2d95551a7K9//asfPXq01jFixIjoVl4Q1HvHHXdEaTJ71jSZAUdpzU463d6//vUvVbTy1ZDa9CSTTOI7qbzN5t3Wd4vd3A8ITDvttO4rX/nKkA7lkEMOcaKMmEwNaT/yNC4zePfwww+7I444IiouXy9OZsROlHiUJgpXz99///0oLevk6quvdvvtt59bZZVVtJjMvt3hhx+u5/ISiW6V2bWex2Mwffzxx5omCjIq1+yk0+1NPvnk7tBDD3XyUnPvvvtus+bbz0++ImzmnUTErsuKQCdm3vJP76+88kqd4YVZLzOyCy+80IsN1TP7E9uqX2uttfyOO+6oJgTwYKZOmQ033NA/99xz/sgjj9RPebH/evqFMOPcaqut/Oabb+6vuuoqTRN3Z70mLcxIg4lB/pv9dttt50WRR/evueaa0UxUE1v81amZN2YATExgkRTGDZ5BJHSBF4XmRXmHpMyjBMXzzF6TMsMMM/ivfe1rUTLlmOWLjdmLbd5/+umnfv311/dzzDGH//vf/x6Va3bSjfboC2a473//+3XNd3rmbWaTOogtoSoItKu8ZYFNlSX/VChO/vEQWXTzEslN02QhzstimCpmyqAkkL333lsVE2lzzz23l4UqLzNGvUcW6PwLL7yg5bAZU+aoo47Sa+y13/72tzXtnHPO0bQHHnhA7cWUwxTANYJZgrR99tlHr9v51SnlHSZ3l19+eWZ3nnrqKe07L6t2BMUvC5KKWbweMAEbnt1CCy2kL9dXXnklXqSl8060t+6663pZAK8zgXVaeZvZRP4CTAYTgZlmmsmdf/75LnzeBxQwYYitVS8POuggJ7NmJ7NkZVDATEBOPvlkt/zyy+v5z3/+c3fJJZc4sdW6nXbaycns1B1//PGaJ4pFj+EXzIglllgiXOpxueWWc1NPPbWyJkaOHOm4Rg444AB39tln66e4JpTg15NPPqm9mHXWWTN7A1tkySWXdDIDzSzXLFMWRNWkJV82NUVPOukkZZfI15KjTzBaMH+1K51ob+GFF3byBeGef/75druTeb8p70x4LHMQEJhsssnqhgl1EIECFwRF/OGHHzphWmhSoMDFlbGwDTTvT3/6U7gt9xHFHpfpp5/e7bLLLo6XTFlEZtTalSzlLV8w7uKLL3byZeFkttly18Vk5U444QQns30nppqaesRk5aAzYndHaQubwwn7xAXbd03hnBedai/QFQNWOZsvXMyUd2HI7IZBRUAoa02HPtdcczmxx7q8i3TxCpPKO55XlvOgHL/0pS817NIf//hHVaTMvFsVvl5kPcEREvVb3/pWTTUsWIrN3R1zzDHu6KOPVl45L1k438JEqSmb96KT7YmJSpsV+3ve5lsqZ8q7JdjsJkMgHYFnnnlGWRfzzTdfeoGM1Coo7xA3HSeZRsIMVhZkG2U3TRfapNt6663dFlts4WR9oK78pZde6iiz3nrraZ6sOehMnwtZfK4r3yyh0+2988472iRsmW6KKe9uomt1DxwC0P2QoFiC2UDYKREW2GkRqHVBUNwokZAX0jHTlEkWXXRR7Y4sDjbs1iabbOJkYTc1n/FhD24kssqoChsTFesNQTDF7LbbbnopTkB6FOeXkK3rFsJKccIUitI46XV7tClOTRwcO+B0U0x5dxNdq7sSCAQFyYwxyJtvvqmncaWLuzaS/BwObuR8esOB5hN+55131rIrrLCCLkZiA2ZHFBbEjj32WM3DbTrUiU0XxYWdmHIswrHohz1XvOy0fBl+MR7MQo1s+ihPofU5cTiq6y5K++tf/7rar+++++66fBJY4GT8vBxwI+eHWfxiiy0WcfExp9AHsOaFh+CSz3OJz/h73Z52RH5h615kkUUcXwRdlSSfJlCBkul2bQiUDYF2qYKiOJUGKLNepZ0Je0RdrYUpolQv+cfzcJUffPDBiN5HmjBCvHwae5ld632iWPyyyy6rPONvfvObXhR/DVS4OEPVgyrGPb/4xS+8LEL673znO9oehUW5KyWOvoijiodSCL+c9gLNsKbSghedogrSrMyAvTAqtI/JbuBKD7dbzEbJLP+Pf/zDyyKsjgm6ZFKEWaN5jDntJx4GQdg9yjcfPny4l815vSwSenHoqelTr9tjPLjr03d5sSSHp8+/kx6WxvOug9gSqoJAu8q73XEG5Y3CElqYKvRGdcrs3ovXnWbDJ0+L9/H222/XOZmQ1gnppPKmTzjI3HTTTaldw1Em7qwTL0Q6L6PTTz89ntzSucy6/YsvvuiffvrpVDyptNft8eKVrxNP7JekdJrn3TqPR14vJoaAIeD0Ex5zQJbA4w4S7ODhOhxxMU9KWlqyTDev01gz9An++f77768bQCdpfFAcGwnmFlH6TgJNNSqSOx3Ticy8M8v3sj1MXr/+9a8dJqEkGwd6aXyNI7PTOTM7oryx+YnHmTvrrLPqaD05+1GKYiyESDQ1ja9Q1F5F7AX+oJNCqMoQqyGZ18tr+LBx+y1tY5cMzii97Eu/tBVs3/zdEAK1nwSeO/FesCEvtdRSTkxF7nvf+140RP6u5evBiTehu/nmm51E9Ivysk4kAqCW74RDTVY7Ia9X7V1zzTW6RgEWLJwGQTdecMEFTkIoOPHadYFGGPLbOian9q3YvMXbSe084mmWrK6n19gJWxFslLjbYqsTMP1jjz1WuBralgUvtQVSBzZNbHR82g2VxPHAdRgXbvrGD+7NssA2VF3rSLtDZTYRBoOGQ+UzGCyJeJdm4+zIIDtUSVGzSd5mMRfhUj7oQjiANFNYu7hkhYTtCNtEFJ+umgcqj/xB91z4JCGoflh9LtKB119/3e2+++6Ru3ORe0NZqF7wOvHyQmThysGJ5dNuKCSJhwTtiVbiJeymGzt2bGdnAUMxyCFqE2cd3LPF9qvUNDz94iyHIerWkDSLuSh4FA5JB0rSKAyjSSedtKe96YjZhB5DDxpKaSekZvAE6wSpPvwhh+NQYZKGR7C7BrfuoepbP7Tbq8/+fsDKxtAdBDoyLZTVXo27G2y+LHJcdNFFDrI+fM1TTz1VbeEE7Qn8WYaDGy1EfLYzwjYk4TOd7JChvFZm0Li7Bk+rCRMmKALYpPG84od8BEUVAgHBryWmbqeFtiQsqAuBiYrUnwcPuMZgttFGG2lAG1mRdyuttJJuJSVmHG2u13jQ3g9+8AO36qqrqtPJr371K+0H/eFLJfzAy0Ukkp777ne/q+nMRoMQd2KHHXZwQunSZxO3vcNzhvdMDAsWsUaNGqVBn8K9djQEDIEGCCRtMkVt3vBTxetKbX+iNLW6PCE1f/e736mtULqltCMJ7uNFues5aaIYtK52Qmomx9bsWhwCdBxpNu+84Tkl1oLWEY93nAePXoQYJfYx2MqnbjMo/D333KMxk7HlQnsSpazbUgWuLWOiLtlbsKYuuLYSiyIKh8m2UKL8tT54vPJpqeFPoXDRH/na0Xrks1P7RT7c6jwyVDbvPH0rW5lu2bzLNs5+60+WzbsjPG/4nvwjB+UNgLJZqabFF3JkRd7LCnaELwqc+2SGGZHrUQ4sHLJvHYtssnqrZeKOCmx1xH0hHjIVslceDg7tSJbyhruK8iEGdJakKW/K58FDVvR1XPH9+nihMVYcI5B28CiivFHCtPv73/9e2xVvQL2WrwO9ZnGG4Pcs2uHsEgSOq2zkq5fyZaXPkYXcIOw7SL1hqyjiV3NNgH/xiNNnznPPI0F5c7/9ZGMQHJEMp2ycyogPE5wUGdcRm3eRkJqisDWkJvcE2yuuryz4ISzysZgGvQb6XUjXzCa/ipRtUlVddgjPWZeRM6FRiNE0PJIhRjFLNHJHzmq+HTwIbYobMrGuxcPQEbMawQyGsDiz7777qjszZq8f//jHau7BDBIWbTEDYf5iV+8guJCzI3gwnQSOMPfIS9uxsFpUWDwM9RS9d1DKs/M74WWh/JlUBwEoiLI5R2qHO6K8U2tOScwTUpPbgsMDtmLx5EqpKT2pHWWVXmN3U/PgMVQhRnEyAH8xi+jmAdiricURFyK+HXfcccpvh0dO/A5iUYRxyVeUcoXh2jaSwMYJ9zQql5VO30K0u6xyg5zHmoO48OsayiDjULWxww9nv9A06ciCZVrF7aSFIOZFw2pWTXnnwajXIUb5EmB2ze4xLB4yW2MhGipUUmCvsGjJYisOWihvlEQQHBLIQ4knRdyak0l2bQgYAgUQKKXyRoFAPST8ZHAlDp/ZjC2EzYTLHATFzSd6yAvpnTyiiJqJ2Ka0SDg2K98sv5MhRpv1KURlQ+nC3gHPwF/G3IEk68CzFhMYjB9MS3ElH8w/sH/i98FiYYdtE0PAEGgdgY4o76DUWg2p+dvf/taF4O4EU2enDD7H+XRvJ6RmUVhC/4Pbc/z+vOE5QyzfcAx1BIpk/CWEfRhJttetEKNPPPGEtsfzEraHnqNUub799tvVXZ6wALj3hpcmtm9m4SG28kMPPeSuu+46vZdfuEVvv/32jnjL7G4SF8wqzM6hebIbCnbzPfbYQymhP/3pT7Uoji4INEMTQ8AQKICA/PPWSFGq4GmnnaYUL2lSmSSyaarPG1JTbDnKEmA1FfoaTAVRHF5iHtf0qdWQmjWVZFzcf//9XhbVlBnBOAh3KTbcmjuaheeUxVWtQ144Oib5EtBQlcILz41HiFLXjRCjYv7QsJmMjx9YIrJgrHTAkMZRNnrVcQun24snplL7hN+urBMJAqShUqEJxkU259V0sZHHk/VcZtnKJAltiA3fy8ta8+Ql5aEIkic2bz9mzJimbJ54A4FtEuiL8Tw7r0XAqIK1eFTlqutUwVaBCMobTrfMSDNjgcjssKWQmq32Le2+ToXnTKubtKC8uxlitFHbaenQAePxTwhlCj87KeJg46FZNhJirBDnWBZfIkpoo7JF0k1550fLlHd+rMpUMkt595RtIjOshkIEM34aSXDtJj980ifLhvCZwsVWb75kfvKaXTskWFMyueF1qL9hgQ5lwMAIjJtGVRbBo1EdzdKhA/ITBNwD9vIHrsn//ve/ncRmztw7kPWIbm8JFfpox84ggIdsMN9RI6wj6IZJwbwpX0y61sH/nXyV6a42RFmUjQfcBhtsEFGCk/c2usaLGTMlZkVopXvuuae2Hy+PmRWKKusn/H0R5RBKaithKTrRnvieKDMuqU9uu+22GpaW+KMoIys+lpbPk2+ZomaT5P1FroPDifCJi9zWtKwoFM84mv2InbVpXb0swE4u8iC9bOnUy2YLtwW+YhtXE5f8c3n5By1cRyduKMPMm6+Kbkqn6i8685YwFPq3iIczX4KyvVnNMPG6lbWOyMTJF9oCCyzgZfd2LyFy/cEHH6z34zxXJNqe+DTofZhSMaXx/8Ax7rgGJhJGw+PwxRedUFH96quv7mX9xct6WU0/m110qj3GL4HxPB7FcSFd9tr0EiJCx9IXO+lI0PLI3ikOLF4oZoWBj4NU5fOqhRjln4ftvvjHWm655bxEZRwS+Idaecsib93aSCeB6GT9rSpv1nKSwt8rypKQEUFkEVvXTx5//PGQ5OVrS/9GWBPJI7wgWGe58847tTieuRJvXusg7G4QXhD87eGdG4S1JdLwvs4rnW4PD2HGHLyHk/1gnamTyrsjbBMBrbDg6SULhRpSk5CssA/6LaB9XlBwUKlSiFE+U/nU5NMV769ZZ50171D7phyf9ARSg07ZDel2/e30WUJeqOMIjllBMCnKS72GiYR3LoKzVx65+uqrdSOUsHmJzL414B334m0dBDYagv4IEthTwawX0rOOnW4PD2FosdBgZcu7rKY7k5d8O/TSbJJs264NgSIItDrzhl1EoC0+c/niYyYXhD0IhdvuJWqlJrE/oihpTeMehIVXPuvlP1CZSQQhEwqlLrpfeOGFHjMgZWDu8HkPi4a+Iu3Uz/0s7kvkTE4LSadm3pgBiEMTD7wWOsIY4wvaBBgjTlEaCyncEz+++uqrGt8mnsY5DDTx+4iSKSfrQhrEjng9LKSvv/76GmuHGER5pRvt0Rd5kXlZT6vrRqdn3h0JTFXXS0swBHqAQCvKm89q6JzYOrFFQm1FCYfPWf75UBakBSFQFsoChY2w8xIsAMqg/Pl8Jy1PZMh26mc3dBQAfSmipOhzp5R3mNyxE1OWsLMM+IwfPz6rWNM8FD+02+Ru8+x8Rf3gsdBCC3norHkDmmU12on2ZGs4pc5iXopLp5X3kJlNBHgTQ6CnCMiWXfpZDmsChyLixuBJymc6zmGYA+QfzCXDMhAsK840YicigqkhODVhAiTt5JNPjnZjwiHpkksucWK/dcSxlxmrttVO/eyByAa3ErVRvVl7Ct7/GnvyySf1rJmpDLYIm5zA6GpHxOdDWWjyFVNTDWZG2CV4VNMngtx1YoOMTrQnfiJO7N8aqK2m0x2+MOXdYUCtuvIiwKYQUBuF1VPTSWK4INhAiwr2/7iESJkhNAB5eKkinYgMueaaa2rMGa1wCH6FuENZylu+LjTODbup87JqVfB4ZpMOme3XRY0U85SDzojdHaXNsyUyZbB9t9Jmp9oLdMWAVSt9yXOPKe88KFmZvkCACG0IPPq4rLzyynpJELCiklTeafczw6fNvAt38Try1B8v3+3zoBwJXdFI2CELRRq2F2xULiudLxVCEovZRUMrxMuyYMmX0zHHHOMkfr7yyompA+db1i/iRXOfd7I9MVFpu8mwF7k7k7Ng7V9xzpusmCFQRQRmmmkm7bbsElTT/dlnn12vMX0UlTzKtdeRIYuOoUj5EHo3xCJKu5cZbAholpbfLA0GT9j+kPg4Sbn00kuV5SMeyZqF6YqIlgjmr6LS6fbeeecd7QJsmW6KKe9uomt1lwoB4aRrf6A5xiXEG2f/VCR86oeAa7LopMoCKlyQoLTFWSUkNTzKgqbmBWXTTv3i8KKbmTRsrMsZRPpEwqYcac2xd62wP9Ky1EaNPbiRgDUKWxYho2BolMUUI7tJ6W3iBKRHWXDWI7+gJRJQTbjbURon2MR72R5thqB03fYqNuUN2iYDgQDKE5sxEQzj9m3x9NUFSWGeKA5EQESIZc7iIG7eKAX8EXAZR4mHnXtkC0CH8hdPRL0n/Aqu5XyOsxDKZz2bYyOt1o8CY+bLjC4enTK02YsjUT4xATWy34MT4ZxXW221uu6gRAn7AHZ33313XT4JLHCycxQvBzb24IdZPAvEIXwG5hT6AK6BZy9eyRqdMz7j73V7YUDYuhdZZBFdzA5pXTnGqSycBypQMt2uDYGyIdAKVRCvOjjcePXi3i2xKJSrLYyFaHhQzsKm2tADZaMJ3SNVWCked3EoYLiHE/1R/imVgxw8C0NwsazIkK3WT5u4gMsCXZ27etT5BiedogpSPfupEnkTT9uk4EoPt1sYO8ksD9VRYr4rZknqH4XZIxY8G/3Eo0cKk0f55sMl0qXspKQRM9l3Nd6nXrfHGHDXp//yYuGyRuSLK6Kk1mRkXGQFpvr/ZNb/VWDKOwNJyyoVAq0o7zAAsUt62fTDZ8W3ES/HSBnIDDrcGh1l1qf3cwwSlHeeyJCt1E8/0voS2m907KTyJrqmbE/o2Xg8TeCgx5114mVIZzNxCWYWT27pHNxffPFFjyNVo/gpvW4PJyxCW/OiTUqnlXfrPB55vZgYAlVFAPvoqquumtl9Pv+DBAZBuObIpzuLZWlCXrPIkK3Un9aPtPY7lZbGkIHzztZ4+++/vxMvysiEFNpkR6VGgrkFU5MEmmpUJHc6GMvMO7N8L9uDGgkPH5NQko3D2kh8zSSz0zkzzeadEygrZgjkQSDQw8KiWp57ylhGzEpqY8aGvNFGG2nsoXg/CcEqZgon3oQu7BIVz290zvoAawydcKhp1EY8vVftscs7jkmMjYlBENY+CJnLYjlOVh19+San9mY2SSJi12VFoB2zSafHhB0cGzqfxvKP64mCl2b37HS7eesrajbJW694reaOXZK3ziqWIxxAI9NNO+PJsnmb2SS8Iu1oCLSBQIgMidt2EKLM9bs0Mw31+/jD+OIbb4e0bh9NeXcbYat/YBDolSlgYAC1gWYiYDbvTHgs0xAwBAyBciJgyrucz8V6ZQgYAoZAJgINzSZjx47NvNEyDYGhRiDQ2CS2s4tvyDzU/Spj+1DV2Cgaj1GT6iCQFZlwGCuh8aGwrVXS1Teeb+eGQBUQgKqHG7zstVmF7lofDYGGCIhXrTvllFOS+ePrlHeyhF0bAlVEAM7tIYccovEuqth/67Mh0ASB8WbzboKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYETHmX8alYnwwBQ8AQaIKAKe8mAFm2IWAIGAJlRMCUdxmfivXJEDAEDIEmCJjybgKQZRsChoAhUEYEJiljp6xPhkARBJ544gk3YsQI98UXX0S3ee8dP5NOOmmUNmzYMLfBBhu4CRMmRGl2YghUFQGbeVf1yVm/IwQWXnhhN3z4cPfZZ59FP59//rkq83jap59+6saMGRPdZyeGQJURMOVd5adnfY8Q2H777d0kk2R/SE422WRuww03jO6xE0OgygiY8q7y07O+RwhsueWWOuuOEhInKPbRo0e7KaecMpFjl4ZANREw5V3N52a9TiAw99xzu6WWWsph104TzCdbb711WpalGQKVRMCUdyUfm3U6DQFMJxNNlP4nPfXUU7tRo0al3WZphkAlEUj/S6/kUKzTg47A2LFjaxgnAQ8YJ5tvvnkN8yTk2dEQqCoCpryr+uSs33UIzDzzzG7kyJF1s29YJmYyqYPLEiqOgCnvij9A634tAtttt11tglzNNNNMbpVVVqlLtwRDoMoImPKu8tOzvtchsPHGG9fMvDGZbLvttjVpdTdZgiFQQQRMeVfwoVmXGyMwzTTTuPXWWy/ifGMy2WqrrRrfYDmGQEURMOVd0Qdn3W6MwDbbbOPwsETmmmsupRA2Lm05hkA1ETDlXc3nZr3OQGD99dd3U0wxhZbYYYcdMkpaliFQXQSy/Ym7PK6XXnrJPfzww11uxaofRARw2Ln77rvddNNN56688spBhMDG3EUEJp98cg1y1sUmmlY9TCKv+aalulTg3HPPdTvvvHOXardqDQFDwBDoDgJf/epX3VtvvdWdyvPVOn7IzSZ4voXwnXb8bxjTQcOBMK0wQjo5btzhL7jggo7W2cn+tVLXuHHj3DzzzNNXY2oFh6G+59RTT82nXrtcasiVd5fHZ9UPKAITTzyxvhAGdPg27AFAwJT3ADxkG6IhYAj0HwKmvPvvmdqIDAFDYAAQMOU9AA/ZhmgIGAL9h4Ap7/57pjYiQ8AQGAAETHkPwEO2IRoChkD/IWDKu/+e6UCO6Prrr9dNiG+99daBHH+eQR933HHu6aefjopeddVVuqfnSiut5A4//HD3+OOPR3l5Tz788EN37LHHurXWWks3uxg/frz7z3/+U3f7vffe6/B2JbrjD37wA/fII4/UlcmT0In2zjjjDHfppZfmaa7cZYQzOWRyzjnneOF5D1n71nA5EBCetxeed1udOemkk3A28z//+c/bqqfdm7/44ot2q2h4v/C8vfC8G+Y3yhBl6mWXIf+zn/0sKnLWWWf5Nddc07/wwgteXnh+4YUXVvyuvvrqqEyzE1Gkep/sC+rFaUXv5xmsuuqqXmLLRLfffPPNXnY48pdccol/++23PW1zffTRR0dl8px0qr1//vOffo011vCHHXZYnmbrygjPW8dbl9HbhHEQ/odMTHkPGfSlargTypsBvfzyy0M6LhTWaqut5sVBqCv9aEV505fVV1/dH3jggTV9WnLJJf1mm20WpV100UWqfGUGHaU1O6FOidjo//Wvf+mYr732Wi8heLUe+RKKbl9wwQVVoUcJckKfUPhFpJPt0ed5553Xn3nmmUW6oGXLorzNbFLuDyPrXQEEvva1rxUo3fmihxxyiLvjjjvUA7LztbdW43nnnafxg4444oiaCmaccUZ3ww03ODxRkREjRujx/fff12OzX6LF3G9+8xsnys8R5wOnqI022ijaJzSYYCj3t7/9zT366KPu448/jqrlvNF+o1Gh2Emn26PPhx56qJMXgnv33XdjLVXn1JR3dZ6V9TQDAWy52G3PPvtsLYUSktmk22STTdwrr7zicGn+1re+5XbaaSf35ptvRjX98Y9/dAcddJA75ZRTHHZzYn+vuOKKDlf0EFZ2//33d1tssYXbcsst9b6//OUvuq0aaXvuuaemobiPP/54PSdeD4ohyB577OF22WWXcNmzIwoSpT1mzBj35S9/uaZd7N1/+MMforjnjAkRc0JNuayLyy+/3E011VQ1RcAYmXXWWfU4bNgwbZ/nAS4y43UPPPCAe/DBB933vvc9LZP3V6fbI3TwJJNM4rDTV1L0O2CIfpnZZIiAL1mz7ZpNLr74Yr/ooovq57ooTR3dD3/4Qy+KRdP4bN900029KGW9lpCxWuZ3v/udlxmnpkn0Qb/EEkt4Ue6ec/ln9rvvvruWkw0d/AwzzKBpATp5Iajdds4559QkUUh+gQUW0DIy+/ZcI//4xz+8KAgt+/e//13TWv1V1GxyxRVXaH9E6TVtUjZo9rPMMouXWWjTslkFRCH7L33pS14ihkbF5MUQ2cTnn39+L9vS+V/84hdRfjsn7ba37rrregkfXMjUVRazidm82/nLsXs7gkC7yptO3HTTTaqogvImTWZWmiamAy5VsLN+5StfCZceBY6iFsaFD4uNKBv5rPZiCvAoaWT55ZfXctGNckJdQXmTvvTSS3uZacaL6Pltt93mf/3rX9elF00oqryPOuoo7fNdd92V2ZREx1OFO2HChMxyzTI/+OADVczCJqkrKmYT7QtYQ1K48cYb68oUTehEe/SVPvHM80pZlLeZTeTJmVQfgckmm6xuEGFDBpl5R3kLLbSQg24W6GzClNC8xRZbzPGJj8js0I0dO1bNJn/+8581Le+vUEe8vLA6IltwPL3b50899ZQ2EUwYjdoj+iIbWGBiakcw0YAjJqe4vPPOO26vvfZSsxV2cfkaUYqivCzixQqfd6I9ts1DAlaFOzGEN5jyHkLwreneI8DCWh75+te/rsXyLuCFOtOUd8jr9TEsEIoZI7Pp3/72t5G9PrNgRuaFF17o5AvDYUvHjhwXoQQ6oSQ6oSE6YaSocpevHLfffvu1vLjbqfbCWoCYtOJdrsS5Ke9KPCbrZK8RCDOx+eabr1DTZVLefEEgH330UeYYxPzjhDaXWSYrEwccFv1Q3tNOO21NUTaAvuyyy5xwv134yjn44IN1pv/Xv/7VPfTQQzXl81x0sj2+ChAxf+VpulRlTHmX6nFYZ8qCgNjCHdRDWQjVLoXZJCYXROyjalZhBhkExQ1DJdDvQvonn3wSmWlCWi+Ooe+wbbKEHa3ShHHADskSKIC77babExu2m2222aKisgbhMIv8+9//VioeWx7GJbBS3nvvvSi51+3R8BtvvKHtt/PyigbQ4xNT3j0G3JrrDgJBqTKbCxIogbKwFZKcOPLoefIzGdNBmKGy5yVbXOFOHkwOQdlA+UNRiYOLQ/G8/vrrSjFEiaO8UOrConIoxCeffNIx82QGzMwu3o+oQ108WWGFFZRL/ac//alhKz/60Y+cLOA6WVCtKYPSxnTEmNgLNE2ee+45J05JyvPG/sxuSPyAFUfy2CkL+uF9992neIR6CGNA3d/85jc1qdfthX7whbXIIou4ueeeOyRV55h3hbUb5Ywq2A1Uq1dnu2yT0047zYuiUdYATJIDDjjAC+daKWDyn+gXX3xxL7xi/+1vfztiPIwcOdLLJ7OXDbA1DdYIdYjCU1pg3JUcRGGdBDoiZXHxhl0isTr8iSeeqFQzKIsy+9YfsecqewXXdCh4Yk7wouzbejhF2SY0JrNidWEPTJpkB/bZZx8dPx6WcYHiOP3002seuCVFvib88OHDIzzBOf6D52UQXOKFa66UyVGjRqmLP9jJSyUUUUplL9uj4WeffVb7HGcjRR3KOCkL24SZwpCJKe8hg75UDbervNsZTFDecLqhnkEZi8flSNaNC35QhMTISAqKKsnnplxa2eS9za5bUd70B946VMo0YSyvvvpqWpaXBU8P3fD0009PzS+aSGySxx57zIvHZeqtvW6PFywva16wRaQsyrt2Wbg6HwzWU0Og4whgPuAnS+Iu+IGpEC+P23lS0soly3Trmv7gdYqXqHyB1NilaRM7/eyzz57aPOYWbNe33HJLan7RREwowQ6fdm8v28O0hakIk1AwjaX1qcxplVTe8rnjTjjhBMeqs3ySumWXXVbP5bO1kqvG4Q8Eu2Eeu6hEiHNPPPGExqbAdhhf9Al12TEfAsH2/dprr+W7oYKlxLvUsWgq3oROIvzp/0yeYRC2lfJJBkmee1sp06v2rrnmGo3JwtjEc7aVrpbinsotWMJdhXb0jW98w/EQWKhi0aXRokozlOVzqa5IWlpdoS4kiG3RyWeuxtlgZoDDwzLLLKM/jJd/QPlk04UfnEyYORTlIdPttPGlpXVhiKWqkn/efffdV/sEzW3HHXfUIEql6mSHOkNcFnjWwXEpT7U41ojJJU/RjpTpVXv8LzHxi39FdWQAPa6kcjNviQms9B5mEazi33///RrNjM+7ohxboqKJ67DDiSBIWlrI6/ZxjjnmcBJjQgPFzzzzzBrIJ9kmM3M+PwmqI4tmGjEuWSbrGlYEHn8oq+CwkpaWVUe/5Mmipf79xMfTq1lmvM1enQfHo161V9Z24h63Ze1jnn5VbuYdSP3hnwz+bTM7ZRoQUMaIIBcix1EmLS3t3m6moZizRBaQNOodZQL3OKt8Mi8tbGlaWvK+frwmLCh/R/Gffhynjak/EajUzPv73/++w3kCQYkFj600Hit2YPi6d955p3vmmWfULk7ITkJYPv/888pBJc4w7rpwdYXx4IQ2pZ/NIU0YCGqywJRx8sknO9pBuWJzFsqT9gOzBXGR+SSlDPf+6le/csyiifGATT4IoUExfUhEtZBU6MinL+aUrAWwrHEnw5bSR8wl8VCmpElEPu1X1rh///vfqxMGePIVRFxnYWqo6QGOr4khYAh0GQH55x0yKUoVFKK/UnsEEk/YTa75YTsn0kIEOAa09tpre3Ft9vBRZXFPI8mJUtGxivOGh3bFPbKvntYlwePr0ijHNlFsPcX2WmIn8/CDuY++I3lCj1Iub2hQMdto/XCJ5cWgP9C9xD6rYUWpJwg0J/oSl6xxp4UtTUujvqxxg2doWxwtvDg56DZmMoPVPsrXUbxLTc+HkirYtHMlKtAKVbBE3e+brpSFKlj7n99jeIsqb7q3zjrrqMKCExpEguJrWlDe5EmUOd2nLpTBKYBtj4LIbFnvie+jl5a28cYbewkiH27zYmPX+yRSW5SWJ/QohcXO3DQ0aFDeKGWhcOkPoUfZ84+0LOWdZ9xpYUvT0pqNW7zrtD/sfSieiYqFbOyqaTitFBFiKjM2+2mOQdhmzLBqjlU3MWKiMsQyrlJmE3kYuYQVdah0mFVgoxCBDHohcRaSkrbIGdJEUSqjBZtoMM1gI5eZuLodUx9207CCH18IIfQoJh5YISFcKQuFeYXFJcw7QbDHh0BDIS15zDvuML74/fG0POMOY8btO5ivGDOCOaqIsB0Wu4qHXWmK3DtIZWFX4cYP48hk6BCApcSuS0Mtfam8AZVIadjFYY+w3RKKNSjgOOhxpRXSQxq2coTttQhfWUQCk6PIPVll55prLt3SqplDQZ5xh/HF24untTruVsdM24yPeCEmjREglggL9oZTY4x6kUPcG9a5hlr6UnkzOyaYPkGK4H9LzARd6EsDO660Qn5ICwuD4kIdsqIjfHNmqFD6eiXxfRHT2sw77jC+eB3xtDzjjt9r54aAIdB7BCpHFcwDEcwPPm3WWmstVdzcI7ElapxTgrLCrBEkmYZ5BDYFrBU2qg0CL3q77bbTl0NIy3OEaRJvL+0e6kbEnpaWnZmWd9wo+XjYUsYdT+v0uDM7bZmGgCHQEgKVm3mH0J+4NUNrQ4JrM7Q90gLvmyDwq6++uu5WzU7ZCHZDYvcSjhLBuWfDDTdUZx3KJtP23ntvpfzh1YnphLqJU8wO43g/Is1Cj9JWCA3KjB37e9h+SSuI/Qp18dXAzD6L9w3FEYHSJ5u65h43LwYoh7L45WRvRsUimdZs3AHPuDs/dnlENrHVo/0yBAyBLiIwlCumRdkmhJkUu6qyEkRxepkNe+F+R0wMNomVPQc1Ktx6662n6TA2oFgJB1uvxeVcd7aWGbDnXKD14ibroQqmpUE1hG0S2B4yS9XQojA7kLyhR6m7WWhQwnOGXcrpF7uRswN4UogCB72OMvwQllTiQuQad1rY0rS0rHHLQqoHf9oGFzZxhUYpMZE1TV5MEZUy2fe0a6MKpqFSn2ZUwXpMhiLFqIKCelHlXfRBEeIzLslrMRUon5ljkLQ08mQWrC8Lwlq2Kp0KDdqs/eQ4k9fwxpNhS9PSaKcT427WX1PezRD6b74p73w4dbtUWZR35cwmMtvLLcF8Em5IXkNRS+6gkZbG/di+l1hiiVBVS8ewENjSzQVuSo4zeZ0WtjQtjSY7Me4CXbeiQ4wAjBaotSHeDxsKs7s8ZkrMihABssK6pnUfaiP1pgkmS3baQQgWRfhaNitebrnlNM6P+CCk3aZpoqQdwdwwaRLrBxFfAw2mhTdyv0tfK+9+f3g2vs4ggBIIi9WdqbG2lm7XX9ta61coT+i1squOVoIiFbOdBkBD+UK5ZWs4FsbFiStXQ3D+CWvBYn2aBMVNzHDCLNA2RAPWplDgRx55pFJk0+796U9/qqEm4owv8ZjWNSy2oDv22GPTbuubtL5km/TN07GBdB0B2D0oENg23ZBu19+pPsPOIqLlL3/5y4j+KlvB6SyWr1MUKru+I6TnFeojnjhxb2BahR/i8fNVF/YGJTQv+1kyY+YrcNddd3USisLJLj6pTbHxcehPvABft3CwISsQdbOfxZR3Pz9dG1tTBLodUbHb9TcdYI4CzIq/+93v6i7wgcHFbShRFGGglY4YMUJrKxJDHucrNmPGOxgHs/CDh6KQCtRDmS8TZugoZNhYQTjHjJkUNivGTMLMO03wesYn4sADD+xr5lM9MmloWJohUDIECE3w4x//2MnGtqoEmIUF6iQ7lW+99dZOYt4orZOus2EH1/yQjySjLPIPDxWVT/eNNtpIwxNgRlhppZXUq1H2X9T72qmfCrifmSyx5MsgzKRffPFF3fE93h/s3VBCQ+hhZs9IMHXEyzY6B2+UaVzeeOMNtW8zI0cwWfEceSkIs8uhnCVgmpNNo9VUE7+Xc2E36TNnZt5IUO70e/z48Y2KVD9d3npDJt1mmwzZwKzhQggUZZvAlGEnd1lA0w1t77nnHqV7EiyIc4TgWPLfqRvoci3mi2j3eP7ukLSIisJv96Js9F6oj/IZr1RM6pIYLspO4t5W6+demRFq/VBDi0i32CYSk8bL7jJNuyJmFaW7Co+/admsArA1wDgENKMsGz8TgA2cZZbuxW/BS+jkumoIHrfUUktF0UIpLy/dunIkEPBMYvB4+XJIzW810dgmgrqJIdAKAnwOC5/fvfTSS9FWVuedd56yIdjKjLwQJCvUz+wuyRZiQQwnKPLCLI40ZtgEFZMwwJFNlhkhn//EPie91frpzwEHHKCOYszuh1owieA0hr05SzBrsIh46aWXtr01GjN6CV0cBTSjXcwq7O7EJsnE1uG5SOTOmi4RU4TgZWz/h4NZM5GIl+qER4C3ZkHdmtVVxnwzm5TxqVifGiIgsyV3+eWX6z92fA9ClA/RDVFEKO8ikmSahCiJcWV/2GGHaZVpwc2atZWsn1g7u+yyi3rFNru32/mwSPD+TSrKZLvQBddff323ySabJLMKXeNBTLyhYDIJN7/zzjuOPSxZwOSlhncxNEK8mYPstNNOaseOR+8MeWnH4MX81FNPpWVXPs2ogpV/hIM1AGaA/GOHf8wweuybzJoJmcrMrcjGuUnlGuqMH1l4Y/GsyGJduD9P/aFsr49hgbBZtEpwPeWUU9ruHjRDok+KqaymLnjl8LuffvppfQljq2YNgpAUvDCY9UMnZDbNlxeCwkdgyvACItpinBce/CrgqPejmPLux6fax2NCKaOoJUSA7jmKUg0ioRD0FN5vI15xKBs/5lGuvBCg/cnuTPFbc53nqT9XRV0oxHjon9ifM2sn1DAxgdoVgryx4EmM/CAoXqh9cVMKC9BsLs6WgoTBZRGTLQXZgDxIYMHwbJjRsyt8XHkH5U7M+X4UM5v041Pt4zExQwzmjCRbQ+K76MYY5AeGhIQGiNAI/+wo4SAoLjjeIS+kJ4+y7Z4mQW9D2q0/BFjTyobwF7ZlXoCyC1VmL7D3N5K8Y+GrCZNJ0vQCc4hgZqxhxCVwwNmXFfYQL+z4T3j+BFEjHQeduMBqQTrx0onXW5ZzU95leRLWj9wICNNDZ4u4XYfdkYhoyCc3MzZm57LHpi56SdAtXWiEDhc87uAuhwiIRHzEji4MFC2HZ16QsFsK4YRZEMXWysIl0k79bNbMzFPYI6GpIT2uvPLKuk7QyFEJmiUhFlgoTAqmFPBO2rCT5bjGZIKMHj1aj+EXLxBm41Ao4/jfeuutGvGy2WJqqCd5xNYt+6vWhcBIlqvqtSnvqj65Ae43/8zXXXedztRQothF2WIOV2rspAj2ThQ55oDddtvN3Xjjjcr5JXQui3Phk5pYHcy+d999d906j0/vILwcsKOjtNnKDgZKYDm0Uz8mHV4YmAvKIMccc4x+eQTlmuwTJgleYGmhfsGXLxnuDTz75P3hGpMJz45nkBTMIbwAYJvIPrU6W2bmTMjmsICcvCfrmoVYuOK45vettMp17MR9xvPuBIrVr6MozzuMWEwdGspXZmw1nOGQz5EokIGXLMpS+cHxfM6TERUJJyz/8F6Uhyf8rSj65C3RdSv1hzajSnKedIvnTfPCpvHLLrtsak/gyItZIjWPRHCCJ87zyBL5MtKyWWXAU6iaXkwsWcWa5skL3cuLXcM8Ny1csIDxvPv2dWgD6xUCsBb4LM4SPsmDBDt1uA7HRhEVYZewEXSWtFp/ozaz2upmHl8p7PeK12pytsqXSVgMTvaBrwfu/c53vqMskmR+/Fri08cvU8/Bs2jUwmRFmMAw8WBfb8aiSd5bpWszm1TpaVlfe4JAoJaFHZp60ugQN4IZCIWHmeInP/lJ7t7AqycaIIuGZRAohawpQB+UjU3K0KWu9cGUd9egtYqrhgALdjAWYK0gONKcf/75VRtGy/2FO8/sm/WDvIKnaZl2s2fNgtC2cQeuvGOpWjnjeVftiVl/u4YAZpiTTjpJf0IjyaBKIb1fj5hIWDSsquT1vqzq+OL9NuUdR8POBx6BuPPIwINhAJQaATOblPrxWOcMAUPAEEhHwJR3Oi6WaggYAoZAqREYcrMJMQtwlDAZXARYIITSZX8H2X8DeAziMGM4ZePU7dxGmyl3u91k/cPgpycTe3VNvIgi++H1ql/WTvURgOZHlLpWXaurj4CNoJsIwMxhg+YhlPFDqryHcODWdJ8jANeXbc4CZ7vPh2vDGzwExpvNe/Aeuo3YEDAE+gABU9598BBtCIaAITB4CJjyHrxnbiM2BAyBPkDAlHcfPEQbgiFgCAweAqa8B++Z24gNAUOgDxAw5d0HD9GGYAgYAoOHgCnvwXvmNmJDwBDoAwRMeffBQ7QhGAKGwOAhYMp78J65jdgQMAT6AAFT3n3wEG0IhoAhMHgImPIevGduIzYEDIE+QMCUdx88RBuCIWAIDB4CprwH75nbiA0BQ6APEDDl3QcP0YZgCBgCg4eAKe/Be+Y2YkPAEOgDBEx598FDtCEYAobA4CFgynvwnrmN2BAwBPoAAVPeffAQbQiGgCEweAiY8h68Z24jNgQMgT5AwJR3HzxEG4IhYAgMHgKmvAfvmduIDQFDoA8QMOXdBw/RhmAIGAKDh4Ap78F75jZiQ8AQ6AMETHn3wUO0IRgChsDgIWDKe/CeuY3YEDAE+gABU9598BBtCIaAITB4CJjyHrxnbiM2BAyBPkDAlHcfPEQbgiFgCAweAqa8B++Z24gNAUOgDxAw5d0HD9GGYAgYAoOHgCnvwXvmNmJDwBDoAwRMeffBQ7QhGAKGwOAhYMp78J65jdgQMAT6AIFJ+mAMNoQBR+CJJ55wI0aMcF988UWEhPfe8TPppJNGacOGDXMbbLCBmzBhQpRmJ4ZAVRGwmXdVn5z1O0Jg4YUXdsOHD3efffZZ9PP555+rMo+nffrpp27MmDHRfXZiCFQZAVPeVX561vcIge23395NMkn2h+Rkk03mNtxww+geOzEEqoyAKe8qPz3re4TAlltuqbPuKCFxgmIfPXq0m3LKKRM5dmkIVBMBU97VfG7W6wQCc889t1tqqaUcdu00wXyy9dZbp2VZmiFQSQRMeVfysVmn0xDAdDLRROl/0lNPPbUbNWpU2m2WZghUEoH0v/RKDsU6PegIjB07toZxEvCAcbL55pvXME9Cnh0NgaoiYMq7qk/O+l2HwMwzz+xGjhxZN/uGZWImkzq4LKHiCJjyrvgDtO7XIrDddtvVJsjVTDPN5FZZZZW6dEswBKqMgCnvKj8963sdAhtvvHHNzBuTybbbbluTVneTJRgCFUTAlHcFH5p1uTEC00wzjVtvvfUizjcmk6222qrxDZZjCFQUAVPeFX1w1u3GCGyzzTYOD0tkrrnmUgph49KWYwhUEwFT3tV8btbrDATWX399N8UUU2iJHXbYIaOkZRkC1UUg25+4pON66aWX3MMPP1zS3lm3yoAADjt33323m2666dyVV15Zhi5ZH0qIAC95XvZVlGESec1XrePnnXee23HHHavWbeuvIWAIlAyBWWed1b3++usl61Wu7oyvrNmEGBUh7Kcd/xv+tF9w+Oc//6l/vTfeeGPLzxh3+AsuuKDl+6uC5eKLL+4OPfTQvh9nN57HySefnEtLlrVQZZV3WQG1fpUDgYknnlgpguXojfXCEOg8Aqa8O4+p1WgIGAKGQNcRMOXddYitAUPAEDAEOo+AKe/OY2o1GgKGgCHQdQRMeXcdYmvAEDAEDIHOI2DKu/OYWo2GgCFgCHQdgUo66XQdFWug8ghALVtjjTXU0xLKoUk6Ascdd5xuyrzgggu6v/3tb+6HP/yh+8Mf/uC++tWvus0228xtsMEGhbaOO/XUU91zzz2X2hj7h/JMkI8++sidcsop7r777tPdjzbddFNHUDFi0zSSu+66y40bN87ddNNNGmjsjDPOUCcstsAbRDHlPYhPfQDG/MUXX7jHH3/cfelLX9INGhrtsNMLKHiRNNqerRftp7XxySefuG9/+9tu+eWXdyjujz/+WMPmHn300e7AAw90p512mkMpLr300qpgic7YTFD+3//+9x11p0lQ3OCx9tprO3Y3op0PPvjAHXDAAe7ggw92jz76qL44kve///77jpg1r7zySvQ8CX3AC+HJJ590xx57bPKWvr82s0nfP+LBHCA872effdY99dRTQxoOlpcISisEyirD06Av66yzjptlllncd77zHe3S7bffrngtvPDCbrbZZtMZ7rzzzuseeeSR3KEofvnLXzpm0H/5y1/cf/7zn+jnhBNOcFNNNZX71re+pW1dccUV+kI46KCDHG0QyuCoo45yb775ppswYUIqRPQTxR2XL3/5y+6GG25wl112mTvrrLPiWQNxbsp7IB7zYA5y2mmndV/5yleGdPCHHHKIu+OOO9QDckg7Emuc8BLEBjriiCOi1BlnnFFntNddd12UNmLECD1n1ptHiOB47rnnuvnnn1+/ePjq4ef666/XML2TTz65VvPWW2/p8f7774+qZeaPTDJJvTEAT9m///3v+mKIbvjfCXXiYcrXwrvvvpvM7u9r+YSpnMgb3ot7fOX6bR3Oh4C4xxNvx4utOt8NKaVEGXgJSOVHjx7txVVeS7z33nv+wgsv9GJb9X/961+92Fz9Wmut5SVOjn/jjTe0jHzCaxn5HPdiu/VHHnmkX3HFFf2YMWO8fNJrmXvvvddLjHAv+2L6q666StPGjx+v16SRj4gZQMfBWGSHHy+KXNPJX3PNNf2dd96p1+38Evd4L8ordxVgKzNrHXPydM70dQAAQABJREFUJsYHbkGoW5SjF+UdkgofJW6IF5OVl9l2dO+rr76qaRI0zP/mN7/xEnPdS3AoP8ccc3hR0lE5Tp5//nk/fPhw/9prr3mxwSuelI8L1/Ly8WKyiSc3PRf3eC+xTZqWK2mBccwIKiemvCv3yAp1uF3lLbZXVZYyi6v5Z5fFOC+f75omdl4vn/iqmFGuKA9k7733VoVF2txzz+3F7utlCzW9hwnDCy+8oOVOPPFETZPPfb0W84gXG7KmnXPOOZr2wAMP+AUWWEDTZPbtuUZklqhp++yzj16386uo8kaJMrbLL788s1kxN2k5XkrtiCxgKp6yQFlTDWOnHzyjhRZaSF+iYhapKcNLd4UVVohekI2UNzetu+66XiIERi/qmooaXFRdeZvZRP6CTPoLAfasPP/881347A+jw4QhM3G9xN4qs2Yns2BdIIPJgBCsiEU85Oc//7m75JJLnMyQ3U477eQImHX88cdrnigcPYZfLEguscQS4VKPyy23nC7KkcfGyFwjLM6dffbZ+rmvCT38xeIeQjS9LDnzzDPdkksuqQuQWeWa5YExi5MEkovLSSedpOwSAojRJ/Ixc8WFRchvfOMbqeaSeDnOsdX/61//cjJTT2b17bUp7759tDawySabrA6EsEkDDIsgKOIPP/xQF9hIC4omrowPO+wwLf6nP/0p3Jb7mGSaTD/99G6XXXbRjZFzV9KhgizgIlnKm63jLr74YidfEKk26LxdYQGSmOosYiZFzFdOTCZqd0dpX3vttarkg+0bCiELkXkj/wWKYRhfsr1+vK5fHejHUdqYDIEMBGCmNBMW46Ab5l28i9eXVN7xvF6fB+XIQmIj+eMf/6iKlJl3O3L11Vc7sIUrHhcWLInH/5Of/MSJ+cTtIJQ/MXsoAwWFzVfOrrvuqi+YOAXwscce02r4amKBlWMQmCcIC5uDIqa8B+VJ2zjbQuCZZ55RNsZ8881XuJ4yKW+YIAhOMo1EFnOdLLw2ys6dzg5G0CST5pBLL71UqZNsFI3I2oLO9KEMcg/KGxYJVE9+ggQ2CQqer5e48n7nnXe02JxzzhmK9/3RlHffP2IbYCcQgO6HBIUTKG04mATBfovA7Q6C4oZXTV64hzzMNENBY1x00UW1a3Cmk3b70OdNNtmkITeecWBWCeancE/yiMMOJpOf/exnySwnzBFNYztDeN4I6xMzzDCDE0aQXv/+97/XY/yXMHwcip/74lhSRthCWjTUF7+vX8/N5t2vT9bGpQoSGJhJBsEOi8SV7ssvv6xpyU9u+MkIC5Vwo7GT77zzzpomLAhdjMQ2DLcZJRU+8XEcCXXi8CJkB7UfU47FORYDmY3i6t1rod+YfxrZ7lGeX/va19xqq61W1zUWBL/+9a+rEw+KOUswmSBhgTheFq9I+gCmwXkJl3zwb3XGj617kUUW0Vl8vK1+Pjfl3c9Pd0DHhrIWXrYLNtKtt97aMZP7v//7P4cnIQLj46GHHnK77bZbFIuDcnEFTpwOGCIobRY/f/e737ngJo6NFXduzA/UQfwUodXpIiSLgeEzfuzYseoav/vuu7snnnhC2RO4j6PQmcH2WjCbYE9mwZA+JAXPSDAIs+N4PoqWFxl2/4suuiieVXeO+eOb3/xm6qIs6dzP1wwzZezdq6++ujv88MPdvvvuW1dXswRiqQgNs21mTLN2SpffgAJZ6mTjeZf68bTduXZ53u12QEwjykHGcQcnEVHEDasU84cXW6zm4ywiirmu7Ntvv13nfEJaJ6Qoz5s2aRsHGQnwlNoFUd41zjrxQjjxwG0//fTT48l1508//XTk+FSX+b8EeRn4F1980VM2DbdG9yXT99tvP+WDy4snmZV5XXWet9m8S/c6tQ6VBQE+7TETZAnBlYIk7bAhHWZEUtLSkmW6dU3b8Mz3339/J8pfzSDxtlgMbCSYW4jqd8sttzQqouninJSZTyb4Dh8+vGm5rALQGX/961+rfT2LQZNVR1XzBlJ5i3uuBrTBNrnSSivp52+VHiCBfhhDXFAc/FPyyb7qqqsOlO0vjkMnzoPpBNMBoVH7UeBeY77BZHHzzTdrkKo84yRQFeWTDJI893a6zDXXXKPrB/SHxc5Bk4G0ebMSjmcddsqwYFKlBw8vFkYD8ZD5gR7FDAb+LHZXZovYahvFVa7SWHvZV/4WwBYFheBIg6dmvwohX1lYbMYciY9/r7320hja8bShOsf7kv9jFlgHUQZSeUNLkkBB+rzzOGiU7Q8D928W1xA+24mhzD8VLsewGfgkxtECZsGDDz5Ytu6Xtj/8LYCh2ISVsoYHYKvsh9IOMtExXvTBOzGRVfpLFpLDAnLpO9uFDg6k2QQcsbfFj3pRoV+NOMIoIGaMUOFQ6lC+YDngCGHSHIEymAOa99JKGAKiwwYFBDiqRx55pJNQnBqIHioTkvR+I8YCn84EqydOcJwPDN2M4Ebwc1m4gf4FxQnaVVz4FOWTlDq++93v1i3uZLVBTAcJU+pCoKR4vUXOZQVeF6MYd6DHcT+zSuJ0SBQ97SP2wiCBAoaTBk4cUOUIoI/HW+BHU5Y8sGR8bJXFSyIuWW3Ey9m5IWAItIFAJpempJlFqYLQucT1VsN+EtOZcJfyuah0MOH+RqMUpeZlsc/fc889XkwPXj7JNKQn9CiZvSodSaDWeMjiEOC33XZbLzM1jU0snGGtR5Svl9V6pZeJ4vTiHuwJHxokqw3K5A0XSn/oi5hNQtV1xxCilH4ihDOdZ555vETL05jTI0eO1DpCCNM8IVOpR3i6ShfjXJS/0s44R5q18d9S2b+HmiqY3bty5bZCFSzXCIauN1WnCkLUr5wUVd4oTBSdbM8UjRVFRVpQ3mIj9mJy0CD9oZCEstQy4hGnSQTn5x4JP+lDfGLZBFXTgoKW7Zo0MDwvCUQ80fyPfvQjPc/TBhxbXhzEpM6SPMpb7LfaNxQ2wiYE4iEYVSs7mWi+MFSiNNknUNPE+y1KE8aFFzONXotrsubLxrVRPrzoIHnaCGUbHYPyBmv7ycaACYZhlI1RFj7CUmn0Z1j29HEDYfPGdXn22WfX7ZnkQaosu+yyegxmEzy+YBuwg3UQvMlE8UWmk7AqD7sjhA0N8SFgsCDEgiYONBHZCGdJRLUQWjRPGyFcaOhDO0fMFwgLnP/4xz8c1CpsusE1mvEyPuz///73vzUYUBhjMmQq3oV437Gyj/0c0wvmk2OOOUbjTdBO3jYom0cIPJSMyZ3nvkEq84Mf/ED3gNxiiy0GadgdGSt8dTjiVZW+V94oVXi7K6+8cs0zCko7HNk0lUXAQBOrKZxxkWSrwE5A0RG3YaONNlKbMUqbDVhbbSOj+cws4kUg0AaJiofggow9vIjExwhexOjApk+MDgIFibed7uzdThtp/cGNGh6ySWME5AtSXe5ZezAphgDBrJo5GxWrsbel+37BEkcEhMVGZo6NhFgVRHpDwSZFXHiTSQ2vibomZh0NJM9s/9Zbb9VFPW7oVBsNG49lsMBIvAeEhcUQ75iNZ5NCjOewKWwyL3nNbJ2vC9nv0BHlDcy2335799vf/rZjbSTbtGtDwBCoR6Dvlbdsaqqxf2FdyEJkhEAyfGcwbcAwEWNXVA72BztT5xXYJShOZuAERqJettEiFnHeNlCIzSQedjRZlvtR2ESIwwWabagwjzD7h2UDBzwI9cB5j0feC3lpR15k1InnYdhthTpg0HSqjbR2Lc0QMARqEeh75c1wiSCHEM4Tk8bjjz/uZMFN01A6UOmEmaEOLxMmTFBTB3brPfbYQ2eXP/3pT7VsoMvF6YMh9GcIFM/M9LTTTtPy2K8xWbC/HrbmPG3kDRcKdxvBVo2y5oWDiYhZPyYi7NpEtAt7LhL3QTbXVbs+7vMSXMjJIqtGfsOGvcwyy2h9WWMMbuOyeW1EHQzxrQkzkLcNbch+GQKGQHsIlH1JNa1/RdkmMsv2srDjRbnoyjzsCfFQVJqdbMfkhVOtzUDzk/jL0eq9bH3lxRygeUSXEwWnebLAp/WJa67uMC5PwIuXmodyJ6YEZWbAuhB+tJcZsNLywjiy2qCMuLtrG0RuayTQCSWOSdRPiWvixSyi9EfaY2du8Q6su53IbbBN6D99Fvu17ngOFRIRRa87cJMHBU28M6Md0UmDWgglEnqkKHy/5557+lGjRvnvfe97XmbfWkezNrRQk1+BbSLhC5qUtGyjCrb+N2BUwdaxa/nOoso7NETISJQwigaqH0oiKeTJ1kseWmBQSMkyWddwuxEJaqShN9PKNmtDWCJpt3UsTVghHtqizNgL1ylfFh4cUfjgxHWatNOGKe80RNPTTHmn45InterKu+/ZJjJjjITP+hDiM1D9osz/ncCmaGcrJfbeQ9hBpZE0a6Pb4UKxfQf7e6M+NkqHVgiOSBZO7bTRqG1L7zwCmA/ZuAJqKGY32CuwlFjTgMEC1bXR/0qj3rB4DU2WdRT5MnPiJFa35Rq00quuukoX9PmfweMXz17+N+LCAjumTcyAmOZgOWHmQ8THQoNkkTaIMhA270F8sDbm9hCQmVt7FTS5u9v1N2lew8ESBoKJAoobxtEqq6ziVlxxRceaBvG4UYpiKiu04w+kABbLecmjlMVBTdd94ovwkAXwp0DBw1NnbUac2zTkQrzfrEWF7eago7IZMQo8hEOm/8Tzhv46kJLn86JsZVo1m5RtHNafdASG2mwiSsMfccQR6Z3rQGon62/FbMIakMTk0VAMYTjXXXedroXIYn5I8vJlpWsjrO3kEXYSkhm0F+etqHgI0RDfeUe2j9N6ZY/QqJzEytE0wi0g7GLEbj/ipBWVEaKArlOJQo/SMFPSz+AFHWXkOKm62cRm3gP5yrZBN0IA9hD8dVhD3ZBu15+nzziQYY6QF1RUnBm4rMU4UeJRWvBuJWBZHsFXgE2YoecGCSF1g6kDXAlZzKYhgalEWSJhIrC8EGb/UF3ZbzMITnTM0iWsg/oZkM7sHnovdN7A+Arl+/1oyrvfn/AAjQ+nJGHAaORIYRGpAgjDh69PDHTcyKGDInzSc80P+bKYrdRJbL/YWYmmiJLjkx8vWTxmKQPNks93bMJhk2O479QT7K84e4X26BPSqH7yoKUGBcZ1twTzCEobO3dw3KItzCU4XsW9bxkDypGx5hGc0pJrKZg+JCic+hpQB5s+szkz9FlMK0FQ+MQVx2UdswpHZNFFFw1F9Mh9SPwlI/F4HDtJsRHJQEmOr4vSFTGzSekeSUc71IrZhABhUEHFdd8TPEu4/foZLgo26hvBw+SfO4qICOsnfNZD8xSOu5dwv1pG7Kn+jjvu0DThx3tRYpoORVIUtBf7sF7LYp5GUsRkQJAj6g+C+QBapsTC0aRG9cPMge5JWQKTFZGiZpMrrrhC+ygz28xmiLzJWEQhZpZrlCmLlUpLpQ558UXFLrvsMq03RLqMMuRkvvnm0zzuFTu8nsMOi4s4hmk6NNy4SBgFpbliEsorZjaRp2NiCAwlAsxomTEyA2PGLfx8dU5iAQ6PUswESAgiphfyC2ZDfKY488wzu8UWW0yz+cxnsY40FtYICYDwWX/JJZeo1ywzc3nRaFvM/ET5aJnwi9lknDnUqH6YOQRIYlu+rM1/Q73tHNlpCcFskSU4ixFcLRmrPeuekMfCIl8lfK0gPJfg6BaCpYUAaOEejiGNLx9+4ml6kSgT0jgyI8eLmr+FQZH//90yKCO2cfYdApg48DQNCjYMEJoawuYYRSVJWQt0ubiyJ7IiEqI0FmkjWT+bhIT+FqmnaFmZUestWcpbviKi0Ae8lIoKLzFMWOJL4YgMiWCXxhwStlzDvp4U2kUwv+CRjMgsWo/hV7xMSOMY6g3ji+f167kp7359sgM0rrDRctyGyvBDJMkQ7bAIJEnlmnYvM3zazLugF68jT/3x8p06x+aNBK5+Wr3EviEeDjPvdiTYoaH7hVlx2CwYxZ4UuN/CMNGd7EM50uISrpNfUcF+H0I4xO/p13NT3v36ZAdoXMQrR+KBx7hmBodgrigqeZQrLwVmkElzSZ628tSfp56iZeaff369JU15hrpwrgkskZDWzpHQvoQV5nnQPi882XGppkpm1JhUwoIkO8MjyXLiuazpoZxeyC8WQRFi7Q+KmPIelCfdx+Mk+BeS3PczxGaHSYEEE0A8sBif8kj4jA9KNSt8sN4gv2RBU08D5S3UHxxS+OSHGhfqpnCj+gldnKdNbbCNX4G9wUYajQRPRzwe0wS8mEUXERgm7PWKbR9zzejRo5XZEoKgUResHcYPFRBhf1gkvscq17B/UNBJExmxuZEsr18t0Ee/THn30cMc1KGgPLEZM0uL27f5x2fBMESVDN56hLJlQwl2WIKXjNxwww0ODnYIawBVDeVP5MW4iGOJXrJQyUIo3olEq0TYrBmB8sfiI4t2cJVff/11x30o8bT6mXUyI0UpxV8sWlmHf4EBM99Gdnr6i8litdVWq2sZpU14CcYg2/vV5WO+AgMJrqYboFBAAqS5p59+2v3iF7+IymP/pg9QNYPIln3qiUlIZYSXDLN/OOHgh7DYSnx8Fo3DGoRmyC9s3bKvbOQ6H9L7+piXVlOmckYVLNPT6HxfWqEKitLxwrNWuphsDqF0Pvm09vIPX9NBqIBEYJRZshel70WpeDG7ePYeFWWtQbeEcaJ0NFFiPngcUlYUgSdPttDzwiTRjZih/wWBGihKR8tBDzzrrLP80ksvrbRCaIrQ2AjqlayftFlmmUWjNTKOIlKUKkjdu+22m+7DKi+TuqbwbIQWCW0vKVAa2VwbHKBYJoXxE7GTfGibwg/XttICrbG3K56RsgDs2SuWZydmqJoqQ4RKng/4ywvFQ+9LCgHSaFNepsmszOuqUwX/Pyk1c5jlyjTlXa7n0enetKK8Qx/E9uklZrtyr0Na8kg0RfHG02SZ9XqURFyIlCiz+JqIiUF5o9zgHtNOI5EZfBSRkrEkJa1+yqWVTd6bvG5FeaNMcT2Xr4tkdXoN15yokWlCujgp+bi7e7wcePJCw5U9jxB9s1Fb4X5ebnC/G4nQRD0u85QrIlVX3sV5QPKKMzEEyoqAOMq4VWWziSyZeuqpo+xgp44S5IRPenjeaUJeiEyZlk9aYEpwHlgQnAdJqz+tXCjf6SOmJMwReIWK8o9MOaGdLK455hZMSo32fgTPIgvEwYwU2k47woxptBBJYCo48phxshg0afVWPc2Ud9WfoPW/JwgEChpsB8KlVl023XRTjSzIBs+sDYjZJteQWAegfOBh57qpS4WuueYa3QSb/vDSHjSxBctBe+I23kIIwBYh9GhgrrAYef755xeqo6yFicPCAm/wbMzTz7322ku52HnKdrsMdEKJeFjzpdPtNstUv828y/Q0rC+lQwB+MkwIfoIQrKlfpJkJqMzjhOkzyGLKe5Cfvo09FwJlMBHk6qgVGigEzGwyUI/bBmsIGAL9goAp7355kjYOQ8AQGCgEKms2IYpccKUdqCc2AIMNLusE18eL0aQxAhK7XBcdWwm+1bjWwcghdG2VZRik9qoNQJwwdOfoqvXb+ts7BKD04S5PUCQTQ6ARAkQxJHZ5BWV8JZV3BYG2LvcYAf4hDznkkCjGRo+bt+YMgW4jMN5s3t2G2Oo3BAwBQ6ALCJjy7gKoVqUhYAgYAt1GwJR3txG2+g0BQ8AQ6AICpry7AKpVaQgYAoZAtxEw5d1thK1+Q8AQMAS6gIAp7y6AalUaAoaAIdBtBEx5dxthq98QMAQMgS4gYMq7C6BalYaAIWAIdBsBU97dRtjqNwQMAUOgCwiY8u4CqFalIWAIGALdRsCUd7cRtvoNAUPAEOgCAqa8uwCqVWkIGAKGQLcRMOXdbYStfkPAEDAEuoCAKe8ugGpVGgKGgCHQbQRMeXcbYavfEDAEDIEuIGDKuwugWpWGgCFgCHQbAVPe3UbY6jcEDAFDoAsImPLuAqhWpSFgCBgC3UbAlHe3Ebb6DQFDwBDoAgKmvLsAqlVpCBgChkC3ETDl3W2ErX5DwBAwBLqAgCnvLoBqVRoChoAh0G0ETHl3G2Gr3xAwBAyBLiBgyrsLoFqVhoAhYAh0GwFT3t1G2Oo3BAwBQ6ALCJjy7gKoVqUhYAgYAt1GwJR3txG2+g0BQ8AQ6AICpry7AKpVaQgYAoZAtxEw5d1thK1+Q8AQMAS6gIAp7y6AalUaAoaAIdBtBEx5dxthq98QMAQMgS4gYMq7C6BalYaAIWAIdBsBU97dRtjqNwQMAUOgCwhM0oU6rUpDoKcIPPHEE27EiBHuiy++iNr13jt+Jp100iht2LBhboMNNnATJkyI0uzEEKgqAjbzruqTs35HCCy88MJu+PDh7rPPPot+Pv/8c1Xm8bRPP/3UjRkzJrrPTgyBKiNgyrvKT8/6HiGw/fbbu0kmyf6QnGyyydyGG24Y3WMnhkCVETDlXeWnZ32PENhyyy111h0lJE5Q7KNHj3ZTTjllIscuDYFqImDKu5rPzXqdQGDuued2Sy21lMOunSaYT7beeuu0LEszBCqJgCnvSj4263QaAphOJpoo/U966qmndqNGjUq7zdIMgUoikP6XXsmhWKcHHYGxY8fWME4CHjBONt988xrmScizoyFQVQRMeVf1yVm/6xCYeeaZ3ciRI+tm37BMzGRSB5clVBwBU94Vf4DW/VoEtttuu9oEuZppppncKqusUpduCYZAlREw5V3lp2d9r0Ng4403rpl5YzLZdttta9LqbrIEQ6CCCJjyruBDsy43RmCaaaZx6623XsT5xmSy1VZbNb7BcgyBiiJgyruiD8663RiBbbbZxuFhicw111xKIWxc2nIMgWoiYMq7ms/Nep2BwPrrr++mmGIKLbHDDjtklLQsQ6C6CGT7Ew/BuO655x73xhtvDEHL1mQ/IYDDzt133+2mm246d+WVV/bT0GwsQ4DAaqut5macccYhaLlxk8Mk8ppvnN37HOyVN910U+8bthYNAUPAEGiAABOBlVdeuUHukCSPL6XZhE/dENLTjv8NbWo41OOw7rrrOrwq07DBHf6CCy5IzUsr389pxx13nJt//vkNi/+FCS7yrF9//fUh0cx5Gi2l8s7TcStjCGQhMPHEEytFMKuM5RkCVUbAlHeVn5713RAwBAYWAVPeA/vobeCGgCFQZQRMeVf56VnfDQFDYGARMOU9sI/eBm4IGAJVRsCUd5WfnvXdEDAEBhYBU94D++ht4CBw/fXX6+bFt956qwHSAAGohk8//XSUe9VVV+leoCuttJI7/PDD3eOPPx7l5T358MMP3SGHHOKWX355h0fsJZdc4v71r3/V3f7www+7Qw89VOPVjBs3zr344ouFynz88ceOcAlvv/123X1VTzDlXfUnaP1vCwGUwcsvv5yqFNqquODNcI/LJp988onD5wLPwgUXXFC7d/bZZ7szzzzTnXzyye6oo45y11xzjVtsscX0WKT/Y8aMcZdffrmbZ5553P3336/x1sePH19Txe233+5WWGEFxy5Ip59+unv22WcdL4xXX301KteszJe//GXHJh28JF566aXovr44kT+aUok4Xnj5gylVn6wz5USAvxVx0mm7c6K8266jnQokiJYX92svjkXtVNPwXpk5e3HSaZiflkFfVl99dX/ggQfWZC+55JJ+s802i9Iuuugi3jp+rbXWitKanVx66aX+hz/8oWfcyF//+lcvG0T7r371q/6LL77QNAmR4SW0gR8xYoRe8+uDDz7wosi9KPTcZcLN8tJRDGTGH5JyHcVJR8cnHpa5yvew0DibeffFK9gG0Q4CX/va19q5ve17MR/ccccd6gHZdmUdquC8885zmCyOOOKImhqZhd9www0OD1ZElKse33//fT3m+cVMmjGH/UbnnHNOnWEPHz482kCaWfl7773ndt1116jKr3zlKzqLZqb+6KOP6sy9WZlwM18QRJr88Y9/HJIqfzTlXflHaANoBwFsudhtMQcgKCGZTbpNNtnEvfLKK+7UU0913/rWt9xOO+3k3nzzzaipP/7xj+6ggw5yp5xyitrNiRm+4oorOuyyIRzt/vvv77bYYgu35ZZb6n1/+ctf1DxA2p577qlpKLHjjz9ez3feeWe174ZG9thjD7fLLruEy54dsROjtDFtYHaIC/buP/zhD1G8dMaErLHGGvFimefEL4oLuPIcZJYfJYf4RosuumiUxsnCCy+s19ddd10UAymrTLhZZvaK+U9+8hP31ltvheRqH3s4zc/VlJlNcsFkhQSBds0mF198sZd/fP0slkUxxZTP+ammmkrTxM7rN910Uy9KWa9lYU3L/O53v9PPefnP10/7JZZYwoty13PSdt99dy0nG0H4GWaYQe/VBPklLwQvM04vs01NeuCBB/wCCyygZWT27blG/vGPf6gpgbJ///vfNa3VX0XNJldccYX2R2a/TZuUjZ39LLPM4t99992mZZMFMJFgQvn617/ul1lmGf/OO+9ERcAeLJ9//vkojROeGengnadM/OaHHnpI7+UZ5xUzmwjaJoZA2RBgthxmvaFvzIRHjx6tl8ysmWnee++9Tuyx7q677tL0VVdd1Z144ol6vtBCC+lM9JxzznGieN3kk0/uzjrrLF1UY7Y333zzhar1OMccc9SEFl1uueV0QW7YsGG6eTLXiLxA3K9//Wt34403uumnn17TevXrySef1KZmnXXWzCb/9re/6UIli4mE3i0qLHjydfLCCy+oiQYTTDC/UDcS4rKHusM1+XnKhPs4hln7n//853hyZc/NbFLZR2cd7wQCk002WV01QUEEhgUFUNLQ2/7zn/9o+SmnnFKPMC1QvAiR+2A2YDYpqiBCHVrR/36tueaabtSoUfGknpw/9dRT2k4z5U3URmh+mJhakaOPPtq99tprar+W2beThUt3xhlnaFXTTjutHmWGXFM129ohs88+u8tTJn4zJiD2NA3ji+dV8dyUdxWfmvW55wgQpTCPoISQMIPMcw9l0pR33ns7XQ6bN/KlL30ps+rf/va3dV8umTc0yOQF+LOf/UxzA2c8LCKL+ajmrnDNyzRPmZqb5YIXs5ihksmVvDblXcnHZp0uKwJhVpc0lzTrb5mUN18QyEcffZTZbUxJ8847b2aZvJnf/OY3tWhQyN/4xjf0GpNKXJipI5hA8pSJ3wtvna8n2C39IKa8++Ep2hhKg4AsZuqMMDAgsHsjKA0EMwBmFVms02t+obhJC/S7kIGyCWaakNaLY+g7bJssOffcc1OzGUeat2Rq4f8lymKi4rDhhhtqiiz66vHmm2+uuQ36IsoXp5s8ZeI3B7ZQp1448bqH4tyU91Cgbm2WBoGgVLG3Bgn/5OIUEpLUC5OL5Cc3poMwQ2WvTGhouJMHkwM0QwTKH4uP4uCi/GV2aME1HyU+22yzqVJn0ROFyIIhtl1mwCiqeD+0si7/wqsRDvaf/vSnhi396Ec/cvCuWVSNC0ob0xFjYuuwNIGGCO+afF5m3EN9BxxwgHpQcg8vEGGyKIUz7GYDLoQx+PnPf+5Yc8hTJt5++CraYIMN4snVPc9LmelVOaMK9grp6rfTLlXwtNNOU5qa/Pd6UURelIcX9okXu6hSyhZffHH/4IMP+m9/+9t6TbmRI0cqpU1mgJoG5Q+qG15/0ALFdlsDLNRAUTJRWWGi+KWXXtqvssoqXhgr6lUJ/U1m3/qz3377qZehzLiVgieLcl4cUWrqLHpRlCpI/bvttpsX00Tk8Zhsc5999tEx4WEZFyiOwo7RPHBLk4MPPljHCp7iHq+YCm+7rqh8eXjhvvuZZprJCzdcvVDFLb+mXJ4y4QaJcaLPSr5yQlLTY5mpgrz5SiWmvEv1OErdmXaVdzuDC8obTjdu2+KsErl7p9WLC35w/f7nP/9ZV0QCJ9XxuSmXVrbu5iYJrShv+oN7ujjLpNbOWCTGSGqeLHh6oQF6oRCm5pPIC0m+cDIxCzfzIsOFPkualcHdHv7+tddem1VNXV6Zlfd/DXLV/XCwnhsCQ44A5gN+siQsxFEm6bVIGm7nSUkrlyzTrWv6g9cpPGz5AlEzSLwt7PTQ9dIEcwsekrfccktatqYFml/DArEMTFDNFhmzyuBCv/baa6tpZqONNorVXO3TvlPeRBwj9gI/RCCTT7RKPiGZArgJEyZoJLVWxgDPOLh8xwEgrsTMM8+s9lQJhhTZZuNl7Lw5AsH2HdgPze+oXgnxLnUsmsoXjmPhUDwpcw3ikUce0fJFFHSuilsoxIIvTlcSwMx997vfbaGG8t7SdwuWeF3hEcfiECv4VRTiNkj0Nl3c4h+hFYFGhQfhhRdeqPE3iEcBnxYnBRaKcK4QW6IurrVS/yDfgyLbd999FYLbbrvN7bjjjpG3X7/hQlyWq6++us7TMWuce+21V0sel1l1tpoH24cFTllLaLWK0t7XdzNvXGy32247DS6U17GiTE8H9gGfgDAW2lkVZ+y4Wi+77LL6+brxxhs7Ah8Fee6555wsKGlQJri0uHQHWlsoY8d0BGTRUmNQx3PLMMuM96eT58HxqJN19qou/g8CH7xXbfaqnb5T3gAXQk2GY6/A7EQ79HmdddbReA+dqK+RLRauK3E7mOH/8pe/dLiJBy+3TrTbz3UQv4QfE0NgKBHoC7MJPNEjjzzSEQsCxQffFol7rbEN0mGHHabmAj4F4+T/vGFAqZNPSO6nHWxo8UWZrDa4t5PSiXChBDw66aSTtFu/+tWvaronq/LKxWWcbEMVuMadworGGrVR0xG7MAQMgXQE6rgxQ5xQlCooThZ+qaWW8oTrhH4kRPyIu/t///d/OhoxCyifVGxfXuzhyisVNLw4RWh+njCgFLzvvvuUw0roSnlheIlhrFxd8pq1QZkiQihM+ihBf+puKxIulF1PqEfiGNfVQ0KgQlEmhN+Ul5yXyHn+nnvu8exAInZyDVsKBawTWNFuVhvk55GhpArm6V9ZyrRCFSxL34e6H+H/o4w76VSe540SQPHAsw2CgiEtKG+x9yrZP+TLThyaL1HTQpKHwM89soNIlMa2TDhvBPnOd77jhUIVOU3wQMUzTLPztBHqyXPMUt7cLwtlXrzbmlbVTHlTAY4gjF0WN71sMuDFTljDqxWalebL3oXaXrtY5Wmj6cCkQOgXfbefbAx4ARtG2Rhl4VNG5V15mzd2WvimIZiOPABdpOOI2YQoZGySyoJScPeFhcLGp9iX//3vf6v9slEYUGJVQDfCJkw8BVausROzASsLihKIP3cb9KlTgomoE8L4g3s37BN2kQEfFjiDiKOI4hVMJ+1gRZ152ghtZx15fvKF0HcUsKwxt5KHqe/OO+/Uv9lW7h/ke+CIi7dpKSGotPKGFgjfduWVV64BN9i6OT7zzDOax1ZXRelCSbYKsRZQ5jI7d5D9iVuBIgpxMVppo6bjQ3Dx2GOPaUAksIKZIi7j6nBSlKKYFys2GWDrLBZSi7aRhIc+D5d9D4kXYtIYAf4HoIoaTo0xapQjnpmlVd6VXrDEgQD5/e9/3zD6WvBSIxpZUohbXGQ/O6Klwcy47LLLdLZPkBz+ITrZRrKP3b7mqwSBVshuKIyFYE1hb8J4+y+++GL8MvO8EVbc1Kk2MjtgmYZAnyNQaeXNllIwJmCbyOJa9KhCaE0405hHmO3BQGHT2CDkwQcPs+aQnnWEXULEOWbgzFgxmfA5itLrVBtZ7cfzeHFhzmkmjLORnH/++eoyLAGVnOwlqMUYEwLDRMzKes4vWayt2SA2ymhw0ggr2etQceO2dtto0LQlGwIDgUCllTdPiDCSCA4omDTYiQMHFwQqGvsO7r333mrHxT4qAXN0/0GCv88999xONj7VsnnCgGILxqyA8NJgtkpQeILS52lDb8z5K7xUght2/LYi4UKfeOIJvTV8YWDj5hMa5UqYUhQ3fG/MDwiOO7jQ45qPWQgbP7REvDV/+tOfapl2sGLtIU8b2pD9MgQMgcYI5FnV72WZolRBmWX7H/zgB168EnU1HYbI1ltv7UUBeXFb9qK8fQgbKQtcWkZspRrmE+obkjcMqCgwZZ/ALGH3auFAK/WQOpq1QZm8cuCBB3rZiSXqq3whRLuKUwcR1NixG5YI9Mg0kS8DLwusWoc8fT2KuULvYaduGCOwcYS3XXc7lEiJxxzdO9dcc3mJW63lOoEVFWW1UdehBglGFWwATCLZqIIJQApclpkqOIxxNFbtvc+RuL06k8W2XEQwIxCUitk0tmwWs4ItOtQDqwK3cEwpzC6LSmCmyANVF/a0SHDttpG3T4wRSY4x7/3NyvFnIXRFxRH3aPAsInmwarcN/lZgyLCAbNIYAaHOOjYLTlvHaHyX5YAAC5ZhY4kkMWKIERpfabZJHDzigYQYDGFn73g+59ilg003mZfnOrhE8zAbSVobsGLwVGwm3//+99V7s1k58rultEPbKOt2tovKg1W7bYS+2rG7CGCGHDNmjFtwwQW1IcxsvAww6a2++upu7NixuqtNK70gKiO76MAEY+KVFIgGmD+h+RIlFO/mZLmsMkxyMNNh8uNF30/SN8q7zA9lmmmmcQcddFDTLsoOK03LWIHyIMCXQ9EvkiK973b9zfrC1yyKD/+GoLgJM3zFFVdoIDO+Yr/3ve/pGhNc8rhvQLO6WYMZP368E8cv9bWgnaTcfvvtbtSoUVq/bOzgjj76aFXg7HcJWQFpVoZJDi8XxvCb3/zGDf/f2k6yrUpeyx9IqaSozbtUnbfO9BSBobR5y+K1bsvFmks3pJP1t2LzZlwyq/asv8RFHNQ8XrtBxM9B10bWWmutkJTrKIvm/umnn45CVbBGExd2vmEnH4kSGiWzYxFrWWw5h+QpE24mzIM48nnCaRSRMtu8K882qeQb0zpdeQQOOeQQd8cdd9TQKTs5qG7X36yvrCNgjmCz4LiwzsNGJ4GOSwhmhIBlRQQv5QUWWKDhDjmXX365btS86667RtXi2MUsWsJbuEcffdTlKRNuZsNj2GI//vGPQ1Llj6a8K/8IbQBFEGAhlX9gbLgseLJLEQvQCFx2YSq5LbbYQqmSpGGP5Zof8hEUq7Bu9ByKKnx1HJvwtsXzloVeKKnYaHHiwicAaaf+cL/McJX+qhV26Rd2YpQ2GCXXVrB3QzUNsd/DIqgEaetob9hGDWGH+LhAzUXYsCRPmXAv/d1zzz2dBGgr5JgX7i/j0ZR3GZ+K9akrCOAgRAgACeilIYRRwpdccoly9dl9acUVV3SsOzCjYxs5RMwGjjUL0uQzX9NQ0MwaEXbR2XDDDXVTC2aJ119/vUPB4pKOwkDZUS/eqe3UT1vUjY0X+3I3hV2oeKFhb04Ks99g/yYPb2O2Rwv+FsnyrV4HPwcJHldTBVv4IbKhc+Rgl1UmfjMvU+L0SDTReHJlz23BsrKPzjpeFAEUMUr5pZdecmFDYMwDMCZQwuQttNBCNdWyIJlkKOGcBdWUvJEjR2p50phh4yiGYxMOTggz83PPPVdn6qS3Wj91oSBhAHV7E90nn3yS5lxSKWpi7BcsKsIr4J2Ll3EnhbqREAQt1B2uyc9TJtzHMczaw4s5nlfFc5t5V/GpWZ8LIyCLVDp7RiEFxU0leNpCLX322Wej2XbeypNMk0BRjSt7NgBBQkTLvHVTLlk/Xr14xXab8iYx8bWbzZQ3dEH2QpWY80WGlassnrgIzy0ueBcjRBLNUyZ+LyYg9nAN44vnVfHclHcVn5r1uTACzNIID0wY2bhg2mDWjIQIlPH8rPOkck0rK96p2mbRBT3qylN/WpvtpgUHMHwnskS8biPbf1a5VvLCC5ZnFpdwzRdMnjLxezln5p4WciJZrgrXtX/JVeix9dEQaAEBPutR1HjhYi+NC7M4JNhT43lZ53mUKy8EgoNJuIOsqlLz8tSfemObiSE2fojz3qg6Yvq048jVqF7Sw6bBbI4dF5x6EEwgecrE74W3zsLynHPOGU+u7Lkp78o+Out4EQSYRQZzBsHK4kJccWZk5KPgkbDxBOeBFheP0IhihXoW8iiXJtAJEZgtSLv1o3y6LYHh8corr2Q2hS0/TcCESJ/tyO677663x/eaJQH6IsoXp5s8ZeJ9CAHVuvXCibfVi3NT3r1A2dooBQInnniimiJOPfVU9eqjU8zCYZFAGWR2Lg4guhh58cUX60IjOzUde+yx2n/4zWHWTogE7LEwF1BiYZGPgrBCEJgNLIjCzmDhEmmnfrwRsfOOGzdO6+rWL/qIeSnLTg+FEuYJzJ24oLQJUxHigcTz0s4DqyRpyuAFQuhlPDoDlROMiaHPwi/rC3nKxNsMtm52wOoHMeXdD0/RxpALARYn4QfDNkFBEU+D7eSOPPJI5WpTCYtaKHJMBmx/BW0ON24WCVnAk82ntS2cRZh9M/sj7G74hCeTlwN2dJQ22+fBQGGhDGmnfj77eWGERTutsAu/MJtAe5Q9TesWDENzzGJ5OUG/jAtfI6Rj44f33kgeeOABXegMcfjhYPMc4kL7hCLmi4iFUcqccMIJTjxro2J5yoTC9IcXSzcWWEMbPT3KH0OpxNzjS/U4St2ZVt3j5bPeS9x3DUsrSjp1jLhRi2LSPFGWGvI3WfDtt9/2MmOMksU0oq7iuG2zgbQo+igvedJK/dRBm0WlFfd42sE9XRxhUpsTE5KX9YPUPFnw9EcddZSXeCSp+UUTCYEsM/TM25qV4ZlI0DgvQa4y60lmltk93njePX1VWmNlQID9NhdZZJHMrsRDBgc7dfKGtJDAlMHkECJcJu8J163W36jNUG+njrSDyWL//fd3iy++uJpB4nXz1REWeuPpnGNuwfvxlltuSWa1dM16RbNFxqwybCK89tprq7dstznyLQ2wxZtMebcInN1mCCQRCHZbGBEwMaoum266qcNUg5mChUM8KfMIC8CUDzzsPPd0qwxbBY4ePdptv/32untUt9oZinpNeQ8F6tZmXyGAnZcFSZQWgiMN2+KhMKouxM/Gfh88G/OMZ6+99spTrCdl+GpigTO+JtGThnvQiCnvHoBsTfQ3AphhTjrpJP0JIw2bUYTrKh+bmYDKPDaeTT8qbjA35V3mvzzrW2UQKIOJoDJgWUc7goBRBTsCo1ViCBgChkBvETDl3Vu8rTVDwBAwBDqCQCnNJjg14ARhYghkIcCmADjB2N9KFkpOo+jhpWg4ZeOUltuum39anZ1KK53yXnnlldX1tVMDtHr6FwE8JhsJdD2CGmWVaXRvv6WzYNevi3bdflawbNgNqVf8+iLjGYZHUZEbrKwhUAUEiAPCTjmBe12FPlsfDYECCIw3m3cBtKyoIWAIGAJlQcCUd1mehPXDEDAEDIECCJjyLgCWFTUEDAFDoCwImPIuy5OwfhgChoAhUAABU94FwLKihoAhYAiUBQFT3mV5EtYPQ8AQMAQKIGDKuwBYVtQQMAQMgbIgYMq7LE/C+mEIGAKGQAEETHkXAMuKGgKGgCFQFgRMeZflSVg/DAFDwBAogIAp7wJgWVFDwBAwBMqCgCnvsjwJ64chYAgYAgUQMOVdACwraggYAoZAWRAw5V2WJ2H9MAQMAUOgAAKmvAuAZUUNAUPAECgLAqa8y/IkrB+GgCFgCBRAwJR3AbCsqCFgCBgCZUHAlHdZnoT1wxAwBAyBAgiY8i4AlhU1BAwBQ6AsCJjyLsuTsH4YAoaAIVAAAVPeBcCyooaAIWAIlAUBU95leRLWD0PAEDAECiBgyrsAWFbUEDAEDIGyIGDKuyxPwvphCBgChkABBEx5FwDLihoChoAhUBYETHmX5UlYPwwBQ8AQKICAKe8CYFlRQ8AQMATKgoAp77I8CeuHIWAIGAIFEDDlXQAsK2oIGAKGQFkQMOVdlidh/TAEDAFDoAACprwLgGVFDQFDwBAoCwKmvMvyJKwfhoAhYAgUQMCUdwGwrKghYAgYAmVBYJKydMT6YQi0isATTzzhRowY4b744ouoCu+942fSSSeN0oYNG+Y22GADN2HChCjNTgyBqiJgM++qPjnrd4TAwgsv7IYPH+4+++yz6Ofzzz9XZR5P+/TTT92YMWOi++zEEKgyAqa8q/z0rO8RAttvv72bZJLsD8nJJpvMbbjhhtE9dmIIVBkBU95VfnrW9wiBLbfcUmfdUULiBMU+evRoN+WUUyZy7NIQqCYCpryr+dys1wkE5p57brfUUks57Nppgvlk6623TsuyNEOgkgiY8q7kY7NOpyGA6WSiidL/pKeeemo3atSotNsszRCoJALpf+mVHIp1etARGDt2bA3jJOAB42TzzTevYZ6EPDsaAlVFwJR3VZ+c9bsOgZlnntmNHDmybvYNy8RMJnVwWULFETDlXfEHaN2vRWC77barTZCrmWaaya2yyip16ZZgCFQZAVPeVX561vc6BDbeeOOamTcmk2233bYmre4mSzAEKoiAKe8KPjTrcmMEpplmGrfeeutFnG9MJltttVXjGyzHEKgoAqa8K/rgrNuNEdhmm20cHpbIXHPNpRTCxqUtxxCoJgKmvKv53KzXGQisv/76boopptASO+ywQ0ZJyzIEqotAtj/xEIzrnnvucW+88cYQtGxN9hMCOOzcfffdbrrppnNXXnllPw3NxjIECKy22mpuxhlnHIKWGzc5TCKv+cbZvc/BXnnTTTf1vmFr0RAwBAyBBggwEVh55ZUb5A5J8vhSmk341A0hPe3439CmhkM9Duuuu67DqzING9zhL7jggtS8tPL9nHbccce5+eef37D4X5jgIs/69ddfHxLNnKfRUirvPB23MoZAFgITTzyxUgSzylieIVBlBEx5V/npWd8NAUNgYBEw5T2wj94GbggYAlVGwJR3lZ+e9d0QMAQGFgFT3gP76G3ghoAhUGUETHlX+elZ3w0BQ2BgETDlPbCP3gYOAtdff71uXnzrrbcaIA0QgGr49NNPR7lXXXWV7gW60korucMPP9w9/vjjUV7Rk9dee83ttdde7sUXX0y99eGHH3aHHnqoxqsZN25carmsMh9//LEjXMLbb7+dWn+VE015V/npWd/bRgCl8fLLL6cqhbYrL1AB3OOyySeffOLwucCzcMEFF9TunX322e7MM890J598sjvqqKPcNddc4xZbbDE9Fun/W2+95fbdd18377zzutNOO8199NFHdbfffvvtboUV/l97Zx5zx/jF8aciCIIQ+xJLGqkltVOprdTyB6otKmJX1JooSiPW+NVSJRolQhFaKiFqiTaWllhasdVOkVr+ELGL4B/P73xOnGvuvDP3zty5974z4znJe+fOsz9n7nvmec75nvOMcJyCdPvtt7vly5c7XhjffPNNo2y7MquvvrrjkI699trLrVixolGvFl/kR1MqEscLLz+YUo0pDKacHOC3Ik46hQcnwrtwG0UakCBaXtyvvTgWFWkmta6snL046aTmJ2UwllGjRvkpU6Y0Ze+8887+6KOPbqQ9+OCDvHX86NGjG2lZvrz11lteVvNeDs/Q+u+++25TNQmR4SW0gd9ll10a6b/88osXQe5FoGtaljJWWV46yoNff/3VkjJdxUlHxycelpnK97HQ/8LKuxav4DCJIhzYYostilQvXHfq1Klu0aJF6gFZuLEuNXDfffc51BFXXHFFU4uswp966imHByskwlWvP//8s16zfshLwG277bZu8803T6wyb94899NPP7mJEyc28tdaay1dRb/22mtu2bJlLksZq8wOgkiT06dPt6TKX4PwrvwjDBMowgF0uehtUQdACCFZTbqxY8e6r7/+2s2cOdMdfPDB7tRTT3Xffvtto6u3337bXXrppe62225TvTkxw/fee2+HXtbC0U6ePNlNmDDBHXfccVrvk08+0ePYSDv33HM1DcF9ww036PfTTjtN9bvWydlnn+1OP/10u+3bFT0xQnv8+PEOtUOU0HfLqrkRL505QQceeGC0WOHvFt9oxx13bGpr++231/v58+c3YiC1KmOVV155ZeX5zTff7FDZ1IFKF1WwDkwNc6gGB+bOneuuv/56NbhhFINmzZrlpk2bpjrYjz76yCEsfv/9d3fvvfeq0evJJ590L774orvwwgtViBG1kJjhrEB5EbAqRCdLOwjljTbayP3www/uoYce0pUmaZTfdNNNVdd75JFHuscee8whBE855ZRGKFt0wLxQ/v77b3fjjTe6ddddt29Mffrppx0xPQ499NABfbL65c/o4Ycf1jlecsklltSV61dffaXtbLzxxk3tcU4phJ0iS5loZfTlPMt77rnH8dKsOoWVd9WfYBh/xxxgtWyrXmuEf+oxY8boLStrVpqvvPKK22CDDdxLL72k6fvtt5+bMWOGft9uu+1UiCMQlixZ4lZbbTV31113qQBntTd06FBrWq+bbbZZU2jRPffcUw1yQ4YM0cOTuYfWXHNN98wzzzgEaT8FN31/+OGHXFxccGpi5OO7775TQyXGRF5i3STahiwuu7Vt9+RnKWP1uNqq/f33348mV/Z7EN6VfXRh4N3gwKqrrjqgGRMQhrCgAEJajF3ur7/+0vJrrLGGXkFaIHghIveBbEBtkldAWBva0D8fBx10UOLqN1qmF9/ZcUDthDdRGzn4AhVTt2mdddbRJsUA2NQ0x9pB7FyylIlWRgXEmaY2v2heFb8H4V3FpxbG3HcOEKUwC2299dZaLK8BL0l4Z+mvF2XQeUOrrLJKy+ZfeOGFATuXlhVyZJoR+bfffmuqZfe8TLOUaaosN7yYUWPVgYLwrsNTDHMoDQdsVRdXl7QbYJmENzsIKAl7HZ0HqiRw2r2gYcOGabNffPFFU/M49UCoQLKUiVYGt87uKQ3hEi1bhe9BeFfhKYUxVoYDixcv1hWhISDQe0MIDQg1AGoVDJFGCG7SDH5n6QgbU9NYWj+uNnbQNq1o9uzZidnM448//kjMy5o4adIkLbpgwYKmKsAXEb443WQpE61saKFevXCiffXjexDe/eBy6KO0HDChasgFBmr/5OIU0hg36AYovuVGdWArVM7KBIaGO7mpHIAZQkD+MD6Kg4vil0Fz4JqPEN9kk01UqGP0RCBiMES3ywoYQRUdhzbW4w+8GldaaSX3zjvvpPYESgfUCUbVKCG0UR0xJ44Oa0fG9zhfeYEce+yxirix02zgC2EM7rzzTofNIUuZaP+2Kzr88MOjydX93kePoExdBQ/LTGwKhYQDRT0sxS3bi6BRDzoRRF7gbl7QJ170opo2fPhwv3TpUn/GGWfovfyXq0fg999/72UFqGkiXLUNvP7WW289f8cddzQ9G1m9ehEyjbKCRPG77bab33fffb0gVtSrcs6cOV5W3/onEEQvAt3LitsLzNCLUc6Ls0pTm3lvOvGwPPPMM72oJnQsSf1dcMEFOic8LKMkOmkv6BjNg29pJJBKf9RRR3mxJWhZ+hJseVNx2Xl4wb779ddf38vZtuqFKm75uctYBYlxos9KdjmW1PZaZg9L3viloiC8S/U4Sj2YosK7yORMeMvW3eO2LTht30oo4IKPUIYEazygawmc5GX12ZROuaSyTYUy3HQivBkP7uniLJPYA3MRPHtinhg8vcQ98QIhTMzPm8iLTFboLau1K4MrvcAv/eOPP96ynXhmmYV3cNKp7qYpjLwkHIg7riQNy5AR5MW9FknD7TxOSeXiZXp1z3hwEsJLVHYgqgaJ9oWeHrheEqFuwUNy4cKFSdm501BBtTMytiqDm/0hhxyiDlk4RdWFaie88W4j9gJ/eFRddtlllXtWOIMQuQ096zbbbKNuvbvvvnuueYAzNpfvaEUitOGlhj5VgiE1dLPRMuF7ew6YjtbQD+1rVK/EuHHjHEZT2eE4DId4i2ahN954Q8sbDjtLnV6VweCL05UEMHPnnHNOr7oZlHZrZ7DE6wqPOIxDWPCrRmOxGJIAAA77SURBVLhh48GHsQePPRwhMCDljTcNjAoPwgceeEDjbxCPAocSnBRoG+cK0SWqca1qPBrs8SLICGcKPfvss+rWbt5+gz22bvdPXBbc981xKUv7xOfutsdlln6TyoD2wcBJOIO6Ue2ENzEmTjzxRH1OWR0ryvJQcexgp0DsDKzwxJpm1cBLKO8Ogrnjar3HHnvo9MQ45Ah8dOWVV7pHHnlEvcx23XVXDcpE0KU4TK0sPCnjOCSMqcYwYTsOEuKWW25R9/kyjrUbYwI9svbaa3ejqb63wf+B4cH73nmPO6yd8IZfwJyiV72pwAcrHFYIgkTQ0aLnI+IdlNfdWivJRzSIkKVxBetK3A50saz2WS0FysYB4pegEoj+ZasZSgUOdI8DtdB5gy0Fd4q6BLUAUdugqNcaxyBx+gfGFPS+6MAsahorXnTkCE/KiEXaET2OIEKE+Izq+igDnpc6rEiOOOIINYbQX6s+yG9HGFUEbtZUjJ0EaRZLwzIJF4o+8u6777ak3FcCHt16660am4L5Csyt0QY84A/cMmMgahyrr27xio7S+mgMInwJHAgcSOdAHBoz2Pd5oYLiZOFl++9Fh6t4WAHiN7C7EkpTpyMutl4Mf150X14EfOP0DnGK0PzrrrtOYUTCJS/BiLwYarzEZlb8Ke0avfrqq4phBecrLwwvMYwVq0t+uz6sjbxXEZaK/41iZsHSii7Pyw5jALws3j6nnjAviWMcz9J7g0JR5vPPP9e0yy+/3Ive3b/88stejJ5eXoheAud7IGDd4BWdtOpDB5HhYzChghmGV5oinUAFSzP4QR6I/X+U8SSdyuO8EQIIHnC2RggY0kx44wwA2N8IBwHyJWqaJXkA/KTJCSKNNInd4HHeMDrrrLO8QKgaThM8UFnxa3aWPqydPFeJLe1lxesF1dBUTQxlXrzbmtKSbtoJb+rgCMLcxbjp5ZABdZyI4mplR6D5goDRLoryKksfSXOJp9m4GHv4a80DXsCBR6151Io/ZRTelVebsNUHb2rBdOQBNIx0qE2IQsYhqegnzd0XAyAQPHTjf/75p8ZgNmt6PAwosSqAGxE6lHgKWK45wgn1Cm62O+20U+Y+GFsewmh50003qYERd+MoES60G8T8zb0b9AmnyMAfDJxGBLCHX+amXYRXtJmlD+u71ZXnBzKnbhCwVnPuJA9VH0ZwfrOB8nEAo7R4m+ar1KfSlRbewLPA244cObKJXabr5vrpp59qHoa/vHChOFqFWAsIc1mdO8D+xK1AECFkoU760IoJHwhM9Omc6mLxMRKKFU6Sg18VaQKvQKZwkjdGTrC6eSgrrzhkgFNjOukjPh7GvOWWW2q8kHheuP+XA/wPABUlrkqgfBwQz8zSCu9Ko00w2EFvvvlmavQ181IjGlmciFuc5zw74HQgMzj6idU+2Gv+IbrZB2Nk5Xv88cfr+Yei644Pu6v37EogYIVgc5kLwZoQsHECupiV0nhF/W71kXUsoVzgQB05UGnhDRoExARoEzGuNZ6PYZaJ2MZ2n9UeCBEOjTUiDzy4rZotvdWV7TkR51iBs2JFZcJ2FKHXrT5Er+sQ2ASb5xguI6LMRbdvvLhQ57Qj5plG999/v6J0QLNwxiLEnCDOdGQsRmKsdVOmTLHbttc0Xv34449d66PtIEKBwIEac6DSwpvnYgef4oCCSuO9995reA0CRcPV/Pzzz9fVLPpRCZij5w/us88+bquttnLmdp4lDCgrYtQKEC8NVqsEhScofZY+tGKbj4suukjDghJL+YQTTtA/XhZ4RxpmO0+40A8++EB7tB0GOm620AhXwpQiuMF7o36AeHEApXz00UdVXYOOH1gi3po4o0BFeIXtIUsf2lH4CBwIHEjnQNyCP9j3eaGCssr2F198sZfANGpNByEiKgcvAsjLadxehLeXVaqiTYDWCSca0Dugb1DWMKAiwBR9ArJEvBL9YYcdptBD2mjXB2XaEbA8xpf2Z4gaIqi1CxcqOwMvBtamtkRdocgSeWEpugY0DlDEOAGJFANpo67g5r3ErdZi3eAVDbXqIz6etPsAFUzjTHN6gAo28yPPXZmhgkOYSLpo73+OxO3VlSy65TyEGoGgVKym0WVjzDJdtLUDquKzzz5TVQqry7xkyBRcoolilhQJrmgfWcfEHKH4HLPWb1eOn4XgvpWPOCPBzzyUhVdF++C3AkIGA3KgdA4IdFZj5CTZMdJrhRw4gMHSDpaIAyMGmUPTKo02iTIPYYqQgeLeiFYOvbTpdC0tzxW3aCgO24u2kdQHqBhZpUeLJX5HZUIgoCzUK6FtfSOsixwXlYVXRfuwsYZr/zkgq3k3fvx4B7SW3zcvCNRxqBAx4gOjTfs/bDVaXuio7JYvX54YzwfoL2o+wAL8xuz0eltczJo1S21QWf+PWo2l7Hm1Ed5lZjRu5VHjY9pY5YSVtKyQXkIOIGhMaPRieL1uv5Mxs8PFZoHPA4KbHSCxeK6++mo1aGMTQnDyW8bITbiKrDR//nwNnLZs2TIN2RCvBxAB/wPxcFa7EHYcQAeUv+aaa7T4ySefrBBbjky79tpr403U6j4I7z48Thx8Asa2D4zuYxegeHCUIiRsHOPejWH0uv1OxojBnh0kRn7xNtYmnnvuOV0lY7hnR0osIJBd+AkAz5UwE5m6Yr7snlnRp50xKcekueeff17P/iTeDoThnRcHsfuJDcSOlDhFnG9JYDdeNHWlyqNN6vpgwrzKzYGpU6e6RYsWNcEpuzniXrffyVixLSCQEaJG2H0QvKyajUywEsQsK+Ety4sBiGwS8eLgcBEJaeGwdRiBmIJARRmhTgHqCrQVaGpdKQjvuj7ZMK9EDmBInT59uuprEQLESccADbHNN+co9K4Q0SonTJigf+RDCFZB3eh3IKoIChyb8LbF8xZDL5BUVoPsuPAJgIq0b/VHjx6t8FdtsI8fqEcQ2ui5o/YWVtaoLaLeyxhGEaDMv1v0+uuvq7qEFT6C3ghfD9SSHLtm/h3kSfwdx0EMeCjXlf7lQl1nGOYVOPAPB1iFEQJAAnqpbhUhPHfuXMXqE04YQYSudt68eY346azeEA6kffzxx9oSAlqiLOp3gaOqjpXQCBMnTtQtPQIWl3SEB8Y12sU7tUj7dPbEE0841BTEKuk3cTIVLzkLoxztHx8Ei3cDjxDmHPoB37pF5kzHyjtOGEnRxYMMMYL3PGs5BFl9PCy9Tteg867T0wxzackBBDGHWqxYsUIPoaAwqoBRo0bpUWbkxbftGCTjCCWcs4Cakrf//vtrn6Sxwl4sjmJs4S0eDSvz2bNn60qd9E7bpxMc0kAADcYhuhgAoSThqRn/fHD2KoHbQE51k4iVD9lLItq2pYF6iR5UzCqdFTk7oWjgumjdKn8PK+8qP70w9swcALnB6hnhEz3JHU9bIG1A0/KeVhRHmhg0LirsJWSxjtEiWmYesBSMt49XLzpesO39JomTr122Et54/s6ZM8dJnHzddXRzjLaKR78eJ/qF4qfZWx0be7xe1e+D8K76Ewzjz8QBVmVghKP6UiqyvWbVDFkESr3J8BEXrklVONWJPvMY76ydLO1b2V5fzSkMREgaETsIxAcr726TvXAtfHG0fZ4r8YWiJ16Rb7p5Io/WkYLwruNTDXMawAH+uRHUeOF++eWXTfm2Yttwww2b0tvdZBGuvBBYLQ4dOrRdcwPys7Q/oFKPEkztkCQ8rUv00sTh6QXRPy9BObGqqXlW3ahUUJHECTw4FFWlxMtU+T4I7yo/vTD2zBxgxWjqDIKVRQlMMnpT8hHwkB08wXdDMUS37AhW4GuWR7kkAk4IGbytaPugWgaDwE1DBExLo7Fjx6rHY1I+fCL6Z6eEumbMmDFqDLXAaLSFnYHomsccc8yAps2AWcRTeECjJUoIwrtEDyMMpbccmDFjhuqRZ86cqSco0RurcBASQAZZnY8YMUKNkehuMTRyUpN56uH8Yat2HFLQo6PfpZwZ9GgTVAjEgRoYRPFExHAJFWkfYyBRGXGE6Tcxbla+abp7TpxBtXHAAQcMGBpCm9AVFiNkQIFIgqFKklQdQDIZA/BNIw7QRu2Fs06c0HXvsMMOGu8onleH+yC86/AUwxwycQDjJM4koE0QRmCT8ZIE1oZggNCTIshRDxA/HYgcWGGMhKz+bCvOSo/V96RJkxxhd4cNG9YYAy8HBApCG+9aECjmJl6kfeBwvDDMQNfosA9fUFsAhZRzThMdk1j9InDlrNUBo2GHwosMvT9Y+DQihIR5RLI7Oumkk9zSpUsbxXH+IaYJzwS9OrBFVvSMCaEeJQLQLVmypOuol2gfg/5dfgylorwhYUs1+DCYvnKg05Cw8g/vJe67hqUVIZ04ZlFPeMGFa54ISw35Gy8oulYvAquRLKoRDaMr23Uv8DQvgr6RF//SSfu0QZ95qVshYelbdide4HeJQ4AXYthMzCP9qquu8oK7TszPm8iB3Gl90Za8mL28oD3hk4tQmUPCBpz3oL8+wwD6zQFikbCdbkXRkMGmp46XTwoJTBlWgRbhMl7H7jttP61Pa7eXV/rGRX3y5Mlu+PDhA6JrAmVMI9QtYK4XLlyYViRXeqvInqiycMSSE981XkquhitUOAjvCj2sMNRyc8D0tKgO8PqrI40bN069GWXX4xYsWDAAnpc2Z4zClEdn30viTFZsA/TFKVF1pmZFUZ1nGuYWONAjDqDTJRQpAgrCkYbzQetKhHzFRd88G7PM87zzzlODcJayRcpgeyDUgeHCi7RV9rph5V32JxTGV3oOoIYB9cCfkR1GYfd1u7ZTCw3WfDES/1coCO//ypMO8+wpB3qtDujp4EPjleRAUJtU8rGFQQcOBA781zkQhPd//RcQ5h84EDhQSQ6UUm2CU0OSu2slORwG3TMOcOAtTjDht9KaxXgaEos78Kk1n5Jyi7j0J7XXzbTSCe+RI0d2dOp0N5kS2qoGB/CYDNSeAyAwoh6g7WuEEsYBEDWchjSY+HobS/w6BO+jeGK4DxwIHAgcCBwoNQemBZ13qZ9PGFzgQOBA4EAyB4LwTuZLSA0cCBwIHCg1B4LwLvXjCYMLHAgcCBxI5kAQ3sl8CamBA4EDgQOl5sD/AWaWJ6oQ2tViAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keras.utils.plot_model(model, './models/chap10-models/mnist_model.png',\n",
    "                       show_shapes=True, show_layer_names=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:07.221449Z",
     "start_time": "2020-01-03T11:05:07.071245Z"
    }
   },
   "outputs": [],
   "source": [
    "# after the model is created, must call its `compile()` method\n",
    "model.compile(loss='sparse_categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              # a list of (multiple) metrics to compute during training and evaluation\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is equivalent to\n",
    "\n",
    "```python\n",
    "model.compile(loss=keras.losses.sparse_categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(),\n",
    "              metrics=[keras.metrics.sparse_categorical_accuracy])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Use the **`loss='sparse_categorical_crossentropy'`** because we have **sparse labels** (i.e., for each instance, there is just a target class index, from 0 to 9 in this case), and the classes are exclusive (互斥的).\n",
    "\n",
    "\n",
    "+ If instead we had **one target probability per class for each instance** (such as one-hot vectors, e.g. `[0., 0., 0., 1., 0., 0., 0., 0., 0., 0.]` to represent class 3), then we would need to **use the `loss='categorical_crossentropy'` loss** instead.\n",
    "\n",
    "\n",
    "+ use the **`'sigmoid'`** (i.e., logistic) activation function in the output layer, and **use `loss='binary_crossentropy'`**  when **doing binary classification** (with one or more binary labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:07.226800Z",
     "start_time": "2020-01-03T11:05:07.223177Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 7, ..., 3, 0, 5], dtype=uint8)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:07.234747Z",
     "start_time": "2020-01-03T11:05:07.228888Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 0., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 1., 0., 0.],\n",
       "       ...,\n",
       "       [0., 0., 0., ..., 0., 0., 0.],\n",
       "       [1., 0., 0., ..., 0., 0., 0.],\n",
       "       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert sparse labels (i.e., class indices) to one-hot vector labels\n",
    "keras.utils.to_categorical(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:05:07.242915Z",
     "start_time": "2020-01-03T11:05:07.236549Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 0, 7, ..., 3, 0, 5])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(keras.utils.to_categorical(y_train), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regarding the optimizer, `\"sgd\"` means that we will train the model using simple Stochastic Gradient Descent. **<font color='blue'>When using the SGD optimizer, it is important to tune the learning rate.</font>** So, you will generally want to use `optimizer=keras.optimizers.SGD(lr=???)` to set the learning rate, rather than `optimizer='sgd'`, which defaults to `lr=0.01`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:16.525037Z",
     "start_time": "2020-01-03T11:05:07.244634Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 55000 samples, validate on 5000 samples\n",
      "Epoch 1/30\n",
      "55000/55000 [==============================] - 5s 85us/sample - loss: 0.7228 - acc: 0.7623 - val_loss: 0.5304 - val_acc: 0.8160\n",
      "Epoch 2/30\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.4838 - acc: 0.8323 - val_loss: 0.4494 - val_acc: 0.8454\n",
      "Epoch 3/30\n",
      "55000/55000 [==============================] - 4s 80us/sample - loss: 0.4379 - acc: 0.8475 - val_loss: 0.4173 - val_acc: 0.8548\n",
      "Epoch 4/30\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.4126 - acc: 0.8555 - val_loss: 0.3898 - val_acc: 0.8660\n",
      "Epoch 5/30\n",
      "55000/55000 [==============================] - 4s 80us/sample - loss: 0.3930 - acc: 0.8626 - val_loss: 0.3987 - val_acc: 0.8610\n",
      "Epoch 6/30\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.3776 - acc: 0.8672 - val_loss: 0.3730 - val_acc: 0.8732\n",
      "Epoch 7/30\n",
      "55000/55000 [==============================] - 5s 83us/sample - loss: 0.3645 - acc: 0.8709 - val_loss: 0.3663 - val_acc: 0.8692\n",
      "Epoch 8/30\n",
      "55000/55000 [==============================] - 5s 88us/sample - loss: 0.3526 - acc: 0.8743 - val_loss: 0.3614 - val_acc: 0.8736\n",
      "Epoch 9/30\n",
      "55000/55000 [==============================] - 5s 87us/sample - loss: 0.3414 - acc: 0.8787 - val_loss: 0.3648 - val_acc: 0.8746\n",
      "Epoch 10/30\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.3323 - acc: 0.8814 - val_loss: 0.3480 - val_acc: 0.8790\n",
      "Epoch 11/30\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.3247 - acc: 0.8839 - val_loss: 0.3387 - val_acc: 0.8798\n",
      "Epoch 12/30\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.3171 - acc: 0.8862 - val_loss: 0.3406 - val_acc: 0.8800\n",
      "Epoch 13/30\n",
      "55000/55000 [==============================] - 4s 73us/sample - loss: 0.3093 - acc: 0.8887 - val_loss: 0.3294 - val_acc: 0.8834\n",
      "Epoch 14/30\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.3029 - acc: 0.8915 - val_loss: 0.3228 - val_acc: 0.8848\n",
      "Epoch 15/30\n",
      "55000/55000 [==============================] - 4s 77us/sample - loss: 0.2960 - acc: 0.8944 - val_loss: 0.3247 - val_acc: 0.8838\n",
      "Epoch 16/30\n",
      "55000/55000 [==============================] - 4s 75us/sample - loss: 0.2896 - acc: 0.8952 - val_loss: 0.3293 - val_acc: 0.8788\n",
      "Epoch 17/30\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2833 - acc: 0.8979 - val_loss: 0.3109 - val_acc: 0.8896\n",
      "Epoch 18/30\n",
      "55000/55000 [==============================] - 4s 70us/sample - loss: 0.2773 - acc: 0.9001 - val_loss: 0.3264 - val_acc: 0.8848\n",
      "Epoch 19/30\n",
      "55000/55000 [==============================] - 4s 82us/sample - loss: 0.2730 - acc: 0.9021 - val_loss: 0.3572 - val_acc: 0.8660\n",
      "Epoch 20/30\n",
      "55000/55000 [==============================] - 5s 84us/sample - loss: 0.2683 - acc: 0.9034 - val_loss: 0.3015 - val_acc: 0.8926\n",
      "Epoch 21/30\n",
      "55000/55000 [==============================] - 4s 78us/sample - loss: 0.2628 - acc: 0.9059 - val_loss: 0.3175 - val_acc: 0.8846\n",
      "Epoch 22/30\n",
      "55000/55000 [==============================] - 4s 72us/sample - loss: 0.2585 - acc: 0.9063 - val_loss: 0.3164 - val_acc: 0.8878\n",
      "Epoch 23/30\n",
      "55000/55000 [==============================] - 4s 72us/sample - loss: 0.2549 - acc: 0.9083 - val_loss: 0.3035 - val_acc: 0.8924\n",
      "Epoch 24/30\n",
      "55000/55000 [==============================] - 4s 74us/sample - loss: 0.2483 - acc: 0.9101 - val_loss: 0.3013 - val_acc: 0.8930\n",
      "Epoch 25/30\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2451 - acc: 0.9123 - val_loss: 0.3071 - val_acc: 0.8890\n",
      "Epoch 26/30\n",
      "55000/55000 [==============================] - 4s 70us/sample - loss: 0.2409 - acc: 0.9144 - val_loss: 0.3194 - val_acc: 0.8828\n",
      "Epoch 27/30\n",
      "55000/55000 [==============================] - 4s 71us/sample - loss: 0.2362 - acc: 0.9146 - val_loss: 0.2990 - val_acc: 0.8896\n",
      "Epoch 28/30\n",
      "55000/55000 [==============================] - 5s 82us/sample - loss: 0.2333 - acc: 0.9157 - val_loss: 0.2989 - val_acc: 0.8946\n",
      "Epoch 29/30\n",
      "55000/55000 [==============================] - 5s 90us/sample - loss: 0.2290 - acc: 0.9178 - val_loss: 0.2978 - val_acc: 0.8946\n",
      "Epoch 30/30\n",
      "55000/55000 [==============================] - 5s 96us/sample - loss: 0.2251 - acc: 0.9181 - val_loss: 0.3308 - val_acc: 0.8816\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, epochs=30, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ `epochs` default to just 1, which would definitely not be enough to converge to a good solution.\n",
    "\n",
    "\n",
    "+ Keras will measure the loss and the extra metrics on validation set (this is optional) at the end of each epoch, which is very useful to see how well the model really performs. **<font color='red'>If the performance on the training set is much better than on the validation set, your model is probably overfitting the training set (or there is a bug, such as a data mismatch between the training set and the validation set).</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At each epoch during training, Keras displays:\n",
    "\n",
    "+ the number of instances processed so far (along with a progress bar)\n",
    "\n",
    "\n",
    "+ the mean training time per sample\n",
    "\n",
    "\n",
    "+ the loss and accuracy (or any other extra metrics passed) on both the training set and the validation set (optional)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red' size=4>Tip:</font>**\n",
    "\n",
    "Instead of passing a validation set using the `validation_data` argument, you could set `validation_split` to the ratio of the training set that you want Keras to use for validation. For example, `validation_split=0.1` tells Keras to use the **last 10%** of the data (before shuffling) for validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ If **the training set was very skewed**, with some classes being overrepresented and others underrepresented, it would be useful to set the **<font color='red'>class_weight</font>** argument when calling the `fit()` method, which would **give a larger weight to underrepresented classes and a lower weight to overrepresented classes**. **These weights would be used by Keras when computing the loss**. \n",
    "\n",
    "\n",
    "+ If you need per-instance weights, set the **<font color='red'>sample_weight</font>** argument (**if both `class_weight` and `sample_weight` are provided, Keras multiplies them**). Per-instance weights could be **useful if some instances were labeled by experts while others were labeled using a crowdsourcing platform**: you might want to give more weight to the former. You can also **provide sample weights (but not class weights) for the validation set** by adding them as a third item in the `validation_data` tuple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The `fit()` method returns a `History` object containing:**\n",
    "\n",
    "+ the training parameters (`history.params`)\n",
    "\n",
    "\n",
    "+ the list of epochs it went through (`history.epoch`)\n",
    "\n",
    "\n",
    "+ most importantly a dictionary (`history.history`) containing the loss and extra metrics it measured at the end of each epoch on the training set and on the validation set (if any)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:16.531405Z",
     "start_time": "2020-01-03T11:07:16.527107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 32,\n",
       " 'epochs': 30,\n",
       " 'steps': 1719,\n",
       " 'samples': 55000,\n",
       " 'verbose': 0,\n",
       " 'do_validation': True,\n",
       " 'metrics': ['loss', 'acc', 'val_loss', 'val_acc']}"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:16.536342Z",
     "start_time": "2020-01-03T11:07:16.533455Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]\n"
     ]
    }
   ],
   "source": [
    "print(history.epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:16.543409Z",
     "start_time": "2020-01-03T11:07:16.538989Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loss', 'acc']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:16.841374Z",
     "start_time": "2020-01-03T11:07:16.545488Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAE3CAYAAAB/8eJFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdeXxcdb3/8dd3zqyZSWYy2ZM2TdOW0pXSDbrRsiOCiFzlckVANkXxp3j16vWKy9WHoFz1epGLgKB4FRUVRMpeoEDL0r10o3RNm63Zl0lmn+/vjzPZ0zZp02aSfJ48zuOcOec7k29OQ975nu/3fI/SWiOEEEKI1GAZ7goIIYQQoosEsxBCCJFCJJiFEEKIFCLBLIQQQqQQCWYhhBAihUgwCyGEEClEglkIIYRIIQMKZqXUnUqpDUqpsFLqt8cpe5dSqlop1aKUekwp5RiSmgohhBBjwEBbzJXAD4HHjlVIKXUp8E3gQmACUAp8/2QqKIQQQowlAwpmrfVTWuu/A/XHKXoj8KjWeofWuhH4AXDTyVVRCCGEGDuGuo95BrC12+utQJ5SKmuIv44QQggxKlmH+PM8QHO31x3b6fRqbSulbgduB3A6nfOKi4uHuCqjWyKRwGKRsXuDIeds8OScDZ6cs8Ebq+fsww8/rNNa5/TeP9TBHAAyur3u2G7tXVBr/TDwMMDUqVP17t27h7gqo9vq1atZsWLFcFdjRJFzNnhyzgZPztngjdVzppQq62//UP+JsgM4q9vrs4AjWuvj9U0LIYQQgoHfLmVVSjkBAzCUUk6lVH+t7d8BtyilpiulfMC3gd8OWW2FEEKIUW6gLeZvA0HMW6GuT25/WylVrJQKKKWKAbTWLwI/AV4HDgFlwHeHvNZCCCHEKDWgPmat9feA7x3lsKdX2Z8BPzupWgkhhBBj1NgbBieEEEKkMAlmIYQQIoVIMAshhBApRIJZCCGESCESzEIIIUQKkWAWQgghUogEsxBCCJFCJJiFEEKIFCLBLIQQQqQQCWYhhBAihUgwCyGEEClEglkIIYRIIRLMQgghRAqRYBZCCCFSiASzEEIIkUIG9DxmIYQQYkyIRyHaDtGQuY4l19Fg332xCKBB6xNbH4UEsxBCiKGTiJsh1iPQkkss2Ou1Waa47EN4c30yq44TZH2OJSAeg0TUDNVE1Hwdj3RtdxzrPB6FRMxcx0I965eIDdOJ6yLBLIQQo5nWEAubAXTMdbcWYTTYz/oY+7qHcDwy6CqWAhw4XikFSvXaTq4NO1isYNjAYgPDmtyX3LbYzGOGHayOnuWsLrA5wZYGNlfy9dH2pXXttzrNz+per+51Gsj6+45+v1MJZiGEOBmxCETbINIOkTbcgYNQ9b7Zkuto0Wl6ve5Y994XN8OxoxXXGXihnsHZ43i31md/oXsCQdlFdYVT5zq57fRCen7PsOoRYsnF6uwbaL1ev7H2HZaft7z/AOsM47FDglkIMfIlEhAP9wyoeNQMpXjkKNvd9sUiPY93BG20HSKBY28noj2qsgBgwyn4Hi22bkHn6rntSAdPbrIVl2zJDWjda5/d3TN8DftpCUZtsYHVfsq/zkghwSyEODXisX4Gz/TX39j7WHuvlmC3y6xHW8fDQ1t3IxlSdrcZUB3bnnywJ1/b3P1se9j+wYfMnDkLlMVcUMnt7i1BS7d9lp77bM6uy6vd14b8uh4r5F9aiNFK664+P50wB7UkYubgnO5rHe92rHu5GP76jbC9HsIBiLQll9au7XC37UjAXDrKnmhYdl7udPVdp2X3Cixn/2WtTrMFZnQstl7bjqPsT64txgmf9rr61TBtxQm/X5weOpEg3tyMjkSPX/goLO40DI9nCGtlkmAWIpXEoz3DLRJIhl9yX7jVDMZwoOtYn+Md729N9mGeuNkA23rt7GxNesDh6daazDX32d3m/o5WpLVb/+NA+iOHsU8xWlVF89//jnI4sU8oxl5cjG38eCxO57DVqbeOQIk3NhJvaCDW0EC8oZF4UyOJUAgSGhJxdDxhrhMa4nF0Ig4Jba77HEugLBYMnxeL14vh9WJ4feba1/Hai5GRgbKn7iVnHYkQa2ggVl9PvL6eWF098QZzHWuoJ15XT6w+ud3QCPH4yX1BpXBMnoTr7Lm45p5N2ty52MaPR53kz7AEsxAnIpFIXoI1B/yY6/ZefZPJlmS0rVsLM9C3ZdnRAg0HBt7KtFiTwZiRDEcPODMgo7DnPoena8SqxQBlJLeTry3dXnce69q/ces25p27vFsAe8wWZT90JEJ4715Cu3YR2ryLSNleLB4PVr8fw+/HmmXB8Huw+t0YWVlY0/1YMjJQluGf5yi8bx/1v36U5mefhVjf22Ws+fnYJ0zAXlyMfUIxtuJi8/X48VjS0k7oa2qt0eEwidZW4oEAiUCARGsrsYZGM3Qbu4VuQwOxpsZkADeZP3/9sVjAMMxgMAzz3HZ7jWFBKUu/x3Q8RqK5hXhLy9E/H7CkpWH4fFh8PQPcmpuDLT8fa14+tvw8rPn5Q9qaTLS3E62qIlpZSbSyY11JrKrKDNv6ehItLf2+VzmdWLOyMLKzsBUV4Zo9y/wZ9PtRjhP/oytWX0dw8xZaXniBpiefBMDIzibt7Dm4zp5L2tyzcUyfjmWQf8xIMIvRSWuznzKSHKQTbSe95UM4YDEv70bauvVvtneFaUd/Z7/H23qWG2A1dFyRSFjRVg/aSDMXSxra4kQbOWg1Hm3Y0R4HGgcaG1pZ0dqG1oY56MeehnKmoZwecKSh7E6UzWb+YrVaOxfztQ1lNfdjtZq/NP1+s/wgtR6IQO6ZffbHAwHCH3xAaNcHZhDv2kV4716ImpcFLWlp2CdOJFpRQXtDA/Hm5v6/gNWKkenD6s/CmuXHyPRjZPmx+rNwzpiB+9xzTqjeAxXcupW6Rx4hsOpVlNNJ5nXXkXXTjVg8HiKHDhEpO0TkUBnRskNEDh2i9dVXiTc09PwWcnPNlvWEYuzFE3BVVVH34R4Sra0k2gLEW7tCN97WZu4PBIgHAv3+EdBJKTP4/H4MfyaOiaUY8/zJ85U8V/7M5HYmRmYmFkf/t98Mhk4kzPo1NxNvajbXzU3Em5qINzeT6LG/mfCRD5N/SDT2+SyL2401Px9bnhnUtvx8rPl5PQLckpEBWhNrbCRaUUm0sqIzcKOVlea+qqq+n28Y5ucWFuCYOhW334+RnYU1K9v8WcrKwppcLG73SZ+X452z8J69BDdvJrh5E+2bNtP6yioAlN2Oc9Ys0uaebbasz56DNTPzmJ+ntD767COny9SpU/Xu3buHuxojyurVq1mxYsVwV+PkdQZoRx9lW1eYdu+77GyBBjpvS+lqiXZ/3Y4Om/t1IgFadd2RortdXtLd5t3p2DC691e60NZkX6bVCYaDeMJBImYlHrWQiBkkwtqcwyAUJx6OkwhGiQcjJNrDxNuCJNqCxANtx/7lexoZXm/nLysjO8sMw+ysHr/AjOxsrH5/Zytw9erVLJ050wzfnbuSIbyTaNmhrs/1+3FOm4Zz+jSc06bhmDYN+4QJPVrCOhollrz0Gm9oIFbfYF5ibGg01/UN5qXHxkbi9fUk2to66+y56EIyLrsM97nnDklIa61pW/s29Y88Qvt772HxevF/+tNkfub64/7CBIi3thI5dIhoR3CXlZkhfqiMeG1dV0GrFcPjwZKejsXjwXC7u7bTPVjcHcfcZjmPud1xhcHwelHGifd1n26JSIRYTQ2x6mqi1UeIHUmuq6uJHjHXsdra5MQgXZTLRSIeR0UiffbbCgt7LQWd29bc3JQ+P7HaWto3bya4aTPtmzcR2rmr8w9X+8SJuOaeTdGPfrRRaz2/93slmEeolAvmaAja63stDX33BRu7XeJtR4cDJMKaWNhCPGIhHu5YDHNf9yViIZFQoC1myGrVY/IfrTH7107zj7QlLQ1LRob5y9aTjiUjHaP7Oj0di8uFstnMxW5Prm0oW69te68yNhvKYkEnEuhoDOIxdCyGjsXRsajZPxgz9xGLoeNxdDSGjidfR6PEm5u7+tzqG4jV15l9bQ0NR7/0l5aGNSuLUHMzRrcytnHjOkPYMW0azmnTsebmnHSfWm+J9nba3n2XlhdeJPDaayTa2rB4vaRfeCEZl11qhvQgLw/qeJzWl1+m7pFHCO/chTU3F/9nP4vvk5/E8AxNiyoeaGPtq6tYeumlKIdjyM/LSKejUWJ1dUSrq4kdOWKuq6o5XF5O6TkLsRZ0Ba/h842q85cIhQht3077ps0EN20iuHkzU9e9128wy6XsEUYnEkT278e+axfBnByzf8fnxeJ2n/wPcSwMoRYIt0CoCR1sJtFYQ7zuCPGGWrO109hIItCKDgXMlmmoHR1uR8ci6IRKhmQyMBNma1VbHGiLAyx2EtpGPGQlHvQQb3cSb0vv8xd0B2WzYfgyMHw+jPF+HJl+LE6X2U9mWFFWsy9UGYZ5CdcwwGqgLEbXsW5l9uzbx5QzpoBS5rnqMYkBXfuhzwQHyqKwJFs1RoYZtmYrx2NeMh6hEpGI2YKtqydeX9cV3PXmAJqW2lpKzl9hhvCZZ2JkZJyWelnS0ki/4ALSL7iARDhM29q1tL70Eq0vv0zzU09hycjoCulFi44Z0olIhOan/079Y48SLTuEvaSEgh/+gIyPfWzQfX/HY3jcJLzelBoslkqUzYatoABbQUGP/btWr8afSg2NU8DidJI2fz5p880c1omE2e/fj5H7G2WMiNXXE9z6PsGtWwm+v5XQtu0kAgEygYO/+J+ugoaRHDXpwfCkYXhcGG47htPA4rRgODSGLY5hi2BRIRKBVuLNrcRb2om3hYi1RYgHdY8WaixigcRAwt5pLkqhDAtYrck+TrMVSEcfqGGg7HaM/EwcmZnmaM/MTKzJ/jHDl5l87cPIzES5XEP6F3NwDPzPP1gWux1Lsu+vP3tS4JxZHI6ukI5EzJB+8SVaV62i+emnzZC+4ALSL70E95IlnWEbDwRo+vOfafjt48Rqa3HOmEHuL35B+kUXpvQlUDE2HGvQowTzCdCxGOF9+4lWlJsDa7LMfjqLx3NSQZIIhwnt2GkG8JbNBLe+T7Sq2jxoUTjGZZMxJx9XoYP2aB3pShFvbSXeGiTeFiQebDFDtdZCtEIR7rj8GzvOqFdlwUjLwEh3YXjd2LzpOH3JASZZ2RhZeRi5BVhzCswBJm53z7BNDjJKhdG1YnSz2O2kn38+6eefb4b022+bIf3qqzT//e9Y0tNJv+B8jOxsmp78C4nWVtIWnUvhj+8lbdGiUXVpVIxeEszHoaNR8xaQnTsJ7dhBaMdOQrt3o0OhPmWV3d5zIE3n2m+OFMz0YjjiWK1tGIlGIgf2EPpgP8G9lQTLGgkdCULyLgVrWgxXVpTMORFcWVGcmVEs1grzlpa0LGzaiTu7CJy54PSBy9dz7fR2bmubxwzs9jDx5hYSgVZzAIovEyPTZ96bKC0IMcJY7HbSV6wgfcUKdCRC2zvv0JIM6URrK+kXX0zWbbfimjVruKsqxKBIMHejIxFCe/Z0BfDOnYR370YnRwta3G6c06aRee21OGfOwD5hgtli7biRva6WWPVhYjVVxA7tJrS1hVhrEOLHHo1ksYEz30bWuVm4SnNxTinBVlAI7mxzpqPOdRY4vGCxsH4Qg78U5j+0/GOL0UrZ7XiWL8ezfDk6EiEeCGD1+4e7WkKckDH7uzoeCBDZu5fQBx+YIbxjB6E9e7ruw0xPxzl9OpnXX49z+nScM6ZjLy5GhRqhfh807If6ldB0yFyaD0GwAjISkAFMBlDo9AISrnHELHnEVSaxuIdYxE48bGCbMAnn3AU4Jk2SFqsQQ0TZ7RLKYkQb9cEcb20lvHcvkX37CO/ZS3jfPsJ79xKrru4sY/F6cc2YTtaNN5ghXFqIzRVCNe5PBvBf4eXkdrj77SUKMorAVwwTlphrXzH4xpvrjHEoqx0DkNgVQggxEKMmmOMtLYT37iO8by+RvXvN7b17iR050llGORzYJ5WStnABjtJJOHLsONLbsVnqUY0HoOEfsPUXsK5b+CqLGbL+STBuAWRNAn+pufgmyKPKhBBCDKkRHcxt77xDw28fJ7RrF7Gams79yunEUVqK+9xzsE+ajGPyZByTJ2FzJ1Blb8H+12H/36A6OUuPMpLhWwrjz0kGbzKAfcUSvkIIIU6bERnM7Zs3U/vfv6D9vfew5uXhXrQI++RJyQCejK2oyLx1J9QMB96C/SvhmdVQv9f8AE8eTL4QSs83W8ESvkIIIVLEiArm0M6d1PziF7S98SZGVhZ53/p3fNde2zVxeywC5eth9e/MVnHFRnOuRpsbSpbA/JvNMM6dNqyPlhNCCCGOZkQEc3jfPmr/535aX3oJi9dLzle/iv/6T5sT7dftgU2vmEF8cK35IANlgaJ5sOxfu1rF0iIWQggxAqR0MEcOH6bulw/Q/OyzWJxOsr9wB/6bbuqar3fXSvjz9YCGrMkw5zoziEuWmpNrCCGEECNMSgZz9MgR6h58kKa//g1lGPhvvJGs227teW9izS54+nNQeDZ86nGzn1gIIYQY4VIqmGMNDdQ//AiNf/wjOh7H98l/Ivvzn8eWl9ezYHsD/PE6sLvhn/8AGYXDU2EhhBBiiKVGMCcS1PziFzQ+/jsSoRDej32M7Du/iH3cuL5l4zH42y3QXA43PSehLIQQYlRJiWC2VlRQ/+CvSP/IZeTceSeOSZOOXvjV78G+1+Bj90PxOaetjkIIIcTpkBLBrJ1OJj79FM5p045d8P0n4e37YcFtMPeG01M5IYQQ4jQa0AN0lVJ+pdTTSqk2pVSZUupfjlLOoZT6lVLqiFKqQSn1rFKq6HifH8/JOX4oV26Gf3wJJiyFy+4ZSLWFEEKIEWegT7Z/AIgAecCngQeVUjP6KfdlYBEwGygEGoH7T7qWgRr406fBnWOOwDZsJ/2RQgghRCo6bjArpdzANcDdWuuA1noN8A/gM/0Unwi8pLU+orUOAX8G+gvwgYtF4MkbzJHY//wH89nEQgghxCiltNbHLqDU2cBarXVat31fA5Zrra/sVXY+8Avgk0AT8GugRmv9lX4+93bgdoCcnJx5Tz75ZL9ff8qHD1JU+SI7pn+N2txlg/neRrVAIIDH4xnuaowocs4GT87Z4Mk5G7yxes7OP//8jVrr+b33D2Twlwdo6bWvGUjvp+we4DBQAcSBbcCd/X2o1vph4GGAqVOn6hUrVvQttOExqHwRlnyFGRffPYCqjh2rV6+m33MmjkrO2eDJORs8OWeDJ+esp4H0MQeAjF77MoDWfso+ADiALMANPAW8cEI1K3sHnv86TL4YLvzOCX2EEEIIMdIMJJg/BKxKqSnd9p0F7Oin7Bzgt1rrBq11GHPg10Kl1OA6hpvL4cnPgG8CXPNrsBiDersQQggxUh03mLXWbZgt3/9USrmVUkuAq4D/66f4euAGpZRXKWUDvgBUaq3rBlyjaNAcgR0NwXV/lIdRCCGEGFMGervUFwAXUAP8EbhDa71DKbVMKRXoVu5rQAizr7kWuBy4esC10Rqe/TJUbYVrHoGcqQN+qxBCCDEaDGjmL611A/Dxfva/hTk4rON1PeZ9zifmnV/C+3+G878NUz9ywh8jhBBCjFQDbTGfentfhVe+A9OvgvO+Nty1EUIIIYZFSgSzJRGFv94MOdPgqv8FpYa7SkIIIcSwSIlgdgWrzDC+7glwjL2bzIUQQogOKRHMlkQUPvlbyCwZ7qoIIYQQwyolgjnoyoPSFcNdDSGEEGLYpUQwx6xy+VoIIYSAFAlmIYQQQpgkmIUQQogUIsEshBBCpBAJZiGEECKFSDALIYQQKUSCWQghhEghEsxCCCFECkmJYG6N6OGughBCCJESUiKY60OaUDQ+3NUQQgghhl1KBDPAlsNNw10FIYQQYtilTDBvONgw3FUQQgghhl1KBLPNAusONg53NYQQQohhlxLB7DQUm8oaiSdkEJgQQoixLTWC2QqBcIxdVS3DXRUhhBBiWKVEMDsMBcB66WcWQggxxqVEMFstUORzSTALIYQY81IimAEWlGSy/mAjWks/sxBCiLErZYJ5fomf2tYwZfXtw10VIYQQYtikTDAvnOgHYJ1czhZCCDGGpUwwT87x4EuzyUQjQgghxrSUCWaLRTF/gtnPLIQQQoxVKRPMAAtK/Byoa6OmNTTcVRFCCCGGRWoFc7KfeaO0moUQQoxRKRXMMwu9OG0WGQAmhBBizEqpYLZbLcwZ75OJRoQQQoxZKRXMAAtL/OysbCEQjg13VYQQQojTLuWCecFEPwkNm8qkn1kIIcTYk3LBfHZxJhYlD7QQQggxNqVcMHscVmYUeiWYhRBCjEkpF8xg3s+8+VATkVhiuKsihBBCnFYpGsyZhGMJtlU0D3dVhBBCiNMqJYN5fok50YjMmy2EEGKsSclgzkl3UJrtln5mIYQQY05KBjPA/BLzgRaJhB7uqgghhBCnTcoG84ISP83BKHtrA8NdFSGEEOK0SdlgXph8oMW6A3I5WwghxNiRssFc7E8jJ90h/cxCCCHGlJQNZqUUC0v8bJBHQAohhBhDBhTMSim/UupppVSbUqpMKfUvxyg7Vyn1plIqoJQ6opT68olWbkFJJhVNQSqagif6EUIIIcSIMtAW8wNABMgDPg08qJSa0buQUiobeBF4CMgCJgMvn2jlOu5nXi/9zEIIIcaI4wazUsoNXAPcrbUOaK3XAP8APtNP8a8CL2mt/6C1DmutW7XWu060ctMKMkh3WKWfWQghxJgxkBbzGUBMa/1ht31bgT4tZuBcoEEp9bZSqkYp9axSqvhEK2dYFHMnZEowCyGEGDOsAyjjAVp67WsG0vspOw6YC1wMbAN+AvwRWNK7oFLqduB2gJycHFavXt3vF8/SEd44EmXly6/jsasBVHdsCAQCRz1non9yzgZPztngyTkbPDlnPQ0kmANARq99GUBrP2WDwNNa6/UASqnvA3VKKa/WuscTKbTWDwMPA0ydOlWvWLGi3y/uKq7nqT3v4hw3nRXT8wZQ3bFh9erVHO2cif7JORs8OWeDJ+ds8OSc9TSQS9kfAlal1JRu+84CdvRT9n2g+xyaJz2f5lnjfdgNi1zOFkIIMSYcN5i11m3AU8B/KqXcSqklwFXA//VT/DfA1UqpOUopG3A3sKZ3a3kwnDaDWeO8rJNgFkIIMQYM9HapLwAuoAazz/gOrfUOpdQypVTnZNZa69eAbwHPJctOBo56z/NALSjxs628mWAkfrIfJYQQQqS0AQWz1rpBa/1xrbVba12stX4iuf8trbWnV9kHtdZFWutMrfWVWuvDJ1vJhRMziSU0Ww43nexHCSGEECktZafk7G5esR+lkH5mIYQQo96ICGZvmo2peekSzEIIIUa9ERHMYPYzbyprJBZPDHdVhBBCiFNmxATz/JJM2iJxdlX1d/u0EEIIMTqMmGBeONF8oIXcNiWEEGI0GzHBXOB1MS7TxQYJZiGEEKPYiAlmMPuZ1x9sQOuTnlBMCCGESEkjLpjrAhEO1LUNd1WEEEKIU2JEBfPCiZkAbDjYOMw1EUIIIU6NERXMk3I8ZKbZZACYEEKIUWtEBbNSivnJfmYhhBBiNEqJYI7q6IDLLizxU1bfTk1L6BTWSAghhBgeKRHMNdEaattrB1R2fonZz7xe+pmFEEKMQikRzAkS3P323QO6DWpmkReXzZDL2UIIIUallAjmTGsmayvW8qfdfzpuWZth4exinwSzEEKIUSklgtlj8bCkaAk/3fBT9jfvP275+SV+dlW10BoaeN+0EEIIMRKkRDAD/GDxD3BZXfz7W/9ONH7swF1Y4iehYWOZ9DMLIYQYXVImmHPScvjuou+ys34nD2598Jhlzy72YViUTDQihBBi1EmZYAa4aMJFXDXpKh7d/ihbarYctZzbYWVGYYZMNCKEEGLUSalgBvjmwm9S4C7g39/6d9qiR58Te0GJny2HmwjH4qexdkIIIcSplXLB7LF7+NHSH1HZVsmP1/34qOUWlPiJxBJsr2g+jbUTQgghTq2UC2aAuXlzuXnmzTy992lePfRqv2U6JhpZd0D6mYUQQoweKRnMAF846wtM80/j+29/n7pgXZ/j2R4HpTluuZ9ZCCHEqJKywWwzbNy77F7aY+3cvbb/WcEWlvjZcLCBROL4M4YJIYQQI0HKBjNAqa+Uu+bdxZqKNTy5+8k+x+eX+GkJxfigunUYaieEEEIMvZQOZoDrzryOxYWL+a8N/8WB5gM9ji2dnI3DauHzv9/IbglnIYQQo0DKB7NFWfjBkh/gsDrMWcESXbOC5Xud/PH2cwlF41z9v2t5cXvVMNZUCCGEOHkpH8wAuWm5fOfc77CjfgcPbX2ox7G5xZk8+6WlnJGXzud/v4mfv/Kh9DkLIYQYsUZEMANcUnIJH5v0MR7Z9kifWcHyMpz86fZzuWbuOH7x6h4+//uNBMKxYaqpEEIIceJGTDCDOStYflo+31rzLdqj7T2OOW0G//XJ2Xznium8+kENn/jftRysO/rMYUIIIUQqGlHBnG5P50fLfkR5azk/Wf+TPseVUty8dCK/u3khNa1hPvbLNbz5Ye0w1FQIIYQ4MSMqmAHm5c3jszM/y9/2/I3XD73eb5klk7P5xxeXUuhzcdNv1vHrt/b3ex+0EEIIkWpGXDAD3DnnTs70n8n33vlev7OCARRnpfG3OxZz6Yx8fvjcLv71ya2EovLACyGEEKltRAZzx6xgbdE2vvv2d4/aGnY7rDzwL3P56sVn8NTmCj710DtUNQdPc22FEEKIgRuRwQwwyTeJu+bdxZvlb3Lna3f2mXykg8Wi+H8XTuHhz8xjX02AK+9fy8YymV9bCCFEahqxwQzmrGD/Ou9f2XhkI5945hP8eN2PaQ73/xjIS2bk8/QXl+B2GPzzw+/yp3WHTnNthRBCiOMb0cFsURZumnkTz139HFdPuZonPniCy5+6nD/s+kOPGcI6nJGXzjNfXMK5pVl886ltfOeZ7UTjiWGouRBCCNG/ER3MHbJcWXxn0Xf4y5V/YXrWdO5ddy+feOYTvFn+Zp/+Z1+and/ctIDbzyvld5GB4UgAACAASURBVO+UceX9a3hqUzmRmAS0EEKI4TcqgrnDGZln8PDFD/PLC34JwBdf/SKfe+Vz7Gnc06Oc1bDwrcun8b+fnks8ofnqk1tZ+uPXeOD1vTS1R4aj6kIIIQQwyoIZzElGlo9fzlNXPcU3FnyDHfU7+Kdn/4kfvPMDGkI9B31dPquAl+86j8dvXsjU/HTue2k3i+55je88s50DMmuYEEKIYWAd7gqcKjaLjeunX8+Vk67kwa0P8ucP/szzB57n9tm38+lpn8Zu2IFkkJ+Rw/IzcviguoVH3zrAn9Yd5v/eLeOiaXncunQiCyf6UUoN83ckhBBiLBh1LebevA4v31z4Tf521d+YmzeXn238GVf9/SpWla3q0/98Zn4G933yLNZ883y+dP5kNhxs4NqH3+Vjv1zLM1sqZKCYEEKIU27UB3OHUm8pD1z4AA9d/BBOq5O7Vt/FzS/dzPa67X3K5qY7+eolU3n7mxfyo6tn0RaJ8eU/beG8n7zOQ2/soznYd8S3EEIIMRTGTDB3WFy4mL9c+RfuPvdu9jXt47rnruPzqz7P5prNfcq67Ab/ck4xq+5azmM3zWditpt7XviARfe8yvf+sYOyeumHFkIIMbQGFMxKKb9S6mmlVJtSqkwp9S/HKW9XSu1SSpUPTTWHltVi5VNTP8Xzn3ieL8/9MjvrdnLDCzdw80s3817Ve30ucVssigvOzOOJ287luf+3lMtm5vOH98pYft9qPvmrt/m/dw5SHwgPzzcjhBBiVBloi/kBIALkAZ8GHlRKzThG+a8DKf+8RY/dw62zbuXFa17k3xb8GwebD3Lry7fymRc+0+890AAzCr387FNzWPONC/jaJWfQHIxy9zM7WPijV/nMo+/xlw2HaQnJpW4hhBAn5rjBrJRyA9cAd2utA1rrNcA/gM8cpfxE4HrgnqGs6KmUZkvjM9M/wwvXvMDd595NbXstX3z1i1y78lpWla0iofsO+srLcHLnBVN4+a7lvPiVZXx+eSkH69v4+l/fZ/4PVnH77zaw8v1KghF5opUQQoiBG8jtUmcAMa31h932bQWWH6X8/cC3gBH3GCeH4eBTUz/F1VOuZuW+lfx626+5a/VdTPJO4rbZt3FZyWUYFqPP+87Mz+DM/Ay+dslUthxu4tmtVax8v5KXdx4hzW5w8fQ8PnZWIcum5GA14EjbEQ61HmKybzJZrqxh+E6FEEKkKnW0RyZ2FlBqGfAXrXV+t323AZ/WWq/oVfZq4Hat9UeUUiuA32utxx3lc28HbgfIycmZ9+STT57M93FKJHSCTe2beLn5ZaqiVeRYc7jEewkL3AswVN+A7i4UD7O+vob1DZXsa68mbq3F6qjFcNSSUOalbitWzvGcw/kZ55NnyxtU3QKBAB6P54S/t7FIztngyTkbPDlngzdWz9n555+/UWs9v/f+gQTz2cBarXVat33/CqzQWl/ZbZ8b2AJcrrXec7xg7m7q1Kl69+7dA/5mTreETvD6odd56P2H2NWwi0J3ITfPvJmPT/k4gUiAA80HONBywFwnl8pAJRrz3CoUfkceRHOpa/ASDmaTbs2loHAPlbG3iOkoK8av4KYZNzE3d+6AJjNZvXo1K1asOMXf+egi52zw5JwNnpyzwRur50wp1W8wD+RS9oeAVSk1RWvdMen0WcCOXuWmACXAW8lgsQNepVQ1cK7W+uAJ1n3YWZSFCydcyAXFF7CmYg0Pvf8QP3zvh9y77l5iOtZZzmk4meidyOyc2Vw1+SomeicyMWMiEzIm4LQ6AQhG4rz2QQ0r369kze4pBGKLsfvf4U39LqsPr2Zi+jQ+P+dmLim5CKtl1E7MJoQQ4iiO+5tfa92mlHoK+E+l1K3AHOAqYHGvotuB8d1eLwZ+CcxlBIzQHgilFMvGLWNp0VLWV6/njfI3KPQUMjFjIhO9E8lz52FRxx5P57IbfHR2AR+dXUAsnmBreRNv7TmbN/Z8nJ0tr7Ev8ibfeOvrfOetbJbkfILbz76W6fk5MiWoEEKMEQNtkn0BeAyoAeqBO7TWO5L9zy9orT1a6xhQ3fEGpVQDkNBaV/f7iSOYUoqFBQtZWLDwpD7HaliYN8HPvAl+vnLRGTQHl7B2bw1/3fUSm5uf4bXah3n1hf/DEVzCstyPc/HUM1gyOQtfmn2IvhMhhBCpZkDBrLVuAD7ez/63gH577LXWq4Hj9i+LLl6XjctnFXH5rJuBm3lx77s8svU3fGi8yqttr/Pi63OI/u08ZuacwXh7hET+EeYV+/Gm2Ya76kIIIYaIdGKmsMsmn8tlk8/lcMthHt/xOE/v/TsR30aqYzPYdWQOz/+xjETUx5ScHOZPyGL+hEzml2RS7E+TS99CCDFCSTCPAOMzxvPtRd/mzrPv5M+7/8wTHzyBw7oDR/J4jXayst7L36t8JN7y4VLZlGYWcVbBRBZPmMySklLS7HL5WwghRgIJ5hHE5/TxubM+x00zb+KPq/5I4dRCqtqqqGqrojJQxcHmCqrbdhGMt7BHw55K+Gsl6Lct2PGRac+lOKOIaTnFFGXk4Xf6O5csVxZeu7ffCVSGQjwRJxANAOajOIUQQvRPgnkEchgOShwlrChZ0e/x9mg71W3V7KwtY335fnbWlFHeWklVsIbqtk2sr3sdpfpOM2pRFnwOX2dQ+51+spxZPV77nX4sykJLpIXWSCuBSIDWSCstkRYCUXO7c4l2bbdFu57Edab/TJaPW87yccuZkT3juCPZhRBiLJFgHoXSbGmU+kop9ZVyxZTzO/e3R2JsOdzExoP1bCivYltVOc2RBpTRhtXeRo43QoYtgjXeTnMwQGWgkoZQQ49QPRqLspBuT8dj85BhzyDdnk5xejEem4d0ezoZ9gw8dg+hWIg1FWt4ZNsjPPT+Q2Q5szhv3HksH7ecRYWLSLOlHfdrCXE8WmsZZyFGLAnmMSTNbmXxpGwWT8oGpqK1pqo5xNbDTWwpb2Lr4Sa27WqmLfngjXSHldnjvcwocjExFwqzYmCYl6PT7ek9ljTrwAec3Tb7NppCTaypXMObh99kVdkqnt77NDaLjQX5C8zW9PjlFHmKTtWpEKPUh40f8t8b/5sPGj7ggQsfYFrWtOGukhCDJsE8himlKPS5KPS5+MisAgDiCc3+2gBbDjextbyJrYebeeytCmIJc3rRvAwHs4p8zCpKZ2ZRBrOKvLhtzkF/bZ/TxxWlV3BF6RVEE1G21GzhjcNv8Eb5G9yz7h7uWXcPk32TO0N6dvbsU9b/LUa+6rZq7t98P8/uexaP3YPLcHHLy7fwyMWPMCP7WE+oFSL1SDCLHgyLYkpeOlPy0vnkfHMit1A0zs6qFrYeTraqK5p59YMjdEyznpPuYFaRl5mFGcwo8jKryEuB1zngFnRHS3lB/gK+tuBrlLWU8cbhN3iz/E0e3/E4j25/FJ/Dx9KipczOmc1k32RKvaX4nX65XDnGNYebeXT7o/xh5x8AuHHGjdw661YC0QC3vHQLt718G7+6+FfMzpk9zDUVYuAkmMVxOW0Gc4szmVuc2bkvEI6xq6qF7RXNbKtoZkdFC6t315BsWON325mZDOuZybAel+kaUJBOyJjADTNu4IYZN9AaaeXtyrd54/AbrKlYw8r9KzvL+Rw+Sr2lTPJNYpJvUud2jkumMB3twvEwf/rgTzz8/sO0Rlq5ctKV3DnnTgo85pUfr8PLby79Dbe8fAu3v3I7v7roV8zJnTPMtRZiYCSYxQnxOKwsKPGzoMTfuS8YibOruoUdybDeXtHCw2/u77wM7nXZmF6QwZQ8D5Nzu5Ycj+OoQZpuT+fSkku5tORStNbUtNewr3kf+5v2d65fOvgSLZGWrvfY0in1lfYI60neScR1nEAkQCgeIhKPEIqHCMfChOPhzu1QPEQ4bu7r/jqeiDPRO5EZWTOYkDFBLqsPk4RO8Nz+57h/8/1UtVWxpGgJd829i6n+qX3KFngKeOzSx7j15Vv53Cuf48GLHmRu3txhqLUQgyPBLIaMy963ZR2KxvnwSCvbK1rYVtHMzqoWntpUQSDc9VSuDKe1M6Sn5KZ3bhf5XFgsXYGtlCLPnUeeO4/FhV3PUNFaUx+q7wzrfU372N+8n9WHV/PUnqd6VvLQiX1vFmUhoc1bzFxWF9P805iWNY3pWdOZ7p9OibdEngZ2ir1d8TY/2/gzdjfuZnrWdH6w5AecU3DOMd+T787nsUsf45aXbuHzqz7PAxc+wIL8BaepxkKcGPlNIk4pp81g9jgfs8f5OvdprTnSEmZvTYA9Na3srQmwtybAax/U8OSG8m7vtVCa3RHY5rok202xPw23o+tHVylFtiubbFd2nweLNIYa2d+8n31N+9i0axNnTj4Th9WB03DiMBw4rA5zbST3dTvmtJpru2EnoRMcaD7AzvqdnctTe57iD7vMvk2n4WSqfyrTs6YzzW8G9iTfJAnrIbCzfic/3/hz3q16lyJPET9e9mMum3jZgO9/z03L5TeX/YZbX7qVL6z6AvdfeD/nFpx7imstxLFFE9GjHpPfGuK0U0qR73WS73WydEp2j2ONbRH21gY6w3pvTYCNZY38Y2tlj3LZHjvF/jRzyTLDekKW+To3vevSeKYzk3nOeczLm0duVS4rZq44oTpblIUpmVOYkjmFqyZfBZizmR1sOdgjrJ/Z+wx/jP0RMCeCOSPzDKZnTWeidyIF7oLOxevwplQ/eCwRoyncRGOokaZwEw2hBhpDjRwOHGZuZC4Z9ozTXqfy1nLu33w/zx94Hp/DxzcWfINPTf0UdmPw08tmu7J59NJHufXlW7nz1Tv5n/P/h8VFvZ9cK8Spp7XmpbKX+NmGnx21jASzSCmZbjsL3D37rgHawjH217ZR1tDGoYZ2DtW3U1bfzvqDZmh3DDoDs6XdGdp+N8V+FxOy3FQHEoSicZy2oekfNixG58CzKyddCZh9oB1hvat+Fzvrd7Jy/8o+k7S4rC7y3fkUuAvId+d3bncsee48HIajvy/br4ROdPaLd/SRd/SbN0eaaQw10hBq6Azf3tvd++h7e+LPT3BOwTlcXHwx5xefj9/pP2rZk9UaaWVd9TreKn+LZ/Y9g1VZuXXWrdw882bS7ekn9dlZriweu/Qxbnv5Nr702pf47/P/m2Xjlg1RzYU4vt0Nu7l33b1sOLKBqZl9x0V0UFrrox48XaZOnap379493NUYUVavXs2KFSuGuxopIRJLUNEUpKy+jcMNZmAfauha2pMTpgAoBfkZTsZ3Bre5dLzO9tiHvCWrtaYh1EB1W3Xn3OZVbVXm64C5XR+q7/O+LGcW+e58/E4/kUSkR+j2HqB2rMti3VmVFZ/TR6YzE7/Db247MvE7/Z37Mx2Z5nGnn5VvrqQhq4FXyl6hPFCORVmYnzefiyZcxIXFF5KblntS5yaWiLGjfgdvV77NO5Xv8H7t+8R1HJfVxeUTL+eOs+4gz513Ul+jt6ZQE7e/cjt7m/by8xU/Z/n45UP6+fL/5uCN9nPWGGrkl5t/yV/3/JUMewZfOvtLXDPlGqyGdaPWen7v8hLMI9Ro/0EeKlpr6gIRDjW089LajbhzJ3CooZ3DydCubgn1KJ9mN3oEdffgHu934bCemtHY4XiYI21HeoR3x3ZjqBG7Ye/sA+/oE+/eD36s1xn2DDNwnZmk29IH9YdHx8+Z1prdjbt5pewVXil7hQPNB1Ao5uTO4aLii7howkUUegoH9JnlreW8U/UO71S+w7tV79IaaUWhmJ41ncWFi1lUuIg5OXOwGafuOePN4WY+98rn2N24m58u/ykXFF8wJJ9b3VbNc289x4J5C7BarBjKwGaxYbVYO193bNssNgyLgVVZx9wo/2AsyM76nbxf+z7b6rbRVNfEZ8/9LIsKF42qcRmxRIw/7/4zD2x5gPZoO/985j9zx1l3dD7IRyklwTyaSDAPXn/nLBSNU94YTLa02zjUEOwR3MFoz9Z2odfFhKy05OJmgj+5zuo5IG20ONrP2b6mfbxS9gqrylaxu9H8f3dG1gwumnARF0+4mAkZEzrLBiIB1lWv62wVH2o1h8bnpeWxpGgJiwoWcU7BOWQ6M/t8nVOpJdLCHa/cwc76ndy3/D4umnDRoD8jEo+wqWYTayvWsqZiDXub9p5QXRQKq8WK0+pketZ05uTM4ezcs5mdM/ukL+EPt4ROUNZS1hnC79e+z4eNHxLX5v9bRZ4iGtsbaU+043f6+cjEj3BF6RXMyJqRUuMwBuvdqnf58bofs7dpL+cWnMs3FnyDyZmTe5SRYB5lJJgHb7DnrHtr+1BDG2XJfu2yenO7vi3So3y2x9EV2n43Jdld4e1Ls43IXzIDOWeHWg6x6tAqXjn4CtvrtwMwJXMKC/MXsqt+F1trt3Zenl6Qv6CzVTwxY+Kwn5NAJMAdq+5gW9027j3vXi4ruey47zncepi1FWtZW7GW96rfIxgLYrPYmJs3l2VFy2g/1M6MWTOIJWLEEjHiOt65HU1Ee7yOJWLEdNd2a6SV7XXb2d24m4ROoFBMzpzcGdRzcuYwLn3csJ+3Y2kON3cGcEcYd4xhcNvczMyeyezs2ZyVcxYzs2eS5cpi1eursEyysHL/St44/AaRRISSjBI+WvpRPlr6Ucanjx/m72rgylvL+a8N/8Wrh16lyFPE1xd8nQvGX9Dvv5kE8ygjwTx4Q33OWkPRrrBuaKOsLrmub6equeclcrfdoMDnosDrpNDrosDXtS7wuij0OUmzp16Le7DnrCpQxapDq1hVtoqttVuZ6p/K4sLFLC5czFk5Z53QiOpTrS3axhdWfYEttVv40dIf8dHSj/Y4HoqFWF+9nrWVZhgfbDkImC29pUVLWVa0jAX5CzqfjDYUP2dt0Ta21W1jc81mttZsZWvt1s7nmWc5s5iTO4c5OXOYkzuH6VnTT/t5jSVi1AfrqWmvoSZYYz5mNnlpuuP8dPxRMTt7NrNzZjM7ezYTvRP7vWzf/Zy1RFpYVbaKlftXsr56PQBn557NFaVXcMmES/A5fX3enwrao+38etuveXzH4xgWg9tm3cYNM2445iBOCeZRRoJ58E7nOQtF4xxuaOdgsoVd3hikqjlIVXOIyqYQdYFwn/dkOK0UJsO7wOei0NsR2i6KfC7yvU7s1tP77OqTOWcJnRgxz9puj7Zz52t3svHIRn645IfMzJ7Jmoo1rK1Yy4YjGwjHwzgMBwvyF7C0aClLi5ZSnF7cbyvoVPycxRNx9jXvY0vNFnOp3cLh1sMA2C12ZmTPYE7OHAo9hTitTpxWJy7D1XlfvsvqMvcbyWNWFzZL36s4Wmsaw43UttdS015DbbCWI+1HqG2vpbY9uR2spT5Yj6ZndvidfmbnmC3h2dmzmZE9A7fNPaDv72jnrCpQxXMHnmPlvpXsa96H1WJlWdEyrii9guXjlw/qzoVTRWvNcwee4+cbf05New1XlF7BV+Z+ZUCDFo8WzKn3J7oQo4DTZnQ+DKQ/4VicmpYwlU3JsG4OUtUUoqo5SGVTiC2Hm2hs7znSWinIS3dSlGkGdff1uOQ6lVrdIyWUwXyG+QMXPsCXXvsS31rzrc79E70T+eQZn2Rp0VLm5c3DaR38k9SGgmExOCPzDM7IPINPTf0UAHXBOrbWbGVL7RY212zm97t+P+DR+WD++3QEtdNwotHUBev6/YxMRya5abnkpOUwLWuaue3KITctt3PJcmYN+SX2Ak8Bt866lVtm3sLuxt2s3LeS5w88z+uHX8dj83BJySVcWHwh6fZ0FD1nCQSz1a5QPV53FOs4ZlGWHgP0ug/S676/v5/nHfU7uPe9e9lSu4XpWdP56fKfDsmc7Knzf7EQY4jDaiRHeqcdtUwwEu8M6sqmIOVNQSoag1Q0tbP5cCPPb6vqnIe8Q2aarSuwfWnJbbPlXeBzku129JjmVHRxWV388oJf8rudv8Pn8LG4cDHj0scNd7WOKtuVzYUTLuTCCRcCEI1HaYm0EIqHCMXMJRgLdr7uvt37dTgeRmtNdlo2eWl5PUI325U97F0QSinO9J/Jmf4zuWveXayrXsfK/St58cCLfafdPVV1SA7Q6z6qvjHUSKYzk/9c/J9cNfmqIftjVIJZiBTlshuU5ngozfH0ezye0NS0hpJhHaQ8ua5oDLKvto03P6zrMaocwGaYs64VeJOXyrtdMu/o9x6pA9WGgtPq5PbZtw93NU6IzbCR5coa7mqccobFYFHhIhYVLuLb536bbbXbiCXMufd18j8wLzF3v9ze/XXHtkaT0ImjDsY73v7stGyun3b9kI+cl2AWYoQyLMoMVK+LPp1UmL98mtqjVCQvl3e0vquSl803lDVyZFsV0XjPVrfLZiT7uZ2oYJhNkd3kdw5UM0M8w2kds+EtUofL6uozP/5oIMEsxCillCLTbScz+Wzs/iQSmrpAmMrmEFVNwc51R7/3wbo4b1fupdcVc9LsRmdIm2unhLcQQ0SCWYgxzGJR5GY4yc1wMmd839tQVq9ezdJl51HTGu5sdVcnR5ZXt5gt8Lf21FHTGuo3vPMznORlmA8syctwkp/h6Nr2OsnxOLAaI2eQmBCngwSzEOKYrIaFQp952xb0PztXNJ6gtjXceUuYOcI8xJGWENUtIdYdaKCmNdTnsrlFmROzdAV3V4hne+z43XYy08xWv9tuSAtcjAkSzEKIk2brEd79SyQ0De0RqrsF9pFmc13dEuZQfTvrDjTQHOz/lh+7YcGXZusW1jZznQxuv9uGL82OP80MdL/bTpqEuRiBJJiFEKeFxaLI9jjI9jiO2ucN5m1iR1pC1LdFaGyL0NhuLg1t0R6vd1e30tQepbE90ucyegeH1UKW247fY8fvdpjbyaVjOyt5zO+2S7+4SAkSzEKIlOKyG5RkuynJHtisUYmEpiUUpbE9SkMyzBvaIzS0mUt9IEJDW5iGtgj7awM0tEV6PAq0O5uhyEyzk5PuIDfdQW66k9wMczsn3dm1P8Nxyp40JoQEsxBiRLNYFL40O740OxMHGOahaJz6tggNgQj1ydBuaIt07qsNhKlpDbGjsoW6QLjfFrnXZesM6dx0ZzK8HdRWxlAf1uJPs3deepdL6mIwJJiFEGOO02YkZ0c7ep94h3hCU98WpqYlTG2rudS0hqhpNffVtIZYf7CBmtYwkVgCgIfeX9fjM+yGpUefuN/dFdq+NLN/vOOY12Ujw2Uj3WnFJiPWxyQJZiGEOAbDopIt4mPPk621piUY4/nX3uKMWXPMPvH2rkvrTW1Rc90e4YPqFhrbozQdo38czFvOvC4bGU4bGS5rt20bGU6ruU7u87ps5KQ7UvZJZWLgUvZfLxqNUl5eTigUOn7hMcjr9bJr1y6cTifjxo3DZrMNd5WEGNOUUnjTbBR4LMyb4B/Qe/rrH28JRWkJRmkJxWgOdmxHaQ5GqWwK8UGolZZglNZwjKM9HNCXZqPA2zVPujli3tk5cj4vXe4fT2UpG8zl5eWkp6dTUlIifTP9aG1txePxUF9fT3l5ORMnThzuKgkhBulE+sc7xBOaQDhGSzDaGeA1reHkFKzm5C/ljUHWHWigJRTr+XUV5GWYs7R1hHVuusO8hO6wku40L6V7nFbSnVbSHTacNov8Lj5NUjaYQ6GQhPJxKKXIysqitrZ2uKsihDjNDIvC6zIvYY8/TtlAONY55WplUzC5mNvbK5p5eeeRzv7xo7FaVFdYO8zgNhdz2+cy7yPPTN5PbvaZm9tyG9rgpGwwA/IPOQByjoQQx+NxWI/5fHCtNc3BKK2hGK2hGIFwjNZQ8nVyO5A81hqKmi31UIzKphCt4VZak5fdj3Zp3bCoZHB3DXjLTDMHvPnS7NSUR4nuPNJ1f7nHTrpj7IZ5SgfzcPN4PAQCgeGuhhBCnFJKdV1SP1HxhKYlmBzwlhzY1rXu2m5oi3C4oZ33y819HS3132zf0OPzbIZKTgbTz8Qwno4JYhz43TY8Dhtuh4Hbbh0VzxuXYBZCCHHSDEvX08wGSmtNMBrn+Vff5IxZczvvI++8pzx5j3l9W4TDje00BCK0hmPH/Ey33cDjtOJ2WPEkl97b6U4rbruB22GOdM90d7XgvS7bsA+Mk2AeAK01//Zv/8YLL7yAUopvf/vbXHvttVRVVXHttdfS0tJCLBbjwQcfZPHixdxyyy1s2LABpRQ333wzd91113B/C0IIkXKUUqTZrWS7LMwe1/fpZv0Jx+I0tkV7TAzTFo7TFjYvu7cll47tQChGQ1s7bRFzuy0cJxI/dn96htNKZp9L7rYe/eYd96L70uz4XLYhnURmRATz95/dwc7KliH9zOmFGXz3yhkDKvvUU0+xZcsWtm7dSl1dHQsWLOC8887jiSee4NJLL+U//uM/iMfjtLe3s2XLFioqKti+fTsATU1NQ1pvIYQYyxxWg3yvQb732PeVH0s4FqctHCeQ7BvvmH+9Y+51c052c7s+EGFvTYCmdrNv/WhsRtdgvI6w9qbZ8Lk6ArznMV/a0W9xHRHBPNzWrFnDddddh2EY5OXlsXz5ctavX8+CBQu4+eabiUajfPzjH2fOnDmUlpayf/9+vvSlL/HRj36USy65ZLirL4QQohuH1cBhNfAP4rI7QCSWoCmYDPDkA1Wag1Ga2qM0JW9ba26P0hSMUN0S4oPqVpqDxw70/oyIYB5oy/Z0O++883jzzTd57rnnuOmmm/jqV7/KDTfcwNatW3nppZf41a9+xZNPPsljjz023FUVQghxkuxWy4BmgestGk90BnhzMtib2qP804/7Lz8ignm4LVu2jIceeogbb7yRhoYG3nzzTe677z7KysoYN24ct912G+FwmE2bNnH55Zdjt9u55pprmDp1Ktdff/1wV18IIcQwshmWzkeeDoQE8wBcffXVvPPOO5x11lkopfjJT35Cfn4+jz/+OPfddx82mw2Px8Pvfvc7Kioq+OxnP0siT1R6+wAADgdJREFUYQ4uuOeee4a59kIIIUaSAQWzUsoPPApcAtQB/661fqKfcl8HbgQmJMv9r9b6vqGr7unVcQ+zUor77ruP++7r+a3ceOON3HjjjX3et2nTptNSPyGEEKPPQFvMDwARIA+YAzynlNqqtd7Rq5wCbgDeByYBLyulDmut/zRUFRZCCCFGs+PeRa2UcgPXAHdrrQP6/7d3/7FRlXsex99faNMBKz9qkQq4Um+AkjJUBHdRU0G4gBi1BIPNgoRlczGCir9i6CJsCKJ778ZrogmB26BACXuxgl25iz+yhFaWK7pUgxawt8mCYHX5UWC7rUtB6rN/zIDtMG1ncMo5035eyUnkOefMfM83j/3mPOfM8zi3B9gOzI081jn3z865L5xzF51zfwHeA+5OdNAiIiJdVSx3zMOBi865mhZtXwIT2jvJQr+0zgf+0Mb+x4DHAAYMGEBFRUWr/X379qWhoSGG8Lqn5ubmy/lpamq6In9ypcbGRuUpTspZ/JSz+ClnrcVSmNOByNk96oHos6H/bAWhO/L10XY654qBYoARI0a4iRMnttr/9ddfc/31HX1F99XQ0HA5P4FAgDFjxngckf9VVFQQ2c+kfcpZ/JSz+ClnrcVSmBuBPhFtfYA2b2fN7ElCz5rznXPnrz48ERGR7iWWmbprgBQzG9aiLQ+IfPELADP7e6AImOycq/3lIYqIiHQfHRZm59wPwLvASjO7zszuBgqATZHHmtkc4BVginPucKKDFRER6epiXdtqEdALOAn8EVjonDtoZvlm1nLB4lXADcA+M2sMb2sTG7KIiEjXFVNhds6dcc7NcM5d55z7q0uTizjn/sM5l97iuGznXKpzLr3F9nhnBX8tzJgxg7Fjx5Kbm0txcTEAH374Ibfffjt5eXlMnjwZCL1VOH/+fILBIKNHj2bbtm1ehi0iIkkqOabk/KAIjlcl9jOzgjD9tx0e9tZbb5GRkcG5c+e44447KCgoYMGCBezevZvs7GzOnDkDwEsvvUTfvn2pqgrFefbs2cTGKyIi3UJyFGYPvfHGG5SVlQHw7bffUlxczD333EN2djYAGRkZAOzcuZMtW36e4Kx///7XPlgREUl6yVGYY7iz7QwVFRXs3LmTvXv30rt3byZOnMhtt91GdXW1J/GIiEjXF+vLX91SfX09/fv3p3fv3lRXV/Ppp5/S1NTE7t27OXLkCMDloewpU6awevXqy+dqKFtERK6GCnM77rvvPi5evMjIkSMpKipi/PjxDBgwgOLiYmbOnEleXh6FhYUALFu2jLNnzzJq1Cjy8vIoLy/3OHoREUlGyTGU7ZG0tDQ++OCDqPumT5/e6t/p6els3LjxWoQlIiJdmO6YRUREfESFWURExEdUmEVERHxEhVlERMRHVJhFRER8RIVZRETER1SYRUREfESFOUHS09Pb3PfNN98watSoaxiNiIgkKxVmERERH0mKmb9+95+/o/pMYheOyMnIYclfL2lzf1FRETfffDNPPPEEACtWrCAlJYXy8nLOnj3Ljz/+yKpVqygoKIjre5uamli4cCGVlZWkpKTw2muvce+993Lw4EHmz5/PhQsX+Omnn9i2bRuDBg3ikUceoba2lubmZpYvX355ClAREemakqIwe6GwsJBnnnnmcmEuLS3lo48+YvHixfTp04e6ujrGjx/PQw89hJnF/LmrV6/GzKiqqqK6upqpU6dSU1PD2rVrefrpp5kzZw4XLlygubmZ999/n0GDBrFjxw4gtKiGiIh0bUlRmNu7s+0sY8aM4eTJk3z//fecOnWK/v37k5WVxbPPPsvu3bvp0aMH3333HSdOnCArKyvmz92zZw9PPfUUADk5Odxyyy3U1NRw55138vLLL1NbW8vMmTMZNmwYwWCQ559/niVLlvDAAw+Qn5/fWZcrIiI+oWfM7Zg1axZbt27l7bffprCwkM2bN3Pq1Ck+//xz9u/fz8CBA2lqakrId82ePZvt27fTq1cv7r//fnbt2sXw4cP54osvCAaDLFu2jJUrVybku0RExL+S4o7ZK4WFhSxYsIC6ujo+/vhjSktLufHGG0lNTaW8vJyjR4/G/Zn5+fls3ryZSZMmUVNTw7FjxxgxYgSHDx/m1ltvZfHixRw7doyvvvqKnJwcMjIyePTRR+nXrx/r1q3rhKsUERE/UWFuR25uLg0NDQwePJibbrqJOXPm8OCDDxIMBhk3bhw5OTlxf+aiRYtYuHAhwWCQlJQUNmzYQFpaGqWlpWzatInU1FSysrJYunQp+/bt44UXXqBHjx6kpqayZs2aTrhKERHxExXmDlRVVV3+78zMTPbu3Rv1uMbGxjY/Y+jQoRw4cACAQCDA+vXrrzimqKiIoqKiVm3Tpk1j2rRpVxO2iIgkKT1jFhER8RHdMSdQVVUVc+fObdWWlpbGZ5995lFEIiKSbFSYEygYDLJ//36vwxARkSSmoWwREREfUWEWERHxERVmERERH1FhFhER8REV5gRpbz1mERGRWKkwi4iI+EhS/Fzq+CuvcP7rxK7HnDYyh6ylS9vcn8j1mBsbGykoKIh6XklJCa+++ipmxujRo9m0aRMnTpzg8ccf5/DhwwCsWbOGu+66KwFXLSIifpcUhdkLiVyPORAIUFZWdsV5hw4dYtWqVXzyySdkZmZy5swZABYvXsyECRMoKyujubm53ek+RUSka0mKwtzenW1nSeR6zM45li5desV5u3btYtasWWRmZgKQkZEBwK5duygpKQGgZ8+e9O3bt3MvVkREfCMpCrNXLq3HfPz48SvWY05NTWXo0KExrcd8teeJiEj3o5e/2lFYWMiWLVvYunUrs2bNor6+/qrWY27rvEmTJvHOO+9w+vRpgMtD2ZMnT768xGNzczP19fWdcHUiIuJHKsztiLYec2VlJcFgkJKSkpjXY27rvNzcXF588UUmTJhAXl4ezz33HACvv/465eXlBINBxo4dy6FDhzrtGkVExF80lN2BRKzH3N558+bNY968ea3aBg4cyHvvvXcV0YqISLLTHbOIiIiP6I45gbQes4iI/FIqzAmk9ZhFROSX8vVQtnPO6xB8TzkSEelafFuYA4EAp0+fVuFph3OO06dPEwgEvA5FREQSxLdD2UOGDKG2tpZTp055HYovNTU1EQgECAQCDBkyxOtwREQkQWIqzGaWAbwJTAXqgH9wzv1LlOMM+C3wm3DTOqDIXcVtb2pqKtnZ2fGe1m1UVFQwZswYr8MQEZEEi/WOeTVwARgI3AbsMLMvnXMHI457DJgB5AEO+HfgCLA2MeGKiIh0bR0+Yzaz64CHgeXOuUbn3B5gOzA3yuHzgN8752qdc98Bvwf+LoHxioiIdGmxvPw1HLjonKtp0fYlkBvl2Nzwvo6OExERkShiGcpOB/43oq0euL6NY+sjjks3M4t8zmxmjxEa+gY4b2YHYgtZwjIJPe+X2Cln8VPO4qecxa+75uyWaI2xFOZGoE9EWx+gIYZj+wCN0V7+cs4VA8UAZlbpnBsXQywSppzFTzmLn3IWP+UsfspZa7EMZdcAKWY2rEVbHhD54hfhtrwYjhMREZEoOizMzrkfgHeBlWZ2nZndDRQAm6IcXgI8Z2aDzWwQ8DywIYHxioiIdGmxzvy1COgFnAT+CCx0zh00s3wza7ne4R+APwFVwAFgR7itI8Wxhyxhyln8lLP4KWfxU87ip5y1YJryUkRExD98O1e2iIhId6TCLCIi4iOeFmYzyzCzMjP7wcyOmtlsL+NJBmZWYWZNZtYY3v7idUx+Y2ZPmlmlmZ03sw0R+yabWbWZ/Z+ZlZtZ1N8Rdjdt5czMhpqZa9HfGs1suYeh+oKZpZnZm+G/Ww1mtt/MprfYr34Wob2cqZ+15vXqUrHOwS2tPemcW+d1ED72PbAKmEbopUUAzCyT0C8MfkPoJcWXgLeB8R7E6DdRc9ZCP+fcxWsbkq+lAN8CE4BjwP1AqZkFCc3noH52pfZydon6GR4W5hZzcI9yzjUCe8zs0hzcRV7FJcnPOfcugJmNA1quiTkTOOiceye8fwVQZ2Y5zrnqax6oj7STM4ki/DPSFS2a/s3MjgBjgRtQP7tCBzn73JOgfMrLoex45uCW1v7JzOrM7M9mNtHrYJJIq7ncw38o/gv1uVgcNbNaM1sfHnmQFsxsIKG/aQdRP4tJRM4uUT/D28Iczxzc8rMlwK3AYEK//fuTmf3K25CSRuRc7qA+15E64A5Cc/qOJZSrzZ5G5DNmlkooJxvDd8TqZx2IkjP1sxa8LMzxzMEtYc65z5xzDc658865jcCfCT2rkY6pz8UpvNRrpXPuonPuBPAkMNXMVGQAM+tBaBbEC4RyA+pn7YqWM/Wz1rwszPHMwS1tc4B5HUSSaDWXe/g9h1+hPhePSzMSdfufWpqZAW8Senn1Yefcj+Fd6mdtaCdnkbp1P/PsouOcg1sAM+tnZtPMLGBmKWY2B7gH+NDr2PwknJsA0BPoeSlfQBkwysweDu//R+Cr7vxCziVt5czM/sbMRphZDzO7AXgDqHDORQ7VdkdrgJHAg865cy3a1c/aFjVn6mcRnHOebUAG8K/AD4Ren5/tZTx+34ABwD5CQ2L/A3wKTPE6Lr9thN78dBHbivC+XwPVwDmgAhjqdbx+2NrKGfC3wJHw/6P/TWihmiyv4/V6I/Qs1AFNhIauL21zwvvVz+LImfpZ601zZYuIiPhItxy/FxER8SsVZhERER9RYRYREfERFWYREREfUWEWERHxERVmERERH1FhFhER8REVZhERER9RYRYREfGR/wev649REJVskgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(history.history).plot(figsize=(8, 5))\n",
    "plt.grid(True)\n",
    "plt.gca().set_ylim(0, 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ The training accuracy and the validation accuracy steadily increase during training, while the training loss and the validation loss decrease. Good!\n",
    "\n",
    "\n",
    "+ **The validation curves are close to the training curves, which means that there is not too much overfitting**.\n",
    "\n",
    "\n",
    "+ In this particular case, the model looks like it performed better on the validation set than on the training set at the beginning of training. But that’s not the case: indeed, **the validation error is computed at the end of each epoch**, while **the training error is computed using a running mean during each epoch**. So **the training curve should be shifted by half an epoch to the left**. If you do that, you will see that the training and validation curves overlap almost perfectly at the beginning of training. **<font color='red'>When plotting the training curve, it should be shifted by half an epoch to the left.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training set performance ends up beating the validation performance, as is generally the case when you train for long enough. You can tell that **the model has not quite converged yet, as the validation loss is still going down**, so you should probably continue training. It’s as simple as **<font color='red'>calling the `fit()` method again, since Keras just continues training where it left off</font>** (you should be able to reach close to 89% validation accuracy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tune the hyperparameters:**\n",
    "1. the first one to check is the learning rate (**always retune the learning rate after changing any hyperparameter**)\n",
    "2. try another optimizer (if 1 does not help)\n",
    "3. if still not great, try tuning model hyperparameters such as the number of layers, the number of units per layer, and the types of activation functions to use for each hidden layer.\n",
    "4. can also try tuning other hyperparameters, such as the batch size (it can be set in the `fit()` method using the `batch_size` argument, which *defaults to 32*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.344361Z",
     "start_time": "2020-01-03T11:07:16.843379Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/1 - 0s - loss: 0.3194 - acc: 0.8681\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.36434496695995333, 0.8681]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Once you are satisfied with your model’s validation accuracy,\n",
    "# you should evaluate it on the test set to estimate the generalization error\n",
    "# before you deploy the model to production.\n",
    "model.evaluate(x_test, y_test, verbose=2)  # `batch_size` and `sample_weight` are supported"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='blue'>It is common to get slightly lower performance on the test set than on the validation set, because the hyperparameters are tuned on the validation set, not the test set</font> (however, in this example, we did not do any hyperparameter tuning, so the lower accuracy is just bad luck). **Remember to resist the temptation to tweak the hyperparameters on the test set**, or else your estimate of the generalization error will be too optimistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.428869Z",
     "start_time": "2020-01-03T11:07:17.346313Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.  , 0.  , 0.  , 0.01, 0.  , 0.01, 0.  , 0.98],\n",
       "       [0.  , 0.  , 1.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],\n",
       "       [0.  , 1.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ]],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make predictions on new instances\n",
    "x_new = x_test[:3]\n",
    "y_proba = model.predict(x_new)\n",
    "y_proba.round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> for each instance the model estimates one probability per class, from class 0 to class 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.435628Z",
     "start_time": "2020-01-03T11:07:17.431093Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 2, 1])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(y_proba, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.479631Z",
     "start_time": "2020-01-03T11:07:17.438192Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([9, 2, 1])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# only care about the class with the highest estimated probability (even if that probability is quite low)\n",
    "y_pred = model.predict_classes(x_new)\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.486044Z",
     "start_time": "2020-01-03T11:07:17.481878Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Ankle boot', 'Pullover', 'Trouser'], dtype='<U11')"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(class_names)[y_pred]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Regression using the sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**dataset:** the California housing data (no `ocean_proximity` feature, and no missing value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.491431Z",
     "start_time": "2020-01-03T11:07:17.488259Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.519489Z",
     "start_time": "2020-01-03T11:07:17.493561Z"
    }
   },
   "outputs": [],
   "source": [
    "housing = fetch_california_housing()\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    housing.data, housing.target)\n",
    "x_train, x_valid, y_train, y_valid = train_test_split(\n",
    "    x_train, y_train, test_size=0.25, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "x_train = scaler.fit_transform(x_train)\n",
    "x_valid = scaler.transform(x_valid)\n",
    "x_test = scaler.transform(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.525899Z",
     "start_time": "2020-01-03T11:07:17.521615Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((11610, 8), (5000, 28, 28), (5160, 8))"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape, x_val.shape, x_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.532587Z",
     "start_time": "2020-01-03T11:07:17.528439Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.442, 1.687, 1.621, ..., 0.68 , 0.613, 1.97 ])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:17.538674Z",
     "start_time": "2020-01-03T11:07:17.534751Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8,)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:29.788304Z",
     "start_time": "2020-01-03T11:07:17.540942Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/20\n",
      "11610/11610 [==============================] - 1s 94us/sample - loss: 1.6321 - mae: 0.9071 - val_loss: 1.5903 - val_mae: 0.6559\n",
      "Epoch 2/20\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 0.7124 - mae: 0.6159 - val_loss: 0.6539 - val_mae: 0.5777\n",
      "Epoch 3/20\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 0.6348 - mae: 0.5782 - val_loss: 0.6039 - val_mae: 0.5565\n",
      "Epoch 4/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.5983 - mae: 0.5596 - val_loss: 0.5738 - val_mae: 0.5388\n",
      "Epoch 5/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5713 - mae: 0.5445 - val_loss: 0.5406 - val_mae: 0.5244\n",
      "Epoch 6/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5482 - mae: 0.5312 - val_loss: 0.5221 - val_mae: 0.5144\n",
      "Epoch 7/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5298 - mae: 0.5231 - val_loss: 0.4965 - val_mae: 0.5036\n",
      "Epoch 8/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.5138 - mae: 0.5138 - val_loss: 0.4843 - val_mae: 0.4950\n",
      "Epoch 9/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.5005 - mae: 0.5066 - val_loss: 0.4684 - val_mae: 0.4863\n",
      "Epoch 10/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.4884 - mae: 0.4998 - val_loss: 0.4558 - val_mae: 0.4795\n",
      "Epoch 11/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.4783 - mae: 0.4938 - val_loss: 0.4469 - val_mae: 0.4754\n",
      "Epoch 12/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.4696 - mae: 0.4894 - val_loss: 0.4383 - val_mae: 0.4707\n",
      "Epoch 13/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.4621 - mae: 0.4852 - val_loss: 0.4320 - val_mae: 0.4677\n",
      "Epoch 14/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.4554 - mae: 0.4822 - val_loss: 0.4255 - val_mae: 0.4630\n",
      "Epoch 15/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.4495 - mae: 0.4784 - val_loss: 0.4184 - val_mae: 0.4605\n",
      "Epoch 16/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.4443 - mae: 0.4759 - val_loss: 0.4126 - val_mae: 0.4564\n",
      "Epoch 17/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.4397 - mae: 0.4729 - val_loss: 0.4087 - val_mae: 0.4549\n",
      "Epoch 18/20\n",
      "11610/11610 [==============================] - 1s 53us/sample - loss: 0.4352 - mae: 0.4702 - val_loss: 0.4057 - val_mae: 0.4538\n",
      "Epoch 19/20\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 0.4316 - mae: 0.4691 - val_loss: 0.4005 - val_mae: 0.4494\n",
      "Epoch 20/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.4280 - mae: 0.4666 - val_loss: 0.3975 - val_mae: 0.4481\n"
     ]
    }
   ],
   "source": [
    "# Since the dataset is quite noisy,\n",
    "# we just use a single hidden layer with fewer neurons than before,\n",
    "# to avoid overfitting.\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mean_squared_error\",\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=['mae'])\n",
    "\n",
    "history = model.fit(x_train, y_train, epochs=20, batch_size=32,\n",
    "                    validation_data=(x_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:29.962254Z",
     "start_time": "2020-01-03T11:07:29.790477Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3515 - mae: 0.4624\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4219739536906398, 0.46242753]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:30.039926Z",
     "start_time": "2020-01-03T11:07:29.964159Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.36892918],\n",
       "       [1.6819053 ],\n",
       "       [3.0886497 ]], dtype=float32)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_new = x_test[:3]\n",
    "model.predict(x_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:30.047595Z",
     "start_time": "2020-01-03T11:07:30.042640Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.477  , 0.458  , 5.00001])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:30.341614Z",
     "start_time": "2020-01-03T11:07:30.050206Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeYAAAE3CAYAAAB/8eJFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3gc1b3/8ffZvqtdSatdFau5qbjJXdg0YyDBlBBaKKFcIJcSyCWdhJAQnPwISSDJvSlcA7lJKAECoTeDKRbYNNu4G9uSu2XZ6m0lbZ/fH7NaFavasi3Z39fzzDOz0/aMzMNnz5kzZ5SmaQghhBBieDAc6wIIIYQQooMEsxBCCDGMSDALIYQQw4gEsxBCCDGMSDALIYQQw4gEsxBCCDGMSDALIYQQw8iAglkp9V9KqVVKqYBS6rF+9v2eUuqAUqpJKfV3pZR1SEoqhBBCnAAGWmOuAO4D/t7XTkqpBcBdwNnAaGAc8IvDKaAQQghxIhlQMGua9qKmaS8Dtf3sej3wN03TNmmaVg/8P+CGwyuiEEIIceIY6nvMk4F1nT6vA9KVUp4h/h4hhBDiuGQa4vM5gcZOn9uXXXSrbSulbgFuAbDZbLNyc3MH/WXV4WpC0QhBfxpZTgPmEdCVLRqNYjCMgIIOgRPlWk+U6wS51uPViXKtw+06S0tLazRNS+2+fqiD2QckdvrcvtzcfUdN0x4FHgUoLCzUtm7dOugvW/jxQt7ZtZR963/EM7fMZc644V8xLykpYf78+ce6GEfFiXKtJ8p1glzr8epEudbhdp1Kqd09rR/qnw6bgGmdPk8DKjVN6+/e9CFJsaXgCzUAUap9gSPxFUIIIcRRNdDHpUxKKRtgBIxKKZtSqqfa9hPAfyqlJimlkoGfAY8NWWm78dq9RImijK1UN0swCyGEGPkGWmP+GdCG/ijUtbHlnymlcpVSPqVULoCmaW8BDwBLgT3AbuDeIS91jNfuBcBk8UkwCyGEOC4M6B6zpmkLgYW9bHZ22/cPwB8Oq1QD5LHr95STnH4JZiGEOApCoRDl5eX4/f5jXZRBS0pKYvPmzUf9e202G9nZ2ZjN5gHtP9Sdv46q9hqzK8Ev95iFEOIoKC8vx+VyMWbMGJRSx7o4g9Lc3IzL5Tqq36lpGrW1tZSXlzN27NgBHTN8+o0fgvZgttlapMYshBBHgd/vx+PxjLhQPlaUUng8nkG1MIzoGrPD5MBmtGFWPg5IMAshxFEhoTw4g/17jegas1IKj92DMvmobQkSiWrHukhCCCGOMKfT2f9OI9iIDmbQO4CFVRORqEZ9a/BYF0cIIYQ4LCM+mL02L0FNH/lT7jMLIcSJQ9M07rzzTqZMmUJRURHPPvssAPv372fevHlMnz6dKVOmsGzZMiKRCN/85jfj+/73f//3MS5970b0PWbQO4D5Ip8DejBPHHWMCySEEOKoePHFF1m7di3r1q2jpqaG4uJi5s2bx9NPP82CBQv46U9/SiQSobW1lbVr17J//342btwIQENDwzEufe9GfDB77B58oUYgIjVmIYQ4in7x2ia+qGga0nNOykzk3gsnD2jf5cuX8/Wvfx2j0Uh6ejpnnHEGK1eupLi4mG984xuEQiEuvvhipk+fzrhx49i5cyd33HEHF1xwAeecc86Qlnsojfym7NgjU8rUIs8yCyGEYN68eXz44YdkZWVxww038MQTT+B2u/n444+ZP38+Dz/8MDfddNOxLmavjosaM4BDnmUWQoijaqA12yPl9NNP55FHHuH666+nrq6ODz/8kAcffJDdu3eTnZ3NzTffTCAQYPXq1Zx//vlEo1Euu+wyCgsLufbaa49p2fsy8oPZFhuW0+WnSoJZCCFOGJdccgmffPIJ06ZNQynFAw88QEZGBo8//jgPPvggZrMZp9PJE088wb59+7j++uvjx/76178+hiXv24gP5vam7ARHG9XNI2/sViGEEIPj8/kAfSyLBx98kAcffLDL9uuvv75LCLdbtmzZUR+S81CM+HvM7U3ZNpu8+lEIIcTIN+JrzHaTnQRzAkYlr34UQggx8o34GjPEmrONzTT5w/hDkWNdHCGEEOKQHRfB7LF5CCn9WboaeWRKCCHECHZ8BLPdQyAqw3IKIYQY+Y6LYPbavTSH6wAJZiGEECPbsAhmS7D+sI732Dy0hn2gQjL6lxBCiBFtWASzNVAHrXWHfHx8WE6jjP4lhBBiZBsWwQwabHv3kI9uD+YkV5sEsxBCHOd27drFhAkTuOGGGygoKOCaa67h3Xff5dRTTyU/P58VK1awYsUKTj75ZGbMmMEpp5zC1q1bAYhEItx5550UFxczdepUHnnkkWN8NQcbFsGsKROUvnXIx7cPMpKUEJBgFkKIE8C2bdv4wQ9+wJYtW9iyZQtPP/00y5cv53e/+x33338/EyZMYNmyZaxZs4Zf/vKX3H333QD87W9/IykpiZUrV7Jy5Ur++te/snPnzmN8NV0NiwFGwiaHXmOOhMBoHvTx7TVmh71V7jELIcTRsvguOLBhaM+ZUQTn/abf3caOHUtRUREAkydP5uyzz0YpRVFREbt27aKxsZHrr7+esrIylFKEQiEAlixZwvr163n++ecBaGxspKysjLFjxw7tdRyGYRLMCeBvhL2fwZjTBn18ii0FAIutheoaCWYhhDjeWa3W+LLBYIh/NhgMhMNh7rnnHs4880xeeukldu3axfz58wHQNI0///nPLFiw4FgUe0CGRTBHjHYw+PXm7EMIZovRQqIlEaNBH5ZT0zSUUkegpEIIIeIGULM9VhobG8nKygLgsccei69fsGABixYt4qyzzsJsNlNaWkpWVhYJCQnHqKQHGyb3mA16IJe+fcjn8Nq9aMYmAuEozYHwEJZOCCHESPOjH/2In/zkJ8yYMYNwuCMTbrrpJiZNmsTMmTOZMmUKt956a5ftw8GwqDEDUHAuvPVjqN0OnvGDPtxj91DZpI/+VdUUINE2+HvVQgghhr8xY8awcePG+OfONeLO20pLS+Pr77vvPpqbmzEYDNx///3cf//9R628gzUsaswAFJyjz8uWHNLhXpsXvwzLKYQQYoQbPsGcMg68hYf82JTH7qEppI8gJj2zhRBCjFTDJ5gBChbAro/A3zToQz12D/5IK6ig1JiFEEKMWMMsmM+FaAh2LB30oe3PMlusPglmIYQQI9bwCuacOWBLOqTe2e3B7HbK6F9CCCFGruEVzEYT5H1ZD+ZodFCHemz6sJwup1/uMQshhBixhlcwg96c3VoDFasHdVh7jdlub5UasxBCiBFr+AVz3tmgDIPune22uVEozBa5xyyEEELndDp73bZr1y6mTJlyFEszMMMvmB0pkDN30MFsMphw29wYTD7qWgJEotoRKqAQQghx5Ay/YAb9sakDG6Bx36AO89g9RFQTUQ1qW6TWLIQQx5u77rqLhx56KP554cKF3HfffZx99tnMnDmToqIiXnnllUGf1+/3c+ONN1JUVMSMGTNYulR/OmjTpk2cdNJJTJ8+nalTp1JWVkZLSwsXXHAB06ZNY8qUKTz77LNDdn0wnIbk7KzgXHj3Xih7G2Z/Y8CHeWweKoINgD76V5rLdqRKKIQQJ7zfrvgtW+q2DOk5J6RM4Mcn/bjX7VdeeSXf/e53+da3vgXAc889x9tvv823v/1tEhMTqampYe7cuXz1q18d1MuMHnroIZRSbNiwgS1btnDOOedQWlrKww8/zHe+8x2uueYagsEgkUiEN998k8zMTN544w1Af2HGUBqeNebUQkgePejHprx2L62RjmAWQghxfJkxYwZVVVVUVFSwbt063G43GRkZ3H333UydOpUvfelL7Nu3j8rKykGdd/ny5Vx77bUATJgwgdGjR1NaWsrJJ5/M/fffz29/+1t2796N3W6nqKiId955hx//+McsW7aMpKSkIb3G4VljVkqvNa9+AkJtYLYP6DCv3UtTsB7QJJiFEOII66tmeyRdfvnlPP/88xw4cIArr7ySp556iurqaj7//HPMZjNjxozB7/cPyXddffXVzJkzhzfeeIPzzz+fRx55hLPOOovVq1fz5ptv8rOf/Yyzzz6bn//850PyfTBca8yg32cOt8HOZQM+xGPzEIwGwBCQZ5mFEOI4deWVV/Kvf/2L559/nssvv5zGxkbS0tIwm80sXbqU3bt3D/qcp59+Ok899RSgv5Vqz549FBYWsmPHDsaNG8e3v/1tLrroItavX09FRQUOh4Nrr72WO++8k9WrB/d4b3+GZ40Z9PczmxP03tntb57qh8ceG2TE0SY1ZiGEOE5NnjyZ5uZmsrKyGDVqFNdccw0XXnghRUVFzJ49mwkTJgz6nLfffju33XYbRUVFmEwmHnvsMaxWK8899xxPPvkkZrM53mS+cuVK7rzzTgwGA2azmUWLFg3p9Q3fYDZZYfyZ+n1mTdObt/vRPshIkstPlQSzEEIctzZs2BBf9nq9fPLJJz3u5/P5ej1H53c322w2/vGPfxy0z1133cVdd93VZd2CBQtYsGDBoRR7QIZvUzbo95mbyqFy04B2b68xO6XGLIQQYoQavjVmgPxYE3bpW5DR/+gs7TVmm62VmkoJZiGEEHrt+rrrriMajWIw6PVRq9XKZ599doxL1rMBBbNSKgX4G3AOUAP8RNO0p3vYzwr8EbgEMAMfAd/UNG1wI4W0c6VD5ky9OXveD/vdPdmajFEZMcmwnEIIIWKKiopYu3Ytzc3NuFyuY12cfg20KfshIAikA9cAi5RSk3vY7zvAycBUIBOoB/58WCUsOBfKV0JLTb+7GpSBFFsKyuijORCmLRg5rK8WQgghjrZ+g1kplQBcBtyjaZpP07TlwKvAdT3sPhZ4W9O0Sk3T/MCzQE8BPnAFCwANyt4Z0O5eu5eI0kdhqZFHpoQQQowwA2nKLgDCmqaVdlq3Djijh33/BvxRKZUJNKDXrhf3dFKl1C3ALQCpqamUlJT0/O2axsmWFBo/epIvGkb1X9pWqA8eAODtDz4hz23s/5ijyOfz9X6tx5kT5VpPlOsEudbj1WCuNSkpiebm5iNboCMkEokcs7L7/f4B/40HEsxOoKnbukagp4b6MmAvsA+IABuA/+rppJqmPQo8ClBYWKjNnz+/9xI0X0jappdIO+0UMFn6LOy7y9+lslzvNp+dP5n5UzL63P9oKykpoc9rPY6cKNd6olwnyLUerwZzrZs3bx4R92l7cizvMdtsNmbMmDGgfQdyj9kHJHZblwj09LPjIcAKeIAE4EV6qTEPSsG5EGiCPT0/p9aZPixnHaDJ6F9CCHGC6+t9zMPVQIK5FDAppfI7rZsG9PRw8XTgMU3T6jRNC6B3/DpJKeU9rFKOOwOM1gG91MJj9xDWwiijPMsshBBi5Om3KVvTtBal1IvAL5VSN6GH70XAKT3svhL4D6VUCdAK3A5UaJrWf5fqvlgSYOw8/Xnmc+/vc9f2Z5ndLr8EsxBCHEEH7r+fwOahfe2jdeIEMu6+u9ftd911Fzk5OfHXPi5cuBCTycTSpUupr68nFApx3333cdFFF/X7XSUlJdx7770kJyezYcMGrrjiCoqKivjjH/9IW1sbL7/8MuPHj+e1117jvvvuIxgM4vF4eOqpp0hPT6elpYU77riDjRs3EgqFWLhw4YC+tz8DfVzqdsAOVAHPALdpmrZJKXW6UqrzeGc/BPzo95qrgfPRn2k+fAULoG471Gzrc7eOYTkDEsxCCHGcufLKK3nuuefin5977jmuv/56XnrpJVavXs3SpUv5wQ9+gKZpAzrfunXrePjhh9m8eTNPPvkkpaWlrFixgptuuok//1l/2ve0007j008/Zc2aNVx11VU88MADAPzqV7/irLPOYsWKFSxdupQ777yTlpaWw77GAQ0womlaHXBxD+uXoXcOa/9ci94Te+h1HgXM22N/MkB/wxRAgqNN7jELIcQR1FfN9kjp/D7m6urq+PuYv/e97/Hhhx9iMBji72POyOi/829xcTGjRulP/IwfP55zztGzpqioiKVLlwJQXl7OlVdeyf79+wkGg4wdOxaAJUuW8Oqrr/K73/0O0Hte79mzh4kTJx7WNQ7vITk7c4+GtEl6MJ/SRzDHxsu2WVvYL8NyCiHEcWco38dstVrjywaDIf7ZYDAQDocBuOOOO/j+97/PV7/6VUpKSli4cCEAmqbxwgsvUFhYOKTXN7xfYtFdwQK9Z3ZbQ6+7JFoSMRvMGM36sJwDbc4QQggxMhyJ9zH3pbGxkaysLAAef/zx+PoFCxbw5z//OZ4za9asGZLvG2HBfC5Ew7D9/V53UUrhsXvQjM0EI1Ga2sJHsYBCCCGOtJ7ex7xq1SqKiop44oknDul9zH1ZuHAhl19+ObNmzcLr7XjI6J577iEUCjF16lQmT57MPffcMyTfN3KasgGyi8Hu1h+bmnJpr7t5bV4CQX1YzmqfnySH+WiVUAghxFEwFO9jnj9/fpeBVTqPzNV520UXXdRjb2u73c4jjzwyuIIPwMiqMRuMeiewsiUQ7f0FFR67B39UD+aqJrnPLIQQYuQYWTVm0O8zr38WyldB7pwed/Havayv3gggPbOFEOIEd1y+j3lYGX82KKPeO7uXYPbYPTQG64GoPMsshBAnuOP1fczDhz0ZRp/S5/CcHpuHqBbFYpXRv4QQYqjJ0y6DM9i/18gLZtCbs6s2QcOeHje3j/6VIsNyCiHEkLLZbNTW1ko4D5CmadTW1mKz2QZ8zMhrygb9saklP9NrzSfdfNDm9mBOdPrlHrMQQgyh7OxsysvLqa6uPtZFGTS/3z+ogBwqNpuN7OzsAe8/MoPZkwcp4/Te2T0Ec/voXw57G9X1EsxCCDFUzGZzfEjKkaakpGTA70Q+lkZmU7ZSeq15xwcQPHjA8PYas8Xqk6ZsIYQQI8rIDGbQ7zNHArDzw4M2OUwO7CY7RnMLda1BQpHoMSigEEIIMXgjN5hzTwGLS39sqhulFCm2FDRDE5oGdS3BY1BAIYQQYvBGbjCbLJB3lt4BrIfegV67l6AWG5ZTmrOFEEKMECM3mEG/z9y8Hw6sP2iTx+ahLaq/hUqCWQghxEgxsoM578uA6nGwEa/dS1OoHoB9DW1HuWBCCCHEoRnZwexMhezZPd5n9tq9NAUbyE6x8NvFW/h4e80xKKAQQggxOCM7mEHvnb3vc/BVdVnd/izzousKyUiyccPfV/L6+opjUUIhhBBiwI6DYD5Xn5ct6bK6PZiNFh///ubJTMtJ4o5n1vDYRzuPdgmFEEKIARv5wZw+BRKzDmrObh9kpLatlmSHhSf/cw5fnpjOwte+4LdvbZFxXoUQQgxLIz+YldKbs7cvhXBH72uPTa8x17Tp95ZtZiOLrp3F1XNyWVSynR/8e50MPCKEEGLYGfnBDHpzdtAHuz+Kr2pvyq7118bXGQ2KX108he9/uYAXV+/jpsdX0RIIH/XiCiGEEL05PoJ57Dww2bs8NmU32XGandS21XbZVSnFt8/O5zeXFrGsrJqr//optfIGKiGEEMPE8RHMZjuMOwO2Lu4yCpjH7ok3ZXd31Um5PHLdbLYcaOayRR+zp7b1aJVWCCGE6NXxEcyg32du2A01pfFVHlvvwQzw5UnpPH3zHBraQly66GM27ms8GiUVQgghenX8BHP+An3eqXe2x+7pco+5J7NGp/D8N0/GajJw1aOfsrxMBiIRQghx7Bw/wZyUBRlFXe4ze+3ePmvM7fLSXLxw2ylkJdu58bEVvLJ235EsqRBCCNGr4yeYQe+dvedTaK0D9GBuDjYTjPT/2seMJBvPffNkZuS6+c6/1vJ/y3Yc6dIKIYQQBzn+glmLwPb3gY5nmbv3zO5Nkt3ME984ifOmZHDfG5u5/83NRKMyEIkQQoij5/gK5syZ4PDG7zO3j/41kObsdjazkb9cPZP/OHk0j364g+8/t5ZgWAYiEUIIcXSYjnUBAFRgiJ4jNhj03tlb3oBIuGNYzn46gHVnNCh+8dXJpCfaePDtrdS2BFl07Syc1mHx5xJCCHEcGxY1ZlNlJU1vHfzqxkNSsAD8DVC+Ij761/rq9YMeG1spxbfOzOOBr03l4+21XPXoJ1Q3y0AkQgghjqxhEcyaxcq+732fuqeeOvyTjTsTDGYofYtUeyoz02by1w1/5ca3b2RL3ZZBn+6K2Tn89T9msa3Kx2WLPmZXTcvhl1EIIYToxbAI5nB6Gs7586n8f/dR9cc/Ht6bn2yJMOZUKH0bo8HI3xf8nXvm3sP2hu1c8doV/OKTX1DnrxvUKc+akM4zN8+l2R/iskUfs7684dDLJ4QQQvRhWAQzSpH95z+RdNml1C56mAM/vxctfBgvl8hfANVboH4XRoORKwqv4PVLXueaidfwctnLfOXFr/DEpicIRUIDPuWMXDfP33YKdouRqx79lA9Lqw+9fEIIIUQvhkcwA8pkYtR99+G59VYa/v1vyr/zXaJ+/6GdrKB9FLAl8VVJ1iR+fNKPeeGrLzA1dSoPrnqQS1+9lGXlywZ82vGpTl687RRGexL4xmMreWlN+aGVTwghhOjFsAlm0DtcpX3vu6T/9Kf43n+fPf95E5HGQxi/2jMePPldhudsNy55HIu+tIiHzn4IDY3b37ud29+9nZ2NOwd06rREG8/eOpfiMSl879l1/M+7pfLqSCGEEENmWAVzu5TrriXrD7+nbf16dl97HaHKysGfpGAB7FoGAd9Bm5RSzMuex0tffYkfzv4ha6rWcOkrl/LgygdpDjb3e+pEm5nHvlHMhdMy+Z93y5h7/3ssfHUT26sP/i4hhBBiMIZlMAMknnceuY8+Qqiigl1f/zqBHYMcIrPgXIgEYUdJr7uYjWaun3w9r13yGhflXcSTXzzJV176Cs+XPk8kGunz9FaTkT9dNZ0Xbz+FL01K56nPdnP27z/gur99xjtfVBKREcOEEEIcgmEbzAAJJ59M7hOPowWC7L76GtrWrRv4wblzwZrUY3N2d167l4WnLORfX/kXYxLH8ItPfsFVb1zFqgOr+jxOKcXMXDf/feV0Pr7rbH54TgFllT5ufmIV8x5YysMfbKe+pf9xuoUQQoh2wzqYAeyTJzPmmacxuFzsvuFGfB9+OLADjWbIOxvKlkB0YENqTvJM4rFzH+OBeQ9Q76/nxrdv5Icf/JD9vv39HpvqsvJfZ+Wz/MdnsuiameSk2PnN4i3M/fV73PnvdWwol3c9CyGE6N+wD2YAS24uY555GsvYMey9/Vs0vPzywA4sOBd8lVC+csDfpZTivLHn8dolr3HbtNso2VvChS9fyP+u/V/awm39Hm8yGjivaBT/uuVk3v7uPC6fnc0bG/Zz4V+Wc8n/fsTHFWEC4b6byYUQQpy4RkQwA5i8XkY/8QSO4tnsv+sn1P7t7/0flP9lsLjg8a/AS7fB/oE3hdtNdm6ffjuvXfwaZ+acyaJ1i7jwpQtZvHPxgAdAKcxwcd/FRXx699nce+EkGlpDPLo+wKm/eZ/fvb2V/Y39B70QQogTy4CCWSmVopR6SSnVopTarZS6uo99ZyqlPlRK+ZRSlUqp7wxVYY1OJzmPPILrvHOpevBBKn/zW7S+mqkdKXDrBzDzevjiFXhkHvzjfNj8GvTTuavdKOcoHjzjQR479zFSbCn86MMfcf1b17OpdtOAy51oM3PjqWN57/tn8MPZVqbnJPNQyTZO++1Sbvvn53yyvfbwRjsTQghx3Bjo65IeAoJAOjAdeEMptU7TtC7ppJTyAm8B3wOeByxA9tAVFwwWC1m//z2VKR7qHnuMcG0tmb+6D2Wx9HyAZzxc8Ds462ew5kn47FF49lpIzoWTboUZ14I9ud/vnZU+i2cueIaXt73Mn9b8ia+//nUuyb+EO2bcEX+LVb9lNyimeE3819eK2VvXyj8/282zK/eyeOMBCtKd/MfJY7hkRhYJ8hYrIYQ4YfVbY1ZKJQCXAfdomubTNG058CpwXQ+7fx94W9O0pzRNC2ia1qxp2uahLTIog4H0n/2U1O9+l6bXXmPv7d8i2tLPyyXsyXDKHfDtNXDFk5CYDUt+Cn+YBG/eCbXb+/1eo8HIZQWX8dolr3HdpOt4ddurnPP8OXzn/e/w1q63BnQPul1OioOfnDeRT39yNg98bSpmo4GfvbxRnokWQogT3ECqZgVAWNO00k7r1gFn9LDvXGCDUupjIA/4DPiWpml7Druk3Sil8H7zVkxeD/t/fi+7b7iRnEcexpSS0veBRhNM+qo+VayFzx6Gzx+DFY/qY2zPvQ3GzQelej1FoiWRO4vv5PKCy3l267O8test3t/7Pg6Tg7Nyz+K8sedxcubJmA3mfq/DZjZyxewcLp+Vzeo9DTzxyS6e+mw3j328i9PzvVwyI4uzJqSR7OilRUAIIcRxRfV3b1MpdTrwb03TMjqtuxm4RtO0+d32LQXSgC8DG4AHgFmapp3aw3lvAW4BSE1NnfXcc88d8kVY160j6f/+RsTtpv7b3ybq9QzqeEugnsyKt8isWIwl1EiLI5fy7AupTD+DqNHa7/FRLUqZv4zPWz9nbeta2qJtJBgSmO6YzuyE2YyzjsOg9MYJn8+H0+ns83yNAY0PykOU7A1T59cwKCh0G5iZbmJmmhGPfWT02RvItR4PTpTrBLnW49WJcq3D7TrPPPPMzzVNm919/UCCeQbwkaZpjk7rfgDM1zTtwm77rgNWa5p2Y+yzB6gBkjVN6/VB3sLCQm3r1q2DuZ6DtK5ezd5v3obBaiXn//6KrbBw8CcJB2DjC/Dp/8KBDWB3w6wbofgmSMoa0CmCkSAf7fuIxTsXU1JeQlu4jXRHOueOOZfzxp1H1foqzjzzzAGdKxrVWL+vkSWbDrDki0q2VenN21OyEjlnUgbnTE6nMN2F6qN2fyyVlJQwf/78Y12MI+5EuU6Qaz1enSjXOtyuUynVYzAPpCm7FDAppfI1TSuLrZsG9NQteT3QOemPWldjx8yZjHnqn+y56WZ2X3sd2Q/9hYSTThrcSUxWmH41TPs67P4YPlsEH/0PfPwnmHQRzLkNcor7PIXFaOHM3DM5M/dMWkOtlOwtYfHOxTy15Ske/+Jx0kxpbF67mfPGnsfYpLF9no/Owt8AACAASURBVMtgUEzPSWZ6TjI/OncC26t9vPNFJUs2HeAP75Tyh3dKyU1xcM6kdM6ZnMGs0W6MhuEZ0kIIIQam32DWNK1FKfUi8Eul1E3ovbIvAk7pYfd/AC8opf6EHtz3AMv7qi0PJWt+PmOeeZo9N9/C3ptuJvN3D5J4zjmDP5FSMOZUfarfrd9/Xv2kXpvOmq3fh550kT66WB8cZgfnjzuf88edT2OgkXd2v8PTq5/m4XUPs2jdIiamTOS8sedx3tjzyEjI6PNcoL92cvwZTr55xniqmvy8u7mKJV8c4IlPdvN/y3fiSbDwpYnpnDM5nVPzvNjMxsFfuxBCiGNqoM/l3A78HagCaoHbNE3bFLv/vFjTNCeApmnvK6XuBt4AHMByoNdnno8Ec2Ymo//5JOXfvI193/0ekZ/fg/uqqw79hO7RsOBXMP8nsO4Z+HQRvPCfsOQeOOkm/RnphP4fl0qyJvG1gq/hrfAy6aRJvL3rbRbvXMwfPv8Df/j8D8xMm8n5Y8/ny2O+TIqtnw5s6K+fvHpOLlfPyaXZH+KD0mqWbKrkzQ37eXbVXhwWI2cUpHLO5HTOKkwnydF/RzQhhBDH3oCCWdO0OuDiHtYvA5zd1i0CFg1J6Q6Rye0m97F/sO+73+PAwl/Q8MKLOIqLcRTPxjFrFsbExMGf1OqEk26G2f8J297V70O/90t98uRD9mzImqXP06f0WZtOc6Rx3aTruG7Sdexp2sPinYtZvHMx9312H79e8WvmZs7lgrEXcFbuWSSYE/otmstm5itTM/nK1EyC4Sif7qhlyRcHWLKpksUbD2A0KOaOS+GcSRl8eVI6mcn2wV+/EEKIo+K4HcnCYLeT/Zc/U/v3f+D78EPqn3ySur//HZTCOmGCHtLFxThmz8bkdg/ixAYoOEefqjbDlteh/HM9rNc9o+9jssGoaXqzd/YsfZ6c2+MjWLmJudw67VZumXoLpfWl8ZC+e/ndWI1Wzso9i0vzL+WkjJPiPbv7YjEZmFeQyryCVH751SldOo/d++om7n11E0VZSZwzKZ2zJ6YzcdTw7TwmhBAnouM2mAGU2Yz31lvw3noLUb+ftnXraV25ktaVK2l49jnqn3gS0O9Nx2vUxcWYvAMbyYu0ifoEoGnQsAf2rdKDet8qWPU3+PQhfXtCKmTNJjeUAts1yJoJtqSOsipFYUohhSmFfGfmd1hXvY43drzBmzvfZPHOxWQmZHJx3sVclHcRmc7MARWvr85jv3+nlN+/U4rXaeHUPC+n5nk5Pd/LqCSpTQshxLF0XAdzZwabjYQ5J5EwR++pHQ0G8W/YEAvqVTS8/DL1Tz8NgGXs2FhQF+M4qRhzenr/X6CUfj/aPRqmXKavi4SgclOXsB5Xsxh2PgUo8BbEmr9jter0yWA0o5Rietp0pqdN54fFP+T9Pe/zUtlLLFq3iEXrFjF31Fwuyb+Es3LPwjqA56zbde889kFpNcu31fDRthpeWVsBwLjUBE6PBfXc8R4SbXJvWgghjqYTJpi7M1gsOGbNwjFrFnwTtFAI/xdf0LpyJS0rV9L05ps0xAY9MefkdAR1cTGW7IE904zRDJnT9an4JgCWv/M6p41zdNSqy5bAOv0HASa73gTe6X61NSkn3nO7wlfBK9tf4ZVtr/CjD3+Ey+LigrEXcGn+pUz0TBzU9acl2rh8dg6Xz85B0zS2HGjmo201LN9Ww3Orynn8k90YDYpp2Umclp/KaXleZuQmYzaOjMFNhBBipDphg7k7ZTZjnzYN+7RpeG66CS0Swb9lS7xG3fzeezS++CIApsxRJBQXY589G8es2VjGjEYZBhZYYbMTxs+H8WfpKzQNGnZD+SrY97k+X/FXiPxF325LhrRJkD6JzLSJ3JY2nVvPu5IVjaW8VPYSL5a9yL+2/osJKRO4OO9iLhh7Acm2/l/K0eXalWLiqEQmjkrkptPHEQhHWLOngeVlelD/5f0y/vReGQkWI3PGeTgtz8tp+V7y05xyf1oIIYaYBHMvlNGIffJk7JMn47nhBrRolEBZGa0rVtK6ahW+ZctpfOVVAAwuF7bJk7FPmYxtyhRsU4owZ2UOLLSUAvcYfSr6mr4uHITKjXpQV27SO5mtfw4CTfr3AXMTs5ibNonGtC+z2BTmpaZSfrPiN/x+1e85O/dsLsm7hDmj5mA0DP5ZZqvJyNxxHuaO8/DDBYU0tob4ZEcty7dV89G2Wt7fUgVAmsvKabFm79PyvaQn2gb9XUIIIbqSYB4gZTBgKyzEVlhIynXXomkawR07aFuzhraNG/Fv3ETt409AKASAMTlZD+miKdinTME2ZcrA7lUDmCx657CsmR3rNA0ay6HqC32q1OdJO0q4KhriKmCrxcrLqZm8tusd3tr1FhmWJC4avYCLJ19PdlLuIV97ksPMuVMyOHeKPghKeX0rH22rYVlZDSWl1by4Zh8A+WlOTsv3clqelznjBjdeuRBCCJ0E8yFSSmEdPx7r+PEkf02v6UaDQQJbS/Fv2kjbhg16WD/6V4hEADClppI8KoPqTZuwFxVhmzKl/7dhdXwhJOfoU8GCjvWRkP7KyqpNFFZt5seVX/C9qo0sDVTzkquNRwPP8kjZc8yJmrk4YRxfyjodW8ZUvXncmdbnW7R6k+12cGVxLlcW5xKNanyxvyl+f/rpz/bwj492YTIosp2K0xo2MD3HzfScZMZ5EzDIkKFCCNEnCeYhZLBYsBdNwV40JT7aWLStDf/mLfg3bsS/aSP+z1ZQ85eH9Bow+v1q+xQ9pO1TJmObPBljUlJfX9OV0QxpE/QpxgIsCPhYUL2VA+Wf8Gr5Ul5q2cFP2rZy/9bNnLe6hUubW5hkdKK8+eDN1wdJ8Rboy+4x/Q43Gr9mg2JKVhJTspK49Yzx+EMRVu+u56PtNby/bicvr6ngn5/qb/102UxMy9Yf35oWe4wr1TXwXuVCCHEikGA+wgx2O46ZM3DMnAFAaUkJp8+ejf+LL/Bv0MO6beMmmpcsiR9jHp2LfbLe/G0tLMCal48pLXVwHa2sTsieRUb2LG7hv7hJi/J55ee8tPkZXi0v4blEF6MNdqZFGina+x5Fm56lIBjCDGAwgXusHtLdQ9vRdw3fZjZySp6XU/K8FFsPcPq8M9he7WPt3gZ92tPAog+2E4nqP0yyku3xZ62n5SRTlJWE3SJjfAshTlwSzMeA0ekk4aSTurz9KtLQQNumTfg3bsK/cQOta9bQ9Oab8e2GxES96TxvPNa8PCzj87Dm52FKSxtQYBuUgeKMYoozivlJsJnFOxezrHwZy2vW86pqBdcorAYzE21pTFF2pgaDFNVvI2vbu6hIsONEDk8sqPP0sG4PbffoHmvZRoOiIN1FQbqLK2bnANAWjLCxopG1expYW66H9Rsb9sf3L0x3MS0nmRk5yUzPTWZ8qlPemiWEOGFIMA8TxuRknKeeivPUU+PrwrW1BMq2Edi2jcD2bQTLttH8zrs0/Pv5+D4Gl0sP7Pw8LOPHY83Lx5o3HlN6eq+B7bK4uKLwCq4ovAJN09jfsp/1NevZUL2BDTUb+HftF/wzEoAESEmZyJSk8RRZPUyNGJnc0kRS3S4ofRvW/LPjpPFadoEe2p583HW1UJkKznSwp+jDmQJ2i5HiMSkUj+mofVc3B1gXq1WvK2/g9fUVPLNCbwJ3Wk0UZSXFm79n5CZLD3AhxHFLgnkYM3k8mDweEubO6bI+XFtLYNt2AtvKCG7fTqBsG83vvU+kc2A7nVjHj8eSn4d1fB7WvDw9sDMyugS2UopMZyaZzkzOHXMuAKFoiG3129hQs4H11evZWLORZVWfo8Verz3GPYai/CsoShpPkdFFYSCAuW471JTpU9kSiIaYBrD+3tgXGfXOZs702NR1OdWZzpfS0/jS+CywFhKNauyoaYmH9dq9Dfzfsh2EY03g6YnW+LPXk2Lzsd4EqVkLIUY8CeYRKB7Yc07qsj5cV6fXrrdtI7htO4Ft2/AtLaHx+Rfi+xgSErDEmsOtY8dizszElDEKc+YoTKmpKKMRs8HMRM9EJnomckXhFQA0B5vZVLuJjTUbWV+9nk/2f8JrO14DwGKwMMEzgaJxRRSddDVTUyaRHY6wdvnbzMgbBb4q8FV2zJv3w/510FIFWvTgCzQnYHCmkedMJ8+ZxmXOdJiaTmhuKrsCLr5osrGmTmNFdQvLy2riYW0zGyhMdzEpMzEe2hMyXLhkWFEhxAgiwXwcMaWkYOp27xogXF9PMBbYgfbALvmAxhde7HoCoxFzejqmUaMwt0+Zo+Kfi0dNYk7GHJRSaJrGgZYDbKjZEK9Zv1j2Ik9tfgoAt9XNWONYvmb/GqflXYnb1sMbvKIRaK2LhXblwQHuq4TqrbDzQ/A3YAbyY9NFAMqI5kmnzZ5OndHLvoibbf5ENmxI4PWVifyVFKo0N+kpyUwc5epSw85222XUMiHEsCTBfAIwud2YYuN8dxbx+Qjv309o/35CFfsJHdivf67YT9vatTS9/XZ8wJR2hoQEPawz9LCemTmKOaPyMI06HcOkVHbbWtjQuJm1VWtZunMpdy+/G4MyMNU7lTNyzmBe9jzyk/P1UDQYwZmqT0zp+yLCgVhgd6p1N+9HNVXgaNqHo2kP2U2fMifYrO/f6Sms5kASlTtT2F2azAEthec0Nw3mVKzubJLTRzMqdzz5uZkUpLuwmaVHuBDi2JJgPoEZnU6M+flY8/N73K5Fo4RranoNb/+mTUTq6roepBTTvV5mZ47iMtNYbIU5lNoaWVm5i+c3/w+Lkv4HT3Im87LncUb2GRRnFGMzDaAjl8naMcBKX/xNemg37YOm/dBUgau5AldTBeMaK4g0rsXsr9X3rY9NW8Cn2SjXUmgypxJKyMCUmIY9OYNk7yi86VlYEtMgwQsOL1gc/ZdXCCEOkQSz6JUyGDCnpWFOS8M+bVqP+0T9fsIHDnQE9/5YeFdUYN62HcP6DRSGQhQC18aOaXVVst/1DGWJT/OZ24QzdxyjC4uZVvQlMsYXYXAcRvDZEvUptfCgTYbYRDgQC+8Koo0VNBzYRWPVHiL1e0ny7Seh6XPcjY1Yy0MHnQMgZLQTsXkwOL2YE9MobA5D8D09uBNS9fBO8MTmqRLkQohBkWAWh8Vgs2EZMwbLmDEHbSspKeGMefMIV9cQ2rdPnyoqCO3bh6d8L749O2B7NcZPtgJbaeSfNALBRDumrEySRudhzsrCnJWFJTY3Z2YeXnCDXvuOvTjEAKRMhe7DprQGQmzeX0XFvr1UV+6jqfYAbQ0HiDRX4wo34gk24WlqwnugjFTVSKhyGWat5yDH7IiFtLej1u1I0Z8Jb58SvB3LtuT4o2VCiBOPBLM4opTBgDk9DXN6GsRGP+tMi0YJV9ewq3QFX2z6gIqytYT3VeBt3EHGql2kvqthDHftuW10uzGPGoUpLU2fUlMPnntSUKZD/8/bYTUzcUwWE8d0ffe2pmkcaPKzo7qFHdU+lla3sGrrXurDZhoa6/GoJjw0kaKayHP4GZfQRo61lXSTjxStEWfjAUyVX6BaayDs7+2PBnZ31+COB7m35/VW1yGNey6EGH4kmMUx1R7c+elfIf/0rwDQGGhk+b7lvFX+IR+VL4P6JkY1GzlJG8P0SCZjWhwYa5sJVVbStmGDfp87NvZ4nMGgP1bWU2h3ng8ywJVSjEqyMyrJzql5XgBKSqqZP38+bcEIO2ta2FHjY0d1C9urfXxc3cKOAz5agpH4OexmI6M9DvLdRiYkBRmfECDH1kamuZUkrQlDWx201nZMdTugfKW+HA33XDCDuWtY25L0cD9oSu762eKUQBdimJFgFsNOkjWJC8ZdwAXjLiAcDbOueh0flH/AsvJlPNGwHICMhAzGJI4h1zWD0Y4sRoeTyQzY8bQoVG0D4aoqQlVVhKur9QDfuJFIbW2PAW70pGBO7SG001IxpaYNOMDtFiOTMhOZlJnYZb2maVQ2BdhR7WN7jV7T3lPbyhc1Lbxd6icYiaK/esSCxZRCbsokxngcjPYkMCYnNvckkJlkxRT2xQK7W3i3Ty210Favv3HM36DvFwn0XmiDqSOkbcn9hrm9tVzvVGd1gjlBmtyFOAIkmMWwZjKYmJU+i1nps/j+rO9T3lzOB+UfsL56PXub9/L27rdpDDTG91coMhIyyC3MJac4h1zXXHITc8l15TLalo65qY1wVZU+VVfH56GqKkLVVbRt2jTwAI+FuKWqkjaPt9cAV0qRkWQjI8nGKbFadrtIVGN/Yxu7a1tjUwu7alvYXdvK8m01+EMdzfgmgyInxcFoj4MxngRGewr0eZaDbLcDi6mXkAy16WF90NRw8Lrm/VC1WV9uf/SskzkAKzqtsDhjU4Ie1hZXbO7sNHd1/dzbNrNDau9CIMEsRphsVzbXTLyGayZeE1/XGGhkT9Me9jTv6Zg37+Hd3e/SEGjocny6Iz0e1DmTcxh98iRyXDnkuHJwmPVOZVooRLiurmt4V1UTrq7qMcDdwK7/XaR/QecA76npPC0NU6oXo9uNwWLBaFBku/VgPTWv67VqmkZVc4BdNXpQtwf2rtoWVu2qxxfoaNY2KMhy2xmdkkC2205mcvtkIyvZTkZSOtbEzMH9sSMh8Dd2Ce4v1nzCpHE5EPRBwAfBFj3AA76OdU37Om3zQah1YN+nDB0hHw97V7fgd/Ye8t33tzilRi9GJAlmMeIlWZMoSi2iKLXooG2NgUbKm8vZ07yH3U272du8lz1Ne1i6dyl1/q7PYKfZ08hJzCHXlUtech6F3kIK82fitiX3+L3tAf7Z4sVMy83tUhMPVVX13YQOGBwOjG43xpQUjO5kTG43xmS3vs7txuhOxpWSwnS3m1nj3RhnZqKM+gAomqZR2xLUa9g1ek17d10ru2paeHdzMzW+g5uvU11WPayTbPHgzkruWPYkWLqOhmY0d/Qkj6mqsDKpeP5A/lk6RCOdgtzXdTng6xrswRYINHda9kFTecfyYIIe9Fp4vEaeAGYbmGKT2QYme7d19vjyqIo9sHZ/x34ma8d2c+xz/Hi7/veSGr8YAhLM4riWZE0iyZrEZO/kg7Y1B5vjQd25tv1B+Qe8tO2l+H5pjjQmpEyg0F1IQUoBhe5Ccl25GM1mzOnphMeMwTV/fq9l6FIDr6oiXF1DpKGBSH09kYZ6wnX1ROrqCW7fQaS+nmhrL8GjFMbExE5h7maUO5kct5v57YE+ORmjM4mQxUZt1ERlWLE/YKC8TaOiKcC+hjZKK5sp2VpNWyjS5fRWkyFey85Mag/ujpp3ZrL9kP4NMBj1zmi2pEM7vrtopKM23lOQB5u7Bnnn2n3Yr08t1frz7OE2CPn1eTjQpad8IUDpYK7TpIe/xRH7QZCgT2ZHbF3sc3w5to85of/tRvlf9YlE/rXFCctlcTHJM4lJnkkHbatpq6G0rpSt9Vv1qW4rH+37iIimh5ndZCcvOY8CdwGGZgNJVUkUuAtIMCccdC4VC3BzevqAyhUNBPTQjk3hej2440FeX0+kvoHQ3r34168n3NBw0NCp7ZJj00RAORwY4pOdqNVO0GKjzWSlxWCmWZmp32+iNmqiOmxkc9jAapOFNpOVNqMVv8mKslv5yyev4/Ik4012kp5kJz3RSlqijTSXlfREG6kuK2bjEWxCNhg7BpIZatGo3lku1MbHy5ZySvGMjuAO+TuCPdTWaTm2Pdiq1+aDLfrUvuxv0Jv3g60QatHn4bZBXrM5VkO3gtEKJoteczfG5iZrp+XO2zof0/s2d10p7DJ3a0Wwd20tkNaAo0aCWYgeeO1evFleTsk6Jb4uEAmwvWE7W+u2Ulqvh/Y7u9+hKdjEvxf/G4BsZzaFKYUUugv1eUohmQmZg3phhsFqxZCRgTkjY0D7a5pGtKUlHuTRlhZ9am3Vp5bWTsud1re2YmxtxVpXTWJrK+mx7YR7eSSrm7DBSIvJSqvJRqvJRpnZyjqTjTazjYjdgSEhAZPLiSUxEXuyiwR3EomeZNypyXjSUkhJdWNJdKEcjuHzQhGDAQx2MNsJWj2QMu7IfE80Egvu9rBu6RrcodZYbT+2HGqFcFD/IRAJdFqOzcNBCDXEav2xKdJpOewHDr6d0k5/RWtfBVY9NP93n9u6NfV3vTWg/xCwdfwg6Lyu8w+M7vsYTrzx6yWYhRggq9F6UA1b0zRefu9l3IVuttbptevS+lLe3/N+/P3VLrOLfHd+PLDz3fmMThxNknVomnaVUvq4504n5PQzlvgAaMFgjyEebW1l08pVFOTmEPW1EPX5CPt8+Bua8Dc0EWxuJtLsQ2upxNDQgsnfhjkc7PE7GmMTQFQpghY7EZsdzWpFWW0YbVaMdjsWhx1Lgh2L3YbRbkNZbRhs+j7KZsVgtaGs1q7rbO3rYnOrFWWzYbDbUfZh8lYxg1HvqGZ1HZ3v0zT9GfhwoFOYd4T26pWfMLNoYrfWgf7mnVoOWuu6fu7conC4DKa+w9tg0v+eyqjPDaZOy+3r9X0KKqug+eV+94t/Npp7b5Xo3AphtHaUp/P+h/ijQoJZiMOglMJtcjM/Zz7zc+bH17eGWilrKIvXrrfUbeHlbS/T1qkJM9maTG5iLqNdo8lNzNWfy07MZXTi6B6bxI8WZbFgtFgwJh/c6c1vNJLSx/307rRQiGhLC8GmZuqq66mrqqehpoGm2gZa6xtpa2wm2NQe6D4IBLCEQlj9Icx19VgjVVgiIazRMLZoCGskjDkawhzuZfjT/hgMGJxODAkJGJ0JGBKc8c8GZwJGpzO+zr6vnMaWFn2ds9N+Cfp+yjyC3vOtVCxkei5zU1IDjJs/9N+raR0/Arr8IOg+72FdpKf1/o7WgvZ10Yj+BEG0TV+OhvX3vMeXY/NoFI+/FZrWxrZFYts67dfT++EPhzL2Et6xeS8kmIU4AhxmB9NSpzEttePlH1EtSnlzOdsbtrOneQ+7mnaxp2kPKw6s4LUdr3U53mPzMDpxNKMTR8fDOteVS25iLnbTIXbCOgaU2YwxORl7cjJZuTlk9bN/JKpR3xqk1hek1hegpiXIAV9A/9wSoCa2vs7np6mxlWCbH2skpId3JIQlqi87VQSPCTzmKMlGjWRjFBdhXNEgCeEA9pAfa8iPOdCGqaERVVGhtxD4fHrnO00jEah46uner81q7RLqBosVZbWiLBaUxYLBakGZLV3WKasFQ/uypdu+lti+5h72tdkxJDj0Wr9xBDXtKqU3Z5uHx3+zn5SUML+vH5aa1imww3rgR4I93x446JZCoNvyAPbvhQSzEEeJQRn0Z6gTcw/a1hZuY2/zXnY37WZ30272NOmPdy3bt4yabTVd9k13pHcEdqfadrYrG4vRcrQu54gwGhRepxWv0wr038zrD0WobdHDutYXpMYX6PJ5V6fl2pYAoYimD7LWjUGB22HBnWAhxW4k1aSh1VYwOSOFFEIkaUGStBDOaICEUABbyI/Z39pxP9/nQwsGiAb02wBaMIgWCOi3BUJBtEAwvq6nR+cGQ9ls+o+B9o58/S0n9L2fOtyXwhxPlIr1gDfR5aXuR8p/9HxbRYJZiGHAbrJT4C6gwF1w0DZf0Bd/nGt30+54bbv7ACoGZWBUwijSHemkO9JJdaSS5kjrMqXaUwf2/usRwmY2khV7rKs/mqbREoxQ3xKkvjVIXXweiq9rX7/NF6KyzcU7ZSE9zDHHpo4QMxoUbocZd6YFt8NCksNMst1Mkt1MskOfJzks+ufY+iS7CZdJYQiHiAZjYR0Mdg3yYCzIQx3bom3+2H3+llhnvpZOHftaiDQ3ETqwX/9R0NJKpLW11576PUkzGtliNqNMJr1GbjajjMbYsgllNOkj2pmM8WVlNIKp03L7fkYjymyC9mOMBpTBqM+NJpTRoN977by++9xo6mW9EQyxuVEvi8Fu63Kbof1HhxrBg8tIMAsxzDktzl4f62of9Wx3c0dNu7K1kk21m6jaW4U/cnDnmyRrEqn21K7hbe8a4Cm2FIzHWW9YpRROqwmn1UROSv+1xJKSEs444wx8gTD1LSHqYsFd39I11Btag9S2BNlb18rGthANraGDnhHvWg5wWU2xILfEAjwW6HYbSXaXHuxuM0mdtifaTCRYTBgMA+u81mMnvp6WW1rZvX07OVlZEAmjhcJokQhaOAThSGw5rG8Ld1oOhWM/GlohfkwYwh3LWiSsnyMahUineSQCkchhtx70RcVaC4yOWFg7nST729j3+hudQtzR0W8goWO/+HabTf/x0f4jpf1HiNF4RDsRSjALMYL1NeqZpmk0h5qpaqmiqrWKqrbYPDZVt1ZTVl9Gjb+GaLdOL0ZlxGP3HBTYtb5azPvMpDpSSbWnkmxNHh69nI8QpRQumxmXzUyuZ+BNvoFwhMa2EE2xoG7sPI+vD8Y/VzS20RjbHo72Hlbtoe6ymUm0m3HZTCTaTCTaYsvxdebYPiZctgQSk5NwZejbEswH/+DaVFJC+iA69Q0VTdO6BbbeaUuLRCAajc+7h3p8eziC1tZKpP2WQkuL/sRAS8+Tob6etobG+GfNfxi9xttr7j0Fd3vLwkHLXfftjQSzEMcppRSJlkQSLYnkufN63S8SjVDrr6W6tZrK1sr4vKq1iuq2avY072FV5Sqagk0APPXuU/FjTQYTXruXVLse1KmO1I7Pjo51bqv7uKuB98VqMpLmMpLmGtxtg/bm9sZOwd0e2M3+MM3+EE3+ME3+EE1t+ud9DX62+JtpagvRHAj3Wwm1mAwHhXlbk5/Xq9fhtJr08I61LMQn28HLVpPhsH+UKaX0sDqsswxcSUkJUzv9ANHCYb3VwOeLh3VHyLei+dv0HwvtrQWRSKflsP5DIaz37O6+Lt5aEOm6HD++j1sNEsxCnOCMBmO8RjyZg4cubdcWbuP1pa8zfup4qtuqqWmrobq1muq2aqpb9QBfXbX6oBeHgF4DCs1TCgAAElVJREFUT7Gl4LV7SXOk6eEdC+7O69w2N1bjUeh0M0x1bm4fyH3z7qJRjZZgmOZYeDf7w3pgd/vc1O1zVUuUfdtq8AXC+AYQ7qC/7aw9wLuEuc2E06LPE6z/v727jZHrqu84/v3P4673wV4bk9S0wUCDWxFhSIKKiAxGBCqQUELdN5AGUNW6IopUiTYKpQ1EAbUiKm8KiOAqDSGkaR3i8NAoFVTUaoLaikBxsFFilAcKDeCYXa9ndnZmdnb/fXHvjGdn78zetde7Z2Z+H+lq7pw5Mz4nZya/PffeOZNjohjdjhWzUb1CR911CvnzYblctMzt5EVYRS6Nf7w/sVjBLCKpjOZG2ZnfyZWXXNmzXn2xHoX2/IucrkS3zfB+cT6ajR8/fZzp6nRrEZbOf2drcSvbittat9uK25gsTEb7I9tWPDZRmCBj/Xuxz3rJZM4det9F+mA/2vY1InenUl9krtagVGtQrjZa+3NxcJeq5/bL1UYr0M9U6vx0ptJ6zly9+7n2dtnMuT9IxorZZTP2lfvZKPQLUdloIcuWQo4thWy8n2Ukl019Lj5ECmYRWVeFbIFd47vYNd77ZyYbSw2mq9PnZt3zLzJbm+VM9Qxnamei/doZnp57mtnaLLP12RXnwpsylmkF92RxshXY7eGdtD9IV6ivFzOLZ7g5XnqBr7UYz+DnWqEeB34ruM+Fe/vjzfD/xWz13B8AtQY9Tr+vMJqPgno0H4X1lkKWWmWeLz3/3dZjrTDPnwv2Zv32wB9pe42RfPaiz/AVzCKyKXKZXOsQehpLvkSpXmoFdnt4N/eb95sXtp2pnVm22lqnbrPzrcWtnD57mtIzpVbZtuI2to5sZSI/MdAXvK2nbMaYHMkzOXLhq6S5O9WFJcptM/dyrcF8fZFKfZFKvcH8QnN/kepCVFapL7bq/HwOTpWqrbJm/XpjbSt+ZQy2FHKtwF4W9Cv2c13rdKNgFpG+kLFM62c8L2PlIi3d1BfrXYN8pjqzYnbevO84Rx4/suL1spZttaMZ4lPFKbaNbGN7cTvbR7czVZxi+8h2pkammBqZ6qvV2kJlZtGMtpBl58T5XYcQHbLft6K8sbjE/MJiW8gvMr/QWBbg863yzv3GsvLZ+YUVdeuLawt+BbOIDLRCtrCmmTlEs/NHv/0oV7zhCmaqMz1n6S+UX+BHp3/EdG2axlLyL3ON5kZXhHVrvzjFjtEdTBXPlY/mAvmxjSGRy2aYyGaYWIeZfZL24G+f1V/9qS7tuSitEBHpYxnLMJYda61Xnoa7U14oM1OdYbo6zUx1hplatN+6X53h9Pxpfnzmx8xUZ6gtJq+XXMwWW6G9fXR76+K31lZM3legh2mtwa9gFhFZB2bGRGGCicJE4nrondydSqOyLLSnq9PM1M7tNx97fvZ5ztbPUq6XE69kb8pn8kwUJlaGd48w31rcSmWpwsLSAvlMH/1i1gBLFcxmth24G3gHcBr4C3fv+rMrZlYAjgET7v7r69FQEZFBYmaM5ccYy4/xGxPpfkd7cWmR8kKZs/Wz0VY723N/ujrdCvVSvdQz1G+971bymTxb8lvYktvCaG40us1Ht+37o7lRtuRX1mmWL6uTHSWXyWkmvwZpZ8yfA+rAJcDrgEfM7Ji7n+hS/xbgRdL8PIyIiKSSzZy78GytlnwpCvWOAC/VSzz51JPsevkuKo0K8415KguVZfunKqeWlVcala5fXUuSsQzFbLG1jeRGlt0v5ooUM9HtSHZkWXnS/UK2EJXnioznx5koTDBeGGc8Pz4Q32dfNZjNbAw4AFzh7mXgcTP7OnAj8JGE+q8A/gD4MPD369tcERE5H83vek8WVq5yteOFHezfuz/1a7k79aU6lYXkIG/fn2/MU1usLd8aNaqLVeqLdaqLVWars9SWlpfXFmtUG9Wes/xORnQUohnUE/mJ1umF8fw40zPTPHf8ueixwkTr8fHCOJOFScbz4xSzxU2f3aeZMb8aaLj7ybayY8BbutT/DPBRoPuXB0VEpG+ZWWsWO8XURft33J3GUoPqYnVFqDeDe25hjlK9RKleorxQbu0375+qnOKZM89QWihRqpX45ve+2fPfzGVyrZAey4+tmLm3z/ab+61ZfNv9Yq64bD/p+d2Yr7IoqpntAx5090vbyv4YuMHd93fUfQ9w0N3faWb7gS93O8dsZgeBgwA7d+686vDhwz3bMSjK5TLj4+Ob3YwNMSx9HZZ+gvo6qIalr6VSicJYgfml+XObR7fVpSqVpQrVpWrrsZrXqHudBV+ItqWFc/vxtki6ZUeTHP/g8e+5+9Wd5WlmzGWg89jHJFBqL4gPed8JvCtNg9z9EHAIYM+ePb5/E35ybDO0r0k76Ialr8PST1BfB9Ww9PVi9LOx1Ggdku+czTdn+c39zjo3c3Pia6YJ5pNAzswud/cfx2V7gc4Lvy4HdgOPxcfnC8BWM/sF8EZ3f37tXRYREQlXLpMjl8mxJZ/+97qbzjuY3X3OzI4Ad5jZHxFdlX0d8KaOqseB9mv+3wR8FriS6AptERERWUXa68pvAkaBU8ADwIfc/YSZ7TOzMoC7N9z9F80NmAaW4vvnfxBeRERkiKT6HrO7TwPXJ5Q/BiReMeDuRwEtLiIiIrIG/f9NbBERkQGiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREApIqmM1su5k9bGZzZvYTM3tfl3q3mNlxMyuZ2XNmdsv6NldERGSw5VLW+xxQBy4BXgc8YmbH3P1ERz0D3g88CbwK+KaZ/dTd/2m9GiwiIjLIVp0xm9kYcAC4zd3L7v448HXgxs667n6nu3/f3Rvu/jTwNeCa9W60iIjIoDJ3713B7PXAd9x9S1vZnwNvcfd393ieAd8HvuDudyU8fhA4CLBz586rDh8+fH496DPlcpnx8fHNbsaGGJa+Dks/QX0dVMPS19D6+da3vvV77n51Z3maQ9njwNmOsllgYpXn3U40I78n6UF3PwQcAtizZ4/v378/RVP639GjR1FfB8uw9BPU10E1LH3tl36mCeYyMNlRNgmUuj3BzG4mOte8z91r5988ERGR4ZLmquyTQM7MLm8r2wt0XvgFgJn9IfAR4G3u/rMLb6KIiMjwWDWY3X0OOALcYWZjZnYNcB1wX2ddM7sB+Gvg7e7+7Ho3VkREZNClXWDkJmAUOAU8AHzI3U+Y2T4zK7fV+ySwA/iumZXjbcWFXyIiIpIs1feY3X0auD6h/DGii8Oa91+xfk0TEREZPlqSU0REJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIApmERGRgCiYRUREAqJgFhERCYiCWUREJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJSKpgNrPtZvawmc2Z2U/M7H1d6pmZfcrMfhVvnzIzW98mi4iIDK5cynqfA+rAJcDrgEfM7Ji7n+iodxC4HtgLOPAt4DngrvVproiIyGBbdcZsZmPAAeA2dy+7++PA14EbE6p/APi0u//M3f8P+DTwwXVsr4iIyEBLcyj71UDD3U+2lR0DXpNQ9zXxY6vVExERkQRpDmWPA2c7ymaBiS51ZzvqjZuZubu3VzSzg0SHvgFqZnY8XZP73kuA05vdiA0yLH0dln6C+jqohqWvofXz5UmFaYK5DEx2lE0CpRR1J4FyZygDuPsh4BCAmT3h7lenaEvfU18Hz7D0E9TXQTUsfe2XfqY5lH0SyJnZ5W1le4HOC7+Iy/amqCciIiIJVg1md58DjgB3mNmYmV0DXAfcl1D9S8CHzexlZrYL+DPgi+vYXhERkYGWdoGRm4BR4BTwAPAhdz9hZvvMrNxW7wvAN4AfAseBR+Ky1RxK3+S+p74OnmHpJ6ivg2pY+toX/bSE078iIiKySbQkp4iISEAUzCIiIgHZsGAehvW2zaxoZnfH/SuZ2Q/M7J1d6n7QzBbNrNy27d/gJl8QMztqZtW29j/dpV7fjilAxxiV43H7TJe6fTWuZnazmT1hZjUz+2LHY28zs6fMrGJm/25mid+5jOvujutU4udce9Ebv0bd+mpmbzSzb5nZtJm9aGYPmtmv9XidVO/7zdSjr7vNzDven7f1eJ2gx7VHP2/o6GMl7vdVXV4nqDHdyBlz+3rbNwCfN7OkVcHa19t+LfBu4E82qpEXKAf8FHgLsBX4K+Cwme3uUv8/3X28bTu6Ia1cXze3tX9Plzr9PKa0jxFwKTAPPNjjKf00ri8AnwT+ob3QzF5C9G2M24DtwBPAP/d4nQeA/wF2AH8JfMXMdl6MBl+AxL4CU0QXBe0mWvChBNyzymuled9vpm59bdrW1v5P9Hid0Mc1sZ/ufn/H5/Ym4Fng+z1eK5gx3ZBgtiFZb9vd59z9dnd/3t2X3P1fiH7EI/GvtCHSt2Oa4ADRtxMe2+yGrAd3P+LuXwV+1fHQ7wEn3P1Bd68CtwN7zey3Ol/DzF4NXAl83N3n3f0hom9mHLi4rV+bbn1190fjfp519wrwWeCaTWnkOukxrqn1w7iuoZ8fAL6UtNhViDZqxjyU622b2SVEfe+2yMrrzey0mZ00s9vMLO2vfYXkb+I+fKfHIduBGVPSfcAHYVyXjVm8nsEzdP/MPuvu7asB9vMYv5nVF0ZK874P2U/M7Gdmdk98dCTJQIxrfArmzUTrbPQSzJhuVDCvy3rbF6ltF4WZ5YH7gXvd/amEKv8BXAG8lOgv0PcCt2xcC9fFrcArgZcRHQr8hpm9KqHeoIzpy4lOU9zbo9ogjCusHDNI/5ntVTdoZvZa4GP0HrO07/sQnQbeQHTI/iqiMbq/S91BGdf3A4+5+3M96gQ1phsVzBdlve1QmVmGaGW0OnBzUh13f9bdn4sPef8QuAP4/Q1s5gVz9/9295K719z9XuA7wLsSqvb9mMZuBB7v9QEfhHGNXchntlfdYJnZbwKPAn/q7l1PVazhfR+c+FTiE+7ecPdfEv3/6R1mlhS2AzGuRMHc64/p4MZ0o4J5aNbbjmeBdxNd5HbA3RdSPtWBvppBJujWh74e0zarfsAT9Ou4Lhuz+DqRV9H9M/vKjv+599UYx0dD/g34hLsnLTfcS7+OMURth+QsGIRxvQbYBXxljU/d1DHdkGAesvW2Pw/8NvBud5/vVsnM3hmfgya+oOY24Gsb08QLZ2bbzOx3zWzEzHJmdgPReZx/Taje72OKmb2J6DBXr6ux+25c47EbAbJAtjmewMPAFWZ2IH78Y8CTSadl4mtHfgB8PH7+e4iuvn9o43qyum59NbOXAd8GPuvud63yGmt532+aHn39HTPbY2YZM9sB/B1w1N07D1n3xbj2eP82fQB4qOM8eedrhDem7r4hG9FXLr4KzAH/C7wvLt9HdFizWc+AO4HpeLuTeOnQ0Dei8zYOVIkOAzW3G4DL4v3L4rp/C/wy/u/xLNEhz/xm92ENfd0JfJfosNYZ4L+Atw/amLb14QvAfQnlfT2uRFdbe8d2e/zYtcBTRF8POwrsbnveXcBdbfd3x3XmgaeBaze7b2n7Cnw83m//zLa/fz8KPBrvd33fh7T16Ot7ib4pMgf8nOiP5kv7dVxXef+OxGP0toTnBT2mWitbREQkIFqSU0REJCAKZhERkYAomEVERAKiYBYREQmIgllERCQgCmYREZGAKJhFREQComAWEREJiIJZREQkIP8PXncA5EZRxX0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.DataFrame(history.history).plot(figsize=(8, 5))\n",
    "plt.grid(True)\n",
    "plt.gca().set_ylim(0, 1)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Complex models using the functional API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, although Sequential models are extremely common, it is sometimes useful to **build NNs with more complex topologies, or with multiple inputs or outputs**. For this purpose, Keras offers the Functional API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Wide & Deep neural network</font>** ([2016 paper](https://homl.info/widedeep)) connects all or part of the inputs directly to the output layer. This architecture makes it possible for the neural network to **learn both deep patterns (using the deep path) and simple rules (through the short path)**. (**<font color='blue'>MLP的一个缺点</font>**) While a **regular MLP forces all the data to flow through the full stack of layers**; thus, simple patterns in the data may end up being distorted by this sequence of transformations.\n",
    "\n",
    "![wide-and-deep-neural-network](./figs/chap10-figs/wide-and-deep-neural-network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'>one input and one output, connect all the inputs directly to the output layer (all input features as the \"wide\")</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:30.392486Z",
     "start_time": "2020-01-03T11:07:30.343774Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_1 (InputLayer)            [(None, 8)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 30)           270         input_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 30)           930         dense[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 38)           0           input_1[0][0]                    \n",
      "                                                                 dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 1)            39          concatenate[0][0]                \n",
      "==================================================================================================\n",
      "Total params: 1,239\n",
      "Trainable params: 1,239\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "input_ = keras.layers.Input(shape=x_train.shape[1:])  # shape: (8,)\n",
    "hidden1 = keras.layers.Dense(30, activation='relu')(input_)  # pass it the input\n",
    "# pass it the output of the 1st hidden layer\n",
    "hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)\n",
    "# concat = keras.layers.concatenate([input_, hidden2])\n",
    "concat = keras.layers.Concatenate()([input_, hidden2])\n",
    "# output layer with a single unit and no activation function\n",
    "output = keras.layers.Dense(1)(concat)\n",
    "model = keras.Model(inputs=[input_], outputs=[output])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **`Input` layer** is a specification of the kind of input the model will get, including its `shape` and `dtype`. A model may actually have multiple inputs.\n",
    "\n",
    "\n",
    "+ **The `Dense` layers were called like a function, passing it the input.** **<font color='blue'>This is why this is called the Functional API.</font>** Note that we are **just telling Keras how it should connect the layers together; no actual data is being processed yet.**\n",
    "\n",
    "\n",
    "+ You may prefer the **`keras.layers.concatenate()` function**, which creates a `Concatenate` layer and immediately calls it with the given inputs.\n",
    "\n",
    "\n",
    "+ Create a Keras Model, specifying which inputs and outputs to use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当构建 Keras 模型后，其他部分与 Sequential 模型一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:42.233758Z",
     "start_time": "2020-01-03T11:07:30.394853Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/20\n",
      "11610/11610 [==============================] - 1s 94us/sample - loss: 1.2474 - mae: 0.7958 - val_loss: 0.8727 - val_mae: 0.5975\n",
      "Epoch 2/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.6341 - mae: 0.5799 - val_loss: 0.8881 - val_mae: 0.5628\n",
      "Epoch 3/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5885 - mae: 0.5552 - val_loss: 0.5573 - val_mae: 0.5347\n",
      "Epoch 4/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5599 - mae: 0.5411 - val_loss: 0.5238 - val_mae: 0.5185\n",
      "Epoch 5/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.5367 - mae: 0.5288 - val_loss: 0.6201 - val_mae: 0.5152\n",
      "Epoch 6/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5170 - mae: 0.5173 - val_loss: 0.4979 - val_mae: 0.5012\n",
      "Epoch 7/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5022 - mae: 0.5120 - val_loss: 0.5240 - val_mae: 0.4968\n",
      "Epoch 8/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.4885 - mae: 0.5031 - val_loss: 0.4538 - val_mae: 0.4851\n",
      "Epoch 9/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.4778 - mae: 0.4984 - val_loss: 0.4906 - val_mae: 0.4818\n",
      "Epoch 10/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.4673 - mae: 0.4921 - val_loss: 0.4332 - val_mae: 0.4730\n",
      "Epoch 11/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.4585 - mae: 0.4876 - val_loss: 0.4430 - val_mae: 0.4723\n",
      "Epoch 12/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.4508 - mae: 0.4832 - val_loss: 0.4182 - val_mae: 0.4649\n",
      "Epoch 13/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.4437 - mae: 0.4793 - val_loss: 0.4282 - val_mae: 0.4643\n",
      "Epoch 14/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4375 - mae: 0.4764 - val_loss: 0.4053 - val_mae: 0.4571\n",
      "Epoch 15/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4316 - mae: 0.4728 - val_loss: 0.4192 - val_mae: 0.4576\n",
      "Epoch 16/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4263 - mae: 0.4696 - val_loss: 0.3946 - val_mae: 0.4503\n",
      "Epoch 17/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4216 - mae: 0.4664 - val_loss: 0.4018 - val_mae: 0.4503\n",
      "Epoch 18/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4168 - mae: 0.4635 - val_loss: 0.3910 - val_mae: 0.4493\n",
      "Epoch 19/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.4131 - mae: 0.4619 - val_loss: 0.3829 - val_mae: 0.4428\n",
      "Epoch 20/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.4093 - mae: 0.4595 - val_loss: 0.3807 - val_mae: 0.4409\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='mean_squared_error',\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=['mae'])\n",
    "\n",
    "history = model.fit(x_train, y_train, epochs=20, batch_size=32,\n",
    "                    validation_data=(x_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:42.395160Z",
     "start_time": "2020-01-03T11:07:42.235653Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3445 - mae: 0.4568\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4045557499855988, 0.45684075]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'>multiple inputs and one output, connect part of the inputs directly to the output layer (part of the input features as the \"wide\")</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q: What if you want to **send a subset of the features through the wide path and a different subset (possibly overlapping) through the deep path**?\n",
    "\n",
    "A: In this case, one solution is to use multiple inputs.\n",
    "\n",
    "![多输入](./figs/chap10-figs/handling-multiple-inputs.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:42.440442Z",
     "start_time": "2020-01-03T11:07:42.396874Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "deep_input (InputLayer)         [(None, 8)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 30)           270         deep_input[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "wide_input (InputLayer)         [(None, 5)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 30)           930         dense[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 35)           0           wide_input[0][0]                 \n",
      "                                                                 dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "output (Dense)                  (None, 1)            36          concatenate[0][0]                \n",
      "==================================================================================================\n",
      "Total params: 1,236\n",
      "Trainable params: 1,236\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "# send 5 features through the wide path (features 0 - 4)\n",
    "# and all features through the deep path\n",
    "input_A = keras.layers.Input(shape=[5], name='wide_input')\n",
    "input_B = keras.layers.Input(shape=x_train.shape[1:], name='deep_input')\n",
    "hidden1 = keras.layers.Dense(30, activation='relu')(input_B)\n",
    "hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)\n",
    "concat = keras.layers.concatenate([input_A, hidden2])\n",
    "output = keras.layers.Dense(1, name='output')(concat)\n",
    "model = keras.Model(inputs=[input_A, input_B], outputs=[output])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>You should name at least the most important layers, especially when the model gets a bit complex like this.</font>**\n",
    "\n",
    "We must pass a pair of matrices `(x_train_A, x_train_B)`: one per input when calling the `fit()` (or `evaluate()` or `predict()`) method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:42.446504Z",
     "start_time": "2020-01-03T11:07:42.442902Z"
    }
   },
   "outputs": [],
   "source": [
    "x_train_A, x_train_B = x_train[:, :5], x_train[:, :]\n",
    "x_valid_A, x_valid_B = x_valid[:, :5], x_valid[:, :]\n",
    "x_test_A, x_test_B = x_test[:, :5], x_test[:, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:52.945985Z",
     "start_time": "2020-01-03T11:07:42.448890Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/20\n",
      "11610/11610 [==============================] - 1s 80us/sample - loss: 1.9107 - mae: 1.0213 - val_loss: 5.3144 - val_mae: 0.6943\n",
      "Epoch 2/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.7137 - mae: 0.6210 - val_loss: 0.7097 - val_mae: 0.5887\n",
      "Epoch 3/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.6270 - mae: 0.5834 - val_loss: 0.5938 - val_mae: 0.5645\n",
      "Epoch 4/20\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.5898 - mae: 0.5642 - val_loss: 0.5712 - val_mae: 0.5453\n",
      "Epoch 5/20\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.5604 - mae: 0.5471 - val_loss: 0.5308 - val_mae: 0.5285\n",
      "Epoch 6/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.5341 - mae: 0.5317 - val_loss: 0.5385 - val_mae: 0.5174\n",
      "Epoch 7/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.5133 - mae: 0.5217 - val_loss: 0.4951 - val_mae: 0.5058\n",
      "Epoch 8/20\n",
      "11610/11610 [==============================] - 0s 43us/sample - loss: 0.4958 - mae: 0.5102 - val_loss: 0.4907 - val_mae: 0.4963\n",
      "Epoch 9/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4815 - mae: 0.5023 - val_loss: 0.4585 - val_mae: 0.4834\n",
      "Epoch 10/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4685 - mae: 0.4941 - val_loss: 0.4505 - val_mae: 0.4768\n",
      "Epoch 11/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4578 - mae: 0.4874 - val_loss: 0.4349 - val_mae: 0.4725\n",
      "Epoch 12/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4490 - mae: 0.4825 - val_loss: 0.4256 - val_mae: 0.4658\n",
      "Epoch 13/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4409 - mae: 0.4775 - val_loss: 0.4162 - val_mae: 0.4626\n",
      "Epoch 14/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4344 - mae: 0.4742 - val_loss: 0.4143 - val_mae: 0.4572\n",
      "Epoch 15/20\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.4282 - mae: 0.4706 - val_loss: 0.4024 - val_mae: 0.4546\n",
      "Epoch 16/20\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.4228 - mae: 0.4672 - val_loss: 0.3991 - val_mae: 0.4496\n",
      "Epoch 17/20\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.4182 - mae: 0.4642 - val_loss: 0.3929 - val_mae: 0.4483\n",
      "Epoch 18/20\n",
      "11610/11610 [==============================] - 0s 43us/sample - loss: 0.4135 - mae: 0.4615 - val_loss: 0.3889 - val_mae: 0.4490\n",
      "Epoch 19/20\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.4098 - mae: 0.4603 - val_loss: 0.3851 - val_mae: 0.4427\n",
      "Epoch 20/20\n",
      "11610/11610 [==============================] - 0s 42us/sample - loss: 0.4061 - mae: 0.4579 - val_loss: 0.3821 - val_mae: 0.4412\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='mse',\n",
    "              optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "              metrics=['mae'])\n",
    "\n",
    "history = model.fit([x_train_A, x_train_B], y_train, epochs=20, batch_size=32,\n",
    "                    validation_data=((x_valid_A, x_valid_B), y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:53.109663Z",
     "start_time": "2020-01-03T11:07:52.947534Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3207 - mae: 0.4544\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4011351616114609, 0.45436996]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate((x_test_A, x_test_B), y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'> multiple inputs and multiple outputs</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>When do we want to have multiple outputs?</font>**\n",
    "\n",
    "+ The task may demand it. (识别并定位到图片中的主要对象）\n",
    "\n",
    "\n",
    "+ **Have multiple independent tasks based on the same data**. Sure, you could train one NN per task, but **<font color='blue'>in many cases you will get better results on all tasks by training a single NN with one output per task</font>**. This is because the neural network can learn features in the data that are useful across tasks. (一个输出用来识别人的表情，一个输出用来识别是否带眼镜了。)\n",
    "\n",
    "\n",
    "+ Another use case is **<font color='blue'>as a regularization technique</font>** (i.e., **a training constraint whose objective is to reduce overfitting and thus improve the model’s ability to generalize**). For example, you may want to **<font color='blue'>add some auxiliary outputs</font>** in a neural network architecture (see Figure below) to **ensure that the underlying part of the network learns something useful on its own, without relying on the rest of the network**.\n",
    "\n",
    "\n",
    "![multiple outputs](./figs/chap10-figs/handling-multiple-outputs-adding-an-auxiliary-output.png)\n",
    "<center>handling multiple outputs, in this example to add an auxiliary output for regularization</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:07:53.168254Z",
     "start_time": "2020-01-03T11:07:53.111604Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "deep (InputLayer)               [(None, 8)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense (Dense)                   (None, 30)           270         deep[0][0]                       \n",
      "__________________________________________________________________________________________________\n",
      "wide (InputLayer)               [(None, 5)]          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 30)           930         dense[0][0]                      \n",
      "__________________________________________________________________________________________________\n",
      "concatenate (Concatenate)       (None, 35)           0           wide[0][0]                       \n",
      "                                                                 dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "main_output (Dense)             (None, 1)            36          concatenate[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "aux_output (Dense)              (None, 1)            31          dense_1[0][0]                    \n",
      "==================================================================================================\n",
      "Total params: 1,267\n",
      "Trainable params: 1,267\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "# add an auxiliary output for regularization\n",
    "input_A = keras.layers.Input(shape=[5], name='wide')\n",
    "input_B = keras.layers.Input(shape=x_train.shape[1:], name='deep')\n",
    "hidden1 = keras.layers.Dense(30, activation='relu')(input_B)\n",
    "hidden2 = keras.layers.Dense(30, activation='relu')(hidden1)\n",
    "aux_output = keras.layers.Dense(1, name='aux_output')(hidden2)  # auxiliary output\n",
    "concat = keras.layers.concatenate([input_A, hidden2])\n",
    "main_output = keras.layers.Dense(1, name='main_output')(concat)\n",
    "\n",
    "model = keras.Model(inputs=[input_A, input_B],\n",
    "                    outputs=[main_output, aux_output])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>Each output will need its own loss function.</font>** When compiling the model, we **should pass a list of losses** (**if we pass a single loss**, Keras will assume that the same loss must be used for all outputs. **By default, Keras will compute all these losses and simply add them up to get the final loss used for training**. We care much more about the main output than about the auxiliary output (as it is just used for regularization), so we want to **give the main output’s loss a much greater weight**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:04.698382Z",
     "start_time": "2020-01-03T11:07:53.171045Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/20\n",
      "11610/11610 [==============================] - 1s 118us/sample - loss: 2.0433 - main_output_loss: 1.7025 - aux_output_loss: 5.1032 - main_output_mae: 0.9097 - aux_output_mae: 1.9629 - val_loss: 6.4848 - val_main_output_loss: 6.8398 - val_aux_output_loss: 3.2592 - val_main_output_mae: 0.7015 - val_aux_output_mae: 1.4675\n",
      "Epoch 2/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.9219 - main_output_loss: 0.7600 - aux_output_loss: 2.3784 - main_output_mae: 0.6321 - aux_output_mae: 1.2029 - val_loss: 1.6743 - val_main_output_loss: 1.6316 - val_aux_output_loss: 2.0519 - val_main_output_mae: 0.6140 - val_aux_output_mae: 0.9602\n",
      "Epoch 3/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.7358 - main_output_loss: 0.6670 - aux_output_loss: 1.3535 - main_output_mae: 0.5969 - aux_output_mae: 0.8474 - val_loss: 0.9784 - val_main_output_loss: 0.9267 - val_aux_output_loss: 1.4405 - val_main_output_mae: 0.5874 - val_aux_output_mae: 0.7420\n",
      "Epoch 4/20\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.6585 - main_output_loss: 0.6263 - aux_output_loss: 0.9468 - main_output_mae: 0.5777 - aux_output_mae: 0.7062 - val_loss: 0.7489 - val_main_output_loss: 0.6974 - val_aux_output_loss: 1.2102 - val_main_output_mae: 0.5611 - val_aux_output_mae: 0.6684\n",
      "Epoch 5/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.6138 - main_output_loss: 0.5947 - aux_output_loss: 0.7850 - main_output_mae: 0.5607 - aux_output_mae: 0.6532 - val_loss: 0.6540 - val_main_output_loss: 0.5997 - val_aux_output_loss: 1.1405 - val_main_output_mae: 0.5405 - val_aux_output_mae: 0.6386\n",
      "Epoch 6/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.5812 - main_output_loss: 0.5674 - aux_output_loss: 0.7043 - main_output_mae: 0.5449 - aux_output_mae: 0.6259 - val_loss: 0.6057 - val_main_output_loss: 0.5462 - val_aux_output_loss: 1.1398 - val_main_output_mae: 0.5263 - val_aux_output_mae: 0.6226\n",
      "Epoch 7/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.5550 - main_output_loss: 0.5432 - aux_output_loss: 0.6615 - main_output_mae: 0.5339 - aux_output_mae: 0.6101 - val_loss: 0.5767 - val_main_output_loss: 0.5146 - val_aux_output_loss: 1.1342 - val_main_output_mae: 0.5158 - val_aux_output_mae: 0.6051\n",
      "Epoch 8/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.5332 - main_output_loss: 0.5231 - aux_output_loss: 0.6240 - main_output_mae: 0.5216 - aux_output_mae: 0.5922 - val_loss: 0.5547 - val_main_output_loss: 0.4882 - val_aux_output_loss: 1.1514 - val_main_output_mae: 0.5039 - val_aux_output_mae: 0.5915\n",
      "Epoch 9/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.5149 - main_output_loss: 0.5057 - aux_output_loss: 0.5984 - main_output_mae: 0.5123 - aux_output_mae: 0.5788 - val_loss: 0.5381 - val_main_output_loss: 0.4704 - val_aux_output_loss: 1.1459 - val_main_output_mae: 0.4925 - val_aux_output_mae: 0.5773\n",
      "Epoch 10/20\n",
      "11610/11610 [==============================] - 1s 43us/sample - loss: 0.4989 - main_output_loss: 0.4903 - aux_output_loss: 0.5762 - main_output_mae: 0.5036 - aux_output_mae: 0.5660 - val_loss: 0.5246 - val_main_output_loss: 0.4561 - val_aux_output_loss: 1.1393 - val_main_output_mae: 0.4848 - val_aux_output_mae: 0.5652\n",
      "Epoch 11/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4855 - main_output_loss: 0.4774 - aux_output_loss: 0.5579 - main_output_mae: 0.4961 - aux_output_mae: 0.5549 - val_loss: 0.5126 - val_main_output_loss: 0.4452 - val_aux_output_loss: 1.1181 - val_main_output_mae: 0.4803 - val_aux_output_mae: 0.5553\n",
      "Epoch 12/20\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.4745 - main_output_loss: 0.4669 - aux_output_loss: 0.5426 - main_output_mae: 0.4905 - aux_output_mae: 0.5456 - val_loss: 0.5012 - val_main_output_loss: 0.4343 - val_aux_output_loss: 1.1016 - val_main_output_mae: 0.4736 - val_aux_output_mae: 0.5453\n",
      "Epoch 13/20\n",
      "11610/11610 [==============================] - 1s 51us/sample - loss: 0.4646 - main_output_loss: 0.4575 - aux_output_loss: 0.5302 - main_output_mae: 0.4849 - aux_output_mae: 0.5367 - val_loss: 0.4905 - val_main_output_loss: 0.4262 - val_aux_output_loss: 1.0674 - val_main_output_mae: 0.4708 - val_aux_output_mae: 0.5380\n",
      "Epoch 14/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4566 - main_output_loss: 0.4497 - aux_output_loss: 0.5192 - main_output_mae: 0.4812 - aux_output_mae: 0.5303 - val_loss: 0.4806 - val_main_output_loss: 0.4181 - val_aux_output_loss: 1.0419 - val_main_output_mae: 0.4640 - val_aux_output_mae: 0.5286\n",
      "Epoch 15/20\n",
      "11610/11610 [==============================] - 0s 43us/sample - loss: 0.4493 - main_output_loss: 0.4426 - aux_output_loss: 0.5094 - main_output_mae: 0.4767 - aux_output_mae: 0.5230 - val_loss: 0.4720 - val_main_output_loss: 0.4128 - val_aux_output_loss: 1.0031 - val_main_output_mae: 0.4621 - val_aux_output_mae: 0.5224\n",
      "Epoch 16/20\n",
      "11610/11610 [==============================] - 0s 42us/sample - loss: 0.4432 - main_output_loss: 0.4368 - aux_output_loss: 0.5010 - main_output_mae: 0.4736 - aux_output_mae: 0.5173 - val_loss: 0.4634 - val_main_output_loss: 0.4070 - val_aux_output_loss: 0.9696 - val_main_output_mae: 0.4566 - val_aux_output_mae: 0.5146\n",
      "Epoch 17/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4381 - main_output_loss: 0.4318 - aux_output_loss: 0.4937 - main_output_mae: 0.4702 - aux_output_mae: 0.5120 - val_loss: 0.4573 - val_main_output_loss: 0.4046 - val_aux_output_loss: 0.9295 - val_main_output_mae: 0.4567 - val_aux_output_mae: 0.5100\n",
      "Epoch 18/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4330 - main_output_loss: 0.4270 - aux_output_loss: 0.4869 - main_output_mae: 0.4676 - aux_output_mae: 0.5069 - val_loss: 0.4521 - val_main_output_loss: 0.4020 - val_aux_output_loss: 0.9015 - val_main_output_mae: 0.4561 - val_aux_output_mae: 0.5058\n",
      "Epoch 19/20\n",
      "11610/11610 [==============================] - 0s 42us/sample - loss: 0.4290 - main_output_loss: 0.4232 - aux_output_loss: 0.4813 - main_output_mae: 0.4658 - aux_output_mae: 0.5033 - val_loss: 0.4462 - val_main_output_loss: 0.3989 - val_aux_output_loss: 0.8704 - val_main_output_mae: 0.4504 - val_aux_output_mae: 0.4993\n",
      "Epoch 20/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4249 - main_output_loss: 0.4192 - aux_output_loss: 0.4762 - main_output_mae: 0.4634 - aux_output_mae: 0.4994 - val_loss: 0.4422 - val_main_output_loss: 0.3978 - val_aux_output_loss: 0.8405 - val_main_output_mae: 0.4485 - val_aux_output_mae: 0.4943\n"
     ]
    }
   ],
   "source": [
    "model.compile(\n",
    "    loss=['mse', 'mse'],  # each output need its own loss function\n",
    "    loss_weights=[0.9, 0.1],  # give the main output's loss a much greater weight\n",
    "    optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "    metrics=['mae']\n",
    ")\n",
    "\n",
    "history = model.fit(\n",
    "    [x_train_A, x_train_B], [y_train, y_train], epochs=20, batch_size=32,\n",
    "    validation_data=([x_valid_A, x_valid_B], [y_valid, y_valid])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:04.860974Z",
     "start_time": "2020-01-03T11:08:04.699897Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3391 - main_output_loss: 0.4158 - aux_output_loss: 0.4657 - main_output_mae: 0.4600 - aux_output_mae: 0.4941\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4215106489122376, 0.41575122, 0.46571365, 0.46003705, 0.49408868]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# When evaluate the model,\n",
    "# Keras will return the total loss, as well as all the individual losses.\n",
    "model.evaluate((x_test_A, x_test_B), (y_test, y_test), verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.033054Z",
     "start_time": "2020-01-03T11:08:04.862672Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3391 - main_output_loss: 0.4158 - aux_output_loss: 0.4657 - main_output_mae: 0.4600 - aux_output_mae: 0.4941\n"
     ]
    }
   ],
   "source": [
    "total_loss, main_loss, aux_loss, main_mae, aux_mae = model.evaluate(\n",
    "    (x_test_A, x_test_B), (y_test, y_test), verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.037404Z",
     "start_time": "2020-01-03T11:08:05.034757Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4215106489122376 0.41575122 0.46571365 0.46003705 0.49408868\n"
     ]
    }
   ],
   "source": [
    "print(total_loss, main_loss, aux_loss, main_mae, aux_mae)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.057108Z",
     "start_time": "2020-01-03T11:08:05.053717Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4207474619150162"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_loss*0.9 + aux_loss*0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.252277Z",
     "start_time": "2020-01-03T11:08:05.066191Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[0.42462975],\n",
       "        [1.6918638 ],\n",
       "        [3.365356  ],\n",
       "        ...,\n",
       "        [1.2852559 ],\n",
       "        [2.5727158 ],\n",
       "        [3.813661  ]], dtype=float32), array([[0.60257137],\n",
       "        [1.6257426 ],\n",
       "        [3.073895  ],\n",
       "        ...,\n",
       "        [1.5508424 ],\n",
       "        [2.1593542 ],\n",
       "        [3.7511044 ]], dtype=float32)]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# return predictions for each output:\n",
    "y_preds = model.predict([x_test_A, x_test_B])\n",
    "y_preds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.257917Z",
     "start_time": "2020-01-03T11:08:05.254139Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.477  , 0.458  , 5.00001, ..., 1.177  , 2.631  , 4.815  ])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.263886Z",
     "start_time": "2020-01-03T11:08:05.259620Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.42462975, 1.6918638 , 3.365356  , ..., 1.2852559 , 2.5727158 ,\n",
       "       3.813661  ], dtype=float32)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_preds[0].flatten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Build dynamic models using the subclassing API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color='blue'>Both the Sequential API and the Functional API are declarative</font> (陈述的、说明的): you <font color='blue'>start by declaring which layers you want to use and how they should be connected, and only then can you start feeding the model some data for training or inference</font>.\n",
    "\n",
    "**<font color='Crimson'>Sequential API 和 Functional API 的优点：</font>**\n",
    "\n",
    "+ the model can easily be saved, cloned, and shared\n",
    "\n",
    "\n",
    "+ model's structure can be displayed and analyzed\n",
    "\n",
    "\n",
    "+ the framework can infer shapes and check types, so errors can be caught early (i.e., before any data ever goes through the model)\n",
    "\n",
    "\n",
    "+ also fairly easy to debug, since the whole model is a static graph of layers\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='Crimson'>Sequential API 和 Functional API 的缺点：</font>** **It is static.** Some models involve loops, varying shapes, conditional branching, and other dynamic behaviors. For such cases, or simply if you prefer a more imperative programming style, the Subclassing API is for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Simply subclass the `Model` class, create the layers you need in the constructor, and then use them to perform the computations you want in the `call()` method.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.273609Z",
     "start_time": "2020-01-03T11:08:05.265508Z"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.281922Z",
     "start_time": "2020-01-03T11:08:05.275325Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create a `WideAndDeepModel` class (== the functional model above)\n",
    "class WideAndDeepModel(keras.Model):\n",
    "    \"\"\"Creating a class gives us an equivalent model to the one we just built\n",
    "    with the Functional API. We can then compile it, evaluate it, and use it\n",
    "    to make predictions, exactly like we just did.\n",
    "    \"\"\"\n",
    "    def __init__(self, units=30, activation='relu', **kwargs):\n",
    "        super().__init__(**kwargs)  # handles standard args (e.g., name)\n",
    "        self.hidden1 = keras.layers.Dense(units, activation=activation)\n",
    "        self.hidden2 = keras.layers.Dense(units, activation=activation)\n",
    "        self.main_output = keras.layers.Dense(1)\n",
    "        self.aux_output = keras.layers.Dense(1)\n",
    "\n",
    "    # separate the creation of the layers in the constructor \n",
    "    # from their usage in the `call()` method\n",
    "    def call(self, inputs):\n",
    "        # do not need to create inputs,\n",
    "        # just use the `input` arg to the `call()` method\n",
    "        input_A, input_B = inputs\n",
    "        # you can do pretty much anything you want in the `call()` method\n",
    "        hidden1 = self.hidden1(input_B)\n",
    "        hidden2 = self.hidden2(hidden1)\n",
    "        aux_output = self.aux_output(hidden2)\n",
    "        concat = keras.layers.concatenate([input_A, hidden2])\n",
    "        main_output = self.main_output(concat)\n",
    "        return main_output, aux_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Do not need to create the inputs; we just use the `input` argument to the `call()` method.\n",
    "\n",
    "\n",
    "+ Separate the creation of the layers in the constructor from their usage in the `call()` method.\n",
    "\n",
    "\n",
    "+ **You can do pretty much anything you want in the `call()` method: `for` loops, `if` statements, low-level TensorFlow operations** (the big difference). This makes it a great API for researchers experimenting with new ideas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>subclassing API 的缺点：</font>**\n",
    "\n",
    "This extra flexibility does come at a cost:\n",
    "\n",
    "+ model’s architecture is hidden within the `call()` method, so Keras **cannot easily inspect it**\n",
    "\n",
    "\n",
    "+ **cannot save or clone it**\n",
    "\n",
    "\n",
    "+ when you call the **`summary()`** method, you **only get a list of layers**, without any information on how they are connected to each other\n",
    "\n",
    "\n",
    "+ Keras **cannot check types and shapes ahead of time**, and it is easier to make mistakes.\n",
    "\n",
    "**<font color='blue'>So unless you really need that extra flexibility, you should probably stick to the Sequential API or the Functional API.</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.290594Z",
     "start_time": "2020-01-03T11:08:05.283781Z"
    }
   },
   "outputs": [],
   "source": [
    "model = WideAndDeepModel(30, activation='relu')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.296346Z",
     "start_time": "2020-01-03T11:08:05.292909Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    model.summary()\n",
    "except ValueError as e:\n",
    "    print(\"ValueError:\", e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.302139Z",
     "start_time": "2020-01-03T11:08:05.298475Z"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    # If we pass a single loss,\n",
    "    # Keras will assume that the same loss must be used for all outputs\n",
    "    loss='mse',\n",
    "    # give the main output’s loss a much greater weight\n",
    "    loss_weights=[0.9, 0.1],  # list or dictionary\n",
    "    optimizer=keras.optimizers.SGD(lr=1e-3),\n",
    "    metrics=['mae'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:05.307410Z",
     "start_time": "2020-01-03T11:08:05.304215Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ValueError: This model has not yet been built. Build the model first by calling `build()` or calling `fit()` with some data, or specify an `input_shape` argument in the first layer(s) for automatic build.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    model.summary()\n",
    "except ValueError as e:\n",
    "    print(\"ValueError:\", e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:17.085677Z",
     "start_time": "2020-01-03T11:08:05.309278Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/20\n",
      "11610/11610 [==============================] - 1s 110us/sample - loss: 2.0433 - output_1_loss: 1.7025 - output_2_loss: 5.1032 - output_1_mae: 0.9097 - output_2_mae: 1.9629 - val_loss: 6.4848 - val_output_1_loss: 6.8398 - val_output_2_loss: 3.2592 - val_output_1_mae: 0.7015 - val_output_2_mae: 1.4675\n",
      "Epoch 2/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.9219 - output_1_loss: 0.7600 - output_2_loss: 2.3784 - output_1_mae: 0.6321 - output_2_mae: 1.2029 - val_loss: 1.6743 - val_output_1_loss: 1.6316 - val_output_2_loss: 2.0519 - val_output_1_mae: 0.6140 - val_output_2_mae: 0.9602\n",
      "Epoch 3/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.7358 - output_1_loss: 0.6670 - output_2_loss: 1.3535 - output_1_mae: 0.5969 - output_2_mae: 0.8474 - val_loss: 0.9784 - val_output_1_loss: 0.9267 - val_output_2_loss: 1.4405 - val_output_1_mae: 0.5874 - val_output_2_mae: 0.7420\n",
      "Epoch 4/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.6585 - output_1_loss: 0.6263 - output_2_loss: 0.9468 - output_1_mae: 0.5777 - output_2_mae: 0.7062 - val_loss: 0.7489 - val_output_1_loss: 0.6974 - val_output_2_loss: 1.2102 - val_output_1_mae: 0.5611 - val_output_2_mae: 0.6684\n",
      "Epoch 5/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.6138 - output_1_loss: 0.5947 - output_2_loss: 0.7850 - output_1_mae: 0.5607 - output_2_mae: 0.6532 - val_loss: 0.6540 - val_output_1_loss: 0.5997 - val_output_2_loss: 1.1405 - val_output_1_mae: 0.5405 - val_output_2_mae: 0.6386\n",
      "Epoch 6/20\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.5812 - output_1_loss: 0.5674 - output_2_loss: 0.7043 - output_1_mae: 0.5449 - output_2_mae: 0.6259 - val_loss: 0.6057 - val_output_1_loss: 0.5462 - val_output_2_loss: 1.1398 - val_output_1_mae: 0.5263 - val_output_2_mae: 0.6226\n",
      "Epoch 7/20\n",
      "11610/11610 [==============================] - 1s 50us/sample - loss: 0.5550 - output_1_loss: 0.5432 - output_2_loss: 0.6615 - output_1_mae: 0.5339 - output_2_mae: 0.6101 - val_loss: 0.5767 - val_output_1_loss: 0.5146 - val_output_2_loss: 1.1342 - val_output_1_mae: 0.5158 - val_output_2_mae: 0.6051\n",
      "Epoch 8/20\n",
      "11610/11610 [==============================] - 1s 47us/sample - loss: 0.5332 - output_1_loss: 0.5231 - output_2_loss: 0.6240 - output_1_mae: 0.5216 - output_2_mae: 0.5922 - val_loss: 0.5547 - val_output_1_loss: 0.4882 - val_output_2_loss: 1.1514 - val_output_1_mae: 0.5039 - val_output_2_mae: 0.5915\n",
      "Epoch 9/20\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 0.5149 - output_1_loss: 0.5057 - output_2_loss: 0.5984 - output_1_mae: 0.5123 - output_2_mae: 0.5788 - val_loss: 0.5381 - val_output_1_loss: 0.4704 - val_output_2_loss: 1.1459 - val_output_1_mae: 0.4925 - val_output_2_mae: 0.5773\n",
      "Epoch 10/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.4989 - output_1_loss: 0.4903 - output_2_loss: 0.5762 - output_1_mae: 0.5036 - output_2_mae: 0.5660 - val_loss: 0.5246 - val_output_1_loss: 0.4561 - val_output_2_loss: 1.1393 - val_output_1_mae: 0.4848 - val_output_2_mae: 0.5652\n",
      "Epoch 11/20\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.4855 - output_1_loss: 0.4774 - output_2_loss: 0.5579 - output_1_mae: 0.4961 - output_2_mae: 0.5549 - val_loss: 0.5126 - val_output_1_loss: 0.4452 - val_output_2_loss: 1.1181 - val_output_1_mae: 0.4803 - val_output_2_mae: 0.5553\n",
      "Epoch 12/20\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4745 - output_1_loss: 0.4669 - output_2_loss: 0.5426 - output_1_mae: 0.4905 - output_2_mae: 0.5456 - val_loss: 0.5012 - val_output_1_loss: 0.4343 - val_output_2_loss: 1.1016 - val_output_1_mae: 0.4736 - val_output_2_mae: 0.5453\n",
      "Epoch 13/20\n",
      "11610/11610 [==============================] - 1s 48us/sample - loss: 0.4646 - output_1_loss: 0.4575 - output_2_loss: 0.5302 - output_1_mae: 0.4849 - output_2_mae: 0.5367 - val_loss: 0.4905 - val_output_1_loss: 0.4262 - val_output_2_loss: 1.0674 - val_output_1_mae: 0.4708 - val_output_2_mae: 0.5380\n",
      "Epoch 14/20\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 0.4566 - output_1_loss: 0.4497 - output_2_loss: 0.5192 - output_1_mae: 0.4812 - output_2_mae: 0.5303 - val_loss: 0.4806 - val_output_1_loss: 0.4181 - val_output_2_loss: 1.0419 - val_output_1_mae: 0.4640 - val_output_2_mae: 0.5286\n",
      "Epoch 15/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4493 - output_1_loss: 0.4426 - output_2_loss: 0.5094 - output_1_mae: 0.4767 - output_2_mae: 0.5230 - val_loss: 0.4720 - val_output_1_loss: 0.4128 - val_output_2_loss: 1.0031 - val_output_1_mae: 0.4621 - val_output_2_mae: 0.5224\n",
      "Epoch 16/20\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.4432 - output_1_loss: 0.4368 - output_2_loss: 0.5010 - output_1_mae: 0.4736 - output_2_mae: 0.5173 - val_loss: 0.4634 - val_output_1_loss: 0.4070 - val_output_2_loss: 0.9696 - val_output_1_mae: 0.4566 - val_output_2_mae: 0.5146\n",
      "Epoch 17/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4381 - output_1_loss: 0.4318 - output_2_loss: 0.4937 - output_1_mae: 0.4702 - output_2_mae: 0.5120 - val_loss: 0.4573 - val_output_1_loss: 0.4046 - val_output_2_loss: 0.9295 - val_output_1_mae: 0.4567 - val_output_2_mae: 0.5100\n",
      "Epoch 18/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4330 - output_1_loss: 0.4270 - output_2_loss: 0.4869 - output_1_mae: 0.4676 - output_2_mae: 0.5069 - val_loss: 0.4521 - val_output_1_loss: 0.4020 - val_output_2_loss: 0.9015 - val_output_1_mae: 0.4561 - val_output_2_mae: 0.5058\n",
      "Epoch 19/20\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.4290 - output_1_loss: 0.4232 - output_2_loss: 0.4813 - output_1_mae: 0.4658 - output_2_mae: 0.5033 - val_loss: 0.4462 - val_output_1_loss: 0.3989 - val_output_2_loss: 0.8704 - val_output_1_mae: 0.4504 - val_output_2_mae: 0.4993\n",
      "Epoch 20/20\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4249 - output_1_loss: 0.4192 - output_2_loss: 0.4762 - output_1_mae: 0.4634 - output_2_mae: 0.4994 - val_loss: 0.4422 - val_output_1_loss: 0.3978 - val_output_2_loss: 0.8405 - val_output_1_mae: 0.4485 - val_output_2_mae: 0.4943\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    (x_train_A, x_train_B), (y_train, y_train), epochs=20, batch_size=32,\n",
    "    validation_data=((x_valid_A, x_valid_B), (y_valid, y_valid)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:17.090791Z",
     "start_time": "2020-01-03T11:08:17.087306Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"wide_and_deep_model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                multiple                  270       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              multiple                  930       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              multiple                  36        \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              multiple                  31        \n",
      "=================================================================\n",
      "Total params: 1,267\n",
      "Trainable params: 1,267\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> only get a list of layers, without any information on how they are connected to each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:17.261584Z",
     "start_time": "2020-01-03T11:08:17.092483Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3391 - output_1_loss: 0.4158 - output_2_loss: 0.4657 - output_1_mae: 0.4600 - output_2_mae: 0.4941\n"
     ]
    }
   ],
   "source": [
    "total_loss, main_loss, aux_loss, main_mae, aux_mae = model.evaluate(\n",
    "    (x_test_A, x_test_B), (y_test, y_test), verbose=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:17.266302Z",
     "start_time": "2020-01-03T11:08:17.263170Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.4215106489122376, 0.41575122, 0.46571365)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_loss, main_loss, aux_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:17.271360Z",
     "start_time": "2020-01-03T11:08:17.267975Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4207474619150162"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "main_loss*0.9 + aux_loss*0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 和上面的 functional API 模型输出一摸一样。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Keras models can be used just like regular layers, so you can easily combine them to build complex architectures.</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Save and restore a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.106860Z",
     "start_time": "2020-01-03T11:08:17.272880Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 79us/sample - loss: 0.7157 - mae: 0.5846 - val_loss: 4.2397 - val_mae: 0.5293\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 0s 37us/sample - loss: 0.4353 - mae: 0.4677 - val_loss: 4.1245 - val_mae: 0.4842\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.4179 - mae: 0.4510 - val_loss: 1.4598 - val_mae: 0.4501\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.3927 - mae: 0.4406 - val_loss: 0.4107 - val_mae: 0.4203\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.3737 - mae: 0.4341 - val_loss: 0.3648 - val_mae: 0.4173\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x168f78128>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation='relu', input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(30, activation='relu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "model.compile(loss='mse', optimizer='sgd', metrics=['mae'])\n",
    "model.fit(x_train, y_train, epochs=5, validation_data=(x_valid, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the Sequential API or the Functional API, saving a trained Keras model is as simple as it gets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.122891Z",
     "start_time": "2020-01-03T11:08:20.108562Z"
    }
   },
   "outputs": [],
   "source": [
    "# save the model\n",
    "model.save('./models/chap10-models/my_keras_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras will use the **HDF5 format** to save\n",
    "+ the model’s architecture (including every layer’s hyperparameters)\n",
    "\n",
    "\n",
    "+ and the values of all the model parameters for every layer (e.g., connection weights and biases). \n",
    "\n",
    "\n",
    "+ also saves the optimizer (including its hyperparameters and any state it may have)\n",
    "\n",
    "\n",
    "In chap19, we will see how to **save a `tf.keras` model using TensorFlow’s `SavedModel` format instead**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will typically have a script that trains a model and saves it, and one or more scripts (or web services) that load the model and use it to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.332291Z",
     "start_time": "2020-01-03T11:08:20.124784Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 30)                270       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 30)                930       \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 31        \n",
      "=================================================================\n",
      "Total params: 1,231\n",
      "Trainable params: 1,231\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# load the saved model\n",
    "model_2 = keras.models.load_model('./models/chap10-models/my_keras_model.h5')\n",
    "model_2.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.478083Z",
     "start_time": "2020-01-03T11:08:20.334424Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.70903134],\n",
       "       [1.8357373 ],\n",
       "       [3.747096  ],\n",
       "       ...,\n",
       "       [1.6408094 ],\n",
       "       [2.638808  ],\n",
       "       [3.9370565 ]], dtype=float32)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_2.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>This will work when using the Sequential API or the Functional API, but unfortunately not when using model subclassing.</font>**\n",
    "\n",
    "\n",
    "You can use `save_weights()` and `load_weights()` to at least save and restore the model parameters, but you will need to save and restore everything else yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.491678Z",
     "start_time": "2020-01-03T11:08:20.479711Z"
    }
   },
   "outputs": [],
   "source": [
    "# save model's weights\n",
    "model.save_weights('./models/chap10-models/my_keras_weights.ckpt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T11:08:20.502210Z",
     "start_time": "2020-01-03T11:08:20.493344Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x14c9d1710>"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load model's weights\n",
    "model.load_weights('./models/chap10-models/my_keras_weights.ckpt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should not only save your model at the end of training, but also save checkpoints at regular intervals during training, to avoid losing everything if your computer crashes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **`fit()`** method accepts a **`callbacks` argument** that lets you specify **a list of objects** that Keras will call **at the start and end of training, at the start and end of each epoch, and even before and after processing each batch**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **<font color='Crimson'>ModelCheckpoint</font> (`tf.keras.callbacks.ModelCheckpoint`)** callback saves checkpoints of your model at regular intervals during training, by default at the end of each epoch (controlled by the `save_freq` argument)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='Green'>one way to implement early stopping:</font>**\n",
    "\n",
    "If you use a validation set during training, you can set **`save_best_only=True`** when creating the **`ModelCheckpoint`**. In this case, it will **only save your model when its performance on the validation set is the best so far**. This way, you do not need to worry about training for too long and overfitting the training set: simply restore the last model saved after training, and this will be the best model on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T13:21:09.236761Z",
     "start_time": "2020-01-03T13:21:06.608584Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 61us/sample - loss: 0.7157 - mae: 0.5846 - val_loss: 4.2397 - val_mae: 0.5293\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.4353 - mae: 0.4677 - val_loss: 4.1245 - val_mae: 0.4842\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.4179 - mae: 0.4510 - val_loss: 1.4598 - val_mae: 0.4501\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.3927 - mae: 0.4406 - val_loss: 0.4107 - val_mae: 0.4203\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.3737 - mae: 0.4341 - val_loss: 0.3648 - val_mae: 0.4173\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x16e176ef0>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation='relu', input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(30, activation='relu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "# model checkpoint\n",
    "checkpoin_cb = keras.callbacks.ModelCheckpoint(\n",
    "    # can contain named formatting options, which will be filled the value of\n",
    "    # `epoch` and keys in `logs` (passed in `on_epoch_end`)\n",
    "    filepath='./models/chap10-models/callbacks/my_keras_model.h5',\n",
    "    monitor='val_loss',       # default\n",
    "    verbose=0,                # default\n",
    "    mode='auto',              # default\n",
    "    # the lastest best model according to the quantity monitored wil not be overwritten\n",
    "    save_best_only=True,      # default False\n",
    "    # if True, only the model's weights will be saved (`model.save_weights(filepath)`)\n",
    "    save_weights_only=False,  # default, the full mode is saved (`model.save(filepath)`)\n",
    "    # if integer, the callback saves the model at end of a batch\n",
    "    # at which this many samples have been seen since last saving ....\n",
    "    save_freq='epoch')  # default, save the model after each epoch\n",
    "\n",
    "\n",
    "model.compile(loss='mse', optimizer='sgd', metrics=['mae'])\n",
    "model.fit(x_train, y_train, epochs=5, validation_data=(x_valid, y_valid),\n",
    "          callbacks=[checkpoin_cb])  # a list of callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T13:22:40.137597Z",
     "start_time": "2020-01-03T13:22:39.764599Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.2804 - mae: 0.4258\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.3640460860359576, 0.4257913]"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rollback to the best model\n",
    "model = keras.models.load_model('./models/chap10-models/callbacks/my_keras_model.h5')\n",
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'>another way to implement early stopping:</font>**\n",
    "\n",
    "**<font color='Crimson'>EarlyStopping</font> (`tf.keras.callbacks.EarlyStopping`)** will **interrupt training when it measures no progress on the validation set for a number of epochs** (defined by the `patience` argument), and it will optionally roll back to the best model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **combine both callbacks** to save checkpoints of your model (in case your computer crashes) and interrupt training early when there is no more progress (to avoid wasting time and resources):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T13:46:20.048986Z",
     "start_time": "2020-01-03T13:46:15.560044Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/100\n",
      "11610/11610 [==============================] - 1s 61us/sample - loss: 0.7157 - mae: 0.5846 - val_loss: 4.2397 - val_mae: 0.5293\n",
      "Epoch 2/100\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.4353 - mae: 0.4677 - val_loss: 4.1245 - val_mae: 0.4842\n",
      "Epoch 3/100\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.4179 - mae: 0.4510 - val_loss: 1.4598 - val_mae: 0.4501\n",
      "Epoch 4/100\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.3927 - mae: 0.4406 - val_loss: 0.4107 - val_mae: 0.4203\n",
      "Epoch 5/100\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.3737 - mae: 0.4341 - val_loss: 0.3648 - val_mae: 0.4173\n",
      "Epoch 6/100\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.3720 - mae: 0.4306 - val_loss: 0.3386 - val_mae: 0.4127\n",
      "Epoch 7/100\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.3613 - mae: 0.4255 - val_loss: 0.3352 - val_mae: 0.4046\n",
      "Epoch 8/100\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.3563 - mae: 0.4211 - val_loss: 0.3435 - val_mae: 0.4163\n",
      "Epoch 9/100\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.3515 - mae: 0.4188 - val_loss: 0.3598 - val_mae: 0.4072\n",
      "Epoch 10/100\n",
      "11610/11610 [==============================] - 0s 37us/sample - loss: 0.3475 - mae: 0.4151 - val_loss: 0.3365 - val_mae: 0.4082\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x16ec96048>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation='relu', input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(30, activation='relu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "# model checkpoint callback\n",
    "checkpoin_cb = keras.callbacks.ModelCheckpoint(\n",
    "    filepath='./models/chap10-models/callbacks/my_keras_model.h5',\n",
    "    monitor='val_loss',\n",
    "    save_best_only=True)\n",
    "# early stopping callback\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(\n",
    "    monitor='val_loss',  # quantity to be monitored\n",
    "    # Minimum change in the monitored quantity to qualify as an improvement,\n",
    "    # i.e. an absolute change of less than min_delta, will count as no improvement.\n",
    "    min_delta=0,               # default\n",
    "    # Number of epochs with no improvement after which training will be stopped.\n",
    "    patience=3,                # default 0\n",
    "    verbose=0,                 # default 0\n",
    "    mode='auto',               # default, one of {'auto', 'min', 'max'}\n",
    "    # Baseline value for the monitored quantity.\n",
    "    # Training will stop if the model doesn't show improvement over the baseline.\n",
    "    baseline=None, # default\n",
    "    # Whether to restore model weights from the epoch with the best value of the monitored quantity.\n",
    "    # If False, the model weights obtained at the last step of training are used.\n",
    "    restore_best_weights=True)  # default False\n",
    "\n",
    "model.compile(loss='mse', optimizer='sgd', metrics=['mae'])\n",
    "model.fit(x_train, y_train, epochs=100, validation_data=(x_valid, y_valid),\n",
    "          callbacks=[checkpoin_cb, early_stopping_cb])  # a list of callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ The number of epochs can be set to a large value since training will stop automatically when there is no more progress. \n",
    "\n",
    "\n",
    "+ There is **no need to restore the best model saved** because the `EarlyStopping` callback will keep track of the best weights and restore them for you at the end of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can **<font color='blue'>write your own custom callbacks</font>**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T13:56:20.456274Z",
     "start_time": "2020-01-03T13:56:20.452495Z"
    }
   },
   "outputs": [],
   "source": [
    "# a custom callback will display the ratio between the val loss and the training loss\n",
    "# during training (e.g., to detect overfitting)\n",
    "class PrintValTrainRatioCallback(keras.callbacks.Callback):\n",
    "    \"\"\"\n",
    "    keras.callbacks.Callback: Abstract base class used to build new callbacks.\n",
    "    \"\"\"\n",
    "    def on_epoch_end(self, epoch, logs):\n",
    "        \"\"\"\n",
    "        Called at the end of an epoch.\n",
    "        Subclass should override for any actions to run.\n",
    "        This function should only be called during TRAIN model.\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        epoch: integer,\n",
    "            index of epoch\n",
    "        logs: dict,\n",
    "            metric results for this training epoch, and for the validation\n",
    "            epoch if validation is performed.\n",
    "            Validation result keys are prefixed with `val_`.\n",
    "        \"\"\"\n",
    "        print(\"\\nval/train: {:.2f}\".format(logs['val_loss'] / logs['loss']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-03T13:57:43.259633Z",
     "start_time": "2020-01-03T13:57:42.336316Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11488/11610 [============================>.] - ETA: 0s - loss: 0.3546 - mae: 0.4203\n",
      "val/train: 1.05\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.3546 - mae: 0.4205 - val_loss: 0.3706 - val_mae: 0.4357\n",
      "Epoch 2/2\n",
      "11456/11610 [============================>.] - ETA: 0s - loss: 0.3542 - mae: 0.4198\n",
      "val/train: 0.95\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.3531 - mae: 0.4193 - val_loss: 0.3367 - val_mae: 0.4075\n"
     ]
    }
   ],
   "source": [
    "val_train_ratio_cb = PrintValTrainRatioCallback()\n",
    "history = model.fit(x_train, y_train, epochs=2,\n",
    "                    validation_data=(x_valid, y_valid),\n",
    "                    callbacks=[val_train_ratio_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "also you can implement:\n",
    "+ `on_train_begin/end()`\n",
    "+ `on_epoch_begin/end()`\n",
    "+ `on_batch_begin/end()`\n",
    "+ `on_train_batch_begin/end()`\n",
    "\n",
    "\n",
    "**callbacks used for evaluation**, you can implement (called by `evaluate()`):\n",
    "+ `on_test_begin/end()`\n",
    "+ `on_test_batch_begin/end()`\n",
    "\n",
    "\n",
    "**callbacks used for prediction**, you can implement (called by `predict()`):\n",
    "+ `on_predict_begin/end()`\n",
    "+ `on_predict_batch_begin/end()`\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.7 TensorBoard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='Crimson'>TensorBoard</font>** is a great interactive visualization tool\n",
    "\n",
    "+ view the learning curves during training, compare learning curves between multiple runs\n",
    "\n",
    "\n",
    "+ visualize the computation graph\n",
    "\n",
    "\n",
    "+ analyze training statistics\n",
    "\n",
    "\n",
    "+ view images generated by the model\n",
    "\n",
    "\n",
    "+ visualize complex multidimensional data projected down to 3D and automatically clustered it for you\n",
    "\n",
    "\n",
    "+ ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use TensorBoard, you **must modify the code** so that it **outputs the data** (that you want to visualize) **to special binary log files** called **<font color='Brown'>event files</font>**. Each binary data record is called a **<font color='Brown'>summary</font>**.\n",
    "\n",
    "The **TensorBoard server will monitor the log directory**, and it will **automatically pick up the changes and update the visualizations**: this allows you to visualize live data (with a short delay), such as the learning curves during training.\n",
    "\n",
    "**<font color='red'>实际使用</font>** In general, you want to <font color='blue'>point the TensorBoard server to a root log directory and configure your program so that it writes to a different subdirectory every time it runs</font>. This way, the same TensorBoard server instance will *allow you to visualize and compare data from multiple runs of your program*, without getting everything mixed up."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:36.518980Z",
     "start_time": "2020-01-04T07:27:36.516519Z"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:37.364939Z",
     "start_time": "2020-01-04T07:27:37.361429Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'.'"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.curdir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:38.326455Z",
     "start_time": "2020-01-04T07:27:38.322703Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./logs/chap10-logs/'"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define a root log directory used for TensorBoard logs\n",
    "root_log_dir = os.path.join(os.curdir, 'logs/chap10-logs/')\n",
    "root_log_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:40.623159Z",
     "start_time": "2020-01-04T07:27:40.620498Z"
    }
   },
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:41.157049Z",
     "start_time": "2020-01-04T07:27:41.153424Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'run_2020_01_04-15_27_41'"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time.strftime('run_%Y_%m_%d-%H_%M_%S')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:44.215775Z",
     "start_time": "2020-01-04T07:27:44.212820Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_run_log_dir():\n",
    "    \"\"\"Generate a subdirectory path based on the current data and time\n",
    "    so that it's different at every run.\n",
    "    \"\"\"\n",
    "    run_id = time.strftime('run_%Y_%m_%d-%H_%M_%S')\n",
    "    return os.path.join(root_log_dir, run_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> You can also include extra info in the log directory in the log directory name, such as hyperparameter values that you are testing, to make ti easier to know what you are looking at in TensorBoard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'>run the program first time:</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:46.107757Z",
     "start_time": "2020-01-04T07:27:46.104151Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./logs/chap10-logs/run_2020_01_04-15_27_46'"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_log_dir = get_run_log_dir()\n",
    "run_log_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Keras provides a nice `TensorBoard()` callback:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:27:50.496588Z",
     "start_time": "2020-01-04T07:27:50.493870Z"
    }
   },
   "outputs": [],
   "source": [
    "# tensorboard callback\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(\n",
    "    log_dir=run_log_dir,  # directory where to save the log files\n",
    "    update_freq='epoch',  # default, or 'batch' or integer (mean xx samples)\n",
    "    # some arguments about embedding\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:29:09.747664Z",
     "start_time": "2020-01-04T07:29:05.171475Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/10\n",
      "11610/11610 [==============================] - 1s 56us/sample - loss: 0.7157 - mae: 0.5846 - val_loss: 4.2397 - val_mae: 0.5293\n",
      "Epoch 2/10\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.4353 - mae: 0.4677 - val_loss: 4.1245 - val_mae: 0.4842\n",
      "Epoch 3/10\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.4179 - mae: 0.4510 - val_loss: 1.4598 - val_mae: 0.4501\n",
      "Epoch 4/10\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.3927 - mae: 0.4406 - val_loss: 0.4107 - val_mae: 0.4203\n",
      "Epoch 5/10\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.3737 - mae: 0.4341 - val_loss: 0.3648 - val_mae: 0.4173\n",
      "Epoch 6/10\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.3720 - mae: 0.4306 - val_loss: 0.3386 - val_mae: 0.4127\n",
      "Epoch 7/10\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.3613 - mae: 0.4255 - val_loss: 0.3352 - val_mae: 0.4046\n",
      "Epoch 8/10\n",
      "11610/11610 [==============================] - 0s 37us/sample - loss: 0.3563 - mae: 0.4211 - val_loss: 0.3435 - val_mae: 0.4163\n",
      "Epoch 9/10\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.3515 - mae: 0.4188 - val_loss: 0.3598 - val_mae: 0.4072\n",
      "Epoch 10/10\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.3475 - mae: 0.4151 - val_loss: 0.3365 - val_mae: 0.4082\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation='relu', input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(30, activation='relu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss='mse',\n",
    "              optimizer=keras.optimizers.SGD(lr=0.01),  # default\n",
    "              metrics=['mae'])\n",
    "history = model.fit(x_train, y_train, epochs=10, batch_size=32,\n",
    "                    validation_data=(x_valid, y_valid),\n",
    "                    callbacks=[tensorboard_cb])  # tensorboard callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all there is to it.\n",
    "\n",
    "**run the code:**\n",
    "\n",
    "+ `TensorBoard()` callback will take care of creating the log directory for you (along with its parent directories if needed)\n",
    "\n",
    "\n",
    "+ during training, it will create event files and write summaries to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:29:25.381651Z",
     "start_time": "2020-01-04T07:29:25.377367Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['train', 'validation']"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(run_log_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:29:26.151132Z",
     "start_time": "2020-01-04T07:29:26.146459Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['plugins',\n",
       " 'events.out.tfevents.1578122945.bingli.51329.484695.v2',\n",
       " 'events.out.tfevents.1578122945.bingli.profile-empty']"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(os.path.join(run_log_dir, 'train'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:29:27.789972Z",
     "start_time": "2020-01-04T07:29:27.785833Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['events.out.tfevents.1578122945.bingli.51329.486656.v2']"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(os.path.join(run_log_dir, 'validation'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **<font color='green'>run the program a second time:</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:29:52.429226Z",
     "start_time": "2020-01-04T07:29:52.425763Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./logs/chap10-logs/run_2020_01_04-15_29_52'"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_log_dir2 = get_run_log_dir()\n",
    "run_log_dir2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:30:42.312339Z",
     "start_time": "2020-01-04T07:30:37.453363Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/10\n",
      "11610/11610 [==============================] - 1s 56us/sample - loss: 1.8631 - mae: 1.0009 - val_loss: 3.2403 - val_mae: 0.6867\n",
      "Epoch 2/10\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.6723 - mae: 0.6092 - val_loss: 0.6939 - val_mae: 0.5660\n",
      "Epoch 3/10\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.5927 - mae: 0.5608 - val_loss: 0.5531 - val_mae: 0.5386\n",
      "Epoch 4/10\n",
      "11610/11610 [==============================] - 0s 37us/sample - loss: 0.5550 - mae: 0.5401 - val_loss: 0.5519 - val_mae: 0.5200\n",
      "Epoch 5/10\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 0.5278 - mae: 0.5238 - val_loss: 0.5715 - val_mae: 0.5111\n",
      "Epoch 6/10\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.5032 - mae: 0.5098 - val_loss: 0.5591 - val_mae: 0.4969\n",
      "Epoch 7/10\n",
      "11610/11610 [==============================] - 0s 41us/sample - loss: 0.4863 - mae: 0.5026 - val_loss: 0.4593 - val_mae: 0.4840\n",
      "Epoch 8/10\n",
      "11610/11610 [==============================] - 0s 42us/sample - loss: 0.4701 - mae: 0.4920 - val_loss: 0.4651 - val_mae: 0.4777\n",
      "Epoch 9/10\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.4586 - mae: 0.4862 - val_loss: 0.4392 - val_mae: 0.4669\n",
      "Epoch 10/10\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.4476 - mae: 0.4793 - val_loss: 0.4257 - val_mae: 0.4623\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "tf.keras.backend.clear_session()\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation='relu', input_shape=x_train.shape[1:]),\n",
    "    keras.layers.Dense(30, activation='relu'),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss='mse',\n",
    "              optimizer=keras.optimizers.SGD(lr=0.001),  # decrease, default 0.01\n",
    "              metrics=['mae'])\n",
    "\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(log_dir=run_log_dir2)\n",
    "history = model.fit(x_train, y_train, epochs=10, batch_size=32,\n",
    "                    validation_data=(x_valid, y_valid),\n",
    "                    callbacks=[tensorboard_cb])  # tensorboard callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:30:50.505943Z",
     "start_time": "2020-01-04T07:30:50.501667Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['run_2020_01_04-15_29_52', 'run_2020_01_04-15_27_46', '.ipynb_checkpoints']"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(root_log_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:30:52.428977Z",
     "start_time": "2020-01-04T07:30:52.424747Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['train', 'validation']"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(run_log_dir2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:30:53.547196Z",
     "start_time": "2020-01-04T07:30:53.543191Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['plugins',\n",
       " 'events.out.tfevents.1578123037.bingli.profile-empty',\n",
       " 'events.out.tfevents.1578123037.bingli.51329.515941.v2']"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(os.path.join(run_log_dir2, 'train'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There’s **one directory per run, each containing one subdirectory for training logs and one for validation logs**. Both contain event files, but the training logs also include **profiling traces** (`train/plugins/profile/2020-01-04_15-09-01/local.trace`): this allows TensorBoard to **show you exactly how much time the model spent on each part of your model, across all your devices**, which is great for locating performance bottlenecks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Start the TensorBoard Server:</font>**\n",
    "\n",
    "+ **run a command in a terminal** (should activate the virtualenv, if needed)\n",
    "\n",
    "`tensorboard --logdir=path_to_your_logs` (anywhere else, as long as you point to the appropriate log directory)\n",
    "\n",
    "`tensorboard --logdir=./logs/chap10-logs --port=6006` (for this notebook，注意 port 不要与 jupyter notebook 的 port 冲突)\n",
    "\n",
    "执行完命令显示如下信息即可：\n",
    "\n",
    "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
    "\n",
    "TensorBoard 2.0.0 at http://localhost:6006/ (Press CTRL+C to quit)\n",
    "\n",
    "> 在浏览器中打开这个链接即可。\n",
    "\n",
    "> If your shell cannot find the tensorboard script, then you must update your `PATH` environment variable so that it contains the directory in which the script was installed (alternatively, you can just replace tensorboard in the command line with `python3 -m tensorboard.main`).\n",
    "\n",
    "\n",
    "+ **use TensorBoard directly within Jupyter**\n",
    "\n",
    "```bash\n",
    "# loads the TensorBoard extension\n",
    "%load_ext tensorboard\n",
    "# starts a TensorBoard server on port 6006 (unless it is already started) and connects to it\n",
    "%tensorboard --logdir=./logs/chap10-logs --port 6006\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![tensorboard](./figs/chap10-figs/tensorboard.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also visualize the whole graph, the learned weights (projected to 3D), or the profiling traces. The `TensorBoard()` callback has options to log extra data too, such as embeddings (see chap13)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow offers a lower-level API in the `tf.summary` package. The following code creates a `SummaryWriter` using the `create_file_writer()` function, and it uses this writer as a context to log scalars, histograms, images, audio, and text, all of which can then be visualized using TensorBoard:\n",
    "\n",
    "```python\n",
    "test_logdir = get_run_logdir()\n",
    "writer = tf.summary.create_file_writer(test_logdir)\n",
    "with writer.as_default():\n",
    "    for step in range(1, 1000 + 1):\n",
    "        tf.summary.scalar(\"my_scalar\", np.sin(step / 10), step=step)\n",
    "        data = (np.random.randn(100) + 2) * step / 100 # some random data\n",
    "        tf.summary.histogram(\"my_hist\", data, buckets=50, step=step)\n",
    "        images = np.random.rand(2, 32, 32, 3) # random 32×32 RGB images\n",
    "        tf.summary.image(\"my_images\", images * step / 1000, step=step)\n",
    "        texts = [\"The step is \" + str(step), \"Its square is \" + str(step**2)]\n",
    "        tf.summary.text(\"my_text\", texts, step=step)\n",
    "        sine_wave = tf.math.sin(tf.range(12000) / 48000 * 2 * np.pi * step)\n",
    "        audio = tf.reshape(tf.cast(sine_wave, tf.float32), [1, -1, 1])\n",
    "        tf.summary.audio(\"my_audio\", audio, sample_rate=48000, step=step)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T07:53:31.017434Z",
     "start_time": "2020-01-04T07:53:31.014299Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function __init__ in module tensorflow.python.keras.callbacks:\n",
      "\n",
      "__init__(self, log_dir='logs', histogram_freq=0, write_graph=True, write_images=False, update_freq='epoch', profile_batch=2, embeddings_freq=0, embeddings_metadata=None, **kwargs)\n",
      "    Initialize self.  See help(type(self)) for accurate signature.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# check out the other available logging options:\n",
    "help(keras.callbacks.TensorBoard.__init__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 Fine-tuning NN hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more best practices regarding tuning NN hyperparameters, check out the excellent [2018 paper by Leslie Smith](https://homl.info/1cycle)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='green' size=3>1. One option</font>**\n",
    "    \n",
    "One option is to **simply try many combinations of hyperparameters and see which one works best on the validation set (or use \"K-fold\" cross-validation)**, like using **<font color='Crimson'>GridSearchCV</font>** (`sklearn.model_selection.GridSearchCV`) or **<font color='Crimson'>RandomizedSearchCV</font>** (`sklearn.model_selection.RandomizedSearchCV`).\n",
    "\n",
    "To do this, we need to **<font color='red'>wrap Keras models in objects that mimic (模仿) regular Scikit-Learn regressors/classifiers</font>**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T08:55:36.411130Z",
     "start_time": "2020-01-04T08:55:36.405488Z"
    }
   },
   "outputs": [],
   "source": [
    "# first step\n",
    "# create a function that will *build* and *compile* a Keras model, given a set of hyperparameters\n",
    "def build_model(n_hidden=1, n_units=30, learning_rate=3e-3, input_shape=(8,)):\n",
    "    np.random.seed(42)\n",
    "    tf.random.set_seed(42)\n",
    "    tf.keras.backend.clear_session()\n",
    "\n",
    "    # build\n",
    "    model = keras.models.Sequential()\n",
    "    model.add(keras.layers.InputLayer(input_shape=input_shape))\n",
    "    for layer in range(n_hidden):\n",
    "        model.add(keras.layers.Dense(n_units, activation='relu'))\n",
    "    model.add(keras.layers.Dense(1))\n",
    "\n",
    "    # compile\n",
    "    optimizer = keras.optimizers.SGD(lr=learning_rate)\n",
    "    model.compile(loss='mse', optimizer=optimizer, metrics=['mae'])\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**It is good practice to provide reasonable defaults to as many hyperparameters as you can, as Scikit-Learn does.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T08:59:38.516132Z",
     "start_time": "2020-01-04T08:59:38.513653Z"
    }
   },
   "outputs": [],
   "source": [
    "# next step\n",
    "# create a `KerasRegressor` based on this `build_model()` function\n",
    "keras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `KerasRegressor` object is a *thin wrapper* around the Keras model built using `build_model()`.\n",
    "\n",
    "Since we did not specify any hyperparameters when creating it, it will use the default hyperparameters (defined in `build_model()`).\n",
    "\n",
    "**Now we use this object like a regular Scikit-Learn regressor:**\n",
    "\n",
    "+ use `fit()` to train it\n",
    "\n",
    "\n",
    "+ use its `score()` method to evaluate it\n",
    "\n",
    "\n",
    "+ use its `predict()` method to make prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:10:30.292253Z",
     "start_time": "2020-01-04T09:10:22.988508Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/50\n",
      "11610/11610 [==============================] - 1s 52us/sample - loss: 1.1815 - mae: 0.7215 - val_loss: 20.7837 - val_mae: 0.6763\n",
      "Epoch 2/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.7013 - mae: 0.5560 - val_loss: 0.6646 - val_mae: 0.5304\n",
      "Epoch 3/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.5197 - mae: 0.5189 - val_loss: 0.5974 - val_mae: 0.4990\n",
      "Epoch 4/50\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.4811 - mae: 0.4962 - val_loss: 0.5292 - val_mae: 0.4749\n",
      "Epoch 5/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.4565 - mae: 0.4816 - val_loss: 0.4813 - val_mae: 0.4618\n",
      "Epoch 6/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.4393 - mae: 0.4711 - val_loss: 0.4501 - val_mae: 0.4576\n",
      "Epoch 7/50\n",
      "11610/11610 [==============================] - 0s 31us/sample - loss: 0.4283 - mae: 0.4671 - val_loss: 0.4231 - val_mae: 0.4488\n",
      "Epoch 8/50\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.4205 - mae: 0.4607 - val_loss: 0.4072 - val_mae: 0.4476\n",
      "Epoch 9/50\n",
      "11610/11610 [==============================] - 0s 37us/sample - loss: 0.4154 - mae: 0.4591 - val_loss: 0.3956 - val_mae: 0.4410\n",
      "Epoch 10/50\n",
      "11610/11610 [==============================] - 0s 36us/sample - loss: 0.4100 - mae: 0.4557 - val_loss: 0.3873 - val_mae: 0.4392\n",
      "Epoch 11/50\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.4060 - mae: 0.4535 - val_loss: 0.3811 - val_mae: 0.4384\n",
      "Epoch 12/50\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.4028 - mae: 0.4517 - val_loss: 0.3743 - val_mae: 0.4364\n",
      "Epoch 13/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.3995 - mae: 0.4496 - val_loss: 0.3691 - val_mae: 0.4344\n",
      "Epoch 14/50\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.3964 - mae: 0.4484 - val_loss: 0.3657 - val_mae: 0.4300\n",
      "Epoch 15/50\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.3932 - mae: 0.4464 - val_loss: 0.3640 - val_mae: 0.4304\n",
      "Epoch 16/50\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.3903 - mae: 0.4446 - val_loss: 0.3618 - val_mae: 0.4273\n",
      "Epoch 17/50\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.3882 - mae: 0.4426 - val_loss: 0.3629 - val_mae: 0.4288\n",
      "Epoch 18/50\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.3856 - mae: 0.4412 - val_loss: 0.3661 - val_mae: 0.4319\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x17008b828>"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# final step\n",
    "# use it like a regular Scikit-Learn regressor\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=2,  # for statement\n",
    "                                                  restore_best_weights=True)\n",
    "# any extra parameters passed to the `fit` will get passed to the underlying keras model\n",
    "keras_reg.fit(x_train,\n",
    "              y_train,\n",
    "              epochs=50,\n",
    "              validation_data=(x_valid, y_valid),\n",
    "              callbacks=[early_stopping_cb])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **<font color='blue'>Note that any extra parameter you pass to the `fit()` method will get passed to the underlying Keras model.</font>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:12:21.194862Z",
     "start_time": "2020-01-04T09:12:21.056469Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.3010 - mae: 0.4402\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "-0.3846614597610725"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# evaluate the KerasRegressor\n",
    "# the score will be the opposite of the MSE \n",
    "mse_test = keras_reg.score(x_test, y_test, verbose=2)\n",
    "mse_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>Note that the score will be the opposite of the MSE because Scikit-Learn wants scores, not losses (i.e., higher should be better).</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Since there are many hyperparameters, it is preferable to use a randomized search rather than grid search.</font>** (see in chap2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:14:27.648144Z",
     "start_time": "2020-01-04T09:14:27.645589Z"
    }
   },
   "outputs": [],
   "source": [
    "from scipy.stats import reciprocal\n",
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:28:17.517702Z",
     "start_time": "2020-01-04T09:26:47.666071Z"
    }
   },
   "source": [
    "```python\n",
    "# 由于 TensrFlow 的原因?，输出中会有很多 == ，\n",
    "# 因此将这个 code snippet 跑完后从 Python code 转换为 markdown\n",
    "param_distribs = {  # just for statement\n",
    "    'n_hidden': [0, 1, 2, 3, 4],\n",
    "    'n_units': [10, 20, 30, 40, 50],\n",
    "    #     'learning_rate': reciprocal(3e-4, 3e-2),\n",
    "    'learning_rate': [0.001, 0.002, 0.005, 0.008, 0.01]\n",
    "}\n",
    "rnd_search_cv = RandomizedSearchCV(keras_reg,\n",
    "                                   param_distributions=param_distribs,\n",
    "                                   n_iter=10,\n",
    "                                   cv=3,\n",
    "                                   verbose=2,\n",
    "                                   random_state=42)\n",
    "# extra params passed to the `fit()` will get relayed(传递) to the underlying keras model\n",
    "rnd_search_cv.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    epochs=50,\n",
    "    # `RandomizedSearchCV` uses k-fold cv, so it does not use X_valid and y_valid,\n",
    "    # which are only used for early stopping.\n",
    "    validation_data=(x_valid, y_valid),\n",
    "    callbacks=[early_stopping_cb],\n",
    "    verbose=0)  # tf.keras evalute 时有很多 == 的问题还没有解决\n",
    "```"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "**output for the code snippet above:**\n",
    "\n",
    "RandomizedSearchCV(cv=3, error_score='raise-deprecating',\n",
    "                   estimator=<tensorflow.python.keras.wrappers.scikit_learn.KerasRegressor object at 0x16fa57550>,\n",
    "                   iid='warn', n_iter=10, n_jobs=None,\n",
    "                   param_distributions={'learning_rate': [0.001, 0.002, 0.005,\n",
    "                                                          0.008, 0.01],\n",
    "                                        'n_hidden': [0, 1, 2, 3, 4],\n",
    "                                        'n_units': [10, 20, 30, 40, 50]},\n",
    "                   pre_dispatch='2*n_jobs', random_state=42, refit=True,\n",
    "                   return_train_score=False, scoring=None, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Note that `RandomizedSearchCV` uses K-fold cross-validation, so it does not use `x_valid` and `y_valid`, which are only used for early stopping.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The exploration may last many hours, depending on the hardware, the size of the dataset, the complexity of the model, and the values of `n_iter` and `cv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:31:02.194483Z",
     "start_time": "2020-01-04T09:31:02.191187Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_units': 40, 'n_hidden': 4, 'learning_rate': 0.01}"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# access the best parameters found\n",
    "rnd_search_cv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:31:19.321277Z",
     "start_time": "2020-01-04T09:31:19.317814Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.39890320162440457"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# best score\n",
    "rnd_search_cv.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> The score will be the opposite of the MSE because Scikit-Learn wants scores, not losses (i.e., higher should be better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:32:38.233516Z",
     "start_time": "2020-01-04T09:32:38.229788Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.wrappers.scikit_learn.KerasRegressor at 0x171f87128>"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rnd_search_cv.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:33:33.993027Z",
     "start_time": "2020-01-04T09:33:33.979719Z"
    }
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "score() got an unexpected keyword argument 'verbose'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-165-ed60e1eb589b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mrnd_search_cv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscore\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_test\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: score() got an unexpected keyword argument 'verbose'"
     ]
    }
   ],
   "source": [
    "rnd_search_cv.score(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 神奇！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:34:27.563147Z",
     "start_time": "2020-01-04T09:34:27.560948Z"
    }
   },
   "outputs": [],
   "source": [
    "# rnd_search_cv.score(x_test, y_test) 这样有很多 =="
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:34:43.821984Z",
     "start_time": "2020-01-04T09:34:43.818282Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.engine.sequential.Sequential at 0x17104afd0>"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the trained Keras model\n",
    "model = rnd_search_cv.best_estimator_.model\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:34:58.068695Z",
     "start_time": "2020-01-04T09:34:57.923858Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5160/1 - 0s - loss: 0.2583 - mae: 0.4653\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.41407177108664844, 0.46533865]"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, y_test, verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 与 `rnd_search_cv.score(x_test, y_test)` 结果一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-01-04T09:35:34.662139Z",
     "start_time": "2020-01-04T09:35:34.658531Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['loss', 'mae']"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now save this model, evaluate it on the test set, and, if you are satisfied with its performance, deploy it to production.\n",
    "\n",
    "Using **randomized search** is not too hard, and it **works well for many fairly simple problems**.\n",
    "\n",
    "**<font color='red'>解决 RandomSearchCV 训练慢的方法（比无脑好）：</font>**\n",
    "\n",
    "When training is slow, however (e.g., for more complex problems with larger datasets), this approach will only explore a tiny portion of the hyperparameter space. You can partially alleviate this problem by assisting (帮助、协助) the search process manually: **first run a quick random search using wide ranges of hyperparameter values, then run another search using smaller ranges of values centered on the best ones found during the first run, and so on (最后一个 range 中使用 `GridSearchCV` 会不会好一点?). This approach will hopefully zoom in on a good set of hyperparameters**. However, **it’s very time consuming, and probably not the best use of your time.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='green' size=3>2. some Python libraries</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many techniques to explore a search space much more efficiently than randomly.\n",
    "\n",
    "Their **<font color='red'>core idea</font>** is simple: **<font color='red'>when a region of the space turns out to be good, it should be explored more</font>**.\n",
    "\n",
    "Such techniques **take care of the “zooming” process for you and lead to much better solutions in much less time**. Here are some Python libraries you can use to optimize hyperparameters:\n",
    "\n",
    "+ [Hyperopt](https://github.com/hyperopt/hyperopt)\n",
    "\n",
    "A popular library for optimizing over all sorts of complex search spaces (including real values, such as the learning rate, and discrete values, such as the number of layers).\n",
    "\n",
    "\n",
    "+ [Hyperas](https://github.com/maxpumperla/hyperas), [kopt](https://github.com/Avsecz/kopt), or [Talos](https://github.com/autonomio/talos)\n",
    "\n",
    "Useful libraries for optimizing hyperparameters for Keras models (the first two are based on Hyperopt).\n",
    "\n",
    "\n",
    "+ [Keras Tuner](https://homl.info/kerastuner)\n",
    "\n",
    "An easy-to-use hyperparameter optimization library **by Google for Keras models**, with a hosted service for visualization and analysis.\n",
    "\n",
    "\n",
    "+ [Scikit-Optimize (skopt)](https://scikit-optimize.github.io/)\n",
    "\n",
    "A general-purpose optimization library. The `BayesSearchCV` class performs Bayesian optimization using an interface similar to `GridSearchCV`.\n",
    "\n",
    "\n",
    "+ [Spearmint](https://github.com/JasperSnoek/spearmint) -- A Bayesian optimization library.\n",
    "\n",
    "\n",
    "+ [Hyperban](https://github.com/zygmuntz/hyperband)\n",
    "\n",
    "A fast hyperparameter tuning library based on the recent [Hyperband 2018 paper](https://homl.info/hyperband) by Lisha Li et al.\n",
    "\n",
    "\n",
    "+ [Sklearn-Deap](https://github.com/rsteca/sklearn-deap)\n",
    "\n",
    "A hyperparameter optimization library **based on evolutionary algorithms**, with a `GridSearchCV`-like interface."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='green' size=3>3. services for hyperparameter optimization by some companies</font>**\n",
    "\n",
    "+ Google Cloud AI Platform’s [hyperparameter tuning service](https://homl.info/googletuning), chap19\n",
    "\n",
    "\n",
    "+ services by [Arimo](https://arimo.com/) and [SigOpt](https://sigopt.com/)\n",
    "\n",
    "\n",
    "+ CallDesk’s [Oscar](http://oscar.calldesk.ai/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='green' size=3>4. evolutionary algorithms</font>**\n",
    "\n",
    "**Hyperparameter tuning is still an active area of research, and evolutionary algorithms are making a comeback.**\n",
    "\n",
    "+ [DeepMind' 2017 paper](https://homl.info/pbt) jointly optimize a population of models and their hyperparameters.\n",
    "\n",
    "\n",
    "+ Google has also used an evolutionary approach, not just to search for hyperparameters but also to look for the best neural network architecture for the problem; their [AutoML](https://homl.info/automlpost) suite is already available as a cloud service.\n",
    "\n",
    "\n",
    "+ In fact, evolutionary algorithms have been used successfully to train individual neural networks, replacing the ubiquitous Gradient Descent! For an example, [Uber's 2017 post](https://homl.info/neuroevol) introduce their Deep Neuroevolution technique."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But despite all this exciting progress and all these tools and services, it **still helps to have an idea of what values are reasonable for each hyperparameter so that you can build a quick prototype and restrict the search space**.\n",
    "\n",
    "The following sections provide guidelines for selecting good values for some of the main hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Number of hidder layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For many problems, you can begin with a single hidden layer and get reasonable results.**\n",
    "\n",
    "An MLP with just one hidden layer can theoretically model even the most complex functions, provided it has enough units.\n",
    "\n",
    "<div class=\"alert alert-block alert-danger\">\n",
    "    <b>为什么含有足够多的 unit 的单层网络就能解决最复杂的问题了，为什么还要使用多层网络呢？</b>\n",
    "</div>\n",
    "\n",
    "But **for complex problems, deep networks have a much higher <font color='Crimson'>parameter efficiency</font> than shallow ones**: they can **<font color='red'>model complex functions using exponentially fewer units than shallow nets, allowing them to reach much better performance with the same amount of training data</font>**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> To understand why, suppose you are asked to draw a forest using some drawing software, but you are forbidden to copy and paste anything. It would take an enormous amount of time: you would have to draw each tree individually, branch by branch, leaf by leaf. If you could instead draw one leaf, copy and paste it to draw a branch, then copy and paste that branch to create a tree, and finally copy and paste this tree to make a forest, you would be finished in no time.\n",
    "\n",
    "**Real-world data is often structured in such a hierarchical way, and deep neural networks automatically take advantage of this fact:**\n",
    "\n",
    "+ **<font color='blue'>lower hidden layers model low-level structures</font>** (e.g., line segments of various shapes and orientations)\n",
    "\n",
    "\n",
    "+ **<font color='blue'>intermediate hidden layers combine these low-level structures to model intermediate-level structures</font>** (e.g., squares, circles)\n",
    "\n",
    "\n",
    "+ **<font color='blue'>the highest hidden layers and the output layer combine these intermediate structures to model high-level structures</font>** (e.g., faces)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Not only does this hierarchical architecture help DNNs converge faster to a good solution, but it also improves their ability to generalize to new datasets.</font>**\n",
    "\n",
    "For example, if you have already trained a model to recognize faces in pictures and you now want to train a new neural network to recognize hairstyles, you can kickstart the training by reusing the lower layers of the first network. Instead of randomly initializing the weights and biases of the first few layers of the new neural network, you can initialize them to the values of the weights and biases of the lower layers of the first network. This way the network will not have to learn from scratch all the low-level structures that occur in most pictures; it will only have to learn the higher-level structures (e.g., hairstyles). This is called **<font color='Crimson'>transfer learning</font>**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>Summary:</font>**\n",
    "\n",
    "+ For **many problems**, you can **start with just one or two hidden layers** and the NN will work just fine.\n",
    "\n",
    "> You can easily reach above 97% accuracy on the MNIST dataset using just one hidden layer with a few hundred neurons, and above 98% accuracy using two hidden layers with the same total number of neurons, in roughly the same amount of training time.\n",
    "\n",
    "+ For **more complex problems**, you can **ramp up the number of hidden layers until you start overfitting the training set**.\n",
    "\n",
    "\n",
    "+ **Very complex tasks** (large image classification or speech recognition) typically require networks with dozens of layers (or even hundreds, but not fully connected ones), and a huge amount of training data. **It is much more common to reuse parts of a pretrained state-of-the-art network that performs a similar task. Training will then be a lot faster and require much less data.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Number of units per hidden layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='green' size=3>1. for input/output layers</font>**\n",
    "\n",
    "The number of neurons in the input and output layers is **determined by the type of input and output the task requires**.\n",
    "\n",
    "<br>\n",
    "\n",
    "**<font color='green' size=3>2. for hidden layers</font>**\n",
    "\n",
    "+ It **used to be common** to size them to form a pyramid (金字塔), with fewer and fewer neurons at each layer —- the rationale (基本原理/根本原因) being that many low-level features can coalesce (合并、结合) into far fewer high-level features.\n",
    "\n",
    "\n",
    "+ **In practice, the above is abandoned.** It seems that **using the same number of units in all hidden layers performs just as well in most cases, or even better**; plus, there is **only one hyperparameter to tune**, instead of one per layer. That said, depending on the dataset, it **can sometimes help to make the first hidden layer bigger than the others**.\n",
    "\n",
    "\n",
    "+ **Just like the number of layers, you can try increasing the number of neurons gradually until the network starts overfitting**. But **<font color='red' size=3>in practice, it’s often simpler and more efficient to pick a model with more layers and units than you actually need, then use early stopping and other regularization techniques to prevent it from overfitting.</font>** On the flip side (另一方面), **if a layer has too few units, it will not have enough representational power to preserve all the useful information from the inputs** (e.g., a layer with two neurons can only output 2D data, so if it processes 3D data, some information will be lost). **No matter how big and powerful the rest of the network is, that information will never be recovered**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>In general you will get more bang for your buck by increasing the number of layers instead of the number of neurons per layer.</b> 增加 layer 数比增加每层的 unit 数收益更好。\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Learning Rate, Batch Size, and Other Hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.1 Learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The learning rate is arguably the most important hyperparameter.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In general, the optimal learning rate is about half of the maximum learning rate** (i.e., the learning rate above which the training algorithm diverges, in Chapter 4)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='red'>One way to find a good learning rate</font>** is to train the model for a few hundred iterations, starting with a very low learning rate (e.g., $10^{-5}$) and gradually increasing it up to a very large value (e.g., 10). This is done by multiplying the learning rate by a constant factor at each iteration (epoch) (e.g., by $e^{log(10^6)/500}$ to go from $10^{-5}$ to 10 in 500 iterations). If you **plot the loss as a function of the learning rate (using a log scale for the learning rate)**, you should see it dropping at first. But after a while, the learning rate will be too large, so the loss will shoot back up: **the optimal learning rate will be a bit lower than the point at which the loss starts to climb (typically about 10 times lower than the turning point)**. You can then reinitialize your model and train it normally using this good learning rate.\n",
    "\n",
    "See in chap-11."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.2 Optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Choosing a better optimizer than plain old Mini-batch Gradient Descent (and tuning its hyperparameters) is also quite important.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.3 Batch size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**<font color='blue'>The batch size can have a significant impact on your model’s performance and training time.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ The **<font color='red'>main benefit of using large batch sizes</font>** is that **hardware accelerators like GPUs can process them efficiently** (see Chapter 19), so the training algorithm will see more instances per second. Therefore, many researchers and practitioners recommend using the largest batch size that can fit in GPU RAM. There’s a catch, though: (**<font color='red'>main weakness of using large batch sizes)</font>** in practice, large batch sizes often lead to training instabilities, especially at the beginning of training, and the resulting model may not generalize as well as a model trained with a small batch size.\n",
    "\n",
    "\n",
    "+ In April 2018, Yann LeCun even tweeted **“Friends don’t let friends use mini-batches larger than 32,”** citing a [2018 paper](https://homl.info/smallbatch) which concluded that using small batches (from 2 to 32) was preferable because small batches led to better models in less training time. (**<font color='red'>main benefit of using small batch sizes</font>**)\n",
    "\n",
    "\n",
    "+ [2017 paper](https://homl.info/largebatch) and [2017 paper](https://homl.info/largebatch2) showed that <font color='red'>it was possible to use very large batch sizes (up to 8192) using various techniques</font> such as **<font color='red'>warming up the learning rate (i.e., starting training with a small learning rate, then ramping it up)</font>** (see in Chapter 11). This **led to a very short training time, without any generalization gap**.\n",
    "\n",
    "\n",
    "**<font color='red'>One strategy is to try to use a large batch size, using learning rate warmup, and if training is unstable or the final performance is disappointing, then try using a small batch size instead.</font>**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.4 Activation function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ In general, the ReLU activation function will be a good default for all hidden layers. \n",
    "\n",
    "\n",
    "+ For the output layer, it really depends on your task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3.5 Number of iterations (epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In most cases, the number of training iterations **does not actually need to be tweaked: just use early stopping instead**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>The optimal learning rate depends on the other hyperparameters—especially the batch size—so if you modify any hyperparameter, make sure to update the learning rate as well.</b>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "367.6px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}