15_autoencoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Chapter 15 – Autoencoders**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains all the sample code and solutions to the exercises in chapter 15._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To support both python 2 and python 3\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"autoencoders\"\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple utility functions to plot grayscale 28x28 image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_image(image, shape=[28, 28]):\n",
    "    plt.imshow(image.reshape(shape), cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_multiple_images(images, n_rows, n_cols, pad=2):\n",
    "    images = images - images.min()  # make the minimum == 0, so the padding looks white\n",
    "    w,h = images.shape[1:]\n",
    "    image = np.zeros(((w+pad)*n_rows+pad, (h+pad)*n_cols+pad))\n",
    "    for y in range(n_rows):\n",
    "        for x in range(n_cols):\n",
    "            image[(y*(h+pad)+pad):(y*(h+pad)+pad+h),(x*(w+pad)+pad):(x*(w+pad)+pad+w)] = images[y*n_cols+x]\n",
    "    plt.imshow(image, cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PCA with a linear Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build 3D dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy.random as rnd\n",
    "\n",
    "rnd.seed(4)\n",
    "m = 200\n",
    "w1, w2 = 0.1, 0.3\n",
    "noise = 0.1\n",
    "\n",
    "angles = rnd.rand(m) * 3 * np.pi / 2 - 0.5\n",
    "data = np.empty((m, 3))\n",
    "data[:, 0] = np.cos(angles) + np.sin(angles)/2 + noise * rnd.randn(m) / 2\n",
    "data[:, 1] = np.sin(angles) * 0.7 + noise * rnd.randn(m) / 2\n",
    "data[:, 2] = data[:, 0] * w1 + data[:, 1] * w2 + noise * rnd.randn(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(data[:100])\n",
    "X_test = scaler.transform(data[100:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's build the Autoencoder..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: instead of using the `fully_connected()` function from the `tensorflow.contrib.layers` module (as in the book), we now use the `dense()` function from the `tf.layers` module, which did not exist when this chapter was written. This is preferable because anything in contrib may change or be deleted without notice, while `tf.layers` is part of the official API. As you will see, the code is mostly the same.\n",
    "\n",
    "The main differences relevant to this chapter are:\n",
    "* the `scope` parameter was renamed to `name`, and the `_fn` suffix was removed in all the parameters that had it (for example the `activation_fn` parameter was renamed to `activation`).\n",
    "* the `weights` parameter was renamed to `kernel` and the weights variable is now named `\"kernel\"` rather than `\"weights\"`,\n",
    "* the bias variable is now named `\"bias\"` rather than `\"biases\"`,\n",
    "* the default activation is `None` instead of `tf.nn.relu`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "reset_graph()\n",
    "\n",
    "n_inputs = 3\n",
    "n_hidden = 2  # codings\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "hidden = tf.layers.dense(X, n_hidden)\n",
    "outputs = tf.layers.dense(hidden, n_outputs)\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_iterations = 1000\n",
    "codings = hidden\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for iteration in range(n_iterations):\n",
    "        training_op.run(feed_dict={X: X_train})\n",
    "    codings_val = codings.eval(feed_dict={X: X_test})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure linear_autoencoder_pca_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARgAAADQCAYAAADcQn7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAE8RJREFUeJzt3W+MXNV5x/Hvs7PrxQ0hCgb5TWT8\nIiClKQ3BfrNRKrbBagSK2gjUUiWOSexgk+BI0LRSLIFq4shIfpFaKEAx4o9NUiIUQCFJUVUIlqBe\nKTVySERT0SZgoIgG3JBgF//dpy/O3O74ev7cmb1nzp07v480Wu/MnZkzu57fnvOcM+eauyMiEsNE\n6gaISH0pYEQkGgWMiESjgBGRaBQwIhKNAkZEolHAiEg0ChgRiUYBIyLRTKZuwCDOO+88X7lyZepm\niIyt55577i13P7/XcSMZMCtXrmT//v2pmyEytszsYJHjNEQSkWgUMCISjQJGRKJRwNTA3Bzcdlv4\nKlIlI1nklQVzc3D55XD8OCxZAk89BTMzqVslEqgHMwK69VD27g3hcupU+Lp377BbJ9KZejBDNjcX\nQmB29syeRrvbevVQZmfD9dnts7PDeBUixShgOugWBIt5zE5h0Xrb5CR84Quwbl37Hkpre2ZmwuOU\n3VaRMihg2ohV1+gWFq23nToFd98Nu3fDzp29eygzMwoWqSbVYNqIVdfIhjONxplhkd1mFr53D899\n6FAIuG3bygs6zTrJsKgH00asuka34Ux22549cN99Idyy5y6zh6JZJxkmBUwbMesa3cIiuy2rvcSo\nqfSq6YiUSQHTQcq6Rszn1qyTDNPYB0yZs0UxZp7KplknGaakAWNm08CdwBrgXOCXwBZ3f2IYz19m\nPWKUahuadZJhST2LNAm8ClwGvA+4GXjYzFYO48nLnC1azGMNe1an6PNptkkWK2kPxt2PAFtbrvqh\nmb0ErAJejv38ZdYjBn2sTgvsYvUwiva0ei0K1BBLiqhUDcbMlgMXAS+0uW0jsBFgxYoVpTxfmfWI\n1sdatmyhB9PrMTstsIs1xCo6i9TpuNbgaTRg/fq4gSgjzt0rcQGmgCeBu3sdu2rVKq+qffvcly51\nbzTC1337ih1v5h6W14X7bt+etn2djtu+PVyXtdWs2OvMP/b27f3dR6oF2O8F3teV6MGY2QTwIHAc\n2JyiDflu/6DDgH7XmXRbYBdD0V5bp+OyoeDRo1nE9LeeZpSK4VKCIikU8wIYcD/wNLC0yH3K7sHk\n/1rffXd/vZBuj1XHv+z79rlff7379HT/r7O1BxSzpyZxMUI9mLuADwFr3P3dFA3I9zoeeWTw1a6L\nqesMY/q4jALtYlYca6HfeEm9DuYCYBNwDHjDsk/6wSZ3/86w2pH/T3/JJfDjH8PExGBvgqquM+ln\nBqlIcAz6Oq+9NnxVcbj+Uk9THyQMkZLKzwDdeCPMz4dZkp076/MmKFIfilkjmZuDyy6DEydgaioE\njNRb6oV2lTEzA1u2hO0Rjh8PATM/DwcO1GexWbftIjIxt+DcsSOEC4SvGzZosV/dVaEGUymtw6XJ\nydNndkZ9xqNIfShmjeT110///he/CI/fqcalGafRpx5MzsxMGBZdfjlcccXCArg6bKhdpLaShVCZ\nG1xlNmw487rjx8MUfTva0Hz01bYHM+hsydxcqMFkPZhGI1w/6jMe/fQGYhWpN24MX7dtg9de6328\nZpxGXy0DZjFd69a/mgDXXQcrVoz+526qstFUFjKbNi1c99GPtj9WW0uMvloOkRbTtc4XQtetC8Vf\nGO1iY5EC77AcOhSWAED4euhQ52Oz4rvCZTTVsgezmK51u7+adSg2Vqk3MDsL09Ma+oyDWgZMtzdT\na20GOm/A3bo1wdatcOxYmLYuMryo6nYGVVkAWKWwk7hqGTDQ/s2U33vFvfsUdHZ8Fi5FVvbWobcz\nDFUJO4mrljWYTvK1mRMnutdpsuOzcFmzpndgaGpVZMFYBUy+0Dk11b3o2Xr89HQYKvX6q1ulYqpI\narUdIrWTH/tD9zrAILUC1RdEFljY2mG0rF692vfv35+6GZUt5orEZmbPufvqXseNVQ+mTCrmivQ2\nVjWYMqmYK9KbAmZAKuaK9KYhUot+aioq5or0poBpGqSmosViIt0lHyKZ2WYz229mx8zsgVTtWGxN\nRTuviZypCj2Y14FvAJ8ElqZqxGI+IKkZJZH2kgeMuz8KYGargQ+kasdiaipV2WtFpGqSB0xRMc5N\nnTdoTUU7r4m0NzIB4+67gF0QVvImbs5pivR+Os1QaTWw1NnIBEzVdev9dKrRqHYjdZd8FmmUFZ05\n6jRDpdXAUnfJezBmNtlsRwNomNlZwEl3P5m2Zd310/voVKNR7UbqLnnAADcDf9vy/VrgVmBrktYU\n1O/MUbvzMWs1sNRd8oBx961UPEzaKdr7yPd08udj1mpgqbPkATOqivY+tEZGxpkCZhGK9D5UZ5Fx\npoCJrLWns2zZwkyRejEyDhQwQ5CFida8yLjROpgh0ZoXGUcKmCHRDngyjgoNkcxsCXAYmOpwyGPu\nflVpraohrXmRcVS0BjMFrG9z/U3ApcAPSmtRjWnNi4ybQgHj7keAb7deZ2Y7COHyVXe/P0LbRGTE\n9T2LZGYG3A7cANzg7neW3ioRqYW+irxmNkHYk+XLwIYsXMxs2szuMbNfmdk7ZvaimX0lQntFZIQU\n7sGYWQPYDVwDrHX3h3KP8wbwJ8CvgD8E/snM/tvdHy6xvSIyQgr1YMxsCvgu8OfANblwwd2PuPst\n7v6f7j7v7j8FHgc+XnqLRWRk9AwYM5sGHgU+BVyVbdLd4z5TwB8BP1t0C0VkZBUZIu0hhMsDwPvN\nbG3u9sfd/Xe5674FvNO8r4iMqa4B05wxuqL57eebl1bzwHtz9/kmMAN8wt2Pl9JKERlJXQPG3R04\np+iDmdlO4HJCuLy1yLaJyIgr7dPUZnY78Angj939zbIeV0RGVykfdjSzC4CvAB8EXjKzw83LEwXu\ne66ZPWZmR8zsoJl9pow2iUh6pfRg3P0gYAPe/Q7gOLAcuAT4kZk97+4vlNE2EUkn6XYNZvYe4Grg\nFnc/7O7PEtbPfC5lu0SkHKn3g7mIcA6kF1uuex74cP5AM9toZvvNbP+bb6rEIzIKUgfM2UB+Dc1v\nyU19Qzg3tbuvdvfV559//lAaVwVFzx4pUkWp9+Q9zJnT4OcQFumNPZ27WkZd6h7Mi8CkmV3Yct1H\nABV4Wdw+vur5SBUk7cG4+xEzexT4upl9kTCL9GfAx1K2qyoGPaeSej5SFal7MBD2llkK/Bp4CPiS\npqiDbB/fbdv6CwmdwUCqInUNBnf/H+DTqdtRVfl9fOfmem8crrNJSlUkDxgprujQZ2YGdu6ERx6B\nq6/W8EjSUcCMkHZDn3bhMTcHN94YjnnmGbj4YoWMpFGFGowUVPTkbarBSFWoBzNCip68rZ8aTJGa\njsigFDAjpsjJ24oGkaazJTYFTE0VCaKiNR2RQakGM8aK1nREBqUezBgrOpQSGZQCZswVGUqJDEpD\nJBGJRgEzpvRpaxkGDZHGkKanZVjUgxlDWukrw6IezJhoXbGrT1vLsChgxkC7IZGmp2UYNEQaA61D\nomPHYOvWcP2WLQoXiUsBMwayIdHEBMzPw5NPhh5N0RkkzTjJoFKfeG1z81xHx8zsgZRtqbNsxe6a\nNQsh06u4m4XKrl0hjG65pb9QEoH0NZjXgW8AnyTsyyuRzMyEodEzz7Qv7rYWgWGhZmMWAqk1lDSs\nkqJSn1XgUQAzWw18IGVbxkGnzx7li8DXXrtQs5mYCB+GNNOMk/QvdQ+mMDPbCGwEWLFiReLWjK52\nnz3Kr4uB06exd+6EQ4c04yT9G5mAcfddwC6A1atXe+Lm1Ep+Xcy6deGiaWxZrGgBY2Z7gcs63Pwv\n7v7xWM8t/ek0dFKwyGJFCxh3n4312FK+frdtiLGXr/YHrp+kQyQzm2y2oQE0zOws4KS7n0zZLuku\nxocl9QHMekq90O5m4F3ga8Da5r9vTtoi6SnGhyX1Acx6Sj1NvRXYmrIN0r8YH5bUBzDraWRmkaQ6\nYuzlq/2B60kBI5Wh/YHrRwEjfVNBVopKXeSVEaSCrBSlgJG+6YRtUpSGSNI3FWSHa5QXICpgZCAp\nCrKj/EYbVJF6V5V/LgoYqZy5OdizJ/x73brwphnXwnK7elfr6676z0UBI5UyNxf+EmfbRtx/Pzz9\ndO83Wl31WoC4mJ/LMHo+ChiplL174cSJhe+zN032Rjt2LGx+9fbbYUvPKg4LytRtk7C9e2HZssFW\nQA+r56OAkUqZnYWpqdM3vsreWDt3wubNcPIk7NgRdtubnq7esKBs+XpXPhwG2RBsWD1CBYxUysxM\n+M+er8FAeBPNz4M3txvLb14+7EJnquJqPhwOHQqnoMk2ai/SnmF99ksBI5XTaYaqdZg0Px96MEuW\nhGHCsAudKYur7cKh3/YMa6mBAkZGRuubYtmyhWFBigJwjOcs2iNqFw633dZ/e4ax1EABIyOl05ti\n2Fs99DvE6BUenXogne6X/znMzsLkZOjZTU4O9jOIMeRTwMjIi93db/fG6+c5iwxfOn2+q2jo/Pzn\nofjtvlCj6vc1xhjyKWCkFsro7rdO/WbDL+j8xiv6nPnw2LPnzIBo1yMqGjoQZtdOnQr/Pnmy//Uw\nW7cu1LbKHGYqYERY+AveWkCenj79JHSDvvFaw2NyEu67Lzxea2Dl60ud1ri0C51XXgmhkpmY6H89\nTL5wXtYwM1nAmNk0cCewBjgX+CWwxd2fSNUmGV/ZG3d+Pnyf/SWHYrWWbvWL1vD4yU/g+98Pw5h8\nYGVfe61xaW3PsmVw660Lw6JGA+64o//1MFm4rFkTejN1qMFMAq8Szp30CnAl8LCZXezuLydsl4yh\ndlPgk813R6+FbEXqF9n3+TDotfQ/W+PS+jhPPbWwTujAgYXeixlcdx1s3Nj/687aXma4QMKAcfcj\nnL7h9w/N7CVgFfByijbJ+GrtZbz9dvh64ADcc0/vomfRKeu9e08Pg/Xrzzyu6OzU7t0LQ65GI1y3\nZElYmNjPbFDsAnllajBmthy4CHihw+0b0bmpJYLWN+TsbOiNHD260NPoVXspGgrtTtGb164Wk12f\naQ00CL2WFSt6F6U7iboext2TX4Ap4Eng7iLHr1q1ykXKsG+f+9Kl7o1G+Hr99eHf2YSvWbh+375w\n2b49fG33OJ1uG/S41na1Ht/ttu3bF9rfaITvYwD2e4H3avJzU5vZBPAgcBzYHKs9Iu3khzew0Mto\nNMIwJutpdOoZ9Dsk6XVMr2njbsOaqp1fKum5qc3MgHuB5cCV7n6ix11EStVu2LJu3Zlv3k5L8Qdd\noNYplIpOG3cKqqptZ5q6BnMX8CFgjbu/m7gtMoY6vSGLFl97FXjbBUm3UCpj2rhK55dKuQ7mAmAT\ncAx4I3RmANjk7t9J1S4ZP0XekJ2CqNuQpFOQdAul2NPGw5ZymvogYD0PFKmIdkHUbUjSKUi6hVLV\nhjiLlXqIJDLyeu1fkw+SXiFSpSHOYilgRCLpFiR1CpFuFDAiEY1LkHSiU8eK1Ey2N+/cXOqWqAcj\nUitVOxGbejAiNdJpk6pUFDAiNZLNXDUaNf+ogIgMX9XW0ShgRGqmSjNXGiKJSDQKGBGJRgEjItGY\nD3KWpsTM7E3gYOp25JwHvJW6EUOm1zwe2r3mC9z9/F53HMmAqSIz2+/uq1O3Y5j0msfDYl6zhkgi\nEo0CRkSiUcCUZ1fqBiSg1zweBn7NqsGISDTqwYhINAoYEYlGASMi0ShgSmJm02Z2r5kdNLN3zOyn\nZnZF6nbFYGbnmtljZnak+Xo/k7pNMY3T77YdM7vQzI6a2bf7va8CpjyTwKuE0+W+D7gZeNjMViZs\nUyx3EE71uxz4LHCXmX04bZOiGqffbTt3AP86yB01ixSRmf0MuNXdH0ndlrKY2XuA3wB/4O4vNq97\nEPgvd/9a0sYNUR1/t+2Y2V8CVwH/BnzQ3df2c3/1YCIxs+XARcALqdtSsouAk1m4ND0P1LkHc5oa\n/25PY2bnAF8H/mrQx1DARGBmU8B3gN3u/u+p21Oys4Hf5a77LfDeBG0Zupr/bvO2Afe6+2uDPoAC\npiAz22tm3uHybMtxE8CDhBrF5mQNjucwcE7uunOAdxK0ZajG4Hf7/8zsEmAN8HeLeRxtmVmQu8/2\nOsbMDLiXUPy80t1PxG5XAi8Ck2Z2obv/R/O6j1D/4cI4/G5bzQIrgVfCS+dsoGFmv+/ulxZ9EBV5\nS2Rmfw9cAqxx98Op2xOLmX0XcOCLhNf7j8DH3L22ITMuv9uMmf0ep/dU/5oQOF9y9zeLPo6GSCUx\nswuATYT/hG+Y2eHm5bOJmxbDl4GlwK+Bhwj/6eocLuP0uwXA3f/X3d/ILoSh8dF+wgXUgxGRiNSD\nEZFoFDAiEo0CRkSiUcCISDQKGBGJRgEjItEoYEQkGgWMiESjgBGRaBQwEoWZLTGz410+gf5o6jZK\nfPo0tcQyBaxvc/1NwKXAD4bbHElBn0WSoTGzHcDfAF9192+mbo/Epx6MRNfcS+V24AbgBne/M3GT\nZEhUg5GomrvA7SJs8bChNVzM7C/M7Nnm1gcvp2qjxKMejERjZg1gN3ANsNbdH8od8hvgW4Rd4m4a\ncvNkCBQwEkVzc+x/AP4UuMbdz5g1cvd/bh776SE3T4ZEASOlM7Np4HuETaOvcvcfJW6SJKKAkRj2\nAJ8CHgDeb2b5k3U97u75U59IDSlgpFTNGaPsvM2fb15azTMm51ASBYyUzMPCqvx5k2RMKWAkmeYs\n01TzYmZ2FiGjjqVtmZRFASMpfQ64v+X7d4GDhPPvSA3oowIiEo1W8opINAoYEYlGASMi0ShgRCQa\nBYyIRKOAEZFoFDAiEs3/AQcsVH7eWduMAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116547be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(4,3))\n",
    "plt.plot(codings_val[:,0], codings_val[:, 1], \"b.\")\n",
    "plt.xlabel(\"$z_1$\", fontsize=18)\n",
    "plt.ylabel(\"$z_2$\", fontsize=18, rotation=0)\n",
    "save_fig(\"linear_autoencoder_pca_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacked Autoencoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train all layers at once"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (ie. 2 stacked Autoencoders). We will use ELU activation, He initialization and L2 regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: since the `tf.layers.dense()` function is incompatible with `tf.contrib.layers.arg_scope()` (which is used in the book), we now use python's `functools.partial()` function instead. It makes it easy to create a `my_dense_layer()` function that just calls `tf.layers.dense()` with the desired parameters automatically set (unless they are overridden when calling `my_dense_layer()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0001\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "he_init = tf.contrib.layers.variance_scaling_initializer() # He initialization\n",
    "#Equivalent to:\n",
    "#he_init = lambda shape, dtype=tf.float32: tf.truncated_normal(shape, 0., stddev=np.sqrt(2/shape[0]))\n",
    "l2_regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "my_dense_layer = partial(tf.layers.dense,\n",
    "                         activation=tf.nn.elu,\n",
    "                         kernel_initializer=he_init,\n",
    "                         kernel_regularizer=l2_regularizer)\n",
    "\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3 = my_dense_layer(hidden2, n_hidden3)\n",
    "outputs = my_dense_layer(hidden3, n_outputs, activation=None)\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "loss = tf.add_n([reconstruction_loss] + reg_losses)\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver() # not shown in the book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train it! Note that we don't feed target values (`y_batch` is not used). This is unsupervised training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0204011\n",
      "19% Train MSE: 0.0114192\n",
      "2 Train MSE: 0.0102221\n",
      "3 Train MSE: 0.00989991\n",
      "4 Train MSE: 0.0103724\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\") # not shown in the book\n",
    "            sys.stdout.flush()                                          # not shown\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})   # not shown\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)           # not shown\n",
    "        saver.save(sess, \"./my_model_all_layers.ckpt\")                  # not shown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function loads the model, evaluates it on the test set (it measures the reconstruction error), then it displays the original image and its reconstruction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_reconstructed_digits(X, outputs, model_path = None, n_test_digits = 2):\n",
    "    with tf.Session() as sess:\n",
    "        if model_path:\n",
    "            saver.restore(sess, model_path)\n",
    "        X_test = mnist.test.images[:n_test_digits]\n",
    "        outputs_val = outputs.eval(feed_dict={X: X_test})\n",
    "\n",
    "    fig = plt.figure(figsize=(8, 3 * n_test_digits))\n",
    "    for digit_index in range(n_test_digits):\n",
    "        plt.subplot(n_test_digits, 2, digit_index * 2 + 1)\n",
    "        plot_image(X_test[digit_index])\n",
    "        plt.subplot(n_test_digits, 2, digit_index * 2 + 2)\n",
    "        plot_image(outputs_val[digit_index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_all_layers.ckpt\n",
      "Saving figure reconstruction_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAGoCAYAAACXNJbuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAHQlJREFUeJzt3VuMX1X5N/Bd2jn3RIHpAEWRqlAl\nGEBoOXkMh8QIUYNGudAgicREEknUQGKi3nipXGjQaADRqFGDBhAMEFBUjpWTiFApB6GHKYW0086h\nMy3/m/dNfN//epbdv5nOPNN+PpfPr3utPXs6v2928qy1Frz55psNADC3DpvrGwAABDIApCCQASAB\ngQwACQhkAEhAIANAAgIZABIQyACQgEAGgAQWzdG8tgdjvlsw1zdwqNiyZYvvC+a1oaGh/fq+8IYM\nAAkIZABIQCADQAICGQASmKumLgBm0YIF5b4iR/Dm4Q0ZABIQyACQgEAGgAQEMgAkIJABIAGBDAAJ\nWPYEQNHk5GSx3tXVVazXllBFy64itbEOO6z8Ljnfl3B5QwaABAQyACQgkAEgAYEMAAkIZABIQJc1\nQEXb7uCZnKM2d9RR3PZ+o07q2hwTExPFek9PTzhW1Bm9Z8+eVv++aZpm3759xfps/K4OJG/IAJCA\nQAaABAQyACQgkAEgAYEMAAnosgao6KQDuu1YkVqncdQBHXVN9/X1Fevj4+PhHNFn0c+xd+/ecKyF\nCxe2GmvRojieorFq80dmqlt9JnhDBoAEBDIAJCCQASABgQwACQhkAEhAIANAApY9AVREBxl08u+j\nz6ampor1aHlP0zTNwMBAsd7f399q7sWLF4dzRPOPjY21+ve1+4oOpGj73GvzRwdYNE28tCxa9tTJ\nfe0vb8gAkIBABoAEBDIAJCCQASABgQwACeiyBg4Z0UECUb1p4gMLok7jWkfvzp07i/WJiYli/aij\njgrHikTdzFFXdm9vbzjW8uXLi/XoZ4wOtmiauJs5er61ju1I7ffY9hqHSwDAIUogA0ACAhkAEhDI\nAJCAQAaABHRZ/4cHH3ywWL/uuuvCa4499thiva+vr1j/7Gc/G461YsWKVnWgLOqcjfYhjrqcmybu\npn7llVeK9Zdeeikc6+WXXy7Wo+7kVatWhWNFHdiLFpW/1t/ylrcU64cffng4R7TH9sjISLFee47R\nfUU/e/Qd2jTxzx7ty93d3R2OFf1+I7VO7ul2ZntDBoAEBDIAJCCQASABgQwACQhkAEhAIANAAgs6\n2Yx7BszJpP/NiSeeWKxv2LBhVuZftmxZsb5u3bpZmf9AO/7448PPrrnmmmI9WqqRwOzvPH+I2rJl\nS+vvi2i5TrQsp7ZcJ1qq9MILLxTrTz75ZDjWpk2bivXo4IUjjjgiHCu65+i+Vq5cWaxHz6pp4uVN\n0VKhY445JhwrOtwimr+2HOuiiy4q1k899dRivbaEKnqO0e+kE0NDQ/v1feENGQASEMgAkIBABoAE\nBDIAJCCQASABh0v8h9/+9rfF+uOPPx5e8+53v7tYf/rpp4v1hx56KBzrd7/7XbH+hz/8oVh/29ve\nVqxHXZadiDaEb5qmOfroo4v1f//7363niTqwv/a1r7UeCyLR5v8LFy4Mr4m6g6N6rTO6q6ur1fxL\nliwJx4q6v6Pu4LaHKDRN02zevLlYHx8fL9ZXr14djjU8PFysP/vss8V6bYXFSSedVKy/4x3vKNb7\n+/vDsWaym3q68twJABzCBDIAJCCQASABgQwACQhkAEjAXtaJRJ2LL774YrEedVlv3Lhxpm6p6e7u\nDj+Luqyj+9q2bVs41i233FKsX3LJJZW7m1P2sp4lnexlHYn2Ta59D0Z7He/atatYf+ONN8Kx9u7d\nW6xH+9jX9tiOVlNMTk4W68cdd1yxvn379nCO559/vliP/saj74SmaZrbbrutWP/FL37Reqwrrrii\nWI/2uF66dGk4VvQ7ieqdsJc1AMwjAhkAEhDIAJCAQAaABAQyACQgkAEgAYdLJNLb21usRxupR9as\nWTMTt/NfRQdlvPbaa8X62rVrw7EuuOCCGbknqImW8dWWPUUHQkRLlY466qhwrOgQiei+du/eHY41\nNDRUrEeHXkQHVdSWPZ1xxhnFenSAxsjISDhWdIjDvn37ivXawRqrVq0q1vv6+or12u83Wt4UXRMd\nUDITvCEDQAICGQASEMgAkIBABoAEBDIAJKDLmqpal+fHPvaxYj3qmvzud78bjhV1R0Inok7YqB51\nANc+i7pzaweyRB3b0X1FHdNN0zSLFy9uNX90vz09PeEcy5cvb3Vf0WEUTdM0zz33XLEedVOvW7cu\nHOutb31rsR51sdc6o6Pfb9RlfSAPZPKGDAAJCGQASEAgA0ACAhkAEhDIAJCALmuqbrzxxvCzLVu2\nFOvRPrdRZyTMtKgTNuo0jrpzm6ZpJicni/Vo7/k9e/aEY42NjbWaf3x8PBwrEt1XtPoheiZNE3cg\nv/TSS8X6r371q3Csxx9/vFiPuqnPP//8cKxjjz22WG+7X3bTxB3YtWsOFG/IAJCAQAaABAQyACQg\nkAEgAYEMAAnosqZpmngP2quvvrr1WA888ECxPjQ01HosmGtR527UMT0xMRGOFV0T7T8ddUw3TdMs\nWlT++o7mj7qGo/2qm6ZppqamivU//vGPxfrtt98ejhV1M69du7ZYP+6448Kxoq70mewkr11zoHhD\nBoAEBDIAJCCQASABgQwACQhkAEhAIANAApY90TRN09x6663FerSxftM0zaWXXlqsn3DCCTNyTzBb\nosMomiZe/hLVa4dLRJ91suwpWvoTLa3q6ekp1qPlSE0TH27x5JNPFuubN28Ox7rooouK9bPPPrtY\nX7p0aThW9OyjZU+1nzF6jrXvvgPFGzIAJCCQASABgQwACQhkAEhAIANAArqsDzFR5+Att9xSrEed\nmU3TNN/+9reL9ahrEeZa1E3dSZd1JOqYbpqmWbx4cbHe1dXV+r527NhRrEed0f39/cV6rZs46qZ+\n6qmnivWTTjopHOucc84p1letWlWsR8+kaeKfcSa/e6JnX+vYni5vyACQgEAGgAQEMgAkIJABIAGB\nDAAJ6LI+xPz4xz8u1u+///5i/TOf+Uw4lj2rmW+iDtlaN3N0TdR9Xeuy7uvrK9anpqaK9dHR0XCs\n6JolS5YU61GX9aZNm8I5fv/73xfrGzZsKNYvvPDCcKy1a9cW69Ge1RMTE+FYbTvfax3b0XPUZQ0A\nhyiBDAAJCGQASEAgA0ACAhkAEhDIAJCAZU8Hoccffzz87Etf+lKxvnz58mL9W9/61ozcE2QWLX2p\nWbSo/PVZO5Alumbfvn2t5x8YGCjWo+VN0c945513hnP85je/aTX3Bz/4wXCsoaGhYj06EKK27Kmt\nTpa1zQVvyACQgEAGgAQEMgAkIJABIAGBDAAJ6LKex8bGxor1T3/60+E10absl112WbHuAAkOJlG3\nbe2wgqgzOlLrmN69e3ex3kmXd3SIxWGHld+zHnvssWL9pptuCufYtm1bsX7xxRcX66eccko4VnTo\nRfTso+7rpmmaPXv2FOtRh3vtdzg+Ph5+Ntu8IQNAAgIZABIQyACQgEAGgAQEMgAkoMt6Hoi6Nj/y\nkY8U688++2w41po1a4r1b37zm+1vDOaZqMu6tp9x9FnUzRytfmiauKM4+hvv7e0Nx4o6h1966aVi\n/Xvf+16xvn79+nCO8847r1j/5Cc/Wawfd9xx4VjRsx8dHS3Wa53nbbupa3tZR6Lf74HkDRkAEhDI\nAJCAQAaABAQyACQgkAEgAYEMAAlY9jQPvP7668X6fffd13qsm2++uVhfsWJF67HgYFFb9tR2qVJN\ndCBEdJBCf39/OFa0LOi2224r1h9++OFi/fTTTw/nuOqqq4r19773vcV67Zns2rUr/KyktuSrq6ur\nWI+eSW3ZU+13P9u8IQNAAgIZABIQyACQgEAGgAQEMgAkoMs6kR07dhTr69atazXOT3/60/CzU089\ntdVYcCiIupybpmkmJyeL9aijOOoArs0TXVPrDn7qqaeK9bvuuqtY37p1a7H+0Y9+NJwj6sDu5GCN\n6Dm27Txvmvbd1J10xHdyEMl0eUMGgAQEMgAkIJABIAGBDAAJCGQASECXdSI33HBDsb5x48ZW45x7\n7rnhZ5n2bYXZFnUH10Qd0FHXcK3LOvosuq+oM7ppmubWW28t1h988MFiPdob+oQTTgjnWLJkSbEe\ndTlHz6Rp4p8x6rLuZP/p2jVtx5oL3pABIAGBDAAJCGQASEAgA0ACAhkAEhDIAJCAZU+zbMOGDeFn\n3/jGN2bvRoD9Ei2L6enpKdZrS6uisUZGRor1P/3pT+FYf/3rX1vNsXbt2mL9+OOPD+fYu3dvsR4t\ne6pZtKgcN50c/BBdE9UzLW2q8YYMAAkIZABIQCADQAICGQASEMgAkIAu61l2//33h5/t3Lmz1Vhr\n1qwp1vv6+lqNA8SiDt2FCxcW67Uu66g7OfrbHxsbC8d6y1veUqyvWrWqWD/ppJOK9RUrVoRzTExM\nFOtR93VUb5r4eUXX1DqjOzlEYj7whgwACQhkAEhAIANAAgIZABIQyACQgC7reeDss88u1u+6665i\nXZc1HHjRvsm1TuOoa3l8fLxYr+0zHXUtR53cUVf24OBgOEe0/3QnXc7RNTPZMT1f9qyOeEMGgAQE\nMgAkIJABIAGBDAAJCGQASEAgA0ACC+Zok+6Dc2dwDiXze33FPLJly5aD5vsi+r6NllBF9U7miOrd\n3d3hWNGyp2hpVy1P5vuSpOkYGhrarx/eGzIAJCCQASABgQwACQhkAEhAIANAAnPVZQ0A/AdvyACQ\ngEAGgAQEMgAkIJABIAGBDAAJCGQASEAgA0ACAhkAEhDIAJCAQAaABAQyACQgkAEgAYEMAAkIZABI\nQCADQAICGQASEMgAkIBABoAEBDIAJCCQASABgQwACQhkAEhAIANAAgIZABIQyACQgEAGgAQEMgAk\nIJABIAGBDAAJCGQASEAgA0ACAhkAEhDIAJCAQAaABBbN0bxvztG8MFMWzPUNHCqGh4d9XxyCFiyI\n/8TefHN+/ZcYHBzcr+8Lb8gAkIBABoAEBDIAJCCQASCBuWrqAuD/iBqYas1LtaanNjppkJrJpqrD\nDvNe+H95EgCQgEAGgAQEMgAkIJABIAGBDAAJCGQASMCyJ+CQN5PLi2pjRUt8omtqY+3bt69VPZp7\n0aI4Bvbu3VusR89kamoqHCuaP5qjthwq+hnbzp3N/LhLADjICWQASEAgA0ACAhkAEhDIAJCALutZ\n9rOf/Sz8bPfu3cX6+vXri/Uf/vCHref/+te/Xqx/6EMfKtY/8IEPtJ4D5ptaJ3XbDuioa7hp4i7k\niYmJYn18fDwca+fOneFnJVGncTR30zTNwoULi/Wurq5ifWBgIByrr6+vWO/t7Q2vaSvqGO/kMIyZ\nOryjDW/IAJCAQAaABAQyACQgkAEgAYEMAAks6KT7bAbMyaSz6Ytf/GKx/oMf/GCW72T/vOtd7yrW\n//znP4fXLFu27EDdznww+y2Yh6jh4eED/n1R+x6MuqajDujR0dFwrOHh4WL92WefLdYfffTRcKx/\n/vOfxfqWLVuK9cHBwWK99rP39/cX62vWrCnWo9UaTdM0p512WrF+1FFHFeu1/aqje446xjvpoo+u\n6SQzBwcH9+v7whsyACQgkAEgAYEMAAkIZABIQCADQAICGQAScLjENM3G8qZTTz21WP/EJz5RrG/Y\nsCEc66abbirW//GPfxTrv/71r8OxPv/5z4efQUbRUpraEptoeVNU37ZtWzjWQw89VKzfe++9xfoT\nTzwRjjU5OVmsr169ulhfvnx5sb506dJwjueee65Y37p1a7G+ffv2cKzocItODoSIljdF1+zZsycc\nq7u7O/ys7X1N90AKb8gAkIBABoAEBDIAJCCQASABgQwACeiy3g8vv/xy+NmPfvSjVmOdccYZ4Wd3\n3nlnsR5t8B51B0ab4TdN0/zrX/8q1v/yl78U66+99lo4Fsw3nXTBRl3AUZfzwoULw7Gizuzomgsu\nuCAc68wzzyzWzzvvvGL98MMPL9affPLJcI6f//znxfobb7xRrL/++uvhWFGn88jISHhNJOqKr3VT\nR8bGxor1vr6+1mNNlzdkAEhAIANAAgIZABIQyACQgEAGgAR0We+HWqdxtK9p1E199913h2MtXry4\n3Y0FbrzxxvCzRx55pNVYl1xyyTTvBvKI9lOu7U8cdVlHHdvRPstN0zRDQ0PF+llnnVWsf/jDHw7H\niva4j1Zl7Nq1q1h/9dVXwzk2btxYrEed0bUVHl1dXcV6Jx3T0TPu6elpNXdtrGj+6P9D09jLGgAO\nCgIZABIQyACQgEAGgAQEMgAkIJABIAHLnvbDaaedFn4WLYmKDn6YjQ3LawdedLL5Osw30fKTaPlL\n7UCIqampYn3lypXFerTsqHZfp5xySqt608T3HC1Jevrpp4v1O+64I5wjuiZapnXCCSeEYx155JHF\nevS8tm/fHo4VLVOLvl9ry5GiQ0Kia6a7tKnGGzIAJCCQASABgQwACQhkAEhAIANAArqsp2nZsmVz\nNvfNN99crD/xxBOtx7rggguK9dWrV7ceC+ZadIhE2+7r//ZZSW9vb+uxOjn0YnR0tFjfunVrsf7w\nww8X6y+88EI4R9RJ/p73vKdYP/HEE8Oxoq7wsbGxYn18fDwcKzrEIuqYHhgYaH1f0aEXtd/JdHlD\nBoAEBDIAJCCQASABgQwACQhkAEhAl/U88NhjjxXrX/jCF4r1iYmJcKyjjz66WL/uuuuK9bYdpjBb\nansKt+2ErY0VdeFGnb61sRYtKn/lRh29tT22d+/eXaxv2bKlWH/++efDsSLR3tRnnnlmsV7bx3vz\n5s2t5q51q7d9XrWO7U463A8Ub8gAkIBABoAEBDIAJCCQASABgQwACQhkAEjAsqd54IEHHijWa8ub\nIldeeWWx/s53vrP1WJBVtGSlk6VK0bKYaOnN1NRUOFa07Kmnp6f1WNEhEvfee2+xfscddxTrS5cu\nDeeIljedfPLJxXptmWT0fRUtlaqNFf1+a7/HSPQ76UQn8/8nb8gAkIBABoAEBDIAJCCQASABgQwA\nCeiyTuTyyy8v1n/5y1+2GufLX/5y+NlXv/rVVmNBVrXN/9t2u9a6maN5oi7rqJO7aeIDE/bs2VOs\nR53UTdM0d911V7F+++23F+vRoRPr1q0L5zj33HOL9VWrVhXro6Oj4VgjIyPF+rJly4r12nPs5NlH\naodYlNT+3033QApvyACQgEAGgAQEMgAkIJABIAGBDAAJ6LKeZbt27Qo/i/aaHR8fL9ZXrlxZrF97\n7bXhHN3d3ZW7g4ND272sa92xUQd09Le0cOHC/3J3/1v0vbB+/frwmnvuuadYj7qp3//+9xfrF198\ncTjHaaedVqxHz6v2HKP9uiOTk5PhZ233n65970X/J6LO+1oHv72sAeAgIJABIAGBDAAJCGQASEAg\nA0ACuqxn2aWXXhp+Njw83Gqsq666qlhfsWJFq3HgUBHtgXzYYfG7SdTt29XVVazXOm2jju2dO3cW\n6xs3bgzHeuaZZ4r1o48+uli/6KKLivXTTz89nGNgYKBYj+63tpd01IEd7X8dzdE0TbN06dJWc9T2\nKm/bGT2Te6j//7whA0ACAhkAEhDIAJCAQAaABAQyACQgkAEgAcueDpBoU/j77ruv9Vgf//jHi/Wr\nr7669VhwKIuWrNQOMogOi4jqtYMPNm3aVKw/9NBDxfrdd98djrVt27Zi/ZxzzinWTz755GK9tozn\ntddeK9Z37NhRrNcOcYiecbRUqJNDOqLla7XlSG0Pl4iWu80Eb8gAkIBABoAEBDIAJCCQASABgQwA\nCeiynqaxsbFi/ZprrinWo83la6LN32sdjcD/FnXb1jqjo67a3t7eYr32Nz4yMlKsP/LII8X63/72\nt3CsoaGhYv1973tfsT44OFis17qZo07jww8/vFivdSBHndlRl3M0R9PE9xwdHhLN3TRxZ/aB7KaO\neEMGgAQEMgAkIJABIAGBDAAJCGQASECX9TRdf/31xfo999zTeqzLL7+8WLdnNbQTdVO3rTdN3NEb\ndQfXOnr//ve/t6rXOo0/9alPFesXXnhhsR51Zde6wkdHR4v1aIVHbeVH9Byj+ZcsWdL6vqK9t8fH\nx8Ox+vv7i/XoZ6n9X5kub8gAkIBABoAEBDIAJCCQASABgQwACQhkAEjAsqdpuvbaa2dsrO985zvF\nukMkoJ3okIHoIIGaaJlLtPRm69at4Vjr168v1rdv316sH3nkkeFYZ599drF+xBFHFOvRz97JYRgD\nAwPFeu2QjmjpUbQcatu2beFY0fKmF198sVivLR9bunRpsR793t98881wrOkuifKGDAAJCGQASEAg\nA0ACAhkAEhDIAJCALutEdu3aVax30hnaVk9PT7EedUA2Tby5/sTEROv5x8bGivXrrruu9ViR6Gep\ndcp3dXXN2PzMrE66XaN67W+s7eESte7g5557rljfsGFDsR4dCNE0TfPoo48W61GXddTlvHv37nCO\nTZs2FevRwQ+1AyEWL15crEe/k6effjoc65lnnmk1//nnnx+OFf0ea/+/DhRvyACQgEAGgAQEMgAk\nIJABIAGBDAAJ6LJO5Nhjj52zua+88spi/Zhjjgmv2bJlS7H+/e9/f0buabbUnvsVV1wxi3dCG53s\nGxztcR2tMmiauNu2tgIhEnXtR3st11Ys3HrrrcX6Pffc0+qeBgcHw89ef/31Yj3q/u7v7w/HWrZs\nWbH+yiuvFOvPP/98ONaKFSuK9TPPPLNYj1aw1ESd9/ayBoCDnEAGgAQEMgAkIJABIAGBDAAJCGQA\nSMCyp2m67LLLivUbbrhhlu9keq6//voDPseiRfF/t7ZLSD73uc+Fn5111lmtxjrnnHNa/Xvmr2hZ\nSrQcqiY6lCBaktM0TXPyyScX6zt27CjWR0dHw7FGRkaK9egAi6mpqWJ9+fLl4RwDAwPF+ubNm4v1\nVatWhWNFf+PR0q7aEqoTTzyxWH/7299erNeWdrX97pnu0qYab8gAkIBABoAEBDIAJCCQASABgQwA\nCSyobZR9AM3JpLPpJz/5SbG+Z8+eGZvjiSeeKNZn8nCHr3zlK+FnUUdj5OKLLw4/q3VBJnXgWi35\nfwwPD7f+vog6YaN6dJBA0zRNd3d3sT4+Pl6sRx3TTdM0r776arG+cePGYr3WZR11Uw8PD4fXlEQH\nSDRNfChD1Jm9evXqcKwlS5YU61EGRYdRNE383bNy5cpi/cgjjwzH6u3tLdajg0Bqq0Uig4OD+/V9\n4Q0ZABIQyACQgEAGgAQEMgAkIJABIAFd1tAZXdazpJMu60jUZR11Uteuib47a3sdT05OFuvRXtq1\njt5oL+2oa3r37t3FerQvde2aqGt5aGgoHKvtfuG1bIqeS09PT7Fe2xc7eo6ddFNHdFkDwDwikAEg\nAYEMAAkIZABIQCADQAICGQASsOwJOmPZ0yyZjWVPCxcuDK+JDp6oLW9qO//U1FSxXlt6Ex1+EC39\niURz1z6Lfo5oCVFtrGg5VG2saPlYJ0uYOjlwpC3LngBgHhHIAJCAQAaABAQyACQgkAEggZnbPRsg\nuWhVSa2jN/osGqu2ciXqKJ7Jju2JiYliPeoart3v2NhYsR51LXdyGEYn2na+155vJ8/+QPGGDAAJ\nCGQASEAgA0ACAhkAEhDIAJCALmvgkNfJnv7RNbWu3WjP7OiaaM/m2vzRntFRl3NtL+uZ7EBu22Ee\n/fummdl9pjM5OH8qAJhnBDIAJCCQASABgQwACQhkAEhAIANAApY9AXQgWq5TW0LV9prasqNoWVDb\nQyS6u7vDOTo5rKHtWJ3o5HnNB96QASABgQwACQhkAEhAIANAAgIZABJY0Mmm6gDAzPKGDAAJCGQA\nSEAgA0ACAhkAEhDIAJCAQAaABAQyACQgkAEgAYEMAAkIZABIQCADQAICGQASEMgAkIBABoAEBDIA\nJCCQASABgQwACQhkAEhAIANAAgIZABIQyACQgEAGgAQEMgAkIJABIAGBDAAJCGQASOB/AJrX8D+M\ncXeHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x12271fd68>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_all_layers.ckpt\")\n",
    "save_fig(\"reconstruction_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tying weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is common to tie the weights of the encoder and the decoder (`weights_decoder = tf.transpose(weights_encoder)`). Unfortunately this makes it impossible (or very tricky) to use the `tf.layers.dense()` function, so we need to build the Autoencoder manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0005"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.transpose(weights2, name=\"weights3\")  # tied weights\n",
    "weights4 = tf.transpose(weights1, name=\"weights4\")  # tied weights\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_hidden3), name=\"biases3\")\n",
    "biases4 = tf.Variable(tf.zeros(n_outputs), name=\"biases4\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)\n",
    "outputs = tf.matmul(hidden3, weights4) + biases4\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "reg_loss = regularizer(weights1) + regularizer(weights2)\n",
    "loss = reconstruction_loss + reg_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0150667\n",
      "19% Train MSE: 0.0164884\n",
      "2 Train MSE: 0.0173757\n",
      "3 Train MSE: 0.0168781\n",
      "4 Train MSE: 0.0155875\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_tying_weights.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_tying_weights.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAFsCAYAAAB7FzYbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAG3lJREFUeJzt3VuMleX1x/EHhDkPMMMgM4jD4CAC\ntoaDBjwQjWm8MdFo44X1okZNSpq0SU3aBpMmrTd6p9w01mhA0QsTE8/nmpiowUNEsJVCOAiDMIMg\nMAdmhoP6v6n5G9ZvdZ539p7ZLOb7uVx5n3e/ew97L9+8P9cz6YcffkgAAEQyudIXAABAUTQvAEA4\nNC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAOFMq9LrMpEK5Tar0BUxU3d3d\nfJ9RVm1tbSN+n7nzAgCEQ/MCAIRD8wIAhFOpZ14AUBZqW6dJk8bvEeh4bSvlvafvv/8++9gi5z3X\ncecFAAiH5gUACIfmBQAIh+YFAAiH5gUACIe0IYDQVFrOSwBOnmz/e73IepXsU+dUx3mqqqqyrun0\n6dNy/QUXXJD1+uo6PaUmGMcDd14AgHBoXgCAcGheAIBwaF4AgHAIbAAITQUJvHDCmTNnTE2FE6ZO\nnSrXqyCHOqd6fe+apkyxP8PDw8NZ15lSSt99952pnTp1ytS896QCH+pavRCL+vyLhGBGizsvAEA4\nNC8AQDg0LwBAODQvAEA4BDYAnHOKTHNQgQlvvQoiqMBDESrwUCREooIYtbW1puZN2Dhx4oSpqfek\nzplSSnV1dVnrvdev1DQO7rwAAOHQvAAA4dC8AADh0LwAAOHQvAAA4ZA2BHDO8UYhqRFDauyRl+xT\nY5PUa/X398v1J0+eNLX6+npTa2hoMDUv7aeoBKOqpaTf/9DQkKmpVGZKOkWozumNd1KfqVJkvFQO\n7rwAAOHQvAAA4dC8AADh0LwAAOEQ2ABQUbl7ZHlUOMELbPT19ZmaCiyoc6aUUk1NjalVVVWZmhqv\n5IVQVGBBHesFHtT1q3NWV1fL9bn7ean37h2bG+IoBXdeAIBwaF4AgHBoXgCAcGheAIBwJmRg46OP\nPjK1devWyWMvuugiU1P/p/yvf/1rub65uTmrBkwEKnSgwgnDw8NyvQoi9Pb2mtrXX38t13v1XOr6\np0+fbmql7selQiBq6kVK+jNRr9XR0SHXt7a2mtq0adNMrbGxUa7PnQbiBVZGizsvAEA4NC8AQDg0\nLwBAODQvAEA4NC8AQDiTvJEjY6wiL/qjyy67zNR27tw5Jq+lkkirVq0ak9cqNy+dtHbtWlNrb28f\n46sZ0eg2BULJuru7s7/PuWlDbzzTt99+a2q7d+82te3bt8v1+/btMzU19sgbb6T2+VJpv8HBQVPz\nko4DAwOmNnfuXFObPXu2XJ87HmrRokVy/dVXX21qP//5z02tqalJrlejsNQ1FRmP1dbWNuL3mTsv\nAEA4NC8AQDg0LwBAODQvAEA4E3I81IsvvmhqW7Zskcdefvnlpvbll1+a2scffyzXv/TSS6b21ltv\nmdr8+fNN7auvvpLnzDVliv7ztrW1mdr+/fuzz6uCHH/+85+z12Piyt2nytvP6/Dhw6amxkOpwERK\nesSSqnmBjePHj8v62dR4KzUGKiX9PZ05c2bW66SU0oEDB0xtz549puZ9pirApq5VjbxKSe/npUIc\nKpiRkr9P2Ui48wIAhEPzAgCEQ/MCAIRD8wIAhDMhAxuLFy/OqnmuuOIKU7vzzjvlsQ8//LCp7d27\n19RUYEM9dC1C7QmUkg5sqNdXD8dT8v9PfeBH3kN49XBf7f3kPdyvq6sztfr6elObM2eOXL9gwQJT\nU+ECL7ChfifUtarAw8mTJ+U5VThD1VQwI6WUDh06ZGp9fX2m5oWy1LUWCVGoY72/n1Lk2J/izgsA\nEA7NCwAQDs0LABAOzQsAEA7NCwAQzoRMG44ntVdQblqvSAKyCDXK6siRI6a2cuVKuf6mm24q+zVh\nYlDjoVTaTH1vUkpp1qxZpqbGO3lpw5aWFlNTycLGxka5Xo1yUmk7dU51nSnptKQa5bR582a5Xo2b\nU9Tegiml1NDQYGpqFJRKhaak05rqM1FJ01Jw5wUACIfmBQAIh+YFAAiH5gUACIfAxnnM2z/otttu\nMzX1IP3RRx+V6719fYAflfpw3luvAhNq3Jk3ckiFExQvMJL7vtR+XtXV1fJYVVejpI4dOybXqzFu\ns2fPNjU11i4lPRpOvU/vvau6+j0pN+68AADh0LwAAOHQvAAA4dC8AADhENg4j23YsEHWe3p6TE3t\nHzRv3rxyXxImCG8/qNy9m9TUhpR0OCB3j6+UdJBATbPwAge5e38VCaycPn3a1A4ePGhqW7dulevV\n9f/sZz8zteuvv16u7+joMLVp06aZmveeVLhETeMod4iDOy8AQDg0LwBAODQvAEA4NC8AQDgENs4T\nu3fvNrX7778/e/2mTZtMrbW1taRrAnKoEIe3fYgKTFRVVZmaFw5QoQMVeOjv75frVThDhUOKTJ1Q\ngYfXX3/d1N599125Xk0dWbp0qaldfvnlcv2MGTNMTX3+XohG1VVgw1ufG+I5G3deAIBwaF4AgHBo\nXgCAcGheAIBwaF4AgHBIG54nXnnlFVNTY2dSSumOO+4wtUsuuaTs1wTkUGkzb884tfeVSrapVKBH\nfU8GBgbksWrslUobqv3A1HWmpJPCH330kant3LlTrl+5cqWpXXvttabW3Nws16sUpPr8vLSkSnuq\nZKGXKvRGiY2EOy8AQDg0LwBAODQvAEA4NC8AQDgENgJSD5hfeOEFU1MPt1NK6aGHHjI172EyUE7q\n4byqeYEL9W8695xFjm1oaJDrVTihqanJ1NR4pcHBQXnObdu2ZdW8a7riiitMrb293dTUHl0p6XCF\nF1hR1O9Rkd8TxkMBACYMmhcAIByaFwAgHJoXACAcAhsBPfnkk6b2/vvvm9qvfvUruZ5pGhhrXmBC\nPZwvMo1B7b2lJj8MDw/L9epY9Vpe2EmFHtSEDRU42b59uzznBx98YGrq8/vFL34h169evdrULrzw\nQlNTe4ylpAMX6m9S5G+qjh1tMMPDnRcAIByaFwAgHJoXACAcmhcAIByaFwAgHNKG57AtW7bI+u9+\n9ztTmzFjhqk9+OCDZb8mIEeRvZvUsSdPnpTrp0yxP1nqnN5edipxp15fvU5KeuzRiRMnTE0lC9ev\nXy/P+fnnn5vakiVLTO22226T65cvX25qKi3p7celkpGq5n0m6rzeseXEnRcAIByaFwAgHJoXACAc\nmhcAIBwCG+eIoaEhU7vzzjvlsWp0y1133WVqjIHCuUaFI9S/fS/wocIBNTU1puaNQsodL+UFDlSQ\noaenx9ReeeUVU3v55ZflOdXIqRUrVpiaCmaklFJra6upqc/U09/fb2oqBON9pkq5R0Ep3HkBAMKh\neQEAwqF5AQDCoXkBAMIhsFEB6gHxzTffbGo7duyQ6xcvXmxqf/vb30q/MKBMvL2fVF0FkKqqquR6\nNTmjtrbW1LzAhXfes6lJGinpyR+7du0ytffee8/UDh06JM+ppmksW7bM1Jqbm+V6RX2m33zzTfZ6\n9Tl5IQw1YaSurs7Uyj11gzsvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDikDSvg6NGjpqbSSZ6NGzea\nWpEkEjDWvGRa7oihIntPqWSg9zpqlJRK1nlpSfXdVftxbdu2zdSmT58uz7l06VJT6+joMLWGhga5\nXo28UtfvJSgbGxtlPZfaS9D7+5UTd14AgHBoXgCAcGheAIBwaF4AgHAIbIyx3t5eU1u1alXW2mee\neUbW1egYIAIVJFCBATUGKiUdxFAjm1QwIyW9z5V6LTXyKKWUtmzZYmqbN282teHhYVO76qqr5Dmv\nueYaU1N7dKmRSynp/bi8Y3OpwIUXYlF/E1Xz1o8Wd14AgHBoXgCAcGheAIBwaF4AgHAIbIyx9evX\nm9qePXuy1l533XWy7k0vACJS4QBvmsTAwICpTZ061dTUflYp6XDH8ePHs2oppbRp0yZTU4ENFZhY\ntGiRPOcll1yStd57T4r6jfCmjuROw/DWq8kdTNgAAECgeQEAwqF5AQDCoXkBAMKheQEAwiFtWCY7\nd+6U9b/+9a/jeyFAMCoZp8Y4pZQ/dshL5KrEnhqv9Omnn8r1X3/9tamp/azUeKfOzk55TpXMU+/f\n249L7eelakWoa/LShkVSkOXEnRcAIByaFwAgHJoXACAcmhcAIBwCG2Xy/vvvy3pfX1/W+sWLF5ta\nbW1tSdcETARFwgkqiKD28/ICHyqcsWLFClPr6OgwtaamJnlONbLqyJEjpubt0aXevxfuUNR7Lffe\nW2OBOy8AQDg0LwBAODQvAEA4NC8AQDgENirgmmuuMbV33nnH1AhsYKLyAhNFJj8o6lgVwrjyyivl\n+gULFphafX29qak9xrzAhvqeq/UqWJKS/qzU51QkxKGOPdf2EeTOCwAQDs0LABAOzQsAEA7NCwAQ\nDs0LABDOpAqNATn3Z48gmnMrCjWBdHd3n3Pf5yK/a2q8kpesU2lFdaza48pLC6qxTyotWKpzLS34\nv7S1tY14sdx5AQDCoXkBAMKheQEAwqF5AQDCqVRgAwCAUePOCwAQDs0LABAOzQsAEA7NCwAQDs0L\nABAOzQsAEA7NCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOzQsAEA7NCwAQ\nDs0LABAOzQsAEA7NCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOzQsAEA7N\nCwAQDs0LABDOlAq97g8Vel2cvyZV+gImqq6uruzv8w8/2EMnTTr//nTqfXpy3793ziifX5Hrb29v\nH/FNcecFAAiH5gUACIfmBQAIp1LPvACgsNxnSd5zoNxnbqU+XypyzlxFnm0VOVZd11g8myz3sznu\nvAAA4dC8AADh0LwAAOHQvAAA4dC8AADhkDYEUFFjkWwrkuz7/vvvs9d/9913pjZ5ct49wMmTJ2W9\nrq7O1NT79z6T6urqrNdS7zMlff25CUTPeEz94M4LABAOzQsAEA7NCwAQDs0LABAOgY0yefbZZ2X9\nxIkTpvbZZ5+Z2uOPP579Wn/5y19M7cYbbzS1G264IfucwHjIHZtUZDyTCiJ4gQF17PDwcNZxKaV0\n9OhRWT+bCnbU1tbKY/v6+kztzJkzpua9JxX4ULUpU/TPvXqtqqoqU/M+k0pt6cKdFwAgHJoXACAc\nmhcAIByaFwAgnEml7jEzShV50XL57W9/a2r/+Mc/KnAl/2/JkiWm9sEHH8hjp0+fPtaXUwlj/7/0\nQ+rq6sr+Po/FflwqHDE4OCjXDw0NmZoKTBw8eFCu//LLL03t2LFjWdfpvXc1IWPu3Lmm1t7eLter\nY1Vgw/veqyCHutYiEzq8cIiiXmvevHkjfp+58wIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEw3ioEYxF\nsnDZsmWm9stf/tLUdu7cKdc/9dRTprZt2zZTe/755+X6e++9d6RLBCrKS+apUUaqdvz4cbl+3759\nprZr1y5T+/e//y3X792719RUWlDVLrjgAnlOlZZUNS8t2NnZmXWs9/rqs1YJQm8/MpVCrK+vl8cq\njIcCAEwYNC8AQDg0LwBAODQvAEA4BDb+q6urS9afeOKJrPVXXXWVrL/55pumpka3qP1z1EPblPQD\n5g8//NDUjhw5ItcDY63UsXPev/3Tp0+bmhrP1NPTI9erwIUKd0ybNk2uv+mmm0yto6PD1FQI4cCB\nA/KcXv1s3meqfjtmzJhhaiqEkZLeo0x9zt763FFY3vrR4s4LABAOzQsAEA7NCwAQDs0LABAOgY3/\n8sIN6sGjCmf885//lOsbGhpGfU0bNmyQ9U8//TRr/a233jrq1wbGi5rQ4E2DUPUi0yDa2tpMTQUu\nvL2z5syZY2ozZ840tcOHD5va22+/Lc+ppnn09vaamvdbsmjRIlNraWkxtalTp8r16vNTgQ1vvfqN\nVFNPvD2+Rhvk4M4LABAOzQsAEA7NCwAQDs0LABAOzQsAEA5pw/9avny5rKsUohrHUltbW/Zr8kZT\nnTp1quyvBZRTkT2aVIrNS6CpsVEqxXbxxRfL9SoZp9KC8+bNk+sbGxtNbWhoyNS6u7tNzdsjbMeO\nHabW1NRkaiqV6VFpQY9KBqpUp5c2VH9r9Xfy/qbs5wUAmDBoXgCAcGheAIBwaF4AgHAIbIxg+vTp\n4/I6GzduNLWtW7dmr1f7DHV2dpZ0TcB4KPXhvgpQqT3zPM3NzabmBbCGh4dNbfv27ab26quvmtoX\nX3whz6muddmyZabm7Rmowinq81PX7h2rAhtesEIFOYqEMEa79xt3XgCAcGheAIBwaF4AgHBoXgCA\ncAhsVMDnn39uar/5zW9MrcieROvWrTM17/+IB8pptA/cf6T+nRY5p1qvpkaklFJNTU3WehUiSSml\nQ4cOmdprr71mamrPPe+cS5YsMbVVq1aZ2sqVK+V6FTg5fvy4qXmTeVTgRYU4ivxNRrtHVxHceQEA\nwqF5AQDCoXkBAMKheQEAwqF5AQDCIW1YAZs2bTI1L1morFmzxtQWLlxY0jUB4yF37ydvPyo1tkit\n975Paj8ulaI7duyYXK9GPP3nP/8xtYGBAVPz9ghbtGiRqV166aWm1tLSIter61fJQi99rJKB6u9U\nZD+xImlD9vMCAEwYNC8AQDg0LwBAODQvAEA4BDbG2D333GNqzz33XNbaP/zhD7L+pz/9qaRrAsqp\n1L2b1MN9L1ygghxq7y1vvaqrfa4OHDgg13/yySemdvToUVNraGgwtTlz5shzzpw509RUsMTbj2tw\ncNDUVGDE25swNwTjyd37yxsvxX5eAIAJg+YFAAiH5gUACIfmBQAIh8BGmagHpCml9MYbb5iaevA6\ne/ZsU3vggQfkOdX+O0ClqAfuXogj90G+FxhQkyNUEEGFCLzX7+7uNrV//etfcv3evXtNLff7rIIl\nKaVUV1dnauo61R5dKaXU29tralOm2J92LxihPit1Td7rK+o3ypvQQWADADBh0LwAAOHQvAAA4dC8\nAADhENgokzvuuEPWv/nmm6z1v//9702tubm5pGsCxkORaQqlqqmpyapVV1fL9Spc8e2335rajh07\n5Pqenh5TU5MzWltbTW3+/PnynJdddpmpqcCDeu2U9PYvs2bNMjX1OaWU0pkzZ2Q9lwrReIGZcuLO\nCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOacNR+Oyzz0ztvffey15/++23m9r9999fyiUB55Qie3yp\nY70RaCpF6KXolGPHjpnanj17TO2rr76S61Uyb9q0aabW2dlpasuXL5fnnDt3rqkVSWuqz0SNd/I+\nU5XAVHuEeSO71HmLjAwbLe68AADh0LwAAOHQvAAA4dC8AADhENgYwdDQkKmtXbvW1NSIFM+KFStM\njT26MBGoB/mqpvajSkmHMyZPtv8N7u09pcY+bdq0ydS6urrk+osvvtjUli1bZmpLly41NTUyKiW9\nz5UKUah9y7z1uSGKlFLq6+vLOqcX2FCBFfX65R4ZxZ0XACAcmhcAIByaFwAgHJoXACAcAhsjeOyx\nx0zt3XffzV5/zz33mBrTNHC+88IBuXt/eYENdazaz0pNzUgppQ8//NDUNm/ebGonTpyQ61evXm1q\nanKGCnZ4oa7e3l5TU+9z6tSpcr0KQqjARX9/v1x/5MgRUzt9+rSpeYGR3MkZXuBjtEEO7rwAAOHQ\nvAAA4dC8AADh0LwAAOHQvAAA4ZA2HMEDDzxQ0vpHHnnE1BgFhfNJkb2nFLX3lEq7paTHQ6lRSt3d\n3XL93r17TU2l/VpaWuT6+fPnm5r6Pqtr8kZWDQwMmFptba2pNTY2yvXq81fv3xt5pT4T9fqLFy+W\n69UIPTWyy0tLjhZ3XgCAcGheAIBwaF4AgHBoXgCAcAhsjDH1MFY9zCxVdXW1qXljV9SYFjVix6Me\n0K5bty57vaKu1QvLlPvBL0qTO/LJO1aNYvJGEanzqrFLajySVz9z5kz2+sOHD5vagQMHTK1IYEJ9\nd+vr603NG5mlAidqPNahQ4fkevXd6+zsNDVvDJS61tyRVSn572sk3HkBAMKheQEAwqF5AQDCoXkB\nAMIhsDHGLrroonF5nTVr1pjanDlz5LE9PT2m9ve//73s11Qq77O77777xvlKUC7e5IyzeYEPtV4F\nLry9s9Q0DBUYOHr0qFyv9gPbtWtX1jV5oSgVTlGhJHXOlPQ0DxXi8AITanKGuqampia5XgXAVCit\n3EE17rwAAOHQvAAA4dC8AADh0LwAAOHQvAAA4ZA2HMFdd91lauvXr6/Alfxvjz32WNnP6Y1t8cZO\nne3uu++W9auvvjpr/bXXXpt1HCqr1P28FC8tqJJt6t9jW1ubXK/241Ij3LZt2ybXf/HFF6bW399v\namqEmrcfl7rW2bNnm5qXVlTJxBkzZphaR0eHXL969WpTU0lfNQYqJf07USRtONp/P9x5AQDCoXkB\nAMKheQEAwqF5AQDCmTQWD1szVORFy+Xpp582Ne8Bc66tW7eaWqkjm/74xz/K+oIFC7LW33LLLbJ+\n4YUXjvqaxpDebAhjrqurK/v7rH5v1NgjNcYppfywkDdKaf/+/aamwhn79u2T63fv3m1qajzU4OCg\nqbW2tspztrS0mFpNTU1WLSUd2FDhDC+wMW/ePFNT46G8/bzU30SFM7y/nfo30d7ePuL3mTsvAEA4\nNC8AQDg0LwBAODQvAEA4BDZwviCwUSGlBjbUPlPe75IKJ6hwhzouJb0fmJraoWop6X2+urq6TE1N\n7fAmVKggRJHfZTVNQ03z8Cbm1NXVmZr6/Lxryr1+L/ChENgAAJyXaF4AgHBoXgCAcGheAIBwaF4A\ngHDYzwtAReWOF0pJJ9ZUMrC6ulquV4k/dU4vGTdr1ixTW7hwYdY1eWk9NcpqeHjY1IrsceZ9fopK\na6q9w4qcs0iykP28AAATBs0LABAOzQsAEA7NCwAQDoENAOMmd5SQ9xBf1dXYIzUGqggVmEhJhyPU\neCvF289KfSYqxOGNd1LUeCcvRKE+q7EIXHjnLPJaP8WdFwAgHJoXACAcmhcAIByaFwAgHAIbACqq\nyH5Wqq4CB956dayaJlFTUyPXqykTavKFmvCh9s1KSe/9pcIdRUIoRfZIU/Ui0zRyz1lu3HkBAMKh\neQEAwqF5AQDCoXkBAMKheQEAwiFtCGDclDpKKPecaoyTJzdBWGS9Gu/U398v16tkoRpPVeoeZ0WM\nR1qwVNx5AQDCoXkBAMKheQEAwqF5AQDCmRThwRwAAD/FnRcAIByaFwAgHJoXACAcmhcAIByaFwAg\nHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIBya\nFwAgHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIByaFwAgnP8Dby1/k4rB\nte8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1173a7f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_tying_weights.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training one Autoencoder at a time in multiple graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many ways to train one Autoencoder at a time. The first approach is to train each Autoencoder using a different graph, then we create the Stacked Autoencoder by simply initializing it with the weights and biases copied from these Autoencoders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a function that will train one autoencoder and return the transformed training set (i.e., the output of the hidden layer) and the model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "def train_autoencoder(X_train, n_neurons, n_epochs, batch_size,\n",
    "                      learning_rate = 0.01, l2_reg = 0.0005, seed=42,\n",
    "                      hidden_activation=tf.nn.elu,\n",
    "                      output_activation=tf.nn.elu):\n",
    "    graph = tf.Graph()\n",
    "    with graph.as_default():\n",
    "        tf.set_random_seed(seed)\n",
    "\n",
    "        n_inputs = X_train.shape[1]\n",
    "\n",
    "        X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "        \n",
    "        my_dense_layer = partial(\n",
    "            tf.layers.dense,\n",
    "            kernel_initializer=tf.contrib.layers.variance_scaling_initializer(),\n",
    "            kernel_regularizer=tf.contrib.layers.l2_regularizer(l2_reg))\n",
    "\n",
    "        hidden = my_dense_layer(X, n_neurons, activation=hidden_activation, name=\"hidden\")\n",
    "        outputs = my_dense_layer(hidden, n_inputs, activation=output_activation, name=\"outputs\")\n",
    "\n",
    "        reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "        reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "        loss = tf.add_n([reconstruction_loss] + reg_losses)\n",
    "\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "        training_op = optimizer.minimize(loss)\n",
    "\n",
    "        init = tf.global_variables_initializer()\n",
    "\n",
    "    with tf.Session(graph=graph) as sess:\n",
    "        init.run()\n",
    "        for epoch in range(n_epochs):\n",
    "            n_batches = len(X_train) // batch_size\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                indices = rnd.permutation(len(X_train))[:batch_size]\n",
    "                X_batch = X_train[indices]\n",
    "                sess.run(training_op, feed_dict={X: X_batch})\n",
    "            loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        params = dict([(var.name, var.eval()) for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)])\n",
    "        hidden_val = hidden.eval(feed_dict={X: X_train})\n",
    "        return hidden_val, params[\"hidden/kernel:0\"], params[\"hidden/bias:0\"], params[\"outputs/kernel:0\"], params[\"outputs/bias:0\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train two Autoencoders. The first one is trained on the training data, and the second is trained on the previous Autoencoder's hidden layer output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0185175\n",
      "1 Train MSE: 0.0186825\n",
      "2 Train MSE: 0.0184675\n",
      "3 Train MSE: 0.0192315\n",
      "0 Train MSE: 0.00423611\n",
      "1 Train MSE: 0.00483268\n",
      "2 Train MSE: 0.00466874\n",
      "3 Train MSE: 0.0044039\n"
     ]
    }
   ],
   "source": [
    "hidden_output, W1, b1, W4, b4 = train_autoencoder(mnist.train.images, n_neurons=300, n_epochs=4, batch_size=150,\n",
    "                                                  output_activation=None)\n",
    "_, W2, b2, W3, b3 = train_autoencoder(hidden_output, n_neurons=150, n_epochs=4, batch_size=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can create a Stacked Autoencoder by simply reusing the weights and biases from the Autoencoders we just trained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28*28\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "hidden1 = tf.nn.elu(tf.matmul(X, W1) + b1)\n",
    "hidden2 = tf.nn.elu(tf.matmul(hidden1, W2) + b2)\n",
    "hidden3 = tf.nn.elu(tf.matmul(hidden2, W3) + b3)\n",
    "outputs = tf.matmul(hidden3, W4) + b4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa8AAAFsCAYAAAB7FzYbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAGuFJREFUeJzt3UtoXeXXx/HVa5ImTdKkSWt6Nb1Z\nrWIrUm8DceBEUFQcqANFBUVQUFBRENSJzrQTUVG8DwRBxYF3rFbRolWLjVJT01aNTRNrk7Rpk17i\nO/F9/3+yfs+b5+Sc9GQl389wsZ+9d048WW72r+uZ9s8//xgAAJFML/cNAABQKJoXACAcmhcAIBya\nFwAgHJoXACAcmhcAIByaFwAgHJoXACAcmhcAIByaFwAgnJllui4zqVBq08p9A1NVT08P32eUVFNT\n06jfZ568AADh0LwAAOHQvAAA4ZTrnRcATAq520pNm8Zr2VLiyQsAEA7NCwAQDs0LABAOzQsAEA7N\nCwAQDmlDAFOaSgEODw/LY1V9+nT/DDB79uzsc6r16tiTJ0/K9al67nEzZszIqk00PHkBAMKheQEA\nwqF5AQDCoXkBAMIhsAEgtGLHLhWyXgUZVDhDnfPEiRPynGq8VCGBDXVsRUWFq6VCGKqee0/lxJMX\nACAcmhcAIByaFwAgHJoXACAcAhsAwsidhlFICEMFIYaGhuSxM2f6P5kq8JA79cLMrLKy0tWOHTvm\namoSRyH3lPpMVDhDXT+1vtjPf6x48gIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEQ9oQwISjEnBmOsWn\naqlk3tGjR11NJetSVDJQXf/48eOuVlVVJc/Z19fnaoXssaXqKgGoxliZ6XtVUuOhVLKw2ARmDp68\nAADh0LwAAOHQvAAA4dC8AADhENgAMOGkAhtqTywVBFAjk8x0OEEFNlLhhFmzZrmaCmLU1ta6Wm9v\nrzynGkWlfqa6ujq5Xv1MKjBRyHgp9TkXEthQ5yxkPFUOnrwAAOHQvAAA4dC8AADh0LwAAOFMycDG\n119/7WqbNm2Sxy5atMjV1Avam266Sa5vaGjIqgH4j9RL/IGBgaxaaj+uAwcOuFp/f7+rpaZuqCCG\nClKo6+/atUue8+DBg6522mmnuVpjY6NcPzg46GpqksiaNWvk+oULF7ranDlzXC0V2FBBEBXOSP1O\nU0GS0fDkBQAIh+YFAAiH5gUACIfmBQAIh+YFAAhn2lhHcxSpLBf9Xyp1097ePi7XUkmkCy64YFyu\nVWrLly+X9QcffNDVli5dOs53Myo9ewbjrqenp6jvsxqFpMYTmekRSypBuHv3brl+7969rqbSeqm0\nojpWjWLq7u52tR07dshzqvVnnHGGq6m9xMzMenp6so5dt26dXH/ppZe62sqVK11NjcYyy08WplKF\n6udvamoa9fvMkxcAIByaFwAgHJoXACAcmhcAIJwpOR7q7bffdrUffvhBHnvWWWe5Wltbm6tt3bpV\nrn/nnXdc7YMPPnC1008/3dVSL51zpfY0UqNnfv/99+zzqiDHAw88kL0ek18qCKbqqqb2qDLTQQ41\n3kmFGMx04EN9Tw4fPizXq2upwIkKJyxZskSeU4W61Cio1H5gKmymjlXBCDOz1atXu5r6jqfWV1RU\nuNqpCALy5AUACIfmBQAIh+YFAAiH5gUACGdKBjbWrl2bVUs555xzXO3666+Xxz7xxBOutmfPHldT\ngY2Ojo7se1Jmz54t6yqwoa6feumt/vU/8N/U1AWz/MkLqfUqXKHCHXPnzpXrW1paZH2k+fPny7ra\nJ0vtc6WunwpQqc9EnVPt+2Wmw2Zq6kgqBKOupT7/QvbdUseqYEvqWlnXGNMqAADKiOYFAAiH5gUA\nCIfmBQAIh+YFAAhnSqYNTyW1r05uWq+QBGQh1Cirv/76y9U2btwo119++eUlvydMDbkptNR+XmpE\nkfqOLV68WK5Xo4yqqqpcLTUeas6cOa6mUoTquFTaT/1M6nP6/PPP5Xr1869YscLVVq1aJdcvWLDA\n1VJpTUWlBVWCMZVWHOsoKZ68AADh0LwAAOHQvAAA4dC8AADhENiYxAYGBmT96quvdjX1gvWpp56S\n69ULbmCs1Av/1N5RajxTfX191jnNzJqamrKupb4PZmazZs1yNfV9UOGM1Hgodf1Dhw652v79+7Pv\naf369a62Zs0auV7t3aVGy6VCNKpeyCgpxkMBAKYMmhcAIByaFwAgHJoXACAcAhuT2EsvvSTrXV1d\nrtbY2Ohqy5YtK/UtAU4qnKGoaQx1dXWuVltbK9dXV1e7WmqaRu71VbhBTfIoZD+v9vZ2V0vtr6cC\nD/PmzXM1FWwxyw+xpPbjUnUV2EhN0iCwAQCYMmheAIBwaF4AgHBoXgCAcAhsTBK//vqrq917773Z\n67/66itXW7hwYVH3BIykXtqrCQ2pcIPaaqSmpsbVCpkCo+4pFeJQQYxjx44Vdf3Ozk5X+/TTT13t\np59+kuvV/avASktLi1yvpomoEMbQ0JBcn2uswYwUnrwAAOHQvAAA4dC8AADh0LwAAOHQvAAA4ZA2\nnCTeffddV1N7CpmZXXfdda7W2tpa8nsCRlKJM1VTe1SZ6bFFlZWVrqZGNpnpfbLU9yS1XqXw1B5j\nKpXY29srz7lz505X+/77710tlYBU+3SpmhqjZaY/P/U7SSVAi00hjhVPXgCAcGheAIBwaF4AgHBo\nXgCAcAhsBKReML/11luupl4am5k9/vjjrlbInkpAKRUSDlCjkNR6NbLJzOzIkSOupsZTpaj7UoGH\nwcFBV+vo6JDnVGGrrVu3uprao8vM7KyzznK1Cy+80NUaGhrk+twQTUrq78xI7OcFAJjyaF4AgHBo\nXgCAcGheAIBwCGwE9MILL7jali1bXO2GG26Q65mmgYlu+nT9/9W5+4Glpj6oujqnCmGY5e8d1tXV\n5WrqO2qm99JT4Yxzzz1Xrt+4caOrNTY2ulp1dbVcrwITKhSW+p2o/cByr1MMnrwAAOHQvAAA4dC8\nAADh0LwAAOHQvAAA4ZA2nMB++OEHWb/rrrtcrb6+3tUee+yxkt8TUGpqj6zUXnTqWDWyKZWAU/t0\nqRTd3Llz5Xo1YkmNgtqzZ4+rffbZZ/Kcaj+wDRs2uNq6devkepUeVn8PUmk/tceZoj77FPU5F7I+\nB09eAIBwaF4AgHBoXgCAcGheAIBwCGxMEOql7fXXXy+PVS8+b7zxRldjDBQiKGQ/KRWuGBgYcDUV\nGDDTo6DUyKfUeCj13du/f7+rffTRR67W1tYmz6n2w1q8eLGrXXTRRXK9Gk+lQiyFBCbUsamRW+qz\nKnU4Q+HJCwAQDs0LABAOzQsAEA7NCwAQDoGNMlD/+v+KK65wtZ07d8r1a9eudbVHH320+BsDxlkh\n4QxFBTbU+v7+frle7XOlwh0q8GCmp1F8++23rrZ9+3ZXSwUeVqxY4WpqmkYqgFVXV+dq6jNRk0DM\ndIhFBS5S+3ml6jnXMRv7Pl88eQEAwqF5AQDCoXkBAMKheQEAwqF5AQDCIW1YBn///berbd68OXv9\nq6++6mpqnyFgolEptkLSZmoUUW7azUynEGfMmOFqhw8fluvVd1elDTs7O12tublZnlOlDdevX+9q\ntbW1cr26f5VsTO2Rpo5VycDUyC11XpXWHGuqMIUnLwBAODQvAEA4NC8AQDg0LwBAOAQ2xllfX5+r\nXXDBBVlrX3vtNVlXL3OBCFS4QoU4Zs2alX1OFSRIhQPUsQcOHHC1gwcPyvXbtm1zta1bt7pad3e3\nq6nRVGZmq1evdrUFCxa4mhorZ6b3M1MhitR4ptzAiwqGmOnfVepeS4knLwBAODQvAEA4NC8AQDg0\nLwBAOAQ2xtmLL77oah0dHVlrL7nkElkv9b9UB8pJBQFS0yBUEEAFDmpqauR6taeVupYKXJiZtbe3\nu9qxY8dcTU28Of/88+U5N2zY4GrqZypkP65CjlN/TwqZWlLs1JSx4skLABAOzQsAEA7NCwAQDs0L\nABAOzQsAEA5pwxJRKSQzs0ceeeTU3ggQTG5azkwn+5QTJ07Iutq7Sp1TJejMzCoqKlztzDPPzLon\ntW+XmVldXZ2rqQRkKsGnPr/cVKaZHu+U+vkVdd5CfqdjxZMXACAcmhcAIByaFwAgHJoXACAcAhsl\nsmXLFlnv7+/PWr927VpXq6qqKuqegAgKGSWUGw5IBTtSQY6RqqurZX3ZsmWu1tLSknXORYsWZV9L\nBS5S46HU3wn1c6YCGyqckdq7SzkV4QyFJy8AQDg0LwBAODQvAEA4NC8AQDgENsrgoosucrWPPvrI\n1QhsAKNTgYHZs2dnr6+srMyqpeoqXKECD6nvs5raoc45c6b+c63CGer6qWBM1P0BefICAIRD8wIA\nhEPzAgCEQ/MCAIRD8wIAhDOtTKM9yjNPBJNZzMjUJNDT0zPhvs8qrZdSyHik1IilkdTf1dTf2ty/\nwYWkAtWxkVKFTU1No94sT14AgHBoXgCAcGheAIBwaF4AgHDKFdgAAGDMePICAIRD8wIAhEPzAgCE\nQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIAhEPz\nAgCEQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIAhEPzAgCEQ/MCAIRD8wIA\nhEPzAgCEQ/MCAIRD8wIAhDOzTNf9p0zXxeQ1rdw3MFV1dHTwfUZJtba2jvp95skLABAOzQsAEA7N\nCwAQTrneeQHAhDVtmn7l8s8//vWeOnZ4eDj7nMXKvafJhicvAEA4NC8AQDg0LwBAODQvAEA4NC8A\nQDikDQFMOCpBZzYxU3TqXmfPnu1qJ06cyF5/8uTJrFrK9On+uWTmTP3nPmpakScvAEA4NC8AQDg0\nLwBAODQvAEA4BDZK5PXXX5f1gYEBV9u2bZurPffcc9nXevjhh13tsssuc7VLL700+5zARFJIYCA3\n8JA6dmhoyNVS4Ypjx4652qxZs1xN3X/qnAcPHnQ1Fa5Q1zEzq6qqcjUVGFEhjtSx6vqpEE3qvOON\nJy8AQDg0LwBAODQvAEA4NC8AQDjTUi/hxllZLloqd955p6s9++yzZbiT/zjzzDNd7YsvvpDH1tXV\njfftlMPEHwkwSXV0dJT8+5z6u6TqKggxODgo16sAlQpM/Pbbb3L9n3/+6WqHDx92td9//93VUiEU\nFS5ZuXKlqzU3N8v1LS0trrZkyRJXq6mpketVOEPVZsyYIdererEhjtbW1lG/zzx5AQDCoXkBAMKh\neQEAwqF5AQDCoXkBAMJhPNQoxiNZuH79ele79tprXa29vV2uf/nll13tp59+crU333xTrr/11ltH\nu0XglFEJwuHhYXmsGs906NAhV/vrr7/k+t27d7vad99952o///yzXN/T0+NqKm2n7lONoTIzq62t\ndbXjx4+7WmVlpVy/dOlSV1PJQnUdM7OKigpXU59/KsGpkoUqQZlKII517zCevAAA4dC8AADh0LwA\nAOHQvAAA4RDY+FdqHMzzzz+ftf7888+X9ffff9/V5syZ42pqT53UnkS7du1ytS+//NLVUi+tgYkk\n9d+5ooIQajzT/v375foff/zR1bq6ulytvr5erldj2BoaGlytsbHR1X755Rd5TlVPjXJSig1MqFFQ\nauSWOs5Mh0vU3mOlHkXIkxcAIByaFwAgHJoXACAcmhcAIBwCG/9KhRvUS0YVzvj444/l+kJevI70\n0ksvyfo333yTtf6qq64a87WBU0VNqFAhADM9+UFNiEh979atW+dqq1atcjU1BcfMrLW1NeueOjs7\nXa2/v1+ec9++fa6mpln09vbK9anJGyOpEIaZDneo66fWqwkZ6pxjnaSRwpMXACAcmhcAIByaFwAg\nHJoXACAcmhcAIBzShv/asGGDrKsUohrlVFVVVfJ7So2mUiNygAhUereQFJoaUaTSdvPnz5fr1Ygk\ntR/WueeeK9er7776G9HW1pZ1nJlZd3e3q51zzjmultqPSyUD1ed89OhRuV4lO9XvJPV3R9XnzZsn\nj1XYzwsAMGXQvAAA4dC8AADh0LwAAOEQ2BhFXV3dKbnOq6++6mrbt2/PXn/55Ze72ooVK4q6J6DU\n1Mt5FS5IjTxSx6rAwNy5c+V6tZfe8uXLXU2NnDIzO3LkiKvt3LnT1TZv3uxqP//8szyn2vuqurra\n1VpaWuR6Vc8NtqSozzk1HkoFQVS4RI0BKwZPXgCAcGheAIBwaF4AgHBoXgCAcAhslMH333/varff\nfrurDQ0NyfWnnXaaq23atMnV1Itg4FRQL/zN9IQLtfdT6r9dFaRQIZDUd6epqcnVVLhjYGBAru/q\n6nK1Dz74wNW2bdvmaqn9vNQ0EBUsWbt2rVzf3NzsaiqwkdojTU0HUuGMw4cPy/X19fWupsIZ6ndf\nDJ68AADh0LwAAOHQvAAA4dC8AADh0LwAAOGQNiyDr776ytVS6SjljjvucLXVq1cXdU/AqTA8POxq\nhSTTVLJQ7bGlxiuZ6RSfStapVKGZ2YcffuhqauyT+pkWL14sz9nY2OhqKgGpxliZ6Z9p7969rpYa\ndac+U5VWTI3MSiVLc65TDJ68AADh0LwAAOHQvAAA4dC8AADhENgYZ7fccourvfHGG1lr77nnHlm/\n//77i7onYLwV8nK+kMBGbjggtXeUGpGkxh59/fXXcn1bW5ur/fHHH66m9v1as2aNPOe8efNcbePG\nja6W2o+rt7fX1dQoKrXHlpkOZ6gAWU1NjVyv9lMrdThD4ckLABAOzQsAEA7NCwAQDs0LABAOgY0S\nSe11895777na4OCgqy1YsMDVHnroIXlONVEAiEBN2Mjd48ssf486dR0zHWRob293tR07dsj1Bw4c\ncDU1oUOFMFKBh6VLl7qamrDR19cn1+/evdvV1Geq7tNMh1hU4CIVwlCf9anYS5AnLwBAODQvAEA4\nNC8AQDg0LwBAOAQ2SuS6666T9e7u7qz1d999t6s1NDQUdU/ARJOanDFSKhyg1qtwQCqwoaZBqO+o\nClWZ6e1HLrnkEldTW7KoEIaZ2cqVK12tvr7e1VTYxMzs6NGjrqYmjKhJHGZ68oYKzKjPzkz/rtT6\n1NSTseLJCwAQDs0LABAOzQsAEA7NCwAQDs0LABAOacMx2LZtm6tt3rw5e/0111zjavfee28xtwRM\nKql9u1RirZD9wA4dOuRqf//9t6ulxhupZJ4a7TZ//nxXS/1MKlmoxs2lEpRqFJUa+ZT6TNTeXWo/\nstT11edfUVGRdVwxePICAIRD8wIAhEPzAgCEQ/MCAIRDYGMUavTKgw8+6Gqp0SnKeeed52rs0YWp\nSgUBZs7Uf5pUuEBJfR/V3ld79uxxtZ07d8r1zc3NrqbCGblrzfQ+X+rnV8GS1LHq508FJtRnWllZ\n6WoDAwNyfa5UYCU1Cmw0PHkBAMKheQEAwqF5AQDCoXkBAMIhsDGKZ555xtU++eST7PW33HKLqzFN\nA5Nd6uW8ogIbqWkO6rwqnHDgwAG5XgU29u7d62r79u2T61esWOFqao+vhQsXupqaxGGmp3moCRmp\nYIP6rFQIo729Xa5XkzcaGxtdTYU4Useq/bxSU0vGiicvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDik\nDUfx0EMPFbX+ySefdDVGQWGySyXjclOIqbTh4OBgVi013mnXrl2uppKFamSTmd57a8mSJa6m9thS\n92lm1t/f72qF7Oel6ipB2dnZKdertOayZctcrbW1Va5Xn5VKYJYaT14AgHBoXgCAcGheAIBwaF4A\ngHAIbIwz9eJVjWMpVkVFhaul9u9Ro1ty90ky03ucbdq0KXu9ou41FZYp9ZgZlJcKcaQCH2rvqr6+\nPldT/42amR08eDDr+ipEYWbW29vrar/88kvW9Y8cOSLPqa6lvo+p77O6VldXl6v9+eefcr3627F8\n+XJXS4UwcvfjYj8vAMCUR/MCAIRD8wIAhEPzAgCEQ2BjnC1atOiUXOeOO+5wtZaWFnmsepn79NNP\nl/yeipX67G677bZTfCcoldxwRirUpEIDKsSR2s9LhRPUsQMDA3L9t99+62pq6kZDQ4Or9fT0yHOq\nIEZtbW3WcWY6RLJ//35XU1M/zMyWLl3qaur+q6ur5Xp1X+r3N9ZgRgpPXgCAcGheAIBwaF4AgHBo\nXgCAcGheAIBwSBuO4sYbb3S1F198sQx38v975plnSn5OleIyS6eeRrr55ptl/cILL8xaf/HFF2cd\nhziKHSWk6iqZt2rVKrlejUhavXq1q33zzTdy/Y4dO1xN7YelviOp/bzU90zdU2osmlo/d+5cV1u5\ncqVcf/bZZ7uaSvpWVlbK9ep3mlsrBk9eAIBwaF4AgHBoXgCAcGheAIBwpqVejI6zsly0VF555RVX\nUy9tC7F9+3ZXK3Zk03333SfrqRe3I1155ZWy3tzcPOZ7GkelfRuMbB0dHdnf59zxUKmX+yoIoc7Z\n3d0t1+/bt8/V2tvbXe3XX3+V69va2lyts7PT1dQItrq6OnlOtXfWGWecIY9V1GeiRsMtWbJErl+2\nbJmrzZ8/P+s6ZjpIoo7NDXqZmbW2to76febJCwAQDs0LABAOzQsAEA7NCwAQDoENTBYENsqkkMBG\nsVQ4QO0dlQoHDA0NudqJEydc7fjx43J9X1+fq6nAhtpjKzWxpqamxtUaGxtdLbXHmFqvAi9qL7RU\nXa1PfaaFBDFyEdgAAExKNC8AQDg0LwBAODQvAEA4NC8AQDikDTFZkDYsk2LThupvUOrvkkrBqRRf\nKtlXUVGRVUuNp0qddySVDEyNkFPnHB4ezjqnmb7XkydPjnaL/0f9/CqBqe4pdf1ikTYEAExKNC8A\nQDg0LwBAODQvAEA4eW8fAWCcFLKfV244QAUOzPKDEKlwQmVlpavljqyqqqqS51Qjq9T1U2ERdWwh\nI7PU+jIF+QrCkxcAIByaFwAgHJoXACAcmhcAIBwCGwDCKDZIkApy5F5HrVeBDxWuUGEPMx2YUNdJ\n3VNuYKOQqRvqnOMxSaMYPHkBAMKheQEAwqF5AQDCoXkBAMKheQEAwiFtCCC0QlJ0xTp+/LirqWSf\nSusdPXpUnjM3GVhI0jI13iqXuqeJZuLfIQAAI9C8AADh0LwAAOHQvAAA4UyLsG8LAAD/jScvAEA4\nNC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQv\nAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBAODQvAEA4NC8AQDg0LwBA\nODQvAEA4NC8AQDj/A27TzGrTpYm2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105ba2f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training one Autoencoder at a time in a single graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another approach is to use a single graph. To do this, we create the graph for the full Stacked Autoencoder, but then we also add operations to train each Autoencoder independently: phase 1 trains the bottom and top layer (ie. the first Autoencoder) and phase 2 trains the two middle layers (ie. the second Autoencoder)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0001\n",
    "\n",
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "weights3_init = initializer([n_hidden2, n_hidden3])\n",
    "weights4_init = initializer([n_hidden3, n_outputs])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=\"weights3\")\n",
    "weights4 = tf.Variable(weights4_init, dtype=tf.float32, name=\"weights4\")\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_hidden3), name=\"biases3\")\n",
    "biases4 = tf.Variable(tf.zeros(n_outputs), name=\"biases4\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)\n",
    "outputs = tf.matmul(hidden3, weights4) + biases4\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "with tf.name_scope(\"phase1\"):\n",
    "    phase1_outputs = tf.matmul(hidden1, weights4) + biases4  # bypass hidden2 and hidden3\n",
    "    phase1_reconstruction_loss = tf.reduce_mean(tf.square(phase1_outputs - X))\n",
    "    phase1_reg_loss = regularizer(weights1) + regularizer(weights4)\n",
    "    phase1_loss = phase1_reconstruction_loss + phase1_reg_loss\n",
    "    phase1_training_op = optimizer.minimize(phase1_loss)\n",
    "\n",
    "with tf.name_scope(\"phase2\"):\n",
    "    phase2_reconstruction_loss = tf.reduce_mean(tf.square(hidden3 - hidden1))\n",
    "    phase2_reg_loss = regularizer(weights2) + regularizer(weights3)\n",
    "    phase2_loss = phase2_reconstruction_loss + phase2_reg_loss\n",
    "    train_vars = [weights2, biases2, weights3, biases3]\n",
    "    phase2_training_op = optimizer.minimize(phase2_loss, var_list=train_vars) # freeze hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training phase #1\n",
      "0 Train MSE: 0.00740679\n",
      "1 Train MSE: 0.00782866\n",
      "2 Train MSE: 0.00772802\n",
      "3 Train MSE: 0.00740893\n",
      "Training phase #2\n",
      "0 Train MSE: 0.295499\n",
      "1 Train MSE: 0.00594454\n",
      "2 Train MSE: 0.00310264\n",
      "3 Train MSE: 0.00249803\n",
      "Test MSE: 0.00979144\n"
     ]
    }
   ],
   "source": [
    "training_ops = [phase1_training_op, phase2_training_op]\n",
    "reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]\n",
    "n_epochs = [4, 4]\n",
    "batch_sizes = [150, 150]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for phase in range(2):\n",
    "        print(\"Training phase #{}\".format(phase + 1))\n",
    "        for epoch in range(n_epochs[phase]):\n",
    "            n_batches = mnist.train.num_examples // batch_sizes[phase]\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])\n",
    "                sess.run(training_ops[phase], feed_dict={X: X_batch})\n",
    "            loss_train = reconstruction_losses[phase].eval(feed_dict={X: X_batch})\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "            saver.save(sess, \"./my_model_one_at_a_time.ckpt\")\n",
    "    loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})\n",
    "    print(\"Test MSE:\", loss_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cache the frozen layer outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training phase #1\n",
      "0 Train MSE: 0.00753817\n",
      "1 Train MSE: 0.00775457\n",
      "2 Train MSE: 0.00734359\n",
      "3 Train MSE: 0.00783768\n",
      "Training phase #2\n",
      "0 Train MSE: 0.200137\n",
      "1 Train MSE: 0.00520852\n",
      "2 Train MSE: 0.00259211\n",
      "3 Train MSE: 0.00210128\n",
      "Test MSE: 0.0097786\n"
     ]
    }
   ],
   "source": [
    "training_ops = [phase1_training_op, phase2_training_op]\n",
    "reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]\n",
    "n_epochs = [4, 4]\n",
    "batch_sizes = [150, 150]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for phase in range(2):\n",
    "        print(\"Training phase #{}\".format(phase + 1))\n",
    "        if phase == 1:\n",
    "            hidden1_cache = hidden1.eval(feed_dict={X: mnist.train.images})\n",
    "        for epoch in range(n_epochs[phase]):\n",
    "            n_batches = mnist.train.num_examples // batch_sizes[phase]\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                if phase == 1:\n",
    "                    indices = rnd.permutation(mnist.train.num_examples)\n",
    "                    hidden1_batch = hidden1_cache[indices[:batch_sizes[phase]]]\n",
    "                    feed_dict = {hidden1: hidden1_batch}\n",
    "                    sess.run(training_ops[phase], feed_dict=feed_dict)\n",
    "                else:\n",
    "                    X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])\n",
    "                    feed_dict = {X: X_batch}\n",
    "                    sess.run(training_ops[phase], feed_dict=feed_dict)\n",
    "            loss_train = reconstruction_losses[phase].eval(feed_dict=feed_dict)\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "            saver.save(sess, \"./my_model_cache_frozen.ckpt\")\n",
    "    loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})\n",
    "    print(\"Test MSE:\", loss_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the Reconstructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_one_at_a_time.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUEAAAD/CAYAAABvuWSAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFoNJREFUeJzt3VlsVVUXwPHNPJVSJlskzBUoiIiKjBJCDIEYJWowUR4w\naiIx0UQSNZiYqC8+Ki9GjQYUh5gYJxwjRohMBlFEQbAMhlJoKUNboMz4PX37W2t93O3l9l5auv6/\np32y7r3n3Oayc9Zi7X3a/fPPPwEAvGrf0hcAAC2JSRCAa0yCAFxjEgTgGpMgANeYBAG4xiQIwDUm\nQQCuMQkCcK1jC52XZSqtR7uWvoC2pLa2lt92K1FaWprVb5s7QQCuMQkCcI1JEIBrLVUTBHCFyR2j\n2rXT5TJ5fPHixYwx+znt27fPGLtacCcIwDUmQQCukQ4DrVwqHc01/bTv69SpUxyfPXs2+VqZLtt0\nWMY6dOigYhcuXLjk6+zn2M8sNO4EAbjGJAjANSZBAK5REwRaoWzrfra2lm37iqwBhhDCmTNn4tjW\nBDt27Jjx2MbkOc6dO6diskZo64WSrYHK72hj+cCdIADXmAQBuEY6DLQCl9PqItNDm9bKNNOmjrJF\nxcZkCmzTaPm+EELo0qVLxphMgW1aLV/bo0cPFUu1xciUO5X+54o7QQCuMQkCcI1JEIBr1ASBVkjW\n9lJLzGRri4117tw542davXv3juOmpiYVs60uDQ0NGc/Rs2fPjO9L7WKTqvvJ6059Zq64EwTgGpMg\nANdIh4FW6PTp03F84sSJjLFUetivXz8Vk6lrr169VEy2vRQXF6uYTbllKmvTUZnWyuu077POnz+f\n8TNlq41NlbP9/BTuBAG4xiQIwDUmQQCutZma4MaNG9Xx0qVL43jgwIEq1q1btzheuHChivXp0+eS\nY6C5Uu0cdvlZY2NjHFdWVqrY/v3747i+vl7FZB1OtquEEMLQoUPjuLS0VMVkTdAuabPXLc9pa3Sy\nJti1a1cVk60u/fv3V7GSkpKM58tWru/jThCAa0yCAFxr10LPCc37SUeNGqWObQqRLdk6MHny5GZd\nUy5kyrJkyRIVGzx4cCFOmf9dKh2rra3N+NtO/Vuz7SQHDx6M482bN6vYzp075flU7NSpU3FsW1tk\nymvPJ1eJ2DTWtszs3bs3jouKilRsxIgRl/zMEEIYNmxYHE+bNk3Fxo0bF8c2HU+1vqQ2XC0tLc3q\nt82dIADXmAQBuMYkCMC1NtMi8+mnn6rjLVu2xPHYsWNVbNu2bXH8008/qdhnn30Wx99++62KyZqG\nrIv8G/swmgEDBsRxVVVVxvfJ+mAIITzzzDNZnxOtg6wD2naS1MPI5fvsDs2yZidrcCGk637V1dVx\n/Oeff6qY3KHaXsuRI0cyvtbWC2WN8vfff88YGz16tIrJZXPyO1h2mWA+cCcIwDUmQQCutZkWmXyR\nKcTff/+tYjId3rNnT9afaTeelOmw/MwQQqirq4vjTz75RMXmzZuX9TkvAy0yeWRbZOS/L5nyhaBb\nOmxMpqA2HZbHcvVTCOndZ44dO3bJc4egU+CTJ0+qmD2W7Wg2dV6zZk0c29+vXCVi279uvfXWOLbf\nSf5t7N8ptVErLTIAkAUmQQCuMQkCcK3NtMjki2w/sP+NL1VUVOR8DtmWc/jwYRWbNGlSHM+ePTvn\nc6D1ST3oyC5Vk8s3bU3ZPnBdksvmbL1fnt8ud5O1PfkZl/oc+VAmWWcMIYTVq1fHsa1lyt2cZF08\nBP0d7Y468toK8X8Y3AkCcI1JEIBrpMNXgG0xuPvuu+PYthi88sorcWxbBXD1kW0btoVDPiPYpnly\nJ5XU84PtZ8rX2l1k5Dls+i3ZVRky/bXn3L17t4rt2LEj4zlkG4x9CJT8Tva6U89g5rnDANBMTIIA\nXGMSBOAaNcErYPny5eq4pqYmjvv27atiQ4YMuRKXhFZA1tZs+4ysy9m6V0NDQxzbVhf5OfZ9srXG\n1t3k+VItOCGEcOjQoTj+6quvVEzugn3bbbep2MyZMzNet5Sq+8k6qn0tD18HgBwwCQJwjXS4QGTr\nwOLFizO+bsOGDeq4rKysYNeE1iWV5kk2VZbvsxunyvTQbk4qU2B7Phmzqapt8ZIbsv74448qJtu6\n7IPK5IPCbPtMajeY1IqRVJtRtrgTBOAakyAA15gEAbhGTbBAVq5cGcd2GdL8+fPjePjw4VfsmtC6\nyFpX6gFCtmWke/fucWx3Wk7txiJj9jPlrjW2Raa+vl4dy4ea2R3W58yZE8f2AeuyDmivTdYBbb1S\nvjZVH80Vd4IAXGMSBOAakyAA16gJ5omt6cgnbdl+rZdeeimOU7sNo22x9StZB7O7MMu6mO0FlLFU\nbc324snfqK0Jyt+hrTOuW7dOHW/atCmOy8vLVUw+EXHo0KEZr9ueX37/1LI5G5NYNgcAOWASBOAa\n6XCevPXWW+pYLid64IEHVIy2GISg07zUw4Vsmiffl2oZsSl2U1PTJV8XQggdO/5vKpC71IQQwg8/\n/KCO5YOYJkyYoGLjx4+P4+LiYhWTD4O331emyrZFRrb22OumRQYAmolJEIBrTIIAXKMmmKMtW7ao\n48cff1wdl5SUxPGLL754Ra4JV5dUS0dqqZismdn2K1kHtA9Rl0+ws3U32YYjl3yGoHeLDkH/tmfN\nmqViffr0iWPbNpb6TqnlfvLYtu/I70GLDADkgEkQgGukw5dBphf333+/itlb+AULFsQxLTF+pVo4\nUjtLy9UeNq2U7Sx2BUUqVU61msiWrhUrVqjYgQMH1PHChQvjeNKkSSom01zZkhOC/jdid6qRaa79\nTvLY/p3YWRoAmolJEIBrTIIAXKMmmGBrE3fccUcc79y5U8UqKirU8QsvvFC4C8NVybZwyBpZajmY\nrAGGoOtnqafUyRpgCLpeV1VVpWJyt+ht27ap2O23366OZ8+eHceyXcZet/1Oss5pW13kd0zttmNj\nPHwdAJqJSRCAa6TDCUePHlXHq1evzvha21YgO+fhl0zRUq0fKanVHTYm00rbtiVbbVatWqViv/zy\nSxyPGTNGxWw7mNw5xqbq8iHulnxtqjRQiI1TU7gTBOAakyAA15gEAbhGTdCQu+pOnjw54+veffdd\ndWx32AUsW8/K9YHjcsmZrcnJNhT7mdXV1XEsH5YUQgh1dXVxfN9996nYxIkTQyaph7/Lh8SHoL+v\nrR3KOqBt7ZG1TPudsq2rpnAnCMA1JkEArpEOG8uWLYvjPXv2ZHzd9OnT1XEh/usebUtqdYclU+BU\nOmxTx5MnT8bx4cOHVez999+P488//1zF5MqPkSNHqljfvn3VsUy5bToqU1f7b0KmzjaNT22cmmoz\nygfuBAG4xiQIwDUmQQCuua8JVlZWquPnn3++ZS4EbV5q5+NUvfBy6s1yN+ft27er2BdffBHH9gHr\nN9xwQxyPGDFCxWxNsr6+Po5tq4usUdqdrVM7Ysvj1N+JZXMAkGdMggBcc58OywfMhBBCY2NjxtfK\njVO7detWsGuCD6nNQiX7oCWZnsrnDIeg00obGzVqVBzbXY5ky1dZWZmKyV1r7LXaByal0nrb+pLp\nM61Ct59xJwjANSZBAK4xCQJwzX1NMGXq1Knq+LvvvotjaoIopNQDi1JL02QbSnl5uYrNnTs3jm0t\nb9CgQXFcVFSkYraWJ1tfUrU8u7O1vNbUjjpXGneCAFxjEgTgWrvU7WwBtchJcUlsf5NHtbW1V/S3\nndqpxZLtK6ln+6be92+vlSlwKuW9EvNOaWlpVr9t7gQBuMYkCMA1JkEArrVUTRAAWgXuBAG4xiQI\nwDUmQQCuMQkCcI1JEIBrTIIAXGMSBOAakyAA15gEAbjGJAjANSZBAK4xCQJwjUkQgGtMggBcYxIE\n4BqTIADXmAQBuMYkCMA1JkEArnVsofPyYJPWg+cO51FNTQ2/7VairKyM5w4DwL9hEgTgGpMgANda\nqiYI4AqTzxhv1y5zuezixYvq2L5Wfk779lf/fdTV/w0AoBmYBAG4RjoMtCEXLlyIY5m22uMOHTqo\nWOfOnTO+73KcOnUqjjt21NOLPKdNo8+fPx/HqVS9ELgTBOAakyAA15gEAbjmvib43nvvqeOTJ0/G\n8ebNm1XsjTfeyPg5zz33nDqeNWtWHM+cObMZVwiPZJuKrJdZp0+fVsdnz57N+L6GhoY4bmxsVDFZ\nv5OfcalzlJSUxHFxcbGK9ejRI467deuWMfZvbThStq09ueJOEIBrTIIAXGvXnP8Ob4YW3Wnjscce\ni+PXX3+9IOcYM2ZMHK9du1bFevXqVZBz5ohdZPIo111kZGtLCDrtO3PmjIo1NTXF8aFDh1Rs7969\ncbxlyxYVk7/DY8eOqdjx48fjuHv37ipWWlqqjseOHRvHc+fOVbGKioo4tr/zLl26xLFtkZHfybbW\nyL+Fbe2RabX9THaRAYAsMAkCcI1JEIBrLlpkZA0whOzrgBMmTFDH9957bxxXVlaq2Ntvv62Ot2/f\nHscfffSRij388MNZnR9+2NaPVIvM4cOH43j9+vUqtnPnzjj++OOPM55v+vTp6nj8+PFxfOTIERXb\nt29fxvMfPHgw4+fKlpgQ9HeydU5ZB7T/TyH/NufOnVOxfCz3404QgGtMggBca7PpsLyFf/PNNzO+\nbuLEier4m2++iWPbKiBvvW1Lw65du9TxunXr4limD8B/pVZCyN+X3JklBL3aw678qKqqiuMpU6ao\nmFzFNGfOHBWTq0B+/vlnFbPlHNlOY3/b1dXVcVxWVqZiMgW2Kb5Mh1OrQjp16qSO5efY9plscScI\nwDUmQQCuMQkCcK3N1gRlrcL+17msA65atUrFioqKsvr85cuXq+NNmzZlfO28efOy+kz4Imtftg6W\nqj/Lutg111yjYjNmzIhjuXQzhBBuuummOL7uuutUbP/+/XFcX1+vYnV1depYxm0bTO/evePY/luS\ndT+7M43dVUZK7Uidax1Q4k4QgGtMggBca7PpsLz1t/+NL1MNu/Fjtmzbjd2IErBSz++1bTDyN2p3\nY+natWsc2zYuafDgweq4X79+cXz06FEVkyugvvzySxX79ddf1bFMwW06LlttZCtNCDqV7dmzp4ql\nVoWkHhCVD9wJAnCNSRCAa0yCAFxrszVBKV87Oa9YsSKOf/vtt+RrZ8+eHccjRozIy/lx9ZG1Ltve\nIVtfbI1MtoykHpRu21Dk7s1yHIKurdm2F7lU7q+//lIx2wYj23BGjhypYvI72SVuMmb/FqkdojN9\nRgjpNqNscScIwDUmQQCuuUiHc2VbAx599NE4tptCDhgwQB0vXbo0jm1aAD9kCmpTOZnm2ocLSbZ9\nRn6OXKFhYzJtDkG3cdl0eMeOHRnfZ3dakhunDho0SMXkri7yGd4h6NYe+33ltdm/U6FxJwjANSZB\nAK4xCQJwjZpgwoYNG9SxrQNKixYtUse2dQCwuxmlal+yvcXW1mQ7if1MGbPLz7Zu3RrHH3zwgYpt\n3Lgxju2O0DfffLM6vuWWW+LYLpuTO1vbVhf5nWw7S6rVRR7b3WZybYuRuBME4BqTIADXSIeNhx56\nKI4//PDDjK978skn1fHTTz9dsGtC2yTTRZvW2k1HJdlOYttn5K5IdqcYufHvmjVrVEy269jdZ268\n8UZ1LFNgu0OTbAezrWHy2mwpQKbuNo1OPZAqH7gTBOAakyAA15gEAbjmviZ44sQJdfz111/Hsa3L\nlJaWxvGzzz6rYnapEWCldpa27SyyZmbrhXJXJLszujzHH3/8oWJy5yP7mddff30c33nnnSpWXl6u\njuW/C/sQ9dSyOVnrsw9zksv/7JK6VNuP/B6p3WdSuBME4BqTIADX3KfD8+fPV8eHDh3K+Nonnngi\njvv06VOwa4IPMnW1G6fK1C71MDCb1u7bty+Ov//+exVbu3ZtHBcXF6uY3Ph33LhxKmY3JZYrp+yD\nnmQ6bEtEjY2NcZxK4+37ZFpt0+98lKG4EwTgGpMgANeYBAG45rImuHnz5jhevXp1xtfdc8896njx\n4sWFuiQ4JFs/bPuMbBOxO6fIGpltGdm+fXsc24eByfrh6NGjVUy2yNhrsUvz5IOXZJ3PXo9dGifr\ngLaWKWugDQ0NKpZ66FQ+cCcIwDUmQQCuMQkCcM1FTdDWNJYsWRLHclsiy+6oy9I4NIetg0mpXaZT\n22zt2rVLxVauXBnH27ZtU7ExY8bE8fjx41VsyJAhcZzaAisEveSttrZWxeR7bS+trCXapXGy9/D4\n8eMqJl9rHwSfD9wJAnCNSRCAay7S4ddee00d2+VEktxZmpYY5JNtPZFtIXYHFLs8TDp27Fgc24eB\n7d69O45t6jhp0qQ4njp1qorJ3aJTabuNFxUVqVj//v3j2LazpNpbZFuMbbuRabX9DPl3yrV9hjtB\nAK4xCQJwjUkQgGsuaoJ2F+iUl19+OY5piUE+pZaK2V2YZVuIfZ98rX2iXF1dXRx37dpVxQYNGnTJ\nsZVathaCrsPZc8jrtm0/TU1NcXzgwAEVq66ujmP7feW/Q3s+iZogAOSASRCAay7S4cshH7yU64Nb\nQgihS5cucWxv02WaIDvlLbvSZenSpVmd255PlgPsagBcOTbNk7+D1CoNu1OMPK6srFSxmpqaOJbt\nKiHolRj79+9XMZlW21VU8rccQvqhSLJFx7a6yGurqqpSMfmdKioqVGzgwIFxbP9NNuffaPyMZn8C\nAFzFmAQBuMYkCMA1aoKGrD80x6JFi+L42muvVTFZG3n11Vfzcr4U+Z0eeeSRgp8Pl2aXzWW7Y7Kt\nCcpdZGwbijy2dTe5XNTuMCPrz7KVJYT/f9qcrBHK+mAIuvVFLu+z7HWPGjUqjm19VLbI2Lqq/Zvm\ngjtBAK4xCQJwzUU6vGDBAnW8bNmygp/T7lyTLdlykEqRHnzwQXU8ZcqUjK+dNm1aTteC/Eqlw6m0\nzm5qWlpaGsczZsxQMZmC2nR4/fr1Gc8nV6H069dPxWwKKnenGT58eMbrTpEPew8hhLKysjgeNmyY\nipWUlMSxbcnJB+4EAbjGJAjANSZBAK61+7ddZAukRU76X++8804cpx60ZMmHWV9Oa8tTTz2ljsvL\nyzO+9q677opjudtvATW/xwBRTU1N1r9tWRO0tS4ZsztEy+VodmmlbFHZunWrisklobZ9RS6j27hx\no4rJep29VvvApu7du8exbZ+RtUxb55RtZPZ9skXncpZ9lpWVZfXb5k4QgGtMggBcc5kOQyEdzqNU\nOmxbnmTKa3dDkf8u7fvkCgobk6tL7KqM1I4rciPVgwcPZnxdCDrllWPLboAqd0yyG7XKNNeWBlIP\npEohHQaALDAJAnCNSRCAay6WzQGtga3RSbZGJmuCdombbIuxMVkTlLvNhKBrifah6bItRT7s/FLn\nkNdmr1vWKO3/N8gHNNmHy8u/TSF2iknhThCAa0yCAFwjHQZaAZsCymO7qapkYzKttKmqbD2RD10K\nQaectg3FtqzIVNau4Eh9TiodTiEdBoACYhIE4BqTIADXqAkCrZCsg6VaVOyD0WWtzdYEZY3O1vlS\ndTdbv5N1SBtLPRRJssv9Wmj5bgiBO0EAzjEJAnCNdBi4yqRS19SDiFIPdkrF7HFqF5tsHx7Vkumv\nxZ0gANeYBAG4xiQIwLWW2lkaAFoF7gQBuMYkCMA1JkEArjEJAnCNSRCAa0yCAFxjEgTgGpMgANeY\nBAG4xiQIwDUmQQCuMQkCcI1JEIBrTIIAXGMSBOAakyAA15gEAbjGJAjANSZBAK4xCQJwjUkQgGtM\nggBcYxIE4BqTIADXmAQBuMYkCMC1/wBFeVZKJWQkbwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdd169fe6a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_test_digits = 2\n",
    "X_test = mnist.test.images[:n_test_digits]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_one_at_a_time.ckpt\") # not shown in the book\n",
    "    outputs_val = outputs.eval(feed_dict={X: X_test})\n",
    "\n",
    "def plot_image(image, shape=[28, 28]):\n",
    "    plt.imshow(image.reshape(shape), cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")\n",
    "\n",
    "for digit_index in range(n_test_digits):\n",
    "    plt.subplot(n_test_digits, 2, digit_index * 2 + 1)\n",
    "    plot_image(X_test[digit_index])\n",
    "    plt.subplot(n_test_digits, 2, digit_index * 2 + 2)\n",
    "    plot_image(outputs_val[digit_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the extracted features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_one_at_a_time.ckpt\n",
      "Saving figure extracted_features_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABWCAYAAACaXQIdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH6ZJREFUeJztXctzHFf1/qZ73qMZPayHHcnBimzLgEsiJASbJCQVijdV\nWacoWEAVG/4dFhQFCyiqWLGASkFRLCBOYooAIcH4FduxkGSNLHs00mgkzbt/i67v9O3bShH/4pS6\nzfk2I0339EyfPvee7zzuuSnP86BQKBQKRdzgHPYPUCgUCoXiIKiBUigUCkUsoQZKoVAoFLGEGiiF\nQqFQxBJqoBQKhUIRS6iBUigUCkUsoQZKoVAoFLGEGiiFQqFQxBJqoBQKhUIRS6iBUigUCkUskT6M\nL/3JT37y0PsrpVIpOI5vb/v9/sO+fAg/+MEPUh/rF/w/8Jvf/Oahy9R1XbiuCwDodDoP+/IhvPzy\ny7GS6c9+9rOPRZ6tVgsAkMlkHvblQ/j+978fK3n+6le/SnRPtVdeeSVW8vzFL37x0OXZ6XQwOjoK\nANjd3X3Ylw/hu9/97oeSp3pQCoVCoYglDsWD+v+iUCig3W4DAAaDQeQ4G9+mUr5xdhwH6XT4Frvd\nrvxNFttsNkP//y/B8zyMjIwAADY3NwEEsnUcJyJv13VFzvQG8vl85Lp8Bv8r4P3u7u6KB18sFgFA\nvNBWqyVePuG6rninvV4vdMxxHJHtxx0ViCscx4lERiivdDotcqdO9vt9kRnlzjE/GAwic8SjAOqN\nPdcdBDMSwvmOnxsaGpIxzWua453nm3MBZftxyTN2BsoUNidHCqZWqyGbzQIIBELhZjIZOUbF7Pf7\n2NvbC12Xyt7r9UKCJszPPirgvdiTI+BPmtvb2wCAI0eOAAgGtOd5ESIwGAxE9nb4r1AooFAohM7f\n3NwU2T+KhoyDl7KgsTePEfl8XmRAOe3v76NWqwEInhNfS6WSyJ9jwXEcCcOQWNmTdNJAvTQNCOU0\nGAzk3nO5nLwH+GSAn+XE2mq1RLalUglAIJ9CoSDvcV7Y3d2NkICk6aRtmCifVCoV0hvAlwllvL+/\nL+fxf3PsA/48eezYsdA1KGvXdVEulwEEIcFOpyPzsEkMzO95EGiIT6FQKBSxROw8KFpfIMoozZCd\n7RHl83kJqfB1f38f9XodALC+vg4gYPuVSgUTExMAwq4rrb0Z5ko6KFPTI6JcGo0GxsfHAQRypjdQ\nLpcxPDwces+UEdkZ2Wi73RZ2RZRKpVBY1fyepIF6wvvu9XoRT951XZEH2SV1LpvNijypo6lUSkKr\nlBOfV6VSEVnxu02P91HQTSBg66Z3Th3LZDIiA8p9Z2cHgK/D9+7dAwDxQk0Pk+ObryMjIyIzM7z1\nQR5a0mCHL9PptMiC97i/vx+Jcpif42cpn8nJSfksZT00NAQAGBsbk3mB0ZfNzU2RI70qXiuVSj2w\nl/9oaLhCoVAoHjnExoOymVOr1RJGSdbT6XQ+sPzRTOaRzZoMlKyUOYIjR45EGG6n0xHPgt9ZqVTk\nWFKZlenV2HmLbreLS5cuAfC9KSDsZX7iE58AAMzOzgLwmb/NsugNHFSA4nmeMDaex+/Z29sL5f/i\nDsqFOpLP5+U+zZwGdY6yJqMcHx/H448/HnrPdV25Hq9F1lsqlSRSQC+rWCzK9W19TKVSifROKbuR\nkREZu2Zug7KiHjN30m63RT4c+91uVzws6imvNT4+Ln9T5s1mUzwEvprFP9TZJKJQKODu3bsAwvkf\ne56kDBuNhhybnp4G4D+b999/X44DwNzcHABfTzl+OS+bHj6fK685NDQUipB9GKgHpVAoFIpY4tA9\nKDvXQxYDRKtM0um0eDR8j7Hnmzdv4tq1awACBvrMM89gfn5ePgsEVr1SqUh1Cr9zfX1dGJNdbtnv\n9xPD9s2Yvolmsyks/V//+hcA4MKFC9ja2gIQVJY99thj8hlWODF3Uq1Wsby8DMCPTwMBo8pkMnIt\nym1qaiqUvwKCfFZScih2lRxlUiwWRU+oN51OR+6PMqBn2Wg0RKeph0tLSyIzsntes16vC2ulZ5TP\n5+X6B7HRJFTy2b+ROdBerycyoww8zxN2vra2BiBg61NTU+LZ01ttt9siY3pSlLXruqLH9+/fBwAs\nLy9HniHlWiwWE6OjQLhaEfDv367YGx8fl/mRekwdu3XrlniW9IJc1xXd+/znPw8AOHfuHAB/vC8t\nLQEAbt++DQC4c+eO/B6zohXwPSg7H/3fcOgGygaFYd4Ib3Rubg5TU1MAAgVmeGppaUmMF5Ois7Oz\nOH36NIBAgO+++y4Af/DzWlTCVCol4QRzLRXgG7gkDH4gMABjY2MAgNXVVQC+2/7nP/8ZAPCXv/wF\ngG+Un3jiCQDA5z73OQDA8ePHAfghpk9+8pMAgsTo+++/j+vXrwMIDA7PqVQqMlHw/KGhIZGlHXrJ\n5XIfe4eKhwGz5BYAZmZmAPiTIsuWSXIKhYLoMO/XXGdH48773tnZkUmTEws/3+v1ZMAzxDc5OSl6\ny/MPSnLHGSRJ5n0Cvr5Rjpw8e72eEE+GmkiMFhYWImM4nU6LvC9fvgwgCOfV63UZ33yvVqvJXELZ\nmYbqQSfUwwCfPw0r9W5ra0v0jDrW6/VEZylHPg9zWQNDfM1mU+RPWfPZpNNpuT5DidVqVcY+522z\n3F+LJBQKhULxSOBQPSjTotqlpJ1OR9gOPaLHHntM2A3dfDKckydPiifAa5pJVIat6Nq3221x/fm6\nvb0tDILslMzMcZxEsCkgYOf0GsmQbt68KUycbv7s7Ky47IuLiwAC2d65c0fYKBnY+Pi4eBD0tOi5\nttttYU8mSzYXYgIIrUw3Q7pxBZk175MhOc/zJMRM1prNZiWczNAJQ1MbGxvifZJVTkxMiF7xGMMx\nk5OTMgZ4/W63K9cnk+U56XRaPhtXDAYDkR/v++jRowDCXU34XqfTwcrKCoBA3z796U8D8HWYHibH\n9cjISGQuoaxzuVwo3Af48qdsTTkCvreRhKITjieOe7Pog/fO+/U8Tzwtu+PG1NQUnn32WQDBXLCx\nsYEbN24AAH73u98BCObEM2fOiOxY/FOpVOT3sMCK6Ha7D+zhqwelUCgUiljiUD2o3d1dYSt2nD+f\nz4faGAE+I+WCWybxTp06BcBnU2Szr7/+urySYT333HMAIEUTb7/9Nv7+978DCBjyxsaGsDrmbxin\ndV33gUskDwOpVEpi6nZrkna7LfdFT2dubg5PP/00gIC1Mp68vr4u90wPanR0NJKnI3vd2dmJeLb1\nel1YHJ8jGVs6nU5UEpo6ytd6vS76xfudnp7GmTNnAAS5J8pwZWVFzqc3Vi6XI+21qPeUMxAshNzZ\n2RHPjO/x/H6/H3sdbTQa4knT26P+HFSGnE6npYiCkRTmQm7cuIE33ngDQPBMZmZmJPpB+Z04cUKO\n8bv5PeVyObJ43ewvF/fCqMFgIL/RLp6ZnJwU2fFYLpeLLPe4evUqAD9/dPbsWQDhPBwX6HJepVyr\n1arMJ4xenTp1KtQrEQieM6/zIDj0Igm7oSMFY/bWowKtra3JZEvB0I0sFosHrpGiIaOweM1//vOf\nIjAm+avVqhg5FldwIk/KZOp5nhgT3ovZj9BsAgn4cuaApjJzsN++fTu0DgwA3njjDfzpT38C4IdV\nAUg15PDwsHy3uS6CEwvDimbzzyQUntghI772+/3IWpvFxUV85jOfARCQLSac33nnHQlXMbwyMTEh\ncnnyyScBBIb/5s2b2NjYABA8m4N00OxwEXd5uq4rhpvjmiG/er0u5IivnU5HwtJ2cUitVhN9oxGa\nnZ0VuZMMUBebzWZoPAP++Obz5DEzDBj3kGkmkxECyHsyjQsJNmU2GAzk/miYSNQdx8HNmzcBBMbr\n1Vdfxa9//WsAQaiUur63txeptC4Wi/I7zHQNf5/dEPm/If4zrkKhUCj+J3GoHpS5SpvMzywXtctu\nNzc3I6vJeY1SqYT33nsPQMBO5+bmJHnHIgmWmf/hD3+QUAwTgq7rSkLb7NjL35MEOI4jv52/md7p\n6uoqrly5AiCQW6/XE8ZPkL1eu3ZNWBCZ/89//nNhV/RiyWzHx8clyc0wAL04ILr+xfRA4gy7e7vZ\ngYQM3vTQydhtj/7EiROREut+vx/ZnoPPb2lpSTx+hlDS6bR4FHb0IQlwHEd0g7pFj6perwu751je\n3d2VKAjX2zHikc1mpXCCere/vy+eE8vMzV6UnA8YzhsMBqFlJuZrEgp4crlcqAACQKhzi90JY2dn\nB7du3QIQLD/hOH788cdFj3//+98DAH7729/K8e985zsAgOeffx6AH7Ljekp6YVtbW5FnYpfxPwjU\ng1IoFApFLHHoboHNWmy2CoT3wiG7J+syV0iTFdHinz59WuL/ZLN//OMfAfgFET/84Q8BAF/+8pcB\n+GyWrItgDuZBY6eHBc/zIgyVHs/169dx8eJFAEFOY2hoKJKs/tvf/gbA95ooN3qeqVQKX/nKVwAE\n/fnISqempiK5xMFg8IFx/F6vF/tNIs3Fr3w1+zpSL5jPvHDhgjByluCSUU5NTclnmVuq1WoiMxYA\nMSeztrYmq/cXFhYA+HF/eqWUKyMGnufF3iPN5XKRjuLU01QqFdk7qNvtSpkzj5nl4NRt6tvIyIjM\nA9RnM0fKa3DRbz6flznC3BcJSEank3a7Hdl3iTqWzWbF+6Scbty4IXrJue2LX/wiAD/KRN3697//\nDQD4whe+IJ7ryy+/HPpco9EQj5d56UuXLon8mKOmXIeHhzUHpVAoFIpHA4fuQdl7EJkMkOzdjO/z\nfOYByDD7/b4wJlr8EydOiDV/8803AUDadiwuLuIb3/gGgMDjKpfLcl27r9UHdVGPGwaDgTAou6xz\neXlZ7oNVikePHpV7JRtiRdTU1JSwecrxpZdewjPPPAMAkb5pKysr8p3MSw0NDQmj5TGWs2ez2dhX\nnZleif1bR0dHI/sR1ev1CNM3l1CYW8MDfiSA0QDqOV+np6dlUSplNj4+Lu29KFdWspq91+IKs2UY\nq7tYPTcYDMTbYQl9rVYTxs4qMnqMOzs7ouvMMZs7xtJDowxnZmYkf0edzWQykfPp0Zmd4+OKXq8X\nWQhvls3zPtli6/79+yJHyp1je3t7W8a0OQdQxtQ3Vp2eOnVKjrH67+rVqyJjyvGj7Ahx6AaKwrW3\nct/Z2ZHJ0dxymMrEAgcmj80eUExct1otvPXWWwCCsMnXvvY1AL5B5PkMG9brdQkV0FCZLmkStttI\npVLiwlMR2dBxe3tb7o+G/fz586JAnFC5LgoIXH0q+pkzZySkR9lQtpcuXZLSVR4bHx8X+dp907rd\nbmitTxxhhp3MbVwAnxTZ20OMjY1JuTjJDWV4/fp1mQQ5yZohROqj2aGD30WZlctlGQMM3/DZdDqd\nRGwPYW4QCgSyyGQyEm7jGH7iiSdw/vx5AIHRotHwPE/KqCmfGzduoFqtAghIACfMXq8XKu+3f4+9\njUy73ZbnGlcc1N+O8jHXjf7nP/8B4N8bZUu5sNDh1KlTkfWft27dEuNDMsbih+npaTFkLLRaWlqS\nkCPnIY77fr8vJPjDQkN8CoVCoYglDt2DIsiAGOrb3NwUS0yGcP/+fUngs28ZE6jr6+vCWHmNarWK\n1157DQDwve99DwAkrLe+vh7ynAA/KU0Gam8H3263E5E0BaIl8WY4id7oU089BcBPvtteDOW3vb0t\noQGGoUzPluySr0eOHBFPmCGvZrMp8qVMzX6HcS8+MUN8dkGH4zgiT3o6Z86ckX5mBFnpq6++Gtm8\nsVqtCuM1l1gAPuOnl2RuL8NQNgtdyFQzmUzs5QkE92kv6j5y5Ii8x/BQvV6XsnL2gKQs+v2+nEev\naXt7WzwtllNfuHABgF/Uw04JvObW1lakmw29rG63G/uFugeB82atVhMPijpSKBQkssKwKIudJiYm\nJOxqFupQx1leTiwtLYnXz3Omp6cjTRHMghfdsFChUCgUjwRi40HR6ptbCNM6k3l3Oh1hNP/4xz8A\nBPsaZTIZfPWrXwUQJP0uXrwoCWXGVMnQ9vf3pdu3uaWxvacUWXO32419STRBBkPmbvZqYzyY95fP\n50Pd34FwboneFfMix44dk/dYKs1nNzs7K94Rc1HvvfeeHGe8mnJstVqxT+oD0SUQ5kJZMkcik8mE\nch7m6+LiYqhABPAXS5LlMvlstlaiB0Uv49q1a+KdkgmbZcVJWLRLHbEXcI6Pj8v4Z9702rVreOed\ndwAEOsjzG42GREGo6y+++KLkUOlVcXz3+/1IX0jP8yLLMfgbOp1OIqIm9hiiruzs7ERynr1eT+TP\nKArLwXu9nsidC6VXV1fxzW9+E0CQt+ac+stf/jKyg0SxWIz0BjR/34Mugzh0A8XBa/dvKhaLkoxn\nxdnZs2elQs9sUAr4rj0nQBqhmZkZUTqub6KCmopn7vTKiZsKb67RSArsKilzV1yG4xh2q1arkc0Z\niUajIQpOA3/y5Ek5nxMrw6zz8/MSOmHHisuXL0s4i7I8aB1LnEEd5e/nBJBKpSKJ+4sXL+LHP/5x\n6HM0PIuLi5FecDMzM/jUpz4V+j4SJ7NXIZPcV69elWfHxsd2c864g+ONv5tjznXd0OaFfCURIvFk\ng+j5+XmpKOV49TxPDBKT+V/60pcA+OEnPiez8IXXtXd8TgrsnqVmw22G8RgOXllZCW0vBCC0a7O9\nYeH6+nqkupEdKG7duiUkg7IuFotC8u1OEsCD7/gcf3qgUCgUiv9JHDrlImunZaUFHx4elnAUGWix\nWBS2QGtOT8rsTk4GNDc3J2ydvaJYUvnCCy/Idc3Ny+yQDb8nm80moswcCEJuZvIZ8Jkn75msZnt7\nW843k6uA72WZ3ijgh0bIqMjK6J2afff4971798SzpSdFD4DXjjuoo9Q1Ms5msylFJGSNa2trwvj5\nHr3P6enp0NYugL/2jDrKQhSGWfh5IPCqVldXhZnSG0valu+E7fHV63WRNb3KhYUFkTvP5xKTsbEx\n8RbI6n/0ox9JH7mvf/3rofObzabI1gwzsyCAnoW9nUzcYfcx5ThzXVeiT/RI19bWRFZMjzAcf/Lk\nSVlTx84ltVpNehsS9FBnZmbE06IXOjExIfrJudrsm0qv+cNCPSiFQqFQxBKH7kHZHZlZhru1tSU5\nDlr8ZrMppaO0ysxBra6uCktgPNose2bSjzHoTqcjncvNBWVkH2SztPhJYVNAIFN7X6jTp0+LB0Wv\nylwtz3s2y56ZsyJ73djYkPfIQvm51157TbxMehajo6OSlzL3qQGCGHjcYXeEoF5evnxZ7p33ePz4\ncbz00ksAgu4P9MonJydFPuamkMzhMc/E6y8sLAj7p+xOnz4t8rOLJJK2vxZzo/x/eXlZ/mb0ZHJy\nUvSEXieRy+XEu6IX/9Of/lQ8oW9961sAgvlje3tb9J/57W63G9lkMwml+qbXzDmK49Lc0p26S89o\nenpa7pfl9/R4mK8CgjFdLpcl2kI58pqf/exnZb7meDfnDHNBPuDP2Q+aJ1UPSqFQKBSxxKF6UGab\nDrIWMsVarSYtNlhmmsvlxHoz/2HuP8R4KL2Du3fvSv6DMVUy183NTWGsxOjoqMSj7cq2pOSfstls\npBszZTozMxNZKOd5nhznsyADGhoaEuZFtn7nzh1hQSynfvvttwEAb731VqSX2uzsrJSi2rvzJqH7\ntuu6IkfKgAz09u3b4oWTJZZKJZEf99Ayl1DQG2BPyHq9LnInQ6VcWf4LBHmsXC4X2ksKCPKJ/H1x\nhuM44uFTrnxdXl4W75HzwZNPPikytheNt1ot0S3OB9/+9rdlHqD3yVzLvXv3cO7cOQDBM2k0GpIT\n5dyShBwUf5u5IwDviWPc3EvLrPTke5zrzPZSzNExUuU4jlRZ2r03e72eRJ84JxQKBfk9dn9Vs1Xd\nh8WhGqh+vx9ZW8SbGQwGeP311wEESfUXX3xROkGwxJZFDFtbWxLaowBzuZwYKAqLkwaNHxCEE1zX\nlQdpl5o6jpMII9XtduV32hNrsVgUJTZLdzm4+SzotpfLZZEfwzHr6+tSaEIlZgFKp9ORCYMhlJGR\nEfl+u+lqnCcAYjAYREqTzc4Z1Dmu0dnd3ZWks11csbGxIYU8fD1+/LgsnSBhovEaGhqSMC0nkWaz\nKRMMx4VZYBR3mXqeJ/K0Q5WlUklkwHt0HEcKJqiX1LupqSnZAoJG5fz58yJ3hv3YdaZSqYg8Oc75\nbMxrmBsYxj1k6rpupMycaLVaIgMafs/zRMZsoksZDg8Py/nmVi6cTznXmr09qf9mRxVzGQmAEOl/\nUP3UEJ9CoVAoYolD9aAcxxFmbm9UZhYs0NtptVqR0mQWP1SrVQlHMQRjbsFBj4HdzbPZrDBXc3M9\nurEMI5idq5OAwWAQ2VacYVDz/hhGajQawkwZRiIjm5qakkS/GU4iwyczYrfpmZkZOZ8eaLvdjmz0\nZyIJjJ9eJ2XAe3z22WdFdiySKJfLwjRZBs5zarWasFH2P3v++eflPG4KSf01y3LNBL7N9O0ty+MO\nO2rC/+fn5yXMSR27e/euyJZyYjL/6aefFp2lN7m9vS3nUWaMkFQqFdFZhqb29/cjHeDt5gFxB38n\n79cMUbKbBgsiVlZWJGLyyiuvAAg82CtXrkg0hNd0XVeKrOxtfNrtdiSkb6YM6AV/lMiTelAKhUKh\niCUOnSKQrdglidPT05JvYl5jcnJSYqOMo3LRWCaTEebKPFOr1QqxVyBIps7Pzwv7MmP6ZP52aTGQ\nnEIJ29szY9RmSTLgM0i73JfPoNVqRfbUeffdd0Um3Pqdnmiz2ZQFpVzMmk6n5fpktrym67qJYf1A\nIAOz/RMTyIznHzt2LNT2BQj0plqtii5TL/P5vOgf5cprmmXUZPyu68rvoI7ymSapPBoIxjx/98zM\njGw/buaFGSXhGOb5c3Nzkl8ymf8LL7wAIHgmjJowDwMEXlg+n5dnYXc1T0IRTyqVkrJ6u89gqVSS\nCBDnzXq9Lu9x7mSE6s0335QyfC7wNfeP4jFGEMxu/nyWY2NjES/VlGGiiiSA4Afb66FKpZJsgsWC\niFQqJYKg60phLywsyEBlEnUwGMh1mbSnESuVSqFJmuD5duJxMBgkonGkuS7CXPMF+AbCXiOVy+Xk\nOF13s0+f3dfv+PHjsp0Eq6QYtrp69WoouQr4cuRxNplkyCYp63aokxyY5k7PHIxsXnzixAkJ33ES\npC7lcjnRV7NrBwcwDROfm1nhRz3c398Xw8TzkmCYTNghSepMv9+XcUrZNRoNMUw05DRe+XxeDDgn\n6aeeekomUlb9kcxubm5GwqOjo6Oiq/YcBDz4hHoY4Ni0O8dUKhUh4c899xwA4Ny5cyIf6tRf//pX\nuY65DQ/gy5w6y89RJltbWyJ3krG1tTUx9GY1NaFFEgqFQqF4JHDoHpQdVqKb2u12xSrztdvtCisi\nc6JVz2Qy4gGQsZfLZfksmRPZ2u7urlh/kzHZ21EnpTjChH0PlEexWJRjlEexWJTjlAcZv+M4Ii/K\n+ezZs8I46WkxdGJ2PmYoFUDkGmT8cQ+f2DDXegC+DlF2ZIlXrlwRT4thEpaIm+v4yOCz2WzoekAw\nBsyQs9lVnzK2iw2Soqt2N3PKdXd3N9IVwfSyGUqmBz4yMiLzgVlMwt5xDPuZGzpSdynz4eFhGf+c\nK+y1kHHHQT0NAT9iwmUi1CkzLG2HrKenp8Vzosw2NjZkXuB5PLa3tyd6zPey2azMH5TjR1lWoh6U\nQqFQKGKJ2HhQtMRkpPv7+8IeybiazaZYY3vVd61Wk1Jyfs4s0+V1+drpdIR5mB4Hy8vJOJIQg7bB\n++IrixNSqVSkC0EulxNWSeZPxm92cCdT3dzcFCbEz/H/SqUinhMX8wFBvousOCnFJoTtqZAhFotF\nuSfG+oHwluFAIAuznJnyz2QywuYJMs9erxfa/h3w9dLuW2nut5Mk2PlNz/NEduZiey4GtTvFeJ4n\n3jnnATNvx/eYy261WvJZytXc1O+gpH4SwN9rb7XuOI7oEgt38vm8eOr0+qlHR48eDW1WSlCezAHy\nWuZeWuY8bi9s/yjyVA9KoVAoFLHEoXtQNshm9vb2xJuhJfY8TzwtVpbwf7MSjxa+VCqFKq7M11Qq\nJdc1+0nZO/s+CjCZjJ1b29vb+0Cv1PM8OUbWbsa7yYwo71qtJs/B7CxvM6oktTo6CGY1J/XVXDTL\n+6LMzC3K7WUVuVwuIltzzxx6EswbNBqNSLQhCbD7xR10rNPpiG5w/DmOI3LhfMAK3k6nI2OXn2O+\nBAjkzorAdDot5/GaZn+4JEZLTNj9Q7vdrtwTx2Cj0Yj0cKT879+/f+COw4S900OhUJDvYtSq1WpF\ncswfZbzHzkCZiVO7XNrsj2evou/1eqEOCEBY4Sg0vg4GAxksDCs8qjAVwy7xrVQqkiy1iyoO6kVW\nKBQivfU4YRQKBZEvw3q9Xi+imEk1TIQ56M1wMhDWKzucms/nD5yoORnz1Wz6Sd1keCVp25ETBxkm\nwjTaJJT2+AaCsBPX2Jm9B0mSut2uFAkwnMcQ4e7ubij8BeCBN9BLAsxiGZtg7u3thTZhBRDqPMO5\ngIY/l8tFQrHm+RznvObDHu8a4lMoFApFLJFKulurUCgUikcT6kEpFAqFIpZQA6VQKBSKWEINlEKh\nUChiCTVQCoVCoYgl1EApFAqFIpZQA6VQKBSKWEINlEKhUChiCTVQCoVCoYgl1EApFAqFIpZQA6VQ\nKBSKWEINlEKhUChiCTVQCoVCoYgl1EApFAqFIpZQA6VQKBSKWEINlEKhUChiCTVQCoVCoYgl1EAp\nFAqFIpZQA6VQKBSKWEINlEKhUChiCTVQCoVCoYgl1EApFAqFIpZQA6VQKBSKWEINlEKhUChiCTVQ\nCoVCoYgl1EApFAqFIpZQA6VQKBSKWOL/AObiuv2zlT0eAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fdd1cf348d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_one_at_a_time.ckpt\") # not shown in the book\n",
    "    weights1_val = weights1.eval()\n",
    "\n",
    "for i in range(5):\n",
    "    plt.subplot(1, 5, i + 1)\n",
    "    plot_image(weights1_val.T[i])\n",
    "\n",
    "save_fig(\"extracted_features_plot\") # not shown\n",
    "plt.show()                          # not shown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unsupervised pretraining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a small neural network for MNIST classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150\n",
    "n_outputs = 10\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0005\n",
    "\n",
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "y = tf.placeholder(tf.int32, shape=[None])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "weights3_init = initializer([n_hidden2, n_outputs])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=\"weights3\")\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_outputs), name=\"biases3\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "logits = tf.matmul(hidden2, weights3) + biases3\n",
    "\n",
    "cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)\n",
    "reg_loss = regularizer(weights1) + regularizer(weights2) + regularizer(weights3)\n",
    "loss = cross_entropy + reg_loss\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "correct = tf.nn.in_top_k(logits, y, 1)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "pretrain_saver = tf.train.Saver([weights1, weights2, biases1, biases2])\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular training (without pretraining):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train accuracy: 0.973333 Test accuracy: 0.9334\n",
      "1 Train accuracy: 0.98 Test accuracy: 0.936\n",
      "2 Train accuracy: 0.973333 Test accuracy: 0.9382\n",
      "3 Train accuracy: 0.986667 Test accuracy: 0.9494\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 4\n",
    "batch_size = 150\n",
    "n_labeled_instances = 20000\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = n_labeled_instances // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            indices = rnd.permutation(n_labeled_instances)[:batch_size]\n",
    "            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train accuracy:\", accuracy_val, end=\" \")\n",
    "        saver.save(sess, \"./my_model_supervised.ckpt\")\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})\n",
    "        print(\"Test accuracy:\", accuracy_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now reusing the first two layers of the autoencoder we pretrained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_cache_frozen.ckpt\n",
      "0 Train accuracy: 0.94\tTest accuracy: 0.9266\n",
      "1 Train accuracy: 0.98\tTest accuracy: 0.94\n",
      "2 Train accuracy: 1.0\tTest accuracy: 0.946\n",
      "3 Train accuracy: 0.98\tTest accuracy: 0.9401\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 4\n",
    "batch_size = 150\n",
    "n_labeled_instances = 20000\n",
    "\n",
    "#training_op = optimizer.minimize(loss, var_list=[weights3, biases3])  # Freeze layers 1 and 2 (optional)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    pretrain_saver.restore(sess, \"./my_model_cache_frozen.ckpt\")\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = n_labeled_instances // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            indices = rnd.permutation(n_labeled_instances)[:batch_size]\n",
    "            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train accuracy:\", accuracy_val, end=\"\\t\")\n",
    "        saver.save(sess, \"./my_model_supervised_pretrained.ckpt\")\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})\n",
    "        print(\"Test accuracy:\", accuracy_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stacked denoising Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the book uses `tf.contrib.layers.dropout()` rather than `tf.layers.dropout()` (which did not exist when this chapter was written). It is now preferable to use `tf.layers.dropout()`, because anything in the contrib module may change or be deleted without notice. The `tf.layers.dropout()` function is almost identical to the `tf.contrib.layers.dropout()` function, except for a few minor differences. Most importantly:\n",
    "* you must specify the dropout rate (`rate`) rather than the keep probability (`keep_prob`), where `rate` is simply equal to `1 - keep_prob`,\n",
    "* the `is_training` parameter is renamed to `training`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using Gaussian noise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "noise_level = 1.0\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "X_noisy = X + noise_level * tf.random_normal(tf.shape(X))\n",
    "\n",
    "hidden1 = tf.layers.dense(X_noisy, n_hidden1, activation=tf.nn.relu,\n",
    "                          name=\"hidden1\")\n",
    "hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book\n",
    "                          name=\"hidden2\")                            # not shown\n",
    "hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown\n",
    "                          name=\"hidden3\")                            # not shown\n",
    "outputs = tf.layers.dense(hidden3, n_outputs, name=\"outputs\")        # not shown\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "    \n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0440489\n",
      "1 Train MSE: 0.0432517\n",
      "2 Train MSE: 0.042057\n",
      "3 Train MSE: 0.0409477\n",
      "4 Train MSE: 0.0402107\n",
      "5 Train MSE: 0.0388787\n",
      "6 Train MSE: 0.0391096\n",
      "7 Train MSE: 0.0421885\n",
      "8 Train MSE: 0.0398648\n",
      "9 Train MSE: 0.0408181\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_stacked_denoising_gaussian.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "dropout_rate = 0.3\n",
    "\n",
    "training = tf.placeholder_with_default(False, shape=(), name='training')\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "X_drop = tf.layers.dropout(X, dropout_rate, training=training)\n",
    "\n",
    "hidden1 = tf.layers.dense(X_drop, n_hidden1, activation=tf.nn.relu,\n",
    "                          name=\"hidden1\")\n",
    "hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book\n",
    "                          name=\"hidden2\")                            # not shown\n",
    "hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown\n",
    "                          name=\"hidden3\")                            # not shown\n",
    "outputs = tf.layers.dense(hidden3, n_outputs, name=\"outputs\")        # not shown\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "    \n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0296476\n",
      "1 Train MSE: 0.0275545\n",
      "2 Train MSE: 0.0250731\n",
      "3 Train MSE: 0.0254317\n",
      "4 Train MSE: 0.0249076\n",
      "5 Train MSE: 0.0250501\n",
      "6 Train MSE: 0.024483\n",
      "7 Train MSE: 0.0251505\n",
      "8 Train MSE: 0.0243836\n",
      "9 Train MSE: 0.0242349\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, training: True})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_stacked_denoising_dropout.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_stacked_denoising_dropout.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGztJREFUeJzt3UlsV2XUx/EHoXSCthalRaTMIuKEYkTQaIzDwkSDxoWy\n0KiJxkQTTdRIYqJudKdsjBqN88JEo8YYNSKaqMEhCMQRi4ACghSZ2lLaMryLN+/G8zu+9+E/tId+\nP8uT597/7S3/ntzcH+cZdfTo0QQAQAQnDPUFAABQFE0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZN\nCwAQBk0LABAGTQsAEMaYKn4WozdQDaOG+gJGgr///rvw97lSU3dGjbK/6pzPUsfnnqMSn1WO6/LO\nUep5q6m9vV3+EDxpAQDCoGkBAMKgaQEAwqjmOy0Ax7Hh8G6kmu+pvM874QT7LFCOz1fn9ZT6/qua\n9yv383jSAgCEQdMCAIRB0wIAhEHTAgCEQdMCAIRBehBAtpzEWaXW5iTOjhw5Ymo1NTWFz6uO99YO\nDg4WqnnHe9c1Zoz9c+2tVUlD7xq8uqI+7/Dhw3JtpVKJPGkBAMKgaQEAwqBpAQDCoGkBAMIgiAEg\nW84L9eEwUkgFCLyRQqNHjy58XhViGBgYKLQupZTGjh1b+PNra2sLr1XhiEOHDhVe6wVPlJzxUuXA\nkxYAIAyaFgAgDJoWACAMmhYAIAyaFgAgDNKDAIYNlRTMGfnkpely1qo0nDeqqOhnNTY2yrUNDQ2F\nPj8lnejLuTfeedV4qJzzevdG3V82gQQAjCg0LQBAGDQtAEAYNC0AQBgEMQAMG+qFvBcgUGu9UEDO\nuKScsMHBgwcLXVfOHlmenHFJOUEMNd4pZ4+snPtIEAMAMKLQtAAAYdC0AABh0LQAAGHQtAAAYZAe\nBJBNpb1ykmHeWpVk8zZQzDmvGoGkEoXeOQ4cOFB4rUoEetfV19cn64pK6eWMovISjIrayDKlvJFP\nOeOwcjbe5EkLABAGTQsAEAZNCwAQBk0LABDGiApifP3116a2fPlyuXby5MmmVl9fL9fecsstptba\n2irXenXgeKVCCN6LfhVM2LNnj1y7fft2U+vu7pZr6+rqCtW8z+vt7ZVri4ZBvACCCp54e2+pIEVT\nU5Ncq/bpam9vl2tV3TuvokIuKemfzQttMMYJAHBcomkBAMKgaQEAwqBpAQDCoGkBAMIYlZPaKFHV\nPsgzZ84cU+vs7KzIZzU3N8v6woULK/J5lTJt2jRTe/jhh+Xajo6OCl9NIXpeDspqx44d5vvsjSpS\nyTm1eWJKKe3fv9/Ufv75Z7lWfXe7urrk2p6eHlPzEox79+4tdHxKesSUStN5CTv191cll1PSf1O8\nBKSqn3766XLtJZdcYmrTp0+Xa9UYJ5US9OreSC51f9ra2uQ/KJ60AABh0LQAAGHQtAAAYdC0AABh\njKgxTu+++66prV27Vq6dN2+eqf30009y7TfffGNq7733nlz78ccfm5r30nPTpk2yXpR6aZpSSpMm\nTTK1LVu2FD6vCmeklNJDDz1U+BwYOdQL+ZwxTt5eVmrckgpRpKRHIHnXoM7rhQ3UnlGKd3xLS4up\njRs3Tq5V9Z07d8q1KtDihe7a2tpMzQuSqev1AjhKTiDFw5MWACAMmhYAIAyaFgAgDJoWACAMmhYA\nIIwRlR6cO3duoZrn7LPPlvWbbrrJ1J588km5dvPmzabmpQc3btxY+NqUsWPHyrpKD3rXoMbieONg\nAGX06NGm5v3bVButTpw4Ua5V/46nTp0q16pNDb0NI1XdS9N5Cd1/8zaRVKnGU089Va5VqcbPPvtM\nrlV/Z0488US5ViUuvTFb/f39puYlKHM2gcxJIPKkBQAIg6YFAAiDpgUACIOmBQAIY0QFMarJ2+cm\nJ8SQExLJocZO7dq1S6698MILTe2qq64q+zXh+KVCFyqckVLxsUgp6bFGra2tha/BG+Okrs0LXKi1\nOftpqetS4YyU9Mg5Fc5IKaV//vnH1Lx7ru5jTjDCu48qdOHdB+9nVnjSAgCEQdMCAIRB0wIAhEHT\nAgCEQdMCAIRBevA45o2OWbJkial5qZ6nn37a1NSoHcCjkmhekk2lyBobG+VaVVfjmrzPGxwclGtr\na2tNzUvTqbSj+i55n+WljBWVFFy/fr1cq65Bjb1KKaWZM2ea2vjx4wtfl7fBpUpcer/3HDxpAQDC\noGkBAMKgaQEAwqBpAQDCIIhxHHv55ZdlfceOHaY2YcIEudbbnwgoSr2o917Iq7FGXghCnUOFKFLS\noQBvZJQ6r3e9XoCpyOenpH/erVu3yrVq76xt27bJtTNmzDA1b4ScCmh4+50dPnzY1Lzfjxrj5N1z\nb58thSctAEAYNC0AQBg0LQBAGDQtAEAYNC0AQBikB48Tv//+u6ndf//9hY9ftWqVrLe3tx/zNeH4\npRJjOQkwj0rjeYmzoiOU/quuqIScd7w3nunfvFRjX1+fqa1YsUKuVXVvA8ZFixaZ2jnnnCPXqqRg\nTsovZ+POcuBJCwAQBk0LABAGTQsAEAZNCwAQBkGM48T7779vat5L4htvvNHU1NgXIEfOOJ+c/am8\ngIeqe3s7qTFMXrhCXZu3N50KQqh9vryfV41hWrlypVy7fft2U1N7YaWU0uLFi01typQpcq0aMeVd\nr1dX1O8y53j3vCWfAQCAKqFpAQDCoGkBAMKgaQEAwiCIEYwXrnjnnXdMzftf+E888YSpefsFAUXl\nTMRQUydS0qEA77zqRX/RCRUp+XtGeWEOpaamxtTq6+tNzQt9qCkXat+slFKqq6sztcsvv1yunT9/\nvql5fw8U7z5WamJJDp60AABh0LQAAGHQtAAAYdC0AABh0LQAAGGQHgzmxRdflPUvvvjC1G6++Wa5\nlpFNKFVOUjBn7y211ksaqoSbSh965/VGCqn0oLd23LhxpqbSg2q/u5RS+vLLL01t165dcu0VV1xh\naldeeaVc29TUZGrefcwZh5VD3bOcfbo8PGkBAMKgaQEAwqBpAQDCoGkBAMIgiDGMrV271tTuuece\nubalpcXUHn/88bJfE5BS3r5I6iW7NyZIjRPLCRB4aw8ePGhq3s+Qs1YFC1SQ4oMPPpDHf/vtt6Y2\nffp0uXbJkiWmdtZZZ8m1KpCSE2jxxi2pe+6Nw1JrcwIXHp60AABh0LQAAGHQtAAAYdC0AABh0LQA\nAGGQHhwG+vr6ZP2mm24yNS8BtHTpUlNjXBMqRaXASk0UpqT/feck9zwDAwOFr0HxxkP19/ebWmdn\np6l9/PHH8nj13Vff+5T0GCc1RiolnQjMSWzmbKaZs6Gnl0rM+bfDkxYAIAyaFgAgDJoWACAMmhYA\nIAyCGFWmXkRec801cu369etNbe7cuXLtY489VtqFARly9siqxGf9V12pra01NW/8kAoQeMGErVu3\nmtrbb79tamvWrJHHz5kzx9Suv/56uXbixImm1tPTI9eqe6MCF56ce55z3nLgSQsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBunBKtu9e7epff7554WPf+2112S9tbX1WC8JyJYzxkmt9cYiKd6YIJXEra+v\nl2vr6upMraGhQa5VP8fevXvl2l9//dXUvv/+e1M7dOiQPH7BggWm1t7eLteqBKN3XpXo88Zelfr7\nyRmn5clJnvKkBQAIg6YFAAiDpgUACIOmBQAIgyBGhezbt0/WFy5cWPgcr7/+uqnNnz//mK8JGAql\nvqj3jj948KCpeS/0c/aMUgGPTZs2ybUqRLV27VpTUyOYUkpp3rx5ptbY2Fj4urwRSjU1NabmhStU\nmMNbm/O7VNebM3rLw5MWACAMmhYAIAyaFgAgDJoWACAMmhYAIAzSgxXy0ksvyfrGjRsLn+Piiy82\ntXKkb4BK8JJ7pW4O6Y1xUgm5gYEBuXbPnj2m1t3dLdd2dXWZ2ieffCLXrl692tRU+u/SSy+Vx597\n7rmm1tLSItcqanPLlHSq0Bv5pHgpQZUI9H6/6vdWjo1CedICAIRB0wIAhEHTAgCEQdMCAIRBEKMM\nOjs7Te3RRx+t/oUAVaICQTkv2b1Akap7a1UowBs/1NPTY2rbt2+Xa9esWWNqao+slFLasmWLqbW1\ntZna3Llz5fHNzc2m5o1mUrx7owIp6n555/DCL+p3XI5wRU7AjCctAEAYNC0AQBg0LQBAGDQtAEAY\nNC0AQBikB8vgiy++MLX9+/cXPt5LFtXX1x/zNQERqSSal6ZTo4ZyNin0NoHs7e01tYaGBrl29uzZ\npjZjxgxTmzVrljxejaJSm1t6vJ9Bpf9yUoke9fvJueflGPXFkxYAIAyaFgAgDJoWACAMmhYAIAyC\nGFW2aNEiU/P26iGIgeGq1NE93vHqpb63D5QKFvT39xe+Bu/7NX36dFOrq6uTa1XAQo1xUrWUdGDi\nwIEDcq26N959VOOsvBFXXpijKMY4AQDgoGkBAMKgaQEAwqBpAQDCoGkBAMIYVY7kR0FV+yCMaMVj\nSDhmO3bsKOn77KXF1N+jnLU5vFRizhglNYZJXW/OKKqcJF0Ob9xSzn3M+f3kfJY6R1tbmzwxT1oA\ngDBoWgCAMGhaAIAwaFoAgDCqGcQAAKAkPGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkB\nAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDC\noGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBp\nAQDCGFPFzzpaxc/CyDVqqC9gJNi7d6/5Po8axa1H+TQ3N8t/UDxpAQDCoGkBAMKgaQEAwqjmOy0A\nx7GjR/Vr61LfdXnnLVU53sGpazvhBPsscOTIkZKOTynvetXnecfn3N/h8N6SJy0AQBg0LQBAGDQt\nAEAYNC0AQBg0LQBAGKQHAWQbDimyHKUm7w4fPizXqrqX/it6fH19feHjveRfTiIwJ8GYQ523HP9u\neNICAIRB0wIAhEHTAgCEQdMCAIRBEKMM3njjDVPr7e2Va1evXm1qzz//fOHPeuSRR2T98ssvN7XL\nLrus8HmBasoZ+eSNQFJhgcHBQbl2//79prZr1y65tqury9S2bt0q13qf92+tra2yPnnyZFPr6OiQ\naydMmGBqpQZMUkppzBjbBrzgiVKOsVM5eNICAIRB0wIAhEHTAgCEQdMCAIRB0wIAhDGqUhusCVX7\noEq5++67Zf25556r8pVYZ5xxhql9+eWXcm1zc3OlL2coxZovFNS+ffsKf59Viizn746X0Dtw4ICp\neYnAVatWmdo333wj165cudLUtm3bJteOHTvW1E4++WRTu+SSS+TxV199taktXLhQrlVJw5yRT959\nVPWcDSO9pKE6R06isLm5WS7mSQsAEAZNCwAQBk0LABAGTQsAEAZjnBwqdFGOwMX8+fNN7YYbbpBr\nOzs7Te2VV16Ra3/++WdTe+utt+Ta22+//b8uESirUvd28o7v6+sztd9//12u/fHHH01tzZo1cq0a\nwTZlyhS5tq6uztRUYMIbdbRp0yZTO+mkk+TapqYmUxs3bpxcqwIPAwMDcm3OvleHDh0ytdGjRxc+\nbznwpAUACIOmBQAIg6YFAAiDpgUACIOmBQAIY8SnB//8809Zf+GFFwqf44ILLjC1jz76SK5taGgw\nNTUKJiU9HmXDhg1y7VdffWVq3kgbYKh5yTL1b94bP6TOoTZKTCml6dOnm5r6LqaUUktLi6l531F1\nDd3d3aa2e/duebxK/XprVXJv8eLFcq1KNR48eFCuVbxEoNLf31/yObz7q/CkBQAIg6YFAAiDpgUA\nCIOmBQAIY8QHMbywgnrBqgIXKaW0YsUKU/PGq+R4+eWXTe27774rfPx1111X8jUARZVjjyxVHzNG\n/5lS37Fp06bJtY2NjYWvQa31RiCpPb1UWGrnzp3y+PXr15va5s2b5doZM2aYmhoLl5IeG+WNklIj\nm3JCFEeOHCl83txzKDxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIY8enB8847T9ZVqtAbNVJfX1/W\na/o/apSUl2IChlpOWkyNa0opL/XW3Nxsal4KTY1m8r5LaiyRSgl6a9V4KO9vhBqt5CX3vHum1NbW\nmlpOys/7XXq/C0Vdr3fenOQpT1oAgDBoWgCAMGhaAIAwaFoAgDBGfBDDo17yVsprr70m6+vWrSt8\njquuusrUZs6ceczXBAyFnCBGTU2Nqak9p1LSAQAv2KACC16IQV2Dut59+/bJ41UQY+rUqXKtGluV\nE37JCTuon8s7hxccUWu9+5gT8OBJCwAQBk0LABAGTQsAEAZNCwAQBk0LABAG6cEqW7Nmjandeeed\ncq0aETNp0iS5dvny5abmJYCAoeaNRMtJkanvR85mgl56UCUQVcovpZT2799val1dXabW29srj58w\nYYKpTZw4Ua5Vo6g86j7mjM7yUok591fJ2VzSw5MWACAMmhYAIAyaFgAgDJoWACAMghhVtmrVKlNT\nL5Q9d911l6yfdtppx3xNwHDmBQhU3XvRr+peqKCvr8/U1P56KaW0ceNGU9uwYYOpeUEOFXhQ45pS\nSqmtrc3UxozRf8LVCCXvPqrAlrc2ZxSU+tm843PGUfGkBQAIg6YFAAiDpgUACIOmBQAIg6YFAAiD\n9GCF3HbbbbL+5ptvFj7HfffdZ2oPPvjgMV8TMBRUYsxLi6n64OBg4c/KGQ/lnXdgYMDUdu/eLdd2\nd3ebmkoDb9u2TR6vUnqtra1ybUNDg6l5Y6/UppNecq+urq7wWpW49JKGOYnAHDxpAQDCoGkBAMKg\naQEAwqBpAQDCIIhRBj09Pab24YcfyrVqnIsaz5JSSsuWLTM170UzMFzlvJBXAQBv3JIaP+TtIacC\nC16IYe/evaamxjWllNLOnTtN7bfffjO1f/75Rx7f0dFhat4YJ3VdXmhD7d/V2Ngo1zY1NZmaNx5K\n/S7U/mMp5d3zHDxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIgPVgGN954o6mpVJHn3nvvlXUvGQRE\nkjPGSa31RgqpJG3Oeb0xTmrkkpce7OzsNDWVFJw5c6Y8fsGCBaY2d+5cuVYl91avXi3XHjhwoNBn\neef1RjOpe6Y+KyU/rVgqnrQAAGHQtAAAYdC0AABh0LQAAGEQxMjgvfT8/PPPC5/j+uuvN7X777//\nWC8JGPZUOCJnvyZvjJMaieaNCVLn6Orqkms3bNhgaipwkZIOXcyaNcvUzjrrLHn8NddcY2onn3yy\nXKv2yPICImr/Ly/g4Y1hKnoN6veQkg5ilGOPLZ60AABh0LQAAGHQtAAAYdC0AABhEMRw9PX1mdrD\nDz8s1w4MDBQ+7/nnn29q7JEF/C/1ot7b20np7u6WdbUX1aZNm+RaNc3GC4OcdtpppnbmmWea2mWX\nXSaPV+GIuro6uba2trbwWrUnlxdSUX+/vCkXf/31l6mNHj1arp0wYULha8gJaPCkBQAIg6YFAAiD\npgUACIOmBQAIg6YFAAiD9KDj2WefNbVPP/208PG33XabrDOyCfCpFJmXTlN7O6kxQyml9Mcff5ia\nGteUkh4x5e2HpeoXXHCBqXkjlMaPH29q3s9bU1Njas3NzXKtuo9eIvDvv/82NTUGKqWUNm/ebGod\nHR1ybX9/v6l5P1tOQpQnLQBAGDQtAEAYNC0AQBg0LQBAGAQxHMuWLSvp+KeeekrWGdmEkUYFGw4f\nPizX5oxx6unpMTUVKkgppfXr15va9u3b5VoV8FBjkVLSe1+p8UXemCIVePD2p9q6daupeXuCqUCK\nul8p6YDGrl275Fp1H6ZOnSrXVgpPWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwSA9WiJfU8TZBK5Xa\nIM4bmaKSW2rkikdtkJlSSsuXLy98DsW7XpXkVCNtEIdK6KWk07U53xlvE8g///zT1H744Qe5Vv37\nbmtrk2vV9aq05MaNG+XxKimoNqFMKaU9e/aYWk56UG2EmZJOD5500kly7cKFC03NS0SrjTO9zTRz\n8KQFAAiDpgUACIOmBQAIg6YFAAiDIEaFTJ48uaqfd9ddd5naKaecItfu2LHD1J555pmyX1O5qHt5\nxx13DMGV4FioYIIXulF1L4jR2NhYqJaSDhp5oQ01Lqmzs1Ou/emnn0xtypQpptbe3i6P/+uvv0xN\nBS5S0iGG3t5euTZnLyt1vaqWUkqzZ882NTXKKiUdDisHnrQAAGHQtAAAYdC0AABh0LQAAGHQtAAA\nYZAedCxdutTUXnrppSG4kmKeffbZipxXbcDnpZCUW2+9VdYvuuiiwudYvHhx4bUYfnI2dsxZW19f\nb2rTpk2Ta+fNm2dqXnpQXcO2bdvkWjUaSaUdVcrQ+yxvg0z1s3mpRJXca2lpkWs7OjpM7fzzz5dr\nZ86caWonnniiXKvGO3mbYebgSQsAEAZNCwAQBk0LABAGTQsAEMYoNWKlQqr2QZXy6quvyvrAwEBJ\n5123bp2slzpa6YEHHpD1WbNmFT7Htddea2oTJ0485muqgtLf9OL/tW/fvpK+z97fHRVi8PZrOnTo\nUOHzqn2nvHDEL7/8Ympq9FlK+ruvfga1b1ZK+nq9YMPpp59uak1NTXKt2m9u3Lhxcm1dXV3h86ow\nR0NDg1yrAls5QYzm5ma5mCctAEAYNC0AQBg0LQBAGDQtAEAYNC0AQBikB3G8IT1YBTnpwVL/xqgk\nnMfbMFLVvXFJamPFvr4+uVYlG9UGjN7Gjoo3mkml9FSCMiWdavRSzureeBs4qt+Fd8/VppWkBwEA\nIwpNCwAQBk0LABAGTQsAEAb7aQGoqJyX72qtFzYoVc4oKW8vKm/E1L+1trYW/ixvvzpV98IkOeOw\n1Frvd6bumXcfy7F3lsKTFgAgDJoWACAMmhYAIAyaFgAgDJoWACAM0oMAqi5ntJOXQlN1NToopZQG\nBwcLX0NOglGtVYk+b4SSOt4bWzVmjP1z7V2X2nSyaNLxv1QqEZiDJy0AQBg0LQBAGDQtAEAYNC0A\nQBgEMQBUXc4LfS9coc7h7e2k9ofyxiWpgIZ33qLXoIIg3md5AREV5vDuTan7Xnn3ZjjgSQsAEAZN\nCwAQBk0LABAGTQsAEAZNCwAQBulBAMNaOZKGKpGXs4Git1aNUVKf5aXx1GfljJfyRjOp++Ddx+Gc\nFFR40gIAhEHTAgCEQdMCAIRB0wIAhDEqZ18bAACGEk9aAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBo\nWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoA\ngDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDD+BxBrc2o8LjrdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846a1b11d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_stacked_denoising_dropout.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparse Autoencoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sparsity_loss_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmczWX7wPHPNRhMxlYjZBmy75RK1lRaiIgUSZbnUUk9\nIaVFSJ6nepQlkqKmBSU8kRZZRpK97Mk69rVhDMOMmbl/f9xn5ndmzIzZv2e53q/XeTHnfM/3XOc7\nM+eae7tuMcaglFJKeZoApwNQSiml0qIJSimllEfSBKWUUsojaYJSSinlkTRBKaWU8kgFnQ4gN4mI\nTklUSikPYoyR7D7X51pQxhi9pXF7/fXXHY/Bk296ffzn2sycaWjTRq9PftxyyucSlFJKpccYeO89\neP55pyNRmaEJSinlN1avhshIaN/e6UhUZmiC8hNt2rRxOgSPptcnfb50bcaPh2efhQIFcu+cvnR9\nPI3kRj+hpxAR40vvRymVew4cgCZNICICgoOdjsY/iAgmB5MkfGoWX3pCQ0M5cOCA02Eoh1SuXJmI\niAinw1AOmzwZevfW5ORN/KIF5criDkSkPIF+/9X58xAaCuvXQ5UqTkfjP3LagtIxKKWUz/vsM2jd\nWpOTt/GLLj6llP9KTIQJE+Djj52ORGWVtqCUUj7thx/gmmugRQunI1FZpQlKKeXT3nkHhg4FyfZI\niHKKJigfFRYWRsuWLZO/Dg4O1plsyu+sXWunlXfr5nQkKjs0QTmsSpUqLFu2LPnr2bNnU7p0aVau\nXMmBAwcICAggMTExW+cWtz8Zo6OjCQ0NzWm4SnmVd96BwYOhUCGnI1HZoQnKg4SFhTFo0CB++OGH\n5NaPeHC/REJCgtMhKJWu3bthxQro18/pSFR2aYLyENOmTeOFF15g8eLF3HrrrVl+fmRkJB07dqRE\niRLcdttt7N27N8XjAQEB7Nu3j7Vr11KuXLkU64Lmz59Pw4YNAVsN/j//+Q/VqlUjJCSERx55hLNn\nzwIkt+hmzJhB5cqVufPOOwH47LPPCA0NJSQkhDFjxqRoFWbmfJ999hmVK1emTJkyjB07NjmuxMRE\nxo4dS7Vq1ShRogRNmzblyJEjAOzcuZN27dpx7bXXUrt2bebMmZPla6Z827hx8NRTdoKE8lJOl2PP\n5dLuJi3p3e8JQkNDzUMPPWTKli1rtm7dmuKxiIgIExAQYBISEq56nu7du5vu3bubixcvmm3btpkb\nbrjBtGzZMvnxgIAAs3fvXmOMMdWqVTNLlixJfqxbt27m7bffNsYY895775lmzZqZo0ePmri4OPPk\nk0+aRx99NDkeETG9e/c2MTEx5tKlS2bHjh2mWLFi5rfffjOXL182Q4cONYGBgWbp0qWZPt8///lP\nExsbazZv3mwKFy5sdu7caYwx5u233zYNGjQwu3fvNsYYs2XLFhMZGWkuXLhgKlasaMLCwkxiYqL5\n448/TEhIiNmxY0ea18aTv/8qbxw/bkzJksacOOF0JP7N9buX/c/0nDzZ027ZTVC2CH/Ob9kRGhpq\nSpQoYR588MErHstsgkpISDCFChUyu3btSr7v5ZdfTpGgRCQ5Qb366qumb9++xhhjzp07Z6655hpz\n6NAhY4wxtWvXNsuWLUt+3tGjR02hQoVMQkJCcjwRERHJj48ePdr06NEj+euYmJgUCSoz5zt69Gjy\n47fccov56quvjDHG1KxZ0yxcuPCK9/vVV1+ZVq1apbhvwIABZvTo0WleH01Q/ueVV4x56imno1A5\nTVDaxUdupafsv/7UqVPZtWsX/bLZWX7q1CkSEhKoUKFC8n2VK1dO9/gePXowf/58Ll++zLx587jp\nppuSn3vgwAE6d+5M6dKlKV26NHXq1KFQoUKcOHEi+fnur3P06FEqVqyY/HXRokW59tprk7/OzPmu\nv/765P8HBQVx/vx5AA4dOkTVqlWviP/AgQOsWbMm+ZylSpVi5syZHD9+PFPXS/m28+fhww/t5Ajl\n3TRBeYAyZcqwdOlSVq5cydNPP53l54eEhFCwYEEOHTqUfN/BgwfTPb527dpUrlyZ77//nlmzZtGj\nR4/kxypVqsQPP/xAZGQkkZGRnDlzhgsXLlCuXLnkY9wnbpQrV47Dhw8nf33x4kX+/vvvLJ0vPRUr\nVrxiLC3p/jZt2qQ457lz55g8efJVz6l838cfwx13QLVqTkeickoTlIcoW7Ysy5Yt46effmKw259+\nxhguXbpEbGxs8s2kaq4FBATQpUsXRo4cycWLF9mxYwdhYWEZvl6PHj2YOHEiK1eupJvbIpEBAwbw\n8ssvJye4U6dOsWDBghTxuOvatSsLFy5kzZo1XL58mddffz3F41k9n7v+/fvz2muvsWfPHgC2bt3K\nmTNn6NChA7t27eKLL74gPj6ey5cvs2HDBnbu3Jnhe1a+7/JlePddeOEFpyNRuUETlMPcWyMVKlRg\n6dKlzJ07l1deeSX58eDgYIKCgihatChBQUEsX778ivNMmjSJ6OhoypUrR9++fenbt2+6rwPwyCOP\nsGLFCu68805Kly6dfP9zzz1Hp06daNeuHSVKlOD2229n3bp16Z6nTp06TJo0ie7du1O+fHlKlChB\nmTJlKFy4cLbO5/714MGDefjhh5Of279/fy5evEixYsVYvHgxs2fPpnz58pQvX56XXnqJuLi4jC+2\n8nmzZ9uWU9OmTkeicoNut6Fy1YULFyhZsiR79uzJcBwsP+n33z8kJEC9evD+++BaAaEcptttKMd9\n9913XLx4kQsXLjBkyBAaNGjgMclJ+Y+5c6FkSWjb1ulIVG7RBKVy7Ntvv6V8+fJUqFCBvXv3Mnv2\nbKdDUn4mMRHGjIHXXtOisL5Eu/iUz9Pvv+/79lsYPRo2bNAE5Um0i08p5deMgTfegFdf1eTkazRB\nKaW82k8/QWwsdOrkdCQqt2mCUkp5raTW0yuvQIB+mvkc/ZYqpbxWeDicPq0bEvoqTVBKKa/1xhvw\n8stQoIDTkai8oAlKKeWVVq2y27m7lZJUPsaxBCUipURkvoicF5H9IvJoOscFishUETkuIqdF5FsR\nuXqlUS8RGhpKkSJFiIyMTHF/o0aNCAgI4ODBgxw5coSuXbsSEhJCqVKlaNiwIZ999hnw/5v+FS9e\nnOLFixMcHEzx4sV1Az/l80aMsK0n3c7ddxV08LWnAJeAEKAJsEhENhlj/kx13L+AW4F6wDngY2AS\n0DUfY80zIkKVKlWYNWsWAwcOBGDbtm1cunQpuS5dr169aNy4MYcOHSIwMJCtW7em2FpCRIiKivLo\n7eGVyk3h4XDgAPTu7XQkKi850oISkSCgC/CqMeaiMWYVsADolcbhocBPxpjTxpg4YDZQN9+CzQe9\nevVKUX08LCyM3q7fPGMM69evp3fv3hQpUoSAgAAaNmzIPffck+IcuhBV+QtjbMWIESO09eSxjIEf\nfsjxaZzq4qsBxBtj3Df72UzaiWc60EJEyrkSW0/g+3yIMd/cdtttREdH89dff5GYmMjXX3/NY489\nBtjWUbNmzXj66af56quvUuz55E4TlPIXP/8Mp05Bz55OR6LSNWECvPRSjk/jVIIqBkSlui8KCE7j\n2F3AQeAIcBaoBbyRq9GMHGmXoKe+jRyZ+ePTOzaTklpRP//8M7Vq1aJ8+fLJSWfOnDm0atWKMWPG\nULVqVZo0acKGDRuSn2uMISQkJHl32dKlS/PXX3/lKB6lPFFS62nkSJ2557F++AHefhvc9n3LLkdq\n8YlII+BXY0wxt/sGA62NMZ1SHfslUAToC8QALwIdjDG3pXFe475hXps2bWjTpo1H12KrUqUK06dP\np1q1arRq1YpmzZrRoUMHHnnkEQoVKkRERASVKlVKPj4yMpIhQ4awZMkSDh06xIEDB6hatSrx8fE6\nBpUOT/7+q6z57jsYPhw2b9aFuZ4ofOZMwv/xD+jeHSpVYtSoUTmqxedUggoCIoG6Sd18IhIGHDHG\nvJzq2K3Ay8aYha6vSwBngOuMMZGpjvW6YrFJCapt27bccccd/PHHHxw7dozAwMA0ExTA9u3badCg\nAadPn+bcuXNUrVqVy5cvE6C/sWny5O+/yjxj4KabbM29Ll2cjkalKT4eNm6EW28FvLRYrDEmBpgH\njBaRIBFpDnQEPk/j8PXA4yJSXEQKAQOxiSwyjWO92owZM1i2bBlFixZNcf9LL73E9u3bSUhIIDo6\nmilTplCtWjVKlSoF2C4+/QBWvm7+fPtv587OxqEyULBgcnLKDU7+yT0QCAJOAl8CTxpj/hSRFiJy\nzu24oUAssBs4AdwL+MyPqHu3XJUqVWjSpMkVj8XExNC5c2dKlSpFtWrVOHToEAvc+ndFhFKlSqVY\nBzV+/Pj8exNK5bHERHj9dbulhvZk+w/dD0r5PP3+e79Zs+zEsNWrNUF5E6/s4lNKqcyKi7PjTv/+\ntyYnj/Pbb3DyZJ6dXhOUUsqjTZsGNWrAHXc4HYlK4a+/4MEH7b95RLv4lM/T77/3io6G6tXhxx+h\nUSOno1HJzpyxkyGGDYP+/dM9TLv4lFI+69134a67NDl5lLg4O8+/Q4cMk1Nu0BaU8nn6/fdOJ09C\n7dqwYQNUqeJ0NAqwi9H69oXISJg376rlPLQFpZTySWPGQK9empw8TpMm8OWX+VJryi9aUKGhoRw4\ncMCBiJQnqFy5MhEREU6HobJg7147xPHnnxAS4nQ0Krty2oLyiwSllPIuPXrY7r3XXnM6EpUTmqDc\naIJSyvv9/rsdf9+1C4oVu/rxynPpGJRSymcYA0OH2s0INTl5gLNn4fBhx15eE5RSymMsWGBn7+Xx\n7GWVGZcvQ7duMHWqYyFoF59SyiPExUG9ejBpEtxzj9PR+DljYMAAOHIEvv3WVinPhpx28WXvVZVS\nKpdNmQLVqmly8gjjxsG6dbByZbaTU27QFpRSynGRkVCrFoSHQ506Tkfj5+bNg2eftaXjK1bM0al0\nkoRSyuuNGgVdu2py8giHDtluvRwmp9ygLSillKP++guaN9dFub5IW1BKKa82bBi8+KImJ3UlnSSh\nlHLM0qWwdSt8/bXTkShPpC0opZQjLl+2Y/H//S8ULux0NH4qPh4OHnQ6inRpC0op5Yj334cKFaBz\nZ6cj8VPGwMCBdlfImTOdjiZNmqCUUvnu+HEYO9Yus5FsD6GrHPn3v+1apxUrnI4kXZqglFL57qWX\noE8fu/ZJOeDzz2HaNPjtNyhe3Olo0qUJSimVr1avhiVL7LRy5YAlS2xF3uXLoXx5p6PJkE6SUErl\nm4QEeOYZeOstCA52Oho/FREBc+Z4xapoXairlMo3H34IX3wBv/yiY0/+QDcsdKMJSinP9fff9o/2\nxYuhYUOno1H5QROUG01QSnmup5+GgAA7vVz5B91uQynl8TZutEWyd+xwOhI/Exdnx5xq1HA6kmzR\nBKWUylPx8fDPf8Lbb0Pp0k5H40cSE+1c/oAAO63cC2mCUkrlqcmToUQJ6NXL6Uj8iDEwZIgtY7R4\nsdPRZJsmKKVUnjl8GMaMgV9/1Vl7+ertt+16p19+gaJFnY4m2zRBKaXyzLPP2nJvNWs6HYkf+eQT\nmDoVVq2CUqWcjiZHNEEppfLEggWwfbvH1iH1XUWKwE8/eXyViMzQaeZKqVx3/jzUrQuffgp33OF0\nNMopug7KjSYopTzD0KFw6hSEhTkdiXKSroNSSnmUTZvsrOZt25yORHk7LRarlMo1SWuexo6FkBCn\no/EDhw7B2rVOR5FnNEEppXLN+PG2Snnfvk5H4gdOnoS774Y1a5yOJM/oGJRSKlfs3g3NmtlNWqtW\ndToaH3f2rJ198sADMHq009GkK6djUI61oESklIjMF5HzIrJfRB7N4NgmIrJCRKJF5JiIDMrPWJVS\nGUtMhH794NVXNTnluQsXoH17aNUKRo1yOpo85eQkiSnAJSAEaAIsEpFNxpgU+2yKyLXAD8BzwDdA\nYaBCPseqlMrA1Kl2/GmQ/umYt4yBbt1s8df33vP58hyOdPGJSBBwBqhjjNnruu8z4LAx5uVUx74J\nVDDG9M7EebWLT6l8duAA3HyzrapTu7bT0fiB336DW26Bgp4/Cdtbu/hqAPFJycllM1A3jWNvA86I\nyCoROSEi34pIxXyJUimVIWPsrL3nn9fklG9uv90rklNucOpdFgOiUt0XBQSncWwFoDFwF7ANeAeY\nBbRI68QjR45M/n+bNm1o06ZNjoNVSqUtLMwuyH3hBacjUZ4gPDyc8PDwXDufU118jYBfjTHF3O4b\nDLQ2xnRKdewmYKMxpp/r69LAaaCEMSY61bHaxadUPjl2zG7dvngxNGrkdDTKE3lrF98uoKCI3Oh2\nX0NgexrHbgFSZx0D+PbooFIezBgYMMB272lyykNvvWX3KvFTjiQoY0wMMA8YLSJBItIc6Aikte3j\nJ0BnEWkgIoWA17Ctr3P5F7FSyt2nn9q98EaMcDoSH/b22zBjBlSr5nQkjnGyksRAIAg4CXwJPGmM\n+VNEWohIcvIxxiwHXga+B44DVYEeDsSrlAIiImDYMFtvLzDQ6Wh81Pjx8OGHsGwZlC3rdDSO0UoS\nSqlMS0yEO++Ee++FF190Ohof9cEHtmtvxQqoXNnpaHJEt9twowlKqbw1fjzMmWPXPBUo4HQ0PujQ\nIVvCaPFinyjJoQnKjSYopfLOzp3QooWtTerHwyJ5LzYWChd2Oopc4a2z+JRSXuTyZejVC8aM0eSU\n53wkOeUGTVBKqav697/h2mvt1HKl8ot/1MtQSmXbhg0weTL8/rvP1ybNf1FRUKKE01F4LG1BKaXS\ndf489OgBkybBDTc4HY2P+f57aNDAbp+h0qSTJJRS6erXz1aNmDHD6Uh8zKJF0KcPLFgAt93mdDR5\nJqeTJLSLTymVpqTp5H/84XQkPmbhQujfH777zm6bodKlLSil1BUOHrR7PC1aBE2bOh2ND1mwwCYn\nP7mwOs1cKZWrEhLgscdg8GC/+AzNX4GBfpOccoO2oJRSKYwZY0vA/fyzVotQOaNjUEqpXLN6tZ2x\nt3GjJiflvEx38YlIGRGZICJ7ROSSiBwSkUUicl9OgxCRyiKSKCJNcnoupVT2nD0LPXvC1KlQoYLT\n0SiVyRaUiFQGfsNuy/4idhPBAOw27B8AoTmMQ7hyU0KlVD4xxo7d33cfdO7sdDQ+4ptvICgI7r/f\n6Ui8VmZbUB9gE8hNxpi5xpjdxpi/jDGTsTvhIiIVRWS+iJxz3eaKSPLSPhGpICL/E5G/ReSCiOwQ\nkYddD+9z/bvB1ZJalltvUCl1dZMnw759MG6c05H4iFmz4JlndHVzDl21BSUipYB7gJeNMRdTP26M\niXL991sgBmjj+noyMB9Imuj/ARAItAaigZpup7kFWAe0w7bO4rL4PpRS2bRhA4waZcefihRxOhof\nMH263Wp4yRKoV8/paLxaZrr4qmG74Hamd4CI3A3UB6oaYw657usB7BGRtsaYZUAl4BtjzDbX0w64\nneKU699IY8zJLL4HpVQ2RUVB9+4wZYpWKc8VkybBf/8L4eFQvbrT0Xi9zHTxZWaKYC3gaFJyAjDG\n7AeOAnVcd00AXhOR30TkDZ0QoZSz3MedunVzOhofcPgwfPSRLb+hySlXZCZB7caOP9XO4JiMJjkY\nAGPMDOxkihlAdeA3ERmR6UiVUrlqyhQ77vTf/zodiY+oUMHWhfLybdo9yVUTlDHmDPAT8IyIBKV+\nXERKADuAG0Skktv9VYHyrseSznXUGPOxMeYRYATwT9dDSWNOuvJCqXywcaMdd/r6ax13ylW6eCxX\nZXYW39PYVtIGEekqIjVEpKaIPAVsNsYswU5u+FJEmojIzcAXwAZjzHIAERkvIveISBURaQTcC2x3\nnf8kcBG4x7XeqnguvkellJvISNulN3ky3Hij09Eolb5MJShjTATQBPgZ+A+wGVgKdOD/W0GdsJMd\nlrseOwq4r6gIACZik9JPwHHgCdf5E4BBQH/gCPC/bL8jpVS6EhLsYtzOnXXcKUfi42HNGqej8Hla\ni08pPzJihB3DX7IECmqhs+yJi7PVdGNi7NYZus1wurQWn1IqUxYuhE8+seueNDllU0wMdO0KhQrZ\nShGanPKUbrehlB/Ys8fujjtnDlx/vdPReKmzZ6FdO7juOpucdHZJntMEpZSPu3DBjjmNGuXTu4vn\nLWPggQfsPk6ffmpbUCrP6RiUUj7MGDspIjDQdu9pj1QORETYNU56ETNNx6CUUumaOBF27oRVq/Rz\nNcdCQ52OwO9oC0opHxUebuvsrVkDVao4HY3yRzltQekYlFI+aN8+eOQRmDlTk1O2HDvmdAQKTVBK\n+ZzoaOjUCV59Fe680+lovNDs2XDTTbbUu3KUdvEp5UMSE+2MvbJl7dbtOu6URePH2+q5P/wA9es7\nHY3X00kSSqlkr75ql+vMmaPJKUsSE2HYMPj+ezujRCuSewRNUEr5iFmz7G3dOjutXGXBs8/Cpk3w\n669QurTT0SgX7eJTygesXw/33w9Ll0KDBk5H44W2b4eqVaFoUacj8Sk57eLTBKWUlzt6FG691a55\n6tz56scrlV90mrlSfuzCBTtjb8AATU7K92iCSkNAQAAFChQgICDgiluBAgXo27ev0yHy008/ERAQ\nQExMjNOhKIck7e1Uty688orT0XiR48edjkBlkmMJSkRKich8ETkvIvtF5NGrHF9IRHaKyMG8ju34\n8eMcO3aM48eP89FHHyEinDhxIvn+CRMmZOu88fHxuRajMSap+Zxr51TeZehQOHcOpk3TGXuZNn8+\nNGwIJ044HYnKBCdbUFOAS0AI8BjwgYjUzuD4YdhdePNcmTJlkm8lS5YEICQkJPm+4OBgAIYMGUKN\nGjUICgqiatWqvPrqqymS0PDhw2natCkfffQRVatWpUiRIiQmJhIdHU2PHj0oVqwYN9xwA++++y53\n3303Tz/9dPJzY2NjGTJkCBUqVKBYsWI0a9aM5cuXA/DXX39x//33AxAcHEyBAgVSPFf5vvffhx9/\nhLlzdcZephgD48bBoEF2jZPuOeIVHJlmLiJBQBegjjHmIrBKRBYAvYCX0zi+CtADGAx8lJ+xZqRk\nyZJ88cUXlC1blq1btzJgwACuueYahg8fnnzMzp07WbBgAfPnz0/uNhw0aBDr1q1j0aJFhISE8Npr\nr7F+/XqqV6+e/LwePXpw+vRp5syZQ9myZfn222+5//772bRpEzVq1GDmzJn07NmT/fv3U7RoUYKC\ngpy4BMoB330HY8fa5TqlSjkdjReIj7eJadUqWL0aKlZ0OiKVWcaYfL8BjYALqe4bAnybzvELgY5A\na+BgBuc1ue2bb74xAQEBmTp2/Pjxpn79+slfv/TSS6Zo0aLm7NmzyfdFRkaaggULmgULFiTfFxUV\nZYKDg81TTz1ljDFm+/btpkCBAubkyZMpzn/vvfeaIUOGGGOM+fHHH01AQIC5cOFCtt+b8j4bNxpz\n3XXGrFnjdCRepE8fY+65x5ioKKcj8Tuuz+Rs5wqnFuoWA1IXuooCglMfKCKdgQLGmAUi0jo/gsus\nWbNm8f7777Nv3z7Onz9PfHw8hQsXTnFMlSpVKFGiRPLXu3fvJjExkaZNmybfV7x4cWrVqpX89e+/\n/05iYiI33nhjijGmuLg4iugunn7r0CHo2NGWMLr1Vqej8SKvvWZbTbrPvddx6jt2Hiie6r7iQLT7\nHa6uwLeA+5LuutqJR44cmfz/Nm3a0KZNmxyEmb4VK1bw+OOPM3bsWO68805KlCjB119/zRtvvJHi\nuGuuuSbF10kJRzIY1U5MTCQwMJBNmzZd8Vjq8yn/EBUFHTrAv/4FDz3kdDReRsu555vw8HDCw8Nz\n7XxOJahdQEERudEYs9d1X0Nge6rjqgOVgZViP9EDgRIichS4zRhzxYw+9wSVl1atWkW1atV44YUX\nku/bv3//VZ9Xo0YNAgICWLduHQ888AAA586dY+fOndx8880ANGnShMuXL3Pq1CluTedP5UDXyHhC\nQkJO34rycJcuwYMPQqtWMGSI09Eolb7UjYJRo0bl6HyOzOIzxsQA84DRIhIkIs2xY0yfpzp0K1AR\nO2bVEOiPncnXEDiUfxFfqUaNGuzfv585c+awb98+Jk6cyLx58676vFKlSvHYY48xePBgVqxYwfbt\n2+nXrx8FChRIblXVq1ePLl260LNnT/73v/8RERHBhg0bePvtt1m0aBEAoa7dPRcuXMjp06d1PZSP\nSkiAXr0gJMQW2tbp5Bkwxk6EUD7DyWnmA4Eg4CTwJfCkMeZPEWkhIucAjDGJxpiTSTcgEkg0xpwy\n7oMzDujatSuDBg1i4MCBNG7cmN9++y3TrbdJkyZxyy230L59e9q1a0fz5s2pW7duivGlmTNn0qNH\nD4YMGUKtWrXo1KkTa9eupVKlSoAd23rllVcYMmQIZcuWZejQoXnxNpWDjIHnnoO//4bPP4cCBZyO\nyIPFxcGTT9rbhQtOR6Nyidbi8wCXLl2iQoUKvPHGGzz11FNOh6M8xJgx8M03sGIFuM2zUamdPg1d\nu0Lx4vDllxB8xVwr5RCtxeeF1q9fz9dff82+ffv4/fff6dmzJ/Hx8XTt2tXp0JSH+PhjmDHDrinV\n5JSBbdvgllugWTNbJUKTk0/ReZcOMMbw1ltvsXv3bgIDA2ncuDG//vorISEhToemPMCCBXZm9IoV\nUK6c09F4sMRE6N8fRo+Gxx5zOhqVB7SLTykP8uuvtir599+D21I5lZ74eF3f5MG0i08pH/H779Cl\nix1G0eSUSZqcfJomKC/Sp08fOnbs6HQYKg/8+Se0bw8ffgjt2jkdjVKeQbv4vEh0dDTGGIoXt0U4\n7rjjDurXr8/EiRMdjkzlxP79dhHu2LF2zZNKw+rVtnx7Dhd+qvylXXw+4vLly1c9Jjg4ODk5Kd9w\n5AjcdRcMH67JKU3G2GZlp07gqrSi/IcmqHT88ssvNGvWjODgYEqWLEmzZs3YsWMHYWFhBAcH8913\n31GzZk2KFi1K27ZtU5Q52rdvHw8++CDlypWjWLFi3HTTTckVIJJUqVKFUaNG0a9fv+TqEgCjR48m\nNDSUIkX+k0wkAAAfFUlEQVSKUK5cOZ544onk57h38fXp04cVK1YwefLk5J1+IyIiqF69Ou+++26K\n19q9ezcBAQFp1vZTzjl9Gu6+G/7xD9DtvNJw6ZK9OBMm2NkjrtJgyn9ogkpDQkICDz74IK1atWLr\n1q2sW7eO5557jgKupfyxsbGMHj2asLAw1qxZQ0JCAl26dEl+/vnz57n//vtZunQpW7ZsoWvXrjz0\n0EPs2rUrxeu899571K5dm40bNzJ27FjmzZvHuHHjmDp1Knv27GHRokXccsstacY4YcIEmjVrRp8+\nfThx4gTHjh2jUqVK9OvXjxkzZqQ4dsaMGTRu3JhGjRrl8pVS2RUVBffea2vsvfSS09F4oGPHoHVr\nOHsW1q6FGjWcjkg5ISd7dXjajVzaDyoyMtIEBASYX3755YrHPv30UxMQEGBWr16dfN+BAwdMgQIF\nzNKlS9M952233WbefPPN5K9DQ0NNx44dUxzz7rvvmlq1apn4+Pg0z/HEE0+YBx54IPnrNm3amEGD\nBqU45vjx4yYwMNCsXbvWGGNMQkKCueGGG8yUKVMyeMcqP0VHG9OihTEDBxqTmOh0NB7q7FljpkzR\nC+TlyOF+UNqCSkOpUqXo3bs37dq1o0OHDrz33nscPnw4+fGAgIAU+zlVqlSJ8uXLs2PHDgBiYmIY\nNmwYdevWpXTp0gQHB7Nx40YOHkxZfP3mVH3q3bp14+LFi4SGhtK/f3+++eYb4uLishT79ddfT/v2\n7ZNbUT/88AORkZH06NEjS+dReePCBbttRo0aMHGiFn9NV4kS8NRTeoH8nCaodMyYMYN169bRunVr\nFixYQM2aNfn5558z9dwhQ4Ywd+5c3nzzTX755Rc2b95M06ZNr0g2qfd2qlChArt27WLatGmUKFGC\noUOHctNNN3Hx4sUsxd6/f3+++uorLl26xCeffEKXLl1SbJqonBETY4dRQkPho48gQH/7lMqQ/opk\noH79+rzwwgssX76c1q1bExYWBtgNBdevX5983MGDBzl69Ch16tQB7F5Rjz/+OA8++CD16tWjfPny\n7N27N83XSC0wMJD77ruPcePGsW7dOrZv386qdLYQCAwMTHM/qHvvvZfixYvzwQcfsHDhQvr165fV\nt65y2cWLdiJahQowfbompxT27LEXSKlU9NckDREREQwfPpzVq1dz8OBBli9fzpYtW5ITUIECBfjX\nv/7FmjVr2LRpE71796Z+/fq0bdsWsHtFzZ8/nz/++IOtW7fSq1cvYmNjr/q6YWFhTJ8+nW3bthER\nEcGMGTMIDAykevXqaR4fGhrKunXrOHDgAH///Xfybr0BAQH06dOH4cOHU6FCBe64445cujIqO5I2\nHCxTBj75RLfNSOHrr22h13XrnI5EeSBNUGkICgpi165dPPzww9SsWZM+ffrQq1cvXnzxRQCKFCnC\nK6+8wuOPP06zZs0QEebOnZv8/HfffZcyZcrQqlUr2rdvT7NmzWjZsmWK10hry/eSJUsyffp0WrVq\nRf369Zk/fz7z58+ncuXKacY5dOhQAgMDqVOnDmXKlOHQof/fw7Fv377ExcXRt2/f3LgkKptiY235\nolKlICxMk1Oy2FgYNMguAPvpJztjT6lUtJJEFoWFhTFo0CDOnTuXp6+TU2vXrqVly5bs27ePChUq\nOB2OX4qNhYcegqJFYdYsLRuX7MAB6NYNbrjBNilLlnQ6IpVHtJJEKj6Ub7MlLi6Ow4cPM2LECLp0\n6aLJySGXLtmWU+HCMHOmJqcU3noLHnkE5s3T5KQy5HMJKj7e6QicNWvWLEJDQ4mMjGTcuHFOh+OX\nkqaSFy8Os2dDoUJOR+RhJk+GwYN1Crm6Kp/r4ouONhQr5nQkyl+dO2eT04032l1xdcxJ+TPt4ksl\nE5Pl8s3IkSOdDkHlo7Nn7VYZderYqeSanLCLv5TKJk1QSuWC06ehbVu47Tb44ANd58SlS3aWnlYw\nUTngc118+/YZqlRxOhLlT06csFtmdOhg93Ty+6GVP/+0kyBq1bJbZehECL+lXXypaAtK5acjR+wS\nnm7dNDlhjB14a9XKtp5mz9bkpHJEE1Qe0jEo37Z7N7RoAX37wogRfp6cAH7+GSZNgl9+gf799YKo\nHPO51RmelKCU7/rjD2jfHkaPtp/FCrv74rp1dvGXUrnA58agfvnFkKqqkFK5KjwcHn4Ypk61i3GV\nUmnTMahUtAWl8tK339rkNHu2nyenCxecjkD5AU1QeUjHoHzLJ5/Ak0/C99/bKeV+KTHR7rRYr55u\nkaHynI5BKZUJ//0vvP++7d6rWdPpaBxy+DD06QPnz9sJEUWLOh2R8nE+14LypJ4HbUF5v8REGDYM\nZsyAX3/14+Q0ezY0aWLn1K9cCdWqOR2R8gM+14KKinI6AuUrLl2CJ56wDYeVK+Haa52OyCGHDsF/\n/mP7Nm++2elolB/xuRaUJyUobUF5r8hIW1cvMRGWLPHj5ARQsaKdV6/JSeUzTVBKpbJ/P9x+O9xy\ni+3ZKlLE6Yg8gC66VQ7QBJWHtAXlfTZsgObNYeBAOzHC74q+rlnjdARKJfO5X7+zZ52OQHmr776D\n++6z1cgHDXI6mnx26pQt8Nq7N5w543Q0SgE+mKC0BaWy44MP4B//sEmqUyeno8ln33wDDRrYsaZN\nm6BUKacjUgrQWXzKz8XHw/PP22U9fjd7+u+/YcAA2LYN5s2DZs2cjkipFHyuFl/t2oYdO5yORHmD\ns2ehe3f7/6++8sOdIWJi7OrjZ5/VmSAqT+S0Fp/PJahy5QxHjzodifJ0u3fDAw/APffAuHFQ0Of6\nEpRyntcWixWRUiIyX0TOi8h+EXk0neOGishWETknIntFZGhG5/WkLj4dg/JMy5bZfZyefx4mTNDk\npJSncnKSxBTgEhACPAZ8ICK10zm2F1ASuA94RkQeTu+ksbFw+XJuh6p8xYcfwqOP2vVNAwY4HU0+\nWbYM7r1XC1Uqr+NIF5+IBAFngDrGmL2u+z4DDhtjXr7KcycAGGOeS+MxExJi2LIFypbNg8CV17p8\nGYYMgcWLYeFCqF7d6YjywalTtpDg0qUwebLt01QqH3lrF18NID4pOblsBupm4rktge3pPVi2LJw4\nkcPolE85cQLuugv27LHrUH0+OSUk2IRUt66d+bF9uyYn5ZWc6n0vBqQeLYoCgjN6koiMAgT4JL1j\nzp8fyTvv2OnCbdq0oU2bNjmNNdtGjhyp41AOW7sWuna1u0SMHOknlSHCw+Hrr23XXr16Tkej/Eh4\neDjh4eG5dj6nEtR5oHiq+4oD0ek9QUSewY5VtTDGpDvK1Lz5SO6+Gx5/PFfiVF7so4/glVfsv361\n+PbOO+2Oilo/T+Wz1I2CUaNG5eh8TiWoXUBBEbnRrZuvIel03YlIX2AY0NIYcyyjE5ctC8eP52qs\n2aatJ2fExsIzz8CqVXbxrV/u4aTJSfkARzo8jDExwDxgtIgEiUhzoCPweepjRaQn8CZwtzHmwNXO\nff31npOgVP47fBhatbLbZaxd6+PJadUq2zxUykc52SM/EAgCTgJfAk8aY/4UkRYics7tuDeA0sB6\nEYl2rYeakt5JtQXlv5YutVtkdO5sy8sFZzii6cUiImwJjEce8cPyF8qfOLZE0RhzBuicxv2/4jY+\nZYypmpXzli0LxzLsBFS+JiEB3ngDpk2Dzz+3QzA+6dw5+Pe/7Rt97jn45BMICnI6KqXyjM+toa9Y\n0e5Q7Qm0BZX3jh+Hnj3BGNi4EcqVczqiPPT88zYbb9kCN9zgdDRK5Tmfq8V38aKhRAlbB7NAAacj\nUnlp2TLo1Qv694cRI/zg+x0fr3WZlFfx1oW6eaZIEbj2Ws/o5tMWVN5ISIBRo+CxxyAszP7f55MT\naHJSfsfnEhRAlSqwf7/TUai8cOyYLSsXHm679O66y+mIctnBg9C3ry23rpSf88kEFRpqJzo5TVtQ\nuWvBAmjSBG6/3W4w6FPjTX//DUOHQuPG9o2VKeN0REo5zif7DKpUgb17r36c8g4xMbbQ648/2unj\nzZs7HVEuiomxe36MG2drMm3dCuXLOx2VUh7BJ1tQtWvDzp1OR6EtqNzwxx9w000QHQ2bNvlYcgLb\nctqyxS66nTpVk5NSbnw2Qem2794tMRH++19o1w5efRW++AJKlHA6qjxQsSLMmuXjJS+Uyh6fm2Zu\njCEmBq67zq5r1IlP3ufwYXjiCbh0yS68rVLF6YhyQXy8rb+kY0vKj+g08zQEBdmafPv2OR2Jygpj\n4NNP7TyB1q3tTD2vT04JCbb5V7s2vPuu09Eo5VV8MkEBNGwImzc7G4OOQWXe0aN2T73x4+0Mvdde\n8/LWb3y8TUz16tmxpWnT4D//cToqpbyKzyaoJk3sOhnl2Yyxn+ONGtnJEOvW2f97NWPg1lvh449h\n4kS758cddzgdlVJexyfHoAAWLfr/v8aVZzpxAgYMsEsCPv3UJiifceSI1stTfk/HoNJx882wYYOd\nDaY8izEwcyY0aAB16tjvk9cmp/T+wNPkpFSO+WyCuv56e3NyHErHoK60b58tVfTWW7BwIYwdC4UL\nOx1VNuzfD//6F9xzj9ORKOWzfDZBge32X77c6SgU2DkD77xjNxRs29a2mm65xemosmHNGnj4YdtE\nDwyEGTOcjkgpn+WzY1AAX39t19EsXOhgUIr16+Gf/7Rr06ZOhRtvdDqibOrRA1avti2nvn19eMte\npXJHTsegfDpBnTwJNWrA6dNePmXZS50/b6eLz5xpq0I89hhItn9UPcCePbYSsf4wKZUpOkkiA2XK\nQIUKtp6bE/x1DMoY+OoruzY1MhK2b7cbC3pNcoqKSvv+atU0OSmVj3z+t+3ee+G776BpU6cj8Q/b\ntsGgQTYxzZwJLVs6HVEmXb4M//sfTJ5s/79qldMRKeX3fLqLD+zCz8cfhz//9KK/4L1QVBSMHGkX\n3b7+Ojz5pJc0No4ds1Uepk2zg2MDB0LnznYChFIqR3LaxecNHyE50rQpxMbabXYaNHA6Gt+TmGgn\norz0EnToYKvIh4Q4HVUW9O4NVavazabq13c6GqWUG59vQQG8+KL9a/7NN/M3npEjR/r0OFR4uN1I\nsGBBmDTJS6eNG6NNa6XyiE6SyIRHH7VdT/HxTkfiG3buhI4doU8fGDbMLg3y2OR08SJ8+SV8+GHa\nj2tyUspj+UWCatTIzubL7/VQvtZ6OnnSDtG0bAmtWtlE1b27B37GJybaFdr9+tmSQ59/bn8AlFJe\nxS8SFNiZZRMnOh2Fd4qOhjFjbN28QoVsYho61ENLFMXE2E2khgyxAW/daseX2rd3OjKlVBb5TYJ6\n6CHYtQt+/z3/XtPbW1AXL8K4cXb5z59/2iIK48fDtdc6HVkGgoJgxQr7jR4yRIu2KuXF/CZBFSoE\nw4fDq686HYnni421y4GqVYPffoOlS+0wTvXqTkfmsm+fLey3fn3aj4eG5ms4Sqm84Rez+JLExUHN\nmhAWZsdQVEpxcXa45o03oG5dGD3ag7bB2L0bvvnG3g4dsmuVnn3WBqqU8khai8/N1RIU2JbAu+/C\n2rVespA0H8TEwPTptlFSs6ZdcNu8udNRuZk1CwYPhi5doGtXO0tDv3lKeTxNUG4yk6CMgbvvtiWQ\nhg7N23g8fR1UVBRMmQITJkCzZrYL1NHp4vHxaSee2Fh7f4EC+R+TUirbdB1UFonYqjZvvWWLmPqj\nw4dtMqpa1VZ+WLoU5s93KDlFRNg9ONq3t4NcaW2BXLiwJiel/JDftaCSfPqpTVLr10OxYnkbl6dY\nu9bOwvvpJ1td/LnnbJJyxPPP24Vp0dHQrp2tk3TPPVCypEMBKaVym3bxuclKggL4xz/s4tO5c313\nSOPyZZg3zyam48ftvIK+faFECYcDS9qPo359D1zpq5TKDZqg3GQ1QcXFwQMPQMWKttsvIJc7PJ0c\ng9qzBz7+2M5YrFnTbgL7wAP50FMWFWXnpq9caW/DhtkXVkr5HR2DyoHAQDtreedOW1fO22v1Xbpk\n92Bq2xZuv92+n+XLbVHXBx/M4+Q0ezY0aWIXxr79tm2Sjhhhg1FKqWzw6xZUkgsX7OzluDg7o7lM\nmTwILo8kJtqGysyZtquycWPbddmpUx6UIoqMtC2kKlWufGzrVjuedNNNHloDSSmV37SLz012ExRA\nQoL9g/+zz2zl89atczm4XGQMbNxok+lXX9nSQz16wCOPQOXKufQiUVF2V9ktW2DzZjub5ORJW9Qw\nv/ctUUp5Ja/t4hORUiIyX0TOi8h+EXk0g2PfEpHTInJKRN7Ki3gKFLCfux98AI89Bk88YT+PcyI3\nx58uXbI1T59+GipVsluIBAXB4sU2f7z4YjaT04ULad+/a5edWXH6NNx3HyxYAGfOaHJSSuUbJ+eu\nTQEuASFAE2CRiGwyxvzpfpCIDAA6AknbnS4Rkb3GmGl5EVSHDrb1NHIk1Kpld2wYPBjKlcuLV0uf\nMXZsbPlyWLLErlWqX9/uw7R4sY0tK5Pfwn/8kTbnz9vEk3T76y9bt27jxiuf0LSpfSE/ER4eTps2\nbZwOwyPptcmYXp+840gLSkSCgC7Aq8aYi8aYVcACoFcahz8OjDPGHDPGHAPGAU+ke/KQEDsO0qWL\nnbr23ntw/nyW4gsOtlW8N2+241K1a9tJBvPmZe1UWWlBXbxoN/57/33bOipf3jZc1q+3Zef27IFf\nf7WT4mrXdktO8fFw7Bhs2GADnJZ23g5fvtzWeYqKsqWC3nrLlijfsCHzb8iHhYeHOx2Cx9JrkzG9\nPnnHqRZUDSDeGLPX7b7NQFolXOu6HnM/Lv0Kodu2wYEDcPCg/Xf//vTnj3fsCNdcA9dfb2/XXmsX\ninbpAgULUrGiLQM0Zoyd7TdlCvTubRsXt95q82DDhrbLLTPzAoyxvWSHD6dsyGzaBLv+MjSudZHb\n60XRte5ZxnWLovw1UXDuHHTrduXJYmPtJnxnz0Lp0jajVa5sV96mtY150aK2XIRSSnkJpxJUMSAq\n1X1RQHAmjo1y3Ze2pGSTmbo9zz5rV6+eOGFv+/bZFkaXLikOCw6209D7vHAdiUUCiN1WhJjNhYm+\nXIRzsYUpZdYQXKog5crZfBcUZMe0Xl7RiiKFC5OYCAlxiRAXS9GAWB6vvoZqNQtQsya0aGF3qa1f\nD4qElIHjwbC+hF1JW7Kk/fehh65MsoUL22R83XVaBkgp5ZMcmcUnIo2AX40xxdzuGwy0NsZ0SnXs\nWeAuY8wG19dNgOXGmCtqIYiI70xJVEopH5CTWXxOtaB2AQVF5Ea3br6GQFrlW7e7HksaLGmUznE5\nuhBKKaU8iyOTJIwxMcA8YLSIBIlIc+xMvc/TOPwzYLCIlBeR8sBg4JP8i1YppZQTnCx1NBAIAk4C\nXwJPGmP+FJEWInIu6SBjzIfAQmArsAVYaIz5yImAlVJK5R+fqiShlFLKd/h1sVillFKey6sSlKeV\nR/I0mb0+IjJURLaKyDkR2SsiQ/M71vyWlZ8d1/GFRGSniBzMrxidlMXfrSYiskJEokXkmIgMys9Y\n81sWfq8CRWSqiBx3ffZ8KyL5XIMmf4nIQBFZLyKXRGTGVY593vXzckZEPhaRQlc7v1clKFKWR3oM\n+EBEaqc+KFV5pAZABxH5Z34G6pBMXR+XXkBJ4D7gGRF5OH9CdExWrg3AMOB4fgTmITL7u3Ut8APw\nAVAKqAb4ek2szP7s/Au4FagHlMeu2ZyUX0E65AjwBjA9o4NE5B7s79QdQChwIzDqqmc3xnjFDTuh\nIha40e2+z4CxaRy7Cujv9nVf4Den34OnXJ80njsBmOD0e/CUawNUwS5luAc46HT8nnR9gDeBMKdj\n9tBrMwX4j9vX9wN/Ov0e8uk6vQHMyODxL4Exbl+3BY5d7bze1IJKrzxSWmWPslYeyTdk5fqk1pJ0\n1pb5iKxem4nAcOxfzf4gK9fnNuCMiKwSkROubqyK+RKlM7JybaYDLUSknKveaE/g+3yI0Ruk9Zlc\nRkRKZfQkb0pQeVceyTdk5fokE5FRgODba8syfW1EpDNQwBizID8C8xBZ+dmpgC3gPAioCEQAs/Iy\nOIdl5drsAg5iu73OArWwLQuV9meycJXPJ29KUOeB4qnuKw5EZ+LY4q77fFlWrg8AIvIMtk/9fmPM\n5TyMzWmZujauv3rfwn74gv0F8gdZ+dm5CMw3xvxujInDjiPcLiIZftB4saxcm6lAYezY3DXAfODH\nPI3Oe6T1mWzI4PMJvCtBJZdHcrvvauWRkqRbHsmHZOX6ICJ9sYOWbY3dxsSXZfbaVAcqAytF5Bgw\nFygvIkdFpFL+hOqIrPzsbMF+sLgz+G4yz8q1aQB8aoyJcv3BNwm4RURK50Ocni6tz+QTxpgzGT7L\n6cG1LA7EzcQOtgUBzYEzQO00jhvguiDlXbdtwD+cjt+Drk9P4BhQ0+mYPenaYP9gK+N26wwcxs7e\nEqffg9PXx3XcHcDf2A/jQsB7wAqn4/eQazMDmINtHRQCXgYOOR1/Hl+bAkARYCx28khhbBd56uPu\nAY4CtbEtzKXAm1c9v9NvMIsXoxS22Xwe2/fd3XV/C+BcqmP/4/pFOg382+nYPen6APuwM5POYZvY\n54ApTsfvCdcm1XNa4wez+LJ6fbB/AB52/X59C9zgdPyecG2A0sAXwAkgEvgFuNnp+PP42rwOJAIJ\nbrcR2PHJaKCC27H/wi7dOAt8DBS62vm11JFSSimP5E1jUEoppfyIJiillFIeSROUUkopj6QJSiml\nlEfSBKWUUsojaYJSSinlkTRBKaWU8kiaoJTKByLSWkQSvbHsjYgsF5GJTseh/I8mKOVTRKSRiMSL\nyMpsPPd1EdmaF3G5eOuq+M7Y7UcAcO0qO9jBeJSf0ASlfM0/gMlAPRGpmY3ne2sSyZbMbLttjDlr\njLmQH/Eo5U4TlPIZIlIE6AF8BHwD9E/jmHIi8qWInBaRCyLyu6v7rTe2rlhdV1dcgog87npOooh0\nSXWeFK0IEXleRDaLyHkROSwiH4lIiSzG38V1jhgR+dvVtRbieux1EdkqIv1E5IDrmPmuLdiTnn+z\niPwkIqdEJEpEVorIbaleI1FEnhaRuSJyHnhTRAqKyEQROSIil1znH+v2nOQuPhFZjq34/o7bdQpy\nvV7qa3S3iMQlvQelskoTlPIl3YAIY8w2bNHOx0WkQNKDrv2efgEqAZ2AesBo18OzgXHAX8D1QDng\nqyy8dgLwHFAHeBRoit2ZN1NE5Hrsxn+fYDe6awl8nuqwUGwl+geAO7Hbg0x3ezwYW1G6uev1/wAW\npTHuNQJYhH3/k4FnsdfjYaAa0B17HdLSBVsodhRQFihnjIlxxd431bF9gAXGmFMZvnml0lHQ6QCU\nykX9sB/QGGNWiMgFoCO2EjXYD/cywC3m//eh2Z/0ZFeLIj47H6jGGPdkdFBEXgT+B/TO5CnKY38f\n5xpjDrnu25HqmCJAL2PMEVe8A7B7V91ojNlrjFnufrCIPAd0Be7FbhmRZLYxZobbcZWBXcaYVa67\nDgNr0nmfZ0QkAThvjDnp9tBHwGoRKWeMOSYiJYEHgYcy+f6VuoK2oJRPEJFq2JaD+/bjM7FjUkka\nAVvM1TZJy97rtxWRxSJySETOAfOAQBEpm8lTbMbukbNdRL4RkSdF5LpUxxxJSk4ua7FbHdR2xRAi\nIh+KyF8icha7jUoItsXobmOqrz8FGovILhF5X0TuF5EsbUBojNmI3XctKSH3xG45oTvKqmzTBKV8\nRX/sz/MhEbksIpeBF4G7ReQG1zHZ3fU1rR1jkycXuHbb/Q67SWZXoAn/390VmKkXMCbRGNMOuBub\nrPoBu0Wkfhbi/Ay4CdvV2Ay7g+mRNGJIMeHBGPMHdlxpOPZ9hgGLs/C6ST7Gduvh+vcTo/v5qBzQ\nBKW8nmuc6XHgJeyHsvttC///ofk70CCDtUhx2B1CUzuFHZNKer3r3b8GbsYmrMHGmLXGmD3ADWSD\n6/lvGGOaYncg7e728A1uyRbgVmxCSeoKbA5MMsb8aIz5E5uI3OPM6HUvGGPmGmMGAu2BO12t0rSk\nd52+cMU4EGiMbZkplW2aoJQv6ABcC3xsjNnhfsNOdOjnOm4mcBL4n4i0EJFQEXlARFq7Ho8AKotI\nYxG5VkSSWh7LgIEicpOINMZOZLjo9vq7sb9Lz7vO+Si2FZNaui04EblVRF5xzcSrKCKdgArYVlmS\nS0CYiDQUkWbAB8B3xph9rsd3AY+JSG0RaYrt7oy9yrVLmoH4iIjUciWlnkAUdiwqLRFASxEp7z6L\n0BhzDjt7chx2G/i9V3ttpTKiCUr5gr7AsnTGluYAlUTkLtdss9bYbq8F2DGTkfz/2qe5wPfYsaCT\nwCOu+4cA+4DlwNfYCQHJEwSMMVuxCel5bELp63pOahl1d0VhW0ALsYnmHWC0McZ9TG0/drbhQmAJ\nsIeUM+f6AMWADdhkPB2bTK4WQzTwAnZMawPQALjPGHMpneckbem9F7fr4DId26U4HaVySLd8V8oL\niMjrwEPGmAZOx5IREemObdmVd0twSmWLTjNXSuWYiBTFjncNB6ZpclK5Qbv4lFK5YRiwEzgNjHE4\nFuUjtItPKaWUR9IWlFJKKY+kCUoppZRH0gSllFLKI2mCUkop5ZE0QSmllPJI/wdwbcUUO5YNogAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846b182e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = 0.1\n",
    "q = np.linspace(0.001, 0.999, 500)\n",
    "kl_div = p * np.log(p / q) + (1 - p) * np.log((1 - p) / (1 - q))\n",
    "mse = (p - q)**2\n",
    "plt.plot([p, p], [0, 0.3], \"k:\")\n",
    "plt.text(0.05, 0.32, \"Target\\nsparsity\", fontsize=14)\n",
    "plt.plot(q, kl_div, \"b-\", label=\"KL divergence\")\n",
    "plt.plot(q, mse, \"r--\", label=\"MSE\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Actual sparsity\")\n",
    "plt.ylabel(\"Cost\", rotation=0)\n",
    "plt.axis([0, 1, 0, 0.95])\n",
    "save_fig(\"sparsity_loss_plot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 1000  # sparse codings\n",
    "n_outputs = n_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kl_divergence(p, q):\n",
    "    # Kullback Leibler divergence\n",
    "    return p * tf.log(p / q) + (1 - p) * tf.log((1 - p) / (1 - q))\n",
    "\n",
    "learning_rate = 0.01\n",
    "sparsity_target = 0.1\n",
    "sparsity_weight = 0.2\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])            # not shown in the book\n",
    "\n",
    "hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.sigmoid) # not shown\n",
    "outputs = tf.layers.dense(hidden1, n_outputs)                     # not shown\n",
    "\n",
    "hidden1_mean = tf.reduce_mean(hidden1, axis=0) # batch mean\n",
    "sparsity_loss = tf.reduce_sum(kl_divergence(sparsity_target, hidden1_mean))\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE\n",
    "loss = reconstruction_loss + sparsity_weight * sparsity_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.134832 \tSparsity loss: 0.421739 \tTotal loss: 0.21918\n",
      "1 Train MSE: 0.0587859 \tSparsity loss: 0.0108979 \tTotal loss: 0.0609655\n",
      "2 Train MSE: 0.053738 \tSparsity loss: 0.0201038 \tTotal loss: 0.0577588\n",
      "3 Train MSE: 0.0476169 \tSparsity loss: 0.0399679 \tTotal loss: 0.0556105\n",
      "4 Train MSE: 0.0447499 \tSparsity loss: 0.0116199 \tTotal loss: 0.0470739\n",
      "5 Train MSE: 0.0403685 \tSparsity loss: 0.0930409 \tTotal loss: 0.0589767\n",
      "6 Train MSE: 0.0388338 \tSparsity loss: 0.0462908 \tTotal loss: 0.048092\n",
      "7 Train MSE: 0.0378196 \tSparsity loss: 0.0758871 \tTotal loss: 0.052997\n",
      "8 Train MSE: 0.0332092 \tSparsity loss: 0.0200693 \tTotal loss: 0.037223\n",
      "9 Train MSE: 0.0314318 \tSparsity loss: 0.0965061 \tTotal loss: 0.050733\n",
      "10 Train MSE: 0.0273777 \tSparsity loss: 0.0670885 \tTotal loss: 0.0407954\n",
      "11 Train MSE: 0.0246779 \tSparsity loss: 0.0900828 \tTotal loss: 0.0426945\n",
      "12 Train MSE: 0.0233311 \tSparsity loss: 0.0577432 \tTotal loss: 0.0348797\n",
      "13 Train MSE: 0.0228954 \tSparsity loss: 0.0623308 \tTotal loss: 0.0353615\n",
      "14 Train MSE: 0.0210913 \tSparsity loss: 0.0258186 \tTotal loss: 0.026255\n",
      "15 Train MSE: 0.0220006 \tSparsity loss: 0.483207 \tTotal loss: 0.118642\n",
      "16 Train MSE: 0.0190526 \tSparsity loss: 0.0361403 \tTotal loss: 0.0262806\n",
      "17 Train MSE: 0.0188885 \tSparsity loss: 0.132695 \tTotal loss: 0.0454275\n",
      "18 Train MSE: 0.0174156 \tSparsity loss: 0.0403093 \tTotal loss: 0.0254774\n",
      "19 Train MSE: 0.0178612 \tSparsity loss: 0.110486 \tTotal loss: 0.0399584\n",
      "20 Train MSE: 0.0168293 \tSparsity loss: 0.0291402 \tTotal loss: 0.0226573\n",
      "21 Train MSE: 0.0183871 \tSparsity loss: 0.364209 \tTotal loss: 0.0912289\n",
      "22 Train MSE: 0.0161226 \tSparsity loss: 0.0556278 \tTotal loss: 0.0272482\n",
      "23 Train MSE: 0.0158919 \tSparsity loss: 0.0792573 \tTotal loss: 0.0317434\n",
      "24 Train MSE: 0.0157006 \tSparsity loss: 0.149254 \tTotal loss: 0.0455514\n",
      "25 Train MSE: 0.0145307 \tSparsity loss: 0.136184 \tTotal loss: 0.0417676\n",
      "26 Train MSE: 0.0144209 \tSparsity loss: 0.110554 \tTotal loss: 0.0365316\n",
      "27 Train MSE: 0.0138508 \tSparsity loss: 0.0744676 \tTotal loss: 0.0287443\n",
      "28 Train MSE: 0.0139305 \tSparsity loss: 0.158476 \tTotal loss: 0.0456257\n",
      "29 Train MSE: 0.0133762 \tSparsity loss: 0.143838 \tTotal loss: 0.0421438\n",
      "30 Train MSE: 0.0137258 \tSparsity loss: 0.185643 \tTotal loss: 0.0508544\n",
      "31 Train MSE: 0.0139518 \tSparsity loss: 0.0635133 \tTotal loss: 0.0266544\n",
      "32 Train MSE: 0.013692 \tSparsity loss: 0.0577956 \tTotal loss: 0.0252512\n",
      "33 Train MSE: 0.0134704 \tSparsity loss: 0.104171 \tTotal loss: 0.0343045\n",
      "34 Train MSE: 0.0124406 \tSparsity loss: 0.136569 \tTotal loss: 0.0397544\n",
      "35 Train MSE: 0.0126563 \tSparsity loss: 0.162903 \tTotal loss: 0.0452369\n",
      "36 Train MSE: 0.0128764 \tSparsity loss: 0.0948648 \tTotal loss: 0.0318493\n",
      "37 Train MSE: 0.0123458 \tSparsity loss: 0.108087 \tTotal loss: 0.0339632\n",
      "38 Train MSE: 0.0121672 \tSparsity loss: 0.20089 \tTotal loss: 0.0523451\n",
      "39 Train MSE: 0.0122532 \tSparsity loss: 0.149409 \tTotal loss: 0.0421351\n",
      "40 Train MSE: 0.0125975 \tSparsity loss: 0.230649 \tTotal loss: 0.0587274\n",
      "41 Train MSE: 0.0124657 \tSparsity loss: 0.100664 \tTotal loss: 0.0325984\n",
      "42 Train MSE: 0.0116947 \tSparsity loss: 0.101108 \tTotal loss: 0.0319164\n",
      "43 Train MSE: 0.0122123 \tSparsity loss: 0.125789 \tTotal loss: 0.0373701\n",
      "44 Train MSE: 0.0117173 \tSparsity loss: 0.110301 \tTotal loss: 0.0337774\n",
      "45 Train MSE: 0.0116897 \tSparsity loss: 0.165081 \tTotal loss: 0.0447058\n",
      "46 Train MSE: 0.011611 \tSparsity loss: 0.130426 \tTotal loss: 0.0376962\n",
      "47 Train MSE: 0.0117358 \tSparsity loss: 0.163508 \tTotal loss: 0.0444374\n",
      "48 Train MSE: 0.0116507 \tSparsity loss: 0.459586 \tTotal loss: 0.103568\n",
      "49 Train MSE: 0.0116655 \tSparsity loss: 0.188114 \tTotal loss: 0.0492883\n",
      "50 Train MSE: 0.0114275 \tSparsity loss: 0.19221 \tTotal loss: 0.0498695\n",
      "51 Train MSE: 0.0113954 \tSparsity loss: 0.266457 \tTotal loss: 0.0646867\n",
      "52 Train MSE: 0.0119332 \tSparsity loss: 0.379985 \tTotal loss: 0.0879303\n",
      "53 Train MSE: 0.0113018 \tSparsity loss: 0.129771 \tTotal loss: 0.037256\n",
      "54 Train MSE: 0.0153057 \tSparsity loss: 0.434827 \tTotal loss: 0.102271\n",
      "55 Train MSE: 0.0134004 \tSparsity loss: 0.0833025 \tTotal loss: 0.0300609\n",
      "56 Train MSE: 0.0123188 \tSparsity loss: 0.297605 \tTotal loss: 0.0718399\n",
      "57 Train MSE: 0.0122943 \tSparsity loss: 0.247148 \tTotal loss: 0.061724\n",
      "58 Train MSE: 0.0239939 \tSparsity loss: 0.215717 \tTotal loss: 0.0671373\n",
      "59 Train MSE: 0.013596 \tSparsity loss: 0.203553 \tTotal loss: 0.0543065\n",
      "60 Train MSE: 0.0179108 \tSparsity loss: 0.180449 \tTotal loss: 0.0540006\n",
      "61 Train MSE: 0.0132258 \tSparsity loss: 0.231824 \tTotal loss: 0.0595906\n",
      "62 Train MSE: 0.0136533 \tSparsity loss: 0.498429 \tTotal loss: 0.113339\n",
      "63 Train MSE: 0.0143277 \tSparsity loss: 0.333901 \tTotal loss: 0.0811079\n",
      "64 Train MSE: 0.0119968 \tSparsity loss: 0.176848 \tTotal loss: 0.0473664\n",
      "65 Train MSE: 0.0156422 \tSparsity loss: 0.173917 \tTotal loss: 0.0504256\n",
      "66 Train MSE: 0.0150095 \tSparsity loss: 1.02187 \tTotal loss: 0.219383\n",
      "67 Train MSE: 0.036823 \tSparsity loss: 0.323619 \tTotal loss: 0.101547\n",
      "68 Train MSE: 0.0148193 \tSparsity loss: 0.230714 \tTotal loss: 0.060962\n",
      "69 Train MSE: 0.0126409 \tSparsity loss: 0.454552 \tTotal loss: 0.103551\n",
      "70 Train MSE: 0.045501 \tSparsity loss: 0.745102 \tTotal loss: 0.194521\n",
      "71 Train MSE: 0.0143786 \tSparsity loss: 0.229362 \tTotal loss: 0.060251\n",
      "72 Train MSE: 0.0151026 \tSparsity loss: 0.826014 \tTotal loss: 0.180306\n",
      "73 Train MSE: 0.0136122 \tSparsity loss: 0.316737 \tTotal loss: 0.0769596\n",
      "74 Train MSE: 0.0309757 \tSparsity loss: 0.289552 \tTotal loss: 0.0888861\n",
      "75 Train MSE: 0.0304744 \tSparsity loss: 0.489417 \tTotal loss: 0.128358\n",
      "76 Train MSE: 0.0204102 \tSparsity loss: 0.201982 \tTotal loss: 0.0608067\n",
      "77 Train MSE: 0.0211023 \tSparsity loss: 0.32347 \tTotal loss: 0.0857964\n",
      "78 Train MSE: 0.0178777 \tSparsity loss: 0.533425 \tTotal loss: 0.124563\n",
      "79 Train MSE: 0.018841 \tSparsity loss: 0.424661 \tTotal loss: 0.103773\n",
      "80 Train MSE: 0.0159234 \tSparsity loss: 0.115559 \tTotal loss: 0.0390352\n",
      "81 Train MSE: 0.0129649 \tSparsity loss: 0.912508 \tTotal loss: 0.195467\n",
      "82 Train MSE: 0.0162278 \tSparsity loss: 2.17347 \tTotal loss: 0.450922\n",
      "83 Train MSE: 0.0146708 \tSparsity loss: 0.681089 \tTotal loss: 0.150889\n",
      "84 Train MSE: 0.0150686 \tSparsity loss: 0.292309 \tTotal loss: 0.0735305\n",
      "85 Train MSE: 0.0250247 \tSparsity loss: 0.949989 \tTotal loss: 0.215023\n",
      "86 Train MSE: 0.0146914 \tSparsity loss: 0.685326 \tTotal loss: 0.151757\n",
      "87 Train MSE: 0.0122667 \tSparsity loss: 1.44823 \tTotal loss: 0.301912\n",
      "88 Train MSE: 0.0197259 \tSparsity loss: 0.861047 \tTotal loss: 0.191935\n",
      "89 Train MSE: 0.0331342 \tSparsity loss: 0.291833 \tTotal loss: 0.0915009\n",
      "90 Train MSE: 0.0295548 \tSparsity loss: 0.445159 \tTotal loss: 0.118587\n",
      "91 Train MSE: 0.0145762 \tSparsity loss: 0.0887034 \tTotal loss: 0.0323169\n",
      "92 Train MSE: 0.0147775 \tSparsity loss: 0.390856 \tTotal loss: 0.0929486\n",
      "93 Train MSE: 0.0166543 \tSparsity loss: 0.155326 \tTotal loss: 0.0477195\n",
      "94 Train MSE: 0.012198 \tSparsity loss: 0.12071 \tTotal loss: 0.03634\n",
      "95 Train MSE: 0.0141104 \tSparsity loss: 0.107212 \tTotal loss: 0.0355529\n",
      "96 Train MSE: 0.018834 \tSparsity loss: 0.230255 \tTotal loss: 0.0648851\n",
      "97 Train MSE: 0.0134663 \tSparsity loss: 0.102045 \tTotal loss: 0.0338754\n",
      "98 Train MSE: 0.013678 \tSparsity loss: 0.0839055 \tTotal loss: 0.0304591\n",
      "99 Train MSE: 0.0245401 \tSparsity loss: 0.335841 \tTotal loss: 0.0917084\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 100\n",
    "batch_size = 1000\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        reconstruction_loss_val, sparsity_loss_val, loss_val = sess.run([reconstruction_loss, sparsity_loss, loss], feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", reconstruction_loss_val, \"\\tSparsity loss:\", sparsity_loss_val, \"\\tTotal loss:\", loss_val)\n",
    "        saver.save(sess, \"./my_model_sparse.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_sparse.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHQtJREFUeJzt3W9sluXZx/ETrNA/FApCW/5YAdmk/kU2om6+WLZIombq\nXEzmTLYFTSSaaGYyjSZLthfLXm7sxeKWGd3UF0tc2KJEzeZwU6MIDLr4BwciCgShSGlpaaGgz6vn\nxZPz99tzndztjUf7/bw8ct7Xdd333fbgyvXjOKd89tlnCQCACKae7QsAAKAqmhYAIAyaFgAgDJoW\nACAMmhYAIAyaFgAgDJoWACAMmhYAIAyaFgAgjIY6novRG6iHKWf7AiaD3t7e7Pf59OnTcu3UqdX/\nbfzpp59mtXPOOUeuVdN8pkzRX79a66YBqet1xx0dHa30+hLu9SMjI5Wvq6mpKau570dxx1Xfj1ur\nnDp1StYbGvJW1N7eLg/MnRYAIAyaFgAgDJoWACCMej7TAjBBlDxPKnnmodaOxU4U6nmOe1am1qpn\nLq5e8vys5L2de+65Wc09/1LPjkq+h7H4zNUx3OdYcj7utAAAYdC0AABh0LQAAGHQtAAAYdC0AABh\nkB4EUEwl79zEBZUMU5MkUtIJt+nTp8u1ajqDqrnjurWKe28q0aeSe+716rrc2pLrVd9PyWdTkjR0\nxisJyp0WACAMmhYAIAyaFgAgDJoWACAMghgAipVsc6G4cT4lwYSScIUad+SOq0IMJcEEdS43Mqrk\nuCXXpQIP06ZNq7x2LIIjKmxTEqpxuNMCAIRB0wIAhEHTAgCEQdMCAIRB0wIAhEF6EMC4qnXDSDUW\nKSWdQCw5bnNzs1yrzufGD6nknEoPqnFP7rrcudRxXequZJRUSbJSXZtLRqq627SyBHdaAIAwaFoA\ngDBoWgCAMGhaAIAwCGIAKFYSrigZ56O48UMle2+pY7hRUioscPLkSblWBSEGBgYqrUup+n5cru6O\n29jYmNXcexgeHs5q7jNX1zsW+5IxxgkAMCHRtAAAYdC0AABh0LQAAGHQtAAAYZAeBFBMJezc+KGS\nxFnJmCC11l3DyMhIVisZ+eTSdCqteN5551W+LpWmU2nLlHR6UL2vlFJqaWnJaiol6M7nUn7uc1Cq\njrhKqXCTzcorAQA4y2haAIAwaFoAgDBoWgCAMCZVEOONN97IauvWrZNrFy5cmNWamprk2u9///tZ\nbc6cOXKtqwORuGCBoh6yl4x8OnDggFy7e/furNbT0yPXHjt2LKt1dnbKtao+f/58uVZRfydmzJgh\n16oghxvjpIINbq36zGbOnCnXtra2yrqighglI5jczw1BDADAhETTAgCEQdMCAIRB0wIAhEHTAgCE\nMaUkBVSjup3Iueiii7Lazp07x+Vcs2bNkvWrr756XM43XhYvXpzVHn74Ybm2q6trnK+mkuoxJJyx\ngwcPZr/PJWOR3JggVd+3b59c++c//zmrbdiwQa5VCTn1s52STsOppGJKKfX29mY1NbbKpYZnz56d\n1dzYKpVAdOnBvXv3ZrXLL79crr3pppuy2hVXXCHXqs/RpQfVd1mSNOzo6JA/UNxpAQDCoGkBAMKg\naQEAwqBpAQDCmFRjnNSD2+3bt8u1l1xySVZ7++235dpNmzZltb/85S9y7YsvvpjVlixZItd+8MEH\nsl5VQ4P+etVIGvXg1nEPsB966KHKx0BsJWN3SvanOnnyZFZra2uTa1euXJnVXLCso6Oj8jXs2LEj\nq+3fv1+uVcEC9Xs3NDQkXz8wMJDV3LglNfKpr69Prt2yZUtWc+OwVEBD/f1LSY/ZcuEKFUhxIRO3\nh5jCnRYAIAyaFgAgDJoWACAMmhYAIAyaFgAgjEk1xqmeRkZGZH3Pnj1ZzaUH3eiYqlw6SqUH3TWo\nMTXr16+Xa2+++eaCqxs3jHGqg0OHDlX+fS5JGioqhZZSSsePH69USymlefPmZbXDhw/LtWpslDtu\nY2NjVlMJOfd6lSpU15qSTunt2rVLrv3Vr36V1c477zy59t57781q1157rVyr3ptKfKbk08uK6kOd\nnZ2McQIAxEbTAgCEQdMCAIRB0wIAhDGpxjjVk3pAm1JKy5cvr3yM7u7usbqc/0ONnXIPpa+66qqs\ntnr16jG/JsTnRvSoB/UunKGO4YIYatyR27fK/T4q6nxNTU1yrdozSo0kcoE3tUeWO9eRI0eymho5\nlVJKLS0tWc39PVF19fqUUjpx4kRWc2Oc1Ht245rUiCqHOy0AQBg0LQBAGDQtAEAYNC0AQBg0LQBA\nGKQHJzC38dy3vvWtrOYSQL/85S+zmks3YXJTSbqUdFLQ/byp0T8qsVZ6Dep8w8PDcq0af+bSh+ra\nVM0lIFVK79SpU3Ltu+++m9U2bNgg1/b392c1t7Fje3t7VnNpR3Vt7r2pJOjUqbXfJ3GnBQAIg6YF\nAAiDpgUACIOmBQAIgyDGBPbEE0/I+scff5zV3F47F1xwwVheEiYIFVZw+yop7uG94kIBao8qFxJS\nQQy3550KXbixU+ra1Gfj3q+6LjdSbePGjVmtp6dHrlXj4lasWCHX1vpdloQrCGIAACYVmhYAIAya\nFgAgDJoWACAMmhYAIAzSgxPE+++/n9UeeOCByq9//fXXZb2zs/OMrwkTlxpV5JJhqu7SaSr958Ya\nqTFBJeOHVGrO1d3Ip6rpQZc+HBwczGrbt2+Xa7dt25bVzj//fLn2O9/5TlZbtGiRXKs+m5KNHdXo\nrZT0ho/uZ8R9bwp3WgCAMGhaAIAwaFoAgDBoWgCAMAhiTBDPPvtsVlMPQlNK6bbbbstqS5cuHfNr\nwsTlggVVuXCFCmiUjP5xxy3ZI0sFC44dOybXqlFSKiDS2toqX69GNj333HNy7b///e+stnbtWrl2\n9erVWW369OlyrXq/LohREqop2Uet5OeJOy0AQBg0LQBAGDQtAEAYNC0AQBgEMYJx4Yr169dnNffg\n9ec//3lWUw+PAUc9UHcP09XDe/ezqY5bEgpwa0+fPl15reKCFIqa6tHc3CzX/uc//8lq//znP+Xa\nuXPnZrUbb7xRrlX747mpHmqvLxdoUcETt1eY+n7YTwsAMKnQtAAAYdC0AABh0LQAAGHQtAAAYZAe\nDOaxxx6T9VdeeSWrffe735VrGdmEWqkUWMkoHrdW1V06zSXcqq5VicKU9N5Obq2qq9q+ffvk61Xq\n1629++67s9qyZcsqX1dJstKllEsSlyXYTwsAMCHRtAAAYdC0AABh0LQAAGFMKXkAVqO6nWii2L59\ne1ZbtWqVXKvGzGzZskWuneBBjNo2ekIlhw8frvz7rEIQ7u9OyZifklFSas8oFzZQe2ep8UUp6ZCI\nOu6f/vQn+fqnnnoqq7mRak8//XRWW7lypVzrrldR53Mjn06cOJHVWlpa5Fq3z5aivrf29nb5ZXKn\nBQAIg6YFAAiDpgUACIOmBQAIg6YFAAiDMU6fAy6pc/vtt2c1N07mjjvuyGoTPCWIs6jWTSAddYyS\nkU/uXCo96H6X1Nq2tja5trGxMav94x//yGrPPPOMfL1KKq5Zs0auXbx4cVZzo6xUOtN9Nirl5z5z\ntXlnyTitscCdFgAgDJoWACAMmhYAIAyaFgAgDIIYdaYeYN94441y7XvvvZfVuru75dqf/vSntV0Y\nUKAkiKHq7uG9GimkHv47IyMjsq5CCC6Y4MYSKX19fVlNjWzasWOHfP0tt9yS1b75zW/KtSr04UYl\nqc9XjWBKSY+ict9lyX5a6hjuekvCOtxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIgPVhnR44cyWov\nv/xy5dc/+eSTsj5nzpwzvSSgWEnaS40UciOUVF2l21LSSUOXblPXO23aNLlWXa9LJfb09GS1t99+\nO6tddtll8vX3339/VrvyyivlWvXeXApTpfTc5pKq7tKD6rNxa9V36ZKg7udB4U4LABAGTQsAEAZN\nCwAQBk0LABAGQYxx0t/fL+tXX3115WM89dRTWc09pAXqSQUA3IN+9fC+1nOlpB/euzFBKiwwc+ZM\nuVaNO1KBi5T0yKbBwcGs9r3vfU++Xv09cMET9TfFfTbqGO77UVzQRo2SqnW003+rK9xpAQDCoGkB\nAMKgaQEAwqBpAQDCoGkBAMIgPThOHn/8cVnfvXt35WNce+21Wa0kZQOMF5Xca2io/ufEbbQ4PDyc\n1dzPvErOuSTb6OhopXOllNLRo0ez2ubNm+XaTZs2ZbV58+Zlteuuu06+Xo2SGhoakmvVe3OfuUoK\numSlSkuWjFty349KIJYkDR3utAAAYdC0AABh0LQAAGHQtAAAYRDEGAM7d+7Maj/5yU/qfyFAnbgH\n9Yoa4+RCASpA4PZaUgGAkjDIsWPHZP3999/Pam6Mk9pna9GiRVnNjWbq6+vLaio0kpIObZTsCeao\nwIQbD6U+85JwmDtuyffGnRYAIAyaFgAgDJoWACAMmhYAIAyaFgAgDNKDY+CVV17JagMDA5Vf393d\nLetNTU1nfE3AeFJpOJd6U+myko0dXVJRJeSam5vlWuXgwYOy/s4772S1N998U65Vv+etra1ZTY1K\ncq8vSdKVrHWbQJaMVlJJQ5dUVPWx2CiUOy0AQBg0LQBAGDQtAEAYNC0AQBgEMersK1/5Slb761//\nKtcSxMDnlQoWqIf0Kekghnsg39jYWOn1jrsGFTZwoY0FCxZkteuvv16uVYEUtXbp0qXy9eq9uWCE\nqqsxUinpz8GFNtQ1lIQr3PW676JW3GkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwphSMj6jRnU7ESa1\n6lEznLHe3t7Kv891/BszJudS46hKNkVUY6fcJpDqekuSey4RqN5DSQrTXUNJ0rDkfGrtvHnz5AG4\n0wIAhEHTAgCEQdMCAIRB0wIAhFHPIAYAADXhTgsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAG\nTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQ\nBk0LABBGQx3P9Vkdz4XJa8rZvoDJYO/evdnv82efVf8VnzJFf02ffvppVps6Vf/bWp1Pvd4dw12D\nqrv3VvV63evdNSijo6OVzpVSSuecc07l46pra2jQreH06dM1Hbfkc+jq6pIfDndaAIAwaFoAgDBo\nWgCAMOr5TAvABFHy/Eo9Hzl16pRcW+vzIPcsRz17KnmedO6558q6es6klJzLUZ9NybOrkud97vtR\n3HtT35t7BleCOy0AQBg0LQBAGDQtAEAYNC0AQBg0LQBAGKQHARSbNm1aVnNJupIkmpq4UJKQK+HS\ndCr15t6Dujb1ObiEnUpWurXqXO6zUWlHN81CfQ4ln41Ld5YkQUtwpwUACIOmBQAIg6YFAAiDpgUA\nCIMgxhh4+umns9rQ0JBcu3Xr1qz229/+tvK5fvzjH8v617/+9az2ta99rfJxgRIqbDAmD9lrHPPj\nwiADAwNZ7aOPPpJr9+/fn9UOHTok1/b19WW14eHhrNbY2ChfP3/+/Kz25S9/Wa5dsmRJVps5c6Zc\nq5R8P25tSRCjJFRTcm3caQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCID1Y4J577pH13/zmNzUdtyQx\n9bOf/UzW169fn9VeffVVuXbWrFmVzwdUVbLRoUuLqXSZO64aJeXSaQcOHMhqu3fvlmvfeeedrLZ3\n7165tre3N6u1tLRktY6ODvn6o0ePyroye/bsrNbW1ibXqjFMJ0+elGtVys+NfFLfhdsgs0TJ30Du\ntAAAYdC0AABh0LQAAGHQtAAAYRDEMFTootbARUopXXnllVnt29/+tly7c+fOrPb73/9erlUPj595\n5hm59s477/xvlwj8v0rG+ahQgAtMqAf9zc3Ncq06hgpGpJTS3//+96y2ceNGufb48eNZrampSa69\n8MILs1p7e3tWc/txbdq0Kau99957cu2JEyey2q233irXLly4MKu5AJZ6v24clqq70EZJQMPt36Vw\npwUACIOmBQAIg6YFAAiDpgUACIOmBQAIY9KnB91GcL/73e8qH2PVqlVZ7YUXXpBrVRJKjaNJSady\ndu3aJde+9tprWe3w4cNyLTAe3CiekhE9KoHo0mn9/f1ZzSUCX3zxxazmNmpVv8+XXXaZXPuFL3wh\nq6kEpBuppt6Do0Y+ubRkQ0P+p13VUtKjoErSnW7Mlvou3dqSEWDcaQEAwqBpAQDCoGkBAMKgaQEA\nwpj0QQwXVlAPEdUD2pRS+tvf/pbVZsyYUduFpZSeeOKJrLZ58+bKr7/55ptrvgagKjeKp+Qhu6LG\nDKWU0v79+7PakSNH5Nrly5dntSVLlsi1V111VVZzY5xUSET9TVHXmlJKfX19Wa2xsVGuVftpuRCX\nCm244MnIyEhWc3+/SgIe6ufBjbMiiAEAmJBoWgCAMGhaAIAwaFoAgDBoWgCAMCZ9enDlypWyrhJA\nLqnjkkW1UqOkTp48OS7nAkqUjPNRXIpMpXbdGCiVWlu2bJlce/HFF2c1lcZz1+aSkWosm1p74MAB\n+Xo1Lqm7u1uuVWlHt9Gi+ptUMk6rJAlasvnnWOBOCwAQBk0LABAGTQsAEAZNCwAQxqQPYjizZs2q\n27mefPJJWe/p6al8jNWrV2e1Cy+88IyvCSjlHsiXUGEBFzZQQQoXxFDjltzIp+nTp2e1rq4uuXZ4\neDir7d27N6u5z+a6667Lapdccolcu2DBgqzmwg4qNFYyWs7tpzU6OprV3Bgn9V2Oyc9IzUcAAKBO\naFoAgDBoWgCAMGhaAIAwaFoAgDBID9bZtm3bstrdd98t1544cSKrzZ8/X65dt25dVnOpK6BWJSmw\nktFM6mfWbYqoknMqzZeS3hTRrV20aFFWmzlzply7Y8eOrLZ169ZK509JJwVd6lclmvv7++VaNcZp\nzpw5cq36HFVKMCX9mbmRXG7sncImkACACYmmBQAIg6YFAAiDpgUACIMgRp29/vrrWU0FLpy1a9fK\n+he/+MUzviagVMmDc8WN/lEP791IIRXaaGtrk2tVsKClpUWuVftWuSDFhg0bspr6HXdjoFpbW7Na\ne3u7XKsCLS7soMIr6lwple3Rp743NSIrJR3wcAGektAYd1oAgDBoWgCAMGhaAIAwaFoAgDBoWgCA\nMEgPjpM1a9bI+h//+MfKx/jhD3+Y1R588MEzviZgrKhkmEsUqrrbvNAl0RSVKnTXoDZ27OjokGvV\niKnt27fLtWqM0+LFi7PaqlWr5OtV6ldda0p6hJJLBKr35kZnubqi0oMlx3VrS8aCcacFAAiDpgUA\nCIOmBQAIg6YFAAiDIMYYGBwczGrPP/+8XDsyMpLV3APhRx55JKuV7FEDjBc1dqckROEeyKuAhhtz\npvZxckEMdb1ulNQHH3yQ1dRoppT0vlUqiHHxxRfL169YsSKruX2+VN3tNab+TpSEX9xa9fmORcCD\n/bQAABMSTQsAEAZNCwAQBk0LABAGTQsAEAbpwTFw2223ZbVDhw5Vfv19990n63PmzDnjawLGU0lS\nUFGbMqakx/kcPHhQrm1ubs5qLl2r0oOHDx+Wa9evX5/VXBq4s7Mzq82dOzerqY0lU9IbUbokndqs\n0Y18UolLt5mm+i7dWCV1DS7BqBKIjHECAEwqNC0AQBg0LQBAGDQtAEAYBDEKbN26VdZffvnlyse4\n9dZbs9oDDzxwppcEfG64h+klI3pUgECNPnNrVQgiJR382Llzp1z71ltvZTUXeLj00kuz2je+8Y2s\npkIj7hr27dsn17owh9Lb25vVVGgkJR2YGBoakmvV6CwX8CjBGCcAwIRE0wIAhEHTAgCEQdMCAIRB\nEMNQe9c8/PDDcq36X+LOl770pazGHlmIpmTagaImVKSkpzO4h/RqbV9fn1z77rvvZrVXX31Vrv3w\nww+zmvq9dXUVuhgYGJCv37ZtW1ZzoQ913JK/Pe646nN0E09UEMMp+Rkp+dnhTgsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBulB49FHH81qL730UuXXr1mzRtYZ2YSJoCTtVfJ6VZ81a5Zc29/fn9V27dol\n127cuDGrvfnmm3Kt2seuqalJrlX7YX3yySdyrbJixYqs5kY+dXR0ZDWXwlTX6/a9KklsqnFYLhmp\nrsFdL/tpAQAmJJoWACAMmhYAIAyaFgAgDIIYxiOPPFLT63/xi1/IOiObMBGoB+fu4b0a56Me6Lu1\nLkCgRgqpfaRSSumNN97IagcOHJBrL7rooqzW3t4u1ypqf6m2trbKr1f7hKWkP183mkkFWlRwxdX3\n7Nkj16r9v+bNmyfXLl26NKs1NOiW4+oKd1oAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMEgPjpPBwUFZ\nr3X8jaNSRCrFlJIe2+ISS4raIDOllNatW1f5GIq7XpXkdONg8PnjUoVVud+ZuXPnZjU3AunYsWOV\nainpjRXdpogHDx7Mavv27at8riNHjmQ1lwjs6urKamrkVEr6eo8fPy7XHjp0KKtt3ry58tru7m65\nVv2OuhQl6UEAwIRE0wIAhEHTAgCEQdMCAIRBEGOcLFy4sK7nW7t2bVZbsGCBXPvxxx9ntV//+tdj\nfk1jRX2Wd91111m4Evyvkv2PFDWuKSUdgnCjz1ToQoUVUtJ7UR09elSuVaOgtmzZIteqa1Pjofr6\n+uTr1V5ULniiPnM3Xkr9zgwNDcm16u/BRx99JNeq/cMuuOACuVZx37sLuijcaQEAwqBpAQDCoGkB\nAMKgaQEAwqBpAQDCID1o3HHHHVnt8ccfPwtXUs2jjz46LsdV41XcuCXlBz/4gaxfc801lY/x1a9+\ntfJa1EfJaCaVenMpMkVt9ujqnZ2dcu0NN9xQ+bhqVJFLS6oNENUGl26EkroGN6JMpRrdyKfW1tas\n1tTUJNeqpOGll15a+bjLly+XaxctWlT5GkpwpwUACIOmBQAIg6YFAAiDpgUACGNKreNYCtTtROPl\nD3/4g6yr0TMlenp6ZL3W0Uo/+tGPZH3ZsmWVj3HTTTdlNTc65nOits2bUMmHH36Y/T67cIb6G+PG\n9qi622vJhRCUTz75JKtt2rRJrv3Xv/6V1dwIpMWLF2c19Tmo86ekR1GVjK2aPXu2XKvCHO64ajST\n2qssJf1dqNenpK/XrVXf+/nnny9/oLjTAgCEQdMCAIRB0wIAhEHTAgCEQdMCAIRBehATDenBOtiz\nZ0/2+1wy3sttdFiSHlR/u0rGBLkxTiMjI1ltcHBQrm1ra6vp9TNmzMhq7j2oVKL7HFWi2V2D+t5c\nMlONo3KpRHXckvFfXV1dpAcBALHRtAAAYdC0AABh0LQAAGGwnxaAYi4AoJTsnaW4kU/quG7ckuKC\nI+p8aiRRyXFV4CIlPW7JBUSqnisl/dm4taruAnpuDJOiQhdu5J3bQ0zhTgsAEAZNCwAQBk0LABAG\nTQsAEAZNCwAQBulBAMVU4qwkJViSCHSpt9HR0azmRj6pY7hrUEk2l6ZTKUo1Akldq+PWquSdG6Gk\nrst9Nq5e9biO+i7duUp+drjTAgCEQdMCAIRB0wIAhEHTAgCEQRADQDG1Z5R7yK7G+bhggwpBlIwq\nckGBkmCCGgXlwhHqfahzuTFQJSEI9X7ddanPseRcjguvKCXhl5KAB3daAIAwaFoAgDBoWgCAMGha\nAIAwaFoAgDBIDwIoptJpKiXouESgSuO5TRHVMdw1qNRaSZLNjUtS16tqbkxRyXgn9Zm761JKvh9H\nvTd33JLRTC5NqnCnBQAIg6YFAAiDpgUACIOmBQAIY0rJAzAAAM4m7rQAAGHQtAAAYdC0AABh0LQA\nAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh\n0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYfwP7yJU8LUryWEAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8467498c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_sparse.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the coding layer must output values from 0 to 1, which is why we use the sigmoid activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.sigmoid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To speed up training, you can normalize the inputs between 0 and 1, and use the cross entropy instead of the MSE for the cost function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "logits = tf.layers.dense(hidden1, n_outputs)\n",
    "outputs = tf.nn.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_mean(xentropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variational Autoencoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 500\n",
    "n_hidden2 = 500\n",
    "n_hidden3 = 20  # codings\n",
    "n_hidden4 = n_hidden2\n",
    "n_hidden5 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "learning_rate = 0.001\n",
    "\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "my_dense_layer = partial(\n",
    "    tf.layers.dense,\n",
    "    activation=tf.nn.elu,\n",
    "    kernel_initializer=initializer)\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_inputs])\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3_mean = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "hidden3_sigma = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "noise = tf.random_normal(tf.shape(hidden3_sigma), dtype=tf.float32)\n",
    "hidden3 = hidden3_mean + hidden3_sigma * noise\n",
    "hidden4 = my_dense_layer(hidden3, n_hidden4)\n",
    "hidden5 = my_dense_layer(hidden4, n_hidden5)\n",
    "logits = my_dense_layer(hidden5, n_outputs, activation=None)\n",
    "outputs = tf.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_sum(xentropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "eps = 1e-10 # smoothing term to avoid computing log(0) which is NaN\n",
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.square(hidden3_sigma) + tf.square(hidden3_mean)\n",
    "    - 1 - tf.log(eps + tf.square(hidden3_sigma)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = reconstruction_loss + latent_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train total loss: 32440.1 \tReconstruction loss: 25031.5 \tLatent loss: 7408.61\n",
      "1 Train total loss: 30017.4 \tReconstruction loss: 23093.3 \tLatent loss: 6924.14\n",
      "2 Train total loss: 23337.9 \tReconstruction loss: 20221.0 \tLatent loss: 3116.88\n",
      "3 Train total loss: 21724.7 \tReconstruction loss: 18698.8 \tLatent loss: 3025.89\n",
      "4 Train total loss: 28219.0 \tReconstruction loss: 21493.3 \tLatent loss: 6725.66\n",
      "5 Train total loss: 25906.5 \tReconstruction loss: 19582.4 \tLatent loss: 6324.09\n",
      "6 Train total loss: 19198.3 \tReconstruction loss: 15831.6 \tLatent loss: 3366.69\n",
      "7 Train total loss: 17638.8 \tReconstruction loss: 14539.6 \tLatent loss: 3099.17\n",
      "8 Train total loss: 16688.3 \tReconstruction loss: 13615.9 \tLatent loss: 3072.4\n",
      "9 Train total loss: 17007.3 \tReconstruction loss: 13783.2 \tLatent loss: 3224.1\n",
      "10 Train total loss: 16550.5 \tReconstruction loss: 13333.8 \tLatent loss: 3216.75\n",
      "11 Train total loss: 16248.7 \tReconstruction loss: 13009.1 \tLatent loss: 3239.6\n",
      "12 Train total loss: 16346.3 \tReconstruction loss: 13150.0 \tLatent loss: 3196.26\n",
      "13 Train total loss: 16067.2 \tReconstruction loss: 12777.2 \tLatent loss: 3290.02\n",
      "14 Train total loss: 16512.1 \tReconstruction loss: 13058.1 \tLatent loss: 3454.07\n",
      "15 Train total loss: 16099.5 \tReconstruction loss: 12739.1 \tLatent loss: 3360.35\n",
      "16 Train total loss: 20827.6 \tReconstruction loss: 16602.6 \tLatent loss: 4224.96\n",
      "17 Train total loss: 38965.4 \tReconstruction loss: 24849.1 \tLatent loss: 14116.2\n",
      "18 Train total loss: 29396.9 \tReconstruction loss: 24286.1 \tLatent loss: 5110.81\n",
      "19 Train total loss: 27910.6 \tReconstruction loss: 21005.3 \tLatent loss: 6905.23\n",
      "20 Train total loss: 26797.9 \tReconstruction loss: 20202.2 \tLatent loss: 6595.64\n",
      "21 Train total loss: 18686.1 \tReconstruction loss: 15251.4 \tLatent loss: 3434.69\n",
      "22 Train total loss: 17034.8 \tReconstruction loss: 13890.0 \tLatent loss: 3144.77\n",
      "23 Train total loss: 16404.0 \tReconstruction loss: 13102.6 \tLatent loss: 3301.37\n",
      "24 Train total loss: 16214.5 \tReconstruction loss: 12803.4 \tLatent loss: 3411.13\n",
      "25 Train total loss: 16253.4 \tReconstruction loss: 12823.9 \tLatent loss: 3429.48\n",
      "26 Train total loss: 16326.2 \tReconstruction loss: 12934.0 \tLatent loss: 3392.18\n",
      "27 Train total loss: 16161.3 \tReconstruction loss: 12767.4 \tLatent loss: 3393.91\n",
      "28 Train total loss: 16990.3 \tReconstruction loss: 13471.8 \tLatent loss: 3518.54\n",
      "29 Train total loss: 15728.4 \tReconstruction loss: 12465.1 \tLatent loss: 3263.28\n",
      "30 Train total loss: 16505.3 \tReconstruction loss: 13219.9 \tLatent loss: 3285.37\n",
      "31 Train total loss: 16961.6 \tReconstruction loss: 13379.0 \tLatent loss: 3582.55\n",
      "32 Train total loss: 17671.7 \tReconstruction loss: 14372.1 \tLatent loss: 3299.55\n",
      "33 Train total loss: 16640.7 \tReconstruction loss: 13332.3 \tLatent loss: 3308.39\n",
      "34 Train total loss: 21943.6 \tReconstruction loss: 15878.3 \tLatent loss: 6065.31\n",
      "35 Train total loss: 15656.0 \tReconstruction loss: 12254.1 \tLatent loss: 3401.86\n",
      "36 Train total loss: 15697.0 \tReconstruction loss: 12231.0 \tLatent loss: 3465.93\n",
      "37 Train total loss: 15769.4 \tReconstruction loss: 12409.7 \tLatent loss: 3359.68\n",
      "38 Train total loss: 17182.6 \tReconstruction loss: 13943.9 \tLatent loss: 3238.67\n",
      "39 Train total loss: 18285.6 \tReconstruction loss: 14796.2 \tLatent loss: 3489.34\n",
      "40 Train total loss: 20053.2 \tReconstruction loss: 14899.0 \tLatent loss: 5154.25\n",
      "41 Train total loss: 16290.2 \tReconstruction loss: 13008.1 \tLatent loss: 3282.09\n",
      "42 Train total loss: 27364.1 \tReconstruction loss: 22713.0 \tLatent loss: 4651.08\n",
      "43 Train total loss: 15450.8 \tReconstruction loss: 12009.0 \tLatent loss: 3441.87\n",
      "44 Train total loss: 15567.6 \tReconstruction loss: 12068.6 \tLatent loss: 3499.0\n",
      "45 Train total loss: 15348.8 \tReconstruction loss: 11840.9 \tLatent loss: 3507.95\n",
      "46 Train total loss: 15435.6 \tReconstruction loss: 11949.6 \tLatent loss: 3486.03\n",
      "47 Train total loss: 15210.5 \tReconstruction loss: 11804.4 \tLatent loss: 3406.18\n",
      "48 Train total loss: 20627.8 \tReconstruction loss: 16485.7 \tLatent loss: 4142.07\n",
      "49 Train total loss: 15147.4 \tReconstruction loss: 11587.1 \tLatent loss: 3560.29\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 50\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss], feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train total loss:\", loss_val, \"\\tReconstruction loss:\", reconstruction_loss_val, \"\\tLatent loss:\", latent_loss_val)\n",
    "        saver.save(sess, \"./my_model_variational.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 500\n",
    "n_hidden2 = 500\n",
    "n_hidden3 = 20  # codings\n",
    "n_hidden4 = n_hidden2\n",
    "n_hidden5 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "learning_rate = 0.001\n",
    "\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "my_dense_layer = partial(\n",
    "    tf.layers.dense,\n",
    "    activation=tf.nn.elu,\n",
    "    kernel_initializer=initializer)\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_inputs])\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3_mean = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "hidden3_gamma = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "noise = tf.random_normal(tf.shape(hidden3_gamma), dtype=tf.float32)\n",
    "hidden3 = hidden3_mean + tf.exp(0.5 * hidden3_gamma) * noise\n",
    "hidden4 = my_dense_layer(hidden3, n_hidden4)\n",
    "hidden5 = my_dense_layer(hidden4, n_hidden5)\n",
    "logits = my_dense_layer(hidden5, n_outputs, activation=None)\n",
    "outputs = tf.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_sum(xentropy)\n",
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)\n",
    "loss = reconstruction_loss + latent_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's train the model and generate a few random digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train total loss: 17792.6 \tReconstruction loss: 14122.9 \tLatent loss: 3669.64\n",
      "1 Train total loss: 17332.2 \tReconstruction loss: 13560.0 \tLatent loss: 3772.24\n",
      "2 Train total loss: 16350.7 \tReconstruction loss: 12579.3 \tLatent loss: 3771.48\n",
      "3 Train total loss: 16581.4 \tReconstruction loss: 12810.6 \tLatent loss: 3770.78\n",
      "4 Train total loss: 16223.9 \tReconstruction loss: 12450.0 \tLatent loss: 3773.86\n",
      "5 Train total loss: 15628.1 \tReconstruction loss: 11819.6 \tLatent loss: 3808.51\n",
      "6 Train total loss: 16080.9 \tReconstruction loss: 12179.7 \tLatent loss: 3901.24\n",
      "7 Train total loss: 15772.8 \tReconstruction loss: 12021.3 \tLatent loss: 3751.55\n",
      "8 Train total loss: 16276.5 \tReconstruction loss: 12404.6 \tLatent loss: 3871.83\n",
      "9 Train total loss: 15589.6 \tReconstruction loss: 11740.6 \tLatent loss: 3849.02\n",
      "10 Train total loss: 15931.3 \tReconstruction loss: 12031.4 \tLatent loss: 3899.94\n",
      "11 Train total loss: 16112.7 \tReconstruction loss: 12238.3 \tLatent loss: 3874.35\n",
      "12 Train total loss: 16002.0 \tReconstruction loss: 12185.1 \tLatent loss: 3816.83\n",
      "13 Train total loss: 15357.7 \tReconstruction loss: 11667.4 \tLatent loss: 3690.35\n",
      "14 Train total loss: 16208.4 \tReconstruction loss: 12264.4 \tLatent loss: 3943.96\n",
      "15 Train total loss: 15970.0 \tReconstruction loss: 12158.5 \tLatent loss: 3811.52\n",
      "16 Train total loss: 15551.6 \tReconstruction loss: 11783.1 \tLatent loss: 3768.49\n",
      "17 Train total loss: 15330.0 \tReconstruction loss: 11555.7 \tLatent loss: 3774.3\n",
      "18 Train total loss: 15251.3 \tReconstruction loss: 11584.5 \tLatent loss: 3666.81\n",
      "19 Train total loss: 15196.0 \tReconstruction loss: 11516.6 \tLatent loss: 3679.44\n",
      "20 Train total loss: 15323.9 \tReconstruction loss: 11525.9 \tLatent loss: 3797.99\n",
      "21 Train total loss: 15358.7 \tReconstruction loss: 11515.6 \tLatent loss: 3843.17\n",
      "22 Train total loss: 15297.9 \tReconstruction loss: 11582.5 \tLatent loss: 3715.37\n",
      "23 Train total loss: 14673.0 \tReconstruction loss: 10940.7 \tLatent loss: 3732.34\n",
      "24 Train total loss: 15293.5 \tReconstruction loss: 11561.7 \tLatent loss: 3731.75\n",
      "25 Train total loss: 15256.3 \tReconstruction loss: 11540.8 \tLatent loss: 3715.53\n",
      "26 Train total loss: 15305.4 \tReconstruction loss: 11475.4 \tLatent loss: 3830.01\n",
      "27 Train total loss: 15276.9 \tReconstruction loss: 11449.7 \tLatent loss: 3827.24\n",
      "28 Train total loss: 14980.6 \tReconstruction loss: 11318.0 \tLatent loss: 3662.56\n",
      "29 Train total loss: 15232.8 \tReconstruction loss: 11520.1 \tLatent loss: 3712.69\n",
      "30 Train total loss: 14872.4 \tReconstruction loss: 11172.9 \tLatent loss: 3699.47\n",
      "31 Train total loss: 14890.3 \tReconstruction loss: 11144.1 \tLatent loss: 3746.17\n",
      "32 Train total loss: 15246.7 \tReconstruction loss: 11439.3 \tLatent loss: 3807.4\n",
      "33 Train total loss: 15063.5 \tReconstruction loss: 11282.1 \tLatent loss: 3781.41\n",
      "34 Train total loss: 15046.7 \tReconstruction loss: 11310.2 \tLatent loss: 3736.47\n",
      "35 Train total loss: 15293.9 \tReconstruction loss: 11599.5 \tLatent loss: 3694.4\n",
      "36 Train total loss: 15134.5 \tReconstruction loss: 11362.8 \tLatent loss: 3771.74\n",
      "37 Train total loss: 14705.7 \tReconstruction loss: 11054.7 \tLatent loss: 3650.98\n",
      "38 Train total loss: 14913.9 \tReconstruction loss: 11077.0 \tLatent loss: 3836.93\n",
      "39 Train total loss: 14848.1 \tReconstruction loss: 11198.5 \tLatent loss: 3649.57\n",
      "40 Train total loss: 14694.2 \tReconstruction loss: 10991.5 \tLatent loss: 3702.73\n",
      "41 Train total loss: 15223.9 \tReconstruction loss: 11465.1 \tLatent loss: 3758.8\n",
      "42 Train total loss: 14585.3 \tReconstruction loss: 11019.3 \tLatent loss: 3566.01\n",
      "43 Train total loss: 14579.1 \tReconstruction loss: 10931.2 \tLatent loss: 3647.84\n",
      "44 Train total loss: 15049.1 \tReconstruction loss: 11381.9 \tLatent loss: 3667.18\n",
      "45 Train total loss: 14855.6 \tReconstruction loss: 11125.6 \tLatent loss: 3730.04\n",
      "46 Train total loss: 14777.7 \tReconstruction loss: 11093.4 \tLatent loss: 3684.3\n",
      "47 Train total loss: 14408.9 \tReconstruction loss: 10788.5 \tLatent loss: 3620.39\n",
      "48 Train total loss: 14479.2 \tReconstruction loss: 10864.3 \tLatent loss: 3614.88\n",
      "49 Train total loss: 14637.6 \tReconstruction loss: 10926.0 \tLatent loss: 3711.55\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "n_digits = 60\n",
    "n_epochs = 50\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\") # not shown in the book\n",
    "            sys.stdout.flush()                                          # not shown\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss], feed_dict={X: X_batch}) # not shown\n",
    "        print(\"\\r{}\".format(epoch), \"Train total loss:\", loss_val, \"\\tReconstruction loss:\", reconstruction_loss_val, \"\\tLatent loss:\", latent_loss_val)  # not shown\n",
    "        saver.save(sess, \"./my_model_variational.ckpt\")  # not shown\n",
    "    \n",
    "    codings_rnd = np.random.normal(size=[n_digits, n_hidden3])\n",
    "    outputs_val = outputs.eval(feed_dict={hidden3: codings_rnd})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAE0CAYAAAD9kEDWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXdgldXd+D935WaRnRACIYGEGRBkuxC1Yi0ILmyte1Wr\nb6tV39YuV20rrXu8llbrqKPyFoE6gFYrgmKBsiFCWJIASUhC9k3uep7fH897vtwbhozcgb/z+YeR\ncc9znnPOd3+PzTRNNBqNRqPRxBZ7rAeg0Wg0Go1GC2SNRqPRaOICLZA1Go1Go4kDtEDWaDQajSYO\n0AJZo9FoNJo4QAtkjUaj0WjiAC2QNRqNRqOJA7RA1mg0Go0mDtACWaPRaDSaOEALZI1Go9Fo4gAt\nkDUajUajiQO0QNZoNBqNJg5wRvGz4vkWC9sRvnYyjvtkHDPE77hPxjGDXh/R5Os01yfjmOHkHbeg\nLWSNRqPRaOKAaFrIGo3m/wi99tRmsxEIBNiwYQMAvXv3JiMjA4CEhISYjE+j0Vh0dHQA1j4FsNst\nOzYSe1MLZI0mRiih3NLSwn//93/z6quvApCVlcWcOXMAOO200+QgiEeCwSAAgUBAxulyuWI+ZjW3\nhxvHoe6Bj/WYQzFNE8MwAOjs7KS1tZWkpCQAEhMTRRjE05i/Tqj10drayrPPPgvAl19+yaWXXsqk\nSZPke7p7/rXLWqPRaDSaOEBbyFHANE3a29t58803AZg7dy6tra0A9O/fn759+3L66acDkJqaSllZ\nGQAZGRnY7faIacF+v5+qqioA3nvvPebMmUNmZiZgWW2XXHIJY8eOBWDkyJG43W7g5NDKTdM8yC18\nqL/HCpvNJuNISkriiiuuYNOmTQAUFBSQnZ0dy+EdFYZh8MwzzwDw7LPPkp6eDsArr7zCiBEjYjm0\nr3zHyrLv7OzE4XAA4HA4Ymrdq/VqGAZNTU188MEHADz++OPU1dWRmJgIwG9+8xumT58OIP93smGa\nJl6vF4A9e/bw73//G4ALLriA7OzsmO9R5aZ+9dVXWbp0KQCZmZlkZmbKeonEGE86gezz+WhoaAAs\n196Jum4i4Xbois/n4+GHH+aNN94ALPdejx49AMjNzaW6uprnnnsOgN27dzNkyBAA7r77bkaNGoXL\n5YrIuDo6Onj99dcBWLp0KatXryYlJQWAU045hQ8//FDGNXnyZB599FEA+Z54Qbn3Qg/ZPXv24PF4\nAEvxyM7Opri4GACn0xnzDR+Kw+GgZ8+e5OXlAdbGV8pPPGMYhrhVwVrngMS/Y8mRXNamacqBW1VV\nRUtLC2DNe//+/XE6nYf92UiiPs9ms7F9+3Z++ctfArBv3z58Pp/ELp988klZy5E8H7oT0zRlfVRU\nVDBz5kzmz58PgMfjkWcvKirizTffZNy4cUBslGe/389HH30EwHPPPSdGS1FRUcTHdFIJ5La2Nl55\n5RX++c9/AnD55ZczY8YMANxu9zFNlDpI1CKPBEpA/OpXv+JPf/qTLMjk5GRyc3MBSE9Pp6Ghgf37\n9wOwfft2tm/fDsD69ev5z3/+E7EDIiUlhRtuuAGAs88+m8TERAYPHgxYQquiooL/+q//AmDOnDlc\ndtllAEycODGi83a0qEPX7/fT0dFBY2MjAPv37+fdd99l9erVAOzcuZMLLriAe+65B4CePXvGZsCH\nwWazUV9fT3t7O2ApacpTEU+KQ1dsNpuM0+PxUFBQABATZSJUsQ71jBxK4Q6Nz3788cd8/PHHAJSV\nlfHjH/845gLOZrPRp08f+vTpAyBng7LM6urqeO211wDIzs6mpKQkLteJaZpyBtbU1PDQQw8B8Le/\n/Y22tjb5Wuj7qqys5Dvf+Q5bt24FkLMvWgSDQT7//HNRhtrb2xk+fDhgWe+GYcg5HgnFPvanqkaj\n0Wg0mpPDQg4EAgA8/fTT/OIXv6CwsBCA0aNHs2fPHgD69OmDy+U6asstGhZeXV0dAH/4wx9obW0l\nOTkZsCz7O++8E4C8vDyam5tZtmwZAD/84Q/FhVZTU8MHH3wglmm3a2N2u1g1ubm5OJ1O0cIBhg8f\nzoMPPgjAzTffLC6m008/PS7KcZSGvWXLFv7+97/z2WefAbBr1y5KS0vFYt6yZQvbt29n5MiRAHz3\nu9+NzYCPgM1mY9euXQB4vd6TwmUNyFrt7OyUMIyKJUcTm80mVq/NZhOrS/0Zut9DLbd169axatUq\nwLLy1f/HEpvNRl5eHo8//jgAL7zwArW1tdTW1gLW+lBetJUrV1JYWBgzr0TXf6uz2jAMmpubefvt\ntwH47W9/S319PXBg36p34nA4wn6uoaGBiooKAIYOHRr5B+GAx3TNmjVceumlkuNz7rnncuuttwJW\niHTv3r0if5KSkrr9TD4pBHJNTQ0Av/71r8nPzxchMXbsWJmQzs7OsASoQ7mpFNFw7xiGwU9/+lMA\nGhsbSUhI4K677gLg3nvvlRIGh8NBenq6xA+ffPJJcbV6PB527dolyQ/qZ7oLm80mm8LpdB6kpNjt\ndolV9enTR1yqgUAgLgRyZ2cnAPPnz+fTTz+VA+vMM89k8uTJsm62bt1KbW2tCOh4JPQwy8vLC1OM\nTgacTqckosV67KFx7WAwGJZAp/6u1npdXZ0cvh0dHTF3VyucTqfkkuTk5NDa2sqAAQMAK1avlJ6s\nrKxDlnBFApUo6ff7AescUGvWNE22bNnCp59+CsDatWuprKwUQ0N9H1jro0+fPtx2222AZVjdcsst\ngJVDEwgE5OxURkCkn0udHddddx0NDQ2iCDz22GOUlpYC1rxv3bo1outbu6w1Go1Go4kD4t5CNk2T\nf/zjH4ClwT766KNcfPHFgJXyr7RbODjrs6ubShENC7mqqkqyqk3TpLi4WJKKUlNTDxqDSl7IycmR\nrzmdTjIyMqLmjvL5fAel9O/btw+A6urqmFs+XVHJFXl5efTs2ZNzzz0XgKlTp5KXl8f69esBKzHD\nNE3y8/NjNtYjYZom5eXlskaj5abrDpqbmwHL6uluD86xEmoFq3PgcNajysDfvHmzuN19Pl/cWMhw\nwCOWk5NDY2OjJNBt3ryZpqYm+Vq0Ep9M06Sqqoo//OEPgBUaUu7bdevW8cUXX8h5AVayZWgYQc3t\n2LFjefTRRxkzZgxgnXPKEp48eTItLS1RdcF7vV6uvfZaAMrLyxk6dCh//etfARg0aFCYaz3SCXQn\nhUBWceKcnBwmTpwoLysYDIpbslevXjgcDjmkQ1+ocmVHKxPR5/Nx3nnniWsnKSmJ++6777A1g4Zh\niEsnJydH/t9ms9GvX7+Ijjv0EPN4PFIOkpWVhWEY7NixA7CUH7X54sFdDUhMPiEhgbq6OhFkWVlZ\ntLe389JLLwFWtx2bzUZlZWXMxnokOjs7ee6558QFP2LEiDDFMl4JBAIsWbIEsJQeJZxjRWiWdegh\nqv5PfY9hGHJu7N69W+b6rLPOiovqAYUa+6WXXspLL73EmjVrAKtut3///oC1L6N1rrW3t3PjjTdK\nzN3lcskYfT4fHR0dEh92OBzY7XY5h9PS0qRi4+abbyYjIyPsHBk0aBBgua8//PBDKW2NNIZhsHXr\nVqmDHjZsGM8++6xUm4QaIep5QpWM7uakEMjq5SgtRm38qqoqERi9evXC7/eLFqYWBESvdERt+rlz\n57Jz5075/6KiIsaPHx9W+K+w2WwEg0ER3l0Vh4yMjKiM32az4ff7xXIA67BSiUZFRUUS84mXQ0vN\n59atW2lra+Pzzz+Xf+/YsYPly5cDlgbudrtF2Yg35s2bx+7du6WZhko+g+jUyR8vfr+f3bt3A/HV\neCU0qSsYDGIYhpwLhmHQ0dHByy+/DFheN2WJXnXVVTEfeyhqnxUUFLBlyxbWrVsHhCdBORyOiI9Z\nzeXMmTNZs2aNCN3CwkIxgHJzc2lvbyctLQ2wzov9+/fLOKdNm8bVV18NWAZKV++l+r7i4mLcbjd9\n+/aN6DMpOjo6ePrpp2V9TJ8+nZEjRx7WGxgqV3TrTI1Go9FovqbEvYUcDAaZMmUKYGlWq1evllZm\nTU1NfOtb3wIs96XL5RJNJxZWnHI5PvLIIxiGIWOYOnUqaWlpoll6PB6x+nv06EF6erp8r9/vF+0s\nKytLusNEGrvdTnp6Onv37gWsUqF58+bx4YcfAjBgwABKSkqA2FtAYGmnKmt69erVrFmzRsaelpbG\niBEjpKA/KSmJmpoayUW455574qLloLI8lixZQnJysow31MKLZwzDCIsZhsYE44VAIMD+/fvJysqS\n/1u0aJHkdxiGIRUORUVFcbG2u6LGpKzInj17MmrUKMDqLhatMQ8fPpwBAwbIWVVcXCyZ4DNmzMDt\ndsu+CgaDfPrpp+JBmTZtmvxcMBjEbreLp9Dn80nb2M2bN5OZmcmPfvSjiD6LOotXrlzJunXrGDZs\nmIwzMTHxkG7p0G6A6nccrqrneImfnXMYbDabbPSdO3cye/ZsPvnkEwCuueYaSeRxOp3HVIfc3Zim\nKR1/tmzZAiDum+9///vk5OSE3d6iDrKOjg6Sk5MlOW3lypXyO0tLS6W2Mxq4XC45uBYvXsw777wj\nQm/48OHi2osXN2qo6yghIUHc/iUlJUyaNIlzzjkHgPr6eq655hqZ24ULFzJ9+vSYP4Pa3Hv27Alr\n7bl3714pa3E4HHHX6lPR3t4uSqjT6WT8+PFAfChsCqWkq8StjRs38sADD0hSlGmaUk4E1qF7qPEf\nqowy0s+pzovKykrcbrf0Ixg3bpwob9E4H9Q4SktLueiii6TTXVlZmZQEZWVlhc2Hz+cLO9dqa2vF\nWEpKSsLr9cr637BhA88//zxgPeugQYMYOHBgRJ9JhT0/++wzkpOTRSA7nU5qamrkbHG73bJ2tm3b\nFpaoNmzYMDGYuisZULusNRqNRqOJA+LeQgbEvTt//nyWL1/O5MmTAbjiiiukFABiq5l7vV7JIgwE\nAjidTundWlxcHDa2/Px86WUNlga6ePFi4EDfWrAyDqOZ0RzaAL61tZXExESZ38suu0y0QNUgINaW\nkMqyLikpwe/3i7Z+9913U1RUJOPNz8/nl7/8Jddddx0Af/zjH5k6dWrMXavV1dWA1TgmLS2Nbdu2\nAVanK5XRnpycjN1uj3qC4tGwfft2aVrjcDjE9RsPhN7NnJ2dLWfIpk2bqKmpkZCAy+USa6ytrY2W\nlhaxgDIyMiR8lJCQEJUEKkVoMmtNTQ333HOPNF5xOp1itUUr4ROsTOiqqirp7te/f385H7reSpeY\nmMjo0aPFW/jee+/JGr7qqqtISEgQN/Wjjz7K2rVrAcviv/rqqyN67hmGwcaNGwHLki8tLRULubGx\nkTfffDPMy6nO5Pb2djwej1j948aN4ze/+Q2A7NcTJa4FsmmaeDwefvGLXwDWS+3fvz8//OEPAcuF\nolxPsWo1qDbvu+++KweszWajtLSU733ve8DB8ezQTkGqjZ+Kb7a3t4ugUVnl0cJms8kB5PF4SEpK\nknkNXXDxIpDVWPPz85kwYQKXXHIJYAmx0LHZ7XYuuugiKa3YvHkz9fX1R1WXHKnnNE2TFStWAFYc\nsL29XVp/pqWlHZSJH2+Ypskf//hHEWxZWVlxJZAVNpsNp9MpoZgzzzyTfv360dbWBljK8nnnnQdY\n66SxsVGUjF27dskzuVwu8vPzD3m7XGNjY1i54oliGAaNjY0S1iouLiY3N1dcvA0NDSKQv4rQa0iP\nt/Qz9Pxavny5XHoxdOjQIyqKdrtdqjYWLVpE7969AUuQ9+rVS66X3Lhxo4zxkksu4aKLLoromvd6\nvbLX+vbty0UXXSSu/zVr1rBy5UrpYZCbm8vo0aMBSwDn5+dL97ElS5ZIV8XevXt3S7g07gXyli1b\nREO54447+NnPfiZCoqmpSXqexur+WJVo8f7774f1aB0/fvxRa3mdnZ1SBwcHhF+ka5APhZrHb3zj\nG2zatIl+/foBVsJLqEXZNfEoFkJDHUp+v5/p06eLInOojZGQkCC1hUuXLqW1tfWoBHKknisQCEh9\nfVlZGbNnzxYh4fV6JTmmax1tJMd0LAQCAT777DNZE2PHjpWciXgkdM727dsnYz3//PPlzu/U1FRS\nU1NlT7e2tspBHXr9IVhrTL2T7hDGoVdCrl27lq1bt4rHJzc3F9M0ReGfM2eOCDeVX3M0nOi6MU2T\nhQsXcsYZZwDhc9D1pi2lUKo69YSEBFFuVqxYQV1dHZs3bwastaQs7YsuuuiQjZO6AzW2xsZGyTcZ\nNmwYffv2FSWsqqqKHj16SLLwlVdeyVlnnQUcuHZWKXBTp06VpNdvfetb3SKQdQxZo9FoNJo4IO4t\n5PLycrF8fvazn4WVD5WXl0s3plmzZsVkjKG3xiit0OFwSCedr8Lv9/PUU0+JtuhyuTj//POB6Lvh\nbTabWJ2LFy+mtLRUmr737t1bnk+VdIVqsdGy4NTn+Hw+ab5yyimnHNV92OqOVa/XG9bsPhYEg0G5\nDWzJkiXs3r1bvBNqvcOBkEa8xZAbGxvDmmpMmjQpbhrGdEXdPATwk5/8hLq6OvGOTJ8+XdzZLpcr\nbH579OgRtt7sdrtYVn6/v1tK59Seam5uZsGCBYDVWCgtLU2a8nz00UeUl5dLdUlnZyeTJk0C4Dvf\n+c4Rf393dij0eDyUl5dLDPW///u/xVJXn6Gex+/3U15eLt4Dm80mjU3q6+tpaGiQn8nIyODHP/4x\nAGeccUZE2pcahiEVAZs2bZKLcvx+P36/X2LY1dXVXHHFFZx99tmA1ZQl9D560zQl/ycrK0vOkf8v\nyp5M02TTpk1MnDgROOAyUK6dxx57TFzWXTdTtFBJUA0NDbJ5nU4niYmJ8rK6uq5D3VMvvfQSTz75\npAj2vn37yuKMdl9d0zTZsGEDYC3MCy+8MCyG1vUWnUN1rIl0i1I1hh07dsgBdskllxykIHSlubmZ\nL7/8Uv4da4Hs8XikDGvjxo0EAgFxT19wwQXy97a2NhITE+OmXama/zfeeCMsjhmtevljRXXmeuGF\nFwCrZt1ms0lXtDPOOOOwim/oelLnS2iv967u2mPFNE0Rbu+8845cVQhWXoHqPLdy5UoaGxvD9p9q\no1ldXU3v3r3lrOiOcR2OmpqasLajixcvZsaMGcCB2nN1jrW2tjJnzhy++OILwEpWVaHHQCCAy+WS\nNXPHHXdIwmWkWoEahiH7f968eWFtdCdNmiRfGzFiBOecc46ce6Edu1Q8XtVWB4NBKQHrLuJaIBuG\nQc+ePSXGo/rQqozk7du3SwF5LA6s0JfT2toqG0FZDSqhQW0SpZVt2bKFm266CYAvv/wSv98vsdp5\n8+Z1+0s+FtTG9vl85OXliRUQmunbtXG/3++Xr0Wyb3ioIvPGG2/IJkpLSzvixReGYfDWW2+J8jRu\n3DhpchIr3G63CIKOjo4w67K2tlaSRex2O4MHD6ZXr14xGWdXlIW4fPlyDMOQeU9JSYmrZiaha7O8\nvJy5c+fK1woLC/nZz34GfPWdtur3dL10pTvWt2EY0uTo73//u9Se9+vXjz59+shnbN68mdbW1rB2\nmcrae/XVV7n++usl0eqrFNMToaioiB49esgefPfddykrKwOsOHdNTY0I69mzZ7No0aKweu/Qu6nz\n8/OZPXs2YDUdimSjHvXZKkdj9+7dcp/0vn37cLlcoqD17t2blJQUUSy6JuB++eWXUj1TX1/fbdnV\nivj0MWk0Go1G8/8ZcW8hJyQkSLZhMBiko6NDuroMHjyYm2++GYhdbE3F+/x+v2iA7e3tfPzxxxJf\nMQyD1atX8/777wOWZqW0TIfDQe/evSUGPnjw4Jg9i91ulw5A/fv3F+1Wocal2t4pjbOxsVG0yJ49\ne0a0vlfV6r7zzjsyVqfTeUjrLNRK+uKLL8QCufvuu2PeOjMhIUGyVT/55BM8Ho/M77Jly8Ku3hsx\nYkS3t+g7HkIvp1fXRaos5OHDh8dVDFm9+6amJv7whz+I5ZaYmMi1114rrSePZT670/NjmiZNTU3M\nmTMHsNzBKpa9Zs0aTNOUOGZJSUlYtUloJ7cvvvgiLP8gMTHxIGu+u3C73aSnp8u+/+STT8TS9Hg8\ntLW1iRequbmZQCAQ5j5Xe+7mm2/mvvvuE09gNK519fv94kVISUkRD4NpmjQ3N8t5XFFRQXV1tYx7\n8ODB8l6ampq48847KS8vB6zqiO7uThfXAtlmszFw4EBqa2sBK8lixYoVEud86623Yhpbs9lskrSg\nbjsBK2noo48+EneU3W6ns7MzbHEq1/CQIUN49tlnGTdunHwtVthsNik/mDBhAnPmzJEDd+zYsTJm\nr9dLRUWF9AMeOnQoU6dOBSLrMgu9Fcbj8UgcbePGjfTs2TNsLYQKj9WrV7N3714pZZg4cWLMhYfT\n6ZS66SVLlrBy5UpRZNra2uQqySlTpoT1Oo8lpmlKzkZnZyc2m01cdikpKXGTcBZKR0cH27dvlwN3\n1KhR3H777cesNHad/2AweMKCxOFwkJqaKr9PlTUVFxdjt9tlPZ999tnMmDFDxtDW1iYKxpo1a+jR\no4cIa5fLJWdSd+fVuFwuJkyYwLvvvgtYirhqXnK4cIUac2lpKY8//jhg3Xkc7Zyf0Ksir7nmGjGW\nioqKGD58uHyto6ODbdu2Sbgx9Maqzs5OUlNTOfPMMwGrNFSFvrprf8Z+l2s0Go1Go8EWxUSMY/4g\n5RZVGXFffPEFzz33nKSd/+UvfxHX3glyJFXtiONW2tM999zDn/70JwCxhkMTneBAsld+fj7f/va3\nAbj11lvp3bv38WqLh/uh436paj3s3buXu+66i6qqKsByYavnWLBgAYmJiZKJ+NRTT0lSRFpa2ldp\ni8c916FZqTfccINclD5y5EjOOussSf7Lz8/H4/GIx6K8vJy9e/dy1VVXHfQsR8lxj/lIqLluaWlh\nxYoV4oEYNGiQNK5wOp0kJCTExfowDEM6yt1www20t7dLid6bb77ZXWV63TLXKvGwrq6O++67j//8\n5z8AXHzxxdx///0nXMFwiA5uxzzXfr9fMu3Xrl0r93cPHTo0rJ1jdnY2brc7zKpXruGqqqqwG59C\nwwipqanHuxcPO+YtW7ZwwQUXANalKF2rFdQ43G432dnZ0q3wzjvvlDV9gpbxca2PUDlnGEbYv7uW\nbHZ2dsrZoaocALnARq3z5ORk+ftRhDOO6qHjWiCD5crZsWMHYB38dXV1/PrXvwas2Eo3uQpOSEiA\nFTN55JFHACsDODRztqioiCuuuIJp06YBlkBQL/IEx9/tAll+wf9lhav2jrNmzZIShoyMDG6++eaw\nWj3l8jmK5zmhAzc0hvk///M/gOVmSklJkbK4wYMHS7tBsFyDffr0kQPhOOY8IgI57JdEpkVnt64P\n1TkPrDhgdna25HOcgFLZlW4VyDU1NcyaNUvyIe69996wDOZu5LjmWmXzdo23HuqGr0PV/SvXufp3\nV+HyFW71Yx6zYRjyzp9++mlR2E3TJDk5WdrTXnnlldx4442yJ7sxThzxvXjQLz0KGXkU6+nrIZAD\ngYDECl977TV69+4tvaxDGyicIFF/yd1ExATyIX9pSFu80HKAY+SE5lqNobOzU66G3LNnDytWrJBD\nt6SkhIEDB8p1homJiWJpHid6fUSPbp1rlXegBN/RNJA5Tr5Oc31URojf75fmJb169Qqz4iMYH/66\n7cUwdAxZo9FoNJo4IO4t5NCWZwsXLqS2tlZKnbqxk9XXTes6GccM8Tvuk3HMoNdHNPk6zfXJOGY4\necd94JviXSDDwT78KMZ/4OR8ySfjmCF+x30yjhn0+ogmX6e5PhnHDCfvuAXtstZoNBqNJg6I68Yg\ninhsOKDRaDQaTXcSTZe1RqPRaDSaw6Bd1hqNRqPRxAFaIGs0Go1GEwdogazRaDQaTRygBbJGo9Fo\nNHGAFsgajUaj0cQBWiBrNBqNRhMHaIGs0Wg0Gk0cEM3GIPFc8Px1a8d2Mo4Z4nfcJ+OYQa+PaPJ1\nmuuIjlndFqeuyLTb7cfS/Onrtj7C0BayRqPRaDRxQNy0zlTaEhz9BfJdL+PWaDQaTfwQekZ3dHQA\nUFlZyT//+U/5/9tvvx2HwxH1sR0K0zRjKkviRiCrF3c0L0YJ7/379+Pz+QDIzMzE7XYftTDXaDQa\nTXTw+XzU19cD8OSTT9LR0UFRUREAtbW19OrVC4idYaXkj/ozVuPQ0kuj0Wg0mjggLixk0zSP2mVh\nGAYNDQ0ALF26lAEDBgCQkpKC0+kUzUa7sDUajaZ7OdxlRIc7b5U3c9++ffz6178GrHP7jDPO4JZb\nbgEgPz8/pue1aZoyTsMwcDgcMRtPXAjkr3r4UHdCZ2cnW7ZsAWDnzp18+eWXAEybNk1cIPGOaZoE\ng0EAAoEAAE6n9SpiuRhOVmId99GcPJimGSZUuoa4Qr9us9lisq66jvFYxmIYBrW1tQB8/vnnJCYm\ncv755wPgcrlOeGyhY/iqfWeaJh6PB4C//e1vLF26VH7HjTfeSH5+PnD0OUORwDRNvF4vu3btAiz3\neWlpKT179gSOLoR6uN+r3uGxPF9cCORQQh/EMAyCwaAkA9TU1LBt2zZWrFgBQEtLC8XFxQC0t7fH\nbAMdCtM0RdgahkFrayvz5s0D4OWXX6aqqgqAYDBIRkYGkydPBuD+++8nIyMDiK2Vb5omfr9fFurb\nb7+N2+0GoLi4mMmTJ5OcnAyEKxHRGnMwGKSzsxOAqqoqfD6feEtOxlwCwzDkeSorK2UeU1NTsdls\ncjCkp6eTmJgY03GG/hn63o/23Ud7XZumSXNzMwDr16+nubmZkSNHApCXlyfKcDAYxOPx0NTUBEDP\nnj1JSkqK6jjBirdWVlbS3t4OQElJCUlJSbIGDjV/6mc3bdrEnXfeCUBFRQVXXHEFF1xwQUTG+1Xv\nMRgM8q9//QuAefPmkZWVBcAPfvADRo0a1S0KwvGi5qu+vp67775bzmbDMBg+fDivvPIKAAMGDDhu\noXw86/zkOrU0Go1Go/maEjcWcqhVrGLEX375JcuXL2f79u2A5XJpa2sTi9npdJKamgpAVlZWzC1K\nZTW0tbV5yOFIAAAgAElEQVRRXl7OqlWrANi+fTurVq2SLMP9+/fT2toKgNfrpba2VtxM/fr147/+\n67+A2FjIfr8fgD/+8Y+8+OKL7Nu3D7CsTmUtpKam8te//pXrr78egEmTJsl7iDRqnbS0tHDXXXcB\nMHfuXEzTZMSIEQBcf/31TJ48mYKCAuBAOCCeCLWI6+rq+Oyzz2S9tLa2ypirq6vZvXu3PMvMmTMZ\nNGgQEN31oTw+e/bsAaw4YN++fRk+fDhg5XCEjsfn88k+dblcsj6U50J9bzTCDcFgUMpsnnrqKSZO\nnMiECRNkPGrfVlZW8sgjj8gzqX0YDUKt+CeffJIlS5YwbNgwAH74wx9SXFx82Axg0zSl2uTxxx9n\n+fLlACQkJHDhhRceNu4bSUzTpLGxkffeew+Abdu2MW3aNMAKLypvWywwTVPW5syZM5k7d6641h0O\nBx6Ph02bNgFQVFQkHqmjWaehcux4LOu4OKlM0xQBtWLFCl577TXAcieYpsnZZ58NWJve7Xazbds2\nwNro6mu5ubkxdVP6fD55iffffz9bt26lX79+AJSVlTF9+nQRuh6PRwTdkiVLaGxsFPfU22+/za23\n3goQ9UXr9/vFvbV06VJKS0u57777ADj99NNFWNfV1bFw4UKWLFkCWC61wYMHA5EXEmrBb9y4kTlz\n5gBWuKJHjx54vV4AXnvtNR588EEJZ7zyyiuUlJTEPJyhxu7xePjf//1fFi9eLP+fnp5OWloaAKNG\njRL33uLFi1m7dq0oqVVVVSKQoznmjo4OXnvtNWbNmgVYB9e1115LYWEhYAk2tT727dvH7Nmzqa6u\nBmDgwIHccccdwIE1Hc3ykvr6elnHKSkp3H777WRmZsrnq3EvXLiQRYsWcdtttwGWQIsWhmHwySef\nAPDqq68SCAQYN24cYCU9HSm3xDRNPv30UwBmz54dVgpaVFQUM8V+w4YN8kx9+vTht7/9LUBMQy5g\nzfWyZcsAeOutt0QYgzW2hIQEFixYAEBGRoYob0lJSQfF0EP/brPZwvKdjmeNa5e1RqPRaDRxQMwt\nZMMw8Hq9/OpXvwLgww8/FEtnzJgxjBkzhtGjRwOWttLc3Mz+/fsBGDx4MLm5uYClzcZCE1SJWzNn\nzuQPf/gDAJ2dnUyYMIGf//znAJx66qk4HA7RxG02m1gPzzzzDG+99ZYkklRXV4srM5oWcjAY5Oqr\nr+bjjz8GYMKECcyfP18stVDXXmtrK0uXLhUrP5rJGcrV9Mwzz0imel5eHr/4xS/45je/CcAXX3zB\nT37yE1auXCnP8vjjjzN9+nTAcrmHupOisW6CwSB79+4FrKS+RYsWSRLaddddR3FxMenp6YBlxam5\nTkpK4vXXXxdrTVn90UJ5ru6//35ee+01GdfZZ5/NsGHDSElJke/dsWMHAL/97W9ZuXKluN3dbrdY\nbeo5orVXDcPgL3/5C3l5eYA19wUFBWHeNDW21atXk5KSwimnnBK1MSorqq6ujp/+9KcANDQ0MGzY\nMG688UYAkpOTD+v9U67um266CbCeRe3Hm266ieLi4qh2wQp9npdfflnc8DfffLOs71g3/6ivr+ep\np54CoKmpCZvNJlb7qFGjGDhwoJy98+bNk6qeIUOG0NjYKOdKRkaGeBRLS0vDLP/jrZaJmUAOdd9t\n2rSJd955B4Dm5mYuuugiAG688UZKSkpkctra2li+fDl9+/YFYPz48fTo0QOITeq8YRjycp5++mla\nWloAGDduHC+88AK9e/cGDqTOq8PINE369OkDwBVXXMGSJUtoa2sDLJdJNONrivLycmbPni0H1/vv\nv09mZuYhP7+trY2lS5dy5plnAtC7d++oHV7r168HLFeuWgcLFiyguLhYxlBSUsKpp54qSt6bb77J\nPffcI27L4uJinnrqKYYMGQJYQk8JDxUrVe/sRA6z0PK29evX88YbbwBW9uukSZP4wQ9+AFj5D10z\n1ZXgq6mpob6+Xkr6CgsLo7YmfD4fTz/9NAB/+tOf8Pl8Mmff+c53GD58uAjkQCAgytzSpUtpbm4W\nF3xxcbEorl3XdKTWuDpfdu3axZNPPslvfvMb4OCsWVVKCbBmzRrKysqi6lJVxsdPfvITdu7cCVgK\nzK233kpJSQlw5DUYDAa5/PLLpWrDbrfTv39/AH70ox9FPZNZzfvq1atZsGCBjP3GG2+Mes5D189T\ne3Hbtm2Ul5cDlms9KSlJSsNuvfVWcnNzWbt2LWCFFB9//HEAGhsbRcEAK0yqjMNf/vKXhyyhg2NT\nQGJuIXu9XpqamiRe0tnZyTnnnANYG7lHjx5yOHm9XhoaGmTycnJyYtoD1ePxyKHa2NgoG/mRRx6h\nT58+ByWwhKIEQEFBAfn5+dTU1ABWqYWKaahEmEguZLVoHnvsMRISEpg9ezbAIYWxinNfd9111NbW\n8v3vfx84OLYSKQKBAM8//zxgHTw33HADYK2T0M1gs9koLCzk2WefBaw5fv7558ULsXHjRr7//e9z\n2WWXAZZSFBoL3bx5swiQUaNGnfCYwcqNUIfmyJEj+fGPf3zE96sOjwceeIBgMMh3v/tdIDpeE7Um\namtr+ctf/gJYe6+goICHH34YgHPPPRe32y1jb29vZ+7cuYC1FwBRIr7xjW+I4O76rJFaN+rMeOGF\nFzjttNPkXR/qvFC5EGvXruWJJ56ImnJvmqaUcM6fP1/WSv/+/bn88suPeLYpb9stt9zC4sWL5XnT\n09NFiUpLS4u6EFTKzSuvvIJhGOIlzM7Ojto44NCJb2pP7d69W/4/ISGBvn37ct111wGWN81ms4mh\nt379eurq6gDLEDFNU87uQYMGMWPGDODQXozjSabTMWSNRqPRaOKAmFnISoNxOBxkZmZy+eWXA5aL\nThXtK/eh0jSCwSDnnnuuuM2O5I4J1Yjsdnu3a72GYbBp0yaJm5mmycCBAwFLy/qqOz7VMzU2NoY1\nfsjIyAhzn0Zaw1Wa9s6dOykqKhJPRdfP9fl8EqNds2YNjz76qMQzo6WFezwe/v3vfwPWux86dOgR\nP1/N44QJE3j11VfFiujRowdut1vc806nU+YhKSmJQYMGdVuGrRrDwIEDJVN62rRpB5UJhWKapngC\ntm7dSmpqKvfeey8QvXg3WN2VVJ6A6vikwhQJCQlheQUVFRVSZWCaJikpKRKnGzJkyGH3aqTWuHIF\nt7W1MWXKlLBYdyiGYfDII48A1po6/fTTu30sh8M0TfFAeDweORPcbvcRM3g9Ho+0nXz77bcxTVPW\n689+9jN5R9GO1Xq9XqmQWbNmDaeffrqMM9YVDqFnbHFxsZS2rV+/nuLiYlnz6vuUVbxr166w3B+X\nyyXzO2vWLAkPdId1DHHgsk5NTWXQoEESD9y5c6e45dxud1i9Znp6Orm5uUfc3Grytm3bJqUAkyZN\norS0NCw2e6IC2jRNtm3bFtb6UrnSv+owD60b3LNnT5iLNCkpKayWLdLuMzVfTU1NjB07NqxmV7WV\nAytGsnr1agBuu+027rjjjqjH7VtbW6WL2fbt20UYHOpQN01TXOyLFi0iOztb1lhhYSHjxo3jtNNO\nA6x1pbqO2e32bqtbDj0ExowZI27x1NTUI77bhoYG7r//fnmOu+6667ACJRKoNbFixQpJoktOTsbh\ncFBZWQlYLsi2tjZx//385z+XUItqjaiUO5fLFfUDWe0vn8/HiBEjDvv5+/btE0XiRz/6UdRLndTZ\nFtrHYMuWLdx+++1SB52SkkJzczOff/45AM8//7y8B8MwcDqdfOtb3wKsRC51fkZzzv1+Px988AFP\nPPEEYIUwRowYIWG8eGhvq/bi8OHDJQSUmpoaVkrrcDjYsGGDtPn897//LWvJZrNRVFQk/Q/69et3\n2D18vM+qXdYajUaj0cQBMbeQHQ4HKSkpor1kZmYe5K5RFrGyKpR7QX0dLDdbS0sLixYtAuChhx4S\nF2FOTg7z5s2jtLQU6J4yHbvdTmpqqlhswWBQuhgFAoFDatqhlq/K1lu2bFlYokD//v3FQoHIa7lq\nLnv27Mlpp50mFrHNZqOzs1MyDP/85z9Lgfzvfve7mCTT5eTkyDvcs2ePdF+aPHlyWGKZYRhUVFRw\n6aWXApanZcqUKeKmcjqdDB8+XN5djx49jpiAdyKo35eSkiIuclW6p8pAlAWprNHf/e534jEpKiri\npz/9aVStC/VZzc3NYZeeVFdX89ZbbwGWJ6u8vFzWscfjkZ/r1asX9913X0wsNYX6zISEBNatWyfv\nXu19ZfU8+OCD8jM//vGPozpWu90uiZH/+Mc/xE3q8Xh4/fXXefPNNwHLa6a6FMKBREGwnjMvL4+r\nrroKQDw90cbv9/OnP/1JLvtxOp1UV1eL1yQhIUG8AXa7nYSEhLASuGjMu/qMxMRELrzwQsDKlJ43\nbx7/+Mc/AHjppZfYs2dPmOdCrZkePXowZcoUzjrrLHmOo+FYvAMxF8jKrac2fltbm2zyzs5OMjMz\npRZyx44d5OTkyGXWdrtdspO9Xi/JyckSxz3nnHPCumHdcMMNfPDBBwDSpUd9/vGOe/To0WHKgnIp\n7dy5k5KSkrBm8KEx7Y6ODmkHWllZGVZm097eflDj/kiiNsWMGTMoKCgQ99327dv54IMPWLhwIWCV\ni6i4ZqyawjudTk499VQAli9fLpvd4/GQkJAgB9Xq1auZNm2aKGRnnHEGZ511lszxvn37SE1NFYH4\nVfH+7kApnmBt0KqqKrm4o7CwEK/XK9m+W7duFff2okWLot7ZSM3FoEGDWLNmDQB9+/Zl0qRJonSu\nX7+ehoaGMAVOrSVVOnKkGPmhPq87UcpAUVER69atCyvRCwaDPPDAAwC8/vrr0tJRrYdoYbfbxa3/\nzjvviBt19+7dYZ2eOjo6wmLMcGDOUlNTGThwoGS2x6JNJljPkpiYKGeXMpxU1cbGjRvlbAG48sor\nOe+88wBL0Y5mSMZut8vnjRs3DqfTKblAu3btEmUNrH2rejEMHTpUWmvCV2exh3btOmkEMoRv5qys\nLLl5Y/fu3fTu3VsshwULFlBWVib1vePHj2fs2LGAdaglJCTIJPz+97+XGuH9+/fj9Xqlzi8jI6Nb\n4p+9evUSzbu6ulrqkFetWkVBQUHYQRoIBCTus3LlSlkA9fX1dHR0yAHS0tISVrMZadS8X3rppWzb\ntk2EwhtvvEFTU5MIwEsvvVSUnViiDrAPPvhADqGtW7cyaNAgeb/PPPMMgLTzfPzxx+nTp48knKi+\nwGoNHGotRKK1o1I609PTxeIEK1YbGrfasGGDrKucnJyw8XT3mA6FUlzOOussaWZy5plnMn78eGmS\nUFNTw4YNG6QOs7GxUZ5v5MiRYXPadR0f6frD7kKt6xtuuIEFCxbIOPfv38+aNWt4//33AcLq1WOB\nmrMzzjhDlPTdu3fz/PPPS5nc4sWLxbgASyFWvc2nTJlCVlaWtFP1er1Rb74ClsdB1faC9VzJycly\njeG6devkfBw0aBCdnZ289NJLgOUVvPbaa4Ho9ZNQn5OQkEBeXp6UOblcrjAPbGJiogjvqqoqAoGA\ntEDOycmR93ekcR/LM+kYskaj0Wg0cUBcWMhwQJtLT08X18bHH38s2ZxwwC2iLpQ466yzxP2sOh0p\n7dvlcon7RLXXPNJ9oseD3W6XwvcVK1aIBnj//ffz8ccfS3mWw+GgoqIirGRHuccqKiro6OgQLWrv\n3r1i+RUUFEQ8O1H97uTkZLKysuRGp1tuuYURI0aI1jtkyJCY3zFst9vlfScnJ4vL94knnqBXr14S\ne3c4HFx88cVScjFkyBCqqqrCGgJMmjTpiPfKRhKbzUZSUpJkfefk5FBQUCCXTXR0dEgjBZvNRiAQ\nkLlXWbXqa5FA7ZNTTz1Vmqn079+fzMxM8ZIkJiYyePBgadL/6aefikdo0KBBB3XjOtJNRZF4DjVf\nOTk5XHrppRITrKmpoby8nLKyMsBq86mszUiO52hQ77W4uJiZM2eKB0WVYqm9efPNN3P33XcD1j5o\na2uT2HHoGRhtEhIS5MwNBoMUFRVJCevgwYPlfOzfvz/r16+XtdPQ0MAll1wCRCdsEAgExAreuXMn\n8+fPlwoSdUuT2gNpaWnioe3o6CA1NVVCqOpZIdw9rTie8zLuBHKPHj3EXffhhx+yY8cO8vPzATjv\nvPO46qqrxG0ZGndQm175/7dt2yYJKC6Xi7Kysm6/Jcdms8mCu/vuu3nssccAa9PPmTNHXEder1de\nKliCVgld1f1FbaiysjJx80RDAKoF5HA4yMjIkI5GGRkZ+Hw+6Xa1atWqqNZoHgqbzSYtR/v168eG\nDRsAyw3Zp08fOWT79+/P4MGDxWXtcDjYt29fWMz2cB2v1IEW6e5oNptNDuDU1FRSUlIkPut2uyWG\nrMajNr9hGBG/KUn93vz8fBlHVVUVFRUVEiKaMGECxcXFokQEg0FZ78r9pzAMQ5Ql1XM+mq5JNb9g\nCbDzzjtPXNh9+vQJK5E63h7E3Y3NZhOBVVNTg81mk0SkmTNnhq3fjIwMETCh6yOaJCYmhgnTjo4O\n5s+fL2fG0KFDJYnS5XLR0NAgCWDt7e1UVFQAMHr06G5ZG4faw2oNbt++Xdzlc+bMob29XZSdgQMH\nsm/fPokTqzUB1lpyOBySQxMMBmWuD3Wt6PEQNwJZ4XK5pM6roqKChQsXilWUlZWFz+dj69atAGFx\n2v3797Nq1So++ugjwLI0lWV9yy23cNlll8mkd/d4wbo7VVnic+fOpbm5WZKK2tvbaW9vl0YW+fn5\nEu/s7OwkOTlZvnbxxReHNayINKH9k0PvPLbb7TQ3N8vdqmlpaWEWaCyw2WxysI4ZM0Y2tNvtJj8/\nX+atsLCQc845R5QcwzDo6OgQzdY0Tdxud7dnVh+LIA89dAKBAH/7298kQXHIkCF873vfA6yDrmvv\n5WgJjMTERBHAixYtIj8/XxIqk5KS2LVrl2QGBwKBwyafhSof6t/RJDSL1+12U1ZWJmvD5XKJQK6t\nraVv374x9wSBlah4++23A9bc9u7dm5dffhk4uN991yTLWIzf5XLxk5/8RBo8+Xw+ueQFrEoI1aeh\nR48e/POf/xSjxOv1snnzZoCwd3M8HE4QGoYhHrLf//73kkPQ0tJCRkaGeKuysrJEboDlYVGKZlpa\nGhMnThSD6UgXGuk6ZI1Go9FoTmLizkK22WxyS8xPf/pTCgoKxNVYX1/Piy++KKUWEydOFIuptbWV\nuXPnSqxo6tSp4uIZM2ZMxNPqExMTufLKKwG44IILaGpqEmtTxeGUG+Szzz4Li93m5eWJV2DUqFFR\nsYwVahxOpzPs1iPDMDAMQ6z++vp6mfdYXjCuPnvGjBlSNtHZ2YnL5aK+vh6wLOTc3NyDLpxQLrPh\nw4eTlJTU7ZbE0WrFXd1aDQ0NLF68WCydW2+9VS5m6OqNiHY9snJDXnrppWHlecFgkMzMzLDSMUVb\nWxs9e/YMa4/b9V3EkoKCAlnnTqdTLKDOzs6YxV9DMQyDyy+/XMo/U1JSeOWVVw55GUnXkIbdbo/J\nM9hsNqZMmcKsWbMAq767ra1NwkrV1dVieY4bNw6HwyGhyJKSEonjn+jZdySLVXnQiouL5fMcDgdj\nxoyR3Ijly5dTWFgoN22dcsop8nOVlZUMGjRI5FPXdd2Vk/K2p0OhHnLIkCE8+OCDUiJUV1fHhg0b\nwtLOVVw4ISGBb37zmyJAcnNzRSBGy8WqDtTs7GzZPGDVPXd2doqrPTc3V2rbMjIyuPjii+UO1mgK\n41BUPXjoInK73UycOBGwanvj4bAKrb8Mrev1+/1hm6yrAPD7/RLbHDly5BE3SbQEhor7rVy5kp07\nd0orz4svvjgu3KZwYC5cLhculyssZpaZmSm3YZWXl8v8tre3R6XW+FgITbpxOp2yV0PH5XA4CAaD\nMauzV3z88cf861//EiFbVlbGmDFjjjiHkezbf7TY7Xa5Nemaa66htbVVrjFcu3athDtOPfVUXC6X\nlFQWFxdLT+hInX8qkRLg3nvv5bbbbgMsQ6Ouro5169YB1rnSq1cvOffy8vLkPaxfv56srCxZ51+1\npo9HIMfHrtdoNBqN5v9z4tpCdrlcOJ1Ohg0bBlgah7qvEsKTG7q2X4tluz7VeUwldQUCAXbt2hWW\n5q+s9rPPPpuxY8fGtM2g+tzQtqSBQIDm5uYwzVu50JTrN5aEvu+ul4UcKlM2MzNT3FChndpihWma\n0nRj1qxZpKSkyI1OqampMbcoD0foGs/MzGTy5MmA1QJWWUBerzcuvCmhdL2wpWtZlvpaIBAI65QX\nzfeg9torr7wiiYcAd95551feOR7NSzGOROj6yMjIYNKkSYB1zoV6KUpKSsKSRKPRnVCdEUlJSRL6\nysrKoqioSMoMU1NTyc7Olq+HhgDGjBkTVhJ1tJ93LMSlQFaoDRH6YLF2Jx0tdrtdsgVra2vDMlLT\n09OlJ/PYsWOPeBVfNAkdQ3V1Nb/61a9YsGABYLUgVK6aeLi5pSuh66JrjNY0TZKSkjj33HMByy0W\n6/H7fD7pELV69WqmTp3K6NGjgdhkyR4rKqShsrCvvPJKESjxEosNJTTGGqqwhWYpt7S00NraKoJQ\nfW+0UDkaTqeTgoICKe88/fTTj7gmYr2Wj4ZDKTdqbqPZha7r5zgcDhITExkwYID8f9eQlxpfSkrK\nQbe0dXcJYlwL5JMV9VJVzCI5OZl+/fqJhRmaVJCZmRmzMqIjsWvXLioqKmTxnX/++UcsR4h0bexX\nEaokhFo8YHkk3n77bWlDOWrUqMPWIUcD0zT54osvpKQsIyODO++8M6bJcseDzWYTb8O1114rddRe\nr5dAIBDTUqeuhDZWgXAhoPaf3++nrq5O3kM0PUGhF8zccccd9O3bV5pl5OTkhDX8ONRcRupylGgQ\n6zHbbLZDKvRd/608KxG14iP2mzUajUaj0Rw12kKOAMrNruI62dnZEqsAwrI8Q+Pg8UDoJd4Oh0OK\n4L/5zW/GRWbykT6/qwWhGj5s2LABj8cjjQhiHW/z+/288MILMtdTp06lpKQk5nN4rIRaFmlpaWJ9\ntrS0hHWMirULPtR70tXKCW3r2dnZyfLlyyUDN5rYbAcu2Bk9ejSnnnrqMVm98ehlOxlQ5Z1HO382\nmy0s/NHde9YWxVhPfAWVwjnSrHb7uLsxBnu4X9ItYw699QS6bdNHbK67CmT176amJn7+859LW9Cz\nzz77WMsrunXMgUCAiooKKY3r3bt3pA7UiK6Pg37p/8231+vF5XId7zNFdS92I1Gd627i6zRmiNK4\njzM8d1TfrF3WGo1Go9HEAdpCtoi51nWcaA33KDlBr4ReH9FDz3X0+DqNGY5j3MFgMFru/qM6fKIp\nkDUajUaj0RwG7bLWaDQajSYO0AJZo9FoNJo4QAtkjUaj0WjiAC2QNRqNRqOJA7RA1mg0Go0mDtAC\nWaPRaDSaOEALZI1Go9Fo4gAtkDUajUajiQOieblEPHcg0d2BosfJONcn45hBr49o8nWa65NxzHDy\njlvQFrJGo9FoNHGAvn4xiqg2pYFAQPqnxvpqOo3mcPj9flmfDoeD0Da7XW/UOtmujtRo4hEtkCOA\nOqTUn4ZhYLPZaGhoAODzzz+X7x03bhwZGRkkJiYC+mDTxB61bmtra+WKyNTU1DAB7ff7aWhooL6+\nHrDumB4yZAjAsV5tqTkB1NlyMpwb8aq8hV48041X4x4X2jzTaDQajSYO+Fpdv3gC2k23JAr4/X7A\nuqB937598m+fz0dpaSmGYcg4y8vLAXjqqacwTZOHHnoIgNLS0mNxY+ukjOgQkTGrvef3+/F4POJB\n2bFjB3V1dQCMHTuW7Oxs0tLSgGO2Pk9ofQSDwbB/G4YRZj1v2LCBzz77DIChQ4dy2WWXAYi35ziJ\n6FyrPaj2WDdaQ1Hbi8FgkPb2dgBqamooLCw8Xg9bRMdsmiadnZ0AtLS0kJycDEBycvKJXHl4XOuj\nq2xQ68Dr9VJeXs67774LwLnnnsvw4cNlv9nt9u5aI0f1S05q35JpmuzcuROA22+/ndTUVJ588kkA\nCgsLoz6WxsZGAB599FH279/PAw88AMCAAQMOErLDhw8HrA30ySefMHfuXADuvffeKI76+DFNMyym\nGA+x8NDxGIYhm87n8+FyuUSYxctYvV4vABs3buSFF14QJQ2gra1N/u71ern66qsB+P73v09OTg4Q\nedef3W6XObTZbNjtdpnjnJwcsrOz5XsrKyvp6OgATlggdyvBYJCdO3fy9NNPA7B//34ATj/9dACu\nu+66sHVhs9lEEXE4HPK1eHCzqnHV1tbyP//zPwAMGzaMvn37xsX4QlHCeMWKFQC8//775OXlATBj\nxgwKCwujug9D58c0TXw+H2Apv8888wwLFy4E4MUXX2TChAncddddAIwZMwa3233Q74gUJ6VAVodC\nZWUlkyZNAmDPnj0UFhaSnp4eszH9/e9/B2D+/Pmcc845ohQcauGpl5yWlkZzczNffPEFEJmX3tUq\nOBZCE9GU1bZt2zbWr1/Pl19+CVgHxe233w5A//79I3rhd9f4PFgWZkdHhxy2q1at4t1332XVqlUA\nVFdXk5WVxS9+8QsArrzySpn/WGGaJvv27QNg3rx5rFy5Uixkt9stB8b+/fvxer389re/BaCjo4Nf\n//rXQHQOCLV2wFo/ag25XC4GDhwo+2/Lli0y5nhAjXvDhg2ce+65tLS0yNcSExPxeDyApRhnZGQA\n0NnZye7du9m9ezcAubm5TJs2TX4mShfZHxLTNGltbQXg5ptvZvny5QDcdNNNXHzxxTEbV1eUkvne\ne+/x0EMPsWPHDsBSilU+wr/+9S+eeeYZiouLgejnHJimKcrjqlWrWLFihawHwzBYunSp7MVvf/vb\nTJ8+HbCUUKfTGdF9F3tTQaPRaDQaTXxYyIeLYyu3aGgGXDAYJBAIADB79mzJ8jQMg5ycHNHCok0w\nGCtVLbQAACAASURBVKSqqgqAXr16cddddx1R81PPVFFRQUdHR1SsC8MwvtJKVpaFcvmq+Z01axZL\nly4FLKsjEAjIM9hsNrGen3nmGTIzMyOiRYa6mhoaGli5ciVgWWe7du0Sl++XX35JR0cHzc3NgGX5\nNDU1MXPmTACmTp1KQkJCTN18ofG1jRs30tTUREJCAgAFBQUkJSUBlkutqqpKxrpr164T8ngcC4Zh\nyF4DyyoOfeeJiYkUFRUB8Pe//11CNnl5eTGf282bNwNw4YUX0tTUJGdMQkICo0aN4tZbbwXglFNO\nkbFWVlby1ltvsX79esDy9px66qkAlJSUdGc88ZgJBoP8/ve/B2DBggUy71dffbWsm1jT2trKNddc\nA1hj9Pl8cgampqbicrkAWLduHU899RQ/+tGPACgqKoq6law8ZBkZGQwdOlQs5Pb2dlwul3j/nnji\nCd577z0AJk+ezPXXX0+PHj2AyHioYi6QldBV8RG/3y8HlXIrKDe0cheozdXc3Cw/Z7fb+cEPfhCz\n+KDNZuOcc84BrJfar1+/I36/Oug2bdqEYRiyWCPxkrvGTw73GaFxYcMwaG9v5+233wYst29mZiYA\n+fn51NfXS5zT5/NJKdfWrVsZPXp0t20wNZ5gMEhnZ6e49v/3f/+XBQsWAFa81efzyZw6nU68Xm+Y\n8ABEYaqqqgqLf8YC0zTFxb5nzx5sNhsFBQWApdANHToUgIEDBzJ//nyZ65ycnIgLBbWnOjo6xK2e\nnJxMenq6HP6maWK320VB2rhxo7yPAQMGxMS9q9aKx+ORXIz6+npM05TxXHjhhbzwwgtkZWUBlpKh\nfi4zM5OdO3eyZ88e+dmNGzcClnCOJfX19bIXi4qKRDiXlpbGXPkB68ybMmWKJPmZpkleXh4XXHAB\nYJV3qvW+ZcsWTNNk7dq1AGRnZ8sZH41nsdlscj6NGDGCXr16hYVi3G63PFdjY6M80+rVq1m7di1P\nPPEEYIUbu1veaJe1RqPRaDRxQMwtZJXVqLTy3bt3s2bNGsByS5aVlYnbSLmMlMuupqZGfi4xMZGL\nLrooBk9wAJXWP2rUqK+0EFTZQmdnJ3a7nTPOOCNi4zqWovdQjc/tdjNjxgzA8lwoC/O1117jo48+\noqmpSX5GaZzdnSwVqoFv3ryZF198EYAPPvhA3ExpaWlkZWXJGLxeL263W7J9Gxsbqa+vF8/LY489\nxssvvxxTV59hGPzzn/8ELO9DR0cHe/fuBaCsrIz8/HzAeraEhARZTx6PJ6JWRKi3qrq6WsIA+fn5\nDBw4MCzr2GazyTvIzs6W57nhhhskUSqaKA/JAw88wCeffAJY1r7D4eD8888H4PXXXyc5OTlsnas1\nlpSUxL59+2SduFwuWSORTuY5EsFgkNdff11cpY8++ijjx48HrEzwWFrIynPz0EMP8Z///EfGMnXq\nVO69915KSkoA64xQyXJjx44N61bY2Ngoe9Xtdkf8eUK9rK2trTQ0NIhMCbWO1bjVWd3c3Mxf//pX\nCYvOnDlTQkvdRcwFMliLSk1IQ0ODxHBcLhfZ2dniFktMTMQwDHF9rF27ViavT58+IhBjgc1mk7rj\nsWPHfqXLVmXx2e123G43ZWVl8nviAXXgKld6U1MTs2bNAmDZsmXs3btXDu6cnByuvfZaoHvdlV03\nRkJCgmwOwzDE7VxYWEheXp7E1err68nIyJAQwj/+8Q/+/Oc/i/BYuHAhzc3N5Obmdss4jxXDMFi/\nfj0vvfQSYB1IhmHIfNbV1ZGSkgJYilBovD405tndqJCFWsfbt28XIVtaWkp+fr4IMrXXlCt3+vTp\nvPnmm4CVRTtlypSoZrKbpilZ9bNmzZJ3bbfbGTt2LK+88oqMu6ubUc1nS0uLuLgBevTowWmnnRb2\nPbGgurqap556inPPPReA8ePHh9Udx6q7lN/vl7ySBQsWkJiYKIrPs88+S1pamozL4/HQq1cvwMqR\n8Pl8otAHg0H5e3p6Oi6XS86QSOWiKHmzbt06VqxYITkwTqeTtLQ0caF3ze3xer0sWbJEnr+7OyzG\nhUAO9en369dPNnlVVRXNzc2iWSUnJ7Nr1y6WLVsGWMk7aiIKCgoOmxwWDYLBoBxSKvHscIvKNE1Z\nyIZh4HQ6Revq7s2lWut9FV3rig3DoLKyUiyNxx9/XEoYlOBQQuPuu+/mBz/4AWApTd0x/q7vsqsF\nXlxcLPW4o0aNIjMzkzFjxsj3DB06VH6mrKyMDz74QGrWvV4vn3/+uXhUon2Ytbe38+CDD8ohoNb/\ngAEDAKsWXf196dKlYTXJkYwZKu+Tyt2orKyktrYWsCyJzMxMscwKCwvDFLbKykpWr14NwF133cXc\nuXO57bbbACtOp/ZGpHI8AoEAzz33HIAIY7CSdv785z/LWjnU56vD+d1338Xn88n3lJWVxayMMnRc\n999/P3V1ddLXICkpKSz5MtqE5nWonI7CwkKmTZvGTTfdBEBWVhbV1dVs374dsPIiVJmT2+0OU6i9\nXi/vv/8+YCn0w4YNk/MwUmtdKZ3Lli2jrq5OyrUCgQBOp1OU+/Hjx0t+zPbt2zFNU4Rw18Y53YGO\nIWs0Go1GEwfEhYUMBzTXlJQU0ab37duHaZpiEbe1teHxeKS8IlRDT0tLo62tTWI+0S5RCG30sGPH\nDqZOnSpxQGUpK42qvb1dWrV5vV5KS0ulVduhbtQ5UY70e0IbbYS6Taurq3n99df54x//CFhuVPW9\ndrudoqIiyWS98cYbu72bTdff43Q6CQaDMoaCggKZ39LSUjIzMznllFMAaw2Fus2Liop4+OGHpdTF\n7/czZ84cvvWtb8nvjgZqbjds2MCGDRvkWbKzs3nwwQe56qqrAKs0R1k+7e3tdHR0yDpX2deRIjS+\n5vP5JJ4aCARoaGiQNe52u2lra5PSs+eee07CMKZp8u677/Kvf/0LsKzUK664ArCs/6SkpG63lP1+\nP1u2bJHPV7//4YcflpKlw6GsoyeffBLTNOX8ueWWW+TnlAcpmlUcqmxv7ty5XHbZZVLl4PP5xJXq\ndrvDXLzRJBgMMnLkSABGjhwZ5lHw+/00NzfTp08fwLKglUdNeQVDf4/yaqxYsYKsrCwGDhwIRO6y\nB5VvsGvXLrxeb1hHusTERK6//noATjvtNObPnw9YJZ0+n0+8V5HIK4gbgayw2+3iJtuxYwdbtmwR\nF/bQoUM588wzxdW3du1aOTBGjBhBbW2tbK6cnBwRztG4DaW9vV3ibXv27BGhDFZsasWKFZK0s2bN\nGnGf2u12cnNz5aALbYPXHYvxqw6QriVR6hB49dVXeeedd2Tjhya4DB06lL/85S9S2hXJhJdQ4TBv\n3jxqamoAK6at3EptbW2cddZZcpB2PZzsdjvnnXeeCDWfz8fmzZvl2UI7UEUStVaXLFmCz+eThJA7\n77yTa6+9NuyQUuPZvn07Xq9XFDbVfjCSqDnfv3+/dLgaMWIEPXv2lDXw6aefMnv2bFGWGxsb5eec\nTidut1t+trm5mZdffhmwSou+973vdasCp+Leas4SExNlbX73u989YuKTYRgyNlVzr+a4f//+skZM\n08TlckW0PLErShlub2+nsLBQ9mZLS4v83el0Mnr06JgIZKfTybhx4wBrz4XuoUAggN1ulz0ZqmSq\nuQt1t6swyZ49e8LCZ5GaZ+V2Puecc1i2bJkIaJXTo2ROUlKSnNWGYZCYmMjgwYPlmbob7bLWaDQa\njSYOiDsL2TAMsYDtdjs9e/aU7MKysjJ69eolWuu6devEshw0aBAvvviiaDMTJkzg29/+NgC9e/eO\nWGcmpeWtXr2aOXPmAAdcZqq3dVNTE+3t7fK9DodDkhZSUlLIzMwUl00wGAxraBHNDErDMKioqAAs\nrTwtLU365yYkJEj52e23306/fv2i2nh/5cqVLFu2TN4vHLDkBg0aRM+ePY9oJXg8HtHglWtehT56\n9ux5UBORSKASSWpqasjNzeXyyy8HLAv5cG7zN954A9M0xd0XjcxlNY81NTViBXi9XhwOB7t27QKs\nrNpPPvlEst7hQAOfK664gqlTp0oji5UrV0pi2n/+8x8uv/xyCTd013w3NzeLm7OlpYUJEyYAlqUW\n2m1MrQEVPli1ahWPPfYYYHkwHA6HZPz+5je/YeLEiYBVOTF06FAZb6TLoAzD4M9//jNgzavdbqey\nshKAvf+PvfOOj7JKF/93StqkJySE3jsC0kFRQAQLgqgf9dpQL/a6ruWn67qufS2rq95Vr+zqqqwu\nKIiiKEqzAFKlhRIIJZCQRvok097398d7z+NMBETJTCZ6vv+YmJA573mfc87TT2EhW7duBayLJtq1\na0fHjh2ByCYoOp3OELkNztB3u93s2bNHKmJOO+006e3vdDpDLipxu91s3rwZsDxCU6dOpVevXmF7\nHpvNJuto+vTpJCYmyt5dXFxMXFyclN6uXbtWkrp8Ph85OTnSvz0cXomoOZCDY6fBXVQGDRokiysj\nIwOHwyHdczZu3CiLp7q6mrVr18oBkpeXx+mnnw4g6fZH+9wTeelKAD/99FNx0TmdThITE2VsdXV1\nIW4Yh8MhLsj+/fvTo0cPyUA80lgidbG36q4D1sZfWVkpylFWVpY0WZ8wYULE6h/VZvraa6+xfv16\nkY34+HhZ4AMHDgzpuHSkv5GbmxtSwlBaWip11ZmZmSHutnApbko2O3XqxHPPPScL+0iHsXKfqkoC\n1ZIwEu0yg+PEKi7cs2dP2rRpE1KqEhMTI+PJzs7m4YcfBn5wE6uSoeeee47PP/8csGS/pqZGDuSm\nHLeqfQ5u3blixQpM0wzJvK6srOSbb74BrJafwT+z2+3yntavXy/uSXXblTrIles6XJimKTkRDoeD\nAQMGSIx8//79Mp+BQICSkhJZC81VnqUy9JXiNWfOHDZs2CC10wMGDKBdu3byu2o+wbqsRil6SUlJ\ndOrUKSI3mYF1plx33XVMnz4dgE2bNvHSSy9JCV9ZWZmsxbi4OE477TQpUQ3HWoy6A9nn83HqqacC\nltbbsWNH0bwdDgeFhYU8+uijgKV5q9T5qqoq6uvr5UC5/PLLRaCPVWx+Ii/eMAx27doFQG5uriRd\nZGVl0blzZ1nYSUlJFBYWyvc2m01iXKWlpZx00klSExusOTbFGFUa/0+hEuRUzGfPnj3k5eXJxjN1\n6lQR2qYqbToe1IGwa9cu3G63aLYTJkyQEhzl/Wh8E1SwBv7uu+9KnEq1BVX1hO3btxe5aeqNNjgG\nrnIIhg0bxpAhQ476WYZhSEmWKqdTfX8j0TRBbY6FhYVSLtShQwd69uwpm1CnTp2w2+1S9nLRRRdJ\n4pYqa1FWfceOHUNagTb11Xs2m42UlBRR3oItrsWLF4fcTd64XCh4rTkcDhwOh6zFMWPGSDvTzMzM\nkBac4cZms4nlrnIc1q5dC8C3334ra7pr165kZWVFPIH1SP/P5/MxZ84cwEqAyszMFC9Qt27dxNui\n5lApyF999ZUYM4MGDYpoL/TgvAOwlPsJEybItZHBzaeUDIXzOkYdQ9ZoNBqNJgqIOgvZ6/WKRbxt\n2zbsdru4ojweD6+++qo0JXc4HPTv3x+wNPjgdPWePXuKJhMO14JpmpSXl4tGeOjQIWl/OXDgQIYO\nHSqalYoRK41w27Zt0pLwwIEDJCYmys8at2ILtvzU9z+Hn1vSo7qkzZ07F8MwxFvx9NNPN3lXmp9D\ncnIyiYmJIgsXX3yxuI6Uxa6sHtXcQX2/ZcsWli5dGnJLksvlkkzV4Azdpka9u+LiYukkNWLEiCO+\nF/W7O3fuFGsILFlWIY5wE2wFFhUViQvSbrcTHx8vVk7nzp157rnnxIWt7ooFRNaDWw6qnImzzz67\nydsNgmW1jB49GoBVq1ZJfkBRUVFIuCg40x4sWVYWcZ8+fejRo4e0i23fvr3IfHp6Ok6nU+Ym3Lkd\nweOqq6ujqKgopKxLeRwmTpwYttvVjkTwDWDqRjiw5vPAgQOSN1BTU8OAAQOYMGECYOXKqDGqi2JU\nWdyCBQtkXs8///ywt7Q9UrMk9X1MTAwjR46U+d27d2+INyU/Pz+s+SZRcyCrh0tISJDFU1ZWRmpq\nKkVFRYA1IQkJCQwcOBCw4hLq64yMDHr06CG9UyPREzU/P1+C/5WVlRKLGj58OL1795bPj4uLw+fz\nifvV5XJJgobauIJdYY0PhxM9LI43Bu31ennkkUcAS/lJTEzkqaeeAiLrpg5GhST69etHUVGRhCFO\nOumkH5U5NXalqYX00UcfUV1dLeOPj4+nV69ekpuQmZkZtkWmxrR69WpJ8ouNjaVfv34hndwMw5CY\n9sSJE0WZczqd/PWvf41Y/avdbmfIkCGAFXpRyUMLFy5k2rRpUleanJxMcnKyHBoej0fGvHHjRlau\nXCnrtrCwkD59+gBWAl445MjhcEi898ILLxTF0jRN0tLSRI527NjB8uXLQ9yPqpb2+uuvZ8iQIfJM\nDodDDu7Y2NiQRK5I3UoElrL/xRdfiBu1Q4cOEh4bOXKkhAYigdfrlduwqqur5XCura1l3bp1cpgO\nHz6cm2++mdatWwPWXq4SvPLy8vjiiy/47rvvAEvm1ME9fPjwsMl6sPKg1p7KhQkOcwVf8RuM6t2u\nfhYOpSxqDmT1ElRQH6y4ZUpKiiwQm81Gz549pRY1ISFBFlZ+fj7JyckRqxNUL1BZbOPGjZPWjT16\n9CAlJUU2KI/Hg9/vF8vA4XDI15WVlRQUFIil0bjXblO2ofypBiGLFy+WjcxmszF27FhOOumkJhvH\nL0FZXXfddRf9+vUT2cjJyflRa1L1vZo/tXDy8/Px+XyyWXTq1IkrrrhCNgFlBYUDVRf/6aefyh29\nS5YsYfz48SGJTe+++y5//OMfAWsDVs80ePDgsF480hi73S4eqttvv13GXFlZyezZs6VneXZ2Ng6H\nQ2Rc1XaD1cd4y5YtIne9e/fmpptuAsKXJR4TEyNZ1t27dw+Jz9vtdklOvP/++/F4PPLzrKws8bIl\nJiaSkpIS0hqysUxFah0EW/UNDQ3k5+dLvopqUwnWOohUYxvDMKiurua9994DLK+PUgacTicxMTFy\n3WLXrl0xTZMPP/wQsNogq317zZo1FBcXS7LtTTfdxDXXXAOEz5AyDEP6AFRVVcn+GxMTg9PplL3C\n7/eTn58vzxUfHy8y7nA4wu6N0DFkjUaj0WiigKixkBUOh0PcHFlZWT9yQ8bHx0sqfVVVlbjFsrOz\nycjIiJhrz2az0adPH+644w7AyhRUMayYmBj8fr9oVoZh4HA4JGNVtUMEK4v4m2++YfHixYBVMtLU\n9b3HmpPg8MC9994bEve+//77I6Z9Hw01Bx06dODiiy+W8QTHiBtbL8oFrDq+FRYWSotBsDI5J0+e\nHJGaXjWfu3fvlvKJpUuX8tJLL0km9VdffcXrr78uFoRpmtKe78MPPwx7iU1jlFU4cuRIafE6a9Ys\nDhw4ICGNTp06MXLkSPE6LFu2jLlz5wKWR8IwDKmNfeCBB455uUNTYLPZJE4dXJ7k9/tDMtxzc3OJ\nj48XF/pZZ50leRIxMTEUFhaKezspKelH4ZBIliAGXxF44MAB8bacd955Ye0WdaSxgCXLS5Ys4csv\nvwSsrloqtyE1NZVhw4aRkZEBwNatW9m/f79k4btcLrE6hw4dSmxsLNdeey1gVR1EwrOp9osDBw6I\n2728vDzkIpry8nLy8vLkd10ul1jPNpsNv98vuREJCQm/Xpd1MMEuyOAD2TRNWWBgxTCUq7dNmzYR\nbR9ns9lITEyUhdGpUyd5qapNnNrwlbCp8WVkZMj/CwQC1NbWymZtGEZY+lkfDbVxzZo1i0OHDsmY\nL7vssrDF+34OwbKQmpoqIYq6ujrZoGJjY0PqkBsn3O3duxfTNKXU7JJLLglJMgnnM6o492mnnSZJ\nXYcPH+bNN9+UZgQ1NTVS9gFWo5JXXnkFsJKlmusdxMXFyZzdd999FBUVsWzZMsByq3/55ZcSB1y3\nbp3kUDgcDvr06cOLL74IHN/94CdK4/a4ShaKioqYP3++KAslJSVkZ2eLa/WSSy6RcsSKioqQcMG2\nbdvEDa6SuiKFzWaTpMMVK1awadMmSWIcPHhwWEtvjobf7yc2Nlaa89TW1soa7Nq1a0jJHFghAKXs\ntGrVStpsdu/enbS0NHmGSOzbNptN1mL//v3lkD1w4ABz586VJMDy8nJKS0vle5/PF7JPuFwuMQLT\n0tJClOXGsehfonxql7VGo9FoNFFAVFrIRyMQCJCfny+Zn7GxsSFJF5G2JFQpCPy4XKkxwRl56enp\nUp71wQcfkJ6eLlmTwZdeRwJlcW7cuBG73S7jGD58uLgAowHV2UdppAkJCdJRKrjdKlha78aNG8Xd\nWldXR3p6OrfffjtgNXwIVyvVxigtecaMGZJlnZeXR01NjVhmyj2mmpO8+uqrYllES8ggNjaWjh07\ncsUVVwBWEtemTZvEW1VcXCxfjx07lieffFJcv5G8IanxuB0OR0hCFFgZ+xMnTpTvlRwVFhbi9/vl\nXmen0ynvJCUlRUJPEP6yJ1ViA/DKK6/g8XikFXBycnJE97pgGRg1apTc8qYSacFaU+p9g1W54Ha7\nZV/MzMyUr5tjrw7+vLi4OEnA7dSpE2vWrBFvVWFhIXv37hVZDk7sS0lJoXXr1rI/B3szGz+PYRi/\nSO6j/kAOri9VX6u4SUZGhhwaSUlJzfKSj/czg3/P4XCIK3Dx4sW43e6QPq+Reo5AICDul5qaGlJS\nUsJau32iBM+3ak8KSAcvlSH+/fffs3LlSikzS0lJYerUqdJK1eVyRWyOg2Pg6vae+++/n9zc3JDO\nbT169OBvf/sbYNUpB99UFi3YbDbZnBISEhgxYgTDhg0DrDr14Jh+JBWJo7VMBeRqPSXP6nY15Wrd\ntWuXfF1eXi5uTbDWpso4d7lctGvXTp7rWJ/ZVKi1OXbsWM466yypj26utanuFlAZ8xA6D8c6aKNB\njoNdz+rrnJwcJk6cKNUkixYtYs6cORIuKCkpkX2mb9++XHXVVXT+vzbHMTExP3ouNR/H2yHxR2OM\nhGD9H7/4g4KTClSLTCBk8TQ0NEgpBvxsATjWL0dsgn4BRxv3McccXE6h7vp85ZVX2L59uxzIkydP\n5r777hMLIdhqP8HF1SRzHXx3cyAQCLGSd+7cycyZM8Xyyc7O5uqrr2bcuHHADy0Sf8bG1iRjVvNu\nGAZVVVUy/tjYWBISEsKR2PKL5KOZOeG5Vnd7g3Wt39atW5k9ezZgyfzw4cNFrrOzsyXBJycnB5vN\nJo2Hzj33XClrVDJzjE22Sefa5/OxatUq+frkk08W5aAJD+TfpHwc8R8GxX49Ho8o8/X19SFKWVxc\nXMgZc7S1egQPynEt6ugzgzQajUaj+Q3SIizkxgRfbB1ctH0CmuNvwkJWpRRq/gKBgMTj//3vf5OX\nlyfN9C+55BIGDBggWl5wCcgJaugnNNfB8hosB6ZpSqa6zWbjf//3f6UzkIp9qpKMX9Dk4TchH1FC\nk1jI6t2qygyFKpdTPw92sx4pLny00rqfMe5fNNfB67TxOJuQ36R8NBPH9fJa5IEcBn5tL/lnu1F9\nPh9ut1ti8mGMAYZ9rlX7O9WZp3H3s1/Ab1Y+mgE915Hj1zRmaLnjFrTLWqPRaDSaKEBbyBa/Nq2r\nJY4ZonfcLXHMoOUjkvya5roljhla7rh/+KUIHsgajUaj0WiOgnZZazQajUYTBegDWaPRaDSaKEAf\nyBqNRqPRRAH6QNZoNBqNJgrQB7JGo9FoNFGAPpA1Go1Go4kC9IGs0Wg0Gk0UoA9kjUaj0WiigEje\nhxzNHUh+bd1fWuKYIXrH3RLHDFo+Ismvaa5b4pih5Y5b0BayRqPRaDRRQCQt5F81R7q6zefzAeD1\nepk4caLcoDR79my5ID0MV6ppNBqNpgXSog5kdUeout/UMAxiYmIA6z7k5jzcGn+2aZrU1dUB8O67\n7xIIBKisrARg165dtG7dOuJj/DkEAgH8fj/V1dWA9Xxq3jMzM3E6nVqZ0ByVIymo0YTq4W8YBh6P\nB7BkPj4+vqnu/Y4oje9O1rRM9JvTaDQajSYKaBEWstL+ysvLWblyJWvXrgVg9+7dtG3bFoArrriC\n7t2743K5gCNbrMFfq5+HU4uPj48HYNmyZZSWlpKeng5A+/btw/aZPwev14vX6wXA7Xbz/vvvs3jx\nYgC+++473G43NTU1gGU9qDm02+3ExsYyduxYAN544w2x+KPZKlKo54jEWNVnmab5IxkMtmRsNluL\nmLtglMcKrPCMCtHU1dXh9XpJTU0FwOVySbgmWp5RvYvDhw/z3HPPAZCfn8/UqVOZNm0aAAkJCVEz\n3qMRCAQ4cOAAn3zyCQAdO3ZkwoQJwA/7jyYU0zTF21dZWcm2bdtk31u2bBlpaWmceuqpAJx//vly\nxrhcrrDLQ9QfyIZhUFFRAcD777/P559/zsGDBwHo3LmzuIULCgpo166dCKHdbg+ZPJvNFuLWCffE\nmqbJ9u3bAcjLy6O8vJz6+noAYmNjw/rZx4vD4ZAD+Z133uHLL7+ksLAQgKysLEzTpKqqCgCPx0Mg\nEACsw9vv94ti9Mgjj/DCCy8A0fNsjfH5fCxZsgSADz74gDvuuIO+ffsC4ZMF0zRlzg4cOMCjjz4K\nwNatW6moqJC5jY2NpWPHjlx11VUATJs2TZS35g7FNEatodraWrZv3877778PWDLhdrsB6yDr3r07\ngwcPBqB///6kpKQAREWoI/i9uN1u9u3bB8CaNWvo1q0bU6ZMAaJHeVAoJcLv97N//34Ann/+eT79\n9FMaGhoAOOecczjzzDObbYzBBCuhjffi5sQ0TdavXw/AE088wZdffimyC9b4lILzxBNPcMYZZwDw\n0ksvkZOTE9bxR/WBbBgGO3fu5NJLLwVg37592O12+vTpA0CfPn3o168fAMnJydTV1ZGUlARA0nOx\nTgAAIABJREFUTExMiAVimqZo8DabTWLP4ZpcwzDYtm0bYC0gu91OWloagFgLzYXaVN1uN++++y4A\n8+fPx+v1ctFFFwFw4YUX0qZNGxISEoBQhcbj8fD666/z6quvArBnz56oUzaCN4P6+npeeOEFXnrp\nJcA6MG699dawjyHY6k1JSRGtvKSkhIMHD4o82u12iouLyc3NBWD58uWcfvrpAEyZMoXs7OyoiAs2\nNDTw+uuvA5YCd+DAAZljl8tFly5dABg/fjxDhw6lU6dOgLUWVVy2uTdjsGRDKaL5+fkUFBQA1rqo\nqakJ8WpEw3jBsoSLiooAePnll1mzZg1gKUY1NTXiGRwwYIDsbZEmOG9m1apVzJkzh2+++Qaw5nbA\ngAEAdOrUiTvvvJNWrVoB1n4YSfm22Wyyr23fvh2v1yvy6XK5SElJkbVZXl7O3LlzAVi7di1bt24l\nMTExbGNr/lWu0Wg0Go0mOi3k4PjOzTffzK5du+RnY8aM4Y477gCgR48eJCcnAxAXF4fNZpN/G6z1\nBAIBNm7cKO7YMWPGkJmZCYRPY6+rq+Prr78GoKqqCpfLJZZ9TU0NWVlZYfnc40HNUWVlpXx9zjnn\ncOGFF9K5c2fg2K7SmJgYhg8fzjPPPAPAhg0bxGWm4oaRJjieqawfpeXOnz+fl19+mfLycgC6du1K\nly5dImL9qM9wOBwMGTIEsDwKpmlKdm99fT2BQIDa2loA5s2bx4oVKwBYtGgRDzzwgFgXkfauKPmo\nrq7muuuuY8GCBfL/k5KSJN46Y8YMOnToID8rLCyU54sGN7VCuauVt2LdunVi1aWkpJCVldVsFubR\nqK+vZ/78+cyaNQuw9rP+/fsDUFZWhmEY4o2YNGlSROdayYfH4+HZZ5/l5ZdfBizLUs0xWOtA7eNO\np5MPP/yQ//7v/wZgyJAh9O7dG4CkpCS8Xq+EHuPi4oiLi2vSMdtsNvm8P/3pT+zYsUPCVyNHjiQp\nKUm8JjfeeKOE5kpLS1mzZo3kzoSDqDyQ1Yt8/fXXxY0H1gH87LPPivA1dkkDIYfw4cOHAVi6dCmP\nPvqoLLz//u//5vzzzweQja4pMU1TEtDAOpD79esnh11wck+kCXbDpaWlccEFFwCWyz8xMfGYrqNg\nF/CDDz5ISUmJ/Fs175Ei2CXt9/spLy9nx44dAOzfv5+ysjKJDS5cuJDKykp57ssuuyysbqdg1GfG\nxcWJG7qmpoaNGzdKLkRFRQUej0eUmqSkJIkhFxQU8Ne//pXf/e53AAwcODCih7JaMzNmzGDu3Lny\nPKeffjqzZs0St2Ow3Hi9XhoaGkQmIi0bP4XNZpMY8r59+6QcMTY2lp49e0ZNApqKa86aNYuPPvpI\nFMzf/e539OrVC7Di3p999hljxowBLHdwpMZtGAZbt24FYOrUqezduzckZux0OmWdxcXFhRhPvXv3\nluf55JNP+OijjwArLNKhQwdxzz/22GNNfiDDD6G1Sy+9lEAgIHOmco9UePHjjz/mnnvuAaxy1XAb\nUtplrdFoNBpNFBB1FrJpmpSVlQGwZMkSampqRPuePHky7du3D9GYlFURnIgBUFRUxCuvvALAhx9+\nyMGDByXhq6SkhB49eoT1ORYtWiQWZExMDL169RIN0efzRbT0Jpjgz4uPjxdNsXESXGNM0xSNfe7c\nuWzZskVceyNGjAhLiUWwNd/4/fr9fslSnjVrFp988olYnH6/H5fLxaBBg4AfMsQzMjIAuO222yKW\nRKLGHxsbK0lPgwcP5ttvv2XPnj0AUiJ07rnnAjBo0CBJCMzNzWX9+vU8//zzAPzxj38U2Q33M9TX\n1/Pss88C8Pnnn+N0OiXBcubMmT9yRQd7fjwej6y3aMoUV2NUsrJ8+XKRo65du3LKKac0e9IlWDKh\nXNTz588H4OabbwZg9OjREp5R3hXlwo5EUqWybB9//HGprlBzqOZu8ODBjB49WjKUhw8fHhJSSktL\nk7/z8ccfs2jRIsBKnEpKShIZD3fplt1ux263y9iCLWWwvIjKtX7gwIGwN3Rqfsk7Aspl7Xa78fl8\nsvFXVlZSWVkpk1ddXc3nn38OwKZNm/B4PPKzvXv3sm7dOsDaWOx2u8SNTznlFD799FPAyiZualRG\npHKLDRo0iMsuu0xi2DU1NfKMzRmvstvtx+xK1Lg8ZNOmTYDlAu7YsSPdu3cH4Lzzzgu7W1LlBygF\nbM+ePcyZMwewMn7LyspkM2jfvj3Tpk2TTeqzzz4DkIx85Y6KJHa7XTJhBw4ciM/nC6nxjomJIScn\nB7CeVS388vJyCgsL2bhxIwBz5szh7rvvBsK3Wal3fscdd/Dee+8BVvbp448/zo033ggcW249Hg8b\nNmyQcFCbNm3CMs5fSm1tLY899hhguSHVJjx69GiysrKaXXkwTZMtW7aIElZfX89NN93E6NGjAcvl\nqzrorVu3Dp/PJ2G8cI89EAjw4IMPAvDiiy9KnoDNZiM1NVVyCm644Qa6d+8eUu7W+BnVXj1lyhR2\n794NWOGmuro6UV4j1StAyXzjz3S73WJI9e/fX7Kzw0VUHshKs+7QoQObNm0SrTY3N5d58+aJNvbN\nN99IwN3j8eD1esV6jouLEw3M6XSSnp4uwfjTTz9dNr9wYBgGpaWlIlRTp06lT58+tGvXDrA2BHWw\nNGfCiyrHUgQnxRmGEdIYpLa2Vt7Lvffey+LFi+X5vF5viEA3FUeaF/U5BQUF7Ny5E7CshLi4OAYO\nHAjALbfcwvjx4yWJpKKigkAgIMpXc5UQKaUlKSmJCy64QOa2rKyMhoYGiZs5nU5pHlNZWUlubi6H\nDh0CLM+LKk3r1atXk8uOYRi8/fbbgNXyVW24d911FzfeeONxKZBVVVWsXr1a1mK3bt2OqbBFskGL\nz+fjjTfekHKchoYGidfPmDEjLPHKn4vf7+ehhx4SBf6ss87iyiuvFEXSZrOJhX/48GH69Okjsdpw\nz2Vubi7/+Mc/AEQ2wOoJ8dhjjzF06FAA0tPTSUhIOK61ZhiG7I3nn38+JSUlXHHFFQBhPwAVKo9g\nzZo1bN68WbybwSWy3bt3JzU1VcYUjn1Ex5A1Go1Go4kCotJCVu6NM888k+3bt1NcXAxYbuhnnnlG\nXB01NTUhzRXi4+Ol7KZ3795iwdXX19O3b19xsaalpYXVSqqsrKS4uFg02lNOOQWbzUZpaSlgxa2U\n+9zlcv2oq1ikMAxDSoScTidOp1Ms97y8PPbv3y9uxy5dusicNTQ0kJCQIN1sDh06xPjx4wFITEwM\n67Oo952fny8xWKfTyaRJk/jDH/4QMlZV0K9K4MIRnvg5qHlJSEhg8uTJonnn5uaybt06TjrpJAAm\nTJggFk9MTAylpaUiO5s3b+bNN98ErC5CTT3XZWVlPP3004DlFVG3kv2///f/ftI6Vu/mzTffZMeO\nHXTs2FH+jpIdu91OIBCQNR7pDOzS0lLmzZsnFpDD4eCaa64BLI9DczZgUVb8jh07WLZsmcz3zTff\nHBK7rK2t5ZFHHgHg+++/58ILL5R5DHczk6+++kqaAAHikn722WcZPXq0jNntduNwOGQ+g2UnEAgQ\nCATE07lr1y6GDRsmf6++vj6kIiXce6NhGLJ3zJ8/n4qKipBGIWotJicnU1ZWxg033ABYJZ5NPbao\nPpBHjRpFWVkZH374IWAdyGVlZeK2tNlsEpdr06YNw4cPZ9y4cYBV2xa8IaiEHkD+TVOjFtSmTZvY\nvn279EBNSkqivLyct956C4Di4mIR1JtuuomUlJSQjSlSh7PH4xFXaFpaGl6vV1w3LpeLs846S5JE\ngscUFxeHw+GQcEFzxGSLioqkrlj171VxNKfTSSAQYOHChYA19szMzGat/Q7G6XSSmpoqiSsZGRnk\n5OSIm7p9+/ayBqZMmUJmZibLly8HLGVIxcQffvjhJo0jm6bJvn37pFUtIIfVT7lyDcMQefjyyy8p\nKCiQsNBXX30lCmggEKBfv37iJo6UMqr2jOeee441a9ZIDkd2djZ33XUX0Pxd5tSYHnzwwZC48Mkn\nnwwgdeo333yzJEHFxcVRV1cnh6TT6QyrkjNw4EAxeuLj47nyyisBGDt2bMi+6nK5QvIkgm+L8/l8\nrFixgtWrVwNw0kknSf/tzMxMHA5HxDu7KXlUiavq87Ozs2UtlpWV8dRTT8m433rrraPenfBL0S5r\njUaj0WiigKi0kJUbtaysjIKCArHiamtrQ1wYDodDXCZnnHEGN9xwg1ilqampIe6SmJiYsGvAykJe\nuXIlRUVFoj1t3bqVvLw8sTxUdx31b9xut1ggMTExEXHRgKXVKc26tLQUp9Mpbv5WrVr9SNMOvu3J\n5XJJIw7VmQkQF2c4ME1TEp/27Nkj7z4nJ4eEhATRxlNSUkLKnJxOp7iZogFliX7xxReA5Srt06eP\ndA9yOBwiD/Hx8YwbN066fM2dO1eaMaxZs0YaQjQFKlNWJa24XK7jbpzj9/vFct+9ezdJSUkcOHAA\ngH//+9/ye06nk7vuuivEQg43pmlKAuDbb79NQ0ODrLHp06eL56S5s6tVWGL9+vXExsZK2V5JSQn1\n9fU8+eSTAMyePVvWcEZGBl6vV2Q/KSkprCWVQ4cOlf7333zzDVdffbV8buMEVYfDEdKARSVZLlu2\njJKSEvGsDRo0SN6BcnNH8l3Y7XYeeughAC644AIOHz4s+2BOTo6U4T788MPMnTuXpUuXAvCf//yH\n6dOny7ibgqg5kINbsKmuLR999BFr166VjELDMHC5XHKwxsfHS7u+zp07h2xkwansqhXbkWomm/LF\nB2coB1/xtW3bNtavXy8xz+TkZIlnOxyOkIsvfD6fHOTh2KwMw5A4cWVlZYjrLiUlRTKpVVmCWviG\nYchcGYbBhg0b5HB0OByyIYQr5tN4PKZpyuHfqlUr3nnnHf7+978DVlZvQ0ODlGnZbDa2bdsmnbti\nYmLwer0iO8F1spHYCPx+PytXrhTFs7S0lH79+omLt3F2anx8PE888QRgxbjUO1u0aFGTHsgAbdu2\nZcSIEYClWCq5PNJ7DS5d2bdvn8S2KysrcTgcsgG3bt1a5D01NTVEWY4EgUCAm266CUCUYnUYXHvt\ntVHRScw0Tens5/P5SEpKkrFedtll7Ny5U2KupmlKqEJ1AFSlOeEmLi5OlMPOnTvLfnG0daPec01N\nDa+99hoAW7ZsISUlRUrounTp8qPLfiJ5INtsNokTDx8+/Ec/Vz+75ZZb2Lhxo5ScrV+/nssvvxz4\nFR7IamFv3ryZp556CrASiwKBgLzU1NRULr74YokNb9q0SV5cZWUlW7dulYWWnJwsQnuk1o7heOHq\nb/r9fhoaGti7dy9g1e1WV1dLzHD69OlyNZ1KplItCjdv3kzPnj2B0PhFU2EYhljFypIBq4yosrJS\nShkqKyupr6+XhDqn0ymNK9auXcucOXNkzOnp6XK4hAulYKmFO27cOPLy8gDYuHEjmzdvllapX3/9\n9Y/KsD788EO589ThcJCamsrjjz8OwLnnntvksaBj4ff7qaysFPlMSkqie/fuR93cbDYb3bp1A6zm\nFeq5VYOZphxzYmKiHP5///vf5UB2u90h9wMbhkFdXZ3Ejf/0pz+JghYTE0MgEJBSlpycHFGe+vfv\nH+JFiUTSzp49e9iwYYN8ntPp5JZbbgGseH1zW8YKtZ4SEhIwDEMUyIKCAtxut+yD8fHxUsJ55513\n0r9/fzFEwq3oBOftOJ3Ooxo2SoFX+/pHH33EqlWrAMvoysjIYPLkyfI80fIOjjQOdXZ07NiR008/\nXZ550qRJTd4GWceQNRqNRqOJAqLGQlaayfLly0UzVNq5clGPHTuW++67T77/4IMP+PLLLwHLJbJ7\n926JRQwdOlQ0m0g131CfcdVVV/Hcc8+JG3fDhg106NCBa6+9FrDcIsEt2g4fPixZk//85z9F2339\n9debtFm8YRhUV1dLCGDBggVSTrF48WK2bt0qWdbKra007qSkJGnn6PV6OXjwoMzv0KFDJUM42KPR\nlKi7hZV1dc4554hLNCMjgyFDhogGvmbNGurq6kK098TERGnI3759+xDXbKQ0dDWeQCBATU2NuNTH\njh1LUlLSMceg5nrMmDES+lCeoqbEZrPJ373wwgulo92DDz7IyJEjxY26c+dODhw4IJe/7Nu3T9Zl\nZmYmvXv35swzzwQsj8CoUaOAH7odRfIChFdeeSXkLvTs7GyuuuoqoHk75TVGdZJLTEykoqJCMn6z\ns7PJzMyU71u1asWdd94JwLBhw4iLiwuphgj33Abn5qh5VR4pZRGrMJDyxs2fP188AA6HQ+4dVn8v\neK1Gi7WsUM8UCATIycmR74cOHdrkHsyoOZAVXbp0+dHLUYfGjTfeSGZmpiQQrVu3TlxR+fn5dO/e\nXVw5cXFxEb+1RX1O165dGThwIN999x1gufsqKyullWdCQoIkCtTX1/PBBx/w1VdfAVbimkp4+f77\n7+nYsWOTjd/v97NlyxaZv4aGBlF++vXrh8vlknEYhkFCQoIsuJSUFFn0aWlpGIYhf6dr166yuMI5\n106nU9y6KSkpcjirOm+1UPLz87nrrrvkwPP5fFx99dXSC1g9i9qMI9VrWW1ac+bM4fvvv5fYVHBJ\n3k/92x07dsiGGK4xq7/ft29faU+6ePFiFi1aJBtseXk5DodD5MPj8YgLvlWrVpx66qlyILdq1UrW\nYkpKSoiCHO55d7vdfPbZZzJ/Ks9EudfDodT8UlRp2ODBg2loaJDa3C5durB+/XpJpho8eLDcsuVy\nuSKeA6EIPkjdbjdFRUVS392+fXvq6+ulvnfXrl3yu/Hx8QwcOFDi3pE+gINzH+DY8V+/3y+K/7x5\n89i5c6eUeiUnJze58RE1B7J6Kb169ZKN3+PxSPNvsNrEfffdd5L0tWTJEllY9fX1pKenyyYdGxvb\nbJqWw+Hg888/F4vs0KFDFBcXS2zO4XDI5uXz+XC73SKssbGxshAHDRp0ws8QHONYv349H3/8sVjg\nQ4cOFa1c9X3esmULYNV8FxUV8e233wKWhaxigvn5+fh8Pvk75eXlkugQrpakSjk7Vqa8kpNOnTox\nffp0afFnt9s588wzZWzN0a7UNE3xTHzyySdUVFRIAolqgnC0f2cYhlj/O3bsEEVC3eHa1Ki5cblc\nYol17dqV8vJyWW/Lly+npKRErDZV6w2WZX3ttddKtnZw0kzweo4Eu3bt4sCBA5IIpyozguP1wReX\nNJeFZrPZxLt36623EhsbKz3A6+rqePPNN2WNxcbGiiIR3HyjOcasZLG4uJjnnntOPJaJiYnU1dWJ\n0m6aZoiid/PNN4cYTJFQKIKVh/z8fMBSFrOzs0MO5eCe+evWrZMLVjZt2sSQIUNCLvJo6vHqGLJG\no9FoNFFA1FnInTt3Fs1/9erV+P1+0cpvvfXWkIxkpZ2Dpc2feeaZUsrS3HGIlJQUqX0cOXIkO3bs\nCGk5p7KZHQ4H2dnZ4r65/PLLpbF6U7irgzvkeDwevv/+e7EUJ0yYIHOtLE9VC5uVlcWePXtEq3Q4\nHFJ2kZaWxqFDh2T+6+rqxPrr3r17s7YfBGusWVlZ4tbzer0kJiZGvPtPMF6vl9///veAZV22bt1a\nOjApKyM4VBN8zeTChQu5/fbbAeuGM9Vis0+fPmEPEajcgBkzZlBXV8f+/fsB6NGjB2+++abc0pOS\nkiLW9NVXX018fPwR22NGeu6/+eabEA+U3W5nxIgRUvHQ2DIKnvtIj1VlL/fv3x+v1yvzt2HDBrZu\n3Srj6tq1a4jHoTlRY8zOzqZnz57iUauoqKCmpiak54Hq8HX99dfTrVu3I1rI4UTtgx999JF0f7zg\ngguYPHmy7H+BQICKigpmzpwJWDfFqfrwNm3acPbZZ0slTzh6RkTNgaxITEyUa8cuvfRS9u3bJ+4D\ndYgF98VV8dYrr7yS22+/XdyozX0gA3LIbt68mf3798tLnjdvntyKMn36dAYMGCAutOB7iZv6GVJT\nUyktLZVks5kzZ0qctV+/fqxbt07aURYVFWG32yU23KZNGxlX69atiYmJkfDAiBEjpM60uTcIsDbW\ntLQ0ic1u3LgRv98f4r6PRLmN+hywNihVIlRVVUVDQ4M0zMjMzKRv375yOKjrO8GSlRdffFFyDlq1\naiX11krhCCdqjtSGpdyoMTExcu80WDcSqVuoVFwz3LHuY6Hm3ev1htx3m56ezo033njEG3vUwdDU\npSzHi5onlSyl8jmefPJJSkpKJCRw5plnRsWdzcEkJCRw9dVXh5Qebtq0SWQ6NjaW0047DbCSGIMT\n+yIlH6o0c+3atZLPU1VVRUFBgcj3hg0b2LBhgzSjgh9CSmPHjmXatGlhPWOaf/fUaDQajUYTfRay\nzWYT6/Hrr7/mxhtvlLtLVUan0hSHDx8uGW8TJkwI26URJ4rdbqfz/90XCvDYY4+FaOHh1hCDP6tP\nnz5yUcH+/ftZsGABYCXF+f1+0bwzMjJITk6WBiZnnXUWXbt2BawsSYfDIa7WxMTEqPJMqIxaZd3H\nxsaye/dueZa4uLiIjVONJTipJRAI4Ha7Ze5zc3NDLJ5AICDZqlVVVTidTrnr+YUXXpBOSZH0Rths\nNpxOp5Q9LVq0iP3794tn5+STTxZvVXO0Pzwao0ePplevXhIuuvHGG+nTp88xwxfNPW673U5VVZW0\nyly1ahWBQIA+ffoAVlgpGjxRwdhsNlq1asX1118PWE2HNm/eLHKdlJQkoZbGmfaRQr3zxMRE8TgV\nFBSwfPnyH61TJctDhw6VUNPQoUN/sjzxRIm6Axl+2GjatWvH/PnzxWXd0NBAfX29bLTBcarmXkQ/\nl0iOV81Rr169eOSRRyQGv2/fPpYsWQLAwYMHaWhokEXfv39/evfuLd//lCBGw/w37rWtvs/MzGTJ\nkiWcffbZwA834kRizEqWExMTpc3ezJkzqa6ulk1Add5SG4bP55Ms/N69e3PVVVcxbdo0wAoXNFer\nR5vNJq7efv36UVtbK/Ixfvx4UdCi4TBWnz948GAWLFggse/BgwdHVCH7JdhsNpKTk8XN7nQ6adeu\nnYQqwn3F6S9FdcAD6wanbdu2Sfy1R48eIsMZGRnNolAot/TEiRP5+OOPAatqwe/3y8+ysrKYMWOG\nXNXapk0bidcfz55xovkHtgjGS5onMHN8HGv2WuK4W+KYoYnG7fP5qKiokOsX8/PzKSkpEYsjPj7+\n527KJzxm0zRFEdqzZw+5ubkhtehFRUUUFBQA1mammvanp6fjcrl+abytyeVDKRF1dXVUVVWJhRzc\nnvYEDwu9Fvnh0hmwmh5lZGSE43KcsI15+fLlzJs3T8oo77nnHlGIT7ART5PIR+N7B4IbNYVJ2Tmu\nPxpdfg+NRqPRaH6jaAvZQmvlkSMic22apsQNDx8+TH19vWRL/oJmClo+Gv+BRvtGE1oVeq4jR9jG\n7PP5KCwslE6AAwcOlDDMCTbU+LXJR+gv6QMZ+PW95JY4ZojecbfEMYOWj0jya5rrJh9zE5YZ/trk\nIwTtstZoNBqNJgrQB7JGo9Fowko0ZoVHI5F0WWs0Go1GozkK2kLWaDQajSYK0AeyRqPRaDRRgD6Q\nNRqNRqOJAvSBrNFoNBpNFKAPZI1Go9FoogB9IGs0Go1GEwXoA1mj0Wg0miggktcvRnPB86+tHVtL\nHDNE77hb4phBy0ckCetc+/1+6c2+fv160tLSAMjOzqZ169bye4ZhhNyt/RNo+YgcunWmRqPRaDQt\nhUhayE2GupM1Gi5C12g0mnCibi7r2rUrADExMfh8PgCefPJJrrnmGrm97GdYx5ooJCrfnmrn2fiw\nNU2T6upq/vWvfwEwYcIEevfuDfBzr9PT/AZQcuT1evH7/XL9m2EYIcqclh1NNGOaJnV1daSnpwOQ\nmJjIk08+CcC4ceOw2WzaMGlilNHndrvl66SkpLArPHon0mg0Go0mCohKC/lo2l4gEOAf//gHS5Ys\nAaB9+/ZiIWuaBmVV+v1+ysrKqKmpAaBVq1YkJiYCJ3zBeNgxDAOv18uePXsAmD17Nt999x3Z2dkA\n9OjRg6ysLCZOnAhATk4ODocDsNyBmh9Q8mAYBvX19ZSXlwOwYcMG0tPTadeuHQAul0sSjeLj47XX\noQlQc79lyxbuvPNOOnXqBMDLL79Mjx49AKLeOg6WH4X6Wq25aJIV0zTxeDy88MILALz++uvU1dUB\ncMEFF/DMM8/IPhgOovJAPhper5evv/6aDRs2AFBbWxvVwtjSME1TYlMFBQU8++yzrFmzBoBRo0Zx\nxRVXADBo0CDi4uKabZxHI9hFvX37dr766isAcnNzKSwslN9LSUlhx44ddOnSBYCsrCxiY2MjP+Ag\nTNOUjUopFAClpaWsXbsWt9sNWO9BjdvhcIRd/tWcVlVVsWDBAl566SUA9u7di8PhICkpCYChQ4dy\n2mmnAXD++efTunVr2XCbg+Bb7EzTlHlqSfuF3+8H4MCBA4wePZp77rkHgNTU1OYc1nETCASoqKgA\nYO3atSxbtgywxj9w4EAGDRoEQHJyshxyzX04G4ZBQUEBH3/8MWCtP/Ueli5dSn19PS6XCwiPLLWo\nA7miooLVq1eL1aY2g5aC2nAbGhpYsWIFYGlga9askZc7atQo/va3vwGQkZER8Q1EjbGoqAiPx0NR\nUREAM2fOZNasWQDceuut/OlPf2rWDbcxpmnK4l+xYgWrVq0S5SIzM5MrrrhClIiqqip27drFp59+\nCkDfvn1JSEiQv/NL5/yn/m3jQ0KNr66ujpKSEubPny/jX758ufwsEAjI2Hv06MFnn30GWJZ9OOXD\nNE1qa2sB+Mc//sHLL78s8hAIBLDZbJSVlQGwf/9+PvnkE8Cy4K677jquu+46wIp5RlKOvV4vixcv\nBuDdd99lxYoVYsmPHTuWadOmiWctNja22Q+Bo1FcXAzAwoULOf/880lOTm7mER0/pmmicN03AAAg\nAElEQVRSVVXFvffeC8Cnn35KdXW1/Lxt27Zi8Xft2pUHH3wQsLyezbGvqLXp8/lYvXq1lJjZ7fYQ\nS76uro7MzMywjSM6JVGj0Wg0mt8YLcJCVtrLypUrxQoCGDx4cLO7oI6WEd74d2pra8Xl+9lnn4kl\n6nQ68Xq98nf27dsnGvsbb7wRcW1RjSMrK4sbbriB6dOnA1Yc9tVXXwXg0Ucf5eqrr5YyjGjA7/ez\nbt06wGqc0KVLFwYPHgxAmzZtSEtLE9fT999/z9q1a9myZQtguaVUfPlEsiiPJANHiqH5/X4qKiok\nF2LlypUUFRWxdOlSAGpqasR6VqhMz2HDhpGRkXHUz2tKTNNk9+7dALz66qscOnRInsNut2O320Oe\nT7nVd+/ezTPPPMO+ffsAeOKJJ8QDEc4xKzf/+++/z+233w5YXjWbzcbBgwcB2LRpE3PmzOGqq64C\n4L/+67+ksYZ6nuASoubaXwKBgKy3ZcuWceedd0atJX8kAoEA69evZ9euXYAl80pu/X4/lZWV8r52\n7dpF3759Abjlllua1UIuLi5mxYoVEjcOBAIynvT09LDLQ4s6kJcuXYrP5xNXtXrBzcnxvKBdu3Yx\nYsSIEGVCbVCTJk0iJiaGL774ArBclOrgaI7NQMVSu3TpElIaNHjwYNlwZ82aRUlJSVQdyDU1Naxc\nuRKw3Lp9+vQJcUvabDbZ0DweDxUVFSJXcXFxctD4/f6wbMR2u10OVY/Hw86dO9m0aRMA5eXlbN26\nVdxkNptNNgGbzUZ8fLwcIM8884y4ryNxICs3Y0NDA06nU9Zev3796Nixo7isd+7cKb/r8Xiora2V\nXI+8vDz69+8PELbN1jAM/v73vwNWba5aa6ZpEh8fL+NOTU3FZrMxe/ZsANatW8fAgQPlGb1eL2ef\nfTZgxcXVv4v0Wjx06JCMMSYmhrZt2x71d4NDIdFioHg8HvLz8+U9nHTSSVx00UXye7t27WL//v2A\npRAHu4FPJGz0S1FrMz8/P0RZMAxDnslut4c9d6blqFwajUaj0fyKaREWsrIcPvzwQ/x+f4iWcjwu\n4+Zi8+bNAAwZMgSfzycW2vjx4/nnP/8JQOvWrWloaOAvf/kLAB999BEPPPAAEPmMw2ArUlnHan4T\nEhLo2bMnYGns0eQ+CwQCfP7551RWVgJWklbfvn1/5H5WbuAVK1awZ88esYyys7N/9NzhQFnhHo8n\npLwqEAjQ0NBA+/btZTwejwewEl4mTJjAZZddBiAZnpHAbrfLmE4++WQCgQDDhg0DYMqUKZimKWVQ\nmzZtYu7cuYCVFezxeCQJqba29kelLk1NbW0tzzzzDGBZXOod5uTkMGnSJMaNGwdYyUSbN28WV/y2\nbdt48803ActdaZomixYtAuC1115j6NChYR33kTAMg1mzZpGXlwfAlVde+ZOWWTRYyaZpinevuLiY\ndevWyVgGDhwo8tC2bVu6dOki7uySkhJxWZum2SwWspJP5eVRFnLwWFJSUkhJSQnr2KL+QDYMg7fe\neguw3Dg2m42TTjoJsGI8ytUQ7OKLBhYuXMh5550HWBtuUlIS77zzDgDnnXdeyIHm9/vlmYYNG0av\nXr0iP+BGNJ7H+vp6Fi5cCFguXtX1Khpwu90sWLBANs/u3buHbKA2m41AICBZzG+88QbJycmceuqp\ngJWtHy75UX8veMP0+XwsXLiQrVu3AnDw4EG6du0qLr0xY8bIeFq1akVCQkJED2KFzWaT7OT77ruP\nvXv3imvRNE2KiorYvn07YNXKqjG3b9+erKwsKYPq0aNHyDw09RybpsnevXtlMzVNU0qDZs6cyfjx\n40OUs9GjR0tM+c9//jOrV68GLJd1IBBg7969wA9uzEjjdruZP3++vPO77777iF0LFaoTHVhrM1yy\nfKR3F5xDUFdXJy7q7777juLiYlHounXrJiFGJdPqHXXr1k1CA5Eo5TsSaj/Ozs6mqqpKwnN+v1+U\n5+zs7LArZlF/INfW1or1aBgGLpeLp556CrAsNbVoDMPA5/PJhDkcjhCrJ5hwv/B9+/YxZcoUGVty\ncjJbtmyhY8eO8jtKkOvr63n44Ydlg3j00UebrZwouBbWNE0CgYBYaqtXr5avx4wZIyUL0YDP56NX\nr16iAMXHx4c0TAgEAmzYsEFKK+rr65k6dSpXXnklEJlGJ6ZpyiJftGgRubm55OfnA5ZC0LNnT2n2\nkJycLJtAXFxcszZiUePo1asXhw8fFqWmoKCA3NxcKUFMTExk+PDhAJx99tl07txZPBCpqamyBsPx\nHIZh8PTTT4t8OhwOzjzzTMBqLRkXFxfyuQ6Hg86dOwPQoUMHeca6ujpM05TDO9Jlh2q/mD9/PgUF\nBWLVt2nTJiQpEH7YPwKBAAcPHpRnyMjICJvyFqwEgPUulSXp9Xqpr6+nqqoKsLwkFRUVspdlZmbS\nr18/wPK2+f1+aSSTnJzc7E1C1Dvv1KkTXq83JKlSze0555yjW2dqNBqNRvNbIKotZNM0KSwslGYE\nDoeDq6++mgEDBgChHY18Ph9ut1tiLbW1tbRp0wawNJxItJhT1sLQoUNDLjPYvHkzHTp0kN8zDEOs\no6uvvpotW7Zw+umnA5Y7p7m0RJ/PJ3N96NAhYmNjRVOsrKxkypQpgNVcIZqaFCQmJjJu3DiysrKA\nH+LAyqpYu3Ytl1xyiWj41157LbfddpuUOjWlXBwtp8Hn80mXq1deeYWKigpxM9rtdvLy8iQGm5qa\nSvfu3QGYMWMG48aNa7bbfNRzOJ1OvvzyS/7zn/8AllvV5/PJz2tqasQFP2jQICZNmiTdl8LthgwE\nApSUlMj7jomJYciQIcCRvR8qhAFWxzGVo6JKnlQ8s02bNhFdizt27ACsxioDBgzgzjvvBKy5r62t\nlW5zrVq1EtkpKiqipqZGrGKHwyF7YFPJSrA13rgCQVm28fHxxMXFiev53HPPxW63S4jjrLPOkv1Q\nNZVR44umdrUul4tDhw6JtyX4GU499dSwy0NUH8iGYfDKK6/IoZCVlcV9990ni+ngwYOsX78esGo5\nY2JipJzo8OHDXHzxxYCVUBDubkF+v5+pU6cCVu1jYmKilOG0b98ewzAkvrJ69WpJJNm0aROBQIBz\nzz0XIKx9Uo+FclcrReHzzz/HNE1xo/bs2VNiQP369YuaWD1Ym26fPn1oaGgArEVlGAbff/89YB1q\n1dXV/OEPfwDgtttuEyWtqWn8N5Ws7t+/X8pygg8PhVIs1c/Ve1i1ahUTJkzg7rvvBqzkqkhuYOp5\nYmNj8Xq98jzqv8G9z1WSzt/+9je2bNki892zZ8+wtia12Wx4PJ4QpUXFgb1er4QwFMEdyPLy8kLc\nkzabTcIxkYzbNzQ0SIc+u93OlClTxOXv8XhYtmyZ1M07nU5KSkoA60D2+Xyy702ePJlLLrkEaDpF\nKLjtaOO/1/jQV7LZs2dP2rRpIwdbIBAIqWF3OBzyu5EOKR6Lffv2cfDgQRmrzWaTkj3lYg8n2mWt\n0Wg0Gk0UENUWstvt5u233xYtvEePHlRXV7N27VrA6h6lugElJSWRnp5OaWkpYKXdHz58GIDnn38+\nbH2v1dgWL14s2ZoOh4Nrr71WNG2fz8f27dulF3SHDh2k0N/v95OQkCDu4OZK6LLZbMTFxTFq1CgA\nOnbsyN69e8UN+cUXX8gcduvWLeoa3Pt8PrEqU1JSWLBggbhXa2pqeOKJJ5gxYwYQWbev0vbdbrd4\nGKqqqvB6vT+68ED9N9jt6PV6WbRokcj5k08+ySmnnAJE9tYth8PBKaecIs1Mtm/fLlnJ8EO5Cljh\njXnz5omHaPbs2WEtH3I4HMyYMUNu9yorK5OKgP79+3P55ZeLBWm32/F4PMyZMwdAZEb9zG63Szim\ncRJTOFCW2HfffSdjGTVqFBdeeKF4ywoLC9m1a5dY9aqLlBrjoUOHQvpEn3POOQBN3sTieGQtOMTh\ncrnEK+j1esX1qxIVgxP91Fw3R8kT/PAeZs6cSU1NjXwfFxcn4Y9I7BtRfSAfPnyYuro6mYgOHTrw\n/vvv8/bbbwOWa1jFDUeOHEl8fLx0Djp8+LC4ycLpBlYb0pw5c0Kyqm02G9999518/vr168Ut3b17\ndyZPngxYrqpevXpJPLM5UV2hADp37hxyLeGsWbNEEWrdujVXXHFFxOOZRyMQCLBx40b+/Oc/A1Ys\nzufzycH1ySef0LZt22Ytp+jXrx/ffvstgBzGSj5jYmJwu90hG60qJ/rLX/7C1q1bOXToEGCFElRL\n0HC53Y/2HOPGjZPPy8vLIzk5mW7dugFW5qy6EONf//oX5eXlEvO89tpr5bKH1q1bh6UL2tSpU6V1\n6qxZs0Qxf/DBB3n//fdFgayqqiI/P19+7vV6Q5SfQCDAqlWrAOvgU27VcMyzaZpSO//Xv/5VLpMY\nP3687CFguc579eolnecGDhxIeno6YNVc//vf/5bS0DVr1ohcNdWY1eGkeikc7991OBwSijlw4IAo\nOi6X66ju9OZyV6txLlq0SOLzYI117NixQGSMpejYURuhtCW3201aWhqtWrUCYPjw4axatUriJzEx\nMbI5nXXWWWzbtk0Oibi4OLnvNhJX61111VUS4+natSvjx4+XNP+kpCQGDx4cknCmLE/TNBk5cmRU\nNdoAa5OLj4/n5JNPBqwYqIrRbty4kYsvvrjZD2QlJ9u3b2fGjBlSOuZwOBgwYIDUfSv5aU6cTucx\nY1Aul0vGaZomOTk5gCUrTz/9tGzWxcXF8h4i4aUItlxSUlI4//zzAWvtBVvoNpuNSZMmAZaFNm3a\nNCmB2bNnD19//TVASPvEpiQ5OZk//vGPgOV1+uijjwBrvr799tuQqy2DLV+HwyEbbSAQkJpmsJQO\nZR2FYzM2TVMs+Q0bNojXrHHb2pSUFCZOnBiSBKX2i4SEBHJycsQY6Ny5s3gDmgKv1yuepksvvfS4\n/51qEqK8FomJiXIgNy5Da25UTT1Y3pVg+UhNTY1oi+DoOgU0Go1Go/mNEpUWsiIrKyukwUZ+fr7E\npcAqgledYDZs2MCsWbMkznL66aeHuBrCpZEprfWUU04Rl53T6QyxeBt/9jvvvCNuEYfDIZp9NKE0\nXKUtjhgxQlyqbrebQ4cO0aVLF6D53EwFBQWAdUFHUVGRuBdPP/10Xn75ZXHrHYloaDV4NIJLQvr2\n7Ut2draU1AXHt8JNcKOY+vp6nE6neHmO5C5XMn/yySeTlpYm7lifzyfr9sILLwxbdruK0T/11FNS\n8fDQQw+xY8eOkHCAYRhi8bZr106yrNXeEfyM4aS+vl5u/Dp8+LB0RUtKSgrJaI6JiQmx0E3TFC9J\nQUEBL774oniHLrrooia15isrKyWcVlxcTKtWrY55uYmSF6/XS3l5uViXbdu2Fctdja/x/eBH+5vh\nxuv18sYbbwA/lK4qWc7JyYmIh1UR1Qey8t+r3rIbNmygpqZGBCIrK0v6Refn53P48GEp0/nd734X\nljrToxHs+joaSlhVOQhYMTXlnowGgi/qLigokGSRjIwMKcNQbqjmZPv27RKSKCsrIy0tTVymkyZN\nwufzyQabmJgY8m4CgUBIqUt8fHzIjS7RgFJCS0tLqa2tJSUlBbBqY9XX4cYwDAnDLFq0iAkTJtCn\nTx/gyGEgNeZ169ZRXl4ekuCj4vnhXIvqbyclJXHGGWcAlqJcWFgosuD1ekMSuXr27Mlzzz0HWNc2\n+nw+md9wX7cXFxcXotSquPbChQuZNm2a5HM0NDRQV1cnMfm8vDyJl2/dupWDBw9K57GbbrqpSWU4\nJSVF9tTly5czfPhw6akQ7Do3DINAICD7RXl5OXa7XdzwLpfrR+NSa87r9TbblZfqmkhlTDVWDNq0\naRPRhLOoPJDVQ6usR7V5HjhwAL/fLwu/vLxchDg9PZ2zzz6bhx9+GLBqxqLN8lFabFVVlYztxhtv\njJpDwDAMaZJQWFjI0qVLZSM777zzRIN3Op1yh2wkMU1TlIEzzjhDcglat27Ngw8+KHLy1Vdf8fbb\nb8smdc4550j2OFgHcCAQkGQ/v9//Iwvkl8pOU1jehmGIXD///PPk5eWJcjlgwIAmT9ppjHqG0tJS\n8d7k5uaydu1aqYceNGgQDodD1mJRURH33XcfAAsWLKC+vl7GecYZZ0hCYyST0MA6CFSTFbCebejQ\noSFtVadNmwbA3LlzcTgcUn8c7j3E4XDIfHbs2JEFCxYAsGTJEr7++mt5DzU1NWzZskUu8khJSRHl\nvmfPnkyfPp3rr78esGLITTnm4NyHcePGsXPnTrk2MTs7Ww5nv9/PoUOHpD1sq1atSEtLC7GGgxuM\n+Hw+SaSqqamR/IlI56X4/X6WL18u3jZ1wY4at9PplEThtLS0EM9QOGQjOk4CjUaj0Wh+40SlhRyM\nz+eTko+ampqQlPSGhga5Ueb++++nV69ezZ75ezT8fr+4VA3DkIzDe+65pzmHJfj9fmpra8Uy+/bb\nb9m2bZvUHge3kMvOzm6WTMny8nIuuOACwIpnqZjUKaecQufOneXig0WLFuFyuaQkp6qqiuLiYokx\npqWlhYy/sUV8ItaxslwCgcBxhTHUv4MfQhp79+7l8ssvB6wSLr/fz+jRowHr2sNIdeqqrq6WTngl\nJSUUFRWJC3vo0KFkZGSIq2/Hjh3SlUm1oBwzZgxglQRGy+1gjbtNORwOqYZwOBz4/X6Jz4a75Wdw\nmeGVV14pGegff/wxy5Ytk7W4f/9+SktL5frTiy66SEIAffv2Dbl9ran3P4fDIRay6oam3OV+v18+\nr7S0lOrqannnqqRLebE6deokcuvz+Vi7dq383Z49ezbbbX1+v5+CgoKQvAy73S5h0ZSUFPESBgKB\nsK+96Dy9/o/Y2FjOPvtsPv74Y8DasAzDEFfYmDFjeOGFFwDrNpFoc1ErAoEAr732mrQWdDgcPPTQ\nQ0DTF+//XJTLsba2lh07dvDFF18AVi/rkpISuTUnMzNTFr0KJUQSv9/PvffeS25uroxbzV1cXBzv\nvfee1O4OGjSIa665RuKI8fHxP3I1hevWITWfwX2eG3+maoQQXN/pdrsltnnVVVfJfb2GYdCqVStu\nvfVWwFKGwi3n6u/n5OSIq7ykpASPx8POnTsBK47Z+HYwhdPp5IwzzpDmG9FyGP8Uyq2qXKnl5eWi\nkEZizpWr/JJLLuHiiy8OUe5sNtsRDy0lS+EaX7DcOp1OBg0aJA2XNm3axNKlSwFrv2jbtq2syU8/\n/ZRdu3aJTMfHx4sc9OzZk7Zt28re0rp167DWex8Lm81GWlqauNoVajwlJSVyIKu5COcYtctao9Fo\nNJooIKotZLvdTkZGBo8//jhgFcGvWrVKsmsfeeQRKW2JJutYadrK9TV79mzuv/9++XnHjh25/fbb\ngeYfd/DlB++9955YNXa7nUmTJskdwykpKc2afOb1eqmrqxMrITY2Vm7l6d69O/379+eOO+4AoHfv\n3sTFxUV8vMHac0lJCfHx8eLSi4+Pl6Szuro6Dh48KG0oi4uLWbp0KRs3bgR+KL0AyzMxe/ZsSUqL\n5DMlJSXJhQe///3vyc3NDbkZKRi73S6NSh544AFuv/32iJaLnAgquS8mJgaPxyPtHvPy8iRpKdIt\nbYMt4p/67EjtISqrWlm2Q4YMkbDQihUrJBscLJnev3+/uKyzs7MZNmwYYF2AMXHiRAkhRbLjXGNi\nY2MZMWIE7777LvBDwq1aZ23atJFk1kisPVsk+rX+H7/4g4LdYoZhyCbXhC/xWH/ouMcdnEVYXV3N\nbbfdBsC8efPweDyyQX377bfSYewEOdq4j2vMpmlK3G/p0qXMnDlTXJLjx4/nD3/4g2Q/NqEw/qK5\nNk2T6upqKXPbv3+/KGbB2ZwQlg3quMYc3AXK7XZTWFgoIQCn0ym5EKWlpWzcuFEyxuvq6sQtCVaG\nqso3ePDBB0+k7ecJyYd6JrDct//5z3/44IMPACtm7HQ65Sac2267TeKHJ3izWpOsxeNFXfEKVgZ7\nRUVFSFz3f/7nf4Djis2e8Fw3A79ozME5D8roqKuro7a2NsS97fV6Rbns1q2buIGdTueJxOebVD6U\n4fTiiy8C8O677+J0Ovn9738PWNdIqlyVEyzJOq5/2CIO5Ahwwi9ZNdIAKyb46KOPirVZVFREfHy8\ntHI866yzmurQOOEDWVnIbreblStXysE2atQoXC5Xsx1uUcbPHrPP56OqqkpiwW+99ZZ87fF4KCgo\nkLiV1+ulQ4cOUud99913SwLPCVoPTXpIBJeuyAc0fQlIxA9k1eJz9OjR7N69W5TPmTNnSrvI47CQ\nfzMH8tE41lnSUuWjCTmuCdAxZI1Go9FoooCojiG3JIJjPoFAgNraWnHROBwORo0aJVm/zR03VgS3\naExKSmLixIlRM7aWTkNDA6mpqRKaGDhwoLjv4uLiQrLE7XZ7RLLAT5RoHdeJYLPZJJP6vvvu4y9/\n+Yt0zps8efKv7nnDiZ6rE0e7rC2a3A3i9Xrl1pi4uDg6dOgQjqSAX5ObDKJ33C1xzKDl42ehwk4q\njPMza+1/TXPdEscMLXfcgnZZazQajUYTBWgL2eLXpnW1xDFD9I67JY4ZtHxEkl/TXLfEMUPLHfcP\nvxTBA1mj0Wg0Gs1R0C5rjUaj0WiiAH0gazQajUYTBegDWaPRaDSaKEAfyBqNRqPRRAH6QNZoNBqN\nJgrQB7JGo9FoNFGAPpA1Go1Go4kC9IGs0Wg0Gk0UEMnLJaK5A8mvrftLSxwzRO+4W+KYQctHJPk1\nzXVLHDO03HEL2kLWaDQajSYK0Ncvan4RpmlSUVEBwOrVq0lNTQVg+PDhx3OZu0ajOU4atzfW1xz+\netEHcjMRvMhaygJTY66vr+fJJ5/kpZdeku8zMjIAWLp0Kb169Woxz6TRRAtqfdlsNkzTlO8Nwwi5\ni1qvreZFXc8ZjneiXdYajUaj0UQBLcpCNgwD0zRDtJKWpC0ahgHAoUOH+PrrrwE46aST6N27N3Z7\ndOtGgUCAw4cPA3DrrbeycOFC3G43AAkJCQwaNAiA9PT0ZhujRvNrwDRNDMPA5/MBUFlZidPpxOVy\nAeBwOIiNjQW0tRwJ/H4/YIXmrrnmmv/f3plHR1WdAfw3SyYJCZAESAgQCEhYhEYFRRERCKsFwWrw\n4MZSj1JLiwuWttYe9VStHrRFXEDQ01ZtZVNPZdMCLihV2SooqwFZAknYwpBk9nmvf7xzP2ZYLAoz\n87T39w+BJMx99917v/277N+/H4CSkhJGjRoFwKRJk2jWrNk5n+O2FsimaRKJRNi3bx8A8+bNIxwO\n0717dwAGDRpEdnY2gO0FGliuXYC1a9fyz3/+E4CtW7fywAMPyAazG2oxfvTRR9x7770A7Nq1i6ys\nLBHCY8eOZcyYMQBkZWXZ8pAwDINQKARY6yojI8OW47Q7yo3q8/lYt24d8+fPByzB0Lx5cwDat29P\naWkpRUVFADRq1IjMzEz5uVQTuxYcDoecHW632xbjM02TQCDAe++9B8DixYsJBoMMGzYMgGHDhpGW\nlgbYYz6/LYZhEIlEaGhoACAUComykZGRgdt9Qiyl6vnUOm9oaODOO+8E4I033sAwDJE5ubm59O7d\nG7DOPSWvwFKavsvYbSmQY2Mn+/fvZ8aMGQAsX74cv99Pz549Acu6VC8vLS0t7kWCvRaraZpUV1cD\n8Pzzz7Np0yYAgsHgKUkbdiESifDWW28BMHnyZI4dOwZAt27duP3227nxxhsBaNq0qSRy2WnO1eao\nqalh27ZtfPjhhwCUlZXRt29f2yWfqXVgmqZYR+FwmLS0NBnrd93o52t8KpFv2rRpLFu2TKyFvLw8\nEbqhUAiXy0Xnzp0BmDhxIgMGDABSM37TNOXwnzNnDk8++SQHDx6U76m5vfnmm5k9ezYZGRlJHZ8a\nh4pNhkIhXnzxRZ5++mkAvF4voVCIhQsXAjBmzBj5XnZ2Nk6n01b7Lha1piORCHV1dQBs376dVatW\nydmyc+dOrrnmGgAeeeQR2rVrl3IDq76+HoDbb7+dBQsWAJYAHjBgAH369AGgZcuWdOnSBUBi/uc6\nbvublRqNRqPR/B9gOwtZxU/A0lI2bNjAxx9/DMC+fftIS0tj8+bNgGVpDho0CICcnBwKCgrIysoC\nIBAISDwzLS0Nj8cjFnSsJpMszTIcDos1sW3bNtEWDx8+nJTP/7aEw2EWLVrEk08+CVjeiiuuuAKA\nP//5z3Tu3FksCbto57GelWPHjvG3v/0NsOb7q6++EjffHXfckTQNXI1JWT9gzW00GhUr2O12E41G\n+eKLLwB46aWXWL58OQDHjx/H6XRy6aWXAjB//nxxDSebcDjMvHnzAFiyZAlOp1Msm8LCQqqqqgDY\nvXs34XCYFi1aAMRZPKmwjmtra/nxj38MwMaNGwmHw3GetWAwCMCCBQsoKSnhd7/7HZDcMFhsjsa6\ndeuYM2eOnA2xljPA0qVLxftwyy23kJ2dLSEvt9ttG29VNBpl586dAEyZMkWeJxAIcOzYMSorKwHL\nelbeq+3bt1NUVJRSC9nn83HfffcBsHDhQnJycgB47LHHKC8vl7mur68X93VaWlqcp+K7zr1tBHKs\nu04tvPXr17Ns2TJ5kcqto9xNb7/9tgi53NxcXC6XTNDx48dp2bIlAAMHDqRFixYcP34csA4IVaaT\njNitYRgYhnFaV1g0Gk25eyYW5eb97LPPePPNNzly5AgA/fr149FHHwWsGKFd4m1wwsVbU1MDwFtv\nvcXChQvlHRcUFNCnTx9++tOfAtC6deukjV19jtPpFAFcVVXFv/71L7Zs2SLjj0ajrFy5EoD9+/cT\nCATkewCffPIJYB0KTz31FEBSXe6GYbBv3z6JGaenpzN69GiJoZmmKfsrOzubnJwcSkpKACsumKo1\nXldXR79+/di+fTsAjRs3ZuzYsfz85z8HrIRE5f79y1/+wty5c5kyZQqAxDWTgYnqpqQAABflSURB\nVGEY4iZdtWoVhw8fFkEcm8iqnmnZsmWAJTwyMzO56qqrAOu9NGvWDLDWvcfjSeo6UevV6/Xy29/+\nljlz5gDWOde4cWMAunbtSqNGjeLWhFLerrjiipSGksLhMEuWLBFlPiMjQ3JnJkyYgMfjEYMxNvxy\nvsIG9pEEGo1Go9H8H2MLCznWTR0Oh9mxYwcAb775JuvWrRNXjsqOTE9PByxtUCUa5ebm0rJlS3FF\neTwe8vPzActKqauri0vyKSgoSMpzgaUdOhwOeQ6fzyfPW1RUZJvkIsMw2L17N2Alv2zcuJHLL78c\ngIcffpj27dsD9spGBaitrWX58uXi9nI4HJSXl0vyhcfjIS8vTzwmqbDWYkv0/H4/FRUVfPnll4Bl\nybjdbrESCgoKZH00NDSwd+9esa4rKirk62Sum1AoxFtvvSWJiVdddRXNmzeP24tNmjQBoE2bNjRt\n2lT2YirWivKy3XPPPWzbtk1CWatXr6akpETmzjRNHnroIcCqJKirqxNLNRkWslrD4XBY3Ltffvkl\nfr8/zmvocDhkzFlZWRLy+uyzz/B6vbz99tuAtV5UAlskEqFHjx5MmzYNQPZvoohEInJ+lJeXSwgG\noFmzZhQWFsrXe/bskfM4JydHxtikSZOUrBe1p3bv3s1zzz0n3sybbrqJcePGAda5FyurIL6Zizrn\n4bvvTVsIZDixgWpqanj55ZcBK92/qqpKXpyK/So3c5s2bRg8eDAAffv2JRQKSdw4Nzc3Lm4Vm025\nYcMGSktLE/5M6sWpl6Vi3z6fT8Zy/fXX20K4wYlDF6yF2bFjR2699VYA2rZtm9ID9mRM05SD8w9/\n+AP19fUSz+zXrx8ej0fGWVtbKxnAqSL2QHW5XOzfv19c6l26dKFjx46iQGZmZorytmPHDhYsWMCe\nPXsAK6NdubOTkQ2s3Kaffvops2fPlsPe6/Vy4MABDhw4AFg1mRdddBFgHaqpVNpM02TFihWAFXN3\nOBw89thjAHTq1ClOIXM4HJJbUFxczCeffCJhmhYtWiT8GWIrLJSQPXn+1J/qPAkGgxK2q6+v59Ch\nQ1JSqfIT1O9VV1fLefmPf/wjYc8RDodZsGABDz74IGDl+2RlZUl2/cCBA2VeV65cyZEjR2T9Pv30\n0/Tt2xdIjbIcawS+8soreDweLrzwQgB69uwpMiUSiRCNRiXnID09Pa787Hz0xbCFQI5Go7LYKioq\nJF5WU1NDJBKJ00gAObhGjRrF0KFDAUvLMgxDeiq7XK64ulO/3y8xRq/Xm/CNpuLGYL2scDgspU7h\ncFjiPH369LGNgNu3b58oDa1bt6asrIyrr74awHZ1u+FwmNWrVwNWvsCoUaNEOfN4PESjUXbt2gVY\nSWjXXXedxNmysrJSZiWDZSF0795d1mdZWRlFRUWy8aPRqFii77//PgcPHhRhXlBQIIdAojFNU3I0\n7r//fo4ePSqH0fr161m3bp0cqmVlZVKXnmoMw5BYdzgcpm3btowdOxY49cA3TVOUis2bNxOJROQZ\nk7neMzIyJHFPjXnRokWAlXNQX18vgrahoUHG6HK5xGBRz3Ny4ySl3CUCNaYtW7bwpz/9SRSFiy++\nmGHDhjF+/HjAMkKUQrB3716ysrIkp+OWW245pWQ1WUQiERk7WHNWWloq1vygQYNkv0UiEQKBgKyh\n9PT0U5IVz3XN6BiyRqPRaDQ2wBYWstPpjPPFq2zN01kxTqeTH/3oRwBcfvnl4vbLyMiQ1HP1c8pV\nozKZlVWak5MT93mJQDWEV19XVlZKKYtpmuISUXHDVBONRvn000/Fi9CvXz9GjhwpcTTl9lekylpW\nXod9+/bxn//8B7DiPL1795b3HQ6HeeKJJ3j++eeBE+VzSlsfMmQIrVu3lvjnyS7MRKH+b4/HQyAQ\niMsxaNy4sXw/EAjwxhtvAFZ3oIaGBjp06ADAbbfdljT3eyQSEVfvjh07iEajMscul4t27drRsWNH\neSblkcjJyaF58+Ypy42IRqPiWs/KyuL2228/o3vf5/MxadIkwGpQkZeXl/BYayyxe0pZZeXl5ZSX\nlzN58mQAfvOb3zBv3ry49aLWrsfjobCwkE6dOgFWFz1ltR48eJD09HTJdk/E2JV34cUXX+To0aNM\nnDgRsEoL27VrJ5Z8dXW1uIUdDgeTJk2S8jL1LMlEzWV9fT0zZsxgw4YNAFx22WWMGzeONm3aAFYe\ngTpzQqEQmZmZsgcS0ZDFFgLZ4XDIS7n44ou57bbbAKt+VMUbwAqqFxUVcdlllwGWq1KVRDVp0gSP\nxyMCJFY4gzWxqp5s8ODBCRcoLpcrbgMtXbo0ruZYudpT5apRxJYpvPvuu6IMtW/fPq5G0+Vyxbln\nnE5nSmpL1Vp47bXX5PN79OiBy+Vi7969ANx111289957kqihBNhXX30FWO61CRMmcMEFFwBWKUyy\nS4hik1q++uqruLrdzZs3M3fuXMCKf1944YXcddddgFUykqz5jkajsmcuueQSunXrJm7Vnj170rJl\nS1kfu3fvZu3atYC1L4cPHy4liMnGNE1xn3u9Xvr37y9rRXVUUjWwEydOlL7y0WiU4uLipHXqinUv\nO53OU9agSkJ88MEHWb16tbQQjo1jZmdnM2LECK6//nrA6rfs9XoBq6Z3x44dIlzON4ZhSGjlwIED\nlJeX88ADDwDWnootYV2zZo383qBBg7jnnntSmtehzr1jx47FhQMuvvhiUYzAUtjU9+rr62nRokXc\nuXe+96J2WWs0Go1GYwNsYyErS7Fp06ZSmO/3+1m1apX0cE1PT+fCCy8U98zatWv5/PPPAUt7ady4\nsST2FBcXy//p9/tPG4BP9DPFfk6sxebxePjJT36StLF8E8odM2/ePFasWEG3bt0Aq3nK3r17xaoP\nBoO0atUKsJKSTNOUEEBmZmZSkqRM02TdunWA1a1IZVVv27aNjz/+WEonamtryc7OFk/KfffdR6dO\nnSSLNRgMUlJSclqXdSJR7zoYDFJVVSUuXofDQSAQkPldu3atWB5ut5thw4YxevRogLjs8UTj8Xgk\na1Y1tlFWnJozZT24XC65DOHVV1+lsLBQkuiSnUDndrvFYvR6vezfv1+8J36/n7lz58qZUltbKyVR\nTqeTm2++OWleq4aGhm9cg+rf2rVrxy9+8QtmzZoFWNnY6veuvPJKrrvuOtmLZWVl/Pvf/wasxLv8\n/HxGjBiRkPE7HA7atWsHwKOPPkp+fn7cZT+RSETc1B9++KF4gEaOHEleXl5Kz77YS1KaNGki2eAl\nJSUcPnyYrVu3AnD06FEZd48ePU5ZG+c79GkLgRxLrGvU4/EwZMgQKVEKBoNxtwl17dpVXCF//etf\ncTqdcuiq2BZYGcOXXXZZyhbA8ePHRZCA5YpSCznVqHaHs2bNwuVySeelnTt3smbNGukIdMEFF0ib\n0uLiYjp27CjZm8XFxdKFJ5GuX8MwpNZ4x44doiy8/vrrHD58WBSeQYMGMW3aNImdxbZMhRMu91Rd\n+N6oUSN69OghbsedO3eSm5sra2T37t1ytWXr1q0ZMWJESm5LcjqdcsCe6XPVXm3ZsqV8vXr1ap54\n4gkpoUt2jNDpdEq1RbNmzVi1ahUzZ84ErHBFQ0OD5J5ce+210pq3pqaGXr16JU25nDp1qrjWx48f\nf8augQ6Hg7KyMnGlZ2dnS4lZNBrF5XKJm3r+/Pl89tlnABw6dIiBAwcmzGXtcDhkHpVCoNaJcler\nbPfq6mpxX/fq1SvlvRdUuCgtLY0+ffrIOt+zZw/Lly9n8eLFgBUKVS2Di4uL49rWntxB7XxgO4F8\n8s1HsQ0T6urqyMvLkzT0aDQqi/TYsWMcOHBAalP//ve/y4E3evRoLrnkkmQ9gqCepaKigtraWtno\nw4cPT1rpyjcRjUZ55plnAKisrCQ3N1diyDNnzmTnzp0SQ7z66qvF6sjNzcXv90s7wuXLl4uGqVqY\nJgLDMMSa8Xg8Ek92Op2UlJTw61//GrDi87G3f50uIS2V2nl2djYPPvggX3/9NWBp6Q0NDZL0t2/f\nPpnDkSNH0r1795SN92w/1+VycdNNNwEwY8YM1qxZw6FDhwASJhDOhMPhkBKynj178vnnn4uylpub\nS0lJCY8//jhg7VF1+LrdbvLz85My16ZpsnLlSpYsWQJY59cvf/lL4NQSw7S0NLp16yYKZuz+Onbs\nGNu3b5frXN9//31JtCosLKRjx44Ja3ASW1uv9pjaZ6FQiL1790qjkMrKSjmDlbKUStQ4A4EAWVlZ\n0pTlnXfeYdOmTVLXfdFFF0lCZU5ODpFIJKE3r+kYskaj0Wg0NsAWFnJsO7JQKCTarMPhwO/3i9WW\nn58fF0NzOp3SJGTcuHEcPXqUF154AbDcfqpIfciQISlJrVfPNH36dAzDkDHceeedKY8dgxVfW7p0\nKWDF1goLC+WCg5qaGkpLS3nuuecA6NChg1ickUgk7qKBI0eOiDvqhhtuSJiF7HQ6JZZqmqa4dbOz\ns5k4caK4zxwOxynt7QzDsM0tOMoVrOL10WiUUCgk63XOnDnivvzZz35Genp6ysd8NihLOD8/n6qq\nKmlrm8zLPBTKA3XppZdSWloqLsqmTZvi8XjEW7VlyxZx96ob45JBKBSipqZGLLEnnnhCrMmHH36Y\n5s2bx4VUXC6XhC0Mw5DyxKVLl/LGG29I6C62I2HHjh0ZPnx4Qt3Dse81Nqva5/MRCARkbg3DEAs/\n1d5BwzDk7Pjggw/YunWrxIx37NhBbm6ueAOvuOIKunfvDpy4kUt5bBMR2rCFQIYTvUQrKyvlFhyw\nBIWKT8Smo4O1AGLdkhkZGXKoVVVVyYQ1adIkJTEL1eLwgw8+AE64alRSWqpQ7pqtW7dKS8ZAIMDX\nX38t/YhbtWrFpEmTJNHIMAxRjLZs2cIzzzwjsarMzExGjhwJJHazuVwu2dQPPPCAJBQ5nc64ZIto\nNEokEpGxGIZxSonJyQdJsgXGye4+n88nN8zU19dLDkQqL2v/tnXn6ueV8FDhjlSg5kzVd588fjXW\n6dOnyz7Nz89PmrAIBoNEIhFRFLxeL6+++ipgJfVNmDBBYpfNmjUjFAqxbds2wLrN7P333wesWuNQ\nKCQKqNvtlvyUm2++mQ4dOiQtJm4YhuzJSCSC3++X8rK0tDRZ06lWLqPRqPT+fvPNN/F6vbRt2xaw\njKVrr71WFOJdu3axfv16wFI4O3XqlFCXtW0EsmLbtm0S0zl+/DgFBQWyMH0+3yn1xQqfz8crr7wi\nSUiRSEQWwNChQ5MukE3T5N133wWs2LfT6Yy78i2Vi1Jt3i+++CLOklSF72D1rt64caPUaG7atEmE\nt9frJRQKiXfi7rvvlqzxRG/+2Pd4crLWmf5Uh4OKpSXjys1vy549eySpyzRNySBPhTWhDlW/3y85\nGv8r89g0Tcmora6ujrviNBVrPfYzT9cuU+WaqD0KlvKTrAzrrKwsOnXqxMaNG2VMqpnJhg0bRAjE\njvlMqCRFsIwPdZfvsGHDEr5+Yvda7Bh9Ph+bNm0SJb6wsFAaOqWyxzlYd2I/8sgjgBWvHzNmDHfc\ncQdg5RgYhiFNh2bNmiX5CP3792fAgAFyfiTiOXQMWaPRaDQaG2ALCznWfVdQUCAZb5s3byY7O1s6\nLFVVVVFWViZaX05OjjS/f+GFF1i4cKG4vlu2bClZn23atEm62y8cDosLKhwOk5OTIxddp9plozTF\nrKwsOnfuDFj1mEqbBUtL/+CDD8SdF3u1WGZmJldeeSW///3vAauMQcXHU/lsyrJTVr/68/jx4wQC\ngbgYXCLa3p0LGzdulLiW2+1mzJgxQGpuv1HW4/r16yV+9r9uPqqvr5duYi6Xi2uuuSZpHa9Oh1rj\n6l2f/D0Vcz106JB4Tn71q18lbU243W5eeeUVaee6efNmqQo5+TKd//X/pKWlUVxcDMC9994rnQ4T\nlXsQO7fqvFVfq2d45513mD9/vvy9f//+kmOQqn2nwgOxXROvvfZaysvLxZsTjUZZu3atXHxRWVnJ\nqFGjACs/Jjs7O6Hjt4VAhhMusU6dOklt3q5du6iurpZa2TVr1vDiiy9KnNPn80lphdfrjSuDGjx4\nMHfeeSeQGrdfZWWlFOirRgWqZCfVqANqwIABcS6n3bt3i8tf1fWq+mI4kbQzefJkrr/+epnrVAs3\nlcSlDjLVu1z9va6ujoyMDEk4ib0yTZHo3ubfhGEY7Ny5Uz67sLCQrl27Jn0cJ7Nq1SppXnLrrbee\ncsCbpimlZ5dffrk032jfvj1Tp05NWexbjQ1OzR8AqyHH7NmzAUt5UL2rk11e1q1bN0mifOmll+Ia\n28TegAfxRktWVpaEi/r168fAgQMpKysDLJdrIt3u0WhUFF91BSdYOSg1NTW89tprgBUO83q9DB8+\nHLDaf6a6TbDCMAxxOzc0NLBnzx4xAtesWcNrr70mse9LLrlESkNzc3MTvj60y1qj0Wg0Ghvg+KZk\ngfPMWX2QaZqSKv/ss88yc+ZM0cJV4sCZLJumTZty4403AvD444+LJX0WWs03/cC3niDTNFm2bJm4\nPaLRKIsXL6ZXr15nO56z5Uz/0VnPdawWrqwJsMYYjUbjEjWUZXmO2YXnda7lF2NKLoLBIE6nU/6u\nvCitW7cGTrT6jO0qpJ7zDFZdwsYMlqXRp08fsUZvuukm5syZA5xz57PvtD5UmGLJkiWsWrUKsDq1\njRgxQvZUQ0MDM2bMkHaOPp9Puhh99NFHdOnS5XyP+X+OO+4HYy52MU1T3mskEmHRokWS+HT06FFx\ntT/22GPnkvB3TnsxEolIUldtbS2BQECqMjIzM6mrq5NnysjIkPcQW8KV6DGbpsnBgwfF3b927VpZ\nK59//jm7du2SKpfS0lJGjBghzU4aNWp0vs6977w+1FlXUVHB3XffDVjJqoZhSPmZcsGrsqdnn31W\n3sM5jv+sftkePoQYlGAFmDJlCr169ZJa2I0bN1JXVyeHVOPGjSkqKgKsBTB16lQp00mle0S9eBXX\nadOmDZ07d7ZVzBLi3WBw6uGf6nrBb0vseEOhkLT2rKmpoVWrVmfMjkzVe1GuvxkzZrB161ZZs1dd\ndVVK3b1qnnr37s2mTZsAqzb6qaeekoPL5/MRCoWk5eCECRP44x//CBAX5kgFsQqWaZoEAgFRyt55\n5x1mzZol+RK9e/dm6tSpQGrPDLfbLefe6TpZpbKETBEMBqmpqZEuZ9u3bxclwu12x/WPf+ihh+jd\nu7etzhC1pzp06MCTTz4JwIIFC6QHA1id3UpLS2W+k70PbWchn0xs4kBDQwNer1e0sKKiIpm4jIyM\nRGiK8B3H7ff7pZ66pKQk7r7b88g5aeUpIiHWJpxQhEKhEMFgUNZJZWUlfr9f2nue3JrwLEjImFXC\nS58+fVi/fr0okxUVFefrarpzWh+GYUhv+Jdffpnp06fLAdy4cWPKysq4//77AejcufP5Ki08rxay\numrx9ddfB6wDuLq6WgTF7NmzZV2co0D+Ie3F045ZNfq44YYbAKvFq1J68/LyGDt2bNwdxwlSdBN2\nfiSYs5oMHUPWaDQajcYG2M5lfTJOp1NKatLT08nLy5OsSDuTmZlJz549Uz2M/yuURu52uwkGg5KL\nsH79ejp37pxSN/DpUJ6fiooKAGmAk4o2r6cj9tak++67T+Ku3wdis6wzMzMlfBQKhcjNzeXWW28F\nLO9Eqm8e+r6gvJDqSk5A4slDhw6lR48etnJRfx+xvcs6SfzQ3CDfxzHDeR73yfXI59DLOqEu6/Hj\nx7NixQo2bNgAIHkR5wG9PpLHD2muv3VoIIl5GD+09RGHvUwGjUaj0Wj+T9EWssUPTev6Po4Z7Dvu\n7+OYQa+PZPJDmuvv45jh+zvuEz+URIGs0Wg0Go3mDGiXtUaj0Wg0NkALZI1Go9FobIAWyBqNRqPR\n2AAtkDUajUajsQFaIGs0Go1GYwO0QNZoNBqNxgZogazRaDQajQ3QAlmj0Wg0GhugBbJGo9FoNDZA\nC2SNRqPRaGyAFsgajUaj0dgALZA1Go1Go7EBWiBrNBqNRmMDtEDWaDQajcYGaIGs0Wg0Go0N0AJZ\no9FoNBoboAWyRqPRaDQ2QAtkjUaj0WhsgBbIGo1Go9HYAC2QNRqNRqOxAVogazQajUZjA7RA1mg0\nGo3GBmiBrNFoNBqNDfgvmTYlgjgWm00AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8465cb9780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,50)) # not shown in the book\n",
    "for iteration in range(n_digits):\n",
    "    plt.subplot(n_digits, 10, iteration + 1)\n",
    "    plot_image(outputs_val[iteration])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure generated_digits_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEGCAYAAAAwpAFeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4VFX6+D9T0zuppBIgoRmQpoiKVFdEsaNrxa77yNe6\n6hZ1ddfeXdeOq+LaAVEXsFEFadI7gZAE0pNJMn3uvb8/5neOk5AgZWYI7P08j488kMw9c09563lf\ng6Zp6Ojo6OjodDWMx3oAOjo6Ojo6HaELKB0dHR2dLokuoHR0dHR0uiS6gNLR0dHR6ZLoAkpHR0dH\np0uiCygdHR0dnS6JLqB0dHR0dLokuoDS0dHR0emS6AJKR0dHR6dLogsoHR0dHZ0uiS6gdHR0dHS6\nJOZjPYAA9KKAOjo6OicehiP9Rd2C0tHR0dHpkugCSkdHR0enS6ILKB0dHR2dLklXikHp6HQZRJ80\ng8HvPlcUhY0bN5KVlUVCQgIAVqv1mI1PRyccuFwu+WeTyQSAxWIJ2/N1AaWj0wmaptHS0gLAn//8\nZz788EMSEhL48MMPARg2bJgUYMcKRVHw+XyAX5haLJawjEnTtE6f074J6rF4R5qmoaoqLpdLzmFU\nVBSRkZFYrdZjPm9dGU3TaG1tBeDtt99m7969TJo0iVNPPRUAs9kctvenC6gwoGkadrud//znP8yc\nOROA1tZWCgoKyM3NlRMfGxtLv379SExMxGj0e1+PdiF4vV7279/PDz/8wNy5cwGIjo7G5XIxYcIE\nTjrpJAD69OlDREREl9m4mqYdYMWEc2wGgwGDwUBkZCQAF154Idu3byc1NZXk5OSwjeNgqKrKyy+/\nzMsvvwxAQkIC06dPp6SkJOTPPthcKIoC+LVvk8mEyWSSWnco51AIJYCmpibmzp3Ls88+S01NDeBf\n93//+9+ZNGmSnNeuiKZpuN1uKisrAVi+fDkTJkwgJSUlLHvA5XLxySefAPDzzz8TFxdHfHx80M6k\nw0GPQeno6OjodElOSAvK6/XS0tJCbGxsWDS338Lj8fDoo48yY8YM6Y6JjY0lNTWVffv28corrwBQ\nUVFBnz59uPvuuxk0aBBw9P5et9vNF198wapVq/jll18AiIuLo7i4mAULFvDee+8BMGLECB544AGi\no6OP6nlHgtB8hUsGYP/+/TidTnw+H0lJSQDk5uZiMpnCOpfC756WlkZycjJJSUlERESE7fkHQ7wz\ngdfrle8q1HTm4tM0DafTCfjXs81mIykpiR49egChdQ8Jqxdg165d/OUvf6GmpgaPxwP45/K5554j\nNzeXk08+GQhvPKUzNE2TY9y+fTtPPfUUs2fPxm63A2A0GsnNzeXDDz9k2LBhQOjOM5/Px08//cT0\n6dMB2Lt3LwUFBXIc4eaEElBiY3z33XcsXbqUs88+mxEjRgAclm/+YP71w0G4Oh577DHefPNNPB4P\nMTExAKSnpxMfH099fT2NjY0AlJaWUlpayoYNG1i5ciVw9Bs6KiqKSy+9lFNOOUUerIWFhZjNZnbv\n3s3DDz8MwNy5cznnnHMYPnx42BaicOH5fD6cTic2m02+i/nz57Nu3ToqKys588wzAbj11ltJTU0N\ny9gE4t3X1dXJ9ZWYmNjm344VBoOB5ORkHA4HAN27dw9Z4kbgnhDz1tE+CXSz/fjjjyxYsIC+ffty\n7733AqEXCGI82dnZ5OTk0NDQIMdrsViora3l/fffJyUlBfDvhWM1j5qmoSgKVVVVPPLIIwB8/vnn\ntLS0oChKm/dcXl7O5Zdfzvbt2wH/uRBsFEVh1apVPPnkk1I4nnTSSZxxxhmoqtpG0OsxqMNEURRm\nzZoFwJtvvklmZiaVlZXU1dUBkJqaesiHfbBefm1tLQCvvfYaLS0txMTEcNFFFwFwxx13kJaWhs1m\n46effgJg2rRpNDc3U1VVxX//+1/AH/s4mvEYjUbS09NJSUmR1oD4f1FREXfccQcA9913H/Pnz2fI\nkCFhE1DiINu5cydz585l5cqV7N+/H/AfMK2trWzdupWdO3cCMGDAAC644IKwjK09BoOBiooK3G53\nl8rea25uxu12A36rXGQYBhuDwSDny2AwyMNT/J1YM+LQBVi3bh2rVq3C6XS2sfRCidgraWlpPP30\n0/zrX/+iuroagJqaGtxuN7t27WLVqlUA5OTkhMwibp8somkaPp9PvgubzcYnn3zC448/Ls8KRVEw\nGAyYTCYphLxeL6qqUldXJwVU3759gzZOMZ6NGzdy3XXX0dLSwhlnnAHA1VdfTWJiIrW1tTQ1NQEQ\nGRmpC6jDpaGhgffffx/wL86pU6dK0xT8ri4x8R1pfYJgvXhVVXnggQcAaGxsxGq1Mm3aNO655x7A\nP8kmk4mEhATS0tIAeOGFF1izZg1Op5M9e/bIcUdFRR3xOAwGA0ajsUPhbDQaycnJAfwCweFw4PP5\nwub2EO68//73v6xYsYL6+nrpehkzZgw1NTXs3LmTqqoqALlBjgXicAkU9F0FMZ5wjU0caOIwhV/d\na2K9gd/qtNvtOByOkGj8B8NsNlNcXEy3bt1kRlqvXr3weDzEx8dLC6q9EDkaAi0en8+H1+uVLn1N\n09i2bRtLly5l7dq1AJSXl7NkyRIp0ME/l927d+eWW25h8ODBANx4441UVlbi8/l48MEHAaQyHowx\nC+F45513UldXx4ABA3jooYcAyM/Px+v1UlpaekxcfHqShI6Ojo5Ol+SEsKA0TWP9+vVSI7/lllsY\nOnQoVqtVxg3Ez6mq2mHacvuU5qOlvLxc3pfRNI38/HzuuusuGYMKfI7QLkUaqclkknGOYLofvF4v\nQBstW2hPNTU1YffFi/GkpqbSrVs3Tj/9dMaNGwdAt27d2LRpE62trXJuhKV5LNi+fTuKolBcXHzM\nxtARTU1Ncv0cjaV9KATuG1VVO7U+xJ7bsmULzc3NeL3eY5KMEBUVRbdu3WRcMykpia1bt9LU1CQt\nqGBaduJ9lJeX89prr1FWViY9FOvXr2fLli3U1NTInxOuO/g1Njd8+HD+8Y9/MGTIEDm22bNnM2HC\nBJqbm4PuXna73dLNv2HDBvr378+rr75KYWEhgLwmkJ+ff0yue5wwAqq6ulouun79+mGxWFAURS7O\n5ORkjEYjmqbJSTYajRiNxjbZP8HA4/Ewbtw4eQBHRUVx//33d3j3QlVV6QYQCQAGg0G6J4MxLvEZ\n4uBwuVwkJiaiqir79u0D/IIwPT09rAeJeB9Wq5WGhgaKi4ulYHY4HHzwwQe0tLTI8VdUVIRtbAKh\n9Lz++us4HA5KSkrCFk/5LXw+H4sXL5YBbZvNFtLnBSZJGI3GNoqOSJhQVVXGESsqKlAUhdNOO+2Y\nuIdMJhMXXHAB77zzDgBr166lsrKSwsJCufaCue/FPEydOpXVq1cTEREhv7fb7ZaxuMBYcEREBAkJ\nCdx+++0A3HDDDSQmJrYRREVFRQwZMoT58+fL8ywYqKrKnj17WLFiBeBPiHj00UcpLCxsM7dGoxGr\n1RpUd+ihcsIIqIaGBs466yz5d3a7nbq6OnlJLzk5GUVRpEYABF0wibHMnDmT0tJS+Xd5eXkMGzas\nTYaTeL6iKFKQBY4n2JliRqNRCkKXy0VTUxNVVVXs3bsXgIyMDK666qqwHiRiwZeWluJ0OlmxYgW7\ndu2Sf7d+/XosFou0IgPLroSLr7/+GvAftiUlJfTv31/+W7CyPY8Ur9crL3MKwjEekSShKEobC0BV\nVZxOJ++++y7gn6/o6Gh+//vfH5P3ZDQaycrKYtu2bYA/acNkMuHxeNqcAcFA0zSeeuopAH755RdU\nVSUtLa2N8tna2kpCQgK5ubkA1NfXoygKkyZN4sorrwT8ymxgwgn4FZHc3FwiIyOlRRYMXC4Xr7/+\nulRKx40bR79+/TqMY5rN5oNmb4aKE0JAqarKqaeeKg+y3bt3s2rVKlpbWzn99NMBv5ZusVhCXqbD\n5XLx97//XQpDgHPPPZeEhAQURZEpwfX19cTFxZGQkCAXpMfjwWw2k5ycTF5eXlDHZTAYiIuLA6C6\nupo9e/bw9ddfywzC/Px8cnNzw7bwNE2TGv+GDRtYu3YtVVVVcozFxcX07t2bqKgoqZEvXryYW265\nJWxVADRNY8mSJYDf2isuLj7mqeWBqKpKdXW1XD+BbqFw4PP5aGhoAJDVNebPn8+MGTPk+Lp16xbW\nddWewOd6vV4yMjI4+eSTQ3JVQCgvvXv3xmAwkJ+fT58+fQC45JJLiIiIIDIyUiZFLF26lPLycs4/\n/3w5h4qiYDQa26R1b9q0ia1bt5KUlMT//d//BWWsiqKwdu1atm3bxoABAwA4++yziYyMbFPFBX69\nb9c+mSwcc6onSejo6OjodElOCAvKYDDQu3dvmY48b948Nm3axOjRo6V2IO4VhErqC+1iwYIFbNu2\nDYPBQHx8POBP2khJSWlTKaGmpka6QEQxS3E3o2fPntKSCCZCu05MTGT58uV88803MnVbWCvhNN+F\n1qiqKhaLBa/XS/fu3QE4/fTTGTFiBPX19UybNg2AlStXsmDBAs4+++ywjE9RFGm9paWlkZeXR1VV\nlZzXztL3w4XdbsfpdMp5HT58eFjHYrFYZKykubmZTZs28cgjj0jLWNM0evXqBdDmDpWgo+sewRy/\nqqrs3btXelYuuugihg0bxoABA4K+v1RVpWfPnoDfY5KRkUHfvn3l3yUnJ8vvJiwjsferqqraJLq4\n3W4URWHDhg0AvPrqq+zdu5eioiJ69+4dlPE2NzezYsUKoqKi5BiNRqNMlhLvrKWlhbKyMnw+n7T8\nioqKyM7ODku8+oQQUOB/4cuWLQP88YuSkhJGjx5NbGwsEHrfvLgs+Yc//AGfz4fZbJZVGgIzYDIy\nMoBfEyJUVWXhwoWA3+0HMHjw4JBeBvV4PLS0tBAdHS2FxLnnniv9zMHOaOwMkXWWm5uLoijk5uZy\n2223Ab9WRUhLS+POO+8E/JUk3nvvPcaOHRsWV1ZVVRXNzc2A/xJsaWkpCQkJZGVlyfEHJtqEm9LS\n0jbxlHBW2RCV00ViUl1dHZs3b2b//v1SGEVERNC7d29aW1vle9Q0jYSEBEwmk1zjoahMIOLS1dXV\n3H333YBfSJjNZpqbm4P+PIPBIDM8KyoqyMrKoqCgQJaeClwjwkV98skns3XrVr766is5h1dccQUR\nERFs3LiRJ598EvDHtOLj4/n9739/1OeCmJvt27fj9XrJycmhX79+gD8jdObMmezcuVOemzabDbfb\njcPhkIp0SUkJf/rTn+Q+CCXHvYDSNA2Xy8Wbb77J6tWrAcjMzOSSSy4hKytLXtKzWCwhk/iqqjJn\nzhzAX0POYDDQs2dPbrrpJqBtDSuxSEVGoaIozJ8/H/BrxNHR0Vx11VUhGafAZDLhcrmIjY2VQkII\nznAKKLEpMzIyGD58OGeffbYcj3i22Wxm7NixgF9z27VrFw0NDb+Zcn602rimaaxZs0ZaSw6HgxUr\nVhAbG8uYMWPkGI9lmZw33ngDVVVl/CfcafjiSgT4D//TTjuNHj16yGy2vLw8Ro8ejdFolJa62+2m\nrKyMtLQ0qWRkZGQc0AKjsbGRbt26HfaYxAHc2NhIdXU1eXl58r34fD7q6+vlQXswxD441NRqsZ8B\nVqxYQVZWFn369DloBXCj0YjD4eDbb78lMzMTgB49epCRkcE333zDpk2b5M9OnjyZSZMmHfV6E9bb\nzz//THZ2NuPHj5fCaNOmTaxfv55169bJ86Bv374MHDiQjIwMli9fLn93w4YNZGZmhnz9nxACqrKy\nktbWVs477zwALr/8ciwWC3a7XaYmi4MmFPh8Pr755hvg1yDnsGHDDknbcblccuINBgO5ubkUFBSE\nZOIDMwRHjhzJtm3b2lSS6CizKZQuP3FQeL1exo8fT3R0dIfPEopFQUEBK1asoLW19TcP46Mds6Io\nVFZWSq145syZtLa24vF4pPujfbZVMJ57qPh8PpYuXYrZbGbo0KFAaNf4byG+d6ALdOzYsQwdOpTY\n2Fh5CPp8PlpaWoiLi5PZq+I9Bh7yhyucRJHadevWAbBjxw569uxJamqqFFr79+9n5syZZGVlySoN\nh/q9DnUMAN988w0jR45s830CU/QDM3kXL16M1WolPT0d8B/+dXV1bNmyRb6fpKQkJk2aRGxs7FEr\nXUJR8Pl8FBcX0717dym0ysvLiYqKYty4cVx88cUADB06VBaQHjlyJOAvf7RgwQLGjBkTck+GniSh\no6Ojo9MlOe4tKPD7fC0WC5dffjkAMTExqKpKWVmZdL3dc889IdNuFUWRmpuqqpjNZnkT+2B4vV5e\nfPFFtm7dCvgthbFjx4a8nYPdbuenn34iPz+fK664AvC7h4R21/4uVLCtKE3T8Hq98g7WgAEDDsna\n3Lt3r2zBEWqEO0jENSsrK0lKSmoTtxMu2mMRg2psbJRJNqNGjQKOTTuEwMKn999/P3V1ddI9dP75\n55OcnHzA3MbFxck1AP5xe71evF7vEV0hUFUVm83G3LlzZUPQ+Ph4ysrK+P7779m8eTMACxcuxOVy\ncdZZZ3HZZZcd9DOPxH0rrpBs3rwZu93OPffcI5N+xGepqiq/9+bNm0lJScFoNMr6fHV1dTQ0NGA0\nGmXh33vvvZfTTjvtqEIUqqridrtlsVm73Y7H48Hr9cr3U1NTw+TJkxk+fLh0OYr4oKZp0pWclJQU\nlj0IJ4CA0jSNHTt2MHz48DbxC7fbzX/+8x9ZyTiUGScej0cmOGiaJm+Ii0kM3KDC5Hc6nbzzzjs8\n//zzMjsmLy+P++67L2RjFc/etm0bNTU1jB07Vga5xUVL+LVNiMlkOsBFEYxDUNxg//777wGYOHHi\nQQPlwhUoLj8HFtcMFU6nk7Vr17Jx40bg1ztqY8eOla5Qu91OREREWKubizmaMWMGNptNuoWPBeJi\nLvgr9q9ZswZAdmkeMWJEh8pW+7YdgUWcDyf+KX62paWFmTNnyi6w4D9Ely1bxsqVK2X1BVHmbM2a\nNTI7s3v37rIVz9HGXkUWsaimsWDBAi655BLg1wxaRVHkev7888/ZsmULDQ0N8j6ZoihYLBZyc3Nl\ndYlrrrnmqCuIi5Yd4uK5uOA9cuRIysrKAP89rhEjRtCtW7cOlVTx/RRFkS7JUHPcCyhVVUlNTW3T\n10VVVTZs2EBtbe0BCyTYaJpGRUWFXHSapsnFJDQqsfjtdrvUYG644QZ2796N1+slPz8fgC+++CIs\nEy9u0wc23xPZaIFJEqIMk/g38V2O1lpwOp18/vnn0oKKjY3tVPCpqsoXX3wB+K2aoUOHyvcVSqxW\nK2azWR7AIsOxtrZWWssGg4FevXpJiyEcCO17xYoVsmyOiO+EsxSNsICE9j179mzAr2SJittRUVGd\nrhWhyAmOZF0JYb106VK+/PJL4uPjZYmw7OxsjEYjW7dulUkbIuPR5XLJRp3XXnst2dnZQVnX4nJ9\nXFwcTqeTOXPmyMu73bp1o6qqiurqailI586di81maxOTMhqNZGRk8Mknn8gU/aO9mC48Iw6HQwrm\n8vJympqasFgscozp6elER0cfUKUe/N4LkVVYX18v31moOSEElMVioXv37lKzdrvdfPbZZ6SnpzN+\n/HggtMHr6OhoeXBomobD4eDHH3+U5r2iKKxZs4avv/5aWlpOp1OW1n/99dcBQl6pQAiB4uJi8vPz\n5cYVtNds7XY7zc3NGI3GoBXX1DSNPXv2MHfuXHn/orPPFIegKFWTkJDALbfcEpaOtlarleHDh7N4\n8WLg1/n66aefZNA4JSWF/v37hy2bL9AttmXLFjRNIy4uTh4w4S5TZbPZeOONNwB/AkJkZCRXXXWV\n7AZ9qO/kSN5fYMD/s88+o6qqiuTkZKk8aJrG6aefTo8ePaRS2NjYSEVFBYmJiWzZsgXwH9TJycmy\n/c3hjLs9Yl0mJCRgt9tZuHChLN3lcrlobm7G5/O1SVQQLnUhhK6//nruv/9+0tPTg9o6xefzkZGR\nIb1M4lqMzWaTdzNLS0upq6tDURS5NxMTE7HZbPz1r3+V+7C4uJiBAwcGbWwHQ0+S0NHR0dHpkhz3\nFpQoCNnY2Ci1kG3btlFRUcGDDz4YlhbT3bp1k5ck7XY7breb77//XmrfJpOpjbsP/DGf4uJiXn75\nZYYNG9bm30I5VvBreIMHD+arr76S76ykpASLxYLb7Wb37t2AP7W6qKiI0aNHB02bE83c7Ha7TA7Z\nvn073bp1a3MXRlgL69evl3HEMWPGMGzYsLBYCmazmfPOO08mSaxatQqz2YzdbpfdTCdMmEB8fHzY\nLBdN06Q14HQ6MRgM5OTkdNjCJRw4HA4ZF3Q4HAwePJhbbrnlkK3s9u8tsH7loSB+NjY2VrZOF242\no9FIREQEZ555pnTzGwwGWltbqa6u5pdffgH8yRQNDQ1YLBaZ2i5iUoeLOGtOPfVU5syZQ1NTk4wt\ndeZ+NZlM9OrVi6effhqA8ePHH/HzD4bZbMZkMjFlyhQAsrKyyMnJaXNXy+12s2fPHhwOR5u4ndPp\nJDIyUl5nOOuss8jPzw/PPgz5E0KM2WymT58+1NTUyCDtZ599Rs+ePWXRxlATFRXFpEmTAHjrrbdw\nuVx4vV7pchQTGR0dLWNMl156KTfffDPdu3cP+8FisVgoKipixowZPPvss4D/sqTJZOLHH3+ULqzU\n1FQmTJjQxv1xtBgMBgoLC+nfv7+8iPjvf/+bHTt2MHDgQCnonU4ndrudnTt3yqD7RRddJOMtocZg\nMNCjRw/eeustAFavXo3JZKJ3796yTI7ZbD7ggmmoEf27WlpasFqtZGdnS4EQ7nUUFRUl44F1dXWc\neuqphzw/nV1cPVQMBoMUzJdffjl9+vTh559/lspDz5496devHykpKdL1ZjabSUtLIzs7W96jS0pK\nkjFi4eo60vJVYh4eeughli1bRkVFxQGCKbCMULdu3bjhhhuYNm2avDsWqvuPZrOZbt26yVbuooh2\newYPHozD4ZDuf5EENHnyZJkMFB0dHbbEIMOx6PHRCUc8EFEz7dNPPwX8vuabb76ZzMzMsEh5TdNk\nKZdHH32UDz/8EKfTKZ+dl5fHpZdeyqRJk+jRowdAm14xxwIRKxPprR988AFlZWXExcXJdP2hQ4fK\nG//B3Dher5cdO3bIPj1Op5OoqChiYmIoKioC/EFukU0kUl7j4uKO2Ts71q01xBhEHOCmm24iOTmZ\nV1555YBU5nAgKqmL+GlTUxN333132ILngSiKgs/nazNHHa3ZQOu8faZq+8vWR6OQqarKP//5T158\n8UWZLaeqKtHR0RQVFUkrZurUqcTExAQ11hRsOpMPhznHR7wgThgBtWvXLubOnQv4s1HOP//8sLVl\nOFFov1FDJQw0TcPtdstgcVVVFWvXrqWxsVEK8IKCAnJzc4mIiJAbOJzp3Dq/jXDXgn8PRkREHHMh\n3lUQLmqRwp2ZmUlERMQxLS58DDniL6wnSejo6OjodElOCAtKNPcSMSibzca4cePC2rxNR0dHR6dD\n/rddfHCgr/R/0IzW0dHR6Yoc8WF8wpgYukDS0dHRObHQY1A6Ojo6Ol0SXUDp6Ojo6HRJdAGlo6Oj\no9Ml0QWUjo6Ojk6XRBdQOjo6Ol0EVVVlpYv2F+f/FznusvhE75RDqXLQvnyJjo6OTlcisIEp+Nt/\nfPvttwDceuutAGErhdQVynm1R7egdHR0dHS6JMedBdW+E2dnqKqKzWaTjbkSExOxWq3HtECrjo6O\nTns8Hg91dXUAvPjii9jtdvLy8mSbmczMzJBaNsKKO9qW96HguBJQhyOcmpqaWLlypewPEx0dLQs1\ndqUJ0NHRObHoKG7U2Zmjqio1NTU8/vjjACxZsoRTTz2VG2+8kYyMjIP+brDGKsImqqpiMpm61Pl4\nXAmog724wICi2+1m165d7N27l/LycgDGjh0r2x0fS8QYRYuAwNL+XW1xhIuu6PvWOXwOVg0/UEsP\npZLY3hqAX8+Ngz1TtA8B+Pnnn7FarYwbN+6IGp4GtvX4rTPL6XQyc+ZMlixZIn936tSppKenh9zb\nI7oK7N27F/B3FejVqxdpaWlHFPcS8x/McR9XAioQsQBVVUVRFJxOpzSTy8rKWLNmDS0tLdKCEh1I\nw3UQBrYiUFWV1tZWZs+ezb///W8A9u7di6qqJCUlMWbMGAD+/Oc/k5iYGJYxinYAQoB/9tlnWK1W\n8vLy5Hiio6Ol0Az2mBRFke7X8vJyvF4vhYWFsplbV3PFqqoqN7N4FzExMRiNRkwmEwkJCQBy/KEc\nR+D/A+fmt+YoFOtK7MOWlha2bNmCw+GguLgYgJSUFEwmk9yfAM3NzaSmpoakFY6maXg8HsC/v+x2\nO4WFhVIx7UwB1DSNTZs2MW3aNMDf4fmyyy5jwoQJRzWe33rfiqLw/fffM2vWLJKTkwH4wx/+wKBB\ng0LeCVzTNOrq6rj77rv58ssv5XhOOukk3n77bXr16gUcfoJGsNdY1zoFdHR0dHR0/j/HnQUlzMiG\nhgYAKioqWLt2Lbt375ZaR2trKy6XC7PZLNuXJyQkhCXQKKylLVu2yPYfu3bt4pdffqGuro76+noA\n7HY7TqeTqqoqqqqqAH+Tvttuuy3kFpTX62X69OlMnz6dmpoaACIjI4mOjiY2NlZ2Jr766qsZOXJk\n0Nusa5pGS0sL99xzDwCzZ89G0zRKSkq46qqrABgzZgyZmZnHrGWKqqq4XC7ZYn3ZsmWsXr2a1tZW\nqVVWVVVRWVlJVlYWf//73wEoKioKmaXi8/lkh9alS5eSk5PDgAED5BoXz/V4PNJisVqt0tITiBbn\nwRin6Ez7008/MX36dAYPHkxJSQngt4JVVWX//v289NJLAPTp04err776qJ/bHtHV+rnnngNg0aJF\n9O/fnzvuuEO2pe/oOwur69lnn+Xnn38G/Fbw2WefHbI7SOJzm5qa+Oabb9i5cyfnnnsuAOeee25I\nrfDAtPannnqKWbNm4XA4AL+1ZLfb2bRpk3xnRqPxkNaJiGUFOyX+uBJQmqbR2trK2rVr5SHa1NSE\npmmccsrGKCziAAAgAElEQVQp8iC1Wq3s2rULi8XCKaecAvjdDaF0GwnXwubNm3n44YfZuXMnBQUF\ngH9TTpo0ierqanlw1NTUsGjRImw2G62trQB8+umn3HTTTSHrHOv1egE4//zzWbJkCYWFhVJInHLK\nKXi9Xurr65k3bx7gD9gWFBTQu3fvoB66wqUyc+ZMwK9QxMfH43a7mTFjBgCPPvoo+fn5vPnmm/I9\nhlpwB27eTz/9lIULF8q/S0hIID4+nkGDBkl3zMKFC1m7di319fVScIiW9cEel9PpZMaMGbzxxhuA\n/zC58sor27RY9/l81NTU8Mknn8h4Sq9evbj11lvbHHrBjPk1NjYC8PTTTxMdHc1VV11FfHw84D/c\nPB4PixYtYunSpQBceeWVIXFfqarKwoULpQvd5/MxbNgwMjIy5KHZmXtvyZIlfPrpp9LlnJSURF5e\nXsjWm9iHGzZsYNGiRWRlZckkiVB3AReu4WXLlvHxxx9L4QQQFRVFREQE8+bNIykpCYDhw4cTFRXV\n5l20jyeKP4v/gvnejgsBJV6qx+PhhRde4Oeff5aLqW/fvpSUlDBgwAAZB2hpaaGhoYHevXvLw8Ri\nsYRswfl8Pp5++mkA3njjDZxOJ6eccgoPPvggAAMHDsRoNOL1euUYqqqqeOWVV/joo4+w2WwA7N+/\nH6fTGRIBpSgK1113HQA//vgjI0aM4KOPPpILUWi7ra2t8jCprq4OiQXjdDr55z//KbXv9PR07r//\nfsaNG8e2bdsA+Mtf/sLKlSs5/fTTeeqppwCYOHEisbGxbbS0YM2poijs27cPgH//+998++23FBYW\ncs011wCQl5dHfHw8MTExcj1GRUXx4YcfYrFYZKwzFLS0tPDwww/z/vvvy2ePGjWK/v37t7FuS0tL\nefLJJ1m1apVUxqxWKx6Pp82aCtY7U1WVWbNmAZCcnMzjjz9Oenp6m8/3+XysX7+euLg4AIqLi4O+\nDzVNo7a2lgceeEB6KPr378/UqVOJjo7uUDEVh6zNZuP666/H7XZLwXn99deTn58fkguyYqwA06dP\nx2azMXXqVCnUQ+3lEe/npZdeksqFiNENGjSI3r17Y7Va5bxu27aNPn360NDQID1C8fHxTJgwgcLC\nwjYCNRRJXl1eQAntEWDHjh18++23NDY2Mm7cOACmTJlCbm4uERER2O12ANatW0dOTg4lJSVyA4fK\nelJVldWrV/Pyyy8D/iDwsGHDeOWVV+jevTvwa6DRarXKjdG9e3cuvvhiFi9eTEtLC+DXnoLpfglk\n69atfPHFFwCkpaXx2WefdZiQ0dLSwrJlywC/VZWVlRXUd6dpGhs3bmTRokXSjTBr1ixyc3MxGAwU\nFhYCfqH++OOP8/HHH3PfffcB8Kc//YkePXrwxBNP0KdPH8C/uUwmU5ukFJPJdMiHiygrs2HDBj78\n8EMAdu7cyZlnnsntt98uFRzh6hDzA34ruK6ujry8PLKzs4HgHjDCKn/55Zd566238Pl8MgHh0ksv\npX///sTExMjvvWDBApYsWUJTU5M88PLz82XZnPZa8NGMVdM0KisrefvttwG47777KCgoOCBzz+12\ns2PHDnJzc4HQJJG43W7++Mc/snv3bimIb775ZgoLCztdB0I5uvjiiykvL8doNNKjRw8A7rzzzpAl\nKWiaxi+//ALAvHnzMBqNXHfddSER2u0/U1EUdu7cCcCmTZvw+XxER0fLs/Smm24iNTWVtWvXsmjR\nIgCef/556uvraW5ulp+TmppKQ0MDf/7znw84G4J9dulJEjo6Ojo6XZIub0HBr5pkc3Mz/fv3x+12\nM3LkSACys7OJjY1tk2JaX1/PmWeeSXJycsjTlR0OB9OmTZPmckREBH/729/Izs6Wz+5IozCbzWRl\nZZGRkSGTJNLT03E4HG3cNsHQRjRN44UXXpDa6/vvv9+h9WS327n11lvleG644Yag3x1TFIXXXnsN\no9HIlVdeCUBubu4B7yo7O5vnnnuOzMxM/vWvfwH+eOP69euZNm0akydPBuCiiy6S71q4BxVFYeDA\ngYc8Jp/Px6pVq2TK/UknncQ999xDbGxsh+9fxBAefvhhfD4fU6ZMCbploGmaTGCZMWMGbrebrKws\nHnroIQDOOussIiIiMBgMMoY5e/ZsmTwkXI5jxowhOjr6gO9xtOtKVVVmzJjBSSedBMD48eM7tFZW\nrVrFmjVr+OSTT4DgejKEJbty5Upmz56Nz+eTFvjFF1/cqfXk9Xq58cYbAb/VqWkacXFxvPjii4Df\nhRWqRBeXy8W7774L+NfpAw88QEpKStCf1VEyiKIoMlYKSNe0SFoZPnw4BoOBuLg41q9fD0BtbS0t\nLS2oqiqtyuLiYi655JIO3afBTizp8gLKYDDIlxAfH895551HTU0Nffv2lT8jXBjCbD/ttNMoLCzs\nNH4SmHEX+PmHi6qqbN68mV27dsnYQFFREcOGDfvN7BdN06RQE+NMSEjAYrEE3UwW953EoTV06NAD\nPt/r9TJ58mR++eUXHnvsMcDvHgr2RnU4HKxYsQKLxSLdVZ09w2w2c8opp8jECXGQREZGkpqaCvg3\nmdfrJSoqit69ewMcdgzPbDbTs2dP6Z8/99xziYmJ6TSo/uqrrwJ+l3NsbCx33XVX0N+TqqrSJVtd\nXU1UVBRjx46VillERISMG+7YsQPwu200TSM2NlZmsxUXF3forjraNebxeGhtbZV3hUQmYfvv8Oqr\nr2KxWBg0aNARP6szxD5+//33cTqdaJomv2tngXyHw8GNN94oBab4nT/96U/y3YYqDuR2u/nggw9Y\nu3Yt4D+nbrjhhrDcexQFAYSrtX///mzYsIH8/HzpIhY/U1tbS1lZGeCfZ4PBQGRkpHw/r776Kj16\n9Ai5cILjQECB/0IkQGFhId27d2fv3r3yELJYLPISpfC7p6SkdCicxOXUPXv2ALB69WqGDx9Ofn6+\nfNmqqh6ywNI0TQonsTHGjBlz0ANSTKLH46GyspKtW7dKyy8qKgpVVQ9rDIeC1+ulqalJHhLi3QRa\nnX/7299Ys2YNN954IzfffDMQmo3a0tJCUlISO3fu7PAQEWiaht1uZ968eTIOlJOTQ1ZWFkOHDpXZ\nmfHx8VKTO5KEDrEpBw8eTE5ODoBMhOhoDurr63nkkUcA/1q544475PoMJh6PR6Y9u1wuoqKiMJvN\n8tZ/cnIydrudyspK/vKXvwB+C9hoNHLttdcydOhQIHTJQR6PB4/HI7MWO3pGQ0MDGzZsCFlmqlAK\nnU6nvLAvrOjbb7+d22+/nZiYGJmEtGzZMl599VXKysrk75rNZs4991ymTp0qreBQvC+v18vcuXN5\n/vnnZax8wIABREZGhq2SislkYsCAAQD8/ve/Z/bs2URERLBr1y7A/y42bNjAkiVL5NoTAionJ4c7\n7rgD4IBYoyAU3+G4EFDCVI+KisJoNJKYmNjmJj38KqjA70YQ1lSgtdTS0sLixYulKV9bW0tqaiqv\nv/66TGU+nEPOaDQSExNDfHy8fN6+ffvw+XwdbsjAulc2m43ly5e30foKCgqk+yiYk60oCunp6Qwf\nPhz4ddG5XC6Z3PHuu+9yyimn8I9//COk5f1TUlIoKCigoqKC77//HvCXoRIJIuL97Ny5kylTpmC1\nWjn77LMB/4Y2mUz069ePxMREAGJjYw/5rkZnGAwGYmJipFXW3NxMY2Mj8fHxcm4MBgNOp5NnnnlG\nzlFeXh5//OMfQ7IxDQaDDEybTCbMZjP79+/no48+AmD37t1s2bKF5uZmeeAZDAaysrK47777QnrY\ngn/tW61WKRCKiork3hHvR1hxN910U0jGIQ7JW2+9lW+//Zba2lqZNv3BBx8wY8YMoqKi2tyPDLQW\nwJ8wdPnll3doAQYTr9fLG2+8wZ49e9rco3M4HFitVlwul/xOVqsVq9V6yBVCDhVhCQFMmDCBlJQU\nZs+eLdt7TJ8+nYqKCpmUBn5vRFxcHBMnTpQW1G8pz8EsOqsnSejo6OjodEmOCwtKSGKhSdrtdqld\nejwe4uPjsdvtVFRUAP5YjriPIe4ceDweoqKiyM/PZ8iQIYDfXbNixQruu+8+pk+fLn/3UCW/wWBg\n8ODBWK1WqZktW7aMPXv2UFhY2CbwL6wnoeGVlpayd+9eFEWRP2e322VcLJhYrVbOP/98srKyAP9l\n4tLSUubOnSutmN69e/P888+HvHKD2WympKSEFStWSM3f4XBgsVhQFEX65y+88ELq6uoYOXIkp59+\nOuDX3Kqrq4mJiZF33o7WehKYTCbpqtM0jYqKCsrKymT6uMfjYcmSJezatUv68b/++uuQXaw0GAzS\nffbLL7+Qk5PDmWeeKYPcGzdupL6+HrfbLb+/1WrlxhtvJDU1tdP4WftnHClWq5Xs7Gw2bNgA+O/Q\nZGRkoKqqtJxmzZrF2LFj5R2oYCP2zbBhw/j888+54oor5BkgLo06nU655wI1e5GI1Lt3bxobG0Pe\nudZoNBIdHY2qqtK6VRSFTz/9lI0bN7Jlyxb5s1OmTOGss86iW7duAEF1IYt3FhMTw7Bhw7BYLPKe\nYVlZmbxfKqy8pKQk+vbti8lkku/xYEkkgQWDg7EvjwsBJTAYDFgsFhITE5k7dy7gv9yakZGBy+WS\nufsFBQVkZWUxZMgQ6XPNysqSpr7w2a9bt46WlpY2RVPFwXeoZGRkMGDAAPbv3w/43QirV68mMzOz\nzeHl8/koLy9n1apVgF9A1dfX43K5pDuwpaVFJnwEEyGghK95yZIlfPzxx9hsNhmXmjRpEj179gzq\ncztj6NChzJ8/X8YGdu7cSa9evSgrK5MuR/Bfwn788celkJgxY4a8OyUWf2eXMI9kcwQmq5hMJqqq\nquR8ibtbmzZtkmtKHCCh6NxsMpmkYK6srGTkyJEMGzZMJkRUVVWxceNGGhsbZeaexWKhpKSkwyri\n7f98tDFOq9XKJZdcwuLFiwG/W7S5uZktW7bIytwlJSXce++9IY+vmM1mTjvtNHbt2iUF1Kuvvkp5\neTk//vijzIYE/zvq3r07EydOBPwHcFFREW63W+7DUMWgNm/e3Kbad3R0NOnp6axbt07uhd69e+N0\nOnnnnXfkvayrrroq6NnIwpWYmpoqY/dms1kq2uLsio2NpaKiAkVR5Hvs1q0bZrO50zH9T1czF2mQ\nW7duBfwWS1paGjabTb6Yvn37ctpppzF06NADNO3AmI+qqm0m50gwGo3cf//9rFy5EvDHlh555BEW\nLlwoNWCLxcK2bdtYuXJlm0Nw27ZtMrANfmHb2NhIZmZmULUQg8FAVFSUrBoRFRXF1KlTGTBggNTc\niouLw9JaWsQQo6OjpVLw4osvkpGRgc/nk2M477zzmDp1KsXFxfLQEf8//fTTO0yqCBbifeXm5soU\n4MzMTBYuXIjD4ZBJGwaDAZ/PJ7PpANlz7GgxmUyypl1jYyMFBQUkJibKNOoLLriA4uJili9fLit/\nREZGHlALsLPyM0cbmDcYDCQlJTF27FjAb2HW1taydetWmS16//33k56eHnStujPMZrNUYJ544gn2\n79/PaaedJp8ZGRnJDTfcwF133SWtkpaWFpmGH0orStM0rFarTIIC//WKkpISioqK5GX9goICNmzY\nwLJly2RW6eTJkw9bce4MIYAURWHPnj18+eWX8uKwqqpy/YpzUVigsbGx0mvVUUsT8edgC9LjUkDF\nxsbKNPMlS5awfft2MjMzGTFiBOC/G1NSUkJ0dHSblymy1kQW3+zZs7FarRQXF8skiSMZT0lJiSzV\n/+yzz1JdXc3nn38uNTKPxyODoKIJWWNjo3RxCW2lb9++pKWlheTulslkkokFYsF7PB7++c9/ArB2\n7VqGDx8eciFlMBjIzs4mNzdXuofq6+vJzs6mb9++UmssKiqiqKgIo9EoNbe9e/eSk5PTYQA58IA5\n2kNQHN4mk0m6gmJiYqisrCQiIkJm+wW6bsWhE6yMLIPBINdKdnY2lZWV7NixQ7qnhw8fTl5eHgsX\nLpRJCfHx8Qe401RVxev1tnlnwVpfIkkI/NZAZGQkZ5xxhrQGMjIy8Hq9siWJ+F7hwGAwsGzZMlmT\nEOB3v/sdTz75ZJs7awkJCSiKgqqqIRVQkZGR8tAXrrI5c+YwYsQI+vTpI/emxWKhoaGBsrIyeT5s\n376dwYMHH/W8eb1eSktLAXj77bf5/PPPcTgc8q5j7969qampweFwyDUl5s5kMknFXlGUAyyoUAl4\nPUlCR0dHR6dLctxZUOA35W+44QbA38piwYIFpKSkSNeLz+djz549pKWlSW2pqamJTZs2sXDhQqmR\nu91upkyZwtlnn31UwW6LxcJtt90G+OtUzZ49G5vNJi/itrS00NraSr9+/UhPTwf8AUmn00l0dDT9\n+vUD/G6ttLS0kCQqGAwGadGJdH2bzcbq1asBvybp9XrDYkFFR0czZMgQ6eKzWq2kp6eTmpoqrZMz\nzzyT6OjoNs3uRBKJuKR6JByqhSM+X7hEPv/8c/bv30/fvn25/vrrAb9WLN5XKFxYYu0OGTKE7777\njvT0dDIzM+Wzy8rKqKurk9abWMOBYzAYDHI9hSodXvzfarXSo0cPqZGbzWa8Xi91dXWyLmW4cDgc\n3HrrrXg8Hvns6dOnt6l3CbS5xBzKqjMWi4V7772Xyy67TCYibN68mQceeICxY8dKV2l8fLysNyp+\nbvv27fTr1++QU+E7WouqqlJZWSmLWn/zzTc0NzeTmJgoXbKJiYnSahPxVRHzHzlypDy7Ortb9z97\nD6o9gVk4d9xxBxkZGezdu1f6bD/44AO8Xi/Dhw+Xk9ra2sr8+fNxuVyMHj0agNGjR3PSSScF5Q6E\nOBwuu+wyxo8fT1NTk/w74fJwOBz89NNP8neio6NJSUmRF+BOPvnkkAkns9nc5uAQLo20tDTAfyfM\n7XaHvNw/+N/VBRdcwFlnnQX4L6KaTCYaGhpkQkS3bt0wGo1omiaFQEJCAieddNIB5f8Ph0P5vcDK\nAyIBYeHChVgsFm644Qa5oUNRVb2jcSQkJHD++ee3idEpikJSUtIBGVWtra1t3MQmk+mgJbeCPd70\n9HS5hoVbyO12hzxLTiCE9cUXX4zNZiMuLk6WFgosXRU4x+JSdijHaDAYmDhxIv/617/kRW+73c76\n9evZt2+fLFcl3OwZGRky3ni4fdE6Ex6pqakyRte9e3fMZjODBw+W3XNXrlxJ9+7dKSwslCWs0tLS\n2Lt3L0VFRTIOFrim2vM/2W6jI8QL6tmzJ3fddRfl5eVSQG3dupW6ujp5KRT8WvqoUaPaWFqBGnCw\nsFgsJCcnt0kNTUpKklWdhWaSkpJCYmIikydPloshlCneIqYCvy6iiIgITj31VMCf0RguhIIRmNbt\n9XrJysqS7yDwUBWaZGxsLAMGDOh0AwT7AFYURSa/7NmzhxEjRnD++eeHvR29yF4VZbDg12STQYMG\nsXnzZgDi4uJobW0Najr5oSJivGaz+YDSSqLtO4R2jYO/lQzADz/8gKqq9OnTR8btOnsPiqIc9NAN\nFkajkauvvlrWoGxpaWHdunWsW7dOWsYlJSVYLBYGDhwohUmPHj2O+r2JxJ+7774b8Fd7r6+vp7a2\nVu792NhYMjMzOf3006Xiqmka69evJykpSRoFB1tPuoD6/wQeYDExMfTu3VtuzEGDBskDuH0gL/D/\noUAIArPZLLVvr9dLWVkZy5cvl0LUaDRyxhlnMHjw4JDf+hefLd6Foij4fD6am5vlwWEymbDZbEHL\nFjqU8QRqsSLLsiNtX2Qf9ujRQ/451Giaxv79+3nrrbcAf5LEnXfe2WkB2XAReCcwKSmJCRMmsHz5\ncsCflBBOayUQYYl0lC2oqqp0lQr3UCjeoaIosmEh+BXQadOmHdBxuD2hahDaEYGKYmJiImeeeSZn\nnHFGm2SuwsJCvF6vFEqKogSnKoPRKL0okZGRJCcnk5eXJzNVY2NjSU5OJjIyUu5DTdMYMmRIm0SX\n33pGMNGTJHR0dHR0uiTHrQUlENpYoOQOVbOxw0HcHAf/pcqKigpqa2ulhdKzZ08GDx7cadXsUBD4\nnOrqav7xj38wf/58wF9XLi4uLmyFK9sTWPNOINxGQusbNWpUSLp2doTH4+Gxxx6TSSQTJ07k5JNP\nDrt7rzOEh2Dw4MFcdtllgD+h41hZUMJ6Cpwf4fJrbW2VqdWiRUgoknHcbrf83MzMTIYNG8aIESMO\nul6OpTUcOIb24xBNOMWfgz1OMQcRERHygr74u/YXvUUFjPZ/H453d9wLqK6ImGiRcBATE0N+fj5N\nTU0yJpaRkUFycnJYLsd2RFlZGbt27ZKLbvTo0Z32fgrHYhTPCDxcVVWlvr6eTz/9FPDH7QYOHBiS\nrqztx7JlyxZWrlwpXYp33HFHWBJIDgeDwUBiYqKMaezbtw+Xy4XP52sTswjHQRKYZBA4h6I6gXBt\nW61WeR8omAhhePvttwN+hWvy5Ml069atjcDs6F0Eq1xWsAnHmER8s7NnCuU/VG7Z30IXUCFATKrw\nbaekpJCUlEROTo48OCwWi9Qmw43JZKJ///5YLBaZOjp+/PhOrYNwbZT2B4jH42Hz5s1S+zabzWGJ\nF3i9Xt58801MJhPnnHMO4I9/dbVDTBwu4sBXVZXm5uY2l05DbfG1T2luH1tUVRWXyyVrLF5xxRUh\nGYdIcx88eDDgj0MfquA5VkrisUZcMD+U728w/NppIJwC3XAs3AGd0GUGEkrCUfLlUBEJEoJjvVHb\nCyhN02hqauLhhx8G/OV9Ro4cGfJMMJ/Px44dO4iJiZF3aI71uzkUNE3D7XZjsViOi/HqHF8cxdl1\nxIdd13Co6+jo6OjotEO3oHS6PF3J6tTROVEQ97/CwBFvXF1A6ejo6OiEEt3Fp6Ojo6NzYqELKB0d\nHR2dLokuoHR0dHR0uiS6gNLR0dHR6ZLoF3XDiKZpsmhmOKon6xzftO9qGpjQ9FvVEXR0TgR0ARUC\n2pd8EXXK6uvrZeVpgKFDh5KYmNhhozmd/200TZONNUVrksCGkqIRYH19vayuUVxcHPJLzCcKoirC\nsSrh055jdZUiUMHpisrO/1SaeTgmwOv14na7qampkRqwx+OhZ8+ebUrQbNmyhRdffBFN02SlhJ49\ne3a5BfK/jOhTBf5mk/X19ZSWllJbWwv4FYyUlBTi4+NDIhjaV/oQhyr4i/1u2LCBn376ib59+wL+\nShvHql6gKGsEdNgypaugqip2u13OYVZW1jErOaZpGi6Xi5aWFlkHMzo6OiR3k9qffaqq4na72bx5\nM3PmzAFgzJgx9O/fn/j4+GDPoZ5mrqOjo6NzYnFC+wM0TWP37t2Avxp1TEwMzzzzDDk5OSF5FkBT\nUxNPPPEEjY2N/PWvfwWgV69eB8Sb+vfvj8FgYPHixcyaNQuAu+66q0vUUGvvogxXrKx9JXNVVfF4\nPLLastlsDutY3G43GzduBOC1115j69ataJqG3W6XP+d2u7niiiu45ZZbANpUzz5axHcVLuLAiuEp\nKSmy0dzevXsBcLlcYbWgFEWR++ull16SDTpFl+ZrrrlGzpl4J6J6gdlsDrvVoigKdXV1fPDBB9Lq\nzMrKOiZuNfDP18qVK5k3b57stH3hhReSk5MT9HUe6MYDv1entLSUl19+mXnz5gHwzjvvMHz4cKZN\nmya7EB8r61JwwgooTdMoLy9nzJgxgL8VQXZ2dsg6xoqJ//LLL5kzZw6jRo0iOzsb6PiAj4iIID4+\nnsbGRtmy+2gXQvueLYczbp/PR21tLTt37pSH8p49e/D5fNx2222yTUgwBKjo8yTwer04nU4aGhpY\ns2YNAF999RVr1qxh37598iB+8MEHmTJlSsjbbYgx1tbWMnv2bABWrVpFQ0MDVqtVuv3q6+txu908\n/vjjuFwuAB599NGgb+hA15mYX4vFQq9evRg1ahRbt24F/IdOuFBVlQ0bNsj91dzcDPg7tTqdTgAG\nDBhAUlISTqeTffv2AX5hmpqaynnnnSeFabiUMrvdzh//+EfWrVsne2idddZZYXm2wO12M3fuXMC/\nVkpLS/F6vbJ33MKFC3nuuefIzc0NidtY7Dun08maNWtYsWKFVLg0TWPx4sXU19czZcoUAM477zxS\nUlKOiUIBx5mA6iheJg67wF5CoqX5p59+Sl1dnfy71NRUYmNjQzI2ES+oqKggMzOTadOmHXSBGQwG\nduzYgdPplAdeMDgUISWsE1VV5ft58803WbJkCRs3bpTjEVp7XV0dL774IuBvU300C1XTNDweD/X1\n9axatQqA7du3U1ZWxubNm9mzZw/g30A2mw2Xy0VTUxMATz31FOeeey5WqzUs/alcLpcU1o2NjVgs\nFrKysuTBWlpaSmVlJUajkbKyMuDIlITOEIKpfbt08M9NZGQkubm5MobQ0NBAampqWN7Ntm3bOOec\nc+TcaJpGREQEJ598MjfeeCMAJ510EgaDgfLycmbMmAHAxo0bKSgoYNCgQRQWFgKhb98g9uYbb7zB\nDz/8QEFBAZMnTwbC2+69tbWVa6+9VlosovJ8bGys9BKsW7eOl156iTvuuIPc3FyAkAiqiIgIEhIS\n6Nu3r2xnY7fbsVqt7Nmzh2effRaAOXPmMH78eK655hri4uKA8MYWjwsBFehyUhRFJiIA8uUmJCTI\niRTCymaztdE+b7/99pC5iMSkjRo1CrvdLi2OzvD5fGzatAlVVTvsJns0Y+gsGaS9C81ut/PZZ58B\n/q6/iYmJpKenywCyw+HA4/GwfPlyduzYAcDJJ5982BtGzBv43Rpbt27ls88+k5pkS0sLXq+3TaM9\nt9t9gFurvLyc8vJykpOTD+v5R4KmadTX10vN32g0kpmZSUZGhnQPFRUVMXv2bFpbW6WVF6zNqyiK\ntERqamqIiYkhPj6+zYFqNBrxeDxs2rQJgPnz59OrV6+QWSRiHhwOB3fffTd1dXVtur6effbZvPrq\nq1a2cZ8AACAASURBVHJ+LBYLmqaRmJgoFQ/RWXrjxo306NEjJONsj3A9zps3j4KCAh544AHy8vKA\n8PXLcjgcnH/++SxbtkzuhbS0NMaPH8+QIUNobGwEYNu2bWiaxoYNG+Saio+PD9q6Ep9jNpspKSkh\nMzNTrher1YrVapVtbgCWLl3KmjVrWLt2rRRagUkUoUZPktDR0dHR6ZIcFxZUYIBVURQqKytlh876\n+nr69u3LwIEDpVQ3Go14vV6qqqqkBRUZGcm5554b8rFGR0fLbp4Hw26343K5MJlMjBgxIijP7qgr\nbWc/J1rSX3jhhYDfoisvL+f999/nhx9+AMBms8murUcT99E0TVq6W7du5e233+a///2v9H3Hx8eT\nnJyM2WyWlrHVaiUqKoqGhgapAbtcLp555hneeeedkLtmVFXlu+++o6qqCvBrwPv27aNfv35kZGTI\ncYvmgMLaCYamKyzO/fv3A/4rCenp6fTu3buNl8BgMOB0OqXF8u2333L11VeTmJh41GPoCOFqfOih\nh1i0aFGbdg1jx47lgw8+ICoqqs3a1zSN6OhoeafL5XLJ9RT4XUKFoigyjhgTE8Nf//pXSkpKwhb3\nam1tBeCxxx5j9erVAEyaNAmAO++8k4KCAkwmExUVFQAMGTJEvldhVUVERAQtWSHQy9LS0kJ9fb20\n6IT1FPhzdrud5uZmPv74YxkeeeKJJ2RafKg5LgSUwGQyoaoq9fX1rF+/HvC7EVJSUvB4PDI2oCgK\nDQ0NrF+/XgqonJyckL5UMaFer5chQ4b8phusvr5etoXv169fm88IB+JZwr1os9l44403+Omnn+Sh\n7PP5SElJ4aqrrqJXr17A4Qe02y94i8WC3W5HURSZuZSdnU1qaip5eXkyJpaYmMioUaP49ttvmT59\nOuBPApg3bx42m43U1NSj+foHRSQAvPPOO9TX18vvIbLAYmJiAP9cK4qC0WhkwIABbb7nkSJiql6v\nV7rFvvvuO3r16kVGRoY8/EVQPT8/n/POOw+Ajz76iAULFnDOOecEXYBrmiYTWN544w0cDgdGo5Fh\nw4YB8O677x4gnMD/Pmw2m5xXTdOIjY3llFNOCct6r62tletn2LBhlJSUtFG2Qnk30uv1snTpUsCv\nPERGRjJ27Fief/55AOLi4qSSkZmZCfizCj0eD01NTVJw2Gy2NsoQHPk6CywesG7dOlasWCFd+haL\nhfj4eBISEtrExUU8dtGiRYD/XAjXpd7jSkAZDAbMZjP5+fkywLp3716ampqorKyUGkdZWRnLli2j\nrKxMbpisrKwOkyyChVhM0dHRqKqKz+frcDGJMSxZsgRFUWSQVPzbkUx64K34zmifOaeqKuXl5Sxc\nuBCAF154gdLSUqklg7+CwZ133skf/vCHI6p2Efi8QIEdERFBQUGBFFADBw4kJSWFk08+WR4effr0\nwWw2079/fxmrKi0txe12s3z5cmkNh2KT2O12Hn74YWpra9v47Hv16sXdd99Nz549Af8cCg1Z/F0w\n4oiqquJyuWTiRU1NDa2trSQmJjJ06FDAr3AZDAasVivl5eUArF69mo0bNzJz5kxuvvlmAEpKSjoU\nHIeLz+fjlVdeAX6N+yYmJvLWW28B/vT6jp6hqipz5syRGYZGo5H+/fuHLJu2/bOfeeYZqqurAfi/\n//s/IiMj21x4DhVCodmyZQsA3bt3Z+LEiVx77bUkJSUB/rjv7t27SU9PJz8/H/BbMaqqkpKSIr0J\nc+fOpUePHvTr10+eFUe7zrxeL8uWLaO+vl7OjTizcnNzpeKxfPlydu7ciaZp8gwIPCNCzXEloMC/\nwGNiYqQ1VFNTg6ZpLFu2TB4WDodDCitxMMbGxsoslVDcdBeHcW1tLbt27WLixInSFSQElaIocnPP\nmTMHt9tNr169iI+Pb/MZRzquzn5PCCexMRVFoaqqig8++IA333wTQAa8jUajzB66++67ue66647Y\nvRD4O2IeRPagSDgA/z2xpKQkBgwYIK0T8c5yc3NlpY1bbrkFn8/H559/zu9+97s2nxsMhJKxceNG\nNm7ciKqqUog+8sgjXH755fIAAb8gczgcWK1W+vTpE7RxCFetODjcbjc+n4/6+nqp7UZGRtLa2srq\n1aul4BDW3ldffcWCBQsASEpK4qKLLuKuu+6Se+ZIhJXX62X79u0Acp088sgjbTLxOsLtdvPCCy/I\ntR0TE8NNN93UJvlFfF6waWlp4bvvvuOcc84B/ALV4/Hg8/mk5yAwMzLYKIrCwIEDAb8S1rdvX+Lj\n46V1YrPZ6N69O9nZ2dIiVlVVrmkhCOx2O6tXryY5OVl6Mo7WgvH5fJSVleFyudpULImKiuKaa66R\nd9nmzJnDiy++iNfrlc8OZ8q5niSho6Ojo9MlOe4sKPBrayLoumfPHnbu3EleXp6M5QwaNIja2lrW\nr18vL1CWlJRQXV2N2+2W6ZuB1tTRIiyjb7/9lsrKSkpLS5k4cSLg1+RWrFjB/v37ZXLH7t27MRqN\npKamyu8i3DaHqx391ndof4vcZrPx3nvvMWvWLKnNiRTT4uJi3nvvPcAf3wiGthRoDcyePZvq6mps\nNpu01FpaWhg5cmSHdciMRiOjR4+WY/T5fGzbtq2NyyhYcyhcKosW/T/23jw+yvLc/3/PnkyWSUI2\nSAKEJWGJgKwWEAERFFEWezgVcan1pZyq1Z6e2tZWu6hdtPW0ao+tFY/LUVrLqiIKSAGRfU0ChiUh\nJGTfl0lmfeb3x/O7LycIKmQmgN/5/KOvYTJzz3Pf97V+ruvaisfjITo6moceegiAxYsXi2Wrvq+4\nuBiPx0N8fDypqakhWYOCpmlCDmltbWXEiBGkpaVJbmn79u28/fbb7NixQ6IFqmTBZrNJ4Wxrayuv\nvvoqiYmJ3HPPPcD5dwdQOTH1N1FRUQwYMIBFixZ9YU5E0zT+93//V/JPoFOr+/fvj8fjkfNosVjC\n4sksW7aMtrY2MjIyAP1ZOJ1OWlpaxIMaPnx42Aq/LRaLdGQIvkfKY1FEJYvF8rkwfXAY0uVyUVFR\n0SVM391nFRUVxdSpU9m+fbvIALPZjNVqZeDAgeJtl5SUSI1bbm4u0LO1Y5elgtI0TYS6yWQiMTGR\nadOmiYLq3bs3brebgoICiT/n5OSwdOlSTp06JfHVhQsX0qdPn24XfgYnkFesWCEhi7Vr1wJ6HYbT\n6SQQCIiQs9vtxMbGkpCQIMpNtbQJLjoO5aVVh/748eM4nU7i4+Npa2sD9EM3atQo/uM//kPi4aF0\n5VVR7o4dOygpKenyb7m5uaSmpp6TgKGej8FgwOfzoWma1GmkpqaGrCO1uqjV1dWkpqayYMECvve9\n7wFnDyW+9dZbaJpGbGxsyIWc2WyWs2uxWHC73RiNRsk3vf/++2zdulXC2qDXAi5cuJAbb7yRt99+\nG9CfuwoRKcZmenr6eT+vlpYWCfG0trYyYcIEfD5fl3xEcGsm9d1/+MMfhK0KesHzb37zG66++mrJ\npw0bNkzyy6E6b5qmsXLlShITE+UzKysrqauro6ioSEKlDz74IBkZGWEJWZlMJgndwWeKXp3n0tJS\nGhsbmTx5snSdUa2hAoGAMEMLCgooLS1lzpw55OTkAN1PA9hsNim+VbWQtbW12Gw2Dhw4IPd1586d\neL1e0tLSmDp1qvyunsJlp6CU4FYC44orrmDUqFGMGzdO6LYmk4nDhw+Tn59PS0sLoFvpe/fupb29\nXYpOp0yZInmQs33PVz0EHo+H999/H9BppepgKgtYHbTgz3Q4HAwfPpxBgwZJ0WB31/FlUM8uJSUF\nq9VKS0uLXNTk5GTmzZvHtdde222m0Jnw+Xy89NJLgJ7It1gsREVFSU/EESNGSFHn2f5WtfJRwrC2\ntlYEdVJSUkhyipqmidfRr18/nnnmGa655pqzKiblEZSWlmI0Glm8eHFIcyiaplFTUyOWalNTE2az\nmd69e4tiVt6SyWQS7+3xxx/nW9/6FiaTiQkTJgA6+WX9+vWYTKYuyux8EQgEJLmflpaGwWBgx44d\nsmdOpxODwUBzczPbtm0D9LZfShir59Pe3s6BAwfIzc2V/TIajUIYChUCgQADBw4kJyeHvLw8QI9a\nnD59mh07dogSbWxsFA8rnFDkl/b2dlauXAnAgQMHiIuLIy8vjz59+sj7VIG6klPl5eXExMTQv3//\nkPZ5TEpK4jvf+Q533HEHAPn5+Tz//PMsW7ZM5EJ9fT02m40pU6aIA9CTc+wuSwXl9XqZPHkyoFt2\nWVlZOBwOEaxVVVU8+eST7N27V5RWc3MznZ2dpKamsmjRIkBXbucKd3zVg6BpGsXFxcLWSUhIEMq0\nEggxMTFUVVXR1tYmn9u/f3/q6+vJy8sTynRw4vh81qEE95e1VlICICYmhtLSUo4dOyavzZ07l8WL\nFxMVFRVya7KxsZHi4mJAV9aapnHttdeKBa082DOZhqp+atmyZYAuBDVNo6OjQyivGRkZpKamdku4\nqRCk6hoxduxYxowZc9bP1DRN2uT4/X6io6N5+OGHQ/rMDAYDZWVlsp5evXqRmZlJTk6OCIe+ffti\nMBg4evQot9xyCwDf/OY3sdlsaJomZJOsrCyGDBnC4MGDv7A35JetJy4ursvokYKCAj766COJZHi9\nXhH66r/BXSbU2UxJSWHy5Mn06dNHCCiBQCDkVrnRaOSnP/1plxDwkSNH2L17NyaTSQxTpXRDjTPv\nsZJbK1as4Pnnnwf0fV2wYAEDBw4UD1w9B4/Hw/bt2wFdxo0YMSLkbayC60ZBNxRnzJjB7t27u+yr\n2ne1xp4sh4mQJCKIIIIIIrgkcdl6UIqafeTIEUwmEwkJCUKIeOmllzh48KDUXIBuSaq4q4ql22y2\nbrmrql/bP//5Tylu/cY3vsGoUaMYPXq0WJIJCQl0dnbi8Xg4evQooFOZT58+LZNSQc9LnZl7+iq5\nqPOlWufn57Nq1So0TePqq68G9OrwcHhPoP8G1WjSbreTkJDAwoULJWSgvlfTtC6NajVN4/Dhw/zr\nX/8CPmvEGh0dLaFbn8/X7ZBDIBCgpqZG8ojjx48/6zMNBAIcO3aMPXv2yGtDhgyRsxgqKI9C5aDa\n2towGo1ERUVJ2C87O5tnnnmGlpYW8UTMZjMdHR14vV7x3lWR56xZs7pVqG6z2SRqsWvXLpqamrp0\nagkEAuJxKgKLwWAgOTmZYcOGyZ375je/SWZmJlFRUeK9mM1mGWkfyvPXq1cvOjs7xRtQ9Tz9+/cX\n4o3D4QhL93mVK1XPR3noK1askL0ZMWIE06dPF28XdK/c5XKxefNmSRuYTCZuvvnmkIRAz1YzGVxE\nP2HCBIYNGyYjXNSQ1ZKSki6NinsKl52CMhgMREdHy4NuaGjA4XBQVVUlwtxutzNq1Cjy8vKkDiEx\nMZFBgwYxYMCAkLqqJ0+e5ODBgyIwOzs7GTduXJcYu81mkzoWlTQ9deoUXq8Xp9MpAvZs9SDnI3y/\n7IIrwfHUU0/hdruJjY3lqaeeAgibcgJdUKgGq5WVlYwYMYLhw4fLswgO7wQrZFXkqXJDqoP3kCFD\nuOqqqwA9B9VdkkQgEGDPnj288847ANLdw2QydWFVnT59muuvv75LSPWZZ54JeUzeaDQyevRoYZse\nPnyYDz/8kLlz50qYLjY2ltjYWFJSUmRf/X4/+fn57Ny5U9okVVVVMWTIkC7n8UJgMpkkQb9gwQIK\nCgqkESzo+3D8+HE2b97cJSR05ZVXct9993HllVcCeojPZDLh8XhE2SpyRKjPn8FgoK6uTro57Nu3\nj4yMDBITE8VwDVYOoYLX66WiooLW1tYutUz79+/HYrEISWvJkiWkpqZSV1cnTMySkhI2btzInj17\n5HlMmzaNsWPHdvucqflqoO+nOt/B9WjBylVBdbtQv6WnukjAZaigVKGuYprdfPPNxMfHdxkUN3jw\nYGpqaoiOjpbLcvLkSWlrH0rSQSAQwOFwCMNlzJgxDBo0iPj4eKGTut1uvF4vUVFRIoyjoqJoamqi\nvLxcEt92u71bCf8vOjiBQEA8kUOHDmEwGLjmmmvkoobzwJnNZr7//e8DOq03KyuL9PT0z/ViMxqN\nXYwHn8/XpbuF1Wqlf//+LFq0SOYQhWJAnyK5KDLGpk2bmDZtWhcCzd///ncef/xxqqqqZI9Gjx7N\npEmTuv39Z8JoNOJwOHjwwQcBnXXZ3NzMihUrWLx4MfCZoPf7/UKPP3r0KH/+8585fPiwCJ0hQ4Zw\n3333dZtlaLFYREENGjRIrHn1LOrq6vjJT34iIyTUGocPH050dHSXYnRN0zCZTF0Ebrg8d7fbLS2j\nqquryczMZMaMGZL3DWXuSwn11tZWiaqozg8mkwmLxcJ1110nXdwDgQDvvPMOlZWVkm/ct28ftbW1\n9OnTR8aW3HHHHd3uxae6kwTP7bJYLJjNZrlf6r4FN0JQ5JXujtm5UFx2CgrowlxKSUn5XOuSqKgo\n4uLiaG1tFUsyJSWlC+MrFDAYDAwdOpSHHnpImEsTJ07EYrHg8/lEQamQTXJysrCa3G43jY2NbN++\nXRTHt771rW5Rbc/12wKBAPX19fz4xz8G9MOamJjIj370o7DMmjkTBoNBLP9vfvObQqVVz+fMBqhq\njTU1NVRUVIhwVSMC5syZE1Jat8/no7i4WDoxbN68mRdeeIGbbrpJyBhLly6VMK4KV61evTqkzLNg\nBDPxVq1axVtvvcXp06d54oknAJ1kM378eCwWi6xx9erVlJSUoGma1Jj96Ec/OmcbovOBwWAQYav2\nzufziUVeWVnJp59+SlRUlHjLs2bNYtKkSVit1i6Ej9jYWAnpKYTaKlehqfb2dvlut9vNjBkzGDBg\nQFh6FarzvGXLFj766CMqKipEMScmJjJ69GgSExNlQGlZWRlHjx7FbrdLNGH06NFYrVa+/e1vn7WG\nqjtQ7c0AKioqaGxslJZioJOZjh8/jqZpoqC8Xi9GoxGfzyeyKzo6OtJJIoIIIogggv+3cVl6UPBZ\nSCDY6lYWmbLsWltbJXzWu3fvkHsLBoOBmJgYcnNzu0y/tNls+P1+sfKVlW0ymbrUavn9fpxOp9Qc\nKKsv+PeFAn6/n7///e9iSVqtVm699VZycnJ6xBIyGAwSSlGdkp1Op4Sm1OiFYJp5Q0MDy5cvp6ys\nTF4bMGAACxcuxG63hzRha7fbmTJlioxDaGpq4rXXXmPlypVSyKxCI2lpafz5z38GdG8gnM9PnZ/+\n/fvzwx/+kOrqamnuW11dzaZNm9i5cycHDhwA9Pyn0WgkNzeX5557DtC7qoQijHVmjigQCFBVVSWj\nLFatWkVNTY0M4QO9EL69vZ2mpiYhMBkMBoqKiqT/IoRnYqzBYKC9vZ39+/dLYXh2djbDhw8P20Tm\n4MnHpaWltLe3yxlXNUyKfAD6uZs4cSK9evUSb2nQoEE4HA5sNltIw48GgwG73S4hfZ/PR0VFBStX\nrhQZ2djYKPkwlRoJ7p6volEOh+OsQ1ZDOU1a4bJVUGeDcrFPnjzJ4cOHu2xyuIrLFLvqyxhSKoSh\nksp33nknq1atIjExUS5quLoEe71eDh48KIIgPj6e8ePHS8imJxCcZ7JYLERHR4tibm5ulma1FRUV\ngJ4nW7t2rXTxBnjggQeYPHlyyAWM0WjknnvukdHpx44do7W1ldbW1i5hqJSUFF588UXJO/VEeBT0\nZ2e1WsnKypIaPo/HQ0FBAW63W56j1+vl6quv5qmnnhKCRTgnSJtMJk6ePAnoDDmDwcDw4cO57rrr\n5H01NTVUVlbK2VaF2ikpKRL+UjmpUIf5ysrKePPNN0VJzJs3j5iYmLAoJ7VHABMmTODhhx/m5MmT\nwl6dPHmyGKdqb5xOJ3a7naSkpC6NfMO1PvgsZzt27Fj69evHnj17WLFiBaCH/UpLS/F4PJI2MZlM\n0spLGUzKkD7TYIkoqHNAMb6C8xc2m43ExEQRwuE8mF/lc9V7lMLMzs5m/fr1dHZ2dmlzEuo1qs7l\nbW1t8ix6MoZ8JtTzMpvNEndXHcELCgqkV+GuXbuor6/H4XDIgLcpU6Z08Z5CuabMzEz++te/AvDo\no49y5MiRLoXVgwcP5o9//CPjx48XQdTTzzDYE42Ojmb8+PGMHTuW3/72t8BnFmyoFee5xtS43e4u\nfeUMBgMpKSmiEE6cOCG5VrXXBoOBjRs3kpCQIK9lZmZ2YZOFas0NDQ2MGzeOGTNmAHpOLJxtepRw\nTk1NlXEnwb/pXMqnp85R8PeYzWbS09OZOXOmzDLbuHEj//znP2lubhZqfkxMDMOGDeOOO+4QYtrZ\niGaBQACfzxf6KFU4ZySdJ7q1kOAkZUtLCy6Xq4sgVl0kgqnDX2eofXW5XLzzzju89NJL0u3CZrNx\n44038sgjjwib6ULmPXUHmqbJfvn9fjRNo66uTkY6vPLKK9TW1pKamsqdd94JwNSpU4UeGw7PIHiY\nW0tLC36/X5RRdHR0WEczXE5Q1nJnZyeHDx8G4J///Ccul4tx48aRlpYG6IL69OnTXXr/HTx4kDlz\n5pCQkCB7GNxpIlRQUQOfzyekjbi4uB5t03O5IJhm7na7aWhokJCsw+EgOjq6SzTqXEb5F3jAF3xp\nIrsVQQQRRBDBJYmvjQd1JoKH8wFhs7ovNZxtOOGRI0dYtmyZNJ/s06cPCxcu5IorrhCLR1F/e+IZ\nBXsqwa8pmjfA3/72N5qamsjKyuK2224DupISIp7MxYOylFVYR8FoNHYJtauQ1tks6zPzFaHez7ON\npoicmYuGC37wX1sFFcFnOLPNf2xsbI8l+M8X6jyqli/BxcsRRBDBZYmIgooggggiiOCSRCQHFUEE\nEUQQwdcLEQUVQQQRRBDBJYmIgooggggiiOCSRERBRRBBBBFEcEkioqAiiCCCCCK4JBFRUCFCcN2F\ngtfrlcaokydPZtq0adTU1FBTUxPSti4RRBBBBF9HXJrFMBeI4AJQNRUyuJN4OAv1ztabyul0Avqw\nO7/fT0tLC8XFxQDSDuZiQc3zCZ5W6/f7SUpK+twgwQguL/TkxNNzfT/o91D161OttC7FgnllXF5q\n64og4kFFEEEEEURwieJr40Fpmiatcnbu3MnevXspLi6mT58+ACxevJiBAweetRv22SZ7hsICVVbj\n5s2baWhoIDExkYyMjG5/7vnA4/FIFwnVVv9f//oXe/bsoaOjQ+Yd+Xw+AoEAZrNZPKhp06bx0ksv\nkZqaesl4U+GYl6Us6DPPwZmteC6VZ6CgWlqp2T1qnz0ej4yyiImJ6fEGyeo5NjY28uyzz1JSUsLc\nuXMBfeTFxeymr6BaoFVUVPD+++/Tt29fpk+fDnx2b7+OCJ6Z19zczKeffioTvbds2YLD4WDy5Mnc\nfPPNgN4WLRwTBL4qvhYKStM0mpqaRAB/+OGHVFVV0bdvXwmznT59mj59+hAVFSWCJ7hHV3BfuFBs\nRiAQoKioCNDHDtTV1eF0OkM+avrLYDKZaG9vZ9myZWzatAnQL6XqYq6Glbndbnw+H52dnSLw9u7d\ny1NPPcXvf//7Hl93MNR6Nm3axMqVK/ne974nHaq7u1eqC35FRQVPPfUUAEeOHKGpqYmWlhb53f37\n92fRokXMmzdP5lNdjM746pw6nU6KiopYsWIFnZ2d8lp0dDSDBw9m1KhRAOTl5REXFxeWUS5nQ/BU\ngY6ODk6dOsWePXsYNGgQADfddNNFE3aqd2B5eTl/+tOfAFi/fj0dHR1cf/31XHvttT2+njPDseF+\nNkpB7d+/n9/+9rds3LhRWqApI+z999/nN7/5DQDTp0/nueeeIy0t7aLs22WvoDRN4/jx49x6662U\nlpYCuuAYOnQow4YNY/jw4YBuSTqdTmJjYyUvpRSV6lWnXguFtaxpmigov9+P0Wjskt8JN5Qg6+jo\n4B//+AfvvvsuHo8HgFtuuYV58+bRu3fvLmM2VM5g6dKlACxdupRTp07R2dnZ4wpKXd7Ozk7++Mc/\nAvDCCy8QHR3N/fffH7LvUXsdFxcnZ6C2tpbKyko8Ho+MGKipqeHw4cNs3bqVKVOmADBnzhxSU1N7\nLHfhcrlkb9566y0qKirQNI2YmBhAV6JTp05lzJgxXSY896QiDQQCcs5KSkqoqKggEAhIrvNsQjnc\nUAqzurqaF198kX379knkoKWlhejoaK644gqRC+GCykvv2rULgOXLl7Nt2zY0TZOZTH379uXhhx8m\nOTlZZEUoz5d67na7naNHj+Lz+eSMx8TE4HA4ZOQG6JOS9+7dS0FBgZyznsRlq6CCwwgPPPAAJ06c\nkH+bOHEiDz74IIMGDZKJljabTTorBysjTdM4ePAg1dXVAEyaNCkko7ydTicff/wxoHspsbGx5Obm\nysVQHky4oJ5Pc3MzgUCAWbNmMX/+fAD69et3TqFlsVhk/PSzzz7LgQMHZABduNeraZoIN9A9pzVr\n1vDCCy8A+hj4gQMHkp2dHfKJumazmTFjxgBw6tQpCZ0p61LTNNrb21m9ejXbt28HYMOGDfz4xz/m\niiuuCKvhoQT8kiVLWLt2rawnLi6O+fPnc/fddwP64D9N06iqqpJ5PuEeS3/mOhX5BnQrva2tDYfD\nIec93ErgTLhcLpmUrMhKQ4cOpb6+XtaclZXFzJkzw/Kc1D10u9384Q9/4IUXXqCxsRH4LKxuMBhk\n0oDFYmHNmjV85zvfYfTo0QAMHTqUmJgYvF6vTLW12Wzy/+cD9Rtzc3N57LHHOHr0qBjxasr26dOn\n+e53vwvAnj17qKurY+/evVxzzTXdeBIXhghJIoIIIogggksSl60Hpay0l19+WaZ65uTkAPD000/T\nt2/fLq6xsmRMJpO4/I2NjWzevJmnnnqK9vZ2AO6++27mzZsnLveFIBAI0NjYyI4dOwA9jDBs2DD6\n9+/fI/VPwSGUhIQE5s+fT2xsrLjo5woZqJDaL3/5S0APdcXHx4dlTHYwKcHn89HQ0MCxY8c42ct1\n0AAAIABJREFUdeoUAPX19ZSVlbFu3TpaWloA3fq79dZbwxJqsFqtYiE6nU7xqpuamgB9IrPb7SYu\nLo7ExERAz2v+8Y9/5KGHHmLEiBEAYfGknE4n9913HytXrpR9nTZtGq+++irJycld9tPj8eByuWTP\nwjni/GwIzueeOnVK8njqbvZULgz08PayZct47733AN0jf/jhhxk8eDB79+4F9Hz15MmT6du3b8jX\npWmayKb58+dTWlra5f6bzWZiYmKwWq1CarHZbOTm5uLxeHj//fcBePfdd3G5XGRlZVFVVQXAE088\ncUEelILVauXf//3f8fv9n5vflZCQwOrVqwF45JFHKCkpCXvE51y4LBVU8HC7zZs309bWhslkYvbs\n2QBkZGTI5qlQhxLaHo9HNvkvf/kLa9asobKyUoRebW2tJHS7gw8//JDa2lpAd9tzc3OJi4sTxRrO\nOHzw50ZFRWG1WrFYLF8Yyw4EAnR0dLBmzRq5VFarlfHjx3frIgQPtwt+zefzieJRQuT06dNiPNjt\ndkaOHCl1NABJSUk88MADIc/5GAwGrFYr/fr1A2DkyJHs2LGDkpISCW86HA5mz57NiBEj+PTTTwE4\nevQohw4d4k9/+hM//elPARg0aFDI1qfID88++ywbNmzAarWycOFCAP76179+TtirZ+zxeOQ893T+\nCXTFDfDxxx/T0tLCwIEDmThxIhAeBX42eDweli1bxrvvviu/f8mSJUyYMIFAICCGh9vtJi8vL+Q5\nVq/Xy69//WvJn6qzbrFYuPLKKwE9FTFjxgzGjh0rSt3r9eJwOPD5fBKaXL9+PXv37iUuLk5kUyiY\nhkajUdIcQBdFpYhA3/nOd6ioqLhodZuXpYKCz5hd7e3tIvQVI625uRlN02htbWX9+vUAFBQU4HK5\n0DSNsrIyQGepuVwuDAYDSUlJgJ6DWrduHQsWLLjgtfn9fqqrq0Wwjhw5kkWLFlFZWdmF1t0T8Xij\n0XjO4sjgybtOp5PCwkI++OADEdT9+/dnzpw5IREq6vAHAgFcLhelpaW8/fbbgJ7wr6urw2w2k5mZ\nCegWZ15eHh9++KF8Rl5enlycUMNoNGK32wEYMWIEbrebtrY2OVtms5m0tDSMRiPp6emA7oFXVFRQ\nUFDAypUrAXj44YdDIjz8fj8/+MEPAD13Yrfb+dWvfsW9994LnDuX43a7OXjwoEQAevfu3e21nA/a\n29uFDXns2DGMRiNXXXWVWOA9xVI7cuQIzz//PB0dHdx3330AXHXVVdhsNtra2jhw4ACgy5GsrKyQ\nrsvv9/PYY4/x/PPPi4FsNBpxOBzMmzdP1jNw4EDi4+M/d7/UvbzpppsAKC4upry8nPb2dvr37w+E\n7jkGsy6DPzd4uOnw4cMvGvX+slVQsbGxgM56KSws7MKaW7NmDc3NzWzbto19+/YB+sX1eDxdkouK\npZWUlMTUqVMBmDJligigC4WmadTV1TFw4EAAbr75ZoYOHUqfPn0klOhyuXok3OHz+T5Hq1cXoKOj\nQ9bjdDqJiYnhBz/4gdRFZGdnC/38QnG23+f3+ykrK5PEcGNjIzabjVGjRklydtq0aZw4cYLm5ma5\nQPPnzw8rY06FwxT5oK2tTRLaHR0dVFdXYzKZyMrKAnRDqKioiMrKSjGE5s+fT05OTrf2VdM03nzz\nTf7+978D+ll5+OGHuffee7/UqGltbWX37t3iEQwYMOCcYb5Q15L5fD5effVVIZG43W4SExO55557\nuuWFnw/UWf3lL39JRUUFs2bN4rbbbgP0cLfBYKCiokIiMEOGDCEmJiakz+LIkSMsXboUt9stn9uv\nXz+efPJJxowZIyHi6OjoLzzPynjMzMxk7ty51NbWsnjxYvnbUEEZ9nv37qWwsFAiP6CHHAcMGEB8\nfLx8Z0923IiQJCKIIIIIIrgkcdl6UMotnjFjBkVFRVRXV0sd1O9//3v8fn+XEI3BYCAqKoqEhARy\nc3MB3UPo6Ohg+PDhEttNSEjotoXQ3NxMTU2NJD4nTZoE6In/LVu2ADr91263S2IyXFDU7eAOEW63\nm+PHj1NeXi6hIEU97+zsFHf+gw8+oLq6mmnTpklOIxRr9Xq9lJSUyH6ZzWZmzZrFo48+SnZ2NqBb\naatWrcLtdosHcMstt3T7u78I6rdFR0dzww03YDabOXLkCACHDh1i+PDhXHvttRIKNJvN1NXVUVdX\nR0FBAQBvvPEGv/rVr7r1nOrr6/n9738vtUOpqak88sgjX+o9eb1eXnvtNY4fPy51UE6nU/JQyhNV\ntVGhRm1tLatWraKmpgbQPdK77rqLnJycHrG6A4EAx44dA/TctNVqZcmSJaSmpsp72tvbefLJJ2W/\nbrnlFkwmU0hzwh9//LGE9lSZyzPPPMPEiROxWCwSPlOh9+B99fv9n+vbOWbMGOLj4+ns7JQQX6jW\nq2kajz/+OACrV6+mpaUFg8EgZzwmJob4+Hjq6+slNBkfH99jec3LXkF94xvfoL6+njVr1ojAq6+v\nF3aKEqy9e/dm7NixTJs2TeoL+vbtS3t7u+SfANmYC0UgEKCgoICjR49KW6PY2FgaGxt5/fXXxX02\nmUwsWbKEuLi4LsIi1Bvv8Xiorq7G4XBI3q65uRm73c7MmTMlFKS+12azybPdt28fDocjLMzDqqoq\nqUXp168f1157Lf369ZPv9vv9rFu3DoPBQK9evYDw144pmEwmHA4HgwYNkrORnp5OZmYmGRkZssab\nbrqJpKQktmzZIgLpgw8+4Gc/+9kFx+wDgQBlZWU0NjbKnnz729/+whCZCgXt27ePTZs2UVZWJknt\nrVu30qtXLzRNk+4byggLdd7l2WefZe/evXLOUlNT+f73v99jRd4+n49f/OIX8v/9+/dn5MiR8u/t\n7e08+OCDfPTRR7Imp9Mp4fZQKe0RI0bgcDiw2WzcfvvtAEydOlVki/qv1+ulra0No9HYpV3Vzp07\n2bVrF3l5eYBuhPfq1QuTySRrDOXeqbxuZ2entDtTSt1isVBXV8fvfvc79uzZA8Brr73WY+2qLlsF\npQo66+vrKS8vp7q6WtoaKevCbDaLFzNt2jTuu+8++vTpI68p68VisYTsEgUCAXbs2EFVVZUcxMOH\nD3PixAmamppEKKtu6x0dHSJ8LBZLyDY9uHVTZ2cn9fX1Ilj79+8vB/7MtQeTBU6cOCGFn8FWaHcQ\nCASoqqqitLRU9iE1NZXo6Gja2trkNb/fT69evTCbzWK59RSUkti4cSNDhgwBYNiwYQwePBiTyST7\nFRUVJZ0bVq1aBej5h3379onXfL4wGAw4HA5iYmIk5v9lJQ8qSvDhhx9SXFxMfHw8FRUVAJLHMpvN\nPPTQQwCMGjUqZB6NMl6OHTvGG2+8QWdnp3z2nXfeSUpKSo9Z23V1dZJztlgsXHHFFdTV1Qkp6ne/\n+x0rVqxA0zQRyh6Ph9bWVmJiYkLW53Hs2LG89dZbfPLJJ9xxxx2AbqSemXNWJS+nTp2SRgNbtmyh\ntraWhIQEYfulpKSItxXqZ2k0GvnZz34G6H0Sm5qa6N+/v+ThGxoa+OUvf8nKlSslN/32229z++23\n90gJw2WnoAKBAG63W2ob1qxZw759+6isrJQQhqotsNlsktDOzs7GaDRitVq7jAPwer1ERUV9jq57\noQdBERCCO1YUFRVx4MABTp06JeSOQYMGSWgh2HpSYb/uQLUsAp3e6vP5SElJ6dJANHidgHicmqZx\n6NAhACorK6WXX3cvb/AzVzVQyiNKSUnhzTff5MUXXxRiicvlknUoWvepU6ewWCx4PB7ZVxW+CrU3\nsHPnTrxer3i8Q4cOpXfv3p9LbEdFRfHkk08KJdjj8bBhw4YLVlCge/vjxo2TOjqv13vWM6n2UNWO\nvfbaa7S0tGA0GoWAkp6ezsCBA3E4HDgcDiC0SW515+6//36hbivhf/fdd/dYHVYgEGD37t2irGNj\nY2lpaeH222+XsF9rayuaphEVFSXeSb9+/SQMFyrYbDZGjx5Nv3795L6f63wajUba2tp46aWXACgs\nLMThcHDvvfdKOC/YcA21ggqOMo0fP/5z/x4TE8N3v/td8vPzhSq/f/9+br311h7Z2whJIoIIIogg\ngksSl50HpWkaBQUF/O53vwPg+PHj+Hw+CY0A/Nu//Rt9+/YlPz9frMXm5maOHDlCQkKCWExRUVHE\nxsZ+zhLojpViMBjw+Xy4XC6xbNetW0drays5OTnceeedgB5mUcQFFZosLCxk8ODBpKamdqv2SNM0\nKfIsKSkhJiaG06dPS9Ld5XLR3NxMZ2enJLQtFgvXX389Bw4ckK7wHR0dxMfHd+mPd6EI9tQsFgtT\np06VsMahQ4coKCigsbFR+hcG12aoqvaPPvoIs9mMw+HgiSeeAODGG28M+TgAr9dLc3MzNptNLOCB\nAweelY5sMBgYMGCAkDuOHz9+To/nqyImJoYnn3ySF198UdbT0dHRJe6vaRpOp5N9+/bJs6ipqcFs\nNhMIBGTMTFpaGikpKVxxxRVdii1DlWQ/efIkAAcPHgT0c6RKBTIyMnq0Kazq5g6flTKUlZUJKUGF\nr6dOncqDDz4IIDU+oW7IarfbZS/UawrBhbGapvHuu++yc+dOQN/rXr16MWfOnC6NnHsCZ/ses9lM\nVlYWV199tfyWmTNn9sh64DJUUAaDga1btwohQgnPqKgoaVXzox/9CIvFwsqVK/noo48AaGtr4+TJ\nk6SkpEhTUJPJFPJaJIPBwO23386zzz4rRbmHDh0iIyODb3/724wbN07eZzQaaWxsZMOGDQC8+uqr\nWCwW/va3vwkL63zXpmkabW1tkoNYu3YtaWlpbN68WTpENDc343a7CQQCXToZFxUV4fV6KS8vB/QL\nPXbsWDIyMkRhXOhFDg5RpKSkMHv2bFFQSUlJjBkzhl27dkkiVoUVg0MQQ4YMISMjg969ezNhwgSA\nz4Vnu4NgJdrW1kZhYaGcqdjY2HN+j8lkkpBeaWmphB8vFAaDgaysLCkW/+CDD/j5z3/O+PHjJYx2\n/PhxKioqOHr0qCgJs9lMr169GDp0qIyO8Pl8XHXVVeTl5YVc4Gmaxl//+lcAqflJS0sTYkBPN4Yd\nNmyYnJWmpiacTidpaWlCdHE6naSkpPDAAw+IDLDZbFit1pCHiVV+W4Xv1f3RNK0L0cHlcrFmzZrP\nKdH4+Pgu0xZCvb6vCr/fj6ZppKeni2IdM2ZMj4VuLzsFBXo+KdgyURdjyZIlgC7wKisr2bdvH/v3\n7wd0S2/gwIFMmTJFktzhKJRVFvXIkSOlrb7T6aSlpYV9+/aJhVdXV4fb7WblypVCPXc6nTgcDg4c\nOCBC7nzX5/P5KCwsFM/I7XZTVlbG0KFD5bs//vhj/H4/drtdLlBcXBwWi4X4+HiGDh0K6Ey77Oxs\n4uLiQvaczGYzsbGxxMfHC/Fi0qRJYk0qYfuf//mf5Ofn4/V6+fa3vw3orWri4+OldROEtpWPEiIr\nV64kPz8fu90uRZVf9ndK2ar2Md2F0WiUfVixYgWbNm1iw4YNwhZsaGgQ6rjKN0ZFRZGamsrEiRNF\nQamxDWomFIRu+GJHRwfr1q0DEPaXxWKRVmLKyOopJCUlCbHA5XIxZswYBgwYIMSJt99+m8zMTCnx\ngPDkMBWMRqPIKVXoXVtbK+zezs5OHnvsMYqLi0X42+12RowYEdI7F4zgXDCcu1ejyuUVFxezatUq\njh07JoZHXFxcjxXrXnYKymAwkJOTI6w7VSdjNBrFuty9ezfvvPMO//rXv2SMhtvtxuFwkJqa+jlq\ndahhMplYt26dMMBqamqoqanhN7/5jQgJm80mjT3VYbFYLIwbN45Ro0ad19qCaeAHDhxg7dq1ooTH\njBkjc7HU+w4fPkxZWRnV1dV88skngO5BZWRkUFJSIoczKiqKhoYG2tvbu91dI9iDOhdj0mg0ilC7\n4447WLp0KWazmRkzZgB6wj9c3TcCgQCVlZWA7nU2NzczduxYSVSf6280TWP37t3SxcRisci+dwfB\ntSgPPvgg2dnZNDQ0yHnetm0b1dXVuFwuERZJSUnMnz+fu+66q4u3FBMTEzLFGYwTJ06Ip+71ejEa\njcTHx0sIXfVgDA4nhrP/ZHJysoQXLRYLvXv3xul08sYbbwB6FCUqKoq+ffuKYA6noDUYDGJI1dTU\n8Ic//IGNGzdK2NjpdFJZWYmmabKOYcOG8d3vfrfLOQ+VAlX9NkE32JOTk4UhGPwel8slhv2zzz7L\noUOHGDNmjBBL1OiinkCEJBFBBBFEEMElicvSg+rfv7+EP/bs2YPX66WqqkoSnyaTiY6ODiEfgN4d\n4LrrriMzM7NHtH98fDxHjx4F9BBWUVGRFMKBbuGpgjgVN7/tttu47bbbzrv1v3qvz+fD7XaTn58v\n4bNrr72WIUOGdMkH5ObmkpKS0qX9v9FopLW1lcTEROrq6gDdwuvo6KCiooIBAwbI+8IJZc2lpqaS\nkpKCx+P5XDgmHPB4PPzwhz8E9BBoWloao0eP7vLczuzM7vP5+PDDD3nooYck35iXl8fQoUNDsk7l\nbWdmZvKd73wHp9MpxJtBgwbx+uuvU1JSIuUDDz/8MLfffjtRUVHyt+Eo7FT45JNPxCJXIb6rrrpK\nurKo7w72oMLZxd9ut0sxsuqecujQIQoLCwE9rNWvXz/xKHsCah9SU1PJyclh+/bt0t+xra1Npm0r\ngte9997LgAEDPudBhQI+n0/Kc1avXs2CBQuYPXu2RDT8fj9NTU288sor0qS5rq6OPn36cMMNN8ga\ne3JkymWnoEAPRz377LMALFq0iFOnTuFyubpMflVdvFVNxuLFi3nwwQd71D1VoY6DBw9SXl7Oyy+/\nLIy0MWPGcOeddzJixAhRUGokRnfW53A4qKurkyawr7zyCmPHjmX48OHitjc2NlJVVYXRaJQ19u7d\nG6PR2IVBmJyczIQJExg4cGCPXWgl/B0OB4mJiRw6dEhyQ+rfQi3kAoEAzc3NkqtoaWnB5XLx1ltv\nSQ5q6NChUlipwmyrV6/m+eefp7GxURLxzz//vHS+CBVUqMhutws7z2Kx4PP5xPACWLBgAXa7vUv3\n+nCddTXaXSkhVfx67733fq6paHCIKpzz0IJDaj6fj23btvH0009LLVtSUhLXXXddj438CEZ0dDR3\n3nknjY2NbNu2DYD8/HzJ202ZMgXQO04otmY4usqoOVj79++npaWF8vJyUVAHDhzgwIEDVFdXy3f3\n79+fKVOmMG/ePEkb9CRZ47JUUAaDQcaSb9myhfvvv5+PP/5YEv6g97obN26cdP+dMWNGt9sYXSiM\nRiP9+vXjiSeeEEpwqKrWg6EE95AhQ4SuferUKdauXYvL5RJBrzq4x8fHSyuYWbNmMWDAAGw2mwgd\nNVCtJ5W6WqPBYBBChCIgXHnllWFZi2Iqqd/t8/nw+XysXbtWioSV1ejz+UTgtba2YjKZGDFiBH/4\nwx8AGD16dFiUueqMoqzvjRs3UlZWRkxMjBADEhISwtZx4GyYOHGi9LVUYy2UIldrPtvvCCeCy0qe\nfvppdu7cKTnVyZMn91hfwDOhcmT33nuv9NjLz8/HZDIRGxsr3UIUmSUcz8lkMokMVB14tm7dKjlw\nJT8SExNFvn7/+99nzJgxX8hiDScuSwUFnx3EjIwMVq5cicvlEoZTZ2cn8fHxXUIdF+PhfhHCsR6z\n2czgwYP5+c9/LuHN8vJyNm3aRGVlpXiYQ4YMIS8vj8GDB0uo9ItGDvTUs1OtluCzcQLJycls2rQJ\nQBq4hjrUZzQaiYmJ4dZbbwX0Kc3t7e1omibKEfQL7vF4hIAwZMgQbrvtNubOnSs1RuGk3wYTJ4YP\nH057eztDhgxh2rRpQGg88PNZy5VXXsmaNWsA/ZyFy4A433WBHmIPBAJYLBZhzT333HMhH61xPjCZ\nTMTHx4syOnr0KLW1teTk5DBv3jxA9/LCpUCtVqvUMK1du5aioiKpSwQ9DHnPPfcwb948mSMWExPz\nle5buEK3EZJEBBFEEEEElyQM4YwJnycumYVEcPGgwrRNTU188MEHlJSUSEjt17/+NVFRUWGx0s+k\n4BYVFVFXVycdOaqqqjh9+jR5eXnSADQxMVG6WPR0CFTV1sXFxQltuSfHu1/qCAQCdHZ20tbWJnnE\nnuqq/kVr6ujokLrH1atXc/jwYX7wgx9www03AKEtPP+ytai+mOr7wuh9X/CHRhRUBJcklIBpbGyU\n0K2aWXUxcgiXGoLvbUQpXV5QRlhlZSWnTp1i5MiREjZWXS2+ZogoqAgiiCCCyxHhpN5fIrjgHxcx\nRSOIIIIILiK+5sqpW4goqAgiiCCCCC5JRBRUBBFEEEEElyQiCiqCCCKIIIJLEhEFFUEEEUQQwSWJ\ny7aTRAQRRBDBV4GqHevs7CQ/P5+EhATpl5iSkgLovQQvRo++CL4YEQ8qgggiiCCCSxJfW5NBtbGP\nUDgjiOD/bahuIOPGjcNsNuPxePjFL34BwMKFCzEajRHv6RLFZbsrZ+sGrl5rbW3l9ddfl1lIEP45\nRhGEHmqkQ/CEXzV9NLg9SwQRnAuapkkLq4SEBGJjY3n00UeZOHEiENb2Ppct/H6/PDO/309sbOxF\nU+CXrYI626FSseZXXnmFTZs2kZWVFZLx2183BAIBfD4fzc3NMjcqKSmJ6OhoLBbLRb2wqvW/x+Oh\ntLSU5cuXywyb5ORkBg0aREpKioyBT0tLw2QydRks+HWD6pumWj+BPs8nMTGRjIwM6XCekJBAVFRU\nRGn//wgEAhw9epTHH38cgL59+/LUU08xYMCAsI+g/6pQe6ug/v9itPQKBAK43W7++Mc/8vLLLwN6\nz8f58+fzzDPPyNy6nsRlq6DOBo/HA+iTPg8cOEBbW9tFP4CXEoInwVZWVvLyyy/L1N/hw4czd+5c\nhg0bdtGaaiqPCfRRBB9//DFHjhyhoqJC3hMXF8exY8fo378/oCe5e2K9SpBomiZrrKurY+/evXR0\ndPCNb3wDgOzs7JA3bQ0EArS0tPDee+/x/PPPA1BWVobBYCA2NpZx48YBcPXVVzNv3jxR2uFE8PBI\noEcb5n5VqHM+evRoAO6//36ZPnwpwO/309LSwqFDh2SIYWxsLCNGjGD48OHSBNhut/eIstI0jfLy\nct577z3q6+sBvW/g5s2b6ezsFEOoJ/c5YmpFEEEEEURwSeJr5UE1NzcDsHv3bpxO50VxSb8ImqZJ\nZ+7t27fzt7/9jb1794pFctVVV/Hcc8+RmJgYVitF0zTq6+vxeDzU1NQAUFhYyLvvvsvixYt58MEH\ngfAO3zsTauz6zp07AdixYwd+v59evXrJdE+bzUZbWxslJSWsW7cO0Eexp6enf+WGm1/0vuDGyYFA\nAK/XK4Mf6+rqWLNmDZ988glbt24FkKGGNpuNnJwcAN5//33S09NDsn9qPe3t7SxdupQ///nPVFVV\nAbp3YDQaaWhooLy8HNCH0L3wwgvce++93HPPPcAXD6K8UHg8HjZt2sSyZcvYsWMHoA8OnTJlCgsW\nLJApu1ar9aKGG+vq6tiwYQOzZ88GEI/kYiM4V/6LX/yCjRs30tLSAujeSWZmJllZWWRmZgLwgx/8\ngD59+oTtWar1eL1edu/eLfkn+CxH53Q6hZrfk/jaKKhAICCXpaGhAdDHb/fUbJUv+p5AIEB7ezu3\n3367CFZN07BYLHg8Hok7l5aWYjKZeOWVV8KqHAKBAImJiSxatIgFCxYAumB98803ee6557jlllsA\nyMrKCtsazoTP52Pfvn3s27cPgIEDBzJy5EjS09NJSEiQ9+Tn57Nv3z6OHDkCQG1tLSkpKV85iXu2\nfTozD+Dz+WhqamLTpk1ypqqqqti8eTNtbW0yLkHtu9/vFyWalJQUsjOnBEdxcTF/+ctfqKmpkTyr\nyWTCZDKhaVqXGVHFxcU8/fTTlJaWAvoMrejo6JCsSYU2ly9fzkMPPURTU5N87unTpzl06BDLly/n\n9ttvB+DWW28lLS0No9Eov0Ux5sJ9L/1+P6+99hq7du1iyZIl8t2XAtRZKygo4NSpU3g8HhH+Pp9P\nRsyoac65ubncfffdYQtlq72pra1l+/btdHZ2yhrNZjO9evW6aOHbr5WC+te//gXolkBsbCxJSUk9\n8t1ftnknTpxgwoQJNDc3y2GIjo5m1qxZWCwWNm7cCOiWstfrDfthsFqtZGVldbmweXl5uFwuVq9e\nLQq+JxVUe3s7u3btYvDgwYA+Tj03N7fLfByj0Yjb7aa5uVkukGL2+Xw+UVIX8vyMRqMIerfbzbFj\nx8jPzxdSwpEjR+js7MRoNIrxYDAYiIqK4o477uCZZ54BCOkwxWBL2+VyYTQaRVkPGzaMvn37Ul9f\nz/HjxwFoaWnB4/HQ3t7OoUOHADh+/Dh5eXndNng0TeMvf/kLoCu9pqYmAoGAzDGKjY3F4XBgNBpZ\nvnw5oBM5RowYgdvtFuV2/fXXM3bsWGJjY8N6zuvq6njvvfewWq2kpaWd831nYwOHE4qIALpB2tzc\nzMiRI7nxxhvlPSUlJZSXl4tXpZRXuMZyqHNfXFxMc3MzLpdL7pf6znAMCf0q+NooqM7OTlavXg3o\nD9xmswE9fwCDUVhYCMCYMWPweDyYTCamT58OwNKlS0lLS8PlcvH0008D8M477/Doo4+GzdILfgbK\nilXPJyoqiuzs7IsSltE0jfXr19Pc3MzQoUMBPXR3plfk9XrZuXMnxcXFXHnllYBOkjAajSGhC6tL\n6Xa7xVpVFPfOzk4yMzNJS0uTupoBAwYwY8YMFi1aRHR0dLe++2xQ+5CZmcno0aPx+XxCiLj55psJ\nBAI0NDSQn58PwKpVq6ioqMDtdkt42+l0omlatxVUe3u7nNO6ujoMBgPp6enMmjULgGnTptG7d28K\nCgooKSkBoKioiNdff53q6mo5Z+vXr+evf/0rY8aMCUuUQO3h8uXLKSoq4lvf+pbIgnPax5i0AAAg\nAElEQVShp2REIBDA7/dTV1cHwMGDBzGZTAwZMoS4uDhAZ6X269ePkydPyvsGDx5MIBAIm4JSz6y1\ntRXQ71nwM4mLi7to5JJLw+eNIIIIIogggjPwtfCgNE3jjTfeoLq6Wl4bOXIkZrO5S8wees6TWrdu\nHTfffDPwWbHb//3f/3HTTTcBn1nHPp+PvLw8AMaOHSsJ5nDjzOfQ2dnJtm3bsNlsX2pxhhpOp5O1\na9cyduxYBg4cCHTdL7WH7733Hq+++ioOh0No3aEoIgz2JEG3IN9//30OHz4sFPdBgwZxyy23MHny\nZFlbcnIy0dHRQr8NNdQeZWRk8Mgjj1BWVkZiYqL8e1VVFUVFRRw+fBjQn1lGRgbJyclcc801sm71\n+y707AcCAUpLS8XCBoiPj+dvf/ubRATUHkycOFGe2a9+9St27dqFy+WSPSwtLZX/DweUd7thwwZi\nYmJ44IEHzpl3BKQQ3GazdVtGfFHzAFUw3NzczIEDBwDdE01PTyc7O1v2NTExkejoaOLi4ujXrx+g\nn/FQly4EQ8mi1NRUWlpa6OjokMiBxWLpkbKFc+FroaDa29v53e9+J65qTEwMTz31FBaLRS6Dpml4\nvV5JLgMSGgpGKA7BqVOnuPnmm2WT4+LiKCwspG/fvvKeQCBAZ2cnv/jFL+RCP/HEE2EnR4D+LBQx\nQOUG8vPz8Xq9jB49mj59+oRtDWeDz+cjNzeXG2+8UXIaqq7G7/dLPuXxxx/H5XJx0003cdtttwF0\nyVF1B4FAQNhL69ev58iRI5SWlkqobPDgwQwaNIj4+HgRxjabLWTf/0WwWCzk5ubS2NjImjVrACgv\nL+fTTz+lra1NFOSECRO4/vrryc7OZsSIEQCSF+rOGjVN4+mnn5bcidFo5LrrrmPatGlizKjPN5lM\nUqPWt29fLBYLTqdTzp7ZbA4bS9Xv9/Phhx8C+vO5+uqrSUlJ6XLu4bNQG+hK3mq1kpCQ0O0wbbCR\nA/ozUffL6/Xicrlob2+X+97W1obZbCYpKUlyr3a7Ha/Xi8PhkLNnNpvDGnZX57lfv374fD4hAal/\nmz17dqSTxIUiEAhQVVVFdXW1CPc777yTESNGdBHAXq+Xzs5OrFarUIfT09Olc0KoLkxbWxtjx47F\n5/PJgS8oKBDCgbokJSUl3HXXXRQWFoq1m5ycHNaDqA5eXV0dDQ0NWCwWUaJOp5Np06Yxfvz4Hqfn\n2+12rrnmmi6/32AwoGka+/btY9GiRYC+13feeSf333+/dKG+kH07mzfh9XqlCPbFF1+kubkZr9cr\nF/P48eOsXLkSh8MhXt4999zD9OnTw97LzWAwYDab+eijj3j77bcB6OjoEEJNW1sboOc8R40axaxZ\ns2QPQ2F5+/1+amtr5exaLBZGjx59VuUc7PGePHmSzs5OAoGA7OuwYcPo3bt3WM75iRMneOWVVwA9\nh3nfffdhNpulW0pNTQ1JSUn4fD4pr2hvb5dCWJV3vJC9DFZ6wSzFYGPYarUSExMjXVCMRiOJiYlM\nmzZNDDO/34/dbr8o3VHsdjuVlZW43W5Zf3JyMpMmTbpoDMjLXkFpmsaLL76I1+slOTkZgEceeQS/\n309FRYW40zt27MBisRAdHS3MrIULFzJy5Ejsdnu3L7ES9HPnzqWpqYnY2Fi2b98O6EluTdNoampi\n9+7dALz66qvk5+fj9/uFwRNuxaAsvPLycrZt24amaSJss7KyGDVqlISEehJWq5WhQ4fidrvlOWqa\nxqFDh1iyZIkI4B//+Mfcf//93W7HdObf+v1+ysvL+Z//+R9AV+BKGKv1qPq12tpaIQHs2rWLa6+9\nlv/6r/8S0kY4hIrBYMBqtXZ5Pj6fTxLn6rXi4mKee+45CgsLefTRRwHIycnpNj3ZYDDg9XpFSJnN\nZqFHB3u88FlJBcCxY8fEKFJ/269fv7CERF0uFy+99JKsY/bs2eTl5eHxeKRU4MiRI5jNZmpra6mt\nrQX052i1Wpk9e7aE3y9EqQcbuWeSkYJhsVgYNGgQoBMiFCFBnTfFEj3bGQ/3vTx16hSVlZX4/X5Z\nd15enjBHLwYiJIkIIoggggguSVz2HlRHRwf/93//RyAQkGr+trY29u/fzz/+8Q+psrfb7SQmJlJf\nXy9kisbGRv77v/+7255LcA3Wnj17sFgs3H333ZLk9Hq9FBUV8eabb0qoLyMjA5/Ph91uFzJFuBOR\nypK+8sor6dOnDxUVFVIM+MknnxAbG0tWVpZQXnsSXq+XkydPynevXbuW5cuX09raypNPPgnA3Xff\nHZZnZDAY6OjoEA+8tbVVQsNnhjbMZrPkXdxuNxs2bKCsrIxf//rXAEyaNCkseSmTycSkSZOkdOHI\nkSNSrxLcF6+pqYmVK1eK1/CPf/yDsWPHduu5mUwm7r77bvEcGxoaWLduHXl5eRJ+jY6Oljq1f/7z\nnwBSLBycA4uNjf1crqa70DSN/fv3U15eLn33brjhBux2OzU1NRQXFwN6GNvpdHYhxTQ0NIiHrggf\nyis8X3yVPTcYDLIX0dHRUrumajatVuvn0g7hpJjDZ2mHpUuX0tbWRiAQEA9qzJgxF3UUyWWvoJqa\nmujo6MBsNovwX758OW+88QbNzc2Sqxg/fjzR0dHU19fT1NQE6IchFOEGv9/PP/7xD0APGSiFt2vX\nLkAP3e3fv585c+ZISG3OnDm4XC5yc3NJTU3t9hq+CtQBt9lswvZSArioqIhPPvmE1NRUCTn21MFU\nRIhf//rX0rzW4/EwadIk1qxZQ+/evbusP9QwGo0MGzaMjz/+WL5bhdVUyK6jo0Nyl8eOHQPgt7/9\nLYWFhVRVVbF+/XpA714Sjo7wRqORadOmyeceP36c+Ph4BgwYIGd48+bNvP7669TX11NZWQnoebIN\nGzaQlpZ2wWsyGo3MnTtXunwsW7aMuro6fvazn0lRbnx8PK2trZSUlEj9jsfjwWaz4fF4JD+j2pCF\n4hkFFzK/+OKL1NfXSz43Li4Og8FAdHS0EBAMBgPDhg0jISFBitFXrFjB22+/TUFBQbfCs4qEBV/d\n0DSZTHg8HqqqqqQNU1RU1FlDjOEM7ylj7MMPP5SQo8qfX3PNNReNwQeXsYJSh9PpdBIfH09KSooU\nMe7cuZPa2losFovkBm644QY+/fRTrFareBIzZ84MSfsQg8HAHXfcAeiJ6oEDBzJ9+nSGDx8O6Fbj\n6NGjsdlsEp8/fPgwgUCAq6666qIkII1GIzabjWHDhgF6bsXlclFQUMDMmTOB8CsotYdHjx5lyZIl\nnD59WoTEFVdcwauvvtpj/b/MZvMXxtrtdjvJyckEAgFRmK2trTzzzDPU1NRI0t3lcuFwOEK6NmVB\nx8fHM2/ePEDPZShPTQmvmTNnMnv2bObPny+U8OLiYrZt2ybtqy4UcXFxPPbYY4BuULzzzjvU1tby\nySefAEin92DvyGw2YzabhSgBOnHixIkTjB49utuCT33mpk2bKCwsJCMjQ4xUdafi4uJEaZnNZlGM\nyktKT09H0zT69u17wZ6Tx+Ph3XfflUjIV2l9BroxW15ejt1uFwXV0x0bFMkMdG9SrV3dhQEDBvTY\nWs6Gy1ZBKaSkpHDzzTfj9/s5efIkgIQ3kpOTpeHi/v37efPNN2lvb2fq1KnAZ9ZBdw+ECr8AbNmy\n5XO00ODPf/PNNwH9kptMJrn0FwN+v18uy/Dhw3E6nXR0dEir/czMzLBeltOnTwNw4403UllZic1m\n4+qrrwbgueeeO6fCCBaCPU3oCA7RDBs2jLS0NFpbW0UhBPf06y6C6dGdnZ1dwotn80CMRiNXXnkl\niYmJ0ibH5/Oxfft2FixY0G1iiQpD/eY3v2Hu3Lk89thj4k2q2pngrhUZGRl4vV7a29vlt9hstpAR\nSRRxZevWrTQ2NpKZmdklImIwGLBYLF0UoWqLpajeL7/8MqdPn2bOnDkXrDBbWlro1auXeI69evU6\nZy1hsKfV1NRE3759SU9PF+Wo1nBm4+JwnXOPx8Orr74KfNZJwmg0kp6eDnDRRu8oREgSEUQQQQQR\nXJK47D0ou93O1KlTWb9+vVDK29vbsdlsJCcnU1BQAOh1R42NjeTk5PDQQw8BeuV0qCwTZfl8kRWm\naZrQfwOBAGlpaWKp9CTUYMDq6mrJqzgcDnJzc6moqAh5EvtsKCoqkjEI9fX1JCYmMnPmTK677joA\naXoaExPzOQtY5YhAt8iD62x6AiqfUldXR1tbGw6HQ8J+oexZpryxwsJC1q9fz3XXXScTos9m2fr9\nfvbt2yf5FdDDWpMnTw7JOQ8mOkyfPp2JEydKeKitrQ2Px8PJkyfl/OTk5PDss8+yYsUK8RocDgcJ\nCQkhWY96Bn379sVgMFBXV8emTZsA3Su3Wq24XC4pwK6pqaG0tJT9+/cLOej06dNkZ2dz1113XfAZ\niouLIzs7WyI3ioQUTJZRXefb29ulzEV5Kmcrcwke3qmaC4Tai/L7/Rw4cIAtW7bIaypkrORSuAka\nX4bLVkEFd7g2Go14vV4JGfn9fpl5pNzuxMREbrjhBn7+859L6KinH3plZaW40QaDgSVLlvSoYFUC\nz+VyUVdXJwlr0Jt9qpYmilgSDqi2Oddff73UoqSlpfHTn/4Ut9stRIVly5aRlZXF7NmzmTBhgvx9\nVFQUPp9PYvY+n+9zYZEv29fuhAg1TZMz9ac//Yljx46Rnp7OyJEjgdB2tlDf89hjj/Hpp5+yd+9e\nfvjDHwJ6Ky+TyYTf7xcl8aMf/Yi1a9fS0dEhwvvaa69l9uzZYSFt2O12If2oNY8dO1a+y+/3M2/e\nPFauXCnnPDo6OmQKSu37/fffT1ZWFh988IEI2x07dkhNlur2XlNTQ0JCAoFAQGqRFi5cyF133SVK\n7kJgNpuJj4+XMH9xcTGnT58mJSVFurKoYufOzk4JlcbHx2MymbqcRxUC9Hg8cjeTkpLCQlTw+Xxs\n2bKFsrIyeU0Vnav8c319PQkJCV1Cyj0pNy9bBRUMr9dLdXW1CH9lebjdbslp/OQnPyEnJ+eiUSZ9\nPh8zZ84UJREfHy/Cpqe+X1mSTU1NHDx4kKNHj3ah2DscDpKSkkS4heMgNjY2snDhQmpqaoQp9I1v\nfIN+/fqxZs0aGT1it9vJzs6mtbVVCAhJSUkkJCR0UQLB1t1XWa8qijxzrtIXvV9B0zRKS0tZvHgx\noJM7fD4fkyZNkiLPUBbqqvO8f/9+6urqqKqqEpr5uHHjSExMZOvWrcJ8dLlcBAIBTCaTEAPefvvt\nC07+ny/O7MhiMplk1EdwwXOovIFgL/qb3/wmc+bMYcOGDYCel1IlJer85ObmisGjSlKsVis2m61b\ncsFkMpGQkCA5MaPRSH5+fhfjqaGhgZaWFiZPniwyoKWlhYaGBjIyMuTceL1e8vPzcTgcZGdnA3ou\nPRx3UZE0gg07lbdTkYD29nb8fn+Pd7VQuOwVlNVq5frrr+fdd9/tUg9isViYPHky//3f/w1wUYdu\n+f1+XnrpJYqLi8WS/NnPftZjTVn9fj8dHR1Sx7J9+3Zqa2upr68XwkhiYiI2m+2s/QlDASWgfvzj\nH/Ppp5/i9/tFcNpsNv7+979z9OhRRo0aBejtqqZOnUpUVJRcDnWButtXzu/3S8gpWLkF/79SZOp9\n6vndcccdUlcTCARITk7m/vvvl1KBUJ2x4DBLWloadXV1uFwuISWcOHFC1qgEnmqJNH36dKlF6inl\ndC6oEFFwc9bGxsaQz4NSlHLFpLvpppvk2QRHW848P6EIX6m/D+6+0NLSQmFhIXv27AF0TyQ1NZW9\ne/d26RdYXFxMVFSU7NOAAQNIS0tj+vTpEskIR9mCWrfD4RBPTb1mtVrFe1d1Y6FsB3c+iJAkIogg\ngggiuCRx2XtQRqORpKQk/r/2zjy46TJ94J+kSdP0Pgm0QKEcArXQrVyCAlauEURBZkHFax1GWZEV\nWJB1WVfHc3Fh1AVXBI/xGkWqIggq1CooKnIv5aiFQgulBy29c3/z+yPzvpsgq/wkSYO+nxmGTiZt\n3nzf4znf53n88cely+jbb79l7NixPProo7KMfajvFng8Hmnyr1mzhoceegiPxyMrms+ZMydkY9I0\njdOnT/Pxxx8D3mrder2eYcOGSVdQbGxsUONhwu0qNLKYmBjZnLBHjx7k5OSQmZkp240Iay7QCE1Q\nxL+ioqIwGAxERUVJa6m1tZVTp06xf/9+6R4qKipi//79NDU1yXlLSUnh3Xff5corrwzKWEWc7dln\nn2X+/PkcOnRItpMQtfjgv7GYhIQEHnroIe6///52Tw8WxMTEYDQa5fw3NDRQUlJC586dg3oBVFwH\n+LnPCOQeFJa+Xq9nxIgR5OTkSDfdt99+i8PhoK2tTSaxnDhxgurqaiwWi6w+f9111zFy5EiSkpIu\nqkP0hRAZGcmQIUN45513AK/LUTw3Yb2np6e3W6FYAF0oMrYukIsaiO8lQU3T/CoKhxJR1bipqYk5\nc+YA3k6nokvrV199BSBLsoRiPA6Hg++++04uxMrKSvLy8rjvvvukAA/2IhRz09zczIEDBygvL5cZ\newkJCSHr1yXWiYjHVVZWsmXLFoxGo0w2qK2tZe/evZSVlcn3ud1u9Ho9KSkp8iLz4sWLSU9PD8mY\n6+rqWLNmDQUFBYA3/mUwGMjJyWH27NkAXHXVVcTExLSbK/tcxCXQ/v37y+otUVFRzJgxgxUrVrRr\nCZ1gc66SKiqRCDcseJW25uZmunfvLl8zGAxB7f107hhtNhvPP/884E1MMhqNzJs3T1aTMZvNgThL\nf/Ev/2oEVHvjezvc6XTyxBNPyPJHp0+fxmw28+abb8oW2aE6RITAtFqtsq+SXq8nNzcXs9kcNodZ\nqBHWUlNTE6Wlpbz++usyRmez2aioqJAtLcBb7b1///4sWLBAls4JVmzgf+EbyxG0V2zgQvB4PDQ2\nNjJ8+HD5bHU6HatXr2batGntWkKnPfhfZ224zl8A+cVf8NerwoQYschE6m9zc7M0+SMiIrjyyivJ\nz89vl8oHBoOBmJgYmQb7G9gQP4vQbOPj48nLy2PAgAGyaKjJZMLtdvu5GX0LeLbX8wtnYXQ+dDod\nsbGxLFy4kCVLlgDe0kITJky4pL5HoPgtfueLRSVJKBQKhSIsUS6+IOFwOGS7AZPJRJcuXdo12KhQ\ntBe+TRWFZaqsid8UKgalUCgUirDkFwsopdIrFAqFIixRAkqhUCgUYYkSUAqFQqEIS5SAUigUCkVY\nogSUQqFQKMISdVFX8SNEZmdDQwM7d+4kPj6egQMHAj/dkFGhCDfOV3lDcemgBFQ7caHN9UKJx+PB\narXyzDPPAPDCCy/IBmufffYZAL169QqrMSsUAt/WGaIslG+7jUutEke4Idq6QOiepXLxKRQKhSIs\n+VVZUELCC00q3DQmTdOoqqoC4OuvvyY7O5s+ffqERYUJt9tNfX09DzzwgLSW2traMJvN5ObmkpiY\n2M4jVCh+Ht+OBk6nk4aGBlkpPDo6moiICL+OzIqfxu12s3v3bubPn09FRQXg7Uw8evRo7rrrLtm+\nPljP85IXUKKMSkVFhawe7nK5yM7OZvTo0bKnTjgIAavVKjtsfvjhhxQXF/PQQw+1a+8eUYJm+/bt\nLFiwgKNHj8pnlpuby6233srUqVNla/j22NhC8XA4HHg8HqKion6zB4w4gNva2ti1axcFBQV+/amy\nsrLIzs6mS5cugLddQiir1muaJns/6XQ69Hp9SFvfiOdjs9n4/PPP2bBhA3a7HYDx48czfvz4kFeh\n//+gaRoul8uvkn5UVBRmszmkbThEl90FCxbw8ccf43K55LkQHx/PoEGDiI6O9jMKgjG+S1ZA+WpK\np06d4l//+pfU/G02G3l5eeTk5EjtyWg0/qj/TKhbJVRXV7NixQoA9u/fj91u/58l+EOBy+Vi3bp1\nAMyfP5+Ghgays7O58847AbjpppuIj48P2cY43/hqamo4fPgwAFu3biU/P5/hw4e3S7KGiGuIg8Pp\ndBIZGSkP4VB8fkNDAwDLli3js88+4+TJk1KLjYqKwuVyERERQc+ePQGYOXMmI0eODNocivXb2trK\n6tWrWbJkiWwXLvqyTZ8+nZUrV8oxBgPRVkYIx5UrV7J06VIaGxvla2vXrmX69OksXbrUT3FtT2El\nFOyWlhYASkpK2L59Oxs2bKC0tBSAsWPHsnjxYrp06RKSsba2tvLAAw8AsG7dOpKSkhg+fDhXXnkl\nABaLhaysLL/2L8EyAC5JASWCnwAtLS3s3r2bbdu2cerUKcDb9Ku4uJgVK1bI5nLx8fFYLBZiYmKk\nRpWYmIjRaCQyMlIeMB6PJyiL1ul0cvLkSY4cOSLHLTZye+B0Otm4cSPLli0DvIfJ0KFD+cc//uHX\n1TbUQlzMa2NjI2+++SaHDx/mhx9+ALxKxt133x3wzSA+17enl/gnBKGmafznP//h5ZdfprCwEPD2\nktLr9QwcOJC3334bgNTU1ICOzRen08natWsB2LRpExEREYwbN45OnToB3r5jJ06cwOVykZaWBkDX\nrl2Ddgh7PB7ZiPD6669n7969OJ1Ov+Z7DoeDgoICevfuDcBf/vKXoBxmwkW9c+dOAFatWsWZM2dw\nu93yPQ6Hg40bN3LZZZdx6623At6uxWL/h6pppu+Yy8rK+Otf/8qZM2cAsNvtnD17lsrKSilYv/rq\nK0pLS8nIyAi6J8hqtfLwww+zYcMGAJKSkli8eDETJ06U89rW1kZ0dDQGg0GOJ1jhlPb3eykUCoVC\ncR4uOQtKmJVCu9i1axeffPIJdXV1UlvS6XTU1NSwfv16aVUlJSXJxn3CnLZYLOTn55OWlkZTUxMA\nmZmZJCcnBzQupGkamqZhNpv9vodoIx5qXC4X33//PR988IHU3K666ioeffRRunXrJjWlUMYNnE4n\n1dXVfPTRRwC8//77JCUlYbFYGDZsGAB33nknGRkZAR+X0P7E+qmsrKSwsJDi4mL5HpfLRVFREadO\nncJqtcpx63Q6vvnmG5566ikAlixZEhT3o6ZpnDx5UrZ8N5lMTJkyhaFDh0rLr7m5mZiYGBISEqSL\nLyoqKmhrrLm5mfz8fMDbgj42NpbbbruNWbNmAd7417Jly3jttddkfHju3LlER0cHfCyaptHS0sLW\nrVsBpPXkm3ouxrxp0yba2trkGK+66ipMJhMpKSmA91yIjIwMyjyKLsMAjzzyCK+88gput5v4+HgA\n+vbtS2xsrJ9F0qFDBwYOHBh0t7bT6aSwsJA1a9ZgMpkAuOeee/j9739PZGSk9G4Id3EoXP+XlIAS\nrhin00lJSQkAH3zwATt37uTs2bPyAba1tWEymYiMjJSLITk5GYvFgsFgkMInLS0NvV5PS0uLTBao\nrq7GYrEEbLzgNeV1Oh11dXVyY3g8Hrp27RrSWIp4PidOnOCVV17hwIEDDB48GIC//e1vUjiFOqB9\n9uxZPv/8c7788kv52ZMnT2bYsGFERkbKGIvFYgnaYev7nW02Gz/88APFxcVyLRgMBlJTU7FYLHLc\nLS0tlJeX43Q6OXr0KODd5MGYU4fDwbp162QW6LBhw0hLS8NkMsnDJD4+noyMDOLj44OuZDgcDubN\nm8ehQ4cAiImJYdu2bfTq1Ut+f4/Hw8MPP8zXX38tuxW3trYGXEAJBefo0aMcOHAA8LqqhDLr2+06\nJiaG5uZmvvvuO8DrSv7oo49obW2ViQEul4u8vDyeeeYZunfvHrBxulwuysvLue222wBvHBq8bmHh\npk1OTub48eM4nU4SEhIAeOyxx4iPjw/aXIqzr6Kigtdee00qPwDTpk3DYDD4ud/B/2wDpMAKNJeU\ngALvxqiurubll18GYMOGDVRVVeF0Ov0CdqJJ4OjRowEYMWIEdrudpKQkkpKSAPw2kvh59+7d9O/f\nPyBjFRMqtPPi4mIpoPR6PVOmTAlpjEdYnevWraO8vJwePXpwyy23ANC5c+eQCydhyT755JO0trYy\nbtw4RowYAfy3xfrZs2eDFlj3xXeDRUREUFlZSXJysozH9ezZk7S0NMxms4y7lJSUUFBQwPHjx6UG\nbLPZAj5eTdPYsWMHq1evluunoaGB06dPU1lZSa9evQDIyckhLi4u6PPo8Xikpi0Uhscff5zevXv7\nKRA6nQ6j0UhmZibffvstAHV1daSmpgZ0fL4WpJgH8Qx8P0fTNOx2OzU1NXLt1dbWYrVacTqdfodt\nVVUVkZGRMrZ4sTidTj788EMee+wxysvLAW/8a8SIEYwaNYq6ujrAmwhUX19PVFQUS5YsAbzKSDCF\nk1Cu1qxZg8FgoF+/fvzud78DICEhAZfL5ZedKWJ2vs9XpZnjldaaplFaWio1oOrqarm4fB9SWloa\nkyZNYty4cYA3IULTNBISEuRB5HQ60TQNq9VKdXU14A18B+JhC7ceeCfP6XSyf/9+mQGWkpIS1IV3\nLh6PR7o7Dx48SKdOnbjmmmsYPnw4QMhTt51OJ9u3bwe8WuykSZMYM2YMRqMR+G8A+fnnn+f6668H\nYPjw4cTExATNihJ/Nzk5mcsvvxy73S5dWBkZGSQlJeF2u+VaKSoqorq6GoPBIC2tQLqGxcF76tQp\nHnzwQc6ePSsPiT179rBnzx6ioqKkUM/JyQnYZ/8Umqbx3nvv4Xa76dq1KwC33Xbbj+bF4/FQWVlJ\ncXGxXPc2my0o6ywqKoqBAwfKz+natSvr16/n9OnTUhi53W5aW1ux2+3yDBDWg6+lJQ7e+vr6ix6X\nEHqHDx9m+fLlVFdXy8M/Pz+fW2+9FZvNxpo1awCvdyM6Oprbb7+dqVOnAgQtQ9TtdlNSUsKLL74o\nX7vsssvo2LGjPBcMBgNutxu73S7nV2Su+s5jsM4OlSShUCgUirDkkrKg9Hq91HREUoNOp/vRXSK9\nXk9OTg5DhgyRgU+TyYTRaESv1/tpeiJRQbwvISHhR4HVX4JvDTBxV2vLli1yrO20TjkAABIaSURB\nVH379pWpwKHA7XazY8cOwBtAHj58OBMmTJDxAN/nGGxLSgT89+7dC8DNN9/MkCFDMBqNUgNesmQJ\nL774Is3NzfJ9M2bMYMyYMaSnp8u4i2+aa6CIjIyUiRBCAxYxAJvNxvvvvw94EzmsVivdu3eXacuB\ndO8J7f7pp5/mhx9+QNM0aWHq9XoyMzPp0aOHtNrKyspITEwkJSUlqLFNYYlER0dz1113Aef/3m1t\nbcyZM4djx45Jt3ogYzoCsW47deokrY6pU6cyZ84cFi1aJBM0RGKSiE+L3+nduzfHjh2Tc11TU4PJ\nZJKu04sZl4gZvvzyy5w9e5Y//OEP8pl16dIFh8PBoUOHZExdr9dzzz33sGDBArnGg4Fwsa9atUom\nA+Xk5DB9+nQ6deok51PE93yfma/1pJIkfNDpdJhMJnJzc5kxYwbgNZ23bNlCfX293JSZmZkMGjSI\npqYmedcoPj4ek8mE2Wz22+TgLYEiSvmMGTMmIA89IiLCL5C4ceNGmTEHMG7cuJBc7gTvImtqauLT\nTz8FvG7MrKwsjEajdBkJwS1u/4vXgrEA6+vrefvtt+V85ebmEhERQUVFBbNnzwa87jOHw+GX+Vha\nWsqhQ4e44447yMrKAiAuLi7gh7GmafI+kXBhORwOUlNTOXjwIO+99x7gjQP16dOHe++9l759+wKB\n3bDCRZyYmEheXh59+vThiiuuACAvLw+LxYLdbufEiRMA7Ny5k+bmZsaPHy8vogYDj8dDbm4uTU1N\njBo1Cviv8ujxeDh58iQAf/zjH9m2bRsul4vMzEwgsALcV6HS6/U/WgcdO3Zk8eLFfP3114A3CUC4\nq8TzmThxIlOmTGHHjh0yoerIkSOUlJTQuXPnixqfpmnSHVxVVcUNN9zAn//8Z/nZ4vDftWuX/J1r\nrrmG++67z2/dBwNxJjQ3N8t1lpOTI13VNptNfoeWlhZSUlKCfufpfFxyAspgMJCQkMC8efMAb7bO\n1q1bKSgokBpH37596d27N99//z379u0DvAHUuLg4xo4dKzeLwWDAarViMpkCromfO4mff/651ESA\nkCZIaJrG2rVrKSoqAiA7O5vOnTtTXl4ug7MOh4NOnTpJTRe8cbJApyl7PB55NUDEB48cOcL27dtZ\nunSpTECIiYlhxIgRPPDAA1KTbW5uxmaz0atXLz9tLlCI+XA4HFRVVVFWViZfs9vtdOrUiV27dnH6\n9GnAu37Gjx/P1KlT5XgCOadCkVq0aBGaphEVFSUPYfG93W63fK2oqIi33nqLjh07ytT8YMTrDAYD\nkydPpqGhQcY1LRYLVquVd999V1qY9fX1xMTEoNPpZDKO+E6BQGTd+e5fX4SVKZQeYZGbTCb5fG68\n8UZSUlLIz8+XMdFdu3bRoUMHJk6ceFHj0+l0suTU3//+d1JTU/1iqC6Xi9LSUrZt2yYveE+cOJGk\npKSgnw2ic0FCQgJDhgwBoEePHtTV1XH06FG5D1NTU8nOzv6R8A+El+lCuKQElCAiIsIvYDd27Fj6\n9+8vK0SIRdC3b19Z++6TTz4hIiKC5uZmeUdEp9ORnp7OoEGDgvqwm5qa2LVrFx6Phw4dOgBIzTwU\nVFVVsWrVKvnMBg8eTFlZGd9//z2bN28GICsri1GjRtGtWzd69OgBeNO/u3btGlArRdM0tm7dypEj\nR6RwXLNmDbW1tTidTpl1+dRTT9GzZ08/K1NsimC7GMxmM1dccQV2u52ysjLAe49uz549lJWVyYOx\nS5cuTJgwIWi17sR8iXsx/+s9QuvV6/V88803/POf/5RurWC4ifR6PfHx8aSmpsp7RytXrpRZqkLJ\nmTRpEl999RVVVVUMGjQICNx8eTweFi5cCHgt8DvvvPO8CSo6nU4mukRFRREbG8uAAQOkOy8iIoLG\nxkbWrFkjE69qa2u59tprL9qC0ul08ln4FlUVlp/D4eD999+ntrZWunPz8vJCcvXE7XZjMBgYOnSo\nrLNZUVFBUVERhYWFMhuyf//+ZGRkyPFD6IQTqCQJhUKhUIQpl6QFdW5ShMFgIC0tTV4GTE5Oxmg0\n4na7pc9b1LdqbW3lrbfeArxum2nTpsm0z2CNs7S0lPr6evR6vXQbBNLV8VO43W6WL19ORUWF1IKa\nmppYtWoVR48elbG3YcOGceONN5KYmCj9zyUlJZSWljJy5Ej5vovV7jRNIzo6msjISBkfFMVNFy5c\nKC0oUdz33CSYUPi/Y2NjWbRoEcePH5f3jlpbWyksLKS8vFxadRMmTCA7Ozvo4/m5vy/mZPr06axY\nsYIdO3bIeGdGRkZQxpOUlEReXp5MYHE6nSQnJ9OzZ0+efPJJwLv+169fj8FgkJ6DQFpQoibixx9/\nTENDA/fff7/c7+JzjEYj2dnZAH6XiEXR3SNHjrBu3TqKioqorKwEvIkTPXv2vOgLxSIk4Ttm3yo4\nFRUVHD9+nJMnT3L55ZcDSMsl2Ih7TWazmWPHjgHw5Zdfsn//fmw2mzwTs7KyiIuLw+VySYs+lMWj\nLykBJW4zOxwOme2l0+mwWq00NzfLrDjR70Wv18uNcccdd1BfX8+LL74og8o1NTWMHTs2aNkyIvj4\n3HPPoWkaJpOJmTNnynGHgqamJjZv3ozNZpMbbuvWrZw+fZqcnByeffZZAFlFwuVyybhCQUEBZ86c\nweFwMHnyZODiBZRer+emm26SPnDwCoSZM2f6+d59y/j7XsAOSRdPvZ7Y2Fj69esnXUFOp5OamhoZ\nswRvGZhQF9T9KdLT07FYLJw+fVoG/NPT04MyPqPRyBVXXCHvXrlcLhITE/16LR08eJDGxkZZsiqQ\niAv74I1DP/300xw/fpxHHnkEQF4G9r2AbTabZeLCxo0bAe8a37Fjh99l/Z49ezJhwoSAu9qEcBLr\n3m6309raisvlknHWYCuuvu1Itm/fzuHDh6Ubu7S0lNTUVIYOHSpdsr1795ZV9EWmcyjLs11SAgr+\nWxVclFgRB11qaiodO3b0e6/H4/Er+RIVFUV1dbXUlIQvPVg+X2GJfPHFF4A3hV1UdQ42YiGKBWi1\nWuWN8bi4ONLT05k1a5Z8Zpqm0dTUJC8UAnz33XfExMT4VTK+WCIiIujVqxeLFi2Sh79vuwrxmsvl\nwmg0+lUZF+8NhUA4t9aY1Wrl9ddfp7W1VR4molJ4sDjXcryQ94t1LsrkBAu9Xk9cXNyPLrf6KhTP\nPfccNpuNDh06BPzgtdvtMm7jcrlobGzkjTfekDHnu+66i6FDh5KSkiItlsOHD/PBBx9QVFRETU0N\n4BV0oi2ISJ665ZZbyMrKCnhykKZpuN1uOW6bzcbJkycxmUwy7hvstS0+e/PmzWzatImWlhapxM+Y\nMYOxY8diNBplJuaBAwdIT08nMzPTr9KKsqB+gsOHD7N+/XrAm9llsVgYOnSodMeI+06+tLW18cYb\nb7Bx40Z5CHbr1o1x48YFrSikb1q3Xq9n1qxZMn00FHeNAIqLi+XPwuo0m8107tyZffv2sW3bNvm+\nEydO0NDQIDe0xWJh9uzZ3HDDDQHdrOJ5+7Y4Ef/7pg6LXjlC4xSuwXMJxWY5ceIEe/bsQdM0xowZ\nAwRX23W73VLBMZlMF6QglJaWUl1dTbdu3WRwPljPRvzd860LUblh8+bN6HQ6MjMzA36lIiYmRip7\n+/btk032du/eDeCXuv1TPdeEpyU+Pl5mBoumhoHgfGtbnFMHDhygsbGRjIwM+vXrJ8cTLDweDwcP\nHgRg6dKlmEwmJk2aJO/wJSQkoGkaxcXFvPrqq4DX5Xj11VczdOhQ+UxC6TFQSRIKhUKhCEsuKQtK\nuF0sFosM7B04cIDY2FhKSkqk6y4/Px+j0UhiYqKMp/z73/9m7dq18r4PeE35zp07B8VN43Q6eeON\nN+TPiYmJzJ07N+SVwmNiYujTpw91dXUyiQRg7969bN26VWrpIghqNpvlHZEHH3yQQYMGhSzOImot\nivFrmkZzc7O0oEQMoT26oO7bt4+WlhYMBgPTpk0DguuLb2lpYc+ePYD33tpPFVgVFsuf/vQnDAYD\nY8aMCXqBXTE/56u/J9xstbW1REdHM3/+/IDPl8Fg4PXXXwe8bViKi4ux2+1+Fbd/7vfBawV369aN\nuXPnyirjgVjv4vkIl5pw7dntdpnc8eGHH+JwOBg+fDjp6elAcK0Tt9vNli1bAG/8fcyYMUycOFEm\nQGmaxr59+5g/fz4VFRWAt6DAddddJ++zhZpLSkCBd2H17t2bAQMGAHDs2DGqq6upqqqSG+Oll14i\nLi4Oq9Uqfc2NjY243W7MZrN00cycOTNobpqTJ0/yzTffAN5NMGXKFHnfIBSIg2PEiBHSvSCCoYWF\nhTL5Qdxq1+l0dO3alXvvvZcbbrgBwO9iaDDR6XSyuK5vXMrj8dDc3CwPW4fDIauc+26WYN7LEAfe\nsWPH0Ol0ZGRkyKoRwUa4X48dO8Ytt9zyo4PT4/FQX1/P1VdfDXjdkFlZWcyfPz/ogexzuxCLcbW2\ntrJq1SrAO4fdu3fn8ssvD8r8iOy8wsJCVq9ezTPPPCMvmPoqO+KzRbuNDh06MHLkSACuvfZa8vPz\nZb+4QCAEU1NTkyx1ZLPZqK2t5e2335bx84aGBsaNG8fChQtDUlXGt2WGyWTCarVy6tQpKYx27txJ\nQUEBp06dIjc3F4AnnniChISEdksE0v2UfzbEXPBAfJt+LV++nBdeeIG6urrzBpV9X0tKSmLq1Kky\nDTZYPVY8Hg+bNm3i7rvvBrwL9qOPPmLw4MEhn2jfRelrneh0OtnQTWA0GkMaAD13nA6HQ8a/RAV4\nkYoO3ow0s9nsZ0EJ4RuMA9l3nY0aNYrS0lKmT5/OypUrgYvPaPwpbDYbn3zyCeDNuuzRowcTJkwg\nLi4O8AqCFStW8NJLL8mLw2lpaRQVFYUkEUfUtfPNsHS5XGzYsIH58+cD3koSs2bN4rHHHgtolffz\n4XK5aG1tlQLKZrORkJCA2WyWngOPx0NUVBTx8fFBqUQinsWZM2fYtWsXO3fulB6K4uJiysrKqK6u\nlsr1mDFjuPfee4mOjg7JntM0jePHjwPe/m8HDx70a3sjrL0JEyYE+oz8xX/gkrOgwHt4iSylefPm\nMWjQIJYvXy4bgDU3NxMREUFcXJy8DT5gwAAWLFhAenp60LUVUSi2W7dugLfX0mWXXdYuB79vmu25\nB2qo7mJdKL7jcTgc1NfXU1NTI12yvinMgmC7RERG48GDBzEajVx99dUhSbONjIyUzST379/Pq6++\nyrJly+SB19bWhtPpJCYmhjvuuAPwNrYTAiyYCKXA4/HI8dTW1vLpp5/y0ksvyULOw4YNY8GCBSGx\nDkQJtPNlLwoXVrARlWyqq6tZunQphw4d8kvcio6OZvDgwbICxuDBg0O6B0XpJ/AKqHXr1mGz2eQV\ngP79+9OvXz/i4+PbpdP3+QiPUSgUCoVCcQ6XpIvvXEQbeOHqaGpqorq6mi5dukjtKdBFT38Oq9Uq\nUzp79erld2dEcX58u3aKi5iVlZUySWLkyJEhbaxot9tlrGLnzp1kZGRw+PDhoFeaFgiXbEtLC6+9\n9hrPPfec1Mjj4uIYNWoUc+fOlS69UMQLBedWLn/nnXcoKCigqqpKWkwrV65k1KhRIava394IC6qx\nsZHbb7+d8vJy2fQwOTmZm2++2a+Nxm/oPPjFX/RXIaAUvw5EXAO8MZbS0lK+/PJL+vTpA3gD2udz\n8wWL1tZWWdS3qamJyZMn884774SN+6M9EUktIubzxRdf8OSTT+LxeGTl8tmzZwetkG4443A42Lp1\nKzqdTiZujR49mtzc3N+MsD4HJaAUvz5ElphYo6FO4LDb7TLRpbCwkB07dsj2CQrFhRCqJqBhjhJQ\nCoVCoQhLfrGAUr4KhUKhUIQlSkApFAqFIixRAkqhUCgUYUk4pZT8pqOICoVCofBHWVAKhUKhCEuU\ngFIoFApFWKIElEKhUCjCEiWgFAqFQhGWKAGlUCgUirBECSiFQqFQhCVKQCkUCoUiLFECSqFQKBRh\niRJQCoVCoQhLlIBSKBQKRViiBJRCoVAowhIloBQKhUIRligBpVAoFIqwRAkohUKhUIQlSkApFAqF\nIixRAkqhUCgUYYkSUAqFQqEIS5SAUigUCkVYogSUQqFQKMISJaAUCoVCEZYoAaVQKBSKsEQJKIVC\noVCEJUpAKRQKhSIs+T8ReevT56akNAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84623f0be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_rows = 6\n",
    "n_cols = 10\n",
    "plot_multiple_images(outputs_val.reshape(-1, 28, 28), n_rows, n_cols)\n",
    "save_fig(\"generated_digits_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the latent loss is computed differently in this second variant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encode & Decode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    }
   ],
   "source": [
    "n_digits = 3\n",
    "X_test, y_test = mnist.test.next_batch(batch_size)\n",
    "codings = hidden3\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    codings_val = codings.eval(feed_dict={X: X_test})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    outputs_val = outputs.eval(feed_dict={codings: codings_val})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the reconstructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAHGCAYAAABaaN0mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGcBJREFUeJzt3WuMXVXdBvBdphfaUodykSIgGkJawFYCBMs1hgIGrVoK\nCVYtGBHFgkQUVGoIKLESICRE2xgFEzDcaaMiICQVjVwqIFERKagoFiyX0tKWtrTTy/vh5YNrrd3O\nzJ9zzsyc8/t9e07WntkF9jzs+XftPWzbtm0VAPTXTgN9AgAMTQoEgBAFAkCIAgEgRIEAEKJAAAhR\nIACEKBAAQhQIACEKBICQ4S38Xp6Z0lmGDfQJ0FSu585Sez27AwEgRIEAEKJAAAhRIACEKBAAQhQI\nACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEIUCAAhCgSAEAUCQEgrXygF0Fa2bSvfq7Vx48Yk\nL1u2rFjz9NNPJ3m33XYr1uy///5JHjduXJK7u7uLY7q6urZ/sk3gDgSAEAUCQIgCASDEDKTFNmzY\nUHy2cuXKJNf9zvT4449P8hlnnFGsOeuss5J84oknRk4R2I58vvGf//ynWHPHHXck+ZlnninWvPTS\nS0levXp1sWb8+PFJ/tjHPpbkL33pS8UxY8aMSfKwYcOKNY3kDgSAEAUCQIgCASBEgQAQMqxuI0yT\ntOwbDSbr169P8ne+851izdVXX92Q7zV8ePp3Ii699NId5iZr7vSOgdZ213Pdz8I333wzyV/5yleS\nvHDhwuKYLVu2JHnvvfcu1uyzzz47PKaqqmr58uVJnjJlSpIvv/zy4ph8zU47NeweofZ6dgcCQIgC\nASBEgQAQYiNhP2zdujXJ//jHP4o1J598cpJfffXVJO+8887FMT/72c+SXPd7y1NPPTXJdRuPvvvd\n7yb5xhtvTPJ5551XHFP3EDfoBPnMY8WKFcWaT33qU0l+6KGHklx3rR544IFJvuiii4o1U6dOTXI+\nK62qqrruuuuS3NPTk+S6uUkLZ9pVVbkDASBIgQAQokAACLEPpB8effTRJB9zzDH9/hqnn3568dnt\nt9+e5OgD0PLfic6cOTPJRx99dHHMN7/5zdD36gP7QNrbkL+e8zliPmesqqp64oknkrzHHnsk+fzz\nzy+O+dznPpfkuhc/5bOTfL5aVVX18MMPJ/kHP/hBkvM9KVVVVUcddVSS871hVRX++WIfCACNo0AA\nCFEgAIQoEABCbCTcjrpNRfkbwfoiH3TNmjUrfE696erqSvJee+2V5PwvAUCn2LRpU/HZmWeemeQl\nS5YUa/bcc88k33rrrUk+7LDDimPywXVfhtZ1GxInTJiQ5PxtpuvWrSuOqRvGN5M7EABCFAgAIQoE\ngBAzkLdt3rw5yXfddVexZtWqVUmu+91mvokofyBaM7311ltJvv/++5O8cuXKlp0LDKR8g/Qtt9xS\nrLnnnnuSPGbMmGLNtddem+TDDz88yfncsapiG/XqNnSvWbMmyc8//3ySly5dWhxzwgknvONz6Q93\nIACEKBAAQhQIACEKBIAQQ/S33XzzzUn+8pe/XKzpy0Dqsssua9g59dfPf/7zJC9fvjzJ+dN5oV3l\nf+Gl7q2A+dOr6zYK55/lQ/NGDanrhuj5G0/Hjh27w1xV9RsSm8kdCAAhCgSAEAUCQIgZyNvWrl2b\n5L78brPuQWr5Awxb6Y033khy/oa12267rZWnAy1R9wDBK664Isl1m2jHjx+f5KuuuqpYM2rUqCQ3\na+bR09NTrHnwwQeTnG92zjc1VlX9xsZmcgcCQIgCASBEgQAQokAACDFE3466jT35pqJ882FVVdXO\nO+/ctHP6X3Xnd/311yf5gAMOaMm5wEBavXp18dndd9+d5LpNd5dffnmS3/Oe9xRrGjE0r7tW889e\ne+21Ys1f/vKXJO+7775J3m+//YpjbCQEYEhQIACEKBAAQsxA3vbyyy8nOd9kVFVVdc011yS5u7u7\nqee0I1deeWXx2ZNPPpnko48+ulWnAwPmoYceKj578803k3zMMccUaz7/+c8nuS/zg3x2EZ2R5A9y\nvP3224s1+cNQDznkkCTvvvvuoe/dSO5AAAhRIACEKBAAQsxA3pbvoah7ucykSZNadTqF/GVRV199\nda/HfOYzn2nW6cCAyR8qeO211xZrhg9Pf7RdcMEFxZoxY8YkuS/zjMjMo24fyCuvvJLkm266qViT\nP+D17LPPTnKrH5xYxx0IACEKBIAQBQJAiAIBIKQjh+hLly4tPssHVgPp6aefLj6bNWtWkvM3pVVV\nVf31r39N8kEHHdTYE4NBYNWqVUmuu57zN4NOnTq1WNOsBw/mQ/P8fKuqqs4888wkv/TSS8WaT3zi\nE0k+4ogjGnB2jeUOBIAQBQJAiAIBIKQjZiBr1qxJ8rHHHlus2bBhQ5LPOOOMpp7T/3rrrbeSPHv2\n7GLNxo0bk5y/DKeqqurggw9u6HnBYPTAAw8kOb92q6qqpkyZkuRWPvg0v1YvuuiiYs0TTzyR5LoZ\nzYIFC5I8YsSIBpxdY7kDASBEgQAQokAACFEgAIR0xBA9f1PfypUrizX528lOOeWUppxLPmCrqqo6\n/fTTk/ynP/2pWLPbbrslecaMGY09MRiktm7dmuRf/OIXSV63bl1xTL7pLvrmwFy+SbDuer7kkkuS\nfO+99xZr8iF//rTtqqqq0aNHR06xpdyBABCiQAAIUSAAhHTEDOQnP/lJkuveLPi9730vyY1629eK\nFSuSPH369GLNY489luSvfe1rxZpvf/vbSR4/fnwDzg4Gv3zu8NRTTyW5br5x+OGHN+R75/OX/KGr\nl112WXHMj3/84yTvu+++xZo777wzyfnbEYcKdyAAhCgQAEIUCAAhCgSAkI4Yok+ePDnJt956a7Fm\n8eLFSf70pz8d+l7PP/98ko8++ugkv/rqq8Ux+dvT5syZU6wxNKdT5YPs3XffPcn//Oc/i2NefPHF\nJG/ZsqVYkw/f8+9TVVX1r3/9K8n5k7Lzp+pWVVW9+93vTnL+s6WqqmrChAnFZ0OROxAAQhQIACEK\nBICQjpiB7L///r2uyecifZmBLF++vPjs4osvTnI+87jwwguLY6655pokN+rBb9AO8k29Rx55ZJIf\nf/zx4pjbbrstyXUbC8eOHZvk++67r1hzxRVXJDmfreQPOa2qqrrnnnuSXLeRsF2ucXcgAIQoEABC\nFAgAIcPyB5U1Ucu+UW79+vVJzl/mUlVV1dPTk+S///3vxZq77747yWeffXaxZs2aNUnOf1/70EMP\nFccMH96Wo6j2+CUv29O6HxzZz6ilS5cm+cQTTyyOWbVqVZL32GOPYk2+N+SNN94o1mzatCnJxx13\nXJJvvPHG4ph85tEm847aP4Q7EABCFAgAIQoEgBAFAkBIRwzRczfccEPx2TnnnNOQr33ssccm+dxz\nz01y9CGNQ1BbTA7ZrgG7nvO/8PLrX/+6WHP++ecneeXKlcWafIPiIYccUqz5/ve/n+SjjjoqySNG\njNjxybYPQ3QAGkeBABCiQAAI6cgZyLJly4rPzjzzzCT/9re/LdYcf/zxSb7qqquKNflD29p0k2Bf\nmIG0t0FzPde9CGrz5s1JrpuBdHd3J3nkyJHFmnxO0sHMQABoHAUCQIgCASBEgQAQ0pFDdFrCEL29\nuZ47iyE6AI2jQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCWvm6PA/X\ng/bhesYdCAAxCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJA\nAAhRIACEKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEKGt/B7bWvh92LgDRvoE6Cp\nXM+dpfZ6dgcCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACE\nKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIGT7QJwAwVG3btq34bOvWrUnesmVLsaanpyfJmzZt\n6vXrjB07NsmjRo0qjhk2bNj2T7YJ3IEAEKJAAAhRIACEKBAAQgzR34Fnn322+Gzx4sVJPu+881p1\nOtXMmTOTPG/evCRPnDixZecCAykfbucD6aoqB9lr164t1rz++utJfuSRR5J8xx13FMf87W9/S/L6\n9euLNXXnkxs3blySp0+fnuQrr7yy12OazR0IACEKBIAQBQJAyLC6jTBN0rJvFLFgwYIk57OMqqqq\nRYsWtep0mmLp0qXFZ02ci7R2RxOtNmDXc/4zq24T3nPPPZfkBx98sFhz3333Jfmxxx7r9Xtv3Lgx\nyXWbBPMNfsOHl6PmXXbZJcl185d8djJ69OgkX3/99cUxM2bMSPJOOzXsHqH2enYHAkCIAgEgRIEA\nENKR+0Aa9cCx+fPnF59NmzYtyY2aMeR7TubOnVus6W1GUzfXsTeEoSbfQ1H33/VNN92U5CVLlhRr\nNmzYkOSRI0cWaw499NAkd3d3J/mggw4qjjnhhBOSPGHChGLNsmXLkvzVr361WPPMM8/s8Hz33HPP\n4hgPUwRgSFAgAIQoEABCFAgAIR05RK8bfueDuHwYXlVVNWfOnKadU39FNjUOpvOHqM2bNyc535RX\nVVW19957J3n27NnFmhNPPDHJU6ZMKdbkbwGs2xTYm7oHJ65ZsybJ+VC9qspNirvttluSP/jBDxbH\nGKIDMCQoEABCFAgAIR6mOAjVvagq3zjYlxlIPutp8QzEwxTb26B5mGI+E6mqcn7Q1dVVrMnnGY2a\nH+Tnt27dumLNwQcfnOS6GUg+f/nhD3+Y5LPOOqs4pokzEA9TBKBxFAgAIQoEgBAFAkCIIfogFB2E\nzZw5M8kLFy5sxOlEGaK3N9fzduRPza0bdt91111JrtugeO211yb53HPP7fWYJjJEB6BxFAgAIQoE\ngJCOfJjiYFO3cbA3+byjqgZ85gFtJ58R5zmfd1RVVU2fPj3Jv//974s1Y8aMSXLdGwnPOeecJLd4\n5tEn7kAACFEgAIQoEABCFAgAIYNvKtPm6gbmkyZN6vW4fGg+b968hp0TUA7Iq6p80u/LL7+c5LpN\ngo888kiSx40bV6y56KKLkvyNb3yjWDNixIjtn+wg4Q4EgBAFAkCIAgEgxAykxfI3C/bVtGnTkjxx\n4sRGnA6wA5s2bUryBRdckOQlS5YUx+y+++5JnjVrVrHm4osvTvJQmHfUcQcCQIgCASBEgQAQ4oVS\nTXbaaacledGiRb0e0yYPSvRCqfbWdtdz3c/C3/3ud0n+yEc+0uvXOeGEE5J82223FWu6u7v7eXYD\nzgulAGgcBQJAiAIBIESBABBiiN5g+cMS+/KgxFwL/500kyF6e2uL/0j/19q1a4vPJk+enORly5Yl\neddddy2Oufnmm5N80kknFWu6uroipziQDNEBaBwFAkCIAgEgxMMUGyzysMT58+c34UyAHclfFvXF\nL36xWLNixYokjx8/Pskf//jHi2MOPfTQJO+0U/v+f3r7/skAaCoFAkCIAgEgRIEAEGKI/g4sWLCg\n+Ky3p+3WPWl3zpw5DTsnoFS3Ofexxx5L8gMPPFCsyQfg733ve5N83HHH9XpMO+ucPykADaVAAAhR\nIACEmIG8A+edd16/j5k3b14TzgTYkRdeeKH47Atf+EKS6+Yk48aNS/L++++f5E2bNhXHbN26tdev\nO2xYezxr1B0IACEKBIAQBQJAiAIBIMQQfTvyNwtWVeztgkuXLk3yxIkTw+cElOqG1I8//niSZ82a\nVawZNWpUkru7u4s1+cbB973vfUk+8MADi2PytxS288bC9v2TAdBUCgSAEAUCQIgZyNvymUdk3lFV\n5cMSzTygsfKZx5IlS4o1s2fPTvKqVauKNWPGjEnyvvvuW6yZMGFCkk899dQkH3HEEcUx+WylnbkD\nASBEgQAQokAACDEDedvcuXP7fUzdy6EWLlzYiNMBtmPjxo1JvuWWW4o1q1ev3uExVVXOUvbbb79i\nzYwZM5J82GGHJXn06NHFMe3yoMS+cAcCQIgCASBEgQAQokAACOnIIXqjhlzTpk0rPss3JNpICO/M\nhg0bknzdddcl+c477yyOWbduXZJHjhxZrMk3C9f9pZhPfvKTSc6H5p00MK/jDgSAEAUCQIgCASCk\nI2YgCxYsaMjXyX9HWjcDMfOAvss38/X09BRr8rni4sWLk7x27drimPyBhscff3yx5oorrkhy3QNU\nR4wYkeROn3nk3IEAEKJAAAhRIACEKBAAQoblQ6wmatk3yvXlbYN9GZDPmTOnsSfW3kwb21tTruct\nW7YUn61fvz7JTz31VJK//vWvF8fsscceSc4H5lVVVZMnT05yV1dXn8+zA9Vez+5AAAhRIACEKBAA\nQjpiBsKAMANpb67nzmIGAkDjKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIaeUbCW0sg/bhesYd\nCAAxCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACE\nKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEKGt/B7bWvh92LgDRvoE6CpXM+dpfZ6\ndgcCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIa3cBwJAH23blm61GTZs8G2tcgcCQIgCASBEgQAQ\nYgYC0GJbtmxJ8htvvFGsede73pXkESNGNPWcItyBABCiQAAIUSAAhCgQAEIM0YeIvmwimjlzZpIX\nLlzYrNMBtiPfAPjcc88Va+bMmZPk1atXF2suueSSJOfX92DYWOgOBIAQBQJAiAIBIMQMZBB69tln\nQ8fNmzevwWcC9NfmzZuTXHdd/vGPf0zy+PHjizUf+tCHGntiTeAOBIAQBQJAiAIBIESBABBiiD4I\n5EPzSZMm9XrM/Pnzi88mTpzYsHMCepdvGqyqqlq+fHmSn3zyyWJNT09Pkt///vcXa/baa68kD4aN\ngzl3IACEKBAAQhQIACFmIIPA3Llz+33MtGnTmnAmQH/kmwarqqp++tOfJvm///1vsWaXXXZJ8tVX\nX12sGT588P94dgcCQIgCASBEgQAQMvh/ydZm6h6UuGjRol6Py18mY88HtN7WrVuT/O9//7tY86Mf\n/SjJ69evL9acfPLJSf7ABz5QrBmM+z5y7kAACFEgAIQoEABCFAgAIcPqHgbWJC37RoNZdDDWwn9P\njTL4J4C8E0PuP8iI/LrbuHFjkj/72c8Wx/zyl79M8tixY4s1f/jDH5J84IEHFmsG2RC99mTcgQAQ\nokAACFEgAITYSNhkCxYsGOhTAILyGcg999yT5AceeKA4Jt9seMwxxxRr8hdIDbJ5R5+5AwEgRIEA\nEKJAAAhRIACE2EjYYPnTdidNmtTvr7F06dLisyH49N2hORWkrzriel63bl2SP/rRjyb54YcfLo4Z\nM2ZMkvNNg1VV/lwYAkN0GwkBaBwFAkCIAgEgxEbCBlu8eHG/j5k/f36Sh+C8A4a8unnwwoULk/zE\nE0/0+nVOOeWUJOebBqtqSMw8+sQdCAAhCgSAEAUCQIgCASDERsJ3IN80WFWN2TjYJkP09pgSsj1t\ndz0vW7as+Gzq1KlJXr58eZLHjx9fHPOrX/0qyUceeWSxpqurK3KKA8lGQgAaR4EAEKJAAAixkfAd\niMw7Zs6cWXzWqJlHPpNpk1kKNMWWLVuSPGfOnGLNq6++muSddkr/n3v27NnFMVOmTEly3abBfPY8\nVDcWugMBIESBABCiQAAIsQ+kHxrxsqhm/vPOzy9/sGPd73ibaGj+Upe+GlLXc91195vf/CbJ06dP\nL9a89dZbSZ4wYcIOv0ZVlbPHvsw3hsAMxD4QABpHgQAQokAACFEgAITYSNgPc+fO7fcx+dsGmykf\n3tlICP/vxRdfLD7LNwHmA/Oqqqrhw9MfkTfccEOSDzjggOKYyEC8bsg/BAbr7kAAiFEgAIQoEABC\nzEC2Y8GCBcVnixYt6vW4/GGJLd68B1TlgxLnzZtXrFmxYkWvX+fDH/5wkk866aQk5zOSqhoas4tG\ncQcCQIgCASBEgQAQokAACPE03u2IDsLyjYMdPETvnEliZxrU1/MLL7yQ5COOOKJY8/rrryd57Nix\nxZpnnnkmyfvss0+SGzUw78vP4QEeznsaLwCNo0AACFEgAITYSPi20047rd/H1D0osYNnHjAgNm/e\nXHx26aWXJjmfd1RVVe20U/r/zxdeeGGxJn8DYbPmEEN186E7EABCFAgAIQoEgBAFAkBIR24kfPbZ\nZ4vPJk2a1O+v08J/dkPR0JwK0leD5j/++++/v/hsxowZSa572+B+++2X5D//+c/Fml133TXJQ3XY\n3QA2EgLQOAoEgBAFAkBIR24knDt3bui4uo2DQGv19PQk+Vvf+laxJp95dHV1FWvyjYPd3d3Fmg6e\nefSJOxAAQhQIACEKBICQjtwH0pffa86cObP4bOHChc04nXbll8ftrXU/OLKfUa+99lqSDz744OKY\nVatWJXny5MnFmkcffTTJo0ePjp5iJ7APBIDGUSAAhCgQAEIUCAAhHbmRsG5AvmjRoiTPmzevVacD\n9MPIkSOTPHXq1GLNK6+8kuR77723WDNq1KjGnlgHcgcCQIgCASBEgQAQ0pEbCWkJGwnbm+u5s9hI\nCEDjKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQ\nokAACFEgAIQMb+H38oY6aB+uZ9yBABCjQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQokAA\nCFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACEKBAAQhQIACEKBIAQBQJAyP8BHvDD\nHXc5YbwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846b0829e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(8, 2.5 * n_digits))\n",
    "for iteration in range(n_digits):\n",
    "    plt.subplot(n_digits, 2, 1 + 2 * iteration)\n",
    "    plot_image(X_test[iteration])\n",
    "    plt.subplot(n_digits, 2, 2 + 2 * iteration)\n",
    "    plot_image(outputs_val[iteration])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpolate digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFVRJREFUeJzt3XvMzvUfx/GPSoXkLMdwoxRSaVIqi4o2MYlSqo2S1R/1\nR2qsTawt65yx0ZpWWa01pxmaVGgkhKmc5SzJObpFh99/r16f7+/63u7cp+u67ufjr1fd9/W9btfn\n+/1en33fn0OVf/75JwAAAAAhhHBeRf8BAAAAyB50DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEA\nACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACAXlON7sYlz2atSCsegncoWbZT9aKPcUNJ2\noo3KHm2U/TK2EU8OAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAA\nIOW5CHap+fvvv5X/+usv5fPPP79Yr69S5d81H//5J/Mam8n/7/993nn/9qn9WJ6RG/7888+M2dv4\nggsuyPj/AQDIR3zTAQAAQOgcAgAAQHKmrOxlXS8rnz59WvmXX36JXlO9enXl2rVrK3tpsLCwUNnL\nh15iTL5PjRo1lKtWrZrx9ZSYK5afL7/99lv0syVLliivX79e2YcoeLsOGDBAuWnTpsoXXnhh6fyx\nQAU4c+aM8muvvRb9rKCgQHngwIHKDKsA0qUNUwsh7lN4H6aoYUsV2Y/gSgcAAIDQOQQAAIDkTFnZ\nH8OeOnVK2cvCf/zxR/SaevXqKfvjWn/0e/HFFyt7WdHfL/menr3EnDbDNYS4BOklS8rPpcfPhcmT\nJytPnDgx+j1vm5o1ayq3adNGuXHjxsrHjh1T9rJysoSQT22ZVh7xaySEeLjFiRMnlP0z27Ztm/KC\nBQuUFy5cmPr+derUUe7atauylzg7duyo7NdXPrVDWfJ2eeutt6Kf9e7dW/nee+9Vpqxc9ooqTRbn\nNWnnP9dF6fH74NatW5UnTZoU/d7u3buVvd/QqVMn5WHDhim3atUqen2yH1GeuNIBAAAgdA4BAAAg\nVc7lEfY5+s9vlDa7Z9euXcpHjx5V9rJiCCG0bdtWuW7duhnfw8skabOgk/994MAB5e+++07ZS2bJ\n8svNN9+s3L9/f2WfRV0Kj5BLo25QbidEafB2mTBhgvKLL76ofPLkyeg1PtzAH+n7sX7++WdlP6/G\njh2r7GXNEIq9CHtOtJGXTfzz279/f/R7XkaZM2eO8uHDh5V9uIdf08kStfPrp1q1aspdunRRnjp1\nqnKzZs2Ui7sYfhFyoo1KatmyZco9e/aMfubDZbzNK7LMlUFJ26lC28i/e33muGe/dpKfvd+v/B7l\nr7/sssuU/ToKodyGCOR0Gzn/vKdMmaI8fvx45eQqJ5dccoly/fr1lS+99FLlDh06KI8ePTp6vb+m\nDIcFZDwwTw4BAAAgdA4BAAAgWVUjSPLykJeg/LG5z4ps0qRJ9PqLLrpI2R/J+nHT/n9Rs439NT6r\n8vjx48obN26MXu/lZy+FjxgxQtlnziKdP7r3UvKoUaMy/k7yvPBH9z771c8lLxX44upHjhzJ+B4h\nlEo5s0J5mctLKH5ez549O3rNtGnTMv5e2v7lXsryaypZ4kpbUcA/fy9jDx06NOPvM7s2nbeRl79C\nCKFhw4YZfw9lw7/TtmzZouznu6+gkPTtt98qb968WdmHVg0ePDh6jX8/IjO/D40ZM0bZV8PwoWEf\nfvhh9PobbrhB2fsw/ns+POaqq66KXu/Dnsr7+4U7JwAAAITOIQAAACSry8ourQTSoEEDZZ/ZE0I8\n4y6tlFyc9wshLjP7+/ssZH9/nwkYQgjz5s1TnjVrlrIv8OvHyvUSZWnzmeSLFi1S9hKxP7b3Wa3T\np0+PjuUz+Pxz9r24vY19wVJfBDvf2sjPeS/5+jXmn1EIITRq1EjZy/ctW7ZU9s+yX79+yq1bt1ZO\nLjrvbbx8+XJlLxP7ntm+2KwfN3kdUyL9l5cpa9WqFf2sffv2ynxmZc9n97/xxhvKfk3deeed0Wt8\nGMzMmTOVv/76a+UhQ4ZkzCie+fPnK/v+4963WLJkibKvmhBC+vAabwvvG/h9N/ma8saTQwAAAAid\nQwAAAAidQwAAAEjOjDn0MX++5IvX6JNLwfj4pP86bqao3/exZv6ePjYtOfbAN7n/4osvlFeuXKnc\nuXNn5eTYrsrIx6GtW7dO+dFHH1WuWrWqsi8L9NJLLykXtUSQv4fvBOJLQPjYLF/iI9/GHDr/t/l1\n5EsrhBDC/fffn/H3/PW+tIxfx/7Z+5IdIYTw+eefK/tuNb47kS8x1K1bN+WixhziX/7ZJHcR2rdv\nn3JFjnuqLPw+5stI+Tl+8ODB6DU+5tbH5fr4RR/7y7JOxeO70vhSZz6m3ccZNm/evFjH9evNd0jx\ne1dyB6rkWOzyxNkCAAAAoXMIAAAAyeqyclpJyDcQ91Xei3ps7qURz/4e51KCSntNskzj5QHfIWXF\nihXKjz322H9+/3ySfIT+448/Kvtn45/t448/ruy7mqTtjhNC3P6+y4mXE9q1a6fsQwT83KssJUv/\ndyZ3VUjbZcFLY96uhYWFyr5MxPPPPx+9fu/evcpezvFlcbwUl7aDEdL5+e6724QQwo4dO5QpK5cN\nP0996IUvHeVLprRo0SJ6/erVq5VPnDih7ENfHnjgAWXKypklv3eefPJJ5d9//13Zh9Bcc801JXpP\nbwsfUpPsN/jOOcllbsoaZwsAAACEziEAAAAkq8vKadJmUiYfD/t/e/kwraxc1Oxmf7zrZS4vn/nm\n5z6bKYS4fOxlZZ+JWRkf+3tbJGdqDR06NOPPevXqpTxu3DhlL80Uxc8Lf39/bO/lnDp16ih7KbMy\nKqpk65/r9u3blSdOnKj8ySefKB86dEi5qNKlXxfeLk899ZSyz/SvjNfRufCZ434fC6H8S1iVkZ/z\nXta/8sorlQsKCpR91YQQ4u8OP5bvpOIlZmTmQ75CiO9Rfr/54IMPSu09vQ+xdu1aZR92EwKzlQEA\nAJAl6BwCAABAcrKsnFbaSj6C9dl4Xsr1RUI3btyovGjRImVfbDeEeMFRLzH7Y/82bdoo+yP/EOKy\ntv/9jRo1UvYyT2Xhj9HHjh0b/czLjr1791Z+++23lYuzWLg/wg8hbj8vx/jMND+ul6uZCRvza87P\n+eHDhysvW7ZMOdkWabw07GV9n0noi55X9nL/ufDz2u9PIaQPvUHp8c946dKlyh06dFC+4oorlJMz\nWX2heDdkyBBlhlhk5p/9yJEjo5/59/DHH3+sXJpDLXbv3q28Z88eZW/v5N9Z3jhzAAAAIHQOAQAA\nIDlZx/Qyh5epfCHQEEI4deqU8vfff6+8Zs0aZZ+Z5DNivfQYQvx412cw+R6JgwYNyvjeIcTlU184\n2GdZVpayspcid+3apbxhw4bo97p27ar86quvKnvJN63k5e117Nix6GebNm1S9rb0v8vbtTjvV1l5\nKdn3CPWySXFm3CXLXz7con///spPPPGEMqXkkvHzOvn5+yK/nPNlw8vE/hn73uB+H/vqq6+i169f\nv17ZF4e/9dZbMx4X//J+Q3K2sm+AcPvtt5fae/rQjVmzZin7SgE+/C2Eih0WwJNDAAAACJ1DAAAA\nCJ1DAAAASM4McvOxFz6GyZcfSS4/s3PnzozZj3X99dcr+zgEHwsYQjyGsHnz5so+HsrHEu7duzd6\nva+A71Pi+/btq+w7v+Qz/5x97EVyDJmPL6tZs+ZZj+vnxcGDB5VnzJgR/Z4vX+TLRtx4440Z/5bK\n0i7F4ctDhRDCTTfdpOznfNoSDD6Gxq8X3xUihBD69eunfO211yoXZ6cjFI/fO5PjnZs0aVLef06l\n4PcoH+PuY5x9iaHNmzcrz507NzqWj1n08dl+XeHs6tWrF/23f3f4MjM+FrS4y+l5G/kY0cWLFyv7\nfayouQ7ljSeHAAAAEDqHAAAAkKwuK/sjWi8n+fIZ/tjdlygJIYT27dsr33333crVqlVT9sfDtWrV\nUk6uhu7lsOIsn+J/V/Lv92P7kh35XBrztvzpp5+Uv/nmG2VvlxDicm5aWf6HH35QHjdunPK+ffuU\nGzRoEB3Xl+nwdl29erWylzXzuV2Kw8/rZ555JvqZX4tpZV4vk7Vs2VJ5wIAByn369ImO6zsSbd++\nXfnIkSPKyXbFf+PXVFHlMJy7ZFnQh2X4Pcq/e3bs2KG8ZMkS5QULFkTH8vvjwIEDlf3aY+hFZn7f\n79KlS/Sz5cuXKw8ePFh5zJgxyv4d4sMz3nvvvehYXpauX7++spf+vW/gS6tVNJ4cAgAAQOgcAgAA\nQLKurJw2G9FLfvPmzVPesmWL8uWXXx4d64477sj4M38c74/aS/rY3Wc0z549O/qZH/vqq69W9pXt\n8423n5cJJ0+erOxlwuSsYC8T+8w+P64PJfCZXrVr11Zu2LBhdNymTZsq+w4t/rfcc889//fvqay8\n7b788svoZ35ee/v57Euf0Tx8+HBl333AZ7CHEO+w4sfy67gidw/IVWmzH5OzlX1YDOXI0uPnue+M\n4WV937HDd/ZKrhTgs2zTysrIzD8jX3EkhBCWLl2q7CXmN998U9mHrHm7+BCYEELo1KmTsq+MsmzZ\nMuU2bdoo9+7dO3p9Rc485+4KAAAAoXMIAAAAybqysvOS38svv6zsM1QLCgqUhw0bFr2+VatWyl42\nKc3H7l4mmDhxorKXxUIIoU6dOsqTJk1SzrcFlr1s5Y/bFy5cqLx161blxo0bKyc/C5+dd+zYMWUv\nM/om6R07dlROK3eGEMJnn32m7CVTL9McOnRI2duusvB29KEeXhoJIYTCwkJlHyLx4IMPKvtMZL8m\nfaHxZFnTZ2969tdQPisZXx3A2ziE+Nplxmvp8XuRL6rsZWUvJR4+fFg5OYzi6aefVvZNAhhucXZ+\nHjdr1iz62bPPPqs8f/585Z49eyp72/mKDf69EUI8pMnbxTdi8A0AfFWV5GvKG2cRAAAAhM4hAAAA\nJOvKymnlLC/f+v/3fRCTC1enzUouzb/RZ06PHz9eOfk4eMiQIcq+V2y+lWn8szlx4oSyt0X37t2V\nvWS4bt266Fhe8vXPyWd3+cxnL2t6SdoX3Q4hXnjb/0YvAeRiu/h1kSylF+ff423nC7suWrQo9Tg9\nevRQvu2225R9uIAvgu1/l88uT5Y1fSanlz8pmZUeL1kmeTtV5P6uuS55vXjJ2Gfee1nZz/ENGzak\nHsv3ns+34UnlKTmkxWcY+3eNf1d5W/gwp+R9zK8dn6F89OhRZe+3JPd5pqwMAACArEDnEAAAAJJ1\nZWUvH3tpymeM+iN0n1W3atWq6FidO3dW9sfAaY9q0/akTP5dvt/lww8/rOxlMt97MYR4BlSy/J3r\n/LPyz8AXlfbZ2z5j1R/V+2P3EEKoXr16xvfwx/5pJUffG9mHHoQQlz99lp+XEHwfzFzh14KXckNI\nLzt5Octn2o0ePVrZy/3eviHEs/bWrFmj7NeizyLv27fvWd87hHh/2bTrGCXjpa3kPdHPf28nypcl\n45+zZ7+/+XfNgQMHlJMlR1/oH+cueU9J+3729krbO76ovbT9+23FihXKvrdzco/zirzf8eQQAAAA\nQucQAAAAQucQAAAAknVjDr3e71PMfdySj0/yqf5TpkyJjrV+/XrlW265RdnHSTVp0kS5Ro0ayr/+\n+mt0rKlTpyp/+umnyj4+xJeoeeedd6LXN23aNOQrH2exadMm5XfffVfZdzjxZVJ86v/KlSuj4/o4\nU9+Z45FHHlH29ktbMsV/J4T/Hw+a6d+Ri/zfnxy74uNl/DP38/yVV15RnjVrlrKf4z7GM8mXCPLr\n2Nu+bt26yr6Ux8mTJ6NjdejQQdnPA5ayKZm0nYOSYwmTy3ug/KxevVrZz/fk7hmM/yx7vpya9w/8\ns08bixhC/F3n9zi/j/rYUd8BqqJxpwUAAIDQOQQAAIBkXe0gbaq/bzJ+1113KU+YMEF55syZ0bGm\nT5+uPGPGDGV/dOtlLn/U66WwEOIp6e6+++5Tfv3115W9FBZCfi/Bcfz4ceX3339f+aOPPlL2x+vO\ny1fJ5WMGDRqkPGLECOXmzZsrF/VIv7IpquTqZWZvi82bNysvXbpU2XeO8bJyYWFhdFxve/89L7v4\n8A7fZN6Xqmrbtm103AYNGih7ubyyt3FpatasmXLyc/VlqFD2/Pr0HYl816devXpFr+FaKBtpy97t\n2bNH2UvMXhZOlvr9Xuv3VO93XHfddcrJ5YpYygYAAABZgc4hAAAAJOvKyml8w3LfIWPSpEnKDz30\nUPSaadOmKfvuDV5W9uN6KdkfG4cQP/p94YUXlL1MVlnLXz4zdd++fWf9fd+ovHv37srPPfdc9Hs+\nY9XbjBmrZ5f8jHwmdtrnlzYDz2c3JzeW91Kyt5GXw3ymfrdu3ZR9dn9yRxc/VmW6lsqTf+bJYTM7\nd+7M+LNsmk2ZT06fPq3sn71/DyVXvPBrmmuk9Ph90HfQmjNnjrLfBxs1aqSc3LXGZ557/8KHUPk9\nMZtmoPMtCwAAAKFzCAAAAMmZsrLzR+g+w7hnz57R7/Xo0UPZy19nzpzJeKy0mdLJ//bX8Dg/5gu1\n+iN2n5XsG40PHDhQOVlaxLlLln/9/PVhAC1atFDu06eP8uHDh5UPHjyYelwvg/js15EjRyr7OeGz\n+IsqoXBdlT2fMZlcXcHLmSyIXfb8fG/fvr2yt4MvGo/y4fdKHwI1d+5c5VWrVmX8/RDifke7du2U\n/b7r35PZdN/jySEAAACEziEAAACkSjnuJ5vbG9fmhtJ4Jl2idko7n7LpcXkFK5c2SrZDWrt4mdhn\npS5evFh57dq1yj4LOYR45rnvX+5lk7SZx1l8TlT4dVTevPwVQrwocxbPUC5pO2VNG/m155s3LFy4\nUHnUqFHRawoKCpSzaZZrQt60kc8o96E2R48eVd62bVv0Gl/U2l/funVrZZ+FXkErcWRsI54cAgAA\nQOgcAgAAQCgr55dKVw7LQbRR9qONckPelCy9jL9//37l5cuXK/vqGyGEUKtWLeU8HqKRNW2Uxygr\nAwAAoGh0DgEAACB0DgEAACCMOcwvjJXKfrRR9qONckPej2fz7+csHldYlLxvozzAmEMAAAAUjc4h\nAAAAhB3VAQDIQjlaSkYe4MkhAAAAhM4hAAAApDxnKwMAACDL8eQQAAAAQucQAAAAQucQAAAAQucQ\nAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAA\nQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQ\nAAAA8j+v65u2SfOmMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8462250c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFQtJREFUeJzt3VeMVdUex/GFiI0qVaVJlR4ICkgVLAlBCKigBgViIUFD\nYnzQRH3hQUkEYgyYkCgKimIEQSkBJIKUhCICUqR3RQEFRIpguw83/PitfWePw52ZM2fmfD9Pv3Bm\nnxn3Onuf5f6vUu6ff/4JAAAAQAghXFXSfwAAAACyB51DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1D\nAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAAyNUZ/F1s4lz8yhXBe9BOxYs2yn60UelQ\n2HaijYofbZT98mwjnhwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwC\nAABAMrkIdpH555/L62KWK1cU69X+7/t6DiGEv//+W/mvv/5SLl++fJ45qSj/ThSdZDvnhbYDAOQS\nnhwCAABA6BwCAABASmVZOROl5AsXLkQ/d+rUKeXz588rV6hQQfmGG25Qvv7666Pjr7nmGuWrry6V\npz3rFbQt9+7dq/zVV18pHz9+XLlfv37KrVq1UvY2vuoq/t+qINLaxXPyXFLKL35+/o8cORK95uf/\npptuUuYzD8R8mNnp06ej13bu3Km8bds2Zf8eueeee5Rr1KgRHV+S1xtXOgAAAITOIQAAAIT6pvHH\nw8lS5I8//qh86NAh5cqVKyt7mcbfK4QQqlWrpty6dWvlihUrKlNKKxg/z3/++aeyl4snTpyovHjx\n4uh4Hxbgx/ts8x07dii//vrryj5cwGewh5CbJTdvi4sXLyrv3r1becqUKcp79uxR9jL+kCFDovd9\n5plnlK+77rqi+WMR+eWXX5Qff/zx6LW6desqT5o0Sblq1arF/4flOL+m/B6TXFmhIPcb/07h+6Vw\n/PyfPXtWec2aNcpz5syJjvF7nN8T/frya+ruu++OjvfhaJluv9z7NgMAAEAqOocAAACQcgVZBLiI\nXPEvKq7Frv1RvZcVz5w5o+zlrxBCOHjwoPL+/fuVjx07puwzk/zfQ4hnKA8aNEj56aefVi6Ckk1R\nnKSMfSCuhH8WvJ0WLFig/PbbbyuvXbtW+ffff4/ey2eYe7nfPxde7u/SpYvyuHHjlOvUqRO9bwFn\noZfqNkreL3z4xdatW5XfeOMN5ZUrVyp7KdOvPZ+9F0IIEyZMUB42bJhyhmb6l+o2Kqhdu3Yp33nn\nndFrPvTC72v169cv/j+s4ArbTlnTRj4kw79rFi1apFylSpXomEaNGin7MCZvu+bNmys3bNgwOt7v\ng8WoVLeR3+98aJkPNfIVL6699tro+Ntuu03Z75U//fSTsrfrmDFjouPbt2+vXIzDlvJsI54cAgAA\nQOgcAgAAQLJ6tnJxLXadnGV6iT8S9sfxIYRQu3ZtZZ9ptH37duXffvtN+ddff42O9xLO+++/r9yp\nUyflHj16KDOzLG4zL5X4jLB33nlHecOGDcrexr6IbwghdO3aVdnb2cs5+/bty/Nv8pJ2sqyci/74\n4w9lL+X7DGNfRPzbb79V9mskWfqfNm2a8gMPPKDswwBw5fyaOnfunHJy8V2/lyVLZSgaaSXLl19+\nWdmHYTz66KPR8X4tzJ49W/nzzz9XrlevnvL06dOj45l5/u/8OvAZ/X4f6969u/L48eOj42+55RZl\nv8f5/c2PefHFF6PjZ82apZzp9uLJIQAAAITOIQAAACSry8rFJW1h0PwWnPQZRf6ouUWLFspePvMF\nL0MI4b333lNO29vXS8xeyqHEHJdX/JG+zyqvWbOmcocOHZSfffbZ6L18BpjPmPXSjO8168MIbrzx\nRuVcbJfkf7MvCj506FDl/v37K2/atEnZz/Hy5cuVfe/yEOKSpw8p8NJKLp7/wvJSps/Ib9q0afRz\nfr0lF/RH0fAhGa+88orysmXLlEeMGKH88MMPR8f7bGOfRe7fPSdPnlTO0Ez/Us9nFXspeenSpco+\nBOyDDz5Q9msqhPge5UNtHnvsMeWPPvpIecuWLdHx69evV04ukF3ceHIIAAAAoXMIAAAAoXMIAAAA\nKdODENJ2f/FxAOXLl8/z5/NbjdyP8XGGPmbRV0YPIV4CZfLkycqHDh1S9rEOLB8Rj3X6+eeflf08\n+diPbt26Kfft21c5v10dfvjhB2UfZ+Vj6Vq2bKlcqVIlZf8c5Co/B5UrV1b28U0+ZtPPn//86dOn\nU9/Xd49A4fi9z8es+fi3EOJx1X7vQtHxXTJWrFihXKtWLWUfL53cRcgdPnxY2ZdM8e8R/37CZcl+\ngi/JNW/ePGUfTzh37lxlv6flNw7aX/Ox0/5dNXHixOiYJUuWKPfp06dAv6eo8OQQAAAAQucQAAAA\nUqbKysnHwwXZFcX5sibJMouXOKtXr67sj/q9FJ187OulTV8Wx8uaXi7NRck28uVM/DVfssY3k+/X\nr5+yr0yf5LsR+G4CZ8+eVe7cubNyr169lCn3x/xz7p9/v5bmz5+vvGrVKuUDBw4oJ5dL8eN9OY4G\nDRoU7g+GeLnYh7eEEF8LLGVTdPw7yZdA8WEVAwYMUPbdnZJDnbxdVq5cmefvu+OOO5QZBpO35G5m\no0aNyvPnpkyZolzYJbV8SEfPnj2Vp06dGv2c79rlnx3KygAAAMgoOocAAACQUl9Wzq907KVhL1Em\nS8aXeCklOUPPVzf3R/3+eNd/f7IUk7arh5eb/Xfkyu4P3n7JsrrP5vOSSJ06dZRvvvlmZS83+/v6\nTOcQQli9erWyt1nXrl2Va9eureyzbfObxY7LNm/erPzxxx8r79u3T9lLxz47PIQQmjRpouzn32W6\nzFLWeGkrWVqrUaOGcrVq1TL2N5V1/pn33bH83v/EE08o5zfD2Gfxf/fdd8p+rxw9erQy18hlft+f\nPn169NrOnTuVO3bsqDxo0CDlojyXvuuWX3chxNeof3YyMfOcbzoAAAAInUMAAABImSore1k4hLh8\n6LOCfYFef1TrJa9kKcVnrHrJ0Y/3902WuH2GrJdPvWSWizNh/TydOHEies0XpT516pRymzZtlL2s\nnFYmmzFjRvS+XvL3mWK++Ky3sZdpKM2k8/P64YcfKvswCr9evETvQwJCCOGll15S9tn9jra4cmmL\nYPui1yHEs/3zW3wZV+b8+fPKfl14adGHVPg1klyNY/v27cp+r/QStW/SwPVymS8UPnbs2Og1vy7e\neuutPP+9sLwt/b7pC20nf2emVw3gySEAAACEziEAAACkVJaV/ZGsl6k2bdoU/dzGjRuV/RG+l599\n8VcvC3fo0CF6Lz/GS5ZeDvDjk2WaLVu25Pn3t27dWrkoH1uXRsk9dH3WmM9m9VKy/7u30Zw5c5R9\nf8oQQmjevLmyz/ryWdD+viweWzB+LX3yySfKfo16actLyc8991z0Xr43uV8XlMaKjn+uk59xH1aR\n6/elouSlYL/ftWvXTtnLwmnfdSGEMGHChDxf84Wv094rhNy+lnwljOS+7o0bN1Zu1KjRv75X8rym\nvebZV0xZt26d8vHjx6Pj27ZtW6DfUxx4cggAAAChcwgAAAChcwgAAAAp9WMOfUp6sl5/5MgR5bSN\n5H0DbR8HkFxW5ujRo8q+Q4qPU/NxOz6mIYQQ1q9fn+fvHzx4sHIu7r7hS9n4cj8hhLB7925lX/Ln\n+++/V16zZo2ybz7vYzx9WY4Q0pcHSBuDk8tjc/6Nj5t64YUXlJNjbi/xMVCPPPKIsu8+EALLB2VC\ncucg5/cizv//L7mkmS+p5ufVxz77d5rfn3y8YgghLFy4UNnb66mnnlLObwexXG7j/D77/t1/8uRJ\n5cqVKyv7ed2/f7/yrl27ovfy89qiRQtlX7LG2zu5e5sfz5hDAAAAlBg6hwAAAJCsLiunPUb1UtbB\ngweVfReNEELo0qWLsj9C98frXnr2JU6SJV5fad7LlF4O8PedO3dudLyvjJ+2Gn6u8HY9c+aM8qxZ\ns6Kf87Ly4sWLlX3ZBl+iyD8XvtyNL4sSQrxMjZcEfImirl27KrOUx2XJa3Lz5s3KX375pbJfC34t\neVs8+OCDyj48IwTKysXF2+XAgQPKyd2l0naywZVJXi81atRQ9p1nDh8+rOy7A3kpc9q0adF7+f3K\ny5TNmjVT9nuif1eFEF9XuXCNeVv4jjLJIWT+neznxb+rduzYoTxz5kzlZFm5UqVKysOHD1e+9dZb\nlX3ImZexQ4iXc/Ol8jKBqx4AAABC5xAAAACSdWXltBXFfUaPlwK9rFylSpXovbyE5Sv++4roXj5J\ne+QfQjzL0h/v+t944cIF5c8++yw63stkAwcOVE5utF1W+Xk6d+6c8uzZs5UXLFgQHeOzkv3cepnX\nZ3d52cRXuW/atGn0vj4swMvV/vnxYQTeRrlQfslPcpeGTz/9VNnb2EuRfl69XN++fXvlZOk+189z\nUfJ28dmvXlpL8pJlpstZZUnyc1yvXr08s8+eXbt2bZ7/nlwBw9u1V69eyj6kJr8hAbl8jfnqF8nV\nLDp27KjsQ8B8OFL9+vWVe/furextGkIIt99+u7LvguPXlw9tSw5DaNiwoXKmh3fw5BAAAABC5xAA\nAACSdfUCf6zqM60WLVqkvHr1auVatWop9+jRI3ovLyV7acRLxF7y8sfG+ZW50srdPpvMHxWHED9u\nHjNmTJ7vW9akldyXL1+uPHHiRGWfsRdCPMPbeZvdf//9yr7hvJcNfMZfCPGsWi+T+u9Lzt7Ef3k5\nJIQQli5dquxt7DMAfWZez549lf0aS5ZMyvJ1kS38s59cINmH2GR68d2yJPk59g0Uhg4dquzlY28L\n/97zYTYhhFC9enXlcePGKfvMf//9uX6N+X+vf4c0aNAg+jlvCz9n3ofwc9ynTx/l7t27R+/lfQo/\n3ldW8Xb14WchxCXuTLcXTw4BAAAgdA4BAAAgWVdW9lKy72fsi0r7Y1/fjze5kK6XRnwxSn90mzYL\nOfkI1x/1e8lx/Pjxyu+++65ycsHRCRMmKPvsqLL8aN/bct26dcqvvvqqsi/Em5wJ67w0PHLkSOXR\no0crexv7zL7kfpVff/21srdlp06dlNNKM7nIP/tLliyJXvOhAGkLX3v2/UXzO69eysz1819YaaXF\n5NAX57M08f9Lfna9zDhs2DBlv3b8vul7Mft9K3l8o0aNlJOlybS/JZd5WTk5G99XQPGVTfw7KO2a\n8vZN/pyvFOD3Td+H3tsxhHijgLR2LS48OQQAAIDQOQQAAIBkXVn5xIkTyv5418uSPmPSZyt7DiGE\nli1bKvtj5LSFp322pc9CDiHeS3HSpEnK8+bNU/ZHyEOGDImO79u3r3KmHw8Xt7RFdv3RuZ8zL5V4\nKTi58Lg/om/Tpo2yL6Tsx/j594VgfXhCCPEMZ5/x5+V+H26Q6+UYb9Mvvvgiei05e/kSL7V4W9Ss\nWTPPn0nOiM31c15c/LweO3Ys9ef8ukTRSRtukbb5w8aNG5X9nhRCPNvZhzFx7fw7P5fJ87p3715l\nH8Lm3w/+HZ7f4tTelr5v8pQpU/J8L+8nhBAPjWO2MgAAAEoMnUMAAAAInUMAAABI1o05dD6N3Ov1\nPjbQl0Lx5W5CiJcz8Q2wfbcSHxPgP79hw4bovbZu3ars4xB894dBgwYpP//889HxySnupVlyfJgv\nQbNt2zblsWPHKvv59J1IqlWrpuzjQkMIoXPnzsq+s0CTJk2UfSeO5JIEl/jm5SHEywWkLUnAuJ3L\n/HrbtGlT6s/558LHQPlY4LSxOsnzzfkvHn5e08brhhCPDaUtip+fYx9378sNeZuEEELjxo3zPB7/\nzu9P9evXj17zJYPmz5+v3KxZM2Ufp5jfmEOfuzBz5kxl3+XNf//w4cOj49O+0zKBJ4cAAAAQOocA\nAACQrCsr165dW7lbt27KY8aMUfbHs/v371f2zaxDiEucvnyKl8n88bCXKD2HEELbtm2VfZeHAQMG\nKPtj5woVKkTHl6XH/r6afwjx1H8vJS9cuFDZz7mfW9+JxNs7hBB69Oih3Lp1a2Uv5RdkCYeydO5L\nQn7l3+TuM5f48jfOPwcoWX7tJe9XZWkYTGngQzJ86IYPw/AluEL43124UHB+XkeMGBG9tmfPHuW1\na9cqz5gxQ7lLly7KVatWVU5+N/p34OTJk5V9SMdrr72mnNwhpSS/u3hyCAAAAKFzCAAAAMm6srKX\nsPxxrc/ieeihh5R9lX/fxSSEeENrn4ncvHlzZS8Feyks+cje/xbfVcNnE+U3a6ksST469/P2zTff\n5Pnvzh+p33XXXcpPPvlk9HM+y9jL/7l4zkuSlxy7d+8evebX3MWLF5V9BrsP/fD3YnZ45vl59tUB\nkisQ7Nu3T9mvY6634uHn2GeyOl9lI4T0XVUc11Xe/Lz4TmohhPDmm28q+85eU6dOVV68eLGyD5VJ\nDsfwfodfbwMHDlT2e2o27Z7GlQ4AAAChcwgAAADJ6rKyS5tJ7CVen0UcQvqjdpf22D15LI/nL0vO\nUD169Khyhw4dlL00X7duXWUvJY8aNUrZF8QOgdnH2cJLHSNHjoxemzdvnrKXULztfXF4v15px5Ll\nGwP4sJkQ4mu8IPdRFI4Pw/ANHypVqqTcqlWr6BjKykUj2efwRal9lZQVK1Yo+6YOPrQtuVC5DwVo\n166dsm/k4EOmsglPDgEAACB0DgEAACBZV1YuSoV5pM7j+HQ+2ziEEHr37q3sC1d7qcNnGPu55Txn\nPy+7eDkkhBCGDRum7DPVfS/swYMHK/tsZdq+ZLVv31552rRp0WvefiW5v2uu8HtlxYoVlX0IVXI1\njo4dOyqnrQKAK+f3JZ99fN999ynfe++9ysnVO9LeqyA5m/ApAgAAgNA5BAAAgJTL4Ew0prwVv6J4\nPk07FS/aKPvRRqVDYdspa9rIZyuvWrVKedmyZcr9+/ePjvH95tP2mM+CEnOZaaMyLM82KvFPDgAA\nALIHnUMAAAAInUMAAAAIYw7LFsZKZT/aKPvRRqUD49myH22U/RhzCAAAgPzROQQAAIDQOQQAAIDQ\nOQQAAIDQOQQAAIBkcrYyAAAAshxPDgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0\nDgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEA\nACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACD/Af6RlFUcprl6AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8465411198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFJhJREFUeJzt3VeMVVUbxvHXAlJEaVKkKE2QYqElCIqGeIElhmBiMCCi\nGENQgUSDN8aKCWpClBgTDZgQwdhQKWJBYwAdIahUpRqKjAhIEUXs35WPz9rf7HGAmeHMnP/v6nE4\n58y419l7r+x3lVP+/vvvAAAAACIiTj3ZfwAAAAAKB51DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1D\nAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAAyOnV+LvYxLnqnVIJn0E7VS3aqPDRRjXD\nibYTbVT1aKPCV2Yb8eQQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQ\nAAAAUp2LYNcaf//977qcf/31V5k///XXX5P3+Ovq1KmjfMYZZyifckplrL0LAACqm/cBsv+dd38v\n1Ps+Tw4BAAAgdA4BAAAglJVzZB8Pe1n4t99+U16/fr3yzJkzlXfu3Jm8v2vXrsq33367cqdOnZTr\n1q17An8xsm2W9/M///xT+Y8//lD2x/uU+0+MH+OVK1cq+5CK7t27K9erV696/jCUya9vWf7951wA\n0uvbL7/8orx8+fLkdWvXrlVu3Lix8hVXXKHctm1bZb8+Rpzc840nhwAAABA6hwAAABDKyjmypcgf\nf/xRec6cOcqvv/668o4dO5S9XBkRsX37duVmzZop33333crZR8r/oJSTtocf2x9++EH53XffVZ4/\nf77yN998k3yWt6WXj8855xzlBx54QNlLAHltFEE7eWly7ty5yhMnTlT2Esozzzyj3L9//+Sziv1Y\nVhU/j37++WflRYsWJa87fPiw8pVXXql8/vnnK596Ks8WqoKfRz/99JPypk2bktc1b968zPf4Ncrv\nNfXr10/ezzl2bLyU7O2yevVq5c8++yx5z759+5S//PJLZe8rDB8+XNmHmUWc3OFNnN0AAAAQOocA\nAAAQysrGHxsfOHAg+bc333xTecaMGcpbtmxR9tnGLVq0SN7vj6G95HbWWWcpjx07tszPKlZePl63\nbp3yE088oeyP8ffu3at85MgR5fJmYp522mnKPsP8hRdeUG7fvr2yP/anrJbykuWaNWuU/VzyNlq8\neLFyv379ks+i5FU1vI12796tPHv27OR1Puty0qRJZWa+/5Xn999/V164cKHyhAkTlP2aFhFx3nnn\nKXfo0EHZ7z2XX3658rBhw5L3e8kS//JzxFcm+e6775T9vu/lfm/HiLQt/N/8+uhDOrzEHBFx5513\nKp955pnK1XF95OwGAACA0DkEAACAFH1Z2UuOvpjl5s2bk9ctWLBA2Wcg+aP9AQMGKHfu3Dl5/8aN\nG5VXrFihvHTpUuWRI0cq+4yzYimxZcu/Xkq+5ZZblL3864/9nS84mi2f5B1b3w/b29h/X5s2bZSZ\n/Zfy9vPy+7nnnqvss8u9fONDOiIoWVYVH6px8OBBZT9fIiIaNmyo7LP7/f3lzdzHf/Pvv5cWvZTo\nJk+enPz3qFGjlP0a5aXokpISZZ91HhHRqlWrY/uDa6nsyiQ+U9/vQX7fbtmypbKvtNCxY8fks/wc\n8SE1r776qrLPYp4+fXryfl/l5KmnnlKujiEBXIEBAAAgdA4BAAAgRV9W9nKWLwo7b9685HV79uxR\n9sVHb7jhBmXfM9nLMhERn3zyibI/qvbZsv47GjRooFwsJbZsWdlLxj5juEmTJso+q3vIkCHKffv2\nVW7dunXyuf6eJUuWKL/xxhtl/i0+xMCHDiDl32VvC599uWHDBmVfoDdbVqZkWXm8bOarJvh3vFu3\nbsl7fJFevy75sABf0BzHzhfnHz16tLIPT/noo4+UL7roouT9fl9o2rSpst+f/L7jw6aKnZ8TPpwo\nIh1C5qth9OrVS3nw4MHKvnlC9l7tbemlYB8m5TOffVONiIh33nlH+f7771f2oTpVNZypOHodAAAA\nqBA6hwAAABA6hwAAAJCiHHPoY218vMG2bduUfdp5RLrhfJcuXZR9xwAfJ5gdQ9W9e3fl3r17K/uq\n6T4m4fTTi69psuM1fBzU448/ruzt50sK+G4z/lnZtigtLVVetWqVsi8H4b8jb/xnsS9dk+XHoyLj\n3Pw1HMvqUdHvr58zW7duVWbc2onxpYCGDh2q7MsFPf3008oXX3yxcnnt5eN9fScN/3l5O0UVGz8W\nPvYzIh3n6WM5vb38vlPROQHeFj7eety4cWX+7oh0d6m33nqrzPcw5hAAAABVjs4hAAAApGhql/4Y\n2ctcXkr2Vep9enpExM0336zcr18/5eySNXkaNWqkPGjQIGUvZVb0s2qr7ONxPx5elvfS1tGjR5V9\n6Rt/HP/2228nn/vcc88p79q1S/nss89W7tmzp7KXdopx55rjsX//fmVfCujIkSPKviyKl1xQufx7\n6sfZl+7yHYEi0jKn76ThS2jg2C1btkzZzwu/9owdO1a5otcYH6Lh1ygfNuUl7WLkx8jvG7NmzUpe\n50PK/F6dN2wpbzhNlr/Hh435Lmt+n4tIlyLatGmTst/r6tWrl/s7TwRPDgEAACB0DgEAACC1uqzs\nM+52796t7KuOe2nF+Q4PEenOGBV5jJstB+TtuNGhQwdlnxVLyTL/GBw6dEjZV5T3nWf8cbzv9hCR\nlhR8hri38fjx45X9UX+x7FZzPLy9vv/+e2UvH/vx9pmAHNeq46UuL2f5sA0/pyLStvGdObz98N+y\nKyU8+uijZb5uxIgRysdzjP3+snr16gr9LcXMd/35+uuvk3/zvkLjxo2V/Rrl51R5peQ8eUM9OnXq\nlLyupKREecWKFco+7MO/L5XZb+CKDAAAAKFzCAAAAKlVZeXsIp++uOX06dOVDx8+rOwLUo8aNUq5\nXbt2yWflzVKt6CNlnw3oZc6uXbsqF+PC1+Xx9jx48KDylClTlF977TXlvFnM2e9F/fr1lX3muS9o\n3r9/f2VvF8r9FeNlZT/f/FheeOGF1fo3FSv//nsJq0mTJsp+fmVfx7CK4+ez8yMitmzZouwzlB98\n8EHl47nG+KobGzZsKPN3ZFfgKLZF6P3/169P69evT17n9we/j+Qt5u+y54efR3kznP2a6Odklg/9\n8L+rqnCmAwAAQOgcAgAAQGpVHTP7CH/evHnKXmJu0aKFspcVfYHXbIk3r5Sct1ds9rHvxx9/rOyz\noXzP5mIvK2dL9F6C8ZL/mjVrlH0x0LySV/a4tm3bVnnkyJHKl156aZnvKYaSS2Xw9vMZgD5L0mfB\n9ujRQ5ljXLnyhrv4OeL7umfLyj7D3PeSp52OjZciI9Lj5wvt+wLLFeXtN3XqVGVf7NoXWC72vZX9\nnPDNJ/weEhHRrVs35dLSUmVfhNpnO/vqI37eRKRDZwYOHKjcpk2bMn9/9jz0Rcx9VrLvn83eygAA\nAKhydA4BAAAgdA4BAAAgNX6Qm4+j8CUzItIp/XljPXx6v4/VyNbxfbyC76qSN07QxxhGRMyYMaPM\n9/v4hmIfz5MdMzpmzBjlvHGGzsdT+fgMHzMVETF48GBl3/2h2MfknCgfA7V06dIyX+M7ADRq1Ei5\n2L/7la0iS2zt2rVLeevWrcm/eXv4OFHa6dhkx5770iY+Ps3Hs/n9yY+3jz+LiJg/f77yBx98oOzL\nrvXp00f5eHbyqE38+u47jPguKBHpDmj+3ff7k88b8HtTtj/g9yFvV59r4PetAwcOJO/3NvdxqX7t\nrCo8OQQAAIDQOQQAAIDUyLKyPx73XTF8SnlEuvtI+/btlX16/9q1a5V9evhLL72UfJavon7JJZco\n+/R0XyLn2WefTd6/bt065UGDBin7auzFyJc5eeyxx5J/82PuJf+8Vef9Eb4f4yFDhiSf66UaL+c4\nb2P/XKT8XPTlHXwZIuclfm9HnDhvCz+v/PvupX8vh3377bfJZ/m1s1mzZpX6dxaT7I4XrVu3VvZ7\nwm233abspeC6desqL1iwIPksL416+dF3RfF7UvY6Vmw7pPh334ecZfmwo2uuuUbZ26JXr17KQ4cO\nVc4OI/D7u7e9l5+97VavXp283+97vtRadVw7eXIIAAAAoXMIAAAAqTFlZX8E7o+H9+3bp7x3797k\nPT4T2Mss7733nrLPqvRZsL6ReUREw4YNlX2Gq3+ul9K++uqr5P3+SNp35fCfFwtvSy/r+ozuiP/f\nXeAf3hY+FMBXo3/ooYfKfH1Euul6SUmJspc8i31mX0X5DECfaeflFP+O+3Et9h2Bqkveig4+bMOv\nYxFpaTI7mxMVl51VOn78eOV77rlH2Xfs8OuTr2yRXY3DS4veXn4d7N27t3Kx32t8hrFnX7EkIj1m\nflz9ePt9pzw+7ClvlzUvcWf7HT5zesSIEWV+VlXhySEAAACEziEAAACkxtR1vJS8fft2ZZ9xV95M\noc8//1x53rx5yl6K9tlEt956a/JZPiPJH/V6KdkXIs3OQu7Ro4fy9ddfr1wMs8SyfIa5l1myC4Dm\nzabzMtdNN92kfOONNyr7IqPZWXr++7t3767s5QVm0h47by+f4ernGCXKquPniH9/fcajz4wsLS1V\nzg6j6Nevn7KXxnBsstcRv/b7fcyvSV6yfP/995V9GEBERPPmzZW9lNymTRvlBg0aKGfb0du8ts5c\n9iEV3lfw4UQtW7ZM3uMzvH3oix+XvGNU3rHzY+xD2B5++GHl7NABL3H7cDbKygAAAKhWdA4BAAAg\nBVdW9kev/kh4//79yl988YWyLyRa3mPzDz/8UNkX6/VHyBMmTFAePnx48llemvF9k33hbV9I1hes\njIiYPHmysi+MWpse4Wf58feSv8/SW7lyZe77vSTjx8wXtb7uuuuUfeHe8vZJ9s/10gxl5RPje8Ue\nPHhQ2WfCepnG24jjXbnySsw+e9ZnbGbLyr4IPGXlyuMrJ0ycOFHZzwXP7dq1U164cGHyWb4Qs9/T\nvKzss22z95qKlElrEx8C5teq7HA0X1w87xpV0eOVV7r3xa4//fRT5ewQqEceeUTZh7NVB856AAAA\nCJ1DAAAASMGVlZ0vgjx37lzl5cuXK/uCoR07dkze72VCn73qP+/Zs6ey77ns5eqIdKHKI0eOKPuM\nP/+s++67L3m/L8hdE0to/ni9vPKEPzr34zR16lRlb0v/3LPOOiv5XG+ntm3bKvtMcH/UnldK9jaK\nSGdFd+7cWbm8mX34bz7Tzs9d/374LGYWGq86fsz9u1ynTh3l7OoA7oILLqiaP6zIebt4CTFvJmv7\n9u2Vr7rqquSzvI18qJUP3fD3e9tn/5bayr/7V199tfK0adOUfaZ4RDor3O/pft/Om92dvab5PWnX\nrl3K48aNK/P3+6ooEek+z9XdXtwBAQAAIHQOAQAAIHQOAQAAIAU35tDHWyxevFh5+vTpyr5kjI/b\n8PGHEekYQh//5uPMnI818HEbEenYjU6dOimPGTNG2Xd/yE47z9uAu6bwsRS+rE9EOhbDd8N48cUX\nlefMmaPsuzT4+EFftiEiok+fPsre5r6BvB9nXzLF/6amTZsmn+vfC/+smjgW9GTz77Xv+ODnsX/f\nffcHxnVWj4pcb7LXO98tqiZer2qCihxXH6Pbt2/f5N/8/PEx2T62sLzzrdjatUuXLsq+28iSJUuS\n1z3//PPKvjya76aVHb/5D9/JLSJdsubee+9V9mV1fD6Ej82P+P+d1qoTV2cAAAAInUMAAABIwZWV\nvTToS9b4sgu+TIaXi72UFZEujeLT/n2j7Q4dOij7rgD+mD4iLV/mlSJr82N6/3/LPjr3JUxmzZql\n/MorryhnS/7/8OVjfOmaiHQZAF/135cF8p/77g/ltUteObM2t19V8eEG3n55ywr5kAKOd/Xz9vLz\nwMuPESe3nFWM/Hzxc8R/nr0++hIo/rpWrVop+3CBYjzf/P/Z7+FTpkxRvvbaa5P3bN26Vdl3TfNd\nz3xomvdN/L0REUuXLlU+dOiQsg9Tmz17trKXvrN/f3XjySEAAACEziEAAACk4MrKzh/3bty4UXnH\njh3KvpF5dlX/O+64Q9kfA3v5Me+xe7FvUp5V3v+/l/ZXrVql7OVmL/N6Wd5neGdnFfssrt69eyv3\n799f2TeWL/YSysngx7lJkybKXuby8tfatWuV82aXo3r4Sg/Z4++lshYtWihzXlUNXwFiz549yps3\nb1b2ITQR6TmWt9sK/uXDKHwI2aJFi5LXzZw5U7mkpER52bJlyitXrlTet2+fsl/TItJ2GT16tPKk\nSZOUfWhbIa3gUDh/CQAAAE46OocAAACQgisr+4yiAQMGKL/88svKXtrwxSjLW+STcsiJ8eOXnRVe\nWlqq7DO6fEFsn0HnZf2BAwcqDxs2LPlcXwA2b4Fx2vXk8uN/2WWXKXsp0stczZs3L/O9qB4+pMMX\nAvZhOxERR48eVfb2o82qhh/jnTt3Km/btk3ZS5QR6ZAaP698OE8hlSkLSV6JOSJi2rRpyj4kxtvC\nS/qefZhbRDrD3P+tJgyB4psDAAAAoXMIAAAAOaUaZzYxharqVcbz6f9sp+zeynmP2P2RvD/G97K0\nL77rJa/sewr10ftxqJY2Ohn8WrJhwwZlH17Qq1cvZR8qUGDtW2vbyGdTPvnkk8q+d3lExF133aXs\nq0AUWJnyRNupYNrIF77etGmTsi+q7K+JSBdM9s0E/Dz012T3z64mtaaNarEy26igznQAAACcXHQO\nAQAAIJSVa5daWw6rRWijwlcUbZS3h29EjVkRoFaWLP2e7O1S3sYMeffxAlixo1a2US1DWRkAAADl\no3MIAAAAoXMIAAAAYcxh7VIUY6VqONqo8NFGNQPj2QofbVT4GHMIAACA8tE5BAAAgNA5BAAAgNA5\nBAAAgNA5BAAAgFTnbGUAAAAUOJ4cAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgc\nAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAA\nQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQP4HbHadSQTP3jkAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84623ac278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFV1JREFUeJzt3XvszvX/x/GXTkJylmP4RClEaVI6LCraxCRKqTZKVn/o\nj9RYm1hb1jljozWtslprTjM0qdDIKUzlLGdJzhHR4ffHb99Hj9d71/vTJ5/r83F9rut+++uBz/u6\nPt7v63q/X3s9X4dKf//9dwAAAABCCOG8c/0LAAAAIHfQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQ\nOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIBcUI7vxSbOZa9SFl6D61S2uEa5j2tU\nMZT2OnGNyh7XKPdlvEb0HAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBo\nHAIAAEDKcxHsgvLHH3+k/vm88/5pk19wwQUZ/x4AAGTHX3/9pfznn38qn3/++SU6vlKlf9aK/vvv\nzGtzJ//e/+zPd38tz7mE1ggAAACExiEAAACEsvJZ8K7iX3/9VXnRokXK69ati47xbuwLL7xQuW/f\nvsqNGzdWvuiii7LzywLnwJkzZ5Rfe+015aKiIuV+/fopM6QCKF5aKTOEeNiSl0+LG7aUq+XMbPJz\n5ufl9OnTyj///HN0TNWqVZVr1qyp7Ofv5MmTyn6Ok8PJ/H2qVaum7G0APz6Xrgl3ZAAAAAiNQwAA\nAAhl5RLybuSJEycqjx8/Xtm7h6tXrx4d37JlS+WGDRsqHz16VNnLyt4dnktdzdmQVh7x0nsIcZf8\n8ePHlf2cbd26VXnevHnK8+fPT33/WrVqKXfu3FnZy5zt2rVT9hJ/vl2LsuLX5a233lLu0aOH8n33\n3adMWbl8FFeaLMkxaZ9/vhfZ4/fBLVu2KE+YMCH6uV27dimfOnVKuX379sqDBw9WbtGiRXS8P6/y\nlZeS/Rz58/z333+PjqlTp46y35f8e3DxxRcr+/Xy90u+p2cvMaetXhJC/OzxUnR5fN+4IwMAAEBo\nHAIAAEAqnU2Z4SyV2xtlg5c0Qwhh3Lhxyi+++KLyiRMnlL072rvzk6/3008/KXv39ujRo5W9rFnS\nRTpDCNnoay7z6+Td8H7+9u3bF/2cl1FmzZqlfOjQIWUvCfhMsWSJ2nk3fpUqVZQ7deqkPHnyZOUm\nTZoo/4drkaZCXKPSWrJkiXK3bt2UvZzi1zvHSlwV/hr5fd1njnv2707y/Pv9yu9Rfvxll12m7N+j\nEMptmEBpr1POfI/8fE+aNEl57NixysmZsJdccoly3bp1lS+99FLltm3bKo8cOTI63o8pwzJluV+j\ntJnbO3fuVD5y5Iiyf75DCKFVq1bKtWvXzvge/vlOmwWd/PP+/fuVv/32W2UfDpX83tx8883Kffr0\nUfZZ1Fm4d2a8RvQcAgAAQGgcAgAAQHKqlnOueXe0l5FDCGHEiBEZf65Ro0bK3m3vM19DiGfYeqnA\nF+A8fPhwxvfIQinznPMyl3e1Hzt2THnmzJnRMVOmTMn4c2l7XHqXvM/ySnbVp8068/PvZexBgwZl\n/Hlm2Kbza+Tlr/r162f8GZQdLwVv3rxZ2T/vvoJC0rJly5Q3bdqk7OW3AQMGRMdUrlz57H7ZAuL3\noVGjRin7ahhePvzwww+j42+44QZlH0bjP+fDY66++uroeB/6lA/PmP/x/4ufF/8e+PPYn+EhxJ9d\nv0f566b9fXGzjf0YXzHDn20bNmyIjvfys5fChw4dqpxcGSVbeLoBAABAaBwCAABACr6s7DONFixY\noJyc2eXd0z6rderUqco+ey/ZTe/7NXqZzRcs9UWw86mbP4S4S9272uvVq6fs5yiEEBo0aKDsXf/N\nmzdX9nPZu3dv5SuuuEI5uTCpX+elS5cqe5nY98z2xWb9dZNlUcqk//AyZY0aNZTbtGmjzPkqHz67\n/4033lD279Rdd90VHeNlt+nTpyt//fXXygMHDsyYUTJz585V9v3HfUb/okWLlH3VhBDSh9f4tZgz\nZ46y33eTx+SrtOEt/tzxWdshxOc/rZRckvcLIS4z+/v7LGR/f1/lIYT4+s2YMUPZN2/w18pmu4Ge\nQwAAAAiNQwAAAAiNQwAAAEhBjjn0MWhr165Vfuyxx5R9k+sQ4qnjL730knLaNPLkODffCcSXgPCx\nWb7MR76NOXT+f/NxfsldZR544IGMP+fH+9IyPr7Dz78v2RFCCJ9//rmy71bjK9j7EkNdunRRLm7M\nIf7h58Y/+3v37lUuhDFPucDvZb6MlH/GDxw4EB3jY259XK6PX/SxvyzrVDK+K40vd+Zj2n2cYdOm\nTUv0uv598x1S/N6V3IEq+YzKR/5M8Ge1j79MPsP9s/xf7/HF/bw/t/w9fd5Bclzp1q1blb/44gvl\nFStWKHfs2FE5OW6/NPhGAwAAQGgcAgAAQAqmrOxd6D/88IPy448/ruzlryeeeCI63nc1SVtB3ctk\nyU3SvZzQunVrZe9G9s3rC6Vk6f/P5K4KabsseGnMr6tvoO7LRDz//PPR8Xv27FH2co4vNeCluLRV\n7pHOP+++A8D27duVKSuXHf+c+tALX8LDl+xo1qxZdPyqVauUjx8/ruxDXx588EFlysqZJUu3Tz31\nlPJvv/2m7ENorr322lK9p18LH1Ljz7cQ4h1DksvcVGRp92h/vvqzpbjPrt+jPPt7nM0zIe2Y5DXy\noR++Q8ry5cuVvQ2TTXyjAQAAIDQOAQAAIHldVvZuYJ+pNWjQoIx/3717d+UxY8ZEr+WlmTReQkiW\nzLzb3ss5vgF3coZ0oSmue97P7bZt25THjx+v/MknnygfPHhQubjypZcU/Lo8/fTTyj4bjPJZyfgs\nQR8GkE/lq1zmn3kv61911VXKRUVFyr5qQgjxLH5/Ld9JxUvMyMzLgiHE9yi/33zwwQdZe08fKrNm\nzRplH3YTQmHMVnZpq2Qkz4P/2YeHpZWVi5vd7KV7vy5+T1y2bJmyz1QPIS4fe1nZv59l9UziSQcA\nAAChcQgAAADJ67Kyd6OPHj1a2UuOPXr0UH777beVS7qYpHcVexdyspTpM9P8tb1czUzYmHfvezf6\nkCFDlH2jcr8WxfFueC/r+0xCX/S80Mv9Z8M/1yUpzSC7/DwvXrxYuW3btspXXnmlcnKWpC8U7wYO\nHKjMEIvM/NwPHz48+jcfbvHxxx8rZ3O4xa5du5R3796t7Nc7+XsWgrT7TbKs7CsteCnXF4DfsGGD\n8oIFC5R9I4UQ4sXkvX3gQzpatmyp7M+5EOJ7p//+DRo0UPbPVDbx7QYAAIDQOAQAAIDkVVk52T28\nc+dO5fXr1yt37txZ+dVXX1X2cm9xJS/vjj969Kjyxo0blX0mWvJ3870vS/qehci72H2PUC+blGTG\nXbL85V3yffr0UX7yySeVKSWXjn+u/fz7Ar983suOl4n9PPve4H4f++qrr6Lj161bp+yLw996660Z\nXxf/8OEtydnKvgHCHXfckbX39PLjjBkzlH1WrJdIQyi8YQH+efVr5Iu8hxDCqVOnlL/77jvl1atX\nK/usc1/xxEvHIcTfMW8TeBugf//+Gd87hHhonC/c7StoUFYGAABAmaNxCAAAAKFxCAAAAMmrMYfJ\npUx87IWPIfOxZdWrVy/Ra/vYtgMHDihPmzZN2ae3+5IRIYRw4403ZvxdfNX2QudLCIQQwk033aS8\nZ88e5bQlGHwMjY/P8F0hQgihd+/eyh06dFBmyZXs8aWbfExMo0aNzsWvUxD8HuXjoHx8ky8xtGnT\nJuXZs2dHr+VjFn2Mtn+v8O/q1KkT/dmfHb7MjI8FLemSK36NfIzowoULlf0+Vtx4uHzl/0c/f35/\nSi4/s2PHjozZX+v6669X9naHjwUMIR5D2LRpU2Uf6+7fKX/OhRDvbuTLHfXq1Uu5rNoQ9BwCAABA\naBwCAABAKnxZ2buKf/zxx+jfvvnmG+UqVaooezdsWrft999/H73WmDFjlPfu3atcr149ZV+mI7lM\nwKpVq5S9rFnoJUvvqn/mmWeif/OlbNLKvF4ma968uXLfvn2Ve/bsGb2ur1q/bds25cOHDyv7dcV/\n598r/44md+LA2UuWBX1Yht+jatSoobx9+3blRYsWKc+bNy96Lb9H9uvXT9m/ewy9yMzv/Z06dYr+\nbenSpcoDBgxQHjVqlLI/R7z8+d5770Wv5WXpunXrKnuZ0ofKJJdXy1d+v/H/vz9PfEiFL0EXQght\n2rRRvueee5S9DeGfd/9+JXe68c9C2nfEv0f+eyV/f39tX46trL579BwCAABAaBwCAABAKmRZ2bth\nvUQ4ceLE6Oe8TOhlEi8R+6w+f91kV7PP9KpZs6Zy/fr1lRs3bqzsu7Mkf5d777034P/59fvyyy+j\nf/Pucr9+PvvSZzQPGTJE2XcfSM5i9x1W/LUuv/xy5ULbPSAb0mY/+mxlL5tQiswu/5z7zhheZvMd\nO3z3h+RKAT7LNq2sjMz8HPms1BBCWLx4sbKXmN98801lL2v6dfEhMCGE0L59e2WfPbtkyRLlli1b\nKvfo0SM6Pp9mnqetNOHDuebMmaO8efNmZb/vhxDCnXfemfHf/Bnk17i03wmf0Txz5szo3/y1r7nm\nGmXftais8AQEAACA0DgEAACAVJiysncVe1f7/Pnzlbds2RId07BhQ2XvEvaZeUePHlX2EqNvkB5C\nCO3atVNOK3d+9tlnyl4uDSEu0xw8eFC5Vq1aodD4tfRygJdGQog3Hfdu9IceekjZZyK3aNFC2Rca\nT25M7rPLPPsxlM9Kx2f2+TX27y6zXbPL70W+qLKXlb2UeOjQIeXkMIphw4Yp+0YBDLf4d/5ZbtKk\nSfRvzz77rPLcuXOVu3XrpuzXzmfY+nMjhHhIk18X34zBNwDwmbfJY/KJD+l6+eWXlX0FkqKiIuXB\ngwdHx/tzxJ8d2bxH+RCQ8ePHK/uQpxDi9sGECROUy2PzjPz8dAAAAOCs0DgEAACAVMiy8vHjx5W9\ne/X222+PjvGS4dq1a5W95OtdxT6zKznz2cuaXpb2hbd90W3/HUOISwAVsYTmpcFkl3ZJ/j9+/Xxh\n1wULFqS+TteuXZVvu+02ZR8u4Itg++/ls8v9dw8hnsnp5c98LbOcC16ydH6NCmFv17KU/L54ydhn\nWXpZ2T/j69evT30t33+e/d/PXnJIi88w9ueNP6v8WvhQp+R9zL8/PkP5yJEjyr5wcnKf53y636UN\nVfLyrf+973GdXLg6bVZyNn9Hnzk9duxY5eQ1GThwoHKHDh3K5PdKkz+fDgAAAJQajUMAAABITpeV\nvRvWy4S+oLTP7vFZRiHEXfXe7V61atWM7+Fd/l5uDCHu7vW9kb172kufPsMvhLiE4PtgVhQ+y9RL\nuSGkl528nOUz7UaOHKns5X6/xiHEs/ZWr16tvHLlSmWfRd6rV69/fe8Q4v1lO3bsqFwRy/25yktb\n/t3xz75fI0qXpefn2bPf47zMtn//fuVkydEX+sfZS95TkiXM/0nbg9fvtcXtpe3Pt+XLlyv73s7+\nfcv0u1Vk/rn2YUc+29fvMX7u/HkSQvozIa0Mn7bfePL38r3MH3nkEWV/7vm+2iHEs9vTPjtlhZ5D\nAAAACI1DAAAACI1DAAAASIUZc7hx40bld999V9l3OPElUkKIp66vWLFC2cch+K4cjz76qHKjRo2i\n10pbMsV/LjlewFX0ZTv8/58cu+JjMfyc//LLL8qvvPKK8owZM5R9TEZynKfzZYJ87IVf/9q1ayv7\nUh4nTpyIXqtt27bK/lnIp6UdzoW0nYM8J5f2QPlatWqVsn/ek7tnMAa07PmSatWqVVP2c582FjGE\n+Hnn9zi/j/rYUd8BKt/4M8HvMT4m3cee+zJOkyZNil5r3bp1yrfccouyj4H3575fO3/mhRDC5MmT\nlT/99FNlf+75EjXvvPNOdHzjxo3DucLTEAAAAELjEAAAAJLTNZ5jx44pv//++8offfSRcrKU7Lx7\n2ZfQ6N+/v/LQoUOVmzZtqpwsMebTtP+zUVzJ1cvMfj02bdqkvHjxYmXfPca710+ePBm9rl9//zkv\nu3gJwDeZ9+UMWrVqFb1uvXr1lL1cXujXOJuaNGmi7OfVl6FC+fDvp+9I5Ls+de/ePTqG70LZSFsa\nZffu3cpepvSycLLU7/dav6f68JrrrrtOOblcUT5d47RlnIYNG6Z89913K48bN055+vTp0WtNnTpV\nedq0acpelvdz7GV8H+YUQrxkjrv//vuVX3/9dWUf5hTCub1G9BwCAABAaBwCAABAcrqs7DOQ9u7d\n+68/75uUhxDC7bffrvzcc88p+2xV7ypmtmrJJM+Tz8ROO4dpM/B8dnNyY3kvJft18nKYz+bq0qWL\nss8AS+7o4q+VT6WVXOLn3EsrO3bsyPj3+TyT8lw7ffq0sp9/L18mZ0X6d5rvSPb4fdB30Zo1a5ay\n3wcbNGignNy1xmeeeznTh1D5PbEQZ6BXrlxZ2XdAmzBhgvLDDz8cHTNlyhRl35nL71H+un7u/TsV\nQlzWf+GFF5R9CFSuDm2iNQQAAAChcQgAAADJ6bKy80VavXvdZyT7JuMhhNCvXz/lZGkRZy9Z/vUy\nsQ8FaNasmXLPnj2VDx06pHzgwIHU1/UyiM9+HT58uLJ/LtI2WU/Kpa77fOWz+fy6eNmFBbHLh3/e\n27Rpo+zXwheNR/nwe6UPgZo9e7byypUrM/58CPGwm9atWyv7fdeflYV+3/P/v88w7tatW/RzXbt2\nVfZzfObMmYyvlTZTOvlnP6YiXAt6DgEAACA0DgEAACCVynHP31K9UdrvWRG6Z8tRNk7Gv16n5LVI\nuzZeJvaZqQsXLlRes2aNss9CDiGefe57XHrZJG3mcQ5/LsrlGuUSL834gsw5PEM5r66Rf/d8gd/5\n8+crjxgxIjqmqKhIOYdnuZb2OuXMNfIZ5T7U5siRI8pbt26NjvFFrf143wPYZ6Gfo9U48uYa5bGM\n14ieQwAAAAiNQwAAAEiFKSujRPKqHJanuEa5L6+ukZfy9+3bp7x06VJln6EZQgg1atRQzuMhGjlz\njfIY1yj3UVYGAABA8WgcAgAAQGgcAgAAQBhzmF/yaqxUnuIa5b6CuEZ+78/hcYXFYTxb7uMa5T7G\nHAIAAKB4NA4BAAAg7HwPAAWogpaSAZQDeg4BAAAgNA4BAAAg5TlbGQAAADmOnkMAAAAIjUMAAAAI\njUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMA\nAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAI\njUMAAAAIjUMAAADI/wEKMpu2hAo8pQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846557df60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_iterations = 3\n",
    "n_digits = 6\n",
    "codings_rnd = np.random.normal(size=[n_digits, n_hidden3])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    target_codings = np.roll(codings_rnd, -1, axis=0)\n",
    "    for iteration in range(n_iterations + 1):\n",
    "        codings_interpolate = codings_rnd + (target_codings - codings_rnd) * iteration / n_iterations\n",
    "        outputs_val = outputs.eval(feed_dict={codings: codings_interpolate})\n",
    "        plt.figure(figsize=(11, 1.5*n_iterations))\n",
    "        for digit_index in range(n_digits):\n",
    "            plt.subplot(1, n_digits, digit_index + 1)\n",
    "            plot_image(outputs_val[digit_index])\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Coming soon..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "nav_menu": {
   "height": "381px",
   "width": "453px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}