Automatic Differentiation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic differentiation\n",
    "\n",
    "__Automatic differentiation__ is a method for evaluating the rate of change in the numerical output of a program with respect to the rate of change in its input. The power of the method is the ability of writing a program that computes a differentiable function and having the derivative immediatelly available.\n",
    "\n",
    "We start with an example.\n",
    "\n",
    "Consider the function\n",
    "$$\n",
    "f(x) = \\cos(x)\\sin(x)\n",
    "$$\n",
    "\n",
    "When we want to evaluate the function numerically at a specific $x$, say $x=1$ we can implement a computer program like\n",
    "~~~\n",
    "x = 1\n",
    "f = cos(x)*sin(x)\n",
    "~~~\n",
    "\n",
    "or \n",
    "\n",
    "~~~\n",
    "def g(x):\n",
    "    return cos(x)*sin(x)\n",
    "    \n",
    "x = 1\n",
    "f = g(x)\n",
    "~~~\n",
    "\n",
    "Now suppose we need the derivative as well, that is how much $f$ changes when we slightly change $x$. For this example, it is a simple exercise to calculate the derivative symbolically as \n",
    "$$\n",
    "f'(x) = \\cos(x)\\cos(x) -\\sin(x)\\sin(x) = \\cos(x)^2 - \\sin(x)^2\n",
    "$$\n",
    "and code this explicitely as\n",
    "~~~\n",
    "x = 1\n",
    "df_dx =  cos(x)*cos(x) - sin(x)*sin(x)\n",
    "~~~\n",
    "\n",
    "But could we have calculated the derivative without coding it up explicitely, that is without symbolically evaluating it a priori by hand? For example, can we code just\n",
    "~~~\n",
    "x = my.Variable(1)\n",
    "f = my.cos(x)*my.sin(x)\n",
    "df_dx = f.derivative()\n",
    "~~~\n",
    "or \n",
    "\n",
    "~~~\n",
    "def g(x):\n",
    "    return my.cos(x)*my.sin(x)\n",
    "\n",
    "x = my.Variable(1)\n",
    "f = g(x)\n",
    "df_dx = f.derivative()\n",
    "~~~\n",
    "\n",
    "to get what we want, perhaps by overloading the appropriate variables, functions and operators? The answer turns out to be yes and it is a quite fascinating subject called __automatic differentiation__. Interestingly, this algorithm, known also as __backpropagation__, is in the core of todays artificial intelligence systems, programs that learn how to program themselves from input and output examples. See https://www.youtube.com/watch?v=aircAruvnKk for an introduction to a particular type of model, known as a __neural network__.\n",
    "\n",
    "\n",
    "To symbolically evaluate the derivative, we use the chain rule. The chain rule dictates that when\n",
    "$$\n",
    "f(x) = g(h(x))\n",
    "$$\n",
    "the derivative is given as\n",
    "$$\n",
    "f'(x) = g'(h(x)) h'(x)\n",
    "$$\n",
    "\n",
    "We could implement this program as \n",
    "~~~\n",
    "x = 1\n",
    "h = H(x)\n",
    "g = G(h)\n",
    "f = g\n",
    "~~~\n",
    "where we have used capital letters for the functions -- beware that the function and its output is always denoted with the same letter in mathematical notation. To highlight the underlying mechanism of automatic differentiation, we will always assign the output of a function to a variable so we will only think of the rate of change of a variable with respect to another variable, rather than 'derivatives of functions'. To be entirely formal we write \n",
    "~~~\n",
    "x = 1\n",
    "h = H(x)\n",
    "g = G(h)\n",
    "f = identity(g)\n",
    "~~~\n",
    "and denote the identity function as $(\\cdot)$. This program can be represented also by the following directed computation graph:\n",
    "\n",
    "<img src=\"../latex_figures/chainrule.png\" width=\"400\">\n",
    "\n",
    "The derivative is denoted by\n",
    "$$\n",
    "f'(x)  =  \\frac{df}{dx}\n",
    "$$\n",
    "As we will later use multiple variables, we will already introduce the partial derivative notation, that is equivalent to the derivative for scalar functions.\n",
    "$$\n",
    "f'(x)  =  \\frac{\\partial f}{\\partial x}\n",
    "$$\n",
    "\n",
    "The chain rule, using the partial derivative notation can be stated as\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial f}{\\partial x} & = &  \\frac{\\partial h}{\\partial x} \\frac{\\partial g}{\\partial h} \\frac{\\partial f}{\\partial g} \\\\\n",
    "& = & h'(x) g'(h(x)) \\cdot 1\n",
    "\\end{eqnarray}\n",
    "\n",
    "This quantity is actually just a product of numbers, so we could have evaluated this derivative in the following order\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial f}{\\partial x} & = & \\frac{\\partial h}{\\partial x} \\left(\\frac{\\partial g}{\\partial h} \\left(\\frac{\\partial f}{\\partial g} \\frac{\\partial f}{\\partial f} \\right) \\right) \\\\\n",
    "& =& \\frac{\\partial h}{\\partial x} \\left(\\frac{\\partial g}{\\partial h} \\frac{\\partial f}{\\partial g} \\right) \\\\\n",
    "& = &\\frac{\\partial h}{\\partial x} \\frac{\\partial f}{\\partial h}  \\\\\n",
    "& = &\\frac{\\partial f}{\\partial x} \n",
    "\\end{eqnarray}\n",
    "where we have included $\\partial f/\\partial f = 1$ as the boundry case.\n",
    "\n",
    "<img src=\"../latex_figures/chainrule_with_df.png\" width=\"470\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we could imagine calculating the derivative using the following program\n",
    "\n",
    "~~~\n",
    "df_df = 1\n",
    "df_dg = 1 * df_df\n",
    "df_dh = dG(h) * df_dg  \n",
    "df_dx = dH(x) * df_dh  \n",
    "~~~\n",
    "\n",
    "If $g$ and $h$ are elementary functions, their derivatives are known in closed form and can be calculated from their input(s) only.\n",
    "\n",
    "As an example, consider\n",
    "$$\n",
    "f(x) = \\sin(\\cos(x))\n",
    "$$\n",
    "\n",
    "The derivative is\n",
    "$$\n",
    "\\frac{\\partial f}{\\partial x}  =  -\\sin(x) \\cos(\\cos(x)) \n",
    "$$\n",
    "\n",
    "~~~\n",
    "df_df =  1\n",
    "df_dg =  1 * df_df\n",
    "df_dh =  cos(h) * df_dg  \n",
    "df_dx = -sin(x) * df_dh  \n",
    "~~~\n",
    "\n",
    "As $h=\\cos(x)$, it can be easily verified that the derivative is calculated correctly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions of two or more variables\n",
    "When we have functions of two or more variables the notion of a derivative changes slightly. For example, when \n",
    "$$\n",
    "g(x_1, x_2)\n",
    "$$\n",
    "we define the partial derivatives\n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial g}{\\partial x_1} & , &  \\frac{\\partial g}{\\partial x_2}\n",
    "\\end{eqnarray}\n",
    "\n",
    "The collection of partial derivatives can be organized as a vector. This object is known as the __gradient__ and is denoted as\n",
    "\\begin{eqnarray}\n",
    "\\nabla g(x) \\equiv \\left(\\begin{array}{c} \\frac{\\partial g}{\\partial x_1} \\\\ \\frac{\\partial g}{\\partial x_2} \\end{array} \\right) \n",
    "\\end{eqnarray}\n",
    "\n",
    "When taking the partial derivative, we assume that all the variables are constant, apart from the one that we are taking the derivative with respect to.\n",
    "\n",
    "For example,\n",
    "$$\n",
    "g(x_1, x_2) = \\cos(x_1)e^{3 x_2}\n",
    "$$\n",
    "When taking the (partial) derivative with respect to $x_1$, we assume that the second factor is a constant\n",
    "$$\n",
    "\\frac{\\partial g}{\\partial x_1} = -\\sin(x_1) e^{3 x_2}\n",
    "$$\n",
    "Similarly, when taking the partial derivative with respect to $x_2$, we assume that the first factor is a constant\n",
    "$$\n",
    "\\frac{\\partial g}{\\partial x_2} = 3 \\cos(x_1) e^{3 x_2}\n",
    "$$\n",
    "\n",
    "\n",
    "The chain rule for multiple variables is in a way similar to the chain rule for single variable functions but with a caveat: the derivatives over all paths between the two variables need to be added.\n",
    "\n",
    "Another example is\n",
    "$$\n",
    "f(x) = g(h_1(x), h_2(x)) \n",
    "$$\n",
    "Here, the partial derivative is \n",
    "$$\n",
    "\\frac{\\partial g}{\\partial x} = \\frac{\\partial g}{\\partial h_1} \\frac{\\partial h_1}{\\partial x} +   \\frac{\\partial g}{\\partial h_2} \\frac{\\partial h_2}{\\partial x}\n",
    "$$\n",
    "The chain rule has a simple form\n",
    "$$\n",
    "\\frac{\\partial f}{\\partial x}  = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial x} \n",
    "$$\n",
    "\n",
    "\n",
    "To see a concrete example of a function of form $f(x) = g(h_1(x), h_2(x)) $, consider \n",
    "$$\n",
    "f(x) = \\sin(x)\\cos(x)\n",
    "$$\n",
    "\n",
    "We define \n",
    "\\begin{align}\n",
    "h_1(x) & = c = \\cos(x) \\\\\n",
    "h_2(x) & = s = \\sin(x) \\\\\n",
    "g(c,s) & = g = c \\times s \\\\\n",
    "f & = g(c,s)\n",
    "\\end{align}\n",
    "\n",
    "that is equivalent to the following program, written deliberately as a sequence of scalar function evaluations and binary operators only\n",
    "~~~\n",
    "x = 1\n",
    "c = cos(x)\n",
    "s = sin(x)\n",
    "g = c * s\n",
    "f = g\n",
    "~~~\n",
    "\n",
    "This program can be represented by the following directed computation graph:\n",
    "<img src=\"cos_sin.png\" width=\"600\">\n",
    "\n",
    "The function can be evaluated by traversing the variable nodes of the directed graph from the inputs to the outputs in the topological order. At each variable node, we merely evaluate the incoming function. Topological order guarantees that the inputs for the function are already calculated. \n",
    "\n",
    "It is not obvious, but the derivatives can also be calculated easily. By the chain rule, we have \n",
    "\\begin{eqnarray}\n",
    "\\frac{\\partial f}{\\partial x} &=& \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c} \\frac{\\partial c}{\\partial x} +  \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial s} \\frac{\\partial s}{\\partial x} \\\\\n",
    "&=& 1 \\cdot s \\cdot (-\\sin(x)) +  1 \\cdot c \\cdot \\cos(x) \\\\\n",
    "&=& -\\sin(x) \\cdot \\sin(x) +  1 \\cdot \\cos(x) \\cdot \\cos(x) \\\\\n",
    "\\end{eqnarray}\n",
    "\n",
    "The derivative could have been calculated numerically by the following program\n",
    "\n",
    "~~~\n",
    "df_dx = 0, df_ds = 0, df_dc = 0, df_dg = 0 \n",
    "df_df = 1\n",
    "\n",
    "df_dg +=  df_df           // df/dg = 1\n",
    "df_dc +=  s * df_dg       // dg/dc = s\n",
    "df_ds +=  c * df_dg       // dg/ds = c\n",
    "df_dx +=  cos(x) * df_ds  // ds/dx = cos(x)\n",
    "df_dx += -sin(x) * df_dc  // dc/dx = -sin(x)\n",
    "~~~\n",
    "\n",
    "Note that the total derivative consists of sums of several terms. Each term is the product of the derivatives along the path leading from $f$ to $x$. In the above example, there are only two paths: \n",
    "\n",
    "- $f,g,c,x$\n",
    "- $f,g,s,x$\n",
    "\n",
    "$$\n",
    "\\frac{\\partial f}{\\partial x}  = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c} \\frac{\\partial c}{\\partial x} +  \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial s} \\frac{\\partial s}{\\partial x}\n",
    "$$\n",
    "\n",
    "It is not obvious in this simple example but the fact that we are propagating backwards makes us save computation by storing the intermediate variables.\n",
    "\n",
    "This program can be represented by the following directed computation graph:\n",
    "<img src=\"cos_sin_with_df.png\" width=\"700\">\n",
    "\n",
    "Note that during the backward pass, if we traverse variable nodes in the reverse topological order, we only need the derivatives already computed in previous steps and values of variables that are connected to the function node that are computed during the forward pass. As an example, consider\n",
    "\n",
    "$$\n",
    "\\frac{\\partial f}{\\partial c} = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c}\n",
    "$$\n",
    "The first term is already available during the backward pass. The second term needs to be programmed by calculating the partial derivative of $g(s,c) = sc$ with respect to $c$. It has a simple form, namely $s$. More importantly, the numerical value is also immediately available, as it is calculated during the forward pass. For each function type, this calculation will be different but is nevertheless straightforward for all basic functions, including the binary arithmetic operators $+,-,\\times$ and $\\div$.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Tutorial introductions to Automatic differentiation\n",
    "\n",
    "Richard D. Neidinger,\n",
    "Introduction to Automatic Differentiation and MATLAB Object-Oriented Programming,\n",
    "SIAM REVIEW, 2010 Society for Industrial and Applied Mathematics,\n",
    "Vol. 52, No. 3, pp. 545–563\n",
    "\n",
    "Baydin, Atılım Güneş, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. 2018. “Automatic Differentiation in Machine Learning: a Survey.” Journal of Machine Learning Research (JMLR) \n",
    "https://arxiv.org/pdf/1502.05767.pdf\n",
    "\n",
    "\n",
    "Two related blog posts from Ben Recht:\n",
    "\n",
    "http://www.argmin.net/2016/05/18/mates-of-costate/\n",
    "\n",
    "http://www.argmin.net/2016/05/31/mechanics-of-lagrangians/\n",
    "\n",
    "\n",
    "Back-propagation, an introduction, by Sanjeev Arora and Tengyu Ma \n",
    "http://www.offconvex.org/2016/12/20/backprop/\n",
    "\n",
    "A nice autodifferentiation package for python\n",
    "https://github.com/HIPS/autograd\n",
    "\n",
    "A good tutorial on Backpropagation by Roger Grosse\n",
    "\n",
    "http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/slides/lec6.pdf\n",
    "\n",
    "http://www.cs.toronto.edu/~rgrosse/courses/csc321_2017/readings/L06%20Backpropagation.pdf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd8G/X5x98nyfJKoiySQAQR2wqIsLehGAjDZrQM82MZ\naAGDGQUXuDKKCgXEMIWCqUtZpkAxlFmLjYGKHVYR4EAYDigDEkKU5am73x/POXWCh7ziJH7evPQ6\nI914lMT3ue8zDdu2URRFUZR2XENtgKIoirJ2ocKgKIqirIIKg6IoirIKKgyKoijKKqgwKIqiKKug\nwqAoiqKsggqDoiiKsgoqDIqiKMoqqDAoiqIoq6DCoCiKoqyCCoOiKIqyCioMiqIoyiqoMCiKoiir\noMKgKIqirIIKg6IoirIKKgyKoijKKqgwKIqiKKugwqAoiqKsgmeoDVAURVlXCZhRN5DpvLwdfs5c\n7X0v4Ebuue4+/NzxvX80RApnDeb3UmFQFGVYEDCjBpANjO3kNRLIBUb08MoFsvifCLjX6JcQ3gQG\nVRgM27YH8/yKoiiDivPUPhnYCNiww7bjzxMQAcjs4XTNwLJOXss7bJvTfLWs9nMKaHO23f3c5WcN\nkUKrD39EvUaFQVGUtR7n5r8FMBXYDNi8wzbAz70fKWA+MM95/QD8CCzq5PUTsARY3hApbB3kr7JO\noMKgKMpaRcCMjgN2BELAds52KuLCaecn4Cvga2f7DTAHEYG5wMKGSGFqDZq9XqHCoCjKkOH4/acC\newF7OK+tO+wyH4gDHzvbT4GvGiKFP61hU4cVKgyKoqxRAmZ0InAgMN3ZTnI+Wgi85bzeBT5uiBQu\nGBIjhzmalaQoyqATMKObAccARwM7O28vBF50XjFkJaBPqmsBumJQFGVQCJhRP3AicCywg/P2u8CT\nwPPAR2sqy0bpHSoMiqIMGE720EHAmUAR0l3hHeBR4F8NkcLZQ2iekiYqDIqi9JuAGR0DlCKCMAVJ\nD70HuKshUvjVUNqm9B4VBkVR+kzAjG4IXICIwkjgFaAKeLIhUtgylLYpfUeDz4qi9JqAGR0LmMC5\nSHuIR4DrGyKFHw2pYcqAoMKgKEraBMxoJnA+8HvABzwAXNUQKfxySA1TBhQVBkVR0iJgRqcDtwNb\nAlHg0oZI4cdDa5UyGKgwKIrSLU5B2u1IDcIs4OCGSOHzQ2uVMpjooB5FUbokYEaPQdpQHAZcDoRU\nFNZ/dMWgKMrPCJjR0cBfgeOA94CTGyKF9UNrlbKmUGFQlNVImLFRSKWuBbzuj+QPq5zugBndHvgX\n0s76D8B1DZHCtiE1SlmjqDAoSgcSZmwC8BEy3AXgdOCuobNozeF0Ov01Ek9YCOzbECl8Y2itUoYC\njTEoyqrcDoxDAq0vAbcnzNgO3R+y7hMwox7gNuDvwH+AHVQUhi8qDIrikDBjhyMdQP/oj+Q/BhyP\nDISJDKlhg0zAjI4CngbKgJuAQ7Td9fBGhUFR/sc5wGzgRgB/JH8B0u/ngIQZmziUhg0WATM6GXgd\nmY1wZkOk8CKdfKaoMCgKkDBjfuAA4H5/JL/j3N8Hkd+T4iExbBAJmNFNkTkIAWSVcOfQWqSsLagw\nKIpwImAA93d80x/J/wwJRp8wFEYNFgEzmoeIwmhg/4ZI4YtDbJKyFqHCoAx7EmbMAE4G3vBH8jvr\n+fMQsGvCjAXWqGGDRMCMbosEmDOAXzRECmcMsUnKWoYKg6LA5kAQeLiLz59ztvusGXMGj4AZ3QIZ\npdkK5GuvI6UzVBgURQbSg4yb7IxPgSSw15oxZ3AImNGNkRTcDODAhkjhF0NskrKWosKgKJKRMxvo\ntHW0P5JvAW+xDgtDwIxOQERhDHBQQ6TwsyE2SVmLUWFQhjUJM+YBCoAXemh98QawTcKMjVkzlg0c\nATOaDTwFbAwUNkQK3x9ik5S1HBUGZbizCzAK8bt3R3sV8B6Da87AEjCjLuA+YDfgxIZI4etDa5Gy\nLqDCoAx3DgBsoK6H/d4FUqx77qSrgWOBSxoihY8PtTHKuoEKgzLc2Qv4xB/J/7G7nfyR/OXAf4Fd\n14hVA0DAjJ4IXIo0AbxpiM1R1iFUGJRhS8KMuYDdkcByOnwMbDt4Fg0cATM6DWmI9xpwdkOkcFi1\nDlf6hwqDMpwJIgPt30xz/zgwKWHGxg+eSf3HGbLzGLAIKG6IFLb2cIiirIIKgzKcaQ8kp7ti+MTZ\nrrWrBifYXA1MAY5piBR+P8QmKesgKgzKcGZP4EdkwH06rPXCAJQDhwPlDZHCdFdCirIKKgzKcGYP\n4K1ejO6ch7hn1kphCJjRnYBrEDfSbUNsjrIOo8KgDEsSZmw0kAe8ne4xjoB8wlooDAEzmoO0CP8B\nOEODzUp/UGFQhis7Otvedhb9BNjW6ci6NlEBbAWc1BApXDTUxijrNioMynBlJ2fb2/YQcSSTyT+w\n5vSdgBk9DCgFbmqIFL4y1PYo6z4qDMpwZSdgdk+FbZ3Q3nwuOMD29ImAGR0D3IkU310xxOYo6wkq\nDMpwZSd6v1qA/3Vg3WIAbekPNwMbAKc2RAqbh9oYZf1AhUEZdiTMmA+5sfdFGOYBK1gLhCFgRg8C\nTgGub4gUfjjE5ijrESoMynCkPfDca2FwMpO+ZIiFIWBGRyIupJlIozxFGTA8Q22AogwB7YHnD/p4\n/JfA1AGypa9cg8xX2KshUtg0xLYo6xm6YlCGIzsACX8kf0Efj/8S2CxhxtwDaFPaBMzojkAZcEdD\npDDddh6KkjYqDMpwZBrwUT+O/xLwMgQpq04vpEpgIXD5mr6+MjxQYVCGFQkzloVUPP+3H6dpz0za\nsv8W9ZpTkVbhFzVEChcPwfWVYYDGGJThxlTATf+Eob3p3hbAS4R9LmDc4pasicvbvFljvCuSOZ62\n7wknl/TX2I4EzOg44HrgdeAfA3luRemICoMy3JjmbPssDBO85yz6oeUvLVmud08jXPQbyyY440d/\nzlsLp5CyXYz0NDF9w1kEwr6FSKX068DTwPuEk/3pYfQnYDQ6eEcZZNSVpAw3piF1CF/1+siwb2fC\nvnu8roaFHuM7Lxg7WTZLH2rYft7rCzZlpKd5RiB30YMtlnvuY9+FeP2HKfVI+4zLkJ5MXxD2XUXY\nN6W3lw6Y0RBwBlDZECmM99p2RekFKgzKcGMaEPdH8lNp7R32GYR9hxL2vYPc3I8F/mHbue80Wbt8\n9ueZ+c993zRyc+B3v37g5V2PuufNE5utjM2B59/5cZNdK+rzfwOMB34DzEZE4kvCvrsJ+9KKUQTM\nqIE0yUsCf+zl91WUXqPCoAwbnI6o00jXjRT27Qy8DESRm/v5wGTCyTNTjJ9h2QSQeoJHkdYUAJTX\n1DYBJyGZQw9X1OcvI5y8m3DyAGBT4K/A8cBMwr47CfvG9WDJwcCBwFXaOVVZE6gwKMMJPzAG+Ljb\nvcI+H2Hf35AVQgg4FwgSTv6FcDLp7NXgMlwjMlyZy4Azy2tqV/H5l9fULgDOQlphn/q/cye/JZw8\nDxGI24DTEBfT6U4QexUCZjQDWS3MAu7o/VdWlN6jwqAMJ3oOPId9hyAzF36DrAI2J5y8nXCypeNu\nc5bP8gJMzArcX15T+1MXZ6tFBgFdUVFclLXqdZLzCSd/C2zvXO9O4FXCvsBq5zgd6eR6UUOksAVF\nWQOoMCjDie2c7c9XDGFfNmFfFfAMsATYk3CyvKuU01lL3j8MYMdxB7zR1cWcVcTlyErl153uFE5+\nAvwCWVVMAz4m7DuJsM9w+iGFgdeQrCZFWSOoMCjDiWnAN/5I/qo3+7Bva+TJ/kzgRmBHwsl3ujpJ\nRXHRFotbftgDINszYmJ3FyyvqX0ZeA84q6K4qPOpb+GkTTh5H/+Lf9wPPDCexRcjLbUv0fRUZU2i\nwqAMJ34eeA77jkG6rG4EHEo4eTHhZE9zDU5vthpTtm03AYE0rlsFbAPs2e1e4WQDsnq4wrY57p/e\nay4NGV+/2BAp7FKkFGUwUGFQhgUJM5aDtLAQYZA01MuAR5C+SdsTTj7b03kqiosykYDx04ZhNADp\n1CQ8jLinSnvcM5xMEU7+6dK2Xz821ljietJ7xR6EfUekcQ1FGTBUGJThwrbIv/f/EvZ5gXuRSuIH\ngP0JJ+ekeZ7DkNTVvwENpCEM5TW1y5EWFsdUFBf5eto/YEY3/mdq/8OPb7nsUbdh1wNPEvZdQtjX\nuStKUQYYFQZluDANIMv1VgPwAlACXAmcnIbrqCPHAd8DLyEFa+lWMT8AZALpPP1fCRif25tcBOwD\n/BOIAPcS9mX2wlZF6RMqDMpwYRpYy8dlXFsD7AGcQDh5VW96F1UUF40ECoFHy2tqU8iKYXzCjOWm\ncfg7iJAUd7dTwIxuhozr/FtDpHA24WQTcAIiFiXAi4R949O1WVH6ggqDMiwwWJrvNb7INAx7DFBA\nOPlQH05zOJCFxAxAbvSQnjvJBmqA6RXFRd1VOl8KtCErBEGylq5CViu7AO8Q9gV7bb2ipIkKg7Le\nY185+iRwbedxJZYBuxNOdll70APHAQmgfWpa2sLgUIN0NP5VZx8GzGgAWRX8vSFSOPdnO4STNUjW\nUi7wJmHffmleV1F6hQqDsv4imUdXpewJ99vk0pTa6Y+Ek73vqgpUFBeNQPoVPVZeU2s5byec7eQ0\nT/Mh8DVwZBefXwpYyMyFzpH6it2BucALhH2npHltRUkbFQZl/STsywIeBK5osnZ5CcBiTH/mIx+I\nBI+f6vDePMAmzRGfjjvpaWB/R2hWEjCjU5Dq57saIoWJzo5fidQ77AW8igSkr9aMJWUgUWFQ1j/C\nvg2Qrqj/B5iL2858HbmBf9KPsx4OLEaG7gDgj+S3AvPp3eznpxGBOXC193/v2Bj52RGdEU4uBg4F\n7kbabjzgiKGi9BsVBmX9QoKy7wA7AscQTl4PxjRglj+Sv7wvp6woLnIDRUC0vKa2dbWPE/ROGF5H\nBObw9jcCZnQTpGju7oZI4XdpnymcbEWa7P0eaeOtGUvKgKDCoKw/hH37I4HhHGBfwsl/OZ+kP4Oh\nc3ZHito6a2Q3h/RjDDjC8gxQ5AgOgOlsr+u1ZZKxFEHSYHcB3kp3AJCidIUKg7J+EPb9GngOeYLf\njXDyXYCEGRsFbEb/hOEQIIUUxq1Ob1cMIO24xwM7B8xoe+fVexsihd/22cJw8hFgP2Qm9NuEffl9\nPpcy7FFhUNZtwj43Yd/1wF1AHbAX4eTsDnu0t9rujzBMB94ur6ld3MlnCWB0woyN6OSzrngRiScc\nBJQjv4e9Xy2sTjj5FrK6WQC8RNh3fL/PqQxLVBiUdZewbxTwJHAx0sG0sMOEtXZ2cLbv9+USTjHa\nznS+WgBxJUHv3EkLgfdTuAqBM4CHGiKFDX2x72dIOu6eiEvtQcK+KzRjSektKgzKuknYtxnwJuLm\nKSOcPItwsq2TPXcE5vsj+fP6eKUDAIOuhaG3tQztPO/C2tmbas4BbuijbZ0TTi5CVjn3A1chKa3e\nAb2Gsl7jGWoDFKXXSMXvv5Ab9kGEky93s/eOwAf9uNp0JIvovS4+bxeGXsUZlnhGvDqqbdllWy3/\n8r3aOy7+FKCytG5nZKRoPjKgpxkpinsWeKCsqmBp2hcIJ1uc4rcvEXHYgrDvGMLJvgqkMozQFYOy\n7iCVzOchT+/fA7t2JwoJM5aFDMjpkzA4E9emAy+V19R2thqB/7mSeiUMD04+bttmw8uOyQ/nVpbW\nbVBZWvcEMANpmNcAPAa8AmwN3AEkKkvrLqosrUv/yV8ylq5GMpa2Bz4g7Nu7N3YqwxMVBmXdQOIJ\njwC3Ik/QexBOftnDUSHATd9XDHnIDb8rNxL+SH4jsIheCEPAjGa0uTIu+CFzgx99bct3tW37Q6RY\n7QpgcllVQWFZVcFZZVUFJ5dVFWyNBJT/g7icZlSW1uX16ltIxtJuwFLgFcK+8zTuoHSHupKUtZ+w\nbxriOtoUCTTflGa77B2dbV+F4SBn+2IP+yXoXYzhOGATr+F7BOYca1s/zjXc43cvqyr4sLOdy6oK\n3gEOqyytOwy4B3ivsrTu+LKqgs7qKjonnPyUsG8XoBoR190J+85MNNU2AlsArcBcR+iUYY6uGJS1\nF3Ed/Rp4G+kouh/h5I29mKGwE/I039f6gOnAF+U1tQ097Jd2LUPAjLqAS0anjC82ztjtQIDWZU/f\n1ZUodKSsquDfiEvoM+CJytK609K55kokY+tXwGVt9vjiH1vMOZBKAvVILOKnhBl7LmHGDkuYMb03\nDGP0L19ZO5F+R48h9QlvAjsQTsZ6eZYdgQ/8kfy0h/G048x2/gXwfBq7zyF9V9KhHpttTliWmely\nj3GB6xvbWrxrunaVVRXMAQqQCXJ3V5bWXVJZWpe+WyictBJNtfXzm+9pabR2HZHrfjF7hPuJx6Dt\nVOCvQBCp8H43YcZ2Tvu8ynqFupKUtY+w73Dg70gV70XAnwknU13tHqoOjQKmAlshwdqJXitj1GPc\nvMMrvne/vrn67LuBhcB3SKHbe/GSeE8uk72AbLqJL3QgAUxImLFMfyS/pzGhl+zfmLE0xzamAIeB\nNR34TUVxUVZ5TW1TGteirKpgmeNWqkaa7uUCf0jn2IQZOwp4GFzvZ7k+PGNMxu2/B44bnXH3BOCk\nRFPtxUjzwQjwTsKMXQVc7Y/kW92cVlnPUGFQ1h4kwPxnpKHcR8D+hJM/64gaqg75kO6k+wJ7I9XN\n7avfNuCHTZsnt3pwuz7J+SoDOBhJ/8xw9mkOVYdeAx4FHo2XxFcvigNxI7Uira17oj1ldUMko6hT\nAmZ0t41bXXtv1+IBuKWsqqC2ovhmGzjX+R4vpXEtAMqqCloqS+tOAFYAV1SW1i0rqyroth4iYcam\nIrOnZwAHj7/2d0sIX3080rupEvjEn1V0KXBHoqn2KeA2IAzsnjBjJ/oj+T+ma5+ybqOuJGXokVhC\nMTATmXd8DdLvaKUohKpDE0PVofNC1aGXkaf/R5H5BQuRPP3DkdVCTrwkPvmWhouvArhg3okHxkvi\nk5FW135nv0qkf9LfgTmh6lBlqDq0+hS2g4A3y2tql6XxDdKqZfDa/O7gxgzLxv4KuMx5+1Wghf8F\nutOmrKrAQiqna4DrK0vrzu7SQDPmRURhGfArfyR/CdCe0voPRFzfAP4CvOHPKtoEmSZXiriuPkiY\nsR07O7ey/mHYdq/dr4oycIR9WwO3IxXG7wNnEU7OAAhVh7KRG/nJyI3TDXwK/BuIAm/HS+Kd1hck\nzNjtznGjO3ODhKpDBtLq4iykdsBAborhU56ZkovMWbi0vKa2xx5GCTO2DTLr4f/8kfyHO9snYEY3\n2aPJ883eTRkuoKCsquCV9s8qioteBsaX19RO6+lanVFZWpeBxGMOA0rKqgru78TGSxD30JH+SP5T\nq38O4KSwHg/cgrjxbgCuSTTVTgUeB8YBJ/gj+U/2xU5l3UFXDMrQEPaNJOz7ExBH2kWXAbuFNt3k\n/VB1aJ9Qdegu5Ob8MPI0ewMwNV4S3zZeEv99vCT+elei4LAj8GFXvvF4SdyOl8RnxEvipyHpmvcj\nDe0+/Xqj5Rc5u6UTX4A0VgwT2oxLd2vyuJqxox1FocN1tqsoLtowzeutQllVQStwLDKc6J7K0roj\nVjHOjI0GLgGiXYoCtK8eHkTqNx5ERo1+4c8q2sZg6W6I+D2eMGMXJcyY1kGsx6gwKGuWsC+TsO98\noN2d8jCwdWjTTV4KbbpJ2Hn/NaRa93Fgf2BKvCR+abwkXp/OJRJmzE17pW8axEvi38VL4r9B/PxL\nLMMub3VbzY/vM+fzNL/VEsRF02ktQ8CMjty52XOaC1KZGGWd7NKe+TQ9zev9jLKqgibgl8iq65HK\n0rqCDh+XA2OQSW89E07+SDh5CrAPMlv6vslZ//fMpMzTrkRceDcAf3fcU8p6iAqDsmaQ9tinAF8g\nropP3sjOOjC06SZvhzbd5Engc+QJ9XPgRGBSvCR+arwkXhcvifc2I2ZrJKOoV4Vt8ZL4G6OXZuw4\nZX7Osu8mNGYuGdH2Zqg61OPQGycdtstahkCr66Jgqzsj6bIfKasqmN3JLh8DP9APYQBweikdCswC\nnqosrdslYcZGAucDj/oj+R/16oSSHrw74l4a4zF+eHZy5mEjPUbDvcgMiecSZmxMf2xW1k5UGJTB\nJezLJuw7AynKurfRMBb+fvy4P4YCGy8unTShPRsmF0lL3TheEj84XhJ/MF4S79MYTofdnG1Xje+6\n5MjYRsGMlGuE2zJuAjYC3gtVhwrTOLTTWoaAGXVv2+K+wAJrrOW6sLMDy2tqLcSdNL2iuKhfv5Nl\nVQU/IgKzEHh2cZtdDowEbu7TCcNJi3Dyn4h76RLDsHedlHnOqT7PnZ+BtTfYbyfM2Bb9sVlZ+1Bh\nUAaHsG8cYd8VwOwU/O217Cx+OXnSy7tO8W9eOzL3SgxjTyQdcvt4SXy7eEn8pnhJfO4AXX13pCNq\nuq6gjkwHmPJ9zi1I5fQs4KlQdeiUHo7rdMWQ1+I+c+tW94gFbjtaVlUwv5vjn0emuu3QzT5pUVZV\nMBcJ5re6DC5N2fZnyBzsvhNONhFO3gAEgPKRnqfHbOD9fYbB8s2g9cM55jP799duZe1BhUEZOCTt\ndG/Cvntt+O5Tb8ZVl44fu2zXwMYLz5k0Yasvvd7dMIynkAwjf7wkXh4vifdnslpX7AG83ceirOnA\np+U1tXPiJfHZSPVzHXBvqDp0UTfHJYANnfjGSoIt7itTYGfblPZw3fZ+TP1yJ7VTVlXw1Q457otH\nuY2Mzxqt8U8tbp04EOclnFxGOHkzsFmm69PSCd7yuR5j/gib7JcWX3b+04R9q6f9KusgKgxK/wn7\nJhH2XdwK9e9kZcb+NG7MCfmbTLaOm7wh/x45wt9iGG8iweSJ8ZJ4Sbwk/kIPGUV9xvGpb4v0V+oV\nFcVFOUjAdWU2UrwkvgxJA30EuCFUHbrWSXVdnTlIOu3KG/Cxv33+qM3aXBPmeKzXLrvjgG5XQ+U1\ntd8jRX29rmfoik28rr1t226c3WKNAOoqS+v6lPXUKbKC+FuGa85mYzNuKs4wvlm4LHX0YYtbT2+w\nrtzgWcK+4wn7ejPuVFmL0MpnpW+EfX7gVwvdruIPcrL3eDUnx6jLzW5b7nKBbacwjJeAJ7Ka7dr7\nb05lIj14Tqu/LrglMAmpRB4P5ABZgBcp9FqOZPjMAb52Xu8DM4Iz61vSsGxXpCbhrT58q3ykEG6V\n/kjxknhzqDp0POKe+j1SXb16C4qOKatzATZpc92YApYZ9q/TvP4LwAUVxUUjy2tq0x/K0wkJM+YB\njjIM48kU/A2p+3itsrSuoKyqIFGfF5yAZG5tj8ysmAhMQGoVPMifIUir7iTy3echbUW+dbazYaMv\ngzM/eKTVjD3hIvm3ZakjTm2ypu0/LiNycIYr0UjY928k8+xZwsm0Wn4oQ48Kg5IeYZ8b2DHpch06\nIyuzOD7GF3w7O4t6rxfbMPC0WYs3nWe8ss8nqfp94nZjditbAGcjRWOjOpxpGXLj/AFJTV0ONCGi\nkIkEokcivuwC5/8BGuvzgm8jDd4eCs6s/6ELS3d3tu/24VsegkxN+1mzvnhJPBWqDp2F/M5cEaoO\nNcdL4td02KWjMLx7/G+f33n3Ns+msz3WR3+9bfrXaV7/eaSt+C+QIr7+sB9yk3+krKrgtcozX5oO\nvOBpa4y/v8O+83JEqNuZ47zmInUlrYCNiMMIwIekuwaRgHxHT0OqPi/4FVK1Xu+d+qtb7S0OPOV7\n646sEZ6n3/d57ikwDOtYYAVhXx3SfuMZwsnOsrOUtQQVBqVzwj4XsHW9N+OQTzO9v/xuzOidPsn0\nZn/m8jL+J4ONE5a151x7/smJ1E8bLSI7sxW/IXn0v3TOMA9p5/wPnJuG85oXnFmfVrl9fV7QQFYX\nuyMungKkl9KN9XnBZ5GK6RdXO98eQL0/kr+4D9+6EKgrr6ld0dmH8ZK4FaoOnYH0XPpTqDrUFC+J\nVzgft09ymwywQcp1mwEsctvn9+L6byC9jw6i/8JwLLCsZdbzdfV5Z5xSABcsGblJ7kfbncN7O/4u\nd6svH71t0vczHgc+Ds6sX5TuSevzgh6kJ9TGSFuRvA6vQ1o+ezyj9auXydqxhGUTj9w7uXi3JdbX\nD72Sm/UeuRs275Q1urXI7bUh7KtHJtT9B4gRTg5U4oEyAGhLDKW9FcKkjzK9e33u9R7yg8u9+w+t\nni0WLfd6M5a4mbzQZvKPNlMW0DZmKZ4ODnYLeerveOOvBz4Pzqzvy425R+rzglORHj4nI6IRAy4P\nzqz/jxP4/RF4xB/JP6M3560oLtoKyWIqK6+pvaO7fUPVIQ9SGXwscHK8JP4PZ35BE1BxSeOKW3Zs\n9sxb4LbmRCoP3LiXdkSBLctrarfqzXEdSZgxj23b31vJ775e8eqfNkbcRJ8Af/lou3M+WTQ2+JDz\n3ollVQWP9/U6q+OIxqbAVDCCGZvvf6h3q0N2N7y5Ga3fvknL51HsFT/i8lhJ76i2tuxxLaMyR7dl\nZI5qwzuq9RtPph1D3IbvAx8RTvYnZVnpByoMw4xlfxw95s3srF3muT37NDe692xc4Q62rvCMZ4nb\nM2qxwUaLbDZcBDktYGPQ5smm2Zvb0uL1zW7KGj9nRfbEhY05G/zUlDVu/vKcSQstt7c986cJcRMt\nBRYgrpXvy6oKumyX3R/q84KZSJHV5cgT7NPZu5VVejac9jxwoj+S/2BvzldRXHQhUAEEymtqe3Rz\nhKpDmYjffl/gsHhJ/LmEGZsNvHbd8sbsqa2eo9/LbDv93lun39VLO85DJqxtVl5T+01vjm3n66Ou\nKPNuOf32xnf/Rtvc918BrgVebl9ZVZbWTQSeRFZiVwDXDdbfU8KMjbJt649gnA22K/Xjl+83f/xw\nwlqSCIAdBCOnfV93VsrK9LW5vCPb8Oa22Z4sK+HJTtV7R7V9kJFjfYQ8gHxBOKlT5gYZFYb1ibDP\n+NTrHRvP8G7XuNy9i7XCvR0txpZ2k8dvNOaOcTeNyMpoHGGMaM4hpyUX25VLa0YObR7ZNmWOSLV4\nRzW3ekb8uf3DAAAgAElEQVS1We5sj+XyZGMY/emJk0JWFP9FMm7eBt4oqyroaWZB2tTnBbOB84Bw\nxhbT7axtj862U60bb3xjQaKnYzviNLKbWF5Tu226xzhzIF4DtgT2e7b+jlst2256cElLfqOLxqsq\nDxjVwyk6syMPWXWVltfU/q03x9bnBXOAW7zBI073bnkwK165+mRr6dwHOnPdVZbWZSNDkI53vsNJ\nZVUF3/XW3nRJmDE/Erg/Hcneesa2Uncuf+6iz+yWZXnAVLC3MVxMAza3LWNkx+PdXouMEW1k5KZw\ne62k22t978qw57i91leeLGtm5ujWD70jUrOAeYSTg5LxNpxQYVibCfuMp725Y5YlM0NWc07IassJ\n2m25ATuVsyFW9lg7lT3SbsvMMuxsL6kst2FnGYadCYYXy+Wl1ZNJm9tDm8uDeH3awG7DJgV2m/Oe\nAdAExjIwlmK4FxtG5o8YWQsMV848wzVqjmF4FyAjMhchrpplzsEWEqQEySwagQSOJyJBWD8SsJyG\n+KNBfOivIkHkh8uqCjqbhdBr6vOCW2bvdeHbruyxY5e/dPnjQElwZn06LbOpKC7yIZXCFeU1tWZv\nrhuqDk1CJsyNfOTzG98zUjm7vpxsG/uBt+2mu/8yvbu6h65sMZCZDh+W19Qeme5x9XnBIJJSu23u\n9Ou+N7LHfL3x9fvu2d0xzuS3EiRW04I0Mny4rKpg0G4KCTO2MdIq/DfAJNu257dYTS/Nb/zm3Q9+\nfOnrFqvRC2R721LjslpaJ2e2pTbITLVtkGWn/BlWaoIrZY+mzch2WbbLZdm4bBu3s/W4U3gz2shw\nW60eT6rJ47aWe1zW4gx3aoHHbc1zuey57gx7rivDnuvy2PPdXmtu5ujWud4RqSW9GBc7LFBh6Ibq\nCw5y283GSHfKNRrDO9ZlZ4xyGe4x4Bnlsl0jwDUSjJFgjbFt92hslw9cI7FdOTbubMN2ZYPLa+Dy\ngjvDsF1uDJfbwOXCdrsNDBe4DQOXYRhuDFwGhoGB4Tyoy9YADMOWLc7WAMO2wbCx7BSWncK2LSzk\n55SdotVqpsVqotVqptVqpim1nJTdp4epeciT/1dI1sr7wAflNbVL0j1BZWmdD0kHPQgZnLMF0Ig0\nZasqqyroS3rpShJmzGXb9gLrp6+/WvGf63dC2nMfEZxZ36M7pqK46GjHjn3Ka2p7Oz6UUHVoK+CN\nc+ce7z1g8V6jHl7a0vZeZlvu438+OJ302s7sqUTmUoxLZ6pbfV6wGLgHWO4en3duzt4XPgxc7o/k\nX9PDoQBUltZticRMdkGCweeXVRX0rq9SD1QUF01GXFfbAlNduPIm524V8OdsPWrDnM3IcHlps1pY\n2DyXhU0JFjbNYXHLDzRbneYB9B77fyIiQmLhsmzcto3btnBj48ay3diWGyvlxm5zGVab27Bb3bbd\n6jLsFhd2swuryYXd6MJeYcAKw7AbDYNGw7BXGAbLDcNeYWAvN2CFgb3CZdvLDVhuYC91WSw1LHtJ\nRspampNqWZ6J1ZSRm2rZ4LHEWncTHlbC8NJZt7ZOztnUA66VN9+V/xkGBq7V3pP9XMbaWQdoO395\nRi/cPZZt/WRjzcG2v7OxZ7daLbOWti6aNWvJ+18nVnzRjDzxj+vw2gBJHd0McZl0LJL6CHgOSbN8\no7ymtjUdG5wn1Z2RGMH/IemsLwBXlFUV9CXNlIQZ2w5xWZUsffKMecjwGgs4Ojiz/tXujq0oLroP\nmfswobymtk/KGaoO7Xpq4rg3j126j/vGxiXP3HprYTr9lbqy5yDkz7WwvKb2ma72c7K2yoEbgdeB\n4pFH3nkAMvJzJ38kP+0mgpWldW7k7+Ma5O+9GriprKrg0z5+h5FI+m8R8kAQcD6ykdqUmUgtxHyv\nK2vBtmPyJ03KDuRlu0ds5zYytmr/N23b9kIL63PbTs1J2ak5Kbt1bkuqac7y1JLkkpaFzYua57Us\naEq0NlsrvMiqNQtpoCg/23a227J9WXbbpEzaJrpse6wh6bcjbMi2MbIs8Fq4PBaGxzIMVwrDZRmG\nkTJcpAxDnsIAFy7nniD3ie7+39V+P3He+9n/y50FVyf7uwz3yvvOqueUbWPLZ88dft99h/Tl7yVd\nhpUwPFN67eIxGZN9YGPbtm1jIz9b8g7yH3b7z1b7T/Imlt3hZ8vZz7KxU2C3GYbdakMz2M0GdiMG\ny1wYyw2MJYbh/sltZPzoMbJ/yHGNmZfpyVmc6fKu8Lqym1yGqw0pmko5244/d/ZeG5BqH3LvZONk\nIEViXufnbOQXYLTzGoNk8WyCpBpugvyytlen2kjmSgx5anzBH8n/afU/w4rioglID6FdkJbYeyJp\nzz8ADwHV5TW1aT9tVpbW5SLDci5BCt6eBM4tq+pdjCBhxn6H3CA39kfyE/V5wS0Qd9XmiDh0mv7p\nNK2bD7xUXlN7fG+uuTr3lUeTB2SMGvVb/59nfD5y1t7xknhfVwyZiGvrwfKa2k5badTnBV1IsPy3\nyGrn5ODM+qaEGatGbsiT+tISpLK0bjRSvFeK/Bt6DictuKyqoNvvU1Fc5EaE4DRkZdj+PV5D/l29\nCXxSXlPbbfDYmR+xExByXlsj/2YnI/GJzmhE3GEdX622bbcCLhs8NmRY2B7LsL02ttuWf7duG1yG\njdsAl4HhEikwDAPkVtyvMNvA8+7SV9//VeUVOw/mNYaVMCir4gxbmYL88m2PzCPYCykqa0NiAU8A\nNV3N+60oLhqFCMQJSOsILxJkvhaoLa+pTesfWGVp3UgkiHwpIoAXA3c64yvT+S4vAxP8kfxQ+3v1\necGxyI1tB+CE4Mz6RzqxfzfH3hPLa2p7lcnUkTPOe2HPAsvzxt4jMjA3uYX/5n7xAJLK2qdfsIri\nokcR0d3Y6b66kvq8oBu4D2lPfitwYXBmvQXgZEa964/kH9PX7wJQWVo3HhGHc5CY0U/AU87r9bKq\ngoUdbB2NxA3ORv49zQX+hUyVe6O8pnZAMp7+c8GrXjfG1m6DbRCh2BDsCZaRmmQZqQmWYY+ysXMN\njCyX7fK6bZfHhdtl2MbKQFhXWwuLNiNFykiRImVZhm1bWJZlWJaFlbIMednYbZZhtVrYrZZhtYDd\nYmO32NACdqtlyM82VouN3WobdrON3WxDk23YzWA32tCEPDg22thN2EajAU0WdjMGjRapFhujKYXV\nZGM3GtDU5GptaqR1hY3ddGXYHJQMso6oMCir4LRS2Bk4AilW2xqpBn4Uaa3wRvtKZXUqiovGIgJx\nIbIaiQNXAk/2QiA2Be5EuoO+ChxfVlUwrwebRyJB8Vv8kfyLO35WnxcchaSV7gmcGpxZv8rYy4ri\noj8h2TITymtq+zzsvvycF2dsY7l2nj7Ky6NjX/zXPROfOBq4Pl4S71Uwu4NdJyFT5XYrr6ld6V5z\nVgr3IEHjy4Fr27OOEmYsAHwDnOuP5N/e1+/SkcrSukyksd/RyL8Jn/PR57bdMqN1Rd04q2XmPmDl\nguctjIzbDNfIRy986J5OXXKOGzELaYUyxnmNdV4dfx6LuLTG29jjbewNDIxRBp0/vqeMNpo8y2n2\nrKDJs5wWT2NLi7t5RZureWmru2Vxm6tlkWVYCy0jtcAyUj+kXK0LWt3NC5rdjfOXZC2cvyh33gJg\nabwknpY7dH1HhUHpFsd3fzpSUDYKebq+EnixG4HIAI5Dnv7zkPjBueU1tV+kc03n5nEq0pZ7CXB0\nWVXBG93YeATigirwR/JXH5tJfV4wF3nSLQCKgzPrH3XsNBBf95zymtqC1Y9Ll9LzXpi8dYs78b3H\n+u6cEVkb29i/PzRYNgV54j4vXhK/rbfnrCguGgN8D9xaXlN7kfM9XIho/hq4Mjiz/qqOxyTMWAmy\nktjOH8mP9/X7dIUjErsCe7U1//fotsZ3tsde5nZ5puDJ3huXZ2X/wFbn1dbh5UFcU9k9Xcci1dzq\nbm5pzFhmr8hYkrHCuySrybPcaMpYTpNnOa3upiWt7uZ5Le6mOc2extkrMpZ8ucKb/NI27AQSt5g3\nWE0ahwsqDEpaJMxYLnAScrPfGPEXX+SP5L/Z1TEVxUUeJAXyKuSGEAGu7kWQOoSM9wwAF5RVFXT6\nFJwwY1XISmWcP5LfqR/cyfF/AbmxHRacWf98RXFRe8D6rPKa2qp0bOqM88998YmtWt1Hvp7VesqN\nWTm3Av84JHj2bxF3yhHAsfGS+L96e96K4qJngKnApof+9yuQvlPnAFcHZ9av3sSPhBm7Fwmib9DH\nluPp2DQRGa50FPAhxojLs0afMQf5OxqHxInGIELQ/spAxKGx/WVjr1iQ+21mYvQXGy3M/W6zJZk/\nbrrcm9yi2bMiy3KlQITlM+c1C5n89wUwK14SH5SqeuV/qDAovSJhxjKR4OLlSEO1u4FLuopBAFQU\nF00CbkJu3m8DJ5TX1KbVWM4Jht6PxC9uAi7umGfvtKKYDbznj+T/svOzCPV5wdGIe2pL4MBnpm1+\nCCJ0G5bX1HbVlK9bzLIXM8darmVJl73iwZHNo19n1KdIr6ajQtWhbGTOwi7AwfGS+M9WM91RUVx0\nCnAvsOuh//1qP+B6JOB8UWdFawkz9jXwkT+S/6u+fJc07DkKWbHkIqvGinSzuELVoQyc1Ybz2hMR\nEZDOrR8gWW7txZAz+xq8V/qPCoPSJxJmbASSvXIB0pb5/J7aUFQUFxUjcQoXcEZ5Te3D6VyrsrTO\nhQRZz0HSKH9TVlXQ5tixF5KqmVYbjPq84EQgZsMGz4c2XWy5XF+X19T2efpY2bkvVExt9Vz4bmZr\npPrWg36fMGMvAiP9kfzdAULVobFIllcAOCheEu/SJbY67e6k0cubnt/zyzlFSPvqE9oDzR1JmLHJ\nSBuSC/yR/Fv6+n26sMODrPbKka61p5TX1Nb3dFyoOrQ5kp10EOLGa8+Am4U0DHwDWXnO7MNcb2UQ\nUWFQ+kXCjG2LPEXuATwAlPkj+V0WvlUUF01Biqn2Aq4DLl8966YznLjD5YhbqhY4pqyqoClhxm4F\nzkQyktIquKvPCwZ+ysmc8daW/vE5za2XnPXk8zekc1xnNi1yWUuAzNez2nKfv/ng1oQZuw/Y3x/J\nX9k8z6mOfg2pATkwXhJPe8zmbb86+E3DZo/96mf/xwXTgzPrO20nkjBjRyGuq939kfz+jfHsgOM6\nqkF6QlUCF5bX1Hb6JO+sCvZD3GcHIanCIJXczyOuvFi8JL5goOxTBgcVBqXfOJlMlyIriG+B4/yR\n/C4L1SqKi7xIbvzpSAzh5PKa2rQ6aVaW1p0F3AE8M8ZtHLXPSM/XwDs9uZF+dp5fHvRwi8ddXPDZ\n7M+8KWuvvnSDLTv3hbOntnoqP/S2PXrXX6YfC5AwY9cgNRmZ/kj+yrTCUHVoMiIO4xFxmNHT+evz\nglPnjh7x7kdTJubmNrUcVfrUC112Qk2YsRuBc4FRXcVZektFcdGmwEuIoJ1ZXlP7j9X3CVWHsoAD\nkZjD4Uh8YQXSUvt55zWrr2m7ytCwdpb0KusU/kh+mz+SfxUyM8EFxBJm7NSu9neeOM9E3FBHAq9V\nFBdtkM61yqoK/uoce2iWixeRm9bP6hO6o6K4KKvJmzE9sy1V501ZWwJP1ucFs3pzjsrSOmNiyvXH\npYZtz/akyjp8NBcpwlrl+8RL4nMQd8pPQF2oOtRtFlR9XnBD4NmJyeVLse3Fy7O8PdUl7AF8MICi\nEERcdGOAX3QUhVB1yBuqDh0eqg49hBQ2Po38PdY62/HxknhRvCR+W7wk/oWKwrqHCoMyYDgZSjsh\nVa73JMzYrQkzltHZvuU1tXZ5Te0tyFPmVCBWUVy0STrXKasquBM4Z8MM194p225b2GZ12TaiCw4D\nxjR6MyJIPcC+wP1OOmhaJNyp48dZrvGfe1N1L958SEfXyCoDezoSL4l/ixQRzgaeDVWHju7s3PV5\nwZFI7cU4t20XYhgPAL90Csl+bosZ8yJ/7r2ec90ZFcVFOyBxETewb3lN7buh6pARqg7tEqoO3YaI\n31PISqEGqbSeEC+JnxwviT8VL4lrW+x1HBUGZUBxspMORiatnQc8mzBjY7rav7ymNor4ozcEXncG\n5vTIEaMzaiZnGG2zWyzPG8tStzkB6nQpRfLd64Iz6/8J/A44Bviz03+oWypL64wRtnHDEsPiE2/b\n2at93KUwwMqVwz7Ae8AjoerQeaHq0Mpr1ucFM5AV0HbAMcGZ9R8gmUmZSGO9zpiGFI31qxEhQEVx\n0R6IG6gR2Oe+Q2cnQ9WhS5FW4O8i7r+XkdYXG8VL4qfHS+LPaQbR+oUKgzLgOK6lC5Eb2T7Amwkz\ntllX+zsdTX+B3NxerygumpbGZU5xGYbnh1b7dqS+4hYnQN0tTu1CAXB7h1YNNwO3IEL2u57OscSw\njhhtuTb6xJt6f8aNh65etNc+onKjro6Pl8QXIU/bTyPZVg+GqkO5jijdgQjrWcGZ9c8ClNfUfoBk\n8Jzr9CNanfY51/1aMVQUF+0PvGhjL3x+1+9vue/Q2X9DVjfXIC6j04FJ8ZJ4cbwkHtUq4fUXFQZl\n0PBH8quRG+BE4O2EGetyPkB5Te2HiJulGXilorhol672ddxTZwOx79vs85DVyblAOA2zzkeehldO\nVnNqAsqRJ/Ub6vOCJ3R1cGVpnWFg3JI0LD7KbOtsnvP3SFfXTlcM7cRL4iuAXwGXAcXAO/NH8xdk\nTsE1wZn1f1/tkFuRDrdFnZxuV2CeP5Lf50E7Nx5XeLiN/eyyrLbWRwrmbDhvfFMFMk/jD8Bm8ZL4\nPvGS+F1aXDY8UGFQBhV/JP815Ik2CdQlzNhxXe3rtMzYByl4ermiuGjvLnY9DZktfL1T7FaOuFv+\nUFla99uuzu8U2p2AdIBd1PEzpzbgZKQA7t76vOABnZ2jDfvwkbYx5aPM1Fef3lD4s5oEfyS/DenW\n2q0wAMRL4la8JH4tcNBB71lTJi3mnM8n89+ys9xXd7L7E4j7q9xp5dGRnYEes5w6I1Qd2uKky/Z+\nHHhqoa8l4997z3M3ZqXa04m3ipfEr46XxPs0YlRZd1FhUAYdfyT/CyRrZgbwz4QZu9zp7PoznDnH\n+yDDgZ533BsrSZixbOQp9k3gGQBHHM5AUl//XFla11VG1GVIQLWisw+dGoEjkf5JT9TnBXfo+Hll\naV1mCv76o8vio8y2S7v5ynNIQxjaeeS6thGnvWjlzNqQueET3NMWjDY+DlWHDu4Ye3AqjG9AZhsc\n2P6+00BwayRmkRah6tAmoerQBaHq0Jtbfjdi1rQvfb/8aWTrT6/tsPC0Zq81MV4SPyNeEn9Ts4mG\nLyoMyhrBH8lfiHRMfQC4GrjXyab5GeU1tQlEHL4GohXFRR2H3piI//6yjk38nEro45EWFHdVltat\n0haiorgogKS53l1eU/tlV3YGZ9YnkSybn4Bn6vOCm3b4+PxMjA1fy26d32LwWDdfdy7dxBg6Up8X\n/AXwsAEztpzHVim3cRAiXs8Cb4aqQ8c6hWMAf0d8/td2WDXsgMxnfb+razgZRVND1aHfhapD7zjn\nuHnaLN8me8XH0ea2Xxm3xOt/66wP79WMIgVUGJQ1iD+S34y4a8JImugLCTM2vrN9y2tqv0cC0p8C\nT1QUFx2VMGO7I0/9D/gj+a+ufkxZVUEz0ir8HeCflaV1Hd1BEWTOQ2dumlUIzqyfgwSAM4Hn6vOC\n4ytL6za3sP84y5PiqwzruoZIYXc98dNaMdTnBfdHVj1fAUXBmfXL4yXxF5D03VIkNlMDzA1Vh/52\n36GzD12a3XozkpraPlSofWDLSmEIVYeyQ9WhXUPVodJQdagaaZXxKTLIyI3N7496ZfKtO8waPRn4\nV0bKdXB5Te0AzdBU1ge08lkZEhJm7ARkrsBCpM9Rpw3mnNz9Z7LdI3c9xP/rRRmuzEakrXSyq3NX\nltaNQWIFmwMHNP1080SkLfeV5TW1V3V13OrU5wX3Bl60Mb54Lf/PK5rdnp3uHtXcuNRl+xsihUu7\n+W6XIpk8Of5IfqdP4PV5welILcCXwP7BmfU/a+IXqg65EYE6Aan3yDVsKHxzUmrk8gz76fy5L172\nw9nbbtbsH3fcVpe8gKT8buS82rOXFgB1yErqpVOemTIHCWSfjbToPr2v40yV9RddMShDgtPwbjdg\nGRKUvithxiatvl95Te3iAzY86awDNjqpBdjg3QXPPlLzzfXd9kQqqyr4CamNmGdby18E991I187r\nemNjcGb968DhszeZnme5M3b/wPjRs9Rl/7U7UXDotpahPi94OJKq+jmwX2eiABAviaectNDjkcE1\n+9gG57+39eJHMlKGe8/4uL02bN1g4qysby2kY+wSRASuRVZOU4CJ8ZL4cfGS+N2nPDPlJ+DfiCjc\nCPxaRUHpDF0xKEOKM+fhD0hmURsyN7oW8YOPBgqBMtu2V/zn+0fj8xu/yUdSTc/raXbwX077+xZt\nTW98bKcWZ3uy9yk//74Lb+6tfZWlddOx7WfH/vixvennD7i/HTXxyGPfeOapHr7TgUjDuH39kfz/\ntL/v1CmUI0Hk94BDgjPr+zQ1rqK4qNxjeG86KnABwBX+SP6feth/cySzaSoyf2L1dFhFWYmuGJQh\nxR/JX+6P5F+CTHq7Hxkh+Rhy43wJqTt4xDCMqfMbv9kPeRr+DfB+RXHR7l2cloriog1alz/1dzu1\nMDMj54CZnqztb6wsrTs7nSK4dipL63YBHkkZfF45aVwq5XInQz9+81h9XtB05i53xc9WDPV5wfHI\njflG5/v9oq+i4HDzhjmbPQ3w4Y8vb9NF4RsVxUXuiuKiM5EZB5sAB6soKD2hKwZlrcIpXtsF6ULa\nDLy9ejyhorjoQKRuYTKSonon8Ep5TW1LRXHRKOD/kBbd44HTs8Zc+DhSvFaI0xq8rKqgW3dUZWnd\nPoi7Z1H1iKZnfvDYZ+8279Odwu/cawLHIoNlfhucWR/r5DuMQuo2Ll765Bm3IhXg1yKjUS8Gbu1s\n0E5v+faSV893Ge5bnv62ksbUsv8AfwJeLa+pbXXmbx+JNCrcFmlzcUp5Te23/b2usv6jwqCsk1QU\nF40ELkRufD7ARgrj2vsyvY8EVj8EqCytcyMZTVci7R0uBR4sqypYpcdPZWndSMTdcwXw5Zee1K+e\nGNHyJvBCQ6TwGMcddAxSbb0RUk/xDyS4+01wZr1VnxfMGnFE1cLUD/WfNL5162SkgjgGnB2cWf/J\nQP0ZJMzYXcARNd/ccAnY1yJZTClgKeKGA4ljXAH8q7ymVn/ZlbRQYVDWaSqKizKRzJ0dgAlIdfDr\nwOud3Qgd91Alsir5HkkX/QQRlu2Q7J+xSKyj9MbRjWchIzV3aYgUriwiq88L5iIV2OcigV+AFmSV\nk5tTEHZZy+bT9G5VHVJQ9+xArBI6kjBjM4CkP5J/gPPncCQQQsRxNpKZNUMFQektKgzKsMPpxHog\nUi29LzLEHiR1tg6oKKsqeDdgRrOAb4BPGyKFnbbIcFYQ2yBtP7YAvMCS3OmRXxpZo1Ib37DfjoPx\nHZzhSEuBO/yR/PLBuIYyfPEMtQGKsqYpqyqwcKaLOcHo0Uj18E9Oe412TgImOdtOcVYBnzivlSTM\nmB+JaQwWWyLdaP87iNdQhikqDMqwxhGCn1Z/P2BG3cBFSJD55T6cOgFMTJgx70BNVVuN7Z3tR4Nw\nbmWYo+mqitI5v0Keyq9viBT2xd/6HbIKSatnUh+YBrQiDf8UZUBRYVCU1QiYUReSyfMFdNssrzsS\nztY/IEb9nGnAZ4O0GlGGOepKUpSf057dc3IPzfK6o31ozsYDY9LPmIakyCrKgKMrBkXpgLNa+APS\n3O6f/TjVoK0YEmZsAtIwT+MLyqCgwqAoq3I48jT+p4ZIYZ8bzPkj+UuQpnaDsWJon4mtGUnKoKDC\noCgOATNqIKuFr4AHB+CUCQYnxqDCoAwqGmNQlP9RhFRQn9af1UIHvmPwhGGOP5LfnyZ8itIlumJQ\nFFauFq5EKp0fGKDTfod0NB1otkfjC8ogosKgKMKhyMjMaxoiha0DdM5vkSK3rAE6Hwkzlom0KFc3\nkjJoqDAow54Oq4UGZCbEQDHb2Q5kAHoq4gJWYVAGDRUGRZFMpF0Y2NUCiNAABAbwnBp4VgYdFQZl\nWOP0RLoWqXK+b4BP375imDKA55wGNCJ1FooyKGhWkjLcORFxzxwzQJlIHZmDDM4ZSGHYHvjYH8nv\na0W2ovSIrhiUYUvAjGYCf0SmvfW1J1KX+CP5bUgtQ2AgzpcwYwayYlA3kjKo6IpBGc6ciTzNn97H\nDqrpMJuBWzH4kelsKgzKoKIrBmVYEjCjo4DLkYltLw3ipRoYuOCzBp6VNYIKgzJcMYENAHMQVwsg\nK4bJCTOWMQDnah/O8/EAnEtRukSFQRl2BMzoFOBC4IGGSOGMQb7cbOT3bPIAnGsa8JU/kr90AM6l\nKF2iwqAMR64DbODSNXCtBmcbGIBz7YC2wlDWACoMyrAiYEZ3B/4PuKkhUvhdT/sPAANSy5AwYz5g\nc2QGtaIMKioMyrDBaX3xZ2A+cP0auuy3gAVs1s/ztAeeP+zneRSlR1QYlOHEScDuwGUNkcJla+KC\nzkzmb5Gn/f6wo7NVYVAGHRUGZVgQMKOjgRuBdxj41hc98SWwZT/PsQMw3x/Jnz8A9ihKt6gwKMOF\nq4HxwNkNkUJrDV/7S2CLfp5jBzS+oKwhVBiU9Z6AGd0ROBv4a0OkcChurrOAsQkzNrYvBzvzHKai\nbiRlDaHCoKzXBMyoC6gEFiKVzkNBeyfUvsYZQoAbFQZlDaHCoKzvnIUEnC9qiBQuHiIb2oWhr+6k\nHZytCoOyRlBhUNZbnArnCPAC8I8hNOVrpKCurwHoHYAkMo9aUQYdFQZlvcSpWbgTMIAzB7kfUrf4\nI/lNSPvt/qwYPvRH8ofsOyjDCxUGZX2lBJgOXNIQKWwYYlugj5lJCTPmQYrb1I2krDFUGJT1joAZ\n3S+9zi0AAAamSURBVAipcI4Bfx1ic9qZRd9cSVsDWagwKGsQFQZlvcLJQroPyAR+MwQ1C13xOTA+\nYcbG9/I4DTwraxwVBmV94zzgQOCChkjhF0NtTAc+cbbb9vK4HYAmYObAmqMoXaPCoKw3BMzodkhz\nvKeRwPPaRF+FYSfgY2d+tKKsEVQYlPWCgBnNBh4CFiEupLUtg2ceYlvawuAEnndB+jspyhpDhUFZ\nX7gN2AY4tSFSuGCojVkdJ9X0E6SKOV1CQA7w1qAYpShdoMKgrPMEzOivgV8D1zZECp8banu64RNg\n24QZM9Lcf09nq8KgrFFUGJR1moAZ3QnphfQS8IchNqcn4sAowJ/m/nsgLqjZPe2oKAOJCoOyzhIw\no+OAfwE/AMc3RApTQ2xST7QHoNN1J+0BvKUVz8qaRoVBWScJmFEv8BiwEXDM2hhX6IS0hSFhxiYi\n40DVjaSscVQYlHWODn2Q9gVOa4gUrhNZO/5I/mLgK2C3NHbf29m+OXgWKUrnqDAo6yKXIr2Qwg2R\nwgeH2phe8gawVxoB6AOBpcCMwTdJUVZFhUFZpwiY0ROAPwEPAFcNsTl94Q1gAj0P7TkQeNUfyW8d\nfJMUZVVUGJR1hoAZPQKoBl5h7SxiS4c3nO3eXe2QMGObI/GFF9aIRYqyGioMyjpBwIweADwCvAcc\n0RApbB5ik/pKPbAY2KubfQ50ti8OvjmK8nNUGJS1noAZ3Rt4Cmkkd2hDpHDpEJvUZ/yRfAsJKPck\nDN8Ba1MTQGUYocKgrNU4K4XnkRvl9IZI4aIhNmkgiAHBhBmbvPoHCTM2CjgEiGr9gjJUqDAoay0B\nM3o4EEWmn+3bECn8fohNGigec7bHdfLZ0UA2cP+aM0dRVkWFQVkrCZjRE4HHgf8C+61HooA/kj8L\nSUM9oZOPT0amvb29Ro1SlA6oMChrFQEzagTMaBj4B+JyOWA9cR+tzkPADgkzFmx/I2HGNkWK9qrV\njaQMJSoMylpDwIxmIfUJVyJpqQc1RAqXDK1Vg8bDgAWcA+AUvN0EtKBuJGWIMWxbH0yUoSdgRqcg\n6ai7ApcB162jdQppkzBjtwNlQDGQC9wDmP5I/vVDapgy7PEMtQGKEjCjhyKuIw9wVEOk8PEhNmlN\nUY4IYY3z/zOAiqEzR1EEXTEoQ0bAjGYibS0uRoLMRzdECr8cWqvWLAkzNgk4EZm7UOuP5CeH2CRF\nUWFQhoaAGd0R8aVvg3RK/W1DpLBxaK1SFAXUlaSsYQJmNBvpjmoCC5BK5meH1ipFUTqiwqCsMQJm\n9DDgL0AAyT46fz1NRVWUdRoVBmXQCZjR7YAI0urhM6Rg7dUhNUpRlC5RYVAGjYAZ3Qz4I1LhmwQu\nAm5tiBTqjAFFWYvR4LPy/+3dvWoUURxA8RMWiaIiaEAwFgMaBQnYamNhIeJUIoIPYOEjbGM/FhaC\nIGl8AUvZysJG1FJsAsHIFBotjAG/cE0iFneWzH/Jp8lmk+z5wWVmd0O4STGHzIR7t1zWbI2TniHc\nAuaBB8C9ssjn+joxSetiGLQlsmarAVwD7lTHn8AEcL8s8pl+zk3SxhgGbUrWbI0Ct6txEvgMPAIe\n+mBZ2p0MgzYsa7ZGgOvATeAy0CBtQzkBPPUZgrS7GQatS7WW0VXgBksxmCYt5/C4LPLpPk5P0hYy\nDFpW1mwdAS4BV6pxpvroHfCkGm/2+kJ30iAyDOo8OD4LXAAuVuMcMAT8Ap6TNqZ/BkwaA2lvMwwD\nJGu2hoATwBgwDpyvxjhpO0mAOdLuYS+BF8Crssjb2z9bSf1iGPaQ6sJ/lPTfQaPVOEUKwRhwmqUA\nAMySVjV9Wx1fA1Nlkf/dxmlL2mEMww5WXegPAMe6xkjt/DhLERgFhru+zTzwnrSPcH1MAjPeFpLU\nzTCsobo4N0jLh9SPnfN9wH7SBXl4A+cHgcO1cajrdee9xirT+0ZaofRjbXzoej1TFvnCZn8PkgbH\nQIUha7buktbtqV/YVzs26N2+2G3ge9f4scLr2Wp8qZ1/LYv8T4/mJmmADdoiep9I99IXgYUVjqt9\nttzXzAO/SRf6zrG9xnvtssgXe/3DStL/GKi/GCRJa+vVbRJJ0i5lGCRJgWGQJAWGQZIUGAZJUmAY\nJEmBYZAkBYZBkhQYBklSYBgkSYFhkCQFhkGSFBgGSVJgGCRJgWGQJAWGQZIUGAZJUmAYJEmBYZAk\nBYZBkhQYBklSYBgkSYFhkCQFhkGSFBgGSVJgGCRJgWGQJAWGQZIUGAZJUmAYJEmBYZAkBYZBkhT8\nA5xYcwwLmTYyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x105ddbe48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from __future__ import absolute_import\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "'''\n",
    "Mathematically we can only take gradients of scalar-valued functions, but\n",
    "autograd's grad function also handles numpy's familiar vectorization of scalar\n",
    "functions, which is used in this example.\n",
    "To be precise, grad(fun)(x) always returns the value of a vector-Jacobian\n",
    "product, where the Jacobian of fun is evaluated at x and the the vector is an\n",
    "all-ones vector with the same size as the output of fun. When vectorizing a\n",
    "scalar-valued function over many arguments, the Jacobian of the overall\n",
    "vector-to-vector mapping is diagonal, and so this vector-Jacobian product simply\n",
    "returns the diagonal elements of the Jacobian, which is the gradient of the\n",
    "function at each input value over which the function is vectorized.\n",
    "'''\n",
    "\n",
    "def tanh(x):\n",
    "    return (1.0 - np.exp(-x))  / (1.0 + np.exp(-x))\n",
    "\n",
    "x = np.linspace(-7, 7, 200)\n",
    "plt.plot(x, tanh(x),\n",
    "         x, grad(tanh)(x),                                 # first derivative\n",
    "         x, grad(grad(tanh))(x),                           # second derivative\n",
    "         x, grad(grad(grad(tanh)))(x),                     # third derivative\n",
    "         x, grad(grad(grad(grad(tanh))))(x),               # fourth derivative\n",
    "         x, grad(grad(grad(grad(grad(tanh)))))(x),         # fifth derivative\n",
    "         x, grad(grad(grad(grad(grad(grad(tanh))))))(x))   # sixth derivative\n",
    "\n",
    "plt.axis('off')\n",
    "plt.savefig(\"tanh.png\")\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py36]",
   "language": "python",
   "name": "conda-env-py36-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}