nonlinear-optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b425e6f4-1b5f-4556-a1fd-ae1c1c30e843",
   "metadata": {},
   "source": [
    "# Assignment 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e67077df-27fa-4a88-b662-d82a3322efb4",
   "metadata": {},
   "source": [
    "The first step is loading some python packages which are going to be used during the whole notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "89db17d2-c4a0-4640-bfe5-e1ab2c82fbd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "import random\n",
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ecb33e-c5e4-4f13-a7bf-3ec6036876e5",
   "metadata": {},
   "source": [
    "The aim is adjusting a multiple linear regression model to explain a variable $y$ as a function of other variables $X$, $y = X \\beta + \\epsilon$, where $\\beta$ are the adjusted regression coefficients and $\\epsilon$ the regression error. For this purpose, the **Ridge Regression** is used, where the objective is to minimize the function:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\underset{\\beta}{\\min} \\quad \\lVert y - X \\beta \\rVert_2^2 + \\rho \\lVert \\beta\\rVert_2^2 \n",
    "\\end{align*}\n",
    "\n",
    "In order to clarify the elements and dimensions of the problem, if there are $n$ **samples** or instances and $k$ **variables**:\n",
    "\n",
    "* Explained variable: $\\underset{(nx1)}{y}$\n",
    "* Variables used to explain $y$: $\\underset{(nx(k+1))}{X}$\n",
    "* Regression coefficients: $\\underset{((k+1)x1)}{\\beta}$\n",
    "* Regression error: $\\underset{(nx1)}{\\epsilon}$\n",
    "\n",
    "\n",
    "Now the data is generated randomly. Remark that variable `nsample` corresponds with $n$ and `nvars` with $k$. The values of $\\beta$ are between -5 and 5, and the ones of $X$ from 0 to 10. Mention that $\\rho$ is a parameter which is fixed during the whole problem to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "49c3b095-4032-44c4-a074-d706b9b4ddee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The problem has n=1000 samples with k=100 variables\n",
      "\n",
      "Dimensions y:\t\t(1000, 1)\n",
      "Dimensions X:\t\t(1000, 101)\n",
      "Dimensions beta:\t(101, 1)\n",
      "Dimensions epsilon:\t(1000, 1)\n"
     ]
    }
   ],
   "source": [
    "# Dimensions of the problem\n",
    "nsample = 1000 # = n\n",
    "nvars = 100    # = k\n",
    "\n",
    "# Original values of beta\n",
    "beta = np.random.randint(-5, 6, size=([nvars+1,1])) \n",
    "\n",
    "# Values for the X's\n",
    "X0 = np.ones([nsample, 1]) # column of ones for beta_0\n",
    "X1 = np.random.uniform(0, 10, ([nsample, nvars]))\n",
    "X = np.concatenate([X0, X1], axis = 1)\n",
    "\n",
    "# Values for the normal errors, they follow a gaussian distribution\n",
    "error = np.random.normal(0, 1,(nsample,1))\n",
    "\n",
    "# Values for the y's\n",
    "y = np.dot(X, beta) + error\n",
    "\n",
    "# Value for rho\n",
    "rho = 1\n",
    "\n",
    "# Print dimesions\n",
    "print(f'The problem has n={nsample} samples with k={nvars} variables\\n')\n",
    "print(f'Dimensions y:\\t\\t{y.shape}')\n",
    "print(f'Dimensions X:\\t\\t{X.shape}')\n",
    "print(f'Dimensions beta:\\t{beta.shape}')\n",
    "print(f'Dimensions epsilon:\\t{error.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6aa321d-157e-4d85-bd6f-d11c71862254",
   "metadata": {},
   "source": [
    "## A. Analytical solution \n",
    "\n",
    "The value of the regression coefficients can be obtained by an analytical solution:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\beta^{\\ast} = (X^TX + \\rho I)^{-1}X^Ty\n",
    "\\end{align*}\n",
    "\n",
    "These ones are the optimal values. For this reason, solutions given by iterative methods can be compared with this one in order to see their performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e18f3219-315c-429d-8e31-4b8bab4d9d7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "beta_exact = np.linalg.inv(X.T.dot(X) + rho * np.eye(X.shape[1])).dot(X.T).dot(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "165496a6-0cae-4e11-931e-7f4d5a68ae8b",
   "metadata": {},
   "source": [
    "With the following function a specific number of $\\beta$ values chosen randomly are printed together with the optimal value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "abb4d172-0f9d-401b-9532-3c7252057af0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 75         | 1.98095         | 1.98095         |\n",
      "--------------------------------------------------\n",
      "| 85         | 1.98991         | 1.98991         |\n",
      "--------------------------------------------------\n",
      "| 60         | -3.9954         | -3.9954         |\n",
      "--------------------------------------------------\n",
      "| 79         | -2.98967        | -2.98967        |\n",
      "--------------------------------------------------\n",
      "| 61         | 4.99514         | 4.99514         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "def print_beta(beta, ks=5):\n",
    "    '''Print ks values of given beta and beta_exact\n",
    "    \n",
    "    Args:\n",
    "        beta (numpy array): obtained beta to compare\n",
    "        ks (int): number of samples to print\n",
    "        \n",
    "    '''\n",
    "    if ks > nvars+1:\n",
    "        print(f'Choose ks smaller than {nvars+1}')\n",
    "        return\n",
    "    if len(beta.shape) == 1:\n",
    "        beta = np.matrix(beta).T\n",
    "    elif beta.shape[0] == 1:\n",
    "        beta = beta.T\n",
    "    index = random.sample(range(nvars+1), ks)\n",
    "    beta_s = beta[index, 0]\n",
    "    beta_opt_s = beta_exact[index]\n",
    "    print('-'*50)\n",
    "    print(f\"| {'Index':<10} | {'Beta':<15} | {'Optimal beta':<15} |\")\n",
    "    print('-'*50)\n",
    "    i = 0\n",
    "    for idx in index:\n",
    "        print(f\"| {idx:<10} | {round(beta[idx, 0], 5):<15} | {round(beta_exact[idx, 0], 5):<15} |\")\n",
    "        i += 1\n",
    "        print('-'*50)\n",
    "        \n",
    "print_beta(beta_exact)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64f7a025-89a7-49db-a5b7-a3039006f5a0",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "source": [
    "## B. Scipy\n",
    "\n",
    "In order to solve this section and next ones, the gradient and hessian of the Ridge regression are calculated. The expression can be formulated as:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\lVert y - X \\beta \\rVert_2^2 + \\rho \\lVert \\beta\\rVert_2^2 = \\\\\n",
    "    \\sum_{i=i}^n (y_i - (\\beta_0 + \\beta_1 x_{1i} + \\dots + \\beta_{ki}))^2 + \\rho \\sum_{j=1}^k \\beta_j^2 = f(\\beta^*)\n",
    "\\end{align*}\n",
    "\n",
    "The gradient corresponds with a vector of dimensions $((k+1)x1)$, where each position $j$ is the partial derivative of the function respect one regression coefficient $\\beta_j$.\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla f(\\left.\\beta^{*} \\right)=\\left[\\begin{array}{c}\n",
    "    \\dfrac{\\partial f}{\\partial \\beta_0}(\\left.\\beta^{*}\\right) \\\\\n",
    "    \\dfrac{\\partial f}{\\partial \\beta_1}(\\left.\\beta^{*}\\right) \\\\\n",
    "    \\vdots \\\\\n",
    "    \\dfrac{\\partial f}{\\partial \\beta_k}(\\left.\\beta^{*}\\right)\n",
    "    \\end{array}\\right]\n",
    "\\end{align*}\n",
    "\n",
    "Each derivative can be formulated as:\n",
    "\n",
    "\\begin{align*}\n",
    "     \\dfrac{\\partial f}{\\partial \\beta_j}(\\left.\\beta^{*}\\right) = \\sum_{i=1}^n 2(y_i - (\\beta_0 + \\beta_1 x_{1i} + \\dots + \\beta_k x_{ki} + \\dots + \\beta_j x_{ji})) (-x_{ji}) + 2 \\rho \\beta_j\n",
    "\\end{align*}\n",
    "\n",
    "Applying vector operations, the **gradient vector** $\\underset{((k+1)x1)}{\\nabla f(\\left.\\beta^{*} \\right)}$ is equal to:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla f(\\left.\\beta^{*} \\right) = -2(y - X\\beta)^T X + 2\\rho \\beta^T\n",
    "\\end{align*}\n",
    "\n",
    "Now, in order to calculate the Hessian matrix, it is required to differentiate the previous expression. In order to illustrate the process, each partial derivative is:\n",
    "\n",
    "\\begin{align*}\n",
    "     \\dfrac{\\partial f^2}{\\partial \\beta_j}(\\left.\\beta^{*}\\right) = \\sum_{i=1}^n 2 x_{ji}^2 + 2 \\rho\n",
    "\\end{align*}\n",
    "\n",
    "Then, the **hessian matrix** $\\underset{((k+1)x(k+1))}{\\nabla^2 f(\\left.\\beta^{*} \\right)}$ is equal to:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla^2 f(\\left.\\beta^{*} \\right) = 2 X^T X + 2\\rho I\n",
    "\\end{align*}\n",
    "\n",
    "Then, these two expression as well as the Ridge regression are written in Python in next cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c92c898d-3c3a-405d-b41f-0d75c448bee3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ridge regression objective\n",
    "def ridge_reg(beta_r, X, y):\n",
    "    '''Implement Ridge Regression\n",
    "    \n",
    "    '''\n",
    "    beta_r = np.matrix(beta_r)\n",
    "    z = y - np.dot(X, beta_r.T)\n",
    "    return np.dot(z.T,z) + rho*np.sum(np.square(beta_r))\n",
    "\n",
    "# Gradient\n",
    "def ridge_reg_der(beta_r, X, y):\n",
    "    '''Implement gradient Ridge Regression\n",
    "    '''\n",
    "    beta_r = np.matrix(beta_r)\n",
    "    pp = -2*np.dot((y-np.dot(X,(beta_r).T)).T,X) + 2*rho*beta_r\n",
    "    aa = np.squeeze(np.asarray(pp))\n",
    "    return aa\n",
    "\n",
    "# Hessian\n",
    "def ridge_reg_hess(beta_r, X, y):\n",
    "    '''Implement Hessian Ridge Regression\n",
    "    \n",
    "    '''\n",
    "    ss = 2*np.dot(np.transpose(X),X) + 2*rho*np.identity(X.shape[1])\n",
    "    return ss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "608a6dbd-7ce6-4d8b-9f3f-4934b8a5e522",
   "metadata": {},
   "source": [
    "The following function is used to print statistics of each solver used. It shows:\n",
    "\n",
    "* **Error**, compared with the values of beta obtained by the analytical solution.\n",
    "* Computational **time** in seconds.\n",
    "* Number of **iterations**.\n",
    "* Number of **function evaluations**.\n",
    "* Number of **gradient evaluations** (if the gradient is used).\n",
    "* Number of **hessian evaluations** (if the hessian is used)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "33cee55a-069e-4c65-837a-2f509f17b47c",
   "metadata": {},
   "outputs": [],
   "source": [
    "solvers_dic = {}\n",
    "\n",
    "def show_solver_stats(methods='all'):\n",
    "    '''Prints statistics of the used scipy solvers\n",
    "    \n",
    "    '''\n",
    "    print('-'*127)\n",
    "    print(f\"| {'Method':<15} | {'Error':<10} | {'Iterations':<10} | {'Function evaluations':<20} | {'Gradient evaluations':<20} | {'Hessian evaluations':<20} | {'Time (sec)':<10} |\")\n",
    "    print('-'*127)\n",
    "    if methods == 'all':\n",
    "        methods = list(solvers_dic.keys())\n",
    "    elif isinstance(methods, str):\n",
    "        methods = [methods]\n",
    "    for method in methods:\n",
    "        print(f\"| {method:<15} | {solvers_dic[method]['err']:<10} | {solvers_dic[method]['nit']:<10} | {solvers_dic[method]['nfev']:<20} | {solvers_dic[method]['njev']:<20} | {solvers_dic[method]['nhev']:<20} | {solvers_dic[method]['time']:<10} |\")\n",
    "        print('-'*127)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c981a05a-e621-4c78-a978-d208750361cc",
   "metadata": {},
   "source": [
    "It is employed `scipy.optimize.minimize` with four different solvers. Next function runs the algorithm, given the solver as `method` and with optional parameters to input the gradient and hessian functions as well as other parameters for the scipy solver. The parameter `ks` can be used to print the solved $\\beta^*$ and compare them with the analytical solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "73b706b8-1798-4cf0-ae05-7966497d1eae",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.optimize import minimize\n",
    "\n",
    "def run_solver(method, ks=5, jac=None, hess=None, options=None):\n",
    "    '''Run given method using scipy.optimie.minimize\n",
    "    \n",
    "    '''\n",
    "    # Init dic specific method\n",
    "    solvers_dic[method] = {}\n",
    "    \n",
    "    # Initial beta\n",
    "    beta_0 = np.zeros(nvars+1)\n",
    "\n",
    "    # Start time\n",
    "    time_start = time.process_time()\n",
    "\n",
    "    # Solve the optimization algorithm using Newton-CG\n",
    "    res = minimize(ridge_reg, beta_0, args=(X, y), jac=jac, hess=hess, method=method, options=options) \n",
    "\n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "\n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(beta_exact.T-res.x,ord=2)/np.linalg.norm(beta_exact.T,ord=2)\n",
    "    \n",
    "    # Gradient evaluations\n",
    "    if jac is None:\n",
    "        njev = 'Not used'\n",
    "    else:\n",
    "        njev = res.njev\n",
    "        \n",
    "    # Hessian evaluations\n",
    "    if hess is None:\n",
    "        nhev = 'Not used'\n",
    "    else:\n",
    "        nhev = res.nhev\n",
    "\n",
    "    # Save stats\n",
    "    solvers_dic[method]['err'] = round(err, 5)               # error\n",
    "    solvers_dic[method]['nit'] = res.nit                     # number of iterations\n",
    "    solvers_dic[method]['nfev'] = res.nfev                   # number of function evaluations\n",
    "    solvers_dic[method]['njev'] = njev                       # number of gradient evaluations\n",
    "    solvers_dic[method]['nhev'] = nhev                       # number of hessian evaluations\n",
    "    solvers_dic[method]['time'] = round(time_elapsed, 5)     # computational time\n",
    "\n",
    "    # Show stats\n",
    "    show_solver_stats(method)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(res.x, ks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bce46cb5-3e26-4ef3-a4b1-cc91657ebee6",
   "metadata": {},
   "source": [
    "### B.1. Newton-CG\n",
    "\n",
    "It employs the gradient and the hessian."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7414695e-e331-4bc5-8785-6907a15ba989",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Method          | Error      | Iterations | Function evaluations | Gradient evaluations | Hessian evaluations  | Time (sec) |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Newton-CG       | 0.0        | 22         | 25                   | 25                   | 22                   | 0.26807    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 29         | 2.99209         | 2.99209         |\n",
      "--------------------------------------------------\n",
      "| 24         | -1.9988         | -1.9988         |\n",
      "--------------------------------------------------\n",
      "| 23         | 3.99253         | 3.99253         |\n",
      "--------------------------------------------------\n",
      "| 53         | 2.98933         | 2.98933         |\n",
      "--------------------------------------------------\n",
      "| 46         | 4.98537         | 4.98537         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "run_solver('Newton-CG', jac=ridge_reg_der, hess=ridge_reg_hess, options={'xtol': 1e-10})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65ae4131-aeb7-4f46-a5a7-f966d405bede",
   "metadata": {},
   "source": [
    "### B.2. Nelder-Mead\n",
    "\n",
    "The gradient and hessian are not used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "80b98d85-f5a2-4353-b9ed-df0803142c36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Method          | Error      | Iterations | Function evaluations | Gradient evaluations | Hessian evaluations  | Time (sec) |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Nelder-Mead     | 1.27506    | 19324      | 20200                | Not used             | Not used             | 38.33262   |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 24         | 0.51097         | -1.9988         |\n",
      "--------------------------------------------------\n",
      "| 46         | 0.6452          | 4.98537         |\n",
      "--------------------------------------------------\n",
      "| 31         | 1.98414         | 3.9958          |\n",
      "--------------------------------------------------\n",
      "| 19         | -1.67058        | -0.99718        |\n",
      "--------------------------------------------------\n",
      "| 42         | -0.23087        | -5.02055        |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "run_solver('Nelder-Mead', options={'xtol': 1e-10})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "520aa21e-e77f-43e4-a4af-50b8a29fb9b9",
   "metadata": {},
   "source": [
    "### B.3. CG\n",
    "\n",
    "It is only used the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a9447f55-b6b3-4a56-9595-6a41750fc841",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Method          | Error      | Iterations | Function evaluations | Gradient evaluations | Hessian evaluations  | Time (sec) |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| CG              | 1e-05      | 219        | 554                  | 543                  | Not used             | 1.47735    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 96         | 1.9994          | 1.9994          |\n",
      "--------------------------------------------------\n",
      "| 47         | -3.00586        | -3.00587        |\n",
      "--------------------------------------------------\n",
      "| 90         | -5.00043        | -5.00043        |\n",
      "--------------------------------------------------\n",
      "| 81         | -4.00837        | -4.00837        |\n",
      "--------------------------------------------------\n",
      "| 72         | 2.99095         | 2.99095         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "run_solver('CG', jac=ridge_reg_der)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c693c68-8f07-46f5-8692-2aa4f09c690b",
   "metadata": {},
   "source": [
    "### B.4. BFGS\n",
    "\n",
    "Only employs the gradient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4e1b6ce5-38ab-4cf1-bc64-1a296a02ea01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Method          | Error      | Iterations | Function evaluations | Gradient evaluations | Hessian evaluations  | Time (sec) |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| BFGS            | 0.0        | 119        | 221                  | 221                  | Not used             | 0.99504    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 11         | -0.02688        | -0.02688        |\n",
      "--------------------------------------------------\n",
      "| 14         | 0.99094         | 0.99094         |\n",
      "--------------------------------------------------\n",
      "| 53         | 2.98933         | 2.98933         |\n",
      "--------------------------------------------------\n",
      "| 26         | 3.02327         | 3.02327         |\n",
      "--------------------------------------------------\n",
      "| 38         | 2.9948          | 2.9948          |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "run_solver('BFGS', jac=ridge_reg_der)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a11825dd-5805-4e11-898f-2980d92bbb1d",
   "metadata": {},
   "source": [
    "The results from the solvers can be easily printed with functon `show_solver_stats()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4e828784-0663-4790-8e34-3e799583f39b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Method          | Error      | Iterations | Function evaluations | Gradient evaluations | Hessian evaluations  | Time (sec) |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Newton-CG       | 0.0        | 22         | 25                   | 25                   | 22                   | 0.26807    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| Nelder-Mead     | 1.27506    | 19324      | 20200                | Not used             | Not used             | 38.33262   |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| CG              | 1e-05      | 219        | 554                  | 543                  | Not used             | 1.47735    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n",
      "| BFGS            | 0.0        | 119        | 221                  | 221                  | Not used             | 0.99504    |\n",
      "-------------------------------------------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "show_solver_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7f0d423-02f7-442d-bdd2-e3fd306e5434",
   "metadata": {},
   "source": [
    "The best method is **Newton-CG**, being the fastest and obtaining zero error. In addition, the number of function and gradient evaluations is much less than in **CG** and **BFGS**. The worse is **Nelder-Mead**, which is much slower than the rest and provides a larger error. Mention in this method the gradient and the hessian are not used, whereas in the best one both are employed. Consequently, it can be seen the benefitial effects in convergence if the gradient or the hessian are implemented."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab66362d-4c3c-4ffb-89e4-4ce97426f061",
   "metadata": {},
   "source": [
    "## C. Algorithm implementations (I)\n",
    "\n",
    "Different iterative algorithms are implemented in this section. Next function is employed to print the results, it shows:\n",
    "\n",
    "* **Error**, compared with the values of beta obtained by the analytical solution.\n",
    "* Reached **tolerance**: calculated with the norm of the gradient.\n",
    "* Computational **time** in seconds.\n",
    "* Number of **iterations**.\n",
    "* **Time per iteration**, in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "20e67f83-a615-4e00-8f97-0909a9a4bedc",
   "metadata": {},
   "outputs": [],
   "source": [
    "alg_dic = {}\n",
    "\n",
    "def show_alg_stats(methods='all'):\n",
    "    '''Print results of each algorithm\n",
    "    \n",
    "    '''\n",
    "    print('-'*90)\n",
    "    print(f\"| {'Algorithm':<19} | {'Error':<10} | {'Tolerance':<12} | {'Iterations':<10} | {'Time (sec)':<10} | {'Sec/iter':<10} |\")\n",
    "    print('-'*90)\n",
    "    if methods == 'all':\n",
    "        methods = list(alg_dic.keys())\n",
    "    elif isinstance(methods, str):\n",
    "        methods = [methods]\n",
    "    for method in methods:\n",
    "        print(f\"| {method:<19} | {alg_dic[method]['err']:<10} | {alg_dic[method]['tol']:<12} | {alg_dic[method]['nit']:<10} | {alg_dic[method]['time']:<10} | {alg_dic[method]['sec']:<10} |\")\n",
    "        print('-'*90)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "788c7f8d-79ef-4022-8741-7e5d91eb31be",
   "metadata": {},
   "source": [
    "### C.1. Gradient method\n",
    "\n",
    "It is not the best method in terms of convergence, but its computational cost is not very high.\n",
    "\n",
    "* From an initial iterate $\\beta_0$\n",
    "\n",
    "* Compute search (descent) directions $p_k=-\\nabla f(\\beta_k)$\n",
    "\n",
    "* Compute a steplength $\\alpha_k>0$ using Armijo rule.\n",
    "\n",
    "* Movement: $\\beta_{k+1} = \\beta_k + \\alpha_k\\ p_k$\n",
    "\n",
    "Until convergence to a local solution. The limit is given by a total number of iteratons `n_iter` or if a tolerance limit `epsilon` is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "ccdac15d-9185-4bec-9ecc-0a3a7e670232",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_method(n_iter=2000, epsilon=1e-4, sigma=1e-5, delta=0.01, alpha_init=1e-5, ks=5):\n",
    "    '''Implement Gradient method\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Gradient'\n",
    "    alg_dic[alg] = {}\n",
    "    \n",
    "    (a,b) = X.shape\n",
    "    \n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b) \n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Newton method\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        # compute gradient\n",
    "        grad = ridge_reg_der(beta,X,y)\n",
    "        # compute descent direction\n",
    "        ddirect = -grad \n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta = beta + alpha*ddirect\n",
    "    \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # if the norm of the gradient is small, we can stop\n",
    "        tol = np.linalg.norm(grad,ord=2)\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "a6abdeb8-e1ea-4554-8545-a49acac261f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Gradient            | 0.09287    | 43.065986502 | 1999       | 10.77377   | 0.00539    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 41         | -5.02018        | -5.01597        |\n",
      "--------------------------------------------------\n",
      "| 27         | -3.02714        | -3.01823        |\n",
      "--------------------------------------------------\n",
      "| 44         | 3.98383         | 3.99035         |\n",
      "--------------------------------------------------\n",
      "| 59         | 5.00098         | 5.00409         |\n",
      "--------------------------------------------------\n",
      "| 14         | 0.98653         | 0.99094         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "n_iter = 2000      # maximum number of iterations\n",
    "epsilon = 1e-8     # tolerance limit\n",
    "sigma = 1e-5       # Armijo rule parameter\n",
    "delta = 1e-2       # Armijo rule parameter\n",
    "alpha_init = 1e-5  # initial alpha for Armijo rule\n",
    "\n",
    "# Run algorithm\n",
    "gradient_method(n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71ce96e0-ab26-4475-915b-0e177b586c96",
   "metadata": {},
   "source": [
    "### C.2. Newton method\n",
    "\n",
    "Better convergence than gradient method, but its computational cost is very high.\n",
    "\n",
    "* From an initial iterate $\\beta_0$\n",
    "\n",
    "* Compute search (descent) directions $p_k=-(\\nabla^2 f(\\beta_k))^{-1} \\nabla f(\\beta_k)$, whenever $\\nabla^2 f(\\beta_k)$ is nonsingular. Computing Hessian.\n",
    "\n",
    "* Compute a steplength $\\alpha_k>0$ using Armijo rule.\n",
    "\n",
    "* Movement: $\\beta_{k+1} = \\beta_k + \\alpha_k\\ p_k$\n",
    "\n",
    "Until convergence to a local solution. The limit is given by a total number of iteratons `n_iter` or if a tolerance limit `epsilon` is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "01074bf1-8002-4014-8d8d-ed8b80698740",
   "metadata": {},
   "outputs": [],
   "source": [
    "def newton_method(n_iter=1000, epsilon=1e-2, sigma=1e-8, delta=1e-8, alpha_init=0.01, ks=5):\n",
    "    '''Implement Newton method\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Newton'\n",
    "    alg_dic[alg] = {}\n",
    "\n",
    "    (a,b) = X.shape\n",
    "\n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b)\n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Newton method\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        # compute gradient\n",
    "        grad = ridge_reg_der(beta,X,y)\n",
    "        # compute hessian\n",
    "        hess = ridge_reg_hess(beta,X,y)\n",
    "        # compute descent direction\n",
    "        ddirect = -np.dot(np.linalg.inv(hess),grad)\n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta = beta + alpha*ddirect\n",
    "        \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # if the norm of the gradient is small, we can stop\n",
    "        tol = np.linalg.norm(grad,ord=2)\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "75bd5717-c208-481b-95f0-18e16b7f081d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Newton              | 0.0        | 2.813e-06    | 499        | 27.89775   | 0.05591    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 28         | 3.9929          | 3.9929          |\n",
      "--------------------------------------------------\n",
      "| 84         | 0.00643         | 0.00643         |\n",
      "--------------------------------------------------\n",
      "| 20         | 0.01171         | 0.01171         |\n",
      "--------------------------------------------------\n",
      "| 21         | 2.99243         | 2.99243         |\n",
      "--------------------------------------------------\n",
      "| 22         | -2.01631        | -2.01631        |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "n_iter = 500       # maximum number of iterations\n",
    "epsilon = 1e-8     # tolerance limit\n",
    "sigma = 1e-3       # Armijo rule parameter\n",
    "delta = 5e-1       # Armijo rule parameter\n",
    "alpha_init = 1     # initial alpha for Armijo rule\n",
    "\n",
    "# Run algorithm\n",
    "newton_method(n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a21b4af-ca8b-4ed6-af8a-e73f64290997",
   "metadata": {},
   "source": [
    "### C.3. Quasi-Newton method\n",
    "\n",
    "The idea is having a smaller computational cost than Newton and better convergence than gradient. This method is equivalent to Newton, but the Hessian is approximated by another function instead of being exactly calculated, saving a lot of time. It starts with a $B_0$ which corresponds with the initial approximation to $\\nabla^2 f(\\beta^*)$ and then iterates:\n",
    "\n",
    "\\begin{align*}\n",
    "    B_{k+1} = B_k + \\text{update rule}\n",
    "\\end{align*}\n",
    "\n",
    "Different methods can be employed. The **BFGS** is used here:\n",
    "\n",
    "\n",
    "\\begin{align*}\n",
    "    B_{k+1} = B_k + \\frac{y_ky_k^T}{y_k^Ts_k} - \\frac{B_k s_k (B_k s_k)^T}{s_k^T B_k s_k}\n",
    "\\end{align*}\n",
    "\n",
    "The inverse can be analytically calculated with:\n",
    "\n",
    "\\begin{align*}\n",
    "    H_{k+1} = B_{k+1}^{-1} = \\bigg( I - \\frac{s_k y_k^T}{y_k^T s_k} \\bigg) H_k \\bigg( I - \\frac{y_k s_k^T}{y_k^T s_k} \\bigg) + \\frac{s_k s_k^T}{y_k^T s_k}\n",
    "\\end{align*}\n",
    "\n",
    "where $s_k = \\beta_{k+1} - \\beta_k$ and $y_k = \\nabla f(\\beta_{k+1}) - \\nabla f(\\beta_k)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6497e210-9233-4d8f-8027-b80818f255ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_approx_inv_hess(inv_hess, points_diff, gradient_diff):\n",
    "    '''Calculate inverse approximation of the Hessian\n",
    "    \n",
    "    '''\n",
    "    sk = np.matrix(points_diff).T\n",
    "    yk = np.matrix(gradient_diff).T\n",
    "    I = np.identity(X.shape[1])\n",
    "    n1 = np.dot(sk, yk.T)\n",
    "    d1 = np.dot(yk.T, sk)[0, 0]\n",
    "    n2 = np.dot(yk, sk.T)\n",
    "    d2 = np.dot(yk.T, sk)[0, 0]\n",
    "    n3 = np.dot(sk, sk.T)\n",
    "    d3 = np.dot(yk.T, sk)[0, 0]\n",
    "    m1 = I - n1/d1\n",
    "    m2 = I - n2/d2\n",
    "    out = np.linalg.multi_dot([m1, inv_hess, m2]) + n3/d3\n",
    "    return np.squeeze(np.asarray(out))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "73b5a72f-b7b7-496c-ad12-de074a768b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def quasi_newton_method(n_iter=1000, epsilon=1e-2, sigma=1e-5, delta=0.001, alpha_init=0.001, ks=5):\n",
    "    '''Implement Quasi-Newton method\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Quasi-Newton'\n",
    "    alg_dic[alg] = {}\n",
    "\n",
    "    (a,b) = X.shape\n",
    "\n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b)\n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    inv_hess = np.identity(b)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Quasi-Newton method with BFGS update rule\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        # compute gradient\n",
    "        grad = ridge_reg_der(beta, X, y)\n",
    "        \n",
    "        # compute approximation inverse of the hessian\n",
    "        if i > 1:\n",
    "            points_diff = beta - beta_old\n",
    "            gradient_diff = ridge_reg_der(beta, X, y) - ridge_reg_der(beta_old, X, y)\n",
    "            inv_hess = get_approx_inv_hess(inv_hess, points_diff, gradient_diff)\n",
    "            \n",
    "        # compute descent direction\n",
    "        ddirect = -np.dot(inv_hess, grad)\n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta_old = beta\n",
    "        beta = beta + alpha*ddirect\n",
    "        \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # if the norm of the gradient is small, we can stop\n",
    "        tol = np.linalg.norm(grad,ord=2)\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "5a96bc69-a535-4a4e-9222-a1a30ad76eb5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Quasi-Newton        | 0.0        | 0.334951676  | 999        | 9.06882    | 0.00908    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 28         | 3.9929          | 3.9929          |\n",
      "--------------------------------------------------\n",
      "| 44         | 3.99035         | 3.99035         |\n",
      "--------------------------------------------------\n",
      "| 56         | -2.0103         | -2.0103         |\n",
      "--------------------------------------------------\n",
      "| 79         | -2.98967        | -2.98967        |\n",
      "--------------------------------------------------\n",
      "| 42         | -5.02055        | -5.02055        |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "n_iter = 1000      # maximum number of iterations\n",
    "epsilon = 1e-8     # tolerance limit\n",
    "sigma = 1e-2       # Armijo rule parameter\n",
    "delta = 1e-8       # Armijo rule parameter\n",
    "alpha_init = 2e-2  # initial alpha for Armijo rule\n",
    "\n",
    "# Run algorithm\n",
    "quasi_newton_method(n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d75e29f-0d97-49db-bee5-a2adf51e843d",
   "metadata": {},
   "source": [
    "The results from the methods can be easily printed with functon `show_alg_stats()` and indicating the desired algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e61c4aff-92bb-4ae9-a0a4-e32b170525d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Gradient            | 0.09287    | 43.065986502 | 1999       | 10.77377   | 0.00539    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Newton              | 0.0        | 2.813e-06    | 499        | 27.89775   | 0.05591    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Quasi-Newton        | 0.0        | 0.334951676  | 999        | 9.06882    | 0.00908    |\n",
      "------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "show_alg_stats(['Gradient', 'Newton', 'Quasi-Newton'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef18163e-b72e-478d-b17e-54d219ca3f23",
   "metadata": {},
   "source": [
    "As it was expected, the fastest method is the **Gradient** taking into account the number of iterations. Different parameters have been tried in order to get a lower error, but it was not possible to reduce it. Incrementing the number of iterations was not decreasing the error and tolerance.\n",
    "\n",
    "The one with smallest error corresponds with **Newton**.  Mention it is so fast since in only a few iterarions is able to reach the global minimum, although in mean of seconds per iteration is the slowest, as expected.\n",
    "\n",
    "An intermmediate option is the **BFGS Quasi-Newton**. It converges better than the Gradient, but it needs more iterations than Newton. In velocity it is also between both previous options. Also in this case, with that small tolerance limit, was not possible to reduce it more incrementing the number of iterations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "119211de-c7f2-4054-8e21-04de5e9e616d",
   "metadata": {},
   "source": [
    "## D. Algorithm implementations (II)\n",
    "\n",
    "Following methods have as common the same goal, trying to reduce the computational time. In these problem there could be millions of samples ($n$) with a large number of variables $(k)$, so previous algorithms could take long time to finish. For this reason, next approaches reduce these periods simplifying or approximating certain calculations of the original Gradient method.\n",
    "\n",
    "It is important to mention that now the tolerance is calculated in another way. Instead of using the squared norm of the gradient, it is compared the difference between the objective function to minimize. If this value is smaller than the tolerance limit, the algorithm stops."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "919adb30-7794-4a82-a49c-ccf8de2e1c2c",
   "metadata": {},
   "source": [
    "### D.1. Coordinate gradient method\n",
    "\n",
    "It follows the same steps than the Gradient method, the only difference is in the gradient calculation. In each iteration, it is optimized **only one component** of the fitted regression coefficients:\n",
    "\n",
    "$$\n",
    "\\beta_{t+1} = \\beta_t - \\alpha \\dfrac{\\partial f}{\\partial \\beta_{j_t}}(\\left.\\beta^{*}\\right) e_{j_t}\n",
    "$$\n",
    "\n",
    "where $e_{j_t}$ corresponds with the unit vector and the component $j_t$ is usually chosen randomly. If the gradient of the function is:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla f(\\left.\\beta^{*} \\right) = -2(y - X\\beta)^T X + 2\\rho \\beta^T\n",
    "\\end{align*}\n",
    "\n",
    "Each partial derivative is defined as:\n",
    "\n",
    "\\begin{align*}\n",
    "     \\dfrac{\\partial f}{\\partial \\beta_j}(\\left.\\beta^{*}\\right) = \\sum_{i=1}^n 2(y_i - (\\beta_0 + \\beta_1 x_{1i} + \\dots + \\beta_k x_{ki} + \\dots + \\beta_j x_{ji})) (-x_{ji}) + 2 \\rho \\beta_j\n",
    "\\end{align*}\n",
    "\n",
    "This expression can be formulated in vector format:\n",
    "\n",
    "\\begin{align*}\n",
    "     \\dfrac{\\partial f}{\\partial \\beta_j}(\\left.\\beta^{*}\\right) = -2 (y - X \\beta)^T X_j + 2 \\rho \\beta_j\n",
    "\\end{align*}\n",
    "\n",
    "where $X_j$ is the column $j$ of the matrix $X$. At the end, the gradient is going to be a vector with all zeroes excluding the chosen component in the iteration, which has the partial derivative:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla_t f(\\left.\\beta^{*} \\right) = \\begin{bmatrix} 0 \\\\ \\dots \\\\ \\dfrac{\\partial f}{\\partial \\beta_{j_t}}(\\left.\\beta^{*}\\right) \\\\ 0\\end{bmatrix}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "02d0f387-ad0d-480a-9ab4-da84152962ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Coordinate gradient\n",
    "def ridge_reg_coor_der(beta_r, X, y, j):\n",
    "    '''Calculate coordinate gradient\n",
    "    \n",
    "    '''\n",
    "    bj = beta_r[j]\n",
    "    beta_r = np.matrix(beta_r).T\n",
    "    Xj = np.matrix(X[:, j]).T\n",
    "    grad = np.zeros(X.shape[1])\n",
    "    grad[j] = -2*np.dot((y - X*beta_r).T, Xj) + 2*rho*bj\n",
    "    return grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "380794f4-6a34-448a-9238-0765632f12aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def coordinate_gradient_method(n_iter=2000, epsilon=1e-4, sigma=1e-5, delta=0.01, alpha_init=1e-5, ks=5):\n",
    "    '''Implement Coordinate gradient method\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Coordinate'\n",
    "    alg_dic[alg] = {}\n",
    "    \n",
    "    (a,b) = X.shape\n",
    "    \n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b)\n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Coordinate method\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        # compute gradient\n",
    "        grad = ridge_reg_coor_der(beta, X, y, np.random.randint(0, nvars))\n",
    "        # compute descent direction\n",
    "        ddirect = -grad \n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta_old = beta\n",
    "        beta = beta + alpha*ddirect\n",
    "    \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # Tolerance calculation\n",
    "        tol = np.absolute(OF_iter[i] - OF_iter[i-1])\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "29c95ec3-8371-445b-9715-0c5c521fbd42",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Coordinate          | 0.09603    | 7e-09        | 7341       | 27.42444   | 0.00374    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 70         | 2.96238         | 2.98507         |\n",
      "--------------------------------------------------\n",
      "| 87         | 0.0121          | 0.00169         |\n",
      "--------------------------------------------------\n",
      "| 31         | 4.00141         | 3.9958          |\n",
      "--------------------------------------------------\n",
      "| 29         | 2.98854         | 2.99209         |\n",
      "--------------------------------------------------\n",
      "| 32         | 1.98996         | 1.99881         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "n_iter = 10000\n",
    "epsilon = 1e-8\n",
    "sigma = 1e-1\n",
    "delta = 1e-1\n",
    "alpha_init = 1e-5\n",
    "\n",
    "# Run algorithm\n",
    "coordinate_gradient_method(n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b690db-741e-4e32-a4bc-0f78f6e92961",
   "metadata": {},
   "source": [
    "### D.2. Mini-batch gradient method\n",
    "\n",
    "The procedure is the same than gradient method, but in this case the gradient is calculated using a **subset** of samples instead of the whole data:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla f(\\left.\\beta^{*} \\right) = -2(y_b - X_b\\beta)^T X_b + 2\\rho \\beta^T\n",
    "\\end{align*}\n",
    "\n",
    "where $X_b$ and $y_b$ correspond with the sampled $X$ and $y$ variables respectively. The number of samples or batch size $b$ is fixed during the process, and the samples are randomly selected in each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ec22bcb4-6d0a-4c08-b40e-604cd7654e4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to select random samples\n",
    "def sample_data(batch):\n",
    "    '''Select random samples given a batch\n",
    "    \n",
    "    Args:\n",
    "        batch (int): number of samples to take\n",
    "        \n",
    "    '''\n",
    "    idx = random.sample(range(nsample), batch)\n",
    "    return X[idx, :], y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bdaa81af-7de4-43a0-a292-cc3eb393efd0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mini_batch_gradient_method(batch=50, n_iter=2000, epsilon=1e-4, sigma=1e-5, delta=0.01, alpha_init=1e-5, ks = 5):\n",
    "    '''Implement Mini-batch gradient method\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Mini-batch'\n",
    "    alg_dic[alg] = {}\n",
    "    \n",
    "    (a,b) = X.shape\n",
    "    \n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b) \n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Mini batch method\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        \n",
    "        # get subsample data\n",
    "        X_b, y_b = sample_data(batch)\n",
    "        \n",
    "        # compute gradient\n",
    "        grad = ridge_reg_der(beta, X_b, y_b)\n",
    "        # compute descent direction\n",
    "        ddirect = -grad\n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta_old = beta\n",
    "        beta = beta + alpha*ddirect\n",
    "    \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # Tolerance calculation\n",
    "        tol = np.absolute(OF_iter[i] - OF_iter[i-1])\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)\n",
    "        \n",
    "    return err, i, time_elapsed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c562f7e-40da-4dd8-8b8a-1f090294f7af",
   "metadata": {},
   "source": [
    "In next cell it is employed the whole data as batch, in order to see if the algorithm is working."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5ab1415c-cc5d-4f98-8c4e-b76da171192b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.0943     | 0.000348951  | 2999       | 78.11931   | 0.02605    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 30         | 2.99925         | 3.00752         |\n",
      "--------------------------------------------------\n",
      "| 82         | -1.01896        | -1.01327        |\n",
      "--------------------------------------------------\n",
      "| 88         | -3.02691        | -3.02075        |\n",
      "--------------------------------------------------\n",
      "| 45         | -1.01399        | -1.00733        |\n",
      "--------------------------------------------------\n",
      "| 10         | 3.00249         | 3.00503         |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "batch = nsample\n",
    "n_iter = 3000\n",
    "epsilon = 1e-8\n",
    "sigma = 1e-1\n",
    "delta = 1e-1\n",
    "alpha_init = 1e-5\n",
    "\n",
    "# Run algorithm\n",
    "err, ite, tim = mini_batch_gradient_method(batch, n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8b51cd2-1ff1-468d-9d65-3348efbb1381",
   "metadata": {},
   "source": [
    "Now, different batches are applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "eb510920-aa0b-4507-aefe-064c49c963c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "batch = 1\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 10.96051   | 0.0          | 20         | 0.14693    | 0.00735    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 100\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 4.92891    | 1e-09        | 19         | 0.20703    | 0.0109     |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 200\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.38626    | 0.0          | 133        | 1.48199    | 0.01114    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 300\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.28516    | 0.0          | 163        | 2.00471    | 0.0123     |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 400\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09746    | 0.0          | 488        | 7.14576    | 0.01464    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 500\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09639    | 0.0          | 565        | 9.02176    | 0.01597    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 600\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09643    | 0.0          | 556        | 10.40822   | 0.01872    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 700\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09582    | 0.0          | 547        | 10.6758    | 0.01952    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 800\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09507    | 0.0          | 583        | 17.54763   | 0.0301     |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 900\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.09496    | 0.0          | 621        | 17.93729   | 0.02888    |\n",
      "------------------------------------------------------------------------------------------\n",
      "batch = 1000\n",
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.0943     | 0.000348951  | 2999       | 77.22793   | 0.02575    |\n",
      "------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "batches = np.array([1])\n",
    "batches = np.append(batches, np.arange(1, 11, 1)*100)\n",
    "errors = []\n",
    "iters = []\n",
    "times = []\n",
    "\n",
    "for batch in batches:\n",
    "    print(f'batch = {batch}')\n",
    "    err, ite, tim = mini_batch_gradient_method(batch, n_iter, epsilon, sigma, delta, alpha_init, ks=0)\n",
    "    errors = np.append(errors, err)\n",
    "    iters = np.append(iters, ite)\n",
    "    times = np.append(times, tim)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a24ae812-4a06-4908-b0af-de13c6459d94",
   "metadata": {},
   "source": [
    "In next cell it is shown how the number of iterations increases exponentially if the batch size is larger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "bb2c168c-a69b-4840-a4d2-86bb34277c1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe2cc27c220>]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAf4AAAF2CAYAAACPjPqQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3NElEQVR4nO3dfXxU9Z33/9eZ3EAgJMxNQkwIUu6qKCWxQYVVUEltq7brz3q5ax/ai1RaqqKPyuqlrVtpL+uKtYiyDd4V683l1tp9FHZtr9bdGDdsm/5+zRZZrVYBgYTMBHIzQ24gYZLM9/fHJEMoCSTMZM5k5v18PNrMnDlz5jOfWt/zPed7zrGMMQYRERFJCQ67CxAREZH4UfCLiIikEAW/iIhIClHwi4iIpBAFv4iISApR8IuIiKQQBb+IiEgKSbe7gHjx+Xwx25bH46G1tTVm20tF6mH01MPoqYexoT5GL9Y9LCwsHPE1jfhFRERSiIJfREQkhSj4RUREUoiCX0REJIUo+EVERFKIgl9ERCSFKPhFRERSiIJfREQkhSj4RUREUkhcrtwXDAZZv349fX199Pf3c+mll3LTTTfR1dXFpk2baGlpIS8vj3vuuYfs7GwAtm3bRnV1NQ6Hg4qKCkpKSgDYt28flZWVBINBSktLqaiowLKseHwNERGRCS8uI/6MjAzWr1/P448/zg9+8AN27drF7t272b59O4sWLWLz5s0sWrSI7du3A9DY2EhtbS1PPPEEDz74IFu3biUUCgHw/PPPs2bNGjZv3syhQ4fYtWtXPL6CiIhIUohL8FuWxeTJkwHo7++nv78fy7Koq6tjxYoVAKxYsYK6ujoA6urqWLZsGRkZGeTn51NQUMDevXsJBAJ0d3ezYMECLMti+fLlkfeIiIhMROa9P9LX1Bi3z4vbTXpCoRD3338/hw4d4rOf/Szz58+nvb0dp9MJgNPppKOjAwC/38/8+fMj73W5XPj9ftLS0nC73ZHlbrcbv98/7OdVVVVRVVUFwIYNG/B4PDH7Lunp6THdXipSD6OnHkZPPYwN9fHsmVCI5mcfo+ez1+OpuDsunxm34Hc4HDz++OMcPXqUH/7whzQ0NIy4rjFmTMuHU15eTnl5eeR5LO96pDtRRU89jJ56GD31MDbUx7NnWg7B8R4cM2cn7935pk6dysKFC9m1axe5ubkEAgEAAoEAOTk5QHgk39bWFnmP3+/H5XKdsrytrQ2XyxXfLyAiIhIr3noA0mfNidtHxiX4Ozo6OHr0KBCe4f/ee+9RVFREWVkZNTU1ANTU1LBkyRIAysrKqK2tpbe3l+bmZpqampg3bx5Op5OsrCx2796NMYYdO3ZQVlYWj68gIiISc2Yw+Is/EbfPjMuu/kAgQGVlJaFQCGMMS5cu5dOf/jQLFixg06ZNVFdX4/F4WLduHQDFxcUsXbqUdevW4XA4uO2223A4wr9RVq9ezZYtWwgGg5SUlFBaWhqPryAiIhJ7vgZw5+OYMhWOdcflIy0zlgPnE5jP54vZtnQ8K3rqYfTUw+iph7GhPp69/u/eBa48Zvzvzcl7jF9ERETA9PXBIS9W0blx/VwFv4iIiB2afdDfB0Wz4vqxCn4REREbGG/4tHarUCN+ERGR5OerB8sB58yM68cq+EVERGxgvPWQfw5WRmZcP1fBLyIiYgdvQ9yP74OCX0REJO5M8Di0NMX9+D4o+EVEROKvqRGMwdKIX0REJPkNXqqXOJ/DDwp+ERGR+PM1QHo65J0T949W8IuIiMSZ8TVAwUys9LjcMuckCn4REZF489bbMrEPFPwiIiJxZbqPgb/FllP5QMEvIiISX76BS/XaMLEPFPwiIiJxFZnRX6gRv4iISPLzNcCkyeDOt+XjFfwiIiJxZLz1UDgLy2FPBCv4RURE4slbj2XTbn5Q8IuIiMSN6TgCne22XLFvkIJfREQkXiIz+jXiFxERSXrGGw5+bLp4Dyj4RURE4sdXD1OnQa7TthIU/CIiInFivPVQNAvLsmyrQcEvIiISB8YY8DXYdo3+QQp+ERGReAi0Qvcx267RP0jBLyIiEg8DE/vsPIcfFPwiIiJxYXz2XqN/kIJfREQkHrz1kOvCys6xtQwFv4iISBwYb4Ptx/dBwS8iIjLuTKgfDh20fUY/KPhFRETGX+thCAY14hcREUkJgzP6bbw5zyAFv4iIyDgz3oEZ/ecU21sICn4REZHx52sAzwysyVl2V6LgFxERGW/ha/Tbv5sfFPwiIiLjyvT1wmGv7VfsG6TgFxERGU+HfdDfrxG/iIhIKhic2GclwKl8oOAXEREZX94GcDhgxky7KwEU/CIiIuPK+OphRhFWRobdpQAKfhERkfHlrU+YiX0A6fH4kNbWViorKzly5AiWZVFeXs4111zD66+/zltvvUVOTvhORTfffDMXXXQRANu2baO6uhqHw0FFRQUlJSUA7Nu3j8rKSoLBIKWlpVRUVGBZVjy+hoiIyJiY4z3hy/UuvcruUiLiEvxpaWnceuutzJkzh+7ubh544AE+9alPAXDttdfyxS9+8aT1Gxsbqa2t5YknniAQCPDwww/z1FNP4XA4eP7551mzZg3z58/n0UcfZdeuXZSWlsbja4iIiIxN00EwJmEm9kGcdvU7nU7mzJkDQFZWFkVFRfj9/hHXr6urY9myZWRkZJCfn09BQQF79+4lEAjQ3d3NggULsCyL5cuXU1dXF4+vICIiMmZm4Br9JMBd+QbFZcQ/VHNzM/v372fevHl8+OGHvPnmm+zYsYM5c+bwla98hezsbPx+P/Pnz4+8x+Vy4ff7SUtLw+12R5a73e4Rf0BUVVVRVVUFwIYNG/B4PDH7Dunp6THdXipSD6OnHkZPPYwN9XFknYFmjmVk4jn/Qqy0tBHXi2cP4xr8PT09bNy4kVWrVjFlyhSuvvpqbrzxRgB+9rOf8fLLL3PHHXdgjBn2/SMtH055eTnl5eWR562trdEVP4TH44np9lKRehg99TB66mFsqI8j69/7IZwzk7ZA4LTrxbqHhYWFI74Wt1n9fX19bNy4kcsvv5xLLrkEgOnTp+NwOHA4HKxcuZKPP/4YCI/k29raIu/1+/24XK5Tlre1teFyueL1FURERMbG25BQM/ohTsFvjOGZZ56hqKiI6667LrI8MOQX0B/+8AeKi8O3KywrK6O2tpbe3l6am5tpampi3rx5OJ1OsrKy2L17N8YYduzYQVlZWTy+goiIyJiYo11wpC2hju9DnHb1f/TRR+zYsYNZs2Zx3333AeFT9373u99x4MABLMsiLy+Pr3/96wAUFxezdOlS1q1bh8Ph4LbbbsPhCP9GWb16NVu2bCEYDFJSUqIZ/SIikph84Yl9iTSjH8AyYzlwPoH5fL6YbUvHs6KnHkZPPYyeehgb6uPwQv/xa8yrT+PY8GMsd/5p103KY/wiIiIpxdcAk7LAlWd3JSdR8IuIiIwD42uAolkJd3VZBb+IiEiMGWPAewCrKLEm9oGCX0REJPY6j0BXJyTYqXyg4BcREYk97+CMfo34RUREkp7x1ocfJNipfKDgFxERiT1fA2TnwLTpdldyCgW/iIhIjBlvPRSdm3Az+kHBLyIiElPhGf2Jd43+QQp+ERGRWPK3wPFuSMCJfaDgFxERia2BiX2Jdo3+QQp+ERGRGDIDp/Il4jn8oOAXERGJLV89OD1YU7LtrmRYCn4REZEYCs/oT8zRPij4RUREYsb090NTI1ZhYk7sAwW/iIhI7LQ0QV+vRvwiIiIpYfAa/Qk6sQ8U/CIiIjFjvPVgWXBOsd2ljEjBLyIiEiPGVw+eGViTJttdyogU/CIiIrHibUjYK/YNUvCLiIjEgOnthWZfQs/oBwW/iIhIbBxuhFAooWf0g4JfREQkJgYv1WtpV7+IiEgK8NZDWhrMKLS7ktNS8IuIiMSA8TXAjCKs9Ay7SzktBb+IiEgseOsTfjc/KPhFRESiZnq6ofVwwt6KdygFv4iISLSaDgKJP7EPFPwiIiJRM9768IMEP5UPFPwiIiLR8zZAZiZ4ZthdyRkp+EVERKJkfPVwziwsR5rdpZyRgl9ERCRa3oaEvhXvUAp+ERGRKJiuDmj3J/zNeQYp+EVERKLhG7xUr0b8IiIiSW/wGv0k+F35Bin4RUREouGrh6yp4HTbXcmoKPhFRESiYLz1UDQLy7LsLmVUFPwiIiJnyRgzoWb0g4JfRETk7LX74VjXhDm+Dwp+ERGRs+edWDP6AdLj8SGtra1UVlZy5MgRLMuivLyca665hq6uLjZt2kRLSwt5eXncc889ZGdnA7Bt2zaqq6txOBxUVFRQUlICwL59+6isrCQYDFJaWkpFRcWEOa4iIiLJ5cQ1+jXiP0laWhq33normzZt4pFHHuHNN9+ksbGR7du3s2jRIjZv3syiRYvYvn07AI2NjdTW1vLEE0/w4IMPsnXrVkKhEADPP/88a9asYfPmzRw6dIhdu3bF4yuIiIicytcA03KxpuXaXcmoxSX4nU4nc+bMASArK4uioiL8fj91dXWsWLECgBUrVlBXVwdAXV0dy5YtIyMjg/z8fAoKCti7dy+BQIDu7m4WLFiAZVksX7488h4REZF4M76GCTXaBxuO8Tc3N7N//37mzZtHe3s7TqcTCP846OjoAMDv9+N2nzgf0uVy4ff7T1nudrvx+/3x/QIiIiKACYXA14A1wYI/Lsf4B/X09LBx40ZWrVrFlClTRlzPGDOm5cOpqqqiqqoKgA0bNuDxeMZW7Gmkp6fHdHupSD2MnnoYPfUwNlK1j/2HfbQe7yF7wUKmRPn949nDuAV/X18fGzdu5PLLL+eSSy4BIDc3l0AggNPpJBAIkJOTA4RH8m1tbZH3+v1+XC7XKcvb2tpwuVzDfl55eTnl5eWR562trTH7Lh6PJ6bbS0XqYfTUw+iph7GRqn00f9oFwNFcN8ei/P6x7mFhYeGIr8VlV78xhmeeeYaioiKuu+66yPKysjJqamoAqKmpYcmSJZHltbW19Pb20tzcTFNTE/PmzcPpdJKVlcXu3bsxxrBjxw7Kysri8RVEREROEpnRP4Eu3gNxGvF/9NFH7Nixg1mzZnHfffcBcPPNN3P99dezadMmqqur8Xg8rFu3DoDi4mKWLl3KunXrcDgc3HbbbTgc4d8oq1evZsuWLQSDQUpKSigtLY3HVxARETmZtwFceVhZIx+6TkSWGcuB8wnM5/PFbFupulsrltTD6KmH0VMPYyNV+9j/vbvB6SHt7oei3lbS7eoXERFJJqavDw41Tqhr9A9S8IuIiIxVSxP09U24c/hBwS8iIjJ2AxP7JtI1+gcp+EVERMbIeBvAckDBTLtLGTMFv4iIyBgZXz3kn4OVOcnuUsZMwS8iIjJW3gaYgLv5QcEvIiIyJiZ4HJqbsAon3sQ+UPCLiIiMzaFGMKEJObEPFPwiIiJjYrwN4QcT8FQ+UPCLiIiMjbce0tIh7xy7KzkrCn4REZExML4GKCjCSo/rne1jRsEvIiIyFt56rAm6mx8U/CIiIqNmuo+Bv2XC3Yp3KAW/iIjIaPnCE/s04hcREUkBxjexZ/SDgl9ERGT0vPWQOQnc+XZXctYU/CIiIqNkfA1QOAvLMXHjc+JWLiIiEm/e+gl7xb5BCn4REZFRMJ3t0HEEJug1+geN+uoDv/3tb5k9ezYzZ87E5/Px7LPP4nA4WL16NUVFReNZo4iIiP2SYEY/jGHE/7Of/Yzs7GwAXn75ZebOncv555/Pj3/843ErTkREJFEYb334Qars6u/o6GD69OkEg0E++ugjbr75Zm688UYOHDgwjuWJiIgkCG8DTMmGXJfdlURl1Lv6c3JyOHToEA0NDcydO5eMjAyOHz8+nrWJiIgkDOOrh6JZWJZldylRGXXwf+lLX+L+++/H4XBwzz33APDee+9x7rkT+1iHiIjImRhjwNuAdclyu0uJ2qiD/4orrmDp0qUATJo0CYD58+fzzW9+c1wKExERSRiBNug+OuFn9MMYgh/Cgd/e3s6RI0fGqRwREZEE5AtP7Jvo5/DDGIJ/165dPP3008OG/s9+9rNY1iQiIpJQjHfgGv0T+K58g0Yd/Fu3buVLX/oSV1xxBZmZmeNZk4iISGLx1kOuCys7x+5Kojbq4O/q6uIzn/nMhJ/NKCIiMlbG1zDhz98fNOrz+K+66irefvvt8axFREQk4ZhQPzQ1YCXBxD4Yw4h/z549/PrXv+Zf/uVfmD59+kmvfe9734t1XSIiIomh9TAEg1BYbHclMTHq4L/qqqu46qqrxrMWERGRxONNjmv0DxrTefwiIiKpJnKN/lQb8QO8/fbb7NixA7/fj8vlYvny5Vx55ZXjVZuIiIj9fA3gzseaPMXuSmJi1MH/i1/8gpqaGr7whS/g8XhobW3lX//1XwkEAtxwww3jWaOIiIhtwjP6k2M3P4wh+N966y2++93vkpeXF1m2ePFi1q9fr+AXEZGkZPp64VAj1qfK7C4lZkZ9Ot/x48fJyTn5wgXTpk0jGAzGvCgREZGEcLgJ+vuT4hr9g0Yd/CUlJWzevBmfz0cwGMTr9fKjH/2IxYsXj2d9IiIitjGRa/QnT/CPelf/V7/6VV544QXuu+8++vr6SE9PZ+nSpVRUVIxnfSIiIvbx1oPDAQVFdlcSM6MO/ilTprB27VruuOMOOjs7mTZtGg7HqHcYiIiITDjG2wD5hVgZyXOPmtMGf3NzM/n5+QAcPnz4pNd6enoij2fMmDEOpYmIiNjMVw/Fn7C7ipg6bfDfe++9vPzyywDcfffdI653ptvybtmyhZ07d5Kbm8vGjRsBeP3113nrrbciEwZvvvlmLrroIgC2bdtGdXU1DoeDiooKSkpKANi3bx+VlZUEg0FKS0upqKjQTYNERGRcmOPHoeUQ1iVX2F1KTJ02+AdDH84c7qdzxRVX8LnPfY7KysqTll977bV88YtfPGlZY2MjtbW1PPHEEwQCAR5++GGeeuopHA4Hzz//PGvWrGH+/Pk8+uij7Nq1i9LS0rOuS0REZESHDoIxSTWxD8Ywq/+FF14YdvmLL754xvcuXLiQ7OzsUX1OXV0dy5YtIyMjg/z8fAoKCti7dy+BQIDu7m4WLFiAZVksX76curq60ZYvIiIyJpFL9SbJ7XgHjTr4a2pqhl2+Y8eOs/7wN998k3vvvZctW7bQ1dUFgN/vx+12R9ZxuVz4/f5Tlrvdbvx+/1l/toiIyGl5GyA9A/LOsbuSmDrjrP7q6moA+vv7I48HNTc3M23atLP64Kuvvpobb7wRCB9GePnll7njjjswxgy7/kjLR1JVVUVVVRUAGzZswOPxnFWdw0lPT4/p9lKRehg99TB66mFsJGsfAy1NhIpn447DBPZ49vCMwf+f//mfAPT19UUeD8rNzeXOO+88qw+ePn165PHKlSt57LHHgPBIvq2tLfLa4A2B/nJ5W1sbLpdrxO2Xl5dTXl4eed7a2npWdQ5n8F4FcvbUw+iph9FTD2MjWfvYf2Av1icvjMt3i3UPCwsLR3ztjMG/fv16AF577TX+9m//NmZFBQIBnE4nAH/4wx8oLg7f7rCsrIzNmzdz3XXXEQgEaGpqYt68eTgcDrKysti9ezfz589nx44dfO5zn4tZPSIiIoPMsS4ItCbVpXoHjfoCPkND3xhz0q73M13I58knn+SDDz6gs7OTb3zjG9x00028//77HDhwAMuyyMvL4+tf/zoAxcXFLF26lHXr1uFwOLjtttsi21+9ejVbtmwhGAxSUlKiGf0iIjI+fA0AWEk2sQ/AMqM8eO73+9m6dSt//vOfOXr06EmvRXOqX7z4fL6YbStZd2vFk3oYPfUweuphbCRjH0M1v8H8ny04NvwYy50/7p8Xz139o57V/9xzz5Gens5DDz3E5MmTeeyxxygrK+NrX/taTIoUERFJGN56mJQFrrwzrzvBjDr4d+/eze23387s2bOxLIvZs2dz++2388tf/nI86xMREYk742uAwuKkvDrsqIPf4XCQlpYGwNSpU+no6GDSpEk6l15ERJKPtz7prtg3aNST++bNm8c777zDxRdfzOLFi9m0aROZmZnMnTt3POsTERGJK9NxBLo6ku6KfYNGHfx33XVXZCb/qlWreOONN+ju7ubaa68dt+JERETibuBSvVYSnsoHowz+UCjET37yE9asWQNAZmYmX/rSl8a1MBERETuYgVP5SNJd/aM6xu9wOHj33XeTcpKDiIjISbz1kD0NcqbbXcm4GPXkvmuvvZbXX3+dvr6+8axHRETEVuEZ/ecm7WB31Mf4f/Ob33DkyBF+9atfkZOTc9JrTz/9dMwLExERiTdjTHhG/9Ir7S5l3Ixpcp+IiEhS87dCT3dSXqN/0KiDf+HCheNZh4iIiP18AzP6k3RiH4wh+Ht7e/nnf/5nfve739HZ2clLL73Ef//3f9PU1KS75ImISFIwA6fyUZic5/DDGCb3vfTSSxw8eJC77747MuGhuLiYf/u3fxu34kREROLK2wDT3VhTs+2uZNyMesT/hz/8gc2bNzN58uRI8LtcLl2yV0REkobx1SftFfsGjXrEn56eTigUOmlZR0cH06ZNi3lRIiIi8WZC/eA7mNTH92EMwX/ppZfyox/9iObmZgACgQBbt25l2bJl41aciIhI3DQfgr7epJ7RD2MI/i9/+cvk5+fzd3/3dxw7doy7774bp9PJjTfeOJ71iYiIxEdkRn9y7+of9TH+9PR0Vq1axapVqyK7+JP1qkYiIpJ6jLcBLAvOKba7lHE16hF/RUVF5HFOTk4k9FevXh37qkREROLNWw+eGViTJttdybgadfD39/efsqyvr++UCX8iIiITkfE1JO0d+YY6467+hx56CMuy6O3tZf369Se91tbWxoIFC8atOBERkXgwvb1w2ItVutTuUsbdGYP/qquuAmDv3r1ceeWJmxZYlkVubi4XXnjh+FUnIiISD4cbIRSCwuQ+vg+jCH6PxwPArbfeGnk81IcffqjwFxGRCc14G4Dkvkb/oDMG/5luuWtZFj/60Y9iVpCIiEjceeshLQ0KiuyuZNydMfgrKyvjUYeIiIhtjK8B8gux0jPsLmXcjXpWv4iISNLyNaTEbn5Q8IuISIozx3ug5VDS35xnkIJfRERSm+8gAFaSX6N/kIJfRERSmhm4Rn8qXLwHFPwiIpLqvPWQkQl5M+yuJC4U/CIiktKMtwHOKcZypNldSlwo+EVEJLX56pP+VrxDKfhFRCRlmaOdcMSfMsf3QcEvIiKpbPBSvSkyox8U/CIiksJOzOjXrn4REZHk522ArCngPPUmdMlKwS8iIinL+OqhcBaWZdldStwo+EVEJCUZY8CbOtfoH6TgFxGR1NQegKOdkEIT+0DBLyIiqWpgYl8qncMPCn4REUlRZuBUvlQ6hx8gPR4fsmXLFnbu3Elubi4bN24EoKuri02bNtHS0kJeXh733HMP2dnZAGzbto3q6mocDgcVFRWUlJQAsG/fPiorKwkGg5SWllJRUZFSEzJERCSGvPUwLRdrWq7dlcRVXEb8V1xxBd/+9rdPWrZ9+3YWLVrE5s2bWbRoEdu3bwegsbGR2tpannjiCR588EG2bt1KKBQC4Pnnn2fNmjVs3ryZQ4cOsWvXrniULyIiScj4GqAwtXbzQ5yCf+HChZHR/KC6ujpWrFgBwIoVK6irq4ssX7ZsGRkZGeTn51NQUMDevXsJBAJ0d3ezYMECLMti+fLlkfeIiIiMhQmFwJd6M/ohTrv6h9Pe3o7T6QTA6XTS0dEBgN/vZ/78+ZH1XC4Xfr+ftLQ03G53ZLnb7cbv94+4/aqqKqqqqgDYsGEDHk/sLs6Qnp4e0+2lIvUweuph9NTD2JiIfexvbqL1eA/Zn1zIlASoPZ49tC34R2KMGdPykZSXl1NeXh553traGlVdQ3k8nphuLxWph9FTD6OnHsbGROyjeW8XAEdz3BxLgNpj3cPCwsIRX7NtVn9ubi6BQACAQCBATk4OEB7Jt7W1Rdbz+/24XK5Tlre1teFyueJbtIiIJIXINfp1jD9+ysrKqKmpAaCmpoYlS5ZEltfW1tLb20tzczNNTU3MmzcPp9NJVlYWu3fvxhjDjh07KCsrs6t8ERGZyLz14PJgTZlqdyVxF5dd/U8++SQffPABnZ2dfOMb3+Cmm27i+uuvZ9OmTVRXV+PxeFi3bh0AxcXFLF26lHXr1uFwOLjttttwOMK/T1avXs2WLVsIBoOUlJRQWloaj/JFRCTJGG9Dyl2xb5BlxnrwfILy+Xwx29ZEPJ6VaNTD6KmH0VMPY2Oi9dH09xNa+z+wVn4Bx40VdpcDpMgxfhEREVs0N0FfX8qO+BX8IiKSWiLX6Ffwi4iIJD3jrQfLgnNm2l2KLRT8IiKSUoy3AfLOwcqcZHcptlDwi4hIavHVQ4rdincoBb+IiKQM0xuEw00pe3wfFPwiIpJKmhrBhFJ2Rj8o+EVEJIWYyIx+7eoXERFJft4GSEuH/JEvcJPsFPwiIpIyjLceCoqw0hPu5rRxo+AXEZHU4WtI6Yl9oOAXEZEUYbqPQVtzSt6KdygFv4iIpAZfA5DaE/tAwS8iIinCDAR/Kp/KBwp+ERFJFd56yMwEzwy7K7GVgl9ERFKC8TXAObOwHKkdfan97UVEJHVoRj+g4BcRkRRgOjugPZDSN+cZpOAXEZHkNzijP8Un9oGCX0REUsDgNfrRrn4Fv4iIpABvPUyZCtNddldiOwW/iIgkPeNtgMJzsSzL7lJsp+AXEZGkZowBX33KX7FvkIJfRESS2xE/HDuq4/sDFPwiIpLcvOGJfZrRH6bgFxGRpBaZ0Z/id+UbpOAXEZHk5m2AXCfWtBy7K0kICn4REUlqxluv0f4QCn4REUlaJhSCJl2jfygFv4iIJK/WwxAMasQ/hIJfRESS18DEPo34T0i3uwAREZFoGGOg+xh0HIGOAKY9/Jf2AGb3n8IrFRbbWWJCUfCLiEhCMsHj4TBvD0DHEUx7YCDQj2AGQn7wNXqDp24gLQ2m5WJdeiXW5Clxrj5xKfhFRCRuTF8fdLXDwKjctA8J744jmIFgpyMQHsUPJzsHcp2QMx1r/kLICT8mdzpWjnPgNSdMzcZy6Ij2X1Lwi4hIVEwohOnsOLF7fWh4R0bqR8IB39Ux/EaypoTDOnc6VvEnIKd0IMydWINBnjsdsnOx0hVd0VD3RETkFJGReUd7OLw72yOjcjoHdrUPLGvubIf+/lM3kpEZCW/yzsGae344vHMGw3z6iZF75qR4fr2UpuAXkXFhjnZBfx+kpUN6evhvWppui2ojc7znL8J7IMw7hwn3o53DbyQ9IxzY03Ih14VV/AmyCorozpg0EOjTw6/nOCFriv73TkAKfhGJigmFoPUQHNyPadiPObgPDu4L3xFtOJEfAmkDPwYGnkd+HAx5PT0j8oOB9HSstIzI45N/UAzdRhqkZQzZRvh1a+j6A397A25M19GT3zf4niHbTtTjxCYUgmNdJ4I7EuRHBp4fibxGZzsc7xl+Q1lTw0GeMx3OmYn1yQth2nTIycXKmT7weHp4nWHCfJrHw/HW1vH7ohJTCn4RGTXT2wu+hnC4N+zDHNwPjfuhpzu8gsMB5xRjnbcYZs6GzEnQ3xveDdzXF94DMPi3vx/6egee90N/H6bvL9btDYYneA08N5H39Z283b4+MKHT1z7MshF+mpzK4Rj+h0Za+hl/iFhn+DFzuu1a6RnhUXrnkSG73I+cCPKRdrFbDsieNjDyno4157yBx+FwtwZDfiDMrYzM0XZCkoCCX0SGZY52hkfxB/fDwX2Yhn1wqPFE0EzKguLZWEuvhOI5WLPmQOEs20LEhPpH+IHRd+LHQX//ST8YcqZMoSPgD/+gGMP7hq5n/nL9/j4IHh/yY2aE7Q7+iDnddxr6ZHAXe850mO7GmjU3EuRMmx4emQ+OyrOnYTnSxqvVMsEp+EVSnDEG/C0DI/iBUfzB/dDWfGKlXBcUfwJr8cXhGdfFcyCvIKF2gVuONHCkhSeUjdIkjwertRW7jkIbY4bs/RjYi9HXP+THQS9kTAoH+uQsHS+XmLA9+O+8804mT56Mw+EgLS2NDRs20NXVxaZNm2hpaSEvL4977rmH7OxsALZt20Z1dTUOh4OKigpKSkrs/QIiE4jp64NDBzEN+wdG8wPH448dDa9gWTCjCGvOJ2HF58Oj+OLZ4XOjJeYsywqP5NMzQJPaJU5sD36A9evXk5Nz4j7J27dvZ9GiRVx//fVs376d7du3c8stt9DY2EhtbS1PPPEEgUCAhx9+mKeeegpHAo06RBKF6T528q76g/vD1y3v6wuvkJEJM2djlV0eHs0XfyL8fNJkewsXkXGVEMH/l+rq6vjud78LwIoVK/jud7/LLbfcQl1dHcuWLSMjI4P8/HwKCgrYu3cvCxYssLdgERsZY8Iz6AfC3TQMjOJbDp1YKTsHZs3BWvmFE8fj8wux0nQcWCTVJETwP/LIIwB85jOfoby8nPb2dpzO8K5Fp9NJR0f4Sk9+v5/58+dH3udyufD7h5+XW1VVRVVVFQAbNmzA4/HErN709PSYbi8VqYfR6f34I47++nXSPv6I3v17wqdtDUgrKCJ93nmkf+aLZHxiPumfWIDD5dHx4WHon8PYUB+jF88e2h78Dz/8MC6Xi/b2dr7//e9TWFg44rrGDHdCzvDKy8spLy+PPG+N4TmmHo8npttLRerh2QvV/Abz0+fCx+MLZ2EtKsMaHMXPnA1ZU+gD+oAeCE8Nb2uzs+SEpX8OY0N9jF6se3i6LLU9+F0uFwC5ubksWbKEvXv3kpubSyAQwOl0EggEIsf/3W43bUP+Beb3+yPvF0l2prcX89NnMf/5b3DhReT9r3/Af3yYO5KJiJyGrbPienp66O7ujjx+9913mTVrFmVlZdTU1ABQU1PDkiVLACgrK6O2tpbe3l6am5tpampi3rx5ttUvEi8m0Ebo8W9h/vPfsK75Hzju+g6OaTlnfqOIyF+wdcTf3t7OD3/4QwD6+/u57LLLKCkpYe7cuWzatInq6mo8Hg/r1q0DoLi4mKVLl7Ju3TocDge33XabZvRL0jO73yf07GNw/DiO2x/AumiZ3SWJyARmmbEcOJ/AfD5fzLal41nRUw/PzBiDeftXmNe3gnsGjju/jVU4K/K6ehg99TA21MfopdQxfhE5lQkex/yfpzG/r4bFF+P46j1YU6baXZaIJAEFv0iCMW0thJ5+FOr3Yn3hZqzr/iahLo0rIhObgl8kgZgP3yX07A+gvw/HnQ9ilVxid0kikmQU/CIJwBiDqfpXzD//BPILw8fzC2baXZaIJCEFv4jNzPHjmJd/hPlDDZReiqPim1hZU+wuS0SSlIJfxEam5VD4eH7jAazrb8H6/I06ni8i40rBL2IT88E7hJ77IZgQjrsewlr0abtLEpEUoOAXiTNjDObNX2B+8QoUFuO441tY+SOfcysiEksKfpE4Mj3dmJf+EfNfv8Uquwzrf96FNTnL7rJEJIUo+EXixDT7CG15FHwHsW5chXX1/6Nb5YpI3Cn4ReLAvPdHQj/+IVgOHN9cj7Ww1O6SRCRFKfhFxpExBvN/f475l1ehaHb4eH5egd1liUgKU/CLjBPTc4zQC0/CO/8v1iUrsG5dizVpkt1liUiKU/CLjANzqDF8PP+wF+tvbsNa+UUdzxeRhKDgF4kxs+v/I/TCJkhLx3HP/8Y671N2lyQiEqHgF4kREwphfvka5o3X4Nx5OG7/FpY7z+6yREROouAXiQFz7CihrU/Au3VYS6/CuuV2rEwdzxeRxKPgF4mS8TWEj+e3HsL68hqsK67R8XwRSVgKfpEomJ21hF54CjIzcaz7PtaCC+wuSUTktBT8ImfBhPox//JPmP/7c/jEgvDxfKfb7rJERM5IwS8yRuZoV/gqfH/aiXX51Vg3r8HKyLC7LBGRUVHwi4yBaTxAaMs/gL8V69Y7cCz/nN0liYiMiYJfZJRCdb/FvPgUZE3Fcd8/YM09z+6SRETGTMEvcgamvx+z7WXMm9tg7nk4vvEA1nSX3WWJiJwVBb/IaZiuDkLPPQ5//m+sKz6P9TersdJ1PF9EJi4Fv8gITMPH4fPz2/1Y//MuHJd9xu6SRESipuAXGcIYA8EgZmct5pVKyM7B8b82YH1igd2liYjEhIJfkpbp74ejnSf+09WJGfjL0Q442oXpGnytI7IOfb3hDSy4AMea+7Fyptv6PUREYknBLwnPGAM93SeFsxkS5ieWdYSfH+sK/+0+OvJG09JgSjZk58DUaZBXgDV7fvhx9jRwerDKLsNK1/9FRCS56N9qEnfm2FF669sxjQfhaMeQUXf4rzkp0DvCQd7fP/IGs6aGw3pKNmRPw5pRFH4+GOJTp2ENeUx2DkzO0vX0RSQlKfhl3JlQP+zfg3l/J+b9d2D/HvwmdOqK6Rknh/M5xVhTsweW5YRDfWiAD/zHSkuL/5cSEZmgFPwyLkygDfP+Tnj/HcwHu8KjdsuC2fOxrr2JnPMuoDPEkFF5DmRmahQuIjLOFPwSE6Y3CHs+ODGq99aHX8h1YZVeAhdchHX+YqzsHAAmezx0tbbaWLGISGpS8MtZMcbAYd+JoP/oXQgGIT0d5l+AtfRKrAsugqJzNYoXEUkgCn4ZNdN9DD58Nxz2f9oJbc3hF/ILsf7qM1gXXgSfXIQ1abK9hYqIyIgU/DIiEwrBwf0Do/qd8PGH4dn1k7Lg/E9hfe4GrAsuwsorsLtUEREZJQW/nMR0HAlPxhvchd/ZHn5h1hysq6/HuuDTMPeTul69iMgEpeBPcaavD/Z9iHn/nfDu+4aPwy9k52AtLIULL8JaWIKV67S3UBERiQkFfwoyrYdPBP2H/x2+Kp7DAXPOw7r+FqwLSmHWXCyHw+5SRUQkxhT8KcAcPw67/3TiWP0hb/gFVx7WxcvDQX/eYqwpU+0tVERExt2EDP5du3bxk5/8hFAoxMqVK7n++uvtLimhGGPAdxDz/h/Dx+l3vx++8UxGJnzyQqwVnw+faldQpFPtRERSzIQL/lAoxNatW/n7v/973G433/rWtygrK2PmzJlx+fz+ZzbQ0rCPUGjgkrODwTk0QCOPrSGvR/5r+HVHev/gn+G2OXT9ocsCbXCkLfz4nGKsK68JB/38hViZk0b7VUVEJAlNuODfu3cvBQUFzJgxA4Bly5ZRV1cXt+C3iueQOS2X48d7wAwuNWBM5OEpy2DI62bgVXPyuoOvDbdshG2ceHzycsszA85fjHVBKZYr7yy+pYiIJKsJF/x+vx+32x157na72bNnT9w+33HtTeR6PLTqcrMiIjIBTbjgN0NHuwOGO05dVVVFVVUVABs2bMDj8cSshvT09JhuLxWph9FTD6OnHsaG+hi9ePZwwgW/2+2mra0t8rytrQ2n89RzzMvLyykvL488j+UI3aMRf9TUw+iph9FTD2NDfYxerHtYWFg44msT7kTtuXPn0tTURHNzM319fdTW1lJWVmZ3WSIiIhPChBvxp6Wl8dWvfpVHHnmEUCjElVdeSXFxsd1liYiITAgTLvgBLrroIi666CK7yxAREZlwJtyufhERETl7Cn4REZEUouAXERFJIQp+ERGRFKLgFxERSSEKfhERkRSi4BcREUkhCn4REZEUYpnh7nojIiIiSUkj/rPwwAMP2F3ChKceRk89jJ56GBvqY/Ti2UMFv4iISApR8IuIiKQQBf9ZKC8vt7uECU89jJ56GD31MDbUx+jFs4ea3CciIpJCNOIXERFJIel2FzCR7Nq1i5/85CeEQiFWrlzJ9ddfb3dJCam1tZXKykqOHDmCZVmUl5dzzTXX0NXVxaZNm2hpaSEvL4977rmH7OxsALZt20Z1dTUOh4OKigpKSkrs/RIJIhQK8cADD+ByuXjggQfUw7Nw9OhRnnnmGQ4ePIhlWdx+++0UFhaqj2Pwy1/+kurqaizLori4mDvuuINgMKgensaWLVvYuXMnubm5bNy4EeCs/v+7b98+KisrCQaDlJaWUlFRgWVZ0RVnZFT6+/vN2rVrzaFDh0xvb6+59957zcGDB+0uKyH5/X7z8ccfG2OMOXbsmLn77rvNwYMHzSuvvGK2bdtmjDFm27Zt5pVXXjHGGHPw4EFz7733mmAwaA4fPmzWrl1r+vv77So/obzxxhvmySefNI8++qgxxqiHZ+Ef//EfTVVVlTHGmN7eXtPV1aU+jkFbW5u54447zPHjx40xxmzcuNG8/fbb6uEZvP/+++bjjz8269atiyw7m5498MAD5qOPPjKhUMg88sgjZufOnVHXpl39o7R3714KCgqYMWMG6enpLFu2jLq6OrvLSkhOp5M5c+YAkJWVRVFREX6/n7q6OlasWAHAihUrIv2rq6tj2bJlZGRkkJ+fT0FBAXv37rWt/kTR1tbGzp07WblyZWSZejg2x44d489//jNXXXUVAOnp6UydOlV9HKNQKEQwGKS/v59gMIjT6VQPz2DhwoWR0fygsfYsEAjQ3d3NggULsCyL5cuXxyR3tKt/lPx+P263O/Lc7XazZ88eGyuaGJqbm9m/fz/z5s2jvb0dp9MJhH8cdHR0AOHezp8/P/Iel8uF3++3pd5E8uKLL3LLLbfQ3d0dWaYejk1zczM5OTls2bKF+vp65syZw6pVq9THMXC5XHzhC1/g9ttvJzMzk8WLF7N48WL18CyMtWdpaWmn5E4seqkR/yiZYU5+iPo4S5Lr6elh48aNrFq1iilTpoy43nC9TXV//OMfyc3Njew5ORP1cHj9/f3s37+fq6++mh/84AdMmjSJ7du3j7i++niqrq4u6urqqKys5Nlnn6Wnp4cdO3aMuL56OHYj9Wy8eqkR/yi53W7a2toiz9va2iK/3ORUfX19bNy4kcsvv5xLLrkEgNzcXAKBAE6nk0AgQE5ODnBqb/1+Py6Xy5a6E8VHH33Ef/3Xf/HOO+8QDAbp7u5m8+bN6uEYud1u3G53ZDR16aWXsn37dvVxDN577z3y8/MjPbrkkkvYvXu3engWxtqz4XInFr3UiH+U5s6dS1NTE83NzfT19VFbW0tZWZndZSUkYwzPPPMMRUVFXHfddZHlZWVl1NTUAFBTU8OSJUsiy2tra+nt7aW5uZmmpibmzZtnS+2J4stf/jLPPPMMlZWVfPOb3+TCCy/k7rvvVg/HaPr06bjdbnw+HxAOsZkzZ6qPY+DxeNizZw/Hjx/HGMN7771HUVGRengWxtozp9NJVlYWu3fvxhjDjh07YpI7uoDPGOzcuZOXXnqJUCjElVdeyQ033GB3SQnpww8/5KGHHmLWrFmRwyE333wz8+fPZ9OmTbS2tuLxeFi3bl1k8ssvfvEL3n77bRwOB6tWraK0tNTOr5BQ3n//fd544w0eeOABOjs71cMxOnDgAM888wx9fX3k5+dzxx13YIxRH8fg9ddfp7a2lrS0NGbPns03vvENenp61MPTePLJJ/nggw/o7OwkNzeXm266iSVLloy5Zx9//DFbtmwhGAxSUlLCV7/61agPMyv4RUREUoh29YuIiKQQBb+IiEgKUfCLiIikEAW/iIhIClHwi4iIpBAFv4jExZ133sm7775rdxkiKU9X7hNJcXfeeSdHjhzB4XCQnp7OggUL+NrXvobH4znt+5qbm1m7di0//elPSUtLi1O1IhItjfhFhPvvv59XXnmFZ599ltzcXF544QW7SxKRcaIRv4hEZGZmcumll/LSSy8B4atVvvbaaxw+fJgpU6Zw5ZVXctNNNwGwfv16AFatWgXAd77zHRYsWEBVVRW/+tWvaGtrw+12c9ddd0VuNnTgwAFefvllWlpaKCkp4c477yQzMxMI35jotddeo6WlhZkzZ/K1r32Nc889F4Dt27fz61//mu7ubpxOJ6tXr2bRokXxbI1I0lDwi0jE8ePHqa2tjdzUZtKkSaxdu5aZM2dy8OBBvv/97zN79mwuvvhivve977F27VpefPHFyK7+3//+9/z85z/nvvvuY+7cuRw+fPikwwC///3v+fa3v01mZibf+c53+I//+A+uvvpq9u3bx9NPP83999/P3Llz2bFjBz/4wQ948sknaWlp4c033+TRRx/F5XLR3NxMKBSypT8iyUDBLyI8/vjjpKWl0dPTQ25uLg8++CAAF1xwQWSdc889l7/6q7/igw8+4OKLLx52O9XV1fz1X/915KYsBQUFJ73++c9/PnJ3sU9/+tMcOHAAgLfeeovy8vLID44rrriCbdu2sWfPHlwuF729vTQ2NpKTk0N+fn5Mv7tIqlHwiwj33Xcfn/rUpwiFQtTV1bF+/Xo2bdpES0sL//RP/0RDQwN9fX309fVx6aWXjrid1tZWZsyYMeLr06dPjzzOzMzE7/dH3ldTU8NvfvObyOt9fX34/X4WLlzIqlWr+PnPf05jYyOLFy/mK1/5im71KnKWFPwiEuFwOLjkkkt47rnn+PDDD3n11Vf57Gc/y7e+9S0yMzN58cUX6ejoABj2DmEej4fDhw+P+XPdbjc33HDDiHe8vOyyy7jssss4duwYzz33HK+++ip33XXXmD9HRDSrX0SGMMZQV1fH0aNHKSoqoru7m+zsbDIzM9m7dy+//e1vI+vm5ORgWdZJQX/VVVfxxhtvsG/fPowxHDp0iJaWljN+7sqVK/n3f/939uzZgzGGnp4edu7cSXd3Nz6fjz/96U/09vaSmZlJZmYmDof+1SVytjTiFxEee+wxHA4HlmWRl5fHnXfeSXFxMatXr+bll1/mhRdeYOHChSxdupSjR48C4Yl/N9xwA9/5znfo7+/n29/+NkuXLqWzs5OnnnoKv99Pfn4+a9euJS8v77SfP3fuXNasWcMLL7xAU1MTmZmZnHfeeZx//vn09vby6quv4vV6SUtL45Of/CRf//rX49EWkaRkGWOM3UWIiIhIfGh/mYiISApR8IuIiKQQBb+IiEgKUfCLiIikEAW/iIhIClHwi4iIpBAFv4iISApR8IuIiKQQBb+IiEgK+f8BzhoamCkpYhYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.xlabel('Batches')\n",
    "plt.ylabel('Iterations')\n",
    "plt.plot(batches, iters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d024065e-8c4a-4526-bff7-44a42f7cab9f",
   "metadata": {},
   "source": [
    "Same issue happens with the computational time, as $b$ increases the spent time is larger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "80d50989-80c6-4f9d-ae1a-b1cd31d6be97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe2cc1e69a0>]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAF2CAYAAAB+h6EdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA44klEQVR4nO3de3xU9Z3/8ddJhoRLSDLJJEASIHIrYrlpuAk2VVPdorZZq7i1bJf60yoItNBaaXdF+2t9iLU01hZFd73VutZaJb+qtd2m2FivG0QL9QYqE5gEch0SQq6T+f7+mCQSSWCSmczJJO/n48FjZs7MnPOZT4tvvufyPZYxxiAiIiJRJ8buAkRERKR/FOIiIiJRSiEuIiISpRTiIiIiUUohLiIiEqUU4iIiIlHKEakNPffcc+zcuRPLspg4cSJr1qyhtbWVgoICqqqqSEtLY8OGDSQkJESqJBERkahmReI68draWm655RYKCgqIi4vjZz/7GWeffTYej4eEhATy8/MpLCykoaGBlStXnnZ95eXlYavN5XJRXV0dtvUNR+ph6NTD8FAfQ6cehi7cPczIyOj1vYjtTvf7/bS2ttLe3k5raytOp5OSkhJyc3MByM3NpaSkJFLliIiIRL2I7E5PSUnhsssuY/Xq1cTFxTF37lzmzp1LXV0dTqcTAKfTSX19fSTKERERGRIiEuINDQ2UlJSwbds2Ro8ezc9+9jNeeumloL9fVFREUVERAFu2bMHlcoWtNofDEdb1DUfqYejUw/BQH0OnHoYukj2MSIjv3buX9PR0EhMTAVi0aBH79u0jKSkJr9eL0+nE6/V2vf9peXl55OXldb0O57EGHf8JnXoYOvUwPNTH0KmHoRtyx8RdLhf79++npaUFYwx79+4lMzOTnJwciouLASguLmbBggWRKEdERGRIiMhIfPr06SxevJibb76Z2NhYsrOzycvLo7m5mYKCAnbu3InL5WLjxo2RKEdERGRIiMglZuGmS8wGF/UwdOpheKiPoVMPQzfkdqeLiIhI+CnERUREopRCXEREJEopxEVERKJUxG6AIiIiMtSZA/to81aCMz0i29NIXEREJEz8v3+C+nvvjNj2FOIiIiLh4nHjmDwlYptTiIuIiISBOX4MjtbgmDwtYttUiIuIiISDpxQAx+SpEdukQlxERCQMTJkbAEe2QlxERCS6eNwwZiwxzsjdylUhLiIiEgbG44asbCzLitg2FeIiIiIhMn4/lB/EysqO6HYV4iIiIqGqroCWZsicHNHNKsRFRERCVRY4M10jcRERkShjPG6wLMiYFNHtKsRFRERCZDxuSBuPFT8yottViIuIiISqrBQivCsdFOIiIiIhMS0tUFmOFeGT2kAhLiIiEprDB8GYiJ/UBgpxERGRkBiPO/AkMzvi21aIi4iIhMLjhrh4SBsX8U0rxEVEREJgykohczJWTGzEt60QFxER6SdjDHjctpzUBgpxERGR/qs/Cg31tlxeBgpxERGR/us4qU0jcRERkShj55npoBAXERHpvzI3JKdgjU20ZfMKcRERkX4yHnfEbz96IoW4iIhIP5j2djh8yJaZ2jo5IrGR8vJyCgoKul5XVlayYsUKcnNzKSgooKqqirS0NDZs2EBCQkIkShIREQlNRRn4fLYdD4cIhXhGRgZ33XUXAH6/n+uvv56FCxdSWFjI7Nmzyc/Pp7CwkMLCQlauXBmJkkRERELSeVKbnSPxiO9O37t3L+PHjyctLY2SkhJyc3MByM3NpaSkJNLliIiI9E9ZKcTGwvgs20qIyEj8RK+88gpLly4FoK6uDqfTCYDT6aS+vr7H7xQVFVFUVATAli1bcLlcYavH4XCEdX3DkXoYOvUwPNTH0KmHwfNWltOeMQnXhAndlkeyhxENcZ/Px5tvvsnVV1/dp+/l5eWRl5fX9bq6ujpsNblcrrCubzhSD0OnHoaH+hg69TB47Qf2Y02deVK/wt3DjIyMXt+L6O70t956izPOOIPk5GQAkpKS8Hq9AHi9XhIT7bnOTkREpC9M43GoqbT18jKIcIifuCsdICcnh+LiYgCKi4tZsGBBJMsRERHpn7JSwN6T2iCCId7S0sKePXtYtGhR17L8/Hz27NnD+vXr2bNnD/n5+ZEqR0REpN9MmTvwxOYQj9gx8fj4eB566KFuy8aOHcvmzZsjVYKIiEh4eNwwagw47T0JUDO2iYiI9JEpK4WsyViWZWsdCnEREZE+MMaAx41l40xtnRTiIiIifVFTCc1Nth8PB4W4iIhI3wySM9NBIS4iItInnXOmkznJ1jpAIS4iItI3ZaXgGoc1crTdlSjERURE+sJ43LbP1NZJIS4iIhIk09YKFWWD4ng4KMRFRESCd/gQ+P0KcRERkWjzyUlt2XaW0UUhLiIiEqyyUhgRB+kTTv/ZCFCIi4iIBMl43DBhIlZsrN2lAApxERGR4JWVDprj4aAQFxERCYqpPwp13kEx3WonhbiIiEgwOqdbHSTXiINCXEREJCimzB14opG4iIhIlPG4YWwSVmKy3ZV0UYiLiIgEwXhKB9UoHBTiIiIip2X87VB+cFCdmQ4KcRERkdOrPAxtrYNmprZOCnEREZHT6TwzXSNxERGR6GI8brBiYEKW3aV0oxAXERE5DeMphXEZWHHxdpfSjUJcRETkdMrcg2qSl04KcRERkVMwzY1QdWTQXV4GCnEREZFTKzsIDL6T2kAhLiIickpd061qd7qIiEiU8ZTCyFGQmm53JSdRiIuIiJyCKXND5mSsmMEXmY5Ibej48eNs376dQ4cOYVkWq1evJiMjg4KCAqqqqkhLS2PDhg0kJCREqiQREZFTMsaAx42Vc57dpfQoYiH+8MMPM2/ePL7zne/g8/loaWlhx44dzJ49m/z8fAoLCyksLGTlypWRKklEROTUvDXQeHxQnpkOEdqd3tjYyHvvvccFF1wAgMPhYMyYMZSUlJCbmwtAbm4uJSUlkShHREQkOB0ntQ3Ga8QhQiPxyspKEhMTuffeeyktLWXKlCmsWrWKuro6nE4nAE6nk/r6+kiUIyIiEhTjCcyZTtYwDvH29nYOHDjANddcw/Tp03n44YcpLCwM+vtFRUUUFRUBsGXLFlwuV9hqczgcYV3fcKQehk49DA/1MXTqYXd11YdpdY0jbVJ20N+JZA8jEuKpqamkpqYyffp0ABYvXkxhYSFJSUl4vV6cTider5fExMQev5+Xl0deXl7X6+rq6rDV5nK5wrq+4Ug9DJ16GB7qY+jUw+7aP/oAJkzsU0/C3cOMjIxe34vIMfHk5GRSU1MpLy8HYO/evWRlZZGTk0NxcTEAxcXFLFiwIBLliIiInJbxtcERz6Ccqa1TxM5Ov+aaa7jnnnvw+Xykp6ezZs0ajDEUFBSwc+dOXC4XGzdujFQ5IiIip3bEA+3tg3Kmtk4RC/Hs7Gy2bNly0vLNmzdHqgQREZGgdZ7UZmWdYXMlvRt808+IiIgMBh43xDpgXO/HpO2mEBcREemBKXPDhIlYjojttO4zhbiIiEhPPKWD+qQ2UIiLiIicxBw/BkdrBu0kL50U4iIiIp/WeVJbZra9dZyGQlxERORTjMcdeKKRuIiISJQpc0PCWEhKsbuSU1KIi4iIfIrxuCEzG8uy7C7llBTiIiIiJzB+P5QfHPRnpoNCXEREpLvqCmhpHtTTrXZSiIuIiJyo46Q2jcRFRESijPG4wbIgY5LdpZyWQlxEROQEpqwU0iZgxY+0u5TTUoiLiIicyOMe9NeHd1KIi4iIdDAtLVB1eNDP1NZJIS4iItKp/CAYg6WRuIiISHQxngOBJ1FwZjooxEVERD5RVgpx8eAab3clQelTiFdXV7Nv376BqkVERMRWgelWJ2PFRMcY1xHMh6qrq/n5z3+O2+0G4LHHHuP111/n7bff5oYbbhjI+kRERCLCGANlbqz5S+wuJWhB/VPjgQceYP78+Tz66KM4HIHcnzNnDnv27BnQ4kRERCKmzgsNx6JiutVOQYX4hx9+SH5+PjEn7F4YPXo0jY2NA1aYiIhIREXRdKudggrxpKQkjhw50m2Zx+PB5XINSFEiIiKRZspKA0+iaCQe1DHxyy67jDvvvJP8/Hz8fj8vv/wyO3bsID8/f4DLExERiRCPG5JTsBIS7a4kaEGF+AUXXEBCQgJ/+ctfSE1Npbi4mKuuuoqFCxcOdH0iIiIRYcrcUXN9eKegQhxg4cKFCm0RERmSjM8Hhw9hzZpndyl9EnSIv/feexw4cIDm5uZuyy+//PKwFyUiIhJRleXg8w3NkfhDDz3Ea6+9xsyZM4mLi+tablnWgBUmIiISKSYKz0yHIEP8b3/7G1u3biUlJWWg6xEREYk8jxtiY2F8lt2V9ElQl5i5XC5GjBgx0LWIiIjYwpSVwvgsLEd0ZV1QI/EbbriB+++/n6VLl5KUlNTtvVmzZgW1oRtvvJGRI0cSExNDbGwsW7ZsoaGhgYKCAqqqqkhLS2PDhg0kJCT0/VeIiIiEwuPGmnam3VX0WVAh/vHHH/PWW2/x3nvvdTsmDnDfffcFvbFbb72VxMRPrr8rLCxk9uzZ5OfnU1hYSGFhIStXrgx6fSIiIqEyjcehtgqyvmh3KX0WVIg/8cQT3HzzzcyZMyesGy8pKeG2224DIDc3l9tuu00hLiIikdUxU1u0ndQGQYZ4fHx80LvNT+X2228H4Atf+AJ5eXnU1dXhdDoBcDqd1NfX9/i9oqIiioqKANiyZUtYp3t1OByaPjZE6mHo1MPwUB9DNxx72FhSxTEgZfZ8YsPw2yPZw6BC/KqrruKRRx7hiiuu6LY7HOh2U5RT+dGPfkRKSgp1dXX8+Mc/JiMjI+gi8/LyyMvL63pdXV0d9HdPx+VyhXV9w5F6GDr1MDzUx9ANxx76P3gHRo+h1sRgheG3h7uHp8rLoEK887j3n//855Pee/LJJ4MqovPytKSkJBYsWMCHH35IUlISXq8Xp9OJ1+s96R8IIiIiA8143JA5OSrnPgkqxH/5y1+GtJHm5maMMYwaNYrm5mb27NnDFVdcQU5ODsXFxeTn51NcXMyCBQtC2o6IiEhfGGOgrBRryfl2l9IvQYV4WlpaSBupq6vjpz/9KQDt7e0sW7aMefPmMXXqVAoKCti5cycul4uNGzeGtB0REZE+qamE5qaom261U68hfv/993P99dcD8Itf/KLX3Qxr16497UbGjRvHXXfdddLysWPHsnnz5mBrFRERCa/O6VYzs20to796DfH09PSu5+PHj49IMSIiIpFkOi4vI3OSvYX0U68h/s///M9dz7/whS+QnJx80meOHj06EDWJiIhEhscNrnFYI0fbXUm/BHV92Le+9a0el2/YsCGsxYiIiESSKSuN2uPhEGSIG2NOWtbY2Bj0NeIiIiKDjWlrhSNlWJmT7S6l3055dvrq1asBaG1t7XreqaGhgaVLlw5cZSIiIgOp/BAYf1ROt9rplCG+bt06jDHccccdrFu3rtt7ycnJfZp1TUREZDAxZe7Ak6Ea4p3zpT/44IPEx8dHpCAREZGI8LhhRBykT7C7kn4L6qC2AlxERIYaU1YKGZOwYmLtLqXfdGaaiIgMTx53VJ/UBgpxEREZhkz9Uag/GtXHw0EhLiIiw1HHTG3RfGY6nOLEtlPNl36iYOZOFxERGUxMx5zpRPnu9F5DXPOli4jIkFXmhsRkrMRkuysJSa8hfuWVV0ayDhERkYgxntKoH4VDkPcTB/D5fJSXl1NfX99t+Wc/+9mwFyUiIjJQjL8dyg9iff6LdpcSsqBC/P333+dnP/sZbW1tNDU1MWrUKJqbm0lNTeWXv/zlQNcoIiISPpWHoa016s9MhyDPTn/00Uf50pe+xMMPP8yoUaN4+OGH+cpXvsJFF1000PWJiIiEV8dJbVZmtq1lhENQIV5eXs7y5cu7LcvPz+f5558fkKJEREQGiikrBSsGMibaXUrIggrx0aNH09TUBARufOLxeGhoaKC5uXlAixMREQk343HDuAysEXF2lxKyoI6JL1q0iLfeeotly5ZxwQUX8MMf/pDY2FiWLFky0PWJiIiEl8eNlT3d7irCIqgQX7VqVdfzyy67jGnTptHc3MzcuXMHqi4REZGwM82NUF0By75gdylhEfQlZic688wzw12HiIjIwCs7CBD1Nz7pFFSIV1ZW8sQTT+B2u086Dn7fffcNSGEiIiLhZsrcgSdD4PIyCDLEf/7znzNu3Di+/vWv697iIiISvTxuGDkKUtPtriQsggpxj8fDj370I2JidNMzERGJXsbjhszJQd3gKxoElcpnnnkmbrd7gEsREREZOMYYKCuN+tuPniiokXhaWhq33347CxcuJDk5udt7V1111UDUJSIiEl7eamg8DkNgprZOQYV4S0sL55xzDu3t7dTU1Ax0TSIiIuFXVgow/Ebia9asGeg6REREBpTpmDOdzEm21hFOQV8nfvjwYV555RVqa2tJSUlh6dKlTJgwoU8b8/v9bNq0iZSUFDZt2kRDQwMFBQVUVVWRlpbGhg0bSEhI6POPEBEROS2PG1LSsEYPnZwJ6sS2Xbt2sWnTJsrKykhISKC8vJxNmzaxa9euPm3sD3/4A5mZmV2vCwsLmT17Nvfccw+zZ8+msLCwT+sTEREJlikrHTLXh3cKKsSfeOIJbrrpJr71rW9x9dVXs379er73ve/xxBNPBL2hmpoadu/ezYUXXti1rKSkhNzcXAByc3MpKSnpY/kiIiKnZ3xtcMQzZGZq6xRUiNfW1p401erMmTP7dJLbI488wsqVK7tdm1dXV4fT6QTA6XRSX18f9PpERESCdsQD7e1DbiQe1DHx7Oxsnn32WfLz87uWPffcc2RnZwe1kTfffJOkpCSmTJnCO++80+cii4qKKCoqAmDLli24XK4+r6M3DocjrOsbjtTD0KmH4aE+hm6o9rDpH7uoB1I+Ow/HAP++SPYwqBC/9tprufPOO3nhhRdITU2lpqaG+Ph4vve97wW1kQ8++IBdu3bx1ltv0draSlNTE/fccw9JSUl4vV6cTider5fExMQev5+Xl0deXl7X6+rq6qC2GwyXyxXW9Q1H6mHo1MPwUB9DN1R76H9/LzgceONGYw3w7wt3DzMyMnp9L6gQz8zMpKCggH379uH1eklJSWHatGk4HMGd3H711Vdz9dVXA/DOO+/w7LPPsn79eh577DGKi4vJz8+nuLiYBQsWBLU+ERGRvjBlpTBhIlaQuRUtgv41sbGxYb8FaX5+PgUFBezcuROXy8XGjRvDun4REREAPG6smXPtriLseg3xDRs2UFBQAMDq1at7XUFfb0V61llncdZZZwEwduxYNm/e3Kfvi4iI9IVpqIejtUPupDY4RYhff/31Xc/XrVsXkWJERETCrnO61SF2eRmcIsRnzpzZ9byuro4lS5ac9JnXX399YKoSEREJk67pVofgSDyo68S3b9/e4/L7778/rMWIiIiEXVkpJCRCktPuSsLulCe2VVRUAIE5zysrKwP3Yj3hvbi4uIGtTkREJETG44bMyd0mGxsqThni69ev73r+6ePiycnJXHnllQNTlYiISBgYvx/KSrHOu8juUgbEKUP8ySefBODWW2/lhz/8YUQKEhERCZvqI9DaAkPwpDYI8pi4AlxERKKSp+PM9KwzbC5kYAQ12Ut7ezt/+tOfePfddzl27Fi39xTwIiIyWBmPGywLMibZXcqACGok/uijj1JUVMSsWbP4+OOPWbRoEXV1dV2TtoiIiAxGpswNaROw4uPtLmVABBXib7zxBj/4wQ9Yvnw5sbGxLF++nJtuuqlfdyQTERGJGE/pkLw+vFNQId7a2kpqaioAcXFxtLS0kJmZidvtHsjaRERE+s20NEPV4SE5U1unoO9i9tFHHzFt2jSmTJnCU089xahRo0hJSRno+kRERPqn/BAYgzXcR+KrVq0iJibw0X/7t3/jwIEDvPnmm3zzm98c0OJERET6y3gOBJ4M4RAPaiTucrlITk4GYMKECdxyyy0AHD16dKDqEhERCU1ZKcSPBNc4uysZMEGNxL/1rW/1uHzDhg1hLUZERCRcjMcNGZOwYoKKuqgU1C87cc70To2NjV272EVERAYTYwyUuYf08XA4ze701atXA4Gz0zufd2poaGDp0qUDV5mIiEh/1Xmh4RhkZttdyYA6ZYivW7cOYwx33HFHjzdAycjIGNDiRERE+qXjHuJW1tC9vAxOE+KzZs0C4MEHHyR+iM52IyIiQ48pcweeDOFrxCHIs9MLCwt7fe+qq64KVy0iIiLh4SmF5FSshES7KxlQQYV4TU1Nt9dHjx7l3XffZeHChQNSlIiISCiMxw1DfFc6BBnia9asOWnZ22+/zcsvvxz2gkREREJhfD44cgjrrPl2lzLg+n2N2Jw5cygpKQlnLSIiIqGrKAefTyPxThUVFd1et7S08PLLL+NyuQakKBERkf7qnG51qF8jDkGG+Pr167u9jouL44wzzuDGG28ckKJERET6rawUYmNhfJbdlQy4oEL8ySefHOg6REREwsJ43DA+C8sxwu5SBlxQIQ7g9/vZt28fXq+XlJQUpk+frmlXRURk8CkrxZo2y+4qIiKoEC8tLeWuu+6ira2NlJQUamtrGTFiBN/97nfJzs4e4BJFRESCYxoboLZqWJzUBkGG+H333cfFF1/MpZdeimVZGGN4/vnnue+++7jzzjsHukYREZHgeEqB4XFSGwR5idnhw4e55JJLsCwLAMuyWL58OUeOHBnQ4kRERPrClAVCfKjf+KRTUCPx+fPns2vXrm4ztO3atYv584O7kL61tZVbb70Vn89He3s7ixcvZsWKFTQ0NFBQUEBVVRVpaWls2LCBhISE/v0SERERjxtGjwFnqt2VRERQIe73+7n77ruZMmUKqamp1NTU8PHHH5OTk8Mvf/nLrs+tXbu2x++PGDGCW2+9lZEjR+Lz+di8eTPz5s3jf//3f5k9ezb5+fkUFhZSWFjIypUrw/PLRERk2DFlbsjK7tpzPNQFFeITJ05k4sSJXa+zsrKYO3du0BuxLIuRI0cC0N7eTnt7O5ZlUVJSwm233QZAbm4ut912m0JcRET6xfj9gTPTl5xvdykRE1SIX3nllSFvyO/3c/PNN3PkyBEuvvhipk+fTl1dHU6nEwCn00l9fX2P3y0qKqKoqAiALVu2hHWmOIfDoZnnQqQehk49DA/1MXTR3MP2inKqm5tImPlZRtv4GyLZw6CvE6+srOTgwYM0Nzd3W75s2bKgvh8TE8Ndd93F8ePH+elPf8rBgweDLjIvL4+8vLyu19XV1UF/93RcLldY1zccqYehUw/DQ30MXTT30Ox9C4DjSS4abfwN4e5hRkZGr+8FFeI7duzgd7/7HRMnTiQuLq5ruWVZQYd4pzFjxjBr1izefvttkpKS8Hq9OJ1OvF4viYlD+76vIiIycIzHHXiSOcnWOiIpqBB/7rnnuPPOO8nK6t88tPX19cTGxjJmzBhaW1vZu3cvX/7yl8nJyaG4uJj8/HyKi4tZsGBBv9YvIiJCWSmkjccaOdruSiImqBBPSEggLS2t3xvxer1s27YNv9+PMYYlS5ZwzjnnMGPGDAoKCti5cycul4uNGzf2exsiIjK8GY8bMofHTG2dggrxVatWcf/993PJJZeQlJTU7b1gDt5PnjyZn/zkJyctHzt2LJs3bw6yVBERkZ6Z1haoKMfKWWp3KREVVIj7fD727NnDK6+8ctJ7usOZiIjY7rAHjH/YTLfaKagQ/6//+i+++tWvsnTp0m4ntomIiAwGn5zUpt3pJ/H7/Zx//vm69aiIiAxOZW4YEQfpE+yuJKKCSuXLLruMwsJCjDEDXY+IiEifGY8bMiZhxcTaXUpEBTUSf+GFFzh69Cg7duw46QYl991334AUJiIiEjSPG2tOjt1VRFxQIb5u3bqBrkNERKRfTL0XjtXBMDupDYIM8VmzZg10HSIiIv3jCdxD3Bom9xA/UdCXmD3zzDO89NJLXdOkfu5zn+Pyyy/H4Qh6+nUREZGwM2WBENdIvBe//vWv+eijj7juuutIS0ujqqqKp59+msbGRlatWjXAJYqIiJyCxw2JyVhjk0770aEmqBB//fXXueuuuxg7diwQuKPKGWecwU033aQQFxERWxmPe1iOwiHIS8x0aZmIiAxGpr0dDh8adjO1dQpqJL5kyRLuvPNOrrjiiq77pD799NMsWbJkoOsTERHpXeVhaGsddjO1dQoqxFeuXMnTTz/Ngw8+iNfrJSUlhXPPPZevfOUrA12fiIhI78rcABqJn/JDDgdXXXUVV1111UDXIyIiEjTjcUNMDEyYaHcptjjlMfH333+fX//61z2+9/jjj7Nv374BKUpERCQYxuOGcZlYI4bnzblOGeI7duzodaKXWbNm8cwzzwxIUSIiIkEpKx22u9LhNCHudruZN29ej+/NmTOHAwcODERNIiIip2WaG6G6Ytie1AanCfGmpiZ8Pl+P77W3t9PU1DQgRYmIiJxW2UFg+J7UBqcJ8czMTP7+97/3+N7f//53MjMzB6QoERGR0zEed+CJRuI9u+SSS3jggQd444038Pv9APj9ft544w3+8z//k0suuSQiRYqIiJzE44aRoyA13e5KbHPKS8yWLVvG0aNH2bZtG21tbSQmJlJfX09cXBxXXnkly5Yti1SdIiIi3ZgyN2RlY1mW3aXY5rTXiV966aVccMEF7Nu3j4aGBhISEpgxYwajR4+ORH0iIiInMcaApxRr4Xl2l2KroCZ7GT16dK9nqYuIiESctxqajg/bG590CuoGKCIiIoNKx0ltVma2rWXYTSEuIiJRR2emByjERUQk+pSVQmo61ugxdldiK4W4iIhEHeNxD/tROCjERUQkypi2NqgoG9YztXVSiIuISHQ54oH2do3ECfISs1BVV1ezbds2jh49imVZ5OXlsXz5choaGigoKKCqqoq0tDQ2bNhAQkJCJEoSEZEoZcrcwPCeM71TREI8NjaWf/3Xf2XKlCk0NTWxadMm5syZw1//+ldmz55Nfn4+hYWFFBYWsnLlykiUJCIi0crjBocD0jPsrsR2Edmd7nQ6mTJlCgCjRo0iMzOT2tpaSkpKyM3NBSA3N5eSkpJIlCMiIlHMeNwwYSKWIyLj0EEt4sfEKysrOXDgANOmTaOurg6n0wkEgr6+vj7S5YiISLQpK9Wu9A4R/WdMc3MzW7duZdWqVX2ae72oqIiioiIAtmzZgsvlCltNDocjrOsbjtTD0KmH4aE+hm6w99BfX0fV0VrGzDiLMYO0zkj2MGIh7vP52Lp1K+eddx6LFi0CICkpCa/Xi9PpxOv1kpiY2ON38/LyyMvL63pdXV0dtrpcLldY1zccqYehUw/DQ30M3WDvoflgLwCNzjSaBmmd4e5hRkbvx/4jsjvdGMP27dvJzMzk0ksv7Vqek5NDcXExAMXFxSxYsCAS5YiISJTSdKvdRWQk/sEHH/DSSy8xadIkbrrpJgC++tWvkp+fT0FBATt37sTlcrFx48ZIlCMiItHK44aEREhy2l3JoBCREJ85cya//e1ve3xv8+bNkShBRESGAFNWClnZWJZldymDgmZsExGRqGD8/sCZ6dqV3kUX2YmIyKBjjAFfG7S2QGsrtLVA5ZHAa11e1kUhLiIiQTHGQFvrCcHa+bzjT8dr09ra7XVXCHc+b23BtJ3mM22tYEyPdViTpkb4lw9eCnEREcHUH8X8/r+pramkvfF4V9h2hWxbR7j2hxUDcXEQFw8jOh47X8fFQ0IiVueyEfG9ftYaEQ+JyViTpoT3x0cxhbiIyDBmjMG8UYx58j+hqQmmz4KRo2Bs0ifBGhffEa6fDti4wGd6fO+E17EOnYg2QBTiIiLDlKmtxv/re2HvLjhjBjGr1pMy5+xBPdmLdKcQFxEZZozfj/nb/2B+9zD427FW/B+sCy/Fiom1uzTpI4W4iMgwYirL8f9qG3ywF2bOIebra7HSxttdlvSTQlxEZBgw/nZM0e8x/+/xwDHqr6/FWvYFHauOcgpxEZEhzpSV4n/0F3BgH8xdSMzXVmM5U+0uS8JAIS4iMkQZXxvmD7/D/OEpGDUa67rvYi04T6PvIUQhLiIyBJkD+/E/ek9gmtKFuVj/ch3W2J5v9yzRSyEuIjKEmJYWzO8fx/z595DkJGbtLVhzdZvnoUohLiIyRJgP/oH/V7+AysNYn7sY6yursEaPsbssGUAKcRGRKGeaGjFPP4Ip/iOkjSfmOz/GmjnH7rIkAhTiIiJRzOzdhf+xe+FoLdYXvoz15ZVY8fF2lyURohAXEYlC5lg95sn/xLxRDBmTiLnhZqwpn7G7LIkwhbiISBQxxmB2vYx54gFobMC67F+wll+J5Rhhd2liA4W4iEiUMEdr8D++Hd5+AyZPI2bjj7Cysu0uS2ykEBcRGeSMMZiX/4x56mHwtWFd8Q2svC9hxeqGJcOdQlxEZBAzVUfwP7YN3vs7zDiLmK+vwxqXYXdZMkgoxEVEBiHjb8fsfB6z4zGIicH62urAtd8xMXaXJoOIQlxEZJAxhw8Fbljy0fswO4eYlauxUtLsLksGIYW4iMggYXw+zJ+ewTz3Gxg5Cuv/bMRalKsblkivFOIiIoOAKf0I/yP3gOcAVs4yrK9+Eysx2e6yZJBTiIuI2Mi0tmCe/Q3mf3bA2GRi1vwAa/5iu8uSKKEQFxGxidn/buDYd0UZ1rIvBC4dG5Ngd1kSRRTiIiIRZpobMc/8CvPiHyA1nZgN/xdr1jy7y5IopBAXEYkg84/dgeu+vdVYF16Glb8Sa+Qou8uSKKUQFxEJgfH7od0HbW3dH31t4Ov+aF57EfPaThifRcz3tmBNO9Pu8iXKRSTE7733Xnbv3k1SUhJbt24FoKGhgYKCAqqqqkhLS2PDhg0kJOhYkIj0jfG3g7cWaqto8cRjamoDgdkZnu1t0Nb52HvA4vN98h1f22mC+YTPtLcHX2xsLNbyFViXrsAaETdwTZFhIyIh/vnPf55/+qd/Ytu2bV3LCgsLmT17Nvn5+RQWFlJYWMjKlSsjUY6IRBlzvAGqj0B1BabqCFRVYKorAstqqgJhChwNdoUOB8SOgBEdjw4HOHp4HD0m8BjrCNwlrLfPffox1gEjRmB1PBLbsTw1DSs1faDaJMNQREJ81qxZVFZWdltWUlLCbbfdBkBubi633XabQlxkmDJtbVBTGQjp6k+FdFUFNB3v/oWEsZA6DmvSVDjnXHCNw0pJJ2n8BOoaGk4bsJo8RYYK246J19XV4XQ6AXA6ndTX1/f62aKiIoqKigDYsmULLpcrbHU4HI6wrm84Ug9DN9R7aIzB762hvaL8kz+V5bQfCTz311aBMZ98YUQcsekTiB2XQexZ84kdN4HYcZmB1+MyiBk9psftOBwO4ny+CP2qoWmo/38xEiLZw6g4sS0vL4+8vLyu19XV1WFbt8vlCuv6hiP1MHRDoYemuRGqKzp2eVdA1ZGO0XQF1FRAa2v3LySnBkbQM87Cco0D13istPHgGgdJToiJoR046YhzY1PgTw+GQh/tph6GLtw9zMjo/a51toV4UlISXq8Xp9OJ1+slMTHRrlJEJAimvR1qqzp2eQdCutvzhk/tTRs5ClzjYXwm1mfPhrTxWK6OkHal68QukTCwLcRzcnIoLi4mPz+f4uJiFixYYFcpInIC4/cHjkV7SjEeN6asFDzuwDK//5MPxsZCSlpgNH32ko5wHh8YVaeNgzFjdexZZIBFJMTvvvtu3n33XY4dO8YNN9zAihUryM/Pp6CggJ07d+Jyudi4cWMkShGRE5iGeugM6rJAaFNWCq0tgQ9YFqSNh6xsrHPO7RhNjwssc7qwYmNtrV9kuItIiH/729/ucfnmzZsjsXmRYc+0tcHhQ12jalPmBk8p1NV+8qGEsZCZjXXeRZA5GSsrGzImYcWPtKtsETmNqDixTUSCY4wJHLf2lGI8Bz4ZXVeUfbIr3OGACROxZs0NhHZWNmROhiSndn+LRBmFuEiUMo3HAyFd5u6+K7yp8ZMPpaYHdoXPXxx4zJwM6RlYDv3VFxkK9DdZZJAzPh9UlHfsAj/hRLPaqk8+NGp0YFS9KLdjdD0ZMiZj9XI9tYgMDQpxkUHCGBM4Ru0p7TpmbTxuOHIoMFc3QEwMjM/CmjoTcv8JKzMbsrIhxaVd4SLDkEJcxEbG14Z5/a/UvvkKfvd+aDj2yZvJKYFd4GfN69gVnh0I8BEj7CpXRAYZhbiIDUxbK+aVIswLT0NtFf6JZ2DNX/LJrvDMyVgJmgBJRE5NIS4SQaalBfO3P2L+tAOO1sKUzxCzcjWpn7+Ympoau8sTkSijEBeJANPciHnxBcyfC+FYHcz4LDHXbICZc7AsS8ezRaRfFOIiA8g0NmB2PocpehaOH4NZ84m5ZAXWjLPsLk1EhgCFuMgAMMfqMUW/x7z4XOC67bkLA+F9xgy7SxORIUQhLhJGps6L+Z9CTPELgfnHz15CzPIVWJOm2F2aiAxBCnGRMDC11Zg/PYP52/+Az4e18Dys5VdiZUyyuzQRGcIU4iIhMFVHMH98BvNqERiDtfjzWF+8Emtcht2licgwoBAX6QdTUY75w1OY11+EmBispXlY//SVwG06RUQiRCEu0gem7CDmD7/FlLwMDgfW+ZdgXfTPWCkuu0sTkWFIIS4SBHPwI/zP/xZ2vwbxI7Eu+jLWRflYiU67SxORYUwhLnIK5uMPAuG9pwRGjca6ZAVW3pc0JaqIDAoKcZEemH3/CIT3u2/DmLFYX/4a1gWXYI1OsLs0EZEuCnGRDsYYeO/v+J9/Eva9A2OTsK5YhZX7RayRo+wuT0TkJApxGfaMMbB3F/7nnoQD+yA5Beuqa7HOuxgrPt7u8kREeqUQl2HL+P3w9uuB3eYHP4bUdKyvrQ5cLqZ7dotIFFCIy7Bj/O2Ykpcxf3gKyg9C+gSsVeuxFn0ey6G/EiISPfRfLBk2jM+H+d9izB9+BxVlMGEi1rXfwcpZhhUba3d5IiJ9phCXIc+0tWFe+wvmhaehugKyziDmhpth/hKsmBi7yxMR6TeFuAxapq0NWpqgueNPS3PXc9OX5Q31gcfs6cT8y3UwZwGWZdn980REQqYQl7AwxkBba4/BSksTpk/LO563+4LbeEwMjBwF8aMCjyNHQfzIwCVi8SMDk7TMWwyz5im8RWRIUYjLaRmfD7zVUFuNqamE2kqoqQo8r6misqEe09wIfn9wK3Q4Tg7ckaMCl3Z1Pv9UKPe2nPiRMCJO4Swiw5JCXDAtzVBbBTWVmJrAIzVVmI6w5mgtmE8FdGJy4JKsiWcwcvwEmo3VLVit3gJ35Cgshy7fEhEJB4X4EGeMgePHAmFcU9kVzJ2jaGoroeFY9y/FxkJyaiCkZ86G1HRIScPqeCQ1DWtEXNfHE10uWqurI/zLRETE9hB/++23efjhh/H7/Vx44YXk5+fbXVJUMf52OOqF2k+Pojue11YFjjGfKC4+EMypaVjZ0yHFFQjsjmUkp2DF6JIrEZHBztYQ9/v9PPjgg/zHf/wHqampfP/73ycnJ4esrCw7yxo0jDHga4Pa6o5RdNUJI+qOkPZWQ3t79y+OGRsI43GZWLPmdQR0WseIOh0SxuoYsojIEGBriH/44YeMHz+ecePGAXDuuedSUlISsRD3v/Qnat/4K+1tbWDMCX/8YPjkOHDnCVsnLu/2eQMY8Hc8fvq9T3/uVO+duI6eWBYkpQRG0VM+A6nLIOWEUXRKmm7WISIyTNga4rW1taSmpna9Tk1NZf/+/ZErwOHAGj0G2trAigkE5Il/sCAm8Gh9+r1ePkfn5CE9fqbzvRiwenk8cV2WFTg+7UzFSukYSTtTdWKYiIgANoe46WG02dNu3qKiIoqKigDYsmULLpcrPAV86Socl38Nny/I65GlRw6HI3z/mwxT6mF4qI+hUw9DF8ke2hriqamp1NTUdL2uqanB6XSe9Lm8vDzy8vK6XleH8Uxol8sV1vUNR+ph6NTD8FAfQ6cehi7cPczIyOj1PVsnjp46dSqHDx+msrISn8/Hq6++Sk5Ojp0liYiIRA1bR+KxsbFcc8013H777fj9fs4//3wmTpxoZ0kiIiJRw/brxM8++2zOPvtsu8sQERGJOroPo4iISJRSiIuIiEQphbiIiEiUUoiLiIhEKYW4iIhIlFKIi4iIRCmFuIiISJRSiIuIiEQphbiIiEiUskxPtxITERGRQW/Yj8Q3bdpkdwlRTz0MnXoYHupj6NTD0EWyh8M+xEVERKKVQlxERCRKDfsQz8vLs7uEqKcehk49DA/1MXTqYegi2UOd2CYiIhKlhv1IXEREJFo57C7ALm+//TYPP/wwfr+fCy+8kPz8fLtLGpSqq6vZtm0bR48exbIs8vLyWL58OQ0NDRQUFFBVVUVaWhobNmwgISEBgB07drBz505iYmL4xje+wbx58+z9EYOE3+9n06ZNpKSksGnTJvWwH44fP8727ds5dOgQlmWxevVqMjIy1Mc+eO6559i5cyeWZTFx4kTWrFlDa2urengK9957L7t37yYpKYmtW7cC9Ovv78cff8y2bdtobW1l/vz5fOMb38CyrNCKM8NQe3u7Wbt2rTly5Ihpa2sz3/3ud82hQ4fsLmtQqq2tNR999JExxpjGxkazfv16c+jQIfPYY4+ZHTt2GGOM2bFjh3nssceMMcYcOnTIfPe73zWtra2moqLCrF271rS3t9tV/qDy7LPPmrvvvtvccccdxhijHvbDL37xC1NUVGSMMaatrc00NDSoj31QU1Nj1qxZY1paWowxxmzdutW8+OKL6uFpvPPOO+ajjz4yGzdu7FrWn55t2rTJfPDBB8bv95vbb7/d7N69O+TahuXu9A8//JDx48czbtw4HA4H5557LiUlJXaXNSg5nU6mTJkCwKhRo8jMzKS2tpaSkhJyc3MByM3N7epfSUkJ5557LiNGjCA9PZ3x48fz4Ycf2lb/YFFTU8Pu3bu58MILu5aph33T2NjIe++9xwUXXACAw+FgzJgx6mMf+f1+WltbaW9vp7W1FafTqR6exqxZs7pG2Z362jOv10tTUxMzZszAsiw+97nPhSV3huXu9NraWlJTU7tep6amsn//fhsrig6VlZUcOHCAadOmUVdXh9PpBAJBX19fDwR6O3369K7vpKSkUFtba0u9g8kjjzzCypUraWpq6lqmHvZNZWUliYmJ3HvvvZSWljJlyhRWrVqlPvZBSkoKl112GatXryYuLo65c+cyd+5c9bAf+tqz2NjYk3InHL0cliNx08MJ+SEflxjimpub2bp1K6tWrWL06NG9fq6n3g53b775JklJSV17NE5HPexZe3s7Bw4c4KKLLuInP/kJ8fHxFBYW9vp59fFkDQ0NlJSUsG3bNu6//36am5t56aWXev28eth3vfVsoHo5LEfiqamp1NTUdL2uqanp+heVnMzn87F161bOO+88Fi1aBEBSUhJerxen04nX6yUxMRE4ube1tbWkpKTYUvdg8cEHH7Br1y7eeustWltbaWpq4p577lEP+yg1NZXU1NSuUc7ixYspLCxUH/tg7969pKend/Vo0aJF7Nu3Tz3sh772rKfcCUcvh+VIfOrUqRw+fJjKykp8Ph+vvvoqOTk5dpc1KBlj2L59O5mZmVx66aVdy3NyciguLgaguLiYBQsWdC1/9dVXaWtro7KyksOHDzNt2jRbah8srr76arZv3862bdv49re/zWc/+1nWr1+vHvZRcnIyqamplJeXA4FAysrKUh/7wOVysX//flpaWjDGsHfvXjIzM9XDfuhrz5xOJ6NGjWLfvn0YY3jppZfCkjvDdrKX3bt38+ijj+L3+zn//PO5/PLL7S5pUHr//ffZvHkzkyZN6jrk8NWvfpXp06dTUFBAdXU1LpeLjRs3dp348cwzz/Diiy8SExPDqlWrmD9/vp0/YVB55513ePbZZ9m0aRPHjh1TD/vI7Xazfft2fD4f6enprFmzBmOM+tgHv/3tb3n11VeJjY0lOzubG264gebmZvXwFO6++27effddjh07RlJSEitWrGDBggV97tlHH33EvffeS2trK/PmzeOaa64J+VDusA1xERGRaDcsd6eLiIgMBQpxERGRKKUQFxERiVIKcRERkSilEBcREYlSCnER6bMbb7yRPXv22F2GyLA3LGdsExmqbrzxRo4ePUpMTAwOh4MZM2Zw3XXX4XK5Tvm9yspK1q5dyxNPPEFsbGyEqhWRUGkkLjLE3HzzzTz22GPcf//9JCUl8dBDD9ldkogMEI3ERYaouLg4Fi9ezKOPPgoEZin8zW9+Q0VFBaNHj+b8889nxYoVANx6660ArFq1CoBbbrmFGTNmUFRUxPPPP09NTQ2pqamsW7eu60YubrebX/3qV1RVVTFv3jxuvPFG4uLigMBNX37zm99QVVVFVlYW1113HZMnTwagsLCQF154gaamJpxOJ9deey2zZ8+OZGtEhgyFuMgQ1dLSwquvvtp1w5D4+HjWrl1LVlYWhw4d4sc//jHZ2dksXLiQH/7wh6xdu5ZHHnmka3f6a6+9xlNPPcVNN93E1KlTqaio6Lar/bXXXuMHP/gBcXFx3HLLLfz1r3/loosu4uOPP+a+++7j5ptvZurUqbz00kv85Cc/4e6776aqqoo//elP3HHHHaSkpFBZWYnf77elPyJDgUJcZIi56667iI2Npbm5maSkJP793/8dgLPOOqvrM5MnT2bp0qW8++67LFy4sMf17Ny5ky9/+ctdN7wYP358t/e/+MUvdt2F6ZxzzsHtdgPwl7/8hby8vK5/PHz+859nx44d7N+/n5SUFNra2vB4PCQmJpKenh7W3y4y3CjERYaYm266iTlz5uD3+ykpKeHWW2+loKCAqqoq/vu//5uDBw/i8/nw+XwsXry41/VUV1czbty4Xt9PTk7ueh4XF0dtbW3X94qLi/njH//Y9b7P56O2tpZZs2axatUqnnrqKTweD3PnzuXrX/+6bm8p0k8KcZEhKiYmhkWLFvHAAw/w/vvv8/jjj3PxxRfz/e9/n7i4OB555BHq6+sBeryTksvloqKios/bTU1N5fLLL+/1zoDLli1j2bJlNDY28sADD/D444+zbt26Pm9HRHR2usiQZYyhpKSE48ePk5mZSVNTEwkJCcTFxfHhhx/y8ssvd302MTERy7K6hfYFF1zAs88+y8cff4wxhiNHjlBVVXXa7V544YX8+c9/Zv/+/RhjaG5uZvfu3TQ1NVFeXs4//vEP2traiIuLIy4ujpgY/WdIpL80EhcZYu68805iYmKwLIu0tDRuvPFGJk6cyLXXXsuvfvUrHnroIWbNmsWSJUs4fvw4EDjp7fLLL+eWW26hvb2dH/zgByxZsoRjx47x85//nNraWtLT01m7di1paWmn3P7UqVO5/vrreeihhzh8+DBxcXHMnDmTM888k7a2Nh5//HHKysqIjY3lM5/5DN/85jcj0RaRIUn3ExcREYlS2o8lIiISpRTiIiIiUUohLiIiEqUU4iIiIlFKIS4iIhKlFOIiIiJRSiEuIiISpRTiIiIiUUohLiIiEqX+PxF9BTpTHbcrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.xlabel('Batches')\n",
    "plt.ylabel('Computational time')\n",
    "plt.plot(batches, times)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95473bd1-ba5e-4edf-a2c6-5bccccca6043",
   "metadata": {},
   "source": [
    "But if the error evolution is studied, it can be seen that this one is also decreased exponentially if a larger batch is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0df3b430-3178-4761-9758-202cfbb50edd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe2cc15da90>]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfEAAAF2CAYAAAB+h6EdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxlElEQVR4nO3de3jU9Z3//dcnmQQIMSGZScBwEIGMgUQ5hJNB8UC23fZut9Z7q1W7/tC1VQRdtR6o6+HyutZKWSloi6K/Ba3errXurljcajUFYTXaDUSUJApBjgqSE6dAzvO5/xiIRJIwyczkO5N5Pq6Lq5nvnF68rb7yPRtrrRUAAIg6cU4HAAAAvUOJAwAQpShxAACiFCUOAECUosQBAIhSlDgAAFGKEgcAIEq5nA7QG/v27QvZZ3k8HtXU1ITs82IRMwweMwwN5hg8Zhi8UM8wKyury+dYEwcAIEpR4gAARClKHACAKEWJAwAQpShxAACiFCUOAECUosQBAIhSlDgAAFGKEgcAIEpR4gAARClKHACAKBXTJW7rqtVU8p7TMQAA6JXYLvF339ShX/1CtqnR6SgAAPRYTJe48eZKbW3S5585HQUAgB6L6RLXuPFSXLzstjKnkwAA0GMxXeJmYJJcY7yyleVORwEAoMdiusQlKTF3krRjm2xLs9NRAADoEUo8d7LU2iLt3OZ0FAAAeiTmSzxh/AWSMewXBwBEnZgv8bjkFGn4aNlt7BcHAESXmC9x6cSpZp9/Ktva4nQUAAACRolLMt48qblZ2v2501EAAAgYJS5J2RMkiU3qAICoQolLMilDpLNHcnAbACCqUOInmOxcaXuFrK/N6SgAAASEEj/Jmys1Nkh7dzqdBACAgFDiJxhvniT2iwMAogclfoJJc0sZw9gvDgCIGpT4KYw3T6qskPX5nI4CAMAZUeKn8uZKx45K+/Y4nQQAgDOixE/x9X5xNqkDACIfJX4qd6aU7pE4uA0AEAUo8VMYY2S8ebLbymStdToOAADdosS/KTtXOnpYOvCl00kAAOgWJf4N7BcHAEQLV198yVNPPaXS0lKlpqZqyZIlkqT6+notXbpU1dXVysjI0J133qnk5OS+iNO9oVlSyhD/fvHZf+t0GgAAutQna+KXXnqp7r///g7LVq9erfPPP19PPvmkzj//fK1evbovopzR1/vFy9kvDgCIaH1S4hMmTDhtLbukpESXXHKJJOmSSy5RSUlJX0QJjDdPOlgj1RxwOgkAAF1ybJ/44cOHlZaWJklKS0vTkSNHnIpyGuPNlcR11AEAka1P9okHq6ioSEVFRZKkRYsWyePxhOyzXS7XaZ9n09NVfVaqBuypVOoPrg7Zd/VXnc0QPcMMQ4M5Bo8ZBq8vZ+hYiaempurgwYNKS0vTwYMHlZKS0uVrCwsLVVhY2P64pqYmZDk8Hk+nn2fHjlfjllK1hPC7+quuZojAMcPQYI7BY4bBC/UMs7KyunzOsc3pU6dO1fr16yVJ69ev17Rp05yK0ilzXq5U/ZVsHf9nBgBEpj5ZE1+2bJkqKip09OhR3XLLLbrqqqt0xRVXaOnSpVq7dq08Ho/uuuuuvogSMJOdJyvJVpbLzLjE6TgAAJymT0r8jjvu6HT5Qw891Bdf3zsjR0uDkqRtZRIlDgCIQFyxrQsmLl4aN4Ej1AEAEYsS74bx5kpffSF75KDTUQAAOA0l3o2T11FXZYWzQQAA6AQl3p1RY6UBA2W3cjMUAEDkocS7YVwuaWwOdzQDAEQkSvwMTHau9OVu2WNHnY4CAEAHlPgZfL1fnKPUAQCRhRI/k3OzJVcCp5oBACIOJX4GJiFRGnMeJQ4AiDiUeACMN1fas0O24bjTUQAAaEeJB8B48yTrk7Z/6nQUAADaUeKBGHOeFB8vW8mpZgCAyEGJB8AMGCiNzma/OAAgolDiATLeXGlXpWxTo9NRAACQRIkHzGTnSW1t0o6tTkcBAEASJR64ceMlE8clWAEAEYMSD5AZlCSNGsN+cQBAxKDEe8B4c6UdW2Vbmp2OAgAAJd4TxpsrtbZIOyudjgIAACXeI9m5ksR+cQBARKDEe8AMPksafo4sdzQDAEQASryHjDdP+vwz2dZWp6MAAGIcJd5DxpsrNTVKez53OgoAIMZR4j3lZb84ACAyUOI9ZFLSpGEjOF8cAOA4SrwXjDdX2l4h62tzOgoAIIZR4r3hzZMajkt7dzmdBAAQwyjxXjAnzxfn/uIAAAdR4r1g0j1SxjDZrewXBwA4hxLvJZOdK20vl/X5nI4CAIhRlHhvefOk+qPS/r1OJwEAxChKvJdM+/nibFIHADiDEu8tz1ApzSNx0RcAgEMo8V4yxsh4c2W3lcla63QcAEAMosSD4c2VjhySDuxzOgkAIAZR4kEw3jxJXEcdAOAMSjwYQ4dLKUMk7i8OAHAAJR4EY4xMNvvFAQDOoMSD5c2V6mqk2iqnkwAAYgwlHiT2iwMAnEKJBytrlDT4LImLvgAA+hglHiQTFydlT2BNHADQ5yjxEDDZuVL1V7IHa52OAgCIIZR4CJjz2C8OAOh7lHgojDxXGjiI/eIAgD5FiYeAiYuXxk2Q5aIvAIA+5HI6wBtvvKG1a9fKGKORI0fq1ltvVWJiotOxesx482T/63eyRw7JpAxxOg4AIAY4uiZeV1enN998U4sWLdKSJUvk8/lUXFzsZKReO3l/cVVWOBsEABAzHN+c7vP51NzcrLa2NjU3NystLc3pSL1zzlgpcQAHtwEA+oyjm9PT09P1/e9/X/PmzVNiYqImTpyoiRMnnva6oqIiFRUVSZIWLVokj8cTsgwulytkn3cw53z5dmyVO4T5okEoZxirmGFoMMfgMcPg9eUMHS3x+vp6lZSUaPny5UpKStKvf/1rbdiwQbNnz+7wusLCQhUWFrY/rqmpCVkGj8cTss/zneuV/ePLqt69S2Zwckg+MxqEcoaxihmGBnMMHjMMXqhnmJWV1eVzjm5O37JlizIzM5WSkiKXy6UZM2Zo27ZtTkYKisnOk6zl1qQAgD7haIl7PB5VVlaqqalJ1lpt2bJFw4cPdzJScMZ4JZeLU80AAH3C0c3p2dnZmjlzpu677z7Fx8dr9OjRHTabRxuTkCid65XdysFtAIDwc/w88auuukpXXXWV0zFCxnjzZP/0H7KNx2UGJjkdBwDQjwVc4hs3blRFRYWOHDnSYfmCBQtCHiqaGW+e7H//Qdr+qZSX73QcAEA/FtA+8VdffVXPPvusfD6fPvzwQyUnJ+vjjz9WUhJrmqcZmyPFx8tyHXUAQJgFtCa+bt06PfDAAxo1apTeffddzZ07VxdddJH+8z//M9z5oo4ZMFA6ZxwXfQEAhF1Aa+LHjh3TqFGjJPlPYm9tbdW4ceNUUcElRjtjsnOlXdtlm5qcjgIA6McCKvFhw4Zp7969kqSRI0fq7bff1oYNG5ScHDsXNOkJc16e1NYq7fjM6SgAgH4soM3pV199tY4ePSpJuu666/TEE0+osbFR//iP/xjWcFFr7HjJxMlWlsuMP/0ysgAAhEJAJT5lypT2n8eNG6ff/OY3YQvUH5ikwdLIczm4DQAQVgFtTr/hhhs6XX7TTTeFNEx/Yry50o6tsi0tTkcBAPRTAZV4W1vbactaW1vl8/lCHqi/MN48qaVZ2lXpdBQAQD/V7eb0hx56SMYYtbS06OGHH+7wXG1trbxeb1jDRbXsCZIku61M5sTPAACEUrclfvnll0uStm/frssuu6x9uTFGqampysvLC2+6KGaSU6Th5/j3i/8/TqcBAPRH3Zb4pZdeKsl/o5KovruYQ4w3V7Z4rWxrq4zL8cvUAwD6mYD2iWdlZamoqEiPPPKI7r77bklSRUWFiouLwxou6mXnSU2N0t4dTicBAPRDAZX4K6+8onXr1qmwsFA1NTWSJLfbrddffz2s4aKd8eZKEpdgBQCERUAlvn79et13332aNWuWjDGSpMzMTFVVVYU1XLQzqWnS0OGcLw4ACIuAStzn82ngwIEdljU2Np62DKcz3lypskLWd/ppegAABCOgEp88ebJeeOEFtZy4cIm1Vq+88ory87lf9hl5c6WGY9IXu5xOAgDoZwIq8euvv151dXWaO3eujh8/ruuvv17V1dW67rrrwp0v6hmv/zQ8NqkDAEItoPOekpKSdO+99+rw4cOqrq6Wx+PRkCFDwhytfzDpGZI7039wW+HfOR0HANCPBLQmLvnvKf7JJ5+ooqJCW7ZsUX19fThz9SvGmydVlsta63QUAEA/ElCJl5WVaf78+XrzzTe1fft2vfXWW1qwYIG2bNkS7nz9w3l5Uv1Rad9ep5MAAPqRgDanr1y5Uj/72c9UUFDQvuyDDz7QypUrtWzZsnBl6zdMdq6sJFtZJjN8lNNxAAD9REBr4gcPHtTMmTM7LJs+fboOHToUjkz9T8YwaYhb4uA2AEAIBVTis2fP1ltvvdVh2dtvv63Zs2eHJVR/Y4zxX0d9G/vFAQCh0+Xm9JO3IZX8F3t555139Mc//lHp6emqq6vT4cOHlZ2d3WdBo543T/rfDVLVfmloltNpAAD9QJclfvI2pCfNmTMn7GH6M+M9sV98W5kMJQ4ACIEuS/zkbUgRIsNGSGel+veLX/wtp9MAAPqBgM8TR3CMMVJ2Lnc0AwCEDCXeh4w3T6qrlq3l7m8AgOBR4n2o/f7iW1kbBwAEjxLvS8PPkZKSpUrOFwcABC+gK7a98sornS5PSEhQenq6Jk2axA1RAmDi4qTsCewXBwCEREBr4vv379frr7+u8vJyffXVVyovL9frr7+unTt36p133tFtt92mzZs3hzlq/2C8uVLVftlDtU5HAQBEuYDWxH0+n+644w5Nnz69fVlJSYnee+89Pfroo3r33Xf10ksvadKkSeHK2W8Yb96J88XLZaZzxTsAQO8FtCb+8ccfa+rUqR2W5efnt699z549WwcOHAh5uH5p5Bhp4CCJTeoAgCAFVOLDhg3T22+/3WHZ22+/raFDh0qSjhw5ogEDBoQ+XT9k4uOlceNluRkKACBIAW1Ov/nmm7VkyRK9/vrr7ddOj4uL089//nNJ0r59+3T11VeHNWh/YrJzZctelD16WOasVKfjAACiVEAlPmbMGD3xxBPatm2bDh06pCFDhsjr9crl8r99woQJmjBhQliD9icn94urslyaUnCmlwMA0KmASlySXC6XJkyYIJ/P177M5/MpLo5TzXts9DgpMdF/cBslDgDopYBKfMeOHVq5cqX27Nmj5ubmDs91dQ45umZcCdKYHM4XBwAEJaASX758ufLz8zVv3jwOYAsR482TXfOy7LF6mcHJTscBAEShgEq8pqZG11xzjf9OXAgJ482VtVba/qk0cZrTcQAAUSigHdrTpk3Txx9/HO4sseVcr+RysUkdANBrAa2Jt7S06PHHH1dOTs5p10hfsGBBOHL1eyZxgDTaK8vNUAAAvRRQiY8YMUIjRowId5aYY7x5sm/9h2zjcZmBSU7HAQBEmYBK/Ec/+lHYAhw7dkwrVqzQ3r17ZYzRvHnz5PV6w/Z9kcR4c2X/9Adp+2dS3hSn4wAAokyXJV5RUdF+AZeysq732+bl5QUV4LnnntOkSZP085//XK2trWpqagrq86LK2BwpLk62slyGEgcA9FCXJb5y5UotWbJEkvT00093+hpjjH7729/2+suPHz+uTz/9VPPnz/eHcbnarwIXC8zAQdI54zi4DQDQK1025skCl/zniYdDVVWVUlJS9NRTT2n37t0aM2aM5s6dq4EDB4bl+yKR8ebKFq2RbW7yH+wGAECAAlrtXbx4se69997Tlj/++OO6++67e/3lbW1t2rlzp2688UZlZ2frueee0+rVq/XjH/+4w+uKiopUVFQkSVq0aJE8Hk+vv/ObXC5XSD+vp5ryC3Toz68ptfYrJZ6f71iOYDg9w/6AGYYGcwweMwxeX84woBIvL+/8NKiulgfK7XbL7XYrOztbkjRz5kytXr36tNcVFhaqsLCw/XFNTU1Q33sqj8cT0s/rKTt0uGSMDpUUK+7scxzLEQynZ9gfMMPQYI7BY4bBC/UMs7Kyunyu2xI/eV301tbW066RfuDAAWVkZAQVbMiQIXK73dq3b5+ysrK0ZcuWmDuVzSQlSyPPZb84AKDHui3x2tpaSf67lZ38+SSPx6Orrroq6AA33nijnnzySbW2tiozM1O33npr0J8ZbUx2ruyGP8u2tvhvjgIAQAC6LfGTher1ejtszg6l0aNHa9GiRWH57GhhvHmyf1kj7aqUxnFfdgBAYALaJ36ywBsaGnT06FH/jTtOGDp0aHiSxZLsXEmS3VomQ4kDAAIUUIl/8cUXevLJJ7V79+7TnuN+4sEzZ6VIWaO4jjoAoEcCuovZv/3bvyk3N1erVq1SUlKSnnvuOf3N3/xN+0VaEDzjzZW2fybb1uZ0FABAlAioxHfv3q3rrrtOgwcPlrVWSUlJ+slPfsJaeCh586SmBmnPDqeTAACiREAlnpCQoLYTa4hnnXWWampqZK1VfX19WMPFEnNyvzinmgEAAhTQPvGcnBx98MEHuvTSSzVz5kz98pe/VEJCgnJzc8OdL2aYIelSZpZ/v/i3f+h0HABAFAioxO+66672n6+55hqNHDlSjY2Nmj17dtiCxSJzXp7spvdlfW0ycfFOxwEARLge3zIsLi6O8g6X7Fzpf96WvtwjjTzX6TQAgAgXUIkfP35cf/rTn7Rr1y41NjZ2eO6BBx4IS7BYZLx5svLvFzeUOADgDAIq8V//+tfy+XyaPn26EhMTw50pZhl3huTO9B/cNuf7TscBAES4gEq8srJSK1eulMvV463v6CHjzZXdsknWWhljnI4DAIhgAZ1ilpOToy+//DLcWSD594vXH5H273U6CQAgwgW0an3rrbfqscce07hx4zRkyJAOz/393/99OHLFLHPeyf3i5TJZo5yOAwCIYAGV+Msvv6za2lplZGSooaGhfTmbe8Mg42wpNV3aViZd+h2n0wAAIlhAJV5cXKwnnnhCaWlp4c4T84wx/v3ileXsFwcAdCugfeJDhw5VfDwXH+kz3lzpUJ1Uvd/pJACACBbQmvjFF1+sxYsX62//9m9P2yeel5cXjlwx7evzxctlMrOcjgMAiFABlfif//xnSf5946cyxui3v/1t6FPFurNHSskp/v3iF/2N02kAABEqoBJfvnx5uHPgFMYYyZsru63c6SgAgAgW0D5x9D3jzZNqq2Rrq5yOAgCIUF2uid95551aunSpJGnevHldfsDTTz8d+lSQyc79er/4hZlOxwEARKAuS/zmm29u//m2227rkzA4xYhzpKTBUmW5dOFlTqcBAESgLks8Jyen/efDhw/rwgsvPO01H374YXhSwX8/8XETZLeWOR0FABChAtonvmLFik6XP/PMMyENg46MN0+q2id7qNbpKACACNTt0ekHDhyQJPl8PlVVVcla2+E5bksaXmb8RP9+8U82ysz+ttNxAAARptsSv/3229t//uZ+8SFDhuhHP/pReFLBb+S5UsYw2U3FEiUOAPiGbkv8lVdekSQ9/PDDeuSRR/okEL5mjJHJnyX7zmrZY0dlBp/ldCQAQAQJaJ/4Nwv8wIEDqq6uDksgdGSmFEhtbbKb/9fpKACACBNQiS9btkxbt26VJK1bt0533XWX7rrrLq1duzas4SBp9DgpPUN20/tOJwEARJiASrysrExjx46VJL3xxht68MEH9ctf/lKrV68OZzbo5Cb1Aqlis+zxY07HAQBEkIBKvLW1VS6XS3V1daqvr1dOTo5Gjhypw4cPhzsfJJn8WVJbq+wnbFIHAHwtoBugjB49Wq+99pqqq6s1ZcoUSVJdXZ0GDRoU1nA44VyvNMTtP0p9JldvAwD4BbQmfsstt2jPnj1qbm7Wj3/8Y0nStm3bdNFFF4U1HPxMXJx/k3pZqWzjcafjAAAiREBr4sOGDdM//dM/dVg2c+ZMzZw5MyyhcDozpUD2L2v8F36ZPtvpOACACBBQiVtr9Ze//EXFxcU6cuSIHn/8cVVUVOjQoUMqKCgId0ZI0rgcKTXNv0mdEgcAKMDN6a+88orWrVunOXPmqKamRpLkdrv1+uuvhzUcvmbi4mUmXyiVbZJtanQ6DgAgAgRU4uvXr9d9992nWbNmyRgjScrMzFRVVVVYw6Ejk18gNTdJZaVORwEARICAStzn82ngwIEdljU2Np62DGGWnSslp3DhFwCApABLfPLkyXrhhRfU0tIiyb+P/JVXXlF+fn5Yw6EjEx8vM3mm7CcbZZubnI4DAHBYQCV+/fXXq66uTnPnztXx48d1/fXXq7q6Wtddd1248+EbTP4sqalBqvjI6SgAAIcFdHR6UlKS7r33Xh0+fFjV1dXyeDwaMmRImKOhU+edLyUly24qlpnEKX4AEMsCKvGTUlNTlZqaGq4sCIBxuWQmz5At/UC2pUUmIcHpSAAAhwS0OR2RxeTPkhqOS59udjoKAMBBlHg0ypkoDRrsv/ALACBmnbHEfT6fysrK1Nra2hd5EACTkCAzcbrs5r/K8s8FAGLWGUs8Li5OixcvlsvVo93nCDOTXyAdr5e2bnE6CgDAIQFtTh8/fry2bdsWthA+n0/33nuvFi1aFLbv6HdyJ0sDBnHhFwCIYQGtXmdkZOixxx7T1KlT5Xa72y+9KklXX3110CH+9Kc/afjw4WpoaAj6s2KFSUiUuWCq7Ecfyl43TyY+3ulIAIA+FtCaeHNzs6ZNmyZjjOrq6lRbW9v+J1i1tbUqLS3VnDlzgv6sWGPyZ0n1R6RtZU5HAQA4wFhrrZMBlixZoh/+8IdqaGjQmjVrtHDhwtNeU1RUpKKiIknSokWL1NzcHLLvd7lcUXvQnm1qVNX/+a4GXfYdpdx8j2M5onmGkYIZhgZzDB4zDF6oZ5iYmNj1dwX6Ifv379f777+vuro6paena9asWTr77LODCrZp0yalpqZqzJgxKi8v7/J1hYWFKiwsbH988naooeDxeEL6eX0ub4oaitep6YfXy8Q5s0k96mcYAZhhaDDH4DHD4IV6hllZWV0+F9Dm9I0bN2rhwoX68ssvlZycrH379mnhwoXauHFjUMG2bt2qjRs3av78+Vq2bJnKysr05JNPBvWZscbkz5KOHJK2f+p0FABAHwtoTfzll1/WPffco7y8vPZl5eXlWrVqlaZOndrrL7/22mt17bXXtn/emjVrdPvtt/f682KROT9fNiHRfy11b96Z3wAA6DcCWhOvq6vT+PHjOyzLyckJyYFtCI4ZmCTlTpEtLZb1+ZyOAwDoQwGV+OjRo7VmzZoOy9544w2NHj06ZEFyc3M7PagNZ2byC6RDddKOrU5HAQD0oYA2p99000361a9+pTfffFNut1u1tbUaMGCA7r333nDnQwDMBdNkXS7Z0mKZcePP/AYAQL9wxhL3+Xw6ePCgFi9erF27drUfnT5u3DguxRohTNJgafwk2U3Fsj+6scPFeAAA/VfA104fOHCgcnJyVFBQoJycHAo8wpj8WVJdtbRru9NRAAB9JCKunY7gmUnTpfh4rqUOADEkIq6djuCZwWdJORf4j1L/f/8Pm9QBIAY4fu10hI7JnyVVfyXt3eF0FABAHwjowDa3260rr7xSCQkJfZEJvWQmzZT9/57yX/hl1Fin4wAAwiygA9vefvttxXOry4hnzkqRzjtfduP7cvi+NgCAPhDQ5vTZs2frnXfeCXcWhICZUiBV7ZO+3O10FABAmAV0YNv27dv11ltv6Y9//ONpB7Y98sgjYQuHnjNTZsr++wr/hV9GjHY6DgAgjAIq8Tlz5mjOnDnhzoIQMClpUnau7KZi6e+udToOACCMAirxSy+9NMwxEEpmSoHs75+V3b9X5uyRTscBAIRJt/vEV61a1eHx2rVrOzx+/PHHQ58IQTNTLpQk/9o4AKDf6rbE169f3+Hxiy++2OHxli1bQp8IQTNpbmlsDiUOAP1ctyXOaUrRy+TPkr7YKXtgn9NRAABh0m2Jc+nO6GWmFEiSbClr4wDQX3V7YFtbW5vKysraH/t8vtMeIzIZd4Z0rte/Sf07f+90HABAGHRb4qmpqXr66afbHycnJ3d4nJKSEr5kCJrJL5D9j+dlq7+SyRjmdBwAQIh1W+LLly/vqxwIAzPlRImXfiDz7R86HQcAEGIBXXYV0clkDJNGjWW/OAD0U5R4P2fyC6QdW2Xrqp2OAgAIMUq8n/v6KPUPHE4CAAg1SryfM8OGS8PP4cIvANAPUeIxwOTPkj7/VPZQrdNRAAAhRInHAJNfIFkr+9GHTkcBAIQQJR4DTNYo6eyRbFIHgH6GEo8RJr9A2lYue+Sg01EAACFCiccI/yZ1n+xHf3U6CgAgRCjxWDF8tJSZJbvpfaeTAABChBKPEcYY/9r41i2y9UecjgMACAFKPIaY/FmSzye7mU3qANAfUOKxZNQYyZ3JUeoA0E9Q4jHEv0l9lvTpx7LH6p2OAwAIEiUeY0x+gdTWKvvx/zodBQAQJEo81pzrldI93J4UAPoBSjzGGGP8dzYrL5VtOO50HABAECjxGGTyC6TWVtlPSpyOAgAIAiUei8bkSKnpXPgFAKIcJR6DTFyczJQLpbJS2cYGp+MAAHqJEo9RJn+W1NIslW1yOgoAoJco8ViVPV46K5ULvwBAFKPEY5SJi5eZfKHslo2yTU1OxwEA9AIlHsNMfoHU1CiVlzodBQDQC5R4LPPmSclnsUkdAKIUJR7DjMslM2mm7Cf/K9vS7HQcAEAPUeIxzuQXSI0NUsVmp6MAAHrI5eSX19TUaPny5Tp06JCMMSosLNR3v/tdJyPFnpwLpKTBspvel5k43ek0AIAecLTE4+Pj9Q//8A8aM2aMGhoatHDhQl1wwQUaMWKEk7FiinElyEycIbv5r7KtLTKuBKcjAQAC5Ojm9LS0NI0ZM0aSNGjQIA0fPlx1dXVORopJJn+W1HBM+uwTp6MAAHrA0TXxU1VVVWnnzp0aN27cac8VFRWpqKhIkrRo0SJ5PJ6Qfa/L5Qrp50UjO3uOqlcuUWJ5qVIv/XaP388Mg8cMQ4M5Bo8ZBq8vZ2istbZPvqkbjY2Nevjhh3XllVdqxowZZ3z9vn37QvbdHo9HNTU1Ifu8aOX7v0tky0sV9/jvZFw9+92OGQaPGYYGcwweMwxeqGeYlZXV5XOOH53e2tqqJUuW6OKLLw6owBEeJr9AOnZU2lbmdBQAQIAcLXFrrVasWKHhw4fre9/7npNRkDdFGjCQC78AQBRxtMS3bt2qDRs2qKysTPfcc4/uuecelZZyCVAnmMQBMudPlf3oA1lfm9NxAAABcPTAtpycHP3hD39wMgJOYfILZDe+J1VWSOed73QcAMAZOL5PHBEkL19KTJTd9L7TSQAAAaDE0c4MHCTl5cuWfiDr8zkdBwBwBpQ4OjBTCqTDB6XPP3M6CgDgDChxdGAumCa5EmRLOUodACIdJY4OzKAkKXey7KZiNqkDQISjxHEaM6VAOlgj7ap0OgoAoBuUOE5jJk6X4l1c+AUAIhwljtOYwcnS+Imym95XBFxaHwDQBUocnTL5BVJtlbTnc6ejAAC6QImjU2bSDCkujgu/AEAEo8TRKZOcIuVc4D9KnU3qABCRKHF0yeQXSFX7pS92OR0FANAJShxdMpNmSiaOC78AQISixNElkzJE8uZyqhkARChKHN0y+bOk/Xtl9+1xOgoA4BsocXTLTJ4pGcPaOABEIEoc3TJD0qWx4znVDAAiECWOMzL5BdKXu2W/+sLpKACAU1DiOCMz5UJJYpM6AEQYShxnZNIzpDHncaoZAEQYShwBMfkF0p4dslX7nY4CADiBEkdAzJQCSWJtHAAiCCWOgBjPUOmccbKlHzgdBQBwAiWOgJn8WdLObbK1VU5HAQCIEkcPmPwTR6mzNg4AEYESR8BMZpY04lwu/AIAEYISR4+Y/ALp889kD9Y6HQUAYh4ljh4x+bMksUkdACIBJY4eMWePkLJGyZaySR0AnEaJo8dMfoFUWSF7+KDTUQAgplHi6DGTP0uyVvYjNqkDgJMocfRc1ihp2HBuiAIADqPE0WPGGJkps6RtZbJHDzsdBwBiFiWOXjH5BZLPJ7v5r05HAYCYRYmjd0aeK2UM48IvAOAgShy94t+kXiB99ol8R484HQcAYhIljl4z+bOktjY1lfyP01EAICa5nA6AKDZ6nJSeofrf/5vshxukwWdJScnS4MHS4LNkkpKlwcn+ZUnJ/ucTE2WMcTo5APQLlDh6zRgj83fXKu69t9W6e7t0rF46fkyyPkmS7exNLtfXZZ/USdkP9v8xpxb/4MFSUrKMK6FP/34AEOkocQQlbtYcuX9wtWpqaiRJ1ueTGhukY0f9hX7sqHS8XvZ4vb/kj9X7H5/4Xx2skf1yt//nhuPtn9vpLwADBn5d9EmnlH2HXwDOOuUXgBN/BiXJxMX3zUAAoA9R4ggpExfnX8NOGtxxeQDvtW1t/uI/Xv91+Z8s+1N/ATj5fNX+E88flZqbv/6c00IZKTVd8mTKuDMlz1DJnSlz4n+VniHj4l8FANGH/3IhYpj4eOmsFP+fk8sCfK9taelQ/jp26i8AR6W6GtnaKtntn0ol/+M/x739S+KktPRTin1ox8JP8/izAUCEocTRL5iEBCk1zf/n5LIuXmvb2qSDNVJtlWzNAammSqo94C/5rVukg+/6rw1/8g1xcVKap+PauydTxj30RMmns7kegCMoccQcEx/vL1/PUJnzzj/tedvaIh2slWoO+Eu+tqq98G3FZulwXceSj4/3l7xn6Im190zJPfTrwh+SRskDCAtKHPgG40qQMoZJGcM6XZu3LS1SXbV/7b2mSjpR9La2Sras1F/yOmXffLxLSj9R8ieL/eRavSdTSknr5FsA4MwcL/HNmzfrueeek8/n05w5c3TFFVc4HQnolklIkIZmSUOzOi/55iZ/yddUnb4mv/mv0ombxrSXvCtB1Wlu+QIO0Nm3drKss9d1uo8hwPdGgRqXS22trWd+YaB/v3DM4YyfeYbnz/j24D6/NiEhsBlGsjPNIBT/XLv5jMOjxkjXzQv+OwLgaIn7fD6tXLlSDzzwgNxut37xi19o6tSpGjFihJOxgKCYxAHSsBHSsBGdl3xTk1RXdWJzvX9/fGJTg5oaG7/xwk5OtOv03LvOXtfpCwN7XSfLbOdfHHFciQPU1tzU/YsC/qv04O8c8LyDfEGg39Pb91uruMTEDmd7RJ0zziiAGZ5xTt0/bQYOPPN3hIijJb59+3YNGzZMQ4cOlSQVFBSopKSEEke/ZgYMkM4eKZ09sr3kUz2e9nPt0XtDmGPQ0phh0FL6cIaOXju9rq5Obre7/bHb7VZdXZ2DiQAAiB6OronbTjZZdHZd7aKiIhUVFUmSFi1aJI/HE7IMLpcrpJ8Xi5hh8JhhaDDH4DHD4PXlDB0tcbfbrdra2vbHtbW1Sks7/UjdwsJCFRYWtj8O5WYKD5uOgsYMg8cMQ4M5Bo8ZBi/UM8zKyuryOUc3p48dO1b79+9XVVWVWltbVVxcrKlTpzoZCQCAqOHomnh8fLxuvPFGPfroo/L5fLrssss0cuRIJyMBABA1HD9PfMqUKZoyZYrTMQAAiDqObk4HAAC9R4kDABClKHEAAKIUJQ4AQJSixAEAiFKUOAAAUYoSBwAgShnb2QXMAQBAxIv5NfGFCxc6HSHqMcPgMcPQYI7BY4bB68sZxnyJAwAQrShxAACiVMyX+Km3OEXvMMPgMcPQYI7BY4bB68sZcmAbAABRKubXxAEAiFaO34rUKZs3b9Zzzz0nn8+nOXPm6IorrnA6UkSqqanR8uXLdejQIRljVFhYqO9+97uqr6/X0qVLVV1drYyMDN15551KTk6WJL322mtau3at4uLidMMNN2jSpEnO/iUihM/n08KFC5Wenq6FCxcyw144duyYVqxYob1798oYo3nz5ikrK4s59sAbb7yhtWvXyhijkSNH6tZbb1VzczMz7MZTTz2l0tJSpaamasmSJZLUq39/d+zYoeXLl6u5uVmTJ0/WDTfcIGNMcOFsDGpra7MLFiywX331lW1pabF333233bt3r9OxIlJdXZ39/PPPrbXWHj9+3N5+++1279699sUXX7Svvfaatdba1157zb744ovWWmv37t1r7777btvc3GwPHDhgFyxYYNva2pyKH1HWrFljly1bZh977DFrrWWGvfCb3/zGFhUVWWutbWlpsfX19cyxB2pra+2tt95qm5qarLXWLlmyxK5bt44ZnkF5ebn9/PPP7V133dW+rDczW7hwod26dav1+Xz20UcftaWlpUFni8nN6du3b9ewYcM0dOhQuVwuFRQUqKSkxOlYESktLU1jxoyRJA0aNEjDhw9XXV2dSkpKdMkll0iSLrnkkvb5lZSUqKCgQAkJCcrMzNSwYcO0fft2x/JHitraWpWWlmrOnDnty5hhzxw/flyffvqpLr/8ckmSy+XS4MGDmWMP+Xw+NTc3q62tTc3NzUpLS2OGZzBhwoT2teyTejqzgwcPqqGhQV6vV8YYzZ49OyS9E5Ob0+vq6uR2u9sfu91uVVZWOpgoOlRVVWnnzp0aN26cDh8+rLS0NEn+oj9y5Igk/2yzs7Pb35Oenq66ujpH8kaS559/Xj/5yU/U0NDQvowZ9kxVVZVSUlL01FNPaffu3RozZozmzp3LHHsgPT1d3//+9zVv3jwlJiZq4sSJmjhxIjPshZ7OLD4+/rTeCcUsY3JN3HZyQH7Q+yX6ucbGRi1ZskRz585VUlJSl6/rbLaxbtOmTUpNTW3fonEmzLBzbW1t2rlzp771rW9p8eLFGjBggFavXt3l65nj6err61VSUqLly5frmWeeUWNjozZs2NDl65lhz3U1s3DNMibXxN1ut2pra9sf19bWtv9GhdO1trZqyZIluvjiizVjxgxJUmpqqg4ePKi0tDQdPHhQKSkpkk6fbV1dndLT0x3JHSm2bt2qjRs36qOPPlJzc7MaGhr05JNPMsMecrvdcrvd7Ws5M2fO1OrVq5ljD2zZskWZmZntM5oxY4a2bdvGDHuhpzPrrHdCMcuYXBMfO3as9u/fr6qqKrW2tqq4uFhTp051OlZEstZqxYoVGj58uL73ve+1L586darWr18vSVq/fr2mTZvWvry4uFgtLS2qqqrS/v37NW7cOEeyR4prr71WK1as0PLly3XHHXcoLy9Pt99+OzPsoSFDhsjtdmvfvn2S/IU0YsQI5tgDHo9HlZWVampqkrVWW7Zs0fDhw5lhL/R0ZmlpaRo0aJC2bdsma602bNgQkt6J2Yu9lJaW6ne/+518Pp8uu+wyXXnllU5HikifffaZHnroIY0aNap9l8M111yj7OxsLV26VDU1NfJ4PLrrrrvaD/z4r//6L61bt05xcXGaO3euJk+e7ORfIaKUl5drzZo1WrhwoY4ePcoMe2jXrl1asWKFWltblZmZqVtvvVXWWubYA3/4wx9UXFys+Ph4jR49WrfccosaGxuZYTeWLVumiooKHT16VKmpqbrqqqs0bdq0Hs/s888/11NPPaXm5mZNmjRJN954Y9C7cmO2xAEAiHYxuTkdAID+gBIHACBKUeIAAEQpShwAgChFiQMAEKUocQA9Nn/+fH3yySdOxwBiXkxesQ3or+bPn69Dhw4pLi5OLpdLXq9XP/3pT+XxeLp9X1VVlRYsWKCXX35Z8fHxfZQWQLBYEwf6mfvuu08vvviinnnmGaWmpmrVqlVORwIQJqyJA/1UYmKiZs6cqd/97neS/Fcp/P3vf68DBw4oKSlJl112ma666ipJ0sMPPyxJmjt3riTpwQcflNfrVVFRkf77v/9btbW1crvduu2229pv5LJr1y698MILqq6u1qRJkzR//nwlJiZK8t/05fe//72qq6s1YsQI/fSnP9U555wjSVq9erXefPNNNTQ0KC0tTTfddJPOP//8vhwN0G9Q4kA/1dTUpOLi4vYbhgwYMEALFizQiBEjtHfvXv3Lv/yLRo8erenTp+uRRx7RggUL9Pzzz7dvTv/ggw/06quv6p577tHYsWN14MCBDpvaP/jgA91///1KTEzUgw8+qHfffVff+ta3tGPHDj399NO67777NHbsWG3YsEGLFy/WsmXLVF1drT//+c967LHHlJ6erqqqKvl8PkfmA/QHlDjQz/zrv/6r4uPj1djYqNTUVP3zP/+zJCk3N7f9Neecc45mzZqliooKTZ8+vdPPWbt2rX7wgx+03/Bi2LBhHZ7/zne+034Xpvz8fO3atUuS9Je//EWFhYXtvzxceumleu2111RZWan09HS1tLToiy++UEpKijIzM0P6dwdiDSUO9DP33HOPLrjgAvl8PpWUlOjhhx/W0qVLVV1drX//93/Xnj171NraqtbWVs2cObPLz6mpqdHQoUO7fH7IkCHtPycmJqqurq79fevXr9dbb73V/nxra6vq6uo0YcIEzZ07V6+++qq++OILTZw4Uddffz23twR6iRIH+qm4uDjNmDFDzz77rD777DO99NJL+va3v61f/OIXSkxM1PPPP68jR45IUqd3UvJ4PDpw4ECPv9ftduvKK6/s8s6AF110kS666CIdP35czz77rF566SXddtttPf4eABydDvRb1lqVlJTo2LFjGj58uBoaGpScnKzExERt375d7733XvtrU1JSZIzpUNqXX3651qxZox07dshaq6+++krV1dVn/N45c+bonXfeUWVlpay1amxsVGlpqRoaGrRv3z6VlZWppaVFiYmJSkxMVFwc/xkCeos1caCf+dWvfqW4uDgZY5SRkaH58+dr5MiRuummm/TCCy9o1apVmjBhgi688EIdO3ZMkv+gtyuvvFIPPvig2tradP/99+vCCy/U0aNH9cQTT6iurk6ZmZlasGCBMjIyuv3+sWPH6uabb9aqVau0f/9+JSYmKicnR+PHj1dLS4teeuklffnll4qPj9d5552nn/3sZ30xFqBf4n7iAABEKbZjAQAQpShxAACiFCUOAECUosQBAIhSlDgAAFGKEgcAIEpR4gAARClKHACAKEWJAwAQpf5/nCNoUziE3z8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.xlabel('Batches')\n",
    "plt.ylabel('Error estimating beta')\n",
    "plt.plot(batches, errors)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c60ed5c-cc9c-484b-b8fa-10b20b0373a1",
   "metadata": {},
   "source": [
    "For these reason, it is advisable to find the balance according to the problem. In this case, as it is shown in the last picture and in the previous running results, a batch size of **400** performs very well in terms of error and computational time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "80c5af98-ba77-4552-b8a6-a52b9d4981d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.10513    | 0.0          | 421        | 6.31816    | 0.01501    |\n",
      "------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "err = mini_batch_gradient_method(400, n_iter, epsilon, sigma, delta, alpha_init, ks=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51571ec9-0d28-43ea-9b4d-f3489539377d",
   "metadata": {},
   "source": [
    "### D.3. Mini-batch gradient with momentum\n",
    "\n",
    "It is the same algorithm than Mini-batch with an additional term in the computation of the regression coefficients. In each iteration is added a term called **momentum** which helps to reach faster the solution:\n",
    "\n",
    "$$\n",
    "\\beta_{t+1} = \\beta_t - \\alpha \\nabla f(\\left.\\beta^{*} \\right) + \\underbrace{\\lambda (\\beta_t - \\beta_{t-1})}_\\text{momentum}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "bb78299c-8cfc-4e32-b177-b2be84f47f29",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mini_batch_momentum_gradient_method(batch=50, momentum=0.5, n_iter=2000, epsilon=1e-4, sigma=1e-5, delta=0.01, alpha_init=1e-5, ks = 5):\n",
    "    '''Implement Mini-batch gradient method with momentum\n",
    "    \n",
    "    '''\n",
    "    # Init algorithm dic\n",
    "    alg = 'Mini-batch momentum'\n",
    "    alg_dic[alg] = {}\n",
    "    \n",
    "    (a,b) = X.shape\n",
    "    \n",
    "    # Initial values for the variables and data containers\n",
    "    beta = np.zeros(b) \n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "    \n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Mini batch with momentum method\n",
    "    i = 0\n",
    "    tol = 10000\n",
    "    # iteration lower than maximum number and tolerance filter\n",
    "    while (i <= n_iter-2) and (tol > epsilon):\n",
    "        i = i + 1\n",
    "        \n",
    "        # get subsample data\n",
    "        X_b, y_b = sample_data(batch)\n",
    "        \n",
    "        # compute gradient\n",
    "        grad = ridge_reg_der(beta, X_b, y_b)\n",
    "        # compute descent direction\n",
    "        ddirect = -grad \n",
    "        \n",
    "        # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "        alpha = alpha_init\n",
    "        while (ridge_reg(beta+alpha*ddirect,X,y) > ridge_reg(beta,X,y) + alpha*sigma*np.dot(ddirect,grad)):\n",
    "            alpha = alpha*delta\n",
    "        \n",
    "        # compute new point\n",
    "        beta_old = beta\n",
    "        beta = beta + alpha*ddirect + momentum*(beta - beta_old)\n",
    "    \n",
    "        # objective value per iteration\n",
    "        OF_iter[i] = ridge_reg(beta, X, y)\n",
    "        \n",
    "        # if the norm of the gradient is small, we can stop\n",
    "        tol = np.absolute(OF_iter[i] - OF_iter[i-1])\n",
    "        tol_iter[i] = tol\n",
    "        \n",
    "        # alpha iteration due to Armijo rule\n",
    "        alpha_iter[i] = alpha\n",
    "    \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # The error compares the obtained solution to the optimal solution (analytical)\n",
    "    err = np.linalg.norm(np.transpose(beta_exact)-beta,ord=2)/np.linalg.norm(beta,ord=2)\n",
    "    \n",
    "    # Save stats\n",
    "    alg_dic[alg]['err'] = round(err, 5)               # error\n",
    "    alg_dic[alg]['nit'] = i                           # number of iterations\n",
    "    alg_dic[alg]['tol'] = round(tol, 9)               # reached tolerance\n",
    "    alg_dic[alg]['time'] = round(time_elapsed, 5)     # computational time\n",
    "    alg_dic[alg]['sec'] = round(time_elapsed/i, 5)    # seconds per iteration\n",
    "    \n",
    "    # Show stats\n",
    "    show_alg_stats(alg)\n",
    "    \n",
    "    # Show beta\n",
    "    if ks > 0:\n",
    "        print_beta(beta, ks)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dee52ad8-6542-410c-8afb-7f8f5a110341",
   "metadata": {},
   "source": [
    "As it has been explained before, a batch of **400** performs well, so this is the value used in this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "bb951765-8cc8-4b51-80b4-60235e3943cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch momentum | 0.10269    | 0.0          | 381        | 6.0768     | 0.01595    |\n",
      "------------------------------------------------------------------------------------------\n",
      "--------------------------------------------------\n",
      "| Index      | Beta            | Optimal beta    |\n",
      "--------------------------------------------------\n",
      "| 10         | 2.90908         | 3.00503         |\n",
      "--------------------------------------------------\n",
      "| 4          | 0.1073          | -0.01799        |\n",
      "--------------------------------------------------\n",
      "| 82         | -1.07782        | -1.01327        |\n",
      "--------------------------------------------------\n",
      "| 6          | 1.86872         | 1.99519         |\n",
      "--------------------------------------------------\n",
      "| 42         | -4.84493        | -5.02055        |\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "batch = 400\n",
    "momentum = 0.5\n",
    "n_iter = 3000\n",
    "epsilon = 1e-8\n",
    "sigma = 1e-1\n",
    "delta = 1e-1\n",
    "alpha_init = 1e-5\n",
    "\n",
    "# Run algorithm\n",
    "mini_batch_momentum_gradient_method(batch, momentum, n_iter, epsilon, sigma, delta, alpha_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07e7eebe-60b4-4daa-8a64-6c8367c13e4d",
   "metadata": {},
   "source": [
    "The results from the methods can be easily printed with functon `show_alg_stats()` and indicating the desired algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "82a64824-cbe9-499f-abf2-0c770d973d8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Coordinate          | 0.09603    | 7e-09        | 7341       | 27.42444   | 0.00374    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.10513    | 0.0          | 421        | 6.31816    | 0.01501    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch momentum | 0.10269    | 0.0          | 381        | 6.0768     | 0.01595    |\n",
      "------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "show_alg_stats(['Coordinate', 'Mini-batch', 'Mini-batch momentum'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "854f0d26-4053-493d-88fa-755d1f27ea5c",
   "metadata": {},
   "source": [
    "The **Mini-batch algorithm** with and without **momentum** perform similar. Mention there are not many samples, so the techniques could not provide all the potential. In addition, the batch size of the method with momentum has not been chosen appopiately, a tuning process of this parameter could lead to better results than the base algorithm.\n",
    "\n",
    "Both mini-batch approaches are significantly better than the **Coordinate** method in terms of iterations.\n",
    "\n",
    "The statistics of all the methods can be printed with `show_alg_stats()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "41a7d87b-9ad7-477e-b47a-89caa4858d13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------------------------------------\n",
      "| Algorithm           | Error      | Tolerance    | Iterations | Time (sec) | Sec/iter   |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Gradient            | 0.09287    | 43.065986502 | 1999       | 10.77377   | 0.00539    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Newton              | 0.0        | 2.813e-06    | 499        | 27.89775   | 0.05591    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Quasi-Newton        | 0.0        | 0.334951676  | 999        | 9.06882    | 0.00908    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Coordinate          | 0.09603    | 7e-09        | 7341       | 27.42444   | 0.00374    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch          | 0.10513    | 0.0          | 421        | 6.31816    | 0.01501    |\n",
      "------------------------------------------------------------------------------------------\n",
      "| Mini-batch momentum | 0.10269    | 0.0          | 381        | 6.0768     | 0.01595    |\n",
      "------------------------------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "show_alg_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1807c78b-fa25-4937-aa9f-bd1aa6d1de78",
   "metadata": {},
   "source": [
    "The results can be compared from two perspectives, the estimation error and the computational resources. \n",
    "\n",
    "First algorithms provide a result more precise, but they take longer. In this problem, the difference in time is not significant, but it is necessary to mention than the algorithms implemented in section D are thought for problem with a huge amount of samples and variables. In these situations, the difference in the computational resources is critical."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f834f3-35eb-4489-9c96-55340bb778e8",
   "metadata": {},
   "source": [
    "## E. Constrained problem\n",
    "\n",
    "The problem formulation is:\n",
    "\n",
    "\\begin{align*}\n",
    "\\underset{\\beta}{\\min} \\quad & ||y - X\\beta||_2^2 \\\\\n",
    "\\text{s.t.}& \\quad \\sum_{j=1}^K \\beta_j = 1\n",
    "\\end{align*}\n",
    "\n",
    "There are only equality constraints, so it is going to be solved using a **penalization algorithm**, through a sequence of unconstrained problems. The Gradient method is employed in each iteration:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\underset{\\beta}{\\min} & \\quad ||y - X\\beta||_2^2 + \\frac{\\rho}{2} (\\sum_{j=1}^K \\beta_j - 1)^2\n",
    "\\end{align*}\n",
    "\n",
    "where $\\rho > 0$ is the penalization parameter. This value is increased in each iteration in order to obtain better solutions. The gradient in this case is:\n",
    "\n",
    "\\begin{align*}\n",
    "    \\nabla f(\\left.\\beta^{*} \\right) = -2(y - X\\beta)^T X + \\rho (\\sum_{j=1}^K \\beta_j - 1) e\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "2d3edfb2-0f49-45ff-86e5-52c4a4ed376c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Objective function\n",
    "def constrained(beta_r, X, y, rho):\n",
    "    '''Implement constrained function\n",
    "    \n",
    "    '''\n",
    "    beta_r = np.matrix(beta_r)\n",
    "    z = y - np.dot(X, beta_r.T)\n",
    "    return np.dot(z.T,z) + (rho/2)*(np.sum(beta_r)-1)**2\n",
    "\n",
    "# Gradient\n",
    "def constrained_der(beta_r, X, y, rho):\n",
    "    '''Implement gradient of constrained function\n",
    "    \n",
    "    '''\n",
    "    beta_r = np.matrix(beta_r)\n",
    "    pp = -2*np.dot((y-np.dot(X,(beta_r).T)).T,X) + rho*(np.sum(beta_r) - 1)\n",
    "    aa = np.squeeze(np.asarray(pp))\n",
    "    return aa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b32d7050-cb44-4124-82a9-a5f66209289d",
   "metadata": {},
   "source": [
    "The following function is used to print statistics of the problem. It shows:\n",
    "\n",
    "* Reached **tolerance**: calculated with the norm of the gradient.\n",
    "* Computational **time** in seconds.\n",
    "* Number of **iterations**.\n",
    "* **Time per iteration**, in seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "788e4f03-7780-4331-8ef7-5fd601d4d026",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_constrained_stats(tol_pen, time_elapsed, iter_pen, sec_pen):\n",
    "    '''Print results\n",
    "    \n",
    "    '''\n",
    "    print('-'*53)\n",
    "    print(f\"| {'Tolerance':<10} | {'Time (sec)':<10} | {'Iterations':<10} | {'Sec/iter':<10} |\")\n",
    "    print('-'*53)\n",
    "    print(f\"| {tol_pen:<10} | {time_elapsed:<10} | {iter_pen:<10} | {sec_pen:<10} |\")\n",
    "    print('-'*53)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "0ae1f555-6bc7-4d45-8d66-1bb000adda07",
   "metadata": {},
   "outputs": [],
   "source": [
    "def constrained_problem(n_iter=2000, n_iter_pen=1000, epsilon=1e-4, epsilon_pen=1e-8, sigma=1e-5, delta=0.01, alpha_init=1e-5, eta=1.5, rho_init=1.5):\n",
    "    '''Implement constrained problem\n",
    "    \n",
    "    '''\n",
    "    (a,b) = X.shape\n",
    "    \n",
    "    # Initial values for the variables and data containers\n",
    "    beta_iter = np.zeros((b, n_iter_pen))\n",
    "    OF_iter = np.zeros(n_iter)\n",
    "    tol_iter = np.zeros(n_iter)\n",
    "    alpha_iter = np.zeros(n_iter)\n",
    "\n",
    "    # Time start\n",
    "    time_start = time.process_time()\n",
    "    \n",
    "    # Unconstrained problem iterations\n",
    "    t=0\n",
    "    tol_pen = 10000\n",
    "    while (t <= n_iter_pen-2) and (tol_pen>epsilon_pen):\n",
    "        t=t+1\n",
    "        if t==1:\n",
    "            beta = np.zeros(b)\n",
    "            rho = rho_init\n",
    "        else:\n",
    "            beta = beta_iter[:, t-1]\n",
    "        \n",
    "        # Gradient method\n",
    "        i=0\n",
    "        tol = 10000\n",
    "        while (i <= n_iter-2) and (tol>epsilon):\n",
    "            i=i+1\n",
    "            # compute gradient\n",
    "            grad = constrained_der(beta, X, y, rho)\n",
    "            # compute descent direction\n",
    "            ddirect = -grad\n",
    "            \n",
    "            # Armijo rule to adjust alph: now alpha is calculated with delta\n",
    "            alpha = alpha_init\n",
    "            while constrained(beta+alpha*ddirect, X, y, rho) > constrained(beta, X, y, rho) + alpha*sigma*np.dot(grad,ddirect.T):\n",
    "                alpha=alpha*delta\n",
    "                \n",
    "            # compute new point\n",
    "            beta = beta + alpha*ddirect\n",
    "            tol=np.linalg.norm(grad, ord=2)\n",
    "        \n",
    "        beta_iter[:, t] = beta\n",
    "        \n",
    "        # update penalization term\n",
    "        rho = rho*eta\n",
    "        \n",
    "        if t>1:\n",
    "            # compare with the previous solution\n",
    "            tol_pen=np.linalg.norm(beta_iter[:,t]-beta_iter[:,t-1],ord=2)\n",
    "            \n",
    "        \n",
    "        \n",
    "    # Measure elapsed time\n",
    "    time_elapsed = (time.process_time() - time_start)\n",
    "    \n",
    "    # Show stats\n",
    "    show_constrained_stats(round(tol_pen, 5), round(time_elapsed, 2), t, round(time_elapsed/i, 5))\n",
    "    \n",
    "    return beta_iter, t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "1cdaa8ea-d4c6-40c0-ab12-0695d1a9d1c1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------\n",
      "| Tolerance  | Time (sec) | Iterations | Sec/iter   |\n",
      "-----------------------------------------------------\n",
      "| 0.0        | 2273.0     | 54         | 2.27527    |\n",
      "-----------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Parameters for the algorithm\n",
    "n_iter = 1000         # maximum number of gradient iterations\n",
    "n_iter_pen = 1000     # maximum number of penalization iterations\n",
    "epsilon = 1e-8        # tolerance gradient limit\n",
    "epsilon_pen = 1e-8    # tolerance penalization limit\n",
    "sigma = 1e-2          # Armijo rule parameter\n",
    "delta = 3e-1          # Armijo rule parameter\n",
    "alpha_init = 3e-1     # initial alpha for Armijo rule\n",
    "eta = 1.5             # penalization parameter\n",
    "rho_init = 1          # initial rho\n",
    "\n",
    "# Run algorithm\n",
    "beta_iter, t = constrained_problem(n_iter, n_iter_pen, epsilon, epsilon_pen, sigma, delta, alpha_init, eta, rho_init)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5ded481-973f-49a5-922f-d56a0f771808",
   "metadata": {},
   "source": [
    "In order to see if the algorithm is working, we can check if the constraint is satisfied as the number of iterations increases. If $\\sum_{j=1}^K \\beta_j = 1$, the logarithmic of the sum of $\\beta$ should converge to zero. It means the unconstrained situation is becoming more constrained, what is the objective of this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "f17e6263-a739-45d3-b23e-0c704fc9b98f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe2cc0db610>]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfQAAAF4CAYAAACipAspAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAA66UlEQVR4nO3deXiU9b3//+dnMlkhZJnJYhIWkTUoBQRBLIgSU8rFOcdycaBSbFGhVrCI7bFitXKOflWKYm0URAXB1h5bK9ZftR6l8Yi0oj1gggsoixVREUI2QkIWZub+/TEkEhNgEjJzz0xej+uaa+ZeZubN+wJecy+f+zaWZVmIiIhIRHPYXYCIiIicPQW6iIhIFFCgi4iIRAEFuoiISBRQoIuIiEQBBbqIiEgUcIbyy3w+H0uWLCE9PZ0lS5a0WmZZFuvWraO0tJT4+HgWLFhA//79Q1meiIhIxArpFvrLL79Mbm5uu8tKS0s5ePAgRUVF/PCHP2TNmjWhLE1ERCSihSzQKyoqKCkpYfLkye0u37ZtGxMnTsQYw6BBg6irq6OqqipU5YmIiES0kO1yX79+PXPmzKG+vr7d5ZWVlbjd7pZpl8tFZWUlaWlpp/3cAwcOdKgOt9tNeXl5h94j6ltnqW+do751jvrWOZHWt5ycnHbnhyTQ33nnHVJSUujfvz87duxod532rkBrjGkzr7i4mOLiYgCWLVvW6kdAIJxOZ4ffI+pbZ6lvnaO+dY761jnR0reQBPquXbvYtm0bpaWlNDU1UV9fT1FREYsWLWpZx+VytfqFVFFR0e7WeUFBAQUFBS3THf1VFWm/xMKF+tY56lvnqG+do751TqT1zdYt9NmzZzN79mwAduzYwYsvvtgqzAFGjx7NK6+8wiWXXMKePXtISko64+52ERER8QvpsLWv27hxIwCFhYWMHDmSkpISFi1aRFxcHAsWLLCzNBERkYgS8kAfNmwYw4YNA/xB3swYw7x580JdjoiISFTQleJERESigAJdREQkCijQRUREooACXUREJAoo0EVERKKAAl1ERCQK2DoOPZxYtTXwyR4wgHGAMSc9Tkw7Trx2nHh8/XWMAxwx/kdM87Oj9bTD0e4lbUVERM6GAr3ZZ5/gK/qv4H+PMf5wj4n1Pzud/keME5yxEBsHsbGtXpvmefGJEB9/4jnB/0hIxMT5n+nRA5J6Qo9kiIvXDwcRkW5Egd6s7wAct90PlnXSw9d62ufzz/Od9LB8WD4LfN6T5nvB6z1p3olprxe8Hv/Dc+Lh9YL3OHg8WJ7jcPw4HG/yPzccAc9xrONN0NQETQ3Q2OB/z0na3tYG/w+EHj1PBHxP6NkLk5IOqf6HSU2HlHRITYOeKRiHjr6IiEQyBfoJJqkH9B/cufd2cS1nYnmO+4O9+dHQAA3H4FgdVt1ROFYLdbVwrPbEdB2UH8L6+COorfF/xskfGBMD6RmQlYPJyj3xnANZuZDmVtiLiEQABXoEMs4Tu+R7JLdddob3Wp7jcKQaqivgSCXWkSqoqvAH/qEDWHs+hMb6rwI/Ng4yz+HIoGH48vphzhsK5/RWyIuIhBkFejdjnLHgyvA/aPsDwLIsOFIJhw5gHfrC//zl5zS+swXr9Zf9QZ/YA84dhDlvCGbAEDh3MCYxKdR/FBEROYkCXVoxxkCqC1JdmMEXtMx3uVyUf/g+1t6P4J8fYX38EdZLv/f/ADAOGHIB5sJLMKMuxiSn2PgnEBHpnhToEhBjDCYzB5OZA+MvB8CqPwaf7MLa9QHWO1uwnl6F9d+rYfAFmNGXYEYq3EVEQkWBLp1mEpMgfyQmfyTWlXPgi31YW9/E2vZ3rN+uwvrdiXAfMwEz7jJMbKzdJYuIRC0FunQJYwzknYvJOxfryu/B5/uwtp0I9988gvXnZzBTZ2C+eYV/XL2IiHQpBbp0OWMM9D4X0/tEuH/0Hr4/P4P1349hvfycgl1EJAgU6BJUxhgY+g0cQ4b7g/3F5mD/I+bbMzATChXsIiJdQIEuIdEq2He9j+/P/431zONY//Mc5l+u8ge7LlUrItJpCnQJKWMMDBmOY/AF/mD///4b67crsd7fhmPuIkw7F8sREZEz0+W+xBbGGMyQ4Th+dh9m1jx4/x18d92EtWen3aWJiEQkBbrYyhiDo+Bfcdy2HGKc+B74Ob6/PIvl8575zSIi0kKBLmHB9B2A4xcPYS68BOuFp/E99J/+68yLiEhAFOgSNkxiEmb+f2C+fyN8/CG+/1qEtaPU7rJERCKCAl3CijEGx4RCHD9/EJJT8D20FN//bLC7LBGRsKdAl7Bkcvvg+PkKzJgJWM8/hW/L/9pdkohIWNOwNQlbJj4erl2MVVPtv3ysKxMz+Hy7yxIRCUvaQpewZpyxOG64DTKy8a26F+vg53aXJCISlhToEvZMj544Ft0JMTH4iu7COlpjd0kiImFHgS4RwWRk41h4O1RV+LfUjx+3uyQRkbASkmPoTU1NLF26FI/Hg9frZdy4ccycObPVOjt27GD58uVkZmYCMHbsWGbMmBGK8iRCmPOGYK5djPX4/Vjri2DeT3T9dxGRE0IS6LGxsSxdupSEhAQ8Hg933nknI0aMYNCgQa3WGzp0KEuWLAlFSRKhHGMm4Cv7EuuFpyHzHMy/zba7JBGRsBCSQDfGkJCQAIDX68Xr9WrLSjrNTP13KPsS66Xf48s8B8fFl9ldkoiI7YxlWVYovsjn83Hrrbdy8OBBvvWtbzFnzpxWy3fs2MGKFStwuVykpaVx9dVX07t37zafU1xcTHFxMQDLli2jqampQ3U4nU48Hk/n/yDdVLj1zTp+nKq7bub4R++RdtcjxA0dbndJ7Qq3vkUK9a1z1LfOibS+xcXFtTs/ZIHerK6ujgceeIBrrrmGPn36tMw/duwYDoeDhIQESkpKWL9+PUVFRWf8vAMHDnTo+91uN+Xl5R2uu7sLx75ZdbX47vkJGINjaREmLt7uktoIx75FAvWtc9S3zom0vuXk5LQ7P+Rnuffo0YP8/Hy2b9/ean5SUlLLbvlRo0bh9XqpqdHwJDk106MnjjkLTux+/4Pd5YiI2CokgV5TU0NdXR3gP+P9/fffJzc3t9U61dXVNO8s2Lt3Lz6fj+Tk5FCUJxHM5I/AXHw51sY/YX3+id3liIjYJiQnxVVVVbFy5Up8Ph+WZXHxxRdz4YUXsnHjRgAKCwt5++232bhxIzExMcTFxbF48WKdOCcBMTOvxfrgHXy/WYljyS8xjhi7SxIRCbmQH0PvajqGHhrh3jffP97AWrMC8935OCb/i93ltAj3voUr9a1z1LfOibS+hc0xdJFgMBdNhPNHYf3pt1gVh+0uR0Qk5BToEhWMMTi+dwNYFr7fPUqE73gSEekwBbpEDePOwvzb9+D9bVjb/m53OSIiIaVAl6hiJv8L9B2A9czjWHW1dpcjIhIyCnSJKiYmBsf3b4S6o1jPrbO7HBGRkFGgS9QxffpjrrgS6+9/xdr1vt3liIiEhAJdopL5l6sgIxvfb1ZiNTXaXY6ISNAp0CUqmfh4HHNugLIDWG++Znc5IiJBp0CX6DV0hP8EuU0vaxibiEQ9BbpELWMMZtK34cB+2LPD7nJERIJKgS5RzYyZCEk9sDb9j92liIgElQJdopqJj8eML8Aq2YJ1pMruckREgkaBLlHPXDoFvF6sv220uxQRkaBRoEvUM9m5kD8Ca/OrWF6v3eWIiASFAl26BcekqVBVDu9ttbsUEZGgUKBL9zB8DKS58W162e5KRESCQoEu3YKJicFM/Bbs3I518Au7yxER6XIKdOk2zIRCiInBeuMVu0sREelyCnTpNkxKGmbUeKwtxViNur67iEQXBbp0K2bSt+FYHdbWzXaXIiLSpRTo0r0MHAY5fbBe1/XdRSS6KNClW/Ff330q7P8Y9u2xuxwRkS6jQJdux4ybBPGJWK9rCJuIRA8FunQ7JjEJc/EkrK1/w6qtsbscEZEuoUCXbslc+m3wHMd68zW7SxER6RIKdOmWTF4/GJiP9cb/YPl8dpcjInLWFOjSbZlvFsLhg/4T5EREIpwCXbotkz8CAGvX+/YWIiLSBRTo0m2Z1HTIzsP66D27SxEROWsKdOnWzJDhsGcnlsdjdykiImfFGYovaWpqYunSpXg8HrxeL+PGjWPmzJmt1rEsi3Xr1lFaWkp8fDwLFiygf//+oShPujEz5AKsTS/7LzIzYKjd5YiIdFpIAj02NpalS5eSkJCAx+PhzjvvZMSIEQwaNKhlndLSUg4ePEhRURF79uxhzZo13HvvvaEoT7qzQRcA/uPoRoEuIhEsJLvcjTEkJCQA4PV68Xq9GGNarbNt2zYmTpyIMYZBgwZRV1dHVVVVKMqTbswk94K8fjqOLiIRLyRb6AA+n49bb72VgwcP8q1vfYuBAwe2Wl5ZWYnb7W6ZdrlcVFZWkpaW1mq94uJiiouLAVi2bFmr9wTC6XR2+D0S3X07OnIsx179E65eyZi4+C797GjuWzCpb52jvnVOtPQtZIHucDi4//77qaur44EHHmD//v306dOnZXl7d776+lY8QEFBAQUFBS3T5eXlHarD7XZ3+D0S3X2z+g6EpibKt27BDL6gSz87mvsWTOpb56hvnRNpfcvJyWl3fsjPcu/Rowf5+fls37691XyXy9WqoRUVFW22zkWCYuAwMA6sjzQeXUQiV0gCvaamhrq6OsB/xvv7779Pbm5uq3VGjx7N5s2bsSyL3bt3k5SUpECXkDBJPaDveTqOLiIRLSS73Kuqqli5ciU+nw/Lsrj44ou58MIL2bhxIwCFhYWMHDmSkpISFi1aRFxcHAsWLAhFaSIAmMEXYBX/GauxAROfYHc5IiIdFpJA79u3L8uXL28zv7CwsOW1MYZ58+aFohyRNsyQ4VivPg97P4RhI+0uR0Skw3SlOBHwX1QmJgZrl3a7i0hkUqCLACYhEc4dpBPjRCRiKdBFTjCDL4B9e7GO1dldiohIhwUU6C+99BL79u0DYPfu3dxwww3ceOON7N69O5i1iYSUGTIcLB/s2Wl3KSIiHRZQoP/lL38hMzMTgGeeeYZp06Yxffp01q9fH8zaRELrvCHgjNVxdBGJSAEF+rFjx0hKSqK+vp59+/bx7W9/m8svv5wDBw4Euz6RkDGxcXDeEI1HF5GIFFCgu1wudu3axZtvvsnQoUNxOBwcO3YMh0OH4CW6mCEXwOf7sGpr7C5FRKRDAkrkOXPm8OCDD/KnP/2JGTNmAFBSUsKAAQOCWpxIqPmPo1uw+wO7SxER6ZCALiwzatQoHnvssVbzxo0bx7hx44JSlIht+g2EuHisj97HjBpvdzUiIgHr0JXi6uvrOXr0aKs7o2VlZXV5USJ2Mc5YGJiv4+giEnECCvTPP/+coqIiPv300zbL/vCHP3R5USJ2MoOHYz3/FFZNFaaXbhAkIpEhoGPoa9asYdiwYTz55JMkJSWxbt06rrjiChYuXBjs+kRCzgwZDqCrxolIRAko0D/99FO+973v0aNHDyzLIikpiTlz5mjrXKJTn/6Q2AN2KdBFJHIEFOixsbF4vV4AkpOTKS8vx7Isamtrg1qciB1MTAwMGqbj6CISUQI6hj5kyBDeeustJk2axLhx47j33nuJjY1l2LBhwa5PxBZmyAVY7/4fVuVhTHqG3eWIiJxRQIH+k5/8pOX1VVddRe/evWloaODSSy8NWmEidjKDh2PhP45uxl9udzkiImcU0C73P//5z1+9weFg4sSJFBYW8te//jVohYnYKrcv9EzWcXQRiRgBBfqGDRs6NF8k0hmHAwZdgKVAF5EIcdpd7h984L/8pc/na3nd7NChQyQmJgavMhGbmXMHYpVswaqrxfToaXc5IiKnddpAf/TRRwFoampqeQ1gjCElJYVrr702uNWJ2Mjk9sMC+OJTGKQTQEUkvJ020FeuXAnAI488wo033hiSgkTCRl4/AKwv9mEU6CIS5gI6hn7jjTfi8Xj48MMP2bJlCwANDQ00NDQEtTgRW6WmQ49k+Hyf3ZWIiJxRQMPW9u/fzy9/+UtiY2OpqKhg/Pjx7Ny5kzfeeIObb7452DWK2MIYA3n9sBToIhIBAtpCf+KJJ5g1axYPPfQQTqf/N0B+fj4fffRRUIsTsZvJ6wdf7Mfy+ewuRUTktAIK9M8//5wJEya0mpeQkEBTU1NQihIJG7l9obEeKsrsrkRE5LQCCvSMjAz++c9/tpq3d+9esrOzg1KUSLgwJ06M03F0EQl3AR1DnzVrFsuWLeOKK67A4/Hwpz/9ib/+9a9cf/31wa5PxF45fcAYrM/3YUaOs7saEZFTCmgL/cILL+S2226jpqaG/Px8Dh8+zH/8x3/wjW98I9j1idjKxCdAxjk6MU5Ewl5AW+gA/fv3p3///sGsRSQ85fXzX1xGRCSMBRToHo+HDRs28Oabb1JVVUVaWhrjx49n+vTpxMXFnfH95eXlrFy5kurqaowxFBQUMHXq1Fbr7Nixg+XLl5OZmQnA2LFjmTFjRif+SCJdy+T2xSp9C6uxERMfb3c5IiLtCijQn3jiCQ4cOMA111xDRkYGhw8f5oUXXmDNmjUsWLDgjO+PiYnh6quvpn///tTX17NkyRKGDx9OXl5eq/WGDh3KkiVLOvcnEQkSk9cPy7LgwH44d6Dd5YiItCugQN+6dSsPP/wwPXr0ACAvL4+BAwfy4x//OKAvSUtLIy0tDYDExERyc3OprKxsE+giYan5ErCff4JRoItImAoo0FNTU2lsbGwJdPDfsKU5pDuirKyMTz75hAEDBrRZtnv3bm655RbS0tK4+uqr6d27d4c/X6TLubMgPkHH0UUkrBnLsqz2Fpx8u9S9e/fy97//nSlTpuByuaioqODVV1/lkksu4corrwz4yxoaGli6dCnTp09n7NixrZYdO3YMh8NBQkICJSUlrF+/nqKiojafUVxcTHFxMQDLli3r8MVtnE4nHo+nQ+8R9a3y1vkQF0/63Y906H3dvW+dpb51jvrWOZHWt1Odu3bKQF+4cOEZP9QYwyOPBPYfnMfj4Ze//CXf+MY3mDZt2hnXX7hwIffddx+9evU67XoHDhwI6Pubud1uysvLO/QeUd98v3kEq/QtHA8+7b/Ge4C6e986S33rHPWtcyKtbzk5Oe3OP+Uu9+Zbp3YFy7JYvXo1ubm5pwzz6upqUlJSMMawd+9efD4fycnJXVaDyFnJ6wd/2whHKiHVZXc1IiJtBDwO/Wzs2rWLzZs306dPH2655RYArrrqqpZfRIWFhbz99tts3LiRmJgY4uLiWLx4cYe2hESCyeT1wwL/JWAV6CIShkIS6EOGDOHZZ5897TpTpkxhypQpoShHpONy+wH4LwF7/oX21iIi0o6ALv0q0t2ZHj0hza0z3UUkbJ0y0Ldt29byOpLO/hMJmrx+uqa7iIStUwb6ww8/3PL6uuuuC0kxIuHM5PWFLz/H8hy3uxQRkTZOeQw9NTWVV155hby8PLxeb6tx6Sc7//zzg1acSFjJ7QdeDxz8ouXqcSIi4eKUgb5gwQKeffZZXn75ZTweD48++mibdToyDl0k0pm8c7EA64tPMQp0EQkzpwz0wYMH84tf/AKAH//4x612wYt0S1k5EOP0D10be6nd1YiItBLQsLXmMC8vL6eyspL09HTcbndQCxMJN8bphHN668Q4EQlLAQV6dXU1v/rVr9i9ezfJyckcPXqUQYMGcdNNN5Genh7sGkXChsnrh/XRe3aXISLSRkDj0B9//HH69u3LunXrePzxx1m3bh39+vXjiSeeCHZ9IuElrx9UV2DVHbW7EhGRVgIK9F27dvH973+fhIQEABISEpgzZw67d+8OanEi4ablZLjPdYEZEQkvAQV6jx49+Pzzz1vNO3DgAElJSUEpSiRs5fYF0HF0EQk7AR1D/9d//VfuvvtuLr/8cjIyMjh8+DCbNm1i1qxZwa5PJLykpEHPXvDFPrsrERFpJaBALygoIDs7m7///e/s37+ftLQ0brrpJl1URrodY4wuASsiYSngu62df/75CnARTpzpvvlVLJ8P49D9jUQkPOh/I5GOyusHTY1QftDuSkREWijQRTrInDgxDu12F5EwokAX6ahz+oBx6Di6iIQVBbpIB5n4eMg6R4EuImEloJPijh07xssvv8y+fftoaGhoteyOO+4ISmEi4czk9sP67J92lyEi0iKgQH/wwQfx+XxcdNFFxMXFBbsmkfCX1xdKtmA11GMSEu2uRkQksEDfs2cPa9euxekMeJSbSFQzef2wLAsO7If+g+0uR0QksGPoQ4YM4Ysvvgh2LSKRI7cfoEvAikj4CGiTe8GCBdx3330MGDCA1NTUVstmzJgRjLpEwpsrExISNXRNRMJGQIH+zDPPUFFRQUZGBvX19S3zjTFBK0wknBmHA3L6YH35md2liIgAAQb6li1b+PWvf01aWlqw6xGJGMadjfXPj+wuQ0QECPAYelZWFjExMcGuRSSyuLOg8jCW12t3JSIigW2hT5gwgeXLlzNlypQ2x9B1wxbpttyZ4PNBVbk/3EVEbBRQoL/66quA/1j6yYwxPPLII11flUgEMO4sLIDyQwp0EbFdQIG+cuXKYNchEnlOhLhVfgidHioidtO13EU6K80NxgEVZXZXIiIS2Bb6DTfccMpljz766BnfX15ezsqVK6mursYYQ0FBAVOnTm21jmVZrFu3jtLSUuLj41mwYAH9+/cPpDwRWxinE9Ld/l3uIiI2CyjQf/zjH7earqqq4uWXX+aSSy4J6EtiYmK4+uqr6d+/P/X19SxZsoThw4eTl5fXsk5paSkHDx6kqKiIPXv2sGbNGu69994O/FFEbODOwlKgi0gYCCjQ8/Pz28wbNmwY99xzT5st7fakpaW1jGFPTEwkNzeXysrKVoG+bds2Jk6ciDGGQYMGUVdXR1VVlca+S1gzrkysnaV2lyEiEligt/tGp5Oyso4fOywrK+OTTz5hwIABreZXVlbidrtbpl0uF5WVlW0Cvbi4mOLiYgCWLVvW6j2B1t3R94j6diq1fc6lbstruHolY+Li2yxX3zpHfesc9a1zoqVvAQX6H/7wh1bTjY2NlJaWMnLkyA59WUNDAytWrGDu3LkkJSW1WmZZVpv127u0bEFBAQUFBS3T5eXlHarB7XZ3+D2ivp2KLykZgPLdH2Ky89osV986R33rHPWtcyKtbzk5Oe3ODyjQKyoqWk3Hx8czbdo0Jk6cGHABHo+HFStWMGHCBMaOHdtmucvlatXQiooK7W6XsPfVWPQyaCfQRURCJeC7rZ0Ny7JYvXo1ubm5TJs2rd11Ro8ezSuvvMIll1zCnj17SEpKUqBL+NNYdBEJEwEF+gcffEBmZiaZmZlUV1fz9NNP43A4mD17dptLwbZn165dbN68mT59+nDLLbcAcNVVV7VskRcWFjJy5EhKSkpYtGgRcXFxZ/0jQiQkUtLA6dTQNRGxXUCBvnbtWm6//XYAnnrqKcA/FO2xxx7j1ltvPeP7hwwZwrPPPnvadYwxzJs3L5ByRMKGcTggPVOBLiK2CyjQm89A93q9vPvuu6xatQqn08n1118f7PpEwp/GootIGAgo0BMTE6muruazzz4jLy+PhIQEPB4PHo8n2PWJhD3jzsLav9fuMkSkmwso0KdMmcJtt92Gx+Nh7ty5AHz00Ufk5uYGszaRyODOgtqjWA3HMAlJZ15fRCQIAgr0K6+8kosuugiHw0F2djYA6enp/OhHPwpqcSIRofnWqeVlkNfP1lJEpPsK+EpxXx/IfqqB7SLdjXFnfnVfdAW6iNhEt08VOVsnjUUXEbGLAl3kbPXsBfEJGromIrZSoIucJWOMhq6JiO0CCvR3332XAwcOtJp34MAB3nvvvaAUJRJx3FlQ0fG7D4qIdJWAAn3t2rUkJia2mpeQkMDatWuDUpRIpDHuLCg/1O5dA0VEQiGgQD9y5EibG6WkpaVRXV0djJpEIo8rExrqoe6o3ZWISDcVUKBnZWXxwQcftJq3Y8cOMjMzg1KUSKQxLWPRdRxdROwR0Dj0f//3f+eBBx7g8ssvJysri0OHDvH666/rjmgizU4O9H4D7a1FRLqlgLbQx4wZwx133EFDQwMlJSU0NDRw++23M2bMmGDXJxIZmsei68Q4EbFJwFeKGzBgAAMGDAhmLSIRyyQmQY9k7XIXEducMtCff/55pk+fDsAf/vCHU37ArFmzur4qkUjkytRYdBGxzSkDvaKiot3XInIK7iw48KndVYhIN3XKQJ8/f37La538JnJmxp2F9d5WLJ8P49BFGEUktAI+ht7Y2MjBgwdpaGhoNX/w4MFdXpRIRHJngec41FRBqsvuakSkmwko0N944w2efPJJnE4ncXFxrZY9+uijQSlMJNIYd9aJ26iWKdBFJOQCCvSnn36an/70pwwfPjzY9YhErpNuo2oGDLW5GBHpbgI60Od0OsnPzw92LSKRzZXhf9aZ7iJig4ACfdasWfzmN7+hpqYm2PWIRCwTFw8paQp0EbFFQLvcc3JyePbZZ3n11VfbLDvdGHWRbkf3RRcRmwQU6A8//DATJ05k/PjxbU6KE5GvGFcW1scf2l2GiHRDAQV6bW0ts2bNwhgT7HpEIps7C7b9DcvrxcTE2F2NiHQjAR1DnzRpEps3bw52LSKRz50JPh9UldtdiYh0MwFtoe/du5dXXnmF559/ntTU1FbL/uu//isYdYlEpK/Goh/66paqIiIhEFCgT548mcmTJwe7FpHId/JYdJtLEZHuJaBAnzRpUpDLEIkSaW4wDg1dE5GQC/ha7u+++y779u1rcy33QG6fumrVKkpKSkhJSWHFihVtlu/YsYPly5eTmZkJwNixY5kxY0agpYmEDeN0QrpbgS4iIRdQoK9du5a33nqLYcOGER8f3+EvmTRpElOmTGHlypWnXGfo0KEsWbKkw58tEnbcWVgVZXZXISLdTECB/uabb7J8+XLcbnenviQ/P5+yMv0HJ92DcWVi7Sy1uwwR6WYCCvTk5GR69OgR1EJ2797NLbfcQlpaGldffTW9e/dud73i4mKKi4sBWLZsWYd/ZDidzk7/MOnO1LfA1fY5l7otr+Hqlay+dZL61jnqW+dES99OGeiHDn11DHDatGkUFRXxne98h5SUlFbrZWWd/dCcc889l1WrVpGQkEBJSQn3338/RUVF7a5bUFBAQUFBy3R5ecfG+7rd7g6/R9S3jvAlJQNQvvtDMs4fob51gv6+dY761jmR1recnJx2558y0BctWtRmXklJSZt5XXEt96SkpJbXo0aNYu3atdTU1NCrV6+z/myRUGs1Fl1EJEROGeihvOlKdXU1KSkpGGPYu3cvPp+P5OTkkH2/SJc6aSy6iEioBHQM/cknn+Taa69tM3/9+vXMnTv3jO9/6KGH2LlzJ0ePHuVHP/oRM2fOxOPxAFBYWMjbb7/Nxo0biYmJIS4ujsWLF+u68RK5UtLA6YRynQgqIqETUKC/8cYb7Qb65s2bAwr0xYsXn3b5lClTmDJlSiCliIQ943BAeqZ2uYtISJ020P/3f/8XAK/X2/K6WVlZmXaLi5yK7osuIiF22kD/29/+BoDH42l53SwlJYWFCxcGrzKRCGbcWVj799pdhoh0I6cN9KVLlwLw+9//nu9+97shKUgkKrizoPYovvo6uysRkW7ilPdDtyyr5fXMmTPx+XztPkSkHSfOdPce+tLmQkSkuzjlFvrcuXN56qmnALjqqqtO+QGhHN4mEimMOxML8JZ9CT1T7S5HRLqBUwb6yXdFe+SRR0JSjEjUOLGF7jt0APoPtbkYEekOTrnLvfm6tj6fj5UrV5KamkpGRkabh4i0o2cviE/wb6GLiITAKQO9ZQWHg7KyslbH1EXk9IwxkJ6B9/BBu0sRkW7ijIEOMGPGDJ544gkOHz6sk+JEAuXOUqCLSMgEdKW4xx57DPBfGe7rdFKcSPuMKwPvJ7sC+9UsInKWAgp0nRQn0gmuTKzao1j1xzCJSWdeX0TkLAQU6Dr5TaQTXJn+54oyyOtnaykiEv0CCnSAbdu2sXPnTmpqalrNv/HGG7u8KJFoYFz+sehUHFagi0jQBXR4749//COPP/44Pp+Pt99+m549e/Luu++SlKTdiCKndGIL3arQTVpEJPgC2kJ//fXXueOOO+jTpw+bNm1i7ty5fPOb32TDhg3Brk8kcvVKhbg4/xa6iEiQBbSFXldXR58+fQBwOp14PB4GDBjAzp07g1qcSCQzxhDjztYWuoiEREBb6NnZ2Xz22Wf07t2b3r17s3HjRnr27EnPnj2DXZ9IRIvJzMarLXQRCYGAAn3WrFkcPXoUgNmzZ1NUVERDQwPz5s0LanEikS4mIxs+3mV3GSLSDQQU6KNGjWp5PXDgQB5++OGgFSQSTWIyz4GjR7AaGzHx8XaXIyJRLKBAP3So/WOAsbGxpKam4nDoWlgi7XFkZPtfVB6Gc/LsLUZEolpAgb5o0aJTLnM4HFx44YXMmzeP1NTUrqpLJCrEZJ7jf1FxSIEuIkEVUKBff/317Ny5kxkzZuB2uykvL+e5555j8ODB5Ofn87vf/Y61a9fy05/+NNj1ikSUmBNb6FbFYYzNtYhIdAtoX/mzzz7LD3/4Q7Kzs3E6nWRnZzN//nw2bNhAbm4uCxYs0BA2kXY40lwQE+PfQhcRCaKAAt2yLA4fbj30pry8vOX2qQkJCXi93q6vTiTCmZgYSM/QxWVEJOgC2uU+depU7rrrLiZNmoTL5aKyspLXX3+dqVOnAlBSUsKgQYOCWqhIxErPwKoos7sKEYlyAQX6v/3bv9G3b1/eeustPvnkE1JTU7nhhhsYMWIEABdddBEXXXRRMOsUiVjGnYm1o9TuMkQkygV8t7URI0a0BLiIdEB6Jhypwjp+HBMba3c1IhKlAgp0j8fD888/z+bNm6mqqiItLY2JEycyffp0nM6AfxOIdE/uTLAsqDoMmTl2VyMiUSqgNH766af5+OOPmT9/PhkZGRw+fJgNGzZw7Ngx5s6dG+QSRSJbq/uiK9BFJEgCCvS3336b+++/n+TkZABycnI499xzueWWWwIK9FWrVlFSUkJKSgorVqxos9yyLNatW0dpaSnx8fEsWLCA/v37d+xPIhKumu+LXn5IY9FFJGgCHrZ2NiZNmsTPf/7zUy4vLS3l4MGDFBUV8cMf/pA1a9ac1feJhJVUFxiH//KvIiJBEtAW+sUXX8wvf/nLVleK27BhA+PGjQvoS/Lz8ykrO/WwnW3btjFx4kSMMQwaNIi6urqWY/Uikc44nZDmgnINXROR4Ako0OfMmcOGDRtYu3YtVVVVpKenM378eGbMmNElRVRWVuJ2u1umm8e6K9AlargysCoV6CISPAEFutPpZNasWcyaNatlns/n449//GOreZ3V3i59Y9o/2lhcXExxcTEAy5Yta/VDIBBOp7PD7xH1rbOa+3Yktw9NO0rVwwDp71vnqG+dEy196/SYM6/Xy/PPP98lge5yuSgvL2+ZrqioOOXWeUFBAQUFBS3TJ78vEM2HDKRj1LfOae6br0cvrIrDHD50yH85WDkt/X3rHPWtcyKtbzk57Y+WCYsbmY8ePZrNmzdjWRa7d+8mKSlJu9slurgyweeDqsj5T0NEIktIrgrz0EMPsXPnTo4ePcqPfvQjZs6cicfjAaCwsJCRI0dSUlLCokWLiIuLY8GCBaEoSyRkWo1Fd2fZXY6IRKHTBvoHH3xwymXNgRyIxYsXn3a5MYZ58+YF/HkiEad5LHpFmcaii0hQnDbQH3300dO+ORpOIhAJifQM/7PuuiYiQXLaQF+5cmWo6hCJaiY2FlLSFegiEjRhcVKcSLfgztR90UUkaBToIiFi0jO0hS4iQaNAFwkVdyZUlmP5fHZXIiJRSIEuEirpmeD1wJEquysRkSikQBcJEeP2D12j4pC9hYhIVFKgi4RKy1h03UZVRLqeAl0kVNKbt9B1YpyIdD0FukiImPh4SE5RoItIUCjQRULJpbHoIhIcCnSRUHJpLLqIBIcCXSSEjCsTKg5jWZbdpYhIlFGgi4SSKxOON8HRarsrEZEoo0AXCSHjaj7TXUPXRKRrKdBFQql5LHq5jqOLSNdSoIuEUvMWeqUCXUS6lgJdJIRMYhIk9QRtoYtIF1Ogi4SaK0Nj0UWkyynQRULNlQWVOilORLqWAl0kxIwrA8rLNBZdRLqUAl0k1NyZ0FgPx2rtrkREoogCXSTETPNd13RinIh0IQW6SKi5dRtVEel6CnSRUGu+uIwCXUS6kAJdJNSSekJ8orbQRaRLKdBFQswYA27dF11EupYCXcQO6bovuoh0LQW6iA2MO1N3XBORLqVAF7GDKxOO1WLVH7O7EhGJEs5QfdH27dtZt24dPp+PyZMnc+WVV7ZavmPHDpYvX05mpv8M4LFjxzJjxoxQlScSUsaViQX+3e55/WyuRkSiQUgC3efzsXbtWu644w5cLhe33XYbo0ePJi8vr9V6Q4cOZcmSJaEoScReriz/8+GDCnQR6RIh2eW+d+9esrOzycrKwul0Mn78eLZu3RqKrxYJTzl9ICYG65PddlciIlEiJFvolZWVuFyulmmXy8WePXvarLd7925uueUW0tLSuPrqq+ndu3ebdYqLiykuLgZg2bJluN3uDtXidDo7/B5R3zrrdH2rOHcQ5tO9pKuvbejvW+eob50TLX0LSaC3d1cpY0yr6XPPPZdVq1aRkJBASUkJ999/P0VFRW3eV1BQQEFBQct0eXl5h2pxu90dfo+ob511ur75+g3CeuN/OHzwS4wzNsSVhTf9fesc9a1zIq1vOTk57c4PyS53l8tFRUVFy3RFRQVpaWmt1klKSiIhIQGAUaNG4fV6qampCUV5IrYwA4bC8Sb49GO7SxGRKBCSQD/vvPP48ssvKSsrw+PxsGXLFkaPHt1qnerq6pYt+b179+Lz+UhOTg5FeSL2GDAUAOvjD20uRESiQUh2ucfExHDttddyzz334PP5uOyyy+jduzcbN24EoLCwkLfffpuNGzcSExNDXFwcixcvbrNbXiSamJQ0yMjG2vshFH7H7nJEJMKFbBz6qFGjGDVqVKt5hYWFLa+nTJnClClTQlWOSFgwA4ZifVCCZVn6ASsiZ0VXihOx04ChcPQIlH1pdyUiEuEU6CI2MgPyAfy73UVEzoICXcRO2Xn++6PrxDgROUsKdBEbGYcDzhuCtWen3aWISIRToIvYzAzMh4OfY9Xqugsi0nkKdBGbmfP849H5+CN7CxGRiKZAF7FbvwEQ49RudxE5Kwp0EZuZuHjoN0BXjBORs6JAFwkD5ryhsG8P1vEmu0sRkQilQBcJA2bAUPB44NO9dpciIhFKgS4SDs4bAugCMyLSeQp0kTBgeqVCVq4CXUQ6TYEuEibMgCHw8YcttxEWEekIBbpIuDhvKNQehYNf2F2JiEQgBbpImDADm2/UovHoItJxCnSRcJGVCz17gY6ji0gnKNBFwoQxxn+jFgW6iHSCAl0kjJiB+VB2AKum2u5SRCTCKNBFwohu1CIinaVAFwknfQeAM1YnxolIhynQRcKIiY2FfgN1HF1EOkyBLhJmzICh8OnHWE2NdpciIhFEgS4SZsyAoeD1wL49dpciIhFEgS4SbnSjFhHpBAW6SJgxPXvBOb0V6CLSIQp0kTBkBgz136ilscHuUkQkQijQRcKQuWA0HKvD99Pv43tiBda7W7E8x+0uS0TCmNPuAkSkLTNyHI5b7sP6xxtY77yJ9X9vQI9kzIXjMRdNhIHDMA79HheRryjQRcKUGTQMM2gY1lXzYed2rH9s9gf85lch1YUZcRHk9MWckwc5vSE51X89eBHplhToImHOOGNh+BjM8DFYjQ1Y7231B/vbm6ChHqt5xaSecE4eJqcPZOdhMs+BNDekuaBnL23Ri0S5kAX69u3bWbduHT6fj8mTJ3PllVe2Wm5ZFuvWraO0tJT4+HgWLFhA//79Q1WeSEQw8QmYMRNgzAQsy4LqSvhyP9aXn8OBz7AOfoa1/R9wdONXQQ/gdEKqC9JcmDS3/3WvFP9u/J7JkJQMPU88kpIxTv3WF4k0IflX6/P5WLt2LXfccQcul4vbbruN0aNHk5eX17JOaWkpBw8epKioiD179rBmzRruvffeUJQnEpGMMf6t7zQXJn9kq2XW0RooPwTVFVhV5VBVAVXlWFUVWJ/s9v8QON7kX7e9D09IhIQk/3N8wonpREx84ol58RAbD3FxEBsLsXEtD9M8HRMDMbH+Z6cTYpwnnmP8rx0xEOPwPzti/PNPzLOsdqsSkdMISaDv3buX7OxssrKyABg/fjxbt25tFejbtm1j4sSJGGMYNGgQdXV1VFVVkZaWFooSRaKKSe4Fyb2AgbR3VN2yLGish7paqD0KdTVYtUf903U1/nkN9dDYgNVQ71+38rB/GN2J+TQ1geVr+9ldUH8ZgDHgcJz07Phq2jjAcOLZnPQ4MR/Ten7z57XMP2mdlqa1M82J9U56os15Cl+bPtN5DGe7/DQqY2PxHj/NaAidY9GuSqcTr8cTtM933HwXJiExaJ/fLCSBXllZicvlapl2uVzs2bOnzTput7vVOpWVlW0Cvbi4mOLiYgCWLVvW6j2BcDqdHX6PqG+dFe19szwerOON0NiIdbwJq6kRq6kJjjdheTzg9fiH23k8/mnP8RPzvVg+L/i8/tde/zM+L5bHg8OAz+MBnw/L5wPL8q9rWf73WYDPB1jgs/w/LCzL/0PF8vmXW9aJ5Sd+dDQvxzpp+UnrNf8SaZ7m6+u082Pl63sS2uxZ+PryMzX0jCucdqkxhrhTfYb2epzSafvWBVLd7ugJ9PZ2n339bNxA1gEoKCigoKCgZbq8vLxDtbjd7g6/R9S3zupWfTNOiHdCfI+z/qhu1bcupL51TrD7VlFbB7V1XfZ5OTk57c4PyWmvLpeLioqKlumKioo2W94ul6tVQ9tbR0RERNoXkkA/77zz+PLLLykrK8Pj8bBlyxZGjx7dap3Ro0ezefNmLMti9+7dJCUlKdBFREQCFJJd7jExMVx77bXcc889+Hw+LrvsMnr37s3GjRsBKCwsZOTIkZSUlLBo0SLi4uJYsGBBKEoTERGJCiEbbDpq1ChGjRrVal5hYWHLa2MM8+bNC1U5IiIiUUWXjhIREYkCCnQREZEooEAXERGJAgp0ERGRKKBAFxERiQIKdBERkSigQBcREYkCCnQREZEooEAXERGJAsZq7zZnIiIiElG63Rb6kiVL7C4hIqlvnaO+dY761jnqW+dES9+6XaCLiIhEIwW6iIhIFOh2gV5QUGB3CRFJfesc9a1z1LfOUd86J1r6ppPiREREokC320IXERGJRk67CwiV7du3s27dOnw+H5MnT+bKK6+0u6SwtGrVKkpKSkhJSWHFihUA1NbW8qtf/YrDhw+TkZHBzTffTM+ePW2uNLyUl5ezcuVKqqurMcZQUFDA1KlT1bszaGpqYunSpXg8HrxeL+PGjWPmzJnqW4B8Ph9LliwhPT2dJUuWqG8BWLhwIQkJCTgcDmJiYli2bFnU9K1b7HL3+XzcdNNN3HHHHbhcLm677TZuuukm8vLy7C4t7OzcuZOEhARWrlzZEuhPP/00PXv25Morr+SFF16gtraWOXPm2FxpeKmqqqKqqor+/ftTX1/PkiVLuOWWW9i0aZN6dxqWZdHY2EhCQgIej4c777yTuXPn8n//93/qWwBeeuklPv7445a/c/q3emYLFy7kvvvuo1evXi3zoqVv3WKX+969e8nOziYrKwun08n48ePZunWr3WWFpfz8/Da/TLdu3cqll14KwKWXXqretSMtLY3+/fsDkJiYSG5uLpWVlerdGRhjSEhIAMDr9eL1ejHGqG8BqKiooKSkhMmTJ7fMU986J1r61i12uVdWVuJyuVqmXS4Xe/bssbGiyHLkyBHS0tIAf3DV1NTYXFF4Kysr45NPPmHAgAHqXQB8Ph+33norBw8e5Fvf+hYDBw5U3wKwfv165syZQ319fcs89S0w99xzDwBXXHEFBQUFUdO3bhHo7R1VMMbYUIlEu4aGBlasWMHcuXNJSkqyu5yI4HA4uP/++6mrq+OBBx5g//79dpcU9t555x1SUlLo378/O3bssLuciHL33XeTnp7OkSNH+H//7/+Rk5Njd0ldplsEusvloqKiomW6oqKi5deYnFlKSgpVVVWkpaVRVVXV6tiTfMXj8bBixQomTJjA2LFjAfWuI3r06EF+fj7bt29X385g165dbNu2jdLSUpqamqivr6eoqEh9C0B6ejrg/7c5ZswY9u7dGzV96xbH0M877zy+/PJLysrK8Hg8bNmyhdGjR9tdVsQYPXo0b7zxBgBvvPEGY8aMsbmi8GNZFqtXryY3N5dp06a1zFfvTq+mpoa6ujrAf8b7+++/T25urvp2BrNnz2b16tWsXLmSxYsXc/7557No0SL17QwaGhpaDlE0NDTw3nvv0adPn6jpW7c4yx2gpKSEp556Cp/Px2WXXcb06dPtLiksPfTQQ+zcuZOjR4+SkpLCzJkzGTNmDL/61a8oLy/H7Xbzk5/8JCKHdATTRx99xJ133kmfPn1aDudcddVVDBw4UL07jU8//ZSVK1fi8/mwLIuLL76YGTNmcPToUfUtQDt27ODFF19kyZIl6tsZHDp0iAceeADwn4T5zW9+k+nTp0dN37pNoIuIiESzbrHLXUREJNop0EVERKKAAl1ERCQKKNBFRESigAJdREQkCijQRaRLXH311Rw6dMjuMkS6LQW6SJRYuHAh7733Hps2beIXv/hFUL/rP//zP3nttddazfvtb39LVlZWUL9XRE5NgS4irXi9XrtLEJFO0IVlRKLEwoULmTZtGk8//TQej4e4uDhiYmJYv349x48f55lnnuGtt97C4/EwZswY5s6dS1xcHDt27ODhhx9mypQp/OUvf2H48OFcc801PPLII+zZswefz8fgwYOZP38+LpeLZ555hhdeeAGn04nD4WDSpElcd911zJw5k6KiIrKzszl27BhPPvkkpaWlxMfHM3nyZL7zne/gcDjYtGkTr732GgMHDuT1118nKSmJefPmMXLkSAA2bdrEc889R01NDcnJyXz3u99lwoQJNndXJPx1i5uziHQXubm5zJ8/n9dee4277767Zf7vfvc7Dh06xP33309MTAy//vWvee6555g9ezYA1dXV1NbWsmrVKizLorGxkUmTJnHzzTfj8/l49NFHWbt2LT/72c+46qqr2LVrFxMmTGh1L+6TPfnkkxw7doxHHnmEo0ePcs8995CWlsbll18OwN69e7n00ktZu3YtxcXFrF69mtWrV9PY2Mi6deu47777yMnJoaqqitra2uA3TiQKaJe7SJSzLIvXXnuNH/zgB/Ts2ZPExESmT5/Om2++2bKOMYaZM2cSGxtLXFwcycnJjBs3jvj4+Jb1P/zww4C+z+fzsWXLFmbPnk1iYiKZmZlMmzaNzZs3t6zjdrspKCjA4XBw6aWXUlVVxZEjR1pq2b9/P01NTaSlpdG7d++ubYhIlNIWukiUq6mpobGxkSVLlrTMsywLn8/XMt2rVy/i4uJaphsbG3nqqafYvn17y93Q6uvr8fl8OByn3w6oqanB4/Hgdrtb5mVkZFBZWdkynZqa2vI6Pj4e8N/9KjU1lcWLF/Piiy+yevVqBg8ezPe//31yc3M794cX6UYU6CJRLjk5mbi4OB588MGWe0F/XfMd4pq9+OKLHDhwgHvvvZfU1FT27dvHz372M5pPufn6+ifr1asXMTExlJeXk5eXB0B5efkpv/vrRowYwYgRI2hqauL3v/89jz32GHfddVdA7xXpzrTLXSTKpKamUllZicfjAcDhcDB58mTWr1/fslu7srKS7du3n/IzGhoaiIuLIykpidraWv74xz+2Wp6SknLKMecOh4OLL76YZ555hvr6eg4fPsxLL70U0Ilt1dXVbNu2jYaGBpxOJwkJCWfcIyAifvqXIhJlzj//fPLy8pg/fz7XXXcdAN/73vfIzs7m9ttv5wc/+AF33303Bw4cOOVnTJ06laamJq677jpuv/12RowY0Wb5P/7xD6655hqefPLJNu+/9tpriY+P58Ybb+TOO+/km9/8JpdddtkZa7csixdffJHrr7+ea6+9lp07dzJv3ryONUCkm9KwNRERkSigLXQREZEooEAXERGJAgp0ERGRKKBAFxERiQIKdBERkSigQBcREYkCCnQREZEooEAXERGJAgp0ERGRKPD/AywtNIlA1PT3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%capture --no-display\n",
    "plt.figure(figsize=(8, 6))\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Logarithmic sum of betas')\n",
    "plt.plot(np.log(np.sum(beta_iter, axis=0)[0:t]));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e4ae26e-43ca-400d-aa81-58cf880836db",
   "metadata": {},
   "source": [
    "As it can be seen in the previous plot, the method is working properly. In next cell it is shown how the values of $\\beta$ change until the tolerance limit is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "a6782142-1060-48fd-b598-c35abc762065",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAF2CAYAAABtUOHpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABAg0lEQVR4nO3deZxU1Z3//9e5Vb3QQAPdDTRNNzS74AIYiIAao3ZivsZkEuIQNIsaojMTt8hkMWM0mZCMzC/RjEYTJ79gNMvXjHGCiZrRBI2aCSaAigsuiIKIoE1vQENvVfd8/7hV1dV7F3TVrbr9fvpo6m5176cO0p+6937uOcZaaxEREZHAcfwOQERERNJDSV5ERCSglORFREQCSkleREQkoJTkRUREAkpJXkREJKCU5EVERAIq7HcA6bB3794h3V9ZWRl1dXVDus8gU3ulTm2WGrVX6tRmqcm19qqoqOh1uc7kRUREAkpJXkREJKCU5EVERAJKSV5ERCSglORFREQCSkleREQkoJTkRUREAkpJXkREJKCU5EVERAJKSV5ERCSglORFREQCKpB914tkE2st0WgU13X7nHZdl4b6+tjy2I/r4rpRXOsCFmNdrAULWOtCbDr+53BSX1fMoYMH/Q4jp6jNUpPu9ppUMYWCwqK07T9OSf4Y7N/zKvVvPID329ZiAUMqv3L9++Vs+l3bf1z9vxe2G4O1FmMG2o9NnvFebPyPrsdJTBsb26br++PrTdJ7DTa2XefxTD/zjrGx95FYnlhmwDHxdRYnaRtjSMw7xia2c+LvSVpmkvbR5f0NUN5va0kXTX4HkIOa/A4gxzSld/ePv7CIeYs/kd6DoCR/TFp23cP7Ktr8DiPw3Fgut/HX2B82Md2Z7jvPdDtfvc1Nr+u6vMa2cZOWu922iwLWTdqGzmnv1XSbBzcWn016jzfvfSWMH7vLZzOdn7Hz60z8a2Q3A33rEpGsUzRhSkaOoyR/lDraWpk1tp1X68O8FT7NK24w4DjeuWDymV+/zEDn1MfwG3yAfTv9lWT0WGW6re6276TtCwpH0NYa//LTuZ2JxWMcJ7ams4EMoc7tHAdjjLcek5g2xtuH4zgY4+A4DjgOBodQ7FQ65IRxjAHj4DgGxzjxgyfFY7AYTPwvzTsP937if39OyIvKxNYZ0xl/t+mhkGvDWvpN7ZU6tVlqgtJeSvJH6dWtv+WsUstfa0ez8Kxz/A4nqwTlH4eISK5Tkj9KI9pfIepC6ZQav0ORNLLWetfnoy64UYhGwY1io1FsxMWNRnGjUaIdEaJR1/uJLe+ITUdcb13EdXFtlIhribgddETBJUrUhVB+iPb2dlysd3nfWqzxjm+7l9dZ2zkdl5WX7NNXc+KEwrjRSNr2H0Rqs9Sku70WHHc8k6omp23/cUryR8NaZo1p58UDLo81PYDd9bvY8u6bZeVvXs8xhNb9V7fptmQwH7vvbbxiuUQRo+l5RItXuJZYmlzgZ5K2Msmvyfu2sav33bdxY1fmY3fNjY0V79nEsuRXY9zEeofYfGyZMS5OyJJ0F0JEJOH55z/IpKrPpP04SvJH4ZVnf8/7il3Wt7RQMOkd2iOFdJZIxdmBbokfM3vMJ0rHEmDX9/YIJYVTzd6/DJlu+0n6KmFj623SdnjtYeLbWdPLtt3eZwE3dm89UblnYj9ObJmJ53uwhs78nzQde398mXG9bR1rO5fFYsPapPnYe2LBGwzG2s5jJrWF6f6X3aUGr/+/R1+/ag6/p/tEBmXq3MqMHEdJ/iiYQ89weHSUIyObOPL2HFadc53fIWUV3ZNPndosNWqv1KnNUhOU9lKPd0dhRnEbTzW34hjLlMg0v8MRERHplc7kU/TGtj9zyqgI9x8+wqED5Xzywxf6HZLQWaAW7yku6lqvRznXe/A96oIbicYK5VxsR4SOaBQ3YolGorgRl0g0iu2IEokX1HW42KjFjUTA9YrvLC4m4uK6FuPaxKV27xK+xVivoxxrjdcJjjWxDnziPd/Ev1cbLMmP95nEDZ8uN35M/DHK7rdHsrjeQ0QGNGJqIbMXz037cZTkU9RS92femtRGNK+N4+pPwH1sK8bJvV+4/UbcY6Xpf3U3DZDoN6DHe3pblrzvLus7E53pZRvTbZ3pNh3fJjHdy7Y9Xh1wCrpHpX8mIjK0fv/KTlCSzz7TRx3h6dYW2t1CPlB2MkX5hubW/qqLsrfyqN/IBgh7oE/VW1Fg90Wu7bqmZ21ZZxd3yau8E+fOhbb78sR0bJ9Jj5zFt0l+TfSeZy1ufJ1N3ib2k9jG25Gb6D/eW+/1MQ/WurixHVvrAm7iSoPBelcDYvvEuLGiOxLLO/u+s0nlhsnn8/E+84YvE2tJGTy1WWrS3V4lJ81O276TKcmnYM/rzzJ7dDt72o/wnoazGTcWNr5WS/Xfn+V3aFklHQUrpo/poAhKkU+mqL1SpzZLTVDaS4V3KWjc/QdedY/gYDilaAH1R2DCOe/xOywREZFe6Uw+BVUjD/GXyGHmHz6N0QUOf9q5lzlLjveKsnphj/FB9mM6Yz2Wy+1u17WDuayfvE1HSzvR1kj3TWLPfif13+YC1vXm3fh1c+sVtVmLdWOX093YZe8oWBvrcc61GBvFulGMa7FuxNtXNOq9ui426i2LulFwXawb8ZbbaKJAzxLrvQ7rDe1q48PARolaF+taojYa64nO4k1ZouBtayyuhdieiBrvkn983jXg4nofNbbOYnCNG3tM32KNdynfNfGWibVR0rrk/xm6X0K0A4z2JyLZ59Sy05h50ulpP46S/CDV7X2dotGHaYs6nBJezDsHLX+d8AybXrgPY5KTfGZ+4ab1kvUxJo3uPeClpvO9nUV4XfeXqD/vqxEMXk9zOdbbnC6riQwfu2vHMxMl+ayx59Xf4pYf5j2R9zIiFOK/W54mf/LLNLVMoL1tVNeNs7k726EQGwa1+6Jk3dN8fy3So/M2231tX3uwSRX4ydV5SVX5sasp3hNsJqm6z5D8sJo1Xd7p9UQX29yYzuPExqhLPBbXtWLfwbEWxxgMTmyceW8kPAOEHG80vfgyxxgcHBwD+eF8opGot8zE3xPfNn4Q442XZzsjCfXyd5FTjrJbyPy8PNrb2zN+3CHoZtI3+fn5x9Zmw0y622vmgsyMe6IkP0jjR9TzbAQ+aE/h9UOH2VO+kfa2sZwzZiXTZy/yO7ysEpSClUxSm6VG7ZU6tVlqgtJeukI4CPW1b9NRfIBFdilhHB4e+T8Y4zLz8HuU4EVEJGvpTH4Qnn7yxxwZ7/CJ6EKeie6krfANnHfms+y0T/DCtl8QsW1Jz2HbWNkViUKqo5bGK4Oxp7370N867939zefl5dHR5TKXTZpK3tbtXJe4DNrlifhY0VnX93TuI/5+m3hu3iTK1uLL3dgdgKTlNva8uumct7HL/da4Xd7bJY6kosHO/dFlSecniP9fQKJwLvkdXffW2yfv9mc/V5dz9wKyyPA1L//DzJuzMu3HUZIfhMKCPUyPLMUCzxY+TH39bBaPc/jdjn8i2r3H0aG8HZ+rt/aj5FzRW8b0V8tgelnW23YikvPaDzZl5DhK8gM4cvggodEOcyPHs8XZzK7msUwd9Ro7C70zvwmHCwlH80l+yileohUfqjxdTBp/9Q/UN3qXtd0eo3OMifX41rmlIVbE1tteerRRz2Fm4/NeuVxSC9v4fr0hYq0xSSf+BmNNrB95B4hN23j5nDekrLVeoZu3LuyVtxmvPN+4IQwhMGEweRgTwonNO04IY8Levo3BOHk4oRCEwjgmjAmFMOF8Qk4IGwoRCnWuN46DCYUxYQecECUlJTQdOAhOfF8GHAdjQr12m9y9bmy4fQkoKSmhoaHB7zByitosNelurzGFmUm/OZHkt27dyk9/+lNc1+Xss8/mYx/7WOaO/ZefsHDie2mjjT9FX2HMiHoOhy3FHQ5zx1xG9QmnZiyWXBGUgpVMKisrozB/uKXqo1c2qgDTmud3GDlFbZaaoLRX1id513VZt24dX//61yktLeVrX/saixYtorKyMiPHH1fcwQw7kyfM/9JGC2NCHVQemsGSJd/MyPElt1hrIdYhD1Gv4x5vCLzYMtfGXr0+6200SnvU5eDBgxxsaCDiQjQ2el7UetORqFcdEO+Pn1j/+a7bWX0w3IwetZdDzc1+h5FT1GapSXd7TZs+mZGjRqZt/3FZn+R37NhBeXk5EydOBGDZsmVs3rw5I0n+D3f/BPLPYPvEOp7Z30rZu8djovnUWvjdU+sSl0iTe3QDuvVOlp6zs2PdazoTQ+/7jj+Tbrpu08vGvffg1vmJuxbvmaTytL4HlOhe3NZbkZtNukvQfb01XQvl+p5OLrCzifd1Htf28dr7pxORYJrzdCnnXPKptB8n65N8Q0MDpaWlifnS0lJee+21Ltts2LCBDRs2ALB27VrKysqG5Nh5I4o55M7hsdoWprbvZBOHINQxJPuWTEuulDCJMdw76xq8ZfH1Jnn7Hstjf8bGhk/e1knaLmlLTOwmurEk7Y8ux+/8GmN6/NnzW53pfbGI5ITx4/OHLFf1J+uTfG/9v5tuVUc1NTXU1HT2HjRU94NPWPpe/uOPOzixcDL7C5Zz2pQXaT74EpHQ2MT5qNcjWm+/bL2+15OusXZZFxvPFMd228bGHuGyxJ5kc71lffW0ZWJFYg7ghAADIccr4HJC3msf+uv0q/u6Hj3Y9SgG65x3nFBsiNXOxcYkpTGna2Lz9uU1oo31EpdYFwrFXk0iqFAohBMyOI6DMcZrAkLefDiEMYZQOIRjQl6vc46DCTleb3Gx94ZMCBMKgXEIh+NtGCIUDnvbOyGvyM90JnlrvH0TK7QDJzZvEt8bDCb+/YGkVbH5vhtcdQypUXulTm2Wmky011Duv6KiotflWZ/kS0tLqa+vT8zX19czbty4jBx7fFU1VZGt3OuO5AvhF9nXfjLhETOozH+U6Wedx4jRYzMSB4Btb4PmQ9B8EJoPYg8dgAON0FiHbayDujporPeW2aTn3PMLoLwSM6kSJlVhKqbApEoYP8lLcmmgXyYiItkh65P8jBkz2LdvH7W1tZSUlLBx40auuuqqjB3/A1Uj+Z8Gh5fe3kdp8e+ZVrKAN1rP4/BjOykc8SInffAzGYnD5BdASQGUeJd3+jontJEIHGyE+v3Yd/bA3rew+3ZjX9sGf3ui84w8vwCmz8HMOh4zax5MPw5TUJCJj5KTrLVYa3GjUW86GsXGiue8Ee68YjrXdXGjNvbqettHXGxsJDzXdTGuN7odbmcB3sGCfFpbWr0KA4tXXBffv3VjVw68vz03aZRAM9BzmgG9wd9QUEB7W5vfYeQUtVlq0t1eo+bMYVRZ+k9YjT3W8VAz4JlnnuHuu+/GdV3OPPNMli9f3u/2e/fuHbJjR9vb+YefP83Y9kNUR16nIFTEgkmF1EfOwLUhphZuwbFHgO6XuLO9Wfsa2aTnGHID3fcduLhwMN279HbDwyRFmfzsfOc1cBt7Vj55WSKmpIGCbJcenE1i3sbv0cf25c17z8xjvOfqvW3j653Edja+XfJy6+AmtnGwtnO5jcVhie/XW57Yt036TLrbLhJoFWMf5D3nfHro9perl+sBTj75ZE4++WRfjh3Kz+e9bi2PjJrBKXlN7Kt9h9fqijjuhBeI7CvijZalvsQVbPG03rWOIZH2TPJ6m5zW+1lnY98DbGLeW9Z1PrHeWEys29zel7lABKf7dsb15o3FWDdp/7FlSTX4Xbr1TYrf2q4x26TPHv9O3nUYXn0hEMk1odLxGTlOTiR5v50zeRQPNYZ5+cAIFpWVs6NuH2+9nM/ffe4DHGhu8C7VAl6Pat4jUG4sfww4ouXRDnmJ17Ncv7vuNv5Q982dXorybEc7vLUT8+IWeH4zNDdD0Uick07GzF8K02d7vbX1OHbnfFlZWaKnqESJnQFjQrFp0+U9JrlSbZhSHUNq1F6pU5ulJijtpSQ/CPP//qOMve1xDkXzOWH269Q3T+LN5t089H8f4sMXftjv8Ibe2DI4cTH27y+Fl5/DbnoC+9Sf4fE/wthSzHvfhzn7PExJ799ECwqKyMs7kuGgRUSkOyX5QQiPGMGSjnd4fOQUHtgX4TMLdnLPllHsatjDXx57ilPPCuYlexMOw4nvwZz4HmxbK/a5TdhNT2If/R32sQcxZ3wIc+75mOLMPO2QDl5BHclPNXrLXK/XuUjE4rqWaDT+6hKJWqIudHREiUYtkahLJOp66yKWqOsSjXhFdlHXK9JzXQturKe6RPGei7XeTwiHqBvxLsdb6z2CaIBo56h7iT9jl/Pjl/WHI8dxvMJEGTS1WWrS3V5zFlQxd9bktO0/Tkl+kN5X4vBwWwG79uQxfmYdJ1dN4m+7ann+5W1s374zsV3mrjj7dGk7bwYsnOFNHwDuebjXzeJ3irsUzUFnTjIDzPc+S2+fu7c0131Zl7v7feTFnp3E2sRLz0Fik/s/SBqKNv6NATcxbWPfIGzS0LgiMryN4ICSfDaZ+cHTKVr/Oo4b5cHa2Xxw6pvsr59GbWtLl45IO59usn0mk6GS7Xew++2gNbbKdJvvXkZmur+hx/Le16f2fpv4QtKzrt3GagY6i9+S6wwS+7Pxwr5YhzjWSWwbLyA0yT8mXkcf77kuuabeJLbp+eyC6RHHcNR1pEMZDLVZatLdXtXH9V4NP9SU5AepoLiYU9r2snnkJB58ZwIXVrzAe06axcSTL/Q7NN/ZPTtxf/t/YevfYFQxoz/5OQ6/9/2Yfnrbk66CUuSTKWqv1KnNUhOU9tJv4RScPipCc14RNDaxuamKyryX6WhVgZmpnEbo8utw/uV7UDWNQ+v+A/e2b2MPH/I7NBGRYS0nOsNJ1VB2hgOd3+ja9r3Dp/+4n9mHdlNRPYZ/nfMQzzctonzRJ4b0eLnMWsvIzU9y6M5bYGwJzj98BTNt9lHtp+uCLi+d9We9TMdf4j3SJf6LDfPqulEMLhYXG42AdTFYXDeKY6NE3Qgm6gJRr6c6G431UhcF4r3Uud60tWC93uxsbJ1DrHAu8ZP0THz8c1nvOXhrY5fpjUl6FDPpsyfuXyQvS75QH7h/voNijOl1XAvpm9osNelur8Kpy8grHbp78jndGU62KJhUzntat/LqiPG82jSKVc2lHDf6eQ49v7vbljZpqu87pz37lsukbveoU7jB23vcncsiI2DE5Uu95HT4t/Bi5/ru98hN0rCyfXdqQ5dtu3ZOQ2I6vs4kOqAhMe30OnztEDFAfBiA9AwHICIBs72pkrFDmOT7oiSfotNGtPKUU0xl87v8qe5EaiZs6zXpWcwgCqP8TPPdorNdlx3dF9heytziz6XFh2RL2i45vXc/E++6TY+vBt3eY7Cxojevu9jk93du65XPxe9QGWzyNol9OJ0lcLFubRPTGDCOdyYdGznPWm8Y2fhodZjO0euMcbDGAeN1HmRj7/GGqI0FHKtbGFE0ktaWlq5/DSbp/6GBewEeVooKiziiW2UpUZulJt3tVVx6XNr2nUxJPkULl5yI87dWyqOH+ENTNR864yxMXp7fYWWV5IIV67rYR9Zj7/85jJ+E849fxVRW+xtgFgpKkU+mqL1SpzZLTVDaS4V3KRo5awYnHdnL2wXjeKMtzPbn/up3SFnNOA7O//kEzj9/G1pbcG/8Evbpv/gdlojIsKAz+aNwWl4zt+VXMaV1P3/YV0TV7ldil0+9y7Lxrtid2Ohlfev/mnj/l8wHeO+A6wd/4B4dy3TrBar7+mhtIS0tLck1ZhA22Euvwn1uM2z9X7DtXve5sQ0SvbpZmyiki48DgHUTXchYa716N1y8TuS82wFuLBDX4vUih7fOu3xvca0lVnfnxRibd4ltE5uP4tXVRTHeSLB4+4xacDHeqzU954GoNbFtTWydtz4am7bxbWJtEn9/fIgbYseKD2MT73PB7fHkvojkuqtPgMo5J6X9OEryR+GUhTO4bRtMizSy4eB4Nvw5vqZ7F4jRDEeWLVrp+eUmBBQBZ0Ah8GrGg0qLEJZw7BZ9GO81ZCxhIBSbDsWXYzHGu3wWMvFBbb118UqAsNO53JjO6oNEBUJyBzyZ/KAiMqRMtD0jx1GSPwrFC05izuY/sSc0hn+c+FbsjKuzEMwm/RyrwZTv9WmA6rn+KuoHHDyv3/2azlPmxEhzkCibsxbaW8C1mMIRieKzLpX08f0kl9qZpHWxeWNt4sqJiZXNEUu6BpOo90v0RJe83DGx2jfjLXMMjgnhON5tBuM4hGLLjBMiFPJeTShMOOTghPLACUEoD0MIxwljQiGc2Ch98df4T/f5+A8E5/5fpqi9Uqc2S01Q2ktJ/iidFmpiXcFcvj6ulZL3LPA7nKwymH8c9p09uP/+VSgswrn2/8OMGZeh6EREhg8l+aO0ZNZE1u2Gvz2/k/+jJJ8yU16Jc9U3cL93He4t38T58o2YEUV+hzU0kju86dL5DUmD2nROG4CWFkJtmbl8Fwhqr9SpzVKT5vaK5oUTVzHTST3eDUJfZ6ZX/eTP5BHl4pIWb4HXi0sKA4gM1PR972HAv7QB/lr7W92j0K7HfOcldGt7buM4DtGom1ja2TVN5xsStzba27GHmyEcxo4cHVvnFcth48+wxwrrYsvjhWudxXjxQjZvvrOwjUQBW7y4zcXECu7oWggXWxclVnQXK4xzY/PR5PfHtnNxEuvdHusNrjGJeZfkAjuDa+LzJukzd2tn070/AREJin+fGaHi1PcO2f7U410anEEdPxsxl6+3+B1JDjPAqNj00dYp9hw6rv/NrYtjLQ4uxtrEtGMtjnW9H7zpUGLaEqLzfSFrcYjE5r2fcGw6ROeyzvWdxXVOrADP2ES3O50fIanuIPm1y2cVkZxXFJqUkeMoyR+Dj/z9WSx68ilcN/6oV/w1/sdQ/EY++vM4019lHT0L70x/K7u/N2lrEyt2i8vLCxOJRLts520T28jpLMaLrzV172Df2YtTWoapmBrb1GCcrp/FYHAcr5rOGCdRWOeEHIgVtzmOt84JJRXAOQ44BicUwoQcr2DOhGKFcwbreMtsOIxxQhAKeT3UOSFwjNdzXSjUeXktUQVoBmyrwQhKkU+mqL1SpzZLTVDaS0n+GOSPGcPUj3zI7zCyztH+43DvXYf9428xn7sGZ+mZaYis50OOvdHlcREJCvV4J1nDnH8JTJ+DvXcd9tBBv8MREcl5SvKSNYzj4Hzmcmg5jL3vp36HIyKS85TkJauYymrMBz+O3fgo9uXn/A5HRCSnKclL1jHnfRLGl+P+4ofY9ja/wxERyVlK8pJ1TH6Bd9m+dh/2oV/7HY6ISM5SkpesZObOxyw9E/vIf2PfftPvcEREcpKSvGQt8/erYEQR7s9v7zG8rYiIDExJXrKWGV3sJfrXX8E++Yjf4YiI5BwleclqZumZMHc+9jd3Y5vq/Q5HRCSnKMlLVjPG4Hz6nyASwf3V/+93OCIiOUVJXrKemVCB+fAKeHoj9rlNfocjIpIzlOQlJ5hzPg4VU3Dv+TE2EvE7HBGRnKAkLznBhPNwln8W6muxT//F73BERHKCkrzkjhMXQflk7B9/i7UaK05EZCBK8pIzjONgav4O3twB27f5HY6ISNZTkpecYpaeCaOKcf94v9+hiIhkPSV5ySkmvwDz/nPhuU3Yd/b4HY6ISFZTkpecY848F8J52D/+zu9QRESymu9J/qmnnmL16tV88pOf5PXXX++ybv369Vx55ZVcffXVbN261Z8AJeuY4rHe4DVPPYY9dMDvcEREspbvSb6qqoovfelLzJ07t8vyPXv2sHHjRm6++Wauu+461q1bh6tBSiTGfODvoKMd+/j/+B2KiEjW8j3JV1ZWUlFR0WP55s2bWbZsGXl5eUyYMIHy8nJ27NjhQ4SSjcykKjhxEfZPD2E72v0OR0QkK4X9DqAvDQ0NzJo1KzFfUlJCQ0NDr9tu2LCBDRs2ALB27VrKysqGNJZwODzk+wyyTLVX+/mfpfEbVzHyxS0UfeCjaT9eOun/sdSovVKnNktNUNorI0l+zZo1NDU19Vi+cuVKFi9e3Ot7UunspKamhpqamsR8XV1dyjH2p6ysbMj3GWSZai87aSpUTePQ+l9yeP4SjOP7hamjpv/HUqP2Sp3aLDW51l69XRGHDCX566+/PuX3lJaWUl/fObRoQ0MDJSUlQxmW5DhjDOaDH8Ou+z5se8brEU9ERBKy9tRn0aJFbNy4kY6ODmpra9m3bx8zZ870OyzJMmbR6TC2FPcP9/sdiohI1vH9nvymTZu48847OXjwIGvXrqW6uprrrruOqqoqli5dyurVq3Ech1WrVuHk8OVYSQ8TDmPOPg/733djd7+BmTLd75BERLKGsQEc6WPv3r1Dur9cuzfjt0y3lz3SjPuVVZiFp+CsWp2x4w4l/T+WGrVX6tRmqcm19urrnrxOjSXnmaJRmNNqsJv/jG3InX+UIiLppiQvgWBqPgquxT7xsN+hiIhkDSV5CQRTNhHmnoR9+i8aa15EJEZJXgLDLFwC774N+97yOxQRkaygJC+BYRacAoB99q8+RyIikh2U5CUwzNhSmHEc9pmn/A5FRCQrKMlLoJiFS2D369j6Wr9DERHxnZK8BIpZuAQA+6zO5kVElOQlUMyECpg8VfflRURQkpcAMguXwmsvYQ82+R2KiIivlOQlcMzJS8Fa7HOb/A5FRMRXSvISPJXVUDZRl+xFZNhTkpfAMcZ4BXgvb8W2HPE7HBER3yjJSyCZk5dCJIJ9YYvfoYiI+EZJXoJp+nFQPBbUMY6IDGNK8hJIxnEwC5ZgX3wa29HudzgiIr5QkpfAMguXQFsrvLTV71BERHyhJC/BddyJMGKker8TkWFLSV4Cy4TzMCctwj63CRuN+h2OiEjGKclLoJmFS6H5ELy2ze9QREQyTklegu2EkyEvXx3jiMiwpCQvgWYKCuH4hdhn/4q11u9wREQySkleAs8sXAKNdbBrh9+hiIhklJK8BJ6Z/15wHOyzG/0ORUQko5TkJfDMyNEw50TdlxeRYUdJXoYFs3ApvPM2dt9bfociIpIxSvIyLJiTFgFgtz3rcyQiIpmjJC/DgimdABMmYdXFrYgMI0ryMmyYeQtg+4vYSMTvUEREMkJJXoYNM3e+N2DNG6/6HYqISEYoycvwMeckMA725ef8jkREJCOU5GXYMCNHQfVM7Mtb/Q5FRCQjlORlWDFzF8DO7dgjh/0ORUQk7ZTkZVgx8xaA68L2F/wORUQk7ZTkZXiZPgfyC7Av6b68iASfkrwMKyYvD2afoPvyIjIsKMnLsGPmzve6uG3Y73coIiJppSQvw46ZtwBAj9KJSOApycvwM3kqFI8F3ZcXkYAL+x3Az3/+c55++mnC4TATJ07kC1/4AiNHjgRg/fr1PPbYYziOwyWXXMKCBQv8DVYCwRiDmTsf+9JWrOtiHH3XFZFg8v2320knncRNN93E9773PSZNmsT69esB2LNnDxs3buTmm2/muuuuY926dbiu63O0EhhzF8ChA7D3Tb8jERFJG9+T/Pz58wmFQgDMnj2bhoYGADZv3syyZcvIy8tjwoQJlJeXs2PHDj9DlQBJ3JfXqHQiEmC+J/lkjz32WOKSfENDA6WlpYl1JSUliS8AIsfKjCuFSVUqvhORQMvIPfk1a9bQ1NTUY/nKlStZvHgxAL/5zW8IhUKcfvrpAFhrB73/DRs2sGHDBgDWrl1LWVnZsQedJBwOD/k+gyxX2uvgyafQ8scHKB1TjMnL9zWWXGmzbKH2Sp3aLDVBaa+MJPnrr7++3/WPP/44Tz/9NDfccAPGGABKS0upr69PbNPQ0EBJSUmv76+pqaGmpiYxX1dXNwRRdyorKxvyfQZZrrSXnXYctN9H3aa/YOac6GssudJm2ULtlTq1WWpyrb0qKip6Xe775fqtW7fy29/+lq9+9asUFBQkli9atIiNGzfS0dFBbW0t+/btY+bMmT5GKoEz+wRwHN2XF5HA8v0RunXr1hGJRFizZg0As2bN4rLLLqOqqoqlS5eyevVqHMdh1apVOHrUSYaQGVEE0+d49+U//hm/wxERGXK+J/kf/OAHfa5bvnw5y5cvz2A0MtyYufOxD96LPdzsjTcvIhIgOjWWYc3MWwDWhVef9zsUEZEhpyQvw1v1bCgcofvyIhJISvIyrJlwGOacqOflRSSQlORl2DNz50PtPmzdu36HIiIypJTkZdjT0LMiElRK8iLllTC2FHRfXkQCRklehj1v6NmTsK++kFJ3yiIi2U5JXgRg+hxv6NmG/X5HIiIyZAad5Pfs2ZMYZKa1tZV7772X++67j7a2tnTFJpIxpnqWN7Fzu7+BiIgMoUEn+VtuuYUjR44A8LOf/YyXX36Z7du38+Mf/zhtwYlkTGU1hMPYXa/5HYmIyJAZdLe2+/fvp6KiAmstmzdv5qabbiI/P58rrrginfGJZIQJ50HVdOxOJXkRCY5Bn8nn5eXR0tLCjh07KC0tpbi4mLy8PDo6OtIZn0jGmOpZ8ObrWDfqdygiIkNi0Gfyp556Kt/61rdoaWnhQx/6EAA7d+5kwoQJaQtOJKOmzYY/PQT73obJU/yORkTkmA06yV988cU899xzhEIhTjjhBMB79Oiiiy5KW3AimWSqZ2EBu+s1jJK8iARASkPNzp8/v8v8jBkzhjQYEV9NrIARRbBrO5x6tt/RiIgcs0En+Wg0yiOPPMJLL73EoUOHuqz713/91yEPTCTTjOPA1JkqvhORwBh04d3dd9/Nhg0bmDdvHm+88QannHIKBw4c4Pjjj09nfCIZZabNgj27sB3tfociInLMBp3k//a3v/Ev//IvnHvuuYRCIc4991y+/OUvs23btnTGJ5JRpnoWRCPw1k6/QxEROWaDTvLt7e2UlpYCkJ+fT1tbG5MnT2bXrl3pik0k86pnA6hTHBEJhEHfk588eTKvv/46M2fOZPr06fz6179mxIgRlJSUpDM+kcwaVwpjxoHuy4tIAAz6TP7iiy/GcbzNL7roInbu3MnTTz/NZZddlrbgRDLNGAPVs3QmLyKBMOgz+bKyMsaOHQvApEmTuP766wESg9aIBIWZNhv73CbskcOYopF+hyMictQGfSZ/9dVX97r8mmuuGbJgRLJBYkS6N3f4G4iIyDEadJK31vZYduTIkcQlfJHAqJ4JqPhORHLfgJfr/+mf/gnwquvj03HNzc2ceuqp6YlMxCdm5GiYMAmrseVFJMcNmOSvvPJKrLXceOONXHnllV3WjR07loqKirQFJ+IXUz0b+5r6gBCR3DZgkp83bx4A69ato6CgIO0BiWSFaTNh0xPYpgbMWD0mKiK5adA31B3H4Z577uGKK65IjDz33HPP8fDDD6ctOBG/mFinOOi+vIjksEEn+bvuuou33nqLq666ynuWGKiqquIPf/hD2oIT8U3VdHAcDVYjIjlt0M/Jb968mVtvvZXCwsJEki8pKaGhoSFtwYn4xRQUwOSpqrAXkZw26DP5cDiM67pdlh08eJDRo0cPeVAi2cBMmw27tvf6+KiISC4YdJJfsmQJt912G7W1tQA0Njaybt06li1blrbgRHxVPQuOHIbafX5HIiJyVAad5C+88EImTJjAP//zP3PkyBGuuuoqxo0bx/nnn5/O+ER8Y6Z5Pd/pkr2I5KpB35MPh8NcfPHFXHzxxdTV1fHaa69RVVVFXl5eOuMT8c+kKZCfDzu3wyln+B2NiEjKBkzyDQ0N3HnnnezZs4fZs2fzkY98hG984xs4jsPhw4e54oor1OudBJIJhWDKTJ3Ji0jOGvBy/Y9//GNGjhzJRRddhLWW73znO/zjP/4jP/nJT1i9ejXr16/PRJwivjDVs2D3G9hIxO9QRERSNmCS3759O5deeikLFy7k85//PAcOHGDx4sUALF68mP3796c9SBHfTJsFHe2wd7ffkYiIpGzAJB+NRgmHvav6BQUFXZ6TFwm6+LCzdpcGqxGR3DPgPfloNMqLL76YmHddt8e8SGCNL4eRo2Hna/C+D/kdjYhISgZM8mPGjOFHP/pRYn7UqFFd5ouLi48pgF/96lds2bIFYwxjxozhC1/4AiUl3oAg69ev57HHHsNxHC655BIWLFhwTMcSSZUxBqbNUvGdiOSkAZP87bffntYAPvrRj7Jy5UoAfv/733Pfffdx2WWXsWfPHjZu3MjNN99MY2Mja9as4ZZbbsFxBv1ov8iQMNWzsA/9GtvWiiko9DscEZFB8z1jFhUVJabb2toS9/s3b97MsmXLyMvLY8KECZSXl7Njxw6/wpRhzFTPBuvC7jf8DkVEJCWD7gwnne655x6efPJJioqK+MY3vgF4z+fPmjUrsY0GwxHfTJsJgN25HTNrns/BiIgMXkaS/Jo1a2hqauqxfOXKlSxevJgLLriACy64gPXr1/Pwww+zYsWKlAYF2bBhAxs2bABg7dq1lJWVDVXogNfb31DvM8gC115lZewvHU9+7duMSdPnClybpZnaK3Vqs9QEpb0ykuSvv/76QW132mmnsXbtWlasWEFpaSn19fWJdQ0NDYmCvO5qamqoqalJzNfV1R1bwN2UlZUN+T6DLIjt5VZMpXXHK3Sk6XMFsc3SSe2VOrVZanKtvSoqKnpd7vs9+X37Okf42rJlSyLQRYsWsXHjRjo6OqitrWXfvn3MnDnTrzBlmDOV1fDOHmxHh9+hiIgMmu/35H/5y1+yb98+jDGUlZVx2WWXAVBVVcXSpUtZvXo1juOwatUqVdaLfyqrIRqFd/ZA1TS/oxERGRTfk/yXvvSlPtctX76c5cuXZzAakd6ZqmlYwL61E6MkLyI5QqfGIoMxoQLy8mHPTr8jEREZNCV5kUEwoRBUTMHu2eV3KCIig6YkLzJIprIa3tqZ0uOdIiJ+UpIXGayqadB8EA40+h2JiMigKMmLDJKpjBXc6b68iOQIJXmRwaqsBsC+tcvXMEREBktJXmSQzMhRUFIGKr4TkRyhJC+SisppWF2uF5EcoSQvkgJTOS3WvW2736GIiAxISV4kFZXV4Lqw9y2/IxERGZCSvEgKTFU1gDrFEZGcoCQvkooJkyBf3duKSG5QkhdJgXFCUDEV+5aSvIhkPyV5kRSZqmmwZ5e6txWRrKckL5Kqymo4fAiaGvyORESkX0ryIilS97YikiuU5EVSVTkVQPflRSTrKcmLpMgUjYLSCereVkSynpK8yNGorNaz8iKS9ZTkRY6CqZoG77yNbW/zOxQRkT4pyYscBVNZDdaFvbv9DkVEpE9K8iJHI1Zhr0v2IpLNlORFjsb4csgvUPGdiGQ1JXmRo2AcByare1sRyW5K8iJHyevedqe6txWRrKUkL3K0KqfBkcPQWOd3JCIivVKSFzlK8bHleWuXn2GIiPRJSV7kaE2uBsCqD3sRyVJK8iJHyYwogrKJoOI7EclSSvIix6JyGvbtXX5HISLSKyV5kWNgqqrh3X3YNnVvKyLZR0le5Bh0dm/7pt+hiIj0oCQvcizi3dvqvryIZCEleZFjUTYRCkaoe1sRyUpK8iLHwDgOVE7VY3QikpWU5EWOkamshj1vqntbEck6SvIix6pyGrQchvpavyMREelCSV7kGJkqr/hOneKISLZRkhc5VpXTwDjY3a/7HYmISBdZk+R/97vfsWLFCg4ePJhYtn79eq688kquvvpqtm7d6l9wIv0wBQUwqRL7ppK8iGSXrEjydXV1vPDCC5SVlSWW7dmzh40bN3LzzTdz3XXXsW7dOlzX9TFKkb6ZKTNg9xt+hyEi0kVWJPm7776bT33qUxhjEss2b97MsmXLyMvLY8KECZSXl7Njxw4foxTpx9TpcKABe6DR70hERBJ8T/JbtmyhpKSE6urqLssbGhooLS1NzJeUlNDQ0JDh6EQGx0yZ4U3obF5Eskg4EwdZs2YNTU1NPZavXLmS9evX8/Wvf73HulSeOd6wYQMbNmwAYO3atV0u+w+FcDg85PsMsuHYXm7RYvYDI+r2MeooPvtwbLNjofZKndosNUFpr4wk+euvv77X5bt376a2tpYvf/nLANTX1/PVr36VG2+8kdLSUurr6xPbNjQ0UFJS0ut+ampqqKmpSczX1dUNYfRQVlY25PsMsmHbXhMmcfiVF2g9is8+bNvsKKm9Uqc2S02utVdFRUWvyzOS5PsyZcoUfvKTnyTmL7/8cm688UaKi4tZtGgRt956K+eddx6NjY3s27ePmTNn+hitSP/MlBnYndv9DkNEJMHXJN+fqqoqli5dyurVq3Ech1WrVuE4vpcQiPRtygzY8r/Yw4cwI0f7HY2ISHYl+dtvv73L/PLly1m+fLlP0YikxkydjgWv+G7ufL/DERHxv7peJDCqvAp7qwp7EckSSvIiQ8SMLoaS8aDubUUkSyjJiwylKdPVh72IZA0leZEhZKbMgHf3YluP+B2KiIiSvMhQMlNmgLXw1i6/QxERUZIXGVJTpwPokr2IZAUleZGhNKYEiseChp0VkSygJC8yhIwxMGUG9i09Rici/lOSFxliZsoM2Lsb29HudygiMswpyYsMMTNlOrgu7HnT71BEZJhTkhcZalNUfCci2UFJXmSolU2EopHq+U5EfKckLzLEEsV3qrAXEZ8pyYukgZkyA95+ExuJ+B2KiAxjSvIi6TBlOkQ64J23/I5ERIYxJXmRNDBTY8POvqnn5UXEP0ryIukwoQIKClV8JyK+UpIXSQPjOFA1TY/RiYivlORF0sRMmQFv7cS6Ub9DEZFhSkleJF2mzIC2Vqjd53ckIjJMKcmLpImJDzur5+VFxCdK8iLpUl4F4TzYrQp7EfGHkrxImphwGCqrVXwnIr5RkhdJIzNlOux+HWut36GIyDCkJC+STlNmwJHDUPeu35GIyDCkJC+SRmaK1/Od7suLiB+U5EXSqXIqOA5WSV5EfKAkL5JGJi8fKqao+E5EfKEkL5JmpnoWvPGqer4TkYxTkhdJt7nz4Ugz7NrhdyQiMswoyYukmZm7AIzBvvSs36GIyDCjJC+SZmZ0MUyZgd221e9QRGSYUZIXyQBz/EJ44xVsyxG/QxGRYURJXiQDzLyF4Lrw6vN+hyIiw4iSvEgmzJgDBYXYbbovLyKZoyQvkgEmnAdzTlSSF5GMUpIXyRBz/ELY/w62dp/foYjIMKEkL5IhZt5CAOxLW/0NRESGDSV5kUyZWAGlE/S8vIhkTNjvAO69914effRRiouLAbjgggs4+eSTAVi/fj2PPfYYjuNwySWXsGDBAh8jFTk2xhjMvAXYLf+LjUQwYd//+YlIwGXFb5kPf/jDfPSjH+2ybM+ePWzcuJGbb76ZxsZG1qxZwy233ILj6OKD5C5z/ELsn/8Au7bDzHl+hyMiAZe1GXPz5s0sW7aMvLw8JkyYQHl5OTt2qO9vyXHHzQfjqPc7EcmIrEjyjzzyCF/60pf44Q9/SHNzMwANDQ2UlpYmtikpKaGhocGvEEWGhBk5CqbN0n15EcmIjFyuX7NmDU1NTT2Wr1y5kg9+8IOcf/75APzXf/0XP/vZz/jCF76AtXbQ+9+wYQMbNmwAYO3atZSVlQ1J3HHhcHjI9xlkaq/+NS86lcP33UVJYT7OKK8WRW2WGrVX6tRmqQlKe2UkyV9//fWD2u7ss8/m3//93wEoLS2lvr4+sa6hoYGSkpJe31dTU0NNTU1ivq6u7hii7amsrGzI9xlkaq/+2WmzwXWp/8vjmPcsA9RmqVJ7pU5tlppca6+Kiopel/t+ub6xsTExvWnTJqqqqgBYtGgRGzdupKOjg9raWvbt28fMmTP9ClNk6FTPhhFF2G3P+B2JiASc79X1v/jFL9i1axfGGMaPH89ll10GQFVVFUuXLmX16tU4jsOqVatUWS+BYMJhmHMS9qWtWGsxxvgdkogElO9J/sorr+xz3fLly1m+fHkGoxHJDHP8QuzWv8K7e6F8st/hiEhA6dRYxAfm+HgXt6qyF5H0UZIX8YEZXw7jy9WPvYiklZK8iE/M8QvhlRewkQ6/QxGRgFKSF/GJmbcQ2lrg9Vf9DkVEAkpJXsQvx50EjqP78iKSNkryIj4xI4pg+nHYbUryIpIeSvIiPjLHL4Ddr+MebPI7FBEJICV5ER+ZeQvBWtqf2+x3KCISQEryIn6qngljSzn8u19hXdfvaEQkYJTkRXxknBBm+WeJ7HgZ+9fH/Q5HRAJGSV7EZ+aUMwjPnItd/zNsa4vf4YhIgCjJi/jMOA6jV30Rmhqwj/zG73BEJECU5EWyQP5xJ2Le+z7sI+ux9fv9DkdEAkJJXiRLmOUXAWB/c7fPkYhIUCjJi2QJUzoec87HsZuexO542e9wRCQAlORFsog5ZzmMLcG9d50eqRORY6YkL5JFTOEIzMc/Czu3Yzc94Xc4IpLjlORFsoxZ8n6onoX9759h21r9DkdEcpiSvEiWMY6D88nPQ1M99mE9UiciR09JXiQLmZlzMYtPxz7yGz1SJyJHTUleJEuZT8QfqfuZz5GISK4K+x2AiPTOlE7AfPBj2IfuJdrWgvORlZipM/0OS0RyiJK8SBYz530S8vKxf7gf99urYf57cT5yAWbqDL9DE5EcoCQvksVMOA/z4RXYMz+MfexB7B/vx/32NUr2IjIoSvIiOcAUjcSc90nsWef1TPZLz4KCQsjLh7y8rq/hPAiFwHHAib9608ZRSY5I0CnJi+SQrsn+Aewff4v73Kaj32E86Zv4q+k2bTqXGTpfMV3XQ+w1vixpm0Tw3eYT74ltS7fJLtuaLsvqw2GikUjfn6v7cVJdn673+qhhoDaTLtLdXs7nrsFMrEjb/uOU5EVykJfsV2JrPgrv7oWODuhoh0gHdHRgE9PtEHXBjYKb/Jo0bWPz1oJru827gPWme/tJrKNzGhLrLfH5xB9J28Q/TWKic10ya3usc/LzoL2j98bpbR9dNxhgdT/rB9p1FjP9tZn0kPb2ytCVNCV5kRxmCougl4r73DzXHLxxZWXU1dX5HUZOUZulJijtpZtyIiIiAaUkLyIiElBK8iIiIgGlJC8iIhJQSvIiIiIBpSQvIiISUEryIiIiAaUkLyIiElBK8iIiIgGlJC8iIhJQSvIiIiIBpSQvIiISUEryIiIiAWWsHXBcRhEREclBOpMfhGuvvdbvEHKK2it1arPUqL1SpzZLTVDaS0leREQkoJTkRUREAkpJfhBqamr8DiGnqL1SpzZLjdordWqz1ASlvVR4JyIiElA6kxcREQmosN8BZLOtW7fy05/+FNd1Ofvss/nYxz7md0hZ54c//CHPPPMMY8aM4aabbgKgubmZ73//++zfv5/x48dzzTXXMGrUKJ8jzQ51dXXcfvvtNDU1YYyhpqaGc889V23Wj/b2dr7xjW8QiUSIRqMsWbKEFStWqM0G4Lou1157LSUlJVx77bVqrwFcfvnlFBYW4jgOoVCItWvXBqLNdLm+D67rcvXVV/P1r3+d0tJSvva1r3H11VdTWVnpd2hZ5aWXXqKwsJDbb789keR/8YtfMGrUKD72sY9x//3309zczKc//WmfI80OjY2NNDY2Mn36dFpaWrj22mv58pe/zOOPP64264O1lra2NgoLC4lEItxwww1cfPHFbNq0SW3WjwcffJDXX3898f+Z/l327/LLL+fGG2+kuLg4sSwIbabL9X3YsWMH5eXlTJw4kXA4zLJly9i8ebPfYWWdefPm9fhmu3nzZs444wwAzjjjDLVbknHjxjF9+nQARowYweTJk2loaFCb9cMYQ2FhIQDRaJRoNIoxRm3Wj/r6ep555hnOPvvsxDK1V+qC0Ga6XN+HhoYGSktLE/OlpaW89tprPkaUOw4cOMC4ceMAL6kdPHjQ54iyU21tLTt37mTmzJlqswG4rstXv/pV3nnnHc455xxmzZqlNuvHXXfdxac//WlaWloSy9ReA/vOd74DwAc+8AFqamoC0WZK8n3o7S6GMcaHSCSIWltbuemmm7j44ospKiryO5ys5zgO3/3udzl8+DDf+9732L17t98hZa2nn36aMWPGMH36dLZt2+Z3ODljzZo1lJSUcODAAb797W9TUVHhd0hDQkm+D6WlpdTX1yfm6+vrE9/opH9jxoyhsbGRcePG0djY2OUel0AkEuGmm27i9NNP55RTTgHUZoM1cuRI5s2bx9atW9VmfXj11VfZsmULzz77LO3t7bS0tHDrrbeqvQZQUlICeP8WFy9ezI4dOwLRZron34cZM2awb98+amtriUQibNy4kUWLFvkdVk5YtGgRTzzxBABPPPEEixcv9jmi7GGt5Y477mDy5Mmcd955ieVqs74dPHiQw4cPA16l/QsvvMDkyZPVZn248MILueOOO7j99tv54he/yAknnMBVV12l9upHa2tr4tZGa2srzz//PFOmTAlEm6m6vh/PPPMMd999N67rcuaZZ7J8+XK/Q8o6//Ef/8FLL73EoUOHGDNmDCtWrGDx4sV8//vfp66ujrKyMlavXp1zj52kyyuvvMINN9zAlClTErd/LrjgAmbNmqU268Obb77J7bffjuu6WGtZunQp559/PocOHVKbDWDbtm088MADXHvttWqvfrz77rt873vfA7ziztNOO43ly5cHos2U5EVERAJKl+tFREQCSkleREQkoJTkRUREAkpJXkREJKCU5EVERAJKSV5E0uYzn/kM7777rt9hiAxbSvIiAXb55Zfz/PPP8/jjj3P99den9Vjf/OY3efTRR7ss+/nPf87EiRPTelwR6ZuSvIgMKBqN+h2CiBwFdYYjEmCXX3455513Hr/4xS+IRCLk5+cTCoW466676Ojo4J577uGpp54iEomwePFiLr74YvLz89m2bRs/+MEP+NCHPsRDDz3ESSedxCWXXMJtt93Ga6+9huu6zJkzh0svvZTS0lLuuece7r//fsLhMI7j8P73v59Vq1axYsUKbr31VsrLyzly5Ah33nknzz77LAUFBZx99tl8/OMfx3EcHn/8cR599FFmzZrFn/70J4qKivj85z/PwoULAXj88ce57777OHjwIKNHj2blypWcfvrpPreuSPbTADUiATd58mQuvfRSHn30UdasWZNY/stf/pJ3332X7373u4RCIW655Rbuu+8+LrzwQgCamppobm7mhz/8IdZa2traeP/7388111yD67r86Ec/Yt26dXzlK1/hggsu4NVXX+X000/vMoZ5sjvvvJMjR45w2223cejQIb7zne8wbtw4zjrrLAB27NjBGWecwbp169iwYQN33HEHd9xxB21tbfz0pz/lxhtvpKKigsbGRpqbm9PfcCIBoMv1IsOQtZZHH32Uiy66iFGjRjFixAiWL1/OX/7yl8Q2xhhWrFhBXl4e+fn5jB49miVLllBQUJDY/uWXXx7U8VzXZePGjVx44YWMGDGCCRMmcN555/Hkk08mtikrK6OmpgbHcTjjjDNobGzkwIEDiVh2795Ne3s748aNo6qqamgbRCSgdCYvMgwdPHiQtrY2rr322sQyay2u6ybmi4uLyc/PT8y3tbVx9913s3Xr1sSocC0tLbiui+P0f75w8OBBIpEIZWVliWXjx4+noaEhMT927NjEdEFBAeCNCDZ27Fi++MUv8sADD3DHHXcwZ84cPvvZzzJ58uSj+/Aiw4iSvMgwNHr0aPLz87n55psT42h3Fx8lL+6BBx5g7969/Nu//Rtjx45l165dfOUrXyFe1tN9+2TFxcWEQiHq6uqorKwEoK6urs9jd7dgwQIWLFhAe3s7v/rVr/jP//xPvvWtbw3qvSLDmS7XiwwDY8eOpaGhgUgkAoDjOJx99tncddddiUviDQ0NbN26tc99tLa2kp+fT1FREc3Nzfz617/usn7MmDF9PhPvOA5Lly7lnnvuoaWlhf379/Pggw8OqniuqamJLVu20NraSjgcprCwcMArByLi0b8UkWHghBNOoLKykksvvZRVq1YB8KlPfYry8nKuu+46LrroItasWcPevXv73Me5555Le3s7q1at4rrrrmPBggU91v/tb3/jkksu4c477+zx/s997nMUFBRwxRVXcMMNN3Daaadx5plnDhi7tZYHHniAf/iHf+Bzn/scL730Ep///OdTawCRYUqP0ImIiASUzuRFREQCSkleREQkoJTkRUREAkpJXkREJKCU5EVERAJKSV5ERCSglORFREQCSkleREQkoJTkRUREAur/AY1bLvlVM1jdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "plt.xlabel('Iterations')\n",
    "plt.ylabel('Betas')\n",
    "plt.plot(beta_iter[:, 0:t].T);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd143ae3-9269-41f6-a5e5-de141b4c6153",
   "metadata": {},
   "source": [
    "Most of the regression coefficients change in the first iterations and then remain constant. There is one which is slightly modified at the beginning and then exponentially until one point where it also does not change."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}