Lesson15.1学习率调度的基本概念.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 15.1 学习率调度基本概念与手动实现方法\n",
    "\n",
    "学习率调整的场景：\n",
    "\n",
    "- 缓解Dead ReLU Problem      \n",
    "&emsp;&emsp;在ReLU激活函数叠加的神经网络中，由于ReLU本身负值归零的特性，可能会出现Dead ReLU Problem，我们可以通过减小学习率来降低模型落入活性失效陷进的概率。不过降低学习率也代表模型收敛速度更慢\n",
    "\n",
    "- 提升模型表现      \n",
    "&emsp;&emsp;学习率绝对数值的调整过程中，学习率对模型性能的影响整体呈现U型特征，即学习率过大或者过小都不好，学习过大可能会导致模型无法穿越狭窄的通道最终抵达最小值点，而学习率太小则容易在最小值点附近停止收敛，因此在进行模型训练时，我们需要找到一个适中的准确率取值\n",
    "\n",
    "**学习率的设置**\n",
    "- 模型学习率设置太大\n",
    "  - 虽然前期收敛速度较快\n",
    "  - 但容易出现收敛过程不稳定、收敛结果不佳、或者神经元活性失效等问题\n",
    "- 学习率设置太小\n",
    "  - 虽然收敛过程将相对平稳，并且能够有效规避神经元活性坏死的问题\n",
    "  - 但容易出现收敛速度慢、收敛结果不佳等问题。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 一、学习率对模型训练影响\n",
    "\n",
    "- 学习率较大的模型收敛速度较快，但效果却不一定最好\n",
    "- 学习率非常小的模型不仅收敛速度较慢，并且效果也不尽如人意\n",
    "- `A.extend(B)` 用于在列表A末尾一次性追加另一个序列B中的多个值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 设置随机数种子\n",
    "torch.manual_seed(420)  \n",
    "\n",
    "# 创建最高项为2的多项式回归数据集\n",
    "features, labels = tensorGenReg(w=[2, -1, 3, 1, 2], bias=False, deg=2)\n",
    "\n",
    "# 进行数据集切分与加载\n",
    "train_loader, test_loader = split_loader(features, labels, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 关键参数\n",
    "num_epochs = 20\n",
    "\n",
    "# 实例化模型  tanh为激活函数\n",
    "tanh_model1 = net_class2(act_fun = torch.tanh, in_features=5, BN_model='pre')\n",
    "tanh_model2 = net_class2(act_fun = torch.tanh, in_features=5, BN_model='pre')\n",
    "tanh_model3 = net_class2(act_fun = torch.tanh, in_features=5, BN_model='pre')\n",
    "tanh_model4 = net_class2(act_fun = torch.tanh, in_features=5, BN_model='pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#————————————————————————4组模型只有学习率不同————————————————————————\n",
    "# tanh_model进行模型训练\n",
    "train_l1, test_l1 = model_train_test(tanh_model1, \n",
    "                                     train_loader,\n",
    "                                     test_loader,\n",
    "                                     num_epochs = num_epochs, \n",
    "                                     criterion = nn.MSELoss(), \n",
    "                                     optimizer = optim.SGD, \n",
    "                                     lr = 0.03, \n",
    "                                     cla = False, \n",
    "                                     eva = mse_cal)\n",
    "\n",
    "train_l2, test_l2 = model_train_test(tanh_model2, \n",
    "                                     train_loader,\n",
    "                                     test_loader,\n",
    "                                     num_epochs = num_epochs, \n",
    "                                     criterion = nn.MSELoss(), \n",
    "                                     optimizer = optim.SGD, \n",
    "                                     lr = 0.01, \n",
    "                                     cla = False, \n",
    "                                     eva = mse_cal)\n",
    "\n",
    "train_l3, test_l3 = model_train_test(tanh_model3, \n",
    "                                     train_loader,\n",
    "                                     test_loader,\n",
    "                                     num_epochs = num_epochs, \n",
    "                                     criterion = nn.MSELoss(), \n",
    "                                     optimizer = optim.SGD, \n",
    "                                     lr = 0.005, \n",
    "                                     cla = False, \n",
    "                                     eva = mse_cal)\n",
    "\n",
    "train_l4, test_l4 = model_train_test(tanh_model4, \n",
    "                                     train_loader,\n",
    "                                     test_loader,\n",
    "                                     num_epochs = num_epochs, \n",
    "                                     criterion = nn.MSELoss(), \n",
    "                                     optimizer = optim.SGD, \n",
    "                                     lr = 0.001, \n",
    "                                     cla = False, \n",
    "                                     eva = mse_cal)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403a8ee00>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403a8ef50>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403a8f340>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403a8f6a0>]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f403a8f220>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABODElEQVR4nO3dd3xT9frA8c83q2nS3dI9KFv2doJ6EcEFbgEHXgfivupP4F5FxY04rturouDGDSriBMHBKnvJhi6690qTfH9/nLS00JaOtGna7/v1yivjnJPzJE2fnDzne54jpJQoiqIo3kfn6QAURVGU5lEJXFEUxUupBK4oiuKlVAJXFEXxUiqBK4qieClDW64sLCxMdu3atS1XqSiK4vWSkpKypZRdjn28TRN4165dWb9+fVuuUlEUxesJIQ7V9bgqoSiKongplcAVRVG8lErgiqIoXqpNa+CKorhHZWUlKSkplJeXezoUxY3MZjOxsbEYjcZGza8SuKJ4oZSUFPz9/enatStCCE+Ho7iBlJKcnBxSUlJITExs1DKqhKIoXqi8vJzQ0FCVvDsQIQShoaFN+lWlEriieCmVvDuepv5NvSKBF3z7Hflff410ODwdiqIoSrvhFQm88JtvSJ/1b/ZPnEjhTz+hepgriqJ4SQKPff01Yv77X3A4Sb3zLg5eeRUlf/6pErmieEh+fj6vvfZak5c7//zzyc/Pd39AbpKbm8vYsWPp2bMnY8eOJS8vr875Fi5cSM+ePenZsycLFy6sfnz8+PEMGjSIfv36MX36dBytXDXwigQudDoCxo+j2zdLiHriCew52Ry+4UYOX/9PyjZt8nR4itLp1JfA7XZ7g8stXbqUoKCgVoqq5Z5++mnGjBnDnj17GDNmDE8//fRx8+Tm5jJnzhzWrFnD2rVrmTNnTnWi//TTT9m8eTPbtm0jKyuLzz77rFXj9aphhMJgIOiySwm46ELyF31K9htvcHDSZPz+8Q+63H035t69PB2iorS5Od9sZ0daoVufs290AA9f1K/e6bNmzWLfvn0MHjwYo9GI2WwmODiYXbt2sXv3bi6++GKSk5MpLy/n7rvvZtq0acDRfkjFxcWcd955nHHGGfz555/ExMSwePFifH1961zfWWedxZAhQ1i1ahUlJSW89957PPXUU2zdupWrrrqKxx9/nJKSEq688kpSUlJwOBzMnj2bq666iqSkJO69916Ki4sJCwtjwYIFREVF1bmexYsXs2LFCgCmTp3KWWedxdy5c2vN88MPPzB27FhCQkIAGDt2LMuWLWPy5MkEBAQA2heZzWZr9R3NXrEFfiydyUTItdfQ48cf6PKvf1G6bh0HLr6Y1P+7H9vhw54OT1E6vKeffpru3buzadMm5s2bx4YNG3jxxRfZvXs3AO+88w5JSUmsX7+el156iZycnOOeY8+ePdx+++1s376doKAgvvjiiwbXaTKZWL9+PdOnT2fixIm8+uqrbNu2jQULFpCTk8OyZcuIjo6u3gIeP348lZWV3HnnnXz++eckJSVxww038MADD9S7joyMjOrkHhkZSUZGxnHzpKamEhcXV30/NjaW1NTU6vvjxo0jPDwcf39/Lr/88obfyBbyqi3wY+msVsKm30Lw5EnkzH+H3Pffp3DZMoIuu4yw227FGBHh6RAVpdU1tKXcVkaOHFnr4JOXXnqJr776CoDk5GT27NlDaGhorWUSExMZPHgwAMOGDePgwYMNrmPChAkADBgwgH79+lUn2m7dupGcnMyAAQO47777mDlzJhdeeCGjRo1i27ZtbNu2jbFjxwLgcDjq3fo+lhCiWVvQP/zwA+Xl5Vx99dX8+uuv1etuDd6xBV5eAFl/1ztZHxhI+L330OPHHwi+6iryv/ySfeeOI+OZedjr2QmhKIr7WK3W6tsrVqzg559/5q+//mLz5s0MGTKkzoNTfHx8qm/r9foT1s+r5tfpdLWW1el02O12evXqxYYNGxgwYAAPPvggjz76KFJK+vXrx6ZNm9i0aRNbt27lxx9/rHcdERERpKenA5Cenk54ePhx88TExJCcnFx9PyUlhZiYmFrzmM1mJk6cyOLFixt8TS3lHQn848nw1S0nnM3QpQuRsx+k+/ffE3DeeeQuWMC+c8aS9cqrOIqL2yBQRekc/P39KSoqqnNaQUEBwcHBWCwWdu3axerVq9skprS0NCwWC9dccw33338/GzZsoHfv3mRlZfHXX38BWg+Z7du31/scEyZMqB5VsnDhQiZOnHjcPOPGjePHH38kLy+PvLw8fvzxR8aNG0dxcXF18rfb7Xz33Xf06dOnFV7pUd6RwLueAemboSy/UbObYmOIfvopui1ZjPX008l+5RUOTLyYigMHWjdORekkQkNDOf300+nfvz/3339/rWnjx4/Hbrdz0kknMWvWLE455ZQ2iWnr1q2MHDmSwYMHM2fOHB588EFMJhOff/45M2fOZNCgQQwePJg///yz3ueYNWsWP/30Ez179uTnn39m1qxZAKxfv56bbroJgJCQEGbPns2IESMYMWIEDz30ECEhIZSUlDBhwgQGDhzI4MGDCQ8PZ/r06a36mkVbjqUePny4bNYZeQ7+DgsugEkfQ5/zm7x46YYNpNxxJ0hJ3Jv/w3fAgKbHoCjtyM6dOznppJM8HYbSCur62wohkqSUw4+d1zu2wGNHgMEMB1c1a3HL0KF0/ehDdFYrh6ZeT/Gq390coKIoStvzigT+yNK97Db1gwMrm/0cpq5dSfjoQ0zx8STfeisF33zjxggVRXGH22+/ncGDB9e6vPvuu167ntbmFcMI80ttLLf1plfGR1CSA9bQEy9UB2N4OAnvv0fK7XeQdv8M7Dk5hF5/vXuDVRSl2V599dUOtZ7W5hVb4PGhVn4sdR1l2cwyShW9vz9xb72J/7nnkvn0XDKffVb1VFEUxSt5RQJPCLGw2dkNp9HaojJKFZ2PDzEvPE/Q5EnkvD2f9H//B1lZ6YZIFUVR2o5XlFASQi3YMZAXNpzQFm6BVxF6PZEPPYShSxeyX3oZe14usS+8gM5iccvzK4qitDav2AKPD9WS6gH/YZC9GwrT3fK8Qgi63HYbkXPmULLqdw7985/qyE1FUbyGVyTwLn4+WEx6Nupc47cPuncYYPBVVxLz4n+p2LmLQ1dfQ2VamlufX1E6GtUPvO5+4A888ABxcXH4+fm1SbxekcCFEMSHWFhTFg3mQDjwm9vXETB2LPHz38aelcXByVMod3VVUxTleKofeN39wC+66CLWrl3bZvF6RQ0ctDr4vqwS6DrKLTsy62IZMYKED94n+aabOXTNtcS9/hqWYcNaZV2K4jbfz4IjW937nJED4Lzjk1cV1Q+87n7gbdU2oIpXbIEDJIRaOZxbijPhDMg/BHmHWmU95t69Sfj4YwwhIRy+4UaKfv21VdajKN5M9QOvvx94W/KaLfD4EAs2u5Oc8JPpAtp48OCEVlmXKTaGhI8+JPmW6aTccSdRj84hqJUbsytKszWwpdxWVD9wz/CiLXBtJMo+4sESBgfcM5ywPoaQEBIWvIv1tNNIf3A2OfPnt+r6FMWbqX7gMcfN1xa8J4GHaB+Qw7llkOiqg7fyEZQ6q5W4114l4PzzyJz3LEW//NKq61MUb6H6gR/fD9wTvCaBRweZMegEh3JLIHE0FKVBzr5WX68wmYh66inM/fuTNmMmFftVT3FFUf3Aj+8HDjBjxgxiY2MpLS0lNjaWRx55pFVfs3f0A3c5c95yBsQE8sq5AfDKMLjwBRh+gxsjrF9lWhoHLrscfWgIiYsWoavxk1FR2prqB95xdbx+4C7xIRYO5ZRCaHfwj2614YR1MUZHE/P8c9j2HyDtgQdVAyxFUTzOqxJ411Arh3JKQAhXHXxVq9fBa7Keeirh995D0bJl5L67oM3WqyidheoH3jReM4wQtJEoheV28kttBCWOhi2LIHMnRPRtsxhCbryRsi1byXz2Wcx9+2I95eQ2W7eidHSqH3jTeNUWeHyINpTwUE6ptiMT2rSMAtrY0Kgnn8TUtSup995LZbp7GmspiqI0lVcl8IRQbcfhodxSCIqHoIQWn+ChOfR+VmJfeRlZUUHK3f/CabO1eQyKoiiNSuBCiHuEENuFENuEEB8LIcxCiEQhxBohxF4hxCIhhKm1g63aAj+cU6I9kDhaS+BOR2uv+jg+3boR9fRTlG/ZQsbjT7T5+hVFUU6YwIUQMcBdwHApZX9AD0wC5gIvSCl7AHnAja0ZKICvSU+4v49WQgEtgZcXuL+RTyMFjB1L6M03k//pp+R//rlHYlAUpfNqbAnFAPgKIQyABUgH/gFUZa2FwMVuj64OCaEWrYQCWmdCaPM6eE1d/nU31tNO5cijj1G2dZvH4lCUtqT6gdfdDzwpKYkBAwbQo0cP7rrrrurhxo888ggxMTHVI16WLl3qlnhPmMCllKnAs8BhtMRdACQB+VLKquYFKUCdzQCEENOEEOuFEOuzsrJaHHB8iGsoIUBAFIT18mgCF3o90c89hz4slJS77sKem+uxWBSlrah+4HX3A7/11lt566232LNnD3v27GHZsmXVy91zzz3VPVnOP/98t8R7wmGEQohgYCKQCOQDnwHjG7sCKeWbwJugHYnZrChr6Bpq4YsNFZRXOjAb9dpW+JZF4KgEvbGlT98shuBgYl96mUNTppB6333Ev/UWwuBVIzQVLzZ37Vx25e5y63P2CenDzJEz652u+oEf3w/8rLPOorCwsLp1wHXXXcfXX3/Neeed16T3vikaU0I5BzggpcySUlYCXwKnA0GukgpALNAmDXGrzo95OLdGHdxWDGkb22L19fLt34/Ihx+m9K/VZL34okdjUZTWpvqBH98PPDU1ldjY2OMer/LKK68wcOBAbrjhhnpLM03VmM3Ew8ApQggLUAaMAdYDy4HLgU+AqcBit0R0AtVDCXNK6RXhX7sOHjeyLUKoV9Bll1K2ZQs5b72Nuf8AAsad69F4lM6hoS3ltqL6gTfs1ltvZfbs2QghmD17Nvfddx/vvPNOi5+3MTXwNWg7KzcAW13LvAnMBO4VQuwFQoFWbZhdtTMgofpgHlcd3BoKEf09WgevKeKB/2AeNJD0f/+bin2t3y1RUdoD1Q88hpiYGFJSUo57vOp59Xo9Op2Om2++2W3nzWzUKBQp5cNSyj5Syv5SymullBVSyv1SypFSyh5SyiuklBVuiej4dfPShpd4Yo021jrIYsTfbDhaQgGtjJK8BuytEkKT6EwmYl98EWE2k3LnXTiKiz0dkqK4neoHfnw/8KioKAICAli9ejVSSt57773q5dNrHLH91Vdf0b9/f7e85nZ/JKYQAru0s+jvRfx6+FeEENpQwpwaCbzrKLCXQ8o6zwVagzEykpgXXsB26BDp//6P6lyodDiqH3jd/cBfe+01brrpJnr06EH37t2rd2DOmDGDAQMGMHDgQJYvX84LL7zgltfsFf3AKx2VXL30atJL0vlywpc88lUyO9ILWf5/Z2kzlOXDM4kw+n44+z9ujbklct55l8xnnqHLffcSdvPNng5H6UBUP/COq8P1AzfqjTw9+mnK7eU88PsDxIWYSc4txe5wajP4BkHUoFY/T2ZThfzzevzPG0/WC/+lpIFvfUVRlObwigQO0C2wG/ePuJ+/0v8iU/yE3SlJL6ixYyRxtFZCsZV4LshjCCGIfvxxfLp3I/X+GTja8RFoitIeqH7gTeNVR5tc0esKVqWuYnnKQnQ+t3Iop5Q416gUEkfDHy/C4dXQY4xnA61BZ7USPW8eBy67nIx584h+QjW+UpT6qH7gTeM1W+CgbdHOOW0OAaYAzDGfsDe7xmHrcaeAzuCR9rInYu7Th9Ab/knBF19S0kZ75BVF6fi8KoEDhJhDePyMx9H7ZPLN4beOTvDxg5jh7WY8+LHCbr8dY3w86Q89jLOOMbGKoihN5XUJHGBU7Bn4lp3F7vJlrEypkbATR2mH1JcXeC64eujMZqIenUPl4cNkv9r0Lm6KoijH8soEDtDXPBmjI5rZf8wmuyxbezBxNEgnHPrLs8HVw3rKKQReeik577xD+S73Nh9SFKXz8doE3jUkkIq0yRTbinnoj4e0g2ViR4Lep92WUQAiZtyPPiiI9AdnIx1tfyYhRXEH1Q+8af3AP/vsM/r164dOp6M5x8LUx2sTeEKohZLiLkwbcBerUlfxyd+fgNGsNbQ62H4TuD4oiMgH/kP5tm3kvv++p8NRlGZR/cCb1g+8f//+fPnll4wePdqt8XrVMMKaElxtZYcFXciomDU8t/45RkSMoEfimbD8cSjNBUuIh6Osm/955+G3eAlZL76E/zljMcXWeS4MRWmUI08+ScVO95bkfE7qQ+R/6j+qWfUDb1o/8NY6ataLt8C17meHc8t49PRHsRqtzFw1E1uCq+/Cwd89GF3DhBBEPvwQQgiOPPKI6pWieB3VD7zp/cBbg9dugccG+yKE1hc8zDeWx05/jNt/uZ0XM/7gfqNVq4P3neDpMOtljI6myz33kPHEExR++y2BF13k6ZAUL9XQlnJbUf3APcNrt8B9DHqiA32r28qOjh3NpN6TeG/Xh/wZP7Bd78isEjxlMuZBA8l48insbjpDh6J4guoH3nA/8NbitQkcID7EcvTEDsB9w++jW2A3HhR55OXugaLjf/60J0KvJ+rRx3AUFZH59NwTL6Ao7YTqB960fuCtxasTeEKopdaJHcwGM3NHzyXfaePhsBCkF2yFm3v3IvTmmyhYvJjiP/7wdDiK0iiqH3jT+oF/9dVXxMbG8tdff3HBBRcwbtw4t7xmr+gHXp/XVuzlmWV/s23OOPx8jpbzF25bwLNJz/GQpTdXXPG529bXWpwVFRy4+BJkZSXdlixGZ7F4OiSlnVP9wDuuDtcPvD4JIVUnOK7dQvbaftdxirDyTMnf7C/Y74nQmkTn46MdZp+SQtYrHaNLmqIorc+7E7hrLPjhmqdXA3RCxxPdLsfH6WDW8vuodFR6IrwmsYwYQdCVV5K7YAFlDdToFKUjU/3Am8ZrhxECxLsS+KHc0uOmhfcYz5w/n+Vf+r28sukV7hl2T1uH12Th/3cfxcuXkz57NomffoowePWfR2llUsp2P8ytqTp7P/CmlrS9egs8wGwkxGqqfYLjKuF9GYOFy4wRvLvtXdamr237AJtIHxBAxIMPUrFjJ7k1+isoyrHMZjM5OTnqILAOREpJTk4OZrO50ct4/SZefIiFw7l1nEZNp4OuZzAjJYmkxEQeXf0oSy5egk607+8s/3PH4jdmDFkvv4L/uediqnHEl6JUiY2NJSUlhaysLE+HoriR2WyudTTniXh9Ak8ItZB0qJ6DYBJHY9mxmFu7/x8zNz7Pn2l/ckbMGW0bYBMJIYh8aDb7L7iQIw8/TNz8+R3uZ7LSckajsdaRj0rn1L43RxshIcRCWn4ZNrvz+ImJZwIwttxBqDmUj3d93MbRNY8xIoLw++6l5M+/KPh6safDURSlnfL6BB4fasUpITW/7PiJoT3ALxLj4T+5ovcVrEpZRXJh8vHztUNBV12F79ChZD79NPY6GgEpiqJ4fQKvGkp4MKeOOrgQ2ll6Dqziip6Xoxd6rW+4FxA6HVGPzsFZWkrGU8f3JFYURfH+BB5S91jwaomjoCST8JI8xiSM4au9X1FaWc+87YxPjx6E3nILhd9+S/HK9t8WQFGUtuX1CbyLvw++Rn3dQwlB2wIHOLCSKX2mUGQrYumBpW0XYAuFTrsZU/fuHJnzKM6KCk+HoyhKO+L1CVwI4WpqVUcJBSC4KwTFw4HfGBI+hN7Bvfl418deM35WZzIROXs2lamp5KlTsCmKUoPXJ3CoaivbQFmk62g4+DtCSib3mczuvN0kZSS1XYAtZD3lZPzOPpvsN/6ndmgqilKtQyTwqrayTmc9W9WJo6E8HzK2cn638/E3+XvNkMIq4fffj7O8nKxXXvF0KIqitBMdIoHHh1qpsDvJLKqnRpw4Srs+sBJfgy+X9riUXw7/QkZJ+z7hQ00+3RIJnjSJ/EWfUrFnj6fDURSlHegQCbxqJMqxbWWrBURDaM/q06xd1ecqnNLJZ7s/a6sQ3SLs9tvQ+fmRMW+ep0NRFKUd6BgJvKorYUN18MTRcOhPcFQS5x/H6NjRfLb7M2wOWxtF2XKG4GDCbr2VkpWrKF71u6fDURTFwxqVwIUQQUKIz4UQu4QQO4UQpwohQoQQPwkh9riug1s72PpEB/mi1wkO1TcSBbQEbiuGtI0ATO4zmdzyXH469FMbRekewVdPwRgfT+Yzc5EnOAmsoigdW2O3wF8Elkkp+wCDgJ3ALOAXKWVP4BfXfY8w6nXEBvueYCRKVR38NwBOjT6VhIAEPtr1URtE6D46k4nw/7uPij17yf/iS0+HoyiKB50wgQshAoHRwHwAKaVNSpkPTASqmlYvBC5unRAbR2sr20ACt4ZCxIDqOrhO6JjcZzJbsrawPdu7zoDjP3YsvsOHkfXSSziKiz0djqIoHtKYLfBEIAt4VwixUQjxthDCCkRIKdNd8xwBIloryMZICD3BWHDQyiiH10BlOQATuk/A1+DrdUMKhRBEzJyFIyeHnDff8nQ4iqJ4SGMSuAEYCrwupRwClHBMuURqhzXWOQhbCDFNCLFeCLG+NZvPJ4RYKSirpKC0gfNfJo4GRwWkaGfn8Tf5M6H7BL4/8D155fX0FG+nfAf0J2DCReQuWEBlaqqnw1EUxQMak8BTgBQp5RrX/c/REnqGECIKwHWdWdfCUso3pZTDpZTDu3Tp4o6Y63T0/JgN7MhMOA2EvrqMAjCp9yRsThtf7vG+enL4PfeAEGS+8F9Ph6IoigecMIFLKY8AyUKI3q6HxgA7gCXAVNdjUwGPnnmgUUMJzQEQPaRWAu8R3IORkSNZ9Pci7E7vGtVhjIoi5IZ/Uvjtt5Rt3uzpcBRFaWONHYVyJ/ChEGILMBh4EngaGCuE2AOc47rvMfEnOpinSuJoSE2CiqLqh6b0mUJ6STq/pfzWmiG2irCbbkLfJYyMp+d6TYMuRVHco1EJXEq5yVUGGSilvFhKmSelzJFSjpFS9pRSniOlzG3tYBtiMRno4u/TuB2ZTjscXl390JlxZxJpjfS6nZkAOquV8LvvpmzjRop++MHT4SiK0oY6xJGYVRJCLBxqaCghQNzJoDdVjwcHMOgMXNX7Ktakr2F//v5WjtL9Ai+5BJ/evcl89jnVM1xROpGOlcBDrfWfmaeKyQKxI2vVwQEu7XkpRp3RK7fChV5PxMwZVKakkPfBB54OR1GUNtLBEriFI4XllFc6Gp4xcTSkb4HSo1WfEHMI5yWex5J9Syi2ed/BMdbTTsPvzDPJfv0N7LkerWYpitJGOlwCB0g+URklcTQg4dAftR6e0mcKpfZSFu/z6ICaZgufcT/OsjKyVc9wRekUOlQCPzoS5QQJPGYYGC3HlVH6hfVjYNhAPtn1CU7pbK0wW41P9+4EX3UVeYs+pWLvXk+HoyhKK+tQCTwh1Apw4h2ZBhPEn3pcAgeY1GcSBwsPsjp9dR0Ltn9hd9yOzmJRPcMVpRPoUAk82GLE38dw4rHgoJVRsnZBUe2z8ozrOo4Qc4hX7swEMISEEDZ9OiW/raT49z9OvICiKF6rQyVwIQTxjWlqBa46OHBwVa2HTXoTl/W8jN+SfyO12Dt7jARfew3G2Fgyn3kG6TjBDl1FUbxWh0rgcPQExycUNQh8Ausso1zZ+0p0QseiXYtaIcLWp/UM/z8qdu8m/0vv6/GiKErjdMAEbiUlrxRHfWeor6LTQ9cz6kzgkdZI/hH/D77Y8wVl9rJWirR1+Y87F9+hQ8l68SUcxY0oKSmK4nU6XgIPsVDpkKTlNyLxJo6GvAOQf/i4SZP7TKbQVsiyA8taIcrWJ4QgYtZMHNnZ5LyteoYrSkfU4RJ4VVvZRpVRqurgB1YdN2l4xHB6BPXgo10feW2TKN+BAwm48EJy311AZVqap8NRFMXNOlwCrx5K2JgdmeEngSWszjKKEILJfSazK3cXm7O8t1Vr+L33AJD53PMejkRRFHfrcAk8MsCMSa9r+MQOVYTQtsIPrIQ6trIv7HYh/kZ/rzvxcU3G6GitZ/h336lhhYrSwXS4BK7XCWJDfDmU3YgtcNASeFEa5Ow7bpLFaOHinhfz08GfyCptvdPBtbaw6dMxJSZy5KGHcJaoHZqK0lF0uAQOjWwrW6W6Dl73yRwm9Z6EXdr5fPfnboqu7el8fIh6/DEq09LI/O+Lng5HURQ36ZgJPNTK4ZySxu18DOkGAbF11sEB4gPiOSPmDBb9vQibw+bmSNuOZdgwgqdMIe+DDyjdsNHT4SiK4gYdNIFbKLE5yClpRMKtqoMfXAXOuhtYXdf3OnLKc/hu/3dujrRtdbn3XgyRkaTPno3T5r1fRoqiaDpsAodGjkQBLYGX5kDmjjonnxJ1Cr2Ce/Hejve8dkghgN7PStScR7Dt20fOG294OhxFUVqoQybw+BBtKOHhxoxEAUgcpV3XU0YRQnB9v+vZm7+XP9K8eySH3+jRBE6cQPabb1H+99+eDkdRlBbokAk8LsQXIZqwBR4YCyHd603gAOO7jifcN5wF2xe4J0gPCp81C31AAOkPPIi02z0djqIozdQhE7iPQU9UgPnE58esKXGUdoYeR90Jzag3cnXfq1mTvoZdubvcFKlnGIKDiZz9IOXbtpG78D1Ph6MoSjN1yAQO2iH1BxvTF7xK4mioKIT0+o+6vLzX5VgMFhZuX+iGCD3Lf/x4/MaMIeull7AdOuTpcBRFaYYOm8ATQqyN64dSpWtVHbzu8eAAAaYALu15KcsOLONIyZEWRuhZQggiH3oIYTKR/uBsZD0jcBRFab86bAKPD7WQXWyjuKKRNV6/cAjv22AdHOCavtfgxMlHO7338PoqxohwwmfcT+m6deR/5r0HKilKZ9VhE3hXV1OrptXBR8Ph1WCvqHeWGL8Yzk04l892f0axrbilYXpc0OWXYzn5ZDLnzaPyiHf/qlCUzqbDJvCE6rayTayD28sgZX2Ds03tN5XiymK+3OP9Z7sRQhD12KNIu50jj8zx6nHuitLZdNgEHt/Ug3kAEk4HoTthGaV/WH+GRQzjg50fYHd6/zA8U3w8Xe6+m+IVKyhcutTT4SiK0kgdNoEHmI0EW4yNb2oF4BuknSvzBAkcYGrfqaSXpPPToZ+aH2Q7EnLdtZgHDiTj8Sew5+V5OhxFURqhwyZwgPhQa9Nq4KCVUVLWga3h0suZcWfSNaArC7Yv6BBlB6HXE/X4YziKi8l48ilPh6MoSiN06ASeENLEseCgJXBnpbYzswE6oePavteyI2cH6zMarpl7C3OvXoRNm0bhN99QtGKFp8NRFOUEOnYCD7WQll+Gzd6EMc7xp4LO0KgyyoTuEwj2Cea97R3naMbQW6bh07MHRx6Zg6PY+0fZKEpH1qETeHyIBaeE1Macob6KyQqxIxqVwM0GM5P6TGJFygr2F+xvQaTth85kIurxx7FnZJD53HOeDkdRlAZ06AR+9ATHzSijpG+CsvwTznpV76vw0fvw/o73mx5gO+U7aBAh111H/sefULpunafDURSlHh06gXetHgvejB2Z0gmH/jzhrKG+oVzU/SK+2fcNOWU5zQmzXepy910YY2NJf3A2zvJyT4ejKEodOnQC7+Lvg69R37Sx4KCVUAzmRpVRQDtjT4WjgkV/L2pGlO2TzmIh6tE52A4dIvvVVz0djqIodWh0AhdC6IUQG4UQ37ruJwoh1ggh9gohFgkhTK0XZvMIIYgPsTQ9gRt8IP6URifwxMBEzoo9i092fUK5veNsrVpPO43Ayy4l5513Kdu23dPhKIpyjKZsgd8N7Kxxfy7wgpSyB5AH3OjOwNwlPtTStMPpqySOhsztUJzVqNmv63cdeRV5LNm3pOnrasciZsxAHxJM2syZ2HM6TolIUTqCRiVwIUQscAHwtuu+AP4BVLWwWwhc3ArxtViCawvc6WziwTaJZ2rXB1c1avbhEcPpF9qP93e8j1N2nNas+sBAYubNozI1lUNXX0NlaqqnQ1IUxaWxW+D/BWYAVZkpFMiXUlY1AkkBYupaUAgxTQixXgixPiurcVuz7pQQaqHC7iSzqP4Og3WKGgwm/0aXUYQQTO03lYOFB/ktuf6e4t7IesopxL8zH3tODgevvoaK/R1jyKSieLsTJnAhxIVAppQyqTkrkFK+KaUcLqUc3qVLl+Y8RYvEN3cood4AXU9vdAIHGJswlihrFAt3eP8Ze45lGTqUhPffQ9rtHLr6Gsq2bvN0SIrS6TVmC/x0YIIQ4iDwCVrp5EUgSAhhcM0TC7TL39YJIa6uhE0dSghaHTx3HxQ07qUZdAauOekakjKS2Jbd8RKcuU8fun74ATqLhcNTp1KyZq2nQ1KUTu2ECVxK+W8pZayUsiswCfhVSnk1sBy43DXbVGBxq0XZAjHBvuh1oulNrUBL4NDoOjjAZb0uw9/o3yHOm1kXU0ICCR99iCE6iuSbb6bo1189HZKidFotGQc+E7hXCLEXrSY+3z0huZdRryMmyLd5W+Dh/cA3pEllFKvRyuW9LufHQz+SWtwuf5S0mDEigoT338enTx9S7ryL/K+/9nRIitIpNSmBSylXSCkvdN3eL6UcKaXsIaW8QkrZxL2EbSch1MLGw3mUNPb8mFV0OkgcpSXwJrSMnXLSFHTo+GDHB02M1HsYgoOJf+cdLCNHkD7r3+S+13EaeimKt+jQR2JWmTa6G2n5Zdz9ySYcTR5OOBoKkiHvQKMXibRGMj5xPF/u+ZJCW2ETo/Ueej8rcf/7H/5jzyHjyafIevmVDtEbXVG8RadI4KN6duHhi/rx884M5i7b1bSFq8aDN6GMAtp5M0vtpXy+u2Of7V1nMhHzwgsEXnop2a++SsYTTyKdHWccvKK0Z50igQNMPa0r156SwJsr97No3eHGLxjaA/yjYO8vTVpfn5A+nBx1Mh/u+JBKR2UTo/UuwmAg6onHCbn+evI++IC0WbOQlR37NStKe9BpEjjAwxf1ZVTPMB74aht/7WvkYeFCwEkXwc4l8O09UNn4XifX97uezLJMlh1c1syIvYcQgvCZM+jyr39RuOQbUu66W3UxVJRW1qkSuEGv45UpQ0kItXDrh0kcyG7kwT3jnoTT74b178D8sZDbuCMRT48+nR5BPTrMeTNPRAhB2PRbiHz4IYpXrCD55mnqrD6K0oo6VQIHCPQ18s71IxDAjQvWUVDaiJ/6eiOMfRQmfQz5h+B/Z8HOb064mBCC6/pex+683axOb/gcmx1J8OTJRM+bR+nGjRy+bir23FxPh6QoHVKnS+Cgnannf9cOJzmvlNs+SqLS0cidbn3Oh1tWQWh3WHQNLPsP2G0NLnJBtwsI8w3jjc1vYHc2cRijFwu88ALiXn2Fiv37tSZYaWmeDklROpxOmcABRiaG8OQlA/hjbw4PL9ne+BJHcALcsAxG3gKrX4UF50N+cr2zm/Qm7hpyFxsyN/DIn490qE6FJ+J35pnEz38be3Y2B664kvyvvlYjVBTFjTptAge4Yngc08/szkdrDvPuHwcbv6DBB85/Bi5/FzJ3wf9GwZ6f6p39kp6XcNug21i8bzHPrX+uU9TDq1iGDSPhww8wxcaS/u9/c3DSZMq2bPF0WIrSIXTqBA4wY1xvzu0bwePf7WD5rsymLdz/Upi2AgJi4MPL4ZdHwVF3mWT6oOlM7jOZ93a8x/xt7bLrQKsx9+pFwscfEfX0U1Smp3HwyqtI+/d/sHugvbCidCSdPoHrdIL/ThrMSVEB3PnxRv4+UtS0JwjrATf9DEOvg1XPwfsXQ9GR42YTQjBr5CzOTzyfFze8yGe7P3PPC/ASQqcj6OKL6f79MkJvvonCb79l3/jzyJk/H2lreD+Coih16/QJHMBiMvD21OFYTHpuWLCO7OImtnUx+sKEl+Hi1yFlPbwxqs4jN3VCx+NnPM6omFE89tdj/HjwRze9Au+h97MSft99dPv2GywjR5I571n2XzSBohUrPB2aongdlcBdogJ9eXvqcHJKKpj23nrKKx1Nf5LBU+DmX8E3CN6bCL/Ng2N22hl1Rp476zkGhw9m5qqZ/Jn2p3tegJcxJSQQ9/prxL31Juh0pEy/lcPTplGxv/E9ZxSls1MJvIaBsUE8f+VgNhzOZ9YXW5q3szGiL9y8HPpfBssf12rjJbWP+vQ1+PLKmFfoFtiNfy3/F1uyOu9OPb9Ro+i2+GvCZ86kbMNG9k+YQMbcZ9QBQIrSCCqBH+P8AVH837m9+HpTGq/8urd5T+LjB5e+BRe+oJ0M4o0zILn22WsCTAG8cc4bhJpDue2X29iXv88N0XsnYTIR+s/r6b7sewIvnkjuggXsG38e+V98qYYdKkoDVAKvw+1n9+CSITE899NuvtuS3rwnEQKG3wA3/gQGEyycAAdqn9mni6ULb577JkadkWk/TSOtuHMf7GIICyP68cfp+umn2rDDBx7g4FWTKNu0ydOhKUq7pBJ4HYQQPHXpAIYlBHPvp5vYnJzf/CeLHgw3/aIdAPTRVXC49iH1cf5x/G/s/yizlzHtp2nklDWyyVYH5jugPwkff0T0M3OxZ2RwcNJkUu+fge3QIU+Hpijtikrg9TAb9fzv2mF08ffhpvfWk5Zf1vwns4bBdUsgIAo+uBxSkmpN7hXci9fGvEZGSQa3/nwrxTZV/xVCEDhhAt2/X0rotGkU/fgj+86/gLQHH8SW0jFPVacoTaUSeAPC/HyYP3UEZTYHNzS28VV9/CNg6jdgDYUPLoH0zbUmDw4fzPNnPc+evD3c+eudlNtVK1YAndVK+L330P2nHwm+egqFS75h3/jxpD/8CJXpzSxvKUoHoRL4CfSO9Of1a4ayP6uEqe+upbip59WsKSBaS+I+AfDexZCxvdbkUbGjeOKMJ0jKSOL+lfd3quZXJ2IMDyfyP//REvmVV5L/5ZfsO3ccRx59jMqMDE+HpygeoRJ4I4zq2YWXpwxha2oBNy5YR5mtGWPEqwTFw9QlYDBrY8WzdteafH638/nPyf9hRfIKHv7z4U7V/KoxjBERRD40mx4/LCPw0kvJ+/RT9o09lyNPPqkOzVc6HZXAG2lcv0iev3IQaw/mMv2DJCrsLUjiId20JI6AhRdBTu0hhJP6TOK2wbexZN+STtf8qrGM0dFEzXmE7su+J2DCReR9+BF7x55LxtxnsOeoHcFK56ASeBNMHBzDU5cM4LfdWdz98Sbsje0jXpewnloSd1ZqQwzzao+wmD5wOlP6TOmUza+awhQbS/Tjj9N96XcEjBtH7sKF7D1nLJnPPYc9L8/T4SlKq1IJvIkmjYxn9oV9Wbb9CDM+34LT2YKt4/CT4LrFYCuGhRdCQUr1JCEEM0fO5IJuF/Dihhf59O9P3RB9x2VKSCB67tN0+/Zb/MeMIeft+ewbcw6Z//0vjvx8T4enKK1CJfBmuPGMRP7v3F58uTGV2Yu3tazEETkArv0KyvK1ckqNToY6oeOx0x9jdOxonljzBMsPL2958B2cT7dEYp6dR7dvlmA9czQ5b/yPveeMJeu113CqrodKB6MSeDPdfnYPpp/ZnQ/XHObJpTtblsRjhsI1X0BxplZOKT66M86oMzJv9Dz6hvRlxsoZnbpvSlP49OhB7AsvkLj4a6ynnkL2Sy9z4OJLKE1KOvHCiuIlVAJvJiEEM8f35rpTE3hr1QFe/GVPy54wbiRM+RQKkrXRKaVHTwRsMVp4ZcwrhPmGceevd3K48HALo+88zL17E/vyy8S9/TayooJDV19D+pw5OIqa2PddUdohlcBbQAjBIxf14/Jhsfz35z28tXJ/y56w6+kw+WPI2asl8bL86kmhvqG8fs7rOKWTW3++ldxydab3pvA743S6fbOEkOuvJ3/Rp+y/8CKKfvnF02EpSouoBN5COp1g7mUDuWBgFE8s3ckHq1vYr6PbWTDpQ8jaBR9cCuWF1ZO6Bnbl5X+8TEZpBnf+eidl9hYc3t8J6SwWImbNpOuiT9AHBZFy+x2k3P0vNX5c8VoqgbuBXid44crBjOkTzuzF2/hyQ8qJF2pIz7FwxULtcPsPr4CKo71RBocPZu6ouWzN2srMlTNxOFswHr2T8h0wgMTPP6PLPfdQvHw5+y64kPwvvlDj7RWvoxK4m5gMOl69eiindgvl/z7bzPdbW9ino8/5cNl8SFkLH08CW2n1pDEJY5g5cibLk5fz9NqnVeJpBmE0EnbLNBIXf425d2/SH3iQw9f/U3U8VLyKSuBuZDbqeeu64QyOC+KuTzay/O8mnuX+WP0uhkvehIO/w0dXQtHRnh9Xn3Q1U/tO5ZO/P2HB9gUtW08n5pOYSPzCBUQ+OofyHTvYP2Ei2W+9haxsQeMyRWkjKoG7mdXHwLv/HEmvCH+mv5/E6v0tPKx74BVw6ZuQsg7eOB32/Fw96d7h9zKu6zieT3qe7w9838LIOy+h0xF85ZV0+/Zb/EaPJuu55zlw5VWUbdt+4oUVxYNUAm8Fgb5G3r/xZOJDLNy4YB0bD7fwkO6BV8K0FWANhw8vg2X/BnsFOqHjiTOeYFjEMB74/QHWHVnnlvg7K2NEOLEvv0TMSy/iyM7m4JVXkvHMPJxlamex0j6pBN5KQqwmPrjpZML8fZj6zlp2pBWeeKGGhJ+knfF+5DRY/Rq8PQayduOj9+HFs18kzj+Ou5ffzd68Zp7HU6kWcO65dPvuW4Iuv5zcd95h/4SJFP/+h6fDUpTjnDCBCyHihBDLhRA7hBDbhRB3ux4PEUL8JITY47oObv1wvUtEgJkPbzoZPx8Dk978i992t3C4mtEM58+DyZ9AQSq8eSYkLSTQFMDr57yOj96HW3+5lczSFtbeFfQBAUQ9OoeE999D6PUk33QTh67/J6UbNng6NEWp1pgtcDtwn5SyL3AKcLsQoi8wC/hFStkT+MV1XzlGbLCFRbecSnSQL/98dy3/+21fy0eN9D4Pbv0TYkfAN3fBZ1OJ1vvy2pjXKKwo5Lafb1OnZXMTy4gRJC7+moh/z6Jizx4OTbmawzfdTNkW1dJA8TzR1GQihFgMvOK6nCWlTBdCRAErpJS9G1p2+PDhcv369c0O1puV2uzc/9kWvtuazoRB0cy9bCC+Jn3LntTphD9fhF8fB/8ouPQt/jA4uf2X2xkZOZJXz3kVo87onheg4CwtJe/jj8l5620c+fn4nXUWYXfegW+/fp4OTenghBBJUsrhxz3elAQuhOgKrAT6A4ellEGuxwWQV3X/mGWmAdMA4uPjhx3qxONspZS8/ts+5v3wNydFBvDmdcOIDba0/IlTkuCLGyH/EJw5k6+ievDQ6jlM7D6Rx05/DO3Po7iLo7iEvA8+IOfdd3EWFOA/9hzC7rgDc+8Gt18UpdlanMCFEH7Ab8ATUsovhRD5NRO2ECJPStlgHbwzb4HXtPzvTO76eCNGvY5XpgzhtO5hLX/SiiL47v9gyycQfyqv9T2T1//+iOmDpnP74Ntb/vzKcRxFReQufI/cBQtwFhfjP348Xe64HZ8ePTwdmtLB1JfAGzUKRQhhBL4APpRSful6OMNVOsF1rfacNdLZvcNZcscZhFhNXDt/Le/+caDldXEff7j0f3DpW3BkG7cuf52LQ4fwxuY3+HLPlydeXmkyvb8/Xe64nR6//Ezo9FsoWbmS/RdNIPX+GVQcOODp8JRO4IRb4K7yyEIgV0r5rxqPzwNypJRPCyFmASFSyhkNPZfaAq+tqLySexZt5uedGVw2NJYnLumP2djCujhA7gH44kYqU5O4o+cg1jgKefbMZxkTP0aVU1qRPS+P3Pnzyf3wI2RFBYETJxJ2262Y4uI8HZri5ZpdQhFCnAGsArYCVSeB/A+wBvgUiAcOAVdKKRvscaoS+PGcTsmLv+zhxV/2MCg2kDeuHUZUoG/Ln9hRCcufpOSP/3J9XBy79JJw3zBGxZ7JqNhRnBp1KhajG+rvynHs2dnkvPU2eZ98gnQ4CLrkEkJvuQVTbIynQ1O8lFt2YraUSuD1+2H7Ee5dtAlfk4E3rhnK8K4h7nni/b9R/PUt/OwsZKXFl798fSnWCYwIhpkjGB0ygFHxZ9M19jSwhIDaQnebyoxMct58k/xPP0VWVmLq3h3LyBFYR47EMmIEhjA37PtQOgWVwL3Anowibn5vPan5ZTwyoR9Xn5zgnicuy9caYuUdoDJ7DxvzdrKq7Agr9ZXsN2nDDOMrKxld4WSUKYzhgT0xhfWEkG5HL9YuKrk3U2V6OoXffUfJunWUrU/CWVICoBK60mgqgXuJgrJK7vp4I7/tzmLKyfE8clE/TIZW6nhgryAldS2rDv7IyswNrCtNoQInvhJOKStjVGkpo0rLiXQ4IKI/XLcErKGtE0snIe12ynfupHTtWkrWrlUJXWkUlcC9iMMpefbHv3l9xT6GJwTz2jVDCfc3t/p6y+xlrDuyjpUpK1mZ8hvpJUcA6G0K5fSsw5xsjWXIlMX4+qquCe4i7XbKd+yondBLtd7vNRO679ChGMLD1U7oTkolcC/07ZY07v9sC4G+Rh6Z0Jcu/j6YjfqjF4MOX5Mes0GPTufef2wpJXvz97IqdRUrU1ayOXMjdunEiGBwxDBGRp3MKVGn0C+snzra040aSui6gAB8EhMxde+OT/fumLp3w6d7d4zR0Qi9G0YvKe2WSuBeakdaIdPeX09KXsMtTU16HT5GHb7VCV5XK9n7+egJtfoQ6mci1Goi1M/HdW0i1OpDoK+xwS+BksoSNvz2OGt2fMTa0Fh22YuQSCwGC8MihnFy1MmcHHUyvYJ7oROqyaW7VCX0sk2bqdi/D9u+/VTs348j52ifeeHjgykxEZ9u3aqTuqlbN0xdu6IzmTwYveIuKoF7seIKO9tTCyi3OymvdNS4OCmrcbveaXYnReWV5JbYyC+t+0wzep0gxGqqldRDrCbC/LRknxBqYUTXEIw/z4a/XiH/Hw+wLmEIa9LXsCZ9DQcLDwIQ5BPEiMgRnBJ1CidHnUy8f7xHfvZLKalwVFBoK6SwohAfvQ+x/rEdpgThyM+nYv9+KvZVJXXtujItDar+p/V6TLGx2hZ7t0SMsXEY42IxxcZijIpCqOTuNVQCVwCodDjJK7GRU2Ijp9hGTknFMdc2cooryCmxkVtso6jCXr1skMXI2D5d+L+iuUQkf6+ds3PA5QAcKTnCuiPrWJ2+mjXpa8go1U7/FmGJ4OSok+kb2hejzohBZ0Av9Oh1egzCgF6nRy/0GHSG4+5Xz6cz4JROCisKKbQVUmQrotBWSEFFQfXtqkuRrah6vkpn7S+reP94zow7k7PjzmZw+OAOWfpxlpVhO3CAihpJvWL/PmyHDkPN08TpdBgiIzDFxmGMjcUUF4sxVruYYmPRh4V1mC+79kBK2aL3UyVwpVnKKx3kltjYklLAD9uP8PPODGzlpXzoM5dBur0kjZrPwDMuwGIyVC8jpeRw0eHqrfO1R9aSX5Hv9th0Qoe/yZ8AUwABpoCjt30Caj0e4BNAfnk+v6X8xtr0tdicNvxN/pwRcwZnx53N6TGnE2AKcHt8doeTvzOK2HAoj7SCcqICzcQG+xITZCEm2Bc/H8OJn8RNpMOBPTOTypQUbMkpVKYkY0tJoTI5hcqUFOxZtXvVC19fjDHR1QneGBODISwMQ1go+tBQDF26oA8MROgaLpc5nZLf9mTx174cIgPMJIRaSAi1EBtscc9Rx+1UTnEFm1Py2ZRcwObkfDan5PPjv0YTHtC8wQgqgStuYbM7+WNfNis37ebanbcQKnOZ4phDTK+hnDcgkn/0iSDQt/aWrVM6KagowCEd2J127E47DunA4XRgl3YcTkeD0+zSjg7dcYnZYrQ0ud5eWlnKX2l/sSJlBStTVpJbnotBGBgaMZSz4s7irNiziAto3qHvBaWVbEjOY8OhPJIO5bE5OZ8SmwPQSlQOZ+3/tUBfoyuh+xLjuo4N9iU22EJMkC9BFmObbQU7y8upTE3FlpxMZUoqlcnJ2FJdCT45uXpHai16PYaQEPRdwjCEhmEIDdUSfFgYzsBg/siTfLavhK2lRkrNFuzy6GsRAiIDzMSHaAk9PsRCfKiVBNf9IIv3lHfKbA62pWmJepMrWSfnavusdAJ6RfgzOC6IO/7Ro9ndR1UCV9zOnnMQ59vnUGrXcTWPs73IilEvOK17GOf1j2Rs3whC/Xwa/XxOpySvVCvjZBdVkF11XVxBhd1JkK+RIKuJYIuRYIuJQF8jwa77vkZ9k5Odw+lga/ZWViSv4LeU39ibr52OrltgNy2Zx53FwLCB6HXHbyk6nZL92cVsOJRP0qE8kg7nsTdTO4mGXic4KcqfofHBDEsIZmh8MDFBvmQXV5CSX0ZqXhkpeWWk5pfWuF1GqSvZV7GY9NVJPSrIt/p1B1m011x1HWwxEeBrRO/mkUhVpJQ4Cwqw5+Rgz87BkZONPTsHe3Y29pxsHNW3tWvs9uOfRAgwm5EmH+xGHyoMRsp0JoqFgUKpp0gaqNAbqTCYKNebwMcHX38rfoF+BAT5ExDkjz4gAJ2/P/oAfwyBARgDAjD5WfFx7bQ36fX4GHXVO/RNeh0GvXt3qDuckr2ZxWxKzqveuv47o6j6yzkmyJfBcUEMigtkUGwQA2IDa/06bS6VwJXWkb4Z3j0fGdyVTed8zNLdxXy/7QgpeWXoBJycGMr4/pEMigvSknOxjeziCrKLtDp7dnEFWa7buSW247ZSQUuIPgbdcQmuJpNBVyPB1U50WoIzYDJo/+Qmg851W7v2cd3PrUgnKesP1mb8zubsDTiknWCfYEbFjqJv8EAO5hZyICef5LwC0ooKsTkqQFTiY7ITaBH4+0p8TA6MBjs2RwXljnLK7eWUO8oRCCKtkURZo4j2i9Yu1mii/KKItkYTYg6hsMxBan4ZKXml1Uk9JU9L+EcKy8kvtVHH2wNo+THQ99jXr10HW4xEBJgZlhBMYpi1VbbqNyXn887vB1i6JQ2LrZQL43y4PNFCd4NNS/h5eciycpzlZa7rGrfLynCUllFZWoqjrAxZXo6uohxdI3KTQ+goNpopMfpSbPSlxOhLidFMicGXYqOZMpOFcrMvFWardrH4YbP4UWnxx2H1w2A0Hvd5qP5M6HUYXY/Z7E62phawNbWg+nPobzZoyTo2iMFxQQyMC2y14zVUAldaz96f4cMrIXEUTPkMqTeyPa2QZduO8P22dPZllRy3iNmoI8zPx3UxVd8OrXG76vGqIY42u5P8Uht5pZXklWojaqrua9c1bx+9rutL4YR0ZRisuzH478TstwOH3lZ7MgZMOjMWoxmryRezwYxZb67z2kfvg1M6SS9JJ70knbTiNApttU9ybdKZiPKLqk7w1YneqiX7cEs4OvQUldtdr1N7/ce+D8c+nldqq/XF18Xfh5GJIZycGMLIxBB6hfs3+xgCu8PJjzsymP/7AZIO5eHvY+CqEXFMPa0rcSEta5QmpYTKSq20U1JKbnYBFQWFVBYUYCsoxFFYiKOwCEdhIbK4CIqLobgYUVKMrqQYfWkx+tISDLbyBtdTbvKlzGylxMdCiclKsclCkcmXQqOFfKOFAoMveXozpT4WIqPC6NY1gt7dIunfK5pu4QFuP/6iPiqBK61r4wew+HYYNBkufr1W35Q9GUUcyC4htEZStrbRDjwpJUUVdorK7djszqMXh4OKWvePv20qSubUfS+QmLOcLL0ehyUG0/hnCT3pnDrLKk1RbCsmrSSN9OL06uvU4tTqBJ9TnnPcMgGmAILNwQT5BFVfat03BxHsE0yQWbsfaApEr9NTXukgJa+UtQfyWHsghzUHckkv0BJboK+REV2PJvR+0QEnLDsUlFXy6bpkFvx5kNT8MuJDLPzz9K5cMTyuTXfMNoasrMRRXIyzoEBL+gUFOPILtOuCfBwFBdq06seOXnDU/4sPQGexoPP3R+fnh97P7+htfz90fv7o/F2Pu25bTz0VvZ9fs16HSuBK61sxF1Y8CaNnwD8e8HQ0zWcrgd9fgD9eAp0ezrgXogfDd/dpp60bfgOcMwfM7h+5UqXcXq5tsbsSfEZpBvnl+eRX5JNXkUdBRQF55Xnkledhc9rqfA6BIMAngGCfYAJ9Agk2BxNiDiHYJxjh9COnwEhKto49RyQp2Xqkw4rFqJVatIQeysDYwOrRIgezS1jw50E+W59Mic3ByYkh3HhGImNOiqiuv9uddkoqSyitLKW4spiSyhJKKktwSAddfLsQZY0i0Cew3QxRdDgdZJVlkVacRmpx6tHrolSyc1IoyTmCb5kDvzJJYKWJXqYYuhmiiNOFEO70Q1dajrO4BGdRkfZFUVxcfVuW1T74rtvSpfh0S2xWnCqBK61PSlhyJ2x8Hy56EYZd7+mImkZK2Po5/PQQFKXBgCu0RB3o6uNtK4HlT8Lq17STSF/4AvQa5+GQJWX2Mi2hV+TVSvL5FflH75fnkVeRV5307bKOHY2AHh9w+GGz+SIdVoTTjy6WUHx0vhzKy0OntxEbKogO1qHTV1Qn6KpLuaPhkgWAWW8m0hpZfYmyRtW6H2mJbFGveiklNqeN0spSSu2llFWWUWgrJK0kjbTitOokXfWLx+6s/V508e1SvZ8ixi+GaL9oLAYL27K3kZSRxN95f+OUTnRCR+/g3gyLGMaQ8CEMjRhKmO/RBmSyshJnSYmW2IuKMHXv3uwjY1UCV9qGoxI+ngT7lsPkj1ue4OwVkL4FbMWQcBoYGj+qpUnSNsH3MyF5NUQNgvOegfhT6p43ZT0svgOydmpJfvzTYPWezoFSSooqi6qTeW55bnWCr7qdWZJDWlEOueW5lDkKkKISPSb8faz4m/ywGq1YjVb8jH5YjBb8jEcfq+txndCRWZrJkZIjpJekc6TkSPUlqywLyTFDLH0CtcRuiSTCGkGoOZRyRzll9rLqxFyVnMvsZdW3qx53Smc9rx5CzaHE+MVUJ+eaiTrKGoXZ0PCOyJLKEjZnbmZD5gY2ZG5ga9bW6i+ueP94hoQPqU7qCQEJbvm1oRK40nYqimHB+ZC9B67/DmKGNm45KaEgBVLWHb2kbwaHq0Rg8ode50KfC6HnWO08oC1VnAW/Pgob3gdLKJzzMAy+WiudNMRug9+fh5XPaqWU856B/pd12J7pdqcdg66Z9W2nE2xFYA6sc3Klo5LMskzSi9M5UnqkVnKvSvaFtkKMOiMWowWLwYKvwReLwYLFePxtX4Nvrcd9jb74G/2rdxL7Gtxwxqtj4t+Zu5MNGVpC35i5sfrAtRBzCEPDhzIkfAgXdr+QEHPzTtSiErjStooyYP45UFkGN/4EIXXU/mylkL7paLJOXgfFWgtbDGaIHgqxwyF2hLblves7+HsplGSB3ge6n60l897nN71Pud0Ga9+E3+ZCZSmcPB3OnFFvkqlXxg5YcgekJkGv8XDB80dLLp1dfjJs+gg2fQD5h6H7GBhxk/arrIk7gR1OR4t3HLcVp3RysOAgSZlJbMzYyIbMDaQWp7L00qXE+TfvIDGVwJW2l7Ub5o/Vygs3/AgVBVr5IXmtlrAztkFV/TE4UUvUsSMgboR2Agl9Hb1KnA5IXgM7v4Wd30DBYRA6SDgdTroI+lwAgbENx7XnZ1g2C3L2QI+xMP4pCOvZ/NfpdMDq1+HXx7WYxz4KQ6fCCQ4z75Aqy2HXt9qopP0rAAmJZ2plqa2fQVE6BMTC8OthyHXgH+HhgNtGRkkG4Zbm93NXCVzxjEN/wXsTQTqhqrmU0aqVVWJHQNxIiBkOfl2a/txSwpEtWiLf+a1WkwZty/2kC+GkCbUTc84+WPZv2PMDhHTXErc7d0Lm7odv7oYDK6HrKG1Hbmh39z1/TU6H9uvGXt64a71RS6JdTgK9m4f6SamVujZ+AFs/hfICCIyHIVdrw0qDXacGdFTC39/D+vlactcZtL/RiBu1L+AOWn5yB5XAFc/Z9yvsWAJRA7Wk3RpJBCB7L+xyJfNU1+csrLeWzB02WP2GVpo5c4ZWMjG0Qr8NKWHDe/Djg9o6z34ATrntxK+3oggK07XRL7Wu06EwDUqzta3bqoTsrLst8AkZLRA12PUFOhxihkFgXPOSZ0mOlrA3fggZW7WyVt8JMOQa6Dq64V8g2Xth/TtaeaW8ALr0geE3wqCrml7G6gRUAlc6l4JUrV6+cwkc/AOkAwZfA2Meapuf7YVp2rjxv5dC9BD4x4NaAq5KyLWu07WdfMfyCYSAKG3Iol84GH3B4AtGcxOuzdpylWWQtlErYaUmuXYOV2jrsYa7kvlQ7ddQ9BDwDar7dTkd2hfyxvdh11LtiyR6iJa0+18GTT3dnq0Utn8J697W4jNaYeAVWjKPGti05+rAVAJXOq/SXG0Lt+qnfFuRErZ/BUvv17agqwg9+EdqiTkgCvyj6742WVsvNrtN2weRmnT0kr376PSwXtrWedXF5AdbPoFNH2u/DnxDYNAkbcROZH/3xJSaBOvegW2fa780Ykdq5ZW+F2tfRt7IVqrtdC/J1t6nZg6DVQlcUTylNFfb8vXroiVta5cmj8JoE2X52lZw6npI3aDFXJJ5dLrQaTt9h1wNvc5rnRIUaO/X5o9h3XzI3ad9WfS7WCut6AzaF6DOoJVoqm/rXbf1NW5XPa7TbhvMWgKt99p1W2eov6Rkt2lfxlVJuaTqdpbr8Rr3S7K1EU5V7ljf7J3lKoEritI0VePyU9dryajPhdovg7bidMKB37Tyyr5ftX0KTgfQyjlL6I5P7kIHpTlavb4uOqP2xWwNq+e6i3YgWjOPXagvgbevzjOKorQfQkBQnHbxBJ1OG+vf/ezajzud2j4Np8N1bXfddmrXTvvR6VXzOCq1mr/dppVn7BV1Xzsq6p7mtGsHeh2blC1h2n1zoEdG0agEriiKd9HpAF3dxwl0Mp3wSANFUZSOQSVwRVEUL6USuKIoipdSCVxRFMVLqQSuKIripVQCVxRF8VIqgSuKongplcAVRVG8VJseSi+EyAIONXPxMCD7hHN5joqvZVR8LaPia5n2Hl+ClPK4pvltmsBbQgixvq5eAO2Fiq9lVHwto+JrmfYeX31UCUVRFMVLqQSuKIripbwpgb/p6QBOQMXXMiq+llHxtUx7j69OXlMDVxRFUWrzpi1wRVEUpQaVwBVFUbxUu0vgQojxQoi/hRB7hRCz6pjuI4RY5Jq+RgjRtQ1jixNCLBdC7BBCbBdC3F3HPGcJIQqEEJtcl4faKj7X+g8KIba61n3c+euE5iXX+7dFCDG0DWPrXeN92SSEKBRC/OuYedr0/RNCvCOEyBRCbKvxWIgQ4ichxB7XdZ2nWhdCTHXNs0cIMbUN45snhNjl+vt9JYQIqmfZBj8LrRjfI0KI1Bp/w/PrWbbB//VWjG9RjdgOCiE21bNsq79/LSalbDcXQA/sA7oBJmAz0PeYeW4D3nDdngQsasP4ooChrtv+wO464jsL+NaD7+FBIKyB6ecD3wMCOAVY48G/9RG0AxQ89v4Bo4GhwLYajz0DzHLdngXMrWO5EGC/6zrYdTu4jeI7FzC4bs+tK77GfBZaMb5HgP9rxN+/wf/11orvmOnPAQ956v1r6aW9bYGPBPZKKfdLKW3AJ8DEY+aZCCx03f4cGCNE25yMTkqZLqXc4LpdBOwEYtpi3W40EXhPalYDQUKINjxTbbUxwD4pZXOPzHULKeVKIPeYh2t+xhYCF9ex6DjgJyllrpQyD/gJGN8W8Ukpf5RS2l13VwOx7l5vY9Xz/jVGY/7XW6yh+Fx540rgY3evt620twQeAyTXuJ/C8Qmyeh7Xh7gACG2T6GpwlW6GAGvqmHyqEGKzEOJ7IUS/to0MCfwohEgSQkyrY3pj3uO2MIn6/3E8+f4BREgp0123jwARdczTXt7HG9B+UdXlRJ+F1nSHq8TzTj0lqPbw/o0CMqSUe+qZ7sn3r1HaWwL3CkIIP+AL4F9SysJjJm9AKwsMAl4Gvm7j8M6QUg4FzgNuF0KMbuP1n5AQwgRMAD6rY7Kn379apPZbul2OtRVCPADYgQ/rmcVTn4XXge7AYCAdrUzRHk2m4a3vdv+/1N4SeCoQV+N+rOuxOucRQhiAQCCnTaLT1mlES94fSim/PHa6lLJQSlnsur0UMAohwtoqPillqus6E/gK7adqTY15j1vbecAGKWXGsRM8/f65ZFSVlVzXmXXM49H3UQhxPXAhcLXrS+Y4jfgstAopZYaU0iGldAJv1bNeT79/BuBSYFF983jq/WuK9pbA1wE9hRCJrq20ScCSY+ZZAlTt8b8c+LW+D7C7uWpm84GdUsrn65knsqomL4QYifYet8kXjBDCKoTwr7qNtrNr2zGzLQGuc41GOQUoqFEuaCv1bvl48v2roeZnbCqwuI55fgDOFUIEu0oE57oea3VCiPHADGCClLK0nnka81lorfhq7lO5pJ71NuZ/vTWdA+ySUqbUNdGT71+TeHov6rEXtFESu9H2UD/geuxRtA8rgBntp/deYC3QrQ1jOwPt5/QWYJPrcj4wHZjumucOYDvaXvXVwGltGF8313o3u2Koev9qxieAV13v71ZgeBv/fa1oCTmwxmMee//QvkjSgUq0OuyNaPtUfgH2AD8DIa55hwNv11j2BtfncC/wzzaMby9a/bjqM1g1KisaWNrQZ6GN4nvf9dnagpaUo46Nz3X/uP/1tojP9fiCqs9cjXnb/P1r6UUdSq8oiuKl2lsJRVEURWkklcAVRVG8lErgiqIoXkolcEVRFC+lEriiKIqXUglcURTFS6kEriiK4qX+H0cTK2ippGZ5AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(list(range(num_epochs)), train_l1, label='train_mse_0.03')\n",
    "plt.plot(list(range(num_epochs)), train_l2, label='train_mse_0.01')\n",
    "plt.plot(list(range(num_epochs)), train_l3, label='train_mse_0.005')\n",
    "plt.plot(list(range(num_epochs)), train_l4, label='train_mse_0.001')\n",
    "plt.legend(loc = 1)\n",
    "\n",
    "# 学习率较大的模型收敛速度较快，但效果却不一定最好\n",
    "# 而学习率非常小的模型不仅收敛速度较慢，并且效果也不尽如人意。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在实际生产生活中，训练一个模型已经是非常耗费时间的了，为了一个超参数同时训练多组模型并不划算。而要通过学习率数值调整来让模型“又好又快”的收敛，就需要采用一种名为学习率调度的优化算法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二、学习率调度基本概念与手动实现方法\n",
    "\n",
    "### 1.学习率调度基本概念\n",
    "\n",
    "先让学习率取得较大数值，从而能够让模型在最开始能够以较快的速度收敛；然后在经过一段时间迭代之后，将学习率调小，从而能够让收敛过程穿过损失函数的“隘口”，抵达更小的解。\n",
    "\n",
    "### 2.手动实现学习率调度\n",
    "\n",
    "- 利用input函数输入epoch,lr\n",
    "\n",
    "#### 2.1 本地实现方法\n",
    "\n",
    "我们可以借助Python中的input函数来手动实现学习率伴随迭代epoch次数的动态调整，具体操作方法如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 实例化模型\n",
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 实例化模型  \n",
    "tanh_model = net_class2(act_fun=torch.tanh, in_features=5, BN_model='pre')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 更新fit函数  \n",
    "&emsp;&emsp;接下来我们对fit函数进行更新，在原有fit函数基础上加上每一轮迭代后模型评估结果的记录功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_rec(net, \n",
    "            criterion, \n",
    "            optimizer, \n",
    "            train_data,\n",
    "            test_data,\n",
    "            epochs = 3, \n",
    "            cla = False, \n",
    "            eva = mse_cal):\n",
    "    \"\"\"模型训练函数（记录每一次遍历后模型评估指标）\n",
    "    \n",
    "    :param net：待训练的模型 \n",
    "    :param criterion: 损失函数\n",
    "    :param optimizer：优化算法\n",
    "    :param train_data：训练数据\n",
    "    :param test_data: 测试数据 \n",
    "    :param epochs: 遍历数据次数\n",
    "    :param cla: 是否是分类问题\n",
    "    :param eva: 模型评估方法\n",
    "    :return：模型评估结果\n",
    "    \"\"\"\n",
    "    train_l = []\n",
    "    test_l = []\n",
    "    \n",
    "    for epoch  in range(epochs):\n",
    "        net.train()\n",
    "        for X, y in train_data:\n",
    "            if cla == True:\n",
    "                y = y.flatten().long()          # 如果是分类问题，需要对y进行整数转化\n",
    "            yhat = net.forward(X)\n",
    "            loss = criterion(yhat, y)\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "        net.eval()\n",
    "        train_l.append(eva(train_data, net).detach())\n",
    "        test_l.append(eva(test_data, net).detach())\n",
    "        # 返回每epoch的训练误差和测试误差\n",
    "    return train_l, test_l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([tensor(37.9793), tensor(32.4801), tensor(30.2007)],\n",
       " [tensor(37.7321), tensor(33.3702), tensor(32.1592)])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#——————————————————————————测试函数性能——————————————————————————\n",
    "fit_rec(net = tanh_model, \n",
    "        criterion = nn.MSELoss(), \n",
    "        optimizer = optim.SGD(tanh_model.parameters(), lr = 0.03), \n",
    "        train_data = train_loader,\n",
    "        test_data = test_loader,\n",
    "        epochs = 3, \n",
    "        cla = False, \n",
    "        eva = mse_cal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 手动动态调整过程\n",
    "\n",
    "&emsp;&emsp;接下来，我们尝试借助Python中的input函数功能，来执行手动动态调整模型训练过程中的学习率。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 实例化模型  \n",
    "tanh_model = net_class2(act_fun=torch.tanh, in_features=5, BN_model='pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#———————————————————————————第一个模型————————————————————————————\n",
    "# 创建用于保存记录结果的空列表容器\n",
    "train_mse = []\n",
    "test_mse = []\n",
    "\n",
    "# 创建可以捕捉手动输入数据的模型训练流程\n",
    "while input(\"Do you want to continue the iteration? [y/n]\") == \"y\":    # 询问是否继续迭代\n",
    "    epochs = int(input(\"Number of epochs:\"))                           # 下一轮迭代遍历几次数据\n",
    "    lr = float(input(\"Update learning rate：\"))                        # 设置下一轮迭代的学习率\n",
    "    train_l, test_l = fit_rec(net = tanh_model, \n",
    "                              criterion = nn.MSELoss(), \n",
    "                              optimizer = optim.SGD(tanh_model.parameters(), lr = lr), \n",
    "                              train_data = train_loader,\n",
    "                              test_data = test_loader,\n",
    "                              epochs = epochs, \n",
    "                              cla = False, \n",
    "                              eva = mse_cal)\n",
    "    train_mse.extend(train_l)\n",
    "    test_mse.extend(test_l)\n",
    "    \n",
    "# `A.extend(B)` 用于在列表A末尾一次性追加另一个序列B中的多个值\n",
    "\n",
    "# 每次输入 是否迭代 -> num_epochs -> lr\n",
    "#  y  -> 30 -> 0.03 -> y -> 30 -> 0.01 -> n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f40326c040>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f403b41f60>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzS0lEQVR4nO3deXhb5Zn38e8ty5a871tsJ3EW4myQhCQEAi2EhmErhDJN4S0M25R2Ci1tGQZK2xna6TrTZUoH6EDZWiiFshRK0xYIYYcEO5udhThks53EdhKvsi1Z0vP+IcmxE3mXLR/7/lyXL+yjI+k5Qfn5yX2eRYwxKKWUsh5btBuglFJqaDTAlVLKojTAlVLKojTAlVLKojTAlVLKouyj+WZZWVlm6tSpo/mWSilleWVlZUeMMdknHh/VAJ86dSqlpaWj+ZZKKWV5IrI/3HEtoSillEVpgCullEVpgCullEWNag1cKTV+dHZ2Ul1dTUdHR7SbMm44nU4KCwuJjY0d0Pka4EqpIamuriY5OZmpU6ciItFujuUZYzh69CjV1dUUFxcP6DlaQlFKDUlHRweZmZka3hEiImRmZg7qXzQa4EqpIdPwjqzB/nlaIsBf31nL/W/sjnYzlFJqTLFEgL9TeZRfrd2Nrl2ulFLHWSLAizLiae/0caTVE+2mKKXGkMbGRu6///5BP+/iiy+msbEx8g0aZdYI8PQEAKoa2qLcEqXUWNJbgHu93j6ft2bNGtLS0kaoVaPHEsMIJ2cGA/xYG4smp0e5NUqpE333z9vYfrA5oq85Z1IK//HpuX2ec9ddd/Hxxx+zYMECYmNjcTqdpKens3PnTnbt2sWqVauoqqqio6OD2267jZtvvhk4vi5Ta2srF110EWeffTbvvfceBQUFvPjii8THx4d9v3PPPZeFCxfy9ttv43K5+O1vf8uPfvQjysvL+dznPsf3v/99XC4Xq1evprq6Gp/Px3e+8x0+97nPUVZWxje+8Q1aW1vJysriscceIz8/f1h/RpYI8ML0wB9mdUN7lFuilBpLfvzjH1NRUcHmzZt54403uOSSS6ioqOgaR/3II4+QkZFBe3s7S5Ys4corryQzM7PHa1RWVvLUU0/x0EMPsXr1ap577jmuueaaXt8zLi6O0tJSfvnLX3L55ZdTVlZGRkYG06dP5+tf/zpvvPEGkyZN4i9/+QsATU1NdHZ28pWvfIUXX3yR7Oxsnn76ab71rW/xyCOPDOv6LRHgCXF2spLiqDqmJRSlxqL+esqjZenSpT0mwdx777288MILAFRVVVFZWXlSgBcXF7NgwQIATj/9dPbt29fne1x22WUAzJ8/n7lz53b1oqdNm0ZVVRXz58/n9ttv58477+TSSy/lnHPOoaKigoqKClauXAmAz+cbdu8bLBLgAIXpCRzQAFdK9SExMbHr+zfeeIPXXnuN999/n4SEBM4999ywk2QcDkfX9zExMbS39/0v/dD5Nputx3NtNhter5dTTjmFjRs3smbNGr797W9z/vnnc8UVVzB37lzef//94V5iD5a4iQlQlJGgNzGVUj0kJyfT0tIS9rGmpibS09NJSEhg586dfPDBB6PSpoMHD5KQkMA111zDHXfcwcaNG5k1axb19fVdAd7Z2cm2bduG/V6W6YFPzohnTfkhvD4/9hjL/N5RSo2gzMxMli9fzrx584iPjyc3N7frsQsvvJBf//rXzJ49m1mzZrFs2bJRaVN5eTl33HEHNpuN2NhYHnjgAeLi4nj22Wf56le/SlNTE16vl6997WvMnTu80pOM5uSYxYsXm6HuyPOHDQe46/ly3v638yjKSIhwy5RSg7Vjxw5mz54d7WaMO+H+XEWkzBiz+MRzLdOVDYW2llGUUirAMiWU0GSe6mPtMD3KjVFKjWu33HIL7777bo9jt912GzfccEOUWhSeZQI8P82JTdCRKEqNIcaYcbki4X333ReV9x1sSdsyJZTYGBuT0uK1hKLUGOF0Ojl69KguMhchoQ0dnE7ngJ9jmR44BMooOplHqbGhsLCQ6upq6uvro92UcSO0pdpAWSvAM+JZ95F+WJQaC2JjYwe89ZcaGZYpoUCgB17f4qbd44t2U5RSKuqsFeDBoYTVWgdXSilrBrjeyFRKqQEEuIg4RWSDiGwRkW0i8t3g8cdEZK+IbA5+LRjpxhZlBJaVrTqmy8oqpdRAbmK6gRXGmFYRiQXeEZG/Bh+7wxjz7Mg1r6fsJAfOWJuORFFKKQYQ4CYwyLM1+GNs8CsqAz9FhMJ0XZVQKaVggDVwEYkRkc1AHfCqMWZ98KEfiMhWEfmFiDh6ee7NIlIqIqWRGC9alB7PAS2hKKXUwALcGOMzxiwACoGlIjIP+CZQAiwBMoA7e3nug8aYxcaYxdnZ2cNu8OSMBKqPtensL6XUhDeoUSjGmEZgHXChMeaQCXADjwJLR6B9JynKSKDF7aWpvXM03k4ppcasgYxCyRaRtOD38cBKYKeI5AePCbAKqBi5Zh5XmB7aoV7LKEqpiW0go1DygcdFJIZA4D9jjHlZRF4XkWxAgM3Al0aumcd1DSVsaGN+YepovKVSSo1JAxmFshVYGOb4ihFpUT9Ck3l0WVml1ERnqZmYACnOWFLjY3UsuFJqwrNcgENgJEpVg9bAlVITmyUDvCgjnmrtgSulJjhrBnh6AtUN7fj9OhZcKTVxWTLACzMS8Pj81LZ0RLspSikVNZYM8KJ0XZVQKaUsGeCTQ+uCax1cKTWBWTLAC9LjEdGNHZRSE5slA9xhjyE32aklFKXUhGbJAIfAUEItoSilJjLrBrhu7KCUmuCsG+AZCRxu7sDt9UW7KUopFRWWDfC5k1IwBkr3NUS7KUopFRWWDfBzZmbjjLXxyrbD0W6KUkpFhWUDPD4uhnNmZvPK9lrdXk0pNSFZNsABLpiTy6GmDipqmqPdFKWUGnWWDvDzZ+diE3hlu5ZRlFITj6UDPCMxjqXFGbyyrTbaTVFKqVFn6QAHuGBOHh/VtrDviCvaTVFKqVFl+QBfOScX0DKKUmrisXyAF2UkMCc/RcsoSqkJx/IBDnDB3FzKDjRQ3+KOdlOUUmrUjI8An5OHMbB2h/bClVITx7gI8Nn5yRSmx/PKdg1wpdTEMS4CXET4h7l5vLP7CK1ub7Sbo5RSo2JcBDgEZmV6vH7e2lUf7aYopdSoGDcBfvqUdDIS43RxK6XUhNFvgIuIU0Q2iMgWEdkmIt8NHi8WkfUisltEnhaRuJFvbu/sMTbOL8lh7c46On3+aDZFKaVGxUB64G5ghTHmNGABcKGILAN+AvzCGDMDaABuGrFWDtAFc/No6fCyfs+xaDdFKaVGXL8BbgJagz/GBr8MsAJ4Nnj8cWDVSDRwMJbPyCTObuP1nXXRbopSSo24AdXARSRGRDYDdcCrwMdAozEmNOSjGijo5bk3i0ipiJTW14/sDcaEODtnTstk3Uca4Eqp8W9AAW6M8RljFgCFwFKgZKBvYIx50Biz2BizODs7e2itHIQVJTnsPeJiry5upZQa5wY1CsUY0wisA84E0kTEHnyoEKiJbNOGZkVJDoCWUZRS495ARqFki0ha8Pt4YCWwg0CQ/2PwtOuAF0eojYNSlJHAjJwk1mmAK6XGuYH0wPOBdSKyFfgQeNUY8zJwJ/ANEdkNZAIPj1wzB2dFSQ7r9x7VWZlKqXHN3t8JxpitwMIwx/cQqIePOefNyuHBt/bwTuURLpyXF+3mKKXUiBg3MzG7Wzw1nWSnXcsoSqlxbVwGeGyMjU/MzGbdR3UYY6LdHKWUGhHjMsABzivJoa7FzbaDzdFuilJKjYhxG+DnzspGRIcTKqXGr3Eb4FlJDk4tTNMAV0qNW+M2wAFWzMphS3UjR1t1r0yl1PgzvgO8JAdj4I2PdJMHpdT4M64DfO6kFLKTHbyui1v16dqH13P/G7uj3Qyl1CD1O5HHymw24bxZ2fy14jCdPj+xMeP699WQldc0kRofG+1mKKUGadwn2oqSHFo6vJTtb4h2U8asNrdPlx1QyoLGfYCfNSMLgA17dZeecDxePx6fH5cGuFKWM+4DPMUZy7SsRMprmqLdlDGpzRMI7pYODXClrGbcBzjA/MJUyqs1wMNxeXwAWkJRyoImRoAXpHK4uYO6lo5oN2XMCZVOtISilPVMmAAHqNAyyklCwd3q9urCX0pZzIQI8LkFqYhAebUubHUilztQQun0Gdxef5Rbo5QajAkR4EkOe/BGZmNEXm9XbQs/f3XXuOixujzHSydaRlHKWiZEgEOgjBKpkSh/2lTDvWsr2X+0LSKvF01t3QJcb2QqZS0TJ8AL06htdlPXPPwbmXUtgcWxNlVZf3JQa7CEAjqUUCmrmTgBHryRGYleeCjAN+5vHPZrRVubW0soSlnVhAnwuZNSEIGtERgPHurFj4ceeGgcOGgJRSmrmTABnuiwMz07KSJDCUM98B2HWnrUkK2oe69bA1wpa5kwAQ5wakEqW4cZ4B6vn2MuD6cVpuLzG8vP8GzzeLFJ4HsNcKWsZUIF+LyCVOpb3NQO40ZmfXB3nwvm5gGw8UBjJJoWNS63j6wkBwCtehNTKUuZUAF+amHgRuZw6uCh+ndJXjLFWYlsOmDtOrjL7SUryYGI9sCVspoJFeBzJqVgk+GNRAnVv3OSnSwsSmPjgUZLT+hxebwkOe0kxdk1wJWymAkV4AlxdmbkJFFe3Tjk1wj1wHNTHCycks6RVjfVDe0RauHoc7l9JMbFkOS0awlFKYvpN8BFpEhE1onIdhHZJiK3BY/fIyI1IrI5+HXxyDd3+OYVpFJe0zzkXnNdixubQGaSg4VFaQBstHAZxeXxkuCwk+TQHrhSVjOQHrgXuN0YMwdYBtwiInOCj/3CGLMg+LVmxFoZQacWpHKk1c3hId7IrGt2k5nkIMYmlOQlEx8bwyYL38hsc/tIirOTqAGulOX0G+DGmEPGmI3B71uAHUDBSDdspMwP3sgc6vC/2pYOclMCozbsMTZOLUxlU1VjpJo36lxuLwmOGJKdGuBKWc2gauAiMhVYCKwPHrpVRLaKyCMikt7Lc24WkVIRKa2vrx9eayNgTn7qsG5k1jW7yUl2dv28aEo62w820dHp6+NZY5MxBpfHS2JcoISiU+mVspYBB7iIJAHPAV8zxjQDDwDTgQXAIeBn4Z5njHnQGLPYGLM4Ozt7+C0epvi4GE7JTR56gLe4yUl2dP28sCiNTp9h20HrTehxe/34TWCWaqJDb2IqZTUDCnARiSUQ3k8aY54HMMbUGmN8xhg/8BCwdOSaGVnzClKpqGka9I1Mr8/PUZebnJTjPfCFkwP/8LDiwlahkkmiI4Ykh50W7YErZSkDGYUiwMPADmPMz7sdz+922hVAReSbNzJOLUzlSKuHQ02Du5F5pNWDMfTogWcnOyjKiLfkwlZtwaVkE+LsJDsDJRQrj2lXaqKxD+Cc5cC1QLmIbA4euxu4WkQWAAbYB3xxBNo3IuZ1W1p2Ulr8gJ9X2zUG3Nnj+MKidDbsPRa5Bo6S0G48SY4YEh12/AbaO30kxA3kY6GUirZ+/6YaY94BJMxDlhg2GM6c/BRibMLW6kb+IbimyUAcn4Xp6HF80eQ0XtpykENN7eSnDvwXQrSFblomBG9iQmA9FA1wpaxhQs3EDHHGxjBvUgof7h1c2aOuJdADz0npGeBWrYOH1gJPDA4jBF0PRSkrmZABDrBsWiabqxoHNfyvttmNCF2r94XMzk/BYbdZbmErV9dNTDuJcRrgSlnNhA3wM6Zl4PH5BzUNvr6lg8zEOGJjev6xxdltzC+w3oSergCPs5PkPF5CUUpZw4QN8MVTM7AJfLBn4Dcf65rdZCc7wz62aEo65TVNeLz+SDVxxLV1lVC61cC1B66UZUzYAE9xxjJ3Uirr9xwd8HO6T6M/0elT0vF4/byzO/qzTQeqtesmZowGuFIWNGEDHGDZtAw2DaIOHphGHz7Az5uVQ36qk1+/sSeSTRxRbR4vMTbBYbcdL6FogCtlGRM6wM8ozsTj9bN5ALVrn99wpLXnOijdxdlt3PyJaWzYd4zSfdYYEx5aC1xEtAeulAVN6ABfUpyBCHwwgDLK0VY3fkOvJRSAq5ZMJiMxjvvf+DiSzRwxLreXxGBwO+w27DbRm5hKWciEDvDU+Fjm5KewfgA3MkOTeHq7iQmBhbJuXD6V13fWsf1gc8TaOVLaPD4S4mIAAr1wXVJWKUuZ0AEOgfHgGw804Pb2XQcPTeLpqwcOcO2ZU0ly2HngzbHfC3d5vF2lE0B35VHKYiZ8gJ9RnIHb62dLVd/LwdY2B6fRp/TeA4dAr/7zyybzl60H2XfEFbF2jgSXu+e0+SRdUlYpS5nwAb50gHXwumCAZyf13QMHuOnsYuwxNv7vrbHdC3e5fSQ6Yrp+1h64UtYy4QM8LSGOkrwU1u/tJ8BbOshIjCPO3v8fWU6yk9WLC3murIbDg1yydjS5PMdvYgIkOXVXHqWsZMIHOATGg5ftb+hzFmVtH2PAw/niJ6bjM4bfvD12x4W73D2Xjk3UTR2UshQNcALjwTs6/Wytbuz1nPqWDrIHEeBFGQlcdtokfr/hAA0uTwRaGXltHi+JccdLKMlaA1fKUjTACdzIhL7r4HUt7pM2cujPTWcX0+bx8dqO2mG1byT4/YY2j69nCUU3NlbKUjTAgfTEOEryklnfy646fr+hvmVwJRSAkrxk7DZhzxgcjdLWeXwt8JBEhx2Xx4fPr9uqKWUFGuBBy6ZlUrqvgU7fyXXwY20evH4z6AC3x9iYnJnA3voxGODd1gIPCW3qENpqTSk1tmmAB51RnEF7p4+t1SePB+9tL8yBmJaVyN4x2ANv7bYWeEhoUo+WUZSyBg3woKV91MG79sLsZxZmOMVZiew96sI/xsoSobXAE+J6llBAN3VQyio0wIMykxzMyk3mzY9OXs+7PjQLs491UHpTnJWEx+vnYFP7sNsYSaFedtIJ48ABHUqolEVogHezamEBG/Ydo7K2pcfxUAllMMMIQ6ZlJwKMuTJKqM6d0L0GriUUpSxFA7yb1YsLiYux8eT6Az2O17W4SY2PxRkb08szezcta4wGuDs4CkVLKEpZlgZ4N5lJDi6an8dzZdW0dRuJUdfHVmr9yU52kBgXw54xNhLFFWYUim7qoJS1aICf4JplU2hxe3lp88GuY4Fp9IOvf0Ngne3i7LE3EsUV2tA47uRhhBrgSlmDBvgJFk9JZ1Zuco8yylAm8XRXnJU05gI8NA48waElFKWsSgP8BCLCNcsmU17TxJaqRowx1LV09LsOeF+KsxKpbmjrd9OI0dTq8RJntxEbc/wjEBtjw2G30aoTeZSyhH4DXESKRGSdiGwXkW0iclvweIaIvCoilcH/po98c0fHqoUFJMTF8MQH+2lo66TTN/hZmN1Ny0rEb+DA0bYItnJ42oIbGp9IN3VQyjoG0gP3ArcbY+YAy4BbRGQOcBew1hgzE1gb/HlcSHbGcvmCAv689WDXkMKhTOIJKQ6ORBlLa6KcuBZ4iO6LqZR19BvgxphDxpiNwe9bgB1AAXA58HjwtMeBVSPUxqi4ZtlkOjr9XXtbDmUafcjUMTiU0OX29riBGaIrEiplHYOqgYvIVGAhsB7INcYcCj50GMjt5Tk3i0ipiJTW1588y3GsmjsplYWT03gjODNzOCWU1PhYspLixtSiVm0eX48bmCGJDjstWkJRyhIGHOAikgQ8B3zNGNPc/TFjjAHCLvZhjHnQGLPYGLM4Ozt7WI0dbdecMaXr+6EOIwwpHmOLWrncPXekD0nWfTGVsowBBbiIxBII7yeNMc8HD9eKSH7w8XygbmSaGD2XnJpPWkIsyU478WFu+A3GtKyksVUDd/t6LGQVovtiKmUdAxmFIsDDwA5jzM+7PfQScF3w++uAFyPfvOhyxsZw2/kzufTUScN+reLsRI60umnu6IxAy4bP5QlfA0/UHrhSlnHy3+CTLQeuBcpFZHPw2N3Aj4FnROQmYD+wekRaGGU3LC+OyOuERqLsO+Li1MK0iLzmcLjc4UehJGsNXCnL6DfAjTHvANLLw+dHtjnjV/dFrcZEgPdyEzPJYcft9dPp8/eY5KOUGnv0b+gomZyZgAh8PAZGonT6/Hi8/l5LKKBLyiplBRrgo8Rhj6EwPX5MjERpCy0l28tEHkDLKEpZgAb4KAosatUa7WZ0beYQbip916YOE2A9FI/XP+a2ulNqMDTAR9G0rET21rsIDJuPnrYwu/GETJQVCY0xnPvf6/jt+/ui3RSlhkwDfBQVZyXi8vioD26SHC2twRJKUribmBNkX8wjrR4ONnWw/VBz/ycrNUZpgI+isbKoVdda4GFuYk6UfTEPNgY2ma5tju4vU6WGQwN8FI2VDY5Du/GEm0o/UUooNV0B3hHllig1dBrgo2hSajxxdlv0A7yrB957CWW8z8asadAAV9anAT6KbDahODMx7AbHNY3tNLZ5RqUdXaNQwvXA4yZIgAd74A1tnXR0jp2dkpQaDA3wURZYlbDnUMK/VRxmxU/f4Kt/2DwqbQi3I31IjE1IiIuZMCUUIOo3lZUaKg3wUVacnciBY214fX4AHnt3L//yZBl2m/B2ZT2Hmtr7eYXhcwVHocTHhl9hMWkCLGh1sLEdhz3w8T+sZRRlURrgo6w4K5FOn+HAsTZ+8Jft3PPn7VwwJ5fnvnwWxsCfNh0c8Ta0ebzEx8YQYwu/xM1E2FatprGd04Jr0mgdXFmVBvgoCy1q9aUnynjo7b1cd+YU7v/86ZTkpXD6lHSe31g94hN9Wt2+sOWTkPHeA3e5vTS2dbJwShoAh5s0wJU1aYCPstBY8F21rdx9cQn3XDa3qyf8mUUFVNa1UlEzspNL2jxeEsNM4gkZiZ3pjTE8+NbHvLLtcFf5KFpCY8Dn5KfgsNuo0xq4sqiBrAeuIigjMY4vfXI6pxWmctH8/B6PXTp/Et99aTvPbaxmfmHqiLUhsBtP3z3wA662iL7n7rpWfrhmJwB5KU6uWlrEVUsmk5c6vK3qhiJ0A7MgLZ68VKf2wJVlaQ98lIkId11UclJ4A6QmxPKpOTm8tOUgnSPYSw3sh9lPDzzCJZTKusDImzsvLOGUvGT+57VKlv/kdb74u1IOHI3sL4v+dAV4ejy5yU6tgSvL0gAfYz6zsJBjLg9vflQ/Yu/R5vH23QMfgX0xd9W2IALXnzWV3964lDfvOJcvnDONtyuP8MM1OyL6Xv052NiO3SbkJDvJTdUAV9alAT7GfHJWNhmJcTy/qXrE3sPl8fVfA3d7I3oztbKulaL0hK7NoadkJnLXRSWsWljAW5X1ozqZpqahnbxUJzE2ITfZQW2zO+orRCo1FBrgY0xsjI3LTpvEa9vraGobmQ2QXe7wGxqHJDrsdPoMbm/kyjiVtS2ckpt00vGVs3Np8/j4YM/RiL1Xfw42djApLR6AvFQn7Z0+msf5xCU1PmmAj0FXLirE4/PzcvnIjAnvbUPjkGRnZFck7PT52XvExYyc5JMeO3N6JvGxMby2ozYi7zUQNY3tFAYDPCclcBO1TssoyoI0wMegeQUpzMxJ4vmNNRF/bWPMgEooELn1UPYfddHpM2F74M7YGD5xShavba8blTKG1+fncPPxHnhusgPQ2ZjKmjTAxyAR4TOLCinb38C+CK9c6Pb68flNnzcxQ73zSO2Luas2MAJlZpgeOMCnZudyuLmDbQdHfnOF2hY3Pr+hIP14CQV0XXBlTRrgY9SqhZMQgd+8s4fNVY1sP9jM7rpWqo61dW2JNhRtwbXAw+2HGRLpTR0qa1sRgRk5J/fAAVaU5CACr24f+TJKaBJPVw88JRTg2gNX1qMTecao/NR4zpmZzRMfHOCJDw70eCzFaec31y1haXHGoF+3r5UIQwazJvjmqkbuW7ebX129EGcvi2PtqmvpMQLlRJlJDk6fnM5rO2r5+spT+n3P4QitA14QDHBnbAyp8bEa4MqSNMDHsHuvWkBFTTMenw+P14/b68fd6ef/3vqYax9ez/2fX8T5s3MH9Zp9rQUekjiIGvj963bz6vZaNh1o5MzpmWHPqaxtYWYvve+Q82fn8pO/7eRQUzv5qfH9vu9Q1XT1wI/PAM1NcehsTGVJWkIZw9IS4jh7ZhYrSnK5cF4+ly8oYPWSIv74pbOYlZfMzb8r4/mNgxsvHlpKNtxuPCHJAwzw+hY3r++sA2D93vDDAEMjUGbmhq9/h6yckwPAazvq+jxvuGoa28lIjOtxDyA3xUmtroeiLEgD3IIyEuP4/ReWcUZxBt94ZguPvLN3wM8NlVDC7YcZ0lVC6ecm5p821eD1G7KSHGzYeyzsOaERKP31wKdnJzE1M4HXRrgOfrCxvUfvGwIBrsMIlRX1G+Ai8oiI1IlIRbdj94hIjYhsDn5dPLLNVCdKcth59IYlXDg3j++9vJ2fvfLRgIbhhW6A9jUKJT42Bpv03QM3xvBMaRULJ6fx6dPy2XigAU+YiT+VwREop/TTAxcRPjU7l/c/PjqiS9nWNLR31b9D8lKc1AVHpyhlJQPpgT8GXBjm+C+MMQuCX2si2yw1EA57DPd9fhFXLSniV6/v5r51u/t9TqiE0tc4cBEhsZ8FrTZXNVJZ18rqxUWcUZxBR6ef8pqmk87b1c8IlO4+NScXj8/P27tGZh0YY0ywB94zwHNTHPj8hqMuLaMoa+k3wI0xbwHh/32soi7GJvzwivlcsbCAn76yi2dKq/o8fyA3MSFQB++rhPJMaTXOWBuXnprPkqmB0TDh6uC76looTI/vdQRKd4unpJMaH8urIzQrs6m9E5fHd1IPvGsoYZMGuLKW4dTAbxWRrcESS3pvJ4nIzSJSKiKl9fUjt8LeRGazCT+58lTOmZnFN58vZ91Hvd8I7OqB91FCAfrsgbd7fPx5y0Eunp9PsjOWzCQHM3OSwtbBd9e2ckovE3hOZI+xsaIkh3U760Zk04fu64B3p2PBlVUNNcAfAKYDC4BDwM96O9EY86AxZrExZnF2dvYQ3071J85u44FrTqckL5kvP7GRLVWNYc9zub3YBJyxff+v72tfzL9WHKLV7WX14qKuY0uLMyjd19Cjjtzp87PnSGu/I1C6+9TsXBraOtl4IHz7h6NrDHj6CTXw4GxMnU6vrGZIAW6MqTXG+IwxfuAhYGlkm6WGInRjMzMpjhsf+zDsNHyXJ7ASoUj4DY1D8lKcbDrQSNn+k3vVz5RWMSUzgTO6TSRaWpxBq9vLjkPHp8MPdARKd584JYvYGOGvFYcG/JyBOnEWZkhmYhw20QWtlPUMKcBFpPt2MlcAFb2dq0ZXTrKT3964FL8x/NMjGzjm8vR4vM3tI6GPG5gh37l0DllJcVz78Abe//h4bXv/URcf7DnGZ08v7PFL4IziwCSe7svCDnQESnfJzlj+YW4ej767j1+/+XFEF7iqaWzHYbeRmRjX47g9xkZ2skN74MpyBjKM8CngfWCWiFSLyE3Af4lIuYhsBc4Dvj7C7VSDMC07iYevX8KhpnZ+8Jeeu920evpeSjZkUlo8z3zxTArS4rn+0Q28GRwZ8mxZNSJw5emFPc7PS3UyJTOhRx08tIjV9JzEQbX/p589jUtPzefHf93J3S+UD2p7uapjbdzy+400nPCLCwLrgBekxYf910duilMXtFKWM5BRKFcbY/KNMbHGmEJjzMPGmGuNMfONMacaYy4zxkT+37tqWBZNTufmT0zjuY3VPXrQbf1s5tBdToqTp794JtOzk/jC46X8reIQz5VV84mZ2WGnuy+dmsGH+47hD9bBK+taKMqI73PMeTjO2BjuvWoht5w3nac2VHHjYx/S3DGwzS3uXVvJX7Ye4ncf7D/pserG9pPq3yGBANceuLIWnYk5jt163kyKMuL59p/KuybZuDy+PqfRnygjMY6nvrCMOZNS+NITGznY1MHnlhSFPXdpcQYNbZ3srg/0vCsHMQLlRDabcMc/lPBfV57K+x8f5R8feI/qhr43Pz7Y2M6fNtdgE/jdB/tPmlh0sLGdSb2ss5Kb4tAAV5ajAT6OxcfF8L3L5vFxvYuH3t4DhHakH1yPODUhlif++QyWTctgUqqT82fnhD1v2bRAHXz9nqNdI1BmhNnEYTBWLyni8RuXcqipg6se/KDPJW4ffmcvfgP/uWoe9S3uHjdCOzp91Le4e+2B56U4aWjrHNW9OZUaLg3wce68khwunp/HvWsrOXC0jTaPj4RBBjgERrg89YVlvHb7J3HYw/fgC9PjyU91sn7vMfYfbQvswjPEHnh3y2dk8fB1S6huaOfetZVhz2lweXhqwwEuP20SVy+ZzLSsRB55d1/X44eCqw2eOAIlJLS1Wr0uaqUsRAN8Avj3S+ditwnfebGCVre3z80c+iIifdazRYSlxRls2HuMytoWAGYOswcesrQ4g6uWFPGbd/b2GKoY8vj7+2jz+PjiJ6djswnXL5/KlqpGNh5oAI4PITxxEk9IXoqOBVfWowE+AeSlOrn9glm8uaue+hb3gEahDNXS4gzqWtxd0+EHsgbKQN15YQmp8bHc/UJ5141SCCzQ9dh7+/jU7Bxm5QV6/J9ZVEiyw85jwV74iRs5nEhnYyor0gCfIP7pzCnMnZQC9L2d2nCFxoO/vPXQkEag9CU9MY5vXTybTQca+cOHx9d8+cOGKhrbOvmXc6d3HUty2Fm9pIg15Yc43NRBTWM7IsdnXZ6oqweuGzsoC9EAnyDsMTZ+cMV8bAJZwZ3YR8L07EQyE+PweP29bmI8HJ9ZVMCyaRn8+K87qG9x4/H6+c3be1hanMHpU3puMXfdmVPxGcOT6/dT09hOTrKDOHv4j3xKvB2H3Uad1sCVhWiATyALitJ49Ruf7LGGSaSF6uAQufr3ia///VXzae/08cM1O3hpy0EONnX06H2HTM5M4PySXH6//gB7j7h6LZ+EXjcv1ak9cGUpGuATzPTspF43H46UUIBHYgRKODNykvjSJ6fzwqYafrRmByV5yZx7SviF0m5YPpWjLg9l+xt6HYESkpusk3mUtWiAq4i7YG4eCyen9brJcSTcct4MpmQmcNTl4V/Ond7r4lxnTc9kVnAtlt7GgIfkpmqAK2vRAFcRV5AWzwtfXt5vj3c4QtPtbzq7mEvm5/d6nkhgSGGoXX3JTXZQ2+yO6AJaSo2kkRtPptQIO60ojdOK0vo974qFBeyua2XlnNw+z8tLddLe6aO5w0tqfGyEWqnUyNEeuBr3nLExfOfSOWEX4OouNBuz+7rgPr/hh2t2cOvvNw54QS2lRov2wJUK6j4bc2ZuMh2dPr761CZe2V6LTeCjwy08cv0SijISotxSpQK0B65UUG5KYHx8bbObBpeHz/9mPa/uqOWeT8/hiX8+g7oWN6vuezfsLkVKRYMGuFJBoen0ZfsbuPLX71Fe08T9/28R1y8v5qzpWbzw5bNIdtq5+qH1vLi5Jsqt7V9FTRP/+sctVB3rexnekdJ9uQM1MjTAlQpyxsaQGh/LUxsOcKTFzRM3ncFF3Ua4TMtO4oUvL2dBURq3/WEz//W3nbT0Uxeva+ng6Q8PsO1g06iNbvH5Dfet282q+97l2bLqQW2IEQnGBN5/3j1/5+ev7jppXXYVOTKaQ6YWL15sSktLR+39lBqsy//3Hepb3Dx249Je9/L0eP1864Vy/lhWTZLDzmcXF3LDWcVMzjxeGy+vbuLRd/fy560H6fQF/o4VZyVy8fw8Lp6fz5z8lH43lh6KqmNtfOOZzXy4r4FL5ufz6dPyufX3mzhzeiaPXr8Ee8zI9tlcbi//+sct/LXiMCV5yew83EJJXjI//expzCtIHdH3Hs9EpMwYs/ik4xrgSh1X3+LGEWsjxdn/MMItVY08+u5eXt56CJ8xrJydyydnZfPCxhpK9zeQGBfDZxcX8ZlFBVTUNLOm/BDvfXwEvwmE+QVzc1k5O5eFk9OJsfUM83aPj3d2H2HtjlpS42O5dcUMkvtokzGGZ8uq+e6ftyPA91bNZdWCAkSEpz88wJ3PlfNPZ07he5fPG+4fUa/2H3Vx82/LqKxr4ZsXzeafzylm7Y467n6hnKMuD7ecO51bV8zsdT2awfL6/Dz+/n7Kqxv5/hXzB71RSW8qappIjY8dUzerNcCVGiG1zR387v39PLl+Pw1tnUzOSOC6s6by2cWFJ/0iONrq5u/ballTfogP9hzF6zdkJMaxoiSH80tyaGrv5LUdtbxdeQS310+yw47L4yUn2cl/rpoXdiz7+j1H+dkru9iw7xhnFGfws9WnUZjeM3x+tGYH//fWHu759ByuX14c8T+DN3fV85Xfb8RmE3519ULOmXl8aYOmtk6++/I2nt9YQ0leMj+4Yj6nT0kf1vttrmrk7ufL2R5cG/6cmYFNP4b7y2H9nqNc+/AGnLE2fnPdkq5lIaJNA1ypEdbR6aOytpU5k1JO6lGH09zRyZsf1bN2Ry2v76yjuSOwXVxBWjwr5+Syck4uS6ZmsP1QM3c9t5Wdh1u4ZH4+/3HZHHKSnWypauSnr3zE25VHyEl28JXzZ/L/lk4O+94+v+FLT5SxdkctD1+/hPNmhd8WL8QYQ9WxdnzGMCUjAVuY13S5vazdWcdfyw/x922HOSU3mQevXdyjlNTd2h213P1CObXNbj6zqIC7LiohJzn88r69aWrv5L//vpMn1x8gJ9nBPZ+eS4vby789u5VVCybx89ULwrZ1IHbXtXLlA++RmRQHENgB6qoFXDiv95m+o0UDXKkxrNPnZ3NVI0kOOyV5ySfVxzt9fh58aw+/XFuJ025jweR03tpVT0ZiHP/yyelce+aUfhcpa/N4+eyv32f/0TauWTaF/FQnuSlO8lOd5KQ4qGlop2x/A2X7G9h4oIEjrR4Akh125hakcGphGvMKUvH7DWvKD/HmrnrcXj/ZyQ4uO20St19wSr/rv7vcXv533W4efnsvcXYbXz1/BtefVRy25+z2+jjY2EF1Qxs1De0cONbGM6XVHHO5uf6sYr6+cmZXWem+dbv5779/xM2fmMbdF88ezB89AEda3Vxx/7u0e3y88OXlJDns3Pj4h2yuauR7l8/j2mVTBvQ6L289yJ821XDD8mKWz8gadDt6owGu1DjwcX0rdz9fzke1Ldy4vJgbzy4eVO33UFM7N/+2jJ2Hm7turp5oamYCi6akc/qUdGJEKK9poqKmiR2HW7pGlOSmOLhoXj4Xz88PnDfIXu/eIy6+//J21u6sY1p2IguK0mhq66ShzUNjeydNbZ0ca/PQPZ5ibMLCojTuuWzuSTdEjTHc89I2Hn9/P9++ZDb/fM60Abel3ePj6oc+YOfhZv5w85ksCC7P0O7xcevvN7J2Zx23njeD2y84pdcbz01tnfz7SxW8uPkgcXYbHq+f82Zl882LZ/d6M3wwNMCVUl38fsOxNg+HmzoCX80dZCc7OH1KOllJ4Tf86PT52VXbQqfPcGpB6pBLFd2t21nHf//9I5raO0lPjCUtPo7UhFjS4mPJTnZQlJ5AYXo8Benx5KU4+xxF4/MbvvrUJv5SfohfXrWAyxcU9Pv+Pr/hy0+W8cr2Wv7vmtO5YG5ej8e9Pj/feqGCp0urOHdWNqsWFHDurGzSEuK6znmn8gj/+sct1Le6+cqKGXzhnGk88cF+/nfdblxuL6sXF/GNlad0LdUwFBrgSqlxr6PTx/WPbmD93mPMyk1maXEGi6dmsGRqOvmp8RhjaG73UtPYTk1jO3+rOMxzG6v5j0/P4YZebu4aY7j/jY959N29HGn1EGMTTp+Szqdm53CwsYPH3tvHtOxEfrF6QY/F1RpcHn71+m5+98E+7DYb91+zqN97D73RAFdKTQgtHZ08/t4+1u89xsb9Dbg8PgCykx20ub1dP4cMtG7u9xu2VDeydkcdr+2oZefhFgCuP2sqd15YQnwve83uP+ri3rW7+dYls8lIjAt7Tn80wJVSE47X52fn4RY+3HeMippmUuNjmZTmpCAtnklpgdJMbyWj/lQda8Pt9TFjhHae6q63AO/37oeIPAJcCtQZY+YFj2UATwNTgX3AamNMQyQbrJRSw2WPsTGvIHVEZoGOhYk+Axn1/hhw4QnH7gLWGmNmAmuDPyullBpF/Qa4MeYt4MT1My8HHg9+/ziwKrLNUkop1Z+hzjvNNcYcCn5/GOh7ryqllFIRN+xVZUzgLmivd0JF5GYRKRWR0vr6+uG+nVJKqaChBnitiOQDBP9b19uJxpgHjTGLjTGLs7OzeztNKaXUIA01wF8Crgt+fx3wYmSao5RSaqD6DXAReQp4H5glItUichPwY2CliFQCnwr+rJRSahT1Ow7cGHN1Lw+dH+G2KKWUGoRRnYkpIvXA/iE+PQs4EsHmRNt4up7xdC2g1zOWjadrgYFfzxRjzEk3EUc1wIdDRErDTSW1qvF0PePpWkCvZywbT9cCw78e3ZVeKaUsSgNcKaUsykoB/mC0GxBh4+l6xtO1gF7PWDaergWGeT2WqYErpZTqyUo9cKWUUt1ogCullEVZIsBF5EIR+UhEdouI5dYeF5FHRKRORCq6HcsQkVdFpDL43/RotnGgRKRIRNaJyHYR2SYitwWPW+56RMQpIhtEZEvwWr4bPF4sIuuDn7enRWRo+2BFiYjEiMgmEXk5+LNlr0dE9olIuYhsFpHS4DHLfdYARCRNRJ4VkZ0iskNEzhzutYz5ABeRGOA+4CJgDnC1iMyJbqsG7THGz6YYXuB2Y8wcYBlwS/D/hxWvxw2sMMacBiwALhSRZcBPgF8YY2YADcBN0WvikNwG7Oj2s9Wv5zxjzIJu46Wt+FkD+CXwN2NMCXAagf9Hw7sWY8yY/gLOBP7e7edvAt+MdruGcB1TgYpuP38E5Ae/zwc+inYbh3hdLwIrrX49QAKwETiDwMw4e/B4j8/fWP8CCoNBsAJ4GRCLX88+IOuEY5b7rAGpwF6CA0cidS1jvgcOFABV3X6uDh6zOstviiEiU4GFwHosej3BcsNmAksivwp8DDQaY7zBU6z2efsf4N8Af/DnTKx9PQZ4RUTKROTm4DErftaKgXrg0WB56zciksgwr8UKAT7umcCvX0uN5xSRJOA54GvGmObuj1npeowxPmPMAgI916VASXRbNHQiEtp8vCzabYmgs40xiwiUUG8RkU90f9BCnzU7sAh4wBizEHBxQrlkKNdihQCvAYq6/VwYPGZ1A94UY6wRkVgC4f2kMeb54GHLXg+AMaYRWEegxJAmIqGVOq30eVsOXCYi+4A/ECij/BLrXg/GmJrgf+uAFwj8krXiZ60aqDbGrA/+/CyBQB/WtVghwD8EZgbvpMcBVxHYUMLqLLkphogI8DCwwxjz824PWe56RCRbRNKC38cTqOXvIBDk/xg8zRLXAmCM+aYxptAYM5XA35PXjTGfx6LXIyKJIpIc+h64AKjAgp81Y8xhoEpEZgUPnQ9sZ7jXEu3i/gBvAFwM7CJQn/xWtNszhPY/BRwCOgn8Jr6JQG1yLVAJvAZkRLudA7yWswn8M28rsDn4dbEVrwc4FdgUvJYK4N+Dx6cBG4DdwB8BR7TbOoRrOxd42crXE2z3luDXttDffSt+1oLtXgCUBj9vfwLSh3stOpVeKaUsygolFKWUUmFogCullEVpgCullEVpgCullEVpgCullEVpgCullEVpgCullEX9fynS8ryh0QuJAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 查看模型训练结果\n",
    "plt.plot(train_mse, label='train_mse')\n",
    "plt.legend(loc = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 以0.03作为学习率迭代60个epochs，对比模型训练效果\n",
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 实例化模型  \n",
    "tanh_model = net_class2(act_fun=torch.tanh, in_features=5, BN_model='pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#———————————————————————————第二个模型- 以0.03作为学习率迭代60个epochs———————————————————————————\n",
    "# 创建用于保存记录结果的空列表容器\n",
    "train_msel = []\n",
    "test_msel = []\n",
    "\n",
    "# 创建可以捕捉手动输入数据的模型训练流程\n",
    "while input(\"Do you want to continue the iteration? [y/n]\") == \"y\":    # 询问是否继续迭代\n",
    "    epochs = int(input(\"Number of epochs:\"))                           # 下一轮迭代遍历几次数据\n",
    "    lr = float(input(\"Update learning rate：\"))                        # 设置下一轮迭代的学习率\n",
    "    train_l, test_l = fit_rec(net = tanh_model, \n",
    "                              criterion = nn.MSELoss(), \n",
    "                              optimizer = optim.SGD(tanh_model.parameters(), lr = lr), \n",
    "                              train_data = train_loader,\n",
    "                              test_data = test_loader,\n",
    "                              epochs = epochs, \n",
    "                              cla = False, \n",
    "                              eva = mse_cal)\n",
    "    train_msel.extend(train_l)\n",
    "    test_msel.extend(test_l)\n",
    "\n",
    "    # y 60 0.03 n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403151960>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403151780>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'epochs')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'MSE')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f4031517b0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABGjklEQVR4nO3dd3zdVf348de5I/fmZu+kSdukezctbWmpIFAqBcoWVCxSF4psFcX1E9fXLYIiiDIFEWTLkFFaQDqgLW2T7t0kzd7j5uaO8/vjc7Oam52bm5u8n49HHr33M9+fEN733DOV1hohhBBjhynUAQghhBhekviFEGKMkcQvhBBjjCR+IYQYYyTxCyHEGGMJdQB9kZycrLOzs0MdhhBChJVt27ZVaK1TTt0eFok/OzubrVu3hjoMIYQIK0qp44G2S1WPEEKMMZL4hRBijJHEL4QQY0xY1PELIcKf2+2msLCQ5ubmUIcy6tjtdrKysrBarX06XhK/EGJYFBYWEhMTQ3Z2NkqpUIczamitqayspLCwkJycnD6dI1U9Qohh0dzcTFJSkiT9IaaUIikpqV/fpCTxCyGGjST94Ojv73VUJ/6d7/yLTY/9INRhCCHEiDKqE79z3zvMP/I3tM8X6lCEEGLEGNWJn4SJOJSLyrKiUEcihAixmpoa/vKXv/T7vAsvvJCampqhDyiERnXit6dMAqCi8ECIIxFChFp3id/j8fR43muvvUZ8fHyQogqNUd2dMz5zKgANJYeBFaENRgjR5if/2c2ek3VDes1Z42L58cWzu91/5513cvjwYXJzc7FardjtdhISEti3bx8HDhzgsssuo6CggObmZm699Vauv/56oH2usIaGBi644AI+8YlPsHHjRjIzM3nppZeIjIwMeL+zzz6bBQsW8P7779PY2Mjjjz/OL3/5S/Ly8vjMZz7Dz3/+cxobG7n66qspLCzE6/Xyox/9iM985jNs27aNb37zmzQ0NJCcnMyjjz5KRkbGkP2uRnXiTx1vJH5PxbHQBiKECLlf/epX5Ofns2PHDjZs2MBFF11Efn5+W9/3hx9+mMTERJxOJ4sXL+bKK68kKSmp0zUOHjzIU089xd/+9jeuvvpqnnvuOdasWdPtPSMiIti6dSv33HMPl156Kdu2bSMxMZHJkydz++23s2HDBsaNG8err74KQG1tLW63m5tvvpmXXnqJlJQUnn76aX7wgx/w8MMPD9nvYlQnfkd0HJXEYaoNOEGdECJEeiqZD5clS5Z0GvB077338sILLwBQUFDAwYMHuyT+nJwccnNzATjttNM4duxYj/e45JJLAJg7dy6zZ89uK7VPmjSJgoIC5s6dy7e+9S2++93vsnr1as4880zy8/PJz89n5cqVAHi93iEt7cMoT/wAFZZ0HE2FoQ5DCDHCREVFtb3esGEDb7/9Nps2bcLhcHD22WcHHBBls9naXpvNZpxOZ4/3aD3eZDJ1OtdkMuHxeJg2bRrbt2/ntdde44c//CErVqzg8ssvZ/bs2WzatGmwj9itUd24C9AQmUliS3GowxBChFhMTAz19fUB99XW1pKQkIDD4WDfvn1s3rx5WGI6efIkDoeDNWvWcMcdd7B9+3amT59OeXl5W+J3u93s3r17SO876kv8LbETSK3bgMfdgsUaEepwhBAhkpSUxPLly5kzZw6RkZGkpaW17Vu1ahUPPPAAM2fOZPr06SxdunRYYsrLy+OOO+7AZDJhtVq5//77iYiI4Nlnn+WWW26htrYWj8fDbbfdxuzZQ1c9prTWQ3axYFm0aJEe6ApcHz53N0vy7uLk2g8Zlz19iCMTQvTV3r17mTlzZqjDGLUC/X6VUtu01otOPXbUV/U4Uo2+/FVF0pdfCCFgDFT1JGROA8BZeiTEkQghRqMbb7yRDz74oNO2W2+9lS9+8Yshiqh3oz7xp2ZNwqsVnsqjoQ5FCDEK3XfffaEOod9GfVWPNcJGqSkFa31BqEMRQogRYdQnfoBqawbRTTJRmxBCwBhJ/I2OTJI90pdfCCFgjCR+b9wEkqnB2Rh48IYQQowlYyLxW5OM+TjKTkiXTiHGqtEwH//atWt59tlnB32dMZH4ozOmAFBTfDDEkQghQkXm428XtO6cSik78B5g89/nWa31j5VSjwKfBGr9h67VWu8IVhwAyeONvvzNZdKlU4gR4fU7oSRvaK+ZPhcu+FW3u2U+/nbB7MfvAs7VWjcopazA/5RSr/v33aG1Hvz3lT5KSs3CqSPQ1TI9sxBjlczH3y5oiV8bkwA1+N9a/T8hmRhImUyUmdOwNUhffiFGhB5K5sNF5uMPEqWUGdgGTAHu01pvUUrdAPxCKfX/gHXAnVprV4BzrweuB5gwYcKgY6mxZRDbLH35hRAGmY8/SLTWXq11LpAFLFFKzQG+B8wAFgOJwHe7OfdBrfUirfWilJSUQcfSHDWeVE8J2ucb9LWEEOFH5uNvNyxz9Wita5RS64FVWuvf+Te7lFKPAN8elhgSJhJT4aS2upy4pLTeTxBCjCoyH3+7oM3Hr5RKAdz+pB8JvAn8GtimtS5WSingbqBZa31nT9cazHz8rba/8Q8WbrqJg5e9wtTcMwd1LSFE/8l8/MHVn/n4g1nizwAe89fzm4BntNavKKXe8X8oKGAH8PUgxtAmLmMyAPXFh0ASvxBiDAtmr55dwIIA288N1j17kjLBWH2rpVzm5RdCDB2Zj38Ei41PopYoVO2JUIcixJiltcao5R09RsJ8/P2tsh8TUza0KjNnENlYGOowhBiT7HY7lZWV/U5SomdaayorK7Hb7X0+Z8yU+AHqI8eR0ngo1GEIMSZlZWVRWFhIeXl5qEMZdex2O1lZWX0+fkwl/pbo8aTVb8Tn9WIym0MdjhBjitVq7TRSVoTOmKrqUYnZRCgP5cXHQh2KEEKEzJhK/JEpkwCoLJTpmYUQY9eYSvwJWVMBaCw5HOJIhBAidMZU4k8dPxWfVniqjoU6FCGECJkxlfhtdgcVKgGL9OUXQoxhYyrxA1RaM4hqkr78Qoixa8wl/gZHJknu4lCHIYQQITPmEr8ndiIpugpXc1OoQxFCiJAYc4nfPj4Xk9Ic/OitUIcihBAhMeYS/4zll+LUETTufCnUoQghREiMucQfGRXDvqhFZFdskGUYhRBj0phL/ADuqReSRiWHdn3Q+8FCCDHKjMnEP/UTn8arFRVbnw91KEIIMezGZOJPSMlgn20u6SfXhToUIYQYdmMy8QPUZ59Pju84hYfyQx2KEEIMqzGb+CeccRUAhZv/HeJIhBBieI3ZxD8uezqHzZOIO/5mqEMRQohhNWYTP0BZ5nlMb9lLRUlBqEMRQohhM6YTf+riKzEpzZEPng11KEIIMWzGdOKfNHsJJ1UqtkOvhzoUIYQYNmM68SuTiROp5zKjaTsNddWhDkcIIYbFmE78ALG5l2FTbg588GKoQxFCiGEx5hP/tEUrqCYW395XQh2KEEIMi6AlfqWUXSn1oVJqp1Jqt1LqJ/7tOUqpLUqpQ0qpp5VSEcGKoS8s1ggOxn+CaXUbcbe4QhmKEEIMi2CW+F3AuVrr+UAusEoptRT4NXC31noKUA18OYgx9EnE7IuJpYl9m/8b6lCEECLogpb4taHB/9bq/9HAuUBr/8nHgMuCFUNfTVt2ES5tpTH/1VCHIoQQQRfUOn6llFkptQMoA94CDgM1WmuP/5BCILObc69XSm1VSm0tLy8PZpg4ouPYH5lLZvn7Qb2PEEKMBEFN/Fprr9Y6F8gClgAz+nHug1rrRVrrRSkpKcEKsY0zewXj9UkKDuUF/V5CCBFKw9KrR2tdA6wHlgHxSimLf1cWUDQcMfRm/OmXA1D04YuhDUQIIYIsmL16UpRS8f7XkcBKYC/GB8Cn/YddB4yIxW/H5czguGk8Ucdljn4hxOgWzBJ/BrBeKbUL+Ah4S2v9CvBd4JtKqUNAEvBQEGPol+LUs5jevEtG8QohRjVL74cMjNZ6F7AgwPYjGPX9I07M3IuIKHmS/E2vsPD8a0MdjhBCBMWYH7nb0bTF51GHA88+mbRNCDF6SeLvwBph41D0YnKqN6J9vlCHI4QQQSGJ/xTeKZ8ihWoO520MdShCCBEUkvhPMWnZZfi0onz7f0IdihBCBIUk/lMkpWVxyDqVxKL1oQ5FCCGCQhJ/AJXjzmGq+wBVZSNibJkQQgwpSfwBJC+8GJPSHN74YqhDEUKIISeJP4DJc8+ggnhMh94MdSgjWt4vz2HTYz8IdRhCiH6SxB+AyWzmSPwZTG34UBZn6cF41wEiKnaHOgwhRD9J4u+GdeYqYmniwFaZu6c7Dt2Mxd3Q+4FCiBFFEn83Jp++GoC6/e+GOJKRqcXVTITyEOFtCnUoQoh+ksTfjdj4JArUOOzlu0IdyojkbKgFwOZtDHEkQoj+ksTfg9KYWWQ27Qt1GCOSs7EOALuWEr8Q4UYSfw88afNIpYqKkhOhDmXEaW40SvyR2hniSIQQ/SWJvwexk4zZo4v2yLw9p3L5S/xRukkmtBMizEji78GE2UvxaUXTse2hDmXEaWmqByBCeXG5pNQvRDiRxN+D6NgECsyZRFYMTQPvsb1b2fTQt0ZFCdnTXN/2uqm+JnSBCCH6TRJ/L8qjZzCuaf+QXKv4/cdZVvB3io7sGZLrhZK3Q+J3NtSELhAhRL9J4u+FJz3XaOA9eXzQ17I0lQFQsuf9QV8r1DolfinxCxFWJPH3InbSYgAK924a9LVszeUAeE98OOhrhZp2tY/YbWmqC2EkQoj+ksTfi4n+Bl7nsa2DvlZMSwUASdU7B32tUNMt7QO3WppqQxiJEKK/JPH3IiomngJzFpEVeYO+VryvCoBsz1GaGsI7WaqW9hK/RxK/EGFFEn8flMXMJMs5uBG8La5mEqjjgGUaFuXjWJiv6WtyN+LVCuhc3y+EGPkk8feBN20+ydRQfvLYgK9RVVoAQOX4lQDUHgz3xN9ElYoHQDdLHb8Q4UQSfx/ET/aP4N098GRdW2Yk/sisecbkbyXbhiS2ULF4GqkzJ+DTCu2SEr8Q4UQSfx9MmH06Xq1wnhh4sm6qOglAVFImJbFzGN+0O6wHclm9TbjMUTRi71TfL4QY+STx94EjOo4C83gcgxjB21JtLNyekDoBX+Zikqmh+MTBoQpx2EV4m3CbI2lSDkyS+IUIKz0mfqXUmg6vl5+y76Zezh2vlFqvlNqjlNqtlLrVv/0upVSRUmqH/+fCwTzAcCmPmUmm88CAS+m++hK8WpGQmknSdONXeTI/fBd5sWknHrMDp8khq3AJEWZ6K/F/s8PrP52y70u9nOsBvqW1ngUsBW5USs3y77tba53r/3mt7+GGjjc9l2RqKDt5dEDnmxtLqVZxmC0Wsmctpknb8BwP34Fcdp8TrzUKl8mBRRZjESKs9Jb4VTevA73vRGtdrLXe7n9dD+wFMvsd4QgRP9kYwXtyz8BG8NqcZdSYkwCwWCM4aptGYhgP5IrUTnzWKFrMUdg8kviFCCe9JX7dzetA77ullMoGFgBb/JtuUkrtUko9rJRK6Oac65VSW5VSW8vLy/t6q6CZOHspXq1oHmADb7S7ksaI5Lb3dckLyHYfptkZfklT+3w4aEZbo3BborH5ZBUuIcJJb4l/hj9B53V43fp+el9uoJSKBp4DbtNa1wH3A5OBXKAY+H2g87TWD2qtF2mtF6WkpPTxcYInMiqGE+aJOAY4gjfOW4XL3v4c9uzTiVDesBzI5Wpuwqw02GLwWqOwS+IXIqxYetk/czAXV0pZMZL+k1rr5wG01qUd9v8NeGUw9xhOFbEzmVSzEe3zoUx97xDlcbeQqGs5GJ3etm38vE/CRqg58AEsWRmMcIOmsb4GO2CyReG1RuNAEr8Q4aTH7KW1Pt7xB2gAFgLJ/vfdUkop4CFgr9b6Dx22Z3Q47HIgf8DRDzNfRi5J1FJadKRf51WVFWFSGlNMe+JPTh/PSZVGRBgO5GpuNAZsmWzRaFsMUdoZ1mMShBhreuvO+YpSao7/dQZGkv4S8A+l1G29XHs5cC1w7ildN3+jlMpTSu0CzgFuH+xDDJc4fwNv8d7N/TqvptRYrN2W0Llt+2T0HLIawuZzr43LPw2zJTIGFRGNWWmcTTJ6V4hw0VtVT47WujUzfRF4S2v9BaVUDPAB8MfuTtRa/4/APX/CovtmINmzTsfzHxPNxz8C1vR6fKumSmPwliOpc+L3jFtE6v51lBYeJi1r8lCGGlQtjcZsnBZ7DC32WACa6mpwRMeFMiwhRB/1VlHt7vB6Bf6k7e+eOea+29sd0RyxTiGurH9z87tqjOka4lKzOm1P9A/kKtz13tAEOExanEbp3hoZgznSSPzOxpoQRiSE6I/eEn+BUupmpdTlGHX7/wVQSkUC1mAHNxJVJS9mSss+mpv6PlrVV1eMTysST0n82bNPp1lbcR/f0s2ZI5PHaVT12KJisfgTf3OYry8gxFjSW+L/MjAbWAt8Rmtd49++FHgkeGGNXJHTziJCeTj08fo+n2OM2o3FGmHrtD3CZudoxFTiq8JrIJen2fjQsznisDqM6h2XlPiFCBu99eop01p/XWt9qdb6zQ7b12utfxf88EaenIUr8WpF/b4NfT4nwllOjTkx4L7apAXktBykxdU8RBEGX+t6u5HRcdiijMTvlnV3hQgbPTbuKqVe7mm/1vqSoQ1n5IuNT+KgZTKxpX2vnoluKafBmhxwn23SGdhKnmTnBy8y/9zPDlWYQeXzz78fGR2L3Z/4W6t/hBAjX2+9epYBBcBTGNMt9Dg/z1hRmbyYBSXP0uxsxB4Z1evxcd4qqmICD3Se/clPU7rx/2Hd9CcIk8RPSyMebcJmiyQyJh4An6zCJUTY6K2OPx34PjAHuAdYCVRord/VWofvnMKDZJ/6SWzKzeGPe/8VeD0eEnUN3qi0gPsjbHaOTv8ys9z57NvyZsBjRhrV0kiTikSZTETFGlMt+WTdXSHCRm91/F6t9X+11tdhNOgeAjb0Nhf/aJdz2kp8WlG3r/cG3uqyIsxKY4rN6PaY+ZfcQjWxuDaER7OJyd2IEzsANlskbm0GWX5RiLDR64QzSimbUuoK4AngRuBe4IVgBzaSxSUkc8QyidiS3uv5a8qNtXat8eO6PSYyKoZ92Z9nvnMLh/P6Nyo4FCyeRppNkQAok4lGFYmpRRK/EOGitykbHgc2YfTh/4nWerHW+mda66JhiW4Eq0hezGTXHlzNPU9Q1lhZCBhr7fZk1qXfpkFHUvPmr4csxmAxe5pwmRxt75uUA5M7/KaXFmKs6q3EvwaYCtwKbFRK1fl/6pVSY7o1zzb5TOzKzeEdPY+6dVW1jtod3+NxcQnJ5I27kty69RQeGtnz90R4m3D7S/wALiXLLwoRTnqr4zdprWP8P7EdfmK01rHDFeRINMlfz1+7t+d6fm9dCQCJaT0nfoCpl34XDxZOvvarIYkxWCJ8TtyW9hJ/s9mBVZZfFCJs9H1SedFJXFIaRy3ZxJT0XCdvaiylmlgibPZer5mcPoEdyReRW/k6ZUUDW9t3ONh8TXgs7d1YW8xRRHhlTn4hwoUk/kEoT1rM5OY9PY66jXCWUWMKPGo3kPGrv4cJH0deHrl1/XbdjLdDid9jicLukxK/EOFCEv8gREw+i0jVwpEe6vmjWiqoj0jq8zXH5cxgR9wK5pU8T01FyVCEOeQc2om2tpf4vdZoImX5RSHChiT+QZh0mrFkYvXed7o9Js5Ticue2q/rxq+4DYdyceD9fw8qvmDweb04lAsdEd2+LSIGh3aGMCohRH9I4h+E+OR0jpqyiS4O3J/f5/WSqGvwOPqX+LNnLcGtzXjLDwxFmEOqqdHozKVs7SV+HRFNlGrG6/GEKiwhRD9I4h+ksqRFTG7ejbvF1WVfdUUxVuVFdVhrty8s1giKzenYakdeA29zQ2vij2nbpuzG60aZk1+IsCCJf5Csk8/EoVwc3vl+l32ta+1GJHQ/arc7VfYJJDhPDDq+oda60pbZ1l7VY/Ivv+hsqAlBREKI/pLEP0jZC/31/Hu61vM3tK61m9jzqN1AmmOyGec9ic/rHVyAQ8zlX1TdbG9P/GZ/ib+5viYUIQkh+kkS/yAlpmZy1DSRuMKuM3W2VBujdmNTJ/T7uip5KjblprTw0KBjHEot/sRvdbSP37P4V+GSdXeFCA+S+IdASfalzHLnc3zvtk7bPbVG4k9Mywp0Wo+iM2cCUHF8z+ADHEJup1GPH9Eh8dscsgqXEOFEEv8QmL7q67RoCyXv/KXTdlNjGbVE9WmxllOlZc8GoKl4/5DEOFQ8Tv96u5HtjbsRbcsvSuOuEOFAEv8QSEzNZFfc2cwqe5WmDj1bIpxlVJv6Pniro6T08TRqO7piZFX1eP0Lrtii49q22aPjAfA4ZWpmIcKBJP4hErX8emKUk/w3Hm7b5nCVU28dWOJXJhPFlkwc9ceGKMKh4WtdaD2qPfFHyfKLQoQVSfxDZMbilRw1TSRhzxNt2+I8VTTbUwZ8zVrHRJJcI6tLp/Ynfkd0ex2/w5/4tSR+IcKCJP4hokwmyqZ/nqneQxzY/i7a5yNRV+Fx9G/wVkct8ZNI95X1utjLsGppwKWtWCNsbZusETaatRVaZE5+IcJB0BK/Umq8Umq9UmqPUmq3UupW//ZEpdRbSqmD/n8TghXDcJu16qs0aRs17z1ATWUpEcqLigm8yHpfWFOmYlaakqN7hzDKwTG5m2hSXaeYblKRmCTxCxEWglni9wDf0lrPwlio/Ual1CzgTmCd1noqsM7/flSIiUskL+l85la/TfGhHQBY47tfZL03sVlGl86qgpGU+Btxqsgu25uUA7OswiVEWAha4tdaF2utt/tf1wN7gUzgUuAx/2GPAZcFK4ZQSD77BiJVC+53fw+AI6n/ffhbpeUYXTpdJSOnS6fF04grQOJvNkVh8cic/EKEg2Gp41dKZQMLgC1Amta62L+rBAhYF6KUul4ptVUptbW8vHw4whwSk+edwX7LDOY3fwRAbMrAE39cQjKVxGGqPjxU4Q2axdNEi6lr4m8xO7B6pMQvRDgIeuJXSkUDzwG3aa07dfvQWmtABzpPa/2g1nqR1npRSsrAe8aEQt2cL7S9Tkzr/3QNHZVZs4huOD7YkIaM1eukxezosr3FHIVNll8UIiwENfErpawYSf9JrfXz/s2lSqkM//4MoCyYMYTC3PPXUkM0dTiIjIrp/YQe1Ednk+ouHKLIBs9Yb7dr4vdYo7HLKlxChIVg9upRwEPAXq31Hzrsehm4zv/6OuClYMUQKvbIKPZN/wZ7E88b9LV8CZNJpoa6msohiGzwbNoZMPF7rVFEakn8QoQDSxCvvRy4FshTSu3wb/s+8CvgGaXUl4HjwNVBjCFkln7uB0NyHVv6dDgCpUd3E7vgrCG55mBEaic+a3SX7doaTZQkfiHCQtASv9b6f4DqZveKYN13tEkcb3TprC3cCyMg8Tt0Mz5r1xK/tsVgV27cLa5Og7uEECOPjNwd4dJzZuLTCndZ6Nffdbe4sCk3RHQt8bcuxdgki7EIMeJJ4h/hbHYHJaZUImqOhDoUmtrW2+2a+E2Rxtw9jZL4hRjxJPGHgQrbeOKaQt+l09loTDltCpD4Lf7E7xoDq3C1uJpH3JKYQvSHJP4w4IzJJsNThPb5QhqHq9Eo8ZvsXRO/tTXxN4zuxVi0z0fVL2fz0TO/CnUoQgyYJP5wkDSFKNVMZUlBSMNw+ZdWtEZ2HZtg9c/P7xrlq3BVlhWRTgWW0p2hDkWIAZPEHwYcGdMBKD22O6RxtPgTv8Ue22VfZFQ8AB7n6J6Tv+qk0dYS7TwZ4kiEGDhJ/GEg2b/+buPJfSGNo2293aiuib91KUbvKE/8DaVG4o9vKQlxJEIMnCT+MJCWNQWXtuIL8fq7raV5m6NrVY8jxlhWYbQvv9hSaTSyJ+sK8LpDHI0QAyOJPwyYzGZOmsdhqzvaZV/JiYPUVpYOSxxe/7KL9g7r7baK8pf4W5dmHLVqjXmTzGhcVaFtcxFioCTxh4nqyAkkNndef3f7G/8g/qFlHP/bNcMSg3bVAxAZ3TXxmy0WmrQN5T9mtLI1FrW9rj05cqbLFqI/JPGHCVdcDhneYjzuFgA2P/V/5G68GS9m5ji3UVoY/CSkXcZCK5EBqnoAGpUDU8voTvwxrlKO6HEANJR1/QYmRDiQxB8mzClTiVBeio/tZfP9X2fp/l+zM3o5ZVe/jElpjrzzaNBjUC0NNGkbZkvgKZ6cJgfmUb4KV7K3lKOOufi0wlN5LNThCDEgkvjDROy4GQB4n1rD0tKn2JJ8JfNuf4mc2aezzzqLjGMvBH2Al3I30hRg2cVWLpMDyyhed7exvoZ4GjAlT6aUBKg90ftJQoxAkvjDRKp//d1s3wk2T7mNJd/4e1vJu3balWT7Cji064OgxmD2NNGs7N3ud5mjiPAObYlf+3xsfuLHfPzmE23VXKFS4a9Oi0nL4SQp2BqKejlDiJFJEn+YSEjOYFPGF9i+9B6WrvkJytT+n27Giuto0RYqP3ishysMntnThCvAerut3EFYfvHE/o9ZeuiPLNh4I1W/mM6mh++grCg0deu1JcZ9o1OyqbSkE90sg7hEeJLEHyaUycSyr/2JhavWdtkXl5hCfswZTC9/A3eLK2gxRHgaaTF1nYu/lccahX2IF2OpPLYLgE2TbqHEPollJx4k8cGFfPzbiyg6sndI79UbZ8UxABIyJ9NgzyDeXQZez7DGIMRQkMQ/SphyryGBOna/93zvBw+Q1efEHWCh9VY+azSR2jmk93QV78GnFblXfod5d66j6Asb+WjcGqY3fETpc3cM6b1646spwK3NJKdPxBWdhRkf1BcPawxCDAVJ/KPE7LOuoJpYfDv+GbR72HxOPJbuq3p8ETFE6aYhbWSOqDpAsSm1bdH6zEmzWfa1P5GXvIoZDR/S7By+XkTW+kLKTUmYLRZ8seMB0DWhny5biP6SxD9KWCNs7E85nzn1G6mtKg/KPey+JjyWrlMyt7FFE6G8uFxDV+pPajpCeeSkrrHMXo1DuTiw+fUhu1dvHM4SaqxpAFiTJwLgLD82bPcXYqhI4h9FkpZfR4TysG9dcBp5I2lGB1hvt5XJP2vnUC2/6G5xMc5bhDNuapd905ddRJO24cz/z5Dcqy8S3aU0RmYA4EjJBsApg7hEGJLEP4pMmbecY6bxxB14bsivrX0+HNqJL8B6u61aE7+zoWZI7nnyyG4ilBdL+swu++yRUeyPXkxO5XvDskCNx91Csq7EE50JQEp8LKU6HnfVsaDfW4ihJol/FFEmE8XZlzPDvYfCQ/lDem2Xy4lF+SAiqttjWpdfdA5Rib/yqLHYSfzEuQH3u6esIpUqDudtHJL79aSi+DgW5cOcMAGA9Dg7RToZU61M1NbF5gdgy4OhjkL0QBL/KDPp3LX4tKLo9d9yYPsGDudt5vj+HZw8tp+mQSyL6GxdaL2HEr/VYST+1gVbBstVvBefVmRNzQ24f8ryK/BpRfnWF4fkfj2pLjbm4bf76/bTYu0U6hRsDYVBv3fY2fRn+N/doHWoIxHdCDzpighbaVmT2RV5GqdXvggvv9hpXx1R7Fn1MLOWrur3dZ0NtSQAJnvgCdoAbP5VuFr6sPzige0baHz718y8+VnskYG/RURU7afYlEpmVOB7JqZmsjdiJikn3+n1foPVWGrU5cdlTAbAbjVTbk4jyvUR+LxgMgc9hrDQVAWt34JqCyB+QmjjEQFJ4h+FJn71n+Tt3ozP48TnduF1u9AtTlLzH2TS62vY0fhncld8tl/XbF1L1xJgofVWrStzefqQ+Bvf/g0Lmjaye/sGZi+/KOAxSU1HKLfnkNnDdWrGn8eyI/dSWniYtKzJvd53oNzVxrw8KZk5bdsa7BlYmj1QXwJxPUU5hpTsan99YrMk/hFKqnpGobikNOaedSnzz/0sC86/jkWrr2fxFbcSe8PbFFizmfPeDXz00l/6dU1XkzHdsqWHEr8j2liFy9vc89TMFSUFzGncDEDd/g0Bj2nr0RM/rcdrjVtyOQDHNg59g3ZHprpCqonF0WEtAld0lvGiRiZra1PsT/wWu5H4xYgkiX8MSUjJIOOWt9hnn8fij7/H5n/+vM/nuv319hGOruvttnLExgOge1l+8dDbD2FVXiqIJ6b0w4DHtPXoSZvR47UmTMulUGUQeeTNHo8bLHvjSSrNKZ226dbSbHcNvCX5cPS9oMY14pTsgthMmLhcEv8IFrTEr5R6WClVppTK77DtLqVUkVJqh//nwmDdXwQWHZvA1NtfZ3vUmSw98Fs2/f32PnWH9PhL8dYeEn+kIwavVm0rdQWifT7SjzzLfssMDqV+isnNe2hxNXc5rnWOnvjseT3GpUwmClPPZobzYxrqqnt9joGKaymh3p7RaZst0Wjo9VV3M3r31W/B09eOrfl8indB+jyYsAzK9oAzeP9NxMAFs8T/KBCoFfFurXWu/+e1IN5fdMNmdzD/9hf5MGE1ywofZvPj3+/1HI+zdb3d7qt6lMlEo4pEtXQ/J/+B7RvI9hVQO+Mz2CZ/gkjVwpFd/+tynOvknh579HQUM/8SIpSHAxtf6vXYgdA+HyneclxR4zptT0qIo1zH4qoIMIjLWQ2FH0JzDRSMkZJvSxNUHoSMeTDhdEBDwUehjkoEELTEr7V+D6gK1vXF4JgtFhbd9DhbY1ey7Nj9fPjCvT0e7/OX4gMttN5REw5MPST+mo2P4NQRzFy5lokLzgOges/6LsdFVO2npMMcPT2Zvvg8aonCuzc45Yi66nKiVDPEje+0PS3WTpFOwVMVoI7/8Dug/d+k9g/ftBIhVbrbeOb0eZB5GpgsY+dDL8yEoo7/JqXULn9VUEJ3BymlrldKbVVKbS0vD87cM2OdyWxm3o1PkGdbyMIdP2bn+n93e6zPv96uI8BC6x01myK7XYXL2VjP7Mq3yI8/h5i4RBJTMzlmGo+jeEuXYxObjlJmzwlwla4s1ggOxp7BlNqNQVmspaLIWIAlIqlzDxWjL383g7gOvg2RCZDzSTjw3yGPaUQqMQbckTHPGOiXMV/q+Ueo4U789wOTgVygGPh9dwdqrR/UWi/SWi9KSUnp7jAxSBE2Ozk3Ps8xSw5TN9zIge3vBj7QVY9Xq2773LcdZorC0s0qXPlv/4No5cRx+tq2baUJpzHZmY/X014P7m5xkekt7LVHT0dqxkUkUM+BbUPfp7+uxBi8FZPWebK49DhjEJe9sQg6tpP4fHDobZh8LsxYDZWHoOLQkMc14hTvAnt8+zejCcugaBt4+vFh3FgBb/8EWkb32s2hNqyJX2tdqrX2aq19wN+AJcN5fxFYdGwC8V99kRpTHMkvrwk43YNyN9KEvdPKX4E02lKZ5NzDvg/f6rLPsfspClV6pwFk5pzlRCsnR3e3lwz72qOno2nLL6VFm6nbPvTdOl2VRlVO4rjOiT8pKoKTOgWzdkNjWfuOkl3G+ykrYbr/WQ+Mgeqekl1GaV8p4/3408HTDMU7+36NV78J//sD7AlOe40wDGviV0p17BZxOTC0E8qIAUtOn4D3mmdRaNSTV1Bd3nmBEZO7EWcPC623GvfZu6kxxTPh1c+z+4NX27YXHdnN7JZdFEy8vNOHx4QFKwGoyG8vqfe1R09HMXGJ5MWexdKyZ9j8+I+GduK2mhM0ayuJKZ0bdy1mEw3+2To79eU/5P/Qm7LCGMCUOhv2j/LqHq8bSvcY9futJiw1/j2xqW/X2POyP+Er2Pdqr4eLgQtmd86ngE3AdKVUoVLqy8BvlFJ5SqldwDnA7cG6v+i/8VPnU7r6MVJ8FRx64rZO+8zuRpp7WG+3Vfr4Kdi/+gZl5lQmv3kduzYYJfAT7/wdn1ZMOu+rnY5PzcyhUKVjK2ov8btO7gEgc0rfEz/A7G88ybaYc1h65F4++vMX+rUM5clj+9n2u0upqSjpsi+i8STlppSA33YCDuI6+DZk5EJ0qvF++gVG8msaxr4OlYehbN/Az+9vF9SKA+B1GfX6raJTIXFy3+r5ndXw2reND47T1sKhdUYvobFkGKu3gtmr53Na6wyttVVrnaW1fkhrfa3Weq7Wep7W+hKttaxbN8LMWLSCbVnXsrj2v51K7BZvE64+lPgBksdNJO6GtyiyjGfG+uv5+I3HyCl8mfzIRQGnVTgZt5Dspl34vF4AIqoPcFKl9dqQfCp7ZBQLbnuOTZlrWVL1H/b9fhV1NZV9Orfwxbs4rWEDe/9zd5d90c3F1ESkBzxPtQ7ial2Jq7Ub59SV7QdNvwC016j3Hw6NlfDw+fDYanB3HSPRo/pSeOkm+L9xkN+PZTxbR+ymn/JhPWGp0bOntwnb3vyhUb9/6Z9h9mXgcRo9o8aKvGfhN5OhanjWd5CRu6KL3Gt+zkmVRvS677QNrrJ6nbSY+5b4wRglnHzjmxy1TmbBpltIpwLP/M8HPnjiGSRQz4kDHwOQ2Hikzz16TmUym1n21Xv4aN5PmdG8k6p7z6b4+P4ezykpOERu9Rt4tWLq8X91GVCW6CnD6cgIeG58fDzVxECNv2fP4fVGl8YpHRL/uIUQlTp83Tpf+5aRRBvLIe+Zvp3jdsJ7v4V7F8DOf0FMOrx0I5Tk9e38kl1giYTkUxbNmbAUmiqNBu7uHF4PHz8By281vjFMXG40Eu97pW/3DjWtjQ+pZ74AHz85sGt89JDxYbdjgOf3kyR+0UVkVAzlZ/6cib5Ctj/1UwAivE24zT336DlVXGIKmbe8we6IeZSQzOxzPhPwuKxcI0mW7lrXoUfPlEE9w+IrbmX/eY+Q6KtAP7qaxh7WCDj2ym8xodk654ckU8OuN9tXMGt2NpJMDd7Y8QHPTY+1U+BLxts6evfQ20bSylrUfpDJBNM+ZVRfeN2Deq5e5T8Pu1+Ac38AaXNh0309l7a1hl3/hj8tgnd+DpPPgRu3wJffAnsc/OvzfauiKt4FabO7zlI6YZnxb3f1/C2N8J9bIGkKfPK7xjazFaatMj4oR/Ko55ZGI2Hfdzr843KjfeK93/Z/OurKw3BiI5issOOfxmyvQSaJXwQ0/9yr2R59FrlHH6ToyF5sPideS/fLLnYnOjaBWXe+S+y3t2OzBz4/Y+I0SknCUriJk0f3+nv0zBrsIzDnzEspXPUI43QZeU8GHp1cU1HCvJIX+Dj+PBZfcTsFahyxO//etr+8yOjKaUkInPhT/fPy+6qPd+7GeWoCnHYBuGrheBAXjWkoM6aJGLcQlt8Oy26E8n3GB053PrgHnv8KOBLhulfgs09C0mSISYPPPAH1xfDsF3tOwFob3ww61u+3SpoCjqTu6/nf+YXRPnLJn8Bqb98+c7Ux6vn4B3169GH33u/gDzONXkjWSLj8r3DBb6H6qDGQrT92PgXKBJ/6OdQVGd+AgkwSv+hW1ufuxYuZymduJlI3DSjxgzGVQ0/19cpkoiB2ARPqd1B5dAfQ/apb/TVr6So+TFjNouKnOJLfdaDY3pd/j0O5SD7/O5jMZk7OuI5pngPs22oky1r/AiyOlMBVT+mxxkpc5rpCo7qjobRz/X6ryeeA2Ra8wVxawyu3G6XQyx8AswXmXAkxGcbCKIFUHYUNv4TpF8H1GyDnzM77sxbB6rvhyAZYd1f3964+ZnyoZQRojFcKxi8NnPgPrYPNf4HFX4GJZ3TeN/lcY4bPkdi7p2wvvPMzyFwEX3rD+N3N/6zRNoGCvf1YB9rnhR1PweQVsOiLEJkIHz8epMDbSeIX3UrNzCFv+s3Ma/6IZGp6XG93sLzjl5FCddu0C5lTA5QeB2ja539PvYrG/eItbQ3IAE0Ntcw88U92OJaRM2sxALMv+Br1OpKGd+8DwFl+DID4jEldrgvtK3GZvC6j5AYw5byuB0ZEQc5ZRvVFMFam2vWMUSd+7g8hZbqxzRIBS66HI+uNmUI70tr4dmCywIW/7X4hmQVrYPFXYeOfjCqhQEq6adhtNWEpVB02vpG03vt/d8OTnzZiXfHjrudERBnJcN+rI28lr90vAAouu994ttZxC9GpRtVWf9omjr4HdYWQew1YbMYHyL7XjAb6IJLEL3q06KrvcMjs74lj7V8df3+kzzOS5byadwbUo6cn8cnpHMq9k+mefXz0/B/btu96+U/E04D97G+1bYuOTWB3+qXMr9tAWdFRPNUF+LTqtABLp7j90zYARsktY357N85TTV9lVAVUHBiqRzPUnYTX7zAGTC27sfO+09aC1WHU9Xe0+3k4vA7O/VHvi8is+qXR4Pryze29dzoq3gXKDKndVM+19effDM4ao93g7btg1qXwlbfB3s2MrzMuMpJi8Y6e4xtOWhuJP/sTRnXYqWauhtJ8o96+L3Y8abSlTPdPVLxgDfjcsOvpoYs5AEn8okcWawS+i/6AVytUTDcJbQhMmDqPKmKxKTfl9uwhv/6iS25gd8Q8Zu7+PRUlBbS4msk+8Ah7IuYyY0nnqpnx59+GGR+HX78XS30hFSqBCJs94HVjIy2Um/y/F1dt5948p5rmH8Xb1949NQW9l3a1hv/cZkyLcNn9XUvujkQjmeT921gpDIzk+/qdxliDJV+lV2YrXPWYMffQM1+A5lNWWCvZBSkzOtfRd5Qx36i22fFPePBsOPgGrPoVfPoRsPUwCd+0VUbd994R1LunbI/xwT37ssD7Z6w2/u1Lqd9ZY1QLzb2q/XeXNttoo/n4iaB+05HEL3o1beHZFH1+A/MvuTlo91AmE8eicgFoip/a88EDvH70lX/Crps59s/b2fn630mnAveyW7scmzlpJjujljG98FliGo9TZQlQsmu9rlJ4YrPaNwSq328VlwXpc/tWz7//dfjjHKPxtSd7XjIS6YofGY2ygSy9AXwe+PBB4/26n0BTBVx8T9/XCo5OgaseNRadeenGzkmpeGfg+v1WFpsxW+eB140pHNa+asTUWkXSnagk45vGSKrnz3/e+DCaeWng/QkTjSqvvnxY7X7B+H3kXtN5+8JroWw3nNw++Hi7IYlf9MmEabm9TtA2WC1ZRpXAUPToCWTi9Fy2jb+ORXVvMXnHrzliymbeJ68MeKx12ddJpI4Z7j00RgYevNUqOjaJBhVtdOPMXNTjscy8xKjyKAi88hgAHhe84e+FtP7/oLybqqHmOvjvnUaiWfK17q+XOMmoNtn6sNFjZOvDcPoNMC6351hPNeF0OO8nRil18/3GtvpSo0G7u/r9Vku+apRsv/Zee9VPX8y4CMr39r3qJJjaqnnOND4IuzPzEmMQX10v41N3PAkpM40SfkdzrjTGRHz8xOBj7oYkfjFiZC+/mv2WGYw/7fyg3WPBNT+jUKWTSB1VC2/sdtK52csv5qjJWGGrJbrnOvC0ODt5phnG/7BmS88BLL3BWJrw5Vu6n7VyywNQdQQuewAiHEYJO1Df7vX/Z1TfrP5j7/c942ZjVPFTn4XYLDin98V3Alp2o1Gd8daPjA+v1obdnkr8ALMvhyv/3n37R3dmXGT8O9DBXFrD5gfgzR8Nvn98SZ7RSD3nip6Pm3mx8e/+Hr6plB+Awo+M0v6p33zscUb7R96zQZu2QhK/GDHSJ0xl+g+3kD5+cIO3emJ3RNN08YNsTvssueev7fY4ZTJRPsvYb4oP3Ie/VVqMjS+13IG+8He9B2CLgYt+b5RiA1XjNJTBu7+FqedD7ufggt8YpcctD3Q+7uQO+PCvsPjLkHVa7/cdf7pR3eJpNnrx2AbYQ0spuPQ+o9rq32uNrp5gVGEFQ/yEvlednMrrMfrZ//e7sPFe+M+tnafP7q/dLxiN2DMu7vm4lOnG+IWeunXueNK41rzAgxpZsAZcdf3rGtoPkvjFmDNt4SdZesNfsVgjejxu3oXXsznts2Qvv6rH49Lj7DjdXupcfSxRTl8Fs6+A937TtRrnnZ8ZQ/fP/4Xxfu5VRiPnup+1V3f4vEaffUey0SunL1oT9uo/woxBLnUdGW809jZWGGMEErKNUmqwzLzYKB23Nk73RUsjPL3GqNZafhuc+W34+B9GFdpAGk21NnpCTfqk0fbQE6WMmI++H3jUs89r9NqZujJwzyAweg0l5BgxB4EkfiG6YXdEs/SGvwacWK6j1FijR0ZZXfscP16f5v9e28tN/9xOXXOAaRou+LXRzbJjKbR4J2z/B5z+9fY5b5QyBlGZI4zulD6fkcxObje6WUbG9/2BUmcag4SGwrhcuOBXxuve6vcHa+bFgDZ+V67ul/Vs01AOj642Gr0v/B2s/IkxvuH0G2DL/bD+F13Paa6D939vnBdo0ZziHcZAtdm9VPO0mnGxMTHfgTe67jv4ljEi+tRG3Y6UMkr9x943qv2GmCR+IQYp3Z/4S/yJv9nt5YYntvHge0d4La+YK/+ykYKqU+pqo1ONIfonNsL2R40S5et3Gt0vz7qj87Gx44xvAMc/MEbarvspTDrbaFMIpdO+CJ/6hdF+EEypM43qsYNvwiMX9NxoWnEIHjrPGF37mSfbu6sqZXxQLvyCMZ/O//5obHdWw/pfGj2o1v0UCrfCPy6D2qLO1939gjHYrbXNoTfjFhhtOadW1ex6Bp77srGvtXtvd3KvMRp/+/NNp496aRESQvQmLdYGQGmdi+rGFr7y+Fa2n6jmrotnMS09hhue2M5l933Ag184jdMmJrafuGCNMXvmWz82evKc2GiU7gOV4hesMaoa3vuNMfXDRX/ovTtksCkFZ9w0PPda/BWIm2DMG/T3FXDN053bFSoOGm0mO/9lVDutfaXzRHmt8a7+o9Fg+vaP4eTH/nn/640G67O+DSh47GIj+X/xv0a1TmtvnknnGB/MfWEyGdfc/lj7PPuvfQd2PGGM7r3y70Y3157EjoMbg7NmsZT4hRikNH+Jf9vxaq58YCN5RbX85ZqFrF2ewxmTk3nhG2cQY7fwub9t4aUdHUqSrYnI22J0y0ybAwuvC3wTpeDieyE6zeiR012f/SGQX1TLt/+9s+u3lGHi83VTBz/tU/Al/xiIh1fBgTeNNX2fXgN/XmwMUjttLXzt3a5Jv5XJbMxlNP1CYwzE1PPg6x8Yk9ONW2BUYX3uX8bEcU9eCa56KNpuvO+tN8+pZq42GtM3/hkePMdo0D3rDmMyvLis3s8PIqVH2jwYASxatEhv3bo11GEI0a35P3mTWqebWLuFv1+3mCU5nUuG1Y0tfO2JbXx4tIpvnD2ZG86eTIzdauz84B5jCoMvvNw2UVpZfTPr95UxJzOOWRmxqNbSvc/b90FX/eT1aR549zB3v3UAj08zNTWa575xBrGtcQaZ1pq/bDjMfesP8ZUzJ3HTOVOIsAQom9YVwz+v9q8VoI0S/pLrjbEMPfWv78jnNRqnu2tc3f9f+Nc1xuRxKTNg26Nwx6H+tal4PfC7qeCsMj6wr3jQqKIbRkqpbVrrLp+CkviFGAKX/vl/lNe7ePRLS5iWFngaghaPjx+8kMe/txUSbbNw1aIsvnhGDhOSHEaDZHQKeYW1PPLBUf6z6yRur/H/Zk5yFBfOTefCuRmdPwSGUEFVE998ZgcfHavmorkZXDw/g5v++THLJifxyNrFWMzBrRxodHn49r938np+CTPSY9hXUs+M9Bh+d9V85mQG6DHkajBGIMdPMEr5PU39MFC7noHn/W0E01YZ1Uv99cG9RpXSBb/u/xiGISCJX4ggKq93YbOa+lQ63llQwyMfHOWVXcV4tWblzDQ+OT2FF7YXsfV4NVERZq5aNJ4rFmaSX1THa3nFbDxcgU8bHwKfmp3GyplpLJiQgNnU+UPA2eLlf4cqWLe3lLhIKzedO6X9m0UAWmue3VbIT/6zBwX89LLZXJabiVKKpz86wXefy+MLyyby00vnDPZX1K3jlY1c//g2DpbV870LZvKVM3NYt7eM77+QR2VjCzeePZmbzp0auPQ/AB6vj8c2HSevsIafXz6XaFsPTZ1bHjQmwLvqUWMQWg/yi2qJi7QyPnFg05cHgyR+IUaY0rpm/rHpOE9uOU51k5sJiQ6uOyObqxZldfkAqWxw8cbuUl7LK2bzkUo8Pk1iVATnzkhlxYxUap1u3t5byvsHK3B5fMTYLDS2eEiNsfOzy+awclbXKo0tRyr5/ZsH+PBYFafnJPL7q+eTldA5af3ytb389b0j3HXxLNYuH9hymD1590A5N/9zOyaT4k+fW8CZU9uramqb3Pzkld08v72IGekx/OLyuZw2MWFQ99tRUMP3n89jT3EdAGdOTeah6xb3/KFSX9p9lZDfliOVXPvQh9itpoBVfaEiiV+IEarZ7eVgaQOzxsV2KcEHUtfs5t395azbW8o7+8qoazZWx8qMj2TlrDRWzkpjcXYie4rruPO5XewrqeeiuRn8+JJZpMbY2VlQw+/e3M/7BytIjbFx84qpXLNkQsB7e32arz+xjXV7S3lo7WLOmd5zdYXWmoIqJ16tmZjowBTgmo0uD+v2lfF6XjFv7C5hWloMD167yKjyCmDd3lK+/0IepXUurliYyZ0XzCA1ppuZQLtR63Tz2zf28eSWE6TG2Ljr4tnUuzx859ldXJY7jj9cnRsw1r44VNbAlfdvJCnaGBBYWO3k3s/msmpO4HWah5MkfiFGIbfXx46CGqJtFmakx3Sp/3d7fTz43hHuWXcQu8VE7oQE3jtQTmJUBDd8cjLXLpuI3dpzY3FTi4erHtjE8com1iydSEacnbRYOxlxdlJjbRRVO9l2vJptx6vZfqKaigZjDqIYm4XZmbHMy4pnTmYcPp/mtbxi3j1QjsvjIyXGxiXzx/GtT03DEdFzz/JGl4c/rz/EQ+8fJcJi4pYVU1h7Rk7AkrrL4+VkTTOF1U0UVTs5UdXEM1sLqWp0sfaMHG5fObWt+uu+9Yf47Rv7uf6sSXz/wpn9+dUDUNHg4vK/fICzxcsL31hOtM3Clx77iB0FNfz00jlcu3Rin67zyq6TvPhxEV9cnsPyKcn9jqM7kviFGMMOlzfw/efz2F9az5eW5/ClT+T0XLd9iuJaJ9c/vo19JXVtjc6nyk5ysHBiAqdNTMCsFHlFteQX1bK3pJ4WjzE6OS3WxgVzMrhwboZxXD9L2UcrGvn5K3tYt6+MSSlR5I6Pp7bJTXVTCzVON7VNbqqaWjrNymA2KRaMj+euS2Z3aSjWWnPXy7t5bNNxfnjRTL5yZuCV1gJxtnj53N82s6+kjn9dv4zc8fFt22/653bW7SvjpnOm8K1PTeu2Qb62yc3/ezmfl3acJMJiosXj45zpKXzvwpnddhLoD0n8QohB8/k0VU0tlNQ2Gz91zaTE2DhtYgLJ0YEHJLm9Pg6U1uP2auZlxg24SqWj9fvK+O0b+6l1ukmIshIfGUGcw0p8pJWUGBvjExxkJUSSmRBJeqy9x15JXp/mlqc+5tW8Yu75bC6X5vayIpn/nG88uY0395Ty1zWn8anZnafu9nh9/OCFfJ7eWsDZ01O4LDeTs6enEO9onx/qfwcr+Pa/d1Le4OLmc6fw1TMn8cTm4/x5/SEaXR6uXjSeb66c1jYlyEBI4hdCiG40u72sfeRDthytYnpaDEtyElmUncji7AQy4iLRWlPn9FBU46Soxsl/80t4bnshP754Fl/sptG7dVzCIx8cpaKhBbNJcdrEBM6bmcrJmmYe3XiMSSlR3H11LvP93xbAGPPxp3cO8Y/Nx7CYTPxlzcJe21a6I4lfCCF6UN/s5rGNx9hytIrtx6tpbDFmW02JsdHk8rS9b9XXdgGfT7OzsIZ1e8t4e28p+0rqAVh7RjbfXTWDyIjAbSzHKxu5d90hfnDRTBKjep5JtjuS+IUQoo88Xh/7Sur56FgV+UV1xEVaGRdvJzM+knHxRhVSd1VbvSmoasLl8TIlNQiDzk7RXeIP2iRtSqmHgdVAmdZ6jn9bIvA0kA0cA67WWlcHKwYhhBgIi9nEnMy4wKOGB2kkDPAK5jjsR4FT5x29E1intZ4KrPO/F0IIMYyClvi11u8Bpy4/cynwmP/1Y8Blwbq/EEKIwIZ7WuY0rXXrKgolQM/joIUQQgy5kM3Hr41W5W5blpVS1yultiqltpaXlw9jZEIIMboNd+IvVUplAPj/LevuQK31g1rrRVrrRSkpfZxjWwghRK+GO/G/DLQuMXQd8NIw318IIca8oCV+pdRTwCZgulKqUCn1ZeBXwEql1EHgPP97IYQQwyho/fi11p/rZteKYN1TCCFE78Ji5K5Sqhw4PsDTk4GKIQwn1EbT84ymZwF5npFsND0L9P15JmqtuzSShkXiHwyl1NZAQ5bD1Wh6ntH0LCDPM5KNpmeBwT9PyLpzCiGECA1J/EIIMcaMhcT/YKgDGGKj6XlG07OAPM9INpqeBQb5PKO+jl8IIURnY6HEL4QQogNJ/EIIMcaM6sSvlFqllNqvlDqklAq7uf+VUg8rpcqUUvkdtiUqpd5SSh30/5sQyhj7Sik1Xim1Xim1Rym1Wyl1q3972D2PUsqulPpQKbXT/yw/8W/PUUpt8f+9Pa2UGth6eSGilDIrpT5WSr3ifx+2z6OUOqaUylNK7VBKbfVvC7u/NQClVLxS6lml1D6l1F6l1LLBPsuoTfxKKTNwH3ABMAv4nFJqVmij6rdHGT2L2XiAb2mtZwFLgRv9/z3C8XlcwLla6/lALrBKKbUU+DVwt9Z6ClANfDl0IQ7IrcDeDu/D/XnO0VrndujvHo5/awD3AP/VWs8A5mP8Nxrcs2itR+UPsAx4o8P77wHfC3VcA3iObCC/w/v9QIb/dQawP9QxDvC5XgJWhvvzAA5gO3A6xkhKi397p7+/kf4DZPkTyLnAK4AK8+c5BiSfsi3s/taAOOAo/o44Q/Uso7bED2QCBR3eF/q3hbuwX8xGKZUNLAC2EKbP468W2YExtfhbwGGgRmvt8R8Sbn9vfwS+A/j875MI7+fRwJtKqW1Kqev928Lxby0HKAce8VfD/V0pFcUgn2U0J/5RTxsf92HVH1cpFQ08B9ymta7ruC+cnkdr7dVa52KUlJcAM0Ib0cAppVYDZVrrbaGOZQh9Qmu9EKOq90al1Fkdd4bR35oFWAjcr7VeADRySrXOQJ5lNCf+ImB8h/dZ/m3hrs+L2Yw0SikrRtJ/Umv9vH9z2D4PgNa6BliPURUSr5RqnfE2nP7elgOXKKWOAf/CqO65h/B9HrTWRf5/y4AXMD6cw/FvrRAo1Fpv8b9/FuODYFDPMpoT/0fAVH/PhAjgsxgLwYS7sFzMRimlgIeAvVrrP3TYFXbPo5RKUUrF+19HYrRV7MX4APi0/7CweBYArfX3tNZZWutsjP9P3tFaf54wfR6lVJRSKqb1NfApIJ8w/FvTWpcABUqp6f5NK4A9DPZZQt14EeSGkQuBAxj1rz8IdTwDiP8poBhwY3zyfxmj7nUdcBB4G0gMdZx9fJZPYHwd3QXs8P9cGI7PA8wDPvY/Sz7w//zbJwEfAoeAfwO2UMc6gGc7G3glnJ/HH/dO/8/u1v/3w/FvzR93LrDV//f2IpAw2GeRKRuEEGKMGc1VPUIIIQKQxC+EEGOMJH4hhBhjJPELIcQYI4lfCCHGGEn8QgSBUurs1lkuhRhpJPELIcQYI4lfjGlKqTX+ufV3KKX+6p98rUEpdbd/rv11SqkU/7G5SqnNSqldSqkXWudAV0pNUUq97Z+ff7tSarL/8tEd5lF/0j96GaXUr/zrEuxSSv0uRI8uxjBJ/GLMUkrNBD4DLNfGhGte4PNAFLBVaz0beBf4sf+Ux4Hvaq3nAXkdtj8J3KeN+fnPwBhtDcYMpLdhrAcxCViulEoCLgdm+6/z82A+oxCBSOIXY9kK4DTgI/8UyyswErQPeNp/zBPAJ5RScUC81vpd//bHgLP8c8Jkaq1fANBaN2utm/zHfKi1LtRa+zCmqMgGaoFm4CGl1BVA67FCDBtJ/GIsU8Bj2lilKVdrPV1rfVeA4wY6r4mrw2svxqImHoyZIp8FVgP/HeC1hRgwSfxiLFsHfFoplQpta7JOxPj/onVWymuA/2mta4FqpdSZ/u3XAu9qreuBQqXUZf5r2JRSju5u6F+PIE5r/RpwO8ZSekIMK0vvhwgxOmmt9yilfoixUpMJYxbUGzEWu1ji31eG0Q4AxvS3D/gT+xHgi/7t1wJ/VUr91H+Nq3q4bQzwklLKjvGN45tD/FhC9Epm5xTiFEqpBq11dKjjECJYpKpHCCHGGCnxCyHEGCMlfiGEGGMk8QshxBgjiV8IIcYYSfxCCDHGSOIXQogx5v8DaYeB2qfQUtIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_mse, label='train_mse')\n",
    "plt.plot(train_msel, label='train_msel')\n",
    "plt.xlabel('epochs')\n",
    "plt.ylabel('MSE')\n",
    "plt.legend(loc = 1)\n",
    "\n",
    "# 手动调整模型学习率的第一个模型在最终模型表现上比第二个模型更好\n",
    "# 在同样遍历了60次的情况下，模型一能够收敛至一个更小的MSE。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#———————————————————————————学习率为0.01———————————————————————————\n",
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 实例化模型  \n",
    "tanh_model = net_class2(act_fun=torch.tanh, in_features=5, BN_model='pre')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 创建用于保存记录结果的空列表容器\n",
    "train_mses = []\n",
    "test_mses = []\n",
    "\n",
    "# 创建可以捕捉手动输入数据的模型训练流程\n",
    "while input(\"Do you want to continue the iteration? [y/n]\") == \"y\":    # 询问是否继续迭代\n",
    "    epochs = int(input(\"Number of epochs:\"))                           # 下一轮迭代遍历几次数据\n",
    "    lr = float(input(\"Update learning rate：\"))                        # 设置下一轮迭代的学习率\n",
    "    train_l, test_l = fit_rec(net = tanh_model, \n",
    "                              criterion = nn.MSELoss(), \n",
    "                              optimizer = optim.SGD(tanh_model.parameters(), lr = lr), \n",
    "                              train_data = train_loader,\n",
    "                              test_data = test_loader,\n",
    "                              epochs = epochs, \n",
    "                              cla = False, \n",
    "                              eva = mse_cal)\n",
    "    train_mses.extend(train_l)\n",
    "    test_mses.extend(test_l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403302260>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f403302080>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'epochs')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'MSE')"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f4033026e0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9OElEQVR4nO3dd3hUVfrA8e+bOuk9ARIg9I6UiCigKKKICtgbyKor629du66ubVdXV9TdtaysHcGyNiwoIgoIFkAgoYZeAyEQQkgnfc7vjzuBRNKTSZjM+3meeWbmzi3nDuGdc8895z1ijEEppZT78GjtAiillGpZGviVUsrNaOBXSik3o4FfKaXcjAZ+pZRyM16tXYD6iIyMNPHx8a1dDKWUcilJSUlHjDFRv13uEoE/Pj6exMTE1i6GUkq5FBFJqW65NvUopZSb0cCvlFJuRgO/Ukq5GZdo41dKuabS0lJSU1MpKipq7aK0aTabjbi4OLy9veu1vgZ+pZTTpKamEhQURHx8PCLS2sVpk4wxZGZmkpqaSpcuXeq1jTb1KKWcpqioiIiICA36TiQiRERENOiqSgO/UsqpNOg7X0O/47Yd+Nd/DKvfbu1SKKXUKaVtB/7NX0LizNYuhVJKnVLaduAPiIT8w61dCqVUK8nOzua///1vg7cbP3482dnZzV+gU0QbD/zRcOwI2O2tXRKlVCuoKfCXlZXVut38+fMJDQ11UqlaX9vuzhkQBcYOhVkQENHapVHKrT3x9SY2p+U26z77dgjmr5f2q/Hzhx56iF27djFo0CC8vb2x2WyEhYWxdetWtm/fzqRJk9i/fz9FRUXcddddTJs2DTiRHyw/P5+LLrqIkSNHsnz5cmJjY5k7dy5+fn7VHm/06NEMHjyYn3/+mYKCAt59912eeeYZNm7cyDXXXMNTTz1FQUEBV199NampqZSXl/PYY49xzTXXkJSUxL333kt+fj6RkZHMmjWL9u3bN+v3VaGN1/gjreeCjNYth1KqVUyfPp1u3bqxbt06nn/+edasWcNLL73E9u3bAZg5cyZJSUkkJiby8ssvk5mZedI+duzYwe23386mTZsIDQ3ls88+q/WYPj4+JCYmcttttzFx4kRmzJhBcnIys2bNIjMzkwULFtChQwfWr19PcnIy48aNo7S0lDvuuIM5c+aQlJTEzTffzCOPPOKU7wTcocYPjsDfu1WLopS7q61m3lKGDRtWZZDTyy+/zBdffAHA/v372bFjBxERVVsHunTpwqBBgwAYOnQoe/furfUYEyZMAGDAgAH069fveK29a9eu7N+/nwEDBnDffffx4IMPcskllzBq1CiSk5NJTk5m7NixAJSXlzuttg9tPfAHRlvPBXqDVykFAQEBx18vXbqURYsWsWLFCvz9/Rk9enS1g6B8fX2Pv/b09KSwsLDWY1Ss7+HhUWVbDw8PysrK6NmzJ2vWrGH+/Pk8+uijjBkzhssuu4x+/fqxYsWKpp5ivbTxpp6KGv+R1i2HUqpVBAUFkZeXV+1nOTk5hIWF4e/vz9atW/n1119bpExpaWn4+/szefJkHnjgAdasWUOvXr3IyMg4HvhLS0vZtGmT08rQtmv8fmEgHtrGr5SbioiIYMSIEfTv3x8/Pz9iYmKOfzZu3Dhee+01+vTpQ69evRg+fHiLlGnjxo088MADeHh44O3tzauvvoqPjw9z5szhzjvvJCcnh7KyMu6++2769XNO85gYY5yy4+aUkJBgGj0D1/PdoffFcOlLzVsopVSdtmzZQp8+fVq7GG6huu9aRJKMMQm/XbdtN/WA1dyjTT1KKXWcUwO/iISKyBwR2SoiW0TkTBEJF5GFIrLD8RzmzDIQEKWjd5VSzer2229n0KBBVR7vvPNOaxer3pzdxv8SsMAYc6WI+AD+wMPAYmPMdBF5CHgIeNBpJQiIggNJTtu9Usr9zJgxo7WL0CROq/GLSAhwNvA2gDGmxBiTDUwEZjtWmw1MclYZAG3qUUqp33BmU08XIAN4R0TWishbIhIAxBhjDjrWOQTEVLexiEwTkUQRSczIaEKvnIBIKMmD0tr73iqllLtwZuD3AoYArxpjBgMFWM06xxmrS1G13YqMMW8YYxKMMQlRUVGNL4X25VdKqSqcGfhTgVRjzErH+zlYPwTpItIewPHs3DuvOnpXKaWqcFrgN8YcAvaLSC/HojHAZuArYKpj2VRgrrPKAGiNXyk3pvn4q+fsfvx3AB+IyAZgEPAPYDowVkR2AOc73juPZuhUym1pPv7qObU7pzFmHXDSqDGs2n/LqJKhUynVar59CA5tbN59thsAF9Vcd2wL+fhffvllXnvtNby8vOjbty8fffRRk7+2tp2rB8AnALwDIF8Dv1LuZvr06SQnJ7Nu3TqWLl3KxRdfTHJy8vHUzDNnziQ8PJzCwkJOP/10rrjiipPSMu/YsYMPP/yQN998k6uvvprPPvuMyZMn13jMinz8L730EhMnTiQpKYnw8HC6devGPffcw9KlS+nQoQPffPMNYCWLq8jHP3fuXKKiovj444955JFHmDlzJtOnT2fPnj34+vo2W/NT2w/8YDX3aI1fqdZVS828pbhiPv6BAwdyww03MGnSJCZNmtTUrwBwm8AfpYFfKeWS+fi/+eYbfvrpJ77++muefvppNm7ciJdX00J320/SBjp6Vyk35er5+O12O/v37+fcc8/l2WefJScnh/z8/CaXwU1q/JGQtra1S6GUamGuno+/Z8+eTJ48mZycHIwx3Hnnnc3S26jt5+MHWPwk/PIiPHYEPNzjIkepU4Hm4285mo//twKiwJRDUXZrl0QppVqdmzT1VOrL7x/eumVRSrm822+/nWXLllVZdtddd3HTTTe1Uokaxk0Cf6XRu1G9al9XKdWsjDGISGsXo1mdavn4G9pk7z5NPaBdOpVqYTabjczMzAYHJlV/xhgyMzOx2Wz13sZNavyODJ06elepFhUXF0dqaipNmlND1clmsxEXF1fv9d0j8PuHA6I1fqVamLe3d5WRsurU4B5NPR6e4B+hgV8ppXCXwA+atkEppRzcJ/AHatoGpZQCdwr8AVE6/aJSSuF2gV9r/Eop5UaBPxKKc6H05LSrSinlTtwo8DsGcR3TWr9Syr25UeB3DOLSnj1KKTfnRoHfUePX0btKKTfnRoG/UqI2pZRyY24U+DVRm1JKgTsFfp8A8PLTwK+UcnvuE/hFHKN3NfArpdyb+wR+0Hw9SimFBn6llHI7bhb4IzVtg1LK7Tl1IhYR2QvkAeVAmTEmQUTCgY+BeGAvcLUxJsuZ5TguINqq8RtjtfkrpZQbaoka/7nGmEHGmATH+4eAxcaYHsBix/uWERAF9jIobJnfGaWUOhW1RlPPRGC24/VsYFKLHfl4X35t7lFKuS9nB34DfC8iSSIyzbEsxhhz0PH6EBDj5DKcoKN3lVLK6ZOtjzTGHBCRaGChiGyt/KExxoiIqW5Dxw/FNIBOnTo1T2l09K5SSjm3xm+MOeB4Pgx8AQwD0kWkPYDjudppsYwxbxhjEowxCVFRUc1ToEDN0KmUUk4L/CISICJBFa+BC4Bk4CtgqmO1qcBcZ5XhJH7hgGjgV0q5NWc29cQAX4jVbdIL+J8xZoGIrAY+EZFbgBTgaieWoSpPL/AP18CvlHJrTgv8xpjdwGnVLM8ExjjruHXS0btKKTfnXiN3QSddV0q5PTcN/FrjV0q5L/cM/PnVdiRSSim34H6BPyQWinOhMLu1S6KUUq3C/QJ/aGfrOTuldcuhlFKtxP0Cf5gj8Gdp4FdKuSc3DPzx1rPW+JVSbsr9Ar9fGPiGaI1fKeW23C/wA4R10hq/UsptuWfgD+2sNX6llNtyz8AfFg/Z+6wpGJVSys24Z+AP7QxlhTqQSynlltwz8IdpX36llPtyz8Afqn35lVLuy00Dv2Mqx+y9rVoMpZRqDe4Z+H38ISBaa/xKKbfknoEfrHZ+beNXSrkh9w382pdfKeWm3Dfwh3WGnFQoL2vtkiilVIty38Af2hlMOeQeaO2SKKVUi3LfwK99+ZVSbsp9A7/25VdKuSn3DfwhcSAeWuNXSrkd9w38nt4QHKc1fqWU23HfwA/al18p5ZY08GuNXynlZtw78IfGQ/4hKC1s7ZIopVSLce/Af7xL5/7WLYdSSrUg9w78odqXXynlfpwe+EXEU0TWisg8x/suIrJSRHaKyMci4uPsMtSoosaftbfViqCUUi2tJWr8dwFbKr1/FnjBGNMdyAJuaYEyVC8wBrxsGviVUm7FqYFfROKAi4G3HO8FOA+Y41hlNjDJmWWolYg1KYs29Sil3Iiza/wvAn8G7I73EUC2MaYiJWYqEFvdhiIyTUQSRSQxIyPDeSXU9MxKKTfjtMAvIpcAh40xSY3Z3hjzhjEmwRiTEBUV1cylq0QHcSml3EytgV9EJld6PeI3n/2pjn2PACaIyF7gI6wmnpeAUBHxcqwTB7RuXuTQzlCUA4XZrVoMpZRqKXXV+O+t9Po/v/ns5to2NMb8xRgTZ4yJB64FfjDG3AAsAa50rDYVmFv/4jqBpmdWSrmZugK/1PC6uvf19SBwr4jsxGrzf7uR+2kemp5ZKeVmvOr43NTwurr3Ne/EmKXAUsfr3cCw+m7rdFrjV0q5mboCf28R2YBVu+/meI3jfVenlqwZfLvxIHlFZVx9eseaV/ILA98QrfErpdxGXYG/T4uUwknmrksjOS2n9sAPEKZ9+ZVS7qPWNn5jTErlB5APDAEiHe9PacO7hpOaVcj+o8dqX1H78iul3Ehd3TnniUh/x+v2QDJWb573RORu5xevac7sFgnAr7sza18xLB6y94Gp920LpZRyWXX16ulijEl2vL4JWGiMuRQ4gzq6c54KesYEEh7gw4r6BP6yQshJbZFyKaVUa6or8JdWej0GmA9gjMnjRBqGU5aIMLxrOL/uysTUVpvvcrb1vH1ByxRMKaVaUV2Bf7+I3CEil2G17S8AEBE/wNvZhWsOZ3aNIC2niP1Ha5llK6oXRPaCza07lkwppVpCXYH/FqAf8DvgGmNMtmP5cOAd5xWr+QzvGgHAit1Hal+x7wRIWQYFdaynlFIurq5ePYeNMbcZYyYaY76vtHyJMeafzi9e03WPDiQy0Jdfdx+tfcU+E8DYYes3LVMwpZRqJbX24xeRr2r73BgzoXmL0/wq2vlXONr5rSkBqtFugHWTd8tXMHRqi5ZRKaVaUl0DuM4E9gMfAitpfH6eVjW8awTzNhxkb+YxukQGVL+SiFXr//VVK1OnX2hLFlEppVpMXW387YCHgf5YKZXHAkeMMT8aY350duGay5ndrHb+Ovvz950I9lLt3aOUatPqauMvN8YsMMZMxbqhuxNYWo9c/KeUrpEBRAX5smJXHYG/wxAIjtXePUqpNq2uph5ExBdr3tzrgHjgZeAL5xareYkIZ3aN4NfddbTze3hAn0sh8R0ozgPfoJYtqFJKtYC6Uja8C6zA6sP/hDHmdGPM340xrTtrViOc2S2Cw3nF7D5SUPuKfSZAeTHs+L729ZRSykXV1cY/GegB3AUsF5FcxyNPRHKdX7zmc7w/f13NPZ2GQ0AUbK61Q5NSSrmsutr4PYwxQY5HcKVHkDEmuKUK2RziI/xpF2yr+wavhyf0vgR2LITSWkb7KqWUi6qrxt9mHM/bs/to7Xl7wBrFW1oAOxe3TOGUUqoFuU3gB6ud/0h+Mbsy8mtfMX4U2EKtwVxKKdXGuFfg72rl56+znd/TG3pfDNsWQFlJC5RMKaVajlsF/o7hfnQIsdWdtweg7yQozoF590BpkdPLppRSLcWtAr+IMLxbBMt2HSG/uKz2lXuMhVH3w7r3YdZ4yHG5HqxKKVUttwr8AFOGdyansJQXF26vfUURGPMYXP0eZGyDN86BlOUtU0illHIitwv8gzuFce3pnXhn+V62HqrHUIS+E+D3i8EWArMvhVVv6ty8SimX5naBH+DPF/Yi2ObFo18kY7fXI4hH94Zbf4Du58P8+63gr5RSLsotA39YgA9/uagPiSlZfLamnhOs20Lg2g+h62hY8jQcq8cNYqWUOgW5ZeAHuHJoHEM7h/HMt1vJPlbPLpseHnDhP6A4F3563rkFVEopJ3HbwO/hITw1qT85haU89922+m8Y0w8GT7aaezJ3Oa+ASinlJE4L/CJiE5FVIrJeRDaJyBOO5V1EZKWI7BSRj0XEx1llqEuf9sFMPTOeD1ftY93+7PpveO6j4OkDCx93WtmUUspZnFnjLwbOM8acBgwCxonIcOBZ4AVjTHcgC7jFiWWo0z1jexAV6MujX26kvD43egGCYmDUPbB1Huxd5twCKqVUM3Na4DeWiqQ43o6HAc4D5jiWzwYmOasM9RFk8+bRS/qSfCCXTxP313/D4bdbs3V99zDY7c4roFJKNTOntvGLiKeIrAMOAwuBXUC2MaZi2GwqEFvDttNEJFFEEjMyMpxZTC4d2J6EzmH88/tt5BWV1m8jH38Y81c4uA42furU8imlVHNyauB3zNk7CIgDhgG9G7DtG8aYBGNMQlRUlLOKCFipHB6/tC9H8kuYsaQBN2wHXAUdBsPiJ6DkmPMKqJRSzahFevUYY7KBJcCZQKiIVMz1GwecEklwBsaFcvmQWGb+sod9mfUM4hXdO3MPwLKXnFtApZRqJs7s1RMlIqGO137AWGAL1g/AlY7VpgJznVWGhvrzhb3x9BCe+XZL/TfqfBb0vxJ+/hcc2ui8wimlVDNxZo2/PbBERDYAq4GFxph5wIPAvSKyE4gA3nZiGRqkXYiN/xvdjW+TD9U9RWNl458HvzD48v80f79S6pTnzF49G4wxg40xA40x/Y0xTzqW7zbGDDPGdDfGXGWMKXZWGRrj1lFd6RBi4+/zNte/e6d/OFz6olXj//lfTi2fUko1lduO3K2Jn48nD17Um01pufXP4wPWjF0Dr4Gf/wlp65xWvlOFMYbp325lY2pOaxdFKdVAGvirMeG0DgzuFMrz322re8KWysZNB/9I+PKPUFbPC5kdC+HdifVf/xSRV1zGaz/u4qv1p8S9eaVUA2jgr4aI8OjFfcjIK+bzhtT6/cPh0pfg8Cb48bm61y/Oh6/uhN1L4UBSo8vbGjLyrB+q9FzX+sFSSmngr9HQzuH06xDMh6v2Yxoy8UqvcTDoBvjlhbqD+U/PQV6a9drFUj9UBP5DuTofsVKuRgN/La4d1oktB3PZeKCB7dgX/gMCY2DOzVBwpPp1MrbBihkwaDLEDIC9Pze9wC2oIvAf1sCvlMvRwF+LCad1wObtwUerG5DDB8AvFK5+F/IOwYfXQmkhAP9buY9p7yZi7HaY/wD4BMD5f4P4EbB/lUt1Ba1c42/QFZFSqtVp4K9FiJ834we056t1aRwracBNXoCOp8Plb0JqInx+K9jL+Xp9Gt9vTmf7kvdgz49w3mMQGAXxI6GsENLWOudEnCAj3wr8RaV2cosa+N0opVqVBv46XHt6J/KLy/hmw8GGb9x3Alz4NGz5GvP9o2xKy8GfIqKXPwntBkLCzdZ6nc6ynl2ouaeixg+Qrs09SrkUDfx1OD0+jK6RAXzc0OaeCsP/CMP+gPz6Xy4r/YaHA74irPwI6aOeBg9Pa52ACIjuBymuc4M3I68YD7Fea+BXyrVo4K+DiHDN6R1JTMliR3peY3YA454hvcMY/ur1Ltfb5/Fp+TnMTPlNxtH4EbBvJZTXMy10K8vIK6ZbVCAAh3I08CvlSjTw18PlQ+Lw8pDG1/o9PPmo41/ZYLoitiASe9zFh6v2Vb1vED8SSgtcZtRvRn4x/WNDADicp335lXIlGvjrISrIl7F9Y/h87QGKy8obtY/16SU8Evoccvtqrjx7MLlFZXy5Nu3ECp1HWM8u0M5fbjdk5hfTMcyPED9vrfEr5WI08NfTNad35GhBCYs2H27U9skHcugVGwmBUSR0DqNfh2BmLd9zoitkQCRE9XaJdv6jBSXYjfWD2C7Ypm38SrkYDfz1NKpHFB1CbHy0el+Dt83IK+ZwXjF9OwQD1n2D350Vz/b0fFbsqpT+OX4k7PsVyk/t7pEVPXqignyJDvbVwK+Ui9HAX0+eHsJVCR35ZecR9h9t2DSLm9Kskb/9OoQcX3bpaR0ID/Bh1vK9J1bsPAJK8uHg+uYostNU9OGvqPFr2galXIsG/ga4KiEOAd77NaVB221KywU4XuMHsHl7ct2wjizakn7ihyR+pPV8irfzH6/xB9qICbaRkVdc/7kLlFKtTgN/A8SF+XPJwA68/2sKWQX1T6+wKS2HTuH+hPh5V1k+eXhnROTED0lgNET2POXb+Q/nWTX8yCAfYkJs2A0cydeePUq5Cg38DXT7ud05VlLOO5WbaOqwKS2X/rHBJy1vH+LHuP7t+GjVPg5kW/l8iB8JKStO6Xb+jLxiAn298PfxIibIF9BBXEq5Eg38DdSrXRAX9I1h1rI95BXVPdgqt6iUlMxjVdr3K7t7TA+MgSlvryQzv9jRzp8HhzY0d9GbTUZeMVGOgN8uxAboIC6lXIkG/kb403ndyS0qq1db/+Zq2vcr6xETxFtTEziQVchNs1ZT0GG49cEp3NyTkVdMVKAV+GOCrcCfroO4lHIZGvgbYWBcKGf3jOLtn/dQWFL7gK6KG7v9a6jxA5zRNYIZ1w9hU1out36eij28O+z9pVnL3Jwy8k/U+CMDffEQSNcav1IuQwN/I/3p3O5kFpTw4ara+/VvOpBDdJDv8UBZk/P7xvD8lQNZviuTZaW9MCnLwd64UcLOVrmpx9NDiArSvvxKuRIN/I00rEs4w7qE88ZPu2tN47ApLZd+NTTz/NblQ+J4/JK+fJrZBSnOxSx4yJqX9xRSVFpOXlFZlR8y7cuvlGvRwN8Ed5zXnUO5RXyWdKDaz4tKy9mZkX88mVl93DyyC51HXccHZWOQVW/Af4fDtgXNVeQmqzxqt0JMsI3DOum6Ui5DA38TjOweyWlxIbz6407Kyu0nfb71UB7ldlPvGn+F/xvTmye4lVm9XrOmZ/zwGvjkRshtxGQwzazyqN0KMVrjV8qlaOBvAhHhT+f1YP/RQr5Ye3Ktv7pUDfXh7+PF8K4RvJfWHv7wM5z3qFXrnzEMPvs9rHrTSuvQCn39T4zardTUE2Ijp7CUotJT856EUqoqDfxNNKZ3NKd1DOXJrzezO6Nqe/ymtFyCbV7Ehfk1eL/n9opiV0YB+3LK4OwH4I8roOeFsOdnmH8/vH42TO8Esy5p0bl6KwJ/dKUaf7QO4lLKpWjgbyIPD2HG9YPx8hSmvZdEfvGJWvimAzn06xCCiDR4v+f2igZgyTZHGuiIbnDFW3DfVrh7I1zxNgyeDIc3w4KHm+Vc6iMjrxgRCA/wOb6sYhBXurbzK+USnBb4RaSjiCwRkc0isklE7nIsDxeRhSKyw/Ec5qwytJS4MH9mXD+E3Rn53PfJOux2Q1m5na2H8hrcvl8hPjKALpEBJwJ/BREI7QQDroTxz8FZd8K+5XB4azOcSd0y8ouJCPDBy/PEn07FIC5t51fKNTizxl8G3GeM6QsMB24Xkb7AQ8BiY0wPYLHjvcs7q3skD4/vw3eb0nn1x13syiiguMzeoB49vzW6VxQrdmXWPkhs8GTw8IakWY0+TkNk5BUTGVh1TMLx0bs6iEspl+C0wG+MOWiMWeN4nQdsAWKBicBsx2qzgUnOKkNLu2VkFyac1oF/fr+NV5fuBGh0jR+s5p7iMju/7s6seaWASOg7Adb/D0oLG32s+qo8eKtCsM0Lm7eHtvEr5SJapI1fROKBwcBKIMYYU9Ev8RAQU8M200QkUUQSMzIyWqKYTSYiPHvFQHq3C+bLdWnYvD3oGhXY6P0N6xKOn7fnyc09vzX0JijKgU1fNvpY9VVd4BcRHcSllAtxeuAXkUDgM+BuY0xu5c+MNeFstTN4GGPeMMYkGGMSoqKinF3MZuPn48kbU4YS6u9N/w4heHo0/MZuBZu3JyO6R/LD1sMn5uatTvxIiOgBiTMbfaz6MMZUydNTWbQO4lLKZTg18IuIN1bQ/8AY87ljcbqItHd83h5o3Ozlp7CO4f588ccR/PvqQU3e17m9o0jNKmRXRkHNK4nA0N9B6ipI39TkY9Ykt6iMkjJ7lT78FbTGr5TrcGavHgHeBrYYY/5d6aOvgKmO11OBuc4qQ2vqEhlApwj/Ju9ntKNb59K6mnsGXQ+evpD4TpOPWZPq0jVUaBdiIz23qPYrE6XUKcGZNf4RwBTgPBFZ53iMB6YDY0VkB3C+472qQWyoH71igvhhax2B3z8c+k6EDR9DSS1XB01QW+CPDvKluMxOTmHdk9O4sqLScv74QRJbDubWvbJSpyhn9ur5xRgjxpiBxphBjsd8Y0ymMWaMMaaHMeZ8Y8xRZ5WhrRjdO4rVe4/WPeNXwk1QnAvJn9e+XiNV5OmJrqHGD21/ENealCzmbzzE1+vTWrsoSjWajtx1Aef2iqa03LBsZy3dOgE6nQmRvSDJOc09hx1t+FGBtpM+c5dBXIkpWQBsPJDTyiVRqvE08LuAoZ3DCPL1qtLOb7cbFm5O56Z3VvFVRe1TxKr1H0iykrhl7YWt38CPz8MnU2Hh401K7JaRX4yPpwfBfl4nfdauYhCXGwV+vZ+hXNXJ/4PVKcfb04NRPSNZsu0wRaXlfL7mAG/9vJvdRwrw8hBW7M6kb/tgukcHwmnXwqK/wRujwVRKFR3SCTZ/CTmpcNkb4Nnwf/qKPvzV5R6qaPdvy6N3y+2GNSlZBPp6kX2slNSsQjqGN/0GvlItTWv8LmJ0r2jSc4sZ9vQiHv5iI/6+nrx83WB+/PO5+Hl7ctdHaykps4NfGIx/HhJugUtehFsWwV9S4Z6NMPZJSP4MvvhDo2r+GXnFRNYwhaTN25Mwf2/S85o38BtjWJB8iPX7s2ud6awlbDuUR35xGVclxAHa3KNcl9b4XcR5vaOJCvKlf4dgpp3djeFdw4/XvJ+9YiDT3kviX99v4y/j+8CQG6vfyYi7rKuARX8D8YDLXgMPz3qXISOvmLiwmmu4McE2DuU0783dxVsOc9v7SQD4eHrQp30Qp3UMZVDHUC4e2B5fr/qXv6mSUqx+CDec0Zn3f01h44Ecxg9o32LHV6q5aOB3EZGBvqx+5PxqP7ugXzuuP6MTr/+0m7N7RjGie2TNOxp5jxX8Fz9p3ROY9Gq9g/+R/GIGd6o5mWpMsK3Z2/jnJx8k2ObFM5cPZMOBbNbvz+azpFTeXZHC6r1Heebygc16vNqs3ptFTLAv3aIC6BkTxMZUrfEr16SBv4147OK+rNydyb2frGPBXWcTVilf/klG3QfGwA9/t54n/Ae8T+6pU1lZuZ3MgpJq+/BXiAn2bdb+7aXldhZtTuf8vjFcPLA9Fw+0atfldsPf521m9oq9XDesEwPjQpvtmLVJSskiobN1pTUwLoT5Gw9hjGnUfAtKtSZt428j/Hw8eenawRwtKOGhzzfU3ePk7PvhvMdg4yfw5nlweEutqx8tKMGY6gdvVWgXbONIfnG18w83xopdmeQWlXFR/6rNKZ4ewr0X9CQiwJfH527Cbnd+75q07EIOZBcytLN1xdM/NoScwlL2H3V+RlSlmpsG/jakf2wID1zYi+82pfOnD9fyt6828dS8zTwzfwvPLdjKJ6v3V50X9+z74YY5UHDY6gW06k3rCqAah6uZa/e3ooNt2A0cyS+ps6w5haVsTqv96uDb5EME+HgyqsfJTVfBNm8euqg36/Zn89ma1DqP11QV3TgT4q3APzA2FHDiDd6UFfB8D8jc5Zz9K7emgb+N+f3Irlw2OJblO4/w2ZpUPly1j1nL9/L6T7v582cbGPXcEl7/cdeJKSJ7jIX/Ww7xo6y5fD+8DgqOnLTf46N2g2uv8UPdg7hKyuzc+PZKJrzyC6lZx6pdp9xuWLj5EFd1LcX2+VQ4lHzSOpcPjmVwp1CeXbCV3LpGNTdR0t6j+Pt40re9Nb9Cz3aBeHsKGw5kO+eAy160fpBXvemc/Su3poG/jfHwEF64ZhBrH7+AjX+7kE1PjmPbUxex8+mL+N/vz6BXTBDPfLuVs55ZzL++38bRghIIjIYbPoVxz8KuxfDfMyFpdpUunxn1qPGfSNtQe+B/bsFW1qfmYDeGN3/aXe06q/cepSw/k/uPPAJbvoYProLcqmkSPDyEJyf0J7OghBcX7qjX99NYiSlZDOoYenzKSV8vT3q3CybZGTX+o3tg+3fg5Qfr/ue03EvKfWngdxMiwlndI3n/92cw9/YRnNktgv/8sJNRz/7AjCU7KSqzw/Db4NYlENYZvr4T/jvcmtzFmEqB3wdSE2H+n+Hf/axRwQ4VVwO1Bf5Fm9N565c9TD2zM1cOjeOj1fs5kn9yF9CFG/bxhu+LBBQehEtegOI8+OBq67mSAXEhXHt6J2av2Mv29LyT9tMc8ovL2HIwl4TOVXs09Y8NYWOqE0bwrn7L6mk18RUozrHGXijVjDTwu6HTOoby+pQEFt5zNmd1j+T577Yx9oUfWZB8EBPTD25ZCNf+zwo+n06FN0YTte9b/mz7HNtrp8NbY6w5fv1CYclTsPw/AEQG+OLpITUG/gPZhdz36Xr6dQjmL+P78IdzulFSbuedZXuqrGcvt5Ow4a8Mky3IpFch4Wa4ehYc3gyf3nTS4LMHLuxFoK8Xf527ySlpFNbuy8JuICE+vMrygXEh5BaVse9o9c1VjVJyDNa+B30uhf5XQHRf64dA00OoZqSB3431iAnizRsTeP+WM/Dz9uS299dw3Zu/svlgHvS+2Gr7n/QqHMvk6j2PchufQUgcTHgF7t8Of/gJ+l0G3z8KiTPx8LCmYJy/8RBLtlWdNay03M6dH66l3G6Ycf0QbN6edIsK5KL+7Xh3RUqVzKPp857kIvuPbOp9Bwy40lrY/Xy4+F+wcyF8+0CVQBge4MP9F/Rkxe5MXvtxd7OP8E3cm4WHwOBOoVWWD4gNAWBDc/bn3/iJNY3msGmO3Es3W3mXDqxpvmMot6eBXzGyRyTz7xzF3yf2Y+uhPC595RdmLduDEQ9rgpc7kngm7An+GP0eTP0ahkyxavsenlben57jYN69sP5j/jahHyVldm56ZzWTZizjp7WbMcte5of3p7M2JZN/XD6A+MiA48f+4+ju5BWV8f6v+6wF6z+m/doX+Nx+NnETHq9a0ISbYMTd1hSTjquMCtef0ZmzukXw7IKtjJi+hJcW7SAzv9iakSxtbZ3fwcbUHJ6at5nCkpN/NBJTjtKrXTBBNu8qy3vGBOHj6dF87fzGWDdzYwZYmVYBBl4DPoGQ+HbzHEOdmsrLIC+9xQ6ngV8B4OXpwZQz41l6/2jO7RXN377ezCNfJlNabgcvXxaWDsIrNLaaDX3gqtnQZRR8+X+MlVUsue8c3h5dzB1ZzzD8y5HIwse4cM+zfB/5AhO6Vh0l3D82hFE9Innn5x2UrpqJ+epPrPHoz7zODxHiX80gtDF/ta4yFj5Wpfupp4fwwe/P4P1bzmBAbDAvLNrO3599itLXRltdVb+4rcb/WDsP5zFl5kre+mUP93+6vsq4gLJyO2v3ZXN6/Mkjln28rBQSNdb4DyXXOT6iin0rID0Zht1q1fYBbMEw8Gqrnf+YTl3RZn33MPxnCBTUkXq9mWjgV1WE+vvwxpSh3HZON/63ch83vr2KrIKS45k5q+Vtg2s/hNih8OlN+Lw+nDG/3sQY743s7XodN/r9h5cC7qRb0WZ4bQTs+qHK5g/2Oco7pX/Ge/49HIsaxE3H7uSCAR2rP5aHh9X81ONCq/vpnJuhyBoPICKM7BHJO787naRzNvCi58tssHfhTTMR+8Y58J+hsGIGlJ9oVkrLLmTK26vw8vDg1lFd+GbjQV5ctP3451sP5XGspPz4wK3f6h8bQnJazsmDyDK2w8wL4bWRsOxlsNdjUNvK18EWCgOuqro84RYoK7J6+Ki2J3OXdUVXkm/d32kBGvjVSTw8hIcu6s2/rjqNpJQsJs5YRl5xWa2jdvENtLqExg4F32CY+F/k3q30nDqD2X+ewp/uexKZtgT8I+C9y61cQUd3wydT6f/9tUR7HeNx7/t4pfPL5EkgY/vG1Hwsbz+47iOr9r95rlWjP7jB+qy8FL66g4iV06H/lcTe+T3zoqYxpmg6KYEDrZrVayNh949kFZRw48xV5BeV8e7Nw3h4fB+uTojj5R92MnfdAcDqVgon39itMDAuhLyiMlIq3+AtzoePJ4OXL/S4wLo6+d9VkJ9R8znlplndVodMAZ/fJMJr1x86DreCQ31+QAB2LDrxnahT2w9PgacPtBtgNWPanZ+FVgO/qtEVQ+P4cNoZHCuxetHU1ocfsNr9b/kObl0Mg284HsBEBE8Pgeg+VnfRIVPg53/By4Ot/uqj/8KGSYt4N28ob/2yh2Fdwomo61geHjDqXvjdPCg9Bm+db9WYP7jKqjWNuh8uf5N2EaF8/Icz6T9gKOek/ZG34v6BvbQQ3p1A0n+uJ/voYd6cmkDfDsGICE9NGsAZXcJ5YM4GklKySEzJon2IjdhQP+u45aXWzVeH/o4bvMdH8BpjdYXN3AFXzrR6R138b9jzs+NqZ0n155M400qel3BL9Z+f/nvrh3LP0tq/F7sdFj0BH1wBsy6xrjxUy9j+PWTva9g2aWth0+cw/I8w8l7IToGdi5xTvkrEFWYRSkhIMImJia1dDLeVmnWMV37YyZ1jetChIgA2VfJnkLLcyhYaEofdbhj30k9sT8/niQn9mHpWfP33VXAEPr/VakLy8IJLX4LBk6usYozhlR928q+F2xkW58eU4o+4KPdTym3h+E74lzVRvaNdPaughNte+Zyzin9huH0dcT4FxNqKoTAbSh2DqXpfAuP/SWlADP3++h2/Oyueh8f3gV9fgwUPwpjHrWR4FdI3WV1Rj2yH066DDoMhsof18I+EF/tbV0vXf1z9OZYVw7/7QqfhcO0H1a9Tcgy+vM26Chp4jfV9+ATCrT+Af/VXLNUqL7Nmb4vsXv9t3N2mL62uz4ExcONcq5JTH+9OtK7M7loH3v7wQn+r5j95TrMUS0SSjDEJJy3XwK9OFQuSD/GXzzfw3d1nEx1ce7bQk9jtsGY2RPWGzmfWcoyD3PPxegpLy3n9fC8u3PW01V2y18VW7qKUZbDpC2v6SmCTvTNBMV3pFNvBmuTGFmq1xa58DTx94cKnmbi8C34+nnw0zgNmjYfuYx3jIH5zQV1yzOr6uuETKKk02MzTB8pLYPLn0H1Mzee46G+w7CW45gPoOrpqk1DeISvdRtpauODvcOafYP8qmH0pxCXAlC+tG/F1SVsLX99lfScDr4GLnrOu5OpijHUfouSYdQUW1A48veveri3I3AWvnwMRXa0OBOUlcOOX0P602rfb9QO8dxlc+A8483Zr2ZJn4Mdn4c41EN61yUXTwK+Uw470PPYdPcaYPjFW7fbXGbDkH1bgAus/bL/LWB0wmvsWZvPeLcPoHBFQdSdHdlpNOinL2BmYwON5l/FB8CuIly9M+5EiryC+2XAQL0/hov7t8fGq9CNgjBWoM3fAEcfD0wvOf/LkH4vKsvfD62dD4VHrx6LjGdYPQFRv+PZBKMyCK96C3uNPbLPhE+tqaPBka/xFTSmki/Os72DlaxAQBb3Gw5p3rRrsxFdO/kEqyIS178La961zKT1WdarPsC5w4dPWfhqSttoYaz8NmCCoVZUWwttjrSlN//CzFfTfnWh1OJj8GXQ8vfrt7HZ4czQcy4I7Eq37QQC5B62rvzNus76/JtLAr1Rtju6G3UuhyzkQ0a1+29jtkPQOpd89hndZAXZPGwWT5zN7TzCzlqccT0URFeTLlOGduf6MTkTWde+iLiUFVrfP3Uth11JI32gtD+oA139UfS3zh6fhp+esqTdH3HXy51vnw/wHIPeANWBszONWLT9trdUNNmOrtXzs3+HoLlj5BiTPsX4o40dBu4HW1Ye3P/gEWLO7rX7L2q7raBg3/eSmj/JSOLTB6vJ6dDdk7bFyFGXttZq1+l9udWuNHdq076sw26pBlxXDWX9qllp0FV/fZY1iv/5T6HmBtSx7H8yeAAUZVtNd/MiTt0v+zOqRNuk1GHRd1c8+mWr9+9675eQb/Q2kgV8pJ9mxYyvr332AfZHn8FZmf46VlHNOzyj+cHZXyuyGmcv2sHRbBj5eHkw8rQPXDuvIgNjQqlcBv5GZX8yeIwV0CvevcYJ7wOopdCARYhMgMKr6dex2mHOT1fZ/9gNWwM7ed+Jx7AhE96Xs4hf4uagrX649wLZDedw6qiuXD4xAljwNy1+xemsV51gBfuA11ujimL7VH7O81LphveRpq5fT6bdYo6/3r4L9K62mtFJHTygPbwjthAnrwgGPdmTl5tM3cyGeZQXQYYh1nH6X1TlZ0El2LISv7oT8Q9Yx7GXWmIhR91n3Vppq/cfwxTTrpuz5f636We5Bq+afnQLnPWqdR0xfq7mwrARmDLO+x9t+PvnqZu8vMOti6wptyJQmFVEDv1JOUlZuZ9CTCykqLWfCaR249eyu9HGkb66w83A+s5fvZU5SKoWl5fh6eTAgNoShncMY3CmM9iE2NqRms2ZfNmv2ZZGSeaJ7aKi/Nz1jgugVE0TPdkF0jwqkW3QAUYEn/yAUlZaTknmM3Rn5RAX5MqRTGB4eYrW9vzsBUldbzUShnaxgG9qZVFsP3jk2kq82HuZIfgkhft60C7axLT2PYfHh/H1Sf3oVJ8OKV6DzWdZobr+ap+CsoiATlv7jRK8l8bRuXnYabjVVxQ7hqFcMn65J44OV+47nPQrkGDcHreRGz4VEFu3F+Ecgg6dYVx5hnWs/ZmE2fP+I1QwV1Rsm/ReCY63xFIkzMeXFHI2/hNR+tzFw6FmNm0Ht8FZ481zrJv2NX4GnF0cLSvD38cTm7QjkBUesXmZpldJtBMda2XDT1sL1n0DPC0/etzFWhlwvH5j2Y8Oayn5DA79STrQjPY9AmxftQ2rv9ZRzrJTlu46QlJLFmn1ZJB/IpaTSjGWRgb4M6RTKkM5h9IgOJDWrkG3peWw7lMf2Q3nkVcyjAAT5etE1OpCukQHkFJayKyOf/UePUXksWXSQLxf1b8f4Ae1J6BSCZ+ERDtuDWbb7KD/vOMKynUdIzy3Gx8uDMb2jmTQ4ltG9ovD28ODTpP088+1W8orKuGVkF+4a04MA37pna80rKsXb0+NEAATrnkjeQegwmFIvf/YeKWB7ej6Lt6Yzb8NBSsrsDOsSzpThVuqNJdsy+Hp9Gr/szGAYm/ij32JGlK9CMNDzQuT0W6HbedY9EWOs+xs5+62R0ouesGr5I+6G0Q+Bly/FZeUs35XJsrWb6bhtJlfaFxAgxWzz6UvQWbfQ4azrT25WMcZqu09PhmOZ1g9KUbb1vHOh1ex22y/keEbw74XbeO/XFMIDfPjdWfFMHt6ZUH8fax+5aVaCwfRN1uPwZojsaXX3rSmor3rTGqD4+8XWzflG0sCv1CmouKyc5AO5pOcWMSA2hLgwvxproMYYDuYUsTujgF0Z+ccfezIKCPH3oVtUAN2iAunm+DHYfaSA+RsOsmTbYYrL7EQG+hIe4M329HwAwvy9Oat7JOf0iOLC/u0I8Tu5F87RghKeW7CVj1bvJyrIl97tggj09bIeNuv5aEEJadmFpGUXkZZdePzHKdjmRbsQGzHBNqKDbBSXlbM9PY89RwooLbfiTqCvF5cNjmXy8M70ahdU7fEXJB/im41p7Nu9g6s9FnG91xIiyOGYfyzGy4ZvwUG8yk9cIR3178rc+EfZ5tmDnMJScgpL2ZiaQ15xGYG+XpzbO5pLuvsQsWMOkds/JJ6DFHkEYAZchV/vsdaPx4Ek61FwuGqBxANsIRAYg338v/gkoxPPfbeN7GMlXHN6Rw7mFLF0Wwb+Pp5ce3onbhnV5cQYkGqUltv5ZccRvlx3gL1HChjeNYJzekWR0M4bn5f6WckSL3+99j+iWmjgV8pNFRSXsWTbYb7deIi84jJGdItgRPdI+rYPtpqB6iEpJYvXftzFkfxi8ovKyC8us55Lygjz96FDqI32IX7EhvrRPsRGmd2QnlvkeBSTnluEl6fQM9pqruoZE0iP6CC6RwdWvTKoRW5RKT9tz2DpplS8tn3NBeU/UYw3aSaSAyaSAyaCNBPJFtMJL28fQvy8CbZ5E+LnTbeoQMb1b8dZ3SPw9TpxvJyCEr6Y+ymhW/7HOI+V2KQUg1Ac2g2JHYpPpwSkwyCrecYWSolXIHnF5ew8nM8/5m9hfWoOCZ3DeGJiP/p1sAbzbTmYy5s/7ear9WkYIKFzGN2jA+kWFUhXx49zRn4xc9ceYN6Gg2QWWM1rPaIDWZ+aTWm5IcDHk/+E/o9z8uaTf/sGQiI7NPSfHWiFwC8iM4FLgMPGmP6OZeHAx0A8sBe42hiTVde+NPArdWoyxjSujbyJysrtJKflUm634+vliZ+jbd3m5UGgzatKcK+PXRn5vPjVKtJ2rmO76UgeVrOPn7cn0cG+HCspJ6+olKLSE81yUUG+PDy+N5MGxVb7HRzILmT28r0k7j3KrowCcgqrTg/q6+XB+X1imDioA6N7RePj5UF+cRnLdx5h6fYM9m5Zw5TC9+j/u5fp2K2Gm+h1aI3AfzaQD7xbKfA/Bxw1xkwXkYeAMGPMg3XtSwO/UqolZB8rITWrkAPZhRxwPGfkFRPg60mQzZtgmxdBNm9C/b05r3f0Sam6a2KM4WhBCbsyCth5OB+btwdj+8bUur0xht1HCugWFdjo82mVph4RiQfmVQr824DRxpiDItIeWGqM6VXXfjTwK6VUw9UU+Fs6SVuMMeag4/UhoMYUjCIyTUQSRSQxI6OWrIZKKaUapNWycxrrUqPGyw1jzBvGmARjTEJUVA0DU5RSSjVYSwf+dEcTD47nw3Wsr5RSqpm1dOD/CpjqeD0VmNvCx1dKKbfntMAvIh8CK4BeIpIqIrcA04GxIrIDON/xXimlVAuqe/x1Ixljrqvho1oSjiullHI2nXpRKaXcjAZ+pZRyMy6Rq0dEMoCURm4eCRxpxuK0trZ0Pm3pXKBtnU9bOhdw3/PpbIw5qT+8SwT+phCRxOpGrrmqtnQ+belcoG2dT1s6F9Dz+S1t6lFKKTejgV8ppdyMOwT+N1q7AM2sLZ1PWzoXaFvn05bOBfR8qmjzbfxKKaWqcocav1JKqUo08CullJtp04FfRMaJyDYR2emY8ctliMhMETksIsmVloWLyEIR2eF4DmvNMjaEiHQUkSUisllENonIXY7lLndOImITkVUist5xLk84lncRkZWOv7ePRcSntcvaECLiKSJrRWSe471Lno+I7BWRjSKyTkQSHctc7u+sgoiEisgcEdkqIltE5Mymnk+bDfwi4gnMAC4C+gLXiUjjJq5sHbOAcb9Z9hCw2BjTA1jseO8qyoD7jDF9geHA7Y5/D1c8p2LgPGPMacAgYJyIDAeeBV4wxnQHsoBbWq+IjXIXsKXSe1c+n3ONMYMq9XV3xb+zCi8BC4wxvYHTsP6NmnY+xpg2+QDOBL6r9P4vwF9au1wNPId4ILnS+21Ae8fr9sC21i5jE85tLjDW1c8J8AfWAGdgjaT0ciyv8vd3qj+AOEcAOQ+YB4irng+wF4j8zTKX/DsDQoA9ODriNNf5tNkaPxAL7K/0PtWxzJXVe+rKU5ljLubBwEpc9JwczSLrsCYTWgjsArKNMWWOVVzt7+1F4M+A3fE+Atc9HwN8LyJJIjLNscwl/86ALkAG8I6jGe4tEQmgiefTlgN/m2asn3qX64srIoHAZ8Ddxpjcyp+50jkZY8qNMYOwasrDgN6tW6LGE5FLgMPGmKTWLkszGWmMGYLVzHu7iJxd+UNX+jvDSp0/BHjVGDMYKOA3zTqNOZ+2HPgPAB0rvY9zLHNlLj11pYh4YwX9D4wxnzsWu/Q5GWOygSVYTSGhIlIxx4Ur/b2NACaIyF7gI6zmnpdw0fMxxhxwPB8GvsD6YXbVv7NUINUYs9Lxfg7WD0GTzqctB/7VQA9HzwQf4FqsqR9dmctOXSkiArwNbDHG/LvSRy53TiISJSKhjtd+WPcqtmD9AFzpWM0lzgXAGPMXY0ycMSYe6//JD8aYG3DB8xGRABEJqngNXAAk44J/ZwDGmEPAfhHp5Vg0BthMU8+ntW9eOPnGyHhgO1b76yOtXZ4Glv1D4CBQivWrfwtWu+tiYAewCAhv7XI24HxGYl2ObgDWOR7jXfGcgIHAWse5JAOPO5Z3BVYBO4FPAd/WLmsjzm00MM9Vz8dR5vWOx6aK//eu+HdW6ZwGAYmOv7cvgbCmno+mbFBKKTfTlpt6lFJKVUMDv1JKuRkN/Eop5WY08CullJvRwK+UUm5GA79STiAioyuyXCp1qtHAr5RSbkYDv3JrIjLZkVt/nYi87ki+li8iLzhy7S8WkSjHuoNE5FcR2SAiX1TkQBeR7iKyyJGff42IdHPsPrBSHvUPHKOXEZHpjnkJNojIP1vp1JUb08Cv3JaI9AGuAUYYK+FaOXADEAAkGmP6AT8Cf3Vs8i7woDFmILCx0vIPgBnGys9/FtaIa7AykN6NNR9EV2CEiEQAlwH9HPt5ypnnqFR1NPArdzYGGAqsdqRYHoMVoO3Ax4513gdGikgIEGqM+dGxfDZwtiMvTKwx5gsAY0yRMeaYY51VxphUY4wdK0VFPJADFAFvi8jlQMW6SrUYDfzKnQkw21gzNQ0yxvQyxvytmvUam9ekuNLrcqxJTcqwskXOAS4BFjRy30o1mgZ+5c4WA1eKSDQcn5e1M9b/i4qslNcDvxhjcoAsERnlWD4F+NEYkwekisgkxz58RcS/pgM65iMIMcbMB+7BmkpPqRblVfcqSrVNxpjNIvIo1mxNHliZUG/HmuximOOzw1j3AcBKf/uaI7DvBm5yLJ8CvC4iTzr2cVUthw0C5oqIDeuK495mPi2l6qTZOZX6DRHJN8YEtnY5lHIWbepRSik3ozV+pZRyM1rjV0opN6OBXyml3IwGfqWUcjMa+JVSys1o4FdKKTfz/1J83NM4A6vyAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_mse, label='train_mse')\n",
    "plt.plot(train_mses, label='train_mses')\n",
    "plt.xlabel('epochs')\n",
    "plt.ylabel('MSE')\n",
    "plt.legend(loc = 1)\n",
    "\n",
    "# 我们能够发现，第一个模型收敛速度明显快于第二个模型，\n",
    "# 据此也能说明手动调整模型学习率的有效性。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 配合tensorboard实现方法\n",
    "\n",
    "\n",
    "tensorboard中writer对象却能够即时记录模型中间结果，并且同步呈现在tensorboard的操作面板上。借此功能，我们便能将在某个cell尚未运行结束之前看到当前阶段的运行结果。要完成该过程，则需要在此前代码基础之上进行修改。\n",
    "\n",
    "- `writer = SummaryWriter(log_dir='l3')`  实例化 \n",
    "  - log_dir (string): Save directory location\n",
    "- `writer.add_scalar('变量名', y, x)`  添加数据\n",
    "  - "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;31mInit signature:\u001b[0m\n",
      "\u001b[0mSummaryWriter\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mlog_dir\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mcomment\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m''\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mpurge_step\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mmax_queue\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mflush_secs\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m120\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m    \u001b[0mfilename_suffix\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;34m''\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\n",
      "\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mDocstring:\u001b[0m     \n",
      "Writes entries directly to event files in the log_dir to be\n",
      "consumed by TensorBoard.\n",
      "\n",
      "The `SummaryWriter` class provides a high-level API to create an event file\n",
      "in a given directory and add summaries and events to it. The class updates the\n",
      "file contents asynchronously. This allows a training program to call methods\n",
      "to add data to the file directly from the training loop, without slowing down\n",
      "training.\n",
      "\u001b[1;31mInit docstring:\u001b[0m\n",
      "Creates a `SummaryWriter` that will write out events and summaries\n",
      "to the event file.\n",
      "\n",
      "Args:\n",
      "    log_dir (string): Save directory location. Default is\n",
      "      runs/**CURRENT_DATETIME_HOSTNAME**, which changes after each run.\n",
      "      Use hierarchical folder structure to compare\n",
      "      between runs easily. e.g. pass in 'runs/exp1', 'runs/exp2', etc.\n",
      "      for each new experiment to compare across them.\n",
      "    comment (string): Comment log_dir suffix appended to the default\n",
      "      ``log_dir``. If ``log_dir`` is assigned, this argument has no effect.\n",
      "    purge_step (int):\n",
      "      When logging crashes at step :math:`T+X` and restarts at step :math:`T`,\n",
      "      any events whose global_step larger or equal to :math:`T` will be\n",
      "      purged and hidden from TensorBoard.\n",
      "      Note that crashed and resumed experiments should have the same ``log_dir``.\n",
      "    max_queue (int): Size of the queue for pending events and\n",
      "      summaries before one of the 'add' calls forces a flush to disk.\n",
      "      Default is ten items.\n",
      "    flush_secs (int): How often, in seconds, to flush the\n",
      "      pending events and summaries to disk. Default is every two minutes.\n",
      "    filename_suffix (string): Suffix added to all event filenames in\n",
      "      the log_dir directory. More details on filename construction in\n",
      "      tensorboard.summary.writer.event_file_writer.EventFileWriter.\n",
      "\n",
      "Examples::\n",
      "\n",
      "    from torch.utils.tensorboard import SummaryWriter\n",
      "\n",
      "    # create a summary writer with automatically generated folder name.\n",
      "    writer = SummaryWriter()\n",
      "    # folder location: runs/May04_22-14-54_s-MacBook-Pro.local/\n",
      "\n",
      "    # create a summary writer using the specified folder name.\n",
      "    writer = SummaryWriter(\"my_experiment\")\n",
      "    # folder location: my_experiment\n",
      "\n",
      "    # create a summary writer with comment appended.\n",
      "    writer = SummaryWriter(comment=\"LR_0.1_BATCH_16\")\n",
      "    # folder location: runs/May04_22-14-54_s-MacBook-Pro.localLR_0.1_BATCH_16/\n",
      "\u001b[1;31mFile:\u001b[0m           c:\\users\\hp\\python\\python310\\lib\\site-packages\\torch\\utils\\tensorboard\\writer.py\n",
      "\u001b[1;31mType:\u001b[0m           type\n",
      "\u001b[1;31mSubclasses:\u001b[0m     \n"
     ]
    }
   ],
   "source": [
    "SummaryWriter?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[3, 4]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "[1, 2]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 3\n",
      "2 4\n"
     ]
    }
   ],
   "source": [
    "#———————————————————————zip函数————————————————————————\n",
    "l1 = list(range(10))\n",
    "l1\n",
    "\n",
    "l1[3:5]\n",
    "\n",
    "l2 = list(range(1, 3))\n",
    "l2\n",
    "\n",
    "for i, j in zip(l2, l1[3:5]):\n",
    "    print(i, j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x1f474e7ef30>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 设置随机数种子\n",
    "torch.manual_seed(24)  \n",
    "\n",
    "# 实例化模型  \n",
    "tanh_model = net_class2(act_fun= torch.tanh, in_features=5, BN_model='pre')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 实例化writer对象\n",
    "writer = SummaryWriter(log_dir='l3')\n",
    "\n",
    "# 创建空列表容器\n",
    "train_mse = []\n",
    "test_mse = []\n",
    "\n",
    "# 创建总遍历次数计数器\n",
    "num_epochs = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "提前打开tensorboard服务，在命令行中输入\n",
    "```shell\n",
    "F:\n",
    "cd \"F:\\Code File\\PyTorch实战\\\"\n",
    "tensorboard --logdir=\"l3\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "while input(\"Do you want to continue the iteration? [y/n]\") == \"y\":\n",
    "    epochs = int(input(\"Number of epochs:\"))\n",
    "    lr = float(input(\"Update learning rate：\"))\n",
    "    train_l, test_l = fit_rec(net = tanh_model, \n",
    "                              criterion = nn.MSELoss(), \n",
    "                              optimizer = optim.SGD(tanh_model.parameters(), lr = lr), \n",
    "                              train_data = train_loader,\n",
    "                              test_data = test_loader,\n",
    "                              epochs = epochs, \n",
    "                              cla = False, \n",
    "                              eva = mse_cal)\n",
    "    train_mse.extend(train_l)\n",
    "    test_mse.extend(test_l)\n",
    "    for i, j in zip(list(range(num_epochs, num_epochs+epochs)), train_mse[num_epochs: num_epochs+epochs]):\n",
    "        writer.add_scalar('train_mse', j, i)\n",
    "    for i, j in zip(list(range(num_epochs, num_epochs+epochs)), test_mse[num_epochs: num_epochs+epochs]):\n",
    "        writer.add_scalar('test_mse', j, i)\n",
    "    num_epochs += epochs\n",
    "\n",
    "# y 30 0.03 y 30 0.01 n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当我们执行完第一阶段的30次遍历之后，我们即可打开浏览器输入`localhost:6006`查看当前记录的模型MSE情况。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "值得注意的是，当我们在使用add_scalar进行数据记录时，绘制折线图的横纵坐标都是必须的，如果缺失了横坐标，tensorboard中图像绘制的最终结果将呈现一条竖线。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    writer.add_scalar('i',i)\n",
    "\n",
    "# 图i中，缺失横坐标"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_epochs          # 上一轮循环截止时总共遍历多少次数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们再遍历30次数据，并设置学习率为0.005进行模型训练，测试代码性能与模型效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#—————————————————遍历30次数据 学习率为0.005————————————————\n",
    "\n",
    "while input(\"Do you want to continue the iteration? [y/n]\") == \"y\":\n",
    "    epochs = int(input(\"Number of epochs:\"))\n",
    "    lr = float(input(\"Update learning rate：\"))\n",
    "    train_l, test_l = fit_rec(net = tanh_model, \n",
    "                              criterion = nn.MSELoss(), \n",
    "                              optimizer = optim.SGD(tanh_model.parameters(), lr = lr), \n",
    "                              train_data = train_loader,\n",
    "                              test_data = test_loader,\n",
    "                              epochs = epochs, \n",
    "                              cla = False, \n",
    "                              eva = mse_cal)\n",
    "    train_mse.extend(train_l)\n",
    "    test_mse.extend(test_l)\n",
    "    for i, j in zip(list(range(num_epochs, num_epochs+epochs)), train_mse[num_epochs: num_epochs+epochs]):\n",
    "        writer.add_scalar('train_mse', j, i)\n",
    "    for i, j in zip(list(range(num_epochs, num_epochs+epochs)), test_mse[num_epochs: num_epochs+epochs]):\n",
    "        writer.add_scalar('test_mse', j, i)\n",
    "    num_epochs += epochs\n",
    "\n",
    "# y 30 0.005 n\n",
    "# 在tensorboard页面中查看总共90轮epochs的训练结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "至此，我们就完成了学习率手动调度实验。在上述操作流程中，我们不仅可以即时调整每一次训练遍历数据集的次数、同时灵活设置该训练过程的学习率，并且能够借助tensorboard即时呈现模型训练结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.常用学习率调度思路\n",
    "\n",
    "伴随模型遍历数据集次数增加、学习率逐渐降低，能够使得模型整体性能有所提升。\n",
    "\n",
    "- 幂调度：        \n",
    "&emsp;&emsp;随着迭代次数增加，学习率呈现幂律变化，\n",
    "  - 例如，$lr$为初始（第一轮）学习率，第二轮迭代时学习率调整为$lr/2$，第三轮迭代时学习率调整为$lr/3$等；\n",
    "\n",
    "- 指数调度：      \n",
    "&emsp;&emsp;随着迭代次数增加，学习率呈现指数变化，\n",
    "  - 例如，$lr$为初始（第一轮）学习率，第二轮迭代时学习率调整为$lr/10^1$，第三轮迭代时学习率调整为$lr/10^2 = lr/100$等\n",
    "\n",
    "- 分段恒定调度：      \n",
    "&emsp;&emsp;即每隔几轮迭代调整一次学习率，\n",
    "  - 例如1-10轮学习率为$lr$，10-20轮时学习率为$lr/100$等，不难发现此前我们所做的实验就是一种特殊的分段恒定调度策略； \n",
    "\n",
    "- 性能调度：      \n",
    "&emsp;&emsp;即每隔一段时间观察误差变化情况，如果误差基本不变，则降低学习率继续迭代；\n",
    "\n",
    "- 周期调度：      \n",
    "&emsp;&emsp;和前面几种学习率调度策略一味将学习率递减有所不同，周期调度允许学习率在一个周期内进行先递增后递减的变化"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.0 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "3296471c9107718e6db2104fe506aa09d2e74f3fa2347eed57e489ba84541490"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}