examples/mask_initialization_and_scaling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Comparison between Different HAT Mask Initialization/Scaling Methods\n",
    "\n",
    "This file aims investigate the effect of different HAT mask initialization/scaling methods on the convergence and regularization of HAT networks.\n",
    "\n",
    "The mask initialization/scaling methods are:\n",
    "- Normal initialization and linear scaling (proposed in in the paper)\n",
    "- Dense initialization and cosine scaling\n",
    "\n",
    "Normal initialization initialize the mask attention with a normal distribution with mean 0 and standard deviation 1. Whereas dense initialization initialize the mask attention with constant 1."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Original HAT Implementation\n",
    "\n",
    "First, we demonstrate the problem with the original HAT implementation.\n",
    "\n",
    "The implementation was taken from [Overcoming Catastrophic Forgetting with Hard Attention to the Task](https://github.com/joansj/hat) with minor modifications."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class OriginalHATNetwork(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(2, 1)\n",
    "        self.efc1 = nn.Embedding(1, 1)\n",
    "        self.gate = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, t, x, s=1):\n",
    "        mask = self.mask(t, s=s)\n",
    "        h = self.fc1(x)\n",
    "        y = h * mask.expand_as(h)\n",
    "        return y, mask\n",
    "\n",
    "    def mask(self, t, s=1):\n",
    "        return self.gate(s * self.efc1(t))\n",
    "\n",
    "    def get_view_for(self, n, masks):\n",
    "        gfc1 = masks\n",
    "        if n == \"fc1.weight\":\n",
    "            return gfc1.data.view(-1, 1).expand_as(self.fc1.weight)\n",
    "        elif n == \"fc1.bias\":\n",
    "            return gfc1.data.view(-1)\n",
    "        return None"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:00:59.502327Z",
     "end_time": "2023-04-30T10:01:00.429196Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The dataset has two features (x0 and x1) and one target (y). The features are in normal distribution, and the target is linearly dependent on x0.\n",
    "We measure the number of batches it takes for the network to converge (i.e. the loss of the test set is less than a threshold)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "num_samples = 1000\n",
    "num_features = 2\n",
    "\n",
    "features = torch.randn(num_samples, num_features)\n",
    "targets = 2 * features[:, 0] + 1\n",
    "targets = targets.view(-1, 1)\n",
    "\n",
    "# Split the dataset into train and test sets.\n",
    "trn_ratio = 0.8\n",
    "num_train_samples = int(num_samples * trn_ratio)\n",
    "trn_x, tst_x = features[:num_train_samples], features[num_train_samples:]\n",
    "trn_y, tst_y = targets[:num_train_samples], targets[num_train_samples:]\n",
    "\n",
    "# Construct dataloaders for the train and test sets.\n",
    "trn_dataloader = torch.utils.data.DataLoader(\n",
    "    dataset=torch.utils.data.TensorDataset(trn_x, trn_y),\n",
    "    shuffle=True,\n",
    "    batch_size=32,\n",
    ")\n",
    "tst_dataloader = torch.utils.data.DataLoader(\n",
    "    dataset=torch.utils.data.TensorDataset(tst_x, tst_y),\n",
    "    batch_size=32,\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:00.429080Z",
     "end_time": "2023-04-30T10:01:00.430889Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The training function is also taken from the original implementation."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "from hat.constants import (\n",
    "    DEF_HAT_MAX_TRN_MASK_SCALE,\n",
    "    DEF_HAT_GRAD_COMP_CLAMP,\n",
    "    DEF_HAT_ATTN_CLAMP,\n",
    ")\n",
    "\n",
    "# device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "device = torch.device(\"cpu\")\n",
    "convergence_threshold = 0.1\n",
    "learning_rate = 0.01\n",
    "momentum = 0.9\n",
    "\n",
    "\n",
    "def original_convergence_trial(\n",
    "    model: OriginalHATNetwork,\n",
    "    max_num_epochs: int,\n",
    "):\n",
    "    _model = model.to(device)\n",
    "    _optimizer = torch.optim.SGD(\n",
    "        _model.parameters(),\n",
    "        lr=learning_rate,\n",
    "        momentum=momentum,\n",
    "    )\n",
    "    _criterion = nn.MSELoss()\n",
    "\n",
    "    def __is_converged() -> bool:\n",
    "        __tst_loss = 0\n",
    "        with torch.no_grad():\n",
    "            for __x, __y in tst_dataloader:\n",
    "                __pred, _ = _model.forward(\n",
    "                    torch.as_tensor(0),\n",
    "                    __x,\n",
    "                    DEF_HAT_MAX_TRN_MASK_SCALE,\n",
    "                )\n",
    "                __tst_loss += _criterion(__pred, __y).item()\n",
    "        __tst_loss /= len(tst_dataloader)\n",
    "        return __tst_loss < convergence_threshold\n",
    "\n",
    "    def __get_trn_mask_scale(\n",
    "        curr_batch: int,\n",
    "        max_num_batches: int,\n",
    "    ) -> float:\n",
    "        ___min_trn_mask_scale = 1 / DEF_HAT_MAX_TRN_MASK_SCALE\n",
    "        return (\n",
    "            DEF_HAT_MAX_TRN_MASK_SCALE - ___min_trn_mask_scale\n",
    "        ) * curr_batch / max_num_batches + ___min_trn_mask_scale\n",
    "\n",
    "    # Train the model for `num_epochs` epochs\n",
    "    _num_batches = 0\n",
    "    for _ in range(max_num_epochs):\n",
    "        for __i, (__x, __y) in enumerate(trn_dataloader):\n",
    "            _optimizer.zero_grad()\n",
    "            __trn_mask_scale = __get_trn_mask_scale(\n",
    "                curr_batch=__i,\n",
    "                max_num_batches=(len(trn_dataloader) - 1),\n",
    "            )\n",
    "            __pred, __masks = _model.forward(\n",
    "                torch.as_tensor(0),\n",
    "                __x,\n",
    "                __trn_mask_scale,\n",
    "            )\n",
    "            __loss = _criterion(__pred, __y)\n",
    "\n",
    "            # L1 regularization for the masks\n",
    "            # From the `criterion()` in the original implementation.\n",
    "            # At this point, we ignore the regularization term because\n",
    "            # we are mostly focusing on the initialization of masks.\n",
    "            # __reg, __cnt = 0, 0\n",
    "            # for __m in __masks:\n",
    "            #     __reg += __m.sum()\n",
    "            #     __cnt += np.prod(__m.size()).item()\n",
    "            # __reg /= __cnt\n",
    "            # __loss += __reg\n",
    "\n",
    "            __loss.backward()\n",
    "\n",
    "            # Restrict layer gradients in backprop\n",
    "            # Since this is the first task, there is restriction\n",
    "\n",
    "            # Compensate embedding gradients\n",
    "            for __n, __p in _model.named_parameters():\n",
    "                if __n.startswith(\"e\"):\n",
    "                    __num = (\n",
    "                        torch.cosh(\n",
    "                            torch.clamp(\n",
    "                                __trn_mask_scale * __p.data,\n",
    "                                -DEF_HAT_GRAD_COMP_CLAMP,\n",
    "                                DEF_HAT_GRAD_COMP_CLAMP,\n",
    "                            )\n",
    "                        )\n",
    "                        + 1\n",
    "                    )\n",
    "                    __den = torch.cosh(__p.data) + 1\n",
    "                    __p.grad.data *= (\n",
    "                        DEF_HAT_MAX_TRN_MASK_SCALE\n",
    "                        / __trn_mask_scale\n",
    "                        * __num\n",
    "                        / __den\n",
    "                    )\n",
    "\n",
    "            _num_batches += 1\n",
    "            _optimizer.step()\n",
    "\n",
    "            # Constrain embeddings\n",
    "            for __n, __p in _model.named_parameters():\n",
    "                if __n.startswith(\"e\"):\n",
    "                    __p.data = torch.clamp(\n",
    "                        __p.data,\n",
    "                        -DEF_HAT_ATTN_CLAMP,\n",
    "                        DEF_HAT_ATTN_CLAMP,\n",
    "                    )\n",
    "\n",
    "            if __is_converged():\n",
    "                return _num_batches\n",
    "    return None"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:00.434822Z",
     "end_time": "2023-04-30T10:01:00.575774Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now let's train multiple model instances and see how many batches it takes for the model to converge (or not)."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original HAT model converged in 88.83 batches on average.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "num_trials = 100\n",
    "max_num_epochs = 100\n",
    "\n",
    "original_hat_num_batches = [\n",
    "    original_convergence_trial(OriginalHATNetwork(), max_num_epochs)\n",
    "    for _ in range(num_trials)\n",
    "]\n",
    "if None in original_hat_num_batches:\n",
    "    __num_failed_trials = original_hat_num_batches.count(None)\n",
    "    print(\n",
    "        f\"{__num_failed_trials} out of {num_trials} trials \"\n",
    "        f\"failed to converge in the given number of epochs.\"\n",
    "    )\n",
    "    original_hat_num_batches = [\n",
    "        __i for __i in original_hat_num_batches if __i is not None\n",
    "    ]\n",
    "original_hat_num_batches = np.array(original_hat_num_batches)\n",
    "print(\n",
    "    f\"Original HAT model converged in \"\n",
    "    f\"{original_hat_num_batches.mean():.2f} batches on average.\"\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:00.578408Z",
     "end_time": "2023-04-30T10:01:20.002039Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Note that the simple network takes absurd number of batches to converge.\n",
    "\n",
    "This is due to the random initialization of the model parameters.\n",
    "If the weight of x0 and bias is negative at the beginning, the model will likely to shut down the hard attention due to negative correlation between input and the output. This will lead to the model not being able to learn anything unless the mask scale is very small, which only happens at the beginning of each epoch.\n",
    "So essentially, the first few epochs are wasted except for the every beginning of each epoch when the mask scale is very small, if the model is initialized with wrong direction.\n",
    "\n",
    "To verify this, we can try to train the model with positive weight for x0 and bias."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original HAT model with modified weights converged in 13.29 batches on average.\n"
     ]
    }
   ],
   "source": [
    "modified_original_hat_num_batches = []\n",
    "for _ in range(num_trials):\n",
    "    _model = OriginalHATNetwork()\n",
    "    _model.fc1.weight.data[0, 0] = torch.abs(_model.fc1.weight.data[0, 0])\n",
    "    _model.fc1.bias.data = torch.abs(_model.fc1.bias.data)\n",
    "    modified_original_hat_num_batches.append(\n",
    "        original_convergence_trial(_model, max_num_epochs)\n",
    "    )\n",
    "if None in original_hat_num_batches:\n",
    "    __num_failed_trials = modified_original_hat_num_batches.count(None)\n",
    "    print(\n",
    "        f\"{__num_failed_trials} out of {num_trials} trials \"\n",
    "        f\"failed to converge in the given number of epochs.\"\n",
    "    )\n",
    "    modified_original_hat_num_batches = [\n",
    "        __i for __i in modified_original_hat_num_batches if __i is not None\n",
    "    ]\n",
    "modified_original_hat_num_batches = np.array(modified_original_hat_num_batches)\n",
    "print(\n",
    "    f\"Original HAT model with modified weights converged in \"\n",
    "    f\"{modified_original_hat_num_batches.mean():.2f} batches on average.\"\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:19.893356Z",
     "end_time": "2023-04-30T10:01:22.856366Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Solution\n",
    "\n",
    "The reason for such slow convergence is because the model is training the weight and the attention at the same time.\n",
    "If the random initialization of the weight is bad, the attention might shut down, which in turn makes the weight almost un-trainable for the rest of the epoch.\n",
    "\n",
    "Note that this is less of a problem if the model is over-parameterized, because the chance of all the parameters being initialized to bad values is very low.\n",
    "However, in the case of continual learning, we might run into this problem when there are too many tasks to learn.\n",
    "The later tasks might not have the luxury of having a large number of trainable parameters.\n",
    "\n",
    "The solution is to train the weight first so that the weights are in the right direction.\n",
    "Then, we can train the attention to gradually mask out the irrelevant features. And finally, we can train the weight again to fine-tune the model with binary attention masks.\n",
    "\n",
    "To achieve this, we need to initialize the attention so that all weights are trainable at the beginning.\n",
    "And we need to modify the scale of the attention masks so that it's high at the beginning and the end, and low in the middle.\n",
    "\n",
    "We propose dense initialization for the attention masks and cosine scaling for the attention masks.\n",
    "Dense initialization means that we initialize the attention masks with a value of 1.\n",
    "And cosine scaling means that we scale the attention masks with a cosine function.\n",
    "Here is a visualization of the cosine scaling."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 1000x500 with 2 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1IAAAHWCAYAAAB9mLjgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAACTBUlEQVR4nOzdd1hTZxsG8DsJJOy9FRAQRcWJgKPOWu3Qulo73KtD7bJfh22t1Q5b21pbO/y0Kmodra22tVqtdW8Q90BRUHEAArJHIDnfH0BaPhdgkjfj/l0X1yWHQ7hPwLx5znnO+8okSZJAREREREREtSYXHYCIiIiIiMjcsJAiIiIiIiKqIxZSREREREREdcRCioiIiIiIqI5YSBEREREREdURCykiIiIiIqI6YiFFRERERERURyykiIiIiIiI6oiFFBERERERUR2xkCKrd+HCBchkMsTFxYmOYla6d++O7t276z7n80hEZLpkMhnee+890TGMZtSoUWjUqFGNbdb2HJDhsZAiixYXFweZTIaDBw+KjmJQu3fvxkMPPYQGDRrAzs4OQUFB6NevH1asWCE6GhER/Z/z58/j2WefRWhoKOzs7ODi4oLOnTvjyy+/RElJieh4enf8+HE89thjCA4Ohp2dHRo0aIAHHngAc+fOFR2N6J7YiA5AJFpwcDBKSkpga2srOkq9rF69Gk888QTatGmDl156Ce7u7khNTcXOnTuxYMECPP3000bJYe7PIxGRMaxfvx6PP/44VCoVRowYgcjISKjVauzevRuvvfYaTp48ifnz5+v955aUlMDGxvhv+/bu3YsePXogKCgI48ePh5+fH9LS0rB//358+eWXeOGFF4yWRdRzQJaLf01k9WQyGezs7ETHuKPi4mI4ODjc8mvvvfcemjdvjv3790OpVNb4WmZmpjHiATCP55GISKTU1FQ8+eSTCA4OxtatW+Hv76/72sSJE3Hu3DmsX7/eID9b1Ovzhx9+CFdXVyQkJMDNza3G14w5RgHingOyXGztI6t3q3t7Ro0aBScnJ1y5cgUDBgyAk5MTvL298Z///AcajabG92u1WsyZMwctWrSAnZ0dfH198eyzz+LGjRs19vvtt9/wyCOPICAgACqVCmFhYXj//fdverzu3bsjMjISiYmJ6Nq1KxwcHPDWW2/dNv/58+cRHR19UxEFAD4+Pjdl/fLLL9GyZUvY2dnB29sbDz74YI3Wx8WLF6Nnz57w8fGBSqVC8+bN8d133xn8eczOzsbw4cPh4uICNzc3jBw5EkePHuV9V0RkMWbNmoXCwkIsXLiwRhFVrXHjxnjppZd0n1dUVOD9999HWFgYVCoVGjVqhLfeegtlZWU1vu/gwYPo06cPvLy8YG9vj5CQEIwZM6bGPv9/f9B7770HmUyGc+fOYdSoUXBzc4OrqytGjx6N4uLim7L98MMPiIqKgr29PTw8PPDkk08iLS3trsd8/vx5tGjR4qYiCrh5jKr+OTExMXBwcIC7uzu6du2Kv/76S/f12o6lt3Ivz0FJSQlefPFFeHl5wdnZGY8++iiuXLnC+66sHK9IEd2GRqNBnz59EBsbi88++wx///03Pv/8c4SFheH555/X7ffss88iLi4Oo0ePxosvvojU1FR8/fXXOHz4MPbs2aNrdYuLi4OTkxMmT54MJycnbN26Fe+++y7y8/Px6aef1vjZ2dnZeOihh/Dkk09i2LBh8PX1vW3O4OBgbNmyBZcvX0bDhg3veExjx45FXFwcHnroIYwbNw4VFRXYtWsX9u/fj/bt2wMAvvvuO7Ro0QKPPvoobGxssG7dOkyYMAFarRYTJ040yPOo1WrRr18/xMfH4/nnn0dERAR+++03jBw5ss4/j4jIVK1btw6hoaHo1KlTrfYfN24clixZgsceewyvvvoqDhw4gJkzZ+L06dNYu3YtgMqrOr1794a3tzfefPNNuLm54cKFC1izZk2tfsaQIUMQEhKCmTNn4tChQ/j+++/h4+ODTz75RLfPhx9+iKlTp2LIkCEYN24crl+/jrlz56Jr1644fPjwLYukasHBwdi3bx9OnDiByMjIO2aZPn063nvvPXTq1AkzZsyAUqnEgQMHsHXrVvTu3RtA3cbS2qrNczBq1Cj89NNPGD58ODp06IAdO3bgkUceqdfPIwsiEVmwxYsXSwCkhISE2+6TmpoqAZAWL16s2zZy5EgJgDRjxowa+7Zt21aKiorSfb5r1y4JgLR8+fIa+23cuPGm7cXFxTf97GeffVZycHCQSktLddu6desmAZDmzZtXq2NcuHChBEBSKpVSjx49pKlTp0q7du2SNBpNjf22bt0qAZBefPHFmx5Dq9XeMWefPn2k0NDQGtu6desmdevWTff5vTyPv/zyiwRAmjNnjm6bRqORevbsedNjEhGZo7y8PAmA1L9//1rtf+TIEQmANG7cuBrb//Of/0gApK1bt0qSJElr16696zgnSZIEQJo2bZru82nTpkkApDFjxtTYb+DAgZKnp6fu8wsXLkgKhUL68MMPa+x3/PhxycbG5qbt/++vv/6SFAqFpFAopI4dO0qvv/66tGnTJkmtVtfYLzk5WZLL5dLAgQNvGr/uNkbdaiwdOXKkFBwcrJfnIDExUQIgvfzyyzX2GzVq1E2PSdaFrX1Ed/Dcc8/V+LxLly5ISUnRfb569Wq4urrigQceQFZWlu4jKioKTk5O2LZtm25fe3t73b8LCgqQlZWFLl26oLi4GElJSTV+jkqlwujRo2uVccyYMdi4cSO6d++O3bt34/3330eXLl0QHh6OvXv36vb75ZdfIJPJMG3atJseQyaT3TJnXl4esrKy0K1bN6SkpCAvL69Wmf7f3Z7HjRs3wtbWFuPHj9dtk8vl9boCRkRkivLz8wEAzs7Otdp/w4YNAIDJkyfX2P7qq68CgO5equqrQX/88QfKy8vrnOtWr8/Z2dm6vGvWrIFWq8WQIUNqjHN+fn4IDw+vMc7dygMPPIB9+/bh0UcfxdGjRzFr1iz06dMHDRo0wO+//67b79dff4VWq8W7774Lubzm29PbjVF3G0v19Rxs3LgRADBhwoQa+xlzogwyTSykiG6j+h6if3N3d69x71NycjLy8vLg4+MDb2/vGh+FhYU1bqQ9efIkBg4cCFdXV7i4uMDb2xvDhg0DgJsKlAYNGtzynqfb6dOnDzZt2oTc3Fzs3LkTEydOxMWLF9G3b19dhvPnzyMgIAAeHh53fKw9e/agV69ecHR0hJubG7y9vXX3aNWnkKrN83jx4kX4+/vfNKFG48aN6/zziIhMkYuLC4DKN/+1cfHiRcjl8pteB/38/ODm5oaLFy8CALp164bBgwdj+vTp8PLyQv/+/bF48eKb7qO6naCgoBqfu7u7A4DuNTo5ORmSJCE8PPymce706dO1mjAiOjoaa9aswY0bNxAfH48pU6agoKAAjz32GE6dOgWgcoySy+Vo3rz5HR+rLmNpbd3tOaj+XYSEhNTYj2MU8R4pottQKBR33Uer1cLHxwfLly+/5derC4jc3Fx069YNLi4umDFjBsLCwmBnZ4dDhw7hjTfegFarrfF9/z7jVhcODg7o0qULunTpAi8vL0yfPh1//vlnre81On/+PO6//35ERERg9uzZCAwMhFKpxIYNG/DFF1/clLM2avM8EhFZOhcXFwQEBODEiRN1+r5/X4253dd//vln7N+/H+vWrcOmTZswZswYfP7559i/fz+cnJzu+P23e42WJAlA5Tgnk8nw559/3nLfuz3+vymVSkRHRyM6OhpNmjTB6NGjsXr16lt2StxKXcfS2rrbc0B0OyykiO5BWFgY/v77b3Tu3PmOxc/27duRnZ2NNWvWoGvXrrrtqampBstWPXnEtWvXdFk3bdqEnJyc216VWrduHcrKyvD777/XOEN3t9aNexUcHIxt27bdNM37uXPnDPpziYiMqW/fvpg/fz727duHjh073nHf4OBgaLVaJCcno1mzZrrtGRkZyM3NRXBwcI39O3TogA4dOuDDDz/EihUrMHToUKxatQrjxo27p8xhYWGQJAkhISFo0qTJPT3Wv91qjNJqtTh16hTatGlzy+8RMZYC//wuUlNTER4ertvOMYrY2kd0D4YMGQKNRoP333//pq9VVFQgNzcXwD9nu/59dkutVuPbb7+95wxbtmy55fbq/vqmTZsCAAYPHgxJkjB9+vSb9q3OdauceXl5WLx48T3nvJM+ffqgvLwcCxYs0G3TarX45ptvDPpziYiM6fXXX4ejoyPGjRuHjIyMm75+/vx5fPnllwCAhx9+GAAwZ86cGvvMnj0bAHQzxt24ceOmKyfVhUht2/vuZNCgQVAoFJg+ffpNP0eSJGRnZ9/x+7dt23bLKzv/P0YNGDAAcrkcM2bMuOnK0p3GKH2NpXfSp08fALjp58ydO9egP5dMH69IkVVYtGiR7mbRf/v3eh310a1bNzz77LOYOXMmjhw5gt69e8PW1hbJyclYvXo1vvzySzz22GPo1KkT3N3dMXLkSLz44ouQyWRYtmyZXtoG+vfvj5CQEPTr1w9hYWEoKirC33//jXXr1iE6Ohr9+vUDAPTo0QPDhw/HV199heTkZDz44IPQarXYtWsXevTogUmTJqF3795QKpXo168fnn32WRQWFmLBggXw8fHRnTU0hAEDBiAmJgavvvoqzp07h4iICPz+++/IyckBcPfWFiIicxAWFoYVK1bgiSeeQLNmzTBixAhERkZCrVZj7969WL16NUaNGgUAaN26NUaOHIn58+frWtri4+OxZMkSDBgwAD169AAALFmyBN9++y0GDhyIsLAwFBQUYMGCBXBxcdEVY/ea+YMPPsCUKVNw4cIFDBgwAM7OzkhNTcXatWvxzDPP4D//+c9tv/+FF15AcXExBg4ciIiICN2x/vjjj2jUqJFuYqXGjRvj7bff1k2YNGjQIKhUKiQkJCAgIAAzZ8406Fh6J1FRURg8eDDmzJmD7Oxs3fTnZ8+eBcAxypqxkCKrcLsFZasHrHsxb948REVF4b///S/eeust2NjYoFGjRhg2bBg6d+4MAPD09MQff/yBV199Fe+88w7c3d0xbNgw3H///bozXfX1/fff47fffsNPP/2Eq1evQpIkhIaG4u2338Ybb7wBG5t//psvXrwYrVq1wsKFC/Haa6/B1dUV7du3161p0rRpU/z8889455138J///Ad+fn54/vnn4e3tfdPijvqkUCiwfv16vPTSS1iyZAnkcjkGDhyIadOmoXPnzlyNnogsxqOPPopjx47h008/xW+//YbvvvsOKpUKrVq1wueff15j9tLvv/8eoaGhiIuLw9q1a+Hn54cpU6bUuKeousBatWoVMjIy4OrqipiYGCxfvvymyRHq680330STJk3wxRdf6LoaAgMD0bt3bzz66KN3/N7PPvsMq1evxoYNGzB//nyo1WoEBQVhwoQJeOedd2qsQTVjxgyEhIRg7ty5ePvtt+Hg4IBWrVph+PDhAAw7lt7N0qVL4efnh5UrV2Lt2rXo1asXfvzxRzRt2pRjlBWTSbyTjohM1K+//oqBAwdi9+7duqKUiIjIFBw5cgRt27bFDz/8gKFDh4qOQwLwHikiMgklJSU1PtdoNJg7dy5cXFzQrl07QamIiIhuHqOAyvvX5HJ5jYkvyLqwtY+ITMILL7yAkpISdOzYEWVlZVizZg327t2Ljz76qN7TwRMREenDrFmzkJiYiB49esDGxgZ//vkn/vzzTzzzzDMIDAwUHY8EYWsfEZmEFStW4PPPP8e5c+dQWlqKxo0b4/nnn8ekSZNERyMiIiu3efNmTJ8+HadOnUJhYSGCgoIwfPhwvP322zXuRSbrwkKKiIiIiIiojniPFBERERERUR2xkCIiIiIiIqojNnUC0Gq1uHr1KpydnbmoGhGREUmShIKCAgQEBEAu57m9ahyXiIjEqe3YxEIKwNWrVznjChGRQGlpaWjYsKHoGCaD4xIRkXh3G5tYSAFwdnYGUPlkubi4CE5DRGQ98vPzERgYqHsdpkocl4iIxKnt2MRCCtC1Tbi4uHDAIiISgO1rNXFcIiIS725jExvSiYiIiIiI6oiFFBERERERUR2xkCIiIiIiIqojFlJERERERER1xEKKiIiIiIiojlhIERERERER1RELKSIiIiIiojpiIUVERERERFRHLKSIiIiIiIjqiIUUERERERFRHQktpHbu3Il+/fohICAAMpkMv/76a42vS5KEd999F/7+/rC3t0evXr2QnJxcY5+cnBwMHToULi4ucHNzw9ixY1FYWGjEoyAiIkvCsYmIiGpDaCFVVFSE1q1b45tvvrnl12fNmoWvvvoK8+bNw4EDB+Do6Ig+ffqgtLRUt8/QoUNx8uRJbN68GX/88Qd27tyJZ555xliHQEREFoZjExER1YpkIgBIa9eu1X2u1WolPz8/6dNPP9Vty83NlVQqlbRy5UpJkiTp1KlTEgApISFBt8+ff/4pyWQy6cqVK7X+2Xl5eRIAKS8v794PhIjIiuSVqCWNRlv/7zfx119RY5OpPy9ERJastq/BNuJKuDtLTU1Feno6evXqpdvm6uqK2NhY7Nu3D08++ST27dsHNzc3tG/fXrdPr169IJfLceDAAQwcOPCWj11WVoaysjLd5/n5+YY7ECIiC1VYVoFh3x9AI09HfPZ4ayhtLP+2W0ONTfoel2ZtTMLWpEz4u9rB380eAa52iA31RFSQO+Ry2T09NhGRKdFqJRxOy8X+lGxczS3BtbxSXMsrRecwT7zTt7lBf7bJFlLp6ekAAF9f3xrbfX19dV9LT0+Hj49Pja/b2NjAw8NDt8+tzJw5E9OnT9dzYiIi61FWocGzyw7i2OU8pOUUIyO/FIEeDqJjGZyhxiZ9j0vJmYVISi9AUnpBje0+zio8FOmHx9sHIrKBq95+HhGRsZ2+lo+fDqZh44l0XMsrvenrPs4qg2cw2ULKkKZMmYLJkyfrPs/Pz0dgYKDARERE5kOjlTD5x6PYcy4bDkoF4kbHWEURZUj6HpfefrgZhsYGIb3qzGxKVhG2J2Uis6AMS/ZdxJJ9F/Fo6wC81qcpf3dEZFau5Jbg87/OYO3hK5Ckym1OKht0a+qNMG8n+Lvawc/VDsFGeG0z2ULKz88PAJCRkQF/f3/d9oyMDLRp00a3T2ZmZo3vq6ioQE5Oju77b0WlUkGlMnyVSkRkaSRJwru/ncD649dgq5Bh/vD2aB3oJjqW0RhqbNL3uNTIyxGNvBxrbCur0GDPuSysOXQFfxy7ht+PXsXGE+kY3bkRXnmgCexsFXr7+URE+qau0OLLLWexYFcq1BVaAMBDkX4Y3K4h7gv3EvIaZrIN7SEhIfDz88OWLVt02/Lz83HgwAF07NgRANCxY0fk5uYiMTFRt8/WrVuh1WoRGxtr9MxERJZuzt/JWH7gEmQy4Isn2uC+cC/RkYzKnMcmlY0CPSN88fXT7fDHC/ehc2NPqDVa/HdnCh6ftw9pOcXCshER3cmV3BI8MX8fvtl2HuoKLTqEeuD3SZ3x3bAo9GruK+xEkNArUoWFhTh37pzu89TUVBw5cgQeHh4ICgrCyy+/jA8++ADh4eEICQnB1KlTERAQgAEDBgAAmjVrhgcffBDjx4/HvHnzUF5ejkmTJuHJJ59EQECAoKMiIrJMS/ddwJdbKtdLmtE/En1bWebrrDWMTZENXPHD2Fj8fToTr/98FMev5OGRr3Zh9pA26NXc9+4PQERkJNvPZOKVH4/gRnE5XOxs8MngVngw0g8ymfiJc2SSVN1daHzbt29Hjx49bto+cuRIxMXFQZIkTJs2DfPnz0dubi7uu+8+fPvtt2jSpIlu35ycHEyaNAnr1q2DXC7H4MGD8dVXX8HJyanWOfLz8+Hq6oq8vDy4uLjo5diIiCzJuqNX8eKqw5Ak4JVeTfBSr3C9PK4pvv6awthkzOflam4JJq44hMOXcgEA0/o1x+jOIQb9mUREtbH8wEW88+sJSBLQsoErvh3azij3ddb2NVhoIWUqTHEgJyIyFbuSr2NMXALKNRJGdAzG9Edb6O1MIF9/b83Yz4u6QosP1p/C0n0XAVROVjG+a6jBfy4R0e3E7UnFe+tOAQCeignEe4+2gMrGOC18tX0NNtl7pIiISLwjabl4dlkiyjUS+rbyx3v99FdEkelQ2sgx/dEWeKFnYwDAhxtO45tt5+7yXUREhvH9rhRdEfVs11B8NLCl0YqoumAhRUREt3QusxCjF8ejWK1Bl3AvzB7Shou5WjCZTIZXezfF5AcqWxQ/3XQG/91xXnAqIrI2i/ek4oP1pwEAk3o0xpsPRZjsCTwWUkREdJNreSUYsfAAbhSXo3WgG+YNi4LShkOGNXjx/nC81qcpAGDmn0nYeOKa4EREZC22nM7AjD8qr0S9dH84/tOnqckWUQALKSIi+j83itQYsTAeV/NKEertiMWjouGoMtllB8kAJvZojJEdgwEAL/94BMcv5wlORESW7vS1fLy4snJSo6diAvGyniY1MiQWUkREpFOsrsCYJQlIziyEn4sdlo2NhYejUnQsEmBq3+bo2sQbpeVajFuagPS8UtGRiMhCXS8ow7glB1Gk1qBTmCdm9I806StR1VhIERERgMqZ2577oXIabFd7WywbG4MGbvaiY5EgNgo5vn66LcJ9nJCRX4Znlx2EukIrOhYRWZgKjRYTlifiSm4JQrwc8e3QdrBVmEeJYh4piYjIoLRaCa/9fBQ7z16Hva0Ci0dHI9zXWXQsEszFzhaLRkXD1d4WRy/n4Yu/z4qOREQW5utt55Bw4QacVDZYOLI93BzMpwuChRQRkZWTJAkz/jiF345chY1chu+GtUO7IHfRschEBHo44ONBLQEA83acx95zWYITEZGlOHghB19tSQYAfDgwEqHetVu03FSwkCIisnLfbDuHuL0XAACfD2mN7k19xAYik/NQS388FRMISQJe+ekIbhSpRUciIjOXV1KOl1YdgVYCBrVtgP5tGoiOVGcspIiIrNiKA5fw2V+V7VrT+jU3y4GMjGNq3+YI9XZERn4Z3vjlGCRJEh2JiMzYO7+ewJXcEgR5OGDGgEjRceqFhRQRkZX68/g1vPPrcQCVix6O7hwiOBGZMgelDb56si1sFTL8dSoD645xfSkiqp9NJ9Ox7uhVKOQyfPVUWziZ6RIbLKSIiKzQ3nNZupaKp2KC8GrvJqIjkRmIbOCKiT0aAwBmrDuJvOJywYmIyNwUlJZj2m8nAQDPdg1Fm0A3sYHuAQspIiIrc/xyHsYvPQi1RosHW/jhgwHmsV4HmYbnu4chzNsRWYVqfLzxtOg4RGRmPv/rLNLzSxHs6YAX7zf9RXfvhIUUEZEVSc0qwqjF8ShSa9Ax1BNznmwDhZxFFNWeykaBjwZWzuK3Mj4N8ak5ghMRkbk4kpaLJfsuAAA+HNASdrYKsYHuEQspIiIrkZFfiuELDyC7SI3IBi6YPyLK7AcxEiM21BNPRgcCAN5aexxlFRrBiYjI1JVrtJiy5jikqln67gv3Eh3pnrGQIiKyAnnF5Ri5KB6Xb5SgkacD4kbHwNnOVnQsMmNTHmoGLyclzmUWYknV9PlERLez4sAlnL6WD3cHW7z9SDPRcfSChRQRkYUrUWswdkkCktIL4OOswrKxsfByUomORWbO1cEWrz8YAQCYu+UcsgvLBCciIlOVV1yOOX9XLrXxau+m8LSQMYiFFBGRBSvXaDFpxSEcvHgDLnY2WDo2BoEeDqJjkYUY3K4hmvu7oKCsAl9uSRYdh4hM1NfbknGjuBzhPk66tmBLwEKKiMhCabUS3vzlOLYkZUJlI8fCUdGI8HMRHYssiEIuwztVLTrLD1zCucxCwYmIyNRczC7Ckr0XAQBvP9IMNgrLKT8s50iIiKiGjzcm4ZdDl6GQy/Dt0HaIbuQhOhJZoE6NvdCrmS80WgkzN3A6dCKq6ZONSVBrtOjaxBvdm/qIjqNXLKSIiCzQvB3nMX9nCgBg1uBWuL+Zr+BEZMmmPBwBG7kMW5Iysedclug4RGQiDl7IwYbj6ZDLgLcftowJJv6NhRQRkYX5KSENH/+ZBKBy4Boc1VBwIrJ0Yd5OGBobBACYtekMJEkSnIiIRJMkCbM2nQEAPBEdiKZ+zoIT6R8LKSIiC7L5VAbeXHMMAPBs11CM7xoqOBFZi0k9w2FnK8fRtFxsO5MpOg4RCbb3fDbiU3OgtJHjxfvDRccxCBZSREQW4kBKNiauOAStBDwe1RBvPhQhOhJZEW9nFUZ2bAQAmL35LK9KEVkxSZIwe3PldOdPxwTB39VecCLDYCFFRGQBTl3Nx7glB6Gu0KJXM1/MHNQSMplMdCyyMs90DYWDUoETV/Kx+VSG6DhEJMiOs9eRePEGVDZyTOgeJjqOwbCQIiIyc5eyizFiUTwKyioQ08gDXz/d1qKmlyXz4emkwujOjQAAX/ydDK2WV6WIrI0kSfii6mrU8A7B8HGxE5zIcDjSEhGZscyCUgxbeABZhWVo5u+CBSPbw85WIToWWbHxXULhpLLB6Wv52HQyXXQcIjKyrUmZOHo5D/a2CjzbzXKvRgEspIiIzFZ+aTlGLkrApZxiBHk4YMmYaLja24qORVbOzUGJMfeFAAC+3JLMe6WIrIgkSfhySzIAYESnYHg7qwQnMiwWUkREZqi0XINxSw7i9LV8eDmpsGxsDHycLbd9gszL2M4hcFQqkJRegO1nrouOQ0RGsvd8No5dzoOdrRzju1j+rLEspIiIzEyFRosXVh5GfGoOnFU2WDImGsGejqJjEem4Otji6ap1pb7bcV5wGiIylnlV/9+HtA+El5NlX40CWEgREZkVSZLw1trj2HwqA0obORaMbI8WAa6iYxHdZOx9obBVyBCfmoPEizdExyEiAztxJQ+7krOgkMus4moUwEKKiMiszNp0Bj8dvAy5DJj7VFt0CPUUHYnolvxc7TCwbQMA/5ylJiLLVX31uW8rfwR6OAhOYxwspIiIzMT3u1Lw3fbKgWrmoJbo08JPcCKiO3umaxhkMmDzqQycyywQHYeIDORCVhH+PH4NAPCchc/U928spIiIzMCaQ5fxwfrTAIDXH2yKJ6KDBCciurvGPk7o3dwXADBvR4rgNERkKPN3pUArAT2aeqOZv4voOEbDQoqIyMRtS8rEaz8fAwCMvS8Ez1vR2T4yf9Vnp389fAUZ+aWC0xCRvmUXluHnxMsArOtqFMBCiojIpCVezMHzyxOh0UoY1LYB3n64GWQymehYRLXWNsgd7YPdUaGVsHz/RdFxiEjPVsZfgrpCi1YNXRET4iE6jlGxkCIiMlFn0gswenECSsu16NHUG5881gpyOYsoMj+jO1cu0Lv8wCWUlmsEpyEifSnXaLGs6gTJ6M6NrO5EHwspIiITdPlGMUYsOoD80gpEBbvj26FRsFXwJZvMU58WvvB3tUN2kRrrjl4VHYeI9OTPE+nIyC+Dl5MKD7f0Fx3H6DgqExGZmOzCMoxYGI+M/DI08XXCwpHtYa9UiI5FVG82CjmGdwwGAMTtvQBJkgQnIiJ9WLwnFQAwrEMQVDbWN06xkCIiMiGFZRUYHZeAlKwiNHCzx9IxsXBzUIqORXTPnooOgp2tHCev5iPhAhfoJTJ3R9JycfhSLmwVMgyNDRYdRwgWUkREJqKsQoNnlx3Esct58HBUYunYGPi52omORaQX7o5K3QK91Wexich8xVX9P+7XKgDezirBacRgIUVEZAI0Wgmv/HgEe85lw1GpQNzoaIR5O4mORaRXozpVTjqx6WQ6ruSWCE5DRPWVWVCK9VUL8FZPJmONWEgREQkmSRLe/e0ENhxPh1Ihx3+Ht0erhm6iYxHpXVM/Z3QM9YRWAlbFXxIdh4jq6aeENJRrJLQLckPLhq6i4wjDQoqISLAv/k7G8gOXIJMBXzzRBveFe4mORGQwwzpU3kuxKiEN5Rqt4DREVFcarYSV8WkA/vn/bK1YSBERCbRk7wV8tSUZADCjfyQeaWV908eSdXmguS+8nFS4XlCGLaczRMchojraefY6ruSWwM3B1iqnPP83FlJERIL8fvQq3lt3EgDwSq8mGG7lZ/bIOiht5BjSviGAygV6ici8LD9QuQDv4HYNYWdrfVOe/xsLKSIiAXacvY7JPx6BJAEjOgbjxfsbi45EZDRPxQRBJgN2JWfhYnaR6DhEVEtXc0uwNSkTAPB0bJDgNOKxkCIiMrLDl27g+R8SUaGV0K91AN7r1wIymUx0LCKjCfRwQNdwbwDACk46QWQ2ViWkQSsBHUI9OLMsWEgRERnVucwCjIlLQLFagy7hXvj88daQy1lEkfUZWnU2++eDl1FWoRGchojuplyj1c22aa0L8P4/FlJEREZyNbcEIxbG40ZxOVoHumHesCgobfgyTNapZ4QP/FzskF2kxqaTnHSCyNRtOZ2JzIIyeDoq0aeFn+g4JoEjOBGREdwoUmP4wgO4mleKUG9HLB4VDUeVjehYRMLYKOQYEh0IoHJNGiIybT8drPx/+lj7hjwJWIXPAhGRgRWVVWB0XALOXy+Cv6sdlo2NhYejUnQsIuEej6qcvW/P+SxcvlEsOA0R3U5Gfim2n6mcZGJI+0DBaUwHCykiIgNSV2jx/PJDOJKWCzcHWywdE4MGbvaiYxGZhEAPB3QK84QkAb8kXhEdh4huY82hK9BKQFSwOyeZ+BcWUkREBqLVSvjP6qPYefY67G0VWDQqGuG+zqJjEZmU6rPbqxPToNVKgtMQ0f+TJAmrq9r6qteAo0ospIiIDECSJMz44xR+P3oVNnIZvhvWDu2C3EXHIjI5D0b6wdnOBpdvlGB/SrboOET0fxIv3kBKVhHsbRV4pFWA6DgmhYUUEZEBfL31HOL2XgAAfD6kNbo39REbiMhE2dkq0K915Zuz1YmXBachov+3+mDl/8tHWvnDiZMk1cBCiohIz5YfuIjPN58FALzXrzn6t2kgOBGRaatu79tw/BryS8sFpyGiakVlFfjj2FUAnGTiVlhIERHp0Ybj1/DOrycAAC/0bIxRnUMEJyIyfa0buqKJrxPKKrRYd/Sq6DhEVGXD8WsoUmvQyNMB0Y3Ynv7/WEgREenJ3nNZeHnVEUgS8HRsECY/0ER0JCKzIJPJ8HhU5dnun9neR2Qyqv8/PhbVEDKZTHAa02PShZRGo8HUqVMREhICe3t7hIWF4f3334ck/TOrjyRJePfdd+Hv7w97e3v06tULycnJAlMTkTU6fjkP45cehFqjxUORfni/fyQHHaI66N82AHIZcPhSLlKzikTHIbJ6l28U40BqDgBgUDvO1ncrJl1IffLJJ/juu+/w9ddf4/Tp0/jkk08wa9YszJ07V7fPrFmz8NVXX2HevHk4cOAAHB0d0adPH5SWlgpMTkTWJOV6IUYtjkeRWoNOYZ6Y82QbKOQsoiwVT/IZho+zHe4L9wYArD3MNaWIRPvtSGWbbYdQDwRw/cNbMulCau/evejfvz8eeeQRNGrUCI899hh69+6N+Ph4AJUD1Zw5c/DOO++gf//+aNWqFZYuXYqrV6/i119/FRueiKxCRn4phi+MR3aRGpENXPDf4VFQ2ShExyID4kk+wxnUtnJill8PX6lRmBKRcUmSpDuhMagtr0bdjkkXUp06dcKWLVtw9mzl7FdHjx7F7t278dBDDwEAUlNTkZ6ejl69eum+x9XVFbGxsdi3b99tH7esrAz5+fk1PoiI6iqvuBwjFsbjSm4JQrwcETc6Bs52tqJjkYHxJJ/h9G7hCwelApdyinHo0g3RcYis1okr+TiXWQiVjRwPtvQTHcdkmXQh9eabb+LJJ59EREQEbG1t0bZtW7z88ssYOnQoACA9PR0A4OvrW+P7fH19dV+7lZkzZ8LV1VX3ERjI6RyJqG5K1BqMXZKAMxkF8HFWYemYGHg5qUTHIiMwxEk+nuCr5KC0wYORlW/a1hxiex+RKGsOV04y0au5L1x4gvC2TLqQ+umnn7B8+XKsWLEChw4dwpIlS/DZZ59hyZIl9/S4U6ZMQV5enu4jLS1NT4mJyBqUa7SYuOIQDl68ARc7GywdG4NADwfRschIDHGSjyf4/lHdRvTHsWsoq9AITkNkfSo0/yxDUN1uS7dm0oXUa6+9phuwWrZsieHDh+OVV17BzJkzAQB+fpVnrTIyMmp8X0ZGhu5rt6JSqeDi4lLjg4ioNrRaCW/8fAxbkzJhZyvHolHRiPDja4g1McRJPp7g+0fHME/4uqiQV1KObUnXRcchsjq7zmUhq1ANT0clujbxFh3HpJl0IVVcXAy5vGZEhUIBrVYLAAgJCYGfnx+2bNmi+3p+fj4OHDiAjh07GjUrEVk+SZLw0YbTWHP4ChRyGb4d2g7tG3mIjkVGZoiTfDzB9w+FXIb+bSrPgq89zDWliIxtbVVbbb/WAbBVmHSpIJxJPzv9+vXDhx9+iPXr1+PChQtYu3YtZs+ejYEDBwKoXMDv5ZdfxgcffIDff/8dx48fx4gRIxAQEIABAwaIDU9EFmfejhR8vzsVADBrcCv0jPC9y3eQJeJJPsMbWNVOtC3pOvJKygWnIbIeRWUV2Hyq8iTQALb13ZWN6AB3MnfuXEydOhUTJkxAZmYmAgIC8Oyzz+Ldd9/V7fP666+jqKgIzzzzDHJzc3Hfffdh48aNsLOzE5iciCzNjwmX8MnGJADA2w83w+AoTgdrrapP8gUFBaFFixY4fPgwZs+ejTFjxgCoeZIvPDwcISEhmDp1Kk/y1UEzfxc08XXC2YxCbDqZjiHtrfeeMSJj+vt0BkrKNQj2dEDrhq6i45g8mcSFGpCfnw9XV1fk5eVZdTsFEd3appPpeP6HRGgl4LluYXjzoQjRkSyGOb7+FhQUYOrUqVi7dq3uJN9TTz2Fd999F0qlEkBlG+i0adMwf/583Um+b7/9Fk2aNKnVzzDH50Xfvt6ajM/+Oosu4V5YNjZWdBwiqzBuSQL+Pp2JF3o2xqu9m4qOI0xtX4NZSIEDFhHd3v6UbIxYFA91hRZD2jfEJ4NbQSaTiY5lMfj6e2t8XoALWUXo/tl2yGXAgbd6wduZywsQGVJusRrRH/6Nco2Ev17piia+zqIjCVPb12CTvkeKiEikk1fzMH7JQagrtHiguS8+GtiSRRSRkTTyckTrhq7QSsCfJ66JjkNk8TadTEe5RkKEn7NVF1F1wUKKiOgWLmYXYeSiBBSUVSAmxANzn2oLG85eRGRU/VoHAAB+P3JVcBIiy/d71dpR1f/v6O74roCI6P9k5pdi+MJ4ZBWWoZm/C74f2R52tgrRsYisTt9WAZDJgIMXb+BKbonoOEQWK7OgFPvOZwMA+rViIVVbLKSIiP4lr6QcIxcn4FJOMYI8HLBkTDRc7GxFxyKySn6udoipWqvtj6O8KkVkKBuOXYNWAtoEuiHI00F0HLPBQoqIqEppuQbjlx7E6Wv58HJSYdnYGPg4cykFIpEebVPV3sdCishgqv9/Pcq2vjphIUVEBKBCo8ULKw8jPjUHziobLBkTjWBPR9GxiKzeQ5H+sJHLcPJqPs5fLxQdh8jiXL5RjEOXciGTAY+08hcdx6ywkCIiqydJEt5aexybT2VAaSPHgpHt0SKACxESmQIPRyU6N/YCUNl+RET6teF45f+rmEYe8HVhF0ZdsJAiIqv3ycYz+OngZchlwNyn2qJDqKfoSET0L4+0rDxLvv44CykifVt/PB0A0JdXo+qMhRQRWbUFO1Mwb8d5AMDMQS3Rp4Wf4ERE9P96t/CFjVyGpPQCpLC9j0hvLt8oxtG0yra+PpEc/+qKhRQRWa1fEi/jww2nAQCvP9gUT0QHCU5ERLfi5qBEp+r2Pl6VItKbP6uuRsWGeHBypXpgIUVEVmnL6Qy8/ssxAMC4+0LwfLcwwYmI6E76VrX3/cH7pIj05o+qExPV7bNUNyykiMjqJFzIwYTlh6DRShjUtgHeergZZDKZ6FhEdAds7yPSL7b13TsWUkRkVZLS8zE2LgFlFVr0jPDBJ4+1glzOIorI1LG9j0i/2NZ371hIEZHVSMspxoiF8cgvrUD7YHd883Q72Cr4MkhkLh5pWXnWvHqWMSKqv/Vs67tnfAdBRFYhq7AMIxbFI7OgDE19nbFwZDTslQrRsYioDno394ONXIbT1/LZ3kd0Dy7fKMYRtvXdMxZSRGTxCkrLMWpxPFKzitDAzR5Lx8bA1cFWdCwiqiN3x3/a+/48watSRPW1ser/T0wjtvXdCxZSRGTRSss1eGZpIk5cyYenoxI/jIvlyu1EZuyhqrPnG1lIEdVb9f+fh9nWd09YSBGRxdJoJbzy4xHsS8mGk8oGS8bEIMTLUXQsIroHDzT3hVwGHL+Sh7ScYtFxiMxOZn4pEi/dAAAuQn+PWEgRkUWSJAlTfzuBP0+kQ6mQY/7wKEQ2cBUdi4jukZeTCtGNPAAAm07yqhRRXW06lQFJAtoGucHPlR0a94KFFBFZpNmbz2LFgUuQyYAvn2yju6+CiMzfg1XtfSykiOpuU1Vb34O8GnXPWEgRkcWJ25OKuVvPAQA+GBCJh9gDTmRRqtuRDl68gcyCUsFpiMzHjSI19qVkA/jnhATVHwspIrIovx25gvfWnQIAvPpAEwyNDRaciIj0LcDNHq0D3SBJwF8nM0THITIbf5/OgEYroZm/C4I9ec/wvWIhRUQWY/uZTLz601EAwKhOjTCpZ2PBiYjIUDh7H1HdVf9/eYhXo/SChRQRWYTDl27g+R8OoUIr4dHWAXi3b3PIZDLRsYjIQKrb+/alZCO3WC04DZHpKyyrwK7kLABs69MXFlJEZPaSMwowOi4BJeUadG3ijc8ebw25nEUUkSUL8XJEhJ8zNFoJm0+xvY/obrYmZUKt0SLUyxHhPk6i41gEFlJEZNau5JZgxKJ45BaXo02gG+YNawelDV/aiKwBZ+8jqr3q/yd9Iv3YsaEnfLdBRGYrp0iNEQsP4FpeKcK8HbF4VDQclDaiYxGRkVS39+1KzkKxukJwGiLTVVahwfakTACc9lyfWEgRkVkqKqvA6LgEnL9eBH9XOywbGwt3R6XoWERkRBF+zgj0sEdZhRY7z2aJjkNksvaez0aRWgM/Fzu05OL0esNCiojMjrpCi+d+SMTRtFy4Odhi2dgYBLjZi45FREYmk8nQu3nl2fW/TrG9j+h2qpcJeKC5L+8h1iMWUkRkVrRaCa+uPopdyVlwUCqweFQ0Gvs4i45FRIL0bu4LANhyOhMVGq3gNESmR/uvCVl6t/AVnMaysJAiIrMhSRKmrzuJdUevwlYhw3fDotA2yF10LCISKCrYHR6OSuSVlCP+Qo7oOEQm53BaLrIKy+BsZ4PYEE/RcSwKCykiMhtzt57Dkn0XIZMBnz3eGt2aeIuORESC2SjkuD/CB8A/7UtE9I/qtteeET6c1VbP+GwSkVn4Yf9FzN58FgAwrW9z9G/TQHAiIjIVvatmIdt8KgOSJAlOQ2Q6JEnSnWCovp+Q9IeFFBGZvPXHrmHqbycAAC/2bIxRnUMEJyIiU9Il3Av2tgpcyS3Byav5ouMQmYxzmYVIzSqCUiFHt6bs4tA3FlJEZNJ2J2fh5R8PQ5KAp2OD8MoDTURHIiITY2erQNcmXgCAv06xvY+oWvX/h86NPeGk4jqL+sZCiohM1tG0XDyz7CDKNRIebumH9/tHcjV2Irql6sV5/zrJadCJqlX/f+jNRXgNgoUUEZmk89cLMTouAcVqDTqFeeKLJ9pAwbUviOg2ekb4QCGXISm9AGk5xaLjEAmXkV+Ko5fzAEA3IQvpFwspIjI56XmlGLEwHjlFarRs4Ir5I9pDZaMQHYuITJibgxLtgyuXQ9hymu19RFtOZwIA2gS6wcfFTnAay8RCiohMSm6xGiMWHcCV3BKEejkibnQ0+7qJqFYeqFqc9++qN5BE1uzvqhMK1f8vSP9YSBGRyShRazAmLgFnMwrh66LCkjEx8HRSiY5FRGaiV7PKN4z7U7KRV1IuOA2ROEVlFdh9LgvAP/8vSP9YSBGRSSjXaDFheSIOXcqFi50Nlo6JRaCHg+hYRGRGGnk5ItzHCRVaCTvOXhcdh0iYXclZUFdoEeThgCa+TqLjWCwWUkQknFYr4fWfj2Hbmeuws5Vj0ahoNPVzFh2LiMxQr+r2Pk6DTlasuq2vVzNfznZrQCykiEgoSZLw4YbTWHv4ChRyGb4d2g7tG3mIjkVEZqq6jWnbmUyUa7SC0xAZn0YrYWtS5X2CvZpztj5DYiFFREJ9t+M8Fu5OBQB8+lgr9IxgLzcR1V+bQDd4OSlRUFqBhNQc0XGIjO7wpRvIKVLDxc4G0TwxaVAspIhImFXxlzBr4xkAwDuPNMOgdg0FJyIic6eQy9Czas2czZwGnaxQ9d99jwgf2Cr4Vt+Q+OwSkRAbT6TjrbXHAQDPdw/DuC6hghMRkaWobu/bfCoDkiQJTkNkXJtP/XN/FBkWCykiMrp957Px4qrD0ErAE+0D8XqfpqIjEZEF6RLuDZWNHJdvlOBMRoHoOERGk3K9ECnXi2Ajl6FbU2/RcSweCykiMqoTV/IwfulBqCu06N3cFx8OjOSMQkSkV/ZKBTo39gIAbOHivGRFqieZiA31gIudreA0lo+FFBEZzYWsIoxaHI/CsgrEhHjgq6fawob920RkAPc3q7xPagvvkyIrUj3t+f2cuMko+A6GiIwiM78UwxcdQFahGs38XfD9yPaws1WIjkVEFqp6wonDabnIKiwTnIbI8PJKypFw4QaAf04kkGGxkCIig8srKceIRfFIyylBsKcDloyJZssBERmUv6s9WgS4QJKA7Weui45DZHA7zl6HRiuhsY8Tgj0dRcexCiykiMigSss1GL/kIJLSC+DtrMKyMbHwcbYTHYuIrMD9EWzvI+uxtbqtj1ejjIaFFBEZTIVGi0krDiP+Qg6cVTZYMjoGQZ4OomMRkZW4v2r6551nr0NdoRWchshwKjRabKu68sr7o4yHhRQRGYQkSZiy5jj+Pp0BlY0c349sj+YBLqJjEZEVadnAFd7OKhSpNTiQmi06DpHBHLqUi7yScrg52KJdkJvoOFaDhRQRGcTHG5OwOvEyFHIZvn66HWJDPUVHIiIrI5fL0LNpdXsfp0Eny1XdvtqjqQ9nwzUiPtNEpHfzd57Hf3ekAABmDmyJB5qzzYCIxNBNg56UAUmSBKchMowtVetHVc9WScbBQoqI9OrnxMv4aEMSAODNhyIwJDpQcCIismb3hXtBaSNHWk4JkjMLRcch0ruL2UU4l1kIG7kM3Zp6i45jVVhIEZHebDmdgTd+OQYAGN8lBM92DRWciIisnYPSBp3CKluL2d5Hlqj67zq6kQeXFjEyky+krly5gmHDhsHT0xP29vZo2bIlDh48qPu6JEl499134e/vD3t7e/Tq1QvJyckCExNZp4QLOZiw/BA0WgmD2zXElIeaQSaTiY5FRKRrd9qWxEKKLM/Wqr9rTntufCZdSN24cQOdO3eGra0t/vzzT5w6dQqff/453N3ddfvMmjULX331FebNm4cDBw7A0dERffr0QWlpqcDkRNYlKT0fY+ISUFahxf0RPvh4cEvI5SyiiMg09KiacOLgxRzkFqsFpyHSn8KyCt2MlLw/yvhMupD65JNPEBgYiMWLFyMmJgYhISHo3bs3wsLCAFRejZozZw7eeecd9O/fH61atcLSpUtx9epV/Prrr2LDE1mJtJxijFgYj4LSCrQPdsfXT7eDLWcMIgvHbgnzEujhgCa+TtBKwI6z10XHIdKb3cnXUa6R0MjTAaHeTqLjWJ16vdtZtmwZOnfujICAAFy8eBEAMGfOHPz22296Dff777+jffv2ePzxx+Hj44O2bdtiwYIFuq+npqYiPT0dvXr10m1zdXVFbGws9u3bd9vHLSsrQ35+fo0PIqq76wVlGL7wADILytDU1xkLR0bDXqkQHYuslLHGJnZLmKeeVYuUbmV7H1mQrbrZ+jg7rgh1LqS+++47TJ48GQ8//DByc3Oh0WgAAG5ubpgzZ45ew6WkpOC7775DeHg4Nm3ahOeffx4vvvgilixZAgBIT08HAPj61vzj8fX11X3tVmbOnAlXV1fdR2AgZxUjqquC0nKMWhyPC9nFaOhuj6VjY+DqwJtcSQxjjk3sljBP1W1PO85eh0bLadDJ/Gm1ErYmVV5hZVufGHUupObOnYsFCxbg7bffhkLxz5nn9u3b4/jx43oNp9Vq0a5dO3z00Udo27YtnnnmGYwfPx7z5s27p8edMmUK8vLydB9paWl6SkxkHUrLNXhmaSJOXs2Hp6MSy8bGwtfFTnQssmLGHJsM0S3BTgnDaxfkBld7W+QWl+PwpRui4xDdsxNX85BVWAZHpQIxIR6i41ilOhdSqampaNu27U3bVSoVioqK9BKqmr+/P5o3b15jW7NmzXDp0iUAgJ+fHwAgIyOjxj4ZGRm6r92KSqWCi4tLjQ8iqh2NVsLLq45gX0o2nFQ2iBsdgxAvR9GxyMoZc2wyRLcEOyUMz0YhR7cmlWvsbGF7H1mA6mnPu4R7Q2nDe5NFqPOzHhISgiNHjty0fePGjWjWrJk+Mul07twZZ86cqbHt7NmzCA4O1mXx8/PDli1bdF/Pz8/HgQMH0LFjR71mIaLKlqV3fj2OjSfToVTIMX94FFo2dBUdi8ioY5MhuiXYKWEcnAadLMm2M9X3R7GtTxSbun7D5MmTMXHiRJSWlkKSJMTHx2PlypWYOXMmvv/+e72Ge+WVV9CpUyd89NFHGDJkCOLj4zF//nzMnz8fACCTyfDyyy/jgw8+QHh4OEJCQjB16lQEBARgwIABes1CRMDnf53Fyvg0yGXAl0+2QafGXqIjEQEw7th0u26JX375BUDNbgl/f3/dPhkZGWjTps0tH1OlUkGlUuk1J92sWxNvyGVAUnoBruSWoIGbvehIRPWSmV+KY5fzAADdI7wFp7FedS6kxo0bB3t7e7zzzjsoLi7G008/jYCAAHz55Zd48skn9RouOjoaa9euxZQpUzBjxgyEhIRgzpw5GDp0qG6f119/HUVFRXjmmWeQm5uL++67Dxs3boSdHe/XINKnRbtT8fW2cwCADwa0xEMt/e/yHUTGY8yxqS7dEtWFU3W3xPPPP6/XLFQ37o5KtAtyx8GLN7A1KRPDOwSLjkRUL9vPVE4y0aqhK3yc+Z5XFJkkSfWeuqa4uBiFhYXw8THvS4r5+flwdXVFXl4e75ciuoVfD1/Byz8eAQD8p3cTTOoZLjYQWQxDvP4aemxKSEhAp06dMH36dF23xPjx4zF//nzdib5PPvkEH3/8MZYsWaLrljh27BhOnTpVqxN9HJcM59vt5zBr4xn0jPDBolHRouMQ1ctzyxKx8WQ6Xro/HK880ER0HItT29fgOl+R+jcHBwc4ODjcy0MQkYnbfiYT/1l9FAAwqlMjTOzRWHAiojsz9NjEbgnz1qOpD2ZtPIO957NQWq6BnS3XviPzoq7QYve5LAC8P0q0WhVSbdu2hUwmq9UDHjp06J4CEZHpOHTpBp7/4RAqtBL6twnAu32b1/q1gMjQRI5Nffv2Rd++fW/7dZlMhhkzZmDGjBl6/bl07yL8nOHvaodreaXYl5KNHk35RpTMy8ELOSgsq4CXkwotG3DCJ5FqVUhx4gYi65OcUYAxcQkoKdegWxNvfPpYa8jlLKLIdHBsovqQyWToEeGDFQcuYVtSJgspMjtbq2ad7N7Um+OyYLUqpKZNm2boHERkQq7klmD4wnjkFpejbZAbvhvWjmtUkMnh2ET11aNpZSG1NSkT0x+VeKWdzEr1tOc8CSAe3xkRUQ3ZhWUYvvAA0vNL0djHCYtGRsNBeU+3UxIRmZTOjT2hVMhx+UYJzl8vFB2HqNYuZRfj/PUiKOQydGnCJUhEq3MhpdFo8NlnnyEmJgZ+fn7w8PCo8UFE5quorAJj4hKQcr0IAa52WDY2Bu6OStGxiO6KYxPVhYPSBrGhlX8X25KuC05DVHvVV6PaB7vDxc5WcBqqcyE1ffp0zJ49G0888QTy8vIwefJkDBo0CHK5HO+9954BIhKRMZRVaPDsskQcvZwHdwdbLB0bC39XLlZJ5oFjE9VV9Wxn1febEJmD6r9XztZnGupcSC1fvhwLFizAq6++ChsbGzz11FP4/vvv8e6772L//v2GyEhEBqbRSnj1p6PYfS4LDkoFFo+OQWMfJ9GxiGqNYxPVVfX9JQkXcpBfWi44DdHdFasrsC8lGwDQg4WUSahzIZWeno6WLVsCAJycnJCXlwegcirY9evX6zcdERmcJEmYvu4k/jh2DbYKGf47PAptAt1ExyKqE45NVFeNvBwR6uWICq2EPclZouMQ3dW+89lQV2jRwM0e4TzZaRLqXEg1bNgQ165dAwCEhYXhr7/+AlC50rtKpdJvOiIyuC+3JGPpvouQyYDZQ9qgS7i36EhEdcaxieqjB9v7yIz8u62PM02ahjoXUgMHDsSWLVsAAC+88AKmTp2K8PBwjBgxAmPGjNF7QCIynGX7LmDO38kAgBmPtkC/1gGCExHVD8cmqo/q9r5tZ65Dq5UEpyG6PUmSsP1M5cQoPSJ4wtNU1HlO448//lj37yeeeALBwcHYu3cvwsPD0a9fP72GIyLD+ePYVbz7+0kAwEv3h2N4x0ZiAxHdA45NVB/RIe5wVCqQVViGU9fyEdnAVXQkoltKzizEldwSqGzk6BjKac9NxT0vDtOhQwd06NBBH1mIyEh2J2fhlR+PQJKA4R2C8XKvcNGRiPSKYxPVhspGgc6NvfDXqQxsS8pkIUUma1tVW1/HME/YKxWC01C1Orf2zZw5E4sWLbpp+6JFi/DJJ5/oJRQRGc7RtFw8s+wgyjUSHmnlj/cebcFeazJ7HJuovqrvk6pen4fIFFX/fVa3o5JpqHMh9d///hcRERE3bW/RogXmzZunl1BEZBjnMgsxOi4BxWoN7mvshdlDWkMhZxFF5o9jE9VX96aV95scTstFTpFacBqim+WXluPghRsAWEiZmnpNf+7v73/Tdm9vb92MSURkeq7llWDkonjkFKnRqqEr5g2PgsqG7QFkGTg2UX35u9ojws8ZkgTsSr4uOg7RTfYkZ6FCKyHU2xFBng6i49C/1LmQCgwMxJ49e27avmfPHgQEcMYvIlOUW6zGiIXxuJJbglAvRyweFQ0n1T3fIklkMjg20b3QtfdxGnQyQWzrM111fic1fvx4vPzyyygvL0fPnj0BAFu2bMHrr7+OV199Ve8BiejeFKsrMCYuAcmZhfBzscPSsTHwdOK6OmRZODbRvejR1AffbT+PHWevQ6OV2PJMJkOSJGyrnvachZTJqXMh9dprryE7OxsTJkyAWl3ZS2xnZ4c33ngDU6ZM0XtAIqq/co0WE5YfwqFLuXC1t8XSsTFo6M62ALI8HJvoXrQLcoOznQ1uFJfj6OVctAtyFx2JCABw8mo+rheUwUGpQHQI/y5NTZ0LKZlMhk8++QRTp07F6dOnYW9vj/DwcK4cT2RitFoJr60+iu1nrsPOVo5Fo6LRxNdZdCwig+DYRPfCRiFH1ybeWH/sGrYnZbKQIpOxvaqtr3NjL97XbILqfI9UNScnJ0RHR8PZ2Rnnz5+HVqvVZy4iugeSJOH99afw65GrsJHL8N2wKEQF840BWT6OTVRf1W1T1W1URKaAbX2mrdaF1KJFizB79uwa25555hmEhoaiZcuWiIyMRFpamt4DElHdfbv9PBbvuQAA+Ozx1nwBJovFsYn0pVuTymnQj1/JQ2ZBqeA0RMCNIjUOX6qc9rx6mn4yLbUupObPnw9393/OaG/cuBGLFy/G0qVLkZCQADc3N0yfPt0gIYmo9lbGX8Knm84AAN7t2xwD2jYQnIjIcDg2kb54O6vQqqErAGAHr0qRCdiZfB1aCYjwc0aAm73oOHQLtS6kkpOT0b59e93nv/32G/r374+hQ4eiXbt2+Oijj7BlyxaDhCSi2tl44hreXnscADChexjG3BciOBGRYXFsIn3qXnX1fjsLKTIB1X+H3dlVYrJqXUiVlJTAxcVF9/nevXvRtWtX3eehoaFIT0/XbzoiqrW957Pw4soj0ErAk9GBeK1PU9GRiAyOYxPpU4+q9qmdyddRruH9dSSOVithx9nq+6PY1meqal1IBQcHIzExEQCQlZWFkydPonPnzrqvp6enw9XVVf8JieiuTlzJwzNLE6HWaNGnhS8+GBAJmYzroJDl49hE+tSqoRs8HJUoKK3AoYs3RMchK3bsSh5yitRwtrNBO04WZbJqPf35yJEjMXHiRJw8eRJbt25FREQEoqKidF/fu3cvIiMjDRKSiG4vNasIIxfFo7CsAh1CPfDlk21ho6j3hJxEZoVjE+mTQi5D13Av/HrkKraduY7YUE/RkchKbUuqnPa8S7gXbDmmm6xaF1Kvv/46iouLsWbNGvj5+WH16tU1vr5nzx489dRTeg9IRLeXkV+K4QsPILtIjeb+Lpg/oj3sbLnOBFkPjk2kbz0ifPDrkavYfiYTbz4UIToOWanq9aN4f5Rpk0mSJIkOIVp+fj5cXV2Rl5dXo9eeyJTlFZfjifn7kJRegGBPB/z8XCd4O3PxUTIvfP29NT4v4twoUqPdB5shScDeN3tytjQyuusFZYj+8G8AQPxb98PHxU5wIutT29dgXiskMkMlag3GLU1AUnoBvJ1VWDYmlkUUEZEeuDsq0TbQDQB0N/sTGdPOqr+7yAYuLKJMHAspIjNTrtFi0opDSLhwA852Nlg6JgZBng6iYxERWYzqRcyr71MhMqZtVW19PdjWZ/JYSBGZEUmS8OYvx7ElKRMqGzkWjoxGM3+2/RAR6VOPiMo3sHvOZaGsQiM4DVmTCo0Wu5KzAPD+KHPAQorIjMz8Mwm/HLoMhVyGb55uh5gQD9GRiIgsTnN/F3g5qVCk1iAhldOgk/EcSctFXkk53Bxs0aaqxZRMV50LqRMnTtz2a7/++uu9ZCGiO/jvjvOYvzMFAPDxoJbo1dxXcCIi08GxifRJLpehe9UiqNWzpxEZQ3VbX5dwbyjkXA/S1NW5kOrTpw9SU1Nv2v7LL79g6NCheglFRDWtPpiGmX8mAQCmPBSBx9sHCk5EZFo4NpG+6e6TYiFFRrQtqXKiiZ4R3oKTUG3UuZAaN24cevXqhfT0dN22H3/8ESNGjEBcXJw+sxERgL9PZeDNNccBAM92DcWz3cIEJyIyPRybSN/uC/eCQi7D+etFuJRdLDoOWYH0vFKcupYPmQzoGs5CyhzUuZCaPn06Hn74YfTq1Qs5OTlYsWIFRo8ejaVLl+Lxxx83REYiq5VwIQcTVxyCRivhsaiGXByS6DY4NpG+udrbIirYHQCw/SyvSpHh7aj6O2vd0A2eTlzSxBzUa7KJuXPnonXr1ujQoQPGjx+PlStXYvDgwfrORmTVTl/Lx5i4BJRVaHF/hA8+HtQSMhn7pYluh2MT6RunQSdjqm7r47Tn5sOmNjv9/vvvN20bNGgQdu3ahaeeegoymUy3z6OPPqrfhERWKC2nGCMWxaOgtALRjdzxzdB2sFFwkk2if+PYRIbWI8Ibn2xMwt7z2Sgt18DOViE6ElkodYUWu89VTnveg/dHmQ2ZJEnS3XaSy2v3Bk4mk0GjMb/1FvLz8+Hq6oq8vDy4uHBNHhLrekEZHp+3FxeyixHh54wfn+0IV3tb0bGIDOJeXn8teWziuGQaJElCp4+34lpeKeJGR3NdHzKYfeez8dSC/fByUiL+rV6Qc8Y+oWr7GlyrUUir1dbqw9wGKiJTU1BajlGL43EhuxgN3e2xdEwMiyii2+DYRIYmk/17GvTrgtOQJaueZr9rE28WUWZEL71Cubm5+ngYIqtWWq7B+KUHcfJqPryclFg2NhY+LnaiYxGZLY5NpA/VV6G2JmWiFk08RPVSPc0+748yL3UupD755BP8+OOPus8ff/xxeHh4oEGDBjh69KhewxFZC41WwkurDmN/Sg6cVDaIGx2DEC9H0bGIzAbHJjKUzo29YKuQ4VJOMVKzikTHIQt0+UYxzmYUQs5pz81OnQupefPmITCwcjHQzZs34++//8bGjRvx0EMP4bXXXtN7QCJLJ0kS3vn1ODadzIBSIcf8EVGIbOAqOhaRWeHYRIbipLJBTIgHgMqrUkT6Vt02GhXsDlcHtvObk1rN2vdv6enpusHqjz/+wJAhQ9C7d280atQIsbGxeg9IZOk+/+ssVsanQS4DvnqqDTqFeYmORGR2ODaRIfVo6oM957Kx4+x1jOsSKjoOWZjq+6M4mYn5qfMVKXd3d6SlpQEANm7ciF69egGoPKvOG3qJ6mbR7lR8ve0cAODDgS3xYKS/4ERE5oljExlS9RvcAyk5KCqrEJyGLElpuQZ7zmUD4P1R5qjOV6QGDRqEp59+GuHh4cjOzsZDDz0EADh8+DAaN26s94BElurXw1cw449TAIDX+jTFUzFBghMRmS+OTWRIYd6OCPJwwKWcYuw5l4XeLfxERyILcSA1ByXlGvi52KGZv7PoOFRHdb4i9cUXX2DSpElo3rw5Nm/eDCcnJwDAtWvXMGHCBL0HJLJE285k4j+rK2+AH925ESZ0DxOciMi8cWwiQ5LJZOhRNQ36Nk6DTnq0req+ux4R3pDJOO25uanVgryWjgsfkjElXryBod/vR2m5FgPaBGD2kDZcM4KsFl9/b43Pi+nZdiYToxcnwN/VDnvf7Mk3vaQX3T/dhgvZxfjv8Cj04ZVOk1Hb1+A6t/ZVO3XqFC5dugS1Wl1j+6OPPlrfhySyeGczCjAmLgGl5Vp0b+qNTx9vzSKKSI84NpGhdAz1hJ2tHNfySpGUXoBm/ixw6d6kXC/Ehexi2Cpk6NyYE02ZozoXUikpKRg4cCCOHz8OmUymW5yu+swMb+olurXLN4oxYmE88krK0TbIDd8ObQdbhV7WxCayehybyNDsbBXoFOaFrUmZ2HYmk4UU3bPqNtHYEE84qep9bYMEqvO7uJdeegkhISHIzMyEg4MDTp48iZ07d6J9+/bYvn27ASISmb/swjKMWBiP9PxShPs4YfGoaDgo+aJJpC8cm8gYekRUzqq2PYn3SdG9+2facy7Ca67q/E5u37592Lp1K7y8vCCXyyGXy3Hfffdh5syZePHFF3H48GFD5CQyW4VlFRgdl4CUrCI0cLPH0rExcHNQio5FZFE4NpExdG9S+YY38dIN5BWXc/FUqreisgocSMkB8E+BTuanzlekNBoNnJ0rp2f08vLC1atXAQDBwcE4c+aMftMRmbmyCg2eXXYQxy7nwcNRiaVjY+Dvai86FpHF4dhExhDo4YBwHydotBJ2JvOqFNXfnnNZUGu0CPJwQKiXo+g4VE91viIVGRmJo0ePIiQkBLGxsZg1axaUSiXmz5+P0FCu9k1UTaOVMPmno9hzLhsOSgUWj4pGmLeT6FhEFoljExlLzwgfJGcWYltSJvq1DhAdh8xU9f1RPSN8OAOkGavzFal33nkHWq0WADBjxgykpqaiS5cu2LBhA7766iu9ByQyR5IkYdrvJ7D+2DXYKmT47/AotA50Ex2LyGJxbCJj6d60sg1rx9nr0GqtfgUZqgdJknh/lIWo8xWpPn366P7duHFjJCUlIScnB+7u7qyoiarM+TsZP+y/BJkM+OKJNugSzhdKIkPi2ETG0r6RO5ztbJBdpMbRy7loG+QuOhKZmdPXCnAtrxT2tgp0CPUUHYfugV7mXvbw8OBARVRl6b4L+HJLMgBgRv9I9G3F1g8iETg2kSHYKuToWnVybFtSpuA0ZI62VV2N6tzYE3a2CsFp6F7U+orUmDFjarXfokWL6h2GyNytO3oV034/CQB46f5wDO8QLDgRkWXj2EQi9Ijwwfrj17D1TCYm924qOg6Zma1VBThn6zN/tb4iFRcXh23btiE3Nxc3bty47Ychffzxx5DJZHj55Zd120pLSzFx4kR4enrCyckJgwcPRkZGhkFzEN3KzrPXMfmnI5AkYHiHYLzcK1x0JCKLZwpjE1mfblXToJ+4ko/M/FLBacic3ChS4/ClytekHk1ZSJm7Wl+Rev7557Fy5UqkpqZi9OjRGDZsGDw8PAyZrYaEhAT897//RatWrWpsf+WVV7B+/XqsXr0arq6umDRpEgYNGoQ9e/YYLRvRkbRcPPdDIso1Eh5p5Y/3Hm3BliIiIxA9NpF18nZWoXVDVxy9nIdtZzLxRHSQ6EhkJnacvQ6tBET4OSPAjcuhmLtaX5H65ptvcO3aNbz++utYt24dAgMDMWTIEGzatAmSZNhZawoLCzF06FAsWLAA7u7/3NSZl5eHhQsXYvbs2ejZsyeioqKwePFi7N27F/v37zdoJqJq5zILMXpxPIrVGtzX2Auzh7SGQs4iisgYRI5N1dgtYZ2q27K28j4pqoPqv5eebOuzCHWabEKlUuGpp57C5s2bcerUKbRo0QITJkxAo0aNUFhYaKiMmDhxIh555BH06tWrxvbExESUl5fX2B4REYGgoCDs27fvto9XVlaG/Pz8Gh9E9XEtrwQjFh7AjeJytGroinnDo6Cy4Y2jRMYkamwC7twtsW7dOqxevRo7duzA1atXMWjQIINmIeOqfiO8OzkL6gqt4DRkDio0Wuw4+8/6UWT+6j1rn1wuh0wmgyRJ0Gg0+sxUw6pVq3Do0CHMnDnzpq+lp6dDqVTCzc2txnZfX1+kp6ff9jFnzpwJV1dX3UdgYKC+Y5MVuFGkxvCF8biaV4pQb0csHhUNJ1WdVxQgIj0y1tgEsFvC2kUGuMLLSYUitQYJF3JExyEzcDgtF3kl5XBzsOW0+RaiToVUWVkZVq5ciQceeABNmjTB8ePH8fXXX+PSpUtwcnLSe7i0tDS89NJLWL58Oezs7PT2uFOmTEFeXp7uIy0tTW+PTdahWF2BMUsScC6zEH4udlg6JgaeTirRsYiskrHHpmr67JZgp4T5kctl6FG1mCrb+6g2qv9OujXx5i0AFqLWp88nTJiAVatWITAwEGPGjMHKlSvh5eVlyGxITExEZmYm2rVrp9um0Wiwc+dOfP3119i0aRPUajVyc3NrXJXKyMiAn5/fbR9XpVJBpeKbXqofdYUWz/1wCIcv5cLV3hZLx8agobuD6FhEVknE2AT80y2RkJBw09fq0y0xc+ZMTJ8+3RBRyYB6RvhgdeJlbEvKxNS+zUXHIRO3jfdHWZxaF1Lz5s1DUFAQQkNDsWPHDuzYseOW+61Zs0Zv4e6//34cP368xrbRo0cjIiICb7zxBgIDA2Fra4stW7Zg8ODBAIAzZ87g0qVL6Nixo95yEFXTaiW89vNR7Dx7Hfa2CiwaFY0mvs6iYxFZLRFjU3W3xObNm/XWLTFlyhRMnjxZ93l+fj7bzs3AfeFesFXIkJJVhNSsIoR4OYqORCbqSm4JktILIJdBt6Azmb9aF1IjRoww+nTOzs7OiIyMrLHN0dERnp6euu1jx47F5MmT4eHhARcXF7zwwgvo2LEjOnToYNSsZPkkScKMP07htyNXYSOX4bth7RAVzB5nIpFEjE2G6JZgp4R5crazRUyIB/acy8bWpEyMvS9EdCQyUdVtfVHB7nB3VApOQ/pS60IqLi7OgDHq74svvoBcLsfgwYNRVlaGPn364NtvvxUdiyzQN9vOIW7vBQDA50NaozsX0iMSTsTYxG4J+reeEb5VhVQGCym6ra2nK5c/6BnhKzgJ6ZPZTTG2ffv2Gp/b2dnhm2++wTfffCMmEFmFFQcu4bO/zgIA3u3bHP3bNBCciIhEYbcE/dv9ET54/49TOJCSg4LScjjb2YqORCamWF2BPeezAQD3N+NJWEtS7+nPiazFn8ev4Z1fK88+T+rRGGN4xpGI7uKLL75A3759MXjwYHTt2hV+fn56vU+LTEcjL0eEejuiQithV3KW6Dhkgvaey4a6QouG7vYI9zHcTKJkfGZ3RYrImPaez8JLq45AKwFPxQTh1d5NREciIhPEbgnrdn+ED1Kup2LL6Uw83NJfdBwyMVuq7o+6P8LH6Pd0kmHxihTRbZy4kodnliZCrdHiwRZ++GBAJF8AiYjoJtX3vWw/kwmNVhKchkyJJEnYmlR1f1Qz3h9laVhIEd1CalYRRi6KR2FZBTqGemLOk224eB4REd1S+0bucLazQXaRGkcv54qOQybk5NV8ZOSXwUGpQGyIh+g4pGcspIj+T0Z+KYYvPIDsIjVaBLhg/ogo2NkqRMciIiITZauQo1uTyrWBtp7OFJyGTEn1tOf3NfbiewkLxEKK6F/yissxclE8Lt8oQSNPB8SNjuEMTEREdFfVs7FV3w9DBPzr/ijO1meRWEgRVSlRazB2SQKS0gvg46zCsrGx8HbmAplERHR33Zr4QC4DTl/Lx9XcEtFxyARcLyjD0bRcAEAPrj1pkVhIEQEo12gxacUhHLx4Ay52Nlg6NgaBHg6iYxERkZnwcFSiXZA7AF6Vokrbqv4OWjZwhY+LneA0ZAgspMjqabUS3vzlOLYkZUJlI8fCUdGI8HMRHYuIiMzM/VWzsv19KkNwEjIFm09X/h304mx9FouFFFm9jzcm4ZdDl6GQy/Dt0HaIbsRZdYiIqO56Vd0Hs+98NorKKgSnIZFKyzXYXbVAM++PslwspMiqzdtxHvN3pgAAPhncSnc2kYiIqK4a+zghyMMBao0Wu6reRJN12nc+GyXlGvi72qFFALtcLBULKbJaPyWk4eM/kwAAbz/cDI9FNRSciIiIzJlMJtO1cf19mu191qy6ra9nhA9kMq5DaalYSJFV+utkOt5ccwwA8Gy3UIzvGio4ERERWYLq9r5tSZnQaCXBaUgESZJ064n1as5OF0vGQoqszoGUbExaeRhaCXg8qiHefDBCdCQiIrIQ0SEecLazQXaRGkeqpr4m63Lyaj7S80vhoFSgY6in6DhkQCykyKqcupqPcUsOQl2hRa9mvpg5qCUvuRMRkd7YKuToXrVmENv7rNPmqlkb72vsBTtbheA0ZEgspMhqXMouxohF8Sgoq0BMIw98/XRb2Cj4X4CIiPSrur1vCwspq7QlidOeWwu+iySrkFlQiuGLDiCrsAwRfs5YMLI9zxIREZFBdG/iA4VchrMZhbiUXSw6DhnRtbwSnLiSD5kM6BHBac8tHQspsnj5peUYuSgBF7OLEehhj6VjYuBqbys6FhERWShXB1tEN3IHwPY+a7OlapKJNoFu8HZWCU5DhsZCiixaabkG45YcxOlr+fByUmHZmFj4uNiJjkVERBauuq2r+n4Zsg7Vv2+29VkHFlJksSo0Wry48jDiU3PgrLJB3OhoNPJyFB2LiIisQO/mfgCA+As5yC1WC05DxlBYVoF957MBAL057blVYCFFFkmSJLy99gT+OpUBpY0c80e0R2QDV9GxiIjISgR5OqCprzM0WgnbzmSKjkNGsOPMdag1WjTydEBjHyfRccgIWEiRRZq16Qx+PJgGuQz46sm26BjGdRyIiMi4HmjO9j5rsvlUOoDK3zuXVrEOLKTI4ny/KwXfbT8PAPhoYEs8GOknOBEREVmj6kJqx5nrKKvQCE5DhlSu0WJrUuWVxwea832HtWAhRRZlzaHL+GD9aQDAa32a4smYIMGJiIjIWrVs4ApfFxWK1Brsrbp3hixTQmoO8ksr4OGoRFSwu+g4ZCQspMhibE3KwGs/HwMAjOkcggndwwQnIiIiayaXyzh7n5X4q+r32zOicg0xsg4spMgiJF7MwYTlh6DRShjYtgHeeaQZ+5OJiEi46va+v09lQKuVBKchQ5AkSVcoP8DZ+qwKCykye2fSCzB6cQJKy7Xo0dQbsx5rBTnPBhERkQnoGOYJJ5UNMgvKcPRyrug4ZACnrxXgSm4JVDZydAn3Eh2HjIiFFJm1tJxijFh0APmlFYgKdse3Q6Ngq+CfNRERmQaVjQLdmnoDYHufpar+vXYJ94aD0kZwGjImvuMks5VVWIYRi+KRkV+Gpr7OWDQyGvZKhehYRERENVQvzvoXCymL9FfVtOdchNf6sJAis1RYVoHRixOQmlWEBm72WDImBq4OtqJjERER3aR7Ux/YKmQ4l1mI89cLRcchPUrLKcbJq/mQy4D7m/mIjkNGxkKKzE5ZhQbPLD2I41fy4OGoxLKxMfBztRMdi4iI6JZc7W3RMazy3plNJ9MFpyF9qr7KGN3IA55OKsFpyNhYSJFZ0WglvPLjEew9nw1HpQJxo6MR6u0kOhYREdEd9WlR2fa16STb+yxJdWHcpwUX4bVGLKTIbEiShKm/ncCG4+lQKuSYP6I9WjV0Ex2LiIjorh5o7guZDDialov0vFLRcUgPsgrLkHAhBwDQuwXvj7JGLKTIbHzxdzJWHLgEmQyY82QbdG7MKUaJiMg8+DjboV2QO4B/Jicg8/b3qQxIEhDZwAUN3R1ExyEBWEiRWViy9wK+2pIMAHi/fyQebukvOBEREVHd/NPex0LKElT/Hh9kW5/VYiFFJu/3o1fx3rqTAIDJDzTBsA7BghMRERHVXfV9NPtTcnCjSC04Dd2LgtJy7DmXDYD3R1kzFlJk0naevY5XfzoCSQJGdgzGCz0bi45ERERUL8Gejojwc4ZGK2FLUqboOHQPtp+5DrVGi1AvRzT24aRX1oqFFJmsI2m5eO6HRJRrJPRrHYBp/VpAJpOJjkVERFRvvauuXrC9z7xtrPr99W7hx/cmVoyFFJmkc5mFGL04HsVqDbqEe+Hzx1tDLucLFRERmbfq+2l2nr2OorIKwWmoPkrLNdhedUWxD2frs2ospMjkXM0twYiFB3CjuBytA90wb1gUlDb8UyUiIvPXzN8ZwZ4OKKvQYvuZ66LjUD3sSs5CkVqDAFc7tAl0Ex2HBOK7UzIpN4rUGLEoHlfzShHq7YjFo6LhqLIRHYuIiEgvZDIZHoysvCq14cQ1wWmoPv48Xvl76xPJtj5rx0KKTEaxugKj4xJwLrMQ/q52WDY2Fh6OStGxiIiI9OrhyMolPLYlZaK0XCM4DdWFukKLzaczAIBLsRALKTIN6gotnvvhEI6k5cLNwRZLx8SggZu96FhERER616qhKxq42aNYrcGOs2zvMyd7zmehoLQCPs4qRFUtsEzWi4UUCafVSvjP6qPYefY67G0VWDQqGuG+zqJjERERGYRMJsNDVe191W1iZB6qf18PRvpxEixiIUViSZKEGX+cwu9Hr8JGLsN3w9qhHc/wEBGRhXuoZWUh9ffpTJRVsL3PHJRrtPjrVGVbX/V9bmTdWEiRUF9vPYe4vRcAAJ8PaY3uTX3EBiIiIjKCtoHu8HVRobCsAruTs0THoVrYn5KN3OJyeDoqEdPIQ3QcMgEspEiY5Qcu4vPNZwEA0/o1R/82DQQnIiIiMg65XIaHqiad+PMEF+c1B9W/p94t/GCj4FtoYiFFgvx5/Bre+fUEAOCFno0xunOI4ERERETGVX2f1F8n06Gu0ApOQ3ei0UrYVFVIPdySbX1UiYUUGd3ec1l4adURSBLwVEwQJj/QRHQkIiIio2vfyANeTirkl1Zgz3m295myAynZyC5Sw9XeFh1CPUXHIRPBQoqM6vjlPIxfehBqjRYPtvDDBwMiuZgdERFZJYX8n9n7/jjK2ftM2bpjVbP1tfCDLdv6qAr/EshoUq4XYtTieBSpNegY6ok5T7aBglOHEhGRFevbqvI+qb9OpXP2PhNVodFi44nKQqpvay7CS/9gIUVGkZFfiuEL45FdpEZkAxfMHxEFO1uF6FhERERCRTfygK+LCgWlFdh1lu19pmjv+WzcqJqtryPb+uhfWEiRweUVl2PEwnhcyS1BI08HxI2OgbOdrehYREREwsnlMjzcsvIqxx/HrgpOQ7dS/Xt5MJKz9VFN/GsggypRazB2SQLOZBTAx1mFZWNj4eWkEh2LiIjIZPRtFQAA2HwqA6XlbO8zJeoKLTZWzdZX/XsiqsZCigymXKPFxBWHcPDiDbjY2WDp2BgEejiIjkVERGRS2gW5oYGbPYrUGmw/kyk6Dv3L7nPXkV9aAW9nFWJCuAgv1cRCigxCq5Xwxs/HsDUpEyobORaNikaEn4voWERERCZHJpPhkapJJ6pnhyPTUD2b4iMt/TlBFt2EhRTpnSRJ+GjDaaw5fAUKuQzfDm2H9o14FoeIiOh2Hqm6T2rr6UwUqysEpyEAKC3X4K9TGQCgK3SJ/o2FFOndvB0p+H53KgBg1uBWuL+Zr+BEREREpq1VQ1cEeTigpFyDLafZ3mcKdpy9jsKyCvi52CEqyF10HDJBLKRIr35KSMMnG5MAAO880gyDoxoKTkRERGT6ZDKZbk2p345w9j5T8HvV76FvK3/I2dZHt2DShdTMmTMRHR0NZ2dn+Pj4YMCAAThz5kyNfUpLSzFx4kR4enrCyckJgwcPRkZGhqDE1m3TyXS8ueYYAOC5bmEY1yVUcCIiIiLzMaBtAwDAjrOZyC1WC05j3QpKy/H36cr3k9W/F6L/Z9KF1I4dOzBx4kTs378fmzdvRnl5OXr37o2ioiLdPq+88grWrVuH1atXY8eOHbh69SoGDRokMLV1OpCSjRdWHoZWAoa0b4g3HmwqOhIREZFZaeLrjGb+LijXSNhwPF10HKu26WQGyiq0CPV2RIsATpZFt2YjOsCdbNy4scbncXFx8PHxQWJiIrp27Yq8vDwsXLgQK1asQM+ePQEAixcvRrNmzbB//3506NBBRGyrc+pqPsYtOQh1hRYPNPfFRwNbQibjJXAiIqK66t8mAKev5ePXI1fwdGyQ6DhW67cjVwAAA9o04Hsaui2TviL1//Ly8gAAHh6VM8AlJiaivLwcvXr10u0TERGBoKAg7Nu377aPU1ZWhvz8/BofVD8Xs4swYlE8CsoqEBPigblPteWq30Rk0dh2Tob0aOsAyGRAfGoOruaWiI5jlTILSrHnXBaAysKW6HbM5h2vVqvFyy+/jM6dOyMyMhIAkJ6eDqVSCTc3txr7+vr6Ij399pfEZ86cCVdXV91HYGCgIaNbrMyCUgxfGI+swjI083fBghHtYWerEB2LiMig2HZOhhTgZo+YqiVDfj/KSSdE+OPoNWgloE2gG4I9HUXHIRNm0q19/zZx4kScOHECu3fvvufHmjJlCiZPnqz7PD8/n8VUHeWXlmPkogRcyilGkIcDloyJhqu9rehYREQGx7ZzMrT+bRrgQGoOfj18Bc91CxMdx+r809bHq1F0Z2ZxRWrSpEn4448/sG3bNjRs+M902n5+flCr1cjNza2xf0ZGBvz8/G77eCqVCi4uLjU+qPZKyzUYv+QgTl/Lh5eTCsvGxsDH2U50LCIiIfTRds6Wc/q3h1v6wVYhQ1J6Ac6kF4iOY1VSs4pw9HIeFHIZHmnFQoruzKQLKUmSMGnSJKxduxZbt25FSEhIja9HRUXB1tYWW7Zs0W07c+YMLl26hI4dOxo7rlWo0GjxwsrDOJCaA2eVDZaMieZlbyKyWvpqO2fLOf2bm4MS3Zv6APjn6ggZR/Xz3bmxF7ydVYLTkKkz6UJq4sSJ+OGHH7BixQo4OzsjPT0d6enpKCmpvPnS1dUVY8eOxeTJk7Ft2zYkJiZi9OjR6NixI1snDECSJLy19jg2n8qA0kaOBSPbo0WAq+hYRETCVLedr1q16p4eZ8qUKcjLy9N9pKWl6SkhmasBbSrXLvr18BVotZLgNNZBkiSsPcy2Pqo9k75H6rvvvgMAdO/evcb2xYsXY9SoUQCAL774AnK5HIMHD0ZZWRn69OmDb7/91shJrcOsTWfw08HLkMuAuU+1RYdQT9GRiIiEqW4737lz523bzv99VepObecqlQoqFc9+0z/ub+YDFzsbXM0rxb6UbHRu7CU6ksU7ePEGLmYXw1GpwIORt79FhKiaSV+RkiTplh/VRRQA2NnZ4ZtvvkFOTg6KioqwZs2aO94fRfXz/a4UfLf9PADg40Gt0KcFn2Misk5sOydjsLNVoF/ryqsiPydeFpzGOvxS9Tw/1NIfDkqTvtZAJsKkCykyDWsOXcYH608DAF5/sCmGRLN3n4isF9vOyVgGR1Ve6dx4Ih2FZRWC01i2ErUG649dAwAMbtfwLnsTVWIhRXe0NSkDr/18DAAw7r4QPM9pWInIyn333XfIy8tD9+7d4e/vr/v48ccfdft88cUX6Nu3LwYPHoyuXbvCz88Pa9asEZiazFHbQDeEejmipFyDDceviY5j0f46lY6Csgo0cLNHbIiH6DhkJlhI0W0lXszBhOWHoNFKGNS2Ad56uBlkMpnoWEREQrHtnIxFJpPprkr9wvY+g6punxwc1RByOd/rUO2wkKJbOpNegNGLE1BarkWPpt745LFWfGEhIiIyskHtGkAmAw6k5iAtp1h0HIuUnleKPeeyAACD2zUQnIbMCQspuklaTjFGLDqA/NIKRAW749uhUbBV8E+FiIjI2Pxd7dE5rHLGvl8O8aqUIaw9fAVaCYhu5M61MalO+O6YasgqLMOIRfHIyC9DU19nLBoZDXulQnQsIiIiqzU4qvIqyS+HLnNNKT2TJElXoHKSCaorFlKkU1BajlGL45GaVYQGbvZYMiYGrg62omMRERFZtQdb+MNZZYO0nBLsS8kWHceiJF68gXOZhbC3VeCRVv6i45CZYSFFAIDScg2eWZqIE1fy4emoxLKxMfBztRMdi4iIyOrZKxXo37ZyTalVCWmC01iWlfGVz2ffVv5wtuPJY6obFlIEjVbCKz8ewb6UbDgqFYgbHYNQbyfRsYiIiKjKk9FBAIBNJ9KRU6QWnMYy5JeWY/3xqwCAJ2O4RibVHQspKydJEqb+dgJ/nkiHUiHHghHt0bKhq+hYRERE9C+RDVwR2cAFao0Waw9fER3HIvx25CpKy7UI93FCuyB30XHIDLGQsnKzN5/FigOXIJMBc55sg06NvURHIiIiolt4ouqq1Kr4S5AkTjpxr35MuAQAeCI6kOtkUr2wkLJii/ekYu7WcwCA9/tH4uGWvMmSiIjIVPVvEwA7WzmSMwtx6FKu6Dhm7cSVPJy4kg+lQo5BnK2P6omFlJX67cgVTF93CgAw+YEmGNYhWHAiIiIiuhMXO1s80rJy0onqqylUP6uqnr/eLXzh4agUnIbMFQspK7Tj7HW8+tNRAMCoTo3wQs/GghMRERFRbTxVNSnCuqPXkF9aLjiNeSpWV+C3w5WTTDwVEyQ4DZkzFlJW5vClG3huWSIqtBL6tQ7Au32bsy+YiIjITEQFuyPcxwkl5RqsSbwsOo5Z+u3IVRSUVSDY0wEdQz1FxyEzxkLKiiRnFGB0XAJKyjXoEu6Fzx9vDbmcRRQREZG5kMlkGN6xsh1/2f6LnHSijiRJwtJ9FwEAw2KD+T6I7gkLKStxJbcEIxbFI7e4HG0C3TBvWBSUNvz1ExERmZuBbRvAUanA+etF2Hs+W3Qcs5J48QZOX8uHykaOx9tzkgm6N3wnbQVyitQYsfAAruWVIszbEYtHRcNRZSM6FhEREdWDs52tbqa5pfsuiA1jZqqvRvVvEwA3B04yQfeGhZSFKyqrwOi4BJy/XgR/VzssGxsLd85OQ0REZNaq2/s2n8rA1dwSwWnMw/WCMvx54hoAYETHRmLDkEVgIWXB1BVaPPdDIo6m5cLdwRbLxsYgwM1edCwiIiK6R018nREb4gGtBKw4wKnQa2NV/CWUayS0DXJDZANX0XHIArCQslBarYRXVx/FruQsOCgVWDw6Bo19nEXHIiIiIj2pvqqyKuESyio0YsOYuAqNFiviKwvOER25dibpBwspCyRJEqavO4l1R6/CViHDvGFRaBPoJjoWERER6VHvFr7wdVEhq1CNDceviY5j0jafysC1vFJ4OirxcEt/0XHIQrCQskBzt57Dkn0XIZMBs4e0Qdcm3qIjERERkZ7ZKuQYFlt5deX7XamcCv0OFuxKAQA8HRsElY1CcBqyFCykLMwP+y9i9uazAID3+rVAv9YBghMRERGRoQzrEAw7WzlOXs3HvhROhX4riRdv4NClXCgVct0kHUT6wELKgqw/dg1TfzsBAHjx/nCM7NRIbCAiIiIyKHdHJR6LqpwK/ftdqYLTmKbvq65GDWgbAB9nO8FpyJKwkLIQu5Oz8PKPhyFJwNDYILzSK1x0JCIiIjKCMZ1DIJMBW5MycS6zQHQck3IpuxibTqYDAMZ1CRWchiwNCykLcOxyLp5ddhDlGgkPt/TDjP6RkMlkomMRERGREYR6O+H+CF8AwMLdvCr1b4v2pEIrAV2beKOJL2cvJv1iIWXmUq4XYtTiBBSpNejc2BNfPNEGCjmLKCIiImsyvksIAOCXQ1eQXVgmOI1pyCsux08H0wD88/wQ6RMLKTOWnleK4QvjkVOkRssGrvjv8PaciYaIiMgKxYR4oFVDV6grtFiy76LoOCZhefxFFKs1iPBzxn2NvUTHIQvEQspM5RarMWLRAVzJLUGolyPiRkfDSWUjOhYREREJIJPJ8EzXynuA4vakoqC0XHAisYrVFVhYNfnGM11DecsDGQQLKTNUotZg7JKDOJtRCF8XFZaOjYGnk0p0LCIiIhLooUh/hHo7Ir+0Asv2W/dVqZXxacguUiPQwx6PcikYMhAWUmamXKPF88sTkXjxBlztbbFsbCwaujuIjkVERESCKeQyTOzeGEDlVOjF6grBicQoLdfgvzvOAwAmdG8MGwXf7pJh8C/LjGi1El7/+Ri2n7kOO1s5Fo1qzxloiIiISKd/mwAEetgjp0iNFQcuiY4jxOrEy8gsKIO/qx0Gt2soOg5ZMBZSZkKSJHyw/jTWHr4ChVyG74ZGISrYQ3QsIiIiMiE2CjkmVF2Vmr8zBaXlGsGJjKtco8W87ZVXo57rFgalDd/qkuHwr8tMfLv9PBbtqbxp8tPHWqFHhI/gRERERGSKBrdrCH9XO2QWlGF11fTf1mLtoSu4klsCb2cVnogOFB2HLBwLKTOwKv4SPt10BgDwziPNMIiXqYmIiOg2lDZyPNctDADwzbbzVnNVSl2hxdxtyQCAZ7qEws6WS8KQYbGQMnEbT6TjrbXHAQATuodhXJdQwYmIiIjI1D0RHYgAVzuk55di6b4LouMYxcr4S0jLqbwaNbRDkOg4ZAVYSJmw/SnZeHHVYWgl4In2gXitT1PRkYiIiMgM2Nkq8PIDTQBUXpXKK7HsdaWKyiowd2vl1agX7w+Hg5Jra5LhsZAyUSeu5GHckoNQV2jxQHNffDgwkovJERERUa0NatsAjX2ckFdSjvk7z4uOY1ALd6ciq1CNRp4OeJL3RpGRsJAyQReyijBqcTwKyyoQE+KBuU+15RoIREREVCc2Crmum2Xh7lRk5pcKTmQY2YVlmL8zBQDwau+msOV7JjIS/qWZmMz8UoxYFI+sQjWa+bvg+5HtebMkERER1Uvv5r5oG+SG0nItvqpqfbM0324/j8KyCrQIcMEjLf1FxyErwkLKhOSVlGPk4gRcyilGkIcDloyJhoudrehYREREZKZkMhneeDACALAyPg3JGQWCE+nXhawiLNt3EQDwxoMRkMt5GwQZDwspE1FarsH4JQdx+lo+vJxUWDY2Bj7OdqJjERERkZnrEOqJXs18odFKeG/dSUiSJDqS3sz44xTUGi26hHuhS7iX6DhkZVhImYAKjRaTVhxG/IUcOKtssGRMNII9HUXHIiIiIgvxbt/mUNrIsedcNv48kS46jl5sOZ2BrUmZsFXI8N6jLTgpFxkdCynBJEnClDXH8ffpDCht5Ph+ZHu0CHAVHYuIiIgsSJCng26R3g/+OIUStXkv0ltarsGMP04BAMbcF4IwbyfBicgasZAS7JONZ7A68TLkMuDrp9oiNtRTdCQiIiKyQM93C0MDN3tczSvFt9vPiY5zT77flYKL2cXwdVHhhZ7houOQlWIhJdCCnSmYt6NyXYeZg1qidws/wYmIiIjIUtkrFZjatzkA4L87UpByvVBwovpJyynG19sqC8G3Hm4GJxUX3yUxWEgJ8nPiZXy44TSAyllmnogOEpyIiIiILF2fFr7o2sQbao0Wr/98DBqteU08odVKeOOXYygt1yI2xAOPtg4QHYmsGAspAbaczsAbvxwDAIzvEoLnuoUKTkRERETWQCaT4aOBkXBUKnDw4g0s3pMqOlKdLD9wEXvPZ8POVo5PBrfiBBMkFAspI0u4kIMJyw9Bo5UwqF0DTHmoGV8EiIiIyGgaujvgnaoWv083ncF5M2nxu5RdjJl/JgEA3nwwAo28OMMxicVCyoiS0vMxNi4BZRVa3B/hg08Gt+LCcURERGR0T0YHoku4F8oqtPjP6qMm3+Kn1Up47eejKFZrEBvigREdG4mORMRCyljScooxYmE88ksr0D7YHV8/3Q62Cj79REREZHwymQyfDG4FZ5UNDl/KxbfbTHsWv4W7U3EgNQcOSgU+faw1T0STSeA7eSPIKizD8IUHkFlQhqa+zlg4Mhr2SoXoWERERGTFAtzs8W6/yha/L/4+i93JWYIT3dqBlGx8vLGype/tR5ohyNNBcCKiSiykDKygtByjFsfjQnYxGrrbY+nYGLg62IqORURERITH2wdiSPuG0ErAi6sO42puiehINWTml2LSysPQaCUMaBOAp2M4yzGZDhZSBlRarsEzSxNx4ko+PB2VWDY2Fr4udqJjEREREenM6B+JFgEuyClSY8LyQyir0IiOBAAo12gxccUhXK/q6PloUEtO0EUmhYWUgWi0El758Qj2pWTDSWWDJWNiEMLZZYiIiMjE2NkqMG9YFFztbXEkLRfTfjsJSRI7+YQkSXj/j1NIuHADziobzBseBQclF94l08JCygAkScI7v57AnyfSoVTIMX94FCIbuIqORURERHRLgR4OmPNEG8hkwKqENHz+11mheeZuPYel+y4CAD4b0pono8kksZAygNmbz2Jl/CXIZcCXT7ZBp8ZeoiMRERER3VGPCB98MCASAPD1tnP4fleKkBzL9l3A7M2Vhdy0fs3Rp4WfkBxEd8NCSs8W70nF3K2VU4h+MKAlHmrpLzgRERERUe0MjQ3Ga32aAgA+WH8aqw+mGfXn/3bkCt79/SQA4MX7wzG6c4hRfz5RXVhMIfXNN9+gUaNGsLOzQ2xsLOLj442e4bcjVzB93SkAwH96N8HTsZxZhoiIiMzLhO5hGHdfZQHz+i/H8P2uFKPcM7V03wW88uMRSBIwomMwXukVbvCfSXQvLKKQ+vHHHzF58mRMmzYNhw4dQuvWrdGnTx9kZmYaLcP2M5l49aejAIBRnRphYo/GRvvZRERERPoik8nw1sPNMLJjMCSp8srUe7+fhEZrmGJKq5Xw4fpTePe3k9BKwFMxgXivXwvO0EcmTyaJnpZFD2JjYxEdHY2vv/4aAKDVahEYGIgXXngBb7755l2/Pz8/H66ursjLy4OLi0udf/6hSzcwdMEBlJRr8GjrAMx5og1X3CYiqoV7ff21VHxeyBRIkoTvd6Xiww2nAQBdwr3wRHQgYkM84e2suufHzy4sw4HUHPyceBlbkypPfr/WpykmdA9jEUVC1fY12OznkVSr1UhMTMSUKVN02+RyOXr16oV9+/bd8nvKyspQVlam+zw/P7/ePz85owBj4hJQUq5B1ybe+Ozx1iyiiIiIyOzJZDKM7xqKBu72ePnHI9iVnIVdyVkAgCAPBzgoFfV+7NJyDS5kF+s+Vyrk+PTxVujfpsE95yYyFrMvpLKysqDRaODr61tju6+vL5KSkm75PTNnzsT06dP18vM3HE9HbnE52gS6Yd6wdlDaWES3JBER6cE333yDTz/9FOnp6WjdujXmzp2LmJgY0bGI6uThlv5o5OmI1Ylp2J+Sg9PX8nEpp/ju31gLEX7O6BDqicHtGqJlQy4VQ+bF7Aup+pgyZQomT56s+zw/Px+BgYH1eqwX728MD0db9G0VwIXiiIhIp/r+3Xnz5iE2NhZz5sxBnz59cObMGfj4+IiOR1QnzQNcMC2gBQDgRpEap9PzodXW//HkMiDC3wUejko9JSQyPrN/5+/l5QWFQoGMjIwa2zMyMuDnd+t1B1QqFVSqe+/tBSovew/v2Egvj0VERJZj9uzZGD9+PEaPHg0AmDdvHtavX49FixbV6v5dIlPl7qhEpzCukUlk9n1oSqUSUVFR2LJli26bVqvFli1b0LFjR4HJiIjIWlXfv9urVy/dtjvdv1tWVob8/PwaH0REZNrMvpACgMmTJ2PBggVYsmQJTp8+jeeffx5FRUW6s4BERETGdKf7d9PT02/af+bMmXB1ddV91LfdnIiIjMfsW/sA4IknnsD169fx7rvvIj09HW3atMHGjRtvGsCIiIhMkT7v3SUiIuOwiEIKACZNmoRJkyaJjkFERFTn+3f1ee8uEREZh0W09hEREZkS3r9LRGT5LOaKFBERkSmZPHkyRo4cifbt2yMmJgZz5szh/btERBaEhRQREZEB8P5dIiLLxkKKiIjIQHj/LhGR5eI9UkRERERERHXEQoqIiIiIiKiOWEgRERERERHVEQspIiIiIiKiOmIhRUREREREVEcspIiIiIiIiOqI058DkCQJAJCfny84CRGRdal+3a1+HaZKHJeIiMSp7djEQgpAQUEBACAwMFBwEiIi61RQUABXV1fRMUwGxyUiIvHuNjbJJJ4GhFarxdWrV+Hs7AyZTFbn78/Pz0dgYCDS0tLg4uJigISmjcfP4+fx8/jre/ySJKGgoAABAQGQy9ltXo3j0r3h8Vv38QN8Dnj8xhmbeEUKgFwuR8OGDe/5cVxcXKzyj7Uaj5/Hz+Pn8dcHr0TdjOOSfvD4rfv4AT4HPH7Djk08/UdERERERFRHLKSIiIiIiIjqiIWUHqhUKkybNg0qlUp0FCF4/Dx+Hj+P31qP31RZ+++Fx2/dxw/wOeDxG+f4OdkEERERERFRHfGKFBERERERUR2xkCIiIiIiIqojFlJERERERER1xEKKiIiIiIiojlhI1dI333yDRo0awc7ODrGxsYiPj7/j/qtXr0ZERATs7OzQsmVLbNiwwUhJDaMux79gwQJ06dIF7u7ucHd3R69eve76fJm6uv7+q61atQoymQwDBgwwbEADq+vx5+bmYuLEifD394dKpUKTJk3M+v9AXY9/zpw5aNq0Kezt7REYGIhXXnkFpaWlRkqrPzt37kS/fv0QEBAAmUyGX3/99a7fs337drRr1w4qlQqNGzdGXFycwXNaK45LHJc4LnFcsrZxCTCxsUmiu1q1apWkVCqlRYsWSSdPnpTGjx8vubm5SRkZGbfcf8+ePZJCoZBmzZolnTp1SnrnnXckW1tb6fjx40ZOrh91Pf6nn35a+uabb6TDhw9Lp0+flkaNGiW5urpKly9fNnJy/ajr8VdLTU2VGjRoIHXp0kXq37+/ccIaQF2Pv6ysTGrfvr308MMPS7t375ZSU1Ol7du3S0eOHDFycv2o6/EvX75cUqlU0vLly6XU1FRp06ZNkr+/v/TKK68YOfm927Bhg/T2229La9askQBIa9euveP+KSkpkoODgzR58mTp1KlT0ty5cyWFQiFt3LjROIGtCMcljksclzguWeO4JEmmNTaxkKqFmJgYaeLEibrPNRqNFBAQIM2cOfOW+w8ZMkR65JFHamyLjY2Vnn32WYPmNJS6Hv//q6iokJydnaUlS5YYKqJB1ef4KyoqpE6dOknff/+9NHLkSLMesOp6/N99950UGhoqqdVqY0U0qLoe/8SJE6WePXvW2DZ58mSpc+fOBs1paLUZrF5//XWpRYsWNbY98cQTUp8+fQyYzDpxXOK4xHGJ41I1ax2XJEn82MTWvrtQq9VITExEr169dNvkcjl69eqFffv23fJ79u3bV2N/AOjTp89t9zdl9Tn+/1dcXIzy8nJ4eHgYKqbB1Pf4Z8yYAR8fH4wdO9YYMQ2mPsf/+++/o2PHjpg4cSJ8fX0RGRmJjz76CBqNxlix9aY+x9+pUyckJibq2ixSUlKwYcMGPPzww0bJLJIlvfaZMo5LHJc4LnFc4rhUe4Z8/bO550ewcFlZWdBoNPD19a2x3dfXF0lJSbf8nvT09Fvun56ebrCchlKf4/9/b7zxBgICAm76IzYH9Tn+3bt3Y+HChThy5IgREhpWfY4/JSUFW7duxdChQ7FhwwacO3cOEyZMQHl5OaZNm2aM2HpTn+N/+umnkZWVhfvuuw+SJKGiogLPPfcc3nrrLWNEFup2r335+fkoKSmBvb29oGSWheMSxyWOSxyXOC7VniHHJl6RIoP6+OOPsWrVKqxduxZ2dnai4xhcQUEBhg8fjgULFsDLy0t0HCG0Wi18fHwwf/58REVF4YknnsDbb7+NefPmiY5mFNu3b8dHH32Eb7/9FocOHcKaNWuwfv16vP/++6KjERE4LlkjjksclwyFV6TuwsvLCwqFAhkZGTW2Z2RkwM/P75bf4+fnV6f9TVl9jr/aZ599ho8//hh///03WrVqZciYBlPX4z9//jwuXLiAfv366bZptVoAgI2NDc6cOYOwsDDDhtaj+vz+/f39YWtrC4VCodvWrFkzpKenQ61WQ6lUGjSzPtXn+KdOnYrhw4dj3LhxAICWLVuiqKgIzzzzDN5++23I5ZZ7/up2r30uLi68GqVHHJc4LnFc4rjEcan2DDk2WfYzpwdKpRJRUVHYsmWLbptWq8WWLVvQsWPHW35Px44da+wPAJs3b77t/qasPscPALNmzcL777+PjRs3on379saIahB1Pf6IiAgcP34cR44c0X08+uij6NGjB44cOYLAwEBjxr9n9fn9d+7cGefOndMN1ABw9uxZ+Pv7m9VgBdTv+IuLi28alKoH78r7Yi2XJb32mTKOSxyXOC5xXOK4VHsGff275+kqrMCqVasklUolxcXFSadOnZKeeeYZyc3NTUpPT5ckSZKGDx8uvfnmm7r99+zZI9nY2EifffaZdPr0aWnatGlmP81sXY7/448/lpRKpfTzzz9L165d030UFBSIOoR7Utfj/3/mPjtSXY//0qVLkrOzszRp0iTpzJkz0h9//CH5+PhIH3zwgahDuCd1Pf5p06ZJzs7O0sqVK6WUlBTpr7/+ksLCwqQhQ4aIOoR6KygokA4fPiwdPnxYAiDNnj1bOnz4sHTx4kVJkiTpzTfflIYPH67bv3qK2ddee006ffq09M0333D6cwPhuMRxieMSxyVrHJckybTGJhZStTR37lwpKChIUiqVUkxMjLR//37d17p16yaNHDmyxv4//fST1KRJE0mpVEotWrSQ1q9fb+TE+lWX4w8ODpYA3PQxbdo04wfXk7r+/v/N3AcsSar78e/du1eKjY2VVCqVFBoaKn344YdSRUWFkVPrT12Ov7y8XHrvvfeksLAwyc7OTgoMDJQmTJgg3bhxw/jB79G2bdtu+X+5+nhHjhwpdevW7abvadOmjaRUKqXQ0FBp8eLFRs9tLTgucVziuMRxydrGJUkyrbFJJklWcE2PiIiIiIhIj3iPFBERERERUR2xkCIiIiIiIqojFlJERERERER1xEKKiIiIiIiojlhIERERERER1RELKSIiIiIiojpiIUVERERERFRHLKSIiIiIiIjqiIUUkZ4sXLgQvXv31n0+atQoDBgwQFygO2jUqBHmzJmj18eMi4uDm5ubXh9TlLs9P1lZWfDx8cHly5eNF4qIqB44NnFsIsNhIUV6MWrUKMhkMshkMiiVSjRu3BgzZsxARUWF6GhGUVpaiqlTp2LatGm6bV9++SXi4uL+1969xkRxtXEA/y8ILCxyVy6tgBak3KSAEAhFUCgrJASlCKEUqDZCLXJpNG4RUuSijRJrjFrUNkFq2nohtlykyiWisigXRVRYV6MofEBaAa1Ii4E97wfj1AEXpEWWt31+yXyYmXPmPDOz2Sdn9pzZaY9lcHAQ6enpeOuttyAUCjFnzhz4+fmhpKSEK9PU1ISEhIRpj20iW7Zs4T5HLy5vv/22qkPjMTExQVxcHO9+E0JmHspNlJumAuUmoswsVQdA/j2WL1+OwsJCDA0NoaKiAklJSdDQ0EB6evqYsk+fPoWmpuZrieN1HluZ4uJi6OnpwcfHh9umr68/rTE898knn6ChoQF79uyBg4MDent7UV9fj97eXq7MnDlzVBLbq3B0dER1dTVv26xZM++ravXq1XB3d0d+fj6MjIxUHQ4hRAnKTZSbpgLlJvIy9IsUmTJaWlowMzODlZUV1q1bh8DAQJSWlgL4ayjB1q1bYWFhATs7OwDAtWvXsGzZMmhra8PY2BgJCQkYGBjgjjk8PIyUlBQYGBjA2NgYEokE8fHxvGEJ/v7+WL9+PdLS0mBiYgKxWAwAuH79OoKDg6GrqwtTU1PExsbiwYMHXL3i4mI4OztzbQcGBuLJkycAgNraWnh6ekIkEsHAwAA+Pj64d++e0nM/cuQIQkNDedtGD5/w9/dHSkoKNm3aBCMjI5iZmWHLli3jXtOmpia89957MDExgb6+Pvz8/HD58uVx65SWlmLz5s0ICQmBtbU13N3dkZycjDVr1nBlRg8PEAgE+Pbbb7Fy5Uro6OjA1taWu3cvHtfW1hZCoRBLly5FUVERBAIBHj58qDSWkpISuLm5QSgUYsGCBcjOzp7wSfCsWbNgZmbGW0xMTHix5+bmIjo6GiKRCG+88Qb27dvHO0ZnZyfCwsKgq6sLPT09REZGoqenh1emrKwMHh4eEAqFMDExwcqVK3n7BwcHsWbNGsyePRuWlpY4ePAgb7+joyMsLCzw008/jXs+hBDVotxEuWk0yk1kqlBHirw22traePr0KbdeU1MDuVyOqqoqlJeX48mTJxCLxTA0NERTUxOOHz+O6upqrF+/nquzfft2fP/99ygsLIRUKsXvv/+On3/+eUxbRUVF0NTUhFQqxf79+/Hw4UMsW7YMrq6uaG5uxqlTp9DT04PIyEgAQHd3N6Kjo7FmzRrIZDLU1tYiPDwcjDEMDw9jxYoV8PPzw9WrV3HhwgUkJCRAIBAoPde6ujosXrx4wmtSVFQEkUiEhoYG7NixAzk5OaiqqlJa/vHjx4iPj0ddXR0uXrwIW1tbhISE4PHjx0rrmJmZoaKiYtwyL5OdnY3IyEhcvXoVISEhiImJQV9fHwCgo6MDERERWLFiBVpbW5GYmIiMjIxxj3f+/HnExcUhNTUV7e3tOHDgAA4dOoStW7dOKq6Xyc/Ph4uLC1paWvD5558jNTWVu44KhQJhYWHo6+vD2bNnUVVVhTt37iAqKoqrf/LkSaxcuRIhISFoaWlBTU0NPD09eW3s3LkTixcvRktLCz799FOsW7cOcrmcV8bT0xPnz5//x+dDCJk+lJvGotxEuYn8TYyQKRAfH8/CwsIYY4wpFApWVVXFtLS02MaNG7n9pqambGhoiKtz8OBBZmhoyAYGBrhtJ0+eZGpqauz+/fuMMcZMTU1Zfn4+t394eJhZWlpybTHGmJ+fH3N1deXFk5uby4KCgnjburq6GAAml8vZpUuXGAB29+7dMefS29vLALDa2tpXOvf+/n4GgJ07d07pNXke57vvvssr4+HhwSQSySu1wxhjIyMjbPbs2aysrExpmbNnz7I333yTaWhosMWLF7O0tDRWV1fHK2NlZcV27drFrQNgmZmZ3PrAwAADwH755RfGGGMSiYQ5OTnxjpGRkcEAsP7+fsYYY4WFhUxfX5/bHxAQwLZt28arc/jwYWZubq409qysLKampsZEIhFvSUxM5MW+fPlyXr2oqCgWHBzMGGOssrKSqaurs87OTm5/W1sbA8AaGxsZY4x5e3uzmJgYpXFYWVmxDz/8kFtXKBRs7ty5rKCggFfus88+Y/7+/kqPQwhRLcpNlJsoN5HXiX6RIlOmvLwcurq6EAqFCA4ORlRUFG94gLOzM298uEwmg4uLC0QiEbfNx8cHCoUCcrkcjx49Qk9PD+9pjLq6Otzd3ce0PXpba2srzpw5A11dXW55Pin09u3bcHFxQUBAAJydnbFq1Sp888036O/vBwAYGRnho48+glgsRmhoKHbv3o3u7m6l5/3HH38AAIRC4YTXaNGiRbx1c3Nz/Prrr0rL9/T0YO3atbC1tYW+vj709PQwMDCAzs5OpXWWLFmCO3fuoKamBhEREWhra4Ovry9yc3NfOTaRSAQ9PT0uNrlcDg8PD1750U/JRmttbUVOTg7vHqxduxbd3d0YHBxUWs/Ozg5XrlzhLTk5Obwy3t7eY9ZlMhmAZ5+refPmYd68edx+BwcHGBgYcGWuXLmCgICAceN/8XoIBAKYmZmNuVfa2trjngshRPUoN1FuehHlJjKVZt4sOfJ/a+nSpSgoKICmpiYsLCzGTMJ8MSlNtdHHHhgYQGhoKLZv3z6mrLm5OdTV1VFVVYX6+npUVlZiz549yMjIQENDA+bPn4/CwkKkpKTg1KlTOHr0KDIzM1FVVQUvL68xxzM2NoZAIOCS3Xg0NDR46wKBAAqFQmn5+Ph49Pb2Yvfu3bCysoKWlha8vb15w1KUtePr6wtfX19IJBLk5eUhJycHEolE6WTnycY2kYGBAWRnZyM8PHzMvvES+/M3a71O2traE5Z5levR19c3oydHE0IoN1Fu4qPcRKYS/SJFpoxIJIKNjQ0sLS1f6U029vb2aG1t5SbRAoBUKoWamhrs7Oygr68PU1NTNDU1cftHRkYmnNAKAG5ubmhra4O1tTVsbGx4y/PEJhAI4OPjg+zsbLS0tEBTU5M3OdPV1RXp6emor6+Hk5MTfvjhh5e2pampCQcHB7S3t08Y12RJpVKkpKQgJCQEjo6O0NLS4k1KflUODg4YHh7Gn3/++bfisLOzQ3NzM2/bi/flZdzc3CCXy8dcfxsbG6ip/bOvnosXL45Zt7e3B/Dsc9XV1YWuri5uf3t7Ox4+fAgHBwcAz57o1dTU/KMYgGeTxl1dXf/xcQghrw/lJspNL6LcRKYSdaSIysTExEAoFCI+Ph7Xr1/HmTNnkJycjNjYWJiamgIAkpOT8eWXX6KkpARyuRypqano7+8fd3ItACQlJaGvrw/R0dFoamrC7du3cfr0aaxevRojIyNoaGjAtm3b0NzcjM7OTpw4cQK//fYb7O3t0dHRgfT0dFy4cAH37t1DZWUlbt26xX0ZvoxYLEZdXd2UXh8AsLW1xeHDhyGTydDQ0ICYmJgJn1j5+/vjwIEDuHTpEu7evYuKigps3rwZS5cuhZ6e3t+KIzExETdu3IBEIsHNmzdx7Ngx7n9IlN2LL774At999x2ys7PR1tYGmUyGI0eOIDMzc9y2hoeHcf/+fd4y+q1GUqkUO3bswM2bN7Fv3z4cP34cqampAIDAwEA4OzsjJiYGly9fRmNjI+Li4uDn58dNus7KysKPP/6IrKwsyGQyXLt27aVPiMczODiIS5cu8f7okhDy/49y08QoN1FuIs9QR4qojI6ODk6fPo2+vj54eHggIiICAQEB2Lt3L1dGIpEgOjoacXFx8Pb2hq6uLsRi8YRjvi0sLCCVSjEyMoKgoCA4OzsjLS0NBgYGUFNTg56eHs6dO4eQkBAsXLgQmZmZ2LlzJ4KDg6Gjo4MbN27g/fffx8KFC5GQkICkpCQkJiYqbe/jjz9GRUUFHj16NGXXB3j2j/T9/f1wc3NDbGwsUlJSMHfu3HHriMViFBUVISgoCPb29khOToZYLMaxY8f+dhzz589HcXExTpw4gUWLFqGgoIB7M5KWlpbSOMrLy1FZWQkPDw94eXlh165dsLKyGrettrY2mJub85bRdTZs2IDm5ma4uroiLy8PX331FfdqYYFAgJKSEhgaGmLJkiUIDAzEggULcPToUa6+v78/jh8/jtLSUrzzzjtYtmwZGhsbJ3VNSkpKYGlpCV9f30nVI4TMbJSbJka5iXITeUbAGGOqDoKQV6VQKGBvb4/IyMgJJ6hOt1WrVsHNze2lf/L4b7R161bs37+fN0xhOlhbWyMtLQ1paWnT2u5oXl5eSElJwQcffKDSOAghqke5aeag3ES5aTrRyybIjPZ8+IKfnx+Ghoawd+9edHR0zMgviPz8fJSVlak6jNfm66+/hoeHB4yNjSGVSpGfn8/7X5X/kgcPHiA8PBzR0dGqDoUQogKUm2YOyk1/odw0/agjRWY0NTU1HDp0CBs3bgRjDE5OTqiurh53TLiqWFtbIzk5WdVhvDa3bt1CXl4e+vr6YGlpiQ0bNvxnnnCOZmJigk2bNqk6DEKIilBumjkoN/2FctP0o6F9hBBCCCGEEDJJ9LIJQgghhBBCCJkk6kgRQgghhBBCyCRRR4oQQgghhBBCJok6UoQQQgghhBAySdSRIoQQQgghhJBJoo4UIYQQQgghhEwSdaQIIYQQQgghZJKoI0UIIYQQQgghk/Q/mBu5TQBXqBsAAAAASUVORK5CYII="
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualizing different mask scaling methods\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from hat.utils import get_hat_mask_scale\n",
    "\n",
    "progress = np.linspace(0, 1, 100)\n",
    "linear_scale = np.array(\n",
    "    [\n",
    "        get_hat_mask_scale(\n",
    "            \"linear\",\n",
    "            progress=__p,\n",
    "        )\n",
    "        for __p in progress\n",
    "    ]\n",
    ")\n",
    "cosine_scale = np.array(\n",
    "    [\n",
    "        get_hat_mask_scale(\n",
    "            \"cosine\",\n",
    "            progress=__p,\n",
    "        )\n",
    "        for __p in progress\n",
    "    ]\n",
    ")\n",
    "\n",
    "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 5))\n",
    "ax1.plot(progress, linear_scale)\n",
    "ax1.set_title(\"Linear Scaling\")\n",
    "ax1.set_xlabel(\"Progress (in a Single Epoch)\")\n",
    "ax1.set_ylabel(\"Mask Scale\")\n",
    "ax2.plot(progress, cosine_scale)\n",
    "ax2.set_title(\"Cosine Scaling\")\n",
    "ax2.set_xlabel(\"Progress (in a Single Epoch)\")\n",
    "ax2.set_ylabel(\"Mask Scale\")\n",
    "plt.show()"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:22.855708Z",
     "end_time": "2023-04-30T10:01:23.346778Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## On the Convergence of HAT Networks\n",
    "\n",
    "Now let's test the convergence of the HAT network with the proposed solution v.s. the original approach, implemented with this library.\n",
    "\n",
    "The network has three different configurations:\n",
    "- Input mask only\n",
    "- Output mask only\n",
    "- Input and output mask"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "# Create a function that trains a model for `num_epochs` epochs\n",
    "# and returns the number of batches it takes to converge.\n",
    "# If the model fails to converge, return None.\n",
    "def convergence_trial(\n",
    "    model: nn.Module,\n",
    "    max_num_epochs: int,\n",
    ") -> int:\n",
    "    _model = model.to(device)\n",
    "    _optimizer = torch.optim.SGD(\n",
    "        _model.parameters(),\n",
    "        lr=learning_rate,\n",
    "        momentum=momentum,\n",
    "    )\n",
    "    _criterion = nn.MSELoss()\n",
    "\n",
    "    def __is_converged() -> bool:\n",
    "        tst_loss = 0\n",
    "        with torch.no_grad():\n",
    "            for __x, __y in tst_dataloader:\n",
    "                try:\n",
    "                    __pred = _model.forward(__x, None)\n",
    "                except TypeError:\n",
    "                    __pred = _model.forward(__x)\n",
    "                __loss = _criterion(__pred, __y)\n",
    "                tst_loss += __loss.item()\n",
    "        tst_loss /= len(tst_dataloader)\n",
    "        return tst_loss < convergence_threshold\n",
    "\n",
    "    # Train the model for `num_epochs` epochs\n",
    "    num_batches = 0\n",
    "    for _ in range(max_num_epochs):\n",
    "        for __i, (__x, __y) in enumerate(trn_dataloader):\n",
    "            _optimizer.zero_grad()\n",
    "            # Progress is required for the HAT mask scaling method\n",
    "            __progress = __i / (len(trn_dataloader) - 1)\n",
    "            try:\n",
    "                __pred = _model.forward(__x, __progress)\n",
    "            except TypeError:\n",
    "                __pred = _model.forward(__x)\n",
    "            __loss = _criterion(__pred, __y)\n",
    "            __loss.backward()\n",
    "            num_batches += 1\n",
    "            _optimizer.step()\n",
    "            if __is_converged():\n",
    "                return num_batches\n",
    "    return None"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:23.346259Z",
     "end_time": "2023-04-30T10:01:23.350807Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline (non-HAT) model converged in 12.58 batches on average.\n"
     ]
    }
   ],
   "source": [
    "# Check how many epochs the baseline model (no masks) takes to converge\n",
    "baseline_num_batches = [\n",
    "    convergence_trial(nn.Linear(2, 1).to(device), max_num_epochs)\n",
    "    for _ in range(num_trials)\n",
    "]\n",
    "if None in baseline_num_batches:\n",
    "    __num_failed_trials = baseline_num_batches.count(None)\n",
    "    print(\n",
    "        f\"{__num_failed_trials} out of {num_trials} trials \"\n",
    "        f\"failed to converge in the given number of epochs.\"\n",
    "    )\n",
    "    baseline_num_batches = [\n",
    "        __b for __b in baseline_num_batches if __b is not None\n",
    "    ]\n",
    "baseline_num_batches = np.array(baseline_num_batches)\n",
    "print(\n",
    "    f\"Baseline (non-HAT) model converged in \"\n",
    "    f\"{baseline_num_batches.mean():.2f} batches on average.\"\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:23.347433Z",
     "end_time": "2023-04-30T10:01:25.439805Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [],
   "source": [
    "from hat import HATConfig, HATPayload\n",
    "from hat.modules import HATLinear, HATMasker\n",
    "from hat.utils import get_hat_mask_scale\n",
    "\n",
    "\n",
    "class HATNetwork(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        num_features: int,\n",
    "        input_mask: bool,\n",
    "        output_mask: bool,\n",
    "        init_strat: str,\n",
    "        scaling_strat: str,\n",
    "    ):\n",
    "        super().__init__()\n",
    "        self.hat_config = HATConfig(\n",
    "            num_tasks=1,\n",
    "            init_strat=init_strat,\n",
    "        )\n",
    "        self.input_mask = (\n",
    "            HATMasker(\n",
    "                num_features=num_features,\n",
    "                hat_config=self.hat_config,\n",
    "            )\n",
    "            if input_mask\n",
    "            else nn.Identity()\n",
    "        )\n",
    "        self.linear = (\n",
    "            HATLinear(\n",
    "                in_features=num_features,\n",
    "                out_features=1,\n",
    "                hat_config=self.hat_config,\n",
    "            )\n",
    "            if output_mask\n",
    "            else nn.Linear(num_features, 1)\n",
    "        )\n",
    "        self.scaling_strat = scaling_strat\n",
    "        self.mask_scale = None\n",
    "\n",
    "    def forward(self, features, progress):\n",
    "        _mask_scale = (\n",
    "            get_hat_mask_scale(\n",
    "                strat=self.scaling_strat,\n",
    "                max_trn_mask_scale=self.hat_config.max_trn_mask_scale,\n",
    "                progress=progress,\n",
    "            )\n",
    "            if progress is not None\n",
    "            else self.hat_config.max_trn_mask_scale\n",
    "        )\n",
    "        # Keep track of the mask scale for reference\n",
    "        self.mask_scale = _mask_scale\n",
    "        _payload = HATPayload(\n",
    "            features,\n",
    "            task_id=0,\n",
    "            mask_scale=_mask_scale,\n",
    "        )\n",
    "        _payload = self.input_mask(_payload)\n",
    "        if isinstance(self.linear, HATLinear):\n",
    "            return self.linear(_payload).data\n",
    "        else:\n",
    "            return self.linear(_payload.data)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:25.440723Z",
     "end_time": "2023-04-30T10:01:25.446169Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [],
   "source": [
    "# Iterate over different configurations and compare the results\n",
    "results = []\n",
    "configs = [\n",
    "    (__input_mask, __output_mask, __init_strat, __scaling_strat)\n",
    "    for __input_mask in [False, True]\n",
    "    for __output_mask in [False, True]\n",
    "    # for __init_strat in [\"normal\", \"dense\"]\n",
    "    # for __scaling_strat in [\"linear\", \"cosine\"]\n",
    "    for __init_strat, __scaling_strat in zip(\n",
    "        [\"normal\", \"dense\"],  # Default settings\n",
    "        [\"linear\", \"cosine\"],\n",
    "    )\n",
    "]\n",
    "for __input_mask, __output_mask, __init_strat, __scaling_strat in configs:\n",
    "    if not __input_mask and not __output_mask:\n",
    "        continue\n",
    "    __num_batches = [\n",
    "        convergence_trial(\n",
    "            HATNetwork(\n",
    "                2,\n",
    "                __input_mask,\n",
    "                __output_mask,\n",
    "                __init_strat,\n",
    "                __scaling_strat,\n",
    "            ).to(device),\n",
    "            max_num_epochs,\n",
    "        )\n",
    "        for _ in range(num_trials)\n",
    "    ]\n",
    "    if None in __num_batches:\n",
    "        __num_failed_trials = __num_batches.count(None)\n",
    "        __num_batches = [\n",
    "            __num_batch\n",
    "            for __num_batch in __num_batches\n",
    "            if __num_batch is not None\n",
    "        ]\n",
    "        __fail_rate = __num_failed_trials / num_trials\n",
    "    else:\n",
    "        __fail_rate = 0\n",
    "    __num_batches = np.array(__num_batches)\n",
    "    __avg_num_batches = __num_batches.mean()\n",
    "    __result = [\n",
    "        __input_mask,\n",
    "        __output_mask,\n",
    "        __init_strat,\n",
    "        __scaling_strat,\n",
    "        __avg_num_batches,\n",
    "        __fail_rate,\n",
    "    ]\n",
    "    results.append(__result)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:01:25.449838Z",
     "end_time": "2023-04-30T10:03:15.395526Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒═══════════╤════════════╤══════════════╤═════════════════╤═══════════════════╕\n",
      "│  In Mask  │  Out Mask  │  Init Strat  │  Scaling Strat  │  Avg Num Batches  │\n",
      "╞═══════════╪════════════╪══════════════╪═════════════════╪═══════════════════╡\n",
      "│   False   │    True    │    normal    │     linear      │       90.33       │\n",
      "│   False   │    True    │    dense     │     cosine      │       12.53       │\n",
      "│   True    │   False    │    normal    │     linear      │       99.40       │\n",
      "│   True    │   False    │    dense     │     cosine      │       12.60       │\n",
      "│   True    │    True    │    normal    │     linear      │      179.63       │\n",
      "│   True    │    True    │    dense     │     cosine      │       12.43       │\n",
      "╘═══════════╧════════════╧══════════════╧═════════════════╧═══════════════════╛\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "result_df = pd.DataFrame(\n",
    "    results,\n",
    "    columns=[\n",
    "        \"In Mask\",\n",
    "        \"Out Mask\",\n",
    "        \"Init Strat\",\n",
    "        \"Scaling Strat\",\n",
    "        \"Avg Num Batches\",\n",
    "        \"Fail Rate\",\n",
    "    ],\n",
    ")\n",
    "# result_df[\"Fail Rate\"] = result_df[\"Fail Rate\"].apply(\n",
    "#     lambda x: f\"{100.0 * x:.1f}%\"\n",
    "# )\n",
    "if result_df[\"Fail Rate\"].max() > 0.0:\n",
    "    result_df[\"Fail Rate\"] = result_df[\"Fail Rate\"].apply(\n",
    "        lambda x: f\"{100.0 * x:.1f}%\"\n",
    "    )\n",
    "else:\n",
    "    result_df.drop(\"Fail Rate\", axis=1, inplace=True)\n",
    "print(\n",
    "    result_df.to_markdown(\n",
    "        index=False,\n",
    "        floatfmt=\".2f\",\n",
    "        tablefmt=\"fancy_outline\",\n",
    "        stralign=\"center\",\n",
    "        numalign=\"center\",\n",
    "    )\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:03:15.404359Z",
     "end_time": "2023-04-30T10:03:15.409222Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Regularization\n",
    "\n",
    "Dense initialization makes the attention values start at 1.0, so naturally it raises the question: are they as easy to regularize as the baseline model? Because we certainly don't want redundant hard attention masks, which leaves less parameters for later tasks.\n",
    "\n",
    "Let's find out.\n",
    "\n",
    "We construct a new dataset where the target `y` is a linear combination of some of the features: `y = x0 - 2*x2 + 3*x4`. Then we compare how long does it take for a HAT model (with input and output masks) to recognize and zero out the redundant attention weights."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [],
   "source": [
    "num_samples = 1000\n",
    "num_features = 5\n",
    "\n",
    "features = torch.randn(num_samples, num_features)\n",
    "targets = features[:, 0] - 2 * features[:, 2] + 3 * features[:, 4]\n",
    "targets = targets.view(-1, 1)\n",
    "\n",
    "# Split the dataset into train and test sets.\n",
    "trn_ratio = 0.8\n",
    "num_train_samples = int(num_samples * trn_ratio)\n",
    "trn_x, tst_x = features[:num_train_samples], features[num_train_samples:]\n",
    "trn_y, tst_y = targets[:num_train_samples], targets[num_train_samples:]\n",
    "\n",
    "# Construct dataloaders for the train and test sets.\n",
    "trn_dataloader = torch.utils.data.DataLoader(\n",
    "    dataset=torch.utils.data.TensorDataset(trn_x, trn_y),\n",
    "    shuffle=True,\n",
    "    batch_size=32,\n",
    ")\n",
    "tst_dataloader = torch.utils.data.DataLoader(\n",
    "    dataset=torch.utils.data.TensorDataset(tst_x, tst_y),\n",
    "    batch_size=32,\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:03:15.409682Z",
     "end_time": "2023-04-30T10:03:15.432838Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "outputs": [],
   "source": [
    "from hat.utils import get_hat_reg_term\n",
    "\n",
    "\n",
    "def regularization_trial(\n",
    "    model: nn.Module,\n",
    "    max_num_epochs: int,\n",
    "    regularization_factor: float = 1.0,\n",
    ") -> int:\n",
    "    _model = model.to(device)\n",
    "    _optimizer = torch.optim.SGD(\n",
    "        _model.parameters(),\n",
    "        lr=learning_rate,\n",
    "        momentum=momentum,\n",
    "    )\n",
    "    _criterion = nn.MSELoss()\n",
    "\n",
    "    def __is_mask_regularized() -> bool:\n",
    "        __input_mask = _model.input_mask.get_binary_mask(0)\n",
    "        __output_mask = _model.linear.masker.get_binary_mask(0)\n",
    "        # Strict condition\n",
    "        __input_mask_reg = torch.all(\n",
    "            __input_mask == torch.BoolTensor([1, 0, 1, 0, 1])\n",
    "        ).item()\n",
    "        # Relaxed condition: mask 0, 2, and 4 are all 1\n",
    "        # __input_mask_reg = __input_mask[0] & __input_mask[2] & __input_mask[4]\n",
    "        __output_mask_reg = torch.all(\n",
    "            __output_mask == torch.BoolTensor([1])\n",
    "        ).item()\n",
    "        return __input_mask_reg & __output_mask_reg\n",
    "\n",
    "    # Train the model for `num_epochs` epochs\n",
    "    _num_batches = 0\n",
    "    for _ in range(max_num_epochs):\n",
    "        for __i, (__x, __y) in enumerate(trn_dataloader):\n",
    "            _optimizer.zero_grad()\n",
    "            # Progress is required for the HAT mask scaling method\n",
    "            __progress = __i / (len(trn_dataloader) - 1)\n",
    "            try:\n",
    "                __pred = _model.forward(__x, __progress)\n",
    "            except TypeError:\n",
    "                __pred = _model.forward(__x)\n",
    "            __loss = _criterion(__pred, __y)\n",
    "            __reg = (\n",
    "                get_hat_reg_term(\n",
    "                    _model,\n",
    "                    strat=\"uniform\",\n",
    "                    task_id=0,\n",
    "                    mask_scale=_model.mask_scale,\n",
    "                    forgive_quota=False,\n",
    "                )\n",
    "                * regularization_factor\n",
    "            )\n",
    "            # print(f\"loss: {__loss}; reg: {__reg}\")\n",
    "            __loss += __reg\n",
    "            __loss.backward()\n",
    "            _num_batches += 1\n",
    "            _optimizer.step()\n",
    "        if __is_mask_regularized():\n",
    "            return _num_batches\n",
    "    return None"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:03:15.437148Z",
     "end_time": "2023-04-30T10:03:15.443399Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "outputs": [],
   "source": [
    "results = []\n",
    "for __init_strat, __scaling_strat in zip(\n",
    "    [\"normal\", \"dense\"],  # Default settings\n",
    "    [\"linear\", \"cosine\"],\n",
    "):\n",
    "    __num_batches = [\n",
    "        regularization_trial(\n",
    "            HATNetwork(\n",
    "                5,\n",
    "                True,\n",
    "                True,\n",
    "                __init_strat,\n",
    "                __scaling_strat,\n",
    "            ).to(device),\n",
    "            max_num_epochs,\n",
    "        )\n",
    "        for _ in range(num_trials)\n",
    "    ]\n",
    "    if None in __num_batches:\n",
    "        __num_failed_trials = __num_batches.count(None)\n",
    "        __num_batches = [\n",
    "            __num_batch\n",
    "            for __num_batch in __num_batches\n",
    "            if __num_batch is not None\n",
    "        ]\n",
    "        __fail_rate = __num_failed_trials / num_trials\n",
    "    else:\n",
    "        __fail_rate = 0\n",
    "    __num_batches = np.array(__num_batches)\n",
    "    __avg_num_batches = __num_batches.mean()\n",
    "    __result = [\n",
    "        __init_strat,\n",
    "        __scaling_strat,\n",
    "        __avg_num_batches,\n",
    "        __fail_rate,\n",
    "    ]\n",
    "    results.append(__result)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:03:15.447067Z",
     "end_time": "2023-04-30T10:04:00.963384Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "╒══════════════╤═════════════════╤═══════════════════╕\n",
      "│  Init Strat  │  Scaling Strat  │  Avg Num Batches  │\n",
      "╞══════════════╪═════════════════╪═══════════════════╡\n",
      "│    normal    │     linear      │      338.50       │\n",
      "│    dense     │     cosine      │       25.00       │\n",
      "╘══════════════╧═════════════════╧═══════════════════╛\n"
     ]
    }
   ],
   "source": [
    "result_df = pd.DataFrame(\n",
    "    results,\n",
    "    columns=[\n",
    "        \"Init Strat\",\n",
    "        \"Scaling Strat\",\n",
    "        \"Avg Num Batches\",\n",
    "        \"Fail Rate\",\n",
    "    ],\n",
    ")\n",
    "if result_df[\"Fail Rate\"].max() > 0.0:\n",
    "    result_df[\"Fail Rate\"] = result_df[\"Fail Rate\"].apply(\n",
    "        lambda x: f\"{100.0 * x:.1f}%\"\n",
    "    )\n",
    "else:\n",
    "    result_df.drop(\"Fail Rate\", axis=1, inplace=True)\n",
    "print(\n",
    "    result_df.to_markdown(\n",
    "        index=False,\n",
    "        floatfmt=\".2f\",\n",
    "        tablefmt=\"fancy_outline\",\n",
    "        stralign=\"center\",\n",
    "        numalign=\"center\",\n",
    "    )\n",
    ")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "start_time": "2023-04-30T10:04:00.963022Z",
     "end_time": "2023-04-30T10:04:00.964295Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "It is clear that dense initialization with cosine scaling strategy works much better for both weight convergence and mask regularization.\n",
    "Despite the fact that such improvement might not be as significant in bigger networks, it is still worth noting that the dense initialization strategy is more robust to the scaling strategy.\n",
    "\n",
    "However, dense initialization usually requires more training epochs or greater learning rate so that the masks are properly regularized.\n",
    "And adjustment to the regularization factor is often required to achieve the desired mask regularization."
   ],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "name": "hat-cl",
   "language": "python",
   "display_name": "Python (hat-cl)"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}