francesco_baiocchi_Sentence_Reordering.ipynb

{
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kaggle": {
      "accelerator": "gpu",
      "dataSources": [],
      "isInternetEnabled": true,
      "language": "python",
      "sourceType": "notebook",
      "isGpuEnabled": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.13"
    }
  },
  "nbformat_minor": 0,
  "nbformat": 4,
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Sentence Reconstruction"
      ],
      "metadata": {
        "id": "ElNaMbLnRdHR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Francesco Baiocchi Matricola 0001138660"
      ],
      "metadata": {
        "id": "9IcaWBsQyCXK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The purpose of this project is to take in input a sequence of words corresponding to a random permutation of a given english sentence, and reconstruct the original sentence.\n",
        "\n",
        "The otuput can be either produced in a single shot, or through an iterative (autoregressive) loop generating a single token at a time.\n",
        "\n",
        "\n",
        "CONSTRAINTS:\n",
        "* No pretrained model can be used.\n",
        "* The neural network models should have less the 20M parameters.\n",
        "* No postprocessing should be done (e.g. no beamsearch)\n",
        "* You cannot use additional training data.\n",
        "\n",
        "\n",
        "BONUS PARAMETERS:\n",
        "\n",
        "A bonus of 0-2 points will be attributed to incentivate the adoption of models with a low number of parameters."
      ],
      "metadata": {
        "id": "oXr4iGUGRms8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Dataset\n",
        "\n",
        "The dataset is composed by sentences taken from the generics_kb dataset of hugging face. We restricted the vocabolary to the 10K most frequent words, and only took sentences making use of this vocabulary."
      ],
      "metadata": {
        "id": "iQ8k-L-WUK7l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install datasets"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2024-06-10T15:17:37.438074Z",
          "iopub.status.busy": "2024-06-10T15:17:37.437723Z",
          "iopub.status.idle": "2024-06-10T15:17:50.521982Z",
          "shell.execute_reply": "2024-06-10T15:17:50.521050Z",
          "shell.execute_reply.started": "2024-06-10T15:17:37.438043Z"
        },
        "id": "nJ02vehGYySk",
        "outputId": "49ec9576-86cd-44e7-ea52-45d117028658",
        "trusted": true
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Requirement already satisfied: datasets in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (2.19.2)\n\nRequirement already satisfied: filelock in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (3.14.0)\n\nRequirement already satisfied: numpy>=1.17 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (1.26.2)\n\nRequirement already satisfied: pyarrow>=12.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (16.1.0)\n\nRequirement already satisfied: pyarrow-hotfix in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (0.6)\n\nRequirement already satisfied: dill<0.3.9,>=0.3.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (0.3.8)\n\nRequirement already satisfied: pandas in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (2.1.4)\n\nRequirement already satisfied: requests>=2.32.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (2.32.3)\n\nRequirement already satisfied: tqdm>=4.62.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (4.66.4)\n\nRequirement already satisfied: xxhash in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (3.4.1)\n\nRequirement already satisfied: multiprocess in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (0.70.16)\n\nRequirement already satisfied: fsspec<=2024.3.1,>=2023.1.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from fsspec[http]<=2024.3.1,>=2023.1.0->datasets) (2024.3.1)\n\nRequirement already satisfied: aiohttp in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (3.9.5)\n\nRequirement already satisfied: huggingface-hub>=0.21.2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (0.23.3)\n\nRequirement already satisfied: packaging in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (24.0)\n\nRequirement already satisfied: pyyaml>=5.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from datasets) (6.0.1)\n\nRequirement already satisfied: aiosignal>=1.1.2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (1.3.1)\n\nRequirement already satisfied: attrs>=17.3.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (23.2.0)\n\nRequirement already satisfied: frozenlist>=1.1.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (1.4.1)\n\nRequirement already satisfied: multidict<7.0,>=4.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (6.0.5)\n\nRequirement already satisfied: yarl<2.0,>=1.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (1.9.4)\n\nRequirement already satisfied: async-timeout<5.0,>=4.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from aiohttp->datasets) (4.0.3)\n\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from huggingface-hub>=0.21.2->datasets) (4.12.1)\n\nRequirement already satisfied: charset-normalizer<4,>=2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests>=2.32.1->datasets) (3.3.2)\n\nRequirement already satisfied: idna<4,>=2.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests>=2.32.1->datasets) (3.7)\n\nRequirement already satisfied: urllib3<3,>=1.21.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests>=2.32.1->datasets) (2.2.1)\n\nRequirement already satisfied: certifi>=2017.4.17 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests>=2.32.1->datasets) (2024.6.2)\n\nRequirement already satisfied: python-dateutil>=2.8.2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pandas->datasets) (2.9.0.post0)\n\nRequirement already satisfied: pytz>=2020.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pandas->datasets) (2024.1)\n\nRequirement already satisfied: tzdata>=2022.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from pandas->datasets) (2024.1)\n\nRequirement already satisfied: six>=1.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Download the dataset"
      ],
      "metadata": {
        "id": "807Wk-ir_bDU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow"
      ],
      "metadata": {
        "id": "6RWiLgLRxz7Z",
        "outputId": "540c7e0c-80ca-4a69-bcbb-30617de06ef8"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Requirement already satisfied: tensorflow in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (2.16.1)\n\nRequirement already satisfied: absl-py>=1.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (2.1.0)\n\nRequirement already satisfied: astunparse>=1.6.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (1.6.3)\n\nRequirement already satisfied: flatbuffers>=23.5.26 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (24.3.25)\n\nRequirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (0.5.4)\n\nRequirement already satisfied: google-pasta>=0.1.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (0.2.0)\n\nRequirement already satisfied: h5py>=3.10.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (3.11.0)\n\nRequirement already satisfied: libclang>=13.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (18.1.1)\n\nRequirement already satisfied: ml-dtypes~=0.3.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (0.3.2)\n\nRequirement already satisfied: opt-einsum>=2.3.2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (3.3.0)\n\nRequirement already satisfied: packaging in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (24.0)\n\nRequirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (4.23.4)\n\nRequirement already satisfied: requests<3,>=2.21.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (2.32.3)\n\nRequirement already satisfied: setuptools in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (69.5.1)\n\nRequirement already satisfied: six>=1.12.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (1.16.0)\n\nRequirement already satisfied: termcolor>=1.1.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (2.4.0)\n\nRequirement already satisfied: typing-extensions>=3.6.6 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (4.12.1)\n\nRequirement already satisfied: wrapt>=1.11.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (1.16.0)\n\nRequirement already satisfied: grpcio<2.0,>=1.24.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (1.64.0)\n\nRequirement already satisfied: tensorboard<2.17,>=2.16 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (2.16.2)\n\nRequirement already satisfied: keras>=3.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (3.1.0)\n\nRequirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (0.37.0)\n\nRequirement already satisfied: numpy<2.0.0,>=1.23.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow) (1.26.2)\n\nRequirement already satisfied: wheel<1.0,>=0.23.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from astunparse>=1.6.0->tensorflow) (0.43.0)\n\nRequirement already satisfied: rich in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras>=3.0.0->tensorflow) (13.7.1)\n\nRequirement already satisfied: namex in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras>=3.0.0->tensorflow) (0.0.8)\n\nRequirement already satisfied: optree in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras>=3.0.0->tensorflow) (0.11.0)\n\nRequirement already satisfied: charset-normalizer<4,>=2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests<3,>=2.21.0->tensorflow) (3.3.2)\n\nRequirement already satisfied: idna<4,>=2.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests<3,>=2.21.0->tensorflow) (3.7)\n\nRequirement already satisfied: urllib3<3,>=1.21.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests<3,>=2.21.0->tensorflow) (2.2.1)\n\nRequirement already satisfied: certifi>=2017.4.17 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests<3,>=2.21.0->tensorflow) (2024.6.2)\n\nRequirement already satisfied: markdown>=2.6.8 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow) (3.6)\n\nRequirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow) (0.7.2)\n\nRequirement already satisfied: werkzeug>=1.0.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow) (3.0.3)\n\nRequirement already satisfied: MarkupSafe>=2.1.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow) (2.1.5)\n\nRequirement already satisfied: markdown-it-py>=2.2.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras>=3.0.0->tensorflow) (3.0.0)\n\nRequirement already satisfied: pygments<3.0.0,>=2.13.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras>=3.0.0->tensorflow) (2.18.0)\n\nRequirement already satisfied: mdurl~=0.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow) (0.1.2)\n"
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from datasets import load_dataset\n",
        "from keras.layers import TextVectorization\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "np.random.seed(42)\n",
        "ds = load_dataset('generics_kb',trust_remote_code=True)['train']"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:17:50.524231Z",
          "iopub.status.busy": "2024-06-10T15:17:50.523922Z",
          "iopub.status.idle": "2024-06-10T15:19:29.727952Z",
          "shell.execute_reply": "2024-06-10T15:19:29.727235Z",
          "shell.execute_reply.started": "2024-06-10T15:17:50.524204Z"
        },
        "id": "_WjtqA8TrHcS",
        "trusted": true,
        "outputId": "fd0e9f8d-38d4-43ff-ac56-8d375c8b7c6f"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "2024-06-10 18:44:03.486140: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n\nTo enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n\n2024-06-10 18:44:04.239467: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Filter row with length greater than 8.\n"
      ],
      "metadata": {
        "id": "lAVLfsdc_ej5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ds = ds.filter(lambda row: len(row[\"generic_sentence\"].split(\" \")) > 8 )\n",
        "corpus = [ '<start> ' + row['generic_sentence'].replace(\",\",\" <comma>\") + ' <end>' for row in ds ]\n",
        "corpus = np.array(corpus)\n"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:19:29.729669Z",
          "iopub.status.busy": "2024-06-10T15:19:29.729238Z",
          "iopub.status.idle": "2024-06-10T15:20:42.784595Z",
          "shell.execute_reply": "2024-06-10T15:20:42.783544Z",
          "shell.execute_reply.started": "2024-06-10T15:19:29.729636Z"
        },
        "id": "iznq8xGNt2Zr",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a tokenizer and Detokenizer"
      ],
      "metadata": {
        "id": "FyYpXLCF_ldR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer=TextVectorization( max_tokens=10000, standardize=\"lower_and_strip_punctuation\", encoding=\"utf-8\",) #con il max prende le piu frequenti. ordina i token del vocab dal piu frequente al meno frequente\n",
        "tokenizer.adapt(corpus)\n",
        "\n",
        "class TextDetokenizer:\n",
        "    def __init__(self, vectorize_layer):\n",
        "        self.vectorize_layer = vectorize_layer\n",
        "        vocab = self.vectorize_layer.get_vocabulary()\n",
        "        self.index_to_word = {index: word for index, word in enumerate(vocab)}\n",
        "\n",
        "    def __detokenize_tokens(self, tokens):\n",
        "        def check_token(t):\n",
        "          if t == 3:\n",
        "            s=\"<start>\"\n",
        "          elif t == 2:\n",
        "            s=\"<end>\"\n",
        "          elif t == 7:\n",
        "            s=\"<comma>\"\n",
        "          else:\n",
        "            s=self.index_to_word.get(t, '[UNK]')\n",
        "          return s\n",
        "\n",
        "        return ' '.join([ check_token(token) for token in tokens if token != 0])\n",
        "\n",
        "    def __call__(self, batch_tokens):\n",
        "       return [self.__detokenize_tokens(tokens) for tokens in batch_tokens]\n",
        "\n",
        "\n",
        "detokenizer = TextDetokenizer( tokenizer )\n",
        "sentences = tokenizer( corpus ).numpy()"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:42.787058Z",
          "iopub.status.busy": "2024-06-10T15:20:42.786757Z",
          "iopub.status.idle": "2024-06-10T15:20:50.419823Z",
          "shell.execute_reply": "2024-06-10T15:20:50.419009Z",
          "shell.execute_reply.started": "2024-06-10T15:20:42.787032Z"
        },
        "id": "T-bE2JpVbU9E",
        "trusted": true,
        "outputId": "35e00e6a-d076-446a-f7cb-72106aeb58b9"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "2024-06-10 18:44:31.998541: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1928] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 20974 MB memory:  -> device: 0, name: NVIDIA L4, pci bus id: 0000:35:00.0, compute capability: 8.9\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Remove from corpus the sentences where any unknow word appears"
      ],
      "metadata": {
        "id": "lZ64sns1_pSK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "mask = np.sum( (sentences==1), axis=1) >= 1\n",
        "original_data = np.delete( sentences, mask , axis=0)"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.421291Z",
          "iopub.status.busy": "2024-06-10T15:20:50.420959Z",
          "iopub.status.idle": "2024-06-10T15:20:50.478797Z",
          "shell.execute_reply": "2024-06-10T15:20:50.477991Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.421266Z"
        },
        "id": "2LPQtryQz5wh",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "original_data.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.480287Z",
          "iopub.status.busy": "2024-06-10T15:20:50.479954Z",
          "iopub.status.idle": "2024-06-10T15:20:50.486478Z",
          "shell.execute_reply": "2024-06-10T15:20:50.485603Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.480259Z"
        },
        "id": "qYfOscVk7U0r",
        "outputId": "4adeb4a1-efdf-4d8f-e4dd-2015e9818624",
        "trusted": true
      },
      "execution_count": null,
      "outputs": [
        {
          "execution_count": 7,
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(241236, 28)"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Shuffle the sentences"
      ],
      "metadata": {
        "id": "5puiiQ2D_uxa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.utils import Sequence\n",
        "\n",
        "class DataGenerator(Sequence):\n",
        "    def __init__(self, data, mode=\"training\", batch_size=32, shuffle=True, seed=42):\n",
        "        self.data = data\n",
        "        self.batch_size = batch_size\n",
        "        self.shuffle = shuffle\n",
        "        self.seed = seed\n",
        "        self.mode = mode\n",
        "        self.on_epoch_end()\n",
        "\n",
        "    def __len__(self):\n",
        "        return int(np.floor(len(self.data) / self.batch_size))\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]\n",
        "\n",
        "        data_batch = np.array([self.data[k] for k in indexes])\n",
        "        #copy of ordered sequences\n",
        "        result = np.copy(data_batch)\n",
        "        #shuffle only the relevant positions for each batch\n",
        "        for i in range(data_batch.shape[0]):\n",
        "            np.random.shuffle(data_batch[i,1:data_batch[i].argmin() - 1])\n",
        "\n",
        "        if self.mode==\"training\" or self.mode==\"validation\":\n",
        "            # teacher forcing\n",
        "            target = np.array([np.append(sentence[1:], 0) for sentence in result])\n",
        "            data_batch_mask = np.where(data_batch==0,0,1)\n",
        "            result_mask = np.where(result==0,0,1)\n",
        "            return (data_batch , result, data_batch_mask, result_mask), target\n",
        "\n",
        "        elif self.mode==\"test\":\n",
        "            return data_batch, result\n",
        "\n",
        "    def on_epoch_end(self):\n",
        "        self.indexes = np.arange(len(self.data))\n",
        "        if self.shuffle:\n",
        "            if self.seed is not None:\n",
        "                np.random.seed(self.seed)\n",
        "            np.random.shuffle(self.indexes)"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.487988Z",
          "iopub.status.busy": "2024-06-10T15:20:50.487670Z",
          "iopub.status.idle": "2024-06-10T15:20:50.527322Z",
          "shell.execute_reply": "2024-06-10T15:20:50.526596Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.487964Z"
        },
        "id": "1ZXLkWB6od0R",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Make a random permutation of training and test set\n",
        "np.random.seed(42)\n",
        "# Shuffle the all data\n",
        "shuffled_indices = np.random.permutation(len(original_data))\n",
        "shuffled_data = original_data[shuffled_indices]"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.528782Z",
          "iopub.status.busy": "2024-06-10T15:20:50.528453Z",
          "iopub.status.idle": "2024-06-10T15:20:50.569684Z",
          "shell.execute_reply": "2024-06-10T15:20:50.568845Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.528750Z"
        },
        "id": "fNo_Jy3N8zHS",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_generator = DataGenerator(shuffled_data[:220000],batch_size=128)\n",
        "test_generator = DataGenerator(shuffled_data[220000:], mode='test', batch_size=128)"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.571001Z",
          "iopub.status.busy": "2024-06-10T15:20:50.570732Z",
          "iopub.status.idle": "2024-06-10T15:20:50.582856Z",
          "shell.execute_reply": "2024-06-10T15:20:50.581978Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.570978Z"
        },
        "id": "uNlq1Khx1oH2",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "x, y = test_generator.__getitem__(1)\n",
        "x = detokenizer(x)\n",
        "y = detokenizer(y)\n",
        "\n",
        "for i in range(7):\n",
        "  print(\"original: \", y[i])\n",
        "  print(\"shuffled: \", x[i])\n",
        "  print(\"\\n\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.586648Z",
          "iopub.status.busy": "2024-06-10T15:20:50.586382Z",
          "iopub.status.idle": "2024-06-10T15:20:50.602473Z",
          "shell.execute_reply": "2024-06-10T15:20:50.601622Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.586625Z"
        },
        "id": "qR5xwMOn4E88",
        "outputId": "c15b3995-877e-493d-8f1d-2679bf0c01cc",
        "trusted": true
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "original:  <start> all organisms cause changes in the environment in which they live <end>\n\nshuffled:  <start> changes cause they live which the organisms all in environment in <end>\n\n\n\n\n\noriginal:  <start> most business people have meetings on a daily basis <end>\n\nshuffled:  <start> basis daily business on meetings a people have most <end>\n\n\n\n\n\noriginal:  <start> humans thrive on stress because it makes life more interesting <end>\n\nshuffled:  <start> it humans interesting because life makes stress thrive more on <end>\n\n\n\n\n\noriginal:  <start> some bacteria can take up dna from the environment <comma> a process called transformation <end>\n\nshuffled:  <start> environment bacteria some from the called dna process transformation <comma> a take up can <end>\n\n\n\n\n\noriginal:  <start> flowers are rich source of vitamin c and have sweet taste <end>\n\nshuffled:  <start> sweet flowers are and c source vitamin have taste rich of <end>\n\n\n\n\n\noriginal:  <start> antihistamines make a good sleep aid for people with occasional insomnia <end>\n\nshuffled:  <start> a sleep make aid insomnia people good with for antihistamines occasional <end>\n\n\n\n\n\noriginal:  <start> fish are one of the most important natural renewable resource of the pacific countries <end>\n\nshuffled:  <start> resource natural of countries of the the most are renewable important one fish pacific <end>\n\n\n\n\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Metrics"
      ],
      "metadata": {
        "id": "Fo8MazCGBTv3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let s be the source string and p your prediction. The quality of the results will be measured according to the following metric:\n",
        "\n",
        "1.  look for the longest substring w between s and p\n",
        "2.  compute |w|/max(|s|,|p|)\n",
        "\n",
        "If the match is exact, the score is 1.\n",
        "\n",
        "When computing the score, you should NOT consider the start and end tokens.\n",
        "\n"
      ],
      "metadata": {
        "id": "G0NOkuO0CfPo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The longest common substring can be computed with the SequenceMatcher function of difflib, that allows a simple definition of our metric."
      ],
      "metadata": {
        "id": "a-aUrdlXDdVf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from difflib import SequenceMatcher\n",
        "\n",
        "def score(s,p):\n",
        "  match = SequenceMatcher(None, s, p).find_longest_match()\n",
        "  #print(match.size)\n",
        "  return (match.size/max(len(p),len(s)))"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.603756Z",
          "iopub.status.busy": "2024-06-10T15:20:50.603487Z",
          "iopub.status.idle": "2024-06-10T15:20:50.608707Z",
          "shell.execute_reply": "2024-06-10T15:20:50.607727Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.603734Z"
        },
        "id": "ulpTRdrF_huh",
        "trusted": true
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's do an example."
      ],
      "metadata": {
        "id": "RB2YfjXNExM-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "original = \"at first henry wanted to be friends with the king of france\"\n",
        "generated = \"henry wanted to be friends with king of france at the first\"\n",
        "\n",
        "print(\"your score is \",score(original,generated))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.609880Z",
          "iopub.status.busy": "2024-06-10T15:20:50.609623Z",
          "iopub.status.idle": "2024-06-10T15:20:50.619007Z",
          "shell.execute_reply": "2024-06-10T15:20:50.618114Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.609858Z"
        },
        "id": "h17C8bVjEwur",
        "outputId": "4da07b5d-d8a9-41d7-bc92-f59a629e551b",
        "trusted": true
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "your score is  0.5423728813559322\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The score must be computed as an average of at least 3K random examples taken form the test set."
      ],
      "metadata": {
        "id": "BET8GqBvFugR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# What to deliver"
      ],
      "metadata": {
        "id": "4fwo7xj4GBW1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You are supposed to deliver a single notebook, suitably commented.\n",
        "The notebook should describe a single model, although you may briefly discuss additional attempts you did.\n",
        "\n",
        "The notebook should contain a full trace of the training.\n",
        "Weights should be made available on request.\n",
        "\n",
        "You must also give a clear assesment of the performance of the model, computed with the metric that has been given to you.\n",
        "\n",
        "# Good work!"
      ],
      "metadata": {
        "id": "i6uITuxOGHfJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install keras_nlp\n",
        "!pip install keras"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T15:20:50.620354Z",
          "iopub.status.busy": "2024-06-10T15:20:50.620049Z",
          "iopub.status.idle": "2024-06-10T15:21:19.500866Z",
          "shell.execute_reply": "2024-06-10T15:21:19.499853Z",
          "shell.execute_reply.started": "2024-06-10T15:20:50.620331Z"
        },
        "trusted": true,
        "id": "1uJleFNGxz7c",
        "outputId": "59bd69c7-db02-4d76-e24e-cc6d2ee160f2"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Requirement already satisfied: keras_nlp in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (0.12.1)\n\nRequirement already satisfied: keras-core in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (0.1.7)\n\nRequirement already satisfied: absl-py in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (2.1.0)\n\nRequirement already satisfied: numpy in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (1.26.2)\n\nRequirement already satisfied: packaging in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (24.0)\n\nRequirement already satisfied: regex in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (2024.5.15)\n\nRequirement already satisfied: rich in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (13.7.1)\n\nRequirement already satisfied: dm-tree in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (0.1.8)\n\nRequirement already satisfied: kagglehub in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (0.2.5)\n\nRequirement already satisfied: tensorflow-text in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras_nlp) (2.16.1)\n\nRequirement already satisfied: requests in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from kagglehub->keras_nlp) (2.32.3)\n\nRequirement already satisfied: tqdm in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from kagglehub->keras_nlp) (4.66.4)\n\nRequirement already satisfied: namex in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras-core->keras_nlp) (0.0.8)\n\nRequirement already satisfied: h5py in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras-core->keras_nlp) (3.11.0)\n\nRequirement already satisfied: markdown-it-py>=2.2.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras_nlp) (3.0.0)\n\nRequirement already satisfied: pygments<3.0.0,>=2.13.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras_nlp) (2.18.0)\n\nRequirement already satisfied: tensorflow<2.17,>=2.16.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow-text->keras_nlp) (2.16.1)\n\nRequirement already satisfied: mdurl~=0.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->keras_nlp) (0.1.2)\n\nRequirement already satisfied: astunparse>=1.6.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (1.6.3)\n\nRequirement already satisfied: flatbuffers>=23.5.26 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (24.3.25)\n\nRequirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.5.4)\n\nRequirement already satisfied: google-pasta>=0.1.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.2.0)\n\nRequirement already satisfied: libclang>=13.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (18.1.1)\n\nRequirement already satisfied: ml-dtypes~=0.3.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.3.2)\n\nRequirement already satisfied: opt-einsum>=2.3.2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (3.3.0)\n\nRequirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (4.23.4)\n\nRequirement already satisfied: setuptools in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (69.5.1)\n\nRequirement already satisfied: six>=1.12.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (1.16.0)\n\nRequirement already satisfied: termcolor>=1.1.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (2.4.0)\n\nRequirement already satisfied: typing-extensions>=3.6.6 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (4.12.1)\n\nRequirement already satisfied: wrapt>=1.11.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (1.16.0)\n\nRequirement already satisfied: grpcio<2.0,>=1.24.3 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (1.64.0)\n\nRequirement already satisfied: tensorboard<2.17,>=2.16 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (2.16.2)\n\nRequirement already satisfied: keras>=3.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (3.1.0)\n\nRequirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.37.0)\n\nRequirement already satisfied: charset-normalizer<4,>=2 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests->kagglehub->keras_nlp) (3.3.2)\n\nRequirement already satisfied: idna<4,>=2.5 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests->kagglehub->keras_nlp) (3.7)\n\nRequirement already satisfied: urllib3<3,>=1.21.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests->kagglehub->keras_nlp) (2.2.1)\n\nRequirement already satisfied: certifi>=2017.4.17 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from requests->kagglehub->keras_nlp) (2024.6.2)\n\nRequirement already satisfied: wheel<1.0,>=0.23.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from astunparse>=1.6.0->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.43.0)\n\nRequirement already satisfied: optree in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras>=3.0.0->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.11.0)\n\nRequirement already satisfied: markdown>=2.6.8 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (3.6)\n\nRequirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (0.7.2)\n\nRequirement already satisfied: werkzeug>=1.0.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (3.0.3)\n\nRequirement already satisfied: MarkupSafe>=2.1.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow<2.17,>=2.16.1->tensorflow-text->keras_nlp) (2.1.5)\n\nRequirement already satisfied: keras in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (3.1.0)\n\nRequirement already satisfied: absl-py in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (2.1.0)\n\nRequirement already satisfied: numpy in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (1.26.2)\n\nRequirement already satisfied: rich in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (13.7.1)\n\nRequirement already satisfied: namex in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (0.0.8)\n\nRequirement already satisfied: h5py in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (3.11.0)\n\nRequirement already satisfied: optree in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (0.11.0)\n\nRequirement already satisfied: ml-dtypes in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from keras) (0.3.2)\n\nRequirement already satisfied: typing-extensions>=4.0.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from optree->keras) (4.12.1)\n\nRequirement already satisfied: markdown-it-py>=2.2.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras) (3.0.0)\n\nRequirement already satisfied: pygments<3.0.0,>=2.13.0 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from rich->keras) (2.18.0)\n\nRequirement already satisfied: mdurl~=0.1 in /system/conda/miniconda3/envs/cloudspace/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->keras) (0.1.2)\n"
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "import keras_nlp\n",
        "import keras\n",
        "import numpy as np\n",
        "from keras.callbacks import EarlyStopping"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T16:56:27.467722Z",
          "iopub.status.busy": "2024-06-10T16:56:27.467310Z",
          "iopub.status.idle": "2024-06-10T16:56:27.472466Z",
          "shell.execute_reply": "2024-06-10T16:56:27.471545Z",
          "shell.execute_reply.started": "2024-06-10T16:56:27.467687Z"
        },
        "trusted": true,
        "id": "0UYn5zNwxz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "The model is instantiated with the following parameters, each selected through a process of trial and error by training various models with different parameter combinations.\n",
        "\n",
        "Two of the most influential parameters were the number of attention heads and the dropout rate:\n",
        "\n",
        "•\tNumber of Heads: This parameter significantly affects how well the model captures underlying dependencies in sequences.\n",
        "\n",
        "•\tDropout Rate: This parameter determines the model's susceptibility to overfitting and underfitting."
      ],
      "metadata": {
        "id": "jETXgyYy1DIa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "BATCH_SIZE = 64\n",
        "EPOCHS = 200\n",
        "MAX_SEQUENCE_LENGTH = 28\n",
        "VOCAB_SIZE = 10000\n",
        "num_layers = 6\n",
        "EMBED_DIM = 256\n",
        "INTERMEDIATE_DIM = 1550\n",
        "NUM_HEADS = 8\n",
        "DROPOUT = 0.1"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:19.478193Z",
          "iopub.status.busy": "2024-06-10T17:25:19.477259Z",
          "iopub.status.idle": "2024-06-10T17:25:19.483243Z",
          "shell.execute_reply": "2024-06-10T17:25:19.482203Z",
          "shell.execute_reply.started": "2024-06-10T17:25:19.478150Z"
        },
        "trusted": true,
        "id": "h-uertD1xz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "For this project, I opted to implement a neural network based on the original Transformer model, as introduced in the influential paper:\n",
        "\n",
        "\t•\tVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All You Need. Advances in Neural Information Processing Systems, 30.\n",
        "\n",
        "Initially, I replicated the model as outlined in the paper and then made iterative improvements, resulting in the current version of the network.\n",
        "\n",
        "One of the first enhancements I implemented was modifying the DataGenerator class to better fit the model’s architecture. The __getitem__() method now supports two distinct modes based on its usage:\n",
        "\n",
        "\t•\tmode=“training”: This mode is designed for creating the training generator used in model.fit(). It also generates masks for padding elements and shifts the target sequence one token to the right, facilitating the teacher forcing technique.\n",
        "\t•\tmode=“test”: This mode is intended for creating both the test generator and the validation generator used for predictions. The validation data is a subset of the training data from the dataset."
      ],
      "metadata": {
        "id": "JF7qH082yXnf"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def get_transformer():\n",
        "    padding_mask_enc = keras.Input( shape=(None,), name=\"padding_mask_enc\",)\n",
        "    padding_mask_dec = keras.Input( shape=(None,),name=\"padding_mask_dec\",)\n",
        "    # Encoder\n",
        "    encoder_inputs = keras.Input(shape=(None,), name=\"encoder_inputs\")\n",
        "    embedding = keras.layers.Embedding(input_dim=VOCAB_SIZE, output_dim=EMBED_DIM)\n",
        "    x = embedding(encoder_inputs)\n",
        "    for i in range(num_layers-1):\n",
        "        x = keras_nlp.layers.TransformerEncoder(\n",
        "            intermediate_dim=INTERMEDIATE_DIM, num_heads=NUM_HEADS,dropout=DROPOUT\n",
        "        )(inputs=x, padding_mask=padding_mask_enc)\n",
        "    encoder_outputs = keras_nlp.layers.TransformerEncoder(\n",
        "            intermediate_dim=INTERMEDIATE_DIM, num_heads=NUM_HEADS,dropout=DROPOUT\n",
        "        )(inputs=x, padding_mask=padding_mask_enc)\n",
        "\n",
        "\n",
        "    # Decoder\n",
        "    decoder_inputs = keras.Input(shape=(None,), name=\"decoder_inputs\")\n",
        "    decoder_embedding = embedding(decoder_inputs)\n",
        "    x = keras_nlp.layers.PositionEmbedding(\n",
        "    sequence_length=MAX_SEQUENCE_LENGTH)(decoder_embedding)\n",
        "    x += decoder_embedding\n",
        "    for i in range(num_layers):\n",
        "        x = keras_nlp.layers.TransformerDecoder(\n",
        "            intermediate_dim=INTERMEDIATE_DIM, num_heads=NUM_HEADS,dropout=DROPOUT\n",
        "        )(decoder_sequence=x, encoder_sequence=encoder_outputs, decoder_padding_mask=padding_mask_dec, encoder_padding_mask=padding_mask_enc)\n",
        "    decoder_outputs = keras.layers.Dense(VOCAB_SIZE, activity_regularizer = 'l2', activation=\"softmax\")(x)\n",
        "    transformer = keras.Model(\n",
        "        [encoder_inputs, decoder_inputs, padding_mask_enc, padding_mask_dec],\n",
        "        decoder_outputs,\n",
        "        name=\"transformer\",\n",
        "    )\n",
        "    return transformer"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:19.626467Z",
          "iopub.status.busy": "2024-06-10T17:25:19.626084Z",
          "iopub.status.idle": "2024-06-10T17:25:19.637432Z",
          "shell.execute_reply": "2024-06-10T17:25:19.636475Z",
          "shell.execute_reply.started": "2024-06-10T17:25:19.626438Z"
        },
        "trusted": true,
        "id": "Jn5Nh-D6xz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = get_transformer()"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:19.769250Z",
          "iopub.status.busy": "2024-06-10T17:25:19.768915Z",
          "iopub.status.idle": "2024-06-10T17:25:20.631659Z",
          "shell.execute_reply": "2024-06-10T17:25:20.630641Z",
          "shell.execute_reply.started": "2024-06-10T17:25:19.769219Z"
        },
        "trusted": true,
        "id": "6XhspMRsxz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The custom scheduler often caused the model to get stuck in plateaus. However, by utilizing the restart technique with CosineDecayRestarts(), the model was able to break free from these plateaus, leading to a marked improvement in overall performance. This adjustment resulted in a significant increase in both training accuracy and the final test score, with an average gain of 0.02."
      ],
      "metadata": {
        "id": "KyoY13DnzhBe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class MyLRSchedule(keras.optimizers.schedules.LearningRateSchedule):\n",
        "\n",
        "    def __init__(self, d_model, warmup_steps=4000):\n",
        "        super().__init__()\n",
        "\n",
        "        self.d_model = d_model\n",
        "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
        "\n",
        "        self.warmup_steps = warmup_steps\n",
        "\n",
        "    def __call__(self, step):\n",
        "        step = tf.cast(step, dtype=tf.float32)\n",
        "        arg1 = tf.math.rsqrt(step)\n",
        "        arg2 = step * (self.warmup_steps ** -1.5)\n",
        "\n",
        "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n",
        "restart_scheduler = keras.optimizers.schedules.CosineDecayRestarts(8e-04,3e04,t_mul=0.5,m_mul=0.01)\n"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:20.634287Z",
          "iopub.status.busy": "2024-06-10T17:25:20.633473Z",
          "iopub.status.idle": "2024-06-10T17:25:20.641325Z",
          "shell.execute_reply": "2024-06-10T17:25:20.640476Z",
          "shell.execute_reply.started": "2024-06-10T17:25:20.634251Z"
        },
        "trusted": true,
        "id": "Z3ANdhOhxz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def masked_loss(label, pred):\n",
        "    mask = label != 0\n",
        "    loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
        "        from_logits=False, reduction='none')\n",
        "    loss = loss_object(label, pred)\n",
        "    mask = tf.cast(mask, dtype=loss.dtype)\n",
        "    loss *= mask\n",
        "    loss = tf.reduce_sum(loss)/tf.reduce_sum(mask)\n",
        "    return loss\n",
        "\n",
        "def masked_accuracy(label, pred):\n",
        "    pred = tf.argmax(pred, axis=2)\n",
        "    label = tf.cast(label, pred.dtype)\n",
        "    match = label == pred\n",
        "\n",
        "    mask = label != 0\n",
        "\n",
        "    match = match & mask\n",
        "\n",
        "    match = tf.cast(match, dtype=tf.float32)\n",
        "    mask = tf.cast(mask, dtype=tf.float32)\n",
        "    return tf.reduce_sum(match)/tf.reduce_sum(mask)"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:20.642791Z",
          "iopub.status.busy": "2024-06-10T17:25:20.642508Z",
          "iopub.status.idle": "2024-06-10T17:25:20.653987Z",
          "shell.execute_reply": "2024-06-10T17:25:20.653111Z",
          "shell.execute_reply.started": "2024-06-10T17:25:20.642767Z"
        },
        "trusted": true,
        "id": "mYc1dpvaxz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_generator = DataGenerator(shuffled_data[:217000], batch_size=200)\n",
        "val_generator = DataGenerator(shuffled_data[217000:220000], batch_size=1000, mode='test')"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:20.668119Z",
          "iopub.status.busy": "2024-06-10T17:25:20.667856Z",
          "iopub.status.idle": "2024-06-10T17:25:20.677827Z",
          "shell.execute_reply": "2024-06-10T17:25:20.677020Z",
          "shell.execute_reply.started": "2024-06-10T17:25:20.668096Z"
        },
        "trusted": true,
        "id": "QI840qBHxz7c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def clean_sentence(x):\n",
        "    x = x.replace('<start>', '').replace('<end>', '').replace('<pad>', '').strip()\n",
        "    return x"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:20.943988Z",
          "iopub.status.busy": "2024-06-10T17:25:20.943663Z",
          "iopub.status.idle": "2024-06-10T17:25:20.948904Z",
          "shell.execute_reply": "2024-06-10T17:25:20.947891Z",
          "shell.execute_reply.started": "2024-06-10T17:25:20.943963Z"
        },
        "trusted": true,
        "id": "u1o3NcE3xz7d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Translator(tf.Module):\n",
        "    def __init__(self, transformer, tokenizer):\n",
        "        self.transformer = transformer\n",
        "        self.tokenizer = tokenizer\n",
        "\n",
        "    def __call__(self, sentences, max_length=28):\n",
        "        batch_size = sentences.shape[0]\n",
        "\n",
        "        # generate word list for each sentence\n",
        "        bow = [[word for word in sentence if word not in [3, 2, 0]] for sentence in sentences]\n",
        "        # starting vector for prediction, it contains the sos index\n",
        "        #output = [[3]+[0]*(MAX_SEQUENCE_LENGTH-1) for _ in range(batch_size)]\n",
        "        output = [[3] for _ in range(batch_size)]\n",
        "\n",
        "        # during inference, output will be filled with the final sentence.\n",
        "\n",
        "        for i in range(1, max_length):\n",
        "            # (enc_input, dec_input)\n",
        "\n",
        "            predictions = np.array(self.transformer((np.array(sentences), np.array(output), np.where(np.array(sentences)==0,0,1),np.where(np.array(output)==0,0,1)), training=False))\n",
        "            # remove useless dimensions\n",
        "            predictions = predictions[:, -1, :]\n",
        "            for j in range(batch_size):\n",
        "                if len(bow[j]) == 0:\n",
        "                    # no more words to use\n",
        "                    cand_token = 2\n",
        "                else:\n",
        "                    # choose index with highest score\n",
        "                    s_prediction = predictions[j, np.array(bow[j])]\n",
        "                    cand_index = np.argmax(s_prediction)\n",
        "                    cand_token = bow[j][cand_index]\n",
        "                    del bow[j][cand_index]\n",
        "                #output[j][i] = cand_token\n",
        "                output[j].append(cand_token)\n",
        "        return output"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:21.414336Z",
          "iopub.status.busy": "2024-06-10T17:25:21.413975Z",
          "iopub.status.idle": "2024-06-10T17:25:21.424603Z",
          "shell.execute_reply": "2024-06-10T17:25:21.423541Z",
          "shell.execute_reply.started": "2024-06-10T17:25:21.414307Z"
        },
        "trusted": true,
        "id": "LPQPwyCsxz7d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The ValidationCallback played a crucial role in significantly enhancing the model’s performance. This custom callback monitors the model’s progress during training by evaluating its predictions on the validation set.\n",
        "This callback enables prolonged training sessions while preventing overfitting on the validation data, resulting in an average test score increase of 0.03."
      ],
      "metadata": {
        "id": "Cw2BZj_JzwDL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from tensorflow.keras.callbacks import Callback, ModelCheckpoint\n",
        "import numpy as np\n",
        "\n",
        "class ValidationCallback(Callback):\n",
        "    def __init__(self, tokenizer,generator, batch_size, max_patience):\n",
        "        super(ValidationCallback, self).__init__()\n",
        "        self.tokenizer = tokenizer\n",
        "        self.generator = generator\n",
        "        self.batch_size = batch_size\n",
        "        self.max_patience = max_patience\n",
        "        self.patience = 0\n",
        "        self.max_score = 0\n",
        "\n",
        "    def on_epoch_end(self, epoch, logs=None):\n",
        "        score_batch_size = self.batch_size\n",
        "        total_val_size = 3000\n",
        "        tot_score = 0\n",
        "        self.model.save_weights('current_model.weights.h5')\n",
        "        model = get_transformer()\n",
        "        model.load_weights('current_model.weights.h5')\n",
        "        translator = Translator(model, self.tokenizer)\n",
        "        for i in range(total_val_size // score_batch_size):\n",
        "            shuffled, ordered = self.generator[i]\n",
        "            y_pred = translator(shuffled)\n",
        "            pred_sentences = detokenizer(y_pred)\n",
        "            original_sentences = detokenizer(ordered)\n",
        "            b_score = 0\n",
        "            for j in range(score_batch_size):\n",
        "                b_score += score(clean_sentence(original_sentences[j]), clean_sentence(pred_sentences[j]))\n",
        "\n",
        "            tot_score += b_score\n",
        "            print(f\"\\nValidation Score after batch {i+1}: {tot_score / ((i+1) * score_batch_size)}\")\n",
        "        final_score = tot_score / total_val_size\n",
        "        if final_score >= self.max_score:\n",
        "            self.patience = 0\n",
        "            self.max_score = final_score\n",
        "            self.model.save_weights('best_model.weights.h5')\n",
        "        else:\n",
        "            if self.patience == self.max_patience:\n",
        "                self.model.load_weights('best_model.weights.h5')\n",
        "                self.model.stop_training = True\n",
        "            else:\n",
        "                self.patience +=1\n",
        "        logs['val_score'] = final_score"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:21.913322Z",
          "iopub.status.busy": "2024-06-10T17:25:21.912996Z",
          "iopub.status.idle": "2024-06-10T17:25:21.924344Z",
          "shell.execute_reply": "2024-06-10T17:25:21.923457Z",
          "shell.execute_reply.started": "2024-06-10T17:25:21.913297Z"
        },
        "trusted": true,
        "id": "iWyiVZf_xz7d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "score_callback = ValidationCallback(tokenizer=tokenizer,generator=val_generator,batch_size=1000, max_patience=8)"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:22.345376Z",
          "iopub.status.busy": "2024-06-10T17:25:22.345082Z",
          "iopub.status.idle": "2024-06-10T17:25:22.349615Z",
          "shell.execute_reply": "2024-06-10T17:25:22.348658Z",
          "shell.execute_reply.started": "2024-06-10T17:25:22.345353Z"
        },
        "trusted": true,
        "id": "hQ1FDWK8xz7d"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "transformer.summary()\n",
        "early_stopping = EarlyStopping(monitor='val_loss', patience=5, verbose=1, restore_best_weights=True) #stop if validation loss does not improve for 5 epochs\n",
        "opt = keras.optimizers.Adam(learning_rate=restart_scheduler,beta_1=0.9, beta_2=0.98,epsilon=1e-09)\n",
        "transformer.compile(optimizer=opt, loss=masked_loss, metrics=[masked_accuracy])\n",
        "history = transformer.fit(train_generator,epochs=EPOCHS,callbacks=score_callback)\n",
        "#  \"Accuracy\"\n",
        "plt.plot(history.history['masked_accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.show()\n",
        "# \"Loss\"\n",
        "plt.plot(history.history['val_score'])\n",
        "plt.title('model score')\n",
        "plt.ylabel('score')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['val'], loc='upper left')\n",
        "plt.show()\n"
      ],
      "metadata": {
        "execution": {
          "iopub.execute_input": "2024-06-10T17:25:22.665768Z",
          "iopub.status.busy": "2024-06-10T17:25:22.665019Z"
        },
        "trusted": true,
        "id": "GLe2b2FBxz7e",
        "outputId": "d528681f-9173-4597-8d68-0a480f0bcb3c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"transformer\"</span>\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1mModel: \"transformer\"\u001b[0m\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
              "┃<span style=\"font-weight: bold\"> Layer (type)        </span>┃<span style=\"font-weight: bold\"> Output Shape      </span>┃<span style=\"font-weight: bold\">    Param # </span>┃<span style=\"font-weight: bold\"> Connected to      </span>┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
              "│ decoder_inputs      │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)      │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ encoder_inputs      │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)      │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ embedding_28        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">2,560,000</span> │ encoder_inputs[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Embedding</span>)         │                   │            │ decoder_inputs[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ padding_mask_enc    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)      │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ embedding_28[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ transformer_enco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ transformer_enco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ transformer_enco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ position_embedding… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │      <span style=\"color: #00af00; text-decoration-color: #00af00\">7,168</span> │ embedding_28[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">PositionEmbedding</span>) │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ transformer_enco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ add_28 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Add</span>)        │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ position_embeddi… │\n",
              "│                     │                   │            │ embedding_28[<span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">…</span> │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ padding_mask_dec    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>)      │          <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │ -                 │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,059,598</span> │ transformer_enco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerEncode…</span> │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ add_28[<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>][<span style=\"color: #00af00; text-decoration-color: #00af00\">0</span>],     │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ transformer_deco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ transformer_deco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ transformer_deco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ transformer_deco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>) │  <span style=\"color: #00af00; text-decoration-color: #00af00\">1,323,278</span> │ transformer_deco… │\n",
              "│ (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">TransformerDecode…</span> │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ dense_28 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>,      │  <span style=\"color: #00af00; text-decoration-color: #00af00\">2,570,000</span> │ transformer_deco… │\n",
              "│                     │ <span style=\"color: #00af00; text-decoration-color: #00af00\">10000</span>)            │            │                   │\n",
              "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n",
              "</pre>\n"
            ],
            "text/plain": [
              "┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓\n",
              "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape     \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m   Param #\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mConnected to     \u001b[0m\u001b[1m \u001b[0m┃\n",
              "┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩\n",
              "│ decoder_inputs      │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)      │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
              "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ encoder_inputs      │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)      │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
              "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ embedding_28        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m2,560,000\u001b[0m │ encoder_inputs[\u001b[38;5;34m0\u001b[0m… │\n",
              "│ (\u001b[38;5;33mEmbedding\u001b[0m)         │                   │            │ decoder_inputs[\u001b[38;5;34m0\u001b[0m… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ padding_mask_enc    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)      │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
              "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ embedding_28[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m…\u001b[0m │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ transformer_enco… │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ transformer_enco… │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ transformer_enco… │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ position_embedding… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │      \u001b[38;5;34m7,168\u001b[0m │ embedding_28[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m…\u001b[0m │\n",
              "│ (\u001b[38;5;33mPositionEmbedding\u001b[0m) │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ transformer_enco… │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ add_28 (\u001b[38;5;33mAdd\u001b[0m)        │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │          \u001b[38;5;34m0\u001b[0m │ position_embeddi… │\n",
              "│                     │                   │            │ embedding_28[\u001b[38;5;34m1\u001b[0m][\u001b[38;5;34m…\u001b[0m │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ padding_mask_dec    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m)      │          \u001b[38;5;34m0\u001b[0m │ -                 │\n",
              "│ (\u001b[38;5;33mInputLayer\u001b[0m)        │                   │            │                   │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_encode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,059,598\u001b[0m │ transformer_enco… │\n",
              "│ (\u001b[38;5;33mTransformerEncode…\u001b[0m │                   │            │ padding_mask_enc… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ add_28[\u001b[38;5;34m0\u001b[0m][\u001b[38;5;34m0\u001b[0m],     │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ transformer_deco… │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ transformer_deco… │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ transformer_deco… │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ transformer_deco… │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ transformer_decode… │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m) │  \u001b[38;5;34m1,323,278\u001b[0m │ transformer_deco… │\n",
              "│ (\u001b[38;5;33mTransformerDecode…\u001b[0m │                   │            │ padding_mask_dec… │\n",
              "│                     │                   │            │ padding_mask_enc… │\n",
              "│                     │                   │            │ transformer_enco… │\n",
              "├─────────────────────┼───────────────────┼────────────┼───────────────────┤\n",
              "│ dense_28 (\u001b[38;5;33mDense\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;45mNone\u001b[0m,      │  \u001b[38;5;34m2,570,000\u001b[0m │ transformer_deco… │\n",
              "│                     │ \u001b[38;5;34m10000\u001b[0m)            │            │                   │\n",
              "└─────────────────────┴───────────────────┴────────────┴───────────────────┘\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">19,434,424</span> (74.14 MB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m19,434,424\u001b[0m (74.14 MB)\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">19,434,424</span> (74.14 MB)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m19,434,424\u001b[0m (74.14 MB)\n"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
              "</pre>\n"
            ],
            "text/plain": [
              "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
            ]
          },
          "metadata": {}
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "Epoch 1/200\n"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:120: UserWarning: Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.\n\n  self._warn_if_super_not_called()\n\nW0000 00:00:1718048761.114022    8384 assert_op.cc:38] Ignoring Assert operator compile_loss/masked_loss/sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/assert_equal_1/Assert/Assert\n\nI0000 00:00:1718048850.659306    8384 asm_compiler.cc:369] ptxas warning : Registers are spilled to local memory in function 'loop_add_subtract_fusion_80', 168 bytes spill stores, 168 bytes spill loads\n\nptxas warning : Registers are spilled to local memory in function 'loop_add_subtract_fusion_77', 48 bytes spill stores, 48 bytes spill loads\n\n\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 5.8896 - masked_accuracy: 0.2630"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/keras/src/ops/nn.py:545: UserWarning: You are using a softmax over axis 3 of a tensor of shape (1000, 8, 1, 1). This axis has size 1. The softmax operation will always return the value 1, which is likely not what you intended. Did you mean to use a sigmoid instead?\n\n  warnings.warn(\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "\n\nValidation Score after batch 1: 0.3733248206567494\n\n\n\nValidation Score after batch 2: 0.37327319030099393\n\n\n\nValidation Score after batch 3: 0.37435333301787105\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m241s\u001b[0m 116ms/step - loss: 5.8888 - masked_accuracy: 0.2631 - val_score: 0.3744\n\nEpoch 2/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 3.5163 - masked_accuracy: 0.6691\n\nValidation Score after batch 1: 0.4186246587297341\n\n\n\nValidation Score after batch 2: 0.41462534626975434\n\n\n\nValidation Score after batch 3: 0.41790206841907834\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 3.5162 - masked_accuracy: 0.6691 - val_score: 0.4179\n\nEpoch 3/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 3.1344 - masked_accuracy: 0.7584\n\nValidation Score after batch 1: 0.4501660724277956\n\n\n\nValidation Score after batch 2: 0.4460021108407554\n\n\n\nValidation Score after batch 3: 0.4455957087568995\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 3.1343 - masked_accuracy: 0.7584 - val_score: 0.4456\n\nEpoch 4/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 3.0059 - masked_accuracy: 0.7922\n\nValidation Score after batch 1: 0.4812291014173816\n\n\n\nValidation Score after batch 2: 0.4721431426398899\n\n\n\nValidation Score after batch 3: 0.4723056349623503\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m129s\u001b[0m 119ms/step - loss: 3.0059 - masked_accuracy: 0.7922 - val_score: 0.4723\n\nEpoch 5/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.9246 - masked_accuracy: 0.8161\n\nValidation Score after batch 1: 0.4988881271492513\n\n\n\nValidation Score after batch 2: 0.4942667984883136\n\n\n\nValidation Score after batch 3: 0.4905504450775516\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.9246 - masked_accuracy: 0.8161 - val_score: 0.4906\n\nEpoch 6/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.8695 - masked_accuracy: 0.8329\n\nValidation Score after batch 1: 0.502883069528301\n\n\n\nValidation Score after batch 2: 0.498965602046753\n\n\n\nValidation Score after batch 3: 0.4964242609254244\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.8695 - masked_accuracy: 0.8329 - val_score: 0.4964\n\nEpoch 7/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.8240 - masked_accuracy: 0.8483\n\nValidation Score after batch 1: 0.5156868152486612\n\n\n\nValidation Score after batch 2: 0.5081487461214014\n\n\n\nValidation Score after batch 3: 0.5054237753432379\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.8240 - masked_accuracy: 0.8483 - val_score: 0.5054\n\nEpoch 8/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.7870 - masked_accuracy: 0.8623\n\nValidation Score after batch 1: 0.5259948621255153\n\n\n\nValidation Score after batch 2: 0.5234893113666462\n\n\n\nValidation Score after batch 3: 0.5206057988571854\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m128s\u001b[0m 118ms/step - loss: 2.7870 - masked_accuracy: 0.8623 - val_score: 0.5206\n\nEpoch 9/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.7567 - masked_accuracy: 0.8741\n\nValidation Score after batch 1: 0.5315711628677333\n\n\n\nValidation Score after batch 2: 0.5267023431216438\n\n\n\nValidation Score after batch 3: 0.5251793953528735\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.7567 - masked_accuracy: 0.8741 - val_score: 0.5252\n\nEpoch 10/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.7281 - masked_accuracy: 0.8859\n\nValidation Score after batch 1: 0.5391469362443949\n\n\n\nValidation Score after batch 2: 0.5330932520259487\n\n\n\nValidation Score after batch 3: 0.5314359793739333\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.7281 - masked_accuracy: 0.8859 - val_score: 0.5314\n\nEpoch 11/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.7052 - masked_accuracy: 0.8961\n\nValidation Score after batch 1: 0.5440017221267422\n\n\n\nValidation Score after batch 2: 0.5405791686340651\n\n\n\nValidation Score after batch 3: 0.5315543438720249\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.7052 - masked_accuracy: 0.8961 - val_score: 0.5316\n\nEpoch 12/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.6836 - masked_accuracy: 0.9053\n\nValidation Score after batch 1: 0.5494436485331012\n\n\n\nValidation Score after batch 2: 0.5431263474511911\n\n\n\nValidation Score after batch 3: 0.5417712732167915\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m128s\u001b[0m 118ms/step - loss: 2.6836 - masked_accuracy: 0.9053 - val_score: 0.5418\n\nEpoch 13/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.6638 - masked_accuracy: 0.9150\n\nValidation Score after batch 1: 0.5565411657288227\n\n\n\nValidation Score after batch 2: 0.550547628447935\n\n\n\nValidation Score after batch 3: 0.5475514638451523\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.6638 - masked_accuracy: 0.9150 - val_score: 0.5476\n\nEpoch 14/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.6466 - masked_accuracy: 0.9232\n\nValidation Score after batch 1: 0.5685229286885857\n\n\n\nValidation Score after batch 2: 0.5649219545364598\n\n\n\nValidation Score after batch 3: 0.5581234620504884\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.6466 - masked_accuracy: 0.9232 - val_score: 0.5581\n\nEpoch 15/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.6303 - masked_accuracy: 0.9316\n\nValidation Score after batch 1: 0.5587684808180458\n\n\n\nValidation Score after batch 2: 0.5576788518124468\n\n\n\nValidation Score after batch 3: 0.553158883887272\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.6303 - masked_accuracy: 0.9316 - val_score: 0.5532\n\nEpoch 16/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 72ms/step - loss: 2.6151 - masked_accuracy: 0.9390\n\nValidation Score after batch 1: 0.5594533874040399\n\n\n\nValidation Score after batch 2: 0.5536740338403372\n\n\n\nValidation Score after batch 3: 0.5534430123166706\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m125s\u001b[0m 116ms/step - loss: 2.6151 - masked_accuracy: 0.9390 - val_score: 0.5534\n\nEpoch 17/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.6015 - masked_accuracy: 0.9463\n\nValidation Score after batch 1: 0.5593440157512204\n\n\n\nValidation Score after batch 2: 0.5588742434357185\n\n\n\nValidation Score after batch 3: 0.5588307855736147\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.6015 - masked_accuracy: 0.9462 - val_score: 0.5588\n\nEpoch 18/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5889 - masked_accuracy: 0.9531\n\nValidation Score after batch 1: 0.5680063628080056\n\n\n\nValidation Score after batch 2: 0.5638003876366471\n\n\n\nValidation Score after batch 3: 0.5627663159731843\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5889 - masked_accuracy: 0.9531 - val_score: 0.5628\n\nEpoch 19/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5786 - masked_accuracy: 0.9592\n\nValidation Score after batch 1: 0.5650700961331713\n\n\n\nValidation Score after batch 2: 0.5670172386999452\n\n\n\nValidation Score after batch 3: 0.5637304797365781\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 117ms/step - loss: 2.5786 - masked_accuracy: 0.9592 - val_score: 0.5637\n\nEpoch 20/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5686 - masked_accuracy: 0.9646\n\nValidation Score after batch 1: 0.5707488838950915\n\n\n\nValidation Score after batch 2: 0.5689812796754797\n\n\n\nValidation Score after batch 3: 0.5666933841615238\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5686 - masked_accuracy: 0.9646 - val_score: 0.5667\n\nEpoch 21/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5604 - masked_accuracy: 0.9688\n\nValidation Score after batch 1: 0.5671600880674522\n\n\n\nValidation Score after batch 2: 0.5695634335482582\n\n\n\nValidation Score after batch 3: 0.5679460355054637\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5604 - masked_accuracy: 0.9688 - val_score: 0.5679\n\nEpoch 22/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5533 - masked_accuracy: 0.9729\n\nValidation Score after batch 1: 0.5711973650668398\n\n\n\nValidation Score after batch 2: 0.5715504941631421\n\n\n\nValidation Score after batch 3: 0.5698719261505748\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5533 - masked_accuracy: 0.9729 - val_score: 0.5699\n\nEpoch 23/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5478 - masked_accuracy: 0.9760\n\nValidation Score after batch 1: 0.5639224989180052\n\n\n\nValidation Score after batch 2: 0.5648932881522635\n\n\n\nValidation Score after batch 3: 0.5675169030797947\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5478 - masked_accuracy: 0.9760 - val_score: 0.5675\n\nEpoch 24/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5431 - masked_accuracy: 0.9787\n\nValidation Score after batch 1: 0.5664309121581396\n\n\n\nValidation Score after batch 2: 0.5678555954266103\n\n\n\nValidation Score after batch 3: 0.5671832504639672\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 117ms/step - loss: 2.5431 - masked_accuracy: 0.9787 - val_score: 0.5672\n\nEpoch 25/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5406 - masked_accuracy: 0.9800\n\nValidation Score after batch 1: 0.5666704610535006\n\n\n\nValidation Score after batch 2: 0.5692718677760504\n\n\n\nValidation Score after batch 3: 0.5689147013199575\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5406 - masked_accuracy: 0.9800 - val_score: 0.5689\n\nEpoch 26/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5385 - masked_accuracy: 0.9814\n\nValidation Score after batch 1: 0.5702055373189823\n\n\n\nValidation Score after batch 2: 0.5704059140064539\n\n\n\nValidation Score after batch 3: 0.5694735115388578\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5385 - masked_accuracy: 0.9814 - val_score: 0.5695\n\nEpoch 27/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5372 - masked_accuracy: 0.9818\n\nValidation Score after batch 1: 0.568291647679986\n\n\n\nValidation Score after batch 2: 0.5693940306968688\n\n\n\nValidation Score after batch 3: 0.5687711783205593\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5372 - masked_accuracy: 0.9818 - val_score: 0.5688\n\nEpoch 28/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5367 - masked_accuracy: 0.9822\n\nValidation Score after batch 1: 0.568494744881126\n\n\n\nValidation Score after batch 2: 0.5698560249514344\n\n\n\nValidation Score after batch 3: 0.5700660608254497\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5367 - masked_accuracy: 0.9822 - val_score: 0.5701\n\nEpoch 29/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5365 - masked_accuracy: 0.9823\n\nValidation Score after batch 1: 0.568022302931682\n\n\n\nValidation Score after batch 2: 0.5686151160439551\n\n\n\nValidation Score after batch 3: 0.5688866455311617\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5365 - masked_accuracy: 0.9823 - val_score: 0.5689\n\nEpoch 30/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5359 - masked_accuracy: 0.9825\n\nValidation Score after batch 1: 0.5699532302053696\n\n\n\nValidation Score after batch 2: 0.5692289688881357\n\n\n\nValidation Score after batch 3: 0.5702065988925248\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m128s\u001b[0m 118ms/step - loss: 2.5359 - masked_accuracy: 0.9825 - val_score: 0.5702\n\nEpoch 31/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5361 - masked_accuracy: 0.9825\n\nValidation Score after batch 1: 0.5691291292623876\n\n\n\nValidation Score after batch 2: 0.5686793281188293\n\n\n\nValidation Score after batch 3: 0.569456769466902\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 117ms/step - loss: 2.5361 - masked_accuracy: 0.9825 - val_score: 0.5695\n\nEpoch 32/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5354 - masked_accuracy: 0.9828\n\nValidation Score after batch 1: 0.5700834444990819\n\n\n\nValidation Score after batch 2: 0.5688067221029512\n\n\n\nValidation Score after batch 3: 0.5694567331398499\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5354 - masked_accuracy: 0.9828 - val_score: 0.5695\n\nEpoch 33/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5350 - masked_accuracy: 0.9827\n\nValidation Score after batch 1: 0.5680425359951938\n\n\n\nValidation Score after batch 2: 0.5688211649459033\n\n\n\nValidation Score after batch 3: 0.5700876249961676\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5350 - masked_accuracy: 0.9827 - val_score: 0.5701\n\nEpoch 34/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5352 - masked_accuracy: 0.9831\n\nValidation Score after batch 1: 0.5696255370099094\n\n\n\nValidation Score after batch 2: 0.5690395277224266\n\n\n\nValidation Score after batch 3: 0.5690515808535365\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5352 - masked_accuracy: 0.9831 - val_score: 0.5691\n\nEpoch 35/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5343 - masked_accuracy: 0.9832\n\nValidation Score after batch 1: 0.5701879890881097\n\n\n\nValidation Score after batch 2: 0.5701439837117855\n\n\n\nValidation Score after batch 3: 0.5700777434765758\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5343 - masked_accuracy: 0.9832 - val_score: 0.5701\n\nEpoch 36/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5345 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5688167986035643\n\n\n\nValidation Score after batch 2: 0.5686484293802435\n\n\n\nValidation Score after batch 3: 0.5695184006491344\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5345 - masked_accuracy: 0.9835 - val_score: 0.5695\n\nEpoch 37/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5342 - masked_accuracy: 0.9834\n\nValidation Score after batch 1: 0.5693023858870137\n\n\n\nValidation Score after batch 2: 0.5695393591431032\n\n\n\nValidation Score after batch 3: 0.5698306898803484\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5342 - masked_accuracy: 0.9834 - val_score: 0.5698\n\nEpoch 38/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5341 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5700815926635254\n\n\n\nValidation Score after batch 2: 0.5705505183076427\n\n\n\nValidation Score after batch 3: 0.5703444902245111\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5341 - masked_accuracy: 0.9835 - val_score: 0.5703\n\nEpoch 39/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5336 - masked_accuracy: 0.9837\n\nValidation Score after batch 1: 0.5699791432779485\n\n\n\nValidation Score after batch 2: 0.5692702279376319\n\n\n\nValidation Score after batch 3: 0.5699936597037392\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 117ms/step - loss: 2.5336 - masked_accuracy: 0.9837 - val_score: 0.5700\n\nEpoch 40/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 72ms/step - loss: 2.5345 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5696377996086979\n\n\n\nValidation Score after batch 2: 0.56888386058328\n\n\n\nValidation Score after batch 3: 0.5697838909008696\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m126s\u001b[0m 116ms/step - loss: 2.5345 - masked_accuracy: 0.9835 - val_score: 0.5698\n\nEpoch 41/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5338 - masked_accuracy: 0.9836\n\nValidation Score after batch 1: 0.5702170375918673\n\n\n\nValidation Score after batch 2: 0.5692972641121536\n\n\n\nValidation Score after batch 3: 0.5701236702652952\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5338 - masked_accuracy: 0.9836 - val_score: 0.5701\n\nEpoch 42/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5339 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5696516262436521\n\n\n\nValidation Score after batch 2: 0.5690145584380462\n\n\n\nValidation Score after batch 3: 0.5697418406927438\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5339 - masked_accuracy: 0.9835 - val_score: 0.5697\n\nEpoch 43/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5339 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5696516564146614\n\n\n\nValidation Score after batch 2: 0.5690145735235508\n\n\n\nValidation Score after batch 3: 0.5697210174164136\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5339 - masked_accuracy: 0.9835 - val_score: 0.5697\n\nEpoch 44/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5337 - masked_accuracy: 0.9839\n\nValidation Score after batch 1: 0.5699060392411373\n\n\n\nValidation Score after batch 2: 0.5690855851615079\n\n\n\nValidation Score after batch 3: 0.5695774590374854\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5337 - masked_accuracy: 0.9839 - val_score: 0.5696\n\nEpoch 45/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5343 - masked_accuracy: 0.9835\n\nValidation Score after batch 1: 0.5695949281300261\n\n\n\nValidation Score after batch 2: 0.5689300296059523\n\n\n\nValidation Score after batch 3: 0.5694604220004483\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5343 - masked_accuracy: 0.9835 - val_score: 0.5695\n\nEpoch 46/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 74ms/step - loss: 2.5336 - masked_accuracy: 0.9838\n\nValidation Score after batch 1: 0.569544928130026\n\n\n\nValidation Score after batch 2: 0.5689050296059522\n\n\n\nValidation Score after batch 3: 0.5694437553337817\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5336 - masked_accuracy: 0.9838 - val_score: 0.5694\n\nEpoch 47/200\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 73ms/step - loss: 2.5338 - masked_accuracy: 0.9837\n\nValidation Score after batch 1: 0.569544928130026\n\n\n\nValidation Score after batch 2: 0.5689050296059522\n\n\n\nValidation Score after batch 3: 0.5694437553337817\n\n\u001b[1m1085/1085\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m127s\u001b[0m 117ms/step - loss: 2.5338 - masked_accuracy: 0.9837 - val_score: 0.5694\n"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABMJ0lEQVR4nO3dd3xUVd7H8e+kTRLSCIEEQiChSBEIECSGoqzGjYKsXXRVmuKzCgpkXQUp1jU2WCworiu6dhTLo4uLYhR8BKQK0pEeShICpJM2c58/QgZiAkKYmZtMPu/Xa17JnNy585u5rPe755x7rsUwDEMAAAAewsvsAgAAAJyJcAMAADwK4QYAAHgUwg0AAPAohBsAAOBRCDcAAMCjEG4AAIBHIdwAAACPQrgBAAAehXADwGn27Nkji8Wit95665xfu3jxYlksFi1evNjpdQFoXAg3AADAoxBuAACARyHcAIALFRUVmV0C0OgQbgAP8uijj8pisWj79u26/fbbFRoaqubNm2vatGkyDEMZGRm65pprFBISoqioKM2YMaPGPrKzs3XnnXcqMjJS/v7+io+P17///e8a2+Xm5mrkyJEKDQ1VWFiYRowYodzc3Frr2rp1q2688UaFh4fL399fffr00RdffFGnz7h3717de++96tSpkwICAtSsWTPddNNN2rNnT601Tpw4UbGxsbJarWrdurWGDx+unJwcxzYlJSV69NFHdcEFF8jf318tW7bU9ddfr507d0o6/Vyg2uYXjRw5UkFBQdq5c6cGDx6s4OBg3XbbbZKk//u//9NNN92kNm3ayGq1KiYmRhMnTtTx48dr/b5uvvlmNW/eXAEBAerUqZOmTJkiSfr+++9lsVj02Wef1Xjd+++/L4vFouXLl5/r1wp4FB+zCwDgfMOGDVOXLl309NNPa8GCBXryyScVHh6u1157TZdddpmeeeYZvffee3rggQd00UUX6ZJLLpEkHT9+XIMGDdKOHTs0btw4xcXF6eOPP9bIkSOVm5ur8ePHS5IMw9A111yjH3/8UX/5y1/UpUsXffbZZxoxYkSNWjZt2qT+/fsrOjpakyZNUpMmTfTRRx/p2muv1SeffKLrrrvunD7bqlWrtGzZMt1yyy1q3bq19uzZo1dffVWDBg3S5s2bFRgYKEkqLCzUwIEDtWXLFo0ePVq9e/dWTk6OvvjiC+3fv18RERGy2Wy6+uqrlZ6erltuuUXjx49XQUGBFi1apI0bN6p9+/bn/N1XVFQoJSVFAwYM0PPPP++o5+OPP1ZxcbHuueceNWvWTCtXrtRLL72k/fv36+OPP3a8/pdfftHAgQPl6+uru+++W7Gxsdq5c6e+/PJL/f3vf9egQYMUExOj9957r8Z3995776l9+/ZKSko657oBj2IA8BiPPPKIIcm4++67HW0VFRVG69atDYvFYjz99NOO9mPHjhkBAQHGiBEjHG2zZs0yJBnvvvuuo62srMxISkoygoKCjPz8fMMwDOPzzz83JBnPPvtstfcZOHCgIcl48803He2XX3650b17d6OkpMTRZrfbjX79+hkdO3Z0tH3//feGJOP7778/42csLi6u0bZ8+XJDkvH222872qZPn25IMj799NMa29vtdsMwDGPu3LmGJGPmzJmn3eZ0de3evbvGZx0xYoQhyZg0adJZ1Z2WlmZYLBZj7969jrZLLrnECA4OrtZ2aj2GYRiTJ082rFarkZub62jLzs42fHx8jEceeaTG+wCNDcNSgAe66667HL97e3urT58+MgxDd955p6M9LCxMnTp10q5duxxtX331laKionTrrbc62nx9fXX//fersLBQS5YscWzn4+Oje+65p9r73HfffdXqOHr0qL777jvdfPPNKigoUE5OjnJycnTkyBGlpKTo119/1YEDB87pswUEBDh+Ly8v15EjR9ShQweFhYVp7dq1jr998sknio+Pr7VnyGKxOLaJiIioUfep29TFqd9LbXUXFRUpJydH/fr1k2EY+vnnnyVJhw8f1g8//KDRo0erTZs2p61n+PDhKi0t1fz58x1t8+bNU0VFhW6//fY61w14CsIN4IF+e2IMDQ2Vv7+/IiIiarQfO3bM8Xzv3r3q2LGjvLyq/6ehS5cujr9X/WzZsqWCgoKqbdepU6dqz3fs2CHDMDRt2jQ1b9682uORRx6RVDnH51wcP35c06dPV0xMjKxWqyIiItS8eXPl5uYqLy/Psd3OnTvVrVu3M+5r586d6tSpk3x8nDdC7+Pjo9atW9do37dvn0aOHKnw8HAFBQWpefPmuvTSSyXJUXdV0Py9ujt37qyLLrpI7733nqPtvffe08UXX6wOHTo466MADRZzbgAP5O3tfVZtUuX8GVex2+2SpAceeEApKSm1bnOuJ+P77rtPb775piZMmKCkpCSFhobKYrHolltucbyfM52uB8dms9XabrVaa4RDm82mK664QkePHtVDDz2kzp07q0mTJjpw4IBGjhxZp7qHDx+u8ePHa//+/SotLdVPP/2kl19++Zz3A3giwg0Ah7Zt2+qXX36R3W6vdoLeunWr4+9VP9PT01VYWFit92bbtm3V9teuXTtJlUNbycnJTqlx/vz5GjFiRLUrvUpKSmpcqdW+fXtt3LjxjPtq3769VqxYofLycvn6+ta6TdOmTSWpxv6rerHOxoYNG7R9+3b9+9//1vDhwx3tixYtqrZd1ff1e3VL0i233KLU1FR98MEHOn78uHx9fTVs2LCzrgnwZAxLAXAYPHiwMjMzNW/ePEdbRUWFXnrpJQUFBTmGUQYPHqyKigq9+uqrju1sNpteeumlavtr0aKFBg0apNdee02HDh2q8X6HDx8+5xq9vb1r9Da99NJLNXpSbrjhBq1fv77WS6arXn/DDTcoJyen1h6Pqm3atm0rb29v/fDDD9X+/sorr5xTzafus+r3F154odp2zZs31yWXXKK5c+dq3759tdZTJSIiQldddZXeffddvffee7ryyitrDDsCjRU9NwAc7r77br322msaOXKk1qxZo9jYWM2fP19Lly7VrFmzFBwcLEkaOnSo+vfvr0mTJmnPnj3q2rWrPv3002pzXqrMnj1bAwYMUPfu3TVmzBi1a9dOWVlZWr58ufbv36/169efU41XX3213nnnHYWGhqpr165avny5vv32WzVr1qzadn/72980f/583XTTTRo9erQSEhJ09OhRffHFF5ozZ47i4+M1fPhwvf3220pNTdXKlSs1cOBAFRUV6dtvv9W9996ra665RqGhobrpppv00ksvyWKxqH379vrPf/5zTnOFOnfurPbt2+uBBx7QgQMHFBISok8++aTafKcqL774ogYMGKDevXvr7rvvVlxcnPbs2aMFCxZo3bp11bYdPny4brzxRknSE088cU7fI+DRzLpMC4DzVV0Kfvjw4WrtI0aMMJo0aVJj+0svvdS48MILq7VlZWUZo0aNMiIiIgw/Pz+je/fu1S53rnLkyBHjjjvuMEJCQozQ0FDjjjvuMH7++ecal0cbhmHs3LnTGD58uBEVFWX4+voa0dHRxtVXX23Mnz/fsc3ZXgp+7NgxR31BQUFGSkqKsXXrVqNt27bVLmuvqnHcuHFGdHS04efnZ7Ru3doYMWKEkZOT49imuLjYmDJlihEXF2f4+voaUVFRxo033mjs3LnTsc3hw4eNG264wQgMDDSaNm1q/M///I+xcePGWi8Fr+17NgzD2Lx5s5GcnGwEBQUZERERxpgxY4z169fX+n1t3LjRuO6664ywsDDD39/f6NSpkzFt2rQa+ywtLTWaNm1qhIaGGsePHz/j9wY0JhbDcOFsQgCAy1RUVKhVq1YaOnSo3njjDbPLAeoN5twAQAP1+eef6/Dhw9UmKQOQ6LkBgAZmxYoV+uWXX/TEE08oIiKi2uKFAOi5AYAG59VXX9U999yjFi1a6O233za7HKDeoecGAAB4FHpuAACARyHcAAAAj9LoFvGz2+06ePCggoODz+uuvwAAwH0Mw1BBQYFatWpV4/5tv9Xows3BgwcVExNjdhkAAKAOMjIy1Lp16zNu0+jCTdXy8RkZGQoJCTG5GgAAcDby8/MVExPjOI+fSaMLN1VDUSEhIYQbAAAamLOZUmLqhOIffvhBQ4cOVatWrWSxWPT555//7msWL16s3r17y2q1qkOHDnrrrbdcXicAAGg4TA03RUVFio+P1+zZs89q+927d2vIkCH6wx/+oHXr1mnChAm666679PXXX7u4UgAA0FCYOix11VVX6aqrrjrr7efMmaO4uDjNmDFDktSlSxf9+OOP+sc//qGUlBRXlQkAABqQBjXnZvny5UpOTq7WlpKSogkTJjj9vWw2m8rLy52+38bA19dX3t7eZpcBAGikGlS4yczMVGRkZLW2yMhI5efn6/jx4woICKjxmtLSUpWWljqe5+fnn/E9DMNQZmamcnNznVJzYxUWFqaoqCjWEgIAuF2DCjd1kZaWpscee+yst68KNi1atFBgYCAn53NkGIaKi4uVnZ0tSWrZsqXJFQEAGpsGFW6ioqKUlZVVrS0rK0shISG19tpI0uTJk5Wamup4XnWdfG1sNpsj2DRr1sx5hTcyVcciOztbLVq0YIgKAOBWDSrcJCUl6auvvqrWtmjRIiUlJZ32NVarVVar9az2XzXHJjAwsO5FQtLJ77C8vJxwAwBwK1MvBS8sLNS6deu0bt06SZWXeq9bt0779u2TVNnrMnz4cMf2f/nLX7Rr1y49+OCD2rp1q1555RV99NFHmjhxolPrYijq/PEdAgDMYmq4Wb16tXr16qVevXpJklJTU9WrVy9Nnz5dknTo0CFH0JGkuLg4LViwQIsWLVJ8fLxmzJihf/3rX1wGDgAAHEwdlho0aJAMwzjt32tbfXjQoEH6+eefXVgVYmNjNWHCBJdcYg8AgKs1qDk3OL1BgwapZ8+emjVr1nnva9WqVWrSpMn5FwUAgAkIN42EYRiy2Wzy8fn9Q968eXM3VAQA9YdhGLIbp/yUIcOQLBbJIossFsnLYpFFJ9pqmVdoGJWvMSTZDUP2qudG5XOp+v6q/a4T+7ecur/f7L+Weqveo+r9Tv0MVX+rel+p8n0qn1tO+b3y528/vww5fq/626mv/W39OuW5r7eXQgN863IonIJw4wFGjhypJUuWaMmSJXrhhRckSW+++aZGjRqlr776SlOnTtWGDRv0zTffKCYmRqmpqfrpp59UVFSkLl26KC0trdrKz78dlrJYLHr99de1YMECff3114qOjtaMGTP0pz/9yYyPC9QLhmGozGZXSbldJeU2VdgN2e2/PdGcPMnY7dVPRFUnQOPEScOQZLdXPznZjMp92uzGyd+r9lnVbjdUYTdks9tP/DRO/rRVtttOnJSqTrzGKScvQ9VPylV/qzopVtV2arvdOPk+lTVV1lNht8tmP/l3x+dTzc9f9R3Y7IbKT9RZYTNUXvXzN21VQeNkwDh5UrVYLPI6EThO/S5PDRqnfu9Gjed1+zdw6vufGiQg9W4Tpk/v7W/a+xNufodhGDpebjPlvQN8vc/qqqMXXnhB27dvV7du3fT4449LkjZt2iRJmjRpkp5//nm1a9dOTZs2VUZGhgYPHqy///3vslqtevvttzV06FBt27ZNbdq0Oe17PPbYY3r22Wf13HPP6aWXXtJtt92mvXv3Kjw83DkfFnAjwzBUVGZTbnGZcovLday4TMeKy5VbXKZjRZXPc4vLlHu8XEWlFTpebtPxMpsjyJSU23S83CY7J7NGrSo81edUY7HUrbzf9iZZKrtlZFHVZz4Zdh2BUSffy+wrZgk3v+N4uU1dp5tz1/HNj6co0O/3D1FoaKj8/PwUGBioqKgoSdLWrVslSY8//riuuOIKx7bh4eGKj493PH/iiSf02Wef6YsvvtC4ceNO+x4jR47UrbfeKkl66qmn9OKLL2rlypW68sor6/TZAFcwDEO5xeXKKihRdn6psgtKlZVfosMFpco+pS27oEQl5Xanva+XRfLx9pKXRfK2WBwnBC8vi7wtFkfPgtcpPQynDktUa1Pldt5elfvx8jqxzxP7qvrp7VX5u69X5e8+3hZ5e3nJp+r5iZ9V+zk5LHFqj8cpwxOWWv5Wbcjh5FCEt8Uib+9T6rBUvn9V3VV1Vn1my29+ep3yvpW1e53yObzk622Rj1flT28vi3y9T17Y+9seGPuJM61xYgil8j1O7v/U95Uqj0lVu5el8nN7WU5ts5xoO9kTZDfkOJlX9axV9QhV9X5V+5xVz71qDjfVGgZ+s+/agkFtUeHU7/i336+3l+W0AcM4pZfJOKXNEWScEEzOdLGQOxBuPFyfPn2qPS8sLNSjjz6qBQsW6NChQ6qoqNDx48erXXJfmx49ejh+b9KkiUJCQhy3WADcLe94ufbkFGl3TpF2nfhZ9bywtOKs9+Pn46Wmgb5qGuinsEBfhQX4qWkTX4UF+qlpYOXPYKuP/P285e/jrQA/bwX4esvf16vy54nnp558gfrOckrYOqXV6e9hJsLN7wjw9dbmx81ZRyfA9/xX9v3tVU8PPPCAFi1apOeff14dOnRQQECAbrzxRpWVlZ1xP76+1SeGWSwW2e3O+3++QG3yjpdr88F8bT6Ur62H8rX7RIA5UnTmf6/hTfzUItiq5sFWtQj2V4sQqyKDrWoR4q8WJ9oigv3OeugXQMNCuPkdFovlrIaGzObn5yeb7ffnBi1dulQjR47UddddJ6myJ2fPnj0urg44M8MwdDCvpDLIHMzXpoN52nwoX/uPHT/ta1oEWxUX0UTtmjdRXEQTxTar/D0mPFBWH275ATRm9f+sjbMSGxurFStWaM+ePQoKCjptr0rHjh316aefaujQobJYLJo2bRo9MHC7knKb1u47ppW7j2rVnqPadDBfucXltW7bummAurYMUZeWIerQIqgyyEQ0UZCV/3wBqB3/dfAQDzzwgEaMGKGuXbvq+PHjevPNN2vdbubMmRo9erT69euniIgIPfTQQ8rPz3dztWhsikortGbvMa3YfUQrdx/V+ow8ldmqh2ofL4s6tAjSha1C1bVViLq2rHyEBpq3VgaAhslimD2l2c3y8/MVGhqqvLw8hYSEVPtbSUmJdu/erbi4OPn7+5tUoWfgu2zcCkrKtWrPUa3YdVQ/7T6qjQfyZPvNddORIVYlxjVT37hw9YwJU8fIIIaTAJzWmc7fv0XPDYDzVlJu09q9x7Rs5xEt25mj9ftrhpnWTQPUNy5cF8c1U2K7cLUJD2QyLwCXINwAOGflNrt+2Z+n5TtztGznEa3ee0xlFdWHmdo2C3QEmcR2zRQdFmBStQAaG8INgLOSV1yu9K1ZWrgxU8t2HqmxnkxkiFX92keoX/tmSmrfTK2bBppUKYDGjnAD4LSOFJbqm81Z+u/GTC3bkaOKU4aawgJ9ldSumfq1b6Z+HSLULqIJw0wA6gXCTS0a2Rxrl+A7bLgO5R3X1xsz9d+NmVq152i1+yddEBmkK7u11B+7RqpryxB5eRFmANQ/hJtTVK3CW1xcrIAA5gecj+LiYkk1VzZG/ZRTWKov1h3UF+sPal1GbrW/dY8O1ZXdonRltyi1bx5kToEAcA4IN6fw9vZWWFiY455JgYFczXGuDMNQcXGxsrOzFRYWJm9vLu2tr0orbPpuS7Y+Wbtfi7cddgw5WSxSQpumurJblFIujFJMOHNnADQshJvfqLqrNjeFPD9hYWGO7xL1h2EYWpeRq0/W7teX6w8p7/jJVYHjY8J0fa9oXdUtSi1CWJsIQMNFuPkNi8Wili1bqkWLFiovr305eJyZr68vPTb1zMHc4/rs5wP6ZO1+7Tpc5GiPCvHXdb2jdUPvaHVoEWxihQDgPISb0/D29uYEjQat3GZX+pZsfbByn3749bCq5nj7+3rpqm4tdX3vaPVrHyFvJgUD8DCEG8DDZBwt1rxVGfpodYayC0od7Ylx4bohobUGd2/JTScBeDT+Cwd4gKpemvdX7tP/ndJLExHkp5v6xOiWi2LUtlkTc4sEADch3AANWMbRYn24ap8+Wr1fh0/ppRnYMUK39m2j5C6R8vPxMrFCAHA/wg3QwBiGoeU7j2ju0t1K35pNLw0A/AbhBmggSspt+vznA3pz6R5tyypwtA/sGKE/922jy+mlAQBJhBug3juUd1zvLN+rD1bu07HiyuUJAv28dWNCa43oF8uqwQDwG4QboB4yDENr9+XqzaW79d+NmbKdWD24ddMAjUiK1c0XxSg0gFtbAEBtCDdAPVJhs+u/GzP1rx93a/0p93hKjAvXqP5xuqJrJOvSAMDvINwA9UBBSbnmrcrQm0v36EDucUmSn4+XrolvpZH9Y3Vhq1CTKwSAhoNwA5joYO5xvbVsjz5YsU8FpRWSpPAmfrrj4ra6I6mtIoKsJlcIAA0P4QYwwcYDeXr9/3ZpwS+HHHfjbt+8ie4a2E7X9YqWvy+3/gCAuiLcAG5iGIYWbzus137YqZ92HXW0X9wuXHdf0k6DLmghL+bTAMB5I9wALmYYhr7dkq0X03/VhgN5kiRvL4uu7tFSYwa2U7do5tMAgDMRbgAXsdsNfbM5Sy+m/6rNh/IlSQG+3rotsY1GD4hTq7AAkysEAM9EuAGczG43tHBTpl5M/1VbMytXEm7i563h/WJ114A4NWOSMAC4FOEGcBKb3dBXGw7ppe9+1fasQklSkNVHI/vF6s4BcWraxM/kCgGgcSDcAOfJZjf0n18O6qXvdmhHdmWoCfb30aj+cRrdP1ZhgYQaAHAnwg1QR3a7of9uzNSsb7fr1xOhJsTfR3cOaKeR/WO5PQIAmIRwA5wjwzD09aYszfp2u2NOTYi/j8YMrAw1wf6EGgAwE+EGOEuGYei7rdmauWi7Nh2svPop2Oqj0QPiNHpAHD01AFBPEG6A32EYhpZsP6x/LNqu9fsr16lp4uetUf3jdNfAOObUAEA9Q7gBzuCnXUf03NfbtGbvMUmV69SM6Beruy9pp3CufgKAeolwA9Rie1aBnvnvVqVvzZYkWX28NDyprf7n0vbczBIA6jnCDXCKrPwS/WPRdn20OkN2Q/LxsujWvm1032Ud1CLE3+zyAABngXADSCooKddrS3bpXz/uUkm5XZJ0Vbco/S2lk9o1DzK5OgDAufAyu4DZs2crNjZW/v7+SkxM1MqVK0+7bXl5uR5//HG1b99e/v7+io+P18KFC91YLTxNWYVd/162R5c+t1gvf79DJeV2JbRtqk/uSdKrtycQbACgATK152bevHlKTU3VnDlzlJiYqFmzZiklJUXbtm1TixYtamw/depUvfvuu3r99dfVuXNnff3117ruuuu0bNky9erVy4RPgIbKMCoX4Ht24VbtOVIsSWoX0UQPXdVZf+waKYvFYnKFAIC6shiGYZj15omJibrooov08ssvS5LsdrtiYmJ03333adKkSTW2b9WqlaZMmaKxY8c62m644QYFBATo3XffPav3zM/PV2hoqPLy8hQSEuKcD4IGZdWeo3rqqy36eV+uJCkiyE8Tki/QsIti5OttemcmAKAW53L+Nq3npqysTGvWrNHkyZMdbV5eXkpOTtby5ctrfU1paan8/atP6gwICNCPP/542vcpLS1VaWmp43l+fv55Vo6GaufhQj3z3636ZnOWpMrLusdc0k53X9JOQVamnwGApzDtv+g5OTmy2WyKjIys1h4ZGamtW7fW+pqUlBTNnDlTl1xyidq3b6/09HR9+umnstlsp32ftLQ0PfbYY06tHQ3L4YJSvZC+XR+szJDNbsjLIg27qI0mJnfkCigA8EANqg/+hRdeUMeOHdW5c2f5+flp3LhxGjVqlLy8Tv8xJk+erLy8PMcjIyPDjRXDTMVlFXrh21816Lnv9e5P+2SzG0ru0kJfT7hEadd3J9gAgIcyrecmIiJC3t7eysrKqtaelZWlqKioWl/TvHlzff755yopKdGRI0fUqlUrTZo0Se3atTvt+1itVlmtLLrWmFTY7Jq/Zr9mLtqu7ILKIcn41qGaPLiLLm7XzOTqAACuZlq48fPzU0JCgtLT03XttddKqpxQnJ6ernHjxp3xtf7+/oqOjlZ5ebk++eQT3XzzzW6oGA3Bsp05evSLTdqeVShJigkP0IMpnTWke0t5eXEFFAA0BqbOokxNTdWIESPUp08f9e3bV7NmzVJRUZFGjRolSRo+fLiio6OVlpYmSVqxYoUOHDignj176sCBA3r00Udlt9v14IMPmvkxUA9k5pXo719t0ZfrD0qSwgJ9dd9lHXX7xW1k9fE2uToAgDuZGm6GDRumw4cPa/r06crMzFTPnj21cOFCxyTjffv2VZtPU1JSoqlTp2rXrl0KCgrS4MGD9c477ygsLMykTwCzldvsenPpbr3w7a8qKrPJyyLdlthWf/3jBdytGwAaKVPXuTED69x4jmU7cjT9i03akV05BNW7TZgev6abukWHmlwZAMDZGsQ6N0BdZeaV6MkFm/WfXw5Jkpo18dNDV3XWjb1bM68GAEC4QcNRVnFiCCr9VxWfGIK6/eK2+usVnRQa6Gt2eQCAeoJwgwZhzd6jmvTJBv3KEBQA4HcQblCvFZSU67mvt+mdn/bKMCqHoCZd1Vk3MAQFADgNwg3qrW83Z2nq5xuVmV8iSbopobWmDOnCVVAAgDMi3KDeyS4o0WNfbtaCExOG24QH6qnrumtAxwiTKwMANASEG9QbhmHo49X79eSCzcovqZC3l0V3DYzThMsvUIAfC/EBAM4O4Qb1wp6cIj382QYt23lEktQtOkRPX9+DCcMAgHNGuIGpbHZD//q/XZq5aLtKK+zy9/VS6hUXaHT/OPl4N6ib1gMA6gnCDUyzI7tQf5u/Xj/vy5UkDegQoaeu6642zQLNLQwA0KARbuB2NruhuT/u1nPfbFNZhV3BVh9Nu7qrburTWhYLl3cDAM4P4QZutetwof42/xet2XtMknTJBc319PXd1SoswOTKAACegnADt7DZDb25dLee+3qbSivsCrL6aOqQLhp2UQy9NQAApyLcwOV25xTpbx+v1+oTvTUDO0bo6Rt6KJreGgCACxBu4DJ2u6G3lu3Rs19vVUm5XU38vDVlSFfd2pfeGgCA6xBu4BI5haWaOG+d/u/XHElS/w7N9MwNPdS6KVdCAQBci3ADp1u+84jGf/izsgtKFeDrrYeHdNHtiW3orQEAuAXhBk5jsxt65fsd+se322U3pI4tgvTKbb3VMTLY7NIAAI0I4QZO8dthqJsSWuuxay5UoB//xAAA7sWZB+ftt8NQT1zbTTcmtDa7LABAI0W4QZ0xDAUAqI8IN6gThqEAAPUVZyKcszV7j+qed9cyDAUAqJcINzgnH63O0JTPNqjcZjAMBQColwg3OCsVNrvS/rtVb/y4W5J0VbcoPX9TvJpY+ScEAKhfODPhd+UdL9d9H/ysH7YfliRNSO6o+y/rKC8vFuUDANQ/hBuc0c7DhRrz79XalVOkAF9vzbg5XoO7tzS7LAAATotwg9Nasv2wxr2/VgUlFWoV6q/XR/TRha1CzS4LAIAzItygBsMw9MaPu/XUV1tkN6SEtk015/YENQ+2ml0aAAC/i3CDakorbJr62UZ9vGa/JOnmPq31xLXdZPXxNrkyAADODuEGDrnFZbrr36u1eu8xeVmkKUO6anT/WO7mDQBoUAg3kCQdLijVHW+s0NbMAgX7++jlP/fWpRc0N7ssAADOGeEGOpB7XLf/a4V25xSpRbBV796VqAtYmA8A0EARbhq5PTlFuu1fK3Qg97iiwwL0/phEtW3WxOyyAACoM8JNI7Yts0C3v7FChwtK1a55E713V6JahgaYXRYAAOeFcNNI/bI/V8PnrlRucbm6tAzRO3f2VUQQl3oDABo+wk0jtHL3UY1+a5UKSyvUMyZM/x7VV6GBvmaXBQCAUxBuGpkl2w/rf95ZrZJyu5LaNdPrI/ooiJtfAgA8CGe1RmThxkO674OfVW4zdFnnFnrltt7y92VxPgCAZyHcNBKf/bxfD3z8i2x2Q0N6tNQ/bu4pPx8vs8sCAMDpCDeNwNIdOY5gc3Of1kq7voe8vVh1GADgmQg3Hi7jaLHGvb9WNruh63tF6+nre8iLYAMA8GCMS3iw4rIK3f3OGh0rLld861A9dX13gg0AwOMRbjyUYRh6cP4v2nIoXxFBfppzRwKThwEAjQLhxkPNWbJL//nlkHy9LXr19gRWHgYANBqmh5vZs2crNjZW/v7+SkxM1MqVK8+4/axZs9SpUycFBAQoJiZGEydOVElJiZuqbRgWb8vWs19vlSQ9+qcLdVFsuMkVAQDgPqaGm3nz5ik1NVWPPPKI1q5dq/j4eKWkpCg7O7vW7d9//31NmjRJjzzyiLZs2aI33nhD8+bN08MPP+zmyuuv3TlFuv+Dn2UY0q192+i2xLZmlwQAgFuZGm5mzpypMWPGaNSoUeratavmzJmjwMBAzZ07t9btly1bpv79++vPf/6zYmNj9cc//lG33nrr7/b2NBaFpRW6++3Vyi+pUELbpnr0T13NLgkAALczLdyUlZVpzZo1Sk5OPlmMl5eSk5O1fPnyWl/Tr18/rVmzxhFmdu3apa+++kqDBw8+7fuUlpYqPz+/2sMT2e2GUuet06/ZhYoMserV23rL6sMEYgBA42PaOjc5OTmy2WyKjIys1h4ZGamtW7fW+po///nPysnJ0YABA2QYhioqKvSXv/zljMNSaWlpeuyxx5xae3300nc79M3mLPl5e2nO7QlqEeJvdkkAAJjC9AnF52Lx4sV66qmn9Morr2jt2rX69NNPtWDBAj3xxBOnfc3kyZOVl5fneGRkZLixYvdYtDlL//h2uyTpyeu6qVebpiZXBACAeUzruYmIiJC3t7eysrKqtWdlZSkqKqrW10ybNk133HGH7rrrLklS9+7dVVRUpLvvvltTpkyRl1fNrGa1WmW1Wp3/AeqJHdkFmjhvnSRpZL9Y3dwnxtyCAAAwmWk9N35+fkpISFB6erqjzW63Kz09XUlJSbW+pri4uEaA8faunFdiGIbriq2nbHZD932wToWlFUqMC9eUIV3MLgkAANOZem+p1NRUjRgxQn369FHfvn01a9YsFRUVadSoUZKk4cOHKzo6WmlpaZKkoUOHaubMmerVq5cSExO1Y8cOTZs2TUOHDnWEnMbk49UZ2nIoX6EBvpp9W2/5ejeoUUYAAFzC1HAzbNgwHT58WNOnT1dmZqZ69uyphQsXOiYZ79u3r1pPzdSpU2WxWDR16lQdOHBAzZs319ChQ/X3v//drI9gmsLSCj3/TeU8m/sv76iIIM8degMA4FxYjEY2npOfn6/Q0FDl5eUpJCTE7HLq7NmFW/XK4p2Ki2iirydcIj8fem0AAJ7rXM7fnBEboP3HivWvH3dLkh4e3IVgAwDAKTgrNkDPLNymsgq7kto1U3KXFmaXAwBAvUK4aWDW7D2mL9cflMUiTb26iywWi9klAQBQrxBuGhDDMPTkgs2SpJsSWuvCVqEmVwQAQP1DuGlAvlh/UD/vy1Wgn7ce+GMns8sBAKBeItw0ECXlNj27cJsk6Z5L23PvKAAAToNw00C88eNuHcg9rlah/hpzSTuzywEAoN4i3DQA2QUleuX7HZKkB6/sLH/fxrcaMwAAZ4tw0wDM/Ga7ispsio8J05/iW5ldDgAA9Rrhpp7bfDBf81ZnSJKmDekiLy8u/QYA4EwIN/VY1aXfhiEN6dFSfWLDzS4JAIB6j3BTj6VvydaynUfk5+OlSVd2NrscAAAaBMJNPVVWYddTX22RJI3uH6eY8ECTKwIAoGEg3NRT81bt066cIjVr4qexf2hvdjkAADQYhJt66stfDkmS7hnUXsH+viZXAwBAw0G4qYeKyyr0875jkqTkLpEmVwMAQMNCuKmHVu4+qnKboeiwALVtxlwbAADOBeGmHlq284gkqX+HZrJYWNcGAIBzQbiph5buyJEk9e8QYXIlAAA0PISbeuZoUZk2HcyXJCW1b2ZyNQAANDyEm3pm+YkhqU6RwWoR7G9yNQAANDyEm3pm6c7KIal+Hei1AQCgLgg39cyyqvk27ZlvAwBAXRBu6pH9x4q150ixvL0sSmzHTTIBAKgLwk09smxH5Xyb+NahrEoMAEAdEW7qkar5NlwCDgBA3RFu6gnDMLT0RM9NP+bbAABQZ4SbemJ7VqFyCkvl7+ul3m3DzC4HAIAGi3BTT1StSnxRbLisPt4mVwMAQMNVp3Dz/fffO7uORm8Z820AAHCKOoWbK6+8Uu3bt9eTTz6pjIwMZ9fU6FTY7Ppp11FJrG8DAMD5qlO4OXDggMaNG6f58+erXbt2SklJ0UcffaSysjJn19corN+fp8LSCoUG+KprqxCzywEAoEGrU7iJiIjQxIkTtW7dOq1YsUIXXHCB7r33XrVq1Ur333+/1q9f7+w6PVrVqsT92jeTt5fF5GoAAGjYzntCce/evTV58mSNGzdOhYWFmjt3rhISEjRw4EBt2rTJGTV6vJP3k2JICgCA81XncFNeXq758+dr8ODBatu2rb7++mu9/PLLysrK0o4dO9S2bVvddNNNzqzVIx0vs2nt3lxJUv/23CwTAIDz5VOXF91333364IMPZBiG7rjjDj377LPq1q2b4+9NmjTR888/r1atWjmtUE+1as9RldnsahXqr7iIJmaXAwBAg1encLN582a99NJLuv7662W1WmvdJiIigkvGz8KpQ1IWC/NtAAA4X3UKN+np6b+/Yx8fXXrppXXZfaNSdbPM/h0YkgIAwBnqNOcmLS1Nc+fOrdE+d+5cPfPMM+ddVGORW1ymjQfzJHE/KQAAnKVO4ea1115T586da7RfeOGFmjNnznkX1Vgs33lEhiF1bBGkyBB/s8sBAMAj1CncZGZmqmXLljXamzdvrkOHDp13UY3FUm65AACA09Up3MTExGjp0qU12pcuXcoVUudg6Yn5Nv24BBwAAKep04TiMWPGaMKECSovL9dll10mqXKS8YMPPqi//vWvTi3QUx3MPa7dOUXyskiJ7Qg3AAA4S53Czd/+9jcdOXJE9957r+N+Uv7+/nrooYc0efJkpxboqZaeuOVCj9ZhCg3wNbkaAAA8R53CjcVi0TPPPKNp06Zpy5YtCggIUMeOHU+75g1qWraTS8ABAHCF87q3VFBQkC666CJ169btvILN7NmzFRsbK39/fyUmJmrlypWn3XbQoEGyWCw1HkOGDKnz+7ubYRj68UTPTX8uAQcAwKnq1HMjSatXr9ZHH32kffv2OYamqnz66adnvZ958+YpNTVVc+bMUWJiombNmqWUlBRt27ZNLVq0qLH9p59+Wu39jhw5ovj4+AZ1H6sd2YU6XFAqq4+XerdtanY5AAB4lDr13Hz44Yfq16+ftmzZos8++0zl5eXatGmTvvvuO4WGhp7TvmbOnKkxY8Zo1KhR6tq1q+bMmaPAwMBaFwmUpPDwcEVFRTkeixYtUmBgYIMKN1XzbS6KDZe/r7fJ1QAA4FnqFG6eeuop/eMf/9CXX34pPz8/vfDCC9q6datuvvlmtWnT5qz3U1ZWpjVr1ig5OflkQV5eSk5O1vLly89qH2+88YZuueUWNWnScG46ufTEfJt+zLcBAMDp6hRudu7c6Zjj4ufnp6KiIlksFk2cOFH//Oc/z3o/OTk5stlsioyMrNYeGRmpzMzM3339ypUrtXHjRt11112n3aa0tFT5+fnVHmaqsNn1U9VkYubbAADgdHUKN02bNlVBQYEkKTo6Whs3bpQk5ebmqri42HnV/Y433nhD3bt3V9++fU+7TVpamkJDQx2PmJgYt9VXmw0H8lRQWqEQfx91iz63ITwAAPD76hRuLrnkEi1atEiSdNNNN2n8+PEaM2aMbr31Vl1++eVnvZ+IiAh5e3srKyurWntWVpaioqLO+NqioiJ9+OGHuvPOO8+43eTJk5WXl+d4ZGRknHV9rrDhQOWNMi+KDZe3l8XUWgAA8ER1ulrq5ZdfVklJiSRpypQp8vX11bJly3TDDTdo6tSpZ70fPz8/JSQkKD09Xddee60kyW63Kz09XePGjTvjaz/++GOVlpbq9ttvP+N2Vqu1Xq2/k3+8XJLUPLj+1AQAgCc553BTUVGh//znP0pJSZFUOQF40qRJdS4gNTVVI0aMUJ8+fdS3b1/NmjVLRUVFGjVqlCRp+PDhio6OVlpaWrXXvfHGG7r22mvVrFnDmpRbUFohSQqy1vkqfAAAcAbnfIb18fHRX/7yF23ZssUpBQwbNkyHDx/W9OnTlZmZqZ49e2rhwoWOScb79u2Tl1f10bNt27bpxx9/1DfffOOUGtypoOREuPEn3AAA4Ap1OsP27dtX69atU9u2bZ1SxLhx4047DLV48eIabZ06dZJhGE55b3crPBFugv25nxQAAK5Qp3Bz7733KjU1VRkZGUpISKixxkyPHj2cUpwnKiipnHMTzLAUAAAuUacz7C233CJJuv/++x1tFotFhmHIYrHIZrM5pzoPVFjKsBQAAK5UpzPs7t27nV1Ho1HgGJYi3AAA4Ap1OsM6a65NY+SYUMywFAAALlGnM+zbb799xr8PHz68TsU0BlXDUkwoBgDANeoUbsaPH1/teXl5uYqLi+Xn56fAwEDCzWkYhnFKuKHnBgAAV6jT7ReOHTtW7VFYWKht27ZpwIAB+uCDD5xdo8c4Xm6TzV55CTvDUgAAuEadwk1tOnbsqKeffrpGrw5OqlrjxssiBfp5m1wNAACeyWnhRqpcvfjgwYPO3KVHyT9lMrHFwk0zAQBwhTqNjXzxxRfVnhuGoUOHDunll19W//79nVKYJ2IyMQAArlencFN1B+8qFotFzZs312WXXaYZM2Y4oy6PVMgaNwAAuFydzrJ2u93ZdTQKVbdeYDIxAACu49Q5NzizAi4DBwDA5eoUbm644QY988wzNdqfffZZ3XTTTeddlKdyrE7MnBsAAFymTuHmhx9+0ODBg2u0X3XVVfrhhx/OuyhPVcitFwAAcLk6hZvCwkL5+fnVaPf19VV+fv55F+WpCksr59yEMCwFAIDL1CncdO/eXfPmzavR/uGHH6pr167nXZSn4qaZAAC4Xp3OstOmTdP111+vnTt36rLLLpMkpaen64MPPtDHH3/s1AI9CROKAQBwvTqdZYcOHarPP/9cTz31lObPn6+AgAD16NFD3377rS699FJn1+gxCplQDACAy9W5C2HIkCEaMmSIM2vxeKxzAwCA69Vpzs2qVau0YsWKGu0rVqzQ6tWrz7soT1V1+wUmFAMA4Dp1Cjdjx45VRkZGjfYDBw5o7Nix512Upzq5zg3hBgAAV6lTuNm8ebN69+5do71Xr17avHnzeRflqVjnBgAA16tTuLFarcrKyqrRfujQIfn4cOKujd1uqLCMu4IDAOBqdQo3f/zjHzV58mTl5eU52nJzc/Xwww/riiuucFpxnqSorEKGUfk7l4IDAOA6dTrLPv/887rkkkvUtm1b9erVS5K0bt06RUZG6p133nFqgZ6iajKxr7dFVh/uVwoAgKvUKdxER0frl19+0Xvvvaf169crICBAo0aN0q233ipfX4ZcanPqfBuLxWJyNQAAeK46j480adJEAwYMUJs2bVRWViZJ+u9//ytJ+tOf/uSc6jxIPldKAQDgFnU60+7atUvXXXedNmzYIIvFIsMwqvVG2Gw2pxXoKaqGpYKt9GwBAOBKdZr8MX78eMXFxSk7O1uBgYHauHGjlixZoj59+mjx4sVOLtEzOFYnpucGAACXqtOZdvny5fruu+8UEREhLy8veXt7a8CAAUpLS9P999+vn3/+2dl1NnhVc26CWeMGAACXqlPPjc1mU3BwsCQpIiJCBw8elCS1bdtW27Ztc151HqSQO4IDAOAWdTrTduvWTevXr1dcXJwSExP17LPPys/PT//85z/Vrl07Z9foEZhQDACAe9TpTDt16lQVFRVJkh5//HFdffXVGjhwoJo1a6Z58+Y5tUBP4RiWYnViAABcqk7hJiUlxfF7hw4dtHXrVh09elRNmzZlDZfTKCw9MaGYOTcAALiU08604eHhztqVRyooYc4NAADuwH0A3IQJxQAAuAfhxk0cE4pZxA8AAJci3LhJ4YlF/Oi5AQDAtQg3blI1LMWEYgAAXItw4yZMKAYAwD0IN25gsxsqLqu8mSjr3AAA4FqEGzeoGpKSpCZWbxMrAQDA8xFu3KDqjuB+Pl6y+hBuAABwJcKNG1T13IQw3wYAAJcj3LhBQQlXSgEA4C6mh5vZs2crNjZW/v7+SkxM1MqVK8+4fW5ursaOHauWLVvKarXqggsu0FdffeWmauuGm2YCAOA+pnYlzJs3T6mpqZozZ44SExM1a9YspaSkaNu2bWrRokWN7cvKynTFFVeoRYsWmj9/vqKjo7V3716FhYW5v/hzUMAaNwAAuI2pZ9uZM2dqzJgxGjVqlCRpzpw5WrBggebOnatJkybV2H7u3Lk6evSoli1bJl/fyl6Q2NhYd5ZcJ1UTioOYcwMAgMuZNixVVlamNWvWKDk5+WQxXl5KTk7W8uXLa33NF198oaSkJI0dO1aRkZHq1q2bnnrqKdlsttO+T2lpqfLz86s93K2QBfwAAHAb08JNTk6ObDabIiMjq7VHRkYqMzOz1tfs2rVL8+fPl81m01dffaVp06ZpxowZevLJJ0/7PmlpaQoNDXU8YmJinPo5zoZjdWKGpQAAcDnTJxSfC7vdrhYtWuif//ynEhISNGzYME2ZMkVz5sw57WsmT56svLw8xyMjI8ONFVdy3FeKnhsAAFzOtLNtRESEvL29lZWVVa09KytLUVFRtb6mZcuW8vX1lbf3yYXwunTposzMTJWVlcnPz6/Ga6xWq6xWq3OLP0cFXC0FAIDbmNZz4+fnp4SEBKWnpzva7Ha70tPTlZSUVOtr+vfvrx07dshutzvatm/frpYtW9YabOoLx4RihqUAAHA5U4elUlNT9frrr+vf//63tmzZonvuuUdFRUWOq6eGDx+uyZMnO7a/5557dPToUY0fP17bt2/XggUL9NRTT2ns2LFmfYSzUjUsxYRiAABcz9Sz7bBhw3T48GFNnz5dmZmZ6tmzpxYuXOiYZLxv3z55eZ3MXzExMfr66681ceJE9ejRQ9HR0Ro/frweeughsz7CWSHcAADgPhbDMAyzi3Cn/Px8hYaGKi8vTyEhIW55zz88v1i7c4r00f8kqW9cuFveEwAAT3Iu5+8GdbVUQ1XAOjcAALgN4cYNmFAMAID7EG5crKzCrtKKyqu7QrgUHAAAlyPcuFjRicnEktTE6n2GLQEAgDMQblysar5NgK+3fLz5ugEAcDXOti5WUFo534bJxAAAuAfhxsWq7gjOfaUAAHAPwo2LcUdwAADci3DjYidXJ+ZKKQAA3IFw42KscQMAgHsRblysgPtKAQDgVoQbF2NCMQAA7kW4cTEmFAMA4F6EGxdjQjEAAO5FuHGxAoalAABwK8KNi3G1FAAA7kW4cbFCrpYCAMCtCDcu5phQTLgBAMAtCDcuxoRiAADci3DjYo51bphzAwCAWxBuXKik3KYym10SV0sBAOAuhBsXqhqSkqQgP8INAADuQLhxoVOHpLy8LCZXAwBA40C4cSGulAIAwP0INy5UUMoCfgAAuBvhxoW49QIAAO5HuHGhwhLWuAEAwN0INy7kWMCPYSkAANyGcONC3DQTAAD3I9y4UAE3zQQAwO0INy7EhGIAANyPcONCTCgGAMD9CDcuxIRiAADcj3DjQo4JxQxLAQDgNoQbF+L2CwAAuB/hxoWqhqW4FBwAAPch3LgQPTcAALgf4cZFDMM4OaGYq6UAAHAbwo2LHC+3yWY3JDEsBQCAOxFuXKRqjRsvixTo521yNQAANB6EGxcpOGUyscViMbkaAAAaD8KNixSwOjEAAKYg3LhIIVdKAQBgCsKNixSWnlidmMnEAAC4FeHGRfLpuQEAwBT1ItzMnj1bsbGx8vf3V2JiolauXHnabd966y1ZLJZqD39/fzdWe3aqhqWCmHMDAIBbmR5u5s2bp9TUVD3yyCNau3at4uPjlZKSouzs7NO+JiQkRIcOHXI89u7d68aKz07VhGKGpQAAcC/Tw83MmTM1ZswYjRo1Sl27dtWcOXMUGBiouXPnnvY1FotFUVFRjkdkZKQbKz47VXNuQhiWAgDArUwNN2VlZVqzZo2Sk5MdbV5eXkpOTtby5ctP+7rCwkK1bdtWMTExuuaaa7Rp06bTbltaWqr8/PxqD3fgppkAAJjD1HCTk5Mjm81Wo+clMjJSmZmZtb6mU6dOmjt3rv73f/9X7777rux2u/r166f9+/fXun1aWppCQ0Mdj5iYGKd/jtrkO+bcEG4AAHAn04elzlVSUpKGDx+unj176tJLL9Wnn36q5s2b67XXXqt1+8mTJysvL8/xyMjIcEudhSziBwCAKUztVoiIiJC3t7eysrKqtWdlZSkqKuqs9uHr66tevXppx44dtf7darXKarWed63nimEpAADMYWrPjZ+fnxISEpSenu5os9vtSk9PV1JS0lntw2azacOGDWrZsqWryqyTghImFAMAYAbTz7ypqakaMWKE+vTpo759+2rWrFkqKirSqFGjJEnDhw9XdHS00tLSJEmPP/64Lr74YnXo0EG5ubl67rnntHfvXt11111mfowaCplzAwCAKUw/8w4bNkyHDx/W9OnTlZmZqZ49e2rhwoWOScb79u2Tl9fJDqZjx45pzJgxyszMVNOmTZWQkKBly5apa9euZn2EWrHODQAA5rAYhmGYXYQ75efnKzQ0VHl5eQoJCXHJe9jthtpP+UqGIa2akqzmwe6f8wMAgCc5l/N3g7taqiEoLrepKjJybykAANyLcOMCVZOJfbwssvrwFQMA4E6ceV2g8JQ7glssFpOrAQCgcSHcuEBBKVdKAQBgFsKNC1RdKRVsZXViAADcjXDjAqxxAwCAeQg3LlA1oTiYNW4AAHA7wo0LVN1XisvAAQBwP8KNCxQwLAUAgGkINy7gmFDsz4RiAADcjXDjAoWllXNuuK8UAADuR7hxgYIS5twAAGAWwo0LMKEYAADzEG5cwDGhmEX8AABwO8KNC1Stc8OcGwAA3I9w4wIMSwEAYB7CjQsUMqEYAADTEG6czGY3VFRmk8Q6NwAAmIFw42RVQ1KS1MTqbWIlAAA0ToQbJ6uaTOzn4yWrD+EGAAB3I9w4WVXPTQjzbQAAMAXhxskKHWvcEG4AADAD4cbJuCM4AADmItw4WUHVGjesTgwAgCkIN05WSM8NAACmItw4WdXVUizgBwCAOQg3Tua49QITigEAMAXhxsmYUAwAgLkIN05W4LivFBOKAQAwA+HGyQpLK+fcsM4NAADmINw4WQF3BAcAwFSEGydzTCgm3AAAYArCjZOdvP0Cc24AADAD4cbJ8hmWAgDAVIQbJ2NCMQAA5iLcOFG5za6Scrskem4AADAL4caJqubbSPTcAABgFsKNE1VdKRXg6y0fb75aAADMwBnYifK5aSYAAKYj3DhRIfeVAgDAdIQbJ+KO4AAAmI9w40TcNBMAAPMRbpyooLRqdWJ6bgAAMAvhxokKmFAMAIDpCDdOxIRiAADMVy/CzezZsxUbGyt/f38lJiZq5cqVZ/W6Dz/8UBaLRddee61rCzxLTCgGAMB8poebefPmKTU1VY888ojWrl2r+Ph4paSkKDs7+4yv27Nnjx544AENHDjQTZX+PiYUAwBgPtPDzcyZMzVmzBiNGjVKXbt21Zw5cxQYGKi5c+ee9jU2m0233XabHnvsMbVr186N1Z5ZAcNSAACYztRwU1ZWpjVr1ig5OdnR5uXlpeTkZC1fvvy0r3v88cfVokUL3Xnnne4o86xxR3AAAMxn6lk4JydHNptNkZGR1dojIyO1devWWl/z448/6o033tC6devO6j1KS0tVWlrqeJ6fn1/nen/PyWEpwg0AAGYxfVjqXBQUFOiOO+7Q66+/roiIiLN6TVpamkJDQx2PmJgYl9XnmFBMuAEAwDSmnoUjIiLk7e2trKysau1ZWVmKioqqsf3OnTu1Z88eDR061NFmt9slST4+Ptq2bZvat29f7TWTJ09Wamqq43l+fr7LAg4TigEAMJ+p4cbPz08JCQlKT093XM5tt9uVnp6ucePG1di+c+fO2rBhQ7W2qVOnqqCgQC+88EKtocVqtcpqtbqk/t9yrHPDnBsAAExj+lk4NTVVI0aMUJ8+fdS3b1/NmjVLRUVFGjVqlCRp+PDhio6OVlpamvz9/dWtW7dqrw8LC5OkGu3uVlphU5mtsheJq6UAADCP6WfhYcOG6fDhw5o+fboyMzPVs2dPLVy40DHJeN++ffLyqv9Tg6qGpCQpyM/0rxUAgEbLYhiGYXYR7pSfn6/Q0FDl5eUpJCTEafvdk1OkQc8vVpDVRxsfS3HafgEAwLmdv+t/l0gDUcB8GwAA6gXCjZOUVtgUZPVRSADhBgAAM3EmdpI+seHa+FiKGtkoHwAA9Q49N05msVjMLgEAgEaNcAMAADwK4QYAAHgUwg0AAPAohBsAAOBRCDcAAMCjEG4AAIBHIdwAAACPQrgBAAAehXADAAA8CuEGAAB4FMINAADwKIQbAADgUQg3AADAo/iYXYC7GYYhScrPzze5EgAAcLaqzttV5/EzaXThpqCgQJIUExNjciUAAOBcFRQUKDQ09IzbWIyziUAexG636+DBgwoODpbFYnHqvvPz8xUTE6OMjAyFhIQ4dd84exyH+oHjUD9wHOoHjsP5MwxDBQUFatWqlby8zjyrptH13Hh5eal169YufY+QkBD+8dYDHIf6geNQP3Ac6geOw/n5vR6bKkwoBgAAHoVwAwAAPArhxomsVqseeeQRWa1Ws0tp1DgO9QPHoX7gONQPHAf3anQTigEAgGej5wYAAHgUwg0AAPAohBsAAOBRCDcAAMCjEG6cZPbs2YqNjZW/v78SExO1cuVKs0vyeD/88IOGDh2qVq1ayWKx6PPPP6/2d8MwNH36dLVs2VIBAQFKTk7Wr7/+ak6xHiotLU0XXXSRgoOD1aJFC1177bXatm1btW1KSko0duxYNWvWTEFBQbrhhhuUlZVlUsWe6dVXX1WPHj0cC8QlJSXpv//9r+PvHANzPP3007JYLJowYYKjjWPhHoQbJ5g3b55SU1P1yCOPaO3atYqPj1dKSoqys7PNLs2jFRUVKT4+XrNnz671788++6xefPFFzZkzRytWrFCTJk2UkpKikpISN1fquZYsWaKxY8fqp59+0qJFi1ReXq4//vGPKioqcmwzceJEffnll/r444+1ZMkSHTx4UNdff72JVXue1q1b6+mnn9aaNWu0evVqXXbZZbrmmmu0adMmSRwDM6xatUqvvfaaevToUa2dY+EmBs5b3759jbFjxzqe22w2o1WrVkZaWpqJVTUukozPPvvM8dxutxtRUVHGc88952jLzc01rFar8cEHH5hQYeOQnZ1tSDKWLFliGEbld+7r62t8/PHHjm22bNliSDKWL19uVpmNQtOmTY1//etfHAMTFBQUGB07djQWLVpkXHrppcb48eMNw+B/D+5Ez815Kisr05o1a5ScnOxo8/LyUnJyspYvX25iZY3b7t27lZmZWe24hIaGKjExkePiQnl5eZKk8PBwSdKaNWtUXl5e7Th07txZbdq04Ti4iM1m04cffqiioiIlJSVxDEwwduxYDRkypNp3LvG/B3dqdDfOdLacnBzZbDZFRkZWa4+MjNTWrVtNqgqZmZmSVOtxqfobnMtut2vChAnq37+/unXrJqnyOPj5+SksLKzathwH59uwYYOSkpJUUlKioKAgffbZZ+ratavWrVvHMXCjDz/8UGvXrtWqVatq/I3/PbgP4QaAU4wdO1YbN27Ujz/+aHYpjVKnTp20bt065eXlaf78+RoxYoSWLFlidlmNSkZGhsaPH69FixbJ39/f7HIaNYalzlNERIS8vb1rzHbPyspSVFSUSVWh6rvnuLjHuHHj9J///Efff/+9Wrdu7WiPiopSWVmZcnNzq23PcXA+Pz8/dejQQQkJCUpLS1N8fLxeeOEFjoEbrVmzRtnZ2erdu7d8fHzk4+OjJUuW6MUXX5SPj48iIyM5Fm5CuDlPfn5+SkhIUHp6uqPNbrcrPT1dSUlJJlbWuMXFxSkqKqraccnPz9eKFSs4Lk5kGIbGjRunzz77TN99953i4uKq/T0hIUG+vr7VjsO2bdu0b98+joOL2e12lZaWcgzc6PLLL9eGDRu0bt06x6NPnz667bbbHL9zLNyDYSknSE1N1YgRI9SnTx/17dtXs2bNUlFRkUaNGmV2aR6tsLBQO3bscDzfvXu31q1bp/DwcLVp00YTJkzQk08+qY4dOyouLk7Tpk1Tq1atdO2115pXtIcZO3as3n//ff3v//6vgoODHfMGQkNDFRAQoNDQUN15551KTU1VeHi4QkJCdN999ykpKUkXX3yxydV7jsmTJ+uqq65SmzZtVFBQoPfff1+LFy/W119/zTFwo+DgYMd8sypNmjRRs2bNHO0cCzcx+3ItT/HSSy8Zbdq0Mfz8/Iy+ffsaP/30k9klebzvv//ekFTjMWLECMMwKi8HnzZtmhEZGWlYrVbj8ssvN7Zt22Zu0R6mtu9fkvHmm286tjl+/Lhx7733Gk2bNjUCAwON6667zjh06JB5RXug0aNHG23btjX8/PyM5s2bG5dffrnxzTffOP7OMTDPqZeCGwbHwl0shmEYJuUqAAAAp2PODQAA8CiEGwAA4FEINwAAwKMQbgAAgEch3AAAAI9CuAEAAB6FcAMAADwK4QZAo7d48WJZLJYa9/wB0DARbgAAgEch3AAAAI9CuAFgOrvdrrS0NMXFxSkgIEDx8fGaP3++pJNDRgsWLFCPHj3k7++viy++WBs3bqy2j08++UQXXnihrFarYmNjNWPGjGp/Ly0t1UMPPaSYmBhZrVZ16NBBb7zxRrVt1qxZoz59+igwMFD9+vXTtm3bXPvBAbgE4QaA6dLS0vT2229rzpw52rRpkyZOnKjbb79dS5YscWzzt7/9TTNmzNCqVavUvHlzDR06VOXl5ZIqQ8nNN9+sW265RRs2bNCjjz6qadOm6a233nK8fvjw4frggw/04osvasuWLXrttdcUFBRUrY4pU6ZoxowZWr16tXx8fDR69Gi3fH4AzsWNMwGYqrS0VOHh4fr222+VlJTkaL/rrrtUXFysu+++W3/4wx/04YcfatiwYZKko0ePqnXr1nrrrbd0880367bbbtPhw4f1zTffOF7/4IMPasGCBdq0aZO2b9+uTp06adGiRUpOTq5Rw+LFi/WHP/xB3377rS6//HJJ0ldffaUhQ4bo+PHj8vf3d/G3AMCZ6LkBYKodO3aouLhYV1xxhYKCghyPt99+Wzt37nRsd2rwCQ8PV6dOnbRlyxZJ0pYtW9S/f/9q++3fv79+/fVX2Ww2rVu3Tt7e3rr00kvPWEuPHj0cv7ds2VKSlJ2dfd6fEYB7+ZhdAIDGrbCwUJK0YMECRUdHV/ub1WqtFnDqKiAg4Ky28/X1dfxusVgkVc4HAtCw0HMDwFRdu3aV1WrVvn371KFDh2qPmJgYx3Y//fST4/djx45p+/bt6tKliySpS5cuWrp0abX9Ll26VBdccIG8vb3VvXt32e32anN4AHguem4AmCo4OFgPPPCAJk6cKLvdrgEDBigvL09Lly5VSEiI2rZtK0l6/PHH1axZM0VGRmrKlCmKiIjQtddeK0n661//qosuukhPPPGEhg0bpuXLl+vll1/WK6+8IkmKjY3ViBEjNHr0aL344ouKj4/X3r17lZ2drZtvvtmsjw7ARQg3AEz3xBNPqHnz5kpLS9OuXbsUFham3r176+GHH3YMCz399NMaP368fv31V/Xs2VNffvml/Pz8JEm9e/fWRx99pOnTp+uJJ55Qy5Yt9fjjj2vkyJGO93j11Vf18MMP695779WRI0fUpk0bPfzww2Z8XAAuxtVSAOq1qiuZjh07prCwMLPLAdAAMOcGAAB4FMINAADwKAxLAQAAj0LPDQAA8CiEGwAA4FEINwAAwKMQbgAAgEch3AAAAI9CuAEAAB6FcAMAADwK4QYAAHgUwg0AAPAo/w8o9IJ1GfCaDgAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkgAAAHHCAYAAABEEKc/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABd7klEQVR4nO3deVzUdf4H8NcczAzHzHDJKQiKIaaAQhJqh4lh7ZputmphHhm2lZWy7hbbT/OoxbL8sZqFP9PSLt1a23a1LMW0Q0KDSEPEEwHllGM4B2bm+/sDGRsORQS+A7yej8c8ZD7f6/1lsnn5+X6+n69EEAQBRERERGQmFbsAIiIiImvDgERERETUAgMSERERUQsMSEREREQtMCARERERtcCARERERNQCAxIRERFRCwxIRERERC0wIBERERG1wIBERH1KTk4OJBIJ3nvvvRve9uDBg5BIJDh48GCX10VEvQsDEhEREVELDEhERERELTAgERH1QjU1NWKXQNSnMSARUZdasWIFJBIJTp06hdmzZ0Or1WLAgAFYtmwZBEFAXl4epk6dCo1GAw8PD7zxxhut9lFcXIwFCxbA3d0dKpUKISEh2LZtW6v1KioqMG/ePGi1Wjg6OmLu3LmoqKhos66TJ0/ioYcegrOzM1QqFcLDw/Gf//ynU+dYVVWFxYsXw8/PD0qlEm5ubpg0aRLS09Mt1ktNTcX9998PJycn2NvbIzg4GP/4xz8s1jlw4ADuuOMO2Nvbw9HREVOnTkVWVpbFOs2/0xMnTuCRRx6Bk5MTxo8fb17+wQcfICwsDLa2tnB2dsasWbOQl5fXqXMjoiYMSETULWbOnAmTyYQ1a9YgIiICL7/8MhITEzFp0iR4e3vj1VdfRUBAAJYuXYpvv/3WvF1dXR3uvvtuvP/++4iJicHatWuh1Woxb948i3AhCAKmTp2K999/H7Nnz8bLL7+M/Px8zJ07t1UtmZmZuP3225GVlYUXXngBb7zxBuzt7TFt2jR89tlnN3xuf/rTn/D2229j+vTpeOutt7B06VLY2tpaBJt9+/bhzjvvxIkTJ/Dcc8/hjTfewIQJE7B7927zOvv370d0dDSKi4uxYsUKxMXF4fDhwxg3bhxycnJaHfePf/wjamtr8fe//x2xsbEAgFdeeQVz5szB0KFDsW7dOixevBjJycm488472w2LRNQBAhFRF3rppZcEAMLChQvNbQaDQRg4cKAgkUiENWvWmNvLy8sFW1tbYe7cuea2xMREAYDwwQcfmNsaGhqEyMhIwcHBQdDpdIIgCMK///1vAYDw2muvWRznjjvuEAAI7777rrl94sSJwsiRI4X6+npzm8lkEsaOHSsMHTrU3PbNN98IAIRvvvnmmueo1WqFp59+ut3lBoNB8Pf3FwYNGiSUl5dbLDOZTOafQ0NDBTc3N+Hy5cvmtl9++UWQSqXCnDlzzG3Nv9OHH37YYl85OTmCTCYTXnnlFYv248ePC3K5vFU7EXUce5CIqFs8/vjj5p9lMhnCw8MhCAIWLFhgbnd0dERgYCDOnTtnbvviiy/g4eGBhx9+2NxmY2ODZ599FtXV1Th06JB5PblcjieffNLiOM8884xFHWVlZThw4ABmzJiBqqoqlJaWorS0FJcvX0Z0dDROnz6Nixcv3tC5OTo6IjU1FZcuXWpz+c8//4zz589j8eLFcHR0tFgmkUgAAAUFBcjIyMC8efPg7OxsXh4cHIxJkybhiy++aLXfP/3pTxbvd+3aBZPJhBkzZpjPq7S0FB4eHhg6dCi++eabGzovIrpKLnYBRNQ3+fr6WrzXarVQqVRwdXVt1X758mXz+wsXLmDo0KGQSi3//RYUFGRe3vynp6cnHBwcLNYLDAy0eH/mzBkIgoBly5Zh2bJlbdZaXFwMb2/vDp/ba6+9hrlz58LHxwdhYWG4//77MWfOHAwePBgAcPbsWQDAiBEj2t1H83m0rBdoOtevvvoKNTU1sLe3N7f7+/tbrHf69GkIgoChQ4e2eQwbG5sOnxMRWWJAIqJuIZPJOtQGNI0n6i4mkwkAsHTpUkRHR7e5TkBAwA3tc8aMGbjjjjvw2Wef4euvv8batWvx6quvYteuXbjvvvtuuub22NraWrw3mUyQSCT48ssv2/zdtgyPRNRxDEhEZFUGDRqEY8eOwWQyWfQinTx50ry8+c/k5GRUV1dbBIHs7GyL/TX36tjY2CAqKqrL6vT09MRTTz2Fp556CsXFxRg9ejReeeUV3HfffRgyZAgA4Ndff233mM3n0bJeoOlcXV1dLXqP2jJkyBAIggB/f3/ccsstN3lGRPRbHINERFbl/vvvR2FhIXbu3GluMxgM2LBhAxwcHHDXXXeZ1zMYDHj77bfN6xmNRmzYsMFif25ubrj77ruxadMmFBQUtDpeSUnJDdVnNBpRWVnZ6hheXl7Q6/UAgNGjR8Pf3x+JiYmt7iRr7i3z9PREaGgotm3bZrHOr7/+iq+//hr333//dWt58MEHIZPJsHLlyla9cIIgWFy6JKIbwx4kIrIqCxcuxKZNmzBv3jykpaXBz88Pn376KX744QckJiZCrVYDAKZMmYJx48bhhRdeQE5ODoYPH45du3a1Ci8AsHHjRowfPx4jR45EbGwsBg8ejKKiIqSkpCA/Px+//PJLh+urqqrCwIED8dBDDyEkJAQODg7Yv38/jh49ap7TSSqV4u2338aUKVMQGhqK+fPnw9PTEydPnkRmZia++uorAMDatWtx3333ITIyEgsWLEBdXR02bNgArVaLFStWXLeWIUOG4OWXX0Z8fDxycnIwbdo0qNVqnD9/Hp999hkWLlyIpUuXdvjciOg3xLuBjoj6ouZb0ktKSiza586dK9jb27da/6677hJuvfVWi7aioiJh/vz5gqurq6BQKISRI0da3Lbf7PLly8Kjjz4qaDQaQavVCo8++qjw888/t7rNXxAE4ezZs8KcOXMEDw8PwcbGRvD29hZ+//vfC59++ql5nY7c5q/X64W//OUvQkhIiKBWqwV7e3shJCREeOutt1qt+/333wuTJk0yrxccHCxs2LDBYp39+/cL48aNE2xtbQWNRiNMmTJFOHHihMU67f1Om/3rX/8Sxo8fL9jb2wv29vbCsGHDhKefflrIzs5u9zyI6NokgtCNoyOJiIiIeiGOQSIiIiJqgQGJiIiIqAUGJCIiIqIWGJCIiIiIWmBAIiIiImqBAYmIiIioBU4U2UkmkwmXLl2CWq02P52biIiIrJsgCKiqqoKXl1erh2L/FgNSJ126dAk+Pj5il0FERESdkJeXh4EDB7a7nAGpk5ofd5CXlweNRiNyNURERNQROp0OPj4+5u/x9jAgdVLzZTWNRsOARERE1Mtcb3gMB2kTERERtcCARERERNSC6AFp48aN8PPzg0qlQkREBI4cOdLuuu+99x4kEonFS6VSWazTcnnza+3ateZ1/Pz8Wi1fs2ZNt50jERER9S6ijkHauXMn4uLikJSUhIiICCQmJiI6OhrZ2dlwc3NrcxuNRoPs7Gzz+5bXEAsKCizef/nll1iwYAGmT59u0b5q1SrExsaa319vsFZnmEwmNDQ0dPl+eyuFQnHNWyqJiIishagBad26dYiNjcX8+fMBAElJSdizZw+2bt2KF154oc1tJBIJPDw82t1ny2Wff/45JkyYgMGDB1u0q9Xqa+7nZjU0NOD8+fMwmUzddozeRiqVwt/fHwqFQuxSiIiIrkm0gNTQ0IC0tDTEx8eb26RSKaKiopCSktLudtXV1Rg0aBBMJhNGjx6Nv//977j11lvbXLeoqAh79uzBtm3bWi1bs2YNVq9eDV9fXzzyyCNYsmQJ5PL2fx16vR56vd78XqfTtbuuIAgoKCiATCaDj48Pe01wdWLNgoIC+Pr6cnJNIiKyaqIFpNLSUhiNRri7u1u0u7u74+TJk21uExgYiK1btyI4OBiVlZV4/fXXMXbsWGRmZrY52dO2bdugVqvx4IMPWrQ/++yzGD16NJydnXH48GHEx8ejoKAA69ata7fehIQErFy5skPnZjAYUFtbCy8vL9jZ2XVom/5gwIABuHTpEgwGA2xsbMQuh4iIqF29ah6kyMhIREZGmt+PHTsWQUFB2LRpE1avXt1q/a1btyImJqbVQO64uDjzz8HBwVAoFHjiiSeQkJAApVLZ5rHj4+MttmueaKotRqMRAHgpqYXm34fRaGRAIiIiqyZaQHJ1dYVMJkNRUZFFe1FRUYfHBtnY2GDUqFE4c+ZMq2XfffcdsrOzsXPnzuvuJyIiAgaDATk5OQgMDGxzHaVS2W54ag8vI1ni74OIiHoL0QbHKBQKhIWFITk52dxmMpmQnJxs0Ut0LUajEcePH4enp2erZVu2bEFYWBhCQkKuu5+MjAxIpdJ275wjIiKi/kXU0cNxcXHYvHkztm3bhqysLDz55JOoqakx39U2Z84ci0Hcq1atwtdff41z584hPT0ds2fPxoULF/D4449b7Fen0+GTTz5p1Q4AKSkpSExMxC+//IJz587hww8/xJIlSzB79mw4OTl17wn3cX5+fkhMTBS7DCIiopsm6hikmTNnoqSkBMuXL0dhYSFCQ0Oxd+9e88Dt3NxcizvAysvLERsbi8LCQjg5OSEsLAyHDx/G8OHDLfa7Y8cOCIKAhx9+uNUxlUolduzYgRUrVkCv18Pf3x9LliyxGF9ERERE/ZtEEARB7CJ6I51OB61Wi8rKylYPq62vr8f58+fh7+/faoB4X+bn54fFixdj8eLFbS7vr78XIupbDEYTymsbIZEAGpUNFHJO5dKbXOv7+7d61V1s1H3+7//+DytWrEB+fr5Fr93UqVPh4uKCF198EXFxcfjxxx9RU1ODoKAgJCQkICoqSsSqiZrmHTtdXA1BAG5xd+iTNwMIgoDcslqkXSjHL3kVUClkCPZ2RPBALQY62d7QOesNRhzPr8RPF8rxU045zpZUw9VBAR9nO/g628HHyQ6+Lk1/uqmVkEo7//s0mQSU1uhRWFmPSxX1KKysQ4GuHkWV9XBQyTHMQ4MgTw0CPdRwUHb868hoavp9nCzQIauwCnlltdDa2mCAWgk3tRJuGlXTn2olnOwU1zwHQRBQ22BEtd6AqnoDavQGlNc2oLS6ASVVepRW/+ZV1YDSaj3Kahvw264FlY0UapUNNCo5NLY2LX6Ww85GDluFFCobGVRyGVQKGVTypve2iqY2G7kENXojdPWNqKo3oKrFn7q6pj8bjCaoVXJobRXQ2tq0+7JVyCCRABJceQQXcOV9U2PzMrlUChuZBHIZQ15LDEg9QBAE1DUaRTm2rY2sQ//z/OMf/4hnnnkG33zzDSZOnAgAKCsrw969e/HFF1+guroa999/P1555RUolUps374dU6ZMQXZ2Nnx9fbv7NIgsCIKA7KIq7DlWgD3HCnCutAYAMMJbg0dvH4QHQrxhq5D1eE0VtY0ortKjuKoexTq9+ef6RhO8tCoMdLaFt6MdvJ1s4aFRQdbOF3d9oxG/XqxE2oVypF0oR3puOUqr235skYu9AiMHahE80BEhA7UYOVALN/XVHtqymgakXSjHTxfKkJZTjmP5lWgwWs7wf760BkdzylvtWyGXwsfJFj7OdnBXqzoUlqr1BhRU1KGgsh5FunoYTB27SDHIxQ7DPNQI8tRgmIcGwz01GOhkC119I7IKqpBdqMPJwipkFVbhVGFVh/+fKpdKzMFJY2uDGr0B1XoDqusNqNI3BaIOlmhBIoE5JNU3mlDfqEdJlf7aG1kxqQSwkUmhkEuhuPKnjawpPCnksqYQJZVcaZNCLpOYw1XzexuptCl4SYCm+IXfhLQr4ew3bR3xzMShcHW4sTvIuwovsXXSjVxiq20wYPjyr0Sp88SqaNgpOpaDp02bBhcXF2zZsgVAU6/SypUrkZeX1+Zs4CNGjMCf/vQnLFq0CAAvsVH3O11Uhd3HCrDneAHOFFeb25svcTQYmr741So5HgobiJiIQQhwc+jyOvLKarE/qwip58pQqKtHSVXTl2PL4HEtcqkEHloVBjpdDU21egPScsvx68VKNBot/9eskEkxwluDUb5OqGs04lh+BU4WVLUZQDy1KgzzUONCWS3OldS0Wu7qoEDYICeED3LGcC8NLtc0IK+stulVXovcslpcqqiHsTPJoQWpBHBTq+ChVcFTq4Kn1hbuGiXKahtwsqAKJwt1KNK1HSwUcqn5M21JZSNFoLsawzw08HO1h66+8Uoobfo8iqv0KKvp+LMwpRLAQSlv6v2xtYGrgwIDHJRwVSvh6qCAq4MSrg5KDFA3/els3zSvW7X+au+Orr6xxc9Nf9Y1GlFvfplQ12BEvcGIugYj9Iam941GE+yVcqhVza+m3ifNld6o5vcKuRS6ukZU1hlQWddoful+83NlXaNo/yjvagf+fBcGD+jav8O8xEY3LCYmBrGxsXjrrbegVCrx4YcfYtasWZBKpaiursaKFSuwZ88eFBQUwGAwoK6uDrm5uWKXTX3cmeLqpp6i45dwqug3oUgmxV2BA/D7YE9MDHJHg8GET37Kw4epucgtq8W7P+Tg3R9yEDnYBY9GDsKk4e6w6eRlBJNJQEZ+BZKzirD/RDGyi6raXdfRzubK5Z2myzwD1EoobWQoqKhDfnkdLlbUoaCyDo1GAfnlTW1AWav9NIeY5tetXlqobCx7xeobjcgq0OH4xUr8kleJY/kVOFNSjYLKehRU1pvXC3BzQPiV/dzm54xBLnbX7Vk2GE0oqKxH7pXg1NHeEVuFDJ5aW3MgclMrr3v55nK1HtmFVThR0NRLdLJQh1NF1eZw5ONs23Q5zkONYZ4aDPNQY5CLfbs9cM0aDCaUVl/pydPVo1pvaAohSjkcVHI4/ObPjva2t9R8SctaCYIAQQCE5p+BK++b2o0mAY1GExoMJjQYTWg0CmgwmNBoNEF/5c8GgwkGkwkNBgEGkwkGY9M2jcam941X3huMJnOvWvNxmn6+WsOVojpcv6OdeBMuMyD1AFsbGU6sihbt2B01ZcoUCIKAPXv24LbbbsN3332H//3f/wUALF26FPv27cPrr7+OgIAA2Nra4qGHHkJDQ8f/hUZ0PXqDEScu6fBzbgV+zqvAz7nlVwJEExuZBHfdMgC/uxKKNKrffDEpgSfuGoLYOwbj29Ml+ODHXBw4WYSUc5eRcu4y3NRKzLrNB+MCXC3+de6gkrf5RVvbYMD3p0uxP6sIB06WoLT6akCQSSUIH+SECcPc4OdiDzeN8moYkl//75zRJKC4qh4XrwSm5qAkl0owepAjwnyd4eN8/bFFKhsZRvk6YZSvE3Bl+rhqvQG/XqzEqaIqeDvaYrSvE5zsb/xLRi6TwsfZDj7O3f+4JBcHJcYGKDE2wNXcZjCakF9eBxcHBdSqzgUQhVwKL0dbeDnadlWpvY5EIsHV/4z63vi87sSA1AMkEkmHL3OJSaVS4cEHH8SHH36IM2fOIDAwEKNHjwYA/PDDD5g3bx7+8Ic/AGh6aHBOTo6I1VJvJwgCLlbUNYWh3Ar8nFeOzIu6Vpep5FIJ7hjqit8Fe2HScPfr/mtdKpXg7kA33B3ohosVdfg4NRc7juaiuEqP9QfOYP2B1jPv2ytk5gG1apUN5FIJMvIqoP/N5R21Uo67AgcgKsgddwcOuKl/2cqkEnhqbeGptUV4p/fSNgelHLcPdsHtg126eM89Sy6Tws/VXuwyqB+z/m9t6lExMTH4/e9/j8zMTMyePdvcPnToUOzatQtTpkyBRCLBsmXLYDJ1fLwFUTNdfSPe+e48dh7NbXPsibO9AqN8HDHK1xGjfJ0QPFDb6R4Eb0dbLI0OxLMTh+KrzEL886c85JfXoaq+Ebp6g/kSTk2DETUNRhRUWm4/0MkWUUHumDTcHbf5OfN2bqJ+hAGJLNxzzz1wdnZGdnY2HnnkEXP7unXr8Nhjj2Hs2LFwdXXF888/D51OJ2Kl1NvUNhjw3uEcbDp0DpV1jQCaeoeGe2muBCInjPJ1hK/z9cfH3CiFXIopIV6YEuJl0a43GK/cSm15W3VtgxG3emn77LQBRHR9vIutkzhR5I3j76V/qm804qPUXLx18Iz5VvWhbg5YHHULJga5tRp4TETUnXgXGxGJqtFowqdp+ViffNp8R5Wvsx2WTBqKB0K8r3sHEhGRmBiQiKhLGU0C/vPLRSTuP40Ll2sBNM3L8+zEoXgobGCnb7UnIupJDEhE1GmCIKC4So9TRVU4VVSN00VVOHK+zDyztauDAk/dHYBHInx5KY2IehUGJCLqkMraRmReappf51RxUxg6VVRtHnD9WxqVHE/cNQTzxvrB/gaesUVEZC34f65uxPHvlvj76J1MJgHvHc7Bq3tPWswL1EwqAfxc7DHU3QG3uKtxi7sad94ywKpnFyYiuh4GpG4gkzVdSmhoaICtbf+dwbWl5lm3m38/ZP0uVtRh6T9/Qcq5ywCa5gUK8tTglithaKibGoMH2PPyGRH1OQxI3UAul8POzg4lJSWwsbFp80Gv/Y3JZEJJSQns7Owgl/M/O2snCAL+lX4RK/+TiSq9AbY2Mrz4uyDERPhyXiAi6hf4TdUNJBIJPD09cf78eVy4cEHscqyGVCqFry+/YK3d5Wo9/vbZcXyVWQQAGO3riHUzQvnYByLqVxiQuolCocDQoUP5MNffUCgU7E2zcvtOFCF+1zGUVjfARibB4qhb8MSdg6/7NHYior6GAakbSaVSzhhNvUJVfSNW7z6Bf/6UDwAIdFdj3cwQ3OqlFbkyIiJxMCAR9XNHzpch7p8ZyC+vg0QCLLxzMOIm3QKlnAOviaj/YkAisiK6+kZoOvnk+s747y+XsGRnBgwmAT7Otnjjj6EY4+/cY8cnIrJWHFhAZCX2/lqIUav24dEtqdDVt558savtOJKLZ3f8DINJwO+DPfHlc3cyHBERXcGARGQF9AYjXt5zAkaTgO9Ol+Khtw/jYkVdtx3vne/O4YVdxyEIwCMRvlg/axQcOOM1EZEZAxKRFfjgx1zkl9dhgFoJN7USp4qqMW3jD/j1YmWXHkcQBKzbdwov78kCADxx12C8Mm0EpFJOvUBE9FsMSEQi09U34s0DpwEAf550Cz57ehwC3dUoqdJjxqYUHDhZ1CXHMZkErNp9AuuTm471l+hAvDB5GOelIiJqAwMSkcg2HTqL8tpGBLg54KGwgfB2tMUnT0ZifIArahuMeHzbT3j/x5ubcNRoEvDCrmN494ccAMCKKcPx9IQAhiMionYwIBGJqLCyHlu+Pw8A+Gt0oHlCRo3KBu/Ovw1/DBsIkwAs+/ev+PsXWTCZbvyBvw0GE579+Gf886d8SCXA638Mwbxx/l16HkREfQ0DEpGI/pF8CvWNJoQNcsKk4e4Wy2xkUrz2UDD+POkWAMD/fXsOiz5OR32jscP7r2swYuH7P2HP8QLYyCTY+MhoPBQ2sEvPgYioL2JAIhLJmeIq7DyaBwCIv6/tsUASiQTPTByK/50ZAhuZBF8cL0TMO6koq2n/ETYGowmVdY3IvVyLuVuP4GB2CVQ2Urwz9zbcN9Kz286HiKgv4X29RCJ5bW82TAIwabg7wv2uPf/QH0YNhKfWFgu3/4S0C+WYsuF7BLg5oEZvQE2DselPvQHVegP0BpPFtmqlHFvn34bbrnMMIiK6igGJSAQ/5ZTh6xNFkEqA5ycHdmib2we7YNdTYzHv3aPIL6+77jxJNjIJ/Fzs8b8zQzHCm89UIyK6EQxIRD1MEASs+fIkAGBGuA8C3NQd3jbATY3/LhqPfVlFkABwUMphr5TDXilr+lMhh4NSDjuljM9SIyK6CaKPQdq4cSP8/PygUqkQERGBI0eOtLvue++9B4lEYvFSqVQW68ybN6/VOpMnT7ZYp6ysDDExMdBoNHB0dMSCBQtQXV3dLedH1NL+rGL8dKEcKhspFkfdcsPbO9krMCPcB38M98F9Iz1x5y0DEDbIGcM8NPBxtoOTvYLhiIjoJonag7Rz507ExcUhKSkJERERSExMRHR0NLKzs+Hm5tbmNhqNBtnZ2eb3bQ1snTx5Mt59913ze6VSabE8JiYGBQUF2LdvHxobGzF//nwsXLgQH330URedGVHbDEYTXt3b1Hv02Dh/eGhV19mCiIjEIGpAWrduHWJjYzF//nwAQFJSEvbs2YOtW7fihRdeaHMbiUQCDw+Pa+5XqVS2u05WVhb27t2Lo0ePIjw8HACwYcMG3H///Xj99dfh5eV1E2dEdG3/Ss/HmeJqONrZ4E93DxG7HCIiaodol9gaGhqQlpaGqKioq8VIpYiKikJKSkq721VXV2PQoEHw8fHB1KlTkZmZ2WqdgwcPws3NDYGBgXjyySdx+fJl87KUlBQ4OjqawxEAREVFQSqVIjU1td3j6vV66HQ6ixfRjahrMGLdvlMAgEUTAqBR2YhcERERtUe0gFRaWgqj0Qh3d8vJ8dzd3VFYWNjmNoGBgdi6dSs+//xzfPDBBzCZTBg7dizy8/PN60yePBnbt29HcnIyXn31VRw6dAj33XcfjMamyfUKCwtbXb6Ty+VwdnZu97gAkJCQAK1Wa375+Ph09tSpn3r38HkU6fTwdrTFo5GDxC6HiIiuoVfdxRYZGYnIyEjz+7FjxyIoKAibNm3C6tWrAQCzZs0yLx85ciSCg4MxZMgQHDx4EBMnTuz0sePj4xEXF2d+r9PpGJKow8prGvD2wbMAgKXRt3AQNRGRlROtB8nV1RUymQxFRZZPKi8qKrruGKNmNjY2GDVqFM6cOdPuOoMHD4arq6t5HQ8PDxQXF1usYzAYUFZWds3jKpVKaDQaixdRR2385gyq6g0I8tRgaoi32OUQEdF1iBaQFAoFwsLCkJycbG4zmUxITk626CW6FqPRiOPHj8PTs/3HJ+Tn5+Py5cvmdSIjI1FRUYG0tDTzOgcOHIDJZEJEREQnz4aofRl5FdiecgEA8MJ9wyCVtr7zkoiIrIuo8yDFxcVh8+bN2LZtG7KysvDkk0+ipqbGfFfbnDlzEB8fb15/1apV+Prrr3Hu3Dmkp6dj9uzZuHDhAh5//HEATQO4//KXv+DHH39ETk4OkpOTMXXqVAQEBCA6OhoAEBQUhMmTJyM2NhZHjhzBDz/8gEWLFmHWrFm8g4263LH8Cjy6JRUNRhPuGeaGO4e6il0SERF1gKhjkGbOnImSkhIsX74chYWFCA0Nxd69e80Dt3NzcyGVXs1w5eXliI2NRWFhIZycnBAWFobDhw9j+PDhAACZTIZjx45h27ZtqKiogJeXF+69916sXr3aYi6kDz/8EIsWLcLEiRMhlUoxffp0rF+/vmdPnvq8Xy9WYvY7qaiqN2CMnzM2PDyqzXm7iIjI+kgEQRDELqI30ul00Gq1qKys5HgkauXXi5WIeScVlXWNCB/khPceGwMHZa+6J4KIqE/q6Pe36I8aIeprTlzSYfaWpnA02tcR786/jeGIiKiXYUAi6kInC3WIeedHVNQ2ItTHEe89NgZqTghJRNTrMCARdZHswio8sjkV5bWNCBmoxfYFYzhbNhFRL8WARNQFThdV4ZHNP6KspgEjvbXYviCC4YiIqBdjQCK6SWeKq/Hw5lRcrmnArV4avL9gDLS2DEdERL0ZAxLRTThbUo2HN/+I0mo9hntq8OHjEXC0U4hdFhER3SQGJKJOqtEbMGfLEZRU6THMQ81wRETUhzAgEXXSpm/P4WJFHXycbfHh4xFwsmc4IiLqKxiQiDqhsLIe//ftWQDA3+4LgouD8jpbEBFRb8KARNQJr3+djfpGE27zc8LkER5il0NERF2MAYnoBv16sRL/Ss8HALz4u+F8vhoRUR/EgER0AwRBwCt7siAIwAMhXgj1cRS7JCIi6gYMSEQ3IDmrGCnnLkMhl+KvkwPFLoeIiLoJAxJRBzUaTfj7l1kAgAXj/THQyU7kioiIqLswIBF10EepuThXUgMXewWeunuI2OUQEVE3YkAi6oDKukYk7j8FAFg86Rao+Zw1IqI+jQGJqAPe+uYMymsbEeDmgIdv8xG7HCIi6mYMSETXkVdWi3d/yAEAvHh/EOQy/rUhIurr+H96outYs/ckGowmjA9wxd2BA8Quh4iIegADEtE1pF0ox55jBZBIgL/dH8RJIYmI+gkGJKJ2CIKAl/ecAADMCPPBcC+NyBUREVFPYUAiasee4wX4ObcCdgoZ/nzvLWKXQ0REPYgBiagN9Y1GrPnyJADgiTuHwE2jErkiIiLqSXKxCyDqSbr6Rnx5vABV9QYYTAIMRhMajE1/GkwCGgwmGEwm5JXVIb+8Du4aJWLv9Be7bCIi6mEMSNRvVNY1YuamFJwsrOrwNkvvDYSdgn9NiIj6G/6fn/qF+kYjHt92FCcLq+DqoMC4AFfYyKSwkUkgl0ohl0mgkDX9KZc2tXs52uIPo7zFLp2IiETAgER9nsFowqKP0nE0pxxqlRzvL4hAkCfvSCMiovZxkDb1aYIg4IVdx7E/qxhKuRRb5t7GcERERNfFgER92pq9J/FpWj5kUgk2PjIaY/ydxS6JiIh6AQYk6rP+79uz2HToHAAg4cGRiBruLnJFRETUWzAgUZ/0aVo+/v5F0zxGL9w3DDPCfUSuiIiIehPRA9LGjRvh5+cHlUqFiIgIHDlypN1133vvPUgkEouXSnV1Ar/GxkY8//zzGDlyJOzt7eHl5YU5c+bg0qVLFvvx8/NrtZ81a9Z02zlSz0rOKsLz/zoGAIi9wx9P3DlY5IqIiKi3ETUg7dy5E3FxcXjppZeQnp6OkJAQREdHo7i4uN1tNBoNCgoKzK8LFy6Yl9XW1iI9PR3Lli1Deno6du3ahezsbDzwwAOt9rNq1SqL/TzzzDPdco7Us47mlOGpD9NhNAl4cLQ34u/jA2aJiOjGiXqb/7p16xAbG4v58+cDAJKSkrBnzx5s3boVL7zwQpvbSCQSeHh4tLlMq9Vi3759Fm1vvvkmxowZg9zcXPj6+prb1Wp1u/uh3ulkoQ4L3jsKvcGEe4a54dXpwZBKGY6IiOjGidaD1NDQgLS0NERFRV0tRipFVFQUUlJS2t2uuroagwYNgo+PD6ZOnYrMzMxrHqeyshISiQSOjo4W7WvWrIGLiwtGjRqFtWvXwmAw3NT5kLjOllRj7tYj0NUbEDbICRsfGQ0bmehXkImIqJcSrQeptLQURqMR7u6Wdxa5u7vj5MmTbW4TGBiIrVu3Ijg4GJWVlXj99dcxduxYZGZmYuDAga3Wr6+vx/PPP4+HH34YGs3VuW+effZZjB49Gs7Ozjh8+DDi4+NRUFCAdevWtVuvXq+HXq83v9fpdDd6ytRN9p8owpKdGajSGxDorsbWubfBViETuywiIurFetVM2pGRkYiMjDS/Hzt2LIKCgrBp0yasXr3aYt3GxkbMmDEDgiDg7bfftlgWFxdn/jk4OBgKhQJPPPEEEhISoFQq2zx2QkICVq5c2YVnQzfLZBKw4cAZ/O/+UwCA2/yc8FZMGLR2NiJXRkREvZ1o1yBcXV0hk8lQVFRk0V5UVNThsUE2NjYYNWoUzpw5Y9HeHI4uXLiAffv2WfQetSUiIgIGgwE5OTntrhMfH4/KykrzKy8vr0M1Uveoqm/EEx+kmcPRnMhB+PDx2zFA3XbAJSIiuhGiBSSFQoGwsDAkJyeb20wmE5KTky16ia7FaDTi+PHj8PT0NLc1h6PTp09j//79cHFxue5+MjIyIJVK4ebm1u46SqUSGo3G4kXiOFtSjWkbf8C+E0VQyKR47aFgrJo6Ago5xxwREVHXEPUSW1xcHObOnYvw8HCMGTMGiYmJqKmpMd/VNmfOHHh7eyMhIQFA0635t99+OwICAlBRUYG1a9fiwoULePzxxwE0haOHHnoI6enp2L17N4xGIwoLCwEAzs7OUCgUSElJQWpqKiZMmAC1Wo2UlBQsWbIEs2fPhpOTkzi/COqw34438tCokPRoGEJ9HMUui4iI+hhRA9LMmTNRUlKC5cuXo7CwEKGhodi7d6954HZubi6k0qu9AuXl5YiNjUVhYSGcnJwQFhaGw4cPY/jw4QCAixcv4j//+Q8AIDQ01OJY33zzDe6++24olUrs2LEDK1asgF6vh7+/P5YsWWIxLomsT8vxRmP8nLExZjQvqRERUbeQCIIgiF1Eb6TT6aDValFZWcnLbd2sqr4Rcf/8BftONI1XmxM5CP/zu+G8pEZERDeso9/fveouNup/quob8eBbh3G6uBoKmRQv/2EEn6tGRETdjgGJrNpnP1/E6eJqDFArsXlOOMcbERFRj+A1CrJqn6blAwCevGsIwxEREfUYBiSyWtmFVTiWXwm5VIKpoV5il0NERP0IAxJZrU/TmibjnBjkBhcH3q1GREQ9hwGJrFKj0YTPfr4IAHgojIOyiYioZzEgkVU6lF2C0uoGuDoocHfgALHLISKifoYBiaxS8+DsaaHesJHxP1MiIupZ/OYhq1NW04Dkk02TQj4UPlDkaoiIqD9iQCKr83nGRTQaBYz01mKYB2cpJyKinseARFbnk5+aLq89FMbeIyIiEgcDElmVzEuVOFGgg0ImxQMhnPuIiIjEwYBEVuVfaU239kcNd4OTvULkaoiIqL9iQCKr0WAw4d8ZzXMf8fIaERGJhwGJrMY32cUoq2nAALUSdw7l3EdERCQeBiSyGs2Dsx8c5Q055z4iIiIR8VuIrEJJlR7fZBcD4OU1IiISHwMSWYXPMy7CaBIQ4uOIoe5qscshIqJ+jgGJRCcIgvnRIuw9IiIia8CARKLLvKTDycIqKORSPBDMuY+IiEh8DEgkuk9+ygMA3DvcHVo7G5GrISIiYkAikekNRnz+yyUAwB/DfUSuhoiIqAkDEokqOasYFbWNcNcoMT7AVexyiIiIADAgkciaB2c/OHogZFKJyNUQERE1YUAi0RTr6nHoVAkA3r1GRETWhQGJRPPZz01zH432dcSQAQ5il0NERGTGgESiMJkEfHLl8hoHZxMRkbVhQCJRfHw0F2eKq2GvkOF3wZ5il0NERGSBAYl6XJGuHmu+OAkAWBodCI2Kcx8REZF1YUCiHvfS55mo0hsQ4uOIOZF+YpdDRETUCgMS9aivMguxN7MQcqkEax4cyVv7iYjIKjEgUY+pqm/ES59nAgAW3jkYQZ4akSsiIiJqm+gBaePGjfDz84NKpUJERASOHDnS7rrvvfceJBKJxUulUlmsIwgCli9fDk9PT9ja2iIqKgqnT5+2WKesrAwxMTHQaDRwdHTEggULUF1d3S3nR1e9tjcbhbp6+LnY4dmJQ8Uuh4iIqF2iBqSdO3ciLi4OL730EtLT0xESEoLo6GgUFxe3u41Go0FBQYH5deHCBYvlr732GtavX4+kpCSkpqbC3t4e0dHRqK+vN68TExODzMxM7Nu3D7t378a3336LhQsXdtt5EpB2oQwfpDZ9Vn//w0iobGQiV0RERHQNgojGjBkjPP300+b3RqNR8PLyEhISEtpc/9133xW0Wm27+zOZTIKHh4ewdu1ac1tFRYWgVCqFjz/+WBAEQThx4oQAQDh69Kh5nS+//FKQSCTCxYsXO1x7ZWWlAECorKzs8Db9lb7RKExad1AY9Pxu4c//zBC7HCIi6sc6+v0tWg9SQ0MD0tLSEBUVZW6TSqWIiopCSkpKu9tVV1dj0KBB8PHxwdSpU5GZmWledv78eRQWFlrsU6vVIiIiwrzPlJQUODo6Ijw83LxOVFQUpFIpUlNTu/IU6YqkQ2dxqqgaLvYKvHh/kNjlEBERXZdoAam0tBRGoxHu7u4W7e7u7igsLGxzm8DAQGzduhWff/45PvjgA5hMJowdOxb5+U0zMjdvd619FhYWws3NzWK5XC6Hs7Nzu8cFAL1eD51OZ/Gi6ztbUo03D5wBACyfMhxO9gqRKyIiIro+0Qdp34jIyEjMmTMHoaGhuOuuu7Br1y4MGDAAmzZt6vZjJyQkQKvVml8+Pnw8xvWYTALidx1Hg9GEu24ZgAdCvMQuiYiIqENEC0iurq6QyWQoKiqyaC8qKoKHh0eH9mFjY4NRo0bhzJmmHorm7a61Tw8Pj1aDwA0GA8rKyq553Pj4eFRWVppfeXl5HaqxP9v5Ux6OnC+DrY0ML08bAYmEcx4REVHvIFpAUigUCAsLQ3JysrnNZDIhOTkZkZGRHdqH0WjE8ePH4enZ9Cwvf39/eHh4WOxTp9MhNTXVvM/IyEhUVFQgLS3NvM6BAwdgMpkQERHR7rGUSiU0Go3Fi9pXrKvH37/IAgD8+d5b4ONsJ3JFREREHScX8+BxcXGYO3cuwsPDMWbMGCQmJqKmpgbz588HAMyZMwfe3t5ISEgAAKxatQq33347AgICUFFRgbVr1+LChQt4/PHHAQASiQSLFy/Gyy+/jKFDh8Lf3x/Lli2Dl5cXpk2bBgAICgrC5MmTERsbi6SkJDQ2NmLRokWYNWsWvLx4CairrPzvCVTVGzDSW4t5Y/3ELoeIiOiGiBqQZs6ciZKSEixfvhyFhYUIDQ3F3r17zYOsc3NzIZVe7eQqLy9HbGwsCgsL4eTkhLCwMBw+fBjDhw83r/PXv/4VNTU1WLhwISoqKjB+/Hjs3bvXYkLJDz/8EIsWLcLEiRMhlUoxffp0rF+/vudOvI/bf6IIe44XQCaVIOHBkZDLetVQNyIiIkgEQRDELqI30ul00Gq1qKys5OW238gq0OHhzT+iorYRT9w5GPG8rZ+IiKxIR7+/+U976jKni6oQ804qKmobEeLjiMVRt4hdEhERUacwIFGXOFdSjUfeSUVZTQNGeGuw/bExsFXwcSJERNQ7MSDRTbtwuQaPbE5FSZUewzzUeP+xCGhtbcQui4iIqNMYkOim5JfX4pHNqSjU1WOomwM+fDyCs2UTEVGvx4BEnVZQWYdHNqfiYkUdBrva48PYCLg4KMUui4iI6KYxIFGnFOvq8cjmVOSW1cLX2Q4fxd4ON7Xq+hsSERH1AgxIdMNKq/V45J1UnC+tgbejLT6KjYCHluGIiIj6DgYkuiHlNQ2Y/U4qzhRXw1Orwsext2OgEx8jQkREfQsDEnVYZV0jHt2aipOFVRigVuKj2Nvh68JwREREfQ8DEnXYq3tP4teLOrjYK/DR4xHwd7UXuyQiIqJuwYBEHWIwmvDl8QIAwBszQjDUXS1yRURERN2HAYk65KcL5SivbYSjnQ3GB7iKXQ4REVG3YkCiDvk6swgAMHGYO+Qy/mdDRER9G7/p6LoEQcDXJwoBAPfe6i5yNURERN2PAYmuK6ugCvnldVDZSHHn0AFil0NERNTtGJDoupp7j+4YOgC2CpnI1RAREXU/BiS6rq+ujD+KvtVD5EqIiIh6BgMSXVNeWS2yCnSQSoCJw9zELoeIiKhHMCDRNX19oqn3aIy/M5zsFSJXQ0RE1DMYkOiavs68cvfacF5eIyKi/oMBidp1uVqPozllAHh7PxER9S8MSNSu5JPFMAnArV4aDHTiQ2mJiKj/YECidjXPns3La0RE1N8wIFGbahsM+O50CQBeXiMiov6HAYna9O2pUugNJvg422KYh1rscoiIiHpUpwNSRUUF3nnnHcTHx6OsrGkgb3p6Oi5evNhlxZF4mu9eix7uAYlEInI1REREPUvemY2OHTuGqKgoaLVa5OTkIDY2Fs7Ozti1axdyc3Oxffv2rq6TelCj0YTkk8UAgHs5ezYREfVDnepBiouLw7x583D69GmoVCpz+/33349vv/22y4ojcRw9X4bKukY42ysQNshJ7HKIiIh6XKcC0tGjR/HEE0+0avf29kZhYeFNF0Xiap49OyrIDTIpL68REVH/06mApFQqodPpWrWfOnUKAwYMuOmiSDyCIFwdf8TLa0RE1E91KiA98MADWLVqFRobGwEAEokEubm5eP755zF9+vQuLZB61q8XdbhUWQ87hQzjAlzFLoeIiEgUnQpIb7zxBqqrq+Hm5oa6ujrcddddCAgIgFqtxiuvvNLVNVIP+vpEU+/RXbcMgMpGJnI1RERE4uhUQNJqtdi3bx92796N9evXY9GiRfjiiy9w6NAh2Nvb39C+Nm7cCD8/P6hUKkRERODIkSMd2m7Hjh2QSCSYNm2aRbtEImnztXbtWvM6fn5+rZavWbPmhuruq8yzZ3NySCIi6sdu+Db/xsZG2NraIiMjA+PGjcO4ceM6ffCdO3ciLi4OSUlJiIiIQGJiIqKjo5GdnQ03N7d2t8vJycHSpUtxxx13tFpWUFBg8f7LL7/EggULWl36W7VqFWJjY83v1WpOhphTWoPsoirIpRLcE8iARERE/dcN9yDZ2NjA19cXRqPxpg++bt06xMbGYv78+Rg+fDiSkpJgZ2eHrVu3truN0WhETEwMVq5cicGDB7da7uHhYfH6/PPPMWHChFbrqtVqi/VutOerL2q+vHb7YBdo7WxEroaIiEg8nbrE9uKLL+Jvf/ubeQbtzmhoaEBaWhqioqKuFiOVIioqCikpKe1ut2rVKri5uWHBggXXPUZRURH27NnT5rpr1qyBi4sLRo0ahbVr18JgMFxzX3q9HjqdzuLV1/DyGhERUZNOzaT95ptv4syZM/Dy8sKgQYNa9b6kp6dfdx+lpaUwGo1wd7f8MnZ3d8fJkyfb3Ob777/Hli1bkJGR0aE6t23bBrVajQcffNCi/dlnn8Xo0aPh7OyMw4cPIz4+HgUFBVi3bl27+0pISMDKlSs7dNzeqKRKj7TccgDApOEMSERE1L91KiC1HBjdE6qqqvDoo49i8+bNcHXt2O3nW7duRUxMjMVs30DTTODNgoODoVAo8MQTTyAhIQFKpbLNfcXHx1tsp9Pp4OPj04kzsU7JWUUQBCBkoBaeWluxyyEiIhJVpwLSSy+9dNMHdnV1hUwmQ1FRkUV7UVERPDxaT1B49uxZ5OTkYMqUKeY2k8kEAJDL5cjOzsaQIUPMy7777jtkZ2dj586d160lIiICBoMBOTk5CAwMbHMdpVLZbnjqC5pnz+az14iIiDoZkJqlpaUhKysLAHDrrbdi1KhRHd5WoVAgLCwMycnJ5h4pk8mE5ORkLFq0qNX6w4YNw/Hjxy3a/ud//gdVVVX4xz/+0ao3Z8uWLQgLC0NISMh1a8nIyIBUKr3mnXN9WbXegO9PlwIA7uXlNSIios4FpOLiYsyaNQsHDx6Eo6MjAKCiogITJkzAjh07Ovy4kbi4OMydOxfh4eEYM2YMEhMTUVNTg/nz5wMA5syZA29vbyQkJEClUmHEiBEW2zcfu2W7TqfDJ598gjfeeKPVMVNSUpCamooJEyZArVYjJSUFS5YswezZs+Hk1D8fzPrtqRI0GE3wd7VHgJuD2OUQERGJrlMB6ZlnnkFVVRUyMzMRFBQEADhx4gTmzp2LZ599Fh9//HGH9jNz5kyUlJRg+fLlKCwsRGhoKPbu3WseuJ2bmwup9MZvtNuxYwcEQcDDDz/caplSqcSOHTuwYsUK6PV6+Pv7Y8mSJRbji/qbI+eb7ka865YBkEj4cFoiIiKJIAjCjW6k1Wqxf/9+3HbbbRbtR44cwb333ouKioquqs9q6XQ6aLVaVFZWQqPRiF3OTZm68Qf8kleBf8wKxdRQb7HLISIi6jYd/f7u1DxIJpMJNjatJxK0sbExD5ym3kFvMCLrUtOcTqN8+uclRiIiopY6FZDuuecePPfcc7h06ZK57eLFi1iyZAkmTpzYZcVR9ztxSYcGownO9gr4OPP2fiIiIqCTAenNN9+ETqeDn58fhgwZgiFDhsDf3x86nQ4bNmzo6hqpG2XkVQBomv+I44+IiIiadGqQto+PD9LT07F//37zrNdBQUEWjw2h3uGXKwEplJfXiIiIzDo9D5JEIsGkSZMwadKkrqyHelhzD1Kor6OodRAREVmTTl1ie/bZZ7F+/fpW7W+++SYWL158szVRDymvaUDO5VoATZfYiIiIqEmnAtK//vUvjBs3rlX72LFj8emnn950UdQzMvIrAAD+rvZwtFOIWwwREZEV6VRAunz5MrTa1j0OGo0GpaWlN10U9YyM3AoAQKiPo6h1EBERWZtOBaSAgADs3bu3VfuXX36JwYMH33RR1DN+udKDxIBERERkqVODtOPi4rBo0SKUlJTgnnvuAQAkJyfj9ddfxz/+8Y8uLZC6hyAIv7mDzVHUWoiIiKxNpwLSY489Br1ej1deeQWrV68GAPj7+yMpKQlz5szp0gKpe1y4XIvy2kYoZFIEefbuR6UQERF1tU5dYqurq8PcuXORn5+PoqIiHDt2DIsWLTI/ZJasX/Pt/cO9NFDIO/WfARERUZ/VqW/GqVOnYvv27QCanr8WFRWFdevWYdq0aXj77be7tEDqHhm8vEZERNSuTgWk9PR03HHHHQCATz/9FO7u7rhw4QK2b9/e5vxIZH2aA9IoThBJRETUSqcCUm1tLdRqNQDg66+/xoMPPgipVIrbb78dFy5c6NICqevpDUacuKQDwB4kIiKitnT6Nv9///vfyMvLw1dffYV7770XAFBcXAyNhgN+rV1WQRUajCY42dnA19lO7HKIiIisTqcC0vLly7F06VL4+fkhIiICkZGRAJp6k0aNGtWlBVLXy8gtBwCE+DhCIpGIXA0REZH16dRt/g899BDGjx+PgoIChISEmNsnTpyIP/zhD11WHHWPX/IrAfDyGhERUXs6FZAAwMPDAx4eHhZtY8aMuemCqPvxDjYiIqJr4wQ4/UxFbQPOl9YAYEAiIiJqDwNSP9Pce+Tvag9HO4W4xRAREVkpBqR+pjkghQzUilsIERGRFWNA6mf4gFoiIqLrY0DqRwRBuDpA29dJ3GKIiIisGANSP5JbVovy2kYoZFIEearFLoeIiMhqMSD1I829R0FeGijlMnGLISIismIMSP2I+QG1HH9ERER0TQxI/QgniCQiIuoYBqR+osFgQuYlHQAGJCIiouthQOonsgp0aDCY4Ghng0EudmKXQ0REZNUYkPqJqxNEOkIikYhbDBERkZVjQOonOEEkERFRx4kekDZu3Ag/Pz+oVCpERETgyJEjHdpux44dkEgkmDZtmkX7vHnzIJFILF6TJ0+2WKesrAwxMTHQaDRwdHTEggULUF1d3VWnZJWuThDpKGodREREvYGoAWnnzp2Ii4vDSy+9hPT0dISEhCA6OhrFxcXX3C4nJwdLly7FHXfc0ebyyZMno6CgwPz6+OOPLZbHxMQgMzMT+/btw+7du/Htt99i4cKFXXZe1qaythHnSmsAAKEDHcUthoiIqBcQNSCtW7cOsbGxmD9/PoYPH46kpCTY2dlh69at7W5jNBoRExODlStXYvDgwW2uo1Qq4eHhYX45OV19rEZWVhb27t2Ld955BxERERg/fjw2bNiAHTt24NKlS11+jtYgI78CAODnYgcne4W4xRAREfUCogWkhoYGpKWlISoq6moxUimioqKQkpLS7narVq2Cm5sbFixY0O46Bw8ehJubGwIDA/Hkk0/i8uXL5mUpKSlwdHREeHi4uS0qKgpSqRSpqant7lOv10On01m8eouM3AoAQAjHHxEREXWIaAGptLQURqMR7u7uFu3u7u4oLCxsc5vvv/8eW7ZswebNm9vd7+TJk7F9+3YkJyfj1VdfxaFDh3DffffBaDQCAAoLC+Hm5maxjVwuh7Ozc7vHBYCEhARotVrzy8fHp6OnKrpfrvQgcYA2ERFRx8jFLqCjqqqq8Oijj2Lz5s1wdXVtd71Zs2aZfx45ciSCg4MxZMgQHDx4EBMnTuz08ePj4xEXF2d+r9PpekVIEgSBM2gTERHdINECkqurK2QyGYqKiizai4qK4OHh0Wr9s2fPIicnB1OmTDG3mUwmAE09QNnZ2RgyZEir7QYPHgxXV1ecOXMGEydOhIeHR6tB4AaDAWVlZW0et5lSqYRSqbyhc7QGeWV1KKtpgEImxXAvjdjlEBER9QqiXWJTKBQICwtDcnKyuc1kMiE5ORmRkZGt1h82bBiOHz+OjIwM8+uBBx7AhAkTkJGR0W5vTn5+Pi5fvgxPT08AQGRkJCoqKpCWlmZe58CBAzCZTIiIiOjisxTfz3nlAIAgLw2UcpnI1RAREfUOol5ii4uLw9y5cxEeHo4xY8YgMTERNTU1mD9/PgBgzpw58Pb2RkJCAlQqFUaMGGGxvaOjIwCY26urq7Fy5UpMnz4dHh4eOHv2LP76178iICAA0dHRAICgoCBMnjwZsbGxSEpKQmNjIxYtWoRZs2bBy8ur506+h/ySVwkACB2oFbkSIiKi3kPUgDRz5kyUlJRg+fLlKCwsRGhoKPbu3WseuJ2bmwuptOOdXDKZDMeOHcO2bdtQUVEBLy8v3HvvvVi9erXF5bEPP/wQixYtwsSJEyGVSjF9+nSsX7++y8/PGmRc6UHiBJFEREQdJxEEQRC7iN5Ip9NBq9WisrISGo11ju1pMJgwYsVXaDCY8M3Su+Hvai92SURERKLq6Pe36I8aoe6TXViFBoMJWlsb+LnYiV0OERFRr8GA1Ic1z6AdPFALiUQibjFERES9CANSH3bsyvxHIXz+GhER0Q1hQOrDjuU33cEWzDvYiIiIbggDUh9V22DA6eIqAHwGGxER0Y1iQOqjMi/pYBIAd40S7hqV2OUQERH1KgxIfdQvV8YfBXP8ERER0Q1jQOqjmscfhXD8ERER0Q1jQOqjjplv8XcUtQ4iIqLeiAGpD6qsbUTO5VoAwEhv9iARERHdKAakPujYxQoAgK+zHZzsFeIWQ0RE1AsxIPVBnP+IiIjo5jAg9UHN4484gzYREVHnMCD1QexBIiIiujkMSH1McVU9CirrIZUAIzhAm4iIqFMYkPqYY3lNvUcBbg6wV8pFroaIiKh3YkDqYzj/ERER0c1jQOpjfuH4IyIiopvGgNSHCILAHiQiIqIuwIDUh+SX16G8thE2MgmCPNVil0NERNRrMSD1Ic239w/z0EApl4lcDRERUe/FgNSHXL28xvFHREREN4MBqQ/5hTNoExERdQkGpD7CZBLw60UdACDYhz1IREREN4MBqY84V1qNar0BKhspAgY4iF0OERFRr8aA1Ef8cmUG7RFeWshl/FiJiIhuBr9J+wjOf0RERNR1GJD6iGMXm3qQQjj+iIiI6KYxIPUBjUYTTly6MkCbPUhEREQ3jQGpD8gurILeYIJGJYefi53Y5RAREfV6DEh9wDHzA2odIZFIRK6GiIio92NA6gM4gzYREVHXEj0gbdy4EX5+flCpVIiIiMCRI0c6tN2OHTsgkUgwbdo0c1tjYyOef/55jBw5Evb29vDy8sKcOXNw6dIli239/PwgkUgsXmvWrOnK0+pRv5h7kBiQiIiIuoKoAWnnzp2Ii4vDSy+9hPT0dISEhCA6OhrFxcXX3C4nJwdLly7FHXfcYdFeW1uL9PR0LFu2DOnp6di1axeys7PxwAMPtNrHqlWrUFBQYH4988wzXXpuPaWuwYhTRVUAOECbiIioq8jFPPi6desQGxuL+fPnAwCSkpKwZ88ebN26FS+88EKb2xiNRsTExGDlypX47rvvUFFRYV6m1Wqxb98+i/XffPNNjBkzBrm5ufD19TW3q9VqeHh4dP1J9bATBToYTQJcHZTw1KrELoeIiKhPEK0HqaGhAWlpaYiKirpajFSKqKgopKSktLvdqlWr4ObmhgULFnToOJWVlZBIJHB0dLRoX7NmDVxcXDBq1CisXbsWBoPhmvvR6/XQ6XQWL2twzPyAWi0HaBMREXUR0XqQSktLYTQa4e7ubtHu7u6OkydPtrnN999/jy1btiAjI6NDx6ivr8fzzz+Phx9+GBqNxtz+7LPPYvTo0XB2dsbhw4cRHx+PgoICrFu3rt19JSQkYOXKlR06bk/67R1sRERE1DVEvcR2I6qqqvDoo49i8+bNcHV1ve76jY2NmDFjBgRBwNtvv22xLC4uzvxzcHAwFAoFnnjiCSQkJECpVLa5v/j4eIvtdDodfHx8Onk2XeeX5jvYOIM2ERFRlxEtILm6ukImk6GoqMiivaioqM2xQWfPnkVOTg6mTJlibjOZTAAAuVyO7OxsDBkyBMDVcHThwgUcOHDAoveoLRERETAYDMjJyUFgYGCb6yiVynbDk1h09Y04V1IDAAhhDxIREVGXEW0MkkKhQFhYGJKTk81tJpMJycnJiIyMbLX+sGHDcPz4cWRkZJhfDzzwACZMmICMjAxzb05zODp9+jT2798PFxeX69aSkZEBqVQKNze3rjvBHvDrlctrA51s4WyvELkaIiKivkPUS2xxcXGYO3cuwsPDMWbMGCQmJqKmpsZ8V9ucOXPg7e2NhIQEqFQqjBgxwmL75oHXze2NjY146KGHkJ6ejt27d8NoNKKwsBAA4OzsDIVCgZSUFKSmpmLChAlQq9VISUnBkiVLMHv2bDg5OfXcyXcBzn9ERETUPUQNSDNnzkRJSQmWL1+OwsJChIaGYu/eveaB27m5uZBKO97JdfHiRfznP/8BAISGhlos++abb3D33XdDqVRix44dWLFiBfR6Pfz9/bFkyRKL8UW9xdUZtB1FrYOIiKivkQiCIIhdRG+k0+mg1WpRWVl53TFO3WXcmgO4WFGHj2IjMHbI9QeuExER9Xcd/f4W/VEj1DmXq/W4WFEHiQQY6c1LbERERF2JAamXap7/aLCrPdQqG5GrISIi6lsYkHqpX8wzaDuKWgcREVFfxIDUSx3jHWxERETdhgGpFxIEwXwH20j2IBEREXU5BqReqKymAaXVDZBIgFu9xLmDjoiIqC9jQOqFcstqAQAeGhVUNjKRqyEiIup7GJB6oeaA5ONsJ3IlREREfRMDUi+UX14HAPBxYkAiIiLqDgxIvVDu5aYeJF/2IBEREXULBqRe6OolNluRKyEiIuqbGJB6obxy9iARERF1JwakXqbRaMKliqYxSAxIRERE3YMBqZcpqKiHSQCUcikGqJVil0NERNQnMSD1Mr+9xV8ikYhcDRERUd/EgNTLNAckXl4jIiLqPgxIvUzzAG0fJ97BRkRE1F0YkHoZzqJNRETU/RiQepl8BiQiIqJux4DUy3AMEhERUfdjQOpFquobUV7bCIA9SERERN2JAakXyStrmiDS2V4BB6Vc5GqIiIj6LgakXoQDtImIiHoGA1Ivks9b/ImIiHoEA1IvwgHaREREPYMBqRdhQCIiIuoZDEi9SB7HIBEREfUIBqRewmQSkFfedBcbe5CIiIi6FwNSL1FSrUeDwQSZVAJPrUrscoiIiPo0BqReonn8kZejCnIZPzYiIqLuxG/aXiL3MgdoExER9RQGpF4izzwHEgMSERFRdxM9IG3cuBF+fn5QqVSIiIjAkSNHOrTdjh07IJFIMG3aNIt2QRCwfPlyeHp6wtbWFlFRUTh9+rTFOmVlZYiJiYFGo4GjoyMWLFiA6urqrjqlbsFZtImIiHqOqAFp586diIuLw0svvYT09HSEhIQgOjoaxcXF19wuJycHS5cuxR133NFq2WuvvYb169cjKSkJqampsLe3R3R0NOrr683rxMTEIDMzE/v27cPu3bvx7bffYuHChV1+fl0pv4x3sBEREfUUUQPSunXrEBsbi/nz52P48OFISkqCnZ0dtm7d2u42RqMRMTExWLlyJQYPHmyxTBAEJCYm4n/+538wdepUBAcHY/v27bh06RL+/e9/AwCysrKwd+9evPPOO4iIiMD48eOxYcMG7NixA5cuXerO070p7EEiIiLqOaIFpIaGBqSlpSEqKupqMVIpoqKikJKS0u52q1atgpubGxYsWNBq2fnz51FYWGixT61Wi4iICPM+U1JS4OjoiPDwcPM6UVFRkEqlSE1Nbfe4er0eOp3O4tVT6huNKNQ19YCxB4mIiKj7iRaQSktLYTQa4e7ubtHu7u6OwsLCNrf5/vvvsWXLFmzevLnN5c3bXWufhYWFcHNzs1gul8vh7Ozc7nEBICEhAVqt1vzy8fG59gl2oYsVTZfX7BUyONnZ9NhxiYiI+ivRB2l3VFVVFR599FFs3rwZrq6uPX78+Ph4VFZWml95eXk9duzfXl6TSCQ9dlwiIqL+Si7WgV1dXSGTyVBUVGTRXlRUBA8Pj1brnz17Fjk5OZgyZYq5zWQyAWjqAcrOzjZvV1RUBE9PT4t9hoaGAgA8PDxaDQI3GAwoKytr87jNlEollErljZ1kF8nnQ2qJiIh6lGg9SAqFAmFhYUhOTja3mUwmJCcnIzIystX6w4YNw/Hjx5GRkWF+PfDAA5gwYQIyMjLg4+MDf39/eHh4WOxTp9MhNTXVvM/IyEhUVFQgLS3NvM6BAwdgMpkQERHRjWfceRygTURE1LNE60ECgLi4OMydOxfh4eEYM2YMEhMTUVNTg/nz5wMA5syZA29vbyQkJEClUmHEiBEW2zs6OgKARfvixYvx8ssvY+jQofD398eyZcvg5eVlni8pKCgIkydPRmxsLJKSktDY2IhFixZh1qxZ8PLy6pHzvlG57EEiIiLqUaIGpJkzZ6KkpATLly9HYWEhQkNDsXfvXvMg69zcXEilN9bJ9de//hU1NTVYuHAhKioqMH78eOzduxcq1dUHvH744YdYtGgRJk6cCKlUiunTp2P9+vVdem5dKY9zIBEREfUoiSAIgthF9EY6nQ5arRaVlZXQaDTddhxBEBC84mtU6Q3YH3cnAtzU3XYsIiKivq6j39+95i62/qqithFVegMAYCCfw0ZERNQjGJCsXPNDat3USqhsZCJXQ0RE1D8wIFk5DtAmIiLqeQxIVo4DtImIiHoeA5KVa+5BGsiARERE1GMYkKxcHi+xERER9TgGJCvXPEjbx8lW5EqIiIj6DwYkK2Y0CbhYfmUMkgt7kIiIiHoKA5IVK6isg8EkQCGTwl2tuv4GRERE1CUYkKyYeYC2ky2kUonI1RAREfUfDEhWrHmAtg8HaBMREfUoBiQrxjmQiIiIxMGAZMVyzT1IvIONiIioJzEgWbHmW/zZg0RERNSzGJCsWJ55kDYDEhERUU9iQLJSNXoDSqsbAHAOJCIiop7GgGSl8q9MEOloZwONykbkaoiIiPoXBiQrZR6gzctrREREPY4ByUrxIbVERETiYUCyUuZZtHmLPxERUY9jQLJS7EEiIiISDwOSleIcSEREROJhQLJCgiCYHzPCQdpEREQ9jwHJCpVWN6Cu0QipBPBy5BgkIiKinsaAZIWaB2h7am2hkPMjIiIi6mn89rVCeXxILRERkagYkKwQ72AjIiISFwOSFeIs2kREROJiQLJC5lv8+ZBaIiIiUTAgWSHzLf68xEZERCQKBiQr02Aw4VIl50AiIiISEwOSlblUUQdBAGxtZHB1UIhdDhERUb8kekDauHEj/Pz8oFKpEBERgSNHjrS77q5duxAeHg5HR0fY29sjNDQU77//vsU6EomkzdfatWvN6/j5+bVavmbNmm47xxuR+5tb/CUSicjVEBER9U9yMQ++c+dOxMXFISkpCREREUhMTER0dDSys7Ph5ubWan1nZ2e8+OKLGDZsGBQKBXbv3o358+fDzc0N0dHRAICCggKLbb788kssWLAA06dPt2hftWoVYmNjze/VanU3nOGNy+Ut/kRERKITNSCtW7cOsbGxmD9/PgAgKSkJe/bswdatW/HCCy+0Wv/uu++2eP/cc89h27Zt+P77780BycPDw2Kdzz//HBMmTMDgwYMt2tVqdat1rUHzHWwcoE1ERCQe0S6xNTQ0IC0tDVFRUVeLkUoRFRWFlJSU624vCAKSk5ORnZ2NO++8s811ioqKsGfPHixYsKDVsjVr1sDFxQWjRo3C2rVrYTAYrnk8vV4PnU5n8eoOeZwDiYiISHSi9SCVlpbCaDTC3d3dot3d3R0nT55sd7vKykp4e3tDr9dDJpPhrbfewqRJk9pcd9u2bVCr1XjwwQct2p999lmMHj0azs7OOHz4MOLj41FQUIB169a1e9yEhASsXLnyBs6wc5pv8eclNiIiIvGIeomtM9RqNTIyMlBdXY3k5GTExcVh8ODBrS6/AcDWrVsRExMDlUpl0R4XF2f+OTg4GAqFAk888QQSEhKgVCrbPG58fLzFdjqdDj4+Pl1zUr9hMAmQSHiJjYiISEyiBSRXV1fIZDIUFRVZtBcVFV1zbJBUKkVAQAAAIDQ0FFlZWUhISGgVkL777jtkZ2dj586d160lIiICBoMBOTk5CAwMbHMdpVLZbnjqSl8+dwcaDCbIpLyDjYiISCyijUFSKBQICwtDcnKyuc1kMiE5ORmRkZEd3o/JZIJer2/VvmXLFoSFhSEkJOS6+8jIyIBUKm3zzjkxKORSBiQiIiIRiXqJLS4uDnPnzkV4eDjGjBmDxMRE1NTUmO9qmzNnDry9vZGQkACgaRxQeHg4hgwZAr1ejy+++ALvv/8+3n77bYv96nQ6fPLJJ3jjjTdaHTMlJQWpqamYMGEC1Go1UlJSsGTJEsyePRtOTk7df9JERERk9UQNSDNnzkRJSQmWL1+OwsJChIaGYu/eveaB27m5uZBKr3Zy1dTU4KmnnkJ+fj5sbW0xbNgwfPDBB5g5c6bFfnfs2AFBEPDwww+3OqZSqcSOHTuwYsUK6PV6+Pv7Y8mSJRbji4iIiKh/kwiCIIhdRG+k0+mg1WpRWVkJjUYjdjlERETUAR39/hb9USNERERE1oYBiYiIiKgFBiQiIiKiFhiQiIiIiFpgQCIiIiJqgQGJiIiIqAUGJCIiIqIWGJCIiIiIWmBAIiIiImqBAYmIiIioBVGfxdabNT+hRafTiVwJERERdVTz9/b1nrTGgNRJVVVVAAAfHx+RKyEiIqIbVVVVBa1W2+5yPqy2k0wmEy5dugS1Wg2JRNJl+9XpdPDx8UFeXh4fgisifg7WgZ+DdeDnYB34OXQNQRBQVVUFLy8vSKXtjzRiD1InSaVSDBw4sNv2r9Fo+BfACvBzsA78HKwDPwfrwM/h5l2r56gZB2kTERERtcCARERERNQCA5KVUSqVeOmll6BUKsUupV/j52Ad+DlYB34O1oGfQ8/iIG0iIiKiFtiDRERERNQCAxIRERFRCwxIRERERC0wIBERERG1wIBkZTZu3Ag/Pz+oVCpERETgyJEjYpfUp3377beYMmUKvLy8IJFI8O9//9tiuSAIWL58OTw9PWFra4uoqCicPn1anGL7qISEBNx2221Qq9Vwc3PDtGnTkJ2dbbFOfX09nn76abi4uMDBwQHTp09HUVGRSBX3XW+//TaCg4PNExFGRkbiyy+/NC/n59Dz1qxZA4lEgsWLF5vb+Dn0DAYkK7Jz507ExcXhpZdeQnp6OkJCQhAdHY3i4mKxS+uzampqEBISgo0bN7a5/LXXXsP69euRlJSE1NRU2NvbIzo6GvX19T1cad916NAhPP300/jxxx+xb98+NDY24t5770VNTY15nSVLluC///0vPvnkExw6dAiXLl3Cgw8+KGLVfdPAgQOxZs0apKWl4aeffsI999yDqVOnIjMzEwA/h5529OhRbNq0CcHBwRbt/Bx6iEBWY8yYMcLTTz9tfm80GgUvLy8hISFBxKr6DwDCZ599Zn5vMpkEDw8PYe3atea2iooKQalUCh9//LEIFfYPxcXFAgDh0KFDgiA0/c5tbGyETz75xLxOVlaWAEBISUkRq8x+w8nJSXjnnXf4OfSwqqoqYejQocK+ffuEu+66S3juuecEQeDfh57EHiQr0dDQgLS0NERFRZnbpFIpoqKikJKSImJl/df58+dRWFho8ZlotVpERETwM+lGlZWVAABnZ2cAQFpaGhobGy0+h2HDhsHX15efQzcyGo3YsWMHampqEBkZyc+hhz399NP43e9+Z/H7Bvj3oSfxYbVWorS0FEajEe7u7hbt7u7uOHnypEhV9W+FhYUA0OZn0ryMupbJZMLixYsxbtw4jBgxAkDT56BQKODo6GixLj+H7nH8+HFERkaivr4eDg4O+OyzzzB8+HBkZGTwc+ghO3bsQHp6Oo4ePdpqGf8+9BwGJCKyGk8//TR+/fVXfP/992KX0m8FBgYiIyMDlZWV+PTTTzF37lwcOnRI7LL6jby8PDz33HPYt28fVCqV2OX0a7zEZiVcXV0hk8la3YlQVFQEDw8Pkarq35p/7/xMesaiRYuwe/dufPPNNxg4cKC53cPDAw0NDaioqLBYn59D91AoFAgICEBYWBgSEhIQEhKCf/zjH/wcekhaWhqKi4sxevRoyOVyyOVyHDp0COvXr4dcLoe7uzs/hx7CgGQlFAoFwsLCkJycbG4zmUxITk5GZGSkiJX1X/7+/vDw8LD4THQ6HVJTU/mZdCFBELBo0SJ89tlnOHDgAPz9/S2Wh4WFwcbGxuJzyM7ORm5uLj+HHmAymaDX6/k59JCJEyfi+PHjyMjIML/Cw8MRExNj/pmfQ8/gJTYrEhcXh7lz5yI8PBxjxoxBYmIiampqMH/+fLFL67Oqq6tx5swZ8/vz588jIyMDzs7O8PX1xeLFi/Hyyy9j6NCh8Pf3x7Jly+Dl5YVp06aJV3Qf8/TTT+Ojjz7C559/DrVabR5HodVqYWtrC61WiwULFiAuLg7Ozs7QaDR45plnEBkZidtvv13k6vuW+Ph43HffffD19UVVVRU++ugjHDx4EF999RU/hx6iVqvN4++a2dvbw8XFxdzOz6GHiH0bHVnasGGD4OvrKygUCmHMmDHCjz/+KHZJfdo333wjAGj1mjt3riAITbf6L1u2THB3dxeUSqUwceJEITs7W9yi+5i2fv8AhHfffde8Tl1dnfDUU08JTk5Ogp2dnfCHP/xBKCgoEK/oPuqxxx4TBg0aJCgUCmHAgAHCxIkTha+//tq8nJ+DOH57m78g8HPoKRJBEASRshkRERGRVeIYJCIiIqIWGJCIiIiIWmBAIiIiImqBAYmIiIioBQYkIiIiohYYkIiIiIhaYEAiIiIiaoEBiYioCxw8eBASiaTVM7KIqHdiQCIiIiJqgQGJiIiIqAUGJCLqE0wmExISEuDv7w9bW1uEhITg008/BXD18teePXsQHBwMlUqF22+/Hb/++qvFPv71r3/h1ltvhVKphJ+fH9544w2L5Xq9Hs8//zx8fHygVCoREBCALVu2WKyTlpaG8PBw2NnZYezYscjOzu7eEyeibsGARER9QkJCArZv346kpCRkZmZiyZIlmD17Ng4dOmRe5y9/+QveeOMNHD16FAMGDMCUKVPQ2NgIoCnYzJgxA7NmzcLx48exYsUKLFu2DO+99555+zlz5uDjjz/G+vXrkZWVhU2bNsHBwcGijhdffBFvvPEGfvrpJ8jlcjz22GM9cv5E1LX4sFoi6vX0ej2cnZ2xf/9+REZGmtsff/xx1NbWYuHChZgwYQJ27NiBmTNnAgDKysowcOBAvPfee5gxYwZiYmJQUlKCr7/+2rz9X//6V+zZsweZmZk4deoUAgMDsW/fPkRFRbWq4eDBg5gwYQL279+PiRMnAgC++OIL/O53v0NdXR1UKlU3/xaIqCuxB4mIer0zZ86gtrYWkyZNgoODg/m1fft2nD171rzeb8OTs7MzAgMDkZWVBQDIysrCuHHjLPY7btw4nD59GkajERkZGZDJZLjrrruuWUtwcLD5Z09PTwBAcXHxTZ8jEfUsudgFEBHdrOrqagDAnj174O3tbbFMqVRahKTOsrW17dB6NjY25p8lEgmApvFRRNS7sAeJiHq94cOHQ6lUIjc3FwEBARYvHx8f83o//vij+efy8nKcOnUKQUFBAICgoCD88MMPFvv94YcfcMstt0Amk2HkyJEwmUwWY5qIqO9iDxIR9XpqtRpLly7FkiVLYDKZMH78eFRWVuKHH36ARqPBoEGDAACrVq2Ci4sL3N3d8eKLL8LV1RXTpk0DAPz5z3/GbbfdhtWrV2PmzJlISUnBm2++ibfeegsA4Ofnh7lz5+Kxxx7D+vXrERISggsXLqC4uBgzZswQ69SJqJswIBFRn7B69WoMGDAACQkJOHfuHBwdHTF69Gj87W9/M1/iWrNmDZ577jmcPn0aoaGh+O9//wuFQgEAGD16NP75z39i+fLlWL16NTw9PbFq1SrMmzfPfIy3334bf/vb3/DUU0/h8uXL8PX1xd/+9jcxTpeIuhnvYiOiPq/5DrPy8nI4OjqKXQ4R9QIcg0RERETUAgMSERERUQu8xEZERETUAnuQiIiIiFpgQCIiIiJqgQGJiIiIqAUGJCIiIqIWGJCIiIiIWmBAIiIiImqBAYmIiIioBQYkIiIiohYYkIiIiIha+H9obKokl3EBLwAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "score_batch_size = 500\n",
        "total_test_size = 3000\n",
        "score_ = 0\n",
        "translator=Translator(transformer,tokenizer)\n",
        "test_generator = DataGenerator(shuffled_data[220000:], mode='test', batch_size=score_batch_size)\n",
        "for i in range(total_test_size//score_batch_size):\n",
        "    shuffled, ordered = test_generator[i]\n",
        "    y_pred = translator(shuffled)\n",
        "    b_score = 0                   # score associated with each batch\n",
        "\n",
        "    pred_sentences = detokenizer(y_pred)\n",
        "    original_sentences = detokenizer(ordered)\n",
        "\n",
        "    for j in range(score_batch_size) :\n",
        "        b_score += score(clean_sentence(original_sentences[j]), clean_sentence(pred_sentences[j]))\n",
        "\n",
        "    score_ += b_score\n",
        "    print(\"\\n====BATCH OVER====\")\n",
        "    print(\"Score as of batch \", i, \": \", score_/((i+1)*score_batch_size))\n",
        "\n",
        "score_ = score_/total_test_size\n",
        "print(\"\\n====ALL OVER====\")\n",
        "print(\"Final score: \", score_)"
      ],
      "metadata": {
        "execution": {
          "iopub.status.busy": "2024-06-10T17:02:45.459214Z",
          "iopub.status.idle": "2024-06-10T17:02:45.459565Z",
          "shell.execute_reply": "2024-06-10T17:02:45.459409Z",
          "shell.execute_reply.started": "2024-06-10T17:02:45.459393Z"
        },
        "trusted": true,
        "id": "AQA9S2SKxz7e",
        "outputId": "aa664cbc-5682-4152-ec78-da6afd715f25"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": "/home/zeus/miniconda3/envs/cloudspace/lib/python3.10/site-packages/keras/src/ops/nn.py:545: UserWarning: You are using a softmax over axis 3 of a tensor of shape (500, 8, 1, 1). This axis has size 1. The softmax operation will always return the value 1, which is likely not what you intended. Did you mean to use a sigmoid instead?\n\n  warnings.warn(\n"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": "\n\n====BATCH OVER====\n\nScore as of batch  0 :  0.5751883901435415\n\n\n\n====BATCH OVER====\n\nScore as of batch  1 :  0.5836768526619389\n\n\n\n====BATCH OVER====\n\nScore as of batch  2 :  0.57737559133173\n\n\n\n====BATCH OVER====\n\nScore as of batch  3 :  0.5786266152461692\n\n\n\n====BATCH OVER====\n\nScore as of batch  4 :  0.5762958641176169\n\n\n\n====BATCH OVER====\n\nScore as of batch  5 :  0.5732276251260583\n\n\n\n====ALL OVER====\n\nFinal score:  0.5732276251260583\n"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Conclusion\n",
        "\n",
        "My final score is: 0.5732276251260583\n",
        "\n",
        "After numerous experiments and iterations, I have developed a final model using the Transformer architecture as proposed in the paper Attention Is All You Need. While I also experimented with different architectures, such as LSTM, their performance was significantly worse.\n",
        "\n",
        "To further enhance the model’s performance, I incorporated a custom Validation Callback, which allowed real-time monitoring of the model’s performance on the validation set. This led to higher accuracy and better test performance. Additionally, I adopted the CosineDecay learning rate schedule to personalize the learning rate and help the model overcome encountered plateaus.\n",
        "\n",
        "The model achieved average performance overall. Parameter tuning efforts, such as:\n",
        "\n",
        "•\tIncreasing the number of attention heads\n",
        "\n",
        "•\tIncreasing the dropout rate\n",
        "\n",
        "resulted in similar or lower scores.\n",
        "\n",
        "For completeness I would like to say that I was inspired by the following papers:\n",
        "\n",
        "* https://aclanthology.org/2021.naacl-main.134.pdf\n",
        "* https://arxiv.org/pdf/1706.03762.pdf"
      ],
      "metadata": {
        "id": "GPNQVm9M0Js3"
      }
    }
  ]
}