Vanilla_Transformer_Translation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ua4wuagXwmaJ"
      },
      "source": [
        "Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pWKoS5WMtccy"
      },
      "source": [
        "## Task Description"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ygNxO9J1lFQ1"
      },
      "source": [
        "Learn the internal working mechanisms of Transformer more closely.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "abqRds1Ms7bI"
      },
      "source": [
        "You need to **implement TransformerEncoder and TransformerDecoder with their layers** by your own. Using your TransformerEncoder and TransformerDecoder **build a Transformer model and train it** on the task of translation from English to French."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NMVrj09fXWzd"
      },
      "source": [
        "If you do not know how Encoder and Decoder could be implemented, check the original paper [Attention is All You Need](https://arxiv.org/pdf/1706.03762) that describes the implementation of Encoder and Decoder in details."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ge-0IvcN3UIB"
      },
      "source": [
        "To implement the layers you can use the following modules provided by PyTorch:\n",
        "\n",
        "1. [MultiheadAttention](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html)\n",
        "2. [FFN](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html)\n",
        "3. [ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html)\n",
        "4. [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n",
        "5. [Dropout](https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html)\n",
        "\n",
        "However, you are **not allowed** to use PyTorch implementation of Transformer, TransformerEncoder, TransformerDecoder and their layers!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Ost9s8P9KJP"
      },
      "source": [
        "You can use the code from the lab to build a Transformer from your layers and to train it on the translation task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "odM0yTqOs8FW"
      },
      "source": [
        "## Grading Criteria"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOT08X3_t1AO"
      },
      "source": [
        "* **TransformerEncoderLayer** is implemented - 1 point\n",
        "* **TransformerEncoder** is implemented - 1 point\n",
        "* **TransformerDecoderLayer** is implemented - 1 point\n",
        "* **TransformerDecoder** is implemented - 1 point\n",
        "* Transformer **training runs successfully** without any errors - 1 point\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UnQpYQwOyWT5"
      },
      "source": [
        "## Machine Translation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fi9DJjnzKRdq"
      },
      "source": [
        "Vanilla Transformer can be used to implement a translator from one language to another. This task is called Machine Translation (MT). Here we will implement Vanilla Transformer using PyTorch to translate English text to French."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Hm1hPUqKoUK"
      },
      "source": [
        "First, let us install necessary packages and import necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gpABGFRnUP62",
        "outputId": "848ed913-5ab0-4dd2-a697-9bdf96c56c51"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting datasets\n",
            "  Downloading datasets-3.0.2-py3-none-any.whl.metadata (20 kB)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets) (3.16.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.26.4)\n",
            "Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (16.1.0)\n",
            "Collecting dill<0.3.9,>=0.3.0 (from datasets)\n",
            "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.2.2)\n",
            "Requirement already satisfied: requests>=2.32.2 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.32.3)\n",
            "Requirement already satisfied: tqdm>=4.66.3 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.66.5)\n",
            "Collecting xxhash (from datasets)\n",
            "  Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
            "Collecting multiprocess<0.70.17 (from datasets)\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",
            "Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.6.1)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.10.10)\n",
            "Requirement already satisfied: huggingface-hub>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.24.7)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0.2)\n",
            "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.4.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (24.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.1.0)\n",
            "Requirement already satisfied: yarl<2.0,>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.16.0)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.23.0->datasets) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.32.2->datasets) (2024.8.30)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)\n",
            "Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from yarl<2.0,>=1.12.0->aiohttp->datasets) (0.2.0)\n",
            "Downloading datasets-3.0.2-py3-none-any.whl (472 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m472.7/472.7 kB\u001b[0m \u001b[31m9.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m11.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hInstalling collected packages: xxhash, dill, multiprocess, datasets\n",
            "Successfully installed datasets-3.0.2 dill-0.3.8 multiprocess-0.70.16 xxhash-3.5.0\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-2.4.3-py3-none-any.whl.metadata (51 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m51.8/51.8 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting portalocker (from sacrebleu)\n",
            "  Downloading portalocker-2.10.1-py3-none-any.whl.metadata (8.5 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (2024.9.11)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (0.9.0)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (1.26.4)\n",
            "Collecting colorama (from sacrebleu)\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (4.9.4)\n",
            "Downloading sacrebleu-2.4.3-py3-none-any.whl (103 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m104.0/104.0 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Downloading portalocker-2.10.1-py3-none-any.whl (18 kB)\n",
            "Installing collected packages: portalocker, colorama, sacrebleu\n",
            "Successfully installed colorama-0.4.6 portalocker-2.10.1 sacrebleu-2.4.3\n"
          ]
        }
      ],
      "source": [
        "!pip install datasets\n",
        "!pip install sacrebleu"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "RxD4BEIMnUYz"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch import nn\n",
        "import torch.optim as optim\n",
        "import math"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-btisUnDdSCj",
        "outputId": "f44d5552-9ca6-4439-e333-e441bf024b5e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "device(type='cuda')"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "device"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NB7MRv42BmuY"
      },
      "source": [
        "## Positional Encoding"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hObFUSZJK6Rz"
      },
      "source": [
        "Now, let's implement calculation of the positional encoding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "CImV0MHb8PIo"
      },
      "outputs": [],
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    def __init__(self, d_model, max_len=5000):\n",
        "        # Initialize the PositionalEncoding class\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "\n",
        "        # Create a tensor to store positional encodings with shape (max_len, d_model)\n",
        "        # max_len: maximum length of the input sequence\n",
        "        # d_model: dimension of the model embeddings (matches token embedding size)\n",
        "        self.encoding = torch.zeros(max_len, d_model)\n",
        "\n",
        "        # Create a tensor of positions from 0 to max_len-1 with shape (max_len, 1)\n",
        "        # Each row represents the position of a word in the sequence\n",
        "        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
        "\n",
        "        # Compute the denominator for the sine and cosine positional encodings\n",
        "        # This creates a scaling factor based on the position in the model dimension (d_model)\n",
        "        # d_model is divided by 2 since we alternate between sine and cosine encodings\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
        "\n",
        "        # Apply sine to even dimensions (0, 2, 4, ...) of the positional encoding\n",
        "        # This is done by multiplying the position by the div_term and applying sin\n",
        "        self.encoding[:, 0::2] = torch.sin(position * div_term)\n",
        "\n",
        "        # Apply cosine to odd dimensions (1, 3, 5, ...) of the positional encoding\n",
        "        # This is done by multiplying the position by the div_term and applying cos\n",
        "        self.encoding[:, 1::2] = torch.cos(position * div_term)\n",
        "\n",
        "        # Add an extra dimension at the start to match the batch size during training\n",
        "        # Shape becomes (1, max_len, d_model), allowing it to be added to input sequences\n",
        "        self.encoding = self.encoding.unsqueeze(0)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Add the positional encoding to the input tensor (x), ensuring that it is on the same device (CPU or GPU)\n",
        "        # The encoding tensor is sliced to match the sequence length of the input (x.size(1))\n",
        "        # This step adds positional information to the input embeddings\n",
        "        x = x + self.encoding[:, :x.size(1), :].to(x.device)\n",
        "        return x\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s89JutSTBpHr"
      },
      "source": [
        "## Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "POtErLmFF7Xb"
      },
      "source": [
        "TASK: Implement Transformer Encoder Layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "tGYT49wrFgPZ"
      },
      "outputs": [],
      "source": [
        "class TransformerEncoderLayer(nn.Module):\n",
        "    def __init__(self, d_model, nhead, dim_feedforward, dropout=0.1):\n",
        "        super(TransformerEncoderLayer, self).__init__()\n",
        "\n",
        "        # Multi-Head Self-Attention mechanism\n",
        "        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)\n",
        "\n",
        "        # Feedforward network\n",
        "        self.feedforward = nn.Sequential(\n",
        "            nn.Linear(d_model, dim_feedforward),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(dim_feedforward, d_model)\n",
        "        )\n",
        "\n",
        "        # Layer Normalization and Dropout layers\n",
        "        self.norm1 = nn.LayerNorm(d_model)\n",
        "        self.norm2 = nn.LayerNorm(d_model)\n",
        "        self.dropout1 = nn.Dropout(dropout)\n",
        "        self.dropout2 = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, src, src_mask=None, src_key_padding_mask=None):\n",
        "        # Self-attention layer with residual connection and layer normalization\n",
        "        attn_output, _ = self.self_attn(src, src, src, attn_mask=src_mask,\n",
        "                                        key_padding_mask=src_key_padding_mask)\n",
        "        src = src + self.dropout1(attn_output)\n",
        "        src = self.norm1(src)\n",
        "\n",
        "        # Feed-forward network with residual connection and layer normalization\n",
        "        ff_output = self.feedforward(src)\n",
        "        src = src + self.dropout2(ff_output)\n",
        "        src = self.norm2(src)\n",
        "\n",
        "        return src\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bGIfB1M9GAUt"
      },
      "source": [
        "TASK: Implement Transformer Decoder Layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "LH_NjP26FkGd"
      },
      "outputs": [],
      "source": [
        "class TransformerDecoderLayer(nn.Module):\n",
        "    def __init__(self, d_model, nhead, dim_feedforward, dropout=0.1):\n",
        "        super(TransformerDecoderLayer, self).__init__()\n",
        "\n",
        "        # Masked Multi-Head Self-Attention (target attends to itself)\n",
        "        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)\n",
        "\n",
        "        # Multi-Head Attention (target attends to encoder's output, i.e., memory)\n",
        "        self.multihead_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout, batch_first=True)\n",
        "\n",
        "        # Feedforward network\n",
        "        self.feedforward = nn.Sequential(\n",
        "            nn.Linear(d_model, dim_feedforward),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(dim_feedforward, d_model)\n",
        "        )\n",
        "\n",
        "        # Layer Normalization and Dropout layers\n",
        "        self.norm1 = nn.LayerNorm(d_model)\n",
        "        self.norm2 = nn.LayerNorm(d_model)\n",
        "        self.norm3 = nn.LayerNorm(d_model)\n",
        "        self.dropout1 = nn.Dropout(dropout)\n",
        "        self.dropout2 = nn.Dropout(dropout)\n",
        "        self.dropout3 = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, tgt, memory, tgt_mask=None, memory_mask=None,\n",
        "                tgt_key_padding_mask=None, memory_key_padding_mask=None):\n",
        "\n",
        "        # Masked self-attention layer (decoder attends to previous positions in tgt)\n",
        "        tgt2, _ = self.self_attn(tgt, tgt, tgt, attn_mask=tgt_mask,\n",
        "                                 key_padding_mask=tgt_key_padding_mask)\n",
        "        tgt = tgt + self.dropout1(tgt2)\n",
        "        tgt = self.norm1(tgt)\n",
        "\n",
        "        # Multi-head attention layer (decoder attends to encoder's memory)\n",
        "        tgt2, _ = self.multihead_attn(tgt, memory, memory, attn_mask=memory_mask,\n",
        "                                      key_padding_mask=memory_key_padding_mask)\n",
        "        tgt = tgt + self.dropout2(tgt2)\n",
        "        tgt = self.norm2(tgt)\n",
        "\n",
        "        # Feed-forward network\n",
        "        tgt2 = self.feedforward(tgt)\n",
        "        tgt = tgt + self.dropout3(tgt2)\n",
        "        tgt = self.norm3(tgt)\n",
        "\n",
        "        return tgt\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkK8bT20GCo1"
      },
      "source": [
        "TASK: Implement Transformer Encoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "VgNAhmZNFofP"
      },
      "outputs": [],
      "source": [
        "class TransformerEncoder(nn.Module):\n",
        "    def __init__(self, encoder_layer, num_layers):\n",
        "        super(TransformerEncoder, self).__init__()\n",
        "\n",
        "        # Create a list of encoder layers (each identical)\n",
        "        self.layers = nn.ModuleList([encoder_layer for _ in range(num_layers)])\n",
        "\n",
        "        # Optionally, you can add a final layer normalization\n",
        "        self.norm = nn.LayerNorm(encoder_layer.self_attn.embed_dim)\n",
        "\n",
        "    def forward(self, src, mask=None, src_key_padding_mask=None):\n",
        "        # Pass the input through each encoder layer in sequence\n",
        "        for layer in self.layers:\n",
        "            src = layer(src, src_mask=mask, src_key_padding_mask=src_key_padding_mask)\n",
        "\n",
        "        # Optionally normalize the output\n",
        "        if self.norm is not None:\n",
        "            src = self.norm(src)\n",
        "\n",
        "        return src\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wemPJyZGFAd"
      },
      "source": [
        "TASK: Implement Transformer Decoder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "JkuhaorYFq59"
      },
      "outputs": [],
      "source": [
        "class TransformerDecoder(nn.Module):\n",
        "    def __init__(self, decoder_layer, num_layers):\n",
        "        super(TransformerDecoder, self).__init__()\n",
        "\n",
        "        # Stack multiple decoder layers\n",
        "        self.layers = nn.ModuleList([decoder_layer for _ in range(num_layers)])\n",
        "\n",
        "        # Optional final layer normalization\n",
        "        self.norm = nn.LayerNorm(decoder_layer.self_attn.embed_dim)\n",
        "\n",
        "    def forward(self, tgt, memory, tgt_mask=None, memory_mask=None,\n",
        "                tgt_key_padding_mask=None, memory_key_padding_mask=None):\n",
        "\n",
        "        # Pass the target sequence and memory through each decoder layer\n",
        "        for layer in self.layers:\n",
        "            tgt = layer(tgt, memory, tgt_mask=tgt_mask, memory_mask=memory_mask,\n",
        "                        tgt_key_padding_mask=tgt_key_padding_mask,\n",
        "                        memory_key_padding_mask=memory_key_padding_mask)\n",
        "\n",
        "        # Apply final layer normalization if defined\n",
        "        if self.norm is not None:\n",
        "            tgt = self.norm(tgt)\n",
        "\n",
        "        return tgt\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uN25j4f7NG10"
      },
      "source": [
        "Here we implement the Transformer architecture in `nn.Module`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "sDa8aZagpjCq"
      },
      "outputs": [],
      "source": [
        "class TransformerModel(nn.Module):\n",
        "    def __init__(\n",
        "        self,\n",
        "        src_vocab_size,        # Vocabulary size of the source language (input)\n",
        "        tgt_vocab_size,        # Vocabulary size of the target language (output)\n",
        "        d_model=512,           # Dimension of model embeddings (default 512)\n",
        "        nhead=8,               # Number of attention heads in multi-head attention (default 8)\n",
        "        num_encoder_layers=6,  # Number of layers in the Transformer encoder (default 6)\n",
        "        num_decoder_layers=6,  # Number of layers in the Transformer decoder (default 6)\n",
        "        dim_feedforward=2048,  # Dimension of the feedforward network inside the Transformer (default 2048)\n",
        "        dropout=0.1            # Dropout rate (default 0.1)\n",
        "    ):\n",
        "        # Initialize the nn.Module parent class\n",
        "        super(TransformerModel, self).__init__()\n",
        "\n",
        "        # Source embedding layer that converts input tokens to embeddings of size d_model\n",
        "        self.src_embedding = nn.Embedding(src_vocab_size, d_model)\n",
        "\n",
        "        # Target embedding layer that converts target tokens to embeddings of size d_model\n",
        "        self.tgt_embedding = nn.Embedding(tgt_vocab_size, d_model)\n",
        "\n",
        "        # Positional encoding for the embeddings to encode the positions of tokens in the sequence\n",
        "        self.positional_encoding = PositionalEncoding(d_model)\n",
        "\n",
        "        # Create a single layer of Transformer encoder\n",
        "        encoder_layer = TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout)\n",
        "\n",
        "        # Stack multiple encoder layers (num_encoder_layers defines how many)\n",
        "        self.transformer_encoder = TransformerEncoder(encoder_layer, num_encoder_layers)\n",
        "\n",
        "        # Create a single layer of Transformer decoder\n",
        "        decoder_layer = TransformerDecoderLayer(d_model, nhead, dim_feedforward, dropout)\n",
        "\n",
        "        # Stack multiple decoder layers (num_decoder_layers defines how many)\n",
        "        self.transformer_decoder = TransformerDecoder(decoder_layer, num_decoder_layers)\n",
        "\n",
        "        # Final linear layer to map the decoder output to the target vocabulary size\n",
        "        # The output dimension is the size of the target vocabulary (tgt_vocab_size)\n",
        "        self.fc_out = nn.Linear(d_model, tgt_vocab_size)\n",
        "\n",
        "        # Store the model embedding dimension (d_model) for scaling later\n",
        "        self.d_model = d_model\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        src,                    # Source sequence (input)\n",
        "        tgt,                    # Target sequence (input for the decoder)\n",
        "        src_mask,               # Mask for the source sequence (to avoid attending to padding tokens)\n",
        "        tgt_mask,               # Mask for the target sequence (prevents attention to future tokens)\n",
        "        src_padding_mask,       # Padding mask for the source (to avoid attention to padding)\n",
        "        tgt_padding_mask,       # Padding mask for the target (to avoid attention to padding)\n",
        "        memory_key_padding_mask # Padding mask for the memory (encoder output) in the decoder\n",
        "    ):\n",
        "        # Embed the source sequence and scale by sqrt of d_model for stable gradients\n",
        "        src_emb = self.src_embedding(src) * math.sqrt(self.d_model)\n",
        "\n",
        "        # Embed the target sequence and scale by sqrt of d_model\n",
        "        tgt_emb = self.tgt_embedding(tgt) * math.sqrt(self.d_model)\n",
        "\n",
        "        # Add positional encodings to the source embeddings\n",
        "        src_emb = self.positional_encoding(src_emb)\n",
        "\n",
        "        # Add positional encodings to the target embeddings\n",
        "        tgt_emb = self.positional_encoding(tgt_emb)\n",
        "\n",
        "        # Pass the source embeddings through the Transformer encoder\n",
        "        # The encoder produces a memory representation for the source sequence\n",
        "        memory = self.transformer_encoder(src_emb, mask=src_mask, src_key_padding_mask=src_padding_mask)\n",
        "\n",
        "        # Pass the target embeddings and the memory (encoder output) through the Transformer decoder\n",
        "        # The decoder attends to both the target sequence and the encoder's memory\n",
        "        output = self.transformer_decoder(\n",
        "            tgt_emb, memory, tgt_mask=tgt_mask, memory_mask=src_mask,\n",
        "            tgt_key_padding_mask=tgt_padding_mask,\n",
        "            memory_key_padding_mask=memory_key_padding_mask\n",
        "        )\n",
        "\n",
        "\n",
        "        # Apply the final linear layer to map the decoder output to the target vocabulary\n",
        "        output = self.fc_out(output)\n",
        "\n",
        "        # Return the final output (logits over the target vocabulary)\n",
        "        return output\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VBzHTdq2BwDS"
      },
      "source": [
        "## Mask Generation Functions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pJgKfWGZO27c"
      },
      "source": [
        "It is also necessary to implement functions that create masks for the input data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Io2Y6MEPByG9"
      },
      "outputs": [],
      "source": [
        "def generate_square_subsequent_mask(sz):\n",
        "    # Create a square matrix of size (sz, sz) filled with ones above and on the diagonal, zeros below the diagonal.\n",
        "    mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)\n",
        "\n",
        "    # Convert the mask to a float tensor and fill the zeros with -inf (indicating they should be ignored in attention)\n",
        "    # and ones with 0.0 (indicating those positions can be attended to).\n",
        "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
        "\n",
        "    # Return the final mask\n",
        "    return mask\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "_WkSP0DQPhQ6"
      },
      "outputs": [],
      "source": [
        "def create_padding_mask(seq, pad_idx):\n",
        "    # Return a boolean mask where the elements in the sequence that are equal to the pad_idx (padding index) are True,\n",
        "    # indicating that those positions are padding tokens.\n",
        "    return (seq == pad_idx).float()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cymWwH0hC913"
      },
      "source": [
        "## Machine Translation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eX3Q4zwIP4ns"
      },
      "source": [
        "After we have defined the model, now we can train the model to translate the text from English to French.\n",
        "\n",
        "So, let's start with loading the dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 222,
          "referenced_widgets": [
            "71dc356a0e4542759e0464c3ddc75d13",
            "1eceb71577b046ac8c290a8e1d956890",
            "938ddd09e5b64fdd8ddeab6cac227ea7",
            "6183acdb31594796b0c029259e50b5bb",
            "92f9e98dbd63427fab99e8d5d3216424",
            "4dbb6b3f0af744a48615613e95b17ca3",
            "f007232de8e6484a87e5c7c5d2914281",
            "bb11eac2ac4f472b926005fded6c84c7",
            "5cc7ff7931854a6e8253dbff2c004fcb",
            "da6ad3da9af743878d29d9d35db2adec",
            "5ba43e22fe794fadb102e910df0d1ed5",
            "b0fcf548ecef42cfb2f52ee67a8af95e",
            "e8a927e48bd543c2887761626133e694",
            "cdd5053a948f49f0aca6ca7b48530cc1",
            "3f36a293747a46c38e619a53ae43200e",
            "89e72264b9ea448e88be0393dc7923d9",
            "2e07f53d528c4c458bce0cdb5a23f2ee",
            "abca53ae51e446688335dab63acd15ba",
            "f032caf44047483fb00170cda80828a0",
            "680fcda83a6e4e4db4d6c66956b23599",
            "58944b48ea6b4c0a8d7226aee65bc98f",
            "4198bd6f4f5348d986fcbe286c80993c",
            "9db9d75ceee042ae85d8f05a89d2828c",
            "38519f7244574b2cb3512d0cb8f0291d",
            "45a2dd9f82c648f994e91e65eaf008ed",
            "5e6a40f40edb4650993dfebc00a8c460",
            "dfee7792702b4357a8e09f346e5b4369",
            "5cf2500202d340a8b945abc86b3b089f",
            "4bf6990f9c0d433ea62174922427ba24",
            "20daec0b6f1240ca9dbff3f5001f138a",
            "8ec6e6a54c8e42b99d58f0aac8135c18",
            "5d21ca98fcc0462e9f9632cf0e8cab8e",
            "4f2a295af32a4c37bb4e7c0f54dd260c"
          ]
        },
        "id": "PGfu2acNDYiu",
        "outputId": "878c46ab-c4a2-4108-d615-89ab9d812427"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "71dc356a0e4542759e0464c3ddc75d13",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "README.md:   0%|          | 0.00/28.1k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b0fcf548ecef42cfb2f52ee67a8af95e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "train-00000-of-00001.parquet:   0%|          | 0.00/21.0M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9db9d75ceee042ae85d8f05a89d2828c",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split:   0%|          | 0/127085 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "texts = load_dataset(\"opus_books\", \"en-fr\", split='train')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mCq7jssjQXSM"
      },
      "source": [
        "Here is just some unwrapping of the data. Nothing that important."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "1e88fd5ba56b461d97ccd724cdcd4392",
            "6053184dda0e442887d7dba1b56a5018",
            "aa1bf0038c9b4072ad0841defd97436c",
            "8d51f461f2e846e7b30c2accac9b01a2",
            "f9ad224fe31145f08ef5c46d4fb9d448",
            "f7034525d71e4e08b2efa14ada2d483a",
            "c6bae8ba94bf41f487357e1806b16a69",
            "be2a1aa3688248109d77b1cd7891f61e",
            "3ff785e3ff8b452998b43520703b8a40",
            "0a669d9abe604ffb8aa1bbdb608c00de",
            "cc57c4ca91d74af9ae81b3317982a7ed"
          ]
        },
        "id": "GS9yuOoctt59",
        "outputId": "20c6468e-39cb-463b-e32a-a736f545ec5e"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "1e88fd5ba56b461d97ccd724cdcd4392",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Map:   0%|          | 0/127085 [00:00<?, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "src_lang = 'en'\n",
        "tgt_lang = 'fr'\n",
        "\n",
        "texts = texts.map(\n",
        "    lambda x: {\n",
        "        'en': x['translation']['en'],\n",
        "        'fr': x['translation']['fr']\n",
        "    }\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYVf6Ll4QjBs"
      },
      "source": [
        "Definition of how many data samples should be used for training and testing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "1TEiGsBboBAV"
      },
      "outputs": [],
      "source": [
        "# Total number of samples\n",
        "max_size = len(texts)\n",
        "# max_size = 10000\n",
        "\n",
        "# Percentage of samples from the total number that will be used for training\n",
        "train_frac = 0.8 # Percen\n",
        "\n",
        "# Calculation of exact number of training and testing samples\n",
        "train_size = int(train_frac * max_size)\n",
        "test_size = max_size - train_size\n",
        "\n",
        "# Selection of the data for training and testing according to the number\n",
        "# of training and testing samples\n",
        "train_texts = texts.select(range(train_size))\n",
        "test_texts = texts.select(range(train_size, train_size+test_size))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZ6GFWtKTC2j"
      },
      "source": [
        "Wrap the dataset into PyTorch dataset in order to load data with DataLoader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "4qjgnaroH6Ht"
      },
      "outputs": [],
      "source": [
        "class TranslationDataset(torch.utils.data.Dataset):\n",
        "    def __init__(self, data, src_lang='en', tgt_lang='fr'):\n",
        "        self.data = data\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.data)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        return {\n",
        "            \"src\": self.data[idx][src_lang],\n",
        "            \"tgt\": self.data[idx][tgt_lang]\n",
        "        }\n",
        "\n",
        "train_dataset = TranslationDataset(train_texts)\n",
        "test_dataset = TranslationDataset(test_texts)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ElCzpkf5V5d7"
      },
      "source": [
        "For the sake of simplicity, we will use the ready-to-use tokenizer of English and French texts"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z9mr3qsZgfe2",
        "outputId": "2f516ccf-3583-4da2-a461-75714cde79be"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (0.2.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install sentencepiece\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 338,
          "referenced_widgets": [
            "2942d2016c314d088d93021234736a0b",
            "1342eaf0b59b405896c6925012aeaa95",
            "811fa4f4aceb4b2f9f3ca693e32f7653",
            "9e14a7de5bb04fb380ade37c60c9a9b1",
            "18e12efe656443ddac10633c8cad9d18",
            "4fb37a15be414565927ddfcc0d4d4fb7",
            "bd6e89b6c1b24c9ea5571b425b596239",
            "fc84e5310f1642239867dc7e994805ad",
            "8db9c78366844c21a1c10a397c88c8e0",
            "5a44a693c15547469cee55aa5e04c440",
            "eae12e4e7956431e9c9064a62a1b2fe2",
            "65046d18cbcb40d6a8bb0fce16344dbd",
            "8feb80f624234837b1bed233d530893b",
            "568ad1c0514f49d4807e2dd66675f325",
            "8236e0f8a2ba4fbba33ca8799ef0a5ba",
            "30b9907bf7b944e3bc8c055d41f84c48",
            "55babeb0171c42ea8db188dd75a1fdd4",
            "c55805daf77f4400a8a5ba6805c54a27",
            "3c6ea794013743c69c3a904232c0eeff",
            "5095e0b4ebb5469a8c0a6dc7815b5acc",
            "bd6db5af71724722a20ded150573a5b5",
            "29b3333bade247d894721aef8111da58",
            "5ae993317a8d4c47869f855a3a36e794",
            "c57554c008194d7a8eeca458629815f2",
            "3c3c0b98b5a74639a3a5a3646650ddde",
            "fad5f7a390e74a65adf9b8c9d53dbed7",
            "1f73419e430144fa953d09321c8e649d",
            "490b705be6cc4b97b8780f5c0efb1a65",
            "b40b2864c9bc430887b81291e11038c6",
            "471b6650bc514e46baba9ff187193fbd",
            "07f03cca599b4e6481ae6e46a211d302",
            "0db82807074a47d49526b73411b2ba90",
            "6811fcf949814149bda262d69b9e8fa0",
            "c83e3e7acba9412c8b53a11aa17350b4",
            "45f7b3c429b742309ddb1dd7491ccc00",
            "9ae603fa708248b8a12e9c6f19ee0435",
            "2de8d676d69e42bea0aa104ca5bf39f0",
            "0c33e0cfd4274ef7a81b502f331a3551",
            "f08365aa0627435cb19ebd895ecaa833",
            "02a1f43de9164591afa05f2d8b87dd98",
            "d10effdc8adb4cacb389dbd62e4cd1c6",
            "a9214205c6424bb6bb19fcdb98815a16",
            "3d72d6627ee54e2889532ecf359c6b44",
            "1b52e3b983844852956ac97542e0be34",
            "e24248d92fe440398ca6f2b4da0fa454",
            "1fb659b20933484f85d2d26c68ba3b51",
            "611b22e7bbc84ba888befa9d13aabf0e",
            "1bfa0435150b45ad8c5fc3f8735b8f6e",
            "93680094a78e4cb699a4a5a13082c3c5",
            "2da1c85833c4453aaba20dadac96d24f",
            "04e85902ec4e4c0481b0717b724313e3",
            "ffaadc561efd45f9b0d8dd525bc06e89",
            "3eb6f9323dc94ce4994345dabf01c400",
            "9aba8dc10ced4424a453436d09f6d702",
            "40b5561ca264427787da2c872f9bb0fb",
            "cfd82949d8e547a590a5240420ddfa6e",
            "cb544e77262141069263bd53273b8a00",
            "165d2b86a67f4a3f9a4669fd3caddaa2",
            "7f484da3b6704464b7d4aa1cf414a71d",
            "a82849856b1e4a7cb138140fe5f2406c",
            "a1b60b5d82c84e5daef5ee1a40f6096c",
            "fd86c0c4742746d7888f88ce5a35c510",
            "2226562ebeaa4ede861ffa4ffce1610c",
            "0bcad4fe67aa49f69d14d8b74a54e6ed",
            "7023fdcc52d443df9220fa73acab72fb",
            "12cceb7a61d54346b4b5ec233f6a593c"
          ]
        },
        "id": "y3JX31ChFCpl",
        "outputId": "0114dcce-9952-47a4-bc65-8a00128be575"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2942d2016c314d088d93021234736a0b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "0it [00:00, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "65046d18cbcb40d6a8bb0fce16344dbd",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "tokenizer_config.json:   0%|          | 0.00/42.0 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "5ae993317a8d4c47869f855a3a36e794",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "config.json:   0%|          | 0.00/1.42k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "c83e3e7acba9412c8b53a11aa17350b4",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "source.spm:   0%|          | 0.00/778k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "e24248d92fe440398ca6f2b4da0fa454",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "target.spm:   0%|          | 0.00/802k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "cfd82949d8e547a590a5240420ddfa6e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "vocab.json:   0%|          | 0.00/1.34M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/models/marian/tokenization_marian.py:175: UserWarning: Recommended: pip install sacremoses.\n",
            "  warnings.warn(\"Recommended: pip install sacremoses.\")\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "execution_count": 17,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"Helsinki-NLP/opus-mt-en-fr\")\n",
        "\n",
        "# Add Begining of Sentence (BOS) token since it is absent in the tokenizer\n",
        "tokenizer.add_special_tokens({'bos_token': '<BOS>'})"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ejpb030dWfa0"
      },
      "source": [
        "Create training and testing DataLoaders. Implement `collate_fn` in which we tokenize the batch of text using [`tokenizer.batch_encode_plus`](https://huggingface.co/docs/transformers/v4.45.2/en/internal/tokenization_utils#transformers.PreTrainedTokenizerBase.batch_encode_plus)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "T7hHGwJzIZVs"
      },
      "outputs": [],
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "# Define the maximum sequence length and batch size\n",
        "max_length = 64\n",
        "batch_size = 8\n",
        "\n",
        "# Custom function to process and batch data\n",
        "def collate_fn(batch):\n",
        "    # Prepend \"<BOS>\" token to each source text in the batch and tokenize it\n",
        "    tokenized_src_texts = tokenizer.batch_encode_plus(\n",
        "        [\"<BOS> \" + item['src'] for item in batch],  # Add \"<BOS>\" to the source text before tokenization\n",
        "        padding=\"max_length\",                        # Pad sequences to the max_length\n",
        "        max_length=max_length,                       # Define the maximum length for padding/truncation\n",
        "        truncation=True,                             # Truncate sequences longer than max_length\n",
        "        return_tensors=\"pt\"                          # Return a PyTorch tensor\n",
        "    )['input_ids']                                   # Extract the 'input_ids' (token IDs)\n",
        "\n",
        "    # Prepend \"<BOS>\" token to each target text in the batch and tokenize it\n",
        "    tokenized_tgt_texts = tokenizer.batch_encode_plus(\n",
        "        [\"<BOS> \" + item['tgt'] for item in batch],  # Add \"<BOS>\" to the target text before tokenization\n",
        "        padding=\"max_length\",                        # Pad sequences to the max_length\n",
        "        max_length=max_length,                       # Define the maximum length for padding/truncation\n",
        "        truncation=True,                             # Truncate sequences longer than max_length\n",
        "        return_tensors=\"pt\"                          # Return a PyTorch tensor\n",
        "    )['input_ids']                                   # Extract the 'input_ids' (token IDs)\n",
        "\n",
        "    # Return the tokenized and padded source and target texts as tensors\n",
        "    return {\n",
        "        'src': tokenized_src_texts,                  # Tokenized source sequences tensor\n",
        "        'tgt': tokenized_tgt_texts                   # Tokenized target sequences tensor\n",
        "    }\n",
        "\n",
        "# Create DataLoader for training data\n",
        "train_dataloader = DataLoader(\n",
        "    train_dataset,                                   # Training dataset\n",
        "    batch_size=batch_size,                           # Batch size for training\n",
        "    collate_fn=collate_fn,                           # Custom collate function for tokenizing and batching\n",
        "    shuffle=True                                     # Shuffle the data each epoch\n",
        ")\n",
        "\n",
        "# Create DataLoader for test data\n",
        "test_dataloader = DataLoader(\n",
        "    test_dataset,                                    # Test dataset\n",
        "    batch_size=batch_size,                           # Batch size for testing\n",
        "    collate_fn=collate_fn,                           # Custom collate function for tokenizing and batching\n",
        "    shuffle=False                                    # Do not shuffle the test data\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aaJKVet2X5CF"
      },
      "source": [
        "Create an instance of the model, optimizer, and loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "BkO46EkUKrR3"
      },
      "outputs": [],
      "source": [
        "# Retrieve special token indices from the tokenizer\n",
        "pad_idx = tokenizer.pad_token_id  # Padding token index\n",
        "bos_idx = tokenizer.bos_token_id  # Beginning of sequence token index\n",
        "eos_idx = tokenizer.eos_token_id  # End of sequence token index\n",
        "\n",
        "# Define vocabulary size, adding 1 due to the new BOS token\n",
        "vocab_size = tokenizer.vocab_size + 1\n",
        "\n",
        "# Initialize the Transformer model\n",
        "model = TransformerModel(\n",
        "    vocab_size,               # Source vocabulary size (input)\n",
        "    vocab_size,               # Target vocabulary size (output)\n",
        "    d_model=512,              # Dimensionality of the model (embedding size)\n",
        "    nhead=8,                  # Number of attention heads in the multi-head attention mechanism\n",
        "    num_encoder_layers=6,     # Number of layers in the encoder\n",
        "    num_decoder_layers=6,     # Number of layers in the decoder\n",
        "    dim_feedforward=2048,     # Dimension of the feedforward network inside each Transformer layer\n",
        "    dropout=0.1               # Dropout rate for regularization\n",
        ").to(device)                  # Move the model to the specified device (e.g., GPU or CPU)\n",
        "\n",
        "# Initialize the optimizer (Adam optimizer with a learning rate of 0.0001)\n",
        "optimizer = optim.Adam(model.parameters(), lr=0.0001)\n",
        "\n",
        "# Define the loss function (Cross-Entropy Loss) with the padding token ignored\n",
        "criterion = nn.CrossEntropyLoss(ignore_index=pad_idx)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UyPAZvRkYcED"
      },
      "source": [
        "Check how many parameters our model has"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6w9MHXJAYaA0",
        "outputId": "5d063d79-8f44-4f58-d1be-090c2c467b0c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "98833019"
            ]
          },
          "execution_count": 20,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sum(p.numel() for p in model.parameters())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fda8qUv2Y1HN"
      },
      "source": [
        "Define the function for training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "F8CeHWg0gfe5"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nyChRVn6MfSE",
        "outputId": "f2d991b6-593b-4090-f3a1-718b0194fd9c"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Loss: 6.372943062909479: 100%|██████████| 12709/12709 [24:02<00:00,  8.81it/s]\n"
          ]
        }
      ],
      "source": [
        "from tqdm import tqdm  # Import tqdm for progress bars\n",
        "\n",
        "def train_model(\n",
        "    model,                # The Transformer model\n",
        "    train_loader,         # DataLoader for the training data\n",
        "    optimizer,            # Optimizer (Adam in this case)\n",
        "    criterion,            # Loss function (Cross-Entropy Loss)\n",
        "    pad_idx,              # Padding token index to ignore in loss calculation\n",
        "    device,               # Device (CPU or GPU)\n",
        "    num_epochs=1         # Number of epochs to train the model\n",
        "):\n",
        "    model.train()  # Set the model to training mode\n",
        "    for epoch in range(num_epochs):  # Loop through each epoch\n",
        "        total_loss = 0  # Initialize total loss for the current epoch\n",
        "\n",
        "        pbar = tqdm(train_loader)  # Wrap train_loader with tqdm for progress display\n",
        "        for i, batch in enumerate(pbar):  # Loop through each batch in the training data\n",
        "            # Move the source and target sequences to the device (e.g., GPU)\n",
        "            src, tgt = batch['src'].to(device), batch['tgt'].to(device)\n",
        "\n",
        "            # Prepare input and output sequences for teacher forcing\n",
        "            tgt_input = tgt[:, :-1]  # Target input (remove the last token)\n",
        "            tgt_output = tgt[:, 1:]  # Target output (shifted by one token)\n",
        "\n",
        "            # Generate masks for attention\n",
        "            src_mask = None  # No source mask in this case\n",
        "            tgt_mask = generate_square_subsequent_mask(  # Generate mask for target input to prevent attending to future tokens\n",
        "                tgt_input.size(1)\n",
        "            ).to(device)\n",
        "\n",
        "            # Create padding masks for both source and target sequences\n",
        "            src_padding_mask = create_padding_mask(src, pad_idx).to(device)\n",
        "            tgt_padding_mask = create_padding_mask(tgt_input, pad_idx).to(device)\n",
        "            memory_padding_mask = create_padding_mask(src, pad_idx).to(device)\n",
        "\n",
        "            optimizer.zero_grad()  # Zero gradients before backpropagation\n",
        "\n",
        "            # Forward pass through the model\n",
        "            output = model(\n",
        "                src, tgt_input, src_mask, tgt_mask,  # Source and target with their masks\n",
        "                src_padding_mask, tgt_padding_mask, memory_padding_mask  # Padding masks for the attention mechanism\n",
        "            )\n",
        "            output = output.view(-1, output.size(-1))  # Reshape the model's output for calculating loss\n",
        "            tgt_output = tgt_output.contiguous().view(-1)  # Flatten the target output for loss calculation\n",
        "\n",
        "            # Compute the loss\n",
        "            loss = criterion(output, tgt_output)  # Cross-entropy loss\n",
        "            loss.backward()  # Backpropagate the gradients\n",
        "            optimizer.step()  # Update the model's parameters\n",
        "\n",
        "            total_loss += loss.item()  # Accumulate total loss for this epoch\n",
        "\n",
        "            # Update the progress bar description with the current average loss\n",
        "            pbar.set_description(f\"Loss: {total_loss / (i + 1)}\")\n",
        "\n",
        "\n",
        "# Call the training function with one epoch of training\n",
        "train_model(\n",
        "    model,                 # The Transformer model\n",
        "    train_dataloader,      # DataLoader for training data\n",
        "    optimizer,             # Adam optimizer\n",
        "    criterion,             # Cross-Entropy Loss\n",
        "    pad_idx,               # Padding index to ignore in the loss\n",
        "    device,                # Device (GPU or CPU)\n",
        "    num_epochs=1           # Number of epochs to train\n",
        ")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbbMRz9nZvis"
      },
      "source": [
        "Check how the model translates the English text"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e3W_uNjAwMYN",
        "outputId": "af1bfc78-199c-4944-dc0b-cf3ad214df1a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Source sentence: Then we will return with any who may wish to follow us to colonize it definitely, and endow the American Republic with a useful station in this part of the Pacific Ocean!\"\n",
            "Translated sentence: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n",
            "Reference sentence: Puis, nous y reviendrons avec tous ceux qui voudront nous suivre pour la coloniser définitivement et doter la république américaine d'une station utile dans cette partie de l'océan Pacifique!\n",
            "\n",
            "Source sentence: \"Hurrah!\" exclaimed Pencroft, \"and that will be no small present which we shall make to our country!\n",
            "Translated sentence: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n",
            "Reference sentence: -- Hurrah! s'écria Pencroff, et ce ne sera pas un petit cadeau que nous ferons là à notre pays!\n",
            "\n",
            "Source sentence: The colonization is already almost finished; names are given to every part of the island; there is a natural port, fresh water, roads, a telegraph, a dockyard, and manufactories; and there will be nothing to be done but to inscribe Lincoln Island on the maps!\"\n",
            "Translated sentence: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n",
            "Reference sentence: La colonisation est déjà presque achevée, les noms sont donnés à toutes les parties de l'île, il y a un port naturel, une aiguade, des routes, une ligne télégraphique, un chantier, une usine, et il n'y aura plus qu'à inscrire l'île Lincoln sur les cartes!\n",
            "\n",
            "Source sentence: \"But if anyone seizes it in our absence?\" observed Gideon Spilett.\n",
            "Translated sentence: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n",
            "Reference sentence: -- Mais si on nous la prend pendant notre absence? fit observer Gédéon Spilett.\n",
            "\n",
            "Source sentence: \"Hang it!\" cried the sailor. \"I would rather remain all alone to guard it: and trust to Pencroft, they shouldn't steal it from him, like a watch from the pocket of a swell!\"\n",
            "Translated sentence: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --\n",
            "Reference sentence: -- Mille diables! s'écria le marin, j'y resterai plutôt tout seul pour la garder, et, foi de Pencroff, on ne me la volerait pas comme une montre dans la poche d'un badaud!»\n",
            "\n"
          ]
        }
      ],
      "source": [
        "def translate_sentence(model, tokenizer, sentence, max_length, pad_idx, device):\n",
        "    # Tokenize the input sentence\n",
        "    model.eval()  # Set the model to evaluation mode\n",
        "    tokens = tokenizer.encode(sentence, return_tensors=\"pt\").to(device)\n",
        "\n",
        "    # Ensure tokens are not longer than max_length\n",
        "    tokens = tokens[:, :max_length]\n",
        "\n",
        "    # Prepare the input tensor (add batch dimension if necessary)\n",
        "    src = tokens\n",
        "\n",
        "    # Initialize the decoder input with <BOS> token\n",
        "    tgt = torch.tensor([[bos_idx]], dtype=torch.long).to(device)\n",
        "\n",
        "    # Create padding masks\n",
        "    src_padding_mask = create_padding_mask(src, pad_idx).to(device)\n",
        "\n",
        "    # Generate a translation by decoding one token at a time\n",
        "    for _ in range(max_length):\n",
        "\n",
        "        with torch.no_grad():\n",
        "            output = model(\n",
        "                src, tgt, src_mask=None, tgt_mask=None,\n",
        "                src_padding_mask=src_padding_mask, tgt_padding_mask=None,\n",
        "                memory_key_padding_mask=src_padding_mask\n",
        "            )\n",
        "\n",
        "        # Get the last predicted token (greedy decoding)\n",
        "        next_token = output[:, -1, :].argmax(dim=-1).unsqueeze(0)\n",
        "\n",
        "        # Concatenate predicted token to target sequence\n",
        "        tgt = torch.cat([tgt, next_token], dim=1)\n",
        "\n",
        "        # Stop if <EOS> token is predicted\n",
        "        if next_token.item() == eos_idx:\n",
        "            break\n",
        "\n",
        "    # Decode the target tokens back into text\n",
        "    translated_tokens = tgt.squeeze().tolist()\n",
        "\n",
        "    # Convert token IDs back to the target language sentence\n",
        "    translated_sentence = tokenizer.decode(translated_tokens, skip_special_tokens=True)\n",
        "\n",
        "    return translated_sentence\n",
        "\n",
        "\n",
        "# Test translation example\n",
        "for example_idx in range(5):\n",
        "\n",
        "    example_sentence = test_texts[example_idx]['en']\n",
        "    refence_sentence = test_texts[example_idx]['fr']\n",
        "    translated_sentence = translate_sentence(\n",
        "        model, tokenizer, example_sentence, max_length=max_length, pad_idx=pad_idx, device=device\n",
        "    )\n",
        "\n",
        "    print(f\"Source sentence: {example_sentence}\")\n",
        "    print(f\"Translated sentence: {translated_sentence}\")\n",
        "    print(f\"Reference sentence: {refence_sentence}\")\n",
        "    print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ji_s-849Z5D-"
      },
      "source": [
        "Calculate the metrics of translation quality: [BLEU](https://en.wikipedia.org/wiki/BLEU) score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f5-k-7FumP61",
        "outputId": "4129899d-10e5-4d1e-c2a2-4848ed17dd90"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 3178/3178 [02:54<00:00, 18.16it/s]\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Test Loss: 6.432986984468992\n",
            "BLEU Score: 0.0\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "(6.432986984468992, 0.0)"
            ]
          },
          "execution_count": 23,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import torch\n",
        "from torch.utils.data import DataLoader\n",
        "from tqdm import tqdm\n",
        "import sacrebleu\n",
        "\n",
        "# Function to evaluate the model on the test set\n",
        "def evaluate_model(model, test_loader, criterion, pad_idx, device):\n",
        "    model.eval()\n",
        "    total_loss = 0\n",
        "    all_references = []\n",
        "    all_hypotheses = []\n",
        "\n",
        "    with torch.no_grad():\n",
        "        for batch in tqdm(test_loader):\n",
        "            src, tgt = batch['src'].to(device), batch['tgt'].to(device)\n",
        "\n",
        "            # Prepare input and output sequences\n",
        "            tgt_input = tgt[:, :-1]  # Input to the decoder\n",
        "            tgt_output = tgt[:, 1:]  # Target to compare with output\n",
        "\n",
        "            src_mask = None\n",
        "            tgt_mask = None\n",
        "\n",
        "            src_padding_mask = create_padding_mask(src, pad_idx).to(device)\n",
        "            tgt_padding_mask = create_padding_mask(tgt_input, pad_idx).to(device)\n",
        "            memory_padding_mask = create_padding_mask(src, pad_idx).to(device)\n",
        "\n",
        "            # Forward pass\n",
        "            output = model(src, tgt_input, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, memory_padding_mask)\n",
        "\n",
        "            # Compute the loss\n",
        "            output_flat = output.view(-1, output.size(-1))\n",
        "            tgt_output_flat = tgt_output.contiguous().view(-1)\n",
        "            loss = criterion(output_flat, tgt_output_flat)\n",
        "            total_loss += loss.item()\n",
        "\n",
        "            # Generate predictions (greedy decoding)\n",
        "            decoded_sentences = torch.argmax(output, dim=-1)\n",
        "\n",
        "            # Convert decoded sentences to strings and store for BLEU score calculation\n",
        "            for i in range(decoded_sentences.size(0)):\n",
        "                # Remove padding tokens, BOS, and EOS tokens from the predictions\n",
        "                pred_tokens = decoded_sentences[i].tolist()\n",
        "                pred_tokens = [token for token in pred_tokens if token != pad_idx]\n",
        "\n",
        "                # Convert tokens back to string\n",
        "                hypothesis = tokenizer.decode(pred_tokens, skip_special_tokens=True)\n",
        "\n",
        "                # Store the generated sentence (hypothesis)\n",
        "                all_hypotheses.append(hypothesis)\n",
        "\n",
        "                # Convert target tokens to string\n",
        "                target_tokens = tgt[i, 1:].tolist()  # Remove <BOS>\n",
        "                target_tokens = [token for token in target_tokens if token != pad_idx]\n",
        "\n",
        "                reference = tokenizer.decode(target_tokens, skip_special_tokens=True)\n",
        "\n",
        "                # Store the reference sentence\n",
        "                all_references.append([reference])  # sacrebleu expects a list of references\n",
        "\n",
        "    # Compute BLEU score\n",
        "    bleu = sacrebleu.corpus_bleu(all_hypotheses, all_references)\n",
        "    avg_loss = total_loss / len(test_loader)\n",
        "\n",
        "    print(f\"Test Loss: {avg_loss}\")\n",
        "    print(f\"BLEU Score: {bleu.score}\")\n",
        "\n",
        "    return avg_loss, bleu.score\n",
        "\n",
        "\n",
        "# Running the evaluation\n",
        "evaluate_model(model, test_dataloader, criterion, pad_idx, device)\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.7"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "02a1f43de9164591afa05f2d8b87dd98": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "04e85902ec4e4c0481b0717b724313e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "07f03cca599b4e6481ae6e46a211d302": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0a669d9abe604ffb8aa1bbdb608c00de": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0bcad4fe67aa49f69d14d8b74a54e6ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "0c33e0cfd4274ef7a81b502f331a3551": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0db82807074a47d49526b73411b2ba90": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "12cceb7a61d54346b4b5ec233f6a593c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1342eaf0b59b405896c6925012aeaa95": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4fb37a15be414565927ddfcc0d4d4fb7",
            "placeholder": "​",
            "style": "IPY_MODEL_bd6e89b6c1b24c9ea5571b425b596239",
            "value": ""
          }
        },
        "165d2b86a67f4a3f9a4669fd3caddaa2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2226562ebeaa4ede861ffa4ffce1610c",
            "max": 1339166,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0bcad4fe67aa49f69d14d8b74a54e6ed",
            "value": 1339166
          }
        },
        "18e12efe656443ddac10633c8cad9d18": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b52e3b983844852956ac97542e0be34": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1bfa0435150b45ad8c5fc3f8735b8f6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9aba8dc10ced4424a453436d09f6d702",
            "placeholder": "​",
            "style": "IPY_MODEL_40b5561ca264427787da2c872f9bb0fb",
            "value": " 802k/802k [00:00&lt;00:00, 26.8MB/s]"
          }
        },
        "1e88fd5ba56b461d97ccd724cdcd4392": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6053184dda0e442887d7dba1b56a5018",
              "IPY_MODEL_aa1bf0038c9b4072ad0841defd97436c",
              "IPY_MODEL_8d51f461f2e846e7b30c2accac9b01a2"
            ],
            "layout": "IPY_MODEL_f9ad224fe31145f08ef5c46d4fb9d448"
          }
        },
        "1eceb71577b046ac8c290a8e1d956890": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4dbb6b3f0af744a48615613e95b17ca3",
            "placeholder": "​",
            "style": "IPY_MODEL_f007232de8e6484a87e5c7c5d2914281",
            "value": "README.md: 100%"
          }
        },
        "1f73419e430144fa953d09321c8e649d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1fb659b20933484f85d2d26c68ba3b51": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2da1c85833c4453aaba20dadac96d24f",
            "placeholder": "​",
            "style": "IPY_MODEL_04e85902ec4e4c0481b0717b724313e3",
            "value": "target.spm: 100%"
          }
        },
        "20daec0b6f1240ca9dbff3f5001f138a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2226562ebeaa4ede861ffa4ffce1610c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2942d2016c314d088d93021234736a0b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1342eaf0b59b405896c6925012aeaa95",
              "IPY_MODEL_811fa4f4aceb4b2f9f3ca693e32f7653",
              "IPY_MODEL_9e14a7de5bb04fb380ade37c60c9a9b1"
            ],
            "layout": "IPY_MODEL_18e12efe656443ddac10633c8cad9d18"
          }
        },
        "29b3333bade247d894721aef8111da58": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2da1c85833c4453aaba20dadac96d24f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2de8d676d69e42bea0aa104ca5bf39f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3d72d6627ee54e2889532ecf359c6b44",
            "placeholder": "​",
            "style": "IPY_MODEL_1b52e3b983844852956ac97542e0be34",
            "value": " 778k/778k [00:00&lt;00:00, 12.0MB/s]"
          }
        },
        "2e07f53d528c4c458bce0cdb5a23f2ee": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30b9907bf7b944e3bc8c055d41f84c48": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "38519f7244574b2cb3512d0cb8f0291d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5cf2500202d340a8b945abc86b3b089f",
            "placeholder": "​",
            "style": "IPY_MODEL_4bf6990f9c0d433ea62174922427ba24",
            "value": "Generating train split: 100%"
          }
        },
        "3c3c0b98b5a74639a3a5a3646650ddde": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_471b6650bc514e46baba9ff187193fbd",
            "max": 1416,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_07f03cca599b4e6481ae6e46a211d302",
            "value": 1416
          }
        },
        "3c6ea794013743c69c3a904232c0eeff": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3d72d6627ee54e2889532ecf359c6b44": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3eb6f9323dc94ce4994345dabf01c400": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3f36a293747a46c38e619a53ae43200e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_58944b48ea6b4c0a8d7226aee65bc98f",
            "placeholder": "​",
            "style": "IPY_MODEL_4198bd6f4f5348d986fcbe286c80993c",
            "value": " 21.0M/21.0M [00:00&lt;00:00, 47.6MB/s]"
          }
        },
        "3ff785e3ff8b452998b43520703b8a40": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "40b5561ca264427787da2c872f9bb0fb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4198bd6f4f5348d986fcbe286c80993c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "45a2dd9f82c648f994e91e65eaf008ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_20daec0b6f1240ca9dbff3f5001f138a",
            "max": 127085,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8ec6e6a54c8e42b99d58f0aac8135c18",
            "value": 127085
          }
        },
        "45f7b3c429b742309ddb1dd7491ccc00": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f08365aa0627435cb19ebd895ecaa833",
            "placeholder": "​",
            "style": "IPY_MODEL_02a1f43de9164591afa05f2d8b87dd98",
            "value": "source.spm: 100%"
          }
        },
        "471b6650bc514e46baba9ff187193fbd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "490b705be6cc4b97b8780f5c0efb1a65": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4bf6990f9c0d433ea62174922427ba24": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4dbb6b3f0af744a48615613e95b17ca3": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4f2a295af32a4c37bb4e7c0f54dd260c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4fb37a15be414565927ddfcc0d4d4fb7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5095e0b4ebb5469a8c0a6dc7815b5acc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "55babeb0171c42ea8db188dd75a1fdd4": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "568ad1c0514f49d4807e2dd66675f325": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3c6ea794013743c69c3a904232c0eeff",
            "max": 42,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5095e0b4ebb5469a8c0a6dc7815b5acc",
            "value": 42
          }
        },
        "58944b48ea6b4c0a8d7226aee65bc98f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5a44a693c15547469cee55aa5e04c440": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5ae993317a8d4c47869f855a3a36e794": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c57554c008194d7a8eeca458629815f2",
              "IPY_MODEL_3c3c0b98b5a74639a3a5a3646650ddde",
              "IPY_MODEL_fad5f7a390e74a65adf9b8c9d53dbed7"
            ],
            "layout": "IPY_MODEL_1f73419e430144fa953d09321c8e649d"
          }
        },
        "5ba43e22fe794fadb102e910df0d1ed5": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "5cc7ff7931854a6e8253dbff2c004fcb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5cf2500202d340a8b945abc86b3b089f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5d21ca98fcc0462e9f9632cf0e8cab8e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5e6a40f40edb4650993dfebc00a8c460": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5d21ca98fcc0462e9f9632cf0e8cab8e",
            "placeholder": "​",
            "style": "IPY_MODEL_4f2a295af32a4c37bb4e7c0f54dd260c",
            "value": " 127085/127085 [00:00&lt;00:00, 178029.97 examples/s]"
          }
        },
        "6053184dda0e442887d7dba1b56a5018": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f7034525d71e4e08b2efa14ada2d483a",
            "placeholder": "​",
            "style": "IPY_MODEL_c6bae8ba94bf41f487357e1806b16a69",
            "value": "Map: 100%"
          }
        },
        "611b22e7bbc84ba888befa9d13aabf0e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ffaadc561efd45f9b0d8dd525bc06e89",
            "max": 802397,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3eb6f9323dc94ce4994345dabf01c400",
            "value": 802397
          }
        },
        "6183acdb31594796b0c029259e50b5bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da6ad3da9af743878d29d9d35db2adec",
            "placeholder": "​",
            "style": "IPY_MODEL_5ba43e22fe794fadb102e910df0d1ed5",
            "value": " 28.1k/28.1k [00:00&lt;00:00, 498kB/s]"
          }
        },
        "65046d18cbcb40d6a8bb0fce16344dbd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_8feb80f624234837b1bed233d530893b",
              "IPY_MODEL_568ad1c0514f49d4807e2dd66675f325",
              "IPY_MODEL_8236e0f8a2ba4fbba33ca8799ef0a5ba"
            ],
            "layout": "IPY_MODEL_30b9907bf7b944e3bc8c055d41f84c48"
          }
        },
        "680fcda83a6e4e4db4d6c66956b23599": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6811fcf949814149bda262d69b9e8fa0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7023fdcc52d443df9220fa73acab72fb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "71dc356a0e4542759e0464c3ddc75d13": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1eceb71577b046ac8c290a8e1d956890",
              "IPY_MODEL_938ddd09e5b64fdd8ddeab6cac227ea7",
              "IPY_MODEL_6183acdb31594796b0c029259e50b5bb"
            ],
            "layout": "IPY_MODEL_92f9e98dbd63427fab99e8d5d3216424"
          }
        },
        "7f484da3b6704464b7d4aa1cf414a71d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7023fdcc52d443df9220fa73acab72fb",
            "placeholder": "​",
            "style": "IPY_MODEL_12cceb7a61d54346b4b5ec233f6a593c",
            "value": " 1.34M/1.34M [00:00&lt;00:00, 18.2MB/s]"
          }
        },
        "811fa4f4aceb4b2f9f3ca693e32f7653": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fc84e5310f1642239867dc7e994805ad",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8db9c78366844c21a1c10a397c88c8e0",
            "value": 0
          }
        },
        "8236e0f8a2ba4fbba33ca8799ef0a5ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bd6db5af71724722a20ded150573a5b5",
            "placeholder": "​",
            "style": "IPY_MODEL_29b3333bade247d894721aef8111da58",
            "value": " 42.0/42.0 [00:00&lt;00:00, 2.37kB/s]"
          }
        },
        "89e72264b9ea448e88be0393dc7923d9": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8d51f461f2e846e7b30c2accac9b01a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0a669d9abe604ffb8aa1bbdb608c00de",
            "placeholder": "​",
            "style": "IPY_MODEL_cc57c4ca91d74af9ae81b3317982a7ed",
            "value": " 127085/127085 [00:23&lt;00:00, 14715.36 examples/s]"
          }
        },
        "8db9c78366844c21a1c10a397c88c8e0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8ec6e6a54c8e42b99d58f0aac8135c18": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8feb80f624234837b1bed233d530893b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_55babeb0171c42ea8db188dd75a1fdd4",
            "placeholder": "​",
            "style": "IPY_MODEL_c55805daf77f4400a8a5ba6805c54a27",
            "value": "tokenizer_config.json: 100%"
          }
        },
        "92f9e98dbd63427fab99e8d5d3216424": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "93680094a78e4cb699a4a5a13082c3c5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "938ddd09e5b64fdd8ddeab6cac227ea7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bb11eac2ac4f472b926005fded6c84c7",
            "max": 28064,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5cc7ff7931854a6e8253dbff2c004fcb",
            "value": 28064
          }
        },
        "9aba8dc10ced4424a453436d09f6d702": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9ae603fa708248b8a12e9c6f19ee0435": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d10effdc8adb4cacb389dbd62e4cd1c6",
            "max": 778395,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a9214205c6424bb6bb19fcdb98815a16",
            "value": 778395
          }
        },
        "9db9d75ceee042ae85d8f05a89d2828c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_38519f7244574b2cb3512d0cb8f0291d",
              "IPY_MODEL_45a2dd9f82c648f994e91e65eaf008ed",
              "IPY_MODEL_5e6a40f40edb4650993dfebc00a8c460"
            ],
            "layout": "IPY_MODEL_dfee7792702b4357a8e09f346e5b4369"
          }
        },
        "9e14a7de5bb04fb380ade37c60c9a9b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5a44a693c15547469cee55aa5e04c440",
            "placeholder": "​",
            "style": "IPY_MODEL_eae12e4e7956431e9c9064a62a1b2fe2",
            "value": " 0/0 [00:00&lt;?, ?it/s]"
          }
        },
        "a1b60b5d82c84e5daef5ee1a40f6096c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a82849856b1e4a7cb138140fe5f2406c": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a9214205c6424bb6bb19fcdb98815a16": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "aa1bf0038c9b4072ad0841defd97436c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_be2a1aa3688248109d77b1cd7891f61e",
            "max": 127085,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3ff785e3ff8b452998b43520703b8a40",
            "value": 127085
          }
        },
        "abca53ae51e446688335dab63acd15ba": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b0fcf548ecef42cfb2f52ee67a8af95e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e8a927e48bd543c2887761626133e694",
              "IPY_MODEL_cdd5053a948f49f0aca6ca7b48530cc1",
              "IPY_MODEL_3f36a293747a46c38e619a53ae43200e"
            ],
            "layout": "IPY_MODEL_89e72264b9ea448e88be0393dc7923d9"
          }
        },
        "b40b2864c9bc430887b81291e11038c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bb11eac2ac4f472b926005fded6c84c7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd6db5af71724722a20ded150573a5b5": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bd6e89b6c1b24c9ea5571b425b596239": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "be2a1aa3688248109d77b1cd7891f61e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c55805daf77f4400a8a5ba6805c54a27": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c57554c008194d7a8eeca458629815f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_490b705be6cc4b97b8780f5c0efb1a65",
            "placeholder": "​",
            "style": "IPY_MODEL_b40b2864c9bc430887b81291e11038c6",
            "value": "config.json: 100%"
          }
        },
        "c6bae8ba94bf41f487357e1806b16a69": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c83e3e7acba9412c8b53a11aa17350b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_45f7b3c429b742309ddb1dd7491ccc00",
              "IPY_MODEL_9ae603fa708248b8a12e9c6f19ee0435",
              "IPY_MODEL_2de8d676d69e42bea0aa104ca5bf39f0"
            ],
            "layout": "IPY_MODEL_0c33e0cfd4274ef7a81b502f331a3551"
          }
        },
        "cb544e77262141069263bd53273b8a00": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a1b60b5d82c84e5daef5ee1a40f6096c",
            "placeholder": "​",
            "style": "IPY_MODEL_fd86c0c4742746d7888f88ce5a35c510",
            "value": "vocab.json: 100%"
          }
        },
        "cc57c4ca91d74af9ae81b3317982a7ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "cdd5053a948f49f0aca6ca7b48530cc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f032caf44047483fb00170cda80828a0",
            "max": 20985324,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_680fcda83a6e4e4db4d6c66956b23599",
            "value": 20985324
          }
        },
        "cfd82949d8e547a590a5240420ddfa6e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_cb544e77262141069263bd53273b8a00",
              "IPY_MODEL_165d2b86a67f4a3f9a4669fd3caddaa2",
              "IPY_MODEL_7f484da3b6704464b7d4aa1cf414a71d"
            ],
            "layout": "IPY_MODEL_a82849856b1e4a7cb138140fe5f2406c"
          }
        },
        "d10effdc8adb4cacb389dbd62e4cd1c6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "da6ad3da9af743878d29d9d35db2adec": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dfee7792702b4357a8e09f346e5b4369": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e24248d92fe440398ca6f2b4da0fa454": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1fb659b20933484f85d2d26c68ba3b51",
              "IPY_MODEL_611b22e7bbc84ba888befa9d13aabf0e",
              "IPY_MODEL_1bfa0435150b45ad8c5fc3f8735b8f6e"
            ],
            "layout": "IPY_MODEL_93680094a78e4cb699a4a5a13082c3c5"
          }
        },
        "e8a927e48bd543c2887761626133e694": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2e07f53d528c4c458bce0cdb5a23f2ee",
            "placeholder": "​",
            "style": "IPY_MODEL_abca53ae51e446688335dab63acd15ba",
            "value": "train-00000-of-00001.parquet: 100%"
          }
        },
        "eae12e4e7956431e9c9064a62a1b2fe2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f007232de8e6484a87e5c7c5d2914281": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f032caf44047483fb00170cda80828a0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f08365aa0627435cb19ebd895ecaa833": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f7034525d71e4e08b2efa14ada2d483a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f9ad224fe31145f08ef5c46d4fb9d448": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fad5f7a390e74a65adf9b8c9d53dbed7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0db82807074a47d49526b73411b2ba90",
            "placeholder": "​",
            "style": "IPY_MODEL_6811fcf949814149bda262d69b9e8fa0",
            "value": " 1.42k/1.42k [00:00&lt;00:00, 47.8kB/s]"
          }
        },
        "fc84e5310f1642239867dc7e994805ad": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "fd86c0c4742746d7888f88ce5a35c510": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ffaadc561efd45f9b0d8dd525bc06e89": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}