09_5_Data Preparation for BERT__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Data Loading and Text Processing for BERT**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **45** minutes\n",
    "(When using the pre-trained model provided)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this hands-on lab, you will learn essential techniques and steps to prepare your data for training BERT models effectively. BERT (Bidirectional Encoder Representations from Transformers) has revolutionized natural language processing tasks by capturing contextual information from both left and right contexts. To harness the power of BERT, you will cover various topics, including random sample selection, tokenization, vocabulary building, text masking, and data preparation for masked language model (MLM) and next sentence prediction (NSP) tasks. By the end of this project, you will have the skills to preprocess your data and create training-ready inputs for BERT models. Let's dive in and get started!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of contents__\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#Objectives\">Objectives</a></li>\n",
    "    <li>\n",
    "        <a href=\"#Setup\">Setup</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Installing-required-libraries\">Installing required libraries</a></li>\n",
    "            <li><a href=\"#Importing-required-libraries\">Importing required libraries</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Tokenization-and-vocabulary-building\">Tokenization and vocabulary building</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Tokenization\">Tokenization</a></li>\n",
    "            <li><a href=\"#Vocabulary-building\">Vocabulary building</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Text-masking-and-data-preparation-for-BERT\">Text masking and data preparation for BERT</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Text-masking\">Text masking</a></li>\n",
    "            <li><a href=\"#Data-preparation-for-MLM\">Data preparation for MLM</a></li>\n",
    "            <li><a href=\"#Data-preparation-for-NSP\">Data preparation for NSP</a></li>\n",
    "            <li><a href=\"#Finalizing-BERT-inputs\">Finalizing BERT inputs</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Preparing-the-dataset\">Preparing the dataset</a>\n",
    "    </li>\n",
    "    <li>\n",
    "        <a href=\"#Exercises\">Exercises</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Exercise-1---Initializing-the-BERTTokenizer\">Exercise 1 - Initializing the BERTTokenizer</a></li>\n",
    "            <li><a href=\"#Exercise-2---Tokenizing-the-dataset\">Exercise 2 - Tokenizing the dataset</a></li>\n",
    "            <li><a href=\"#Exercise-3---Building-the-vocabulary-with-special-tokens\">Exercise 3 - Building the vocabulary with special tokens</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab, you will be able to:\n",
    "\n",
    "- Understand the process of random sample selection and tokenization\n",
    "- Apply tokenization techniques and build a vocabulary for text processing\n",
    "- Implement text masking and prepare data specifically for BERT models\n",
    "- Gain proficiency in text masking to create masked language model (MLM) training data\n",
    "- Prepare data for next sentence prediction (NSP) training\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing required libraries\n",
    "\n",
    "The following required libraries are __not__ pre-installed in the Skills Network Labs environment. __You will need to run the following cell__ to install them:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: portalocker>=2.0.0 in /opt/conda/lib/python3.11/site-packages (3.0.0)\n",
      "Requirement already satisfied: torchtext==0.16.0 in /opt/conda/lib/python3.11/site-packages (0.16.0)\n",
      "Requirement already satisfied: tqdm in /opt/conda/lib/python3.11/site-packages (from torchtext==0.16.0) (4.66.4)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from torchtext==0.16.0) (2.31.0)\n",
      "Requirement already satisfied: torch==2.1.0 in /opt/conda/lib/python3.11/site-packages (from torchtext==0.16.0) (2.1.0)\n",
      "Requirement already satisfied: numpy in /opt/conda/lib/python3.11/site-packages (from torchtext==0.16.0) (1.26.4)\n",
      "Requirement already satisfied: torchdata==0.7.0 in /opt/conda/lib/python3.11/site-packages (from torchtext==0.16.0) (0.7.0)\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (3.16.1)\n",
      "Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (4.12.2)\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (1.13.1)\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (3.4.2)\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (3.1.3)\n",
      "Requirement already satisfied: fsspec in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (2024.10.0)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.105)\n",
      "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (8.9.2.26)\n",
      "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.3.1)\n",
      "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (11.0.2.54)\n",
      "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (10.3.2.106)\n",
      "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (11.4.5.107)\n",
      "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.0.106)\n",
      "Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (2.18.1)\n",
      "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (12.1.105)\n",
      "Requirement already satisfied: triton==2.1.0 in /opt/conda/lib/python3.11/site-packages (from torch==2.1.0->torchtext==0.16.0) (2.1.0)\n",
      "Requirement already satisfied: urllib3>=1.25 in /opt/conda/lib/python3.11/site-packages (from torchdata==0.7.0->torchtext==0.16.0) (2.2.1)\n",
      "Requirement already satisfied: nvidia-nvjitlink-cu12 in /opt/conda/lib/python3.11/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.0->torchtext==0.16.0) (12.4.127)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.16.0) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.16.0) (3.7)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->torchtext==0.16.0) (2024.8.30)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.11/site-packages (from jinja2->torch==2.1.0->torchtext==0.16.0) (2.1.5)\n",
      "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/lib/python3.11/site-packages (from sympy->torch==2.1.0->torchtext==0.16.0) (1.3.0)\n",
      "Collecting transformers==4.39.1\n",
      "  Downloading transformers-4.39.1-py3-none-any.whl.metadata (134 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: filelock in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (3.16.1)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (0.26.3)\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (1.26.4)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (24.0)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (2024.11.6)\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (2.31.0)\n",
      "Collecting tokenizers<0.19,>=0.14 (from transformers==4.39.1)\n",
      "  Downloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (0.4.5)\n",
      "Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.11/site-packages (from transformers==4.39.1) (4.66.4)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.39.1) (2024.10.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.11/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.39.1) (4.12.2)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests->transformers==4.39.1) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests->transformers==4.39.1) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests->transformers==4.39.1) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests->transformers==4.39.1) (2024.8.30)\n",
      "Downloading transformers-4.39.1-py3-none-any.whl (8.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.8/8.8 MB\u001b[0m \u001b[31m86.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading tokenizers-0.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m91.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: tokenizers, transformers\n",
      "  Attempting uninstall: tokenizers\n",
      "    Found existing installation: tokenizers 0.20.3\n",
      "    Uninstalling tokenizers-0.20.3:\n",
      "      Successfully uninstalled tokenizers-0.20.3\n",
      "  Attempting uninstall: transformers\n",
      "    Found existing installation: transformers 4.46.3\n",
      "    Uninstalling transformers-4.46.3:\n",
      "      Successfully uninstalled transformers-4.46.3\n",
      "Successfully installed tokenizers-0.15.2 transformers-4.39.1\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.11/site-packages (2.2.1)\n",
      "Requirement already satisfied: numpy<2,>=1.23.2 in /opt/conda/lib/python3.11/site-packages (from pandas) (1.26.4)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.11/site-packages (from pandas) (2.9.0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.11/site-packages (from pandas) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /opt/conda/lib/python3.11/site-packages (from pandas) (2024.2)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "! pip install 'portalocker>=2.0.0'\n",
    "! pip install 'torchtext==0.16.0'\n",
    "! pip install transformers==4.39.1\n",
    "! pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing required libraries\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In this section, you will import the necessary libraries and modules to prepare your dataset for training with PyTorch. The focus is on text processing and creating data loaders that will be used for training your models.\n",
    "\n",
    "- **DataLoader**: A PyTorch utility that allows you to load data in batches, making it easier to manage large datasets during training.\n",
    "- **build_vocab_from_iterator**: A function from `torchtext.vocab` that creates a vocabulary object from an iterator. The vocabulary is crucial for text processing, as it maps tokens (words) to integers.\n",
    "- **Vocab**: Represents the vocabulary, a mapping of tokens to indices. This is used to convert text data into a numerical form that the model can understand.\n",
    "- **Tensor, torch, nn, Transformer**: Core PyTorch modules and classes for tensors (the fundamental data structure in PyTorch), neural network layers, and the Transformer model architecture.\n",
    "- **get_tokenizer**: A function from `torchtext.data.utils` that returns a tokenizer to convert text strings into token lists.\n",
    "- **pad_sequence**: A utility from `torch.nn.utils.rnn` that pads sequences to the same length, a common requirement for batch processing in models that deal with variable-length sequences.\n",
    "\n",
    "This setup is essential for processing text data, converting it into numerical format, and preparing batches of data for training neural networks, particularly for tasks like sequence modeling and classification.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from torchtext.vocab import Vocab\n",
    "from torch import Tensor\n",
    "from torch.nn import Transformer\n",
    "from torchtext.data.utils import get_tokenizer\n",
    "from torch.nn.utils.rnn import pad_sequence\n",
    "from itertools import chain\n",
    "from itertools import islice\n",
    "from torchtext.datasets import IMDB\n",
    "from copy import deepcopy\n",
    "import random\n",
    "import csv\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "import pandas as pd\n",
    "\n",
    "# You can also use this section to suppress warnings generated by your code:\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization and vocabulary building\n",
    "\n",
    "In this section, you will define functions and set up necessary components for text processing, crucial for preparing your dataset for model training.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "- The `tokenizer` is initialized to tokenize text using basic English tokenization rules, converting text samples into lists of tokens.\n",
    "- `yield_tokens` is a generator function that iterates through the data, yielding tokenized versions of the text samples. This function facilitates vocabulary building by providing a stream of tokens.\n",
    "- `word_dict` defines special tokens used in text processing, such as padding `[PAD]`, class `[CLS]`, separator `[SEP]`, mask `[MASK]`, and unknown `[UNK]` tokens, with their corresponding indices.\n",
    "- Special symbols and their indices are explicitly defined for clarity and used throughout data preparation.\n",
    "- `text_to_index` and `index_to_en` functions are utility converters. The former converts text into a list of numerical indices based on the vocabulary, and the latter reverses this process, translating a sequence of indices back into readable English text.\n",
    "\n",
    "- **`CLS` (Classification Token)**: This token serves as the Start of Sentence (SOS) marker. It represents the overall meaning of the entire sentence. Commonly used in tasks that require understanding the entire input, like classification.\n",
    "\n",
    "- **`SEP` (Separator Token)**: Used as the End of Sentence (EOS) marker. It also acts as a delimiter in scenarios where a model needs to understand and differentiate between multiple sentences, like in question-answering or sentence-pair tasks.\n",
    "\n",
    "- **`PAD` (Padding Token)**: This token is added to sequences to ensure all inputs are of equal length. During training, it's important to note that the `[PAD]` token, typically with an ID of 0, does not contribute to the gradient calculations.\n",
    "\n",
    "- **`MASK` (Masked Token)**: Utilized for word replacement in tasks like masked language modeling. It allows models to predict the identity of masked-out words, facilitating learning of bidirectional representations.\n",
    "\n",
    "- **`UNK` (Unknown Token)**: Acts as a placeholder for words that are not found in the tokenizer's vocabulary. This token replaces any unknown or out-of-vocabulary item in the input data.\n",
    "\n",
    "These components are foundational for preprocessing text data, ensuring it is in the correct format for model training, including tokenization, numerical conversion, and handling special tokens necessary for models like BERT.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = get_tokenizer(\"basic_english\")\n",
    "\n",
    "def yield_tokens(data_iter):\n",
    "    for label, data_sample in data_iter:\n",
    "        yield tokenizer(data_sample)\n",
    "\n",
    "# Define special symbols and indices\n",
    "PAD_IDX,CLS_IDX, SEP_IDX,  MASK_IDX,UNK_IDX= 0, 1, 2, 3, 4\n",
    "\n",
    "# Make sure the tokens are in order of their indices to properly insert them in vocab\n",
    "special_symbols = ['[PAD]','[CLS]', '[SEP]','[MASK]','[UNK]']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vocabulary building\n",
    "This section focuses on building the vocabulary from the IMDB dataset.\n",
    "- You can utilize the `IMDB` dataset from `torchtext.datasets`, splitting it into training and testing sets.\n",
    "- The vocabulary is built using the `build_vocab_from_iterator` function, incorporating special symbols (`[PAD]`, `[CLS]`, `[SEP]`, `[MASK]`, `[UNK]`) at the beginning.\n",
    "- The `UNK_IDX` is set as the default index for unknown words, and the total vocabulary size is printed.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'would', 'put', 'this', 'at', 'the', 'top', 'of', 'my', 'list', 'of', 'films', 'in', 'the', 'category', 'of', 'unwatchable', 'trash', '!', 'there']\n"
     ]
    }
   ],
   "source": [
    "# create data splits\n",
    "train_iter, test_iter = IMDB(split=('train', 'test'))\n",
    "all_data_iter = chain(train_iter, test_iter)\n",
    "\n",
    "# check tokenizer\n",
    "# list(yield_tokens(all_data_iter))[5][:20]\n",
    "fifth_item_tokens = next(islice(yield_tokens(all_data_iter), 5, None))\n",
    "print(fifth_item_tokens[:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "124179\n"
     ]
    }
   ],
   "source": [
    "# create vocab : vocab is only built using train data\n",
    "vocab=build_vocab_from_iterator(yield_tokens(all_data_iter),specials=special_symbols,special_first=True)\n",
    "\n",
    "vocab.set_default_index(UNK_IDX)\n",
    "VOCAB_SIZE=len(vocab)\n",
    "print(VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now, create functions that transform token indices to token texts and vice versa. You will use them later on.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_to_index=lambda text: [vocab(token) for token in tokenizer(text)]\n",
    "index_to_en = lambda seq_en: \" \".join([vocab.get_itos()[index] for index in seq_en])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the mappings:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ". i windy\n",
      "[16, 12, 148, 119, 10566, 121, 24, 899, 938, 6, 1106, 7, 92, 41, 89, 12, 27, 1772, 6]\n"
     ]
    }
   ],
   "source": [
    "seq_en = [0, 1, 2, 3, 4, 5, 6]  # Example input sequence\n",
    "english_sentence = index_to_en(seq_en)\n",
    "seq2=[6,16,26131]\n",
    "english_sentence = index_to_en(seq2)\n",
    "\n",
    "print(english_sentence)\n",
    "\n",
    "text = \"I've seen R-rated films with male nudity. Nowhere, because they don't exist.\"  # Example input text\n",
    "text_to_index = lambda text: [vocab[token] for token in tokenizer(text)]\n",
    "index_sequence = text_to_index(text)\n",
    "\n",
    "print(index_sequence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text masking and data preparation for BERT\n",
    "\n",
    "This section introduces functions for preparing data for BERT's Masked Language Model (MLM) and Next Sentence Prediction (NSP) tasks, crucial steps for fine-tuning BERT for specific NLP tasks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text masking\n",
    "\n",
    "The `Masking` function applies BERT's MLM strategy, deciding whether each token in a sequence should be masked, left unchanged, or replaced with a random token. This process is essential for training the model to predict masked words based on their context.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, define a function that returns random 0/1 from bernouli distribution for random sampling.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bernoulli_true_false(p):\n",
    "    # Create a Bernoulli distribution with probability p\n",
    "    bernoulli_dist = torch.distributions.Bernoulli(torch.tensor([p]))\n",
    "    # Sample from this distribution and convert 1 to True and 0 to False\n",
    "    return bernoulli_dist.sample().item() == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, define the masking function:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Masking(token):\n",
    "    # Decide whether to mask this token (20% chance)\n",
    "    mask = bernoulli_true_false(0.2)\n",
    "\n",
    "    # If mask is False, immediately return with '[PAD]' label\n",
    "    if not mask:\n",
    "        return token, '[PAD]'\n",
    "\n",
    "    # If mask is True, proceed with further operations\n",
    "    # Randomly decide on an operation (50% chance each)\n",
    "    random_opp = bernoulli_true_false(0.5)\n",
    "    random_swich = bernoulli_true_false(0.5)\n",
    "\n",
    "    # Case 1: If mask, random_opp, and random_swich are True\n",
    "    if mask and random_opp and random_swich:\n",
    "        # Replace the token with '[MASK]' and set label to a random token\n",
    "        mask_label = index_to_en(torch.randint(0, VOCAB_SIZE, (1,)))\n",
    "        token_ = '[MASK]'\n",
    "\n",
    "    # Case 2: If mask and random_opp are True, but random_swich is False\n",
    "    elif mask and random_opp and not random_swich:\n",
    "        # Leave the token unchanged and set label to the same token\n",
    "        token_ = token\n",
    "        mask_label = token\n",
    "\n",
    "    # Case 3: If mask is True, but random_opp is False\n",
    "    else:\n",
    "        # Replace the token with '[MASK]' and set label to the original token\n",
    "        token_ = '[MASK]'\n",
    "        mask_label = token\n",
    "\n",
    "    return token_, mask_label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check how the random masking startegy works:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[MASK] apple \t Actual token *apple* is masked with '[MASK]'\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "[MASK] grzegorz \t Actual token *apple* is replaced with random token #grzegorz#\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n",
      "apple [PAD] \t Actual token *apple* is left unchanged\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(100)\n",
    "for l in range(10):\n",
    "  token=\"apple\"\n",
    "  token_,label=Masking(token)\n",
    "  if token==token_ and label==\"[PAD]\":\n",
    "    print(token_,label,f\"\\t Actual token *{token}* is left unchanged\")\n",
    "  elif token_==\"[MASK]\" and label==token:\n",
    "    print(token_,label,f\"\\t Actual token *{token}* is masked with '{token_}'\")\n",
    "  else:\n",
    "    print(token_,label,f\"\\t Actual token *{token}* is replaced with random token #{label}#\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation for MLM\n",
    "\n",
    "`prepare_for_mlm` prepares tokenized text for MLM training by applying the masking strategy. It returns sequences of masked tokens along with their corresponding labels, optionally including the original (raw) tokens for reference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_for_mlm(tokens, include_raw_tokens=False):\n",
    "    \"\"\"\n",
    "    Prepares tokenized text for BERT's Masked Language Model (MLM) training.\n",
    "\n",
    "    \"\"\"\n",
    "    bert_input = []  # List to store sentences processed for BERT's MLM\n",
    "    bert_label = []  # List to store labels for each token (mask, random, or unchanged)\n",
    "    raw_tokens_list = []  # List to store raw tokens if needed\n",
    "    current_bert_input = []\n",
    "    current_bert_label = []\n",
    "    current_raw_tokens = []\n",
    "\n",
    "    for token in tokens:\n",
    "        # Apply BERT's MLM masking strategy to the token\n",
    "        masked_token, mask_label = Masking(token)\n",
    "\n",
    "        # Append the processed token and its label to the current sentence and label list\n",
    "        current_bert_input.append(masked_token)\n",
    "        current_bert_label.append(mask_label)\n",
    "\n",
    "        # If raw tokens are to be included, append the original token to the current raw tokens list\n",
    "        if include_raw_tokens:\n",
    "            current_raw_tokens.append(token)\n",
    "\n",
    "        # Check if the token is a sentence delimiter (., ?, !)\n",
    "        if token in ['.', '?', '!']:\n",
    "            # If current sentence has more than two tokens, consider it a valid sentence\n",
    "            if len(current_bert_input) > 2:\n",
    "                bert_input.append(current_bert_input)\n",
    "                bert_label.append(current_bert_label)\n",
    "                # If including raw tokens, add the current list of raw tokens to the raw tokens list\n",
    "                if include_raw_tokens:\n",
    "                    raw_tokens_list.append(current_raw_tokens)\n",
    "\n",
    "                # Reset the lists for the next sentence\n",
    "                current_bert_input = []\n",
    "                current_bert_label = []\n",
    "                current_raw_tokens = []\n",
    "            else:\n",
    "                # If the current sentence is too short, discard it and reset lists\n",
    "                current_bert_input = []\n",
    "                current_bert_label = []\n",
    "                current_raw_tokens = []\n",
    "\n",
    "    # Add any remaining tokens as a sentence if there are any\n",
    "    if current_bert_input:\n",
    "        bert_input.append(current_bert_input)\n",
    "        bert_label.append(current_bert_label)\n",
    "        if include_raw_tokens:\n",
    "            raw_tokens_list.append(current_raw_tokens)\n",
    "\n",
    "    # Return the prepared lists for BERT's MLM training\n",
    "    return (bert_input, bert_label, raw_tokens_list) if include_raw_tokens else (bert_input, bert_label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's check how MLM preparations transform the raw input into the input ready for training:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Without raw tokens: \t  \n",
      " \t original_input is: \t  The sun sets behind the distant mountains. \n",
      " \t bert_input is: \t  [['[MASK]', 'sun', 'sets', 'behind', 'the', '[MASK]', 'mountains', '.']] \n",
      " \t bert_label is: \t  [['the', '[PAD]', '[PAD]', '[PAD]', '[PAD]', 'grzegorz', '[PAD]', '[PAD]']]\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "With raw tokens: \t  \n",
      " \t original_input is: \t  The sun sets behind the distant mountains. \n",
      " \t bert_input is: \t  [['[MASK]', 'sun', 'sets', 'behind', 'the', '[MASK]', 'mountains', '.']] \n",
      " \t bert_label is: \t  [['the', '[PAD]', '[PAD]', '[PAD]', '[PAD]', 'grzegorz', '[PAD]', '[PAD]']] \n",
      " \t raw_tokens_list is: \t  [['the', 'sun', 'sets', 'behind', 'the', 'distant', 'mountains', '.']]\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(100)\n",
    "original_input=\"The sun sets behind the distant mountains.\"\n",
    "tokens=tokenizer(original_input)\n",
    "bert_input, bert_label= prepare_for_mlm(tokens, include_raw_tokens=False)\n",
    "print(\"Without raw tokens: \\t \",\"\\n \\t original_input is: \\t \", original_input,\"\\n \\t bert_input is: \\t \", bert_input,\"\\n \\t bert_label is: \\t \", bert_label)\n",
    "print(\"-\"*200)\n",
    "torch.manual_seed(100)\n",
    "bert_input, bert_label, raw_tokens_list= prepare_for_mlm(tokens, include_raw_tokens=True)\n",
    "print(\"With raw tokens: \\t \",\"\\n \\t original_input is: \\t \", original_input,\"\\n \\t bert_input is: \\t \", bert_input,\"\\n \\t bert_label is: \\t \", bert_label,\"\\n \\t raw_tokens_list is: \\t \", raw_tokens_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, each token in a sentence is labeled, depending on the masking operation that is applied on that token. In this example, the first \"the\" is **masked**, therefore, bert_input is [MASK] and its bert_label is 'The'. Tokens 'sun', 'sets', 'behind' and the last 'the' are not changed, so their corresponding labels are [PAD]. \"distant\" is **masked and replaced with a random token**, therefore, bert_input is [MASK] and its bert_label is 'human-scaled'. Finally, 'mountains' and '.' are **unchanged** so their corresponding labels are [PAD].\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preparation for NSP\n",
    "\n",
    "`process_for_nsp` prepares data for the NSP task by creating pairs of sentences. It labels these pairs to indicate whether the second sentence is the subsequent sentence in the original text, facilitating the model's learning of sentence relationships.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_for_nsp(input_sentences, input_masked_labels):\n",
    "    \"\"\"\n",
    "    Prepares data for Next Sentence Prediction (NSP) task in BERT training.\n",
    "\n",
    "    Args:\n",
    "    input_sentences (list): List of tokenized sentences.\n",
    "    input_masked_labels (list): Corresponding list of masked labels for the sentences.\n",
    "\n",
    "    Returns:\n",
    "    bert_input (list): List of sentence pairs for BERT input.\n",
    "    bert_label (list): List of masked labels for the sentence pairs.\n",
    "    is_next (list): Binary label list where 1 indicates 'next sentence' and 0 indicates 'not next sentence'.\n",
    "    \"\"\"\n",
    "    if len(input_sentences) < 2:\n",
    "       raise ValueError(\"must have two same number of items.\")\n",
    "\n",
    "\n",
    "    # Verify that both input lists are of the same length and have a sufficient number of sentences\n",
    "    if len(input_sentences) != len(input_masked_labels):\n",
    "        raise ValueError(\"Both lists must have the same number of items.\")\n",
    "\n",
    "    bert_input = []\n",
    "    bert_label = []\n",
    "    is_next = []\n",
    "\n",
    "    available_indices = list(range(len(input_sentences)))\n",
    "\n",
    "    while len(available_indices) >= 2:\n",
    "        if random.random() < 0.5:\n",
    "            # Choose two consecutive sentences to simulate the 'next sentence' scenario\n",
    "            index = random.choice(available_indices[:-1])  # Exclude the last index\n",
    "            # append list and add  '[CLS]' and  '[SEP]' tokens\n",
    "            bert_input.append([['[CLS]']+input_sentences[index]+ ['[SEP]'],input_sentences[index + 1]+ ['[SEP]']])\n",
    "            bert_label.append([['[PAD]']+input_masked_labels[index]+['[PAD]'], input_masked_labels[index + 1]+ ['[PAD]']])\n",
    "            is_next.append(1)  # Label 1 indicates these sentences are consecutive\n",
    "\n",
    "            # Remove the used indices\n",
    "            available_indices.remove(index)\n",
    "            if index + 1 in available_indices:\n",
    "                available_indices.remove(index + 1)\n",
    "        else:\n",
    "            # Choose two random distinct sentences to simulate the 'not next sentence' scenario\n",
    "            indices = random.sample(available_indices, 2)\n",
    "            bert_input.append([['[CLS]']+input_sentences[indices[0]]+['[SEP]'],input_sentences[indices[1]]+ ['[SEP]']])\n",
    "            bert_label.append([['[PAD]']+input_masked_labels[indices[0]]+['[PAD]'], input_masked_labels[indices[1]]+['[PAD]']])\n",
    "            is_next.append(0)  # Label 0 indicates these sentences are not consecutive\n",
    "\n",
    "            # Remove the used indices\n",
    "            available_indices.remove(indices[0])\n",
    "            available_indices.remove(indices[1])\n",
    "\n",
    "\n",
    "\n",
    "    return bert_input, bert_label, is_next"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look into some sample input sentences and create NSP pairs:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BERT Input:\n",
      "[['[CLS]', 'she', 'enjoys', 'reading', 'books', '[SEP]'], ['he', 'likes', 'playing', 'guitar', '[SEP]']]\n",
      "BERT Label:\n",
      "[['[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], ['he', '[PAD]', '[PAD]', '[PAD]', '[PAD]']]\n",
      "Is Next:  [1]\n",
      "--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "BERT Input:\n",
      "[['[CLS]', 'he', 'likes', 'playing', 'guitar', '[SEP]'], ['i', 'love', 'apples', '[SEP]']]\n",
      "BERT Label:\n",
      "[['[PAD]', 'he', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], ['[PAD]', '[PAD]', '[PAD]', '[PAD]']]\n",
      "Is Next:  [0]\n"
     ]
    }
   ],
   "source": [
    "# flatten the tensor\n",
    "flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "# Sample input sentences\n",
    "input_sentences = [[\"i\", \"love\", \"apples\"], [\"she\", \"enjoys\", \"reading\", \"books\"], [\"he\", \"likes\", \"playing\", \"guitar\"]]\n",
    "# Create masked labels for the sentences\n",
    "input_masked_labels=[]\n",
    "for sentence in input_sentences:\n",
    "  _, current_masked_label= prepare_for_mlm(sentence, include_raw_tokens=False)\n",
    "  input_masked_labels.append(flatten(current_masked_label))\n",
    "# Create NSP pairs and labels\n",
    "random.seed(100)\n",
    "bert_input, bert_label, is_next = process_for_nsp(input_sentences, input_masked_labels)\n",
    "\n",
    "# Print the output\n",
    "print(\"BERT Input:\")\n",
    "for pair in bert_input:\n",
    "    print(pair)\n",
    "print(\"BERT Label:\")\n",
    "for pair in bert_label:\n",
    "    print(pair)\n",
    "print(\"Is Next: \", is_next)\n",
    "print(\"-\"*200)\n",
    "random.seed(1000)\n",
    "bert_input, bert_label, is_next = process_for_nsp(input_sentences, input_masked_labels)\n",
    "\n",
    "# Print the output\n",
    "print(\"BERT Input:\")\n",
    "for pair in bert_input:\n",
    "    print(pair)\n",
    "print(\"BERT Label:\")\n",
    "for pair in bert_label:\n",
    "    print(pair)\n",
    "print(\"Is Next: \", is_next)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two examples demonstrate how sentence pairs are randomly created from the sentence bank and labeled for NSP task. Special symbols [CLS] and [SEP] are first added to the input sentences. BERT label is created using the `prepare_for_mlm` function. In the first example, the second sentence follows the first sentence. Therefore, `Is Next` is 1. In the second example, the second sentence does not follow the first sentence. So, `Is Next` is 0. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finalizing BERT inputs\n",
    "\n",
    "`prepare_bert_final_inputs` consolidates the prepared data for MLM and NSP into a format suitable for BERT training, including converting tokens to indices, padding sequences for uniform length, and generating segment labels to distinguish between pairs of sentences. This function is the final step in preparing data for BERT, ensuring it is in the correct format for effective model training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_bert_final_inputs(bert_inputs, bert_labels, is_nexts,to_tenor=True):\n",
    "    \"\"\"\n",
    "    Prepare the final input lists for BERT training.\n",
    "    \"\"\"\n",
    "    def zero_pad_list_pair(pair_, pad='[PAD]'):\n",
    "        pair=deepcopy(pair_)\n",
    "        max_len = max(len(pair[0]), len(pair[1]))\n",
    "        # append [PAD] to each sentence in the pair till the maximum length reaches\n",
    "        pair[0].extend([pad] * (max_len - len(pair[0])))\n",
    "        pair[1].extend([pad] * (max_len - len(pair[1])))\n",
    "        return pair[0], pair[1]\n",
    "\n",
    "    # flatten the tensor\n",
    "    flatten = lambda l: [item for sublist in l for item in sublist]\n",
    "    # transform tokens to vocab indices\n",
    "    tokens_to_index=lambda tokens: [vocab[token] for token in tokens]\n",
    "\n",
    "    bert_inputs_final, bert_labels_final, segment_labels_final, is_nexts_final = [], [], [], []\n",
    "\n",
    "    for bert_input, bert_label,is_next in zip(bert_inputs, bert_labels,is_nexts):\n",
    "        # Create segment labels for each pair of sentences\n",
    "        segment_label = [[1] * len(bert_input[0]), [2] * len(bert_input[1])]\n",
    "\n",
    "        # Zero-pad the bert_input and bert_label and segment_label\n",
    "        bert_input_padded = zero_pad_list_pair(bert_input)\n",
    "        bert_label_padded = zero_pad_list_pair(bert_label)\n",
    "        segment_label_padded = zero_pad_list_pair(segment_label,pad=0)\n",
    "\n",
    "        # convert to tensors\n",
    "        if to_tenor:\n",
    "\n",
    "            # Flatten the padded inputs and labels, transform tokens to their corresponding vocab indices, and convert them to tensors\n",
    "            bert_inputs_final.append(torch.tensor(tokens_to_index(flatten(bert_input_padded)),dtype=torch.int64))\n",
    "            # bert_labels_final.append(torch.tensor(tokens_to_index(flatten(bert_label_padded)),dtype=torch.int64))\n",
    "            bert_labels_final.append(torch.tensor(tokens_to_index(flatten(bert_label_padded)),dtype=torch.int64))\n",
    "            segment_labels_final.append(torch.tensor(flatten(segment_label_padded),dtype=torch.int64))\n",
    "            is_nexts_final.append(is_next)\n",
    "\n",
    "        else:\n",
    "          # Flatten the padded inputs and labels\n",
    "            bert_inputs_final.append(flatten(bert_input_padded))\n",
    "            bert_labels_final.append(flatten(bert_label_padded))\n",
    "            segment_labels_final.append(flatten(segment_label_padded))\n",
    "            is_nexts_final.append(is_next)\n",
    "\n",
    "    return bert_inputs_final, bert_labels_final, segment_labels_final, is_nexts_final\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the results using the `bert_input`, `bert_label` and `is_next` from previous example:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input:\t\t [[['[CLS]', 'he', 'likes', 'playing', 'guitar', '[SEP]'], ['i', 'love', 'apples', '[SEP]']]] \n",
      "inputs_final:\t [tensor([    1,    35,  1139,   418,  4700,     2,    16,   135, 13793,     2,     0,     0])] \n",
      "bert labels final:\t [tensor([ 0, 35,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0])] \n",
      "segment labels final:\t [tensor([1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 0, 0])] \n",
      "is nexts final:\t [0]\n"
     ]
    }
   ],
   "source": [
    "bert_inputs_final, bert_labels_final, segment_labels_final, is_nexts_final=prepare_bert_final_inputs(bert_input, bert_label, is_next,to_tenor=True)\n",
    "torch.set_printoptions(linewidth=10000) # this assures that whole output is printed in one line\n",
    "print(\"input:\\t\\t\",bert_input,\"\\ninputs_final:\\t\",bert_inputs_final,\"\\nbert labels final:\\t\",bert_labels_final,\"\\nsegment labels final:\\t\",segment_labels_final,\"\\nis nexts final:\\t\",is_nexts_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentences are zero-padded and each token is mapped to its vocab index(`[CLS]`>>1, `he`>>33, ..., `[SEP]`>>2,`[PAD]`>>0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Mask labels are also padded and mapped to vocab indices. In this case, all tokens are **unchanged** except the token, `he` which is masked:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input:\t\t [[['[CLS]', 'he', 'likes', 'playing', 'guitar', '[SEP]'], ['i', 'love', 'apples', '[SEP]']]] \n",
      "mask_label:\t [[['[PAD]', 'he', '[PAD]', '[PAD]', '[PAD]', '[PAD]'], ['[PAD]', '[PAD]', '[PAD]', '[PAD]']]] \n",
      "labels_final: \t [tensor([ 0, 35,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0])]\n"
     ]
    }
   ],
   "source": [
    "print(\"input:\\t\\t\",bert_input,\"\\nmask_label:\\t\",bert_label, \"\\nlabels_final: \\t\",bert_labels_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, segment labels are created, where tokens of the first sentence are labeled with 1, tokens of the second sentence are labeled with 2 and zero-paddings are labeled with 0.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "inputs_final:\t [tensor([    1,    35,  1139,   418,  4700,     2,    16,   135, 13793,     2,     0,     0])] \n",
      "segment_labels:\t [tensor([1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 0, 0])]\n"
     ]
    }
   ],
   "source": [
    "print(\"\\ninputs_final:\\t\",bert_inputs_final,\"\\nsegment_labels:\\t\",segment_labels_final)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the dataset\n",
    "\n",
    "- A CSV file is created to store the data set prepared for BERT training and testing. Each row contains the original text, BERT inputs, labels, segment labels, and the NSP task label.\n",
    "- The data from the IMDB data set is tokenized, processed for MLM, and then for NSP. The results are formatted and written to the CSV file, providing a comprehensive data set for BERT model training.\n",
    "\n",
    "This process is critical for ensuring the data is in the right format for effective training of BERT on the IMDB data set, focusing on understanding text context and relationships between sentences.\n",
    "\n",
    ">This training process might take about 2 to 3 hours.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing samples: 11016it [47:04,  3.90it/s]\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[19], line 11\u001b[0m\n\u001b[1;32m      9\u001b[0m tokens \u001b[38;5;241m=\u001b[39m tokenizer(sample)\n\u001b[1;32m     10\u001b[0m \u001b[38;5;66;03m# Create MLM inputs and labels\u001b[39;00m\n\u001b[0;32m---> 11\u001b[0m bert_input, bert_label \u001b[38;5;241m=\u001b[39m \u001b[43mprepare_for_mlm\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtokens\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43minclude_raw_tokens\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m)\u001b[49m\n\u001b[1;32m     12\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(bert_input) \u001b[38;5;241m<\u001b[39m \u001b[38;5;241m2\u001b[39m:\n\u001b[1;32m     13\u001b[0m     \u001b[38;5;28;01mcontinue\u001b[39;00m\n",
      "Cell \u001b[0;32mIn[11], line 15\u001b[0m, in \u001b[0;36mprepare_for_mlm\u001b[0;34m(tokens, include_raw_tokens)\u001b[0m\n\u001b[1;32m     11\u001b[0m current_raw_tokens \u001b[38;5;241m=\u001b[39m []\n\u001b[1;32m     13\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m token \u001b[38;5;129;01min\u001b[39;00m tokens:\n\u001b[1;32m     14\u001b[0m     \u001b[38;5;66;03m# Apply BERT's MLM masking strategy to the token\u001b[39;00m\n\u001b[0;32m---> 15\u001b[0m     masked_token, mask_label \u001b[38;5;241m=\u001b[39m \u001b[43mMasking\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtoken\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     17\u001b[0m     \u001b[38;5;66;03m# Append the processed token and its label to the current sentence and label list\u001b[39;00m\n\u001b[1;32m     18\u001b[0m     current_bert_input\u001b[38;5;241m.\u001b[39mappend(masked_token)\n",
      "Cell \u001b[0;32mIn[9], line 17\u001b[0m, in \u001b[0;36mMasking\u001b[0;34m(token)\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[38;5;66;03m# Case 1: If mask, random_opp, and random_swich are True\u001b[39;00m\n\u001b[1;32m     15\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m mask \u001b[38;5;129;01mand\u001b[39;00m random_opp \u001b[38;5;129;01mand\u001b[39;00m random_swich:\n\u001b[1;32m     16\u001b[0m     \u001b[38;5;66;03m# Replace the token with '[MASK]' and set label to a random token\u001b[39;00m\n\u001b[0;32m---> 17\u001b[0m     mask_label \u001b[38;5;241m=\u001b[39m \u001b[43mindex_to_en\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtorch\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrandint\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mVOCAB_SIZE\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m1\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     18\u001b[0m     token_ \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m[MASK]\u001b[39m\u001b[38;5;124m'\u001b[39m\n\u001b[1;32m     20\u001b[0m \u001b[38;5;66;03m# Case 2: If mask and random_opp are True, but random_swich is False\u001b[39;00m\n",
      "Cell \u001b[0;32mIn[6], line 2\u001b[0m, in \u001b[0;36m<lambda>\u001b[0;34m(seq_en)\u001b[0m\n\u001b[1;32m      1\u001b[0m text_to_index\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mlambda\u001b[39;00m text: [vocab(token) \u001b[38;5;28;01mfor\u001b[39;00m token \u001b[38;5;129;01min\u001b[39;00m tokenizer(text)]\n\u001b[0;32m----> 2\u001b[0m index_to_en \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m seq_en: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin(\u001b[43m[\u001b[49m\u001b[43mvocab\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_itos\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[43mindex\u001b[49m\u001b[43m]\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mfor\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mindex\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01min\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mseq_en\u001b[49m\u001b[43m]\u001b[49m)\n",
      "Cell \u001b[0;32mIn[6], line 2\u001b[0m, in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      1\u001b[0m text_to_index\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mlambda\u001b[39;00m text: [vocab(token) \u001b[38;5;28;01mfor\u001b[39;00m token \u001b[38;5;129;01min\u001b[39;00m tokenizer(text)]\n\u001b[0;32m----> 2\u001b[0m index_to_en \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mlambda\u001b[39;00m seq_en: \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mjoin([\u001b[43mvocab\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_itos\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m[index] \u001b[38;5;28;01mfor\u001b[39;00m index \u001b[38;5;129;01min\u001b[39;00m seq_en])\n",
      "File \u001b[0;32m/opt/conda/lib/python3.11/site-packages/torchtext/vocab/vocab.py:158\u001b[0m, in \u001b[0;36mVocab.get_itos\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    152\u001b[0m \u001b[38;5;129m@torch\u001b[39m\u001b[38;5;241m.\u001b[39mjit\u001b[38;5;241m.\u001b[39mexport\n\u001b[1;32m    153\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mget_itos\u001b[39m(\u001b[38;5;28mself\u001b[39m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m List[\u001b[38;5;28mstr\u001b[39m]:\n\u001b[1;32m    154\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    155\u001b[0m \u001b[38;5;124;03m    Returns:\u001b[39;00m\n\u001b[1;32m    156\u001b[0m \u001b[38;5;124;03m        List mapping indices to tokens.\u001b[39;00m\n\u001b[1;32m    157\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 158\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mvocab\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_itos\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "csv_file_path ='train_bert_data_new.csv'\n",
    "with open(csv_file_path, mode='w', newline='', encoding='utf-8') as file:\n",
    "    csv_writer = csv.writer(file)\n",
    "    csv_writer.writerow(['Original Text', 'BERT Input', 'BERT Label', 'Segment Label', 'Is Next'])\n",
    "\n",
    "    # Wrap train_iter with tqdm for a progress bar\n",
    "    for n, (_, sample) in enumerate(tqdm(train_iter, desc=\"Processing samples\")):\n",
    "        # Tokenize the sample input\n",
    "        tokens = tokenizer(sample)\n",
    "        # Create MLM inputs and labels\n",
    "        bert_input, bert_label = prepare_for_mlm(tokens, include_raw_tokens=False)\n",
    "        if len(bert_input) < 2:\n",
    "            continue\n",
    "        # Create NSP pairs, token labels, and is_next label\n",
    "        bert_inputs, bert_labels, is_nexts = process_for_nsp(bert_input, bert_label)\n",
    "        # add zero-paddings, map tokens to vocab indices and create segment labels\n",
    "        bert_inputs, bert_labels, segment_labels, is_nexts = prepare_bert_final_inputs(bert_inputs, bert_labels, is_nexts)\n",
    "        # convert tensors to lists, convert lists to JSON-formatted strings\n",
    "        for bert_input, bert_label, segment_label, is_next in zip(bert_inputs, bert_labels, segment_labels, is_nexts):\n",
    "            bert_input_str = json.dumps(bert_input.tolist())\n",
    "            bert_label_str = json.dumps(bert_label.tolist())\n",
    "            segment_label_str = ','.join(map(str, segment_label.tolist()))\n",
    "            # Write the data to a CSV file row-by-row\n",
    "            csv_writer.writerow([sample, bert_input_str, bert_label_str, segment_label_str, is_next])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n",
    "\n",
    "Learn to utilize Hugging Face's pre-trained BertTokenizer for text tokenization, including handling special tokens and preparing the IMDB dataset for NLP model training, without manually building a vocabulary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Initializing the BERTTokenizer\n",
    "1. **Import `BertTokenizer`**:\n",
    "   Begin by importing the `BertTokenizer` class from the `transformers` library. This library provides access to a wide range of NLP models and their corresponding tokenizers.\n",
    "\n",
    "2. **Load pretrained tokenizer**:\n",
    "   Utilize the `from_pretrained` method to load the `bert-base-uncased` tokenizer. This tokenizer is pre-configured with a vocabulary that suits the BERT model trained on uncapitalized English text. It's ideal for understanding the basics of BERT tokenization.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import BertTokenizer\n",
    "\n",
    "# Load a pre-trained BERT tokenizer\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from transformers import BertTokenizer\n",
    "\n",
    "# Load a pre-trained BERT tokenizer\n",
    "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - Tokenizing the dataset\n",
    "1. **Define the `yield_tokens` Function**:\n",
    "   Implement a function named `yield_tokens` that accepts an iterator over the dataset. This function is responsible for processing and tokenizing the text data.\n",
    "\n",
    "2. **Tokenize Text Samples**:\n",
    "   Within the `yield_tokens` function, iterate through the dataset. For each text sample, use the `BertTokenizer` to tokenize the text into sequences of token IDs. Make sure to handle longer texts by setting the `truncation` parameter to `True` and specifying a `max_length` to ensure all tokenized outputs are of a manageable size.\n",
    "\n",
    "3. **Yield Tokenized Texts**:\n",
    "   After tokenizing each text sample, yield the list of token IDs. These lists will be used in subsequent steps to build data structures suitable for training NLP models.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def yield_tokens(data_iter):\n",
    "    for _, data_sample in data_iter:\n",
    "        # Use the BERT tokenizer to tokenize the text\n",
    "        # This returns a dictionary with 'input_ids' among other things\n",
    "        tokens = tokenizer(data_sample, return_tensors='pt', truncation=True, max_length=512)['input_ids'][0]\n",
    "        yield tokens.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "def yield_tokens(data_iter):\n",
    "    for _, data_sample in data_iter:\n",
    "        # Use the BERT tokenizer to tokenize the text\n",
    "        # This returns a dictionary with 'input_ids' among other things\n",
    "        tokens = tokenizer(data_sample, return_tensors='pt', truncation=True, max_length=512)['input_ids'][0]\n",
    "        yield tokens.tolist()\n",
    "```\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3 - Building the vocabulary with special tokens\n",
    "1. **Define Special Tokens and Indices**: Start by defining indices for special tokens such as `[PAD]`, `[CLS]`, `[SEP]`, `[MASK]`, and `[UNK]`. Create a list named `special_symbols` that includes these tokens, ensuring they are in the correct order according to their indices.\n",
    "\n",
    "2. **Load Dataset**: Ensure you have the IMDB dataset's training split loaded. This data will be used to build the vocabulary.\n",
    "\n",
    "3. **Build Vocabulary**: Utilize the `build_vocab_from_iterator` function, passing the `yield_tokens` generator function as an argument. This function iterates over the tokenized dataset and builds a vocabulary. Make sure to include the special tokens by specifying them in the `specials` argument of the `build_vocab_from_iterator` function.\n",
    "(Since you are using a pre-trained tokenizer, you don't need to build the vocab from scratch. Instead, you can directly use the tokenizer's vocab.) \n",
    "\n",
    "4. **Set Default Index for Unknown Tokens**: After building the vocabulary, use the `set_default_index` method to specify the index for unknown tokens (`UNK_IDX`). This ensures that any tokens not found in the vocabulary are handled correctly.\n",
    "(Since you are using a pre-trained tokenizer, you don't need to build the vocab from scratch. Instead, you can directly use the tokenizer's vocab.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary Size: 30522\n"
     ]
    }
   ],
   "source": [
    "from torchtext.data.functional import to_map_style_dataset\n",
    "from torchtext.datasets import IMDB\n",
    "\n",
    "# Define special symbols and indices\n",
    "PAD_IDX, CLS_IDX, SEP_IDX, MASK_IDX, UNK_IDX = tokenizer.pad_token_id, tokenizer.cls_token_id, tokenizer.sep_token_id, tokenizer.mask_token_id, tokenizer.unk_token_id\n",
    "special_symbols = ['[PAD]', '[CLS]', '[SEP]', '[MASK]', '[UNK]']\n",
    "\n",
    "# Load IMDB dataset\n",
    "train_iter, test_iter = IMDB(split=('train', 'test'))\n",
    "\n",
    "# Convert to map-style datasets to be compatible with transformers' tokenizers\n",
    "train_dataset = to_map_style_dataset(train_iter)\n",
    "test_dataset = to_map_style_dataset(test_iter)\n",
    "\n",
    "# Since you are using a pre-trained tokenizer, you don't need to build the vocab from scratch. \n",
    "# Instead, you can directly use the tokenizer's vocab.\n",
    "VOCAB_SIZE = len(tokenizer)\n",
    "\n",
    "print(\"Vocabulary Size:\", VOCAB_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "```python\n",
    "from torchtext.data.functional import to_map_style_dataset\n",
    "from torchtext.datasets import IMDB\n",
    "\n",
    "# Define special symbols and indices\n",
    "PAD_IDX, CLS_IDX, SEP_IDX, MASK_IDX, UNK_IDX = tokenizer.pad_token_id, tokenizer.cls_token_id, tokenizer.sep_token_id, tokenizer.mask_token_id, tokenizer.unk_token_id\n",
    "special_symbols = ['[PAD]', '[CLS]', '[SEP]', '[MASK]', '[UNK]']\n",
    "\n",
    "# Load IMDB dataset\n",
    "train_iter, test_iter = IMDB(split=('train', 'test'))\n",
    "\n",
    "# Convert to map-style datasets to be compatible with transformers' tokenizers\n",
    "train_dataset = to_map_style_dataset(train_iter)\n",
    "test_dataset = to_map_style_dataset(test_iter)\n",
    "\n",
    "# Since you are using a pre-trained tokenizer, you don't need to build the vocab from scratch. \n",
    "# Instead, you can directly use the tokenizer's vocab.\n",
    "VOCAB_SIZE = len(tokenizer)\n",
    "\n",
    "print(\"Vocabulary Size:\", VOCAB_SIZE)\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations! You have completed the lab\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Joseph Santarcangelo](https://author.skills.network/instructors/joseph_santarcangelo)\n",
    "\n",
    "Joseph has a Ph.D. in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Ashutosh Sagar](https://www.linkedin.com/in/ashutoshsagar/) is completing his MS in CS from Dalhousie University. He has previous experience working with Natural Language Processing and as a Data Scientist.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "\n",
    "- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf)\n",
    "- [Mastering BERT Model: Building it from Scratch with Pytorch](https://medium.com/data-and-beyond/complete-guide-to-building-bert-model-from-sratch-3e6562228891)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "© Copyright IBM Corporation. All rights reserved.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "prev_pub_hash": "6c8fa70732ed928bd648243a39ecd88f18424f64edfa13622ab688097a48508d"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}