ch06/01_main-chapter-code/ch06.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c024bfa4-1a7a-4751-b5a1-827225a3478b",
   "metadata": {
    "id": "c024bfa4-1a7a-4751-b5a1-827225a3478b"
   },
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfabadb8-5935-45ff-b39c-db7a29012129",
   "metadata": {
    "id": "bfabadb8-5935-45ff-b39c-db7a29012129"
   },
   "source": [
    "# Chapter 6: Finetuning for Text Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5b7e01c2-1c84-4f2a-bb51-2e0b74abda90",
    "outputId": "9495f150-9d79-4910-d6e7-6c0d9aae4a41"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.10.0\n",
      "numpy version: 2.0.2\n",
      "tiktoken version: 0.8.0\n",
      "torch version: 2.6.0\n",
      "tensorflow version: 2.18.0\n",
      "pandas version: 2.2.3\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"matplotlib\",  # Plotting library\n",
    "        \"numpy\",       # PyTorch & TensorFlow dependency\n",
    "        \"tiktoken\",    # Tokenizer\n",
    "        \"torch\",       # Deep learning library\n",
    "        \"tensorflow\",  # For OpenAI's pretrained weights\n",
    "        \"pandas\"       # Dataset loading\n",
    "       ]\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a445828a-ff10-4efa-9f60-a2e2aed4c87d",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/chapter-overview.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "946c3e56-b04b-4b0f-b35f-b485ce5b28df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Utility to prevent certain cells from being executed twice\n",
    "\n",
    "from IPython.core.magic import register_line_cell_magic\n",
    "\n",
    "executed_cells = set()\n",
    "\n",
    "@register_line_cell_magic\n",
    "def run_once(line, cell):\n",
    "    if line not in executed_cells:\n",
    "        get_ipython().run_cell(cell)\n",
    "        executed_cells.add(line)\n",
    "    else:\n",
    "        print(f\"Cell '{line}' has already been executed.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c",
   "metadata": {
    "id": "3a84cf35-b37f-4c15-8972-dfafc9fadc1c"
   },
   "source": [
    "## 6.1 Different categories of finetuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ede3d731-5123-4f02-accd-c670ce50a5a3",
   "metadata": {
    "id": "ede3d731-5123-4f02-accd-c670ce50a5a3"
   },
   "source": [
    "- No code in this section"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac45579d-d485-47dc-829e-43be7f4db57b",
   "metadata": {},
   "source": [
    "- The most common ways to finetune language models are instruction-finetuning and classification finetuning\n",
    "- Instruction-finetuning, depicted below, is the topic of the next chapter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c29ef42-46d9-43d4-8bb4-94974e1665e4",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/instructions.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7f60321-95b8-46a9-97bf-1d07fda2c3dd",
   "metadata": {},
   "source": [
    "- Classification finetuning, the topic of this chapter, is a procedure you may already be familiar with if you have a background in machine learning -- it's similar to training a convolutional network to classify handwritten digits, for example\n",
    "- In classification finetuning, we have a specific number of class labels (for example, \"spam\" and \"not spam\") that the model can output\n",
    "- A classification finetuned model can only predict classes it has seen during training (for example, \"spam\" or \"not spam\"), whereas an instruction-finetuned model can usually perform many tasks\n",
    "- We can think of a classification-finetuned model as a very specialized model; in practice, it is much easier to create a specialized model than a generalist model that performs well on many different tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b37a0c4-0bb1-4061-b1fe-eaa4416d52c3",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/spam-non-spam.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf",
   "metadata": {
    "id": "8c7017a2-32aa-4002-a2f3-12aac293ccdf"
   },
   "source": [
    "## 6.2 Preparing the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f628975-d2e8-4f7f-ab38-92bb868b7067",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-1.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d",
   "metadata": {
    "id": "9fbd459f-63fa-4d8c-8499-e23103156c7d"
   },
   "source": [
    "- This section prepares the dataset we use for classification finetuning\n",
    "- We use a dataset consisting of spam and non-spam text messages to finetune the LLM to classify them\n",
    "- First, we download and unzip the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "def7c09b-af9c-4216-90ce-5e67aed1065c",
    "outputId": "424e4423-f623-443c-ab9e-656f9e867559"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sms_spam_collection/SMSSpamCollection.tsv already exists. Skipping download and extraction.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import zipfile\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "zip_path = \"sms_spam_collection.zip\"\n",
    "extracted_path = \"sms_spam_collection\"\n",
    "data_file_path = Path(extracted_path) / \"SMSSpamCollection.tsv\"\n",
    "\n",
    "def download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path):\n",
    "    if data_file_path.exists():\n",
    "        print(f\"{data_file_path} already exists. Skipping download and extraction.\")\n",
    "        return\n",
    "\n",
    "    # Downloading the file\n",
    "    with urllib.request.urlopen(url) as response:\n",
    "        with open(zip_path, \"wb\") as out_file:\n",
    "            out_file.write(response.read())\n",
    "\n",
    "    # Unzipping the file\n",
    "    with zipfile.ZipFile(zip_path, \"r\") as zip_ref:\n",
    "        zip_ref.extractall(extracted_path)\n",
    "\n",
    "    # Add .tsv file extension\n",
    "    original_file_path = Path(extracted_path) / \"SMSSpamCollection\"\n",
    "    os.rename(original_file_path, data_file_path)\n",
    "    print(f\"File downloaded and saved as {data_file_path}\")\n",
    "\n",
    "download_and_unzip_spam_data(url, zip_path, extracted_path, data_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1",
   "metadata": {
    "id": "6aac2d19-06d0-4005-916b-0bd4b1ee50d1"
   },
   "source": [
    "- The dataset is saved as a tab-separated text file, which we can load into a pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "da0ed4da-ac31-4e4d-8bdd-2153be4656a4",
    "outputId": "a16c5cde-d341-4887-a93f-baa9bec542ab"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ham</td>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>spam</td>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ham</td>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>spam</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>ham</td>\n",
       "      <td>Will ü b going to esplanade fr home?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>ham</td>\n",
       "      <td>Pity, * was in mood for that. So...any other s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>ham</td>\n",
       "      <td>The guy did some bitching but I acted like i'd...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5571</th>\n",
       "      <td>ham</td>\n",
       "      <td>Rofl. Its true to its name</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5572 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Label                                               Text\n",
       "0      ham  Go until jurong point, crazy.. Available only ...\n",
       "1      ham                      Ok lar... Joking wif u oni...\n",
       "2     spam  Free entry in 2 a wkly comp to win FA Cup fina...\n",
       "3      ham  U dun say so early hor... U c already then say...\n",
       "4      ham  Nah I don't think he goes to usf, he lives aro...\n",
       "...    ...                                                ...\n",
       "5567  spam  This is the 2nd time we have tried 2 contact u...\n",
       "5568   ham               Will ü b going to esplanade fr home?\n",
       "5569   ham  Pity, * was in mood for that. So...any other s...\n",
       "5570   ham  The guy did some bitching but I acted like i'd...\n",
       "5571   ham                         Rofl. Its true to its name\n",
       "\n",
       "[5572 rows x 2 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(data_file_path, sep=\"\\t\", header=None, names=[\"Label\", \"Text\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109",
   "metadata": {
    "id": "e7b6e631-4f0b-4aab-82b9-8898e6663109"
   },
   "source": [
    "- When we check the class distribution, we see that the data contains \"ham\" (i.e., \"not spam\") much more frequently than \"spam\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "495a5280-9d7c-41d4-9719-64ab99056d4c",
    "outputId": "761e0482-43ba-4f46-f4b7-6774dae51b38"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     4825\n",
      "spam     747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f773f054-0bdc-4aad-bbf6-397621bf63db",
   "metadata": {
    "id": "f773f054-0bdc-4aad-bbf6-397621bf63db"
   },
   "source": [
    "- For simplicity, and because we prefer a small dataset for educational purposes anyway (it will make it possible to finetune the LLM faster), we subsample (undersample) the dataset so that it contains 747 instances from each class\n",
    "- (Next to undersampling, there are several other ways to deal with class balances, but they are out of the scope of a book on LLMs; you can find examples and more information in the [`imbalanced-learn` user guide](https://imbalanced-learn.org/stable/user_guide.html))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7be4a0a2-9704-4a96-b38f-240339818688",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "7be4a0a2-9704-4a96-b38f-240339818688",
    "outputId": "396dc415-cb71-4a88-e85d-d88201c6d73f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label\n",
      "ham     747\n",
      "spam    747\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "%%run_once balance_df\n",
    "\n",
    "\n",
    "def create_balanced_dataset(df):\n",
    "    \n",
    "    # Count the instances of \"spam\"\n",
    "    num_spam = df[df[\"Label\"] == \"spam\"].shape[0]\n",
    "    \n",
    "    # Randomly sample \"ham\" instances to match the number of \"spam\" instances\n",
    "    ham_subset = df[df[\"Label\"] == \"ham\"].sample(num_spam, random_state=123)\n",
    "    \n",
    "    # Combine ham \"subset\" with \"spam\"\n",
    "    balanced_df = pd.concat([ham_subset, df[df[\"Label\"] == \"spam\"]])\n",
    "\n",
    "    return balanced_df\n",
    "\n",
    "\n",
    "balanced_df = create_balanced_dataset(df)\n",
    "print(balanced_df[\"Label\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6",
   "metadata": {
    "id": "d3fd2f5a-06d8-4d30-a2e3-230b86c559d6"
   },
   "source": [
    "- Next, we change the string class labels \"ham\" and \"spam\" into integer class labels 0 and 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd",
   "metadata": {
    "id": "c1b10c3d-5d57-42d0-8de8-cf80a06f5ffd"
   },
   "outputs": [],
   "source": [
    "%%run_once label_mapping\n",
    "balanced_df[\"Label\"] = balanced_df[\"Label\"].map({\"ham\": 0, \"spam\": 1})    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e6f7f062-ef4e-4020-8275-71990cab4414",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Label</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4307</th>\n",
       "      <td>0</td>\n",
       "      <td>Awww dat is sweet! We can think of something t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4138</th>\n",
       "      <td>0</td>\n",
       "      <td>Just got to  &amp;lt;#&amp;gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4831</th>\n",
       "      <td>0</td>\n",
       "      <td>The word \"Checkmate\" in chess comes from the P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4461</th>\n",
       "      <td>0</td>\n",
       "      <td>This is wishing you a great day. Moji told me ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5440</th>\n",
       "      <td>0</td>\n",
       "      <td>Thank you. do you generally date the brothas?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5537</th>\n",
       "      <td>1</td>\n",
       "      <td>Want explicit SEX in 30 secs? Ring 02073162414...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5540</th>\n",
       "      <td>1</td>\n",
       "      <td>ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5547</th>\n",
       "      <td>1</td>\n",
       "      <td>Had your contract mobile 11 Mnths? Latest Moto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>1</td>\n",
       "      <td>REMINDER FROM O2: To get 2.50 pounds free call...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>1</td>\n",
       "      <td>This is the 2nd time we have tried 2 contact u...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1494 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Label                                               Text\n",
       "4307      0  Awww dat is sweet! We can think of something t...\n",
       "4138      0                             Just got to  &lt;#&gt;\n",
       "4831      0  The word \"Checkmate\" in chess comes from the P...\n",
       "4461      0  This is wishing you a great day. Moji told me ...\n",
       "5440      0      Thank you. do you generally date the brothas?\n",
       "...     ...                                                ...\n",
       "5537      1  Want explicit SEX in 30 secs? Ring 02073162414...\n",
       "5540      1  ASKED 3MOBILE IF 0870 CHATLINES INCLU IN FREE ...\n",
       "5547      1  Had your contract mobile 11 Mnths? Latest Moto...\n",
       "5566      1  REMINDER FROM O2: To get 2.50 pounds free call...\n",
       "5567      1  This is the 2nd time we have tried 2 contact u...\n",
       "\n",
       "[1494 rows x 2 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "balanced_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f",
   "metadata": {
    "id": "5715e685-35b4-4b45-a86c-8a8694de9d6f"
   },
   "source": [
    "- Let's now define a function that randomly divides the dataset into training, validation, and test subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "uQl0Psdmx15D",
   "metadata": {
    "id": "uQl0Psdmx15D"
   },
   "outputs": [],
   "source": [
    "def random_split(df, train_frac, validation_frac):\n",
    "    # Shuffle the entire DataFrame\n",
    "    df = df.sample(frac=1, random_state=123).reset_index(drop=True)\n",
    "\n",
    "    # Calculate split indices\n",
    "    train_end = int(len(df) * train_frac)\n",
    "    validation_end = train_end + int(len(df) * validation_frac)\n",
    "\n",
    "    # Split the DataFrame\n",
    "    train_df = df[:train_end]\n",
    "    validation_df = df[train_end:validation_end]\n",
    "    test_df = df[validation_end:]\n",
    "\n",
    "    return train_df, validation_df, test_df\n",
    "\n",
    "train_df, validation_df, test_df = random_split(balanced_df, 0.7, 0.1)\n",
    "# Test size is implied to be 0.2 as the remainder\n",
    "\n",
    "train_df.to_csv(\"train.csv\", index=None)\n",
    "validation_df.to_csv(\"validation.csv\", index=None)\n",
    "test_df.to_csv(\"test.csv\", index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8d7a0c5-1d5f-458a-b685-3f49520b0094",
   "metadata": {},
   "source": [
    "## 6.3 Creating data loaders"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c",
   "metadata": {
    "id": "7126108a-75e7-4862-b0fb-cbf59a18bb6c"
   },
   "source": [
    "- Note that the text messages have different lengths; if we want to combine multiple training examples in a batch, we have to either\n",
    "  1. truncate all messages to the length of the shortest message in the dataset or batch\n",
    "  2. pad all messages to the length of the longest message in the dataset or batch\n",
    "\n",
    "- We choose option 2 and pad all messages to the longest message in the dataset\n",
    "- For that, we use `<|endoftext|>` as a padding token, as discussed in chapter 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0829f33f-1428-4f22-9886-7fee633b3666",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/pad-input-sequences.webp?123\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "74c3c463-8763-4cc0-9320-41c7eaad8ab7",
    "outputId": "b5b48439-32c8-4b37-cca2-c9dc8fa86563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50256]\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "print(tokenizer.encode(\"<|endoftext|>\", allowed_special={\"<|endoftext|>\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f582ff-68bf-450e-bd87-5fb61afe431c",
   "metadata": {
    "id": "04f582ff-68bf-450e-bd87-5fb61afe431c"
   },
   "source": [
    "- The `SpamDataset` class below identifies the longest sequence in the training dataset and adds the padding token to the others to match that sequence length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "d7791b52-af18-4ac4-afa9-b921068e383e",
   "metadata": {
    "id": "d7791b52-af18-4ac4-afa9-b921068e383e"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "\n",
    "class SpamDataset(Dataset):\n",
    "    def __init__(self, csv_file, tokenizer, max_length=None, pad_token_id=50256):\n",
    "        self.data = pd.read_csv(csv_file)\n",
    "\n",
    "        # Pre-tokenize texts\n",
    "        self.encoded_texts = [\n",
    "            tokenizer.encode(text) for text in self.data[\"Text\"]\n",
    "        ]\n",
    "\n",
    "        if max_length is None:\n",
    "            self.max_length = self._longest_encoded_length()\n",
    "        else:\n",
    "            self.max_length = max_length\n",
    "            # Truncate sequences if they are longer than max_length\n",
    "            self.encoded_texts = [\n",
    "                encoded_text[:self.max_length]\n",
    "                for encoded_text in self.encoded_texts\n",
    "            ]\n",
    "\n",
    "        # Pad sequences to the longest sequence\n",
    "        self.encoded_texts = [\n",
    "            encoded_text + [pad_token_id] * (self.max_length - len(encoded_text))\n",
    "            for encoded_text in self.encoded_texts\n",
    "        ]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        encoded = self.encoded_texts[index]\n",
    "        label = self.data.iloc[index][\"Label\"]\n",
    "        return (\n",
    "            torch.tensor(encoded, dtype=torch.long),\n",
    "            torch.tensor(label, dtype=torch.long)\n",
    "        )\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "    def _longest_encoded_length(self):\n",
    "        max_length = 0\n",
    "        for encoded_text in self.encoded_texts:\n",
    "            encoded_length = len(encoded_text)\n",
    "            if encoded_length > max_length:\n",
    "                max_length = encoded_length\n",
    "        return max_length\n",
    "        # Note: A more pythonic version to implement this method\n",
    "        # is the following, which is also used in the next chapter:\n",
    "        # return max(len(encoded_text) for encoded_text in self.encoded_texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "uzj85f8ou82h",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uzj85f8ou82h",
    "outputId": "d08f1cf0-c24d-445f-a3f8-793532c3716f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "120\n"
     ]
    }
   ],
   "source": [
    "train_dataset = SpamDataset(\n",
    "    csv_file=\"train.csv\",\n",
    "    max_length=None,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "\n",
    "print(train_dataset.max_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15bdd932-97eb-4b88-9cf9-d766ea4c3a60",
   "metadata": {},
   "source": [
    "- We also pad the validation and test set to the longest training sequence\n",
    "- Note that validation and test set samples that are longer than the longest training example are being truncated via `encoded_text[:self.max_length]` in the `SpamDataset` code\n",
    "- This behavior is entirely optional, and it would also work well if we set `max_length=None` in both the validation and test set cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e",
   "metadata": {
    "id": "bb0c502d-a75e-4248-8ea0-196e2b00c61e"
   },
   "outputs": [],
   "source": [
    "val_dataset = SpamDataset(\n",
    "    csv_file=\"validation.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")\n",
    "test_dataset = SpamDataset(\n",
    "    csv_file=\"test.csv\",\n",
    "    max_length=train_dataset.max_length,\n",
    "    tokenizer=tokenizer\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20170d89-85a0-4844-9887-832f5d23432a",
   "metadata": {},
   "source": [
    "- Next, we use the dataset to instantiate the data loaders, which is similar to creating the data loaders in previous chapters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64bcc349-205f-48f8-9655-95ff21f5e72f",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/batch.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8681adc0-6f02-4e75-b01a-a6ab75d05542",
    "outputId": "3266c410-4fdb-4a8c-a142-7f707e2525ab"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "num_workers = 0\n",
    "batch_size = 8\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "train_loader = DataLoader(\n",
    "    dataset=train_dataset,\n",
    "    batch_size=batch_size,\n",
    "    shuffle=True,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=True,\n",
    ")\n",
    "\n",
    "val_loader = DataLoader(\n",
    "    dataset=val_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")\n",
    "\n",
    "test_loader = DataLoader(\n",
    "    dataset=test_dataset,\n",
    "    batch_size=batch_size,\n",
    "    num_workers=num_workers,\n",
    "    drop_last=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab7335db-e0bb-4e27-80c5-eea11e593a57",
   "metadata": {},
   "source": [
    "- As a verification step, we iterate through the data loaders and ensure that the batches contain 8 training examples each, where each training example consists of 120 tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "4dee6882-4c3a-4964-af15-fa31f86ad047",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train loader:\n",
      "Input batch dimensions: torch.Size([8, 120])\n",
      "Label batch dimensions torch.Size([8])\n"
     ]
    }
   ],
   "source": [
    "print(\"Train loader:\")\n",
    "for input_batch, target_batch in train_loader:\n",
    "    pass\n",
    "\n",
    "print(\"Input batch dimensions:\", input_batch.shape)\n",
    "print(\"Label batch dimensions\", target_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cdd7947-7039-49bf-8a5e-c0a2f4281ca1",
   "metadata": {},
   "source": [
    "- Lastly, let's print the total number of batches in each dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "IZfw-TYD2zTj",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "IZfw-TYD2zTj",
    "outputId": "6934bbf2-9797-4fbe-d26b-1a246e18c2fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "130 training batches\n",
      "19 validation batches\n",
      "38 test batches\n"
     ]
    }
   ],
   "source": [
    "print(f\"{len(train_loader)} training batches\")\n",
    "print(f\"{len(val_loader)} validation batches\")\n",
    "print(f\"{len(test_loader)} test batches\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c",
   "metadata": {
    "id": "d1c4f61a-5f5d-4b3b-97cf-151b617d1d6c"
   },
   "source": [
    "## 6.4 Initializing a model with pretrained weights"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e1af8b-8bd1-4b44-8b8b-dc031496e208",
   "metadata": {},
   "source": [
    "- In this section, we initialize the pretrained model we worked with in the previous chapter\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-2.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2992d779-f9fb-4812-a117-553eb790a5a9",
   "metadata": {
    "id": "2992d779-f9fb-4812-a117-553eb790a5a9"
   },
   "outputs": [],
   "source": [
    "CHOOSE_MODEL = \"gpt2-small (124M)\"\n",
    "INPUT_PROMPT = \"Every effort moves\"\n",
    "\n",
    "BASE_CONFIG = {\n",
    "    \"vocab_size\": 50257,     # Vocabulary size\n",
    "    \"context_length\": 1024,  # Context length\n",
    "    \"drop_rate\": 0.0,        # Dropout rate\n",
    "    \"qkv_bias\": True         # Query-key-value bias\n",
    "}\n",
    "\n",
    "model_configs = {\n",
    "    \"gpt2-small (124M)\": {\"emb_dim\": 768, \"n_layers\": 12, \"n_heads\": 12},\n",
    "    \"gpt2-medium (355M)\": {\"emb_dim\": 1024, \"n_layers\": 24, \"n_heads\": 16},\n",
    "    \"gpt2-large (774M)\": {\"emb_dim\": 1280, \"n_layers\": 36, \"n_heads\": 20},\n",
    "    \"gpt2-xl (1558M)\": {\"emb_dim\": 1600, \"n_layers\": 48, \"n_heads\": 25},\n",
    "}\n",
    "\n",
    "BASE_CONFIG.update(model_configs[CHOOSE_MODEL])\n",
    "\n",
    "assert train_dataset.max_length <= BASE_CONFIG[\"context_length\"], (\n",
    "    f\"Dataset length {train_dataset.max_length} exceeds model's context \"\n",
    "    f\"length {BASE_CONFIG['context_length']}. Reinitialize data sets with \"\n",
    "    f\"`max_length={BASE_CONFIG['context_length']}`\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "022a649a-44f5-466c-8a8e-326c063384f5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "022a649a-44f5-466c-8a8e-326c063384f5",
    "outputId": "7091e401-8442-4f47-a1d9-ecb42a1ef930"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "File already exists and is up-to-date: gpt2/124M/checkpoint\n",
      "File already exists and is up-to-date: gpt2/124M/encoder.json\n",
      "File already exists and is up-to-date: gpt2/124M/hparams.json\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.data-00000-of-00001\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.index\n",
      "File already exists and is up-to-date: gpt2/124M/model.ckpt.meta\n",
      "File already exists and is up-to-date: gpt2/124M/vocab.bpe\n"
     ]
    }
   ],
   "source": [
    "from gpt_download import download_and_load_gpt2\n",
    "from previous_chapters import GPTModel, load_weights_into_gpt\n",
    "\n",
    "model_size = CHOOSE_MODEL.split(\" \")[-1].lstrip(\"(\").rstrip(\")\")\n",
    "settings, params = download_and_load_gpt2(model_size=model_size, models_dir=\"gpt2\")\n",
    "\n",
    "model = GPTModel(BASE_CONFIG)\n",
    "load_weights_into_gpt(model, params)\n",
    "model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab8e056c-abe0-415f-b34d-df686204259e",
   "metadata": {},
   "source": [
    "- To ensure that the model was loaded correctly, let's double-check that it generates coherent text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d8ac25ff-74b1-4149-8dc5-4c429d464330",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Every effort moves you forward.\n",
      "\n",
      "The first step is to understand the importance of your work\n"
     ]
    }
   ],
   "source": [
    "from previous_chapters import (\n",
    "    generate_text_simple,\n",
    "    text_to_token_ids,\n",
    "    token_ids_to_text\n",
    ")\n",
    "\n",
    "\n",
    "text_1 = \"Every effort moves you\"\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_1, tokenizer),\n",
    "    max_new_tokens=15,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69162550-6a02-4ece-8db1-06c71d61946f",
   "metadata": {},
   "source": [
    "- Before we finetune the model as a classifier, let's see if the model can perhaps already classify spam messages via prompting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "94224aa9-c95a-4f8a-a420-76d01e3a800c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is the following text 'spam'? Answer with 'yes' or 'no': 'You are a winner you have been specially selected to receive $1000 cash or a $2000 award.'\n",
      "\n",
      "The following text 'spam'? Answer with 'yes' or 'no': 'You are a winner\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Is the following text 'spam'? Answer with 'yes' or 'no':\"\n",
    "    \" 'You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.'\"\n",
    ")\n",
    "\n",
    "token_ids = generate_text_simple(\n",
    "    model=model,\n",
    "    idx=text_to_token_ids(text_2, tokenizer),\n",
    "    max_new_tokens=23,\n",
    "    context_size=BASE_CONFIG[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(token_ids_to_text(token_ids, tokenizer))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce39ed0-2c77-410d-8392-dd15d4b22016",
   "metadata": {},
   "source": [
    "- As we can see, the model is not very good at following instructions\n",
    "- This is expected, since it has only been pretrained and not instruction-finetuned (instruction finetuning will be covered in the next chapter)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522",
   "metadata": {
    "id": "4c9ae440-32f9-412f-96cf-fd52cc3e2522"
   },
   "source": [
    "## 6.5 Adding a classification head"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6e9d66f-76b2-40fc-9ec5-3f972a8db9c0",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/lm-head.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "217bac05-78df-4412-bd80-612f8061c01d",
   "metadata": {},
   "source": [
    "- In this section, we are modifying the pretrained LLM to make it ready for classification finetuning\n",
    "- Let's take a look at the model architecture first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b23aff91-6bd0-48da-88f6-353657e6c981",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1d8f7a01-b7c0-48d4-b1e7-8c12cc7ad932",
    "outputId": "b6a5b9b5-a92f-498f-d7cb-b58dd99e4497"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPTModel(\n",
      "  (tok_emb): Embedding(50257, 768)\n",
      "  (pos_emb): Embedding(1024, 768)\n",
      "  (drop_emb): Dropout(p=0.0, inplace=False)\n",
      "  (trf_blocks): Sequential(\n",
      "    (0): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (1): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (2): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (3): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (4): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (5): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (6): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (7): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (8): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (9): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (10): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "    (11): TransformerBlock(\n",
      "      (att): MultiHeadAttention(\n",
      "        (W_query): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_key): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (W_value): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (out_proj): Linear(in_features=768, out_features=768, bias=True)\n",
      "        (dropout): Dropout(p=0.0, inplace=False)\n",
      "      )\n",
      "      (ff): FeedForward(\n",
      "        (layers): Sequential(\n",
      "          (0): Linear(in_features=768, out_features=3072, bias=True)\n",
      "          (1): GELU()\n",
      "          (2): Linear(in_features=3072, out_features=768, bias=True)\n",
      "        )\n",
      "      )\n",
      "      (norm1): LayerNorm()\n",
      "      (norm2): LayerNorm()\n",
      "      (drop_resid): Dropout(p=0.0, inplace=False)\n",
      "    )\n",
      "  )\n",
      "  (final_norm): LayerNorm()\n",
      "  (out_head): Linear(in_features=768, out_features=50257, bias=False)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f640a76-dd00-4769-9bc8-1aed0cec330d",
   "metadata": {},
   "source": [
    "- Above, we can see the architecture we implemented in chapter 4 neatly laid out\n",
    "- The goal is to replace and finetune the output layer\n",
    "- To achieve this, we first freeze the model, meaning that we make all layers non-trainable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fkMWFl-0etea",
   "metadata": {
    "id": "fkMWFl-0etea"
   },
   "outputs": [],
   "source": [
    "for param in model.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72155f83-87d9-476a-a978-a15aa2d44147",
   "metadata": {},
   "source": [
    "- Then, we replace the output layer (`model.out_head`), which originally maps the layer inputs to 50,257 dimensions (the size of the vocabulary)\n",
    "- Since we finetune the model for binary classification (predicting 2 classes, \"spam\" and \"not spam\"), we can replace the output layer as shown below, which will be trainable by default\n",
    "- Note that we use `BASE_CONFIG[\"emb_dim\"]` (which is equal to 768 in the `\"gpt2-small (124M)\"` model) to keep the code below more general"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7e759fa0-0f69-41be-b576-17e5f20e04cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "num_classes = 2\n",
    "model.out_head = torch.nn.Linear(in_features=BASE_CONFIG[\"emb_dim\"], out_features=num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30be5475-ae77-4f97-8f3e-dec462b1339f",
   "metadata": {},
   "source": [
    "- Technically, it's sufficient to only train the output layer\n",
    "- However, as I found in [Finetuning Large Language Models](https://magazine.sebastianraschka.com/p/finetuning-large-language-models), experiments show that finetuning additional layers can noticeably improve the performance\n",
    "- So, we are also making the last transformer block and the final `LayerNorm` module connecting the last transformer block to the output layer trainable"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be7c1eb-c46c-4065-8525-eea1b8c66d10",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/trainable.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7",
   "metadata": {
    "id": "2aedc120-5ee3-48f6-92f2-ad9304ebcdc7"
   },
   "outputs": [],
   "source": [
    "for param in model.trf_blocks[-1].parameters():\n",
    "    param.requires_grad = True\n",
    "\n",
    "for param in model.final_norm.parameters():\n",
    "    param.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f012b899-8284-4d3a-97c0-8a48eb33ba2e",
   "metadata": {},
   "source": [
    "- We can still use this model similar to before in previous chapters\n",
    "- For example, let's feed it some text input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f645c06a-7df6-451c-ad3f-eafb18224ebc",
    "outputId": "27e041b1-d731-48a1-cf60-f22d4565304e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inputs: tensor([[5211,  345,  423,  640]])\n",
      "Inputs dimensions: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "inputs = tokenizer.encode(\"Do you have time\")\n",
    "inputs = torch.tensor(inputs).unsqueeze(0)\n",
    "print(\"Inputs:\", inputs)\n",
    "print(\"Inputs dimensions:\", inputs.shape) # shape: (batch_size, num_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbf8481-772d-467b-851c-a62b86d0cb1b",
   "metadata": {},
   "source": [
    "- What's different compared to previous chapters is that it now has two output dimensions instead of 50,257"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "48dc84f1-85cc-4609-9cee-94ff539f00f4",
    "outputId": "9cae7448-253d-4776-973e-0af190b06354"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Outputs:\n",
      " tensor([[[-1.5854,  0.9904],\n",
      "         [-3.7235,  7.4548],\n",
      "         [-2.2661,  6.6049],\n",
      "         [-3.5983,  3.9902]]])\n",
      "Outputs dimensions: torch.Size([1, 4, 2])\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad():\n",
    "    outputs = model(inputs)\n",
    "\n",
    "print(\"Outputs:\\n\", outputs)\n",
    "print(\"Outputs dimensions:\", outputs.shape) # shape: (batch_size, num_tokens, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75430a01-ef9c-426a-aca0-664689c4f461",
   "metadata": {},
   "source": [
    "- As discussed in previous chapters, for each input token, there's one output vector\n",
    "- Since we fed the model a text sample with 4 input tokens, the output consists of 4 2-dimensional output vectors above"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7df9144f-6817-4be4-8d4b-5d4dadfe4a9b",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/input-and-output.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3bb8616-c791-4f5c-bac0-5302f663e46a",
   "metadata": {},
   "source": [
    "- In chapter 3, we discussed the attention mechanism, which connects each input token to each other input token\n",
    "- In chapter 3, we then also introduced the causal attention mask that is used in GPT-like models; this causal mask lets a current token only attend to the current and previous token positions\n",
    "- Based on this causal attention mechanism, the 4th (last) token contains the most information among all tokens because it's the only token that includes information about all other tokens\n",
    "- Hence, we are particularly interested in this last token, which we will finetune for the spam classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "49383a8c-41d5-4dab-98f1-238bca0c2ed7",
    "outputId": "e79eb155-fa1f-46ed-ff8c-d828c3a3fabd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.5983,  3.9902]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8df08ae0-e664-4670-b7c5-8a2280d9b41b",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/attention-mask.webp\" width=200px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32aa4aef-e1e9-491b-9adf-5aa973e59b8c",
   "metadata": {},
   "source": [
    "## 6.6 Calculating the classification loss and accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "669e1fd1-ace8-44b4-b438-185ed0ba8b33",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-3.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a7df4ee-0a34-4a4d-896d-affbbf81e0b3",
   "metadata": {},
   "source": [
    "- Before explaining the loss calculation, let's have a brief look at how the model outputs are turned into class labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "557996dd-4c6b-49c4-ab83-f60ef7e1d69e",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/class-argmax.webp\" width=600px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c77faab1-3461-4118-866a-6171f2b89aa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last output token: tensor([[-3.5983,  3.9902]])\n"
     ]
    }
   ],
   "source": [
    "print(\"Last output token:\", outputs[:, -1, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edd71fa-628a-4d00-b81d-6d8bcb2c341d",
   "metadata": {},
   "source": [
    "- Similar to chapter 5, we convert the outputs (logits) into probability scores via the `softmax` function and then obtain the index position of the largest probability value via the `argmax` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b81efa92-9be1-4b9e-8790-ce1fc7b17f01",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "probas = torch.softmax(outputs[:, -1, :], dim=-1)\n",
    "label = torch.argmax(probas)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414a6f02-307e-4147-a416-14d115bf8179",
   "metadata": {},
   "source": [
    "- Note that the softmax function is optional here, as explained in chapter 5, because the largest outputs correspond to the largest probability scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f9f9ad66-4969-4501-8239-3ccdb37e71a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class label: 1\n"
     ]
    }
   ],
   "source": [
    "logits = outputs[:, -1, :]\n",
    "label = torch.argmax(logits)\n",
    "print(\"Class label:\", label.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcb20d3a-cbba-4ab1-8584-d94e16589505",
   "metadata": {},
   "source": [
    "- We can apply this concept to calculate the so-called classification accuracy, which computes the percentage of correct predictions in a given dataset\n",
    "- To calculate the classification accuracy, we can apply the preceding `argmax`-based prediction code to all examples in a dataset and calculate the fraction of correct predictions as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "3ecf9572-aed0-4a21-9c3b-7f9f2aec5f23",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_accuracy_loader(data_loader, model, device, num_batches=None):\n",
    "    model.eval()\n",
    "    correct_predictions, num_examples = 0, 0\n",
    "\n",
    "    if num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "\n",
    "            with torch.no_grad():\n",
    "                logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "            predicted_labels = torch.argmax(logits, dim=-1)\n",
    "\n",
    "            num_examples += predicted_labels.shape[0]\n",
    "            correct_predictions += (predicted_labels == target_batch).sum().item()\n",
    "        else:\n",
    "            break\n",
    "    return correct_predictions / num_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7165fe46-a284-410b-957f-7524877d1a1a",
   "metadata": {},
   "source": [
    "- Let's apply the function to calculate the classification accuracies for the different datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "390e5255-8427-488c-adef-e1c10ab4fb26",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 46.25%\n",
      "Validation accuracy: 45.00%\n",
      "Test accuracy: 48.75%\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Note:\n",
    "# Uncommenting the following lines will allow the code to run on Apple Silicon chips, if applicable,\n",
    "# which is approximately 2x faster than on an Apple CPU (as measured on an M3 MacBook Air).\n",
    "# As of this writing, in PyTorch 2.4, the results obtained via CPU and MPS were identical.\n",
    "# However, in earlier versions of PyTorch, you may observe different results when using MPS.\n",
    "\n",
    "#if torch.cuda.is_available():\n",
    "#    device = torch.device(\"cuda\")\n",
    "#elif torch.backends.mps.is_available():\n",
    "#    device = torch.device(\"mps\")\n",
    "#else:\n",
    "#    device = torch.device(\"cpu\")\n",
    "#print(f\"Running on {device} device.\")\n",
    "\n",
    "model.to(device) # no assignment model = model.to(device) necessary for nn.Module classes\n",
    "\n",
    "torch.manual_seed(123) # For reproducibility due to the shuffling in the training data loader\n",
    "\n",
    "train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=10)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=10)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device, num_batches=10)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30345e2a-afed-4d22-9486-f4010f90a871",
   "metadata": {},
   "source": [
    "- As we can see, the prediction accuracies are not very good, since we haven't finetuned the model, yet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4a9d15-8fc7-48a2-8734-d92a2f265328",
   "metadata": {},
   "source": [
    "- Before we can start finetuning (/training), we first have to define the loss function we want to optimize during training\n",
    "- The goal is to maximize the spam classification accuracy of the model; however, classification accuracy is not a differentiable function\n",
    "- Hence, instead, we minimize the cross-entropy loss as a proxy for maximizing the classification accuracy (you can learn more about this topic in lecture 8 of my freely available [Introduction to Deep Learning](https://sebastianraschka.com/blog/2021/dl-course.html#l08-multinomial-logistic-regression--softmax-regression) class)\n",
    "\n",
    "- The `calc_loss_batch` function is the same here as in chapter 5, except that we are only interested in optimizing the last token `model(input_batch)[:, -1, :]` instead of all tokens `model(input_batch)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4",
   "metadata": {
    "id": "2f1e9547-806c-41a9-8aba-3b2822baabe4"
   },
   "outputs": [],
   "source": [
    "def calc_loss_batch(input_batch, target_batch, model, device):\n",
    "    input_batch, target_batch = input_batch.to(device), target_batch.to(device)\n",
    "    logits = model(input_batch)[:, -1, :]  # Logits of last output token\n",
    "    loss = torch.nn.functional.cross_entropy(logits, target_batch)\n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a013aab9-f854-4866-ad55-5b8350adb50a",
   "metadata": {},
   "source": [
    "The `calc_loss_loader` is exactly the same as in chapter 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "b7b83e10-5720-45e7-ac5e-369417ca846b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as in chapter 5\n",
    "def calc_loss_loader(data_loader, model, device, num_batches=None):\n",
    "    total_loss = 0.\n",
    "    if len(data_loader) == 0:\n",
    "        return float(\"nan\")\n",
    "    elif num_batches is None:\n",
    "        num_batches = len(data_loader)\n",
    "    else:\n",
    "        # Reduce the number of batches to match the total number of batches in the data loader\n",
    "        # if num_batches exceeds the number of batches in the data loader\n",
    "        num_batches = min(num_batches, len(data_loader))\n",
    "    for i, (input_batch, target_batch) in enumerate(data_loader):\n",
    "        if i < num_batches:\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            total_loss += loss.item()\n",
    "        else:\n",
    "            break\n",
    "    return total_loss / num_batches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56826ecd-6e74-40e6-b772-d3541e585067",
   "metadata": {},
   "source": [
    "- Using the `calc_closs_loader`, we compute the initial training, validation, and test set losses before we start training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f6f00e53-5beb-4e64-b147-f26fd481c6ff",
    "outputId": "49df8648-9e38-4314-854d-9faacd1b2e89"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 2.453\n",
      "Validation loss: 2.583\n",
      "Test loss: 2.322\n"
     ]
    }
   ],
   "source": [
    "with torch.no_grad(): # Disable gradient tracking for efficiency because we are not training, yet\n",
    "    train_loss = calc_loss_loader(train_loader, model, device, num_batches=5)\n",
    "    val_loss = calc_loss_loader(val_loader, model, device, num_batches=5)\n",
    "    test_loss = calc_loss_loader(test_loader, model, device, num_batches=5)\n",
    "\n",
    "print(f\"Training loss: {train_loss:.3f}\")\n",
    "print(f\"Validation loss: {val_loss:.3f}\")\n",
    "print(f\"Test loss: {test_loss:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e04b980b-e583-4f62-84a0-4edafaf99d5d",
   "metadata": {},
   "source": [
    "- In the next section, we train the model to improve the loss values and consequently the classification accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "456ae0fd-6261-42b4-ab6a-d24289953083",
   "metadata": {
    "id": "456ae0fd-6261-42b4-ab6a-d24289953083"
   },
   "source": [
    "## 6.7 Finetuning the model on supervised data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9b099b-0829-4f72-8a2b-4363e3497026",
   "metadata": {},
   "source": [
    "- In this section, we define and use the training function to improve the classification accuracy of the model\n",
    "- The `train_classifier_simple` function below is practically the same as the `train_model_simple` function we used for pretraining the model in chapter 5\n",
    "- The only two differences are that we now \n",
    "  1. track the number of training examples seen (`examples_seen`) instead of the number of tokens seen\n",
    "  2. calculate the accuracy after each epoch instead of printing a sample text after each epoch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "979b6222-1dc2-4530-9d01-b6b04fe3de12",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/training-loop.webp?1\" width=500px>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "Csbr60to50FL",
   "metadata": {
    "id": "Csbr60to50FL"
   },
   "outputs": [],
   "source": [
    "# Overall the same as `train_model_simple` in chapter 5\n",
    "def train_classifier_simple(model, train_loader, val_loader, optimizer, device, num_epochs,\n",
    "                            eval_freq, eval_iter):\n",
    "    # Initialize lists to track losses and examples seen\n",
    "    train_losses, val_losses, train_accs, val_accs = [], [], [], []\n",
    "    examples_seen, global_step = 0, -1\n",
    "\n",
    "    # Main training loop\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train()  # Set model to training mode\n",
    "\n",
    "        for input_batch, target_batch in train_loader:\n",
    "            optimizer.zero_grad() # Reset loss gradients from previous batch iteration\n",
    "            loss = calc_loss_batch(input_batch, target_batch, model, device)\n",
    "            loss.backward() # Calculate loss gradients\n",
    "            optimizer.step() # Update model weights using loss gradients\n",
    "            examples_seen += input_batch.shape[0] # New: track examples instead of tokens\n",
    "            global_step += 1\n",
    "\n",
    "            # Optional evaluation step\n",
    "            if global_step % eval_freq == 0:\n",
    "                train_loss, val_loss = evaluate_model(\n",
    "                    model, train_loader, val_loader, device, eval_iter)\n",
    "                train_losses.append(train_loss)\n",
    "                val_losses.append(val_loss)\n",
    "                print(f\"Ep {epoch+1} (Step {global_step:06d}): \"\n",
    "                      f\"Train loss {train_loss:.3f}, Val loss {val_loss:.3f}\")\n",
    "\n",
    "        # Calculate accuracy after each epoch\n",
    "        train_accuracy = calc_accuracy_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_accuracy = calc_accuracy_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "        print(f\"Training accuracy: {train_accuracy*100:.2f}% | \", end=\"\")\n",
    "        print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "        train_accs.append(train_accuracy)\n",
    "        val_accs.append(val_accuracy)\n",
    "\n",
    "    return train_losses, val_losses, train_accs, val_accs, examples_seen"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9624cb30-3e3a-45be-b006-c00475b58ae8",
   "metadata": {},
   "source": [
    "- The `evaluate_model` function used in the `train_classifier_simple` is the same as the one we used in chapter 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "bcc7bc04-6aa6-4516-a147-460e2f466eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as chapter 5\n",
    "def evaluate_model(model, train_loader, val_loader, device, eval_iter):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        train_loss = calc_loss_loader(train_loader, model, device, num_batches=eval_iter)\n",
    "        val_loss = calc_loss_loader(val_loader, model, device, num_batches=eval_iter)\n",
    "    model.train()\n",
    "    return train_loss, val_loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e807bfe9-364d-46b2-9e25-3b000c3ef6f9",
   "metadata": {},
   "source": [
    "- The training takes about 5 minutes on a M3 MacBook Air laptop computer and less than half a minute on a V100 or A100 GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "X7kU3aAj7vTJ",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "X7kU3aAj7vTJ",
    "outputId": "504a033e-2bf8-41b5-a037-468309845513"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ep 1 (Step 000000): Train loss 2.153, Val loss 2.392\n",
      "Ep 1 (Step 000050): Train loss 0.617, Val loss 0.637\n",
      "Ep 1 (Step 000100): Train loss 0.523, Val loss 0.557\n",
      "Training accuracy: 70.00% | Validation accuracy: 72.50%\n",
      "Ep 2 (Step 000150): Train loss 0.561, Val loss 0.489\n",
      "Ep 2 (Step 000200): Train loss 0.419, Val loss 0.397\n",
      "Ep 2 (Step 000250): Train loss 0.409, Val loss 0.353\n",
      "Training accuracy: 82.50% | Validation accuracy: 85.00%\n",
      "Ep 3 (Step 000300): Train loss 0.333, Val loss 0.320\n",
      "Ep 3 (Step 000350): Train loss 0.340, Val loss 0.306\n",
      "Training accuracy: 90.00% | Validation accuracy: 90.00%\n",
      "Ep 4 (Step 000400): Train loss 0.136, Val loss 0.200\n",
      "Ep 4 (Step 000450): Train loss 0.153, Val loss 0.132\n",
      "Ep 4 (Step 000500): Train loss 0.222, Val loss 0.137\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Ep 5 (Step 000550): Train loss 0.207, Val loss 0.143\n",
      "Ep 5 (Step 000600): Train loss 0.083, Val loss 0.074\n",
      "Training accuracy: 100.00% | Validation accuracy: 97.50%\n",
      "Training completed in 5.31 minutes.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "torch.manual_seed(123)\n",
    "\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5, weight_decay=0.1)\n",
    "\n",
    "num_epochs = 5\n",
    "train_losses, val_losses, train_accs, val_accs, examples_seen = train_classifier_simple(\n",
    "    model, train_loader, val_loader, optimizer, device,\n",
    "    num_epochs=num_epochs, eval_freq=50, eval_iter=5,\n",
    ")\n",
    "\n",
    "end_time = time.time()\n",
    "execution_time_minutes = (end_time - start_time) / 60\n",
    "print(f\"Training completed in {execution_time_minutes:.2f} minutes.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1261bf90-3ce7-4591-895a-044a05538f30",
   "metadata": {},
   "source": [
    "- Similar to chapter 5, we use matplotlib to plot the loss function for the training and validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "cURgnDqdCeka",
   "metadata": {
    "id": "cURgnDqdCeka"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_values(epochs_seen, examples_seen, train_values, val_values, label=\"loss\"):\n",
    "    fig, ax1 = plt.subplots(figsize=(5, 3))\n",
    "\n",
    "    # Plot training and validation loss against epochs\n",
    "    ax1.plot(epochs_seen, train_values, label=f\"Training {label}\")\n",
    "    ax1.plot(epochs_seen, val_values, linestyle=\"-.\", label=f\"Validation {label}\")\n",
    "    ax1.set_xlabel(\"Epochs\")\n",
    "    ax1.set_ylabel(label.capitalize())\n",
    "    ax1.legend()\n",
    "\n",
    "    # Create a second x-axis for examples seen\n",
    "    ax2 = ax1.twiny()  # Create a second x-axis that shares the same y-axis\n",
    "    ax2.plot(examples_seen, train_values, alpha=0)  # Invisible plot for aligning ticks\n",
    "    ax2.set_xlabel(\"Examples seen\")\n",
    "\n",
    "    fig.tight_layout()  # Adjust layout to make room\n",
    "    plt.savefig(f\"{label}-plot.pdf\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "OIqRt466DiGk",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "OIqRt466DiGk",
    "outputId": "b16987cf-0001-4652-ddaf-02f7cffc34db"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEiCAYAAAA21pHjAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAToxJREFUeJzt3Qd0VGXaB/B/Jr2ShPQQCJAQeu9FEJCiotgXXUHWsiK6KLquWEDkU+yggiC6ih0QBVwFFOm9SDGU0AkJkAYhpNf7needzGQmJCEhZWaS/++ce2bmzp2Zdy5hnvvWx07TNA1ERERklXSWLgARERGVj4GaiIjIijFQExERWTEGaiIiIivGQE1ERGTFGKiJiIisGAM1ERGRFWOgJiIismIM1ERERFaMgZqIKmXQoEF4+umnLV0MogaHgZqojjz00EOws7O7ahsxYoSli0ZEVszB0gUgakgkKH/xxRdm+5ydnS1WHiKyfqxRE9UhCcpBQUFmm4+Pj3puw4YNcHJywubNm43Hv/322wgICEBiYqJ6vHr1avTv3x/e3t5o3Lgxbr31Vpw8edJ4/JkzZ1QtfcmSJRgwYABcXV3Ro0cPHDt2DLt370b37t3h4eGBkSNHIjk52ay2P3r0aEyfPh3+/v7w8vLC448/jry8vHK/S25uLp577jmEhobC3d0dvXr1Ut/BIDY2FqNGjVLfT55v164dVq5cWe77ffzxx4iMjISLiwsCAwNx9913G58rKirCzJkz0bx5c/WdOnXqhKVLl5q9/uDBg+p7yfeT1z/44INISUkxa7r/17/+heeffx6+vr7q3L/66quV+ncjsiQGaiIr6wOWAJOWloZ9+/bhlVdewWeffaYCj8jMzMTkyZOxZ88erF27FjqdDnfccYcKZKamTZuGl19+GXv37oWDgwPuv/9+FaA++OADdSFw4sQJTJ061ew18n5HjhxRwfb777/HTz/9pAJ3eZ588kls374dixYtwl9//YV77rlHtRgcP35cPT9x4kQVzDdt2oTo6Gi89dZbKoiWRb6PBNHXXnsNR48eVRckN9xwg/F5CdJfffUV5s+fj0OHDuGZZ57B3//+d2zcuFE9f/nyZQwePBhdunRR7yWvl4ube++91+xzvvzyS3XRsHPnTnURJJ+3Zs2aKv9bEdUpSXNJRLVv3Lhxmr29vebu7m62vf7668ZjcnNztc6dO2v33nuv1rZtW+3RRx+t8D2Tk5MlTa0WHR2tHp8+fVo9/uyzz4zHfP/992rf2rVrjftmzpypRUVFmZXN19dXy8zMNO6bN2+e5uHhoRUWFqrHAwcO1CZNmqTux8bGqu9y7tw5s/IMGTJEmzJlirrfoUMH7dVXX63Uufnxxx81Ly8v7cqVK1c9l5OTo7m5uWnbtm0z2//www9rY8aMUfdnzJihDRs2zOz5uLg49b2PHj1qLH///v3NjunRo4f2n//8p1JlJLIU9lET1aEbb7wR8+bNM9snzbAG0vT97bffomPHjmjWrBlmzZpldqzUVqUmLDVCadY11KTPnj2L9u3bG4+T1xsYauMdOnQw25eUlGT23tKc7ObmZnzcp08fZGRkIC4uTpXFlNSQCwsL0apVK7P9UoOWJnkhNeQJEybg999/x9ChQ3HXXXeZlcvUTTfdpD6jRYsWqlYum7QUSHmk9p+VlaWOMSXN8lKDFgcOHMD69evLrLFL14ChnKU/Pzg4+KrzQGRtGKiJ6pA0u0ZERFR4zLZt29TtpUuX1CavMZA+Xwlon376KUJCQlSglgBdui/Z0dHReF/6rMvaV7q5vCokgNvb2+PPP/9Ut6YMwfKRRx7B8OHD8euvv6pgLc3X7733Hp566qmr3s/T01M100uzuxwrFyPSfyz96vJZQt5H+sPLGognx8i5keb10iQYl3VeauI8ENUFBmoiKyK1P+l/lUC8ePFijBs3Dn/88Yfqi7548aLqv5XnZKCY2LJlS419ttRKs7Oz1WAtsWPHDhV0w8LCrjpWarJSo5baqKEsZZHXyqA02aZMmaLKXlagFtKXLjVv2aSPXQbMrVu3TtWkJSBLq8HAgQPLfG3Xrl3x448/Ijw8XL0PUX3Cv2iiOiRNwwkJCWb7JLD4+fmpwCcDpKQWOn78eNX8K83VUgv997//rUZPS7PyggULVC1RAtcLL7xQY2WTWvnDDz+sBqHJ6HEJljJgTC4SSpOm5AceeABjx45V5ZPALaPIZUCaNC/fcsstamCcjMKWY1NTU1XTdJs2bcr87F9++QWnTp1SA8jke8rocKnpRkVFqdq2jC6XCxjZJ6PeZbDd1q1b1eh0uZiRgWtyETBmzBjjqG5pMpeBbjIYr3Stn8iWMFAT1SEZjWzaFCskGMXExOD1119XU5okaAk5ToKyBJ9hw4apPmQJPNL3K83d8roPP/xQjRavCUOGDFHToyRYygWFfG5F05dkPvj//d//4dlnn8W5c+fUxUbv3r3VlDEhFx4SQOPj41VAlQuP0n3uBlJ7llHm8nk5OTmqHDLyXKZ0iRkzZqhpY9J8LgFdjpda9Isvvqiel24ACdz/+c9/1LmS8ksXgXxmWRcaRLbETkaUWboQRGRZMo9apjgtX77c0kUholJ4qUlERGTFGKiJiIisGJu+iYiIrBhr1ERERFaMgZqIiMiKMVATERFZMQbqapg7d65aCUnS8kmKv127dqG+kgxIskSjzFeVZRdLT+ORoQ6y7KPM/ZWVrWR1KUMWJQNZDlMWyZA5tTIPVhbXMCwPaSBZmGSlKzmnsqqVZDiyBTK/V9JJyuIckpZSUkbKKmKmZH6wzCuWRUtkxS9Z+9qQvtJAFjGRxUJkjWt5H1nopKCgwOwYWWZT5hDLal2yHOnChQthC2SNc1kMRf79ZZO1xFetWmV8vqGfn7K8+eab6v+bLB5jwPMENd9ezovp1rp16/p7jiyWDsTGLVq0SHNyctI+//xz7dChQyrLkbe3t5aYmKjVRytXrtReeukl7aefflIZiZYtW2b2/Jtvvqk1atRIW758uXbgwAHttttu05o3b65lZ2cbjxkxYoTWqVMnbceOHdrmzZu1iIgIY/YjkZaWpgUGBmoPPPCAdvDgQZX1ydXVVfvkk080azd8+HDtiy++UOXev3+/dvPNN2tNmzbVMjIyjMc8/vjjWlhYmMpitWfPHq13795a3759jc8XFBRo7du314YOHart27dPnXM/Pz9jNipx6tQplUlq8uTJ2uHDh7WPPvpIZbFavXq1Zu1+/vln7ddff9WOHTumMlq9+OKLmqOjozpnoqGfn9J27dqlhYeHax07djRmLRM8T5o2bdo0rV27dtqFCxeMm2SSq6/niIH6OvXs2VObOHGi8bGkAgwJCVHpA+u70oG6qKhICwoK0t555x3jvsuXL2vOzs4q2Ar5Q5fX7d6923jMqlWrNDs7O2OqxI8//ljz8fFRqR4NJAWhaTpGW5GUlKS+78aNG43nQ4LSDz/8YDzmyJEj6pjt27erx/JjodPptISEBLNUk5L+0XBOnn/+efUDZeq+++5TFwq2SP69JSUnz4+59PR0LTIyUluzZo1ZelGep5JALRf9ZamP54hN39e5JrJkDZLmXQNZplAeb9++HQ3N6dOn1frVpuejUaNGqjvAcD7kVpq7u3fvbjxGjpfzJikbDcfI8pWS6tFA1r2WJmRZK9qWyFrUpiks5e8lPz/f7BxJU13Tpk3NzpGs7W1IS2n4/leuXMGhQ4eMx5i+h+EYW/u7k+VFZTnUzMxM1QTO82NOmm2lWbb0d+F5KiFda9IVJ6lRpUtNmrLr6zlioL4OkgdYfmhM/5GFPC6dcKEhMHznis6H3Eo/UOlkFBLITI8p6z1MP8MWSOII6VPs16+fMUe0lF8uQORipaJzdK3vX94x8gMjma+sneSxlj5D6fOTjFrLli1D27ZteX5MyAWMpPyUcQ+l8TzpSSVA+otl7XwZ+yCVBRnbkp6eXi/PEZNyENVCbejgwYM1moKyvpBEIvv371ctDkuXLlWZrzZu3GjpYlmNuLg4TJo0CWvWrFEDKqlskpXNQAYoSuCWJCxLliwxpmmtT1ijvg6SJUjS5pUeRSiPg4KC0NAYvnNF50NuJXexKRlhKSPBTY8p6z1MP8PaSVpIyX4lKR2bNGli3C/lly4TSXxR0Tm61vcv7xgZRW0LP1BS05HRs926dVM1RskI9sEHH/D8FJNmW/l/IiONpcVJNrmQkSxpcl9qdDxPV5Pas6RTldSm9fFviYH6On9s5IdGcu+aNnfKY+lva2iaN2+u/qhNz4c0D0nfs+F8yK38x5EfIoN169ap8yZXw4ZjZBqY9C8ZSM1CamGSo9iayRg7CdLSlCvfS86JKfl7cXR0NDtH0vcu/Wqm50iahk0vaOT7yw+DNA8bjjF9D8Mxtvp3J//+kpKS56ck1ah8R2l1MGwyrkP6YA33eZ6uJtM8T548qaaH1su/pTofvlaPpmfJqOaFCxeqEc2PPfaYmp5lOoqwPpFRqDKNQTb5s3n//ffV/djYWOP0LPn+K1as0P766y/t9ttvL3N6VpcuXbSdO3dqW7ZsUaNaTadnyWhNmZ714IMPqik7co5leoQtTM+aMGGCmp62YcMGsykjWVlZZlNGZMrWunXr1JSRPn36qK30lJFhw4apKV4yDcTf37/MKSP//ve/1UjWuXPn2sy0mhdeeEGNgj99+rT6G5HHMur/999/V8839PNTHtNR34LnSdOeffZZ9X9N/pa2bt2qplnJ9CqZbVEfzxEDdTXIvDr5Y5D51DJdS+YH11fr169XAbr0Nm7cOOMUrVdeeUUFWrmAGTJkiJora+rixYsqMHt4eKhpEOPHj1cXAKZkDnb//v3Ve4SGhqoLAFtQ1rmRTeZWG8hFyxNPPKGmJMkPwB133KGCuakzZ85oI0eOVPPH5YdHfpDy8/Ov+rfo3Lmz+rtr0aKF2WdYs3/84x9as2bNVLnlR1H+RgxBWjT081PZQM3zpKlpUsHBwars8jshj0+cOFFvzxGzZxEREVkx9lETERFZMQZqIiIiK8ZATUREZMUYqImIiKwYAzUREZEVY6AmIiKyYgzU1SArKkkCc7ml8vE8XRvP0bXxHF0bz1H9PEcWnUcta/3+9NNPiImJUWun9u3bF2+99ZZaMrI8kjFl/PjxZvskE09OTg7qmiyTKekcJcGALD1HZeN5ujaeo2vjObo2nqP6eY4sWqOWxeYl09COHTvUGqqyxvOwYcNUjtqKyMm9cOGCcYuNja2zMhMRETWYNJeSS7R0bVlyFkvihhtuuKHc19nZ2dlMNiUiIqJ6k49amiKEr6/vNTOlSO5Rybwj6eDeeOMNtGvXrlKfIakV9+3bp9LF6XTVa1CQJOXi3LlzqjmFysbzdG08R9fGc3RtPEe2c44kfknazC5duqgUphWxmrW+pdC33XabSoW4ZcuWco/bvn07jh8/rpKFS2B/9913VWrEQ4cOmeX/NZABA6aDBqS2Pnjw4Fr7HkRERJW1a9cu9OjRwzYC9YQJE7Bq1SoVpMsKuOWRfu02bdpgzJgxmDFjxlXPy+i+6dOnl3lyJHcpERFRXZPxVT179lRjrJo2bWr9gfrJJ5/EihUrVM24efPmVX79Pffco5oOvv/++2vWqKW5QxKDx8XFVemCgIiIqKbEx8cjLCysUrHIoqO+5RpBgvSyZcuwbt266wrShYWFiI6OLrd2LFO3ZJS4YfP09KyBkhMRETWAwWQyNeu7775TtWkJoAkJCWq/zHGTedVi7NixCA0NVXOuxWuvvYbevXsjIiJC9We/8847qungkUceseRXISIiqn+Bet68eep20KBBZvu/+OILPPTQQ+r+2bNnzUZnp6am4tFHH1VB3cfHB926dcO2bdtUczYREVF9YxV91NbaL0BEDY90p8kgVaLqcHR0hL29fY3EIquaR01EZClSZ5GWOulSI6oJ3t7eanEuWaSrOhioqyP7MnB2B9CoCRDU3tKlIaJqMARpWR3Rzc2t2j+u1LAv+rKyspCUlKQeV3cqMAN1daz7P2D3p0Cvx4GRb1m6NERUjeZuQ5Bu3LixpYtD9YBr8YBoCdbyd1VRM/i1MM1ldYT309+e2WrpkhBRNRj6pKUmTVRTDH9P1R3zwEBdHc2KA3XiQSDrkqVLQ0TVxOZussa/Jwbq6vAIAPxaSY8EcHa7pUtDRET1EAN1dYX319+y+ZuI6onw8HDMnj270sdv2LBB1R5re8T8woUL1UjqhoaBuqaav89stnRJiKiBkeBY0SZJia7H7t278dhjj1X6+L59+6okE7KqJNU8jvquqRp1QrR+upZrw7vaIyLLkOBosHjxYkydOhVHjx417vPw8DCbMiSj26+V+1j4+/tXqRxOTk5qvjDVDtaoq8szCGgcUdxPvcPSpSGiBkSCo2GT2qzUog2PY2JiVA4FSR8sSy1LgiJJI3zy5EncfvvtCAwMVIFcciH/8ccfFTZ9y/t+9tlnuOOOO9RI5sjISPz888/lNn0bmqh/++03lYZYPmfEiBFmFxYFBQX417/+pY6TKXH/+c9/MG7cOIwePbrKS1G3bNlSXSxERUXh66+/Nrs4kVYFSSMp3z8kJER9psHHH3+svouLi4s6H3fffTesEQN1TWDzN1H9XLQir8AiW02u7PzCCy/gzTffxJEjR9CxY0dkZGTg5ptvxtq1a7Fv3z4VQEeNGqXyKlRk+vTpuPfee/HXX3+p1z/wwAO4dKn82S6y4Me7776rAqekMJb3f+6554zPv/XWW/j2229VboetW7fiypUrWL58eZW+27JlyzBp0iQ8++yzOHjwIP75z39i/PjxWL9+vXr+xx9/xKxZs/DJJ5/g+PHj6v07dOigntuzZ48K2pLoSVohVq9ejRtuuAHWiE3fNdX8vfdLIJYDyojqi+z8QrSd+ptFPvvwa8Ph5lQzP88SiG666SbjY19fX3Tq1Mn4eMaMGSrgSQ1Z0g6XRxIljRkzRt1/44038OGHH2LXrl0q0JdF5g7Pnz9f1XaFvLeUxeCjjz7ClClTVC1dzJkzBytXrqzSd3v33XdVuZ544gn1ePLkydixY4faf+ONN6qLA2ldGDp0qFp7W2rWPXv2VMfKc+7u7rj11ltVy0OzZs3QpUsXWCPWqGuyRn3hAJCTZunSEBEZde/e3eyx1KilZitN0tLsLM3SUtu+Vo1aauMGEuC8vLyMS2SWRZrIDUHasIym4fi0tDQkJiYag6aQlbukib4qjhw5gn79in9/i8lj2S/uueceZGdno0WLFirrolyQSJO7kIsXCc7y3IMPPqhq99IKYI1Yo64JjUIBn+ZA6mng7E6g1TBLl4iIqsnV0V7VbC312TVFgqopCdJr1qxRtc6IiAi11KX0zebl5VX4PlIjNSV90kVFRVU6vq6TNYaFhalmbemDl+8sNe933nkHGzduVLXovXv3qv7133//XQ3Ek/5sGfFubVPAWKOuKVE3A61GAE7m/ymIyDZJYJHmZ0tstblCmvQHS3OxNDlLf600DZ85cwZ1SQa+yeAtCYoGMiJdAmdVtGnTRn0fU/K4bdu2xsdyISJ98NJUL0F5+/btiI6OVs/JCHhpFn/77bdV37uch3Xr1sHasEZdU0a8YekSEBFdk4xy/umnn1TwkguCV155pcKacW156qmnMHPmTFWrb926teqzTk1NrdJFyr///W81wE36liXg/u9//1PfzTCKXUafywVAr169VFP8N998owK3NHn/8ssvOHXqlBpA5uPjo/rH5TzIyHFrw0BNRNSAvP/++/jHP/6hFinx8/NT06JkxHVdk8+V1KJjx45V/dOywMrw4cOrlGVq9OjR+OCDD1Qzvoz+bt68uRpFPmjQIPW8NGHLiHcZZCYBW1oQJJjLdDB5ToK6NHfn5OSoC5jvv/8e7dq1g7Wx0+q608DC4uPjVb9FXFwcmjRpUu33Kygsgr1OvwqQcjkO0DkAXtXLP0pEdUd+qE+fPq1+6GVOLdU9qc1KU7bUkGUken3/u4qvQixiH3U1PL/0ALrOWIOD54qvRle/CMxuD+xaYOmiERFZtdjYWHz66ac4duyY6jOeMGGCCmr333+/pYtmdRioqyE1Kx9Xcgqw8VjxFIXAdoCdPZB10dJFIyKyajqdTvUhy8poMqVKgrX0LUutmsyxj7oaBrbyx5rDidh4LBlPDo4E2o0G2t4GOHtaumhERFZNmn1Lj9imsjFQVzNQi71nLyMtOx+NXDk1i4iIahabvqshzNcNLf3dUVikYeuJFPMnLTDdgYiI6h8G6moa2CpA3W48mqzfce5P4NPBwFe3WbZgRERULzBQV9PAKH3zt/RTq5luLt76YB23E8jPtnTxiIjIxjFQV1Ov5r5wdtAh4UoOjiamA74tAM9goDAPiC9ZHo+IiMjmArUsHydD82Vx9ICAALXKjCygfi0//PCDWnJOJpDLSjNVTY1Wk1wc7dGnZeOS5m9Z+ETSXoozHNFIREQ2HKglg8nEiRNV/lDJbCL5S4cNG4bMzMxyX7Nt2zaVE/Xhhx9WSc8luMsmScMtPfpbmr/N0l6e2WKxMhERVZYsufn0008bH4eHh2P27NkVvkZWY1y+fHm1P7um3qciskxo586dYassGqhXr16tsrjI2qqSyFwmv0tO1D///LPc18i6rpKoXBZjl4nxstRc165dVdJxSwfq3WcuITO3oKRGLU3f+TkWKxcR1W+SWEN+D8uyefNmFQQlK1RVSVYrWXu7LoLlhQsXMHLkyBr9rPrGqvqoJZm48PX1LfcYSVEmWVJMyULusr8subm5asF5w5aenl7DpQaa+7mjqa8b8gs1bDt5EWgcAXgEAoW5+oFlRES1QFoWpTVS1o0uTZJTdO/eHR07dqzy+/r7+6tsU3VB0mw6OzvXyWfZKp01LcguTS+ylFz79u3LPU6yrUgeU1PyWPaX1w8uuU8Nm2me0poiV60lzd9J+n5qNn8TUS279dZbVVCV1khTGRkZaiyPBPKLFy+q7sLQ0FAVfGVcj2SJqkjppu/jx4+rdJAyLkh+Q+XioKxsWK1atVKf0aJFC5U+U7ozhZRv+vTpOHDggPq9lM1Q5tJN37KU6ODBg1U6Ssly9dhjj6nvYyCtsNLdKRmzgoOD1THShWr4rMrGm9dee00lw5CLBKnpSwuvQV5eHp588kn1/vKdJS2mxBIhs3ukdaBp06bqtSEhIfjXv/6FBhGo5URLP/OiRYtq9H2nTJmiauqG7fDhw6gNhkC94WjxNK3w4kAdy0BNZNPyMqu+FRaUvF7uy77S0zXLe20VODg4qDSREvRMEyFKkJa0jhKgJYNTt27d8Ouvv6rfWAl8Dz74IHbt2lXpoHbnnXfCyckJO3fuxPz581VQLk0GBUs55DdWuigl4casWbPUc/fddx+effZZ1c0pTd2yyb7SZHyStJBKfmhpfpfv8ccff6igaWr9+vU4efKkuv3yyy/V55a+WKmIlO+9995TwV66BuQzb7vtNnVBIj788EP8/PPPWLJkiRrg/O2336qLF/Hjjz+q7/XJJ5+o4+UiQy5+6v0SovKPIEm8N23adM10X9JMkpiYaLZPHsv+ssgVj2mzSm3lXZWR3072OsSnZuNUSiZaNivup47bDRTkAg5s2iGySW+EVP019ywE2t2hvx/zP+CHhwD5TRj/a8kxszuUncDnVX0XYGVJbul33nlHDc415GGWZu+77rrL2JL43HPPGY9/6qmn8Ntvv6kg1LNnz2u+vwTKmJgY9RqpPYo33njjqn7ll19+2Xhfgpp8plS8nn/+eVU79vDwUBcW5f1Wi++++05dWHz11Vdwd9cvyTxnzhzVF//WW28ZW1MlkMt+yV0tM4BuueUWrF27Fo8++milzpkEaLnY+Nvf/qYey3tL0JdWhLlz56qxUpKfun///qrGLzVqA3lOvoN0wTo6OqqadWXOo83WqOUKUIL0smXLsG7dOpWz81r69Omj/kFMSTOM7Lckd2cH9GjuUzJNyz8KcPMDCrKBc3stWjYiqr8kUPXt2xeff/65enzixAk1kEyavYXUrGXQrdT6ZPyPBEwJuhJwKuPIkSMqgYYhSIuyfm8XL16sui4liMlnSOCu7GeYfpYMLDYEadGvXz9Vqzeduis1cwnSBtJEnZRUnMXwGqSydv78efW+puSxfL6heX3//v2IiopSzdq///678bh77rkH2dnZqnlfLgwkfhUUmLSg1LcatTR3yxXUihUrVLOJoZ9ZrgDlCkxIs470rRj6ByZNmoSBAweqZgu5ipIrtj179mDBAsvngJbm760nLqppWv/o31zf/H14hb75u5llLySI6Dq9eL7qr7E3aUFrPUr/Hnal6kVPR6OmSFCWmrLUBqU23bJlS/U7KaS2LU29UluUYC1BUMYDST9sTZHBvA888IDqh5ZmZPkNl99m+Z2uDY6OjmaPpdYrwbymyEwiyY29atUq1aJw7733qhr00qVL1UWLXDTIfqkkPvHEE8YWjdLlqhc16nnz5ql+Y2mukSsiwyZXZgZyRSb9GQZy5SjBXQKzXHnJiZM+gooGoNWVQVH6db93nLqInPxCfVOXofmbiGyTk3vVN3uTOpDcl32OrpV73+sggUTyO8tvozQbS3O4BC8hqSRvv/12/P3vf1e/mVITPHbsWKXfW6bBxsXFmf0Oy9oXpde3kObhl156SY00l2bj2NhY86/r5KRq99f6LBlwZrqWxtatW9V3k9ptTfDy8lKtA6VTbMpj08HGcpz0o0tfu8Qk6Zu+dOmSek4qktIcL33ZGzZsUBcqMgiuXtaoTQc/lEdOQmnS9CCbtYkM8EBwIxdcSMtRwXpQ29uB0G5AcCdLF42I6jFpapagIoNnpWlXmm4NJGhKhUaCqfTtvv/++2pcT2VnwEhNUkZzjxs3TtUc5f0lIJuSz5BKldSiZbVJGbgmTcKmpN9aaqnSpCxjkaQVtfS0LKmVT5s2TX2WjKxOTk5WLQUy+K30bJ/qkHU45HOk5UFGfEsrhJRLBo0JOUdSaezSpYu6SJBBbdKk7+3trQatyQVHr1691Aj3b775RgVu037sejvquz4wn6aVDHgGAk26mV9dExHVAmn+Tk1NVU3Ppv3J0lcsTbmyX1ovJeDI9KbKkkAlQVf6ZWXQ1COPPILXX3/d7BgZMf3MM8+oMUcS+OSiQKZnmZLBbbI4y4033qimlJU1RUwCn/SfS81VAv7dd9+NIUOG1PiCVtLvPHnyZDUSXboDZGqWjPKWCw4hFxFvv/22ah2Qcpw5c0YtVS3nQoK11LKlT1vmqEsT+P/+9z81Tay22GmVqdbWI7IwgPQxSFPOtUaYX49V0Rcw4du9aOHvjnXP6kdgEpF1k5HGUtuTAa0yb5aotv+uqhKLWNWrYf0i/WCvs8Op5EzEXcpCWGE8sP0jwM4eGFXx2rlERESlsem7hnm5OKJbU/00rQ3S/C3LiO79Coj+wXwRBCIiokpgoK4FA6P8S+ZTB7QD+k8G7pY5jg2ql4GIiGoAA3UtMAwo23YyBblFGjB0GtBqOGBfO3PsiIio/mKgrgVtg73g5+GMrLxC/Hkm1dLFISIiG8ZAXQt0Ojvc0MqvZJpWUSFwYi2w7nX9fSKySjW5uhVRUQ39PXHUdy2uUvbT3nMqUE8Z0Qr4YTyQmwa0vhkI6WLp4hFRqVWzZI6srAEtc3zlsWFlL6KqklnPskSrLNgif1fy91QdDNS1ZECEn0pLHZOQjgvpeQiWtb6PrQbObGWgJrIy8mMqc11lmUwJ1kQ1QRZwkexa8vdVHQzUtcTH3Qmdmnhjf9xlbDqWjPua9SsO1FuAvua5VYnI8qTWIz+qkgnpWmtSE12LZPeStJ410TLDQF3Lo78lUEvz932DilOqnd2m76fWlaRoIyLrID+qkgGptrIgEV0PDiarRYOK51NvPp6CgoAOgJMnkJMGJB6ydNGIiMhGMFDXoo5NvOHt5oj0nALsO5cBNO2tf0Kav4mIiCqBgboWyZrfAyJNVikLL27+jjXPg0pERFQeBupaNqh4lbINx5KAZv1LAjXnaxIRUSUwUNeyAcULnxw8dwXJnm0AR3cgOxVIOmzpohERkQ1goK5lAZ4uaBfipe5vPnUZaNpL/wSbv4mIqBIYqOtw9LdaTlTmUwsOKCMiokpgoK4DA1sFqFtZ+KTQtJ9aY9pLIiKqGBc8qQNdmnrD09kBqVn5OKi1QKfI4fom8IJcwNHF0sUjIiIrxkBdBxztdegf6YdVBxOw4UQaOj2wxNJFIiIiG8Gm7zpcTtQ4TYuIiKiSGKjryA3FgfpA3GWkZuYB6YnAoeXspyYiogoxUNeREG9XtAr0QJEGbD12HvigI/DDOODiCUsXjYiIrJhFA/WmTZswatQohISEqKw1y5cvr/D4DRs2qONKbwkJCbAFg6L0o7+lnxphvYCgjkDWJUsXi4iIrJhFA3VmZiY6deqEuXPnVul1R48eVQneDVtAgD4A2ko/tcynLnrgR+DxzSULoBAREVnbqO+RI0eqraokMHt7e8PWdA/3gZuTPZLTc3EkKQvtQhpZukhERGTlbLKPunPnzggODsZNN92ErVttZylOZwd79G3ZuGSVMpGfDeRlWbZgRERktWwqUEtwnj9/Pn788Ue1hYWFYdCgQdi7d2+5r8nNzcWVK1eMW3p6OqximpakvVz5PPBmUyD6B4uWiYiIrJdNLXgSFRWlNoO+ffvi5MmTmDVrFr7++usyXzNz5kxMnz4d1rWc6CHsjU1FbnMPOBfm6ZcT7TbO0kUjIiIrZFM16rL07NkTJ06UP8VpypQpSEtLM26HD1s2vWTTxm5o4eeOgiINB+w7lCTo4HxqIiKqj4F6//79qkm8PM7OzvDy8jJunp6esJbFT35JbQLoHIEr54DUM5YuFhERWSGLBuqMjAwVaGUTp0+fVvfPnj1rrA2PHTvWePzs2bOxYsUKVYM+ePAgnn76aaxbtw4TJ06ELRlYnPbyj+NXoIV21e9k2ksiIrK2Puo9e/bgxhtvND6ePHmyuh03bhwWLlyo5kgbgrbIy8vDs88+i3PnzsHNzQ0dO3bEH3/8YfYetqBPi8ZwdtDhfFoOUtv1hG/cTn0/ddcHLV00IiKyMnaa1rA6R+Pj49Vo8bi4ODRp0sRi5Rj7+S6Vn3pe78sYuf8JoFFT4Jloi5WHiIisMxbZfB+1rTJM01qaFArY2QNpZ4HUWEsXi4iIrAwDtYUD9ebYbBSGdNHvlOZvIiKi6gZqqapLtd1g165damDXggULruftGqSW/u5o4uOKvMIixHsVB+ozDNRERFQDgfr+++/H+vXr1X3JXCVLeUqwfumll/Daa69dz1s2OJL1y1Cr3pRbvIjLmc2WLRQREdWPQC1To2ShEbFkyRK0b98e27Ztw7fffqtGa1PlGAL1dwkh+n7qy7FAWklLBRER0XUF6vz8fLWQiJDpUbfddpu637p1azWliiqnb4QfHO3tcOQSkOvfAXBwAZKPWrpYRERk64G6Xbt2KjnG5s2bsWbNGowYMULtP3/+PBo31meHomvzcHZA92a+6v7/omYCL5wFIoZYulhERGTrgfqtt97CJ598ojJXjRkzBp06dVL7f/75Z2OTOFVtlbJfzzoADvpWCiIiomqtTCYBOiUlRaWN9PHxMe5/7LHH1IphVHmDovzx5qoYbD91ETn5hXBxtNcn6LCzs3TRiIjIVmvU2dnZKs+zIUjHxsaqdbiPHj2KgABJ40iVFRXoiUAvZ+TkF+H8yreBub2Bgz9aulhERGTLgfr222/HV199pe5fvnwZvXr1wnvvvYfRo0dj3rx5NV3GBjNNK/F8LJB8hAk6iIioeoF67969GDBggLq/dOlSBAYGqlq1BO8PP/zwet6yQRsUpW+F+DyjN3Dv18DgVyxdJCIisuVAnZWVZczr/Pvvv+POO++ETqdD7969VcCmqukX4Qd7nR3WXPRHfPBQwJ0j54mIqBqBOiIiAsuXL1dLif72228YNmyY2p+UlAQvL6/recsGrZGrI7qEeav7m46lWLo4RERk64F66tSpeO655xAeHq6mY/Xp08dYu+7SpXjdaqoSQz/14eg/gQ1vAjs/sXSRiIjIVgP13XffjbNnz2LPnj2qRm0wZMgQzJo1qybL1+D6qTPiooENM4E9n1u6SEREZKvzqEVQUJDaDFm0JPE1Fzu5fu1CvNDY3QkbMyMBFwDJMUBmCuDuZ+miERGRrdWoi4qKVJasRo0aoVmzZmrz9vbGjBkz1HNUdTqdHW5o5Y9UeCHJtaV+J/NTExE1eNcVqCWd5Zw5c/Dmm29i3759anvjjTfw0Ucf4ZVXOLWoOquUiR1FbfQ7OJ+aiKjBu66m7y+//BKfffaZMWuW6NixI0JDQ/HEE0/g9ddfr8kyNhj9I/zUyqGr0lviNicJ1KxRExE1dNdVo7506ZJKaVma7JPn6Po09nBGx9BG2FVUfG6TDgFZPJ9ERA3ZdQVqyZYlTd+lyT6pWdP1GxgVgItohAtOzfQ7YrdZukhERGRrTd9vv/02brnlFvzxxx/GOdTbt29XC6CsXLmypsvY4OZTf7j2ODblReE+xOr7qdvcauliERGRLdWoBw4ciGPHjuGOO+5QSTlkk2VEDx06hK+//rrmS9mAdGrSSK1UtjkvSr8jlgPKiIgasuueRx0SEnLVoLEDBw7gv//9LxYsWFATZWuQHOx16B/ph51/FY/8TjgIZKcCriV5v4mIqOG4rho11a5BrfyRDG/E2zcBoAGx2y1dJCIiaoiBetOmTRg1apSqnUteZkn0cS0bNmxA165d4ezsrJKDLFy4EPV13e9Nea30O7jwCRFRg2XRQJ2ZmalGkM+dO7dSx58+fVoNYrvxxhuxf/9+PP3003jkkUfM1huvDwK8XNAm2Av/K+yDI1ETgfZ3WbpIRERkC33UMmCsIjKorCpGjhyptsqaP38+mjdvjvfee089btOmDbZs2aISgQwfPhz1bZWyeRfaYYEuFLNCO1u6OEREZAs1alnbu6JN1vweO3ZsrRVWpoANHTrUbJ8EaNlfb5u/jyWjqEizdHGIiMgWatRffPEFLCkhIQGBgYFm++TxlStXkJ2dDVdX16tek5ubqzaD9PR02IJuzXzg4eyA/MxLiNu2BM38PIHWN1u6WEREVMfq/ajvmTNnmtX627ZtC1vgaK9Dv4jGGKzbj2Z/PAZsftfSRSIiIguwqUAt+a8TExPN9sljLy+vMmvTYsqUKUhLSzNuhw8fhq0Y2CoAO4vaIM4+DAjtDmhsAiciamhsKlDLcqVr164127dmzRrjMqZlkWlcEsgNm6enJ2zFwCh/XEBjDMx6C2mDXodKrUVERA2KRQN1RkaGmmYlm2H6ldw/e/assTZsOjjt8ccfx6lTp/D8888jJiYGH3/8MZYsWYJnnnkG9VGotysiAzwgY8m2nEixdHGIiKihBeo9e/agS5cuahOTJ09W96dOnaoeX7hwwRi0hUzN+vXXX1UtWuZfyzQtyYtd36ZmlTX6e0vMOSAh2tLFISKiOmanaQ2r4zM+Ph5hYWEq01eTJrJEp3XbfDwZT/93Dba6TIKzrgh2L5wFnNwtXSwiIqqGqsQim+qjboh6hPsiy9EXKZoX7IoKgLidli4SERHVIQZqK+fiaI8+LRtjZ1Fr/Q7JT01ERA0GA7WN9FPvKCqe/32GCTqIiBoSBmobCdQyn1po5/4E8rIsXSQiIqojDNQ2INzPHTqfcFzQfGFXlA/E77Z0kYiIqI4wUNuIgVEB2FFcq2Y/NRFRw8FAbUOrlBmbv2MZqImIGgoGahvRu0Vj7LXTDyjT4v8E8nMsXSQiIqoDDNQ2ws3JAYHh7ZCoeUNXmAuc22PpIhERUR1goLaxfmpD8zf7qYmIGgYGahsyyKSfuvA0AzURUUPAQG1DWvp74JR7F+Rp9kjLLWJ+aiKiBoCB2obY2dkhPKozOuZ+hg9D3mF+aiKiBoCB2gb7qXPgjE3Hki1dFCIiqgMM1DamX0RjOOjscColE3EJKZYuDhER1TIGahvj6eKIQU3s8IvTiwj6tANQkGfpIhERUS1ioLZBXdtEINjuIhwLs4DEg5YuDhER1SIGahs0KCoQ/8x7BgOL5iE3sJOli0NERLWIgdoGtQn2RKxHJ8TmNcKeM6mWLg4REdUih9p8c6q9aVqSo3rpn/HI3fAesCUaCGwPBMnWAfBvDTg4W7qYRERUAxiobXiVMgnUrhd2AoV/Amc2lzypcwD8WpUEb3XbAfAIsGSRiYjoOjBQ26j+EX5qmta0rHvRUdcdbXVn0c35HCK1M3ArvAIkHdZv0UtKXuQeoA/cHf8GdLrPksUnIqJKYqC2Ud5uTvhwTBcs3xeAjXERWJqeC+TLMxqCcAltdbHo4hSPnm7nEVl0Bj45cbDLTAJOrgOa9i15o9RYYPHfgdBuwKjZFvxGRERUFgZqG3Zzh2C1aZqG82k52H/2MvadTcX+OF9sPeePdTldgeK01a7IQZRdPPp7XkDRmRYIdDyDLk290SbtLzgm/KUCvJnvimvcxubzDoBvc0BnX/dflIioAWOgrieDy0K9XdV2S8dgtS+/sAgxF9KxPy4V++IuqyC+P8UF+69EAFcAHDmkjgt0yMKdjV9Gc3d3uB44r4J3qKcD7KTmXZgHHFtd8kGObkBAW/N+bxm45upd699RLkbScwtwMSMPFzNykZKRh+z8AvQI90UTH7da/3wiIkux0+QXsAGJj49HWFgY4uLi0KRJEzQkl7PysF+CdvG27+xlpGWr9nIzge4OuDPwPHq7XUBrnIZf5nHYJ8cABdllv7FHoH7w2tDpQJNu+n2F+fpBbRUkDsktKMSlTAm8eUjJyNUH4Uz9bUrxfeP+jDzkFRaV+T6dmjTCiPbBGNk+COF+7td5doiIrDMWWUWgnjt3Lt555x0kJCSgU6dO+Oijj9CzZ88yj124cCHGjx9vts/Z2Rk5OcVtvNfQkAN1afJPf+ZiVnFzuT54Hz5/BQVF5n8SEmtb+7thaGAGertfQGu7WPimH4OdrIqWft54XNEj65Hm014FWPvdnyJs3zs4GnoXfmvyL1ULvpieC6e0kzic0xiJmYVIzymocpk9nB3Q2MMJjd2dVGO9lNn0L7hNsBdubh+EkR2CEBHgWb0TRERUS6oSiyze9L148WJMnjwZ8+fPR69evTB79mwMHz4cR48eRUBA2dOJvLy81POmTb9UdXLemvu5q+3Orvo/lJz8Qhw6n6Zq24Ym83OXs3EkKQtHknT4CKEAQuHuNAAdmjSCp2c2XK+cgk/2Gfz48RlkFF1Q7/OawzaMdcjCppOX8eHR42qfP1Kx22Ui8jV7xGqBOOkYglMIRaJTU6S6NUeWVwt4ePmoINzYw1kFZD8VlJ3h5+ms9rs4mveRJ6fn4vfDCVgVnYDtpy7iyIUrantvzTFEBnioWvbIDsFoHeTJvxMiskkWr1FLcO7RowfmzJmjHhcVFamrjKeeegovvPBCmTXqp59+GpcvX76uz2ONuuqS0osHqhUH7r/iLyMzr7Dc4xu5OiLQ3Q7tXC7B1d0T9j5NVdBtVXgcw3Y9AgdZo7w8niGAfyt9U7phC+sFOLpcs5ypmXlYczgRKw9ewNYTKcgvLPnTDm/spgK2BO4OoY0YtInIomym6TsvLw9ubm5YunQpRo8ebdw/btw4FYhXrFhRZqB+5JFHEBoaqoJ6165d8cYbb6Bdu3ZlfkZubq7aDM6dO4e2bdsyUFdDYZGG40npiI5Pg4O9narx6mu/zvBxc4KTQwUr0xYV6ZvLk48CKceBlOJbeSzTx8ry3PGSxVoO/gRcPgtE3gQElv1vLqTvfe2RRKw6mICNx5KRV1DSvy2D7gw17S5h3tDpGLSJqG7ZTNN3SkoKCgsLERgYaLZfHsfExJT5mqioKHz++efo2LEj0tLS8O6776Jv3744dOhQmV925syZmD59eq19h4bIXmeH1kFeaqsynQ5o1ES/RQwxfy47tTh4HysO5MeA9ATA3b/kmL8W60eiO7mXBOpLp4F93+jngsvmGahq9dKcL1tGbgHWxyRh1cELWB+TrJryP9tyWm1BXi4Y0T5IbTKCXL4bEZE1sWiN+vz586pmvG3bNvTp08e4//nnn8fGjRuxc+fOa75Hfn4+2rRpgzFjxmDGjBlXPc8adT2z61Pg7A6gzxP6oCz2fg38/GTJMY3CgNCuJYE7uDPg7KGeys4rxMZjErQTsPZIkgriBtIfPqxdEG5uH4xeLXzhaM+cNUTUwGvUfn5+sLe3R2Jiotl+eRwUFFSp93B0dESXLl1w4sSJMp+XEeGyGVy5IpOIyWb1fFS/mWrcEujyd+DcXiDpCJAWp98OF3ed2OkA/zYqeLuGdsMI2e7pgJwiO9WXvTI6AWsOJ6gpYd/tPKs2bzdHDGsbiJHtg9Evwq/i5nwiolpk0UDt5OSEbt26Ye3atcY+aul3lsdPPmlSQ6qANJ1HR0fj5ptvruXSktVq1le/idx04MIBIH4PcO5PffC+Eg8kHdJv+77WHxfSBS6PbcCQNoFqy7scgO2J9lh9KAG/HUpU87uX7IlXm6eLA4a2kaAdhBta+V818pyIqDZZfHqWTM2SwWPdu3dXc6dlelZmZqZxrvTYsWNV87j0NYvXXnsNvXv3RkREhBpwJvOvY2Nj1QAzIjh7AuH99ZuB9HOroG3Y9poPRCvMh9Oczhjo5IGBj2/BjNvbY9fpS/gtOh4rD6eoKWDL9p1Tm5uTPQa3DlA1bUmM4uZsr5KjcBQ5EdXbQH3fffchOTkZU6dOVQuedO7cGatXrzYOMDt79ix0MgCpWGpqKh599FF1rI+Pj6qRSx+39DsTlckzCGh9i34zjDzPzyx5XgajFRUCRflqlTUHnQ59I/zQd//zeNVzPy6Ftceu/Ob4MSEQm9OD8ctfF9RmysleB0d7Ozg6yK2u5LG61an9TqaP5RgHO/VZhvtmzxmONb7f1e/l7+mMVoGe8HRxrOMTSkQNah51XeM8aipTfo5+2pfM4TaY3RG4HGt2WJHOEYmuEdie2wy7spsgQfNBkuaDRM0Hl+AJDXXfly3TzaKCPPVboP62hb87nB3YRE9krWxmHrUlMFBTpWVdAs7v0zeVn9uj7/fOSin3cE3ngOT+M5DS+u8qKYrd5bPwPvETMjzCcT50pNon65XnFxQhv0hTj2VRlnzDPvW8YX/x44JSj+X5Av37nEvNRsKVspfOleZ4WXGuVZAnWgd66m+DPBHm48Z540RWwGZGfRNZNTdf/Vxvw3xvuaaV0eTSzy1BW+Z6ZyQA6YlAZjLsigoQ4B+AgJDi+eWZ24ADs4CQrmh700Ml7zunB5CXpW+SN918gwEPw+Ng/edfo+9bEq0cS8zA0YQrOJqYjqMJ6YhJSFfrqB9PylDbryhppnd1tEerQA9V65Zmc5kL3yrIA/4ezuxnJ7JSDNRElSWBzLupfmt3h/lzki0sIwlwMVkExjMQ6PKg/ngDCfaX4/SZyGQ0ekV0jiVBfMCzQNRI/f7MFOD8fsA7DN7+UejZ3FdtJR+hqZq2BG3jlpiugnZ2fiEOxKepzZSvu5MK4CpwFzefyyZJUIjIsvi/kKgm2DsCjSRhiQnDgiulPbVHPxJdbReAjET9rXpcfF+a2GVwm2FOeL7J+uiy4MviB4AmPYFH1pTs/3QwUJALOzdfBLv6ItjNF4PcGgNNfYHWvih08cGFfA+cSHfCocuOiE4uwrGkDJy5mKmmo+04dUltZl/B21U1mRuaziWIt/T3uO555XIRIUvQSoY289si/W1h2fsNm2G/TJHj8q/UUDBQE9V1rdywhGpFCvL0a58bArqstGagswcC2gGNI8xfk3i4/Jzhci0BoEnxNki9jwNw62zkdLgfJ5IycP74Pvgf+i+O5Afjw6zhqlYuy602SjuC00edsEjzQBo8oNPZo1ljNxUsywqixqArjwvN95fKoFotLf3d8c+BLTG6cygXpKF6jYPJiOoD+W8sA9+yLwFZqUDWxeL7l0rdv6S/b6ih3/050P4u/f3DPwNLHtRnK3v4d9X/Lc3m7Rf3hnuuPmFKEexwRXNDquaBbLggF47I0Zz0t9Df5mqOWFbUH9uL9HPVg3ERt9pvR5LmjRVFJfPbe9jFwMGuUB2fCycU6AybCwrsnFCoc1aj7O3tdWoNdhkgp7/V4fzlbKQXL/8a3MgFjwxogb/1CIM7m+rJRnAwGVFDrKmb1rqvJT9bH7RdGpXsk5SiN75szFTm7eaEXi0aA16+wJVcIDcNOmjwtstUW0VuuGEkMtoPVMHVLX4zApZ/hwK/Npj60Ksq0Nrb28FtwTToLupzlZdJEp4VSdO2C2DnDOhcgX6TgN4TkJ6Tj0XbTyJmy0/4I60lZvySg4/WHce4PuF4qG84fNydKn8uiKwcAzVRQ+ToenWfekBr/VbaxJ0lA+Ykw5mxVp4NFOQUb7nFj3PV46CIfkCAPhEKCpoAHe+Dg2cwGnuUrLsP3xb6ZnyT1xk3I03fnC9bzmXjc7LIy6ORGcDGt5Dr6Y3hjl/gzKVsfLD2OL7edBi394zEowNaIMTbtebPHVEdY6AmosoPmJPatiE3eGUFtQfuXHD1/geWlN+MX5hXKoDLbbZaOc4oJw3wi4KzXyTW3nsjVh9MwMfrj+OTS+ORs9sJG3e1QVHTvug7ZBSat4iq4pclsh7soyYi2yY1fbmIkBifdg52s65eTjjZIRi65v3QuO1gILwf4N3smnPUiWoT+6iJqOEoDtLCTprznz+tprAlRa9F1onNCMs5Bv+CC8DxpfpNArpXKOya9dNnXZMELjKCnoGbrBQDNRHVL7KiW+ubEdBan/r2ZPx5rP/9fyg4vQU97I6go90pOF45B0Qv0W/imUMlU+bUIDtvwCQZEJElMVATUb3WskkIWv7jnzh/eSw+23waj+w6jjaFMeili8FAp6No7poNF/dgGIe5LXsciN8F3DYHaHMrGrqM3AKcTMpQc+1PJGcg7lKWGs0v8+hLNp1antZw3/Q5V5N9ct/Z5L5kg6NrY6AmogZBRoBPHdUWTw2OwJfb2+CLbWcwKysfdllF8H9rPR7u3xz39wyDZ0K0fnS76aj4gz8BB77XN5VLk3lwZ8Ch/kwBk6FKKRl5xmBsCMwnkzNwIa3sxC81QebFuzjo4Opkr7K9qYDvZA8Xdd888LsW3/d1d0a/iMZoH9KowaxMx8FkRNQgZeYWYNHuOHy2+ZQxGHm6OOChXqF4uGUavFv2AuyL6zIrJgL7vil5sazqJtPLZO652RZhPjfdyhQVaYhPzcaJ5HR9IE7KVIFZ7qdl55f7Oj8PZ0QEuCMiwAPhjd3Vvuy8QuQUFCInv0itIZ+TX4hck/uyZecXIdd4X3+svKYmok5jdyfc0Mofg6L8MSDSX61Xb0uY5rICDNREZCqvoAgr9p/D/I0ncTJZv5CLs4MO93YPw2M3tECYrxuQdAQ4uR6I3arfpMZdHsmA5hcJtL5FLc5iJD+1dTRgLbegEKdTMvWBuLiWLLenkjOQWyAryVxNiiZpUCUYy/Kscqs2f080cisZsFddEnKkDLnFQdss4BffzzUN7Pkl92W/fK9tJy+qJnnTsndq4q2C9sBW/ujYxFvV1q0ZA3UFGKiJqLza5pojifh4w0kciLus9smP/aiOwXh8UEuVWaz4QCD9vD7NacpxIOVY8XZcn/bUoPs/gFtn6e/nZQLvttLXwv/xG+Dkpt8vSVikBu7ocl1lvpKTb9Z/bLh/9lJWueuqO9nr0MLfXSVXaWkMxh5qnzQx28rF1Z+xqdhwLAkbjyar1K6mfNwcjbXtGyL9zRfasRIM1BVgoCaiishP4vZTFzFvw0lsPp5i3D+4dQAmDGqJHuElKUWvIouwpJzQB27f5kDT3vr9Fw4An9wAuPkBz59EfqG+idh50X1wOrMO+V5hyPZqiUzPFrji0RypbuFIcW6GNDsv5BToa5pyvNryClUgloCclJ5bblE8nR1KAnFxMJZbaSGw9tpmVSWk5WDjsSRsOJqMLcdTjOvAG2rbHUIbYVArfwyMCkDnMOuobTNQV4CBmogq6+C5NMzbeBIroy8Y+1W7N/PBrR2DVVaw3FJBNKdUQDU02+bn5cM3/zw88i9ha34r9Vrxq9MUtNPFlvv5kvzkpBaCk0UhOCG3Wgiii5ojGT7qeWfkIcojG2F+XmgcHG4MyFGOifB1LoKdVgTIJq0A6n4hUFRYct/0ucYt9ZuQpv2T6wB7Z/OR70dX6dOwynuorcBkK+exDMBrN7pk6tvK5yT0AHf/t+R9170OnN1ewXvmlzy2d9LPe4+8Cej1z6vOmVwE7Y1NxcZjySpwH75wxez5Rq6OGBDph0FRAaqZ3N/TMrVtBuoKMFATUVVJv+iCTSfx45/nkFdYdh/v9bCz09DEMQOtHRIQqbuAlrrzCNfOIawoHn6FSSoJSmnbwifiXPsJKii3ytwD98V3A4HtgQlbSw76sCtw6WTVCiMJWQb+W39fRr7P769fsvW5YyXH/HcYEFe89ntl9fwncPPb+vuSsvW9KMDOHphmkvt80QNAzC9Ve9/ODwCjPy5JC/tBJ/2Fxt++A1yKuynyMpGUrcOG4ykqcG8+lowrOSW1bdE+1AuDWgVgYJS/ynHuUEdTxrgyGRFRDWru546Zd3bE00Nb4cttZ3A8KUNNF1KbTCcy3i+ZQ1z28yX7ZT6xDFqzK2+AWV6WPtga+r+L+8L79rkBiArTH3PaBXBwMVudTXH3A/IyADudPijKrSzgYvZYbmWz0983XcPdyQMIHwC46mvuRlI7luZ7OV5Gvsvnyq3hsXEzedykR8nrnb2AEW/q95vqMxFof2c57+Fovk++l+paaFHy+kun9OMGctMBZ8+S/cv+iYBTG3GvXyvc6x+FwiGROIVQbLzoi5/POuCv85k4eO6K2uasPwEvFwc1glyCtjSVB3hd39iBmsYaNRER2baCXCDxIJCRDESNKNk/tzeQfKTs19g7o8C3JS44NkN0biA2XPLBgZxAnNaCkQf9hU+bYC81IE2CdtdmPjW6QAubvivAQE1E1IAC+EVplTgKJB8rvi0erV9Y9kC8HU3+gZk5d+Gvc2lopKVjsG4fjmphOOsUif6Rfqpf+9ZOIfBwrl6DNJu+iYiIHJyBwLb6zZQMSrscaxK8jwHJMapJvXevfljRoT8uZuQiZstP6LdjvmouH5zzDlYdTMBvhxIwvF2QjOSru69Rdx9FRERkBXT2+j5u2UybyqWBWUbAy8pnHs7o1yoESBiAcN+WWN6lHzYcTULilRz41PEqaFaxIvrcuXMRHh4OFxcX9OrVC7t27arw+B9++AGtW7dWx3fo0AErV66ss7ISEVE9ZVc8sM6gxUDgoV+gu+0DNf9aBhPKoMK6ZvFAvXjxYkyePBnTpk3D3r170alTJwwfPhxJSUllHr9t2zaMGTMGDz/8MPbt24fRo0er7eDBg3VediIiotpm8cFkUoPu0aMH5syZox4XFRWpDvannnoKL7zwwlXH33fffcjMzMQvv5TMuevduzc6d+6M+fPnX/PzOJiMiIgsrSqxyKI16ry8PPz5558YOnRoSYF0OvV4+/btZb5G9pseL6QGXt7xREREtsyig8lSUlJQWFiIwMBAs/3yOCYmpszXJCQklHm87C9Lbm6u2gzS080XbyciIrJmFu+jrm0zZ85Eo0aNjFvbtqWG6RMREVkxiwZqPz8/2NvbIzEx0Wy/PA4KCirzNbK/KsdPmTIFaWlpxu3w4cM1+A2IiIjqcdO3k5MTunXrhrVr16qR24bBZPL4ySefLPM1ffr0Uc8//fTTxn1r1qxR+8vi7OysNoPLl/V5Zi9cuFDD34aIiKhyDDFIYt41aRa2aNEizdnZWVu4cKF2+PBh7bHHHtO8vb21hIQE9fyDDz6ovfDCC8bjt27dqjk4OGjvvvuuduTIEW3atGmao6OjFh0dXanP27Vrl4xy58aNGzdu3DRLbxKTrsXiK5PJdKvk5GRMnTpVDQiTaVarV682Dhg7e/asGglu0LdvX3z33Xd4+eWX8eKLLyIyMhLLly9H+/btK/V5Xbp0UQuqyPubvu/1kIFp0uctzemeniYZW6hMPF9Vx3NWNTxfVcPzZbnzJTVp6baVmGT186ht2ZUrV9QANen79vIqzn9K5eL5qjqes6rh+aoani/bOF/1ftQ3ERGRLWOgJiIismIM1NUgo8lljXLTUeVUPp6vquM5qxqer6rh+bKN88U+aiIiIivGGjUREZEVY6AmIiKyYgzUREREVoyBuhrmzp2L8PBwuLi4qLzaspAKlW3Tpk0YNWoUQkJCYGdnpxapofITyUiOdllQISAgQC2ve/ToUUsXy2rNmzcPHTt2VPNaZZPlhFetWmXpYtmMN998U/2fNF2Wmcy9+uqr6hyZbq1bt0ZdYaC+TosXL8bkyZPVCMC9e/eiU6dOKi92UlKSpYtmlTIzM9U5kosbqtjGjRsxceJE7NixQ61jn5+fj2HDhqlzSFdr0qSJCjaS237Pnj0YPHgwbr/9dhw6dMjSRbN6u3fvxieffKIudKhi7dq1U+tzG7YtW7agzlz3It0NXM+ePbWJEycaHxcWFmohISHazJkzLVouWyB/dsuWLbN0MWxGUlKSOmcbN260dFFsho+Pj/bZZ59ZuhhWLT09XYuMjNTWrFmjDRw4UJs0aZKli2S1pk2bpnXq1Mlin88a9XXIy8tTV+9Dhw417pN1w+Xx9u3bLVo2qn9kuULh6+tr6aJYvcLCQixatEi1PpSXUY/0pNXmlltuMfsdo/IdP35cdd21aNECDzzwgMpDUVcsnpTDFqWkpKgfBEPiEAN5HBMTY7FyUf0jC/dL32G/fv0qnXimIYqOjlaBOScnBx4eHli2bJlKnkBlk4sZ6bKTpm+6NhmDtHDhQkRFRalm7+nTp2PAgAE4ePBgnSQzYaAmsvJaj/wY1Gl/mA2SH9D9+/er1oelS5di3Lhxqq+fwfpqcXFxmDRpkhr/IANh6dpGjhxpvC/9+RK4mzVrhiVLluDhhx9GbWOgvg5+fn6wt7dXKcpMyeOgoCCLlYvqlyeffBK//PKLGjEvA6aofE5OToiIiFD3u3XrpmqKH3zwgRooReak204GvXbt2tW4T1oI5e9szpw5yM3NVb9vVD5vb2+0atUKJ06cQF1gH/V1/ijIj8HatWvNmijlMfvFqLpkvJ0EaWm+XbduHZo3b27pItkc+f8oAYeuNmTIENVVIC0Qhq179+6q31XuM0hfW0ZGBk6ePIng4GDUBdaor5NMzZLmNfkD79mzJ2bPnq0GsIwfP97SRbPaP2zTq8/Tp0+rHwUZINW0aVOLls0am7u/++47rFixQvV/JSQkqP2SB9fV1dXSxbM6U6ZMUU2T8neUnp6uzt2GDRvw22+/WbpoVkn+pkqPd3B3d0fjxo05DqIczz33nFoHQpq7z58/r6blygXNmDFjUBcYqK/Tfffdh+TkZEydOlX9kHbu3BmrV6++aoAZ6cn81htvvNHsQkfIxY4M0iDzBTzEoEGDzPZ/8cUXeOihhyxUKuslzbhjx45Vg3zkYkb6ECVI33TTTZYuGtUT8fHxKihfvHgR/v7+6N+/v1rnQO7XBWbPIiIismLsoyYiIrJiDNRERERWjIGaiIjIijFQExERWTEGaiIiIivGQE1ERGTFGKiJiIisGAM1ERGRFWOgJqJaY2dnh+XLl1u6GEQ2jYGaqJ6S5UYlUJbeRowYYemiEVEVcK1vonpMgrKsEW7K2dnZYuUhoqpjjZqoHpOgLDnSTTcfHx/1nNSuJQGIZJ6SrFwtWrTA0qVLzV4v6RAHDx6snpfsSo899pjKhGbq888/R7t27dRnSdo/SdFpKiUlBXfccQfc3NwQGRmJn3/+2fhcamqqSq8oyQ3kM+T50hcWRA0dAzVRA/bKK6/grrvuwoEDB1TA/Nvf/oYjR46o5yRt6/Dhw1Vg3717N3744Qf88ccfZoFYAr2k5ZQALkFdgnBERITZZ0yfPh333nsv/vrrL9x8883qcy5dumT8/MOHD2PVqlXqc+X9/Pz86vgsEFk5yZ5FRPXPuHHjNHt7e83d3d1se/3119Xz8t//8ccfN3tNr169tAkTJqj7CxYs0Hx8fLSMjAzj87/++qum0+m0hIQE9TgkJER76aWXyi2DfMbLL79sfCzvJftWrVqlHo8aNUobP358DX9zovqFfdRE9ZjkADfktzbw9fU13u/Tp4/Zc/J4//796r7UcDt16gR3d3fj8/369UNRURGOHj2qms7Pnz+PIUOGVFgGyQ9tIO/l5eWlckiLCRMmqBr93r17MWzYMIwePRp9+/at5rcmql8YqInqMQmMpZuia4r0KVeGo6Oj2WMJ8BLshfSPx8bGYuXKlVizZo0K+tKU/u6779ZKmYlsEfuoiRqwHTt2XPW4TZs26r7cSt+19FUbbN26FTqdDlFRUfD09ER4eDjWrl1brTLIQLJx48bhm2++wezZs7FgwYJqvR9RfcMaNVE9lpubi4SEBLN9Dg4OxgFbMkCse/fu6N+/P7799lvs2rUL//3vf9VzMuhr2rRpKoi++uqrSE5OxlNPPYUHH3wQgYGB6hjZ//jjjyMgIEDVjtPT01Uwl+MqY+rUqejWrZsaNS5l/eWXX4wXCkSkx0BNVI+tXr1aTZkyJbXhmJgY44jsRYsW4YknnlDHff/992jbtq16TqZT/fbbb5g0aRJ69OihHkt/8vvvv298LwniOTk5mDVrFp577jl1AXD33XdXunxOTk6YMmUKzpw5o5rSBwwYoMpDRCXsZESZyWMiaiCkr3jZsmVqABcRWS/2URMREVkxBmoiIiIrxj5qogaKvV5EtoE1aiIiIivGQE1ERGTFGKiJiIisGAM1ERGRFWOgJiIismIM1ERERFaMgZqIiMiKMVATERFZMQZqIiIiWK//B2WD9xNrcWhBAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_losses))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_losses))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_losses, val_losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbd28174-1836-44ba-b6c0-7e0be774fadc",
   "metadata": {},
   "source": [
    "- Above, based on the downward slope, we see that the model learns well\n",
    "- Furthermore, the fact that the training and validation loss are very close indicates that the model does not tend to overfit the training data\n",
    "- Similarly, we can plot the accuracy below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "yz8BIsaF0TUo",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 307
    },
    "id": "yz8BIsaF0TUo",
    "outputId": "3a7ed967-1f2a-4c6d-f4a3-0cc8cc9d6c5f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeEAAAEiCAYAAADONmoUAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAU8JJREFUeJztnQdYFFcXhj/poliwY0XFrth77yVGjT0aiRr9NbYkmhiNLSZG0zTGGk3UNHtPbCEae+8du2BBsYJIZ//n3HWXBQFZXNhl93sf52Hv7OzMnSvsN+fcc8/JpNFoNCCEEEJIumOX/pckhBBCiEARJoQQQswERZgQQggxExRhQgghxExQhAkhhBAzQREmhBBCzARFmBBCCDETFGFCCCHETFCECSGEEDNBESaEJErjxo3xwQcfmLsbhFg1FGFC0oh3330XmTJlemlr3bq1ubtGCLEQHMzdAUKsGRHcxYsXx9vn7Oxstv4QQiwLWsKEpCEiuPnz54+35cyZU723c+dOODk5Yc+ePfrjv/nmG+TNmxf37t1T7a1bt6J+/frIkSMHcuXKhTfeeANXr17VH3/jxg1lXa9cuRINGjRA5syZUaNGDVy6dAlHjhxB9erVkTVrVrRp0wZBQUHxrPSOHTvi888/R548eZAtWzYMGjQIkZGRSd5LREQERo0ahYIFCyJLliyoVauWugcdN2/eRPv27dX9yfvly5fH5s2bkzzf3Llz4eXlBRcXF+TLlw9dunTRvxcbG4upU6fC09NT3ZO3tzdWr14d7/Nnz55V9yX3J59/55138ODBg3ju9OHDh+OTTz6Bu7u7GvtJkyal6P+NkPSCIkyImedcRTyePn2KEydOYPz48fj555+VqAihoaH46KOPcPToUWzfvh12dnbo1KmTEilDJk6ciHHjxuH48eNwcHDA22+/rcRn5syZSuSvXLmCCRMmxPuMnO/ChQtKSJctW4a1a9cqUU6KoUOH4sCBA1i+fDlOnz6Nrl27Kkv/8uXL6v0hQ4Yood69ezfOnDmDr7/+WglkYsj9iEBOnjwZfn5+6mGjYcOG+vdFgH/77TfMnz8f586dw4cffojevXtj165d6v0nT56gadOmqFKlijqXfF4eXLp16xbvOr/++qt6IDh06JB6wJHr+fr6Gv1/RUiaIaUMCSGmx8fHR2Nvb6/JkiVLvG3KlCn6YyIiIjSVK1fWdOvWTVOuXDnNgAEDkj1nUFCQlB7VnDlzRrWvX7+u2j///LP+mGXLlql927dv1++bOnWqpnTp0vH65u7urgkNDdXvmzdvniZr1qyamJgY1W7UqJFmxIgR6vXNmzfVvdy+fTtef5o1a6YZM2aMel2xYkXNpEmTUjQ2a9as0WTLlk0THBz80nvh4eEaV1dXzf79++Pt79+/v6Znz57q9RdffKFp2bJlvPcDAgLUffv5+en7X79+/XjH1KhRQzN69OgU9ZGQ9IBzwoSkIU2aNMG8efPi7RPXqA5xR//555+oVKkSihYtihkzZsQ7VqxMsWDFkhNXq84C9vf3R4UKFfTHyed16KzoihUrxtt3//79eOcWF6+rq6u+XadOHTx79gwBAQGqL4aIZRsTE4NSpUrF2y+Wr7jJBbFsBw8ejH/++QfNmzdH586d4/XLkBYtWqhrFC9eXFnTsomFL/0Rq/358+fqGEPEVS6Wr3Dq1Cn8999/iVra4q7X9TPh9QsUKPDSOBBiTijChKQh4gotWbJkssfs379f/Xz06JHa5DM6ZI5VxGrhwoXw8PBQIizim3Du1tHRUf9a5ogT25fQhW0MIs729vY4duyY+mmITgjfe+89tGrVCps2bVJCLC7l77//HsOGDXvpfG5ubsp1Lq5wOVYeNGS+Vuax5VqCnEfmnxMLapNjZGzE5Z0QEdrExsUU40CIqaEIE2JGxGqT+U4R2RUrVsDHxwf//vuvmvt9+PChmi+V9yToSti7d6/Jri3WZFhYmAp8Eg4ePKgEtXDhwi8dKxaoWMJiRer6khjyWQnwkm3MmDGq74mJsCBz12IxyyZz2hJ8tmPHDmUBi9iKtd+oUaNEP1u1alWsWbMGxYoVU+chJKPC315C0hBx1wYGBsbbJ6KRO3duJWoSbCTWY9++fZVLVlzIYj1+/PHHKspYXL0LFixQ1p2I0qeffmqyvok13b9/fxXQJVHWIoQSfCUPAAkR926vXr3Qp08f1T8RZYm2luAucfm2a9dOBZlJtLIc+/jxY+UuLlu2bKLX/vvvv3Ht2jUVjCX3KVHUYqGWLl1aWckShS0PJ7JPosMlcG3fvn0qilseVCQITAS+Z8+e+uhncWNL0JgEtiW01gmxVCjChKQhErVr6B4VRGguXryIKVOmqGU9IkiCHCeCK8LSsmVLNWcroiJzreKCls/9+OOPKqraFDRr1kwtERIhlIcFuW5yS3hkvfOXX36JkSNH4vbt2+pBonbt2mrZlCAPFSKOt27dUmIpDxUJ57h1iNUr0dhyvfDwcNUPidCWZU3CF198oZZOiUtbxFqOF+t37Nix6n1xzYsojx49Wo2V9F/c9nLNxB4iCLFUMkl0lrk7QQhJX2SdsCzzWb9+vbm7QohNw0dGQgghxExQhAkhhBAzQXc0IYQQYiZoCRNCCCFmgiJMCCGEmAmKMCGEEGImKMKpZM6cOSpbj5Rhk5Juhw8fhjUiFXEkPaCsy5SUfwmXtEhIgaQclDWuknlJsh/pqurokFSMkuhB1o7Kek9JEKFLTahDqvJIJiYZT8m6JBVvLB1ZwyplAyW5hJQflNKAkuHKEFkDK2tnJemGZKOSfMq6MoU6JAmHJLuQvMlyHknUER0dHe8YSe8o62Qlk5SkwVyyZAksGcmXLUk85P9cNslLvWXLFtj6uCTFtGnT1N+XJDzRYctjNGnSJDUehluZMmWsc2zSpUyElbF8+XKNk5OTZtGiRZpz586pyjc5cuTQ3Lt3T2NtbN68WfPZZ59p1q5dqyrUrFu3Lt7706ZN02TPnl2zfv16zalTpzRvvvmmxtPTUxMWFqY/pnXr1hpvb2/NwYMHNXv27NGULFlSXw1HePr0qSZfvnyaXr16ac6ePauqAGXOnFnz008/aSyZVq1aaRYvXqz6fPLkSU3btm01RYoU0Tx79kx/zKBBgzSFCxdWFY2OHj2qqV27tqZu3br696OjozUVKlTQNG/eXHPixAk13rlz59ZXJhKuXbumqgp99NFHmvPnz2tmzZqlKhpt3bpVY6ls3LhRs2nTJs2lS5dUVaOxY8dqHB0d1VjZ8rgkxuHDhzXFihXTVKpUSV+1ytbHaOLEiZry5ctr7t69q9+kgpg1jg1FOBXUrFlTM2TIEH1bSr95eHiocnHWTEIRjo2N1eTPn1/z7bff6vc9efJE4+zsrIRUkF9u+dyRI0f0x2zZskWTKVMmfVm8uXPnanLmzKnK+umQcnOGpfcyAvfv31f3umvXLv1YiPCsWrVKf8yFCxfUMQcOHFBt+XKws7PTBAYGxispKGX+dOPxySefqC8kQ7p3764eAjIS8n8sJRc5LnGEhIRovLy8NL6+vvFKR9r6GE2cOFE9uCeGtY0N3dGpyLcrlWTE7apD0uRJWwqe2xLXr19XeZENxyJ79uzKPa8bC/kpLujq1avrj5HjZcykPJ/uGEmdKGX9dEg+ZXHtSg7ijILkNzYsVSi/J1FRUfHGR1xqRYoUiTc+ki9aV35Qd+/BwcGqmL3uGMNz6I7JKL9vks5S0m+GhoYqtzTHJQ5xqYrLNOF9cIygprVkGkzKXcp0lriXrXFsKMJGIjVd5UvF8D9XkHbCRP3Wju5+kxsL+SnzMQkLGIhQGR6T2DkMr2HpSKEBmc+rV6+evs6v9F0eLOQhJLnxedW9J3WMfKFIFSRLRWoQy3ydzLdJVaV169ahXLlyNj8uOuTBRMo5SmxBQmx9jGrVqqXmZyX3usQXyAO/xIyEhIRY3diwgAMhJrJozp49a9JSgxkdKThx8uRJ5SFYvXq1qn60a9cuc3fLIggICMCIESPg6+urghFJfKQalw4J8BNRlgIdK1eu1JfetBZoCRuJVI6RMmkJI/GknT9/ftgSuvtNbizkp9SgNUQiFCVi2vCYxM5heA1LRsr/SSUkKd1XqFAh/X7pu0xfSKGE5MbnVfee1DESdWzJX0hirUjEabVq1ZS1J1WhZs6cafPjonOpyt+FROaKZ0g2eUCRKlnyWiwyWx8jQ8TqlRKZUq7S2n5/KMKp+GKRLxWpo2roipS2zHfZEp6enuoX2XAsxJUjc726sZCf8sciXzo6pHC7jJk83eqOkaVQMs+jQywEsaSk1qylIrFqIsDiZpV7kvEwRH5PHB0d442PzHPL3Jbh+Ijb1vBBRe5dvgjEdas7xvAcumMy2u+b/J9LyUGOi7aMpNyfeAp0m8RNyNyn7rWtj5EhsqTx6tWraimk1f3+pGsYmBUtUZII4CVLlqjo34EDB6olSoaReNaCRG9KiL9s8usyffp09frmzZv6JUpy7xs2bNCcPn1a06FDh0SXKFWpUkVz6NAhzd69e1U0qOESJYl2lCVK77zzjlrCIuMrSwcsfYnS4MGD1fKsnTt3xltK8fz583hLKWTZ0o4dO9RSijp16qgt4VKKli1bqmVOsjwiT548iS6l+Pjjj1UU6Jw5cyx+mcmnn36qosSvX7+ufi+kLRHx//zzj02PS3IYRkfb+hiNHDlS/V3J78++ffvUUiNZYiQrEKxtbCjCqUTWlMkvgawXliVLsgbWGvnvv/+U+CbcfHx89MuUxo8fr0RUHkyaNWum1oUa8vDhQyW6WbNmVUsE+vbtq8TdEFljXL9+fXWOggULKnG3dBIbF9lk7bAOeRh5//331fIc+YPv1KmTEmpDbty4oWnTpo1aGy1fNPIFFBUV9dL/Q+XKldXvW/HixeNdwxLp16+fpmjRoqq/8uUnvxc6AbblcTFGhG15jLp3764pUKCA6rN8H0j7ypUrVjk2rKJECCGEmAnOCRNCCCFmgiJMCCGEmAmKMCGEEGImKMKEEEKImaAIE0IIIWaCIkwIIYSYCYrwayDZf6T4tPwkL8PxSRqOTfJwfJKH42M9Y8N1wq+BpGiU0n2SoF7SoZH4cHyShmOTPByf5OH4WM/Y0BImhBBCzARFmBBCCDETNldPWMronThxQpUKs7N7vWcQKTAt3L59W7lASHw4PknDsUkejk/ycHwse2ykYpiURaxSpYoqTZkcNjcnfOTIEdSsWdPc3SCEEGLlHD58GDVq1Ej2GJuzhMUC1g2O1KYkhBBCTMndu3eVsafTm+SwORHWuaBFgAsVKmTu7hBCCLFSUjLlycAsQgghxEyYVYR3796N9u3bw8PDA5kyZcL69etf+ZmdO3eiatWqcHZ2RsmSJbFkyZJ06SshhBBiVSIcGhoKb29vzJkzJ0XHX79+He3atUOTJk1w8uRJfPDBB3jvvfewbdu2NO8rIYQQYmrMOifcpk0btaWU+fPnw9PTE99//71qly1bFnv37sWMGTPQqlUrk/YtJiYGUVFRJj0nIZaAk5PTay/PI4SYhgwVmHXgwAE0b9483j4RX7GITYWs2AoMDMSTJ09Mdk5CLAkRYHmYFTEmlkl4VAyO3niMqJhYc3fF5sjj5owKBbOn2/UylAiLOCYM+Za2LMgOCwtD5syZX/qMJPE2TOStW8id3DVEgPPmzQtXV1c1V02ItSBJBO7cuaOWUBQpUoS/3xbIjov3MHHjOQQ8CjN3V2ySNyoVwOy3q6bb9TKUCKeGqVOn4vPPP0+xC1onwLly5UrzvhFiDvLkyaOEWLLHOTo6mrs75AW3Hj/H53+dh+/5e6qdO6sTPHK8bFiQtKWIuyvSkwwlwvnz51epwAyRtlTKSMwKFsaMGYOPPvpI35ZUZuXKlUv0WN0csFjAhFgrOje0PHRShM1PRHQMft5zHbN2XEZ4VCwc7DKhf31PDG/mhSzOGeormqSCDPU/XKdOHWzevDnePl9fX7U/KWQpk2w6UpJLlC46Ys3w99ty2HflAcZvOItrQaGqXcvTHV90rIBS+dzM3TViCyL87NkzXLlyJd4SJFl65O7uruarxIoVy/W3335T7w8aNAizZ8/GJ598gn79+mHHjh1YuXIlNm3aZMa7IIQQ47gXHI4v/j6Pv0/fVe3cWZ0xrl1ZdKiszZlAbAezrlM4evSoqjIhmyBuY3k9YcIE1ZbgEX9/f/3xEtEpgivWr6wvlqVKP//8s8mXJxEtxYoVww8//JDi4yWRinyBMLKckMSJjonFz3uuodn3u5QA22UC3q1bDNtHNkLHKgUpwDaIWS3hxo0bqyVBSZFYNiz5jJQiJHG86g934sSJmDRpUqoqTmXJkiXFx9etW1c9OGXPnn7h/YRkFI7ceITx68/iYqB2hUaVIjnwRYcK6bochlgeGWpOmCSOCJ+OFStWKE+Cn5+ffl/WrFn1r+WhRwJyXlXjUhdFa2zAjwTP2SKRkZFcd0sS5cGzCEzdfBFrjt9S7Zyujvi0TRl0rVYYdmIKE5uGaXOsABE+3SZWqFjGuvbFixfh5uaGLVu2oFq1aipITbKMXb16FR06dFDrrEWkpeblv//+m6w7Ws4r7v9OnTqpCHIvLy9s3LgxSXe0eDJy5Mih0opKdjO5TuvWreM9NMgymeHDh6vjZFnY6NGj4ePjg44dOyZ5vw8fPkTPnj1RsGBB1Y+KFSti2bJlL62H/eabb1R+cblniTGYMmWK/v1bt26pc0j8gVj71atXx6FDh9R777777kvXl4Qw4oXRIa+HDh2q9ufOnVs/JTJ9+nTVHzln4cKF8f7776vYB0P27dunPi99z5kzp/rs48ePVeyDjIHhunZB+vLOO+8kOR7EMomJ1eD3gzfR9LudegHuWbMwdoxsjO41ilCAiYIi/ArEcnweGW2WLTlXvbF8+umnmDZtGi5cuIBKlSopYWjbti22b9+u3PsijlJMw3AOPjFkzXW3bt1w+vRp9flevXrh0aNHSR7//PlzfPfdd/j9999VwQ45/6hRo/Tvf/311/jzzz+xePFiJU4Svf6qQh7h4eHqgULiA86ePYuBAwcqkZIa0TokqE/ud/z48Th//jyWLl2qT/Qi996oUSMV9CcPEadOnVLBfiLcxvDrr78q61f6LSlVddmofvzxR5w7d069L8GDcm4dEnjYrFkztUxOMsDJA5GMu3gnunbtqn4aPtjcv39f3acEIpKMw6mAJ+g0d59yPweHR6O8Rzasfb8upr5VCTmz0GNC4qA7+hWERcWg3ATzFIg4P7kVXJ1M8180efJktGjRQt8WC1CC23R88cUXWLdunRIAsfCSQqxEsSCFr776SgmOiJ+IeFJrr0WgSpQoodpybumLjlmzZinBFOtakOj3hMvQEiIWsKGQDxs2TFnbEikvhbQlK9rMmTPVucSqFuT69evXV69FkIOCgtSct4yDIBazsYgnQKxtQwxTqIon4csvv1RR/XPnzlX75HixunVtoXz58vrXb7/9tnogEUEW/vjjD2XFG1rhxHJ58jwS327zw9LD/pBnaDcXB4xqWRq9axeFPS1fkggUYRtBvvgNEWtQgrXEyhL3sLiFJfXnqyxhsaJ1iMtVEqWItZYU4nLVCbBQoEAB/fFPnz5VyVZEOHXY29srKzc5q1SsRXkAENEVa1bmY8WFq0uyIta+tMXiTAyxRiUKXyfAqUX6mRBx6UuWNpkGEKtexlUsd/EISP/k2jqBTYwBAwaoqQG5L3nYEJe+PPgwatayiY3VYPXxW5i25SIehUaqfW9VKYgxbcuqXMSEJAVF+BVkdrRXFqm5rm0qEkY5iyUpS73EVSxWoGQc69KlixK05EiYYUnEITnBTOz413Wzf/vtt8rSlflq3fyrWKC6vieVPU3Hq94Xl3LCPiZWUSvhmN64cQNvvPEGBg8erOafReTF3dy/f3/VNxHhV11bHg7EQyHzwy1btlRuba6Dt2zO3wlWCTeO3Xys2qXyZVVRz7WKM/UteTUU4VcgomEql7AlIfOYYmHp3MBiGYuIpCcSRCbztOIWbtiwod7KPX78OCpXrpxs3yWorHfv3qotDwGXLl3SpyMVN7GIncx3S73pxKx5CTCTuezErGGJCpe5ZkPEgn1Visdjx46pvsj6dV2pQLHWE15b+pVcPnPpszxgiDUsVcMkwItYHiHhUZjhexm/HrihgrBcnezxQXMv9K3nCUf71wy3kQfbx9eBmETKqWYvCDi/yKgV9gQICQScXIEcReKOCboEaIyswOSWD8icU/s64hnw9Bbg4Ay4e8Yd8/Bq4n1Kjix5gCwvHkiiwoDHNwE7ByC3wRTQ4xtAVLhx55W+Sp8F6ZP0TTxGeUrHHfMkAIjUZiNLES7ZgWwFkJ5Yn7qQFCFCtXbtWhUUJA8aEsBkbGCSKZD5XHHfijVepkwZNUcskcLJuV+l76tXr8b+/ftVdLFEJItbWyfCLi4uKspaAqIkcKpevXpqDlisSrFKZU5b3NkSdSzXFhe5BKd5eHioFKhNmzZV1rZYo9KWeVkRZV1SmaSQexCLWe5BxtUwYEuHzH+L9S5R0zJXLP3777//lItaoqx188LiqVi4cKE+WxyxHMRLsvHUHUzZdAH3Q7SR7O0qFsC4N8qiQPbXLLggQnRmJbB/NvAgbplhPHouB0q/qMN+aSuw7n9AiWbAO2vjjlnYBIiMH5X/St6cBVTto33tfxD4szNQwBv43+64Y/54SyuYxtBsItDgRf7+oIvAgsZAtoLAR+fjjlndH7h91Ljz1hkKtHqx4uHZPWBuLcDeGRhvMD22eZR2jFJKld5AhzlITyjCNooIl0TcSoIN+fIX0UpJXm1TI9eV8pF9+vRR88ES6SxLduR1UowbNw7Xrl1Tx4mLVz4jgipzzDrkoULWQsuaaakYJEIroieI8P3zzz8YOXKkivCWeVsR8DlztH98cl75vIi4zOfKOEn/zpw5k+y9iBtZxlUivkVsxboXkZfP6ihVqpS69tixY9VcuFjstWrV0ge76TwEnTt3Vm7o5JZqkfTnyv0QTNhwDvuvPlRtz9xZ8Pmb5dGwlHFr6l/i+SPg6C/AoQVA6AsREUFxjlvjr8fewCNj7wS45gJcssU/JrO71oo1BgcXg/M6vDhv9petz4jky8G+hKPBg4ndi/PqLG4dch3ZbwyOBoV2MtlpPy9jZoh4DIw5r1Mi453GZNKYch1MBkDWh4p7LyAgAIUKFYr3nnzhSv5qSY8p1hRJf8QalzXFsgxKIrZtFQkqk6hpiT43Nfw9Nx5ZMjhrxxWVcjIqRgNnBzsMbVISAxsVh7PDa8RuiFV5YC5w4ncg6rl2X7ZCQO3BWqs0obiSDK8zCaElTMzKzZs3lWUo63YlolmWFYlAiEvWFhFXvCQ9kc1wGRMxD2KjbDt3TxVbuP0kTO1rXjYvJrYvj8KmqDsrbucjC7Wv81UE6g0HyneKb+0Sq4YiTMyKBDDJMhyZA5UvvAoVKqhlPmIN2yIy7yxCLC7t0qUNAkxIunPzYSgmbjyHnX5Bql0wR2ZMerM8WpR7EQxkLBJzccVXOx+av4J2X533tQFYMr9ZvLE2sIjYFBRhYlbEZSMBTERLekeok5cJj4rB/F1XMXfnVURGx8LRPhP+17AEhjQpicxOr+F63vEFsHc6UPZNoPvv2n3uxYHea0zWd5LxoAgTQsgL/vO7j0kbz+HmQ+38bP2SufF5h/IokSdr6oKtoiPilrxU6gYc+UUrvBKKQ6uXUIQJIQS48yQMk/86j63nAlU7XzZnjH+jnFp6ZHS2Mgm2OjgPOP47ULY98NZP2v15ywKj/OJHCxObhyJMCLFZxN38y97r+HH7ZZUnXvI796tXDCOal0JWZyO/Hm8fB/bPAs6vj0uUIWt9Y6K1S34ECjBJAEWYEGKT7L/6QK35vXJfm9SiZjF3TO5YHmXyZzM+2ErE98aeuP0lmgJ1hzPYirwSijAhxKa4HxyOKZsvYMPJO6qdO6sTxrQpi7eqFky561nmek+vBA7M1maB0iWiqNAFqDssLvqZkFdAESaE2ATRMbH47cBNzPC9hJCIaGWgvlO7KEa2LI3smVO4LjfsMXB0EXDoJ22qRME5G1DtXaDWIG1eZ0KM4DWzjBNrQmrWJqyHK4UEkkMsh/Xr17/2tU11HkISQyoctZ+9D5P/Pq8E2LtwDmwcUh+TO1RIuQALawcC2ydrBVjW+7b8EvjwLNDyCwowSRW0hK0AKRYghQO2bn05UfmePXtUDuNTp07FqwWcEqS6UcJyfa+L1DAWsZWqRIZITWMpxkCIKXn4LAJfb72IlUdvqbYI7ujWZdCjRmHY2aXA9XznBJC9MJBFW1wDNQYAwXe1LucKbzGzFXltKMJWgFQGkoT/kq80YZ7SxYsXo3r16kYLsK6kX3qRP39+2CJSZ1gKShDTEhurwbIj/vhmqx+ehmlL73WvXhij25SBe5YUjvfmT4DDPwGNRgNNxmr3ebXQbgy2IiaC7mgrQArJi2BK+kdDpEbwqlWrlEg/fPhQVeopWLCgqjwk5fSWLVuW7HkTuqMvX76srGpJ+i9Vh3x9fROtiiSVguQaxYsXV9WIxEoXpH9SR1escnE/y6brc0J3tFQskpKCUmUoV65cqlKS3I8OqYUsFYa+++47VSFJjhkyZIj+Wolx9epVVYdYahhnzZoVNWrUUCkyDZH81XIPksnL2dlZlSf85Zdf9O9LOUQZ72zZssHNzQ0NGjRQ503MnS9IH6WvhmMqhSmkspKcQ+7rVeOm46+//lJ9lvGXyle6WtCTJ09W6T4TIjWZ5Ty2xplbT9Fp3n58tu6sEuCyBbJhzeA6+LpLpeQFWIKtIl8UURCK1tEGW4XHVedS4ksBJiaElnBKMaYwtA4pq6VbHyhrBWMitCW3DNcKJnVep5S7gaVkn3ypi6B99tln+ghPEeCYmBglviJg1apVU1/28uUvZfLeeecdlChRQpXUS0l1o7feeksJ2KFDh1TZwISCI4gwST+kNq8I6YABA9Q+KQvYvXt3VZdX3OY68ZOyfQkJDQ1V5QSllq+4xO/fv68K3Q8dOjTeg4bU4RUBlp9XrlxR5xfhkWsmhoyBlC6cMmWKElip1SuufD8/PxQpoi2ILuN44MABVb1IShNKMYkHDx6o927fvq0eQkRsd+zYocZRUm5KKURjkAcHKbE4ceLEFI2bIP9fIrry/yv9Fgt68+bN6j0ptSgPNzJWItKC1Ec+ffq0qhltKzx9HoXv/vHDH4duqoRUss53ZMtSKvjKwd4uZcFWUr2o/ofa/ZJecsRpzvWStEVjYwQEBEjpRvUzIWFhYZrz58+rny8xMZvx29m1cZ+X17JvUdv45/3aM/HPGsmFCxfUff3333/6fQ0aNND07t07yc+0a9dOM3LkSH27UaNGmhEjRujbRYsW1cyYMUO93rZtm8bBwUFz+/Zt/ftbtmxR11y3bl2S1/j222811apV07cnTpyo8fb2fuk4w/MsWLBAkzNnTs2zZ8/072/atEljZ2enCQwMVG0fHx/Vv+joaP0xXbt21XTv3l1jDOXLl9fMmjVLvfbz81P98PX1TfTYMWPGaDw9PTWRkZGJvp9w/IQOHTqovuqQPnfs2PGV/Uo4bnXq1NH06tUryePbtGmjGTx4sL49bNgwTePGjRM9Ntnf8wxIbGysZvXRAE3Vyf9oio7+W23Dlx3X3Hv6ivt7dEOj2Txao/myQNzf3U+N5YTp1XVigzqTEFrCVkKZMmVQt25dLFq0SFlqYhlKUJa4KgWxiL/66iusXLlSWXRiSYnrVdyfKeHChQvKRSuWmg6xVBOyYsUKZUWKi1YsT7ESxWI0BrmWWKGGQWH16tVT1rhYrWKNC1Jv194+LqG+WMViRSaF9EcCw8SqlEAw6VtYWBj8/f3V+xIsJueTsoqJIe+L+9nR8fWCcWSO3thxk2snZeEL8p5YxNOnT1eVqZYuXYoZM2bA2rkYGIwJ68/h8I1Hql0yb1ZM7lAedUu8CKRKKthKkmuck8xWMdp9ecu/KCP4Ft3NJF2hCKeUsdqF/Ua7o3WUaa89h7ijDfkgadEwFpn7HTZsGObMmaMCssTVrBOUb7/9FjNnzlRzvDIfLAIn7mQRY1MhbtxevXop16i4k8XVvHz5cnz//fdICxKKobjhRaiTQsolyjy2uINlrlfmm7t06aIfA2knx6veF/HTGvVxJDZHnTDiPCXj9qpri1tdXOzr1q1TgV5yXbk3a+VZRDR+8L2ExftvICZWg8yO9hjR3Av96nnCycEuicxW/wL7f4yf2UoyWklmK8lwRfElZoAinFKMmKNNFJkb1s0Pm/K8BnTr1g0jRoxQVpDMGw4ePFg/PyxzlxKU1Lt3b9UWsbp06ZIKsEoJUt83ICBAWZBicQoHDx6Md8z+/ftRtGhRNW+p4+bNm/GOEYEQq/xV15L5UZkb1gmW9F9E7nVq7Mo5JEhKF9AkFqdh6UB5OJFx2bVrF5o3b/7S5yXC/Ndff1UCl5g1LMFxMj465D5lDrxJkybJ9isl4ybX3r59O/r27ZtkXICPj496+JIx7tGjxyuFOyMiDzmbztzFF3+fx73gCLWvdfn8GN++nKr3m2iw1ZlVWstXl9kqkz1QobN2mVEB41cNEGJKGB1tRUjErwQnjRkzRomBYVSul5eXsgLlC1/cvf/73/9w796LjD8pQERJonfli16im8XVbSgaumuIa1esOHGrintVLDNDJDpYgp3EvSoBT+IST4hYhRIBLNcSEZPAK7HwJZBM54pODdI/CVSSa8s9vP322/EsZ+mbXFPcuhKpLf3cuXOncuELEhgWHBysBO7o0aMqWvz3339XLnJBornF1S3bxYsX1UPQkydPUtSvV42bBHFJNLv8lP8/cbt//fXX8Y6R4DUJGJPAN7kHa+Nq0DO888thDF16Qglw0VyuWNK3Bua/Uy1xARYWtQY2DNEKsFNWoM5QYMQpoPNCCjCxCCjCVoa4pB8/fqzcmobzt+PGjUPVqlXVfpkzlnW5snwmpYgVKsIgc6gSTS1f+BJlbMibb76JDz/8UImVRCmL4CdcIiPrmVu3bq2sQ7EcE1smJfPU27Ztw6NHj1S0r7hVmzVrhtmzZ+N1kPlSSQgic+fivpWxkDExZN68eep677//vppnl7lWscgFWQYlIicWtLj5Jdp84cKFeqtYhE9EXCKs5X1ZavQqKzil4yb/ZxLtvnHjRnWMCP7hw4dfEnO5N+l3rVq1YC2ERcbgu21+aP3Dbuy98kC5mz9o7oVtHzRE49J54x/8xF+7EkFH+Y6AWwGgxWTgw3NAqylAjsLpfg+EJEUmic6CDSEJLSTASFyrCRNbhIeHK+vH09NTWWKEZCTkT1mEWB4gPvrooySPy0i/577n72HSxnO4/SRMtZuUzoNJb5ZH0VyJTONs/hg48ovWyhV3sxAVpnU/OzAhCrEMnUkI54QJsQKCgoKUOzswMDDJeeOMRMCj50p8t1+8r9ribp7QvhxalssXV+lIZz/o2q65tNHOAYfjRJj1e4mFQxEmxArImzevyqK1YMGCDJ2DOyI6Bj/tuoY5/11BRHQsHO0z4b0GxTGsaUm4Or34uoqO1AZbSRnBZhOB0q21+2sOBEq3AQp4m/UeCDEGijAhVoA1zCrtvhSEiRvP4foD7Rx83RK5VJUjWfurCHsCHFuszWwV8iIK/cjCOBF2ddduhGQgKMKEELNy92mYWnK0+Uygaud1c8a4N8qhfaUCWtezBFsdnA8c/xWIfJE/XIKtpH6v1PElJANDESaEmIWomFgs3ncdP/x7Gc8jY2Bvlwk+dYrhwxZecHNxBO6e0q7vPbs2fmYrVUawM4OtiFVAEU6E5LIuEZLRsQTX9cFrDzFhw1lcuqe1bKsXzalcz+UKuAFXtmszW13flSCz1TCgRDNmtiJWBUXYAMk0JOth79y5o9awSlsfiUmIlQiwRFLL7/Xr5sBODfdDwjF180WsO3FbtaW04Jg2ZdC5aiHYibW7oDFw92SCzFZDGWxFrBaKsAEiwLJ2UrJNiRATYo2IAMvaRcPiF2mN5Hf+4+BNlXQjJCJaGbNv1yyCj5sUQo4cumhuByBvOeDhFe1cr8z5MrEGsXIowgkQ61dqy0oVm1flOCYkIyIWcHoK8HH/xxi//izO3QlW7YoFs+PLDuXhffF7YO4SoN9WIH8F7cHNJwKtpwKZc6Rb/wgxJxThRNC56szhriPEWngcGolvtl3EssMBqp3NxQEfty6jLGAJwsJBfyAyBDi7Ok6E3fKbt9OEpDMUYUKISYmN1WDl0QB8vfUiHj+XUo4ajC19F+/iLzh5zQBEgIVGnwJV+gAlm5m7y4SYDYowIcRknL39FOM3nMUJ/ydwRDSGuh/H+05b4HpTW2kKB+YAb0zXvs5XTrsRYsNQhAkhr01weBSm/3MJvx24gayaUAxz+g+DMvsiy/Mg4LkEW2QFqvoAtQebu6uEWBQUYULIay15Wn/yNqZsuginZ7cxxmErejvtRObY50BEgsxWDLYi5CUowoSQVHHpXoiKen524zg+c9iEN10OwB6xkH9qqZHKbNWFma0ISQY7mJk5c+agWLFiqq6pFCJPWKjckKioKEyePBklSpRQx3t7e2Pr1q3p2l9CbJ3QiGhM3XwB3WZuw7BbI7HJeSw62e/TCrBnI6DXGmDwfqDy2xRgQizZEl6xYoUqPj5//nwlwD/88ANatWoFPz8/VZotIePGjcMff/yBhQsXokyZMti2bRs6deqE/fv3o0qVKma5B0JsyfW85cxdfLHpAu4+DQfggkJu0dBE2iNThbeAOkMBj8rm7iYhGYpMGjMmkhXhrVGjBmbPnq3P2Vy4cGEMGzYMn3766UvHe3h44LPPPsOQIUP0+zp37ozMmTMrcU4Jt27dUtcICAhQWYMIIa/m+r3HOLj0S1R/vAWdIychu3tufP5meTTNdldbPjBHEXN3kRCLwRidMdoSFtdxv3798O6776rMUqklMjISx44dw5gxY+KljWzevDkOHDiQ6GciIiKUG9oQEeC9e/cmeR35jGw6QkJCUt1nQmyK6EjceRaDRXuvq6jnv+y3wsvuNmaWPY86b4+Hi6Nk3cpn7l4SYltzwh988AHWrl2L4sWLo0WLFli+fHk8kUspDx48UGkh8+WL/0cs7cBAbV3RhIirevr06bh8+bKymn19fVVfJNdzUkydOhXZs2fXb+XKcV0iIUkSfBc4uhghi95C+FdF8cY3f+HnvdcRGaPBprwDEdRsBpr0GvNCgAkhZhHhkydPqgCqsmXLKtdxgQIFMHToUBw/fhxpycyZM+Hl5aXmgyXHs1yzb9++yoJOCrG0nz59qt/Onz+fpn0kJEMhs1GBZ4Bd30CzoAkwvQzw9wdw898Ol9jnqIlzqFM8Fxb3rYEPhwxHngb9AAdnc/eaEKsh1YFZVatWVdv333+PuXPnYvTo0Zg3bx4qVqyI4cOHK3FMrgxg7ty5VRL5e/fuxdsv7fz5E88fK+UF169fj/DwcDx8+FDNEcvcsVjlSeHs7Kw2HcHB2iTyhNgs0RHAjb2A3xbtFnxL7db9tZ6MLYHtsdUQUbI1hjRrhoqFub6XEIsTYVkutG7dOixevFi5hWvXro3+/furCemxY8fi33//xdKlS5P8vFiy1apVw/bt29GxY0e1T1zM0hYLNzlkXrhgwYKqD2vWrEG3bt1SexuE2A7n1mm3K9uByGf63eFwwp6Yivg3tioO2FdD8xre6FuvGAq7u5q1u4TYAkaLsLicRXiXLVum3MB9+vTBjBkzlItYhywbkqjnVyHLk3x8fFC9enXUrFlTLVEKDQ1VVrQg5xaxlXld4dChQ7h9+zYqV66sfk6aNEkJ9yeffGLsbRBi/Ty6BrgbeInOrAYu/q1ehjjmxtZIb2yJqoL9seXh5pZNCe/YmkWR3ZXVwwixWBEWcZWALHE9iwWbWLk/T09P9OjR45Xn6t69O4KCgjBhwgQVjCXiKsk3dMFa/v7+8eZ7xQ0ta4WvXbuGrFmzom3btvj999+RIwfdZYToiYkG5tcHgi4AQ48BuUuq3f5FO+NikDvmBZbCyfBi0MAOXnmzYnLD4uhQ2QPODgy2IsTi1wnfvHkTRYsWRUaF64SJVREeDFz5F7h/AWj6Wdz+X98Ebu6H5q2F2ONUHwv3XMOeyw/0b0uw1cCGxdGoVB7Y6UoLEkIsf53w/fv3ldUqiTYMEVexBFqJa5kQkoY88Qf8tgJ+m7UBVrFRL9xU/QE3bVBjZJsZ2Ho9CnP/vY+LgdpUsPZ2mdC2YgEMaOCJSoXoPSLEEjBahCVblczBJhRhmaP9+uuvlRgTQkxIbCxw5wRw6UU0872z8d/PVRIo3UYtN5KSgssP+2PR3hsIDJbUkoCrkz261yiMfvU8GWxFSEYXYVlnK0uTEiK5m7kGlxATEfkcuL5LK7qXtgLPDJbyZbIDitQBSrXWim9uL9x5EoYle29g6aHTeBYRrQ7L4+aMd+sWQ+9aDLYixGpEWNbcylrehGtzJWuVgwMrIxLy2kiYxuwa+vW7Cic3oGQzoHRbwKuFNl+zPBTfCcbCFSfx16k7iI7VhndIsNUABlsRkiEwWjVbtmypslBt2LBBpYEUnjx5otYGS9Q0IcTIwKrDPwG3jgE9lwGS4Ea2YvWBm/u0lq5sRevrywJKLOXey0FYsDt+sFXt4u74X8MSDLYixJpF+LvvvkPDhg1VhLSufKCksZRlRbJciBCSDNGRWgtXt37X3gnYMx2Ieg4EngYKeGv3t/secMqiFeQXRMXEKotXxPdioLYQiWhtu0oeDLYixFZEWJJnnD59Gn/++SdOnTqlqhhJco2ePXsmumaYEJvn+SPgsq82sEqyVWXzAIa8CGB0dAEajgJccwHZC8d9xjmr/mVIeBSWHfbH4n03XtTxZbAVIdZCqiZxs2TJgoEDB5q+N4RYCw+uxEUz+x8ENDFx7z130Qrzi3ldNBiZ6CnuPg1TwrvskD9CEgRb9apVBDlcte5pQkjGJdWRVBIJLRmtpC6wIW+++aYp+kVIxstSdetwXFGEh5fjv5+3/Iv53baARxUpnp3kqSTY6uc917DRINiqZN6sGNigODpUYbAVITYtwpIyUnJDnzlzRlVJ0iXc0lVMkhrBhNgMESHAplHA5X+AsEdx++0ctcFVIryylChn8lnmVLDVlQcvBVvV8nTH/xoVR+NSeRlsRYgVYrQIjxgxQuWGlmpH8lPqCktZwZEjR6qgLUKsmicBwMMrQIkm2rZTVuDGHq0Au+QASrXSCm+JZoBLtleeToKt/j4twVbXceGutsymaK02s1VxeLOMICFWjdEifODAAezYsUPVA5biCrLVr19fVTqSOsInTpxIm54SYm5kGdHPTYHM7sDHVwA7e230cutp2sCqwrUA+5T9SUmw1fLDAVi077o+2CqzozbYqn99BlsRYisYLcLibnZzc1OvRYjv3LmD0qVLqyVLfn5+adFHQtKXqDDg2i5tYJVbAaDxp9r9snzINbfKUIXQIH2eZpRLeRyEBFst2SeZreKCrXJndVZlBBlsRYjtYbQIV6hQQS1NEle05I/+5ptv4OTkhAULFryURYuQDMOz+9r0kBJUdfU/IDpMu1+WDTUarbV4xcr94AzgZLyVKq5mqWS08WRcsFWJPFlUJaMOlQvCxZHBVoTYIkaLsNTzDQ0NVa8nT56MN954Aw0aNECuXLmwYsWKtOgjIaZHAgrvn4+LZr59THbGvZ+tUFy2KjlWlzTDCAGWYKt9Vx5iwZ5r2H0pKF6wlYhvk9IMtiLE1jFahFu1aqV/XbJkSVy8eBGPHj1Czpw59RHShFhstipJBaks3s3akoCGyNIhWUIkwpuvQrxsVcYgwVabTt9Vkc7nDYKt2lQsoJYZMdiKEJIqEY6KilIZsiRNpbildbi7v0g6QIgllgHUrcl9GgD83jHuPQcXwLNR3DKibAVe61ISbLXiSAAW7b2OOwy2IoSYWoQlLWWRIkW4FphYPgGHge2TgSy5ga5LtPtyldAWQnAvprV4izfW5md+TQKfhmPxvusMtiKEpL07+rPPPlMVk6RYAy1gYhHExgC3jmjX7OZ/4aGxd9Su33XMonVDv6hAhL6bTHbZi4HByuXMYCtCSLqJ8OzZs3HlyhV4eHioZUmSR9qQ48ePp7ozhBiVqerqDsBvK3B5G/D8IeD9NtBpnvb9ApWBdtO1NXh1AmwCGGxFCDGrCHfsaDCnRkh6c34DcPw34PpuIMYgb7lLdsBZu35dIUFVNfqb7LLJBVtJZqvKDLYihKSHCE+cODE11yHk9RNobP4YOGFQszqnZ1w0c5HaWhe0iUku2ErKCBbJxWArQogZqigRkq5lAVf5APfOiokL1B0GVOkN5C6V6mVEKQq22v8i2Co8Ltjq3bpF0atWUeTMwmArQogZRFhyRSe3HpiR08SknF0LbBwORIYAWfIAnX/WRjWnERJstXD3dWw8dRtRMXHBVuJy7liFwVaEEDOL8Lp1615aOyxFG3799Vd8/vnnpuwbsXUOzge2jta+LloP6PzLa6/lTSrYav/Vh2q+d5dBsFVNCbZqUBxNyzDYihBiISLcoUOHl/Z16dIF5cuXV2kr+/c3XTAMsXHKvgHs/gao2gdoMi7FFYqMCbbafEYbbHXujkGwVYUCeK+BJ6oUyWnS6xFCSEJM9q1Wu3ZtDBw40FSnI7ZKkB+Qp7T2dfZCwNCjgKtp16M/i4jG8sP+WLzvBm4/CdMHW3WrXgj96nuiaK7XT+BBCCHpJsJhYWH48ccfUbBgQVOcjtgiUiTBdwKwfxbQYylQpq12vwkF+F5wuKrfGz/Yygk+dYqhd20GWxFCMoAIJyzUIPNpISEhcHV1xR9//GHq/hFbQX6nYkUYNcCdE3EibAL8AkNUGcENJ+OCrYq/CLbqxGArQkhGEuEZM2bEE2GJls6TJ4+qLSwCTYhRxETHzfU2/xzwagGUaPrap5WHwwNXH+KnhMFWxbSZrRhsRQjJkCL87rvvpk1PiO3le945Fbh5AOizQSvEkl7yNQVYF2wllu/Z23HBVq0r5FeWL4OtCCEZWoQXL16MrFmzomvXrvH2r1q1Cs+fP4ePj48p+0eskZB7wJr+2gILwqUtQNn2Jg+2cnG0Q/fqhRlsRQixHhGeOnUqfvrpp5f2582bV0VHU4RJskjO59X9gdD72gpH7We+lgBLsJUI75+HbjLYihBi/SLs7+8PT0/Pl/ZLRSV5j5BEiY0F9n4P/PcVoIkF8pQFuv0G5CmVqtMx2IoQYpMiLBbv6dOnUaxYsXj7T506hVy5cpmyb8RaCH0IrB0AXN2ubVfuBbT9DnAyvvjBsZuPMWvHZez0ix9sNaBhcTRjsBUhxNpFuGfPnhg+fDjc3NzQsGFDtW/Xrl0YMWIEevTokRZ9JBkZ/0PA6r5A8G3AITPQ7jtt8QUjiY3VYO7OK5juewmxGgZbEUJsVIS/+OIL3LhxA82aNYODg/bjsbGx6NOnD7766qu06CPJqMk3DswG/p2kXf+bywvo9iuQr7zRp3oUGokPVpzE7hdLjTpW9sCHLUox2IoQYnsi7OTkpHJEf/nllzh58iQyZ86MihUrqjlhQhRhT4D17wN+m7TtCp21AVjObkaf6uiNRxi69AQCg8NVtPPkDhXQrXph0/eZEEIyUtpKLy8vtRHyEnb2wAM/wN4JaD0NqN7P6Lq/kmzj5z3XMW3rRcTEalTQ1dxeVVEmf7Y06zYhhFi8CHfu3Bk1a9bE6NEvSsy94JtvvsGRI0fUemFio+5nQcRWLN5uvwMxkYBHZaNP9fR5FEatPgXf8/dU+01vD3z1VkVkdTZtFSVCCDE3dsZ+YPfu3Wjb9uW8vm3atFHvERskPFgbfHVwXty+fOVSJcCnbz1Bu1l7lAA72dvhy44VMLNHZQowIcQqMfqb7dmzZ2peOCGOjo4IDtamCSQ2xoW/gHPrAL+tQKVuQJbcRp9C3M+/H7yJL/++gMiYWBRxd1Xu5woFs6dJlwkhJENawhKEJYFZCVm+fDnKlStnqn6RjETlt4FagwGfjakS4JDwKAxddgITNpxTAtyqfD78Naw+BZgQYvUYbQmPHz8eb731Fq5evYqmTbXJ9rdv346lS5di9erVadFHYmlEhgI7pwENRwEu2bXzwG2mpepU5+8EY8jS47j+IBQOdpkwpm1Z9KtXLF6lLkIIsVaMFuH27dtj/fr1ak2wiK4sUfL29saOHTvg7m66AuzEQgnyA1b6AEEXgCf+2rW/qUDczyuPBijrNyI6Fh7ZXTC7V1VUZeINQogNYbQ7WmjXrh327duH0NBQXLt2Dd26dcOoUaOUGBvLnDlzVApMFxcXVZP48OHDyR7/ww8/oHTp0kr8CxcujA8//BDh4eGpuQ1iLKdXAguaaAU4az6g5oBUneZ5ZDRGrjqF0WvOKAFuUjoPNg1vQAEmhNgcqQ45lUjoX375BWvWrIGHh4dyUYugGoPMLX/00UeYP3++EmAR2FatWsHPz0/lqE6IuLw//fRTLFq0CHXr1sWlS5dUfWNxXU6fPj21t0JeRVQ4sHU0cGyJtu3ZCOj8M5D15f+jV3HlfggG/3Ecl+8/U6knR7UqjUENSzDnMyHEJjFKhAMDA7FkyRIlvhIJLRZwRESEck+nJihLhHPAgAHo27evaosYb9q0SYmsiG1C9u/fj3r16uHtt99WbbGgJZf1oUOHjL42SSEPrwKrfIDAM7IIGGj0CdBotDYhh5GsP3EbY9edwfPIGOR1c8aPPaugdnEW/SCE2C52xswFixtYKiiJxXrnzh3MmjUr1ReOjIzEsWPH0Lx587jO2Nmp9oEDBxL9jFi/8hmdy1pc4Zs3b0503TIxAec3AAsaawXYNRfQew3QZKzRAhweFYMxa8+o/M8iwPVK5lLuZwowIcTWSbElvGXLFlU9afDgwSZJV/ngwQPExMQgX7588fZL++LFi4l+Rixg+Vz9+vVVYE90dDQGDRqEsWPHJnkdsdRl0xESEvLafbd6oiMB3wnAoRfJN4rUAbosArJ5GH2qGw9C8f6fx3H+brAKoh7e1AvDm3nBnu5nQghJuSW8d+9eJWDVqlVT87ezZ89Wgpie7Ny5U0Vlz507F8ePH8fatWuV+1oqOyXF1KlTkT17dv3GtcyvQCKeF7eOE+B6IwCfv1IlwFvO3MUbs/YqAc6VxQm/9aupqh9RgAkhREsmjZiURiAR0RJQJfO24hYWa1bmdvv166dqDBvjjnZ1dVXLnDp27Kjf7+PjgydPnmDDhg0vfaZBgwaoXbs2vv32W/2+P/74AwMHDlSZvMSd/SpL+Pbt20qIAwICUKhQIWNu3TZY8Q5wYSPgkgPoNB8o3cboU0RGx2LqlgtYvO+GatcolhOzelZF/uwuadBhQgixLG7duqVW76REZ4xeopQlSxYluGIZnzlzBiNHjsS0adNUNPObb76Z4vNI6kuxqiXRhw6pSyztOnXqJPqZ58+fvyS09vba+cmkniWcnZ2RLVs2/WbMg4JN0vY7oHRb4H+7UyXAtx4/R9efDugFeFCjElg2oDYFmBBCTLVOWIcEakn1JFH9ZcuWGf15WZ60cOFC/Prrr7hw4YKabxZLWxct3adPH4wZMyZecNi8efNUiszr16/D19dXZfCS/ToxJkYSfBc49FNc2y0f0HMZkNP4+tDbL9xDux/34lTAE2TP7IhffKrj0zZl4GD/Wr9mhBBitZikNI0IoLiUDd3KKaF79+4ICgrChAkT1PKnypUrY+vWrfpgLX9//3iW77hx49SaYPkpbuU8efIoAZ4yZYopbsP2CH8K/NQQCL2vjX6u2CVVp4mOicW3//jhp13XVNu7cA7MebsKCuV0NXGHCSHExueEbclXbxNs/wK4tE2bfjJXCaM/Hvg0HMOXncDhG49Uu2+9YhjTpiycHGj9EkJsk1tG6AyLtNoaz4KA6HAgR2Ftu/EYbSEGx8xGn2rP5SB8sPwkHoZGqnq/33SphLYVC5i+z4QQYqVQhG2JG/uA1f0At/xA/38AB2fA3kG7GUFMrAYzt1/GrB2XIX6UcgWyqdq/xXJnSbOuE0KINUIRtgViY4H9M7WuZ02MtvxgaBCQ3Xh3fFBIBD5YcQL7rjxU7Z41i2Bi+3JwcWRgHCGEGAtF2Np5/ghY9z/g8j/adqUewBvTASfjrdZD1x5i2LITuB8SAVcne3zVqSI6Vilo+j4TQoiNQBG2ZgKOAKveBYJvAQ4uQJtvgKp9oPJHGkFsrAbzd1/Fd9v8EKsBvPJmxbzeVVEyL9dcE0LI60ARtkZkovbgPMB3PBAbDbgXB7r9BuSvaPSpHodG4qOVJ/GfX5Bqv1WlIL7sVAGuTvzVIYSQ14XfpNZG2BNgwxDg4t/adrmOwJuzAJdsRp/quP9jDP3zOO48DYezgx0mdyiPbtULq7XahBBCXh+KsDVx56S29u/jG4CdI9DqK6DmAKPdz7J0fNG+G5i6+QKiYzXwzJ0Fc96uinIexgs5IYSQpKEIWwvX9wB/dAZiIoDsRYBuS4CC1Yw+zdOwKHyy+hS2nbun2u0qFsC0zhXh5uKYBp0mhBDbhiJsLRSqDuT2ArIXBjrOBVzdjT7F2dtPVe1f/0fP4WifCePfKId3ahel+5kQQtIIinBG5tE1IEcxQPJrS8YrqfubOWeq3M9/HvLH5L/OIzImFoVyZlbuZ8kBTQghJO1ggt+MyqkVwNy6wJ7v4/aJ9WukAD+LiMaI5Scxbv1ZJcDNy+bDpmENKMCEEJIO0BLOqMjSo+gwIOCQNiNWgjrLKeFiYLByP18LCoW9XSZ82roM3mvgSfczIYSkExThjERsDGD3Ij1klV5a13OpVqkS4FVHAzB+w1mER8UifzYXzH67CqoXM34emRBCSOqhOzqjcHYNMLcOEKrN2awo0zZOlFNIWGQMPl51Ch+vPq0EuGGpPNg0vD4FmBBCzAAtYUsnOgLYNhY48rO2fXAO0GxCqk51NegZhvx5HBcDQ2CXCfioRSm837gk7KRBCCEk3aEIWzKPrmtzP989qW03GKWt/5sKNp66gzFrTiM0Mga5szrjx56VUbdEbtP2lxBCiFFQhC2VC38D698HIp4Cmd2BtxYAXi2MPk14VAy+3HQefxz0V+3axd3xY88qyOvmkgadJoQQYgwUYUsjJgr4dxJwYLa2Xagm0HVxqmr/+j98jveXHsPZ28GqPaxpSYxo5gUHe4YCEEKIJUARtiSe3gJW9QVuHda26wwFmk8C7I1PGbntXCBGrTqFkPBo5HR1xIzuldG4dF7T95kQQkiqoQhbCpd9gbUDgbBHgHN2berJsm8YfZqomFh8veUift57XbWrFc2JWT2rwCNH5jToNCGEkNeBImwJa3//mxKX+apAZaDrEsDd0+hT3X4ShqFLj+OE/xPVHtDAE5+0LgNHup8JIcQioQibnUzAvXPalzXe05YfdHA2+iz/+d3HhytO4snzKGRzccB3Xb3Rsnx+03eXEEKIyaAImwuNRpvnWbJddZwH3NgDlOtg9GmiY2Ix499LmPPfVdWuVCi7Kr5Q2N01DTpNCCHElFCE0xvJ8yyu58c3gA6ztUIshRdSIcD3g8MxbNkJHLr+SLX71CmKz9qVhbODcVm0CCGEmAeKcHpz7wyw8ytAEwtU7gkUq5+q0+y/8gDDl5/Ag2eRyOJkj2mdK6G9t4fJu0sIISTtoAinNwW8gRZfaIsvpEKAY2M1mP3fFeWCFo92mfxumNurKornyZom3SWEEJJ2UITTGlHKA3MAr5ZAnlLafXWHpupUD59F4IMVJ7Hn8gPV7l69MD7vUB4ujnQ/E0JIRoQinJaEPQbWDQYubQFO/AEM3Ak4pi5d5JEbjzBs6QkEBofDxdEOX3asiC7VjM+iRQghxHKgCKcVt49piy888QfsnYBaA1O19Ejczwv3XMM32/wQE6tBiTxZMLdXNZTO75Ym3SaEEJJ+UITTwv18eKG2/GBsFJCzGND1V8CjstGnevI8UqWe/PfCfdXuUNkDX3WqiCzO/G8jhBBrgN/mpiQ8GNg4DDi/Xtsu2x7oMAdwyW70qU4GPFG1fyULlpODHSa1L4+eNQsjkyxpIoQQYhVQhE1F4BlgpQ/w6Cpg5wC0/BKoNUi7DtgINBoNft1/A1M2X0BUjAZFc7mq5BsVChov5IQQQiwbirAp3M/HfwO2fAJEhwPZCmlzPxeuYfSpgsOj8Oma09h8JlC121TIj6+7VEI2F+OrKBFCCLF8KMKvQ2Qo8PdHwOnl2rYsQ+r0kzYDlpGcu/NUuZ9vPHwOR/tMGNu2LN6tW4zuZ0IIsWIowq/DoflaAc5kDzQbD9Qdoc0FbaT7efmRAEzceA6R0bEomCMzZr9dBVWK5EyzbhNCCLEMKMKvQ51hwO3jQO33gWL1jP54aEQ0xq0/i3Unbqt20zJ5Mb2bN3K4OqVBZwkhhFgaFOHXwcEJ6PFnqj56+V4IBv95HFfuP4O9XSZ83Ko0BjYoDjs7up8JIcRWoAibgbXHb+GzdWcRFhWDfNmcMatnVdT0NH4emRBCSMaGIpyOhEfF4PO/zmHZ4QDVrl8yN37oURm5sxqfSYsQQkjGhyKcTlx/EIr3/zyOC3eD1dLhEc28MKypl3JFE0IIsU0owunAptN3MXrNaTyLiEauLE6Y2aMK6nvlNne3CCGEmBmKcBoSER2DrzZdwK8Hbqq2zPvO6lkF+bKlrpISIYQQ64IinEYEPHqOoUuP49Stp6o9uHEJjGxRCg72xq0jJoQQYr1QhNMA3/P3MHLlSQSHRyN7ZkfM6O6NpmXymbtbhBBCLAyKsAmJionFd9v88NPua6pduXAOlf2qUE5Xc3eNEEKIBWIRvtE5c+agWLFicHFxQa1atXD48OEkj23cuLHKp5xwa9euHczJ3adh6LngoF6A+9XzxMr/1aEAE0IIsVxLeMWKFfjoo48wf/58JcA//PADWrVqBT8/P+TNm/el49euXYvIyEh9++HDh/D29kbXrl1hLnZfCsIHK07iUWgk3Jwd8G3XSmhdoYDZ+kMIISRjYHZLePr06RgwYAD69u2LcuXKKTF2dXXFokWLEj3e3d0d+fPn12++vr7qeHOIcEysBtP/8YPP4sNKgMt7ZMPfw+tTgAkhhFi+JSwW7bFjxzBmzBj9Pjs7OzRv3hwHDhxI0Tl++eUX9OjRA1myZEn0/YiICLXpCAkJMUHPgfsh4Rix7CQOXHuo2r1qFcH4N8rBxdHeJOcnhBBi/ZjVEn7w4AFiYmKQL1/8yGFpBwZqC9snh8wdnz17Fu+9916Sx0ydOhXZs2fXb2Jtm4KAR2E4cuMRXJ3sMbNHZUzpVJECTAghJGO5o18HsYIrVqyImjVrJnmMWNlPnz7Vb+fPnzfJtasVzYlvulTCxqH10aFyQZOckxBCiG1hVnd07ty5YW9vj3v37sXbL22Z702O0NBQLF++HJMnT072OGdnZ7XpCA4Ohql4q2ohk52LEEKI7WFWS9jJyQnVqlXD9u3b9ftiY2NVu06dOsl+dtWqVWqut3fv3unQU0IIIcQKlyjJ8iQfHx9Ur15duZVliZJYuRItLfTp0wcFCxZUc7sJXdEdO3ZErly5zNRzQgghJIOLcPfu3REUFIQJEyaoYKzKlStj69at+mAtf39/FTFtiKwh3rt3L/755x8z9ZoQQgh5fTJpNBoNbIhbt26hcOHCCAgIQKFCnNMlhBBiPp3J0NHRhBBCSEbG7O7o9EYCv4S7d++auyuEEEKsEJ2+6PQmOWxOhHXLoZJbW0wIIYSYQm+KFCmS7DE2NyccHR2NEydOqMCvhAFfxiIpMCUDlyQAcXNzM1kfrQ2OU8rhWKUcjlXK4Dil/1iJBSwCXKVKFTg4JG/r2pwImxJJ/CGpMCUTV7Zs2czdHYuF45RyOFYph2OVMjhOlj1WDMwihBBCzARFmBBCCDETFOHXQHJST5w4MV5uavIyHKeUw7FKORyrlMFxsuyx4pwwIYQQYiZoCRNCCCFmgiJMCCGEmAmKMCGEEGImKMKpZM6cOShWrBhcXFxQq1YtHD582Nxdskh2796N9u3bw8PDA5kyZcL69evN3SWLREp11qhRQyUIyJs3ryrTKdXCSHzmzZuHSpUqqTWcsknd8S1btpi7WxbPtGnT1N/fBx98YO6uWByTJk1SY2O4lSlTJt2uTxFOBStWrFB1kCWK7vjx4/D29karVq1w//59c3fN4pDa0DI+8tBCkmbXrl0YMmQIDh48CF9fX0RFRaFly5Zq/EgcUpFGBOXYsWM4evQomjZtig4dOuDcuXPm7prFcuTIEfz000/q4YUkTvny5VW+Z90mpXLTDYmOJsZRs2ZNzZAhQ/TtmJgYjYeHh2bq1Klm7ZelI79u69atM3c3MgT3799X47Vr1y5zd8XiyZkzp+bnn382dzcskpCQEI2Xl5fG19dX06hRI82IESPM3SWLY+LEiRpvb2+zXZ+WsJFERkaqp/DmzZvr90kOamkfOHDArH0j1oOkzRPc3d3N3RWLJSYmBsuXL1feAnFLk5cR70q7du3ifV+Rl7l8+bKaMitevDh69eoFf39/pBc2V0XpdXnw4IH645cCEIZI++LFi2brF7EeJPm7zN3Vq1cPFSpUMHd3LI4zZ84o0Q0PD0fWrFmxbt06lXSfxEceUGS6TNzRJGkkpmfJkiUoXbq0ckV//vnnaNCgAc6ePZsuBS8owoRYoPUiXwDpOi+VgZAvy5MnTypvwerVq+Hj46Pm1CnEcQQEBGDEiBEqvkCCR0nStGnTRv9a5s1FlIsWLYqVK1eif//+SGsowkaSO3du2Nvb6+sS65B2/vz5zdYvYh0MHToUf//9t4oqlyAk8jJOTk4oWbKkel2tWjVl6c2cOVMFHxEtMmUmgaJVq1bV7xMPnvxezZ49GxEREep7jLxMjhw5UKpUKVy5cgXpAeeEU/EFIH/427dvj+c+lDbnpUhqkbg1EWBxre7YsQOenp7m7lKGQf7+RFRIHM2aNVNue/EY6Lbq1aur+U55TQFOmmfPnuHq1asoUKAA0gNawqlAlieJC0x+qWvWrIkffvhBBYf07dvX3F2zyF9owyfK69evqy8BCTgqUqSIWftmaS7opUuXYsOGDWoeKjAwUO2X2qaZM2c2d/cshjFjxij3ofzuSAF2GbOdO3di27Zt5u6aRSG/QwnjCbJkyYJcuXIxziABo0aNUrkMxAV9584dtfRUHlJ69uyJ9IAinAq6d++OoKAgTJgwQX1ZVq5cGVu3bn0pWItAreVs0qRJvAcYQR5iJBiCxCWhEBo3bhxv/+LFi/Huu++aqVeWh7hY+/TpowJo5AFF5vBEgFu0aGHurpEMyq1bt5TgPnz4EHny5EH9+vXVen15nR6wihIhhBBiJjgnTAghhJgJijAhhBBiJijChBBCiJmgCBNCCCFmgiJMCCGEmAmKMCGEEGImKMKEEEKImaAIE0IIIWaCIkwIMRmZMmXC+vXrzd0NQjIMFGFCrARJbykimHBr3bq1ubtGCEkC5o4mxIoQwZV804Y4OzubrT+EkOShJUyIFSGCK3WtDbecOXOq98QqlkIRUoVIKjMVL14cq1evjvd5KX/XtGlT9b5U3Bk4cKCqhGXIokWLUL58eXUtKfcmJRgNefDgATp16gRXV1d4eXlh48aN+vceP36syulJcny5hryf8KGBEFuCIkyIDTF+/Hh07twZp06dUmLYo0cPXLhwQb0n5ThbtWqlRPvIkSNYtWoV/v3333giKyIuZRdFnEWwRWBLliwZ7xqff/45unXrhtOnT6Nt27bqOo8ePdJf//z589iyZYu6rpwvd+7c6TwKhFgQUkWJEJLx8fHx0djb22uyZMkSb5syZYp6X/7cBw0aFO8ztWrV0gwePFi9XrBggSZnzpyaZ8+e6d/ftGmTxs7OThMYGKjaHh4ems8++yzJPsg1xo0bp2/LuWTfli1bVLt9+/aavn37mvjOCcm4cE6YECtCajfrahPrcHd317+uU6dOvPekffLkSfVaLFNvb29V/F1HvXr1EBsbCz8/P+XOlqLnzZo1S7YPUuNXh5wrW7Zsqg6wMHjwYGWJHz9+HC1btkTHjh1Rt27d17xrQjIuFGFCrAgRvYTuYVMhc7gpwdHRMV5bxFuEXJD56Js3b2Lz5s3w9fVVgi7u7e+++y5N+kyIpcM5YUJsiIMHD77ULlu2rHotP2WuWOaGdezbtw92dnYoXbo03NzcUKxYMWzfvv21+iBBWT4+Pvjjjz/www8/YMGCBa91PkIyMrSECbEiIiIiEBgYGG+fg4ODPvhJgq2qV6+O+vXr488//8Thw4fxyy+/qPckgGrixIlKICdNmoSgoCAMGzYM77zzDvLly6eOkf2DBg1C3rx5lVUbEhKihFqOSwkTJkxAtWrVVHS19PXvv//WPwQQYotQhAmxIrZu3aqWDRkiVuzFixf1kcvLly/H+++/r45btmwZypUrp96TJUXbtm3DiBEjUKNGDdWW+dvp06frzyUCHR4ejhkzZmDUqFFK3Lt06ZLi/jk5OWHMmDG4ceOGcm83aNBA9YcQWyWTRGeZuxOEkLRH5mbXrVungqEIIZYB54QJIYQQM0ERJoQQQswE54QJsRE480SI5UFLmBBCCDETFGFCCCHETFCECSGEEDNBESaEEELMBEWYEEIIMRMUYUIIIcRMUIQJIYQQM0ERJoQQQswERZgQQgiBefg/prMItlIYrpkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 500x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "epochs_tensor = torch.linspace(0, num_epochs, len(train_accs))\n",
    "examples_seen_tensor = torch.linspace(0, examples_seen, len(train_accs))\n",
    "\n",
    "plot_values(epochs_tensor, examples_seen_tensor, train_accs, val_accs, label=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90aba699-21bc-42de-a69c-99f370bb0363",
   "metadata": {},
   "source": [
    "- Based on the accuracy plot above, we can see that the model achieves a relatively high training and validation accuracy after epochs 4 and 5\n",
    "- However, we have to keep in mind that we specified `eval_iter=5` in the training function earlier, which means that we only estimated the training and validation set performances\n",
    "- We can compute the training, validation, and test set performances over the complete dataset as follows below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "UHWaJFrjY0zW",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UHWaJFrjY0zW",
    "outputId": "e111e6e6-b147-4159-eb9d-19d4e809ed34"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training accuracy: 97.21%\n",
      "Validation accuracy: 97.32%\n",
      "Test accuracy: 95.67%\n"
     ]
    }
   ],
   "source": [
    "train_accuracy = calc_accuracy_loader(train_loader, model, device)\n",
    "val_accuracy = calc_accuracy_loader(val_loader, model, device)\n",
    "test_accuracy = calc_accuracy_loader(test_loader, model, device)\n",
    "\n",
    "print(f\"Training accuracy: {train_accuracy*100:.2f}%\")\n",
    "print(f\"Validation accuracy: {val_accuracy*100:.2f}%\")\n",
    "print(f\"Test accuracy: {test_accuracy*100:.2f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6882649f-dc7b-401f-84d2-024ff79c74a1",
   "metadata": {},
   "source": [
    "- We can see that the training and validation set performances are practically identical\n",
    "- However, based on the slightly lower test set performance, we can see that the model overfits the training data to a very small degree, as well as the validation data that has been used for tweaking some of the hyperparameters, such as the learning rate\n",
    "- This is normal, however, and this gap could potentially be further reduced by increasing the model's dropout rate (`drop_rate`) or the `weight_decay` in the optimizer setting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a74d9ad7-3ec1-450e-8c9f-4fc46d3d5bb0",
   "metadata": {},
   "source": [
    "## 6.8 Using the LLM as a spam classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72ebcfa2-479e-408b-9cf0-7421f6144855",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch06_compressed/overview-4.webp\" width=500px>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd5408e6-83e4-4e5a-8503-c2fba6073f31",
   "metadata": {},
   "source": [
    "- Finally, let's use the finetuned GPT model in action\n",
    "- The `classify_review` function below implements the data preprocessing steps similar to the `SpamDataset` we implemented earlier\n",
    "- Then, the function returns the predicted integer class label from the model and returns the corresponding class name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "aHdn6xvL-IW5",
   "metadata": {
    "id": "aHdn6xvL-IW5"
   },
   "outputs": [],
   "source": [
    "def classify_review(text, model, tokenizer, device, max_length=None, pad_token_id=50256):\n",
    "    model.eval()\n",
    "\n",
    "    # Prepare inputs to the model\n",
    "    input_ids = tokenizer.encode(text)\n",
    "    supported_context_length = model.pos_emb.weight.shape[0]\n",
    "    # Note: In the book, this was originally written as pos_emb.weight.shape[1] by mistake\n",
    "    # It didn't break the code but would have caused unnecessary truncation (to 768 instead of 1024)\n",
    "\n",
    "    # Truncate sequences if they too long\n",
    "    input_ids = input_ids[:min(max_length, supported_context_length)]\n",
    "\n",
    "    # Pad sequences to the longest sequence\n",
    "    input_ids += [pad_token_id] * (max_length - len(input_ids))\n",
    "    input_tensor = torch.tensor(input_ids, device=device).unsqueeze(0) # add batch dimension\n",
    "\n",
    "    # Model inference\n",
    "    with torch.no_grad():\n",
    "        logits = model(input_tensor)[:, -1, :]  # Logits of the last output token\n",
    "    predicted_label = torch.argmax(logits, dim=-1).item()\n",
    "\n",
    "    # Return the classified result\n",
    "    return \"spam\" if predicted_label == 1 else \"not spam\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f29682d8-a899-4d9b-b973-f8d5ec68172c",
   "metadata": {},
   "source": [
    "- Let's try it out on a few examples below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "apU_pf51AWSV",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "apU_pf51AWSV",
    "outputId": "d0fde0a5-e7a3-4dbe-d9c5-0567dbab7e62"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam\n"
     ]
    }
   ],
   "source": [
    "text_1 = (\n",
    "    \"You are a winner you have been specially\"\n",
    "    \" selected to receive $1000 cash or a $2000 award.\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_1, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "1g5VTOo_Ajs5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1g5VTOo_Ajs5",
    "outputId": "659b08eb-b6a9-4a8a-9af7-d94c757e93c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "not spam\n"
     ]
    }
   ],
   "source": [
    "text_2 = (\n",
    "    \"Hey, just wanted to check if we're still on\"\n",
    "    \" for dinner tonight? Let me know!\"\n",
    ")\n",
    "\n",
    "print(classify_review(\n",
    "    text_2, model, tokenizer, device, max_length=train_dataset.max_length\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf736e39-0d47-40c1-8d18-1f716cf7a81e",
   "metadata": {},
   "source": [
    "- Finally, let's save the model in case we want to reuse the model later without having to train it again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "mYnX-gI1CfQY",
   "metadata": {
    "id": "mYnX-gI1CfQY"
   },
   "outputs": [],
   "source": [
    "torch.save(model.state_dict(), \"review_classifier.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba78cf7c-6b80-4f71-a50e-3ccc73839af6",
   "metadata": {},
   "source": [
    "- Then, in a new session, we could load the model as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "cc4e68a5-d492-493b-87ef-45c475f353f5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_state_dict = torch.load(\"review_classifier.pth\", map_location=device, weights_only=True)\n",
    "model.load_state_dict(model_state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4",
   "metadata": {
    "id": "5b70ac71-234f-4eeb-b33d-c62726d50cd4"
   },
   "source": [
    "## Summary and takeaways"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dafdc910-d616-47ab-aa85-f90c6e7ed80e",
   "metadata": {},
   "source": [
    "- See the [./gpt_class_finetune.py](./gpt_class_finetune.py) script, a self-contained script for classification finetuning\n",
    "- You can find the exercise solutions in [./exercise-solutions.ipynb](./exercise-solutions.ipynb)\n",
    "- In addition, interested readers can find an introduction to parameter-efficient training with low-rank adaptation (LoRA) in [appendix E](../../appendix-E)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "V100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}